As Defense, Intelligence Agencies Drown in Data, Technology Comes to the Rescue 

November 2014   http://www.nationaldefensemagazine.org/archive/2014/November/Pages/AsDefenseIntelligenceAgenciesDrowninData,TechnologyComestotheRescue.aspx

By Sandra I. Erwin 

Director of National Intelligence James R. Clapper has asked the government’s tech gurus and the private sector to “help us find the needles without having the haystacks.”

Clapper’s clarion call comes at a time of unprecedented demand for data-intensive products and services at all levels of the U.S. national security apparatus. The task of filtering and sorting through massive loads of data is only going to get bigger as the military and intelligence agencies collect more information than they can handle. There are more drones and satellites collecting video and imagery than ever before, and human analysts desperately need automated tools to find those needles in ever-expanding haystacks.

“Our next big investment is big data,” says Dawn Meyerriecks, deputy director of the CIA’s directorate of science and technology. The challenge for data scientists is “figuring out how we deal with high volume intelligence.”

Government agencies find that software tools that can parse huge loads of information into actionable information are becoming increasingly more sophisticated, but there are still many gaps to be filled.

As the United States steps up the fight against elusive extremist groups, the traditional methods of finding and tracking targets are inadequate. The amount of data being collected has made it nearly impossible to track and identify suspicious activities and potential security threats solely through human analytical processes.

The intelligence community sees its future in “activity based intelligence,” which is computer-assisted problem solving to help understand how enemy networks operate by following their movements and financial transactions.

The government’s gargantuan appetite for data has spurred an arms race within the tech industry. Much of the innovation these days comes from Silicon Valley, where there is a burgeoning crop of firms that are jumping in to fill big data needs.

“When the agencies first saw our software, they didn’t know software could do what our software did,” says Sean Varah, CEO of MotionDSP. The company’s image processing software initially was created to clean up grainy cell phone videos from the pre-iPhone days. U.S. military and intelligence analysts now use it in the war against the Islamic State. Agencies have rooms full of people who manually, frame by frame, clean up images that may be hard to see, or are clouded by bad weather or smoke. That typically takes weeks, says Varah, whereas the software improves the quality of the video in real time, he adds. “Operators are good. They can see things, but with our technology they can see it a lot faster.”

These technologies fall into the category of “computer vision,” a rapidly growing field that focuses on acquiring, processing, analyzing and understanding images in order to produce actionable information. This technology will explode in the coming years as sources of imagery multiply. Commercial companies like Google’s Skybox Imaging are going to make it easier and cheaper to obtain sophisticated satellite imagery that is now only available to governments.

“What do you do with all that imagery?” Varah asks. “You have to use computer vision technology to extract information.”

The good news about cutting-edge Silicon Valley info-tech products is that they are all privately funded, and the government can acquire them at a fraction of the cost of government-developed systems. “The government should evaluate the best in breed before they pay billions of dollars for contractors to write code from scratch,” Varah says.

The tech revolution is only just beginning. Giants like Google, Amazon, Facebook, Microsoft and Adobe are pouring billions of dollars into computer vision and another emerging discipline called “deep learning.”

The goal is to teach a computer to see the way a human sees. Google and Amazon are betting big on drones, but they know the industry won’t take off until these drones can “see” and avoid hitting people when they deliver packages. The technology that lets a Google car drive around without a driver is also computer vision. “Private investment in computer vision is going to start pouring out new products,” Varah says.

Deep learning is another term for the use of artificial intelligence to solve problems and to find patterns in huge reams of imagery. “The technology in image recognition has gone from laughably bad to super human good,” says Paul Cohen, program manager at the Defense Advanced Research Projects Agency. Most of the big data technology today is insufficient to tackle increasingly complex challenges, he says. At DARPA, Cohen oversees a program called “big mechanism,” which he describes as a “poke in the eye” to big data. “It’s based on the distinction between crunching numbers and understanding what the data is telling you.”

A big mechanism is a large, explanatory model of complicated systems. While the collection of big data is largely automated, the creation of big mechanisms remains a human endeavor that is made more difficult by the fragmentation and distribution of knowledge. DARPA believes that if the creation of models can be automated, it could change how science is done. The program now focuses on cancer biology but the overarching goal of the program is to develop technologies for a new kind of science that is based on models, not on raw data. The implications for military-focused applications are huge.

Another major tech battle that defense and security agencies are fighting is the flood of data generated by social media. What used to be innocuous social media platforms are now bursting with potential threat intelligence that is of great value to the U.S. government.

The data in social media is very unstructured and the government needs tools to make sense of it, says Peg Grayson, president of MTN Government. The company’s social media predictive analytic tool integrates tweets and posts from other sites in real time, using a mathematical algorithm to scan keywords, sources and pictures.

The software integrates media feeds in a way that might provide some predictive information — such as the location of potential terrorists — when it’s combined with human intelligence, says Grayson. “This saves thousands of people’s time.”

The data deluge, meanwhile, is creating a market for storage and archiving. Analysts who collect imagery and video from drones and satellites in many cases are required to keep the data for years. These are exabytes of information that they may have to retrieve when military commanders, for example, want to compare drone feeds of a particular area over a certain period.

The government’s data storage needs are astronomical, says Brian Houston, vice president of engineering Hitachi Data Systems Federal, a company that develops optical storage systems. One exabyte could hold a hundred thousand times the printed material, or 500 to 3,000 times the content of the Library of Congress.

Photo: Example of software that increases the fidelity of video images (MotionDSP)