Too Much Information: Taming the UAV Data Explosion
TMI! That’s what US military commanders are saying about the explosion of data being collected and processed (or not) by thousands of UAVs. Because UAVs provide valuable information, the US military has been asking for more and more of them to be sent to Iraq and Afghanistan. Be careful what you wish for. You might just get it.
All that information needs to be processed so that it is useful for the commanders in the field. Software that can archive and retrieve information when needed and display it on a user-friendly interface is available in the commercial sphere. But the technology is not being developed and deployed fast enough in the military sphere.
As Lt. Gen. Deptula, USAF deputy chief of staff for intelligence, surveillance, and reconnaissance, said recently, “We are going to find ourselves in the not too distant future swimming in sensors and drowning in data.” This free-to-view DID Spotlight article examines the problem of the UAV data explosion, some possible solutions, and future challenges.
Proliferation of UAV Data: The Stats
(click to view full)
UAVs have played a crucial role in the US military’s wars in Iraq and Afghanistan. With all the useful information that UAVs provide comes the problem of how to sort through it all and find actionable data. The scope of the problem is apparent. There are thousands of UAVs deployed in Iraq and Afghanistan. In 2009, the USA’s UAVs alone generated 24 years’ worth of video if watched continuously. New UAV models are expected to produce 30 times as much information in 2011.
The USAF flies 39 orbits over Afghanistan and Iraq every day, and the service expects that number to increase to 50 by 2011. An orbit is a 24-hour combat flight by a single UAV. The USAF uses two shifts of operators per orbit for its high-flying, long-endurance UAVs (MQ-1 Predator, MQ-9 Reaper and RQ-4 Global Hawk), so increasing the number of orbits to 50 is expected to double the requirement for operators.
New technological developments are expected to compound the data explosion problem. For example, the USAF is planning to add a wide area airborne surveillance sensor to its MQ-9 Reaper and, eventually, its other UAVs. This system is expected to add 50 video streams per sensor within a few years [PDF]. The USAF is aiming to have a version deployed on the MQ-9 by the summer of 2010. Made by Sierra Nevada, the Gorgon Stare sensor system is named after the three sisters of Greek mythology who had a gaze that would turn anyone who beheld it to stone.
The USAF has increased the number of UAVs over the last 2 years by 330%. It also plans to shift 3,600 manpower billets to analyze data streaming from UAVs. It is also doubling the number of ISR liaison offers assigned to ground forces to assist with integration of UAV data collection and exploitation.
The Deluge: Breaking Down Barriers
(click to view larger)
In addition to the proliferation of UAVs and the exponential expansion of sensor capability, the Pentagon is engaged in an effort to break down proprietary barriers between UAV systems. This effort is intended to allow commanders on the battlefield as well as analysts back in CONUS to access important information no matter which system collects it.
For example, the popular MQ-1 Predator UAV system comes in a package with 4 vehicles, 1 ground control solution (GCS), and a data link suite that consists of UHF and VHF radio relay links, a C-band line-of-sight data link, and Ku-band satellite data links.
Unfortunately, the Predator GCS can only control and process information from Predator vehicles. The RQ-4 Global Hawk GCS controls and processes information from Global Hawks. And other UAVs use their own proprietary GCS systems.
In 2008, the Pentagon launched an effort to develop and demonstrate a common, open GCS architecture supporting everything from MQ-8 Fire Scout unmanned helicopter to long-range Global Hawk. The intent is to end the packaging of UAVs and GCS by manufacturers as 1 proprietary system. The Pentagon wants GCSs to be able to control multiple types of UAVs and share information across platforms. See “It’s Better to Share: Breaking Down UAV GCS Barriers” for more information.
While breaking down barriers sounds like a great idea, it adds to the data explosion problem. The good news is that there is a lot of information out there; the bad news is it’s hard to find the right information. If all of the UAV systems can share data, who or what is going to sort out the data so that useful information can emerge out of the raw feeds?
The Madden Effect: Video Tagging and Retrieval
(click to view full)
One solution to data explosion is to tag the data, store it, and retrieve it when needed. An application of this technique is used in coverage of NFL football games.
A new $500 million computer system being installed by the US Air Force will enable it to use TV broadcast techniques and send out highlight reels of the greatest battlefield moments, i.e., the most important video feeds for the commander. The video is tagged with time, geographic coordinates, and other essential data.
Not to be outdone, the US Navy is climbing into networks’ broadcasts trucks located outside of football stadiums to get a first-hand view of the technology. Cmdr. Joseph Smith, a Navy officer assigned to the National Geospatial-Intelligence Agency, told the New York Times that he and other officials learned a lot from watching the technology in action (besides the scores of their favorite teams).
“There are these three guys who sit in the back of an ESPN or Fox Sports van, and every time Tom Brady comes on the screen, they tap a button so that Tom Brady is marked.” Then, to call up the highlights later, he said, “they just type in: ‘Tom Brady, touchdown pass.’ ” This retrieves the video they need at that moment.
The US military would like to implement a similar system for its UAV videos. However, tagging can be labor intensive. In addition, the right tags need to be used so that the person searching for the video in a time sensitive situation doesn’t get frustrated by not being able to find the video he needs. If the tags don’t make sense to the commander searching for the video, the technology will be useless.
Like John Madden, the US military is using telestrators, such as the one on the Remotely Operated Video Enhanced Receiver (ROVER), via systems like L-3’s companion VideoScout. This technology is similar to that used by Madden to mark and analyze football plays on the video screen. The telestrator enables US military commanders in the field to circle images of vehicles or individuals they want the UAVs to track.
Data Fusion: Telling a Story
(click to view full)
Data fusion involves the use of techniques and software that combine data from multiple sources and analyze that data to make it useful for the end user.
Data fusion can involve the combining of data (such as UAV video) with a geographical information system (GIS), which adds location and time data to the images gathered by UAVs. To accomplish this, the raw data has to be combined with metadata, which is information about the data that enables the data to be combined with a GIS.
According to the Belgian Royal Military Academy, data fusion can provide the following military benefits:
An example of a basic data fusion system is the Link 16 standard embedded in the MIDS-LVTs carried by fighters. A target seen and identified by any fighter jet in a formation, or any linked ground station or ship, is seen and identified for all.
Data fusion is a subset of information fusion, which is such an important issue that the US Navy has set up a center to tackle it – the NAVAIR Information Fusion Center. It is run by NAVAIR’s Naval Air Warfare Center Weapons Division (NAWCWD).
Robert Reddit, NAWCWD’s Director of Information Fusion, describes the purpose of information fusion this way:
“Information fusion is the science behind Critical Infrastructure Protection, Homeland Security, ForceNet and Maritime Domain Awareness. Currently there are hundreds of rooms with hundreds of individuals all tracking tens of thousands of aircraft, maritime vessels, ground vehicles and individuals with everyone looking for the needle in the haystack. Information fusion reduces the rooms and individuals and finds the needle, pulls it out of the hay and puts it where it won’t hurt anybody.”
To get the ball rolling, in 2009 the center awarded a $95 million contract to General Dynamics to support the center’s work. The firm is helping with research and development, integration and testing, continual advancement and operation of the Information Fusion Center; training for newly developed software, hardware and other products; and independent verification and validation of sensors and systems relating to critical infrastructure protection and force protection.
To support the center, General Dynamics is using its Quarterback Information Fusion capability and Story Maker fusion system. Story Maker provides an overall reconnaissance architecture that stresses multi-service/ multi-platform utility, interoperability among existing and planned airborne reconnaissance components, timely dissemination of intelligence information to operational forces, enhanced combat identification capability, and high payoff multi-use technology. Through the application of algorithms, Story Maker fuses and reasons with collected data, building evidence for track identification. Story Maker enables identification of 10 times as many tracks with 98.6% accuracy.
While Story Maker was originally developed to integrate information collected by the Navy’s EP-3E ISR aircraft, the technology used by the system can be applied to data fusion for a range of platforms.
Googling the Enemy: Intelligent Search
Intelligent search is another tool that can be used to make UAV data more accessible. Probably the most famous and widely used intelligent search engine is Google.
Google uses a patented algorithm called PageRank that ranks pages that match a given word search string. The algorithm, which is a list of well-defined instructions for completing a task, analyzes human-generated links, assuming that the web pages linked from many important pages are themselves likely to be important. This produces results that tend to be in line with human concepts of importance. The founders of Google, Sergey Brin and Lawrence Page, laid out their Google vision as researchers at Stanford University.
Of course, searching for UAV video is a lot different than searching the web for a favorite recipe, but the technology is similar. Algorithms can be used to develop intelligence search engines for UAV video.
To help users retrieve information, one promising method, called natural language processing [pdf], discovers user preferences and needs by either extracting knowledge from what users are looking for or interactively generating explanatory requests to focus users on the information they are interested in. In particular, using techniques for automatically generating natural language sentences allows the system to produce a useful dialog with the user and guide preferences.
The use of natural language to improve the performance of the intelligent search engines is one aspect of artificial intelligence (AI). Other AI technologies include object recognition and statistical machine learning.
Developer of the Powerset AI search engine Barney Pell describes the use of AI for intelligent search in the following way:
“Search engines try to train us to become good keyword searchers. We dumb down our intelligence so it will be natural for the computer…The big shift that will happen in society is that instead of moving human expressions and interactions into what’s easy for the computer, we’ll move computers’ abilities to handle expressions that are natural for the human.”
An intelligent search engine being developed for photos is called Riya, which looks “inside” photos to extract information about their qualities using AI. Riya uses algorithms to calculate a photo’s shape densities, patterns and textures and extract this information into a visual signature. Each photo is represented by 6,000 numbers; Riya uses AI to match one visual signature to another.
This technology could be applied to UAV photos and videos so that related videos from different platforms or from a vast archive could be matched within seconds. This would enable commanders to search and retrieve valuable data in real-time on the battlefield.
Another application of AI in intelligence search is facial recognition. The USAF UAV Battlelab is working on software that can pick out face patterns in UAV video. The software is based on that used by the Nevada gaming industry to pick out problem gamblers.
The UAV Battlelab tested the software using the photo of a USAF captain’s face. The staff launched a Pointer UAV, which began beeping at a clump of trees 2 miles away.
As the UAV approached the trees, the operators could detect a vehicle underneath the trees. As the UAV got closer, the operators were able to detect the captain sitting in the vehicle underneath the trees. Yet the UAV had indicated the captain was there from 2 miles away through foliage and a vehicle windshield using the facial recognition software.
Blind Men and an Elephant: Sensor Fusion
(click to view full)
One area of UAV autonomy that could help with the data overload is sensor fusion.
To better understand the concept of sensor fusion, it might help to recall the story of the blind men and the elephant. As the story goes, a group of blind men examine an elephant and try to determine what it is. Each man grabs a different part of the elephant. One grabs the tail and believes the elephant is like a rope. Another grabs the trunk and believes that the elephant is like a snake. Another grabs the leg and believes the elephant is like a tree. Each has information about a part of the elephant, but no one has the whole picture.
UAVs are like blind men. They are able to collect information about a particular narrow area they are examining. UAV operators compare looking through a UAV camera to looking through a soda straw. The video is partial and needs to be combined with data from other UAVs to create a complete and accurate picture. That is the goal of sensor fusion.
One application is to use a multi-sensor fusion algorithm that can bring together information from multiple UAVs about a particular target of interest in a unified display. Without sensor fusion, each UAV platform would track the target separately, which creates redundant capability and leads to less than optimal tracking results. Together, the UAVs can provide a full and accurate picture based on which the commander can take action.
In fact the USAF awarded a contract on April 23/10 to Aurora Flight Sciences Corp. for research on collaborative sensor fusion and management of multiple UAVs.
Working with researchers from MIT, the Office of Naval Research, the Air Force Research Lab, and the Rome Labs, Aurora developed autonomy technologies for multi-unmanned vehicle coordination, sensor management and sensor fusion, to enable coordinated search, track and prosecute missions with various unmanned vehicles carrying a variety of sensors.
Aurora’s multi-vehicle system can function either fully autonomously, in a distributed way, or with humans in the loop at the command level (to supervise the unmanned vehicles and make strategic decisions with the aid of a centralized planning interface) and/or at the sensor level (to assist with target identification and tracking).
Future Shock
(click to view larger)
What does the future hold for the UAV data explosion? It seems that the US military will continue to demand more eyes in the sky to detect potential threats. And the need to sort through and integrate that information in a way that enables commanders in the field to save US lives and defeat the enemy will become more acute.
The industry will need to develop smart technologies that automate the process of archiving, tagging, retrieving, managing, and displaying UAV videos and other info gathered by increasingly sophisticated sensors. Machine-to-machine interfaces will need to become much more sophisticated to handle the data deluge.
Of course, there are other sensors beside UAVs collecting information about the battlespace. How to bring those thousands of ground, air and sea-based sensors, as well as human intelligence, together in a usable format will be a huge challenge for the military.
Perhaps the US military could take a cue from the commercial sector, which is facing a similar problem of data overload. The world is expected to create 1,200 exabytes (billion gigabytes) of information in 2010.
To tackle this data tsunami, industry is turning to such technologies as business analytics – performing statistical operations for forecasting or uncovering correlations – machine-learning, and visualization software; these are all technologies that have military applications.
The need is urgent, the solution complex. The coming years will see if the US military will be able to adopt UAV data management solutions used in the commercial world, while retaining the ability to protect information that needs to be protected. US soldiers’ lives depend on it.
Additional Readings and Sources
- Defense Update – Unmanned Systems (UAV) Persistent Surveillance
- Wikipedia – Unmanned aerial vehicle
- DID – Walking Back the Cat: The US Army’s Constant Hawk. They key component of “change detection.”
- DID – DARPA’s VIRAT video search engine. Instead of rewind capability, think automated play-by-play…
- L-3 Communications – VideoScout. One of several solutions that help militaries capture, archive, analyze and retrieve video and metadata.
- Defense Systems (Sep 09/11) – Better image tagging improves warfighter situational awareness
- New York Times (Jan 16/11) – In New Military, Data Overload Can Be Deadly
- USAF (Feb 9/10) – USAF Posture Statement 2010
- The Economist (Feb 25/10) – The data deluge: Businesses, governments and society are only starting to tap its vast potential
- Defense Systems (Feb 4/10) – Sensory overload: Military is dealing with a data deluge
- DefenseTech.org (Jan 12/10) – The UAV Data Firehose
- New York Times (Jan 10/10) – Military Is Awash in Data From Drones
- National Defense (January 2010) – Military ‘Swimming in Sensors and Drowning in Data’
- Signal Online (Dec 2/09) – Surveillance Data Fusion Defines Future Army Systems
- Airforce-Technology.com (Sept 11/09) – The New Breed of Pilot
- Military & Aerospace Electronics (July 2009) – The UAV video problem: Using streaming video with unmanned aerial vehicles
- COTS Journal (February 2009) – Large UAVs Ramp Up Sensor Processing Capabilities
- Flight International (Jan 28/09) – USAF to unleash ‘Gorgan Stare’ sensor in 2010
- DID (Jan 26/09) – NAVAIR’s Information Fusion Center
- Batten College of Engineering and Technology (2009) – UAV-based Multi-Sensor Data Fusion for Time-Critical Disaster Response [pdf]
- Australian Centre for Field Robotics (2006) – Information Fusion and Control for Multiple UAVs
- Lockheed Martin (2005) – Data Fusion for the Apache Longbow: Implementation and Experiences [pdf]
- Aviation Week (April 13/04) – UAV Battlelab Experiments With Feature Recognition Software
