The Spy Who Tweeted Me: U.S. Spooks Launch Effort to Mine Social Media

A research arm of the U.S. intelligence community says it want to sweep up public data on everything from Twitter to public webcams in the hopes of predicting the future.

The project is the brainchild of the Intelligence Advanced Research Projects Activity, or IARPA, a relatively new part of the spy community that is supposed to help investigate breakthrough technologies.   While other projects exist for predicting political events, the Open Source Indicators program would be perhaps the first that mines data from social media websites.

The idea is to use automated analysis to sift through the deluge of publicly available data to help predict significant societal events, like a popular revolution. The nascent Open Source Indicators project is just the latest move by the national security community to come to grips with the flood of information now available on social media.

The science underlying the project is that the notion that there may be early indicators of major social upheavals. “Some of these changes may be indirectly observable from publicly available data, such as web search queries, blogs, micro-blogs, internet traffic, financial markets, traffic webcams, Wikipedia edits, and many others,” the announcement, published August 25, says. “Published research has found that some of these data sources are individually useful in the early detection of events such as disease outbreaks, political crises, and macroeconomic trends.”

Indeed, social media sites, such as Twitter and Facebook, have garnered major attention during recent events, such as the Arab Spring, and have been credited with helping to organize protesters and even foment revolution.  Authoritarian governments trying to hold on to power also noted this trend, and attempted at times to shut down access to those sites in the hopes of stymieing efforts to organize protests.

The idea of the U.S. intelligence community culling data from social media is still a new one, and is likely to raise a number of questions. For example, what constitutes public data?

IARPA, for its part, defines public data as “lawfully obtained data available to any member of the general public, to include by purchase, subscription or registration.” That raises a host of questions, for example, whether the intelligence community could register a fake profile on Facebook, in order to “friend” people and obtain more information.

For those who fear the all-seeing surveillance state, IARPA says there are some things the program won’t do: it won’t be used to predict events in the United States, nor will it be used to track specific individuals.