A Brief History of Predicting the Future

Where did OSINT come from, and where can it take us?

Open Source Intelligence (OSINT) is a discipline in constant transformation. Although some form of intelligence has always been part of the decision maker's toolkit, it has constantly needed to adapt to match the dynamics and demands of society. Over time, it has evolved from a nebulous concept into a fully-fledged industry, relevant and accessible to individuals and businesses across the spectrum - but the challenges of properly exploiting it have evolved in tandem. 

Technology has both driven revolutions in intelligence, and at the same time provided the means to deal with them. History reveals just how important it is for leaders to understand and master this relationship, as we move ever further into an information-centric age.

OSINT at War: The Power of Everyday Information

Well into the 19th century, Intelligence with a capital “I” relied almost entirely on human intelligence (HUMINT), supplemented with some OSINT, and what we might recognize as rudimentary signals intelligence or SIGINT (the interception of enemy communications).

Early 20th century advances in radio technology saw increasing reliance on SIGINT over HUMINT, in the name of efficiency and lower operational risk. With the emergence of global ideological threats and weapons of mass destruction, the role of Intelligence as a whole became ever more central to international relations. It was in the fires of the two world wars that the first formal OSINT capabilities were forged, as both the US and UK moved towards institutionalizing their intelligence services. 

The Foreign Broadcast Information Service (FBIS), and the Research and Analysis Branch of the newly created Office of Strategic Services (the precursor to the CIA), have been variously noted as the United States’ first formal OSINT capabilities. 

The R&A’s purpose was to test OSS chief William J. Donovan's hypothesis that the “answers to many intelligence problems could be found in libraries, newspapers, and the filing cabinets of government and industry.” It was where academia met the war effort: applying linguistic skills and analytical training to translate and process data from across government departments, and from the open source environment. The FBIS, meanwhile, was established to monitor, translate, and analyze foreign media reporting: in particular, propaganda generated by axis powers.

The primary challenge for OSINT at this time was collection. In a pre-internet age, open source material had to be physically collected via international embassies or bureaus, inevitably limiting the breadth and scope of coverage.

OSINT in Business: The Commercialization of Intelligence

Public-sector OSINT would remain sleepy for a few decades following the Second World War.  But this was when private intelligence saw its first green shoots. Kroll, the company broadly credited with legitimizing the corporate intelligence industry, was founded in 1973 and expanded into the financial sector in the 1980s. They were equipped and staffed for business, financial crime, and industrial espionage. With neither the formal powers nor the scope for the clandestine activity of government agencies, their intelligence meat was the open source environment, and the human landscape. What they needed wasn’t secret agents, but a dynamic team of “former prosecutors, accountants, investigative journalists, academics”: resourceful researchers who knew how to comb publicly accessible sources for seams of gold. 

In the 1990s, as the business world grew increasingly globalized and technological capabilities developed apace, private intelligence firms proliferated. At the First International Symposium on Open Source in 1992, FBIS Deputy Director J Niles Riddel remarked on a “general awakening to the availability, affordability, and above all, the utility of open source information for supporting analysis and decision making in all sectors of society, both public and private.” His words foreshadowed a new era of OSINT, in which Intelligence would become an ever-increasingly hot commodity.

The OSINT Revolution: For the People, By the People

The first decade of the 21st century opened vast new possibilities for OSINT, as well as new challenges. User-generated content, participatory culture, and interoperability multiplied and redefined sources, methods, and uses. Data that previously had little to no physical, collectible manifestation became suddenly and increasingly prolific: from casual comments and conversations to personal and popular sentiment, social and professional networks, real-time photos and videos of events, and even an individual’s location, the pattern of life or pattern of travel. New platforms, from social media to satellite imagery, generated increasing overlap with sister disciplines including HUMINT, SIGINT, and even geospatial intelligence (GEOINT).

Many contribute the defining moment of this shift back to 2009, when Iran’s “Green Revolution” made international waves. It was the birthplace of so-called “citizen journalism”, as millions of young Iranians co-ordinated to capture the world’s attention via Facebook, Twitter, Youtube, blogs, and other websites, in an unprecedented demonstration of the power of the Open Source internet. 

As ever, with new opportunities have come new challenges. Innovative methods and novel sources require greater technical expertise, whilst greater volumes of information make collection and analysis more time and labor intensive. The user-generated nature of much digital content enhances risks of bias and selection, requiring subtlety of analysis to match speed of collection. And increasingly covert collection methods are required to access information held on parts of the internet that were once public; much of Facebook, for example, is not searchable or accessible without a profile, whilst investigators using a non-anonymous profile risk exposure through algorithmic quirks. Whilst hundreds of tools have been developed to try and manage these challenges, not all technology is created equal. This is becoming increasingly apparent as the internet continues to evolve and grow exponentially: leaving many individuals and businesses on the back foot.

OSINT for the Future: Intelligent Intelligence

Manual intelligence methods are already outmatched by the sheer size, speed, and agility of the internet: and arguably, so are the tools that once made digital OSINT manageable. It’s been said that we are now on the verge of a transition to “Web 3.0". The internet and the information environment are growing and evolving at an unprecedented rate, and Intelligence is now an integral part of everyday life - from business, finance, and litigation, through to healthcare and dating. 

Living in an “information age” means living with misinformation, disinformation, information warfare, and subconsciously making countless intelligence-based decisions every day. Where the challenge was once to collect sufficient data, now the risk is being overwhelmed with data and under-equipped to use it. Investigators need to be able to scour through reams of information at lightning speed, without putting themselves at risk through high-risk overt strategies, but equally without crossing ethical or legal boundaries. They need to be able to identify, collect, and manage massive amounts of relevant data, analyse that data with rigor and subtlety, and process actionable intelligence reports for decision-makers to act on. And they need to do all of this fast enough to match the speed at which information is generated, and decisions are made, in a modern world.

Research by RAND in 2018 assessed that, like Web 2.0 before it, Web 3.0 signals a move towards a new phase of OSINT. The increasing role of technologies like artificial intelligence, machine learning, and natural language processing can help us harness the power of this evolving environment. They can help us make conscious decisions based on quality intelligence, rather than subconscious decisions led by suboptimal intelligence. This means not just faster, better collection of data, but subtler and more creative analyses. Most often, the difference between useless information and actionable intelligence lies in the ability to think outside the box: Jason Matheny, for example, has cited the example of aggregated data from apps like Open Table as one of the earliest indicators for disease outbreaks: “if people are sick, they are going to cancel their dinner date.”

The ultimate goal for modern-day OSINT is to match the speed of information generation with both speeds of collection and rigor of analysis, disseminating actionable intelligence to decision-makers in real-time. Properly used, technology appropriate to “OSINT 3.0” can move us realistically closer towards this ideal.