Crowdsourcing U.S. National Security: In Progress Review

Those familiar with this blog have likely participated in some of my crowdsourcing experiments.  You may have needed to fall asleep one night and decided to read an article I co-authored on its application in national security issues.  Well, this week, the LA Times published an in-progress update on a recent IARPA study to see if groups of experts can accurately predict future national security events.

In “U.S. intelligence tests crowd-sourcing against its experts“, one finds Dr. Philip Tetlock, the Godfather of Expert Political Judgment, taking on a MITRE team to see if crowds can do better than internal U.S. government experts in predicting future national security events.

The study, known as Aggregative Contingent Estimation, is designed to see whether the 17 agencies in the U.S. intelligence community can aggregate the judgment of its thousands of analysts — rather than rely on the expertise of just a few — to issue more accurate warnings to policy makers before and during major global events.

Tetlock notes that his group of experts:

“In year one, we beat the unweighted average by about 57%, which was big,” he said. A control group, run by Mitre Corp., averages scores without giving weight to participants who tally the best results.

Mark Lowenthal, a veteran intelligence professional, disagrees with the notion of crowdsourcing in the national security forecasting space.

“I don’t believe in the wisdom of crowds,” said Mark Lowenthal, a former senior CIA and State Department analyst (and 1988“Jeopardy!” champion) who now teaches classified courses about intelligence. “Crowds produce riots. Experts produce wisdom.”

Well, thanks to those that have participated in my crowdsourcing experiments (1, 2, & 3). I have my own opinions based on your contributions.  I’ll keep my mouth shut for now, but expect something quite soon that strays from both Tetlock and Lowenthal’s positions as I’m currently analyzing the results of the 1 Year After Bin Laden survey.  Thanks to all of you for contributing, and if new to this blog and interested in seeing what crowdsourcing is all about, click on this link – “One Year After Bin Laden Crowdsourcing“.  Results will be forthcoming at this blog beginning in September.

And to see the results of the first crowdsourcing experiment, see this page with associated highlights.

Two Promising New Blogs: Rocky Shoals & AlleyesOnJihad

This week brought the introduction of two promising new blogs in the areas of security studies, terrorism and counterterrorism.

First, @robertcaruso jumped into the analytical game beyond Twitter a couple weeks back with an article he co-authored with @stcolumbia on “Defense Intelligence Reform“. A nice bit of work by two members of the #STRATPACK. Next, @robertcaruso stepped it up another notch by launching his own blog this week (Rock Shoals), which discusses the bureaucratic processes and politics of the U.S. intelligence, military and counterterrorism communities.  In his first post, he helps navigate the defense and intelligence bureaucracy:

 Unsurprisingly, in the last 96 hours, many tears have been spilled over the Defense Clandestine Service. Let’s address some of these half-truths and outright falsehoods.

I’m looking forward to the Caruso-style enhancing the debate on how the U.S. government architecture works.

Second, I’m excited to see Kevin Jackson (@alleyesonjihad) jumping into the blogosphere.  He’s been dropping some serious knowledge about al Qaeda the past year on Twitter and now he’s opened up his new blog at AlleyesOnJihad.  He starts off revealing a mystery player in the original AQ propaganda video. In doing so, he cites a document that’s been needing analysis for a long time – the biography of Fazul Abdullah Mohammed, which he cites here:

This confusion made me want to share an anecdotical, but quite interesting story I read in Harun’s autobiography involving the other Abu’l Husayn, the Egyptian one. He was an al Qa’ida member who having been part of a small team of operatives whose mission was to explore and find alternative hiding places for the organization leadership. To be more precise, Harun recounts that during the course of the year 2000, as the preparation for the 9/11 attacks was on its way, he was chosen by the leader of the mission, Sayf al Adl, a Shura council member and head of the security committee, to accompany him for a « very secret trip » to Jalalabad, to the point that the Comorian operative was not informed of the purpose of the mission. He then learned that it was meant to find new suitable safe havens into which the high command could hide if « things go wrong » (in expectation of the forthcoming big attack and the troubles it might get al Qa’ida into). Those who joined were Shaykh Abu’l Husayn al Misri, « a specialist in the relations with the tribes » and a fluent Farsi speaker, according to Harun, as well as an unnamed « Algerian brother », married to a woman of Waziristan. In Jalalabad, the crew payed a visit to Yunus Khalis, a well-known and powerful Afghan mujahid commander, who assured his foreign guests of his unfailing support (the Khalis’ protection granted to al Qa’ida would turn out to be critical during the organization’s escape in eastern Afghanistan following the US invasion).

I’m already looking forward to the second post.  Nice work on the first!

Foxes, Hedgehogs & Confidence – Part 2

Last week, I posted on Daniel Kahneman’s NYT article explaining how confidence and accuracy appear to have little correlation when it comes to forecasting.  Kahneman noted that his forecasts of soldier leadership ability generated from the personal observations of his assessment team were only slightly more accurate than random guessing.

Kahneman’s notion echoes the research of Dr. Philip Tetlock; author of Expert Political Judgement and the basis for much of Dan Gardner’s book Future Babble. Over 20 plus years, Dr. Tetlock surveyed more than 100 experts on a host of different issues building a database of more than 27,000 predictions.  Armed with this data, Tetlock conducted a thorough analysis of expert opinion and, like Kahneman, generally found highly confident experts commonly cited in the media were less accurate than random guessing on any given prediction.  Tetlock labeled these confident but off-based forecasters “Hedgehogs”.  Meanwhile, Tetlock found the more accurate predictors of future outcomes tended to have lower confidence in their predictions.  Tetlock labeled these less confident but more accurate experts “Foxes”. Dan Gardner explains in Future Babble that “Foxes”:

“had no template. Instead, they drew information and ideas from multiple sources and sought to synthesize it. They were self-critical, always questioning whether what they believed to be true really was. And when they were shown they had made mistakes, they didn’t try to minimize, hedge, or evade. They simply acknowledged they were wrong and adjusted their thinking accordingly.  Most of all, these experts were comfortable seeing the world as complex and uncertain—so comfortable that they tended to doubt the ability of anyone to predict the future.”

I believe Tetlock’s research provides valuable perspective for both policymakers and policy advisers.  Policymakers often seek the counsel of experts and routinely put faith in expert analysis depending on the level of confidence expressed by the adviser.  Yet, by Kahneman’s admission and Tetlock’s research, those advisers most confident in their predictions and prescriptions may in fact be less accurate than random guessing.  Likewise, for policy advisers (so-called experts), they often feel pressured to appear aggressively confident when making their predictions to ensure the respect of policymakers and to sustain their status amongst other experts.  Essentially, when policymakers turn to experts, they are seeking certainty about an expert prediction as much or more than the content of the prediction itself.

I’ve lamented many times at this blog my disdain for “Hedgehogs” vaguely predicting every potential scenario with high confidence. I’ll follow up soon with a part 3 related to the polling conducted here in May. Meanwhile, FORA hosts a great series of segments where Tetlock presents some of his findings and I’ll embed his introduction here below.

Why Foxes Are Better Forecasters Than Hedgehogs from The Long Now Foundation on

How Bin Laden Narratives Hindered Analysis

Immediately following 9/11/2001, hopes were high that Bin Laden and his gang would quickly be caught.  In early 2002, Bin Laden escaped the Tora Bora cave network slipping into Pakistan beginning the longest, most expensive and most exhaustive man hunt in world history.

In 2003, the Bin Laden mission lost focus; distracted by Iraq and the hunt for new villains.  By 2004, the American public narrative changed and repeatedly stated that Bin Laden was hiding in a cave, sickened, weak, and irrelevant.  By 2006-2007, this speculation was cemented into the minds of Western analysts, media pundits and the general public.

Looking back, this narrative hindered my analysis and I imagine the analysis of many others seeking the demise of Bin Laden.  Analysts were seeking to confirm a narrative constructed on two brief periods in Bin Laden’s Afghan existence: a hiding period in the so-called “Lion’s Den” during the mid-1980’s and the 2002 Tora Bora siege.  This narrative, derived from an appealing perceived pattern of Bin Laden’s behavior, drove many to look for things that weren’t there: guys in a cave, living on bread and water, coordinating through sophisticated electronic communication. Instead, he was killed in a compound similar to others he resided in, surrounded by family and communicating by courier.

Resources were poured into detecting a pattern that suited our narrative more than the realities described throughout Bin Laden’s life (See Patternicity for more on this).  During the 1980’s, he founded AQ in Peshawar guesthouses.  In the 1990’s, he occupied a Khartoum estate and later lived fairly openly in several different Afghan camps.  This pattern of life, rather than the cave narrative we created, turned out to be consistent with where Bin Laden was discovered.  His Khartoum residence looks strikingly similar to his Pakistani hideout. (See below)

In hindsight, Bin Laden hid not in caves but within people-social networks of loyalty sealed by ideology, bought with Gulf donations and maneuvered through political brokering.  Bin Laden lasted ten years because he leveraged his financial pull to sustain operations, his political value to engender Pakistani supporters, and his ideological credibility to garner protection from the Haqqani network.  People hid Bin Laden, not caves.

How was he identified and killed? Through the persistent work of dedicated analysts, investigators, military operatives and intelligence officers using human skills to turn interview results into a victory.  In the end, it was pursuing good analysis on Bin Laden’s human network, not adhering to narratives that brought mission completion.

Bin Laden’s Khartoum residence, early 1990’s (Source: PBS)





Bin Laden’s Pakistan hideout, 2011 (Source: Guardian)

Counterterrorism “Patternicity” Analysis

Michael Shermer’s recent TED presentation “The Pattern Behind Self-Deception” provides an excellent discussion on the weaknesses of human pattern detection.  Shermer’s description of “patternicity” reminded me of our nation’s counterterrorism analysis immediately following the 9/11 attacks.  I often joke that, “if you leave an intelligence analyst alone long enough, they’ll find Bin Laden in either Pakistan, the local mall or your basement depending on their pattern analysis.”  In counterterrorism, we always find the pattern we are looking for- whether it’s there or not.  This video should be required viewing for intelligence analysts, investigators and academics researching counterterrorism issues.

Here is a quick recap of Shermer’s key concepts.

Humans make two types of errors when attempting to identify patterns.

“Type I Error- False Positive- Believing a pattern is real when it is not (finding a nonexistent pattern)”
“Type II Error-  False Negative- Not believing a pattern is real when it is (not recognizing a real pattern)”

Patternicity is:

“The tendency to find meaningful patterns in both meaningful and meaningless noise.”

Patternicity will occur:

“whenever the cost of making a Type I error (finding a nonexistent pattern) is less than the cost of making a Type II error (not recognizing a real pattern).”

Shermer explains how humans evolved into a default position of making Type I errors (to ensure survival) and thus tend to assume all perceived patterns are real.

Shermer’s “patternicity” lens describes the default fears of counterterrorism personnel between 2001 and about 2006.  Post 9/11, investigators, analysts, and policymakers understood the high cost of a Type II error (in Shermer speak) and thus we assumed that all screen blips, chatter increases and tan male movements were indicators of terrorist attacks.  Usually, these leads turned out to be dirt on screens, people talking excitedly about Middle Eastern soccer matches, and outdoor workers riding the bus to work (Type I errors).    Unable to think our way through the terrorism problem, the U.S. fell back on a second physiological response to uncertainty: spending.   I’ll follow up in a future post about counterterrorism spending.  For now, I encourage all those in counterterrorism to watch Shermer’s talk.  It’s been useful for me as I scan for ‘patterns’ amidst a sea of data.

Busy Week in Counter Terrorism

Counterterrorism efforts around the world hit peak levels this past week. The flurry began with reports last week of potential “Mumbai Style” (not to be confused with “Hunan Style” which would be breaded and deep-fried) terrorist attacks in Britain, France, Germany and maybe the U.S.  The pace thickened with several significant counterterrorism actions.  Here is a quick recap. I may be missing some events so chime in if I left something out.

  1. Background: July 2010: A German citizen of Afghan origin from Hamburg was captured in Afghanistan.  Prior to his capture, Ahmed Sidiqi had traveled to Waziristan and received weapons training.
  2. Wednesday, Sept. 29, 2010: German officials believe up to 70 Germans had undergone training in Pakistan and up to 40 fought in Afghanistan.  German nationals have been reported leaving Europe to join the Islamic Movement of Uzbekistan.
  3. Saturday, Oct. 2, 2010: A French citizen of Algerian origin, Ryan Hannouni, was arrested in Italy near the Naples train station allegedly carrying bomb-making materials.
  4. Saturday, Oct. 2, 2010:  Kenyan anti-terrorism units detain a German convert to Islam near Mombasa.  The German, Sascha Alessadro Bottcher, penned a letter to his mother saying he “would never return alive” and allegedly wanted to join al Shabaab in Somalia. Kenyans deported him back to Germany on Tuesday, October 5.  (This one’s probably unrelated but still interesting)
  5. Sunday, Oct. 3, 2010: U.S. State Department issues a travel advisory for Europe warning of potential terror attacks in European cities.
  6. Monday, Oct. 4, 2010:  Between three and eight German Nationals were killed in a drone stike in the town of Mir Ali, FATA, Pakistan. “The militants were said to be members of Jehad al-Islami and their deaths follow reports that a group of jihadists from Hamburg is at the center of an al Qaeda plot for coordinated terrorist attacks in European cities.”
  7. Tuesday, Oct. 5, 2010: French police arrest 12 people in two separate raids.  Three are suspected of providing false papers for jihadists returning from Afghanistan, while eight are being held for trafficking firearms and explosives.  The contact information for three of the arrested men came from the cell phone of Ryan Hannouni, caught in Naples on Saturday, Oct. 2.
  8. Wednesday, Oct. 6, 2010:  French authorities issue a travel warning to their citizens that the risk of a terrorist attack in Britain is high. (Ohh the French, nothing hurts worse than a retaliatory travel warning, take that Britain.)

Wow, so what do we make of this?  Here are some of my thoughts and questions.

1)    Significantly improved counterterrorism efforts

Yes, I believe recent events illustrate massive improvement in counter terrorism.  I know, I should stay with the “Terrorism Fear Posse” (TFP for short). But, this week’s actions represent a global disruption effort across at least seven or more countries; hitting operational safe havens with drone strikes, rolling up known al Qaeda logisticians, and preemptively arresting those that can facilitate foreign fighter returnees from AFPAK.  This past week, effective information sharing between multiple countries produced rapid action against a decentralized al Qaeda related threat.  Finally, we are getting there.

True, there could still be an attack (in fact, there will ultimately be another attack in the West, we need to accept that). But deliberate, simultaneous CT actions in Pakistan, France, Britain, and Germany will put any terrorist plot that might be in motion into disarray.  I see this recent counter terrorism flurry as a positive sign.  We’re much closer to defeating al Qaeda.  However, one last step remains, the most challenging one; derailing al Qaeda recruitment.

2)    Lessons learned in countering violent extremism

Al Qaeda and affiliated groups will survive as long as they can replenish their recruitment pool.  Countering violent extremism (CVE) and disrupting al Qaeda recruitment remains the biggest challenge. German national villages emerging in Pakistan.  German nationals training and fighting in Afghanistan to then return and attack in Europe.   Big problems!

Why German nationals? Some are radical converts but most are of Turkish descent from what I’ve read.  The UK, Germany and France provide forces to ISAF in Afghanistan.  Does this really radicalize such a large number of European recruits?  If so, why so many Germans; more than Brits and French it seems?

I don’t know the answer to these questions but I do wonder how each country’s approach to CVE has affected their indigenous recruitment to al Qaeda.  From my limited knowledge, it appears each country chose a different CVE strategy post 9/11.  The Brits established relationships, funded organizations, allowed open dialogue and tried to work with Muslim groups to build bridges.  France constructed an organized council of Muslim groups tied directly into the government.  Meanwhile, Germany appears to have rejected any and all dialogue; banning entire Muslim groups from the country and disengaging from vulnerable populations.

I don’t know enough to accurately gauge how Germany’s CVE approach relates to the current surge in German recruitment, but I do believe the U.S. should examine these three countries to identify the risks and rewards of utilizing different CVE techniques in the States.

3)    The government had to issue travel warnings

Stop crying! The media and public bashing of the U.S. government for issuing a European travel warning is ridiculous.  They have to issue a warning.  If they don’t issue the warning and an attack occurs, then the American public would be outraged that the government wasn’t “doing anything” or “wasn’t aware” of the terrorist threat.

“Well, it was too vague, what should I do, wawawawa…”

Look if the U.S. government knew there was a terrorist plot at a specific place, on a specific day, at a specific time, they wouldn’t issue a warning.  They would just go stop the plot.

So, stop crying about the warnings, the government is doing the best they can, and they are getting a lot better at counterterrorism.  So be happy, not angry!

Connecting Counterterrorism Dots: Need Better Analysis, Not More Dots

Recent terrorist near misses on Christmas Day 2009 and in Times Square in May 2010 received the usual call from policymakers and pundits: “We need to connect the dots.” However, calls for “More Intelligence” and “Connecting the Dots” quickly contort into counterproductive policy and unnecessary expenditures that hamper our counterterrorism abilities for several reasons.

1-We still don’t understand the threat

The Christmas Day underwear bombing attempt suggests ‘a lack of intelligence’ isn’t the problem.  A plethora of intelligence clues turned up in the aftermath of this failed bombing attempt.  The bomber, Umar Farouk Abdulmutallab, bought a ticket in cash, had been denied a visa to enter Britain, was detected by the NSA and tipped off to the U.S. Embassy.  All of these tell tale terrorism dots were not connected.  Why? Analysis is the culprit, not data.

2-We are overwhelmed with data, most of it worthless

Analysts failed to detect the Christmas day plot, but to their credit, they are overwhelmed with reporting.  Unfortunately, policymakers and counterterrorism officials will rush to create “more intelligence” such that this “never happens again.”  To prove to Congress expanded capability, government officials will publish more intelligence reports. These officials will brief Congress in subsequent rounds of testimony where they state, “Senator X, since the last time we spoke, our agency has generated 300 gazillion intelligence reports on al Qaeda which is a 30% increase in intelligence over last fiscal year”.  The session will end, Senator X will be happy there is more intelligence reporting and a few months later, another terrorist plot will sneak by U.S. CT analysts.

Why does ‘more intelligence’ fail to stop terrorism?

Imagine you are a kid taking swimming lessons. You show up to the lesson, receive a quick briefing from your swim instructor and then you are dumped into the center of the pool.  You throw your arms haphazardly at the water, thrashing violently, barely keeping your head above the surface.  You try to reach the side and you don’t make it.  The instructor jumps in and saves you.

The instructor gets you out and says, “well you didn’t make it to the side of the pool, but maybe we’ll try a different method next time.” During the next swimming lesson, the instructor boats you a couple miles into the ocean and throws you in saying, “now, try to swim to shore.”  You quickly drown.

This silly story mirrors our repeated approach to counterterrorism.  Except the analyst or investigator is the swimmer, intelligence data is the water, and detection of the terrorist plot is the edge of the pool or shore.  Our counterterrorism approach has been to detect the next terrorist plot by turning our intelligence pool into an ocean.  Subsequently, we are less likely, not more likely, to detect and disrupt the next plot.  There is just too much data, too much reporting, to make sense of what is happening.

3- We invest in CT stuff far more than we invest in CT people

Each terrorist attempt results in further spending to improve counterterrorism.  Except buying more counterterrorism stuff remains far easier than investing in CT people.  Rather than train our analysts and investigators sensibly to build accurate analytical models of the threat, officials will buy more plasma TV’s, databases and software that produces really neat looking charts with lines connecting colorful shapes of differing sizes.

Each increase in CT technology stuff distracts the CT personnel who must learn how to query an intelligence database they will seldom use.  Each database further segregates information bits creating technological silos where key information is overwhelmed by other worthless information.

So what do we do to fix this?

First, train analysts and investigators on analytical methods representative of the threat we face.  We have the intelligence, but we don’t routinely recognize the equation.  The Christmas attempt follows the exact model al Qaeda has used for more than a decade; Recruit, Indoctrinate, Train, Equip, Attack.  Al Qaeda recruited Abdulmutallab during university time in London which has been a worry for years.  Indoctrination occurred in Yemen where he received training, an explosive device previously utilized in Saudi, and targeting guidance.  He attacked via commercial aviation defeating weak security measures previously defeated by Richard Reid, the 9/11 hijackers, the Bojinka plot, etc. Al Qaeda has not significantly changed their approach, and it appears, unfortunately, we have not significantly changed our approach either.

Second, narrow our intelligence streams to what is useful.  Many agencies maintain upwards of 30 or more counterterrorism databases; each requiring special access passwords and dedicated training.  These databases contain thousands of reports; 98% of which say something like, “Osama Bin Laden is awful and we should stop him!”  Two percent of these intelligence reports prove to be useful, but our technology silos prevent rapid discovery and utilization.

Third, eliminate redundant counterrorism bureaucracy.  I’ll write much more on the counterterrorism behemoth, but the volume of people working on CT is counterproductive.  Too many people in too many agencies trying to do the same thing with limited information.  Consolidate and reduce the counterterrorism effort and we will likely have greater success.