Crowdsourcing U.S. National Security: In Progress Review

Those familiar with this blog have likely participated in some of my crowdsourcing experiments.  You may have needed to fall asleep one night and decided to read an article I co-authored on its application in national security issues.  Well, this week, the LA Times published an in-progress update on a recent IARPA study to see if groups of experts can accurately predict future national security events.

In “U.S. intelligence tests crowd-sourcing against its experts“, one finds Dr. Philip Tetlock, the Godfather of Expert Political Judgment, taking on a MITRE team to see if crowds can do better than internal U.S. government experts in predicting future national security events.

The study, known as Aggregative Contingent Estimation, is designed to see whether the 17 agencies in the U.S. intelligence community can aggregate the judgment of its thousands of analysts — rather than rely on the expertise of just a few — to issue more accurate warnings to policy makers before and during major global events.

Tetlock notes that his group of experts:

“In year one, we beat the unweighted average by about 57%, which was big,” he said. A control group, run by Mitre Corp., averages scores without giving weight to participants who tally the best results.

Mark Lowenthal, a veteran intelligence professional, disagrees with the notion of crowdsourcing in the national security forecasting space.

“I don’t believe in the wisdom of crowds,” said Mark Lowenthal, a former senior CIA and State Department analyst (and 1988“Jeopardy!” champion) who now teaches classified courses about intelligence. “Crowds produce riots. Experts produce wisdom.”

Well, thanks to those that have participated in my crowdsourcing experiments (1, 2, & 3). I have my own opinions based on your contributions.  I’ll keep my mouth shut for now, but expect something quite soon that strays from both Tetlock and Lowenthal’s positions as I’m currently analyzing the results of the 1 Year After Bin Laden survey.  Thanks to all of you for contributing, and if new to this blog and interested in seeing what crowdsourcing is all about, click on this link – “One Year After Bin Laden Crowdsourcing“.  Results will be forthcoming at this blog beginning in September.

And to see the results of the first crowdsourcing experiment, see this page with associated highlights.

Two Promising New Blogs: Rocky Shoals & AlleyesOnJihad

This week brought the introduction of two promising new blogs in the areas of security studies, terrorism and counterterrorism.

First, @robertcaruso jumped into the analytical game beyond Twitter a couple weeks back with an article he co-authored with @stcolumbia on “Defense Intelligence Reform“. A nice bit of work by two members of the #STRATPACK. Next, @robertcaruso stepped it up another notch by launching his own blog this week (Rock Shoals), which discusses the bureaucratic processes and politics of the U.S. intelligence, military and counterterrorism communities.  In his first post, he helps navigate the defense and intelligence bureaucracy:

 Unsurprisingly, in the last 96 hours, many tears have been spilled over the Defense Clandestine Service. Let’s address some of these half-truths and outright falsehoods.

I’m looking forward to the Caruso-style enhancing the debate on how the U.S. government architecture works.

Second, I’m excited to see Kevin Jackson (@alleyesonjihad) jumping into the blogosphere.  He’s been dropping some serious knowledge about al Qaeda the past year on Twitter and now he’s opened up his new blog at AlleyesOnJihad.  He starts off revealing a mystery player in the original AQ propaganda video. In doing so, he cites a document that’s been needing analysis for a long time – the biography of Fazul Abdullah Mohammed, which he cites here:

This confusion made me want to share an anecdotical, but quite interesting story I read in Harun’s autobiography involving the other Abu’l Husayn, the Egyptian one. He was an al Qa’ida member who having been part of a small team of operatives whose mission was to explore and find alternative hiding places for the organization leadership. To be more precise, Harun recounts that during the course of the year 2000, as the preparation for the 9/11 attacks was on its way, he was chosen by the leader of the mission, Sayf al Adl, a Shura council member and head of the security committee, to accompany him for a « very secret trip » to Jalalabad, to the point that the Comorian operative was not informed of the purpose of the mission. He then learned that it was meant to find new suitable safe havens into which the high command could hide if « things go wrong » (in expectation of the forthcoming big attack and the troubles it might get al Qa’ida into). Those who joined were Shaykh Abu’l Husayn al Misri, « a specialist in the relations with the tribes » and a fluent Farsi speaker, according to Harun, as well as an unnamed « Algerian brother », married to a woman of Waziristan. In Jalalabad, the crew payed a visit to Yunus Khalis, a well-known and powerful Afghan mujahid commander, who assured his foreign guests of his unfailing support (the Khalis’ protection granted to al Qa’ida would turn out to be critical during the organization’s escape in eastern Afghanistan following the US invasion).

I’m already looking forward to the second post.  Nice work on the first!

Challenging Social Network Analysis

@will_mccants posted an interesting Slate article yesterday questioning the statistical underpinnings of Nicholas Christakis and James Fowler’s social networking analysis as outlined in their popular book Connected.  Here’s a snippet:

Two other recent papers raise serious doubts about their conclusions. And now something of a consensus is forming within the statistics and social-networking communities that Christakis and Fowler’s headline-grabbing contagion papers are fatally flawed. Andrew Gelman, a professor of statistics at Columbia, wrote a delicately worded blog post in June noting that he’d “have to go with Lyons” and say that the claims of contagious obesity, divorce and the like “have not been convincingly demonstrated.” Another highly respected social-networking expert, Tom Snijders of Oxford, called the mathematical model used by Christakis and Fowler “not coherent.” And just a few days ago, Cosma Shalizi, a statistician at Carnegie Mellon, declared, “I agree with pretty much everything Snijders says.”

Despite the statistical weaknesses of Christakis and Fowler’s argument, their book and discussion provides a useful perspective for understanding social networks.  Christakis and Fowler’s recent fall likely represents the needed plateau of social networking analysis (SNA) – a useful analytical method most effective when utilized in combination with other research approaches rather than seen as an analytic panacea.  Post 9/11 funding flowed like water to any outfit producing a cool looking SNA diagram with a big “Bin Laden” bubble in the center.  Ten years later, I still like and utilize SNA, but I recognize some of its limitations as well.  Here’s my thoughts on SNA as used in counterterrorism.

  1. Self-Fulfilling Prophecy– Many analysts exclusively using SNA routinely fall into the trap of using the method to confirm their preferred theory.  An analyst begins with a seemingly logical story and then searches out bits of data and cobbles connections to ‘prove’ the theory. Sought after data points get put on the diagram and other evidence fails to make the chart.  The analysis satisfies the analyst theory, appears convincing, and quickly falls apart when tested on the ground.  I once argued repeatedly with an analyst who would vigorously trace all information back to Bin Laden.  With sufficient time, this analyst could link every person you’ve ever met to Bin Laden and do it with a convincing chart.  Never mind that almost any person can be linked to any other person in an average of six or so connections.  (I think I read this somewhere, maybe in Connected so you might want to check my facts.)  See Patternicity for a common example of how SNA is misapplied.
  2. Confusing data samples for populations–  Analysts often believe that their data represents the population they are studying when it really is only an unrepresentative sample of that population.  Coupling unrepresentative data samples with SNA results in focus on hubs and links of dubious importance.  This misstep leads analysts to miss outlying actors that are in fact key but don’t have the necessary data to be properly evaluated.  Example, I recently (2011) saw an unclassified, academic SNA showing AQ in Iraq as the key hub of AQ activity globally.  This SNA advocated that AQ in Iraq be the new counterterrorism focus.  I knew this to be off base and then realized that all of the data utilized in the SNA came from open source conflict reports; 80% of which originated in Iraq.  However, no one mentioned this in the briefing.
  3. Can be overly complex–  As technology improves, analysts have increased the amount of data displayed in SNA producing diagrams that look remarkably similar to my iPhone headphones when removed from my pocket- a big mess. Rather than using SNA to create clarity, the diagrams become almost indecipherable resulting in faulty conclusions.
  4. Centrality measured by links can be misleading– Most SNA suggests actors with the most links or actors linking different hubs are the most important.  For the most part, this is true. However, key people (bad guys in my work) often deliberately stay in the shadows, push low-hanging fruit forward, and appear tertiary in SNA.  Here’s an important new article discussing alternative perspectives to the commonly touted centrality notion; “Networks dominated by rule of the few.”
  5. SNA represents cultural factors and strength of relationships poorly- SNA provides a more quantitative method for diagramming relationships.  Unless an analyst has  good software and training, all links can easily appear equal.  However, strength of connections and the cultural reasons for their existence usually prove most important in understanding complexity.  Analysts relying almost solely on SNA will miss this.

Despite my cautions above, Christakis and Folwer’s work is beneficial and I think SNA remains particularly useful.  Here’s some caveats and endorsements.

  1.  Just because the math doesn’t work, doesn’t necessarily mean its wrong–  Unfortunately for statisticians, all human behavior has yet to be proven by numbers. Christakis & Fowler’s analysis of “friends of one’s friends” making one fat, divorced, angry, etc. may in fact be true.  These correlations may just be explained better by something other than math like cultural factors specific to certain populations.
  2. SNA is still great for mapping complex relationships– Law enforcement has been doing SNA with yarn and pictures for decades because it works and helps sort out complex problems.
  3. Technology makes SNA easy– Current tools provide a simple way to track data and relationships creating a central repository for locating information and its relationships.

In the end, I am enjoying the challenges to Christakis and Fowler’s approach but imagine that Internet enabled social networks will not be keen to promote discussions that might undermine their own strength.

Lastly, for those really interested in the mechanics of social networking, collective intelligence and current research, I highly recommend four places:

How Bin Laden Narratives Hindered Analysis

Immediately following 9/11/2001, hopes were high that Bin Laden and his gang would quickly be caught.  In early 2002, Bin Laden escaped the Tora Bora cave network slipping into Pakistan beginning the longest, most expensive and most exhaustive man hunt in world history.

In 2003, the Bin Laden mission lost focus; distracted by Iraq and the hunt for new villains.  By 2004, the American public narrative changed and repeatedly stated that Bin Laden was hiding in a cave, sickened, weak, and irrelevant.  By 2006-2007, this speculation was cemented into the minds of Western analysts, media pundits and the general public.

Looking back, this narrative hindered my analysis and I imagine the analysis of many others seeking the demise of Bin Laden.  Analysts were seeking to confirm a narrative constructed on two brief periods in Bin Laden’s Afghan existence: a hiding period in the so-called “Lion’s Den” during the mid-1980’s and the 2002 Tora Bora siege.  This narrative, derived from an appealing perceived pattern of Bin Laden’s behavior, drove many to look for things that weren’t there: guys in a cave, living on bread and water, coordinating through sophisticated electronic communication. Instead, he was killed in a compound similar to others he resided in, surrounded by family and communicating by courier.

Resources were poured into detecting a pattern that suited our narrative more than the realities described throughout Bin Laden’s life (See Patternicity for more on this).  During the 1980’s, he founded AQ in Peshawar guesthouses.  In the 1990’s, he occupied a Khartoum estate and later lived fairly openly in several different Afghan camps.  This pattern of life, rather than the cave narrative we created, turned out to be consistent with where Bin Laden was discovered.  His Khartoum residence looks strikingly similar to his Pakistani hideout. (See below)

In hindsight, Bin Laden hid not in caves but within people-social networks of loyalty sealed by ideology, bought with Gulf donations and maneuvered through political brokering.  Bin Laden lasted ten years because he leveraged his financial pull to sustain operations, his political value to engender Pakistani supporters, and his ideological credibility to garner protection from the Haqqani network.  People hid Bin Laden, not caves.

How was he identified and killed? Through the persistent work of dedicated analysts, investigators, military operatives and intelligence officers using human skills to turn interview results into a victory.  In the end, it was pursuing good analysis on Bin Laden’s human network, not adhering to narratives that brought mission completion.

Bin Laden’s Khartoum residence, early 1990’s (Source: PBS)





Bin Laden’s Pakistan hideout, 2011 (Source: Guardian)

Counterterrorism “Patternicity” Analysis

Michael Shermer’s recent TED presentation “The Pattern Behind Self-Deception” provides an excellent discussion on the weaknesses of human pattern detection.  Shermer’s description of “patternicity” reminded me of our nation’s counterterrorism analysis immediately following the 9/11 attacks.  I often joke that, “if you leave an intelligence analyst alone long enough, they’ll find Bin Laden in either Pakistan, the local mall or your basement depending on their pattern analysis.”  In counterterrorism, we always find the pattern we are looking for- whether it’s there or not.  This video should be required viewing for intelligence analysts, investigators and academics researching counterterrorism issues.

Here is a quick recap of Shermer’s key concepts.

Humans make two types of errors when attempting to identify patterns.

“Type I Error- False Positive- Believing a pattern is real when it is not (finding a nonexistent pattern)”
“Type II Error-  False Negative- Not believing a pattern is real when it is (not recognizing a real pattern)”

Patternicity is:

“The tendency to find meaningful patterns in both meaningful and meaningless noise.”

Patternicity will occur:

“whenever the cost of making a Type I error (finding a nonexistent pattern) is less than the cost of making a Type II error (not recognizing a real pattern).”

Shermer explains how humans evolved into a default position of making Type I errors (to ensure survival) and thus tend to assume all perceived patterns are real.

Shermer’s “patternicity” lens describes the default fears of counterterrorism personnel between 2001 and about 2006.  Post 9/11, investigators, analysts, and policymakers understood the high cost of a Type II error (in Shermer speak) and thus we assumed that all screen blips, chatter increases and tan male movements were indicators of terrorist attacks.  Usually, these leads turned out to be dirt on screens, people talking excitedly about Middle Eastern soccer matches, and outdoor workers riding the bus to work (Type I errors).    Unable to think our way through the terrorism problem, the U.S. fell back on a second physiological response to uncertainty: spending.   I’ll follow up in a future post about counterterrorism spending.  For now, I encourage all those in counterterrorism to watch Shermer’s talk.  It’s been useful for me as I scan for ‘patterns’ amidst a sea of data.

Busy Week in Counter Terrorism

Counterterrorism efforts around the world hit peak levels this past week. The flurry began with reports last week of potential “Mumbai Style” (not to be confused with “Hunan Style” which would be breaded and deep-fried) terrorist attacks in Britain, France, Germany and maybe the U.S.  The pace thickened with several significant counterterrorism actions.  Here is a quick recap. I may be missing some events so chime in if I left something out.

  1. Background: July 2010: A German citizen of Afghan origin from Hamburg was captured in Afghanistan.  Prior to his capture, Ahmed Sidiqi had traveled to Waziristan and received weapons training.
  2. Wednesday, Sept. 29, 2010: German officials believe up to 70 Germans had undergone training in Pakistan and up to 40 fought in Afghanistan.  German nationals have been reported leaving Europe to join the Islamic Movement of Uzbekistan.
  3. Saturday, Oct. 2, 2010: A French citizen of Algerian origin, Ryan Hannouni, was arrested in Italy near the Naples train station allegedly carrying bomb-making materials.
  4. Saturday, Oct. 2, 2010:  Kenyan anti-terrorism units detain a German convert to Islam near Mombasa.  The German, Sascha Alessadro Bottcher, penned a letter to his mother saying he “would never return alive” and allegedly wanted to join al Shabaab in Somalia. Kenyans deported him back to Germany on Tuesday, October 5.  (This one’s probably unrelated but still interesting)
  5. Sunday, Oct. 3, 2010: U.S. State Department issues a travel advisory for Europe warning of potential terror attacks in European cities.
  6. Monday, Oct. 4, 2010:  Between three and eight German Nationals were killed in a drone stike in the town of Mir Ali, FATA, Pakistan. “The militants were said to be members of Jehad al-Islami and their deaths follow reports that a group of jihadists from Hamburg is at the center of an al Qaeda plot for coordinated terrorist attacks in European cities.”
  7. Tuesday, Oct. 5, 2010: French police arrest 12 people in two separate raids.  Three are suspected of providing false papers for jihadists returning from Afghanistan, while eight are being held for trafficking firearms and explosives.  The contact information for three of the arrested men came from the cell phone of Ryan Hannouni, caught in Naples on Saturday, Oct. 2.
  8. Wednesday, Oct. 6, 2010:  French authorities issue a travel warning to their citizens that the risk of a terrorist attack in Britain is high. (Ohh the French, nothing hurts worse than a retaliatory travel warning, take that Britain.)

Wow, so what do we make of this?  Here are some of my thoughts and questions.

1)    Significantly improved counterterrorism efforts

Yes, I believe recent events illustrate massive improvement in counter terrorism.  I know, I should stay with the “Terrorism Fear Posse” (TFP for short). But, this week’s actions represent a global disruption effort across at least seven or more countries; hitting operational safe havens with drone strikes, rolling up known al Qaeda logisticians, and preemptively arresting those that can facilitate foreign fighter returnees from AFPAK.  This past week, effective information sharing between multiple countries produced rapid action against a decentralized al Qaeda related threat.  Finally, we are getting there.

True, there could still be an attack (in fact, there will ultimately be another attack in the West, we need to accept that). But deliberate, simultaneous CT actions in Pakistan, France, Britain, and Germany will put any terrorist plot that might be in motion into disarray.  I see this recent counter terrorism flurry as a positive sign.  We’re much closer to defeating al Qaeda.  However, one last step remains, the most challenging one; derailing al Qaeda recruitment.

2)    Lessons learned in countering violent extremism

Al Qaeda and affiliated groups will survive as long as they can replenish their recruitment pool.  Countering violent extremism (CVE) and disrupting al Qaeda recruitment remains the biggest challenge. German national villages emerging in Pakistan.  German nationals training and fighting in Afghanistan to then return and attack in Europe.   Big problems!

Why German nationals? Some are radical converts but most are of Turkish descent from what I’ve read.  The UK, Germany and France provide forces to ISAF in Afghanistan.  Does this really radicalize such a large number of European recruits?  If so, why so many Germans; more than Brits and French it seems?

I don’t know the answer to these questions but I do wonder how each country’s approach to CVE has affected their indigenous recruitment to al Qaeda.  From my limited knowledge, it appears each country chose a different CVE strategy post 9/11.  The Brits established relationships, funded organizations, allowed open dialogue and tried to work with Muslim groups to build bridges.  France constructed an organized council of Muslim groups tied directly into the government.  Meanwhile, Germany appears to have rejected any and all dialogue; banning entire Muslim groups from the country and disengaging from vulnerable populations.

I don’t know enough to accurately gauge how Germany’s CVE approach relates to the current surge in German recruitment, but I do believe the U.S. should examine these three countries to identify the risks and rewards of utilizing different CVE techniques in the States.

3)    The government had to issue travel warnings

Stop crying! The media and public bashing of the U.S. government for issuing a European travel warning is ridiculous.  They have to issue a warning.  If they don’t issue the warning and an attack occurs, then the American public would be outraged that the government wasn’t “doing anything” or “wasn’t aware” of the terrorist threat.

“Well, it was too vague, what should I do, wawawawa…”

Look if the U.S. government knew there was a terrorist plot at a specific place, on a specific day, at a specific time, they wouldn’t issue a warning.  They would just go stop the plot.

So, stop crying about the warnings, the government is doing the best they can, and they are getting a lot better at counterterrorism.  So be happy, not angry!

Connecting Counterterrorism Dots: Need Better Analysis, Not More Dots

Recent terrorist near misses on Christmas Day 2009 and in Times Square in May 2010 received the usual call from policymakers and pundits: “We need to connect the dots.” However, calls for “More Intelligence” and “Connecting the Dots” quickly contort into counterproductive policy and unnecessary expenditures that hamper our counterterrorism abilities for several reasons.

1-We still don’t understand the threat

The Christmas Day underwear bombing attempt suggests ‘a lack of intelligence’ isn’t the problem.  A plethora of intelligence clues turned up in the aftermath of this failed bombing attempt.  The bomber, Umar Farouk Abdulmutallab, bought a ticket in cash, had been denied a visa to enter Britain, was detected by the NSA and tipped off to the U.S. Embassy.  All of these tell tale terrorism dots were not connected.  Why? Analysis is the culprit, not data.

2-We are overwhelmed with data, most of it worthless

Analysts failed to detect the Christmas day plot, but to their credit, they are overwhelmed with reporting.  Unfortunately, policymakers and counterterrorism officials will rush to create “more intelligence” such that this “never happens again.”  To prove to Congress expanded capability, government officials will publish more intelligence reports. These officials will brief Congress in subsequent rounds of testimony where they state, “Senator X, since the last time we spoke, our agency has generated 300 gazillion intelligence reports on al Qaeda which is a 30% increase in intelligence over last fiscal year”.  The session will end, Senator X will be happy there is more intelligence reporting and a few months later, another terrorist plot will sneak by U.S. CT analysts.

Why does ‘more intelligence’ fail to stop terrorism?

Imagine you are a kid taking swimming lessons. You show up to the lesson, receive a quick briefing from your swim instructor and then you are dumped into the center of the pool.  You throw your arms haphazardly at the water, thrashing violently, barely keeping your head above the surface.  You try to reach the side and you don’t make it.  The instructor jumps in and saves you.

The instructor gets you out and says, “well you didn’t make it to the side of the pool, but maybe we’ll try a different method next time.” During the next swimming lesson, the instructor boats you a couple miles into the ocean and throws you in saying, “now, try to swim to shore.”  You quickly drown.

This silly story mirrors our repeated approach to counterterrorism.  Except the analyst or investigator is the swimmer, intelligence data is the water, and detection of the terrorist plot is the edge of the pool or shore.  Our counterterrorism approach has been to detect the next terrorist plot by turning our intelligence pool into an ocean.  Subsequently, we are less likely, not more likely, to detect and disrupt the next plot.  There is just too much data, too much reporting, to make sense of what is happening.

3- We invest in CT stuff far more than we invest in CT people

Each terrorist attempt results in further spending to improve counterterrorism.  Except buying more counterterrorism stuff remains far easier than investing in CT people.  Rather than train our analysts and investigators sensibly to build accurate analytical models of the threat, officials will buy more plasma TV’s, databases and software that produces really neat looking charts with lines connecting colorful shapes of differing sizes.

Each increase in CT technology stuff distracts the CT personnel who must learn how to query an intelligence database they will seldom use.  Each database further segregates information bits creating technological silos where key information is overwhelmed by other worthless information.

So what do we do to fix this?

First, train analysts and investigators on analytical methods representative of the threat we face.  We have the intelligence, but we don’t routinely recognize the equation.  The Christmas attempt follows the exact model al Qaeda has used for more than a decade; Recruit, Indoctrinate, Train, Equip, Attack.  Al Qaeda recruited Abdulmutallab during university time in London which has been a worry for years.  Indoctrination occurred in Yemen where he received training, an explosive device previously utilized in Saudi, and targeting guidance.  He attacked via commercial aviation defeating weak security measures previously defeated by Richard Reid, the 9/11 hijackers, the Bojinka plot, etc. Al Qaeda has not significantly changed their approach, and it appears, unfortunately, we have not significantly changed our approach either.

Second, narrow our intelligence streams to what is useful.  Many agencies maintain upwards of 30 or more counterterrorism databases; each requiring special access passwords and dedicated training.  These databases contain thousands of reports; 98% of which say something like, “Osama Bin Laden is awful and we should stop him!”  Two percent of these intelligence reports prove to be useful, but our technology silos prevent rapid discovery and utilization.

Third, eliminate redundant counterrorism bureaucracy.  I’ll write much more on the counterterrorism behemoth, but the volume of people working on CT is counterproductive.  Too many people in too many agencies trying to do the same thing with limited information.  Consolidate and reduce the counterterrorism effort and we will likely have greater success.