International perspectives increase confidence – Poll results #11b

After comparing the confidence of different groups during the AQ Strategy Poll and the Post UBL Poll, I wondered how international experience might influence one’s confidence.

I re-aggregated the AQ Strategy 2011-2012 survey results and then compared the average confidence of those born in the U.S. with those born outside the U.S. Next, I looked at the international experience of respondents and broke respondents into two groups.  Those that had lived in 3 or more foreign countries for 30 days or more (High International Experience) and those that had lived in 2 or fewer countries for 30 days or more (Low International Experience).

The results were fascinating.  Those born outside the U.S. were much more confident than those born in the U.S.  And those that have lived in 3 or more countries for 30 days or more were significantly more confident than those living in 2 or fewer countries for 30 days or more.  A summary of the results are in the chart below.  The results would seem to make sense as the more one knows about foreign cultures the more one might expect them to know about future AQ developments in these foreign cultures.  What’s most surprising is international experience, not academic degree or professional group, produced the most significant difference in respondent confidence.

Academics are confident – before & after Bin Laden’s death – Poll Results #11

Building on last week’s discussion of expert confidence, I returned to the results of the AQ Strategy and Post UBL polls conducted in late April and early May of 2011.  In each of these polls, I asked respondents the following question.

On a scale of one to ten, how confident are you on this topic area and the answers you provided in this survey?

Example:
10= Extremely confident, I work in or study the field of terrorism exhaustively
5= Confident, I’m not a terrorism novice, but I’m not an absolute expert on terrorism
1= Not very confident, I’m interested in the topic, but I don’t really follow the specifics of terrorism on a daily basis

Two weeks ago, I mentioned how Daniel Kahneman’s assessment team was just barely more accurate than random guessing at predicting the future leadership potential of soldiers.  Last week, I expressed my admiration for Philip Tetlock’s research which examined the correlation between expert confidence and prediction accuracy.  In government offices, intelligence agencies, and investment firms, policy makers and investors often rely on an analyst/adviser/expert’s confidence in predicting the outcome of a future issue, trend or market.

When we ask experts how “confident” they are, what does that mean?  How do they determine their confidence? What is their track record?  We usually have no idea what the answers to these three questions are for a particular expert.  Yet, we feel much better if the expert tells us they are “confident” whether they really are or not. The U.S. has executed grand plans based on assertions of confidence. (It’s a Slam Dunk!)

While I can’t assess these two authors analysis in the polls at SelectedWisdom, it did get me curious about how respondents rated their confidence to the AQ Strategy poll and the Post UBL poll. When I took the poll, I rated myself a “6″.  And on average, 268 voters on the AQ Strategy Poll and another 130 in the Post UBL poll estimated their confidence as a “6″.  I then dug a little deeper and wanted to examine how the death of Bin Laden, an unexpected shock, may have affected voter confidence as most respondents answered the AQ Strategy poll the week before Bin Laden’s death and the Post UBL poll the week after Bin Laden’s death.

Below I developed a chart comparing the average confidence of different groups between the two polls.  I broke the comparison down by professional groups, education level and academic focus areas.  Two quick notes – the groups are not exclusive, a government worker with a master’s degree in business will be averaged in the ‘Government’, ‘MA/MS Degree’ and ‘Business’ groups.  Also, some groupings have only a few responses so averages may appear more volatile than they may actually be. (Example: Only 6 respondents in Media-Int’l Development).

Here are the results and I’ll post what I found interesting below. The first 5 categories are professional groups, the next 5 are education levels and the last 7 are academic majors.

Some interesting results:

  1. ‘Academic’ professional group and those majoring in ‘History’ were the most confident on average and held their confidence after Bin Laden’s death.
  2. ‘Government’ professional group had less confidence after Bin Laden’s death.  Maybe those closest to CT action were more cautious in their analysis after a major change in the system.
  3. Those with PHD’s and MA’s were less confident after Bin Laden’s death while those with Associate and BA degrees were more confident after Bin Laden’s death.
  4. Political Science and History majors were more confident than other academic focus areas.  Political science majors were more confident after Bin Laden’s death.  History majors were the most confident throughout.  I wonder if History majors believe “history repeats itself” so they are best equipped to anticipate the future.  I also imagine this leads them to Status Quo bias – a belief that tomorrow will most likely be like today and yesterday.  A safe bet as things on average don’t change drastically from day-to-day.  However, historians are often wrong in their predictions of the long-run future as the only thing that is certain about the future is that it will not be like the past.

I have some more break downs of the poll results on confidence coming in the next few days to include my favorite breakdown coming up in the next post.

Foxes, Hedgehogs & Confidence – Part 2

Last week, I posted on Daniel Kahneman’s NYT article explaining how confidence and accuracy appear to have little correlation when it comes to forecasting.  Kahneman noted that his forecasts of soldier leadership ability generated from the personal observations of his assessment team were only slightly more accurate than random guessing.

Kahneman’s notion echoes the research of Dr. Philip Tetlock; author of Expert Political Judgement and the basis for much of Dan Gardner’s book Future Babble. Over 20 plus years, Dr. Tetlock surveyed more than 100 experts on a host of different issues building a database of more than 27,000 predictions.  Armed with this data, Tetlock conducted a thorough analysis of expert opinion and, like Kahneman, generally found highly confident experts commonly cited in the media were less accurate than random guessing on any given prediction.  Tetlock labeled these confident but off-based forecasters “Hedgehogs”.  Meanwhile, Tetlock found the more accurate predictors of future outcomes tended to have lower confidence in their predictions.  Tetlock labeled these less confident but more accurate experts “Foxes”. Dan Gardner explains in Future Babble that “Foxes”:

“had no template. Instead, they drew information and ideas from multiple sources and sought to synthesize it. They were self-critical, always questioning whether what they believed to be true really was. And when they were shown they had made mistakes, they didn’t try to minimize, hedge, or evade. They simply acknowledged they were wrong and adjusted their thinking accordingly.  Most of all, these experts were comfortable seeing the world as complex and uncertain—so comfortable that they tended to doubt the ability of anyone to predict the future.”

I believe Tetlock’s research provides valuable perspective for both policymakers and policy advisers.  Policymakers often seek the counsel of experts and routinely put faith in expert analysis depending on the level of confidence expressed by the adviser.  Yet, by Kahneman’s admission and Tetlock’s research, those advisers most confident in their predictions and prescriptions may in fact be less accurate than random guessing.  Likewise, for policy advisers (so-called experts), they often feel pressured to appear aggressively confident when making their predictions to ensure the respect of policymakers and to sustain their status amongst other experts.  Essentially, when policymakers turn to experts, they are seeking certainty about an expert prediction as much or more than the content of the prediction itself.

I’ve lamented many times at this blog my disdain for “Hedgehogs” vaguely predicting every potential scenario with high confidence. I’ll follow up soon with a part 3 related to the polling conducted here in May. Meanwhile, FORA hosts a great series of segments where Tetlock presents some of his findings and I’ll embed his introduction here below.

Why Foxes Are Better Forecasters Than Hedgehogs from The Long Now Foundation on FORA.tv
 

How Confident Are You? Markets, CT and Analysis, Part 1

Daniel Kahneman published a short article summarizing points of his new book Thinking, Fast and Slow.  As both a good writer and former winner of the Nobel Prize in Economics, Kahneman’s book will probably be a good read.

Kahneman’s NY Times article “Don’t Blink! The Hazards of Confidence” explains how people evaluate things and assign confidence to their judgements.  I’ve spent the good part of this year looking at similar issues and will try in several posts to bridge Kahneman’s discussion and our polling here at Selected Wisdom.

Kahneman’s conclusion: future forecasts on the performance and patterns of humans and markets – really any complex system – on average will be no better than random chance.  In fact, Kahneman finds in many cases individual and collective predictions by experts produce results worse than random chance.

I’ll spend some of the next few posts discussing “confidence” as it relates to counterterrorism, economic markets and analysis in general.  Meanwhile, Kahneman provides some interesting commentary.

While assessing the future potential of soldiers, Kahneman learned that his team’s assessments were not particularly correct.  Kahneman noted:

We knew as a general fact that our predictions were little better than random guesses, but we continued to feel and act as if each particular prediction was valid.  I was reminded of visual illusions, which remain compelling even when you know that what you see is false…the illusion of validity.

Kahneman asserts his team incorrectly assessed the soldier’s ability based on a story they constructed from very little information.  He sums this up with a cool new acronym I’ll start abusing:

This was a perfect instance of a general rule that I call WYSIATI, “What you see is all there is.”

The rest of the article provides an interesting take on forecasters in a variety of roles. With regards to confidence, he notes:

The bias towards coherence favors overconfidence. An individual who expresses high confidence probably has a good story, which may or may not be true.

With regards to markets, Kahneman illustrates the research results of Odean and Barber on stock market traders noting:

the most active traders had the poorest results, while those who traded the least earned the highest returns…[and] men act on their useless ideas significantly more often than women do, and that as a result women achieve better investment results than men.

Kahneman also rightly notes:

Facts that challenge such basic assumptions – and thereby threaten people’s livelihood and self-esteem – are simply not absorbed…the mind does not digest them.

Overall, a great read and I’ll contrast his findings here with the results of the confidence questions on the Post UBL and AQ Strategy polls from earlier this year during an upcoming post.

 

 

Challenging Social Network Analysis

@will_mccants posted an interesting Slate article yesterday questioning the statistical underpinnings of Nicholas Christakis and James Fowler’s social networking analysis as outlined in their popular book Connected.  Here’s a snippet:

Two other recent papers raise serious doubts about their conclusions. And now something of a consensus is forming within the statistics and social-networking communities that Christakis and Fowler’s headline-grabbing contagion papers are fatally flawed. Andrew Gelman, a professor of statistics at Columbia, wrote a delicately worded blog post in June noting that he’d “have to go with Lyons” and say that the claims of contagious obesity, divorce and the like “have not been convincingly demonstrated.” Another highly respected social-networking expert, Tom Snijders of Oxford, called the mathematical model used by Christakis and Fowler “not coherent.” And just a few days ago, Cosma Shalizi, a statistician at Carnegie Mellon, declared, “I agree with pretty much everything Snijders says.”

Despite the statistical weaknesses of Christakis and Fowler’s argument, their book and discussion provides a useful perspective for understanding social networks.  Christakis and Fowler’s recent fall likely represents the needed plateau of social networking analysis (SNA) – a useful analytical method most effective when utilized in combination with other research approaches rather than seen as an analytic panacea.  Post 9/11 funding flowed like water to any outfit producing a cool looking SNA diagram with a big “Bin Laden” bubble in the center.  Ten years later, I still like and utilize SNA, but I recognize some of its limitations as well.  Here’s my thoughts on SNA as used in counterterrorism.

  1. Self-Fulfilling Prophecy- Many analysts exclusively using SNA routinely fall into the trap of using the method to confirm their preferred theory.  An analyst begins with a seemingly logical story and then searches out bits of data and cobbles connections to ‘prove’ the theory. Sought after data points get put on the diagram and other evidence fails to make the chart.  The analysis satisfies the analyst theory, appears convincing, and quickly falls apart when tested on the ground.  I once argued repeatedly with an analyst who would vigorously trace all information back to Bin Laden.  With sufficient time, this analyst could link every person you’ve ever met to Bin Laden and do it with a convincing chart.  Never mind that almost any person can be linked to any other person in an average of six or so connections.  (I think I read this somewhere, maybe in Connected so you might want to check my facts.)  See Patternicity for a common example of how SNA is misapplied.
  2. Confusing data samples for populations-  Analysts often believe that their data represents the population they are studying when it really is only an unrepresentative sample of that population.  Coupling unrepresentative data samples with SNA results in focus on hubs and links of dubious importance.  This misstep leads analysts to miss outlying actors that are in fact key but don’t have the necessary data to be properly evaluated.  Example, I recently (2011) saw an unclassified, academic SNA showing AQ in Iraq as the key hub of AQ activity globally.  This SNA advocated that AQ in Iraq be the new counterterrorism focus.  I knew this to be off base and then realized that all of the data utilized in the SNA came from open source conflict reports; 80% of which originated in Iraq.  However, no one mentioned this in the briefing.
  3. Can be overly complex-  As technology improves, analysts have increased the amount of data displayed in SNA producing diagrams that look remarkably similar to my iPhone headphones when removed from my pocket- a big mess. Rather than using SNA to create clarity, the diagrams become almost indecipherable resulting in faulty conclusions.
  4. Centrality measured by links can be misleading- Most SNA suggests actors with the most links or actors linking different hubs are the most important.  For the most part, this is true. However, key people (bad guys in my work) often deliberately stay in the shadows, push low-hanging fruit forward, and appear tertiary in SNA.  Here’s an important new article discussing alternative perspectives to the commonly touted centrality notion; “Networks dominated by rule of the few.”
  5. SNA represents cultural factors and strength of relationships poorly- SNA provides a more quantitative method for diagramming relationships.  Unless an analyst has  good software and training, all links can easily appear equal.  However, strength of connections and the cultural reasons for their existence usually prove most important in understanding complexity.  Analysts relying almost solely on SNA will miss this.

Despite my cautions above, Christakis and Folwer’s work is beneficial and I think SNA remains particularly useful.  Here’s some caveats and endorsements.

  1.  Just because the math doesn’t work, doesn’t necessarily mean its wrong-  Unfortunately for statisticians, all human behavior has yet to be proven by numbers. Christakis & Fowler’s analysis of “friends of one’s friends” making one fat, divorced, angry, etc. may in fact be true.  These correlations may just be explained better by something other than math like cultural factors specific to certain populations.
  2. SNA is still great for mapping complex relationships- Law enforcement has been doing SNA with yarn and pictures for decades because it works and helps sort out complex problems.
  3. Technology makes SNA easy- Current tools provide a simple way to track data and relationships creating a central repository for locating information and its relationships.

In the end, I am enjoying the challenges to Christakis and Fowler’s approach but imagine that Internet enabled social networks will not be keen to promote discussions that might undermine their own strength.

Lastly, for those really interested in the mechanics of social networking, collective intelligence and current research, I highly recommend four places:

Counterterrorism “Patternicity” Analysis

Michael Shermer’s recent TED presentation “The Pattern Behind Self-Deception” provides an excellent discussion on the weaknesses of human pattern detection.  Shermer’s description of “patternicity” reminded me of our nation’s counterterrorism analysis immediately following the 9/11 attacks.  I often joke that, “if you leave an intelligence analyst alone long enough, they’ll find Bin Laden in either Pakistan, the local mall or your basement depending on their pattern analysis.”  In counterterrorism, we always find the pattern we are looking for- whether it’s there or not.  This video should be required viewing for intelligence analysts, investigators and academics researching counterterrorism issues.

Here is a quick recap of Shermer’s key concepts.

Humans make two types of errors when attempting to identify patterns.

“Type I Error- False Positive- Believing a pattern is real when it is not (finding a nonexistent pattern)”
“Type II Error-  False Negative- Not believing a pattern is real when it is (not recognizing a real pattern)”

Patternicity is:

“The tendency to find meaningful patterns in both meaningful and meaningless noise.”

Patternicity will occur:

“whenever the cost of making a Type I error (finding a nonexistent pattern) is less than the cost of making a Type II error (not recognizing a real pattern).”

Shermer explains how humans evolved into a default position of making Type I errors (to ensure survival) and thus tend to assume all perceived patterns are real.

Shermer’s “patternicity” lens describes the default fears of counterterrorism personnel between 2001 and about 2006.  Post 9/11, investigators, analysts, and policymakers understood the high cost of a Type II error (in Shermer speak) and thus we assumed that all screen blips, chatter increases and tan male movements were indicators of terrorist attacks.  Usually, these leads turned out to be dirt on screens, people talking excitedly about Middle Eastern soccer matches, and outdoor workers riding the bus to work (Type I errors).    Unable to think our way through the terrorism problem, the U.S. fell back on a second physiological response to uncertainty: spending.   I’ll follow up in a future post about counterterrorism spending.  For now, I encourage all those in counterterrorism to watch Shermer’s talk.  It’s been useful for me as I scan for ‘patterns’ amidst a sea of data.