Building on last week’s discussion of expert confidence, I returned to the results of the AQ Strategy and Post UBL polls conducted in late April and early May of 2011. In each of these polls, I asked respondents the following question.
On a scale of one to ten, how confident are you on this topic area and the answers you provided in this survey?
Example:
10= Extremely confident, I work in or study the field of terrorism exhaustively
5= Confident, I’m not a terrorism novice, but I’m not an absolute expert on terrorism
1= Not very confident, I’m interested in the topic, but I don’t really follow the specifics of terrorism on a daily basis
Two weeks ago, I mentioned how Daniel Kahneman’s assessment team was just barely more accurate than random guessing at predicting the future leadership potential of soldiers. Last week, I expressed my admiration for Philip Tetlock’s research which examined the correlation between expert confidence and prediction accuracy. In government offices, intelligence agencies, and investment firms, policy makers and investors often rely on an analyst/adviser/expert’s confidence in predicting the outcome of a future issue, trend or market.
When we ask experts how “confident” they are, what does that mean? How do they determine their confidence? What is their track record? We usually have no idea what the answers to these three questions are for a particular expert. Yet, we feel much better if the expert tells us they are “confident” whether they really are or not. The U.S. has executed grand plans based on assertions of confidence. (It’s a Slam Dunk!)
While I can’t assess these two authors analysis in the polls at SelectedWisdom, it did get me curious about how respondents rated their confidence to the AQ Strategy poll and the Post UBL poll. When I took the poll, I rated myself a “6″. And on average, 268 voters on the AQ Strategy Poll and another 130 in the Post UBL poll estimated their confidence as a “6″. I then dug a little deeper and wanted to examine how the death of Bin Laden, an unexpected shock, may have affected voter confidence as most respondents answered the AQ Strategy poll the week before Bin Laden’s death and the Post UBL poll the week after Bin Laden’s death.
Below I developed a chart comparing the average confidence of different groups between the two polls. I broke the comparison down by professional groups, education level and academic focus areas. Two quick notes – the groups are not exclusive, a government worker with a master’s degree in business will be averaged in the ‘Government’, ‘MA/MS Degree’ and ‘Business’ groups. Also, some groupings have only a few responses so averages may appear more volatile than they may actually be. (Example: Only 6 respondents in Media-Int’l Development).
Here are the results and I’ll post what I found interesting below. The first 5 categories are professional groups, the next 5 are education levels and the last 7 are academic majors.
Some interesting results:
- ‘Academic’ professional group and those majoring in ‘History’ were the most confident on average and held their confidence after Bin Laden’s death.
- ‘Government’ professional group had less confidence after Bin Laden’s death. Maybe those closest to CT action were more cautious in their analysis after a major change in the system.
- Those with PHD’s and MA’s were less confident after Bin Laden’s death while those with Associate and BA degrees were more confident after Bin Laden’s death.
- Political Science and History majors were more confident than other academic focus areas. Political science majors were more confident after Bin Laden’s death. History majors were the most confident throughout. I wonder if History majors believe “history repeats itself” so they are best equipped to anticipate the future. I also imagine this leads them to Status Quo bias – a belief that tomorrow will most likely be like today and yesterday. A safe bet as things on average don’t change drastically from day-to-day. However, historians are often wrong in their predictions of the long-run future as the only thing that is certain about the future is that it will not be like the past.
I have some more break downs of the poll results on confidence coming in the next few days to include my favorite breakdown coming up in the next post.
2 comments