Experts Fail To Predict The Market – Look for Outliers Next Time

Today, I thought I’d take a break from discussing terrorism and focus on other complex challenges – economic markets. In the U.S., we are encouraged to invest in financial markets as a way to plan for retirement. In fact, at one time, a U.S. president proposed the idea of eliminating social security payments for retirees and instead giving Americans a stake in the markets as a substitute. The idea seems sound in theory, and as Dr. Daniel Kahneman pointed out in “Thinking Fast and Slow” and more recently Nate Silver captured in “The Signal and the Noise”, if one invest in the market for the long-run by simply buying index funds tied to the market average, then one is likely to make solid earnings for retirement assuming the market behaves in the future the way it did in the past.

Well, many financial experts work hard to try and beat the market and earn their investors returns greater than the market. Like experts in other fields, namely terrorism, the track record for beating the market is really bad. In 2012, the experts missed the market by a lot. The recent Bloomberg article, “Almost All Of Wall Street Got 2012 Market Calls Wrong”, reinforces why it will be outliers not the majority that will accurately predict the future in complex systems.

“The ill-timed advice [of financial experts] shows that even the largest banks and most-successful investors failed to anticipate how government actions would influence markets. Unprecedented central bank stimulus in the U.S. and Europe sparked a 16 percent gain in the S&P 500 including dividends, led to a 23 percent drop in the Chicago Board Options Exchange Volatility Index, paid investors in Greek debt 78 percent and gave Treasuries a 2.2 percent return even after Warren Buffett called bonds “dangerous.”

Why did they miss the call? Many biases and heuristics emerged to have the experts miss – most notably status quo bias (a belief that tomorrow will be like the past). With economic markets performing poorly in recent years, they continued to predict doom. They also overvalued, unimportant distracting news stories pulling them away from fundamentals.

“They paid too much attention to the fear du jour,” Jeffrey Saut, who helps oversee about $350 billion as the chief investment strategist at Raymond James & Associates in St. Petersburg, Florida, said by phone on Jan. 2. “They were worrying about a dysfunctional government in the U.S. They were worried about the euro quake and the implosion of Greece and Portugal. Instead of looking at what’s going on around them, they were letting these macro events cause fear to creep into the equation.”

Essentially, the ‘hedgehog’ experts strayed from the strength of their economic knowledge and drifted into politics. They paid for their drift.

“It’s always more challenging for investors to try and predict political actions,” Khiem Do, the head of Asian multi- asset strategy at Baring Asset Management, which oversees about $50 billion worldwide, said in a Jan. 2 phone interview from Hong Kong. “In general they’re trained to analyze the economic data, balance sheets and so on. They’re not trained to predict political decisions. These factors have ruled the lives of fund managers in a more significant manner than what used to be over the past 20 or 30 years.”

So don’t feel bad political forecasters, the financial experts have trouble predicting politics and international affairs too.

Listen To Your Friends, Read Academic Publications, Build Your Confidence, Poll Results #11c

Finally back to my computer and ready to wrap up the analysis of confidence I started last month.  In the lead up posts, I noted two excellent articles/books on analysis: Kahneman’s analysis of forecasting leadership potential and the amazing work of Dr. Tetlock who examined confidence levels and the accuracy of predictions.  I then compared this to the confidence levels annotated by respondents to the AQ Strategy and Post UBL Poll.  The first post examined the confidence levels of different professional groups and the second post focused on comparing confidence levels of those with different academic degrees and international experience.

The last analysis of confidence examines the relationship between respondent confidence levels and their preferred information source.   Below is a chart showing the average confidence of respondents selecting each of the following types of information source.  The question specifically asked:

What source do you rely on the most for getting relevant information on terrorism/counterterrorism? (You can only pick one.) 


No surprise, those with the highest confidence selected academic publications as their primary source.  This makes sense for a couple of reasons. First, academics were the most confident overall and they selected academic publications at much higher rates than any other group; a likely correlation.  Second, academic journals conduct peer review and require data and research to back up assertions.  One would expect readers of these journals to have confidence in the information within them.

Overall, those with the highest confidence levels relied on personal relationships as their most trusted information source.  Essentially, trusted advisers – physical friends – have the greatest influence on respondents understanding of terrorism and counterterrorism.  This isn’t surprising but demonstrates the power of physical, social networks.  Friends make us feel more confident in our assertions.  Unfortunately, over reliance on personal relationships with regards to analysis can also lead to group think.  Social circles lacking dissenting opinions can be self-reinforcing (WMD in Iraq comes to mind.)

Social media, magazines and newspapers all clocked in at about the same confidence levels which also seems quite reasonable since they tend to cite the same sources.  In contrast, a small group of respondents selected radio as their primary source and overall had markedly lower confidence levels than any other group.  I have no idea why but maybe we don’t trust what we hear as much as what we read or what we see.

International perspectives increase confidence – Poll results #11b

After comparing the confidence of different groups during the AQ Strategy Poll and the Post UBL Poll, I wondered how international experience might influence one’s confidence.

I re-aggregated the AQ Strategy 2011-2012 survey results and then compared the average confidence of those born in the U.S. with those born outside the U.S. Next, I looked at the international experience of respondents and broke respondents into two groups.  Those that had lived in 3 or more foreign countries for 30 days or more (High International Experience) and those that had lived in 2 or fewer countries for 30 days or more (Low International Experience).

The results were fascinating.  Those born outside the U.S. were much more confident than those born in the U.S.  And those that have lived in 3 or more countries for 30 days or more were significantly more confident than those living in 2 or fewer countries for 30 days or more.  A summary of the results are in the chart below.  The results would seem to make sense as the more one knows about foreign cultures the more one might expect them to know about future AQ developments in these foreign cultures.  What’s most surprising is international experience, not academic degree or professional group, produced the most significant difference in respondent confidence.

Academics are confident – before & after Bin Laden’s death – Poll Results #11

Building on last week’s discussion of expert confidence, I returned to the results of the AQ Strategy and Post UBL polls conducted in late April and early May of 2011.  In each of these polls, I asked respondents the following question.

On a scale of one to ten, how confident are you on this topic area and the answers you provided in this survey?

10= Extremely confident, I work in or study the field of terrorism exhaustively
5= Confident, I’m not a terrorism novice, but I’m not an absolute expert on terrorism
1= Not very confident, I’m interested in the topic, but I don’t really follow the specifics of terrorism on a daily basis

Two weeks ago, I mentioned how Daniel Kahneman’s assessment team was just barely more accurate than random guessing at predicting the future leadership potential of soldiers.  Last week, I expressed my admiration for Philip Tetlock’s research which examined the correlation between expert confidence and prediction accuracy.  In government offices, intelligence agencies, and investment firms, policy makers and investors often rely on an analyst/adviser/expert’s confidence in predicting the outcome of a future issue, trend or market.

When we ask experts how “confident” they are, what does that mean?  How do they determine their confidence? What is their track record?  We usually have no idea what the answers to these three questions are for a particular expert.  Yet, we feel much better if the expert tells us they are “confident” whether they really are or not. The U.S. has executed grand plans based on assertions of confidence. (It’s a Slam Dunk!)

While I can’t assess these two authors analysis in the polls at SelectedWisdom, it did get me curious about how respondents rated their confidence to the AQ Strategy poll and the Post UBL poll. When I took the poll, I rated myself a “6”.  And on average, 268 voters on the AQ Strategy Poll and another 130 in the Post UBL poll estimated their confidence as a “6”.  I then dug a little deeper and wanted to examine how the death of Bin Laden, an unexpected shock, may have affected voter confidence as most respondents answered the AQ Strategy poll the week before Bin Laden’s death and the Post UBL poll the week after Bin Laden’s death.

Below I developed a chart comparing the average confidence of different groups between the two polls.  I broke the comparison down by professional groups, education level and academic focus areas.  Two quick notes – the groups are not exclusive, a government worker with a master’s degree in business will be averaged in the ‘Government’, ‘MA/MS Degree’ and ‘Business’ groups.  Also, some groupings have only a few responses so averages may appear more volatile than they may actually be. (Example: Only 6 respondents in Media-Int’l Development).

Here are the results and I’ll post what I found interesting below. The first 5 categories are professional groups, the next 5 are education levels and the last 7 are academic majors.

Some interesting results:

  1. ‘Academic’ professional group and those majoring in ‘History’ were the most confident on average and held their confidence after Bin Laden’s death.
  2. ‘Government’ professional group had less confidence after Bin Laden’s death.  Maybe those closest to CT action were more cautious in their analysis after a major change in the system.
  3. Those with PHD’s and MA’s were less confident after Bin Laden’s death while those with Associate and BA degrees were more confident after Bin Laden’s death.
  4. Political Science and History majors were more confident than other academic focus areas.  Political science majors were more confident after Bin Laden’s death.  History majors were the most confident throughout.  I wonder if History majors believe “history repeats itself” so they are best equipped to anticipate the future.  I also imagine this leads them to Status Quo bias – a belief that tomorrow will most likely be like today and yesterday.  A safe bet as things on average don’t change drastically from day-to-day.  However, historians are often wrong in their predictions of the long-run future as the only thing that is certain about the future is that it will not be like the past.

I have some more break downs of the poll results on confidence coming in the next few days to include my favorite breakdown coming up in the next post.