Last month the World Economic Forum released its 2010-2011 Global Competitiveness Report.  Among the 131 countries analyzed, the United States ranks fourth overall for global competitiveness (down from ranking second in 2009 and first in 2008) but ranks number one for innovation. Such a finding should comfort policy analysts and policy makers who have long augured America is losing its innovation edge. It seems, while we could do better in overall global competiveness, when it comes to innovation the United States is the gold standard.  All is well.

But what are studies like the Global Competitiveness Report actually measuring?  According the methodology section of the report, over two-thirds of the indicators are derived from what the WEF calls the “Executive Opinion Study.”  The survey asks business leaders throughout the world questions such as, “How would you rate the protection of property rights, including financial assets, in your country? [1 = very weak; 7 = very strong].”  For the report’s innovation subsection only one of the indicators—utility patents per million population—is based on hard data.

The WEF argues surveys help form qualitative data for metrics that hard data are otherwise unavailable.  But because of limited knowledge, and likely respondent biases, surveys such as the WEF’s risk being a better reflection of a nation’s reputation than its actual position.  Fareed Zakaria summed up this issue well when he said in a 2009 Newsweek column:

I’d always viewed the rankings that routinely show America on top as authoritative. But they may be misleading. Most traditional competitiveness studies use polls—of CEOs, scientists, investors—as a key part of their measurements. The World Economic Forum report, for example, relies upon surveys for almost two thirds of its data.  Like a star that still looks bright in the farthest reaches of the universe but has burned out at the core, America’s reputation is stronger than the hard data warrant.

To illustrate the point, ITIF released a report gauging international competitiveness and innovation that only used hard data.  In that report the United States ranks fifth for venture capital, while in the WEF’s 2009 study the United States ranked first.  The difference is our study takes total venture capital as a percent of GDP while WEF asks survey respondents “where is the best place to look for venture funds?”  The most likely reason for the discrepancy within the hard and survey data is that while the United States was clearly the best place for venture funds in the early 2000s, in the last decade that position has declined.  But, the opinions of executives seem to lag the empirical shift.

One may argue that in certain areas the only way to get data is to use survey data; in which case the question becomes: does the bias within these surveys cause more harm than simply leaving the indicator out?  But within the WEF’s study there are clearer cut examples of using survey data when hard data is readily available.  For example, the report asks respondents, “To what extent do companies in your country spend on R&D?”  Yet governments collect data on such spending, what purpose could there possibly be for using survey data instead of hard data?  (And for what it’s worth the United States ranks sixth amongst survey respondents, behind South Korea, Denmark, Luxembourg, Sweden, and Singapore—all of which have higher corporate R&D as a percent of GDP than the United States.)

Yes, the United States fares better then all countries for items such as “business impact of malaria” and “available airline seats per kilometer.”  However, as countries in Asia invest magnitudes more in clean energy than us, or as the majority of European nations offer a far more generous R&D tax credit, celebrations over our top ranking in innovation might be premature.

The bottom line is that it is nice to have a sterling reputation but it is even better for that reputation to survive rigorous inquiry.

photo credit:  Wolfie Fox