Sections

Services

Information

UK United Kingdom

The Reinhart-Rogoff error – or how not to Excel at economics

Last week we learned a famous 2010 academic paper, relied on by political big-hitters to bolster arguments for austerity cuts, contained significant errors; and that those errors came down to misuse of…

Data and computer code should be made publicly available at an early stage – or else … esarastudillo

Last week we learned a famous 2010 academic paper, relied on by political big-hitters to bolster arguments for austerity cuts, contained significant errors; and that those errors came down to misuse of an Excel spreadsheet.

Sadly, these are not the first mistakes of this size and nature when handling data. So what on Earth went wrong, and can we fix it?

Harvard’s Carmen Reinhart and Kenneth Rogoff are two of the most respected and influential academic economists active today.

Or at least, they were. On April 16, doctoral student Thomas Herndon and professors Michael Ash and Robert Pollin, at the Political Economy Research Institute at the University of Massachusetts Amherst, released the results of their analysis of two 2010 papers by Reinhard and Rogoff, papers that also provided much of the grist for the 2011 bestseller Next Time Is Different.

Reinhart and Rogoff’s work showed average real economic growth slows (a 0.1% decline) when a country’s debt rises to more than 90% of gross domestic product (GDP) – and this 90% figure was employed repeatedly in political arguments over high-profile austerity measures.

Carmen Reinhart Wikimedia Commons

During their analysis, Herndon, Ash and Pollin obtained the actual spreadsheet that Reinhart and Rogoff used for their calculations; and after analysing this data, they identified three errors.

The most serious was that, in their Excel spreadsheet, Reinhart and Rogoff had not selected the entire row when averaging growth figures: they omitted data from Australia, Austria, Belgium, Canada and Denmark.

In other words, they had accidentally only included 15 of the 20 countries under analysis in their key calculation.

When that error was corrected, the “0.1% decline” data became a 2.2% average increase in economic growth.

Kenneth Rogoff Wikimedia Commons

So the key conclusion of a seminal paper, which has been widely quoted in political debates in North America, Europe Australia and elsewhere, was invalid.

The paper was cited by the 2012 Republican nominee for the US vice presidency Paul Ryan in his proposed 2013 budget The Path to Prosperity: A Blueprint for American Renewal.

Undoubtedly, without Reinhart and Rogoff, Ryan would have found some other data to support his conservative point of view; but he must have been delighted he had heavyweight economists such as Reinhart and Rogoff apparently in his corner.

Mind you, Reinhart and Rogoff have not tried to distance themselves from this view of their work.

Keeping records

As said at the outset, this is not the first time a data- and/or math-related mistake resulted in major embarrassment and expense. In a summary of such historical clangers, Bloomberg journalist Matthew Zeitlin recently pointed to:

  • NASA’s Mariner 1 spacecraft, destroyed minutes after launch in 1962, thanks to “a missing hyphen in its computer code for transmitting navigation instructions”
  • an Excel spreadsheet error by a first-year law firm associate, who added 179 contracts to an agreement to buy the bankrupt firm Lehman Brothers' assets, on behalf of Barclay’s bank
  • the 2010 European flight ban, following the eruption of Iceland’s Eyjafjallajokull, for which “many of the assumptions in the computer models were not backed by scientific evidence"

While many different types of errors were involved in these calamities, the fact that the errors in the Reinhart-Rogoff paper were not identified earlier can be ascribed by the pervasive failure of scientific and other researchers to make all data and computer code publicly available at an early stage – preferably when the research paper documenting the study is submitted for review.

Images of Money

We’ve discussed this topic in a previous article on Math Drudge and another in the Huffington Post – emphasising that the culture of computing has not kept pace with its rapidly ascending pre-eminence in modern scientific and social science research.

Most certainly the issue is not just one for political economists, although the situation seems worst in the social sciences. In a private letter now making the rounds – which we have read – behavioural psychologist Daniel Kahneman (a Nobel economist) has implored social psychologists to clean up their act to avoid a “train wreck”.

Kahneman specifically discusses the importance of replication of experiments and studies on priming effects.

Traditionally, researchers have been taught to record every detail of their work, including experimental design, procedures, equipment, raw results, data processing, statistical methods and other tools used to analyse the results.

In contrast, relatively few researchers who employ computing in modern science – ranging from large-scale, highly parallel climate simulations to simple processing of social science data – typically take such care in their work.

In most cases, there is no record of workflow, hardware and software configuration, and often even the source code is no longer available (or has been revised numerous times since the study was conducted).

We think this is a seriously lax environment in which deliberate fraud and genuine error can proliferate.

Raising standards

We believe, and have argued, there should be new and significantly stricter standards required of papers by journal editors and conference chairs, together with software tools to facilitate the storage of files relating to the computational workflow.

ktpupp

But there’s plenty of blame to spread around. Science journalists need to do a better job of reporting such critical issues and not being blinded by seductive numbers. This is not the first time impressive-looking data, later rescinded, has been trumpeted around the media. And the stakes can be enormous.

If Reinhart and Rogoff (a chess grandmaster) had made any attempt to allow access to their data immediately at the conclusion of their study, the Excel error would have been caught and their other arguments and conclusions could have been tightened.

They might still be the most dangerous economists in the world, but they would not now be in the position of saving face in light of damning critiques in the Atlantic and elsewhere.

As Matthew O’Brien put it last week in The Atlantic:

For an economist, the five most terrifying words in the English language are: I can’t replicate your results. But for economists Carmen Reinhart and Ken Rogoff of Harvard, there are seven even more terrifying ones: I think you made an Excel error.

Listen, mistakes happen. Especially with Excel. But hopefully they don’t happen in papers that provide the intellectual edifice for an economic experiment — austerity — that has kept millions out of work. Well, too late.


A version of this article first appeared on Math Drudge.

Join the conversation

28 Comments sorted by

  1. Miriam Goodwin

    Research management consultant

    A great example of why we have to demolish the idea that research outputs = papers. The data themselves and the way they have been analysed are as significant - and sometimes even more significant to others or in the longer term - as the papers that are written.
    Australian research has to lift its game in curating and making data and related code available to other researchers.

    report
  2. Peter Ormonde
    Peter Ormonde is a Friend of The Conversation.

    Farmer

    Excellent article. Many thanks.

    I'd be most wary myself of anything or anyone in economics that attempts to lay down cast iron rules. Just doesn't seem to work like that in practice - not everywhere, not always.

    This hunger for aphoristic rules is perhaps why there are so few rich Nobel economists. Why banks and trading desks are not populated by PhDs but by pimply drop-outs who know just enough to make serious money but not enough to know they shouldn't.

    Peer review doesn't seem to work very well in economics does it? Criticisms tend to be ideological or conceptual - but rarely does anyone delve into the detail of the spreadsheets.

    A timely reminder. Food for thought.

    report
    1. Albert Rogers

      logged in via Facebook

      In reply to Peter Ormonde

      I suspect that Adam Smith, the darling of those who think that the less the government interferes with private corporations, the more of a free market we have in the USA, would not agree with them. Smith's primary target for criticism was monopoly.

      After all, Adam Smith was a Scots professor of Moral Philosophy, and I think that he depended very little upon religion for that philosophy.

      report
  3. john tons

    retired redundant

    It is time we debunked the idea of Economics as a science - how many scientific theories would have survived in the light of a consistent and persistent failure to have any predictive force? Yet in the face of such a monumental failure to have any predictive relevance we find that the Milton Friedman argues that their theories are based on "wildly inaccurate descriptive representations of reality, and in general, the more significant the theory, the more unrealistic the assumptions" (Davies Economia p69)

    report
    1. Russell Walton

      Retired

      In reply to john tons

      Agreed.

      It's already been done- "Debunking Economics" by Prof Steve Keen.

      I can't understand how anyone who has actually studied economics could regard it as a science. Progress has been made, some recent Nobel prizes in Economics have been awarded to recipients who are not economists.

      report
    2. Albert Rogers

      logged in via Facebook

      In reply to Russell Walton

      "The General Theory of Employment, Interest and Money" by Keynes probably qualifies as scientific. He defines his terms quite meticulously, and is explicit in his acceptance of the assumption that when a worker receives real wages at twice the rate of another's it implies that his production is twice as valuable to the consumer.

      We can see that in today's economy this does not apply to directors of enormous companies. Keynes also observed, although not in The General Theory, that "the market can stay irrational for longer than you and I can stay solvent".

      report
  4. Charles Jenkins

    logged in via Facebook

    As a journal editor, I wonder how we would actually implement these sensible suggestions within the peer review system. As it is, the biggest problem an editor faces is actually finding reviewers who have the time and inclination to review papers. If those reviewers had to review underlying code and datasets as well, they would virtually be repeating the work that went into the paper in the first place and I think we would run out of reviewers real quick. Also, preparing that sort of material in a publication-ready-form is a big task for somebody. Perhaps the solution is for journals to offer unrefereed data and code repositories on a "what you see is what you get" basis. I think this happens in some areas already.
    There is no doubt that taking this aspect of the publication cycle more seriously is a lot more work and would slow down the rate at which papers get published. Maybe that wouldn't be a bad thing.

    report
  5. Tony Grant

    Student

    There was no criticism of Abbott's the "sky would fall" when the Carbon Price started last July, he didn't concern himself or the coalition with "test, retest reliability" this is the what is going on in this country!

    The power elite have their agenda and nothing will get in the way and the use of "scary stats" will be used until the election.

    The "hip-pocket" is usually the best identifier of how the economy is performing, let the numbers roll and lets see which are "cherry-picked"?

    report
  6. Blair Donaldson

    logged in via Twitter

    I honestly wish somebody could explain to me why economists are afforded so much credibility when their forecasts are almost always of the mark?

    Of course they are always exceptionally wise in hindsight, the latest world economic crisis is a case in point – economists were full of advice on what went wrong but strangely, hardly any of them raised concerns before it happened.

    report
  7. Jennifer Raper

    Mrs

    I cannot rid myself of the thought that Economic theory is on a par with Theology as a predictor of the future

    report
    1. Peter Ormonde
      Peter Ormonde is a Friend of The Conversation.

      Farmer

      In reply to Jennifer Raper

      Yeah - but you've gotta admit we're getting better and better at predicting the past. Perhaps because there's more of it.

      Must admit I'd be very very suspicious of any economic theory built on a single excel spreadsheet. At least they should've checked the sums before declaring a universal truth. But even more, I'd be suspicious of any economic theory based on numerology... that's got more to do with accountancy than economics - which should be primarily about people - and Excel doesn't do…

      Read more
    2. Blair Donaldson

      logged in via Twitter

      In reply to Jennifer Raper

      Given the casual correlations between economics and reality, I'm surprised the Inghams chicken folks haven't offered up all the chook guts from their processing plants to Treasury to aid in economic modelling/forecasting – and get a tax break for their kind offer.

      report
    3. Suzy Gneist
      Suzy Gneist is a Friend of The Conversation.

      Multi-tasker at Graphic Design & Montville Coffee

      In reply to Jennifer Raper

      As a student of Futures Studies I can tell you that 'expert's' predictions are just as probable to come about as a layman's prediction :)
      I don't place too much importance on the credentials of the person, I rather examine their logic, their observations and their conclusions (plus gain an insight into their worldview/ideology/bias). A good sailor or gardener predicts weather often with more accuracy than a meteorologist.

      report
    4. Russell Walton

      Retired

      In reply to Jennifer Raper

      Your "Theology" comment is on the mark.

      There's an informative book on the abject failure of forecasting and the fact that, despite the incompetence of most forecasters, we still rely on their "expertise".

      "Future Babble" by Dan Gardner.

      report
  8. John Campbell

    farmer

    What this clearly shows is the analysis these people made proves nothing one way or other whatsoever.

    There seems to have been a dumbing down of our tertiary institutions at the same time as more money and credence has been given to those who enact at the best pseudo-science.

    Unfortunately , (a little like political parties) these institutions can become entrenched in their ways and self perpetuating in their mediocrity. Many have such a poor grasp of meaningful statistical analysis and experimental…

    Read more
  9. Peter Sommerville

    Scientist & Technologist

    Occasionally a gem appers on TC - this article is one of them. It mirrors somewhat a recent article by Henry Ergas in "The Australian".

    http://www.theaustralian.com.au/opinion/columnists/honesty-would-prove-priceless/story-fn7078da-1226625436024

    Unfortunately it is behind a paywall, but unlike TC the newspaper actually has to generate revenue.

    However, readers of science would be well aware of the steady increase in the use of models as experimental systems for deriving scientific conclusions. In the main these models are not audited or subject to any independent examination.

    Computers together with software are powerful tools, but too readily allow mistakes and bias in conclusions derived therefrom. In this electronic world it should be mandatory for the model to be published along with the conclusions.

    In my own worklife I have seen some absolute mathematical clangers embedded in excel spreadsheets.

    report
  10. Seán McNally

    Market and Social Researcher at eris strategy

    Great article. One of the great things about Excel, and models in general, is that assumptions, data and their relationships are laid out for their consequences to be seen . . . and rejected.

    I'd love to see the actual spreadsheet used, to see the model that swings the economies of nations.

    report
  11. Alex Cannara

    logged in via LinkedIn

    Aww, it's just economics. Who could be hurt by mistaking austerity's real effects?

    But maybe this was really a sneaky way to set up our self-advertizing economic 'geniusses' like Ryan, etc?

    In that case, thanks Rogoff & Reinhart!

    Just be sure when you go in for your bypass or transplant surgery, the folks in charge there use a checklist, and not on a uSoft product.
    ;]
    The French learned that at the cost of just one Arianne 5 launch (new code was 32-bit, old code wasn't changed from 16-bit), and we in the US learned it when JPL & NASA designed & launched a Mars orbiter whose modules didn't agree on metric vs English thrust & velocity units. That $35 million is peanuts today!

    report
    1. Albert Rogers

      logged in via Facebook

      In reply to Alex Cannara

      There is a world-wide-web Consortium, W3C, that publishes recommendations for all sorts of computer data. Some of the best of these are for HTML, XML, and codes for describing how they might be presented. Compliance with these is voluntary, but there is a facility by which you may verify and publish that yoour document does indeed comply with whatever standard you aimed at.

      The silliest thing about a data source that I use, is that EIA publish actual factual energy data in American quadrillions of British thermal units. The British no longer use Btu's and the European/British quadrillion is 10^24, the fourth power of a million, whereas the American quadrillon is a measly thousand million million, 10^15.

      report
  12. John Kelmar

    Small Business Consultant

    I agree that mistakes can occur, but these should have been picked up by colleagues and reviewers, or is it that some academics are held in such high regard, some erroneously, that it is considered blasphemous to even think of questioning their work.

    I know this from personal experience that when I dared to question my Head of School (when I was a Lecturer at Curtin University) and he responded by threatening to "get even with me for that comment."

    When I reviewed papers for publication I always…

    Read more
  13. Albert Rogers

    logged in via Facebook

    I believe that there are computer-aided proofs of at least two long-unproven hypotheses in mathematics. One I think is the four color flat map theorem, and the other I think is that the sum of the squares of two integers equalling the square of another, is the highest integer power for which such a relationship can be true, in other words it cannot be said for cubes etc.

    I do not know how these proofs were verified by the referees, but it seems that in the much easier realms that are aided by spreadsheets, far better rigor than these examples can be obtained, and should be demanded.
    A thousand numbers on a spreadsheet could easily be bungled or even faked in a column or a row or two. and very difficult to find if the originally coded relationships cannot be subjected to scrutiny, preferably by computer.

    report
    1. Peter Ormonde
      Peter Ormonde is a Friend of The Conversation.

      Farmer

      In reply to Albert Rogers

      Sadly Albert this wasn't a mathematical error - but a conceptual one. And even the best of computers won't fix that. In fact it was a series of conceptual errors.

      There's the great whoops moment of knocking off a decent slab of the planet in the figgering. But worse, that correlation is proof. That by adding up a pile of snapshots at a particular point in time and coming up with a magic number there is a divine law at work. Like gravity or the direction of time.

      That wonderful romp through absurdity by Douglas Adams - the Hitchhiker's Guide to the Galaxy - captured it beautifully. The answer to everything from the world's most brilliant computer - 42.

      It's not the answer alone that was wrong. It was the question.

      report
    2. Albert Rogers

      logged in via Facebook

      In reply to Peter Ormonde

      Peter, I agree with you, although what I'm advocating is an improvement in the scrutiny of such work, and a culture that insists on requiring computer worksheets (of whatever sort) to be published in a manner that can be so scrutinised. The Regulatory Commission for which I worked demanded that Enron send all their relevant working papers. We got cartloads of paper rubbish. I do not know how to fix this.

      report
  14. Albert Rogers

    logged in via Facebook

    "Science journalists need to do a better job of reporting such critical issues and not being blinded by seductive numbers."

    Lord Kelvin computed the total mass of oxygen in the atmosphere as 1020 millions of millions of tons, and at the then current rate of consumption by industrial consumption of coal (and probably oil) how long the oxygen would last. He maybe wasn't aware of the directly deleterious effect of carbon dioxide.

    But as reported, even in the Washington Post, the "millions of…

    Read more
  15. rory robertson

    logged in via email @gmail.com

    Q: What is the difference between the Reinhart-Rogoff errors and the growing scandal involving the University of Sydney's faulty Australian Paradox "finding"?

    A: Reinhart and Rogoff have conceded their simple errors. By contrast, the University of Sydney's high-profile "scientists" are defending an obviously faulty self-published "finding" with further false and misleading information.

    The main details of the University of Sydney's Australian Paradox scandal are as follows.

    (a) The…

    Read more