Guest Post: Scoring The Reinhart-Rogoff Debate

Tyler Durden's picture

Submitted by FF Wiley of Cyniconomics blog,

HAP vs. RR vs. the Pundits: Scoring the Reinhart, Rogoff Dispute

For those surfers who haven’t already lost interest in the spirited debate over a 2% calculation discrepancy in an historical average, please read on.

I’ll share some thoughts on the academic paper that was released early this week by three University of Massachusetts Amherst scholars and the Internet frenzy that followed. My goal is to clear up a handful of fallacies that have taken hold in the media and blogosphere, while also throwing in my editorial comments and “scores” on each of the parties involved.

The three Massachusetts authors – Thomas Herndon, Michael Ash and Robert Pollin, now known on the Internet as “HAP” – argued that a heavily cited 2010 paper by Carmen Reinhart and Kenneth Rogoff (RR) contained fatal errors. In a critique that was released to the public last Tuesday, they revealed a calculation error in one of RR’s spreadsheets. And they also argued that RR omitted data points without justification and used an unconventional weighting method in their statistical averages. In response, RR acknowledged the calculation error but defended their data set and weighting methods.

This is more than an obscure academic debate only because RR’s conclusions are well-known in both academic and political circles. In their 2010 paper and a second paper released last year (with Vincent Reinhart), they suggested that economic growth tends to slow after government debt rises above 90% of GDP.

They’ve backed their thesis with calculations covering a few different country groupings and time periods, and also by summarizing the work of a handful of other researchers who’ve reached similar conclusions. It’s also clear that they truly believe that excessive government debt leads to lower growth – on a conceptual basis – as do many other people.

It’s always politics. Never personal.

But their beliefs don’t sit well with HAP, who argue that RR have too much influence over public policy decisions in both the U.S. and Europe. HAP suggest that policy could be less austere in both regions. They don’t like to hear politicians repeat RR’s warnings about the dangers of high debt, and they hope to discredit RR to bring an end to their perceived role in current policies.

HAP’s paper concludes with the statement: “RR’s findings have served as an intellectual bulwark in support of austerity politics. The fact that RR’s findings are wrong should therefore lead us to reassess the austerity agenda itself in both Europe and the United States.”

But HAP haven’t challenged the full breadth of RR’s thinking and research on fiscal policy matters, at least in the critique they issued last week. Instead, they focus on just one of the calculations in one of RR’s papers – arithmetic average returns for 1946 through 2009 in the first paper.

Here are the competing views on these average returns:

rr paper

From this simple chart, pundits and bloggers launched a days-long game of “whisper down the lane.” We’ve been fed a succession of incomplete, exaggerated, misleading and erroneous reports, as I’ll explain in these six observations:

  1. For all the public focus on RR’s spreadsheet calculation error, it didn’t have a meaningful effect on their results. As reported by HAP in their paper (see page 7), it changed the arithmetic average in the >90% bucket on the right hand side of the chart by 0.3%. That’s pocket change. But the error’s insignificance was remarkably emphasized in only two of the many accounts I’ve read (kudos to Justin Fox of the Harvard Business Review and Brad Plumer of the Washington Post, with apologies to those I didn’t read). In a couple of the very earliest reports on the paper (see here and here), the authors eventually backtracked by adding a mix of clarifications, corrections and updates to their original posts, presumably after recognizing that they overstated the error’s significance. But they both left their prose written in a way that continued to emphasize it. And their later clarifications didn’t stop other commentators from reporting that the return differences shown in the chart are explained entirely by the error, which is simply untrue. Nor did they prevent sensationalistic titles such as these: How an Excel error fueled panic over the federal debt (LA Times), FAQ: Reinhart, Rogoff and the Excel Error That Changed History (BusinessWeek), Math in a Time of Excel: Economists’ Error Undermines Influential Paper (DailyFinance)
  2. Much of the reporting extended beyond the 2010 paper, leading readers to believe that HAP’s critique invalidates RR’s other work, including their 2009 bestseller, This Time is Different. An LA Times report, which was also picked up by RealClearPolitics, even claimed that RR “popularized” the 90% threshold in their book. In fact, the book did no such thing, nor did RR publish any similar results before 2010.
  3. The dispute centers on the slope and significance of the line in the chart above, and particularly the last segment leading to the 90% bucket, not whether it’s rising or falling. But that didn’t stop pundits from writing their accounts in ways that suggested disagreement about the line’s direction. Moreover, RR pointed out that they placed more emphasis on medians than averages in their discussions (which is entirely consistent with a reread of their papers), and the medians escaped HAP’s critique without comment (more on this below). The fact that HAP’s average return calculations yield similar results to RR’s medians has received almost no attention in the public discussion.
  4. RR never presented 90% as a magic number – where 89.9 is a clear, sunny day and 90.1 a class 5 hurricane – nor did they neglect to recognize that correlation is not causation. I’ve cited the paper on several occasions and never saw it in the way that their critics claim it was presented. Marginal Revolution – probably the most heavily trafficked economics blog – recently republished a 2010 post that likewise didn’t consider 90% to be either “sacred” or “stable.” I doubt that any of those who read RR carefully saw 90% as more than the upper limit on one of their buckets and a reasonable marker to use in their conclusions.
  5. The austerity push in Europe wasn’t triggered in any way, shape or form by RR’s research. It’s based on northern Europe’s struggle to limit the potential damage to their own economies from fiscal crises in the peripheral countries, as explained by OpenEurope here.
  6. Similarly, RR aren’t the puppet masters controlling Republican budget strategies in the U.S., notwithstanding Paul Ryan’s reference to their research, which was discussed by HAP in their paper and repeated many times in articles last week. I’m not aware of any public comments from Ryan on the matter, but it seems unlikely that we’ll wake up tomorrow and read all about his conversion to the “debt doesn’t matter” school based on HAP’s critique.

Keeping score

And now for the scorecard that I promised. I’ll start with RR:

-0.5 for an Excel error that should have been caught before publication. But this is a minor issue, as I pointed out above.  I’ve reread the paper to check the effect, and the error didn’t change a single word. We all make mistakes, and this one wasn’t even a factor. It’s like the stumble that costs a distance runner a fraction of a second but doesn’t change his position in the race.  I’ll say it again: It didn’t change a single word.

Dead-even on the debate over the weighting method. RR have a clear and logical defense for their approach, while HAP offered a reasonable criticism. This happens all the time in academia. People think and act differently, and they also approach research differently.

Dead-even on the data omissions raised by HAP. I have no reason to doubt RR’s defense that their data set wasn’t complete when they wrote the paper. I’ve used their data on several occasions and seen it evolve over time, with significant additions to their government defaults in 2011, for example. And it’s not easy to build such a large data set that you can use with confidence, let alone share with your peers as RR have done graciously. One of the disputed countries (New Zealand), in particular, didn’t surprise me in the least, since earlier this year I collected significantly different debt and growth figures from different sources.  Unless you choose to be careless, these types of discrepancies take time to sort out.

-1 for the interactional effects of their various methods. Based on the mix of methods that RR chose, HAP pointed out that their average return for the >90% bucket assigned a 14% weight to a single year’s growth in New Zealand. The year happened to be 1951, when New Zealand’s economy contracted by 7.6%. This seems too much weight for such an extreme result and it would have been helpful to report its effect on the results.  But it’s not the intellectual travesty that many have made it out to be.  Empirical work is always vulnerable to outlying data.  The important thing is not to make your methods perfect, which is impossible, but to recognize their limitations.

+10 for their contribution to their field. Yes, I’m biased in that I believe they’ve built the world’s most comprehensive history of the types of risks that are most threatening to us today. Their data set and book are tremendous accomplishments. And remember, they operate in the field of macroeconomics. If you were to review all of the published papers in this field for the last, say, 100 years, and weigh them up against real life events, the vast majority could be shown to have major shortcomings. Many have done real damage, leading policymakers to adopt views that are hopelessly disconnected from reality. It’s no exaggeration to say that the foundations of conventional macroeconomic theory have been discredited repeatedly in the last century. And it’s the papers that rely on unrealistic, abstract theories that should concern us most, not a 2% disagreement in an historical average. If you’re interested in an example of the type of paper that deserves to be stamped out, see this post from earlier this year. By comparison, HAP vs. RR is pretty ho-hum to me.

Here are my scores for HAP:

+2 for delivering a thoroughly researched critique on one aspect of RR’s paper.

-1 for the way it was done. I don’t know this for sure, but it seems as though RR gave HAP their data and spreadsheets and yet didn’t even receive an advance copy of the critique. Their initial response suggests they saw it for the first time last Tuesday afternoon, well after blogger Rortybomb had already read the critique, interviewed the authors, examined their spreadsheets and written his article. Because of this apparent ambush, many people formed their opinions without seeing both sides of the story.

-1 for failing to even acknowledge the majority of RR’s results on the empirical relationship between growth and debt. They had no comment whatsoever on the very first result cited in RR’s 2010 paper – the finding that the median return is about 1% lower when debt rises above 90% of GDP. And they also failed to comment on the first result cited in RR’s 2012 paper, which also referenced a return difference of about a percent. Instead, they focused exclusively on arithmetic averages over a single time period and calculated a revised return difference of, well, about 1%.  In other words, we’re being asked to cross out RR’s 1% and replace it with the more “accurate” 1% that’s been reported by HAP (see chart below).  Can someone please remind me what we’re arguing about?

rr paper 2

Overall, HAP certainly offered some analysis for people to consider, while pointing out weaknesses in the 2010 paper, as should be expected of a critique.  But they’ve just as certainly failed to disprove RR’s thesis that high debt tends to be associated with lower growth. (Note that I’ve not claimed causation, and nor did RR in their papers.)

And lastly, let’s assign a score to the pundits

In the meantime, pundits with a predisposition toward loose fiscal policy have launched a character assassination of remarkable force.  One only needs to read a few of the more critical essays and comment threads to see RR subjected to a treatment normally reserved for crooks and felons.

And the most amazing thing about the past week may be how many people became instant experts on exactly how RR described their research to policymakers all over the world.  I must have been the only one who missed the nightly Reinhart and Rogoff Hour on national tv.  Since I follow their work and use their data more than most, it seems odd that I was the guy left out of the loop.

Which brings me to the scoring for the pundits who unleashed this frenzy. To me, their contribution isn’t so much a number but a smell. They’ve left a stench of hypocrisy and a strong whiff of political trickery, by using sensationalistic language and misrepresenting the real issues.

It’s easy to see why they sided with HAP – they’re philosophically opposed to RR’s policy advice.

And it’s easy to see why they emphasized the insignificant spreadsheet errors – no-one would have paid attention if they hadn’t.

But in the process of using the critique as an opportunity for political chest banging, they made clear errors in articles that were written to heap scorn on someone else’s errors. And they should own up to their mistakes exactly as RR did.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
syntaxterror's picture

Who cares what these psuedoscientists publish or predict? It's always 100% hope-inspired bullshit. No one trades off these psuedoscientists spreadsheet macros anyway. Garbage in garbage out.

hannah's picture

it is always 100% political. the gov needs 'data' to prove they are right...

GernB's picture

The real world doesn't care what your political beliefs are and it is unaffeected by people's passion for the topic. You can argue all you want, but in the end relaity wins. The only question people should be asking is how many people am I willing to hurt or kill by trying to deny reality for the sake of supporting what I want to believe rather than what is real.

NoLongerABagHolder's picture

The problem with the RR report has never been a rounding/decimal error...... it has always been the foundational misunderstanding of the monetary systems in which they are studying that makes the study structurally wrong and unusable:

GernB's picture

And that articla makes assertions about the significance of factors that could affect the outcome of the paper without backing them up with anything. They claim significance but don't do anything to prove it. That's called propaganda.

hannah's picture

gernby - propaganda is reality when you are looking at it down the barrel of a mosin-nagant 91/30.....reality bites commrade

Lux Fiat's picture

I respect Cullen Roche and read Prag Cap every now and then.  But he is a dyed in the wool proponent of MMT, which belongs in the pantheon of failed monetary "paradigms".

NoLongerABagHolder's picture

He actually doesn't agree fully with MMT, so you must not read him too closely.

And I would argue he has been pretty right on just about all macro calls and hyper inflation rants. Zero Hedge was mocking Dow 10,000 as a farce.


Lux Fiat's picture

Perhaps the change is more recent?  I have read plenty of his articles where he goes full court for MMT [at least from my perspective], whether in the article, or comments section.  I have not visited his site much in the last few months, and read even fewer articles.

Zer0head's picture

TORONTO (Reuters) - Canadian police and intelligence agencies will announce multiple arrests on Monday after an operation to thwart a "major terrorist attack," the Canadian Broadcasting Corporation reported, citing unidentified "highly placed" sources.

The Royal Canadian Mounted Police said in a statement that they would provide details about a "national security criminal investigation" at a news conference at 3:30 p.m. ET

James_Cole's picture

Good excuse for the Cons to strip Canadians of their rights. 

Scro's picture

In the end it won't matter. We are sofa king screwed.

Bunga Bunga's picture

This study is obsolete anyway, because it does not contain iTunes songs.

HD's picture

The money printers love this BS. Any excuse to fuel up the helicopters...

marathonman's picture

Hell, even Max Keiser blew this story out of proportion.  It's getting to the point that I don't trust anyone anymore.  I still like the book.  Let's not lose their point about countries' finances blowing up because of high debt and out of control spending.  Monetary Ponzi schemes like fractional reserve lending always implode.  It is their nature.  Don't shoot the academic messengers for pointing out the obvious in their academic way.

Gromit's picture

OK you like the book.

Have you been able to read it through, every page?

marathonman's picture

A bit of a slog in some places, but yes.  They've crawled through the entrails of national economic roadkill over the last eight decades - economic autopsy from history.  The warnings are clear.  Once you cross the >90% debt to GDP rubicon, a financial crisis most often results in a sovereign debt crisis.  Very few exceptions - (there are a few but the countries and time period escape me at the moment).  This time is not different.

Gromit's picture

Excuse my heresy but yes R & R's book is a tough read, endless boring irrelevant statistics with very little explanatory narrative.

Basically it's unreadable and people seem to conclude therefore it must be very good.

MUCH better book on this topic is "The Volatility Machine" by Michael Pettis. Don't worry about the title, it's about how emerging market financial crises develop. Primarily they develop because of a financial panic in the lendng countries which suddenly pull the rug up from under their debtors.



spine001's picture

I recognize that its a hard read but is very good solid block of data that kills Ben's assumptions. Reinhart fails misserably though at the opportunity of identifying why "this time is different" is false over and over again. If you put her book together with Schiller's "Irrational exhuberance" book it is obvious what the source of the bubbles is. Basically the reason is that financial operators, professional investors and bankers make tons of money pumping the bubbkes as big as they can for as long as they can. I cant believe that Bernanke doesnt understand this. HE Is only giving fuel to the same people responsie of all previous bubbles. The data is there in plain sight for all to see.

Until next time,


Lux Fiat's picture

Will have to check it out - I look forward to each new blog post on his site.  However, would have to recommend Hyman Minsky's "Stabilizing an Unstable Economy".  Even if you don't agree with him on everything, his take on things is still eerily relevant and prescient 27 years after it was first published. 

Ghordius's picture

I did too. Mixed feelings about it.

Nevertheless, re "The austerity push in Europe wasn’t triggered in any way, shape or form by RR’s research. It’s based on northern Europe’s struggle to limit the potential damage to their own economies from fiscal crises in the peripheral countries, as explained by OpenEurope..." OpenEurope is wrong.

The need for balanced budgets was discussed as a strategy/policy in case of a currency war in the 1980's, while discussing the yet unborn EuroProject. It was practically an expected result, in the same way as gold's resurgence was an expected result to the eventual weakening of the USD complex

ChanceIs's picture

Every now and then Max really blows some smoke.  It was pretty bad with his critique of RR.  I think that Stacy didn't have the book title correct.  They were reaching for it big time and it showed.

Max is an asset.  He is 85% facts and 15% schtick.  He was a stand-up comedian for a while.  He plays to the outrage segment.  You know...Madonna, Howard Stern, Lady Gaga, etc.

Its just that when he doesn't have something to say, he should take a vacation from the show instead of pulling it out of his schfinf-schfing.

You gotta love the Jamie Dimon tapeworm characterization.  Jamie won't come on his show, but a lot of heavyweights do: Steve Keen, Jim Rickards, JAmes Turk (?), Doug Naylor Lealand, Fekete, "Das," etc, etc.

Monkeyfister's picture

This is all anyone needs to know about Reinhart-Rogoff:

Contractionary Policy is CONTRACTIONARY.

It is utterly impossible for an economy to contract its way to growth and prosperity.

It's never been done, and it will never, ever happen in the real world.

An economy contracting its way to growth and properity only happens in the fevered dreams of Teabaggers, Tories, Glibertarians, and Fascists.

Cugel's picture

Why You've Never Heard of the Great Depression of 1920

malek's picture

You mean in La-La-land an economy can go on without ever contracting, forever?

venturen's picture

what are you are not a student of history. So your point fiscal irresponsibility wins?? So how much is too much.. 50% GDP, 100 GDP%, 150, 200 300 500 1000%? I love how socialists love to spend other people's money....while carving out rules for themselves. 

tradewithdave's picture

+2 for R 'n R's boss' parents naming their son twice...Pete Peterson, father of the platinum platinum trillion trillion dollar dollar coin coinson.

Skin666's picture

Seriously... Who gives a fuck?

rayban's picture

The original idea was good but the computational error was unacceptable. It's just another blow to the residual goodwill, if any, of the economic "science", which is based on blatantly false assumptions to deliver even worse prescriptions. The biggest lie of all: "efficient markets".

syntaxterror's picture

Nothing is more efficient than POMO and "doing God's work".

THE DORK OF CORK's picture

Stick Greenbacks into the money system and not interest bearing debt and both sides lose their little arguments.

THE DORK OF CORK's picture

Stick Greenbacks into the money system and not interest bearing debt and both sides lose their little arguments.

malek's picture

Thanks for the concise post!

It seems however that the second graph should source RR's 2012 paper, not same January 18, 2010 as the first graph?

macroeconomist's picture


"It’s easy to see why they sided with HAP – they’re philosophically opposed to RR’s policy advice."

Oh yes, it is very easy to see why you wrote this post: Because you are philosophically opposed to HAP’s policy advice. In order to defend austerity and attack anti-auesterity authors, you can shamelessly defend a dishonest data manipulation. Disgraceful indeed.

Radical Marijuana's picture

Certain individuals sometimes become "social lightning rods."

The political becomes personal, in the sense that the social storms stike some individuals. It is usually no fun for those who get struck, at the time, although, IF they survive, that can turn out to have been for the best in the end. Of course, the debate over whether it is a "good idea" for governments to take on more and more debts should be placed in this kind of overall political context:

Money for Nuttin—Debt for Free

By Preston James, April 21, 2013.

poydras's picture

Two interesting items in the past week -

1) Debunking austerity (You really can live beyond your means forever!)

2) More serious talk of wealth seizure (no serious protests so far...why not go directly for the pocketbook?)

"Crisis Countries Should Seize Assets

It would be more sensible -- and fairer -- for the crisis-ridden countries to exercise their own power to reduce their debts, namely by reaching for the assets of their citizens more than they have so far. As the most recent ECB study shows, there is certainly enough money available to do this."


itstippy's picture

Any study using reported GDPs and Sovereign Debt loads for data is a waste of time.  These guys are spending their acedemic lives studying crap data.  Garbage in, garbage out. 

rjs's picture

one might think that one academic paper couldn't possibly have so great an influence as to condemn the entire civilized world to a deep and prolonged recession because policy makers were handcuffed into not using fiscal policy remedies known to be corrective and stimulative; but testament to the pervasiveness of it's influence was evident just last week; in an exclusive document released to Reuters just last Saturday (4/13) a working group representing financial ministers of the G-20 propsoed that they would all cut their debt levels to well below 90% of GDP, the exact prescription advocated by Rogoff-Reinhart; however, by the end of the G-20 meeting on Thursday, 3 days after the dismantling of the Rogoff-Reinhart thesis was made public, the G-20 was in disarray, backpedaling from their plan to reduce international sovereign debt; with financial and monetary authorities both expressing "uncertainty about the way economies work and how to influence recoveries with policy”.

sockratte's picture

i suppose the '+10' is the same error that RR did *g*