Make ‘em Prove the Causality before They Cause Any More Suffering: Part Two, the Fall and After

In Part One, I asked whether the Carmen Reinhart/Kenneth Rogoff study and book didn’t show that, on average, nations experiencing debt-to-GDP ratios above 90% had negative rates of economic growth? And I said the answer to the question was “no.” But I didn’t explain why that was true. So, here goes.

The Fall

When Reinhart and Rogoff published their work they did not make their data set available to people to replicate, analyze, critique their findings, and augment to improve the data set. They ignored the scientific norm that you do that when you’re claiming that you’ve made an important empirical discovery. Other researchers wrote them and requested access to their data set in vain for at least the past three years.

Then a few weeks, ago, they finally yielded to a request for the data set made by Thomas Herndon, a Graduate Student in economics at the University of Massachusetts (UMass) in Amherst. Herndon tried to replicate their analysis and findings and could not do so. In fact he found errors. Here’s a summary from the paper he co-authored with two of his professors, Michael Ash, and Robert Pollin, both of the economics department (hereafter called HAP).

RR has made significant errors in reaching the conclusion that countries facing public debt to GDP ratios above 90 percent will experience a major decline in GDP growth.9 The key identified errors in RR, including spreadsheet errors, omission of available data, weighting, and transcription, reduced the measured average GDP growth of countries in the high public debt category. The full extent of those errors transforms the reality of modestly diminished average GDP growth rates for countries carrying high levels of public debt into a false image that high public debt ratios inevitably entail sharp declines in GDP growth.

Moreover, as we show, there is a wide range of GDP growth performances at every level of public debt among the 20 advanced economies that RR survey.

Specifically, “actual average real growth in the high public debt category is +2:2 percent per year compared to the -0.1 percent per year published in RR.” That change in the findings is very important because even though the new average growth level found is still less than in the 60% to 90% category, where the average growth found was 3.2% annually, the claim that there’s a sharp drop-off between these two categories isn’t supported since the 1 percent difference is not statistically significant. In addition, neither the 3.2% nor the 2.2% average growth rates are representative of their debt-to-GDP ratio level categories, since as HAP say just above there’s a wide range of GDP growth performance in all the categories.

So, that does it. That one finding shows that RR did not show that, on average, nations experiencing debt-to-GDP ratios above 90% had negative rates of economic growth, or even that they had an average rate of growth significantly different from the average in the 60 – 90 % category. Given this finding, what happens to the further inference that high debt levels cause lower growth?

In Part One, I showed that even assuming that the R-R finding was correct it still would not have provided any test of the inference that high debt levels cause lower growth. I stated three reasons. First, R-R committed the ecological fallacy in implying that the high level debt category group growth average could be extended to individual nations and times in each group. We can see from the conclusions of HAP, that there was good reason to be concerned about the ecological fallacy because the group growth average was not found to be representative of individual nations and times.

Second, I pointed out that R-R ignored currency regime variables, failing to include them in the analysis, when it is very likely that any association between the debt-to-GDP ratio and economic growth would vary with these variables. Since HAP end up showing that there is only a small difference between averages in the over 90% category and the 60 – 90% category, it is even more likely that including these variables would have washed out the small differences found, or even reversed the relationship claimed by R-R.

Third, I pointed out that control variables that might have shown that the relationship stated by R-R was spurious were not included in the study, so that possible causes of both a high level of debt-to-GDP and economic growth could not be tested. HAP has nothing to say on this score, but it does raise the question of causality and the failure of R-R to analyze it in any rigorous way, and it concludes by questioning the claim that the R-R findings support the view that high levels of debt inevitably cause low growth.

After the Fall Empirical Research

The HAP analysis and the new availability of the R-R data quickly led to three other analyses, all of which began to explore the question of causality, each one by using more rigorous and more sophisticated though not novel techniques of analysis, than used in the study by Reinhart and Rogoff. A question which immediately occurs is why R-R with all the resources they cold call upon didn’t pursue the same or similar analyses either before or after publication of their results in 2010. After all, those replicating their study only a took a few days to begin to explore questions of causality once they had the R-R data set, yet R-R with three years of opportunity or more to do the same or similar analyses of their own data sets, evidently never did anything of the kind. One simply has to ask whether they were afraid of what they would find if they took a deeper dive into their own data.

Dube’s Distributed Lag Cross-country Panel Analysis

The first of the three studies following on HAP was done by Arindrajit Dube in a guest post at Next New Deal entitled “Reinhart/Rogoff and Growth in a Time Before Debt.” Professor Dube is also in the economics department at UMass. Working on 20 OECD nation corrected panel data set of R-R produced by HAP, Dube used LOWESS regressions and distributed lag models. His results speak to the question of whether slow GDP growth causes higher debt-to-GDP ratios, or whether, as R-R opine, while alternately protesting that correlation isn’t causation, higher debt-to-GDP ratios cause relatively low or even negative growth. They suggest that the causation is more likely to run from growth to debt-to-GDP ratios, than from those ratios to growth. Dube also found that 1) any negative relationship between debt ratios and growth is strongest at lower levels of debt, rather than at higher levels as found by R-R, and 2) there is a stronger association between past economic growth, and current debt ratio levels than the association between current debt ratio levels and future economic growth.

Basu’s Time-Series Analysis of the US, Italy and Japan

The second new follow-on to the R-R and HAP studies was done by yet another UMass economics professor, Deepankar Basu and reported at Next New Deal. He addressed the question of causality by examining time series data in Italy, Japan, and the United States, using vector autoregression (VAR) models, accompanied by Granger non-causality tests and impulse response analysis. VAR analysis isn’t enough to determine causality without making additional assumptions about an underlying causal model. But in cases, where one is analyzing a two-variable relationship using time series data and one assumes that causality can only run way or another, or perhaps both ways, the VAR technique can produce evidence about which of the two variables, if any, is prior to the other. I’ll quote Basu’s summary of his results

To summarize, I find that the time series pattern of the dynamic relationship between public debt and economic growth in the postwar U.S., Italian, and Japanese economies is consistent with low growth causing high debt rather than the high debt causing low growth. I draw this conclusion from two types of analyses: Granger non-causality tests and an investigation of impulse response function plots.

Granger non-causality tests allow one to ask the following questions: (a) do debt levels in the past help in better predicting current economic growth, and (b) does economic growth in the past help in improving predictions of current debt levels? The evidence suggests that for the U.S., Italy, and Japan, the answer to the first question is a NO and the answer to the second is a YES.

Impulse response analysis allows one to address the following questions: (a) what is the impact of an unexpected increase in current debt levels on the future time path of economic growth, and (b) how does an unexpected decline in economic growth affect future levels of debt? The data suggests that an unexpected increase in debt levels has only a small effect on future economic growth but an unexpected decline in economic growth is associated with large and long-lasting increases in public debt levels.

So, Basu’s analysis further extends HAP’s suggestion that it’s more likely that growth causes debt than debt causes growth. Like Dube’s it falsifies the austerity conjecture that debt causes growth at least in the context of a two variable model.

Berg and Hartley’s 20 Nation Panel Study

Perhaps the most important of the recent analyses of the R-R data comes from Matthew Berg and Brian Hartley who are Graduate Students in the economics department at the University of Missouri at Kansas City. They followed Dube in analyzing the R-R 20 nation panel data, used and corrected by HAP, using LOWESS regressions, and distributed lag with impulse response analysis.

First, they addressed the important question of whether the relationship between current debt-to-GDP levels and future growth is the same or at least similar across nations. They found (through an examination of individual “backwards/forwards” graphs) that this relationship varied widely across nations. That is, nations were heterogeneous, not homogeneous with respect to this key relationship. They say:

. . . Even if some sort of relationship between debt-to-GDP and growth can in fact be found in cross-country panel analysis, that relationship does not appear to hold up on the level of individual countries. Because economic policy is made on the level of individual countries, this heterogeneity appears to undercut the rationale for any given particular country to make important policy decisions on the basis of government debt-to-GDP ratios.

I’ve italicized their key point in the paragraph for emphasis. Any attempt to generalize across all the 20 nation panel data, such as the R-R attempt to say that a debt-to-GDP ratio above 90% leads to relatively low or negative economic growth contradicts what the data show at the individual nation level for the two key variables, and is therefore just a false inference.

Second, Berg and Hartley also say:

We find that the correlation between government debt-to-GDP ratios and future growth in Reinhart and Rogoff’s . . . dataset results from outliers which come from the country most suggestive of the hypothesis that slow growth causes high levels of government debt – Japan. . . .

That is, the Japanese data disproportionately distort the overall relationship and create a misleading picture, because of the unique history of Japan. But, nevertheless historical examination of both Japan and the other nations provide evidence consistent with the “reverse causation” causation hypothesis that growth causes debt to-GDP ratio levels rather than the alternative hypothesis of a debt-GDP ratio causal ordering priority. Berg and Hartley show this with distributed lag/impulse response analyses and LOWESS regressions with and without the Japanese data. The analysis, for all practical purposes, shows that there is no relationship between current year association between GDP growth and the debt-to-GDP ratio as claimed by R-R.

They also summarize:

. . . This evidence strengthens and reinforces criticisms recently made by Herndon, Ash, and Pollin . . . of research suggesting a negative relationship between government debt-to-GDP ratios and real GDP growth rates. . . . we . . . find evidence suggesting that correlation of government debt-to-GDP ratios and future growth are much more likely explained by “reverse” causation running from slow GDP growth to high government debt-to-GDP ratios than by “forward” causation running from high government debt-to-GDP ratios to slow growth. Furthermore, what little evidence there is for forward causation appears to stem almost entirely from Japanese outliers. Because – as economists generally recognize – Japan is the clearest of all cases of reverse causation, this considerably weakens the argument for forward causation. In addition, we find tremendous heterogeneity on the level of individual countries in the relationship between current government debt-to-GDP ratios and future growth. This suggests that even if substantial evidence for forward causation is eventually discovered in cross-country studies, the effect will likely be small in size and unreliable, and therefore not relevant to economic policy decisions in any particular individual country. Our findings are suggestive, but not conclusive, and more research is needed. We suggest that simultaneous equations models may offer a way forward on the “frontier question” of causality.

Conclusion: You Can’t Generalize Across All Nations and Times About the Impact of the Debt-to-GDP Ratio on Economic Growth

Actually, I think the findings of Berg and Hartley following on and taking into account the findings of HAP, Dube, and Basu are pretty conclusive and not just suggestive. What they say is that the data included in the two-variable analyses flatly contradict the idea that the debt-to-GDP ratio causes economic growth in the individual nations comprising the 20 nation OECD panel. If anything, the evidence is much more consistent with the idea that it is growth that impacts the debt-to-GDP ratio.

I think there has hardly ever been a clearer finding in the Social Sciences than this one. After all, it took HAP, Dube, Basu, and Berg and Hartley only a matter of days to arrive at it. The only way R-R could have missed it is if they weren’t looking for it. It’s as if they just weren’t looking for the truth; but were only looking for an argument that could be used to justify the austerity policies they favored. That’s not economic science. It’s bias, pure and simple.

In Part Three, I’ll end this series on the R-R affair with a retrospective.