Erik Gahner Larsen

Did welfare reforms cause Brexit?

Did welfare policy reforms directly cause Brexit? That’s the conclusion in this article: “Tory austerity and welfare cuts directly caused Brexit, according to a ground-breaking new academic study.”

The results in the new academic study, ‘Did Austerity Cause Brexit?‘, “suggest that the EU referendum could have resulted in a Remain victory had it not been for a range of austerity-induced welfare reforms”. In this post, I outline why I do not believe this conclusion is warranted on the basis of the evidence presented in the study.

I should emphasise that I enjoyed reading the working paper and I sympathise with the empirical approach. The study is interested in a complex topic and combines multiple data sources in order to gather as much evidence as (humanly) possible. Accordingly, regardless of what you might think of my comments on the study, I can highly recommend reading it.

Brexit or UKIP?
While the study is interested in explaining Brexit, it is not really looking at Brexit. Instead, it focuses on the support for the UK Independence Party (UKIP), a party strongly in favour of leaving the European Union. The study presents a lot of results, including some interesting difference-in-difference estimates, but there is only a cross-sectional analysis on the determinants of people’s propensity to vote Leave (available in Table 6 in the manuscript).

There is nothing impressive about these cross-sectional models that provides strong evidence for the story that specific welfare reforms caused Brexit. While the models do not speak against such an interpretation per se, I do believe the manuscript is making too strong claims about Brexit on the basis of such an analysis. In other words, I would say that the manuscript is more about UKIP voting than Brexit. The author is just making it about Brexit as that’s the hot potato at the moment.

Model estimates and aggregate predictions
Despite the fact that the model estimates are interested in UKIP, the author makes the interpretation that three welfare reforms caused Leave to win the Brexit referendum: “Due to the tight link between UKIP vote shares and an area’s support for Leave, simple back of the envelope calculations suggest that Leave support in 2016 could have been up to 9.51 percentage points lower and thus, could have swung the referendum in favor of Remain.” (p. 3) A similar interpretation is available at the LSE British Politics blog: “Had austerity not happened, Leave support could have been up to 10% lower”.

I have three main issues with this interpretation. First, if the manuscript is interested in providing reliable estimates on whether or not welfare reforms caused Leave to win, I would like to see such an analysis be based on more than ‘simple back of the envelope calculations’. Or, alternatively, not be made with such confidence at all.

Second, there is no way I believe in an effect size of 10 percentage points lower support to Leave. From the research I have read on the topic, I know of no single study that find similar effect sizes of specific policy reforms on social outcomes, let alone election outcomes (more on this point below).

Third, while I understand the temptation to look for monocausal explanations of an inherently complex social outcome, the interpretations made in the manuscript are misleading. As my colleague Matthew Goodwin points out, people will read the paper and think “austerity is THE factor”. Such an interpretation, i.e. that austerity is the key driver, would be too simplistic and one should in general be cautious when taking specific model estimates and using them to make aggregate counterfactuals that are unrealistic overestimates at best and heavily misleading and wrong at worst.

What’s the treatment?
Due to a lot of moving parts in British politics, it is difficult to say that one specific policy (our “treatment”) had a causal effect on whether a person voted Leave or Remain. Accordingly, the key challenge is to identify the exact welfare reforms to study and ensure that no other factors related to the reforms drive the results.

In the manuscript, the author describes that he will “focus on three smaller welfare reforms – the abolishment of council tax benefit, the so-called ‘bedroom-tax’ and the introduction of Personal Independence Payments replacing Disability Living allowance” (p. 19). Well, three smaller welfare reforms. Everything is relative but this definitely adds to my skepticism, i.e. that specific welfare reforms caused the Leave vote to win.

Importantly, the manuscript also pays attention to aggregate data, but these analyses are more difficult to link to specific reforms. The question here is: What is the actual treatment!? Take the following two results from the paper on the correlates of UKIP support (after 2010):

  1. Human capital: “[…] support for UKIP gradually trends up as a function of the share of the resident population with low educational attainment. The correlation between support for UKIP and the measure of low human capital only becomes sharply stronger after 2010.” (p. 14)
  2. Employment shares: “Areas with larger employment shares in Retail, and Manufacturing saw significant increases in electoral support for UKIP after 2010. To get a sense of the magnitude, for the Manufacturing sector (ca. 15.4% of employment in 2001), the point estimate of 0.53 in 2015 suggests that the average area saw an expansion in support for UKIP by 2015 by 8.1 percentage points.” (p. 15)

These findings are not related to specific policies. However, the first empirical analysis in the manuscript concludes that UKIP gained “support after 2010 in areas with low skilled, working in routine jobs or the retail sector remain intact.” (p. 15)

I see no reason to question this finding, but human capital and employment shares are not only linked to austerity-induced welfare reforms but a series of factors. The interesting question is whether we would still believe the main findings (i.e. those based on the individual-level analysis) if we did not find a stronger correlation between support for UKIP and human capital only after 2010. This is not necessarily entering the garden of forking paths, but we are heading towards that territory. There is simply no convincing evidence for an effect of specific policies in this part of the analysis.

However, this points to two more serious issues with the inferences made in the study. First, we should question the parallel trend assumption in relation to the exposure to welfare reforms. The specific reforms are much more likely to affect poorer places (one of the studies cited in the manuscript shows that) and I see no reason to believe that nothing else happened in the poorer places that would not also correlate with UKIP support. In other words, a person that is more likely to be affected by austerity-induced welfare reforms is also more likely to be affected by other social, political and economic changes compared to a person not affected by the reforms.

Second, and returning to the point on the effect sizes, for the three reforms being studied, only 10% of the sample experienced one of the three reforms. This is not a problem as it is similar to the proportion in the population (10% of all UK households). However, if we are working with strong effects that could basically change the outcome of the referendum, this should mean that a nontrivial proportion of the affected group should change their vote choice to Leave (not taking differential turnout into account and assuming that they would vote Remain otherwise) as a result of the reforms – and not other factors.

What’s the mechanism?
This might come as a huge surprise for the economists reading this post, but here we go: Political scientists have been studying the impact of welfare policies on political behaviour for decades… (Including my recent study on the impact of welfare retrenchments on government support.)

This is my main issue with the study. There is too much going on data-wise with limited theoretical reflections and no engagement with the relevant literature in political science. One of the main findings in the political science literature is that there are multiple different mechanisms linking welfare policies to mass publics. There are no reflections on these different mechanisms in the manuscript. A good starting point to this literature is the seminal research by Paul Pierson from the early 1990s, including his book Dismantling the Welfare State? Reagan, Thatcher and the Politics of Retrenchment.

However, we do see the acknowledgement that there is something going on that might explain why welfare reforms are linked to the Brexit vote. Specifically, the study looks at “Perception of politics more broadly”, or as political scientists call it, political efficacy (the manuscript does not mention political efficacy at all). These measures include the perceptions that “Public officials do not care”, that respondents “Don’t have a say in what government does” and that “your vote is unlikely to make a difference” (p. 34). Again, there is a lot of relevant literature within political science on how welfare policies matter for political efficacy.

In a forthcoming article in Policy Studies Journal, I review 65 studies interested in estimating the impact of policies on mass publics. None of the 65 studies are cited in the manuscript. However, for people interested in the topic, I will recommend a few that are of interest in this context. First, for some studies using strong and interesting research designs to study how policies matter for political participation, see Davenport (2015), Flavin and Griffin (2009) and Flavin and Hartney (2015).

For the specific study of welfare reforms and the Brexit vote, three studies come to mind. First, Watson (2015) looks at the evolution of conditional welfare programs in Britain from 1996 to 2013 and uses panel data to trace the behaviour of individuals affected by these policies over time. Second, Gingrich (2014) looks at how welfare policies, and in particular how visible they are to the public, matter for whether people link their policy preferences to voting for a right-wing party. Third, in Spain, Muñoz et al. (2014) use panel data to study how an austerity package shaped political participation.

Last, for some additional studies interested in the impact of welfare policies on political efficacy and participation, see Guo and Ting (2015), MacLean (2011), Mettler and Stonecash (2008), Soss (1999), Soss and Schram (2007) and Swartz et al. (2009). All of these studies can help inform the manuscript in question on the mechanism(s) linking austerity-induced welfare reforms to the Brexit vote.

Economic versus cultural explanations
Altogether, this brings us to another issue with the manuscript, namely the focus on a single explanation for the Brexit vote rooted in economics. This issue goes beyond the specific manuscript and is symptomatic for a trend in the literature, i.e. to make it about a horserace between economic and cultural factors in understanding populist sentiments.

The literature is still in an early stage of disentangling the interplay between different factors, and it might as well be that the individual propensity to vote Remain was shaped by the interplay between economic and cultural factors. This is a possibility Pippa Norris outlines in a new working paper: “These theories can also be regarded as complimentary rather than rivals, for example if economic deprivation catalyzed resentment about immigrants and the rejection of open borders.”

Still, academics tend to reduce these complex questions to the simple horserace. A recent example is the new study in PPNAS by Diana Mutz, arguing that “Status threat, not economic hardship, explains the 2016 presidential vote”. I am sure the study is great (although see this paper), but simply ruling out the potential direct and/or indirect role of economic hardship based on a couple of regressions seems unfair to the theory.

This is why I am not convinced that the manuscript provides convincing evidence that the absence of specific welfare reforms would have resulted in a Leave win (the counterfactual claim). There is no data in the manuscript on immigration/culture that allows the author to study these alternative explanations or their respective role in shaping the Brexit vote. In other words, even if we believe in the results in the study (including the simple back of the envelope calculations), it is not possible to conclude that “austerity measures, not X” caused Brexit.

Last, the author of the study told HuffPost UK “that the withdrawal of welfare in poor areas allowed the Vote Leave campaign to “exploit” underlying worries about EU immigration and claims that billions were being spent on Brussels rather than at home in Britain.” While this is all plausible it yet again points to the fact that we are working with multiple potential mechanisms linking policies to political behaviour, and going for a simple economic explanation unrelated to any cultural factors is too simplistic.

Wrapping up
In my opinion, the evidence presented in the manuscript is not strong enough to make the point that welfare reforms caused Brexit. Did austerity-induced welfare reforms play a crucial role in British politics with implications for the Brexit referendum? Maybe, but I am not convinced.

The manuscript makes a good case for why we need to take welfare reforms serious in order to understand political behaviour, and I do believe there are reasons to expect that certain reforms made people, directly or/and indirectly, more likely to vote Leave. However, I have yet to see convincing evidence that “welfare cuts directly caused Brexit”.

Indlæg i Berlingske: Nedskæringer bliver ikke straffet af vælgerne

Har d.d. en kommentar i Berlingske med titlen “Nedskæringer bliver ikke straffet af vælgerne”. I kommentaren konkluderer jeg:

En kløgtig regering, der agerer strategisk i forhold til næste valg, skal dermed ikke holde sig fra nedskæringer, men tværtimod se dem som et politisk redskab til at bevare magten. Med andre ord kan regeringer trygt skære ned i de politikker, som ikke bruges af de vælgere, der vil stemme på dem – og bruge ressourcer på de politikker, som potentielle vælgere favoriserer.

Den kan ligeledes læses online her (kræver abonnement).

Er Socialdemokratiet gået tilbage siden folketingsvalget?

Hos Politiken kan man læse, at “Socialdemokratiet går tilbage i ny meningsmåling”. Artiklen bærer titlen “Efter enegang: Socialdemokratiet er gået tilbage siden folketingsvalget, viser ny måling”. Lad os kigge nærmere på den nye måling.

Som altid – når der er en ny måling – er det vigtigt at placere den i den rette kontekst. Ingen måling står sig godt ud alene, hvorfor Figur 1 viser Socialdemokratiets opbakning i målingerne fra 2018, hvor jeg ligeledes har angivet, hvilken der er den nyeste fra Megafon.

Figur 1: Socialdemokratiets opbakning i meningsmålingerne, 2018

Socialdemokratiet fik som bekendt 26,3% af stemmerne ved folketingsvalget i 2015. I omtrent alle målinger foretaget i år ligger Socialdemokratiet på niveauet omkring folketingsvalget eller højere. Der er ingen systematisk evidens for, at Socialdemokratiet er gået tilbage siden valget.

Det eneste sted vi finder denne historie er i en artikel om én måling fra Megafon. For et par år siden var jeg ude og kritisere Megafons målinger (og dækningen af samme) i forhold til Socialdemokraternes opbakning, og intet tyder på, at det er blevet meget bedre. Det kan undre mig, at journalister og politiske kommentatorer hopper i med begge ben.

Endnu mere interessant er det da også, at artiklen forsøger at koble denne tilbagegang på Mette Frederiksens udmelding om ikke at danne regering med Det Radikale Venstre: “Målingen kommer, efter at partiformand Mette Frederiksen annoncerede, at hun vil gå til valg på at danne en regering kun bestående af Socialdemokratiet. Dermed ønsker hun at droppe 25 års parløb med Det Radikale Venstre.”

Hvorfor er dette interessant? Fordi der ikke er nogen evidens for et statistisk signifikant fald i meningsmålingerne fra den forrige Megafon til den seneste fra samme institut. I den forrige måling fra Megafon (fra 31. maj) fik Socialdemokratiet 25,1% af stemmerne. Som altid kan jeg anbefale denne side, hvor du kan indtaste tal fra to målinger og få svar på, om der er en signifikant forskel mellem to målinger. Det er der ikke i nærværende tilfælde.

Artiklen hos Politiken afsluttes blandt andet med ordene: “Politiken har forsøgt at få en kommentar fra Nicolai Wammen, politisk ordfører for Socialdemokratiet. Han er ikke vendt tilbage”. Dette er der absolut intet at sige til, når det vedrører den slags jammerlige målinger fra Megafon.

Hvor vigtigt er klimaet for vælgerne?

I en ny meningsmåling foretaget for tænketanken Concito vises det, at klimaet er det vigtigste emne for vælgerne. Dette er dog ikke uden væsentlige forbehold.

Min første bekymring, da jeg hørte om meningsmålingen via en journalist fra Mandag Morgens TjekDet, var, at respondenterne i undersøgelsen nok var stillet andre spørgsmål om klimaet forud for spørgsmålet omkring, hvilket emne de fandt vigtigt. Dette ville føre til, at de ville finde klimaet vigtigere.

Det viste sig også at være korrekt, at respondenterne havde fået stillet spørgsmål omkring klimaet forud for det relevante spørgsmål. Dette gør at undersøgelsen ikke er retvisende, hvilket jeg har udtalt mig om sammen med andre forskere. Artiklen kan findes her.

Hvor mange vil stemme på Nye Borgerlige? #5

Udtaler mig hos Altinget omkring, hvordan det går Nye Borgerlige i meningsmålingerne. Det kan findes her. I artiklen, der desværre er bag en betalingsmur, citeres jeg blandt andet for:

“Når det er sagt, skal man altid være skeptisk, når nogle målinger er markant anderledes. I mange målinger ligger Nye Borgerlige omkring 2 procent, og derfor tror jeg, at 4-5 procent til partiet er for højt sat. Kort sagt er der langt mere evidens for, at Nye Borgerlige ligger tættere på spærregrænsen, end at de er langt over den.”

Tidligere indlæg om samme emne kan findes her, her, her og her.

At what age are people considered old?

At what age are people being described as being old? I saw the figure below getting a lot of attention on Twitter with the description: ‘As a kid my dad told me “The age you consider ‘old’ is the square-root of your age times 10”. At 9 you think 30 is old, at 16 you think 40, etc. Turns out he was wrong. It’s the square-root of your age times 8.’

That is a great fit, i.e. the overlap between the blue and the orange line, but is it true? A follow-up tweet describes that the figure shows the answers from ~200 people. That is not a lot.

I always tell my students that they should never go out and collect low-quality data if they can download high-quality secondary data for free. Luckily, the European Social Survey (round 4) provides data on this exact question for more than 50,000 respondents. Using this data, I replicated the figure:

In this figure we do not see as great a fit as in the other figure. For 18-year-olds the average answer is 57 years. For 25-year-olds the average answer is 60. This is far from the estimates we get with the square-root of the respondent’s age times 8.

The lesson? Do not overfit your model (especially not to your N≈200 sample).

Potpourri: Statistics #46

Why you should not trust the Facebook experiment

Recently, there has been a lot of focus on the implications of using Facebook. One study, “The Facebook Experiment: Quitting Facebook Leads to Higher Levels of Well-Being“, argues that people who leave Facebook feel better with their lives. Matthew Yglesias talks about the study in this clip from Vox:

The study also got some attention back in 2016 when it was published (see e.g. The Guardian). This is not surprising as the study presents experimental evidence that people who are randomly assigned to not using Facebook felt better with their lives on a series of outcomes.

The only problem is that the study is fundamentally flawed.

The study finds that people who did not use Facebook for a week reported significantly higher levels of life satisfaction. The design relied on pre and post test measures from a control and treatment group, where the treatment group did not use Facebook for a week. The problem – and the reason we should not believe the results – is that people who took part in the study were aware of the purpose of the experiment and signed up with the aim of not using Facebook! In short, this will bias the results and thereby have implications for the inferences made in the study. Specifically, we are unable to conclude whether the differences between the treatment and the control group is due to an effect of quitting Facebook or is an artifactual effect.

First, when respondents are aware of the purpose of the study, we face serious challenges with experimenter demand effects. People assigned to the treatment group will know that they are expected to show positive reactions to the treatment. In other words, there might not be a causal effect of not being on Facebook for a week, but simply an effect induced by the design of the study.

An example of the information available to the respondents prior to the experiment can be found in the nation-wide coverage. The article (sorry – it’s in Danish) informs the reader that the researchers expect that using Facebook will have a negative impact on well-being.

Second, when people know what the experiment is about and sign up with the aim of not using Facebook, we should expect a serious attrition bias, i.e. that people who are not assigned to their preferred treatment will drop out of the experiment. In other words, attrition bias arises when the loss of respondents is systematically correlated with experimental conditions. This is also what we find in this case. People who got the information that they should continue to use Facebook dropped out of the study.

Figure 1 shows the number of subjects in each group before and after the randomisation in the Facebook experiment. In short, there was a nontrivial attrition bias, i.e. people assigned to the control group dropped out of the study.

Figure 1: Attrition across conditions

The dashed line indicates the attrition bias. We can see that the control group is substantially smaller than the treatment group.

Third, when people sign up to an experiment with a specific purpose (i.e. not using Facebook), they will be less likely to comply with their assigned treatment status. This is also what we see in the study. Specifically, as is described in the paper: “in the control group, the participants’ Facebook use declined during the experiment from a level of 1 hour daily use before the experiment to a level of 45 minutes of daily Facebook use during the week of the experiment.” (p. 663)

These issues are problematic and I see no reason to believe any of the effects reported in the paper. When people sign up to an experiment with a preference for not being on Facebook, we cannot draw inferences beyond this sample and say anything about whether people will be more or less happy by not using Facebook.

Potpourri: Statistics #45