Erik Gahner Larsen

Potpourri: Statistics #49

House effects in Danish opinion polls #2

A year ago, Zoltán Fazekas and I looked into house effects in Danish opinion polls on the support for political parties. In brief, we found some interesting differences in the house effects among different polling firms (do read the post from last year if you are unfamiliar with the concept of house effects).

However, with the upcoming Danish general election in 2019, we found it necessary to update the results. You can read a lot more about the method and results here. Below you can find one of the figures from the analysis.

Unsurprisingly, the polling firms have a hard time polling the new right-wing political party, Nye Borgerlige, and there is no agreement among the firms (for my previous posts in Danish on this, look here, here, here, here and here). As the figure shows, YouGov believes the party is doing better than what other firms predict, whereas Voxmeter is less optimistic about the prospects for the party.

Last, Hans Redder from TV 2 uses our results in his new and interesting piece on what you need to be aware of when you see a new opinion poll. It is great to see how these aspects of the polling coverage are getting more and more attention, and I do hope that more journalists will show awareness of this in their coverage when we get closer to the election. To be continued…

Potpourri: Statistics #48

Skaber sociale medier ekkokamre? #2

For over et år siden skrev jeg et indlæg om sociale medier og ekkokamre. Mit argument var og er, at der ikke er overbevisende evidens for, at sociale medier skaber ekkokamre, hvor borgerne udelukkende eksponeres for holdninger, der harmonerer med, hvad de allerede mener.

Der er selvfølgelig al mulig grund til at være kritisk i forhold til sociale medier. Der er dog intet der giver grundlag for dommedagsprofetier om, at sociale medier vil placere os alle i vores eget lille meningsbekræftende mikrokosmos, hvor ingen længere vil blive eksponeret for udfordrende synspunkter.

Siden jeg skrev indlægget, er der (heldigvis) kommet flere undersøgelser, der i overvejende grad afkræfter denne demokratiske dommedagsprofeti.

Hos Politiken kunne man således i september 2017 læse en artikel med titlen “Mytedræber: Facebook er ikke kun et ekkokammer“, der formidlede en undersøgelse fra IT-Universitetet. Undersøgelsen viste, at borgere møder holdninger, de er uenige i, mens de diskuterer nyheder eller politik på Facebook.

Undersøgelsen har sine begrænsninger, men det interessante er, at Vincent F. Hendricks, leder af et center med navnet Center for Information og Boblestudier (da et simpelt nysgerrigt prik kan punktere det Carlsbergfondet-finansierede vrøvl, der kommer ud derfra), stadig mener, at Facebook er med til at skabe ekkokamre. Som han siger: »Ekkokamrene opstår simpelthen ved, at de, der i øvrigt er ligesindede, finder sammen i grupper og forstærker hinandens argumenter. Det er med til at polarisere den generelle debat«.

Jeg har som altid stor sympati for det synspunkt, at man skal være på vagt i forhold til nye tendenser i samfundet og de potentielle negative effekter, de kan have på den demokratiske samtale (hvis den da overhovedet eksisterer eller nogensinde har eksisteret). Der er bare ingen evidens for, at vi er vidne til, at sociale medier er i fuld sving med at ødelægge den generelle debat. De undersøgelser, jeg har været i stand til at finde, taler om noget for det modsatte.

I en undersøgelse foretaget af Richard Fletcher og Rasmus Kleis Nielsen, finder de, at sociale medier ikke fører til, at folk kun bliver eksponeret for information, der harmonerer med deres verdensopfattelse. Tværtimod. Som Rasmus Kleis Nielsen udlagde studiet til Mandag Morgens TjekDet: “Folk, der bruger sociale medier, bliver eksponeret for signifikant flere forskellige nyhedskilder end dem, der ikke bruger sociale medier.”

Dette er ligeledes konklusionen i en gennemgang af litteraturen foretaget af Andrew Guess, Benjamin Lyons, Brendan Nyhan og Jason Reifler, der bærer titlen ‘Avoiding the Echo Chamber about Echo Chambers: Why selective exposure to like-minded political news is less prevalent than you think’. I rapporten, der ligeledes opsummeres her, konkluderes det: “the data frequently contradict or at least complicate the “echo chambers” narrative”. Der gennemgås flere interessante studier, der ikke alene taler for, hvorfor vi ikke udelukkende ser ekkokamre, men også belyser hvilke mekanismer, der kan forklare hvorfor.

Der er flere interessante nye studier, der er værd at nævne i denne sammenhæng. Andrew Guess, en af forfatterne bag omtalte gennemgang af litteraturen, undersøger amerikanske borgeres nyhedsforbrug online og finder et betydeligt overlap i, hvilke medier borgere med forskellige politiske holdninger konsumerer. På den baggrund konkluderer han: “if online “echo chambers” exist, they are a reality for relatively few people who drive the traffic and priorities of the most partisan outlets.”

I et andet studie foretaget i forbindelse med den amerikanske valgkamp i 2016, vises det, at dem der fik nyheder via Facebook ikke var mere tilbøjelige til at være i et ekkokammer, da de blev eksponeret for forskellige typer af nyheder, herunder også nyheder der udfordrede deres politiske overbevisninger.

Dette harmonerer med konklusionen i et nyt studie, der undersøger hvilke nyheder algoritmer anbefaler via Google News. Nærværende studie finder, at folk med forskellige politiske overbevisninger får ensarterede nyheder og dermed ikke blot nyheder, der stemmer overens med deres egne overbevisninger. Altså igen ikke meget evidens for idéen omkring ekkokamre.

Det kan dog tænkes, at dette blot er for moderat politisk interesserede, og jo mere politisk interesseret man er, desto mere vil man søge væk fra udfordrende synspunkter. Dette argument udfordres dog af dette studie, ‘The echo chamber is overstated: the moderating effect of political interest and diverse media’, der viser, at især politisk interesserede ikke ender op i ekkokamre.

Før vi går videre er det værd at minde os selv om citatet fra Vincent F. Hendricks: »Ekkokamrene opstår simpelthen ved, at de, der i øvrigt er ligesindede, finder sammen i grupper og forstærker hinandens argumenter. Det er med til at polarisere den generelle debat« (min fremhævning). Jeg vil mene at ovenstående taler for, at der ikke er overbevisende evidens for, at vi lever i ekkokamre. Hvad vi dog også kan udfordre er argumentet om, at ekkokamre – skulle de eksistere – er med til at polarisere den generelle debat. Der er sågar noget der tyder på, at det modsatte kan være tilfældet.

I et nyt studie, ‘Exposure to opposing views on social media can increase political polarization’, foretager en række forskere et felteksperiment på Twitter, hvor brugere af forskellig politisk overbevisning bliver eksponeret for indhold, der enten konvergerer med deres egen overbevisning eller udfordrer samme. Som altid er der aspekter, der kan diskuteres ved studiets metode og resultater, men noget tyder på, at polariseringen var størst hos de brugere, der blev eksponeret for synspunkter i konflikt med deres egne.

Vi ved ikke hvad fremtiden bringer og hvordan sociale medier vil påvirke den politiske debat på længere sigt. Det ville dog være at foretrække, skulle forskere have interesse i at udtale sig om sociale medier og ekkokamre, at de også bringer en flig af evidens med til bordet i stedet for udelukkende at betro sig til egne overbevisninger.

Som det ser ud nu, er der intet der taler for, at sociale medier er ved at placere os alle i ekkokamre.

Did welfare reforms cause Brexit?

Did welfare policy reforms directly cause Brexit? That’s the conclusion in this article: “Tory austerity and welfare cuts directly caused Brexit, according to a ground-breaking new academic study.”

The results in the new academic study, ‘Did Austerity Cause Brexit?‘, “suggest that the EU referendum could have resulted in a Remain victory had it not been for a range of austerity-induced welfare reforms”. In this post, I outline why I do not believe this conclusion is warranted on the basis of the evidence presented in the study.

I should emphasise that I enjoyed reading the working paper and I sympathise with the empirical approach. The study is interested in a complex topic and combines multiple data sources in order to gather as much evidence as (humanly) possible. Accordingly, regardless of what you might think of my comments on the study, I can highly recommend reading it.

Brexit or UKIP?
While the study is interested in explaining Brexit, it is not really looking at Brexit. Instead, it focuses on the support for the UK Independence Party (UKIP), a party strongly in favour of leaving the European Union. The study presents a lot of results, including some interesting difference-in-difference estimates, but there is only a cross-sectional analysis on the determinants of people’s propensity to vote Leave (available in Table 6 in the manuscript).

There is nothing impressive about these cross-sectional models that provides strong evidence for the story that specific welfare reforms caused Brexit. While the models do not speak against such an interpretation per se, I do believe the manuscript is making too strong claims about Brexit on the basis of such an analysis. In other words, I would say that the manuscript is more about UKIP voting than Brexit. The author is just making it about Brexit as that’s the hot potato at the moment.

Model estimates and aggregate predictions
Despite the fact that the model estimates are interested in UKIP, the author makes the interpretation that three welfare reforms caused Leave to win the Brexit referendum: “Due to the tight link between UKIP vote shares and an area’s support for Leave, simple back of the envelope calculations suggest that Leave support in 2016 could have been up to 9.51 percentage points lower and thus, could have swung the referendum in favor of Remain.” (p. 3) A similar interpretation is available at the LSE British Politics blog: “Had austerity not happened, Leave support could have been up to 10% lower”.

I have three main issues with this interpretation. First, if the manuscript is interested in providing reliable estimates on whether or not welfare reforms caused Leave to win, I would like to see such an analysis be based on more than ‘simple back of the envelope calculations’. Or, alternatively, not be made with such confidence at all.

Second, there is no way I believe in an effect size of 10 percentage points lower support to Leave. From the research I have read on the topic, I know of no single study that find similar effect sizes of specific policy reforms on social outcomes, let alone election outcomes (more on this point below).

Third, while I understand the temptation to look for monocausal explanations of an inherently complex social outcome, the interpretations made in the manuscript are misleading. As my colleague Matthew Goodwin points out, people will read the paper and think “austerity is THE factor”. Such an interpretation, i.e. that austerity is the key driver, would be too simplistic and one should in general be cautious when taking specific model estimates and using them to make aggregate counterfactuals that are unrealistic overestimates at best and heavily misleading and wrong at worst.

What’s the treatment?
Due to a lot of moving parts in British politics, it is difficult to say that one specific policy (our “treatment”) had a causal effect on whether a person voted Leave or Remain. Accordingly, the key challenge is to identify the exact welfare reforms to study and ensure that no other factors related to the reforms drive the results.

In the manuscript, the author describes that he will “focus on three smaller welfare reforms – the abolishment of council tax benefit, the so-called ‘bedroom-tax’ and the introduction of Personal Independence Payments replacing Disability Living allowance” (p. 19). Well, three smaller welfare reforms. Everything is relative but this definitely adds to my skepticism, i.e. that specific welfare reforms caused the Leave vote to win.

Importantly, the manuscript also pays attention to aggregate data, but these analyses are more difficult to link to specific reforms. The question here is: What is the actual treatment!? Take the following two results from the paper on the correlates of UKIP support (after 2010):

  1. Human capital: “[…] support for UKIP gradually trends up as a function of the share of the resident population with low educational attainment. The correlation between support for UKIP and the measure of low human capital only becomes sharply stronger after 2010.” (p. 14)
  2. Employment shares: “Areas with larger employment shares in Retail, and Manufacturing saw significant increases in electoral support for UKIP after 2010. To get a sense of the magnitude, for the Manufacturing sector (ca. 15.4% of employment in 2001), the point estimate of 0.53 in 2015 suggests that the average area saw an expansion in support for UKIP by 2015 by 8.1 percentage points.” (p. 15)

These findings are not related to specific policies. However, the first empirical analysis in the manuscript concludes that UKIP gained “support after 2010 in areas with low skilled, working in routine jobs or the retail sector remain intact.” (p. 15)

I see no reason to question this finding, but human capital and employment shares are not only linked to austerity-induced welfare reforms but a series of factors. The interesting question is whether we would still believe the main findings (i.e. those based on the individual-level analysis) if we did not find a stronger correlation between support for UKIP and human capital only after 2010. This is not necessarily entering the garden of forking paths, but we are heading towards that territory. There is simply no convincing evidence for an effect of specific policies in this part of the analysis.

However, this points to two more serious issues with the inferences made in the study. First, we should question the parallel trend assumption in relation to the exposure to welfare reforms. The specific reforms are much more likely to affect poorer places (one of the studies cited in the manuscript shows that) and I see no reason to believe that nothing else happened in the poorer places that would not also correlate with UKIP support. In other words, a person that is more likely to be affected by austerity-induced welfare reforms is also more likely to be affected by other social, political and economic changes compared to a person not affected by the reforms.

Second, and returning to the point on the effect sizes, for the three reforms being studied, only 10% of the sample experienced one of the three reforms. This is not a problem as it is similar to the proportion in the population (10% of all UK households). However, if we are working with strong effects that could basically change the outcome of the referendum, this should mean that a nontrivial proportion of the affected group should change their vote choice to Leave (not taking differential turnout into account and assuming that they would vote Remain otherwise) as a result of the reforms – and not other factors.

What’s the mechanism?
This might come as a huge surprise for the economists reading this post, but here we go: Political scientists have been studying the impact of welfare policies on political behaviour for decades… (Including my recent study on the impact of welfare retrenchments on government support.)

This is my main issue with the study. There is too much going on data-wise with limited theoretical reflections and no engagement with the relevant literature in political science. One of the main findings in the political science literature is that there are multiple different mechanisms linking welfare policies to mass publics. There are no reflections on these different mechanisms in the manuscript. A good starting point to this literature is the seminal research by Paul Pierson from the early 1990s, including his book Dismantling the Welfare State? Reagan, Thatcher and the Politics of Retrenchment.

However, we do see the acknowledgement that there is something going on that might explain why welfare reforms are linked to the Brexit vote. Specifically, the study looks at “Perception of politics more broadly”, or as political scientists call it, political efficacy (the manuscript does not mention political efficacy at all). These measures include the perceptions that “Public officials do not care”, that respondents “Don’t have a say in what government does” and that “your vote is unlikely to make a difference” (p. 34). Again, there is a lot of relevant literature within political science on how welfare policies matter for political efficacy.

In a forthcoming article in Policy Studies Journal, I review 65 studies interested in estimating the impact of policies on mass publics. None of the 65 studies are cited in the manuscript. However, for people interested in the topic, I will recommend a few that are of interest in this context. First, for some studies using strong and interesting research designs to study how policies matter for political participation, see Davenport (2015), Flavin and Griffin (2009) and Flavin and Hartney (2015).

For the specific study of welfare reforms and the Brexit vote, three studies come to mind. First, Watson (2015) looks at the evolution of conditional welfare programs in Britain from 1996 to 2013 and uses panel data to trace the behaviour of individuals affected by these policies over time. Second, Gingrich (2014) looks at how welfare policies, and in particular how visible they are to the public, matter for whether people link their policy preferences to voting for a right-wing party. Third, in Spain, Muñoz et al. (2014) use panel data to study how an austerity package shaped political participation.

Last, for some additional studies interested in the impact of welfare policies on political efficacy and participation, see Guo and Ting (2015), MacLean (2011), Mettler and Stonecash (2008), Soss (1999), Soss and Schram (2007) and Swartz et al. (2009). All of these studies can help inform the manuscript in question on the mechanism(s) linking austerity-induced welfare reforms to the Brexit vote.

Economic versus cultural explanations
Altogether, this brings us to another issue with the manuscript, namely the focus on a single explanation for the Brexit vote rooted in economics. This issue goes beyond the specific manuscript and is symptomatic for a trend in the literature, i.e. to make it about a horserace between economic and cultural factors in understanding populist sentiments.

The literature is still in an early stage of disentangling the interplay between different factors, and it might as well be that the individual propensity to vote Remain was shaped by the interplay between economic and cultural factors. This is a possibility Pippa Norris outlines in a new working paper: “These theories can also be regarded as complimentary rather than rivals, for example if economic deprivation catalyzed resentment about immigrants and the rejection of open borders.”

Still, academics tend to reduce these complex questions to the simple horserace. A recent example is the new study in PPNAS by Diana Mutz, arguing that “Status threat, not economic hardship, explains the 2016 presidential vote”. I am sure the study is great (although see this paper), but simply ruling out the potential direct and/or indirect role of economic hardship based on a couple of regressions seems unfair to the theory.

This is why I am not convinced that the manuscript provides convincing evidence that the absence of specific welfare reforms would have resulted in a Leave win (the counterfactual claim). There is no data in the manuscript on immigration/culture that allows the author to study these alternative explanations or their respective role in shaping the Brexit vote. In other words, even if we believe in the results in the study (including the simple back of the envelope calculations), it is not possible to conclude that “austerity measures, not X” caused Brexit.

Last, the author of the study told HuffPost UK “that the withdrawal of welfare in poor areas allowed the Vote Leave campaign to “exploit” underlying worries about EU immigration and claims that billions were being spent on Brussels rather than at home in Britain.” While this is all plausible it yet again points to the fact that we are working with multiple potential mechanisms linking policies to political behaviour, and going for a simple economic explanation unrelated to any cultural factors is too simplistic.

Wrapping up
In my opinion, the evidence presented in the manuscript is not strong enough to make the point that welfare reforms caused Brexit. Did austerity-induced welfare reforms play a crucial role in British politics with implications for the Brexit referendum? Maybe, but I am not convinced.

The manuscript makes a good case for why we need to take welfare reforms serious in order to understand political behaviour, and I do believe there are reasons to expect that certain reforms made people, directly or/and indirectly, more likely to vote Leave. However, I have yet to see convincing evidence that “welfare cuts directly caused Brexit”.

Indlæg i Berlingske: Nedskæringer bliver ikke straffet af vælgerne

Har d.d. en kommentar i Berlingske med titlen “Nedskæringer bliver ikke straffet af vælgerne”. I kommentaren konkluderer jeg:

En kløgtig regering, der agerer strategisk i forhold til næste valg, skal dermed ikke holde sig fra nedskæringer, men tværtimod se dem som et politisk redskab til at bevare magten. Med andre ord kan regeringer trygt skære ned i de politikker, som ikke bruges af de vælgere, der vil stemme på dem – og bruge ressourcer på de politikker, som potentielle vælgere favoriserer.

Den kan ligeledes læses online her (kræver abonnement).

Er Socialdemokratiet gået tilbage siden folketingsvalget?

Hos Politiken kan man læse, at “Socialdemokratiet går tilbage i ny meningsmåling”. Artiklen bærer titlen “Efter enegang: Socialdemokratiet er gået tilbage siden folketingsvalget, viser ny måling”. Lad os kigge nærmere på den nye måling.

Som altid – når der er en ny måling – er det vigtigt at placere den i den rette kontekst. Ingen måling står sig godt ud alene, hvorfor Figur 1 viser Socialdemokratiets opbakning i målingerne fra 2018, hvor jeg ligeledes har angivet, hvilken der er den nyeste fra Megafon.

Figur 1: Socialdemokratiets opbakning i meningsmålingerne, 2018

Socialdemokratiet fik som bekendt 26,3% af stemmerne ved folketingsvalget i 2015. I omtrent alle målinger foretaget i år ligger Socialdemokratiet på niveauet omkring folketingsvalget eller højere. Der er ingen systematisk evidens for, at Socialdemokratiet er gået tilbage siden valget.

Det eneste sted vi finder denne historie er i en artikel om én måling fra Megafon. For et par år siden var jeg ude og kritisere Megafons målinger (og dækningen af samme) i forhold til Socialdemokraternes opbakning, og intet tyder på, at det er blevet meget bedre. Det kan undre mig, at journalister og politiske kommentatorer hopper i med begge ben.

Endnu mere interessant er det da også, at artiklen forsøger at koble denne tilbagegang på Mette Frederiksens udmelding om ikke at danne regering med Det Radikale Venstre: “Målingen kommer, efter at partiformand Mette Frederiksen annoncerede, at hun vil gå til valg på at danne en regering kun bestående af Socialdemokratiet. Dermed ønsker hun at droppe 25 års parløb med Det Radikale Venstre.”

Hvorfor er dette interessant? Fordi der ikke er nogen evidens for et statistisk signifikant fald i meningsmålingerne fra den forrige Megafon til den seneste fra samme institut. I den forrige måling fra Megafon (fra 31. maj) fik Socialdemokratiet 25,1% af stemmerne. Som altid kan jeg anbefale denne side, hvor du kan indtaste tal fra to målinger og få svar på, om der er en signifikant forskel mellem to målinger. Det er der ikke i nærværende tilfælde.

Artiklen hos Politiken afsluttes blandt andet med ordene: “Politiken har forsøgt at få en kommentar fra Nicolai Wammen, politisk ordfører for Socialdemokratiet. Han er ikke vendt tilbage”. Dette er der absolut intet at sige til, når det vedrører den slags jammerlige målinger fra Megafon.

Hvor vigtigt er klimaet for vælgerne?

I en ny meningsmåling foretaget for tænketanken Concito vises det, at klimaet er det vigtigste emne for vælgerne. Dette er dog ikke uden væsentlige forbehold.

Min første bekymring, da jeg hørte om meningsmålingen via en journalist fra Mandag Morgens TjekDet, var, at respondenterne i undersøgelsen nok var stillet andre spørgsmål om klimaet forud for spørgsmålet omkring, hvilket emne de fandt vigtigt. Dette ville føre til, at de ville finde klimaet vigtigere.

Det viste sig også at være korrekt, at respondenterne havde fået stillet spørgsmål omkring klimaet forud for det relevante spørgsmål. Dette gør at undersøgelsen ikke er retvisende, hvilket jeg har udtalt mig om sammen med andre forskere. Artiklen kan findes her.

Hvor mange vil stemme på Nye Borgerlige? #5

Udtaler mig hos Altinget omkring, hvordan det går Nye Borgerlige i meningsmålingerne. Det kan findes her. I artiklen, der desværre er bag en betalingsmur, citeres jeg blandt andet for:

“Når det er sagt, skal man altid være skeptisk, når nogle målinger er markant anderledes. I mange målinger ligger Nye Borgerlige omkring 2 procent, og derfor tror jeg, at 4-5 procent til partiet er for højt sat. Kort sagt er der langt mere evidens for, at Nye Borgerlige ligger tættere på spærregrænsen, end at de er langt over den.”

Tidligere indlæg om samme emne kan findes her, her, her og her.