Most of the papers I review for scientific journals are not accepted. However, there are several examples of good papers I reviewed that ended up getting accepted. I had a look at some of these reviews and in this post I bring a few examples. In all cases, none of my feedback is significant and crucial for the papers to address, and I was happy to see the papers getting published in the respective journals.
Policy feedback via economic behavior: A model and experimental analysis of consumption behavior (Policy Studies Journal)
This is an interesting manuscript on the policy feedback effects of targeted cash assistance policies on consumer spending behaviour (the “feed” effect) and government provision of basic utilities (the “back” effect). I generally like this paper and I am positively inclined towards recommending publishing the paper in a revised form.
There are three contributions of the manuscript to the literature, all acknowledged throughout the manuscript. First, the focus upon consumption behaviour within the policy feedback literature (as most studies look directly at political attitudes and behaviour). Second, the focus on Mexico (as most studies on policy feedback effects are from the U.S.). Third, the focus on not only the short-term policy feedback effects but also the medium-term effects (studies tend to look at only one of these).
The theoretical argument is well-developed and provides a convincing case for studying consumption behaviour in the context of policy feedback effects. That being said, I believe the insights could be stronger linked to – and discuss – the implications for the findings and focus in the policy feedback literature. Specifically, the focus in the policy feedback literature, especially the literature on policy feedback effects on mass publics, is on how policies matter for the support of politicians and parties. Does the model introduced in this manuscript have implications for governments and politicians? We know that consumption behaviour (and economic factors) matter for government support, and can the insights presented in the manuscript speak stronger towards this? If so, I believe the paper will make a stronger contribution to the policy feedback literature. I agree that it is important to study the case from an economic development perspective, and that we should consider the importance of non-political feeds for mass policy feedbacks, but I would prefer to see more of the policy feedback literature being picked up again in the discussion.
The policy feedback effect model summarised in Figure 1 is helpful to understand the mechanisms linking policies to subsequent policy-making. I suggest that the author considers presenting a similar figure in the data section of the manuscript that makes it more clear how the data from 1995, 2000, and 2005 speak to the different levels of the model. Specifically, I did find it slightly difficult to follow the empirical case. A figure showing how the data and analysis is going to shed light on the theoretical model would make it easier for the reader to follow the case and the implications of the findings for the model.
It also required a lot of attention to fully understand the different variables, e.g., septic drainage, pipes (indoor and outdoor), and communal taps. I recommend that the author provides a table (or figure) showing how the different measurements are linked to the theoretical concepts used in the model. This will make it a lot easier to interpret the regression coefficients in the analysis. I did find it difficult to follow the logic throughout all stages of the analysis.
The discussion should preferably begin with discussing the main finding of the article, and not a discussion of a mismatch in the effects. Related to this, it is difficult to assess the overall finding of the manuscript with the mismatch in the effects. What are the implications of the medium-term effects for how we interpret the short-term effects, and vice versa? Ideally, the discussion should make it easy for the reader to answer these questions.
Again, my main comment with the empirical setup is that it can be difficult to follow at times (especially with no prior experience of the experiment and the case). I also found it difficult to follow the difference-in-difference logic of the design, and in particular to assess how strong the difference-in-difference estimator is in this setting. It might, for example, help if there was an explicit discussion of the parallel trends assumption at the different time points of the analysis.
As a minor comment to the analysis is that I suggest the author, when conducting randomisation tests, rely on Bonferroni correction, to assess whether 2 out of the 44 tests are still statistically significant (they should not be).
In sum, I find the manuscript great and I believe it is of publishable quality upon taking my concerns introduced above into account.
Getting under the Skin: The Impact of Terrorist Attacks on Native and Immigrant Sentiment (Social Forces)
This is an interesting and relevant manuscript on the impact of three terrorist attacks in Nice, Würzburg and Ansbach in 2016 on a series of outcomes among the general public as well as refugees and asylum seekers. I fully agree with the author that a focus on how minority groups respond to terrorist attacks is lacking in the literature. Accordingly, the author provides a good overview of the literature and positions the contribution of the manuscript well within this literature.
That being said, I have some suggestions and concerns I would like to see considered. While the focus on refugees and asylum seekers is important, I would like to see a stronger connection to the results in the ‘German sample’. This is especially relevant as the two surveys rely on different outcomes. For example, if the German sample had showed nothing but null results, should that shape our interpretations of the results in the refugee sample? Hypothetically, if anti-refugee sentiments had decreased in the wake of the attack, would we then expect to find other effects in the refugee sample? Ideally, I would prefer for the theoretical framework to answer these questions.
The framing of the manuscript focusing on ‘in- and out-group sentiment’ gave me the impression that we would be able to compare similar sentiments across both in- and out-groups. However, the focus on distinct outcome measures in the two samples makes it more difficult to compare the outcomes. One possibility is for the author to discuss and compare the different outcomes in greater detail and maybe see how they correlate with similar measures that might be present in both datasets? In other words, a greater focus on construct validity would ideally provide a stronger connection between the two samples.
For the structure of the manuscript, I would like for the author to consider a table that can introduce and compare the two datasets. For example, a column for each dataset with rows introducing 1) data collection period, 2) sample size(s) and 3) sample composition.
For the design and data, the manuscript follows a logic that is well-established in the literature now. However, there are a few issues here that I slightly concerned about. For the three events, i.e. Nice, Würzburg and Ansbach, we are looking at a compound treatment with events spanning 10 days. Nice was by far the most impactful (though taking place in France) followed by the non-deadly events in Würzburg and Ansbach. Should we think of this as one big treatment or three small but different treatments? The answer to this question has implications for how we should understand the effects and in particular the generalizability of the findings. My sense is that we should understand this as one treatment. Preferably (from a methodological view point!), we would like the three events to take place in the same day to have a strong treatment with little ambiguity in relation to the delivery or the nature of the treatment. How many respondents, for example, ‘experienced’ Ansbach? And did they experience Ansbach as a small, isolated event or directly in the narrative of the two other events? I do not expect that the manuscript can (or should) provide answers to all of these questions, but I would like to see a discussion picking up at these issues and discussing how the findings speak to the existing literature.
Related to this: Four weeks is a long time, especially around July 14th 2016. It’s important to keep in mind that a lot happened in this period. For example, we had the Brexit referendum on June 23rd 2016 that shaped a lot of the political discourse in this period, also in Germany. One alternative explanation of the results could be that people in the control group felt more positive towards refugees and asylum seekers following Brexit and the effects we see are merely these effects fading out (and not solely the results of the terrorist attacks). I do not see this as a major limitation of the study but it is something I would like to see addressed or at least mentioned (from what I could see there is no mention of other events in the context beyond the Munich shooting in the treatment period). Given the fact that a lot of respondents were interviewed in April/May, it should be possible to examine a “Brexit effect” as well and rule out this explanation. I do not expect this to go into the main manuscript but could be addressed in an appendix (or a response letter).
There are big differences in the reachability between the control and treatment group in both samples (i.e. the number of times each respondent was contacted). This is often the case in these types of studies (where the treatment is correlated with time). However, I am concerned that reachability will correlate with trust. Might it be that people that are more likely to have negative emotions will require more contacts before they will participate in the survey? I am not convinced that controlling (or balancing out these differences) will fully shed light on this issue. I have two suggestions here. First, provide bivariate correlations between all measures (outcomes and independent variables) in the samples (what is, for example, the correlation between reachability and negative emotions?). Second, if there is sufficient data, estimate the main models only with respondents that were (relatively) easy to reach.
For the measures on emotions; pity, fear, anger and affection are different emotions and I would like to see more details on the principal component analysis (e.g. in the Appendix). Specifically, I find it difficult to think theoretically about how people should feel more or less pity as a result of the terrorist attack (and this finding is also insignificant). A scree plot confirming that we are working with a single component would also be good here (again, in the Appendix).
Again, I believe the primary (comparative) advantage of this manuscript is the survey of refugees. That being said, the use of this data requires more information than we normally expect from nationally representative surveys. Accordingly, I would like to see additional information on the survey of refugees. Is it representative? And if so, representative for what group? How exactly was the population frame developed? Are there specific possibilities in Germany to collect this data that would make it difficult to replicate a refugee survey in other contexts? These questions are not rhetorical and I find it important to provide further info on the data collection (or provide explicit references to where the reader can find more information).
The discussion in the ‘Conclusion’ section is brief and I would address some of the points above here as well. One limitation with survey data mentioned here is that ‘talk is cheap’ but that the triangulation of multiple surveys addressed this problem. I might have misunderstood the point here but I am unable to see how the triangulation of different surveys help overcome this limitation without the use of behavioural measures. I suggest the author rephrase the point here to ensure that nobody will misunderstand the point.
In sum, I believe that the manuscript with some specific revisions will be of great interest to a wide audience, including the readers of Social Forces.
Poll Wars: Perceptions of Poll Credibility and Voting Behaviour (International Journal of Press/Politics)
This is an interesting manuscript on how the perceptions of poll credibility matter for voting behaviour. The manuscript first shows, using a conjoint experiment, that the poll results and the distributing newspaper affect the perceived credibility of an opinion poll. Using these results, the researchers conduct a survey experiment to examine how the perceived credibility matter for vote choice and turnout.
The main finding is that credible, non-credible and mixed (i.e., both credible and non-credible) polls lead to lower turnout, relative to a control group, although the credible polls lead to a greater turnout compared to the non-credible polls. Overall, I believe the two studies are well-executed and show the advantages of using state-of-the-art techniques to study the causal impact of opinion polls on voting behaviour.
That being said, I have some concerns with what we can conclude based upon the manuscript in terms of how opinion polls matter for vote choice and turnout outside the experiment (i.e., “in the real world”).
To illustrate my concern, when interpreting the results for turnout, the authors write: “Rather than considering the methodology and reputation of the poll, it seems plausible that individuals may be disincentivised to turn out if the polls suggest a result contrary to their beliefs or wishes” (page 25). However, the participants in the studies are not faced with information about the methodology being used in the opinion poll. In other words, the inferences we make – or the choice the respondent makes – will be in the absence of any other information (or, at best, all else equal).
In doing this, the study follows a logic similar to Madson and Hillygus (2020), cited in the main text, where the methodological information is held constant between the conditions. However, in a conjoint analysis where the respondents are forced to choose between two polls, I am not surprised that people are more likely to pick the poll that shows their own party ahead, when they had to pick one. The interesting counterfactual question is whether they would have done the same if presented with information about the poll showing their party ahead was of an actual lower quality. Kuru et al. (2017), also cited in the main text, for example, manipulated whether respondents had access to methodological details (margin of error, sampling mode, sample size, subsample statistics, response rate, and question wording). Without such a manipulation, I am not convinced that the authors can make strong conclusions about what people do instead of considering the methodology of a poll.
This is particularly relevant taking the recent study by Kuru et al. (2020) into account. They find that when presented with two conflicting polls, participants are able to identify the best poll in terms the quality of the methodology and therefore not simply picking the poll that is best aligned with their own preferences as the most credible. Again, I am not surprised that, when only presented with information on the results of the poll, the polling vendor, and the distributing newspaper, and then forced to pick the most credible, the results of the poll will matter for the perceived credibility. This might not have significant implications for the interpretation of the findings, but I would like to see the authors pay more attention to this issue. For example, when the main conclusion in the abstract is “that polls perceived not to be credible – and conflicting poll environments – can substantially decrease the likelihood of an individual turning out to vote”, we should consider whether these polls are actually not credible. Might it simply be that these polls are not found less credible, but that participants have a stronger preference for certain polls all else equal?
This is especially relevant if we are to understand what can explain the findings. The literature on poll effects is mostly interested in the impact on voting for specific parties and less so on turnout effects. However, some studies have devoted attention to this (as reviewed by Barnfield 2020, cited in the main text). This literature is only briefly mentioned in relation to Hypothesis 3 but I miss a greater discussion on the mechanism and in particular what it is about the credibility of polls (or lack hereof) that can lead to a lower turnout.
My concern with the setup in Study II is that it will be difficult to generalize these findings outside the experimental setup. The participants are asked to “imagine that there are local elections tomorrow”, and the control group is then asked “In this situation, how likely would you be to turn out to vote?” on a seven-point scale from 1 (extremely unlikely) to 7 (extremely likely). For the control group, this question is more identical to a traditional poll question asking about vote preference if there was an election tomorrow. This can explain why the average value in the control group for the turnout question is high. However, for the three experimental conditions, the material talk about “hypothetical pre-election poll(s) for that election” which might be the reason why people are less likely to vote across these conditions. Can a reason why it is difficult to find a bandwagon effect be that the respondents are told that the information is not real?
In sum, I am not confident we can conclude that 1) poll credibility does not matter for the bandwagon effect or/and 2) poll credibility matter for turnout. While the authors do a good job in setting up the study, my concern is that it is limited what we can conclude about the relevance of the coverage of opinion polls for electoral behaviour. Accordingly, I do not see the current setup and findings providing a significant contribution to the literature.
The context of the Turkish municipal elections seems sensible for the study. The authors argue that Turkey serves a crucial case “due to the highly polarised bi-partisan nature of political and media discourse”. However, I am not sure how I should understand this with the multi-party system and in light of how the authors conclude that some of the findings could suggest in-bloc bandwagon effects. Also, one can think of other countries with “high stakes competitive elections where political discourse is largely defined by an increasingly polarised bipartisan divide”. It would be good to see some additional thoughts on how these findings are expected to generalize (or not) to other countries. Noteworthy, most of the studies in the literature are from the United States or Western Europe, with one of the few exceptions interested in these dynamics being the study by Chia and Chang (2017) in Taiwan. I believe the authors can highlight the need for additional studies from countries such as the study in question. Last, what did the most credible polls show at the time of the survey? Can this explain the difference between what polls government and opposition supporters find credible (with opposition supporters paying more attention to the distributor of information).
To conclude, while the findings speak to a growing body of literature on how partisan motivated reasoning matter for the processing of opinion polls, I have some concerns with the validity and relevance of the findings to shed light on how poll credibility matters for both vote choice and turnout.
References
Chia, S. C., and T. Chang. 2017. Not my horse: Voter preferences, media sources, and hostile poll reports in election campaigns. International Journal of Public Opinion Research 29(1): 23- 45.
Kuru, O., J. Pasek, and M. W. Traugott. 2020. When polls disagree: How competitive results and methodological quality shape partisan perceptions of polls and electoral predictions. International Journal of Public Opinion Research 32(3): 586-603.
Did Terrorism Affect Voting in the Brexit Referendum? (British Journal of Political Science)
This is a novel manuscript on the impact of terrorism on support for the EU. The manuscript is ambitious and sets out to shed light on whether the Brexit vote was affected by domestic terrorist events. The authors do a great job in using different data sources to test the hypotheses and find consistent evidence that people in areas with more terrorism are more likely to possess pro-EU attitudes.
Noteworthy, this is the second time I review this manuscript and I can see that the authors have taking several of my suggestions into account. I have read the manuscript again and removed my suggestions from my previous review that are no longer relevant. In addition, I have elaborated more on a few points to make my existing concerns more explicit. My apologies if any of the comments listed below are no longer relevant.
There is still something about the framing of the manuscript that makes it difficult to easily understand the expected impact of terrorism on EU support. Specifically, the manuscript describes that terrorism was a “top concern by the British public, more than in any other European country”. However, if this is the case, why do we see the British public being so Eurosceptic? Or, in this context, why Brexit? This is not directly relevant for the theory or the empirics, but something I would consider for the framing of the manuscript. Upon my second reading of the manuscript, I believe this is even relevant for the question asked in the title of the manuscript, i.e. ‘Did terrorism affect the Brexit vote?’. If I understand the interpretations provided in the manuscript correctly, we should have seen _more_ terrorist attacks if these attacks should have affected the outcome of the referendum. My concern is that people – upon seeing the framing of the question – will infer that terrorism could be a cause of the Brexit vote (i.e. a Leave vote). The (counterfactual) implication of the findings is, all else equal, that without terrorist attacks in the UK, we would have seen a bigger win for the Leave vote.
For the research design using aggregate data, while the timing of a terrorist event is as-if random, distance from a terrorist attack is not. Proximity to major cities is the best predictor of terrorist attacks (based on Table A2 it looks like education and population density are the variables contributing to the high R^2 in the empirical models) and I am not convinced that population density captures all relevant variation in geographical proximity. The authors rely on different measures of proximity, but I would like to see a series of tests using various measures that rule out terrorist attacks as a proxy for proximity to big cities. These will of course correlate with population density but additional robustness tests would make the findings more credible. Another reason for this suggestion is the finding in Figure A5, i.e. that Scotland drives the findings and the exclusion of these data points make the findings insignificant. It might be that some of the new tests provided in the appendix (such as the identification tests in Appendix B.2) addresses this, and in that case, I would like to see a description on how.
The findings are not easy to interpret as the parameter of interest is distance from the event of interest. Specifically, the effect is negative indicating that as we get further away from terrorist attacks, the support for Remain decreases. The interpretation here is that as we get closer to a terrorist attack, the support for Remain increases. This is not necessarily an issue but when we get to the interaction results it gets a lot more difficult to intuitively interpret the negative interaction effect between distance and high media coverage. The authors conclude that “Substantively, the estimates reveal that the distance-induced Remain effect is at least three times as large when the attacks are extensively covered by media.” (page 18). I suggest that the tables with the interaction analyses are moved to the Online Appendix and, instead, plots with marginal effects are introduced in the main text. This will make it easier for the reader to interpret the findings and compare the marginal effects of distance to terrorist attacks (the “treatment”) at different levels of the conditioning variables.
The matching analysis is a great addition to the analysis in order to test the robustness of the findings when ensuring a better overlap and balance between the “treatment” and “control” cases. However, I would like to see the authors go into greater detail with what exact cases are matched to each other. Specifically, I find it difficult to imagine a reliable control case (counterfactual) for London. In other words, what district in the sample not experiencing terrorist events is a good and realistic counterfactual to London? When matching is applied to individual-level data (e.g. two respondents in a survey), we normally do not care about the exact two individuals beyond the exposure to the treatment, but when looking at aggregate data, I believe that the authors should go into greater detail with the exact districts that are matched to each other and how that leads to more credible inferences.
The analysis of the wards is also relevant but I did not see any good arguments in the manuscript for why this analysis was limited to the 367 wards located in the 19 terrorist-hit districts. The 1,261 wards in the sample should all have a distance from the nearest terrorist attack. I suggest that the authors either 1) use all 1,261 wards in the analysis and model the impact of the events or 2) provide an explicit reason for the omission of these data points.
The individual-level analysis is a novel addition to the manuscript and I applaud the authors for going to such great lengths in order to increase the validity of the inferences in the manuscript. That being said, I do believe there is a discrepancy between the focus on distance in the aggregate and individual-level analysis. If distance is of importance for the identifying assumption in the aggregate analysis, why not use proximity information in the individual-level data as well? At the moment, there is only a subsample test (with no formal test of whether these effects are significantly greater).
For the individual-level analysis, I encourage the authors to provide more information on the groups and in particular the group sizes. The 72,828 respondents included in the analysis with a lot of untreated cases (i.e. respondents interviewed prior to the event) should provide a lot of possibilities to test the credibility of the effects. I suggest three possible ways to do this. First, to use similar matching strategies as for the aggregate analysis. Second, to be more explicit about when the respondents are interviewed and zoom in on the few days before and after the attack. Third, to leverage the panel component of the data. Given the sample size, a lot of respondents in the sample must in one wave be interviewed prior to an attack and after an attack in a second wave. If the authors are able to provide within-subject variation due to the interview timing, that would rule out a lot of potential confounders.
In the tables, consider reporting standard errors in the parentheses (instead of p-values). It is easier to divide by two than having to get a sense of the standard errors based on the p-values (especially when they are less than .001).
Last, I like the new Appendix C.6, “Accounting for pre-existing trends”, and I would consider also trying to regress the pro-EU question in a previous survey wave to show that there is no significant effect of the event. This would be a strong test to show that it is indeed the impact of the terrorist events.
There is no doubt that this is a well-crafted project and I believe that the theoretical insights and empirical findings will be of interest to a wide audience (also outside academic circles). I also see specific improvements to the paper since I reviewed it for another journal.
Think Twice before Jumping on the Bandwagon: Clarifying Concepts in Research on the Bandwagon Effect (Political Studies Review)
This is an interesting manuscript on the concept of bandwagon effects on public opinion. The manuscript makes a strong case for the importance of additional theoretical scrutiny of the concept of bandwagon effects. Without paying sufficient theoretical and methodological attention to the challenges in studying bandwagon effects, we are unable to link different empirical findings in the literature and fully understand how and when information matter for people’s propensity to change their support for politicians and parties.
While I see several merits in the manuscript, I have specific concerns in relation to the manuscript.
First, despite the interesting arguments and points raised in the manuscript, I lacked a basic introduction to what is at stake in the literature. The manuscript argues that by adopting the typology proposed in the manuscript, “scholars will make their contributions clearer, rather than offering up empirical evidence of an elusive concept and struggling to situate this evidence theoretically.” I do not necessarily disagree with this contribution, but the objective of the review should ideally be to situate the evidence and help the reader understand the evidence in the literature in the light of the typology. In other words, I do believe the typology should be put to better use in the manuscript than to help scholars in the future.
Second, and related, I was immediately interested in a review of the studies on bandwagon effects, but I found the actual review brief and casual. Rather than introducing the reader to a big literature, the examples are anecdotical and it is not clear how the review should alter my understanding of the reviewed studies. Accordingly, there is no systematic presentation of the literature in the main text (the only classification of the studies is provided in the Appendix). I suggest that the author devotes significantly more attention to 1) presenting the literature in a systematic manner (with tables and figures) and 2) outline and discuss how exactly we should understand the studies in light of the typology.
Third, for the typology, the theoretical starting point is that people go with “the most popular, or an increasingly popular preference”. Independent of whether a person goes with the most popular or an increasingly popular preference, that person is making a comparison, and the relevant question is what type of comparison is most salient. It might be a comparison to another option (e.g. support for another party) or a temporal comparison (e.g. support for the same party a month ago). I believe there is a potential to have a more interesting theoretical discussion of the information-processing of opinion polls, how they are framed and how they might differ across countries. We would expect that such comparisons are more difficult to make in certain countries, especially in countries with multiparty systems, that might have significant implications for how and when people rely on information that could factor into them updating their preferences. If the author could elaborate on such aspects and show how this (or other elements) have implications for the findings in the literature, the contribution of the review would be much stronger.
Fourth, when specific studies are discussed in relation to the typology, it is not clear what the implications are for the literature – or, specifically, what is at stake. For example, when the author describes that “Some experimental studies have used a single treatment emphasising both static and dynamic popularity, but this precludes theoretically relevant distinctions about which of these factors is of importance (e.g. Nadeau et al., 1993)” (p. 7), it is not clear what the lack of such a distinction will have for the literature. The author concludes that studies “should be designed to isolate this”, but what is the main limitation in not doing this?
Fifth, an important distinction in the manuscript is between conversation and mobilisation. I would suggest that the importance of this distinction is made clear in the introduction as well. Now, it is simply introduced in the introduction but it is not clear from the beginning why we should care about this distinction. This should be easily fixed with an added paragraph in the introduction.
Sixth, I like the discussion on the Titanic effect and the challenges in isolating the bandwagon effect. However, I felt that there was a potential for going into more detail with the psychological mechanisms that could be at play here. One suggestion could be to discuss the potential for a negativity bias, e.g. that increasing unpopularity will matter more for people’s preferences than increasing popularity.
Seventh, there was a lack of real-world examples and illustrations throughout the manuscript. This made it difficult to understand the relevance of all distinctions and points made throughout the manuscript. It would increase the relevance of the manuscript to apply the typology to a case and outline what we should see in the different cases and how that would have significant implications for our understanding of bandwagon effects. An example that comes to mind is the 2016 presidential election between Hillary Clinton and Donald J. Trump. Ideally, the author should be able to discuss theoretically distinct processes in relation to those candidates (e.g. how a static bandwagon effect should predict a bandwagon effect from information about opinion polls showing majority support for Hillary Clinton and a dynamic perspective should predict a bandwagon effect for Donald Trump). This would also enable a discussion of how potential underdog effects, strategic voting dynamics (people voting for a third candidate in the election) etc. could relate to the bandwagon effect.
In sum, while I see several interesting contributions in the manuscript, I do believe that the actual review of the literature should take up a much larger role in the manuscript to be of interest to readers of the Political Studies Review. In its current form, the focus is predominantly on the development of the typology and a discussion of the different mechanisms, but if this can be linked to a more systematic review of the literature (with additional tables and figures), I see a good and solid contribution to the literature on bandwagon effects.
Minor comments:
– Make changes to Table 1. First, add a title. In all cells in the table, it says “Exposure to new information about the distribution of preferences across the electorate induces an individual to”. Consider removing this or making it part of the title.
– There is no need to have the “(The rest were unclear.)” (page 10) in a parenthesis.