Ergodicity and weighted averages of opinion polls

The more likely it is that a weighted average of opinion polls is wrong, the more we need it. If several opinion polls show the exact same numbers, a weighted average will – all else equal – be more correct than if the opinion polls differ a lot, but such an average is not really needed. If the polls differ a lot, and beyond what can be attributed to random sampling, a weighted average is needed, despite the fact that the estimate from such an average is more likely to be incorrect.

Conducting an opinion poll is an ergodic system if 100 polling firms each conducting one opinion poll will provide the same results as one polling firm conducting 100 opinion polls. That is, there is no difference between the 100 results from one polling firm and the individual results from 100 polling firms. Practical limitations aside, we are dealing with an ergodic system if there are no house effects. (The greater the differences in the results from different polling firms, the greater the house effects.)

The greater the house effects, the greater the non-ergodicity. If a polling firm with a tendency to systematically overestimate the support of a specific party conducts 100 opinion polls, this bias will be greater than if 100 polling firms with a smaller bias each conduct an opinion poll. In practise, as we often see non-trivial house effects, we are dealing with non-ergodicity. The greater the house effects, the more we need weighted averages – but the more likely it is that the polls are getting it wrong.

In other words, if we are operating with an ergodic system, an average of the opinion polls will be consistent with the results of a single opinion poll (i.e., we do not need averages to make sense of all opinion polls). Importantly, this is not the same as the opinion polls will be correct as all opinion polls might be biased. We need quality-adjusted, weighted averages when we are dealing with non-ergodic systems; not because a weighted average will be more precise than all individual polls, but beceause we put more trust in an average of different polls than the results from a single poll.

The more we need a weighted average of opinion polls to make sense of what opinion polls show, the greater the likelihood that the weighted average is wrong. If we did not need a weighted average at all, the polls would most likely all be correct (or equally wrong in a systematic manner that a weighted average would not adjust). The reason we still like to use a weighted average is that we are, on average, less wrong, compared to if we just put confidence in a random poll.

For media outlets reporting opinion polls, there is an important implication. If it is relevant to report a weighted average of opinion polls, it is also relevant to report the variation in the estimates going into the weighted average. And the greater the variation in individual polls, the greater the importance of reporting the limitations of weighted averages.