Praveen Chakravarty
September 17, 2015
There is an opinion poll blitzkrieg in the run-up to every Indian election. But an analysis of 82 electoral surveys since 1996 shows that not one has been fully accurate
Man’s fascination for predicting the future is an ancient weakness. Thomas Malthus’ disastrous forecast of the demise of the human race couched in scientific logic was made 60 years before Charles Darwin could explain the very evolution of that human race. This temptation to predict, albeit with an ignominious track record of doing so, continues unabated. It is no surprise that our own electoral prediction season has kicked off yet again in its characteristic pomp and splendour with the sounding of the Bihar poll bugle. In true Malthusian tradition, these electoral surveys are often wrong, but unlike Malthus, unaccountable and unashamed yet.
Partial accuracy
Of the 82 electoral surveys that surveyed more than 1.25 million voters for 13 Lok Sabha and State Assembly elections since 1996 that we analysed, zero (yes, zero) have been fully accurate and 15 have been partially accurate. We define “fully accurate” as being within a +/-5 per cent range in terms of seats predicted vis-à-vis actual results for both the winner and the runner-up. The 5 per cent test is a fairly standard statistical norm to determine if a certain outcome is purely by chance or by design. We classify partial accuracy as those that predicted either the winner or the runner-up but not both, within the range of 5 per cent. If we relax the +/-5 per cent accuracy levels to 10 per cent, then 9 out of 82 surveys have been fully accurate and 27, partially accurate. Accuracy levels of these surveys show no signs of an improving trend since 1996 though sample sizes and number of surveys have increased substantially. Similarly, there is no evidence of these surveys being able to predict a State Assembly election better than a Lok Sabha election or vice-versa. Neither is there proof of any specific survey organisation(s) getting their predictions consistently right vis-à-vis others. Accuracy levels of these poll surveys are understandably as random as electoral outcomes. Thus, at best, one in three poll surveys over the last two decades have been partially accurate and at worst, zero has been fully accurate. Either way, they are of no Nostradamical proportions. Yet, over the next few weeks, we will be inundated with an opinion poll blitzkrieg in the run-up to the five phase Bihar elections. Many will tout a successful track record by claiming credit for predicting a winner correctly in previous elections, which in most bi-polar alliance situations, is a simple 50/50 chance, akin to a coin toss. Many others will claim use of more sophisticated techniques such as stratified random sampling, better vote share to seat share predictive models etc. The naked truth is 7 out of 10 polls are likely to get it wrong, as per our analysis and definitions of accuracy.
We compiled from various sources, information on electoral surveys since 1996 of both Lok Sabha and State Assembly elections (raw data at www.idfcinstitute.org). Of the 82 surveys, 48 were pan-India surveys for six Lok Sabha elections since 1996, and 34 for Assembly elections of Rajasthan, Madhya Pradesh, Bihar and Delhi. Only 41 out of these 82 disclosed their sample sizes and methodology. Many pollsters acknowledge the hopelessness of predicting seats accurately even if one predicts vote shares correct. While psephology may be an established science in other electoral democracies, India’s complex plurality mixed with an inappropriate first-past-the-post electoral system and a generic human fallibility for voting along identity lines renders this science useless in the Indian context. A multi-cornered contest with no requirement for winners to secure at least 50 per cent of the votes makes electoral seat predictions a futile exercise in India. Not only do opinion polls amplified by their noisy media partners routinely get it wrong, but they have the potential to inflict damage to the very notion of a free and fair election.
Opinion polls play an inordinately profound role in shaping perceptions of likely voters. The average Indian voter is generally deemed to be vulnerable. This prompted the Election Commission recently to include photographs of candidates on the ballot to prevent confusion from multiple similar names. In this context, it is not difficult to fathom the potential for inaccurate and misleading opinion polls to sway voters. It is not possible for the average voter to distinguish between deliberate or unintended imprecision of these opinion polls, i.e. sins of omission versus commission.
Though bans are in vogue these days, this is not to imply a ban on election surveys. Perhaps, we can start by issuing some basic guidelines for publishing opinion polls by requiring pollsters to disclose their past accuracy track record, as per a standard definition. It is similar to airlines being asked to publish their on-time performance data periodically. It is imperative we acknowledge that opinion polls have a definitive impact (intended or otherwise) on voter behaviour, especially in a largely illiterate democracy such as ours. Publishing accuracy track record and making voters aware of this can be a baby step in the direction to restrict the abilities of opinion polls to impinge on fair electioneering. To be sure, the increasing complexity of electoral outcome predictions is not unique to India as we witnessed in the recent U.K. elections. What is unique is the size and complexity of the diversity of India’s voter base and the relatively lower education levels of the average voter. To curb infringement on fair electioneering, it is necessary to build accountability and transparency to the opinion poll process. Creating a measurement framework and making publicly available, accuracy data of opinion polls can be an important first step in this process.
(Praveen Chakravarthy is at IDFC Institute and a founder of the data journalism website, IndiaSpend.)
https://www.thehindu.com/opinion/op-ed/predictions-that-never-come-true/article7659933.ece