Pre-election polls: Do they work, do they influence voters, and shouldthey be banned?

by Rajeeva Karandikar, Director, Chennai Mathematical Institute.

I have been a psephologist for about 16 years and have had a fair amount of success in predicting seats in an upcoming election. Here is a post-mortem of what I said and what happened in the current round of elections.

From 2005, I have been working with CNN-IBN and Centre for Studies in Developing Societies (CSDS). CNN-IBN engages CSDS for the survey. CSDS does a great job of running surveys `by the book'. I use vote share data from the survey to come out with seat projections, which CNN-IBN carries on air.

How did we fare in the recent state elections?

Seat count predictions on air (CNN-IBN)

	BJP	Congress	AAP	Others
Madhya Pradesh	136-146	67-77	-	13-21
Rajasthan	126-136	49-57	-	12-20
Chhatisgarh	45-55	32-40	-	7-13
Delhi	32-42	9-17	13-21	1-5

The outcome

	BJP	Congress	AAP	Others
Madhya Pradesh	165	58	-	7
Rajasthan	162	21	-	16
Chhatisgarh	49	39	-	2
Delhi	31	8	28	3

These results show the power and the limitations of opinion poll based projections. If one simply counts the number of cases out of 13 that the actual results are within the interval projected, the score is just 4. However, one should see these as having correctly predicted clear and decisive victories for the BJP in Rajasthan and Madhya Pradesh. For Chhatisgarh, we predicted correctly that the BJP will win, with a much smaller gap than Madhya Pradesh and Rajasthan. So I would count all the three as being good predictions, with Chhatisgarh being very good.

As for Delhi, we underestimated AAP support and marginally overestimated BJP but we had the ordering right: Congress in third position with the possibility of touching a single digit, and BJP as the single largest party.

Vote share estimates on air (CNN-IBN)

	BJP	Congress	AAP	Others
Madhya Pradesh	41%	35%	-	24%
Rajasthan	43%	33%	-	24%
Chhatisgarh	42%	38%	-	20%
Delhi	33%	23%	27%	17%

The outcome

	BJP	Congress	AAP	Others
Madhya Pradesh	44.9%	36.4%	-	18.7%
Rajasthan	45.1%	33%	-	21.9%
Chhatisgarh	41%	40.3%	-	18.7%
Delhi	34%	24.5%	29.5%	12%

Here, the survey has worked well, and the errors are generally within the statistically acceptable range. The conversion from votes to seats is a non trivial transformation - since one needs to estimate the distribution of votes across the state in addition to the overall percentage of votes in the state. This requires building an appropriate statistical model. I will explain my methodology for this stage in a future article.

One can see that in MP and Rajasthan, there was underestimation of BJP votes by a few percentage points. The error in vote to seat conversion went in the same direction, and as a result our prediction was much lower than the outcome for the seats obtained by BJP. In Chhattisgarh on the other hand, the survey estimated the gap between BJP and Congress as 4% while the actual gap turned out to be less than one percent. In this case, the error in vote to seat conversion and the error in the vote share estimate cancelled out, and we got a result that was bang on. Of the four, I was the least confident about Chhattisgarh (and I had said so on air) since the estimated gap in vote shares was small.

In my experience, the predictive power of any opinion poll that is done a while before voting is rather poor. For one, any such poll can only measure the mood of state or nation at the time of poll and cannot estimate the potential change that can happen close to the voting day. Some psephologists claim to estimate this change by conducting polls at regular intervals and then extrapolate to get an estimate of this change. However, this assumes there are linear time trends in vote share, which is unlikely.

The other problem is the selection process that determines who in the general population (that is sampled in a survey) shows up to vote. The propensity to vote is not uniform across socio-economic stratums of the society. One can try to factor these in but that can inflate the error.

Exit polls are designed to take care of both these issues. However, choosing respondents in a randomised manner as they exit the booth is rather difficult and our experience has shown that it does not produce a representative sample (as measured by the gap between the socio-economic profile of the sample and population). Hence, we prefer to do a post-election poll, where in the days following actual voting, we do a household survey with sound methodology. In the current round, various exit polls had got numbers close to the actual results for MP, Rajasthan and Chhattisgarh. But for Delhi there was wide variation. And in the past, there have been occasions when exit polls had given an incorrect picture while we got it right with our post-poll.

Going beyond forecasting, these polls are valuable in understanding what was happening on the ground. As an example, the CSDS poll in MP, Rajasthan and Chhattisgarh showed that the gap between BJP and Congress vote shares was higher in rural areas. This diverges from the common view that the BJP is an urban party. The CSDS website gives the breakup of voting intention by various socio-economic groups, and this is valuable knowledge.

Regulation of opinion polls

Do opinion polls influence voter behaviour? In each of the surveys done by CSDS, one standard question is: `Who did the respondent vote for in the last election'. This refers to the last Vidhan Sabha election, if this is a poll for the Vidhan Sabha and the last Lok Sabha election, if the current one is for the Lok Sabha. Almost invariably the recall for whoever won the last time is much higher than the actual votes, even when the winner from previous poll is set to lose the current election.

Thus in 2011, a much larger percentage of voters seem to recall having voted for the left front in 2006 though in 2011 they were voting for Trinamool Congress. I may add that our estimate of the Trinamool vote share and seats in 2011 was very accurate. The same was true in Tamilnadu where, while voting out the DMK, a much larger number of respondents seem to recall having voted for them.

We have observed this time and again across various states and consistently over the last 15 years. The only explanation that I could come up with is that there is a general tendency to go with the winner.

This raises the concern that a political party may run a media campaign claiming that it is ahead in the polls. This justifies regulation (though not a ban). I feel this regulation could be self regulation by the media, e.g. through the Press Council. The regulation should require that each published poll reveals, in public domain, the detailed methodology of sample selection, the sample size, the socio-economic profile of the sample, the dates when sampling was done, the names of the core team members who supervised the survey and the methodology used to convert vote estimates to seat conversion. All agencies that release such information should be open to an audit by an expert group formed by an autonomous body such as Press Council.

CSDS and I have been very open about our methods. The sampling methodology is on the CSDS website, the sample size is always given on air and the socio economic profile of the sample is also given on the CSDS website. I have written about the vote to seat conversion in an academic article: Predicting the 1998 Indian parliamentary election, Karandikar, R. L.; Payne, C.; Yadav, Y., Electoral Studies; 21, 1; 69-89, and have been talking about it in seminars.

Fortunately, over the years, the visibility of any one opinion poll has declined, with so many agencies doing polls and making contradictory predictions. In the recent Delhi elections, the range of seats projected for the Aam Admi Party was from 6 to 31 out of 70 seats! Hence, the salience of this debate has probably declined greatly.

I do believe that there is a feedback loop: if all surveys point in a certain direction, at least some voters tend to get influenced. But this is no reason for a ban. After all, newspapers and TV channels also talk about their assessment of the political situation and if they all seem to point in a certain direction, this too has an effect on the electorate. In addition, under the Indian legal system, while the government can easily do censorship on television, this is harder with newspapers and more generally on the Internet. Hence, even if a ban were desirable, it is not feasible.