The Problem With Market Research

The Irish Government shouldn’t be the only ones reeling from last Friday’s decision by the electorate to reject a constitutional amendment to abolish the Seanad or Upper House of government. It was another bad day for the market research industry as well. Consider this. Just three days before the election an Irish Times/IpsosMRBI opinion poll predicted the following result. Those in favour – 62 per cent and those against – 38 per cent when the undecided were excluded. How did the market researchers get it so wrong?

Let’s first examine a couple of ‘mitigating’ factors. After the poll, government ‘sources’ described themselves as ‘confident, but not complacent’ because 21 per cent of those surveyed responded that they were undecided while a further 8 per cent said they would not vote. There was some worry about which way the ‘undecideds’ would go but with a solid 17 point lead among the majority that had already made up their minds, the ‘Yes’ camp had reasons to be confident. And second, there was the issue of what might happen in the intervening three days up to the vote that might swing opinions, though, in hindsight, little of significance occurred during this period.

More to the point though is that this election demonstrated again, what is a common issue throughout the world namely, that opinion polls are not a reliable guide to how voters are going to behave. Earlier this year in the British Columbia elections in Canada, the NDP were given an eight to nine point lead the day before the election by two respected opinion polling organisations only to lose the election to the Liberals by five percentage points. Even early exit polls conducted on the day of the election did not show the extent of the swing. And of course, one of Margaret Thatcher’s election wins in the UK, came famously after polls predicted a win for Labour.

The commercial arena is similarly littered with famous examples of market research ‘mistakes’, the launches of New Coke, Red Bull and the Millennium Dome in London to name just three. While experts can argue forever about data collection instruments and sample sizes, one thing is certain – what people say they are going to do is not a good guide to what they actually will do. It is not that humans are dishonest though this can be the case. Instead there is a much bigger issue. People are constantly being asked to rationally explain what they have done in the past or will do in the future when in fact most of the decisions we make are emotionally driven and happen at a sub-conscious level. So it is little wonder that the market researchers so frequently get it wrong.

Be First, If You Can!

Interviewing for that dream graduateship or prestigious MBA programme is challenging enough but is it possible that your chances of success may be a function of whether you happen to be the first interviewee that day or the last. Or what about the exam paper that you have just written? Are your chances of getting a good grade influenced by whether you happen to be the first paper that is graded on a given day or not? If you are a regular reader of this blog, you no doubt sense where this is going and you probably know you won’t like it!

The new psychological term that you need to add to your vocabulary is ‘narrow bracketing’. It occurs in any kind of activity where a small subset of cases from a larger pool is examined at intervals. So for example, an academic may take 10 papers from an overall total of 200 each day or a venture capitalist may look at 5 proposals each day from a larger pool of submissions. Narrow bracketing means that people avoid making subsets of judgements that deviate much from what they expect the overall set of judgements to be like. If you have already seen two good VC proposals and you expect about 40 per cent to be good, the remaining three on that day may be judged more harshly. So getting on that MBA programme may not just be a function of the total pool of applicants that year but also on the few others randomly interviewed that day!

This effect was demonstrated in a recent study published by psychologists Simonsohn and Gino on ten years of MBA applications at a US university. They found a persistent effect where the scores of previous applicants on a given day either positively or negatively affected that of the next applicant. For example, if the score of previous applicants increased by one standard deviation it meant that the expected score for the next applicant dropped, meaning that she would need an extra 30 points on the GMAT, 23 more months of experience or .23 more points (one a 1-5 scale) on her written application to make up the deficit. Once again, this shows the limitations of our so-called ability to make rational decisions either at work or at home!

Related articles  

What Not to Include on Your Resume!