Is Consumer Research Losing Its Focus?

Focus groups continually fail to tell us what customers want. The fundamental problem is that, in spite of what conventional wisdom says, it's not the voice of the customer that matters. What matters is the mind of the customer. The big mistake is in believing that what the mind thinks, the voice speaks. It is time to start embracing methods that can deliver stronger predictive value.


If you listen carefully you can hear it. The sound of marketers and product developers bemoaning the usefulness of their focus group research data, and the stifled groans of companies falling on their metaphorical swords. The noises are not going unheeded. The ground swell of opinion among marketers and researchers alike is that all is not well with focus groups, and that something needs to be done about it. 

A New Consensus

In his recent Slate Magazine article, "Lies, Damn Lies and Focus Groups?" Daniel Gross strongly challenges the effectiveness and value of focus groups for informing product development and marketing. He draws attention to the widely documented mismatch between what people say about product concepts in focus groups, and the way they actually behave when it comes to making purchases - a mismatch that costs companies millions of dollars in misdirected product development efforts. 

Gross is not alone in his views on what is unquestionably the most widely used research method on the planet. Kay Polit, principal analyst at the global management consultant company A.T. Kearney, refers to focus groups as "a faulty process".  Mary Lou Quinlan, founder and CEO of Just Ask A Woman, calls focus groups "a dangerous way to get market intelligence". Dev Patnaik of design strategy firm Jump Associates likens focus groups to “…a customer terrarium, with people behind glass—taken out of their natural surroundings and observed for scientific purposes… Focus groups are the crack cocaine of market research. You get hooked on them and you're afraid to make a move without them.” And authors Joseph Pine and James Gilmore refer to focus groups as "the great lie". In their opinion, "The guidance from focus groups can be downright dangerous.“

Lies? Dangerous? A faulty process? These are not encouraging testimonials upon which to stake millions of dollars, or a company's future. But how justified are these concerns?

Some Real Examples

Over reliance on focus groups failed NBC whose sit-com, Coupling (a re-make of a Brit-com and intended to replace Friends), relied for direction, as most TV pilots do, on focus group responses. NBC had to pull the show from the air after only three disastrous episodes within less than a month of launch. Beryl Vertue, a lead writer on the original British show, says: “There's a huge reliance on ratings and focus groups and far, far too little reliance on a gut instinct, and I think that's a pity. And ultimately, I think it's a mistake.”

Poor management of focus group data failed the Pontiac Aztek which sold below original expectations. Its styling was poorly received by focus group respondents, and should have caused concern and a possible re-design, points out Kay Polit: “Ideally, GM should have stopped Aztek in its tracks when it did so poorly in clinics. They might have been able to save it if they changed a few pieces of sheet metal, but instead somebody edited the data they got and senior management was making decisions on some pretty intensive editorialization… selling the vehicle at this point is probably going to cost them more than it did to design and build it.” 

Focus groups failed the Chrysler PT Cruiser even though its sales now exceed expectations. Focus group data led the Chrysler planners to believe that they had, not a mass-appeal vehicle, but a niche vehicle. They geared up accordingly, and … underestimated volume. 

Focus groups failed a company targeting products to teenage girls. MIT Professor, Justine Cassell, author of a thought-provoking piece entitled "What Women Want" reports her experience working with the company. Following a series of focus groups the company concluded that what teenage girls wanted was technologically-enhanced nail polish. This was a happy coincidence as technologically-enhanced nail polish was precisely what the company produced! However, in Cassell's own research with 3,062 children (60% of whom were girls) in 139 countries, in which the children were invited to describe what they would like to use technology for, not a single one of them said technologically-enhanced nail polish! 

Reflecting what is now a well documented lack of positive correlation between what people say and what they actually do, the Yankelovich Minute Monitor recently provided data listing the top six attributes that respondents said will most strongly influence their purchase decisions for an SUV; and a list of the actual decision criteria used at the point of purchase.  You can guess the punch line. Not one of the six attributes nominated actually played a role in the final purchase decision. 

And so on and so forth. You get the picture. That these cases are not exceptional is evidenced by the fact that a staggering 80% of new products fail within the first six months of launch in spite of most of them going to market on the back of seemingly strong market research data. Data, incidentally, that cost $1.1 billion in 2001. An 80% failure rate?! This is not a subtle clue that something is wrong. It is like turning on the light and getting an electric shock eight times out of ten!

Why Focus Groups Fail

So why do focus groups often result in costly blunders? After all, focus groups:

  • Have a long history (they were first used over 60 years ago to assess the effectiveness of WWII propaganda movies)
  • Are widely used 
  • Have unquestionable face validity 
  • Get data quickly 
  • Seem to be in direct touch with "the voice of the consumer". 

There are a number of important reasons, familiar to most of us, why most focus groups do not fare as well as they could. It is easy to point to methodological design flaws, or badly moderated sessions. Fingers can be pointed at poor and unrepresentative sampling, or at misleading data interpretation or badly written reports, or at recommendations that are ignored. None of these things are conducive to good research, no matter what the method. But none of these are the fundamental problem. The fundamental problem is that, in spite of what conventional wisdom tells us, it is not the voice of the consumer that matters. What matters is the mind of the consumer. The mistake is in believing that what the mind thinks, the voice speaks. 

As an aside, I am reminded of an experience in Newcastle (UK) which was , ironically, one of the most useful focus groups I have conducted, where the participants were all mildly drunk. They had been waiting in the bar of a hotel for the focus group to begin, so were well-primed in the alcohol department even before we got started. And they all arrived at the room carrying a pint of beer in each hand! Now I am not advocating this as a research technique, but the beer seemed to lower any barriers, and I did get the distinct feeling that we were circumventing conscious awareness and accessing genuine thoughts and beliefs! 

Insight or Hindsight?

There is a reason why "unarticulated needs" go unarticulated. Behavioral researchers have long known that expert behaviors (consumers are nothing if not experts at their own daily behaviors) are all but impossible to introspect upon, and so are difficult to reliably articulate. We have known for 30 years (see Nisbett and Wilson's classic 1977 Psychological Review paper) that people have little, if any, reliable access to the cognitive reasoning that underlies decision-making; and that in most instances people are unaware of the factors that influence their responses. This does not mean that respondents cannot provide answers to the "Why?" questions by which most focus group moderators live and breathe. But it does mean that responses are not made on the basis of true introspection. Instead, responses frequently reflect a priori implicit causal theories about the extent to which particular stimuli may plausibly be associated with given responses. In other words, rather than reflecting deep and veritable cognitive processing, respondent's explanations for their decisions are frequently created on the fly in order to fit the situation.

But, even if (which is not the case but let's pretend it is) even if respondents could reliably access their own reasoning processes, and could reliably report on their decision making so that the researcher was indeed collecting bona fide data, we cannot escape the fact that most conventional focus groups actually measure the wrong thing. They do not measure what people think when making a purchase. They measure what people think when participating in a focus group. The psychological, sociological, neurological, and even pecuniary factors bearing on a person's decision making while they are participating and responding in a focus group are not the same psychological, sociological, neurological, and pecuniary factors that bear on decision making when the same person makes an actual purchase. According to Harvard Business Professor, Gerald Zaltman, focus group methods can tap into only about 5% of people's thought processes - the 5% that lies above the level of consciousness. But … it is the 95% of cognition lying below the respondent's level of awareness - the bit that is not visible to focus groups - that is largely responsible for decision making.

Beyond the Voice of the Consumer

So we need to start considering more effective and more reliable methods for discovering consumer needs and preferences. We need to put aside simplistic and overt questioning that telegraphs the researcher's intent, and approach the investigation (rather like a detective might proceed, in fact) from unexpected directions. Indeed, this is how Experimental Psychologists and Cognitive Scientists work. They do not tap into the complexities of human behavior by simply asking people "What are you thinking?" and they seldom rely on people's introspection and self-report.  Instead they use indirect methods of tapping into cognition and behavior. Consumer research can learn much from the methods and tools of the Experimental Psychologist. 

The key is to try and actually bypass the direct voice of the consumer. As an industry we agree that what people say and what people do are seldom the same thing. So it remains somewhat puzzling that we keep basing major decisions on what people say, while paying far less attention to what people do. What we ultimately want to know is the consumer's actual intent. This can be secured by methods other than expecting focus group respondents to inspect their cognitive machinery, understand what they find there, translate that into language, and then articulate unambiguously. It should come as no shock to learn that methods that directly exploit and capture the actual behavior of consumers result in extremely strong predictions of … actual behavior! 

Cultural or social anthropology and ethnography (in the hands of expert Anthropologists and Ethnographers), and structured field research methods such as Contextual Inquiry/Design are highly effective ways of revealing unarticulated consumer needs. Surely it is no coincidence that these methods actually observe people as they engage in daily activities. These findings can then drive the conception, development and marketing of real product solutions - solutions that actually solve something. Although the resource investment of this approach to consumer research is often high compared to the costs of a few focus groups, it is minimal compared to the cost of getting development and marketing decisions wrong.

Later in the development process, lab-based or in-home user tests with high-fidelity prototypes can be used to refine the understanding of needs and to validate a solution's fit to a consumer's problem. Although usability testing methods are typically employed to identify user-interaction problems, they can lend themselves effectively to understanding the extent to which a product is actually useful, and also to identifying missing functions. These behavioral approaches are 'direct' methods that are effective because they eliminate the need for consumer introspection and conjecture.

Undoubtedly, focus group methods are prime for a rethink. The method may have been innovative back in 1945, but it is now time to start exploring methods that can deliver stronger predictive value. Until the industry starts consistently adopting methods that get to the core of consumer behavior, rather than depending so heavily on obvious top-of-mind consumer opinion, billions of dollars will continue to be invested each year in throwing that light switch, only to feel the shock of market failure.

 

Philip Hodgson, June 2004