Complicating market research: why it’s broken, how you can fix it

Now in planner-friendly presentation format.

Complicating marketing: market research (2/2)

If exploratory research is what you do when you don’t know what to do, confirmatory research is what you do when you think you know what to do. And if the former was broken, the latter is completely shattered in pieces.
Part of this is because the premises of both branches of research are the same, and so are the problems: people don’t know what they want today and tomorrow, are inclined to lie, and to make things worse they’re also unable to assess what they’re willing to do. Eg.: “Who cares about an MP3 player? It’s so wrong! It’s so stupid!” (Macrumors Web Forum, October 2001)

Part of if however has to do with tools: we don’t let them experience the real thing we’re going to talk about, opting instead for a blunt surrogate, whether it’s a product concept or an ad storyboard (or the embarrassment we force upon ourselves when probing the reaction to a website offering only a poorly-designed jpeg); and part has to do with us: we don’t spend enough time figuring out what we really want to understand, and opting instead for a standard set of widely accepted (thus probably insignificant) KPIs.

The bottom line is, we don’t know what we’re asking people that wouldn’t know what to tell us about something that we’re not giving them.

Problems in exploratory research could be addressed via diversification; in the case of confirmatory research, the key is preciseness. Here are some tips:

1. Refine your audience. Do your homework, and find out exactly who are the key target consumers that will be buying your product at launch. And then refine it further to find out who key influencers are going to be. Any mainstream, “Joe public” target is the hardest to convert, because they’re defined by the lack of any meaningful attribute to leverage on.
   
2. Improve your inputs. Forget storyboards if you’re testing an advertising campaign. Forget positioning statements if you’re testing a product. Make them “feel” what your product is going to be like, rather than telling them what it is exactly. Get an actor that feels like your brand to tell them a story, use physical analogies to give them a sense of the experience…
3. Invisibility still applies. Even more than exploratory research, confirmatory research should be invisible. Don’t ask when you can show; don’t show when you can simulate; don’t simulate when they can experience.
4. Define project-specific objectives. “Purchase intent” is too vague an objective: investigate what kind of behaviour you need to put in place if you’re to increase sales, and then evalute against that objective. Also, steer away from industry-standard objectives: if you’re selling toilet paper, do you really care about “online buzz”? When was the last time you bought toilet paper because you were “engaged in a conversation” around it? If you’re a luxury brand, increasing your “brand for me” ranking could do more harm than good to your equity.
5. Regularly re-evaluate your KPIs. Over time every performance indicator becomes insignificant, as actions are going to be designed to address the formal indicator, rather than the substantial goal that it is meant to represent. Make sure that the metrics serve the work, instead of the other way around.
6. Look into standard deviation. “Lies, damned lies and statistics” applies to a superficial approach to figures: look deeper. An average rating of 7/10 could be a mediocre result if everyone gives you a 7 that is not nearly good enough to stand out from the crowd, or a very promising combination of most people rating you 5 and a a good amount of them rating you 10. Unless you’re Calgonit, you don’t need to be everyone’s best friend: being loved by a fraction of your market is really a more profitable and safer option than being “taken into consideration”by a majority of it.
7. Don’t ask consumers to fix it. There’s a reason why people spend money to go to the cinema (or at least used to before piracy came along): they recognize that the stories told by Hollywood are better than those they could tell themselves. If you think that superficial thoughts from random consumers can be more effective than what your creative agency came up with, there’s only one thing to do: fire your creative agency!
8. Be clever! Again, there’s no way around this. If you think that you don’t have the time or skills for a well-designed research, let someone else do it. If you think that your company can’t deal with this level of complexity, it’s time to quit your job and move somewhere else.

Confirmatory research is like a compass: it’s not going to tell you where you should go, nor what road you should take to get there. All it does is telling you if you’re heading in that direction.

Complicating marketing: market research (1/2)

BBH has recently embarked on a much-needed and much-welcome crusade against what they labeled as Wind Tunnel Marketing, emphasizing how consumer research makes innovation and differentiation less likely to happen.

So, is research bad?

As always, it’s a complicated matter. Let’s distinguish between exploratory research (that will be discussed in this post) and confirmatory research (that will follow in the next one). Let’s define exploratory research as what you do when you don’t know what to do, whether we’re talking about a product launch, an advertising strategy or virtually any business issue. In this case turning to your users sounds like a good idea, but:

• People don’t know what they will want in the future: as Henry Ford put it, “If I’d asked my customers what they wanted, they’d have said a faster horse”. And you can’t blame them for that: figuring out the future of your industry is your job, you shouldn’t ask them to do it for you.

• People don’t know what they want today: “Consumers’ ability to discriminate and make consistent preferences is low, and lower that they believe it is” (P. Raghubir, “The sense and nonsense of consumer product testing”)

• People lie: according to a survey that we run when I was working on a toiletries brand, consumers replaced their toothbrush once a month. If that were true, we could have all retired and moved to Florida. (However, lies themselves can be quite insightful…)

All in all, we’re asking consumers to look at our life and make it easier, while we should be looking at their life and make it better.

Having said this, exploratory research can still be incredibly valid and valuable, and here are some tips to make it work:

• Diversify your methodology: when a fellow planner from my Publicis days was working on a beer brand, he knew that if he used focus groups he wouldn’t have gotten many actionable insights from a bunch of teens locked into an office and put under scrutiny by an adult moderator. So he opted instead for ethnographic research, going out with them for a few weekends in a row (yes, advertising can be a tough job!) and observing group dynamics first hand. What he got out of it was a promising insight that no competitors had been working on yet, and that led to a campaign that didn’t look like a buddy com or a soft-porn.

• Diversify your participants: don’t just talk to current users v non-users; talk to former users of your product; to recent adopters; talk to the distributors and the influencers; talk to those who designed it and built it; to the person who came up with the idea for it. There’s a wealth of interesting insights behind a product, and all of them could be inspiring, provided that they can generate a human response.

• Look for unexpected analogies: people wouldn’t know how they’d react to the change that you’re about to bring in your industry, but they might have experienced something similar in a very different environment. For instance, there are similarities between mobile phones and politics; or between The Economist and a toilet cleaner. (No offense intended for either of them!)

• Diversify your inputs: It’s hard to be imaginative while questioned by a stranger amidst other strangers, in an estranged environments. Use stories and objects (people think better with their hands) to stimulate the imagination and whenever possible take them out of that brain-annihilating cage that is a focus group room!

• Probe for more: don’t take the “what” into account, ask the “why”, and then again. And then again. If we asked a car manufacturer what’s good about his car, we wouldn’t be happy with “it’s got four wheels, and it takes you places”. We would keep probing, until something revealing comes out: maybe it’s safer than the last car he was driving with his family, maybe he was determined to make it look un-German… So why do we accept shallow answers when it comes to consumer research? (And yes, we do that more often that not)

• Don’t care about what people say. Just like jazz, research is about the silences and the notes not played. A certain facial expression, a hesitation or a word not pronounced are more insightful than what people are willing to share.

• Be invisible. Just like technology, the best research is invisible. Don’t ask, if you can simulate; don’t simulate, if you can make it happen for real.

• Be clever. You have to be real f%&ing clever to make research work! If you think you don’t understand people, let someone else do it. If you think your research partners are less than world-star brilliant, fire them.

Exploratory research is a bit like jazz and technology, but very much like cars: if you know where you want to go, it can take you there faster and show you landscapes you wouldn’t have seen otherwise. But it can also take you to the middle of nowhere, or even worse get you killed. If you’re not a good driver, you shouldn’t be given the keys.

To be continued with confirmatory research...

Further reading here and here and especially here.