If exploratory research is what you do when you don’t know what to do, confirmatory research is what you do when you think you know what to do. And if the former was broken, the latter is completely shattered in pieces.
Part of this is because the premises of both branches of research are the same, and so are the problems: people don’t know what they want today and tomorrow, are inclined to lie, and to make things worse they’re also unable to assess what they’re willing to do. Eg.: “Who cares about an MP3 player? It’s so wrong! It’s so stupid!” (Macrumors Web Forum, October 2001)
Part of if however has to do with tools: we don’t let them experience the real thing we’re going to talk about, opting instead for a blunt surrogate, whether it’s a product concept or an ad storyboard (or the embarrassment we force upon ourselves when probing the reaction to a website offering only a poorly-designed jpeg); and part has to do with us: we don’t spend enough time figuring out what we really want to understand, and opting instead for a standard set of widely accepted (thus probably insignificant) KPIs.
The bottom line is, we don’t know what we’re asking people that wouldn’t know what to tell us about something that we’re not giving them.
Problems in exploratory research could be addressed via diversification; in the case of confirmatory research, the key is preciseness. Here are some tips:
- Refine your audience. Do your homework, and find out exactly who are the key target consumers that will be buying your product at launch. And then refine it further to find out who key influencers are going to be. Any mainstream, “Joe public” target is the hardest to convert, because they’re defined by the lack of any meaningful attribute to leverage on.
- Improve your inputs. Forget storyboards if you’re testing an advertising campaign. Forget positioning statements if you’re testing a product. Make them “feel” what your product is going to be like, rather than telling them what it is exactly. Get an actor that feels like your brand to tell them a story, use physical analogies to give them a sense of the experience…
- Invisibility still applies. Even more than exploratory research, confirmatory research should be invisible. Don’t ask when you can show; don’t show when you can simulate; don’t simulate when they can experience.
- Define project-specific objectives. “Purchase intent” is too vague an objective: investigate what kind of behaviour you need to put in place if you’re to increase sales, and then evalute against that objective. Also, steer away from industry-standard objectives: if you’re selling toilet paper, do you really care about “online buzz”? When was the last time you bought toilet paper because you were “engaged in a conversation” around it? If you’re a luxury brand, increasing your “brand for me” ranking could do more harm than good to your equity.
- Regularly re-evaluate your KPIs. Over time every performance indicator becomes insignificant, as actions are going to be designed to address the formal indicator, rather than the substantial goal that it is meant to represent. Make sure that the metrics serve the work, instead of the other way around.
- Look into standard deviation. “Lies, damned lies and statistics” applies to a superficial approach to figures: look deeper. An average rating of 7/10 could be a mediocre result if everyone gives you a 7 that is not nearly good enough to stand out from the crowd, or a very promising combination of most people rating you 5 and a a good amount of them rating you 10. Unless you’re Calgonit, you don’t need to be everyone’s best friend: being loved by a fraction of your market is really a more profitable and safer option than being “taken into consideration”by a majority of it.
- Don’t ask consumers to fix it. There’s a reason why people spend money to go to the cinema (or at least used to before piracy came along): they recognize that the stories told by Hollywood are better than those they could tell themselves. If you think that superficial thoughts from random consumers can be more effective than what your creative agency came up with, there’s only one thing to do: fire your creative agency!
- Be clever! Again, there’s no way around this. If you think that you don’t have the time or skills for a well-designed research, let someone else do it. If you think that your company can’t deal with this level of complexity, it’s time to quit your job and move somewhere else.
Confirmatory research is like a compass: it’s not going to tell you where you should go, nor what road you should take to get there. All it does is telling you if you’re heading in that direction.