Product Testing Explained by an Expert Market Researcher

Product testing is the process of gathering feedback directly from potential customers about your product. This could be targeted to look for a wide range of specific insights like It helps you understand usage behavior, consumer preferences or reactions to a product–before it hits the shelves. Product testing usually occurs when launching a new product, but can be done at any point, helping you refine, optimize or validate your product. 

Thorough product testing answers the following questions with astounding accuracy:

• Do my target customers like my product?
• Are there any potential defects or vulnerabilities in the design or materials?
• Is my product safe?
• How does my product compare to competitors’?

While your product might be interesting in concept, or test well across a smaller cohort of consumers, your product will only be successful if it performs across your target market. This is where product testing proves to be so important. By testing your product, you collect critical, structured data on how your target customers interact with your product. You are conducting customer journey and experience research. Product testing enables data-driven innovation, to ultimately make your product better. 

Innovators know all about the high-profile product failures that could’ve been avoided with proper research ahead of launch. Even Coca Cola has had some misses.

On the other hand, product Product research can helps define, accelerate, orand magnify the success of a product. For example, we’ve worked with brand partners where we’ve discovered missed opportunities (this energy bar should be called cacao, not chocolate!)  

Bottom line - product testing is a game-changer.

Introducing Dana Kim, Highlight CEO & Founder

Dana Kim is a former qualitative market researcher who has conducted in-person research for conglomerates like Coca-Cola & Nike. Her frustration with outdated methodologies inspired her to create a better way for companies to do product testing. And that’s how Highlight was born.

Today, Highlight–a product testing company providing in-home usage testing (IHUT) –represents major brands including Nestle, Pepsi, King’s Hawaiian, Colgate, and more.

Learn what Dana has to say about product testing, and why CPG companies can’t afford to skip it.

Product Testing Q&A with Highlight’s CEO, Dana Kim

How do you think product testing is evolving?

In a pre-covid world, the primary forms of product testing were always in person, so you can distribute the products, and watch as consumers test and experience those products. It’s super compelling, but here’s where it falls short: the unreliable testing environment.

You're in a focus group facility. You're not in the comfort of your home. You're not consuming a product or trying a toothpaste or drinking a beverage the way you normally would, as if you naturally grabbed it in your home and in your space. Consumers are really starting to appreciate that–and brands should, too. 

The closer you can get to the most organic, natural consumer experience, the more valuable that feedback is, because that's how consumers will actually experience your product in their realistic day to day, not in a sterile facility or in over-observed situations.

I also think there's some really cool movement around testing a product that's not in its final packaging and not yet on shelf. You're making sure that you're getting feedback in the R&D process and actually using consumer feedback to help inform product development. This has definitely been done before, but it's being done more often, earlier in the process.

It's awesome that brands are better able to ensure product success, by making sure that they have data-driven product development processes. Consumers are now able to actually have a hand in the innovation process. As a CPG or as a consumer product company, all you can ask for is to be as close to the consumer as possible at all phases, and this approach really speaks to that. Putting the consumer at the center is a win-win for all.

How important and valuable is constructive or negative feedback to product development and testing?

I love this question.

There's a big challenge in traditional market research where survey respondents are wrongly motivated, and think they need to say what you want to hear. They're incentivized to take the surveys by gift cards or prizes. Sometimes it's gamified by a point system. But at the end of the day, participants want to be chosen again for the next survey and they want to be rewarded for their survey responses.

So, oftentimes they tell you what they think you want to hear. And that just results in 

• A) skewed data, and 
• B) an echo chamber of "Yes, this is great. Love it. Thank you for the opportunity," which is far less valuable than "This is great but X could be better." 

It's so important to reinforce that 'honesty is the best policy' - as an unbiased research platform, we're really proud that we enable honest, critical feedback - good or bad. 

The genuine desire to make products better, that incentive alignment, that motivation to be constructive and co-create better products alongside brands - those are the things we're really proud of about our community of Highlighters.

We always hammer home that fact that we want to hear what you really think, the good and the bad. What do you like about it? What do you not like about it? Honesty is the best policy, and really critical to product development and better products for all.

The way we position the survey is always, "you have a hand in helping make this better." So by couching the entire feedback process in that mission, it really speaks to consumers by saying to themselves, 'How can I make this better and how can I help the brand succeed or improve?' It's funny, coming from client-facing market research agencies and the traditional market research side, I've always dreaded those meetings when you have to present results to clients and tell them they didn't perform so great. But, those always end up being the most rewarding research efforts. 

It's because of that research that brands avoid making really big "wrong" decisions or place big bets on the wrong things. Calling out the critical feedback is where you create a lot of value and really unlock opportunity.

When is product testing important? What makes Highlight different?

Product testing is a very general term, but really where Highlight comes in is when there's a need to get a physical product in hand, and when there's a prototype or a product. Prior to that, there's so much higher-level diligence on the consumer product manufacturer’s side around "What grand ideas can we pursue? What product categories should we approach? What is the market size of those? Is there a consumer need or a job to be done?"

Ok, now that we have, you know, 10, 15 ideas, really just concepts at this phase, let's test those with consumers and get down to the five that people are really interested in. That whole phase is concept testing and development. The next phase is: now that we have narrowed down to a few product ideas and concepts that we're excited about, let's see if we can bring them to life, create prototypes and actually execute on these ideas that we've invested a lot in.

That's where Highlight comes in - and where things get really interesting.

You've done all this hard legwork and invested in research around making sure the concept is perfect. But, if what's on screen or what's on paper doesn't translate to the physical product, then all that work was for nothing.

A product not living up to its claims or its expectations is disastrous because you might get that purchase, or capture that first interest. The consumer opens the bag and takes a bite, or takes a sip, or whatever it might be. If they don't like it, they never buy it again. And that repurchasing is oftentimes key to a product’s success. You can't afford to let a consumer down.

How important is a customer’s first impression of a product?

There’s a framework General Mills and P&G use where they define consumer's experiences with products as "moments of truth." The first moment of truth between a consumer and a product is when they see a product on shelf or on a screen. At that point, you need to somehow stand out.

• Is it a cool flavor? 
• Unique benefit? 
• Beautiful branding? 
• Interesting functionality?

Whatever it might be, that first moment of truth, you need to win because you need to have someone give you a chance - reach over, click through, take a bet and both consider and try you.

The second moment of truth is when they actually consume the product - when they open it and actually experience it. That's where we come in - we're making sure the product lives up to expectations. It's complex and nuanced - that delicate cycle of consumer trial and repeat, tied to that first and second moment of truth. There's a lot to think about and get right.

How large of an audience can you reach? How do you do make sense of all the data you get back?

Yeah, a core differentiator for us as a platform is our ability to offer product testing is scale. Formerly, you could bring people into a room, have them surround your newest products, try each one, answer questions. Central Location Tests (otherwise known as CLTs, or product focus groups) were core to product research, but also so restricted in scale. On average, you get 6 people at a time with those approaches.

But we can get product feedback from hundreds, thousands of consumers in days.

In terms of the data get-back - if you're asking more scoring, rating, ranking quantitative questions, that's super easily synthesized. If you're asking more open-endeds and collecting verbatims, it's of course less so. You're like, great, now I have three hundred "Why did you like this product?" responses, what do I do with them? This is an age-old challenge. There's always been an overwhelming problem of, “How do I make sense of all of this free-form text?”

We do the typical coding and sentiment analysis to help with this understanding. It comes down to some really basic things, like word clouds and things that help us synthesize all that data. What our product team is working on, that I think is really interesting, is building out more robust algorithms around how to identify the most helpful responses and therefore how to highlight the most actionable responses.

Our algorithm can then tag those "most helpful responses" and actually keep intact the verbatims that are most useful and representative and actionable, and only spotlight a few of them.

As we grow in terms of project scale, we don't want to sacrifice that qualitative, open-ended response quality, and want to be able to collect unstructured thoughts from our audience. They're so engaged - they give paragraphs of really great responses to our open ended questions. We want to keep encouraging that, but to make it more digestible for our brand partners. So we will continue to play with those different ways of synthesizing all of that data in really interesting ways. It is a fun part of our internal product development. We want to practice what we preach and ensure all Highlight platform innovation continues to create value for our community!

To understand more about how Highlight^TM Agile Product Testing, drop us a note here, and an Insights Associate will reach out!