Businesses strive daily to provide what customers want. Their success depends mainly on how well they understand the needs and motivations of their target audience. 

In the past, this frequently translated into a scattershot approach to meeting customer demands—build more products, design more features, and so on—with, at best, a goal of growing sales. 

But this slapdash strategy occasionally resulted in overspending, overcommitment of resources, and other strains on business operations that could threaten the business’s existence. 

The organised process of data collection in market research has changed all that. Now the focus is on collecting and analyzing high-quality data—information relevant to meeting customer demands—and how this data is obtained. The goal is the “systematic method of collecting and measuring data gathered from different sources of information,” as Medium notes, adding that an “accurate evaluation of collected data can help researchers predict future phenomenon and trends.”

Broadly speaking, there are two chief forms of data:

  • Primary data refers to first-hand information gathered straight from a primary source. 
  • Secondary data encompasses information found in public records, trend reports, market statistics, etc. 

Armed with high-quality data, businesses can better understand their prospective customers—what they want, what they already like, where they conduct their research, and much more. Companies come away with a deeper grasp of their markets, how their products will benefit that market, and the potential challenges they may face later. 

At its best, market research offers a blueprint of how a brand can move forward while avoiding the pitfalls it might otherwise encounter (without the benefit of high-quality data). 

It’s helpful to remember that a wealth of relevant data may already exist in your company. Information gleaned from business analytics and customer service scores offer vital insights into why consumers act the way they do. It’s an excellent place to begin research and avoid any duplication in data mining. 

What sources of data collection work best? What should brands know about the methodologies employed to acquire and measure such data?

Stay ahead

Get regular insights

Keep up to date with the latest insights from our research as well as all our company news in our free monthly newsletter.

The Value of Quantitative and Qualitative Data

Within the broader scope of primary and secondary data, there are other aspects of data collection worth noting:

  • Quantitative research relies on hard facts and numerical data to gain an objective view of consumer opinion. In general, this approach focuses on uncovering insights about large groups of consumers or the population as a whole. It enables brands to easily compare purchasing and other behaviours of different groups (age, gender, market) and to identify potential buying trends on the horizon. 
  • Qualitative research is less concerned with statistics and trends and more focused on the “human” aspect of buying. This research digs deep into the more intangible and subjective reasons why customers behave the way they do. 

As we have noted before, “People are complex and often unpredictable,” so qualitative research “means getting to know your customers and their motivations better.” As a result, brands can more effectively study customer pain points and barriers to consumer use while also guiding the way to a more personalised approach to marketing.

Where Qualitative Data Comes From

So, what are the sources of data collection? Here’s a quick rundown:

Focus groups. A group consisting of a small number of customers (usually no more than 15) meets to discuss a specific issue. Information derived from this approach often leads to rich insights around consumer attitudes and behaviours, underlying motivations, and perceptions about a brand. 

One-to-one, in-depth interviews. Researchers talk to consumers directly, seeking to understand participant opinions better. This method can be in the form of face-to-face interviews and phone or online interviews. 

Expert interviews. Industry experts are another rich source of data collection. Leveraging their knowledge through expert interviews can help brands explore the impact of emerging trends, thus helping to “future-proof” their business. 

Ethnography. In this realm, researchers immerse themselves in customers’ worlds to learn more about the role brands and products play in their daily lives. This can entail visiting consumers and accompanying them as they go about their day or through self-ethnography, where consumers take on video tasks to show us how they live. 

Online communities. Through an online platform, consumers undertake individual or group tasks that enable researchers to explore potentially sensitive issues and better grasp the attitudes and values that lead to that all-important decision to purchase a product or service. 

The personalized focus of qualitative research goes hand-in-hand with more quantitative research methods, adding context and depth to more numerical and data-based metrics.  

Survey Research Plays a Key Role

Sending out surveys is another key method for drawing insights to understand target customers or explore a new market. Surveys can be conducted in a variety of ways, including:

  • Email. This approach offers the benefit of reaching many people at an affordable cost.
  • Phone. Phone surveys are helpful for researchers seeking feedback from a particular demographic, i.e., older consumers who may not use online resources. 
  • Post. Postal surveys are another option, though of increasingly limited use. Prohibitive costs and a long time lag for responses often rule out this approach.
  • In-person. This method is useful when researchers want to know more about how consumers physically interact with a product or a similar situation. Again, the costs and logistics of this approach make it a less appealing process in general.  

These days, online surveys are often the primary method for collecting quantitative data. Existing customers can complete online surveys or respondents sourced from online panels (groups of people matching a brand’s target market who agree to participate in online research). Based on the results, brands can build accurate representative samples and extrapolate findings to the broader population. 

When it comes to quantitative research, survey questions usually include closed rather than open questions. For example, a survey participant being asked, “How satisfied are you with our delivery policy?” would be restricted to answers such as “Very satisfied/Satisfied/Don’t Know/Dissatisfied/Very Dissatisfied.” This method generates data that can be categorized and analyzed in a quantitative, numbers-driven way. 

How Technology Facilitates Data Collection  

Social media has emerged as a valuable source for insights into consumer perceptions and behaviours. Platforms like Facebook, Twitter, Instagram, and others have potentially vast data reservoirs on a target audience. 

On social media, consumers provide direct, unfiltered feedback about their needs, emotions, pain points, and hopes for the future. These platforms offer a relatively easy and inexpensive way to share surveys and questionnaires and enlist participants for upcoming focus groups.

In this respect, “social listening” offers an expedient method of gauging customer sentiment—what they like and don’t like about the buying experiences, preferences regarding how a purchase is made, and so on. 

Technology also makes it possible for researchers to dramatically expand their horizons, connecting with audiences in far-flung areas of a brand’s home country and around the world. Researchers can conduct real-time interviews and focus groups with consumers in multiple time zones using tools like Zoom and Skype. In this way, data collection for international research often yields a more powerful and richer understanding of consumer behaviour. 

Working with a Research Partner

It’s crucial to remember that every customer group is different. Some brands have a strong command of their markets and may conduct research on their own.

For many other brands, partnering with a professional research firm is the best approach to broad-based marketing research. At Kadence, we draw upon our extensive toolkit of qualitative and quantitative methodologies for a deep understanding of the needs of these under-served communities. The result is:

  • More productive research
  • Valuable insights into different demographics
  • Gaining a step on the competition 

By bringing companies closer to their customers, a third-party research firm can embed rich understanding across your organisation and promote more effective, customer-centric decision-making. This understanding often leads to more informed marketing strategies and greater success with untapped consumer populations.

As soon as you think you understand market research, something brand new comes along to challenge everything you thought you knew. With the rapid evolution of technology, those moments seem to be happening more than ever.

Today’s market research campaign looks very different from how it would ten or even five years ago, and technology is one of the major driving forces behind the evolution of market research. Every year brings a slew of new tools, techniques, and platforms built to make research more accessible and more effective.

In this article, we’ll take a look at some of the most important things that have changed in market research due to technology, and we’ll also explore what new changes could be lying in wait in the near future.

Market research technology: what has changed?

Over the last several years, there have been several significant technological changes that have impacted market research. 

Technology has changed how industries operate, and market research is no exception. Advancements in technology have seen a rise in the self-service model, where brands can implement their own short surveys. But perhaps the most significant impact technology has had on the market research industry is agility. Market research technology allows researchers to quickly test, measure, and pivot projects.

Technology allows traditional research briefs to move past online surveys. For example, eye-tracking technology enables researchers to observe shoppers exhibiting their natural behaviours while walking around a real or virtual store, making note of fixation and gaze points. This can be highly revealing in usability studies, product and package testing, and shopping research.

Here are some of the main forces driving the evolution of market research.

Social media

It’s hard to believe that social media has only been around since the early 2000s. In the two decades since its inception, social media has transformed our lives, and market research is no exception.

Social media platforms like Facebook, Twitter, Instagram, and more offer unique insights into your market and customers. These platforms are home to enormous reservoirs of data on your audience, providing unfiltered and direct feedback around how they’re feeling, what they want, their pain points, and hopes and desires. It’s also easier than ever to share surveys and questionnaires, collect attendees for focus groups, and much more.

Another benefit of social media is more accessible competitor analysis, allowing you to gain an easy glimpse into what other companies in your space are doing and saying, what marketing techniques they are using, and what’s working.

Observing customer behaviour

The explosion in personal technology devices like smartphones, IoT gadgets, Alexa, smart cars, and wearables make it far easier to observe the behaviour of your audience members and collect valuable insights in real-time. Even mobile apps can collect customer data and establish behaviour patterns.

Wearable technology is one of the most recent developments in this area. Gadgets like the Oura Ring, Fitbit, Apple Watch, Rayban stories, and much more allow researchers to observe customer behaviour as it takes place naturally in the real world. This gives a unique insight into how customers behave that can never be truly replicated in experimental conditions and can pick up on responses that customers themselves might not even notice.

Automation

It’s never been easier to analyze data and draw valuable conclusions, thanks to the vast leaps made in automation technology in recent years. The growing amount of audience data available to marketers can now be processed and analyzed much more efficiently, allowing you to gain valuable insights and learn as much as possible about your customers and how they behave.

Chatbots are another example of the power of automation in market research. These tools can ask questions and conduct basic surveys from social media apps and websites, allowing you to communicate with customers and collect valuable information in seconds without relying on time-consuming manual work by human staff.

Increased reliance on video and remote collaboration

Spurred on by the pandemic, video collaboration tools, and remote meetings have skyrocketed in popularity and ease of access. There are many advantages to this for market research, such as a shift away from in-person focus groups. 

Researchers no longer need to hire a venue, convince large numbers of people to take the time out of their day to attend, employ multiple staff on the ground, and do all the other logistical tasks involved in a physical interview. Instead, the whole thing can occur via a Zoom call, saving enormous amounts of time and resources for both interviewer and interviewees. This also allows you to contact a much broader sample of participants without being bound by geographic location.

Reach much larger and more diverse audiences 

Not so long ago, market researchers were confined to methods like postal surveys, local interview sessions, and phone calls. Technology — specifically the internet — has allowed researchers to radically expand their horizons, reaching audiences in far-flung parts of the country and even distant parts of the world.

It’s now easy to conduct real-time interviews and focus groups with people several timezones away, allowing companies to gain a much bigger and richer picture of their audiences. This is especially important for international market research but is also helpful to achieve a complete understanding of your audience as a whole.

Build richer buyer personas

As it becomes increasingly easier to collect data from your audience and analyze it in vast amounts, it becomes possible to build much richer and more detailed profiles for your audience members.

The buyer personas of the past were often vague, two-dimensional things, often built around vague generalizations and assumptions. The small amount of data available to researchers decades ago made it challenging to construct genuinely accurate and useful personas.

Today, with the enormous amount of data made available by technology and the internet, you can learn a lot about the people in your audience and build genuinely valuable and richly detailed buyer personas to inform your marketing decisions, product development, and much more.

Stay ahead

Get regular insights

Keep up to date with the latest insights from our research as well as all our company news in our free monthly newsletter.

What the future holds for market research technology

As time goes on, market research will likely continue to evolve, driven by entirely new advancements in technology. Some of these developments will come as a surprise and may be unexpected side-effects of existing or planned technology. We can also predict some future trends based on what we already know.

Tools like AR and VR

Virtual reality and augmented reality open up a whole host of exciting new possibilities for market research. Both tools allow for much more interactive research where participants can use products and experience services in a completely immersive way without leaving their homes.

This makes it much easier to gauge responses, observe behaviour, and collect meaningful feedback without shipping large amounts of physical products to participants’ homes or asking them to travel to a research site.

One example is how Kadence worked with Asahi, using augmented reality to research a new packaging design. Participants used AR to generate models of beer bottles, allowing them to visualize what the bottle would look like in their own homes and provide more accurate and detailed feedback around specific details.

More use of voice assistants

Today, voice assistants are already popular, with tools like Siri and Alexa quickly becoming a central part of people’s everyday lives. These voice tools allow a unique insight into customers’ daily experience and behaviour, and if this data can be collected ethically, it has excellent value for market researchers.

More agility

Increased agility is a fascinating prospect for market researchers as technology advances. In the past, researchers were forced to take risky gambles instead of using a more flexible approach and making adjustments as needed.

With the ubiquity of data in today’s world, businesses can now take a more agile approach to market research, making quick and frequent changes and course corrections in response to the feedback they get from various channels. This trend could mark one of the most significant changes in how we conduct research over the coming years and suggests a step away from over-reliance on guesswork and individual opinion.

Technology has had an enormous impact on how businesses conduct market research, and as time goes on, that impact is likely to increase. The best thing market researchers and companies can do is be open-minded and prepared to embrace new technologies as they evolve.

At Kadence, we can help you harness all the newest and future technologies to conduct market research most productively and effectively, gaining valuable insights into your audience and getting ahead of the competition. Find out more.

A successful product is one that not only looks great but also solves a real problem. Ticking off both boxes requires understanding your customers’ motivations, goals, and behaviours—and user research is the best method for gaining those insights.

User studies are conducted in person traditionally, but it’s possible to accomplish the same goals in an online environment.

Remote user studies can provide rich data. They’re ideal if you work with a global audience, need alternatives due to lockdown or distancing requirements, or face budget and time constraints.

This article will examine how to conduct user studies in an online environment. We’ll discuss the benefits of usability studies, compare lab studies vs. online studies, and share essential research methods and tools.

What is a User Study?

User research is the study of target customers’ needs and their behaviours in achieving them. The aim is to uncover insights that will assist in designing products that best meet user expectations.

There are two broad categories of user research:

● Quantitative: What’s happening; measurable, numerical results (ex: how many people clicked a button?).

● Qualitative: Why it’s happening; motivations behind the behaviour (ex: why didn’t some people click a button?).

Researchers usually start with qualitative research to discover customer needs and motivations and test their initial designs using quantitative measures.

There are also two basic approaches to user research:

● Attitudinal: Listening to users’ words (ex: interviews, surveys).

● Behavioural: Watching their actions (ex: card sorting, usability testing).

The best user research applies quantitative and qualitative research and attitudinal and behavioural approaches.

The Benefits of Usability Testing

Used correctly, user research should lead to a product, service, or website that better meets the specific needs of your customers. 

When people feel like you “get them,” they’re likely to:

● Buy more quickly the first time.

● Spend more money.

● Make more repeat purchases.

● Remain customers for longer.

● Tell others about your product, service, or website.

Increasing customer acquisition, retention, loyalty, and referrals will positively impact your bottom line. User studies can also improve ROI by minimizing design and development costs and reducing support calls.

The more complex your product, service, or website, the greater the risk of skipping user research. 

Unfortunately, many companies bypass user studies. They don’t want to invest time or money, or believe they already know what their customers want.

Basing design on unvalidated assumptions can lead to wasting time, resources, and money on a product, service, or website that flops—and you won’t know if you’ve missed the mark until after launch.

Stay ahead

Get regular insights

Keep up to date with the latest insights from our research as well as all our company news in our free monthly newsletter.

How to Conduct User Studies

While user studies are most effective and efficient when performed early in the development process, it’s never too late to conduct this type of research.

To begin, set concrete goals. What are you trying to learn? What do you want to understand about your users? What problem are you trying to solve? Write a research brief that asks straightforward questions that lead to definitive, measurable answers.

Next, choose your research method or methods wisely (more on the most common methodologies next). Don’t simply pick what you know best. Take time to consider which methods are most applicable to your specific project.

After completing the user studies, share your findings. The results are only helpful if clearly communicated with key stakeholders, including product or website designers and developers, marketing managers, and C-suite leaders.

It’s vital that these colleagues understand, believe in, and know how to act on the data.

Finally, remember that user research should never end. It’s important to continue learning from your users. The marketplace and technology evolve, so must your product, service, or website.

Research Methods

The most common user study methods include interviews, focus groups, surveys, card sorting, A/B testing, and usability testing. Let’s look at the pros and cons of each research methodology.

●     Interviews

Asking non-leading, open-ended questions with a user to learn about their attitudes, desires, and experiences. 

Pros: Provides the most detailed information, can see non-verbal cues

Cons: Expensive and time consuming to conduct and analyse

●     Focus Groups

Interviewing several people at the same time. 

Pros: Increased reach, reduces time and expenses

Cons: Potential for moderator bias, loudest users can influence others, difficult to analyse

●     Surveys

Gathering quantitative and qualitative information from users via (typically anonymous) questionnaires.

Pros: Easy, inexpensive, most expansive reach

Cons: Low response rates, poorly worded questions can skew results, limited ability for follow up

●     Card Sorting

Asking users to categorize a set of terms to understand how they organize information.

Pros: Quick, simple, and inexpensive to perform

Cons: Provides limited information; analysis can take time

●     A/B Testing

Showing two different concepts to an equal number of users to determine which better accomplished a specific goal.

Pros: Conclusive answers to specific questions

Cons: Limited use cases, limited data, time-consuming

●     Usability Testing

Observing users perform predefined tasks.

Pros: Measures actual behaviours (not perceived preferences)

Cons: Expensive and difficult to run

Online Studies vs. Lab Studies

Many types of user research can be conducted remotely. In fact, online user research has key advantages over traditional testing.

● Reach: Access to an unlimited geographic area allows for a larger pool of participants.

● Environment: Users are in their space using their hardware and software, which creates a greater comfort level.

● Cost: Eliminating travel expenses and reducing logistical challenges makes remote studies quicker and less expensive to complete.

● Neutrality: Less potential for bias introduced by a lab setting and/or a moderator’s body language.

● Flexibility: Online research bypasses potential barriers, such as lack of transportation or a global pandemic.

Despite these myriad benefits, there are reasons that online user studies may not be feasible or preferred.

● If information security is paramount, it’s generally more challenging to maintain control over online proprietary information (and user data).

● If your user base is mainly rural or lower-income, you may find it challenging to find participants who have reliable high-speed internet connections.

● If your study depends heavily on tracking a user’s movements or interaction with a product, it may not be possible in a virtual setting.

Moderated vs. Unmoderated User Studies

Remote research is divided into two categories—moderated and unmoderated.

In moderated sessions, a facilitator speaks directly to participants and guides them through questions and/or tasks. 

This type of qualitative research provides the most in-depth insights into precisely what users think and do and why. They are also more expensive and limited to the availability of a trained moderator.

Unmoderated studies are conducted online at the user’s convenience. Participants follow on-screen instructions and are encouraged to speak their thoughts aloud, which are recorded. 

This type of research doesn’t allow for explanation or follow-up questions, which can create confusion and limit the quality of feedback. On the upside, you can run unlimited sessions at all times using a variety of online technology solutions at a lower cost per participant.

Recruiting for Online User Studies

Whether you choose moderated or unmoderated studies, it’s important to ensure that participants fit your user base or target audience. The right users make all the difference in the quality and usability of your results.

Generally, the most reliable recruitment will happen using your database of customers. If that doesn’t exist or doesn’t produce enough participants, however, you can also try:

● A recruitment agency or panel company can find specific participants but will be costly.

● User testing software companies that specialize in recruiting, which will generate a list of participants who applied for your project through a project board or email blast.

● Posting on social media outlets like Reddit, Craigslist, and Linkedin. In this case, it’s important to screen potential candidates because almost anyone could see and apply.

● Asking family and friends is a low-cost solution, but could be create issues of bias and mismatches with your actual user base.

How to Choose the Best Research Method

Some types of user research (interviews, usability studies) are easier to recreate online than others (focus groups, card sorting). 

The ideal situation is to combine insights from multiple types of user research methods and testing rounds. However, it can be time and cost prohibitive to implement several methods in the real world.

Generally, in-person moderated studies are the best choice if your study requires hands-on participation and has a lot of room for confusion. To conduct these studies online, it’s imperative to consider all potential challenges and to thoroughly test the study before recruiting actual participants.

If you’re most concerned with asking open-ended questions that elicit a great depth of insights, a moderated study conducted online will be a great choice.

If your budget is tight or you’re most concerned with getting a larger quantity of responses, a remote unmoderated study is probably the best option.

Best Practices for Moderated Remote Studies

Moving from moderating in-person research to remote research can feel like a big shift, but by and large, a lot of the practices stay the same. 

Here are some best practices for conducting remote moderated testing.

Study design

Shifting from in-person to remote moderated research requires thoughtful preparation. Start with the problem you want to solve or the hypothesis you want to test, and create questions or tasks that address it.

Qualitative methods can generate a lot of information. When every question or task has a clear purpose, you’ll waste less time, reduce participant fatigue, and avoid “analysis paralysis.”

Knowing exactly what you’ll ask also helps identify where you’ll need tools to support your online research (read more about virtual research tools below).

Also, decide who will observe sessions. Allowing developers, project managers, marketers, and others to witness (and even engage with) the research can increase the likelihood that the results will be understood and put to use.

Make sure observers know in advance of the research session what’s expected of them. Should they mute themselves? What types of questions, if any, can they ask and when? How should they communicate with you or each other during the session?

Session management

Before jumping into your research, create a welcoming atmosphere. Let participants know that you appreciate their time and value their input, and review confidentiality measures.

Disclose upfront whether there are observers and if they will also ask questions. Share the session length, the types of questions or tasks they can expect, the desired answer complexity. 

Confirm that users have—and know how to use—anything necessary for the session. Let them know what to do in case of technical difficulties.

Ask permission to record video and/or audio, and both the moderator and user should turn off any potential distractions.

During the session, observers should take notes on any insights they have about what the participant says or does

End the session by thanking participants for their time and insights, and letting them know how they’ll receive any compensation.

Stop the recording when participants leave or leave it on if your team plans to stay and compare notes about the session.

Data analysis

After all of your remote research sessions, gather all observers’ notes and work together to distill the findings. Look for and discuss patterns and themes and how the team will apply the information to your product, service, or website.

Any tools you’ve used may also produce data, which may need to be aggregated before analysis and compilation with your team’s findings.

Best Practices for Remote Unmoderated Studies

Instead of having a human direct the study, unmoderated research relies on a software application to instruct users, ask questions, and record their actions or answers.

By and large, remote unmoderated studies share the same best practices as those above for moderated research. However, there are a few unique considerations to keep in mind.

Study design

Start with your problem or hypothesis and create questions or tasks that address it. Beware that clarity is paramount without a moderator offering explanations or answering questions. 

Make instructions explicit, so users know exactly what to do. If you’re asking participants to record themselves, include clear triggers so they know when they should start and stop. Be specific with open-ended questions to avoid rambling responses unrelated to your goals.

Run a trial session to uncover any ambiguous instructions, questions, or potential problems with the study design or technology. Replicate actual testing situations by using real participants with their equipment.

Finally, a 10 to 15 per cent drop-off rate is typical in unmoderated studies, so plan to recruit more participants than you need.

Session management

Lower your drop-off rate by recording or writing a warm welcome that thanks participants for their time and insights.

Also, be clear up front with instructions and expectations. Let the user know exactly what they will be doing and how long the session will take. 

Before they log off, include a final thank-you message and any information about compensation or follow-up they may receive.

Data analysis

Unmoderated studies can accumulate a lot of data and typically require extra manpower to analyse. 

Quantitative data is straightforward. The testing tool will automatically collect and generate data visualizations for metrics like success rate, task time, and ratings.

For qualitative data, however, you’ll need to review interview questions and session recordings to take notes on user behaviour and identify positive and negative reactions.

Unmoderated testing tools with robust video analysis features can help by aggregating, exporting, sharing, and visualizing any notes you add to recordings and creating compilations of important moments.

Tools for Remote User Studies

Conducting user studies online necessarily requires software or apps. There are many options available (at a wide range of price points) for every type of research task.

● User Recruitment (Respondent, Ethnio)

● Video Conferencing/Recording (Zoom, Google Meet, Skype, GoTo Meeting)

● Note-Taking (ConfirmKit, Pear Note)

● Transcription (Rev, Reduct, Otter.ai, Trint)

● Surveys (Alchemer, Typeform, Survey Monkey, Survey Legend, YesInsights, Survicate)

● Usability Testing (Lookback, PingPong, UserTesting, Loop11, Userbrain, Userlytics, UsabilityHub, UserZoom, FocusVision, Qualtrics, InVision)

● A/B Testing (Optimizely, VWO)

Conclusion

User research is a crucial method for validating assumptions about your users’ needs and experiences. Done well, remote user testing can provide rich data that will help you understand how your target audience interacts with your product, service, or website.

Unfortunately, user studies are not “one size fits all.” There are many methods, each with distinct advantages and disadvantages. The choice depends on your goals for the type of data you want to gather.

Conducting user studies in the online environment isn’t always the best option. Still, it can be incredibly helpful if you’re working with a global audience, are under budget or time constraints, or face limitations due to the COVID pandemic.

How is your product received by consumers or business decision-makers? What are the pros and cons of a change in an existing product feature or new varieties of your current big-sellers? Why is a product failing to perform? You want to tweak a formulation, messaging or packaging to cut costs or reach new audiences – but will that scupper or supercharge your sales? The answers to all these questions can be found in product testing research.

What is product testing?

With product testing, we’re not looking to establish general consumer attitudes or behaviours. Nor is this about standing up a new concept or looking for gaps in the market. The primary job of product testing is to tell us how people respond to an actual product – including how they use it and what they think its qualities are – allowing brands to decide whether and how to market it.

When should I do product testing research?

Let’s look at the natural marketing life-cycle to explain how product testing research can support the emergence, and successful exploitation, of a product – and place it in the context of a wider field of market research:

  • Ideation – dreaming up an idea worth pursuing. Research helps identify unmet consumer needs, value-chain opportunities, potential applications of innovation and new markets.
  • Screening – a rigorous approach to deciding which ideas are worth pursuing, again drawing on research and feasibility work, and assessing potential audiences.
  • Concept testing – seeing how the manifestation of ideas might work in the market, leading to additional screening out of less viable concepts.
  • Prototyping – designing products to prove mass production feasibility, form factors and feature sets.
  • Early product testing – evaluating consumer attitudes to the product itself, either in controlled settings, in the field or in everyday contexts; typically unbranded or with highly simplified packaging.
  • Late product testing – which might include feedback from earlier tests to refine messaging, packaging and final form factor.
  • Testing iterations of a product to forecast the impact (on sales and usage) of changes to features, formulations, targeting and marketing – often in response to changes in sales patterns or negative customer feedback.

In other words, product testing is distinct from concept testing. It’s all about refining the delivery of something that is (or soon will be) a finished product. This might include changes to feature sets, the marketing pitch, pricing, ideal target audience and other details. It’s not so much whether the product works – it’s how the product will work best.

Use cases for product testing research

In summary, then, you can apply product testing to:

  • Find out how a close-to-final version of a new product might perform.
  • Tweak that product to optimise its performance on launch.
  • See how a new product is performing post-launch.
  • Test the effect of changes to product design or presentation.
  • Evaluate or explore how a product is marketed.
  • See how well consumers in a new market will accept an existing product.
  • Undertake ‘penalty analysis’ to see which qualities, when changed, alter consumer options about a product.

Product testing in action – a case study

One way to think about the value of product testing is as a way to optimise the introduction or evolution of a particular product. A good example is work we’ve done with a beverage brand to launch a new range of iced teas. The client wasn’t launching a brand-new product – they had worked up some new flavours and wanted to know which they could launch successfully, how these might affect brand perception and what consumers made of them.

There were eight formulations to be tested. We added in an established variant to act as a benchmark, giving us a way to test the relative strength of the products. We measured various metrics with consumers to provide comparable scores as a key insights.

The product isn’t necessarily going to change in this case. It’s a chance for the client to check which variants might work best, to optimise the roll-out and then make minor refinements if the research delivers consistent feedback on particular elements. How research subjects describe the teas might also help shape marketing and packaging for instance.

In other cases, clients might test out product names and straplines on consumers while they’re testing the product to create a range of possibilities, not just on the branding and marketing, but also on likely target audiences and even pricing. Does the product live up to a premium positioning? Or will it chime with more down-to-earth messaging?

The role of product testing guidelines

We work with many clients that have well-established product testing guidelines – a set of standards that enable them to better evaluate products over time and give them clearer benchmarks for making decisions. For large corporates in particular, the product testing research project is an exercise in generating fairly standardised numbers – data that fits a well-established, almost algorithmic approach to evaluating potential product performance.

When this isn’t the case – or if the guidelines are relatively basic – it’s a good idea to establish some clear ground rules at the start of a project to ensure it delivers insights that will shape client decisions. You might need to agree:

  • How the product should be stored, prepared, presented and used.
  • The audience it should be tested with, and how to recruit them.
  • Where to conduct tests with participants (see below).
  • What metrics to record.
  • How to record their experiences – and other feedback.
  • The research methodologies that will work best.
  • Whether to use a control product for comparative purposes.

Introducing a framework helps everyone understand what success looks like for a product: for it to go forward, what will the research need to show? Is it being on par with an existing or rival product on overall performance? Does it need to be statistically stronger on key metrics?

In some ways, it’s akin to a science experiment: you outline the aim (proving it’s better than the existing product) based on your prediction (the product design); we provide a sound, rigorous methodology to test that assumption (the research); and the result gives conclusive result to tell you how to proceed.

How does product testing work? Where and how to test

Different objectives of product testing will suit different methodologies. A lot depends on what brands already know about the product and the way it’s perceived; on what they want to learn from the tests (see below); and the type of product under review.

There are broadly three environments to conduct product tests. Let’s look at the use-cases and the pros and cons.

1. Central Location Testing (CLT)

This is where participants are invited to a facility to undertake the test. This is ideal for evaluating products in controlled conditions, especially when testing a variety of use cases. It’s also suited to products that won’t be used as much in the home – especially in food outlets, for instance.

A good example would be a new foamless cappuccino we tested. To get comparable results, the same machine was used in different central testing centres, with the client providing an expert barista to produce the same product every time.

CLT is ideal for evaluating products under controlled conditions – testing different fragrances, say, is hard if the conditions allow for cross-contamination of scents. But it’s also very useful where confidentiality is important. We set up a CLT in a hotel, for example, so that a new tech product could be tested by invited consumers without the design leaking. Non-disclosure agreements might be a feature of any product test, but for this kind of commercially or technologically sensitive research, the controlled setting can be helpful too.

The other advantage of CLT is liability management. Some products – foodstuff and cosmetics, in particular – might cause adverse reactions with test subjects, and it’s easier to screen and monitor them on-site.

You can find out more about central location testing in our guide.

2. Street Intercept Testing (SIT)

This is literally grabbing participants in an ambient setting for a few minutes to get them to try something and test their reactions. This works well for relatively simple research – the questionnaire will need to be relatively quick in a supermarket or street setting – and for targeting particular participants. Testing a new cheese at the deli counter in a supermarket would be one application.

It’s also ideal for capturing insights within specific use locations – when a central facility would be a little abstract. We worked with a sports beverage brand to test a special protein-rich drink. The use-case is post-exercise, so intercepts with the target market in a gym setting yielded much more insights than a central location could have.

3. In-Home Testing (IHT)

For many products, the consumer’s home (or, in some cases, their workplace) will be the usual usage location. Getting the products into the home for a period of use, then running online, telephone or face-to-face follow-up questionnaires is a great way to see how they work ‘as intended’.

In-home testing tends to be ideal for more sustained testing. The taste of a new iced tea or reformulated cheese can be tested fairly immediately. But a toothpaste, cleaning product, in-home device or even lightbulb, will only reveal itself properly over a few days’ use. Out of the control conditions, we can learn more about how good instructions for use are; we can see how consumers might use the product in their daily lives or in combination with other products; and we can monitor evolving opinions about the product as they get used to it.

Obviously in-home testing has been popular during Covid-19 lockdowns – not least because many products are now being consumed or used in the home that might otherwise have been ambient products; but also because centralised or street intercept tests have been harder to run for biosecurity reasons.

Note also that IHT allows for different research methodologies. As well as post-use surveys, we can get consumers to keep diaries of use, highlighting a wider variety of situations and providing more qualitative inputs.

Woman scanning food in her fridge with her phone

How to do product testing

Where to start

For many companies, product testing isn’t the start of their journey with us. This kind of research is often part of a much bigger engagement process around a brand or product line; or it might be commissioned by a brand we already do different kinds of work with. So the starting point is rarely a cold introduction to a product.

But even with some engagement beforehand, the first step in product testing is to look at the product and the client’s requirements, and then design an approach that will answer their key questions.

For some, those questions will be extremely precise. For example, one detergent brand asked us to test out a new toilet cleaning product. They knew exactly what segment they wanted to target – ABC1 consumers in their 30s and 40s who were already familiar with the brand – and even the methodology they wanted (in-home testing).

That’s largely a logistical challenge – getting the product and a control cleaner into their homes, in plain packaging, so they can be tested side-by-side; then running an online survey to generate some quantitative data and some qualitative feedback comparing the product to a known comparator.

Another example might be a commercial-kitchen mayonnaise we tested. The client was keen to assess not just how the product performed against other formulations of mayo, but also what professional chefs thought of it in different applications. Will it be at least on par with the existing product? What recipes or dishes did it suit? And how did it compare commercially?

One thing to bear in mind is that you should be testing for things you might change as a result of the insights we generate. Knowing what can change (from packaging and marketing, to cosmetic attributes or even key design features) as a result of the research findings – and what you definitely can’t alter – will ensure the tests are focused and useful.

Methodology reflections

We find that CLT is generally better for ‘sequential monadic’ testing. ‘Monadic’ means the consumer is evaluating a single product, and this is obviously possible in any environment. Even ‘paired comparison’ testing – head-to-head – can be done in-home. But sequencing the comparisons scientifically (standalone, then in head-to-head, for example) often generates more reliable data.

In terms of participant recruitment, clearly targeting the audience accurately – whether in the field, via lists of consumers, or panels – is key. They are often motivated because they receive free products. But in some cases, especially with the more in-depth or time-consuming studies, the chance to earn money is also a motivator.

Hard and soft questions

Product testing can answer a lot of questions. For seasoned clients, they’re often very precise ones – they’re seeking standardised data on usage and performance that will help them contextualise the product within a portfolio.

A good example of a ‘hard’ question might be pricing. Using techniques such as the Gabor–Granger method (to understand price elasticity) or the Van Westendorp Price Sensitivity Meter (which creates an optimal price point for a given audience), research can reveal a lot about the economics of a product.

Hard questions like that are often central, even when we’re working with smaller companies that are looking to take a product from prototype to production and need to calculate the risk/rewards involved.

Smaller companies, however, are more likely to be asking ‘soft’ questions, too, where quantitative surveys are augmented with qualitative insights. They might be trying to learn more about consumer attitudes to the category as well as the product; or develop a deeper understanding either to tweak the product being researched – or inform future innovations.

A good market research agency can really help with this part of the process. For example, in some companies there might not be rigid product testing guidelines in place. But by explaining what they need to know to market the product, what they might be able to change about it and what they’re not sure about, we can help companies come up with fieldwork that will deliver clear metrics and provide answers to their key questions.

What’s the outcome?

A well-planned, well-run product testing project is rarely just looking for a blunt ‘go/no-go’ answer to a product roll-out or adjustment. Although many big brands have a well-established formula for conducting product tests – designed to plug data into their tried-and-tested algorithms – even these clients will often use the test as an opportunity to learn more about the product in different dimensions.

Sometimes that’s just a by-product of a sufficiently expert and thoughtful product test. As market research professionals, we learn a lot more about products during tests than the raw data suggests. Often it’s the degree of flexibility the market research team brings to the product test that makes it most valuable.

That’s true whether the primary objective is standardised data on product attributes – or semi-quantitative work with a healthy dose of qualitative inputs to shape decisions. By making sure the parameters for the product’s adaptation are clear and the questions about it well framed, we can ensure the right blend of methodology and insights meet the client needs.

A good example of that would be taste tests for a new formulation at a chocolate brand. ‘Super tasters’ working at the client will arrive at some finely calibrated formulation, created to be aligned to brand values and differentiate the product. But it’s ordinary consumers whose verdict will shape its ultimate success.

Looking to embark on product testing research?

With experience in product testing research, we can combine the inputs and recommend robust methodologies to make sure the product hits the sweet spot in the market. Find out more about our product testing capabilities or get in touch to discuss with our team.

Concept and pack testing is an area that’s ripe for innovation – a need made ever more pressing during the Covid-19 pandemic. Watch this 15 minute video to hear how we worked with Asahi UK to pilot the use of augmented reality for this purpose, testing pack designs for Fuller’s London Pride.

We’ll share our key findings, focusing specifically on what we discovered about the value of using AR versus static 2D images. These insights will have value to any researcher looking to broaden their toolkit and harness new technologies in the “new normal”.

This is is the first of Kadence International’s Micro Masterclasses, 15 minute videos designed to provide fresh thinking and a new perspective on research methodologies.

As internet penetration has increased across the world, online market research has seen rapid growth. The COVID-19 pandemic acted as a catalyst for further adoption as restrictions on face-to-face contact meant that businesses had to embrace new research approaches. The result? Many companies experiencing the advantages of online research for the first time. What’s more, the widespread adoption of tech during the pandemic – particularly amongst audiences that were traditionally easier to reach offline – means that online market research is a now a viable way of engaging more people than ever before.

In this guide we’ll:

  • Explore the key advantages and disadvantages of online market research
  • Explain the different methodologies available to you and when to use them
  • Share our top tips for setting your project up for success, moderation and analysis

What is online market research?

As the name suggests, online market research is a type of market research where data collection takes place on the internet. In this way it differs from more traditional forms of market research where data collection takes places offline (either in person or over the phone) such as focus groups or telephone interviews.

What are the main advantages and disadvantages of online market research?

Advantages of online market research include being faster and in some cases, cheaper to conduct than face-to-face market research, meaning that you can obtain results more quickly. One of the major disadvantages of online market research is that it relies on the people you’re trying to reach having access to the internet and as such it may not be suitable for all audiences. Some online qualitative methodologies also require a degree of tech savviness, and a longer time commitment than offline approaches.

Beyond this:

  • Online market research is particularly useful for global projects. If you’re looking to enter a new market and don’t have boots on the ground, online market research can help you build your understanding without having to fork out for flights and focus groups. It’s also well suited to multi-market research as you can conduct research in a number of countries in parallel to identify common trends.
  • Online research can help you get closer to customers and bring them to life for stakeholders. In a focus group or a face to face interview, you only have a short amount of time to get to know respondents. In contrast, some forms of online qualitative research take place over a series of days, weeks or even months. This allows us to dig deeper and build a richer understanding of customers and their needs than we might offline. Some methodologies even allow you to set photo and videos tasks. These result in rich multimedia outputs can be used to bring customers to life for stakeholders.
  • Online techniques allow for a more iterative approach to research. The longitudinal nature of some online qualitative methodologies means that it’s possible to adapt your discussion guide as the research progresses to capitalise on emerging insights. This is particularly effective for product development research. For example, in an online community, you can share initial concepts with consumers, iteratively improve them based on feedback and then put them back into the community for further testing. This approach allows you to build and refine the concepts as the research progresses.
  • Online market research can be good for sensitive topics. Many online methodologies offer a greater degree of anonymity than face to face research. As such, they can be better suited to discussing difficult subjects such as illness or sex.

What type of business questions can online market research help me answer?

Online market research is extremely versatile and can be harnessed to explore a broad range of challenges, including:

What are the main methodologies used in online market research and when should I use them?

There are a number of different research tools you can use to collect data online. They include:

Online surveys

An online survey is the primary method for collecting quantitative data online. Online surveys can be completed by your customers or respondents can be sourced from an online panel (a group of people that have agreed to take part in online research). If you opt for this route you can build a representative sample and extrapolate your findings to the wider population.

When to use an online survey

  • To understand trends and patterns of behaviour e.g. buying habits
  • To understand attitudes towards your brand versus the competition 
  • To compare different groups (e.g. by age, gender or market) to understand similarities or differences
  • To track metrics over time e.g. brand awareness, customer satisfaction

When to avoid an online survey

  • Projects where you need to qualitatively explore an issue with consumers to understand motivations and the “why” behind behaviours
  • Projects where you need to co-create with consumers to iteratively improve an idea

Top tips

  • It can be tempting to pack your survey full of questions but you need to be mindful of time. An online survey should take a maximum of 15 – 20 minutes to complete. Any longer than this and you risk seeing respondent fatigue or a significant proportion of people dropping off before the end. This will mean that fieldwork will take longer to complete.
  • Mix up the type of questions you use to keep the survey engaging for respondents

Online communities

Online communities are a qualitative technique for exploring a topic over a number of days, weeks or even months. Pre-screened respondents are invited to a secure online platform where they take part in a number of activities each day. This can involve discussing topics in groups, one on one or taking part in video, photo or audio tasks.

When to use an online community

  • To deepen understanding of a target audience
  • To explore content preferences and consumption
  • To test audience attitude and brand perceptions
  • To test new ideas with consumers – this can be anything from products and services to packaging or new marketing concepts. Online communities are particularly well suited to this. Many platforms enable consumers to mark-up concepts so you can gather in-depth feedback whilst also protecting the confidential nature of the stimulus, through functionality such watermarking or setting videos to self-destruct once they’ve been watched.  

When to avoid an online community

  • When you need to explore a topic with consumers on an individual level

Top tips

  • Do your research on the types of platforms available. Each offer different functionality so go back to your objectives to ensure you’re selecting most suitable for your project and what you want to achieve.

Digital depths

A digital depth is essentially an audio or video interview which takes places online. These tend to last about 30 – 90 minutes. A typical programme might involve 15 – 20 interviews per market, although this can vary based on your business objectives.

When to use a digital depth

  • To explore in-home brand or product usage or explore attitudes towards particular brands or products
  • To gain B2B market insights

When to avoid a digital depth

  • Projects where you require respondents to share and bounce ideas off each other
  • Projects that require large sample sizes as running 100s or 1000s of depths online or offline is just not feasible

Tops tips

  • One of the challenges levelled against online research is that it can be harder to build rapport online. To overcome this, consider pre-tasks that allow you to get to know the respondent beforehand and help them feel comfortable. This can be anything from asking them to complete a short diary task to some quickfire WhatsApp questions before you get started.
  • Be aware of market differences. In markets that have traditionally favoured face to face methods, using video as part of a digital depth is a must for engendering trust.  

Digital ethnographies

The objective of a digital ethnography is to view a consumer’s life as it happens. Ethnography emerged as an offline discipline, but developments in technology mean that it is now possible to conduct ethnography online. From 360 cameras to smart home technology right through to the humble old smartphone, there are a number of tools you can use. And in some cases, these new approaches can uncover insights that you just would not gather when there’s a researcher in the room.

When to use digital ethnographies

  • To explore in-home brand or product usage or to explore attitudes towards particular brands or products
  • To identify unmet needs
  • To understand cultural differences between markets

When not to use digital ethnographies

  • Projects where you need respondents to bounce ideas off one another

Top tips

  • Not everyone will be suited to take part in digital ethnography. You need respondents who are open and willing to let you into their lives. One way of doing this is to include a casting phase as part of the project to identify the right people to participate.

Online focus groups

An online focus group is where a group of pre-screened respondents are invited to join a private online platform for a few hours. Like an in-person focus group, a moderator will guide the discussion, which can be either text or video based.

When to use an online focus group

  • To test attitudes and responses to concepts and ideas at a high level
  • To sense check ideas quickly with consumers

When not to use an online focus group

  • To obtain detailed inputs from respondents at a one-to-one level

Top tips

  • If you are used to running focus groups in person, don’t automatically assume that online focus groups are the next best thing. Depending on what you’re trying to achieve, another methodology may better meet your objectives.
  • If you’re running a video based online focus group make sure that you’re able to see all of the participants at once. This will help in interpreting non-verbal cues and bringing people into the conversation.

Questions to ask yourself when designing your online research approach

As you can see, there are a number of different online methodologies available to use, each with their own distinct use cases, benefits and drawbacks. To help you design the best research approach, we’d recommend asking yourself 5 key questions:

  1. What am I trying to achieve? Some clients come to us dead set on using a particular methodology. We’d advise against this. By starting with your business objectives and then considering which methodology best allows you to meet these, you’ll be in a much stronger position to design an effective research approach.
  2. Do I need to collect quantitative or qualitative data? If you’re looking to conduct research at scale, you’ll need to opt for an online survey whereas if you want to explore a topic qualitatively, there are a number of different options available.
  3. What depth of insight do I require? Are you sense checking an idea or do you need to explore needs in real detail? This will have implications for the methodology you choose.
  4. Do I need to speak to consumers one-on-one or in a group? Different methodologies and platforms facilitate a different type of engagement with respondents. Think carefully about which you require up front.
  5. Which markets am I exploring? Not every methodology will work in every market. There are cultural factors and connectivity issues you’ll need to consider. For instance, if you’re conducting research in India outside metros or tier 1 cities, you’ll need to make sure your approach is mobile optimised but doesn’t require too much bandwidth. Or if looking to test concepts in China, you’ll need to do so in a one-on-one setting due to the Chinese habit to moderate answers and avoid causing offence in group environments. Our guide to conducting online research in Asia, can help you consider the best approach to take in each market. Remember, if you’re running a multi-market project, you don’t need to use the same methodology in every country. Harnessing an approach that will ensure you can answer your business objectives is more important than consistency of approach.
Stay ahead

Get regular insights

Keep up to date with the latest insights from our research as well as all our company news in our free monthly newsletter.

Setting your online research study up for success

So you’ve decided on the best methodology for your study. The next step is to get into the nitty gritty of setting up the project. Here there are a number of key things to consider.

  • Simplicity. When you’re writing your questionnaire or discussion guide, focus on making it as easy as possible to understand. Cut out the jargon to ensure a consistent understanding amongst respondents.
  • Flexibility. For online qualitative projects, it’s worth building in an element of flexibility into your discussion guide. This will allow you to delve into emerging insights during the course of the research.  
  • Cultural nuances. We talked about the importance of culture when selecting your methodology. It’s crucial to bear this in mind throughout the entire research project. When you’re writing a questionnaire, preparing a discussion guide or designing specific tasks, consider how cultural differences might affect participants’ responses. For instance, in Japan people tend to avoid giving negative feedback so you use a traditional 5 point Likert scale here you’ll find that most people will answer somewhere in the middle, making it hard to ascertain the trend. To get a clearer go / no-go result, you’ll want to consider a 4 point scale in this market.
  • Testing and training. Some online qualitative research approaches rely on external technology platforms. Make sure you build in a comprehensive testing phase to ensure these work across different browsers and in different markets, particularly if you’re using a tool for the first time. You should also consider preparing training materials for respondents. What might seem intuitive to us as researchers, may be not be as straight forward for the person taking part in the research.

How to moderate online research

Moderation is arguably even more important online than it is offline. So what should you bear in mind?

  1. Don’t forget the basics. Just because the research is taking place online doesn’t mean that you can forget everything you’ve learnt about great moderation. Dress appropriately and try and mimic face-to-face interactions – even something as simple as waving hello at the start of an interview can help to build rapport and set someone at ease.
  2. Read the (virtual) room. Sometimes what people choose not to say can be as revealing as what they do.
  3. Do your homework. Earlier in this guide, we mentioned the value that pre-tasks can bring. Even a short WhatsApp exchange can be useful in helping you to understand more about a respondent and how they feel about certain topics so you can get the most out of them in the session.
  4. Curate the conversation. Unlike a focus group or an interview which might only last 90 minutes, online communities take place over a longer period and therefore require ongoing moderation. Moderators should think about how they curate the conversation by connecting individual respondents up to foster group discussion.
  5. Conduct research in the local language. If you’re running a project in another country, use local language moderators who’ll be able to understand and draw out local nuances.

Analysing online research projects

Online market research is often considered to deliver more bang for your buck than a face-to-face approach because of the sheer amount of data you can collect. This is particularly true of online communities due to their longitudinal nature. As such, you need to carefully structure your analysis, ensuring you’re focusing on the ‘so what’ for your stakeholders.

There are also cultural considerations to take into account when it comes to analysis. You’ll need to bear in mind that consumers in different markets can answer questions differently when you’re interpreting the data. For instance, when asking about interest levels or purchase intent, the figures you see in ASEAN developing countries tend to be higher than in other markets. Even within markets, there can be regional differences to take account of. In Vietnam, for instance, there are cultural differences between consumers in the two major cities – Hanoi in the North and Ho Chi Minh in the South. Those in the North tend to favour products from well-known brands, while those in the South are more open to trying new things, and this is reflected in the data you see from these areas.

Considering harnessing online research in your own organisation?

This is where we’re in our element. Find out about some of our online methodologies like online surveys or online communities or get in touch to discuss a project.

In the world of market research, we can only get so far by relying on hard, numerical data.

Hard metrics like (generated from quantitative research) are extremely useful and should form a core part of any business strategy. But they only tell part of the overall story.

To dig deeper and gain a fuller picture of why our customers behave the way they do, it’s important to consider supplementing quantitative research with a more  qualitative approach. Qualitative research is based on conversational and open-ended communication and aims to dive a little deeper than quantitative metrics and explore the why behind customer’s actions.

If you want to get the most out of your research, you should be using both approaches. In this guide, we’ll take a look at what qualitative research is, what makes it so useful, and how you can employ it in your own work.

How is qualitative research different from quantitative research?

Quantitative research:

  • Is more data-based, relying on hard data points and objective measurements
  • It uses statistics and numerical data to identify trends and patterns
  • Allows you to quickly establish what’s happening, and look at possible causes 

Quantitative studies are extremely valuable. They allow us to gain a reliable, accurate understanding of what’s happening in our market  and amongst our customers, and make clear-headed decisions that influence the bigger picture. But quantitative data alone isn’t enough.

Qualitative research is more human-focused. It’s less concerned with numbers and figures, and more focused on what customers have to say. It can take the form of interviews, focus groups or online communities  and its goal is to dig into the more intangible and subjective reasons why customers behave the way they do.

Why is qualitative research useful?

Qualitative research is useful because it helps us dive into the human factors driving our customers’ actions. People are complex and often unpredictable, and our behaviour can’t really be boiled down into a series of metrics..

For example, we might know that sales for one product are outperforming another. But why is this happening? Our hard metrics can show us the overall trend and might allow us to pinpoint certain glaring patterns, but they don’t tell us what’s going on in our customers’ minds.

For this, we need qualitative studies. We need to gain insight into the microtrends that lie beneath bigger patterns. 

The benefits don’t end there, though. Qualitative research means getting to know your customers and their motivations better. Here’s how that helps:

  • It can help you to understand customer needs, generating new ideas for products and services. 
  • It can provide valuable feedback on your existing offering. Using qualitative research you can explore pain points and barriers to use, helping you understand how to improve your current products and services.
  • It can be a useful input to your marketing. By truly understanding your audience, you can take a more personalized approach, speaking their language and talking to your customers in a way they can really relate to. It can also provide useful input to campaign or content development. By understanding customer needs you can create marketing content that solves specific problems for your audience and delivers real value in response to the challenges they face and the pain points they grapple with.
Stay ahead

Get regular insights

Keep up to date with the latest insights from our research as well as all our company news in our free monthly newsletter.

Qualitative research methods 

Qualitative research is made up of a range of different methods and techniques. Each has its own use cases, and the best approaches will combine several methodologies based on your customers and your goals. Here are some of the main methods:

  • Focus groups. This is where you bring a small number of customers (usually less than 15) together in a group to discuss a particular issue. By tapping into the power of group dynamics, we’re able to uncover rich insights around attitudes and behaviours, and explore underlying motivations, need states and perceptions.
  • One-to-one, in-depth interviews. Here, researchers speak to customers directly, in a one-to-one setting. It’s a good way to get truly in-depth on a topic, delving into the participant’s opinions and gaining valuable feedback and insight. In depth interviews can be carried in person, on the phone or online. 
  • Expert interviews. Similar to in-depth interviews, expert interviews involve speaking to industry experts to build a rich understanding of the market and where it’s heading. This approach can help you explore the impact of emerging trends to help future proof your business.
  • Ethnography. This is where researchers immerse themselves in customers’ worlds to understand more about their day to day lives and the role that brands and products play. Ethnography can take different forms, from visiting consumers and accompanying them as they go about their day, to mobile self-ethnography where consumers complete video tasks to show us how they live. 
  • Online communities. This is where groups of consumers are brought together over a series of days on an online platform to explore specific issues. Consumers then complete individual or group tasks, enabling the researcher to uncover rich insights. Like mobile self-ethnography, online communities can involve photo and video tasks and are a great way of bringing an audience to life for key stakeholders. What’s more, as online communities consumers over a longer time period than an in-depth interview or a focus group, they allow you to explore complex or sensitive issues and uncover deep insights into attitudes and values to inform your decision-making.

Traditionally qualitative research was done according to the grounded theory method. This is a framework for research that involves collecting qualitative data through the above methods and then using that data to form a theory or hypothesis. However, it’s easy to underestimate the sheer amount of data you can collect through qualitative research and this is particularly true of online methods such as online communities. As such, it’s often not feasible to use the grounded theory method. At Kadence we take a different and more structured approach, exploring hypotheses with key stakeholders and designing the research so that we can test these. This means that the research is tightly focused on the areas that matter most to stakeholders, ensuring that the insights we uncover are actionable.  

Some examples of qualitative research questions you might ask:

  • How important is corporate responsibility to our customers?
  • What are the main reasons people use social media?
  • Why do people want to work for our organization?
  • How do adult males feel about hair loss?
  • What are the key motivations for  undertaking a weight loss programme?

Qualitative research is essential if you want to truly understand your customers and improve your product or service to deliver what they want and need. It goes hand in hand with more quantitative methods of research and helps add context, explanation, and depth to the more numerical and data-based metrics.

At Kadence, we can help you get the most out of qualitative research, to better understand your customers and market on all levels. To find out how, get in touch with us.

In the present circumstances, many of our clients are asking us questions about online research, both in the short and the long-term, in addition to more traditional techniques. Across our offices we’re experienced in a range of methodologies including online communities, video depth interviews and digital ethnographies. In this short video our Global Head of Qualitative Research explains some of the options available and how they work.

Stay ahead

Get regular insights

Keep up to date with the latest insights from our research as well as all our company news in our free monthly newsletter.

Imagine you’re a digital marketer for an online retailer specialising in fitness gear. You’ve just launched a new line of eco-friendly yoga mats, and you’re tasked with maximising sales through your website. You test two different product page versions to see which drives more purchases. 

Version A features a prominent “Limited Time Offer” banner at the top, while Version B includes a series of customer testimonials right beneath the product title. The results of this A/B test could significantly affect your sales figures and offer deeper insights into what motivates your customers to buy.

Such is the power of A/B testing, a method companies of all sizes use to make data-driven decisions that refine user experiences and improve conversion rates. 

A/B testing provides a data-driven solution to optimise website effectiveness without the guesswork. By comparing two versions of a page or element directly against each other, brands can see which changes produce positive outcomes and which ones do not, leading to better business results and a deeper understanding of customer behaviour.

Whether you’re looking to increase conversion rates, enhance user engagement, or drive more sales, effective A/B testing is the key to achieving your goals precisely and confidently.

A/B testing, or split testing, is a method in which two versions of a webpage or app are compared to determine which performs better. Imagine you’re at the helm of a ship; A/B testing gives you the navigational tools to steer more accurately toward your desired destination—increased sales, more sign-ups, or any other business goal. It involves showing the original version (A) and a modified version (B), where a single element may differ, such as the colour of a call-to-action button or the layout of a landing page, to similar visitors simultaneously. The version that outperforms the other in achieving a predetermined goal is then used moving forward.

›

The Importance of A/B testing and ROI

The compelling advantage of A/B testing is its direct contribution to enhancing business metrics and boosting return on investment (ROI). 

A recent study by eConsultancy shows that companies that use A/B testing are twice as likely to report significant improvements in conversion rates. 

Online retailers frequently use A/B testing to optimise website leads and increase conversion rates. This includes split testing product pages and online advertisements, such as Google Shopping Ads. By A/B testing different product page layouts, retailers can identify a version that increases their sales, impacting annual revenue. Similarly, SaaS providers test and optimise their landing pages through A/B testing to find the version that increases user sign-ups, directly improving their bottom line.

A/B testing is less about guessing and more about evidence-based decision-making, ensuring every change to your interface is a strategic enhancement, not just a cosmetic tweak.

Preparing for A/B Testing

1. Setting Objectives

Before launching an A/B test, defining clear, measurable objectives is critical. These objectives should be specific, quantifiable, and aligned with broader business goals. Common goals include increasing conversion rates, reducing bounce rates, or boosting the average order value. The clarity of these objectives determines the test’s focus and, ultimately, its success.

2. Identifying Key Elements to Test

Choosing the right elements on your website for A/B testing can significantly affect the outcome. High-impact elements often include:

  • CTAs: Testing variations in the text, color, or size of buttons to see which drives more clicks.
  • Layouts: Comparing different arrangements of elements on a page to determine which layout keeps visitors engaged longer.
  • Content: Tweaking headlines, product descriptions, or the length of informational content to optimise readability and conversion.
  • Images and Videos: Assessing different images or video styles to see which leads to higher engagement or sales.

3. Understanding Your Audience

Effective A/B testing requires a deep understanding of your target audience. Knowing who your users are, what they value, and how they interact with your website can guide what you test and how you interpret the data from those tests.

Data Analytics Snapshots:

Utilising tools like Google Analytics, heatmaps, or session recordings can provide insights into user behaviour. Heatmaps, for example, can show where users are most likely to click, how far they scroll, and which parts of your site draw the most attention. These tools can highlight areas of the site that are performing well or underperforming, guiding where to focus your testing efforts.

Importance of Audience Insights:

Understanding user behaviour through these tools helps tailor the A/B testing efforts to meet your audience’s needs and preferences, leading to more successful outcomes. For instance, if heatmaps show that users frequently abandon a long signup form, testing shorter versions or different layouts of the form could reduce bounce rates and increase conversions.

These preparatory steps—setting objectives, identifying key elements, and understanding the audience—create a strong foundation for successful A/B testing. By meticulously planning and aligning tests with strategic business goals, companies can ensure that their efforts lead to valuable, actionable insights that drive growth and improvement.

Designing A/B Tests

Developing Hypotheses

A well-crafted hypothesis is the cornerstone of any successful A/B test. It sets the stage for what you’re testing and predicts the outcome. A strong hypothesis is based on data-driven insights and clearly states what change is being tested, why, and its expected impact.

Guidance on Formulating Hypotheses:

  • Start with Data: Analyze your current data to identify trends and areas for improvement. For instance, if data shows a high exit rate from a checkout page, you might hypothesise that simplifying the page could retain more visitors.
  • Be Specific: A hypothesis should clearly state the expected change. For example, “Changing the CTA button from green to red will increase click-through rates by 5%,” rather than “Changing the CTA button colour will make it more noticeable.”
  • Link to Business Goals: Ensure the hypothesis aligns with broader business objectives, enhancing its relevance and priority.

Examples:

  • Good Hypothesis: “Adding customer testimonials to the product page will increase conversions by 10% because trust signals boost buyer confidence.”
  • Poor Hypothesis: “Changing things on the product page will improve it.”

Creating Variations

Once you have a solid hypothesis, the next step is to create the variations that will be tested. This involves tweaking one or more elements on your webpage based on your hypothesis.

Instructions for Creating Variations:

  • Single Variable at a Time: To understand what changes affect outcomes, modify only one variable per test. If testing a CTA button, change the color or the text, but not both simultaneously.
  • Use Design Tools: Utilise web design tools to create these variations. Ensure that the changes remain true to your brand’s style and are visually appealing.
  • Preview and Test Internally: Before going live, preview variations internally to catch potential issues.

Choosing the Right Tools

Selecting the appropriate tools is crucial for effectively running A/B tests. The right tool can simplify testing, provide accurate data, and help interpret results effectively.

By following these steps—developing a strong hypothesis, creating thoughtful variations, and choosing the right tools—you can design effective A/B tests that lead to meaningful insights and significant improvements in website performance. This strategic approach ensures that each test is set up for success, contributing to better user experiences and increased business outcomes.

Implementing A/B Tests

Effective implementation of A/B tests is critical to achieving reliable results that can inform strategic decisions. 

Test Setup and Configuration

Setting up an A/B test properly ensures that the data you collect is accurate and that the test runs smoothly without affecting the user experience negatively.

Step-by-step Guide on Setting Up Tests:

  • Define Your Control and Variation: Start by identifying your control version (the current version) and the variation that includes the changes based on your hypothesis.
  • Choose the Type of Test: Decide whether you need a simple A/B test or a more complex split URL test. Split URL testing is useful when major changes are tested, as it redirects visitors to a different URL.
  • Set Up the Test in Your Chosen Tool: Using a platform like Google Optimise, create your experiment by setting up the control and variations. Input the URLs for each and define the percentage of traffic directed to each version.
  • Implement Tracking: Ensure that your analytics tracking is correctly set up to measure results from each test version. This may involve configuring goals in Google Analytics or custom-tracking events.

Interactive Checklists or Setup Diagrams:

A checklist can help ensure all steps are followed, such as:

  • Define control and variation
  • Choose testing type
  • Configure the test in the tool
  • Set traffic allocation
  • Implement tracking codes

Best Practices for Running Tests

Once your test is live, managing it effectively is key to obtaining useful data.

Tips for Managing and Monitoring A/B Tests:

  • Monitor Performance Regularly: Check the performance of your test at regular intervals to ensure there are no unexpected issues.
  • Allow Sufficient Run Time: Let the test run long enough to reach statistical significance, usually until the results stabilise. You have enough data to make a confident decision.
  • Be Prepared to Iterate: Depending on the results, be prepared to make further adjustments and rerun the test. Optimisation is an ongoing process.

Visual Dos and Don’ts Infographics

To help visualise best practices, create an infographic that highlights the dos and don’ts:

  • Do: Test one change at a time, ensure tests are statistically significant, and use clear success metrics.
  • Don’t Change multiple elements at once, end tests prematurely, and ignore variations in user behaviour.

Statistical Significance and Sample Size

Understanding these concepts is crucial for interpreting A/B test results accurately.

Explanation of Key Statistical Concepts:

  • Statistical Significance: This measures whether the outcome of your test is likely due to the changes made rather than random chance. Typically, a result is considered statistically significant if the probability of the result occurring by chance is less than 5%.
  • Sample Size: The number of users you need in your test to reliably detect a difference between versions. A sample size that is too small may not accurately reflect the broader audience.

Graphs and Calculators:

  • Provide a graph showing how increasing sample size reduces the margin of error, enhancing confidence in the results.
  • Link to or embed a sample size calculator, allowing users to input their data (like baseline conversion rate and expected improvement) to determine how long to run their tests.

By following these guidelines and utilising the right tools and methodologies, you can implement A/B tests that provide valuable insights into user behavior and preferences, enabling data-driven decision-making that boosts user engagement and business performance.

Analyzing Test Results

Once your A/B test has concluded, the next crucial step is analyzing the results. This phase is about interpreting the data collected, understanding the statistical relevance of the findings, and making informed decisions based on the test outcomes.

Interpreting Data

Interpreting the results of an A/B test involves more than just identifying which variation performed better. It requires a detailed analysis to understand why certain outcomes occurred and how they can inform future business decisions.

How to Read Test Results:

  • Conversion Rates: Compare the conversion rates of each variation against the control. Look not only at which had the highest rate but also consider the context of the changes made.
  • Segmented Results: Break down the data by different demographics, device types, or user behaviours to see if there are significant differences in how certain groups reacted to the variations.
  • Consistency Over Time: Evaluate how the results varied over the course of the test to identify any patterns that could influence your interpretation, such as a weekend vs. weekday performance.

Statistical Analysis

A deeper dive into the statistical analysis will confirm whether the observed differences in your A/B test results are statistically significant and not just due to random chance.

Understanding Statistical Significance and Other Metrics:

  • P-value: This metric helps determine the significance of your results. A p-value less than 0.05 typically indicates that the differences are statistically significant.
  • Confidence Interval: This range estimates where the true conversion rate lies with a certain level of confidence, usually 95%.
  • Lift: This is the percentage increase or decrease in the performance metric you are testing for, calculated from the baseline of the control group.

Making Informed Decisions

With the data interpreted and the statistical analysis complete, the final step is to decide how to act on the insights gained from your A/B test.

Guidelines on How to Act on Test Results:

  • Implement Winning Variations: If one variation significantly outperforms the control, consider implementing it across the site.
  • Further Testing: If results are inconclusive or the lift is minimal, running additional tests with adjusted variables or targeting a different user segment may be beneficial.
  • Scale or Pivot: Depending on the impact of the changes tested, decide whether to scale these changes up to affect more of your business or to pivot and try a different approach entirely.

Decision Trees or Flowcharts:

Create a decision tree or flowchart that outlines the decision-making process following an A/B test. This could include nodes that consider whether the test was statistically significant, whether the results align with business goals, and what follow-up actions (like further testing, full implementation, or abandonment of the change) should be taken based on different scenarios.

By thoroughly analyzing A/B test results through data interpretation, statistical analysis, and strategic decision-making, organisations can ensure that they are making informed decisions that will enhance their website’s user experience and improve overall business performance. This data-driven approach minimises risks associated with website changes and ensures that resources are invested in modifications that provide real value.

Beyond Basic A/B Testing

Once you have mastered basic A/B testing, you can explore more sophisticated techniques that offer deeper insights and potentially greater improvements in user experience and conversion rates. This section delves into advanced testing strategies and the importance of ongoing optimisation through iterative testing.

Advanced Testing Techniques

Advanced testing methods allow you to explore more complex hypotheses about user behaviour and website performance, often involving multiple variables or entire user journeys.

Multivariate Testing (MVT):

  • Overview: Unlike A/B testing, which tests one variable at a time, multivariate testing allows you to test multiple variables simultaneously to see which combination produces the best outcome.
  • Application: For example, you might test different versions of an image, headline, and button on a landing page all at once to determine the best combination of elements.
  • Benefits: This approach can significantly speed up the testing process and is particularly useful for optimising pages with multiple elements of interest.

Multipage Testing:

  • Overview: Also known as “funnel testing,” this technique involves testing variations across multiple pages that make up a user journey or funnel.
  • Application: You might test variations of both the product and checkout pages to see which combination leads to higher conversion rates.
  • Benefits: Multipage testing helps ensure consistency in messaging and user experience across multiple stages of the user journey, which can improve overall conversion rates.

Continuous Improvement and Iteration

The goal of A/B testing is not just to find a winning variation but to continually refine and enhance your website based on user feedback and behaviour.

Importance of Ongoing Optimisation:

  • Iterative Process: Optimisation is an ongoing process that involves continually testing and refining website elements based on user data and business objectives.
  • Learning from Each Test: Each test provides valuable insights into whether a variation wins. These insights can inform future tests, leading to better user experiences and higher conversion rates.

Iterative Testing Strategies:

  • Start with Broad Tests: Begin with broader tests to identify which elements have the most significant impact on user behaviour.
  • Refine and Repeat: Use the insights gained to refine your hypotheses and test more specific variations.
  • Expand Testing: Once you’ve optimised major elements, expand your testing to less prominent components that could still affect user experience and conversions.

Timelines and Case Studies:

  • Timeline Example: Show a timeline that outlines an annual testing strategy, with phases for broad testing, refinement, and expansion.
  • Case Study: Present a case study of a company that implemented continuous testing. Highlight how iterative testing helped them achieve a significant, sustained increase in conversion rates over time. For instance, a tech company could use iterative testing to fine-tune its sign-up process, resulting in a 50% increase in user registrations over a year.

By advancing beyond basic A/B testing and embracing more complex and continuous testing strategies, companies can optimise their websites more effectively and foster a culture of data-driven decision-making. This approach leads to improvements that align with user preferences and business goals, ensuring sustained growth and a competitive edge in the market.

Common Pitfalls and How to Avoid Them

A/B testing is a powerful tool for website optimisation, but common pitfalls can undermine its effectiveness. This section explores typical errors that occur during the testing process and provides strategies to ensure the validity and reliability of your tests.

List of Common Mistakes

Identifying Errors and Solutions:

  • Testing Too Many Changes at Once: It can make determining which change affected the outcome difficult.
    • Solution: Focus on testing one change at a time or use multivariate testing for simultaneous changes and analyze the impact of each element separately.
  • Not Allowing Enough Time for the Test to Run: Ending a test too soon can lead to conclusions that aren’t statistically significant.
    • Solution: Ensure each test runs long enough to collect adequate data, reaching statistical significance before making decisions.
  • Testing Without a Clear Hypothesis: Starting tests without a clear, data-backed hypothesis leads to unclear outcomes.
    • Solution: Develop a precise hypothesis for each test based on thorough data analysis and clear business objectives.
  • Ignoring User Segmentation: Different segments may react differently to the same change.
    • Solution: Segment your audience and analyze how different groups respond to each variation.

Visuals of Pitfalls vs. Best Practices:

  • Create side-by-side infographics showing examples of these mistakes versus best practices. For example, visually compare the outcome of a test that changed multiple elements simultaneously against one that tested a single change.

Ensuring Validity and Reliability

Maintaining the integrity of your A/B tests is crucial for obtaining reliable, actionable insights.

Tips on Maintaining Test Integrity:

  • Use Proper Randomisation: Ensure that the distribution of users between the control and test groups is random to avoid selection bias.
    • Tool Tip: Utilise tools that automatically handle randomisation to avoid manual errors.
  • Control External Factors: Holidays, marketing campaigns, or significant news events can skew test results.
    • Solution: Monitor external factors, adjust the testing period, or filter the data to account for anomalies.
  • Ensure Consistent Test Conditions: Changes in the testing environment or platform during the test can invalidate results.
    • Solution: Keep the testing conditions consistent throughout the test period and verify configuration settings regularly.
  • Validate Test Setup Before Going Live: A misconfigured test can lead to incorrect data interpretation.
    • Solution: Run a smaller pilot test or use a checklist to ensure every test element is correctly set up before full deployment.

Troubleshooting Guide with Graphic Aids:

  • Develop a troubleshooting guide that includes common scenarios where A/B test integrity might be compromised. Include flowcharts or decision trees that help identify and resolve issues such as data discrepancies, unexpected user behaviour, or sudden changes in conversion rates.
  • Example Graphic Aid: A flowchart that helps determine actions when test results seem inconsistent with historical data or benchmarks. Steps might include checking configuration settings, reviewing segmentation criteria, or extending the test duration.

By understanding and avoiding these common pitfalls and maintaining rigorous standards for validity and reliability, organisations can ensure that their A/B testing efforts lead to meaningful improvements and robust data-driven decisions. This approach not only enhances the effectiveness of current tests but also builds a foundation for future testing strategies that are even more successful.

A/B Testing Case Studies

A/B testing has proven to be a critical tool for businesses aiming to optimise their online presence based on data-driven decisions. Here, we delve into some specific real-life case studies from different industries, highlighting the successes and lessons from A/B testing.

Success Stories

E-commerce: Humana

  • Overview: Humana, a well-known health insurance company, conducted an A/B test to increase click-through rates on one of their primary campaign landing pages. They tested the simplicity and message of their banner and CTA.
  • Changes Tested: The original banner had a lot of information and a standard “Shop Medicare Plans” button. The test variation simplified the message and changed the button text to “Get Started Now.”
  • Results: The variation led to a 433% increase in click-through rates to the insurance plans page.

B2B: SAP

  • Overview: SAP, a leader in enterprise application software, tested the copy of their CTA on a product page. The hypothesis was that a more action-oriented CTA would increase engagement.
  • Changes Tested: The original CTA read “Learn more,” which was changed to “See it in action” in the variation.
  • Results: This simple change in wording resulted in a 32% increase in clicks.

.

Digital Media: The Guardian

  • Overview: The Guardian tested different wordings for their support and donation CTAs to determine which would more effectively encourage readers to contribute financially.
  • Results: The test revealed that a direct ask for contributions using emotive language resulted in a higher click-through rate than a more generic request for support.
  • Lesson: This A/B test highlighted the importance of emotional resonance in messaging, especially for non-profit or cause-based initiatives.

Travel Industry: Expedia

  • Overview: Expedia conducted A/B testing to optimise hotel booking conversions on their site by altering the display of discount offers.
  • Changes Tested: They tested the visibility and presentation of savings messages (e.g., showing a percentage off versus a specific dollar amount saved).
  • Results: Showing the amount of money saved led to a slight decrease in conversion rates, contrary to expectations.
  • Lesson: The test underscored the potential for “over-optimising” to backfire and the need to balance how offers are presented to avoid overwhelming customers.

Final Checklist of A/B Testing Steps

To help ensure your A/B testing journey is structured and effective, here is a visual checklist encapsulating the process:

  1. Define Objectives: Clearly state what you aim to achieve.
  2. Formulate Hypotheses: Base your assumptions on data and prior insights.
  3. Select the Testing Tool: Choose a platform that suits your scale and complexity needs.
  4. Design the Test: Create variations based precisely on your hypotheses.
  5. Run the Test: Ensure the test is long enough to gather meaningful data.
  6. Analyze Results: Use statistical analysis to interpret the outcomes.
  7. Implement Changes: Apply successful variations or further refine and test.
  8. Repeat: Use insights gained to continuously improve further testing.

Regardless of the outcome, every test is a step forward in understanding your users better and refining your digital offerings to meet their needs more effectively. The journey of optimisation is continuous, and each effort builds upon the last, opening new doors to innovation and growth.

Harness the power of A/B testing to start making informed decisions that propel your business forward. Your next breakthrough could be just one test away.