Concept and pack testing is an area that’s ripe for innovation – a need made ever more pressing during the Covid-19 pandemic. Watch this 15 minute video to hear how we worked with Asahi UK to pilot the use of augmented reality for this purpose, testing pack designs for Fuller’s London Pride.

We’ll share our key findings, focusing specifically on what we discovered about the value of using AR versus static 2D images. These insights will have value to any researcher looking to broaden their toolkit and harness new technologies in the “new normal”.

This is is the first of Kadence International’s Micro Masterclasses, 15 minute videos designed to provide fresh thinking and a new perspective on research methodologies.

As internet penetration has increased across the world, online market research has seen rapid growth. The COVID-19 pandemic acted as a catalyst for further adoption as restrictions on face-to-face contact meant that businesses had to embrace new research approaches. The result? Many companies experiencing the advantages of online research for the first time. What’s more, the widespread adoption of tech during the pandemic – particularly amongst audiences that were traditionally easier to reach offline – means that online market research is a now a viable way of engaging more people than ever before.

In this guide we’ll:

  • Explore the key advantages and disadvantages of online market research
  • Explain the different methodologies available to you and when to use them
  • Share our top tips for setting your project up for success, moderation and analysis

What is online market research?

As the name suggests, online market research is a type of market research where data collection takes place on the internet. In this way it differs from more traditional forms of market research where data collection takes places offline (either in person or over the phone) such as focus groups or telephone interviews.

What are the main advantages and disadvantages of online market research?

Advantages of online market research include being faster and in some cases, cheaper to conduct than face-to-face market research, meaning that you can obtain results more quickly. One of the major disadvantages of online market research is that it relies on the people you’re trying to reach having access to the internet and as such it may not be suitable for all audiences. Some online qualitative methodologies also require a degree of tech savviness, and a longer time commitment than offline approaches.

Beyond this:

  • Online market research is particularly useful for global projects. If you’re looking to enter a new market and don’t have boots on the ground, online market research can help you build your understanding without having to fork out for flights and focus groups. It’s also well suited to multi-market research as you can conduct research in a number of countries in parallel to identify common trends.
  • Online research can help you get closer to customers and bring them to life for stakeholders. In a focus group or a face to face interview, you only have a short amount of time to get to know respondents. In contrast, some forms of online qualitative research take place over a series of days, weeks or even months. This allows us to dig deeper and build a richer understanding of customers and their needs than we might offline. Some methodologies even allow you to set photo and videos tasks. These result in rich multimedia outputs can be used to bring customers to life for stakeholders.
  • Online techniques allow for a more iterative approach to research. The longitudinal nature of some online qualitative methodologies means that it’s possible to adapt your discussion guide as the research progresses to capitalise on emerging insights. This is particularly effective for product development research. For example, in an online community, you can share initial concepts with consumers, iteratively improve them based on feedback and then put them back into the community for further testing. This approach allows you to build and refine the concepts as the research progresses.
  • Online market research can be good for sensitive topics. Many online methodologies offer a greater degree of anonymity than face to face research. As such, they can be better suited to discussing difficult subjects such as illness or sex.

What type of business questions can online market research help me answer?

Online market research is extremely versatile and can be harnessed to explore a broad range of challenges, including:

What are the main methodologies used in online market research and when should I use them?

There are a number of different research tools you can use to collect data online. They include:

Online surveys

An online survey is the primary method for collecting quantitative data online. Online surveys can be completed by your customers or respondents can be sourced from an online panel (a group of people that have agreed to take part in online research). If you opt for this route you can build a representative sample and extrapolate your findings to the wider population.

When to use an online survey

  • To understand trends and patterns of behaviour e.g. buying habits
  • To understand attitudes towards your brand versus the competition 
  • To compare different groups (e.g. by age, gender or market) to understand similarities or differences
  • To track metrics over time e.g. brand awareness, customer satisfaction

When to avoid an online survey

  • Projects where you need to qualitatively explore an issue with consumers to understand motivations and the “why” behind behaviours
  • Projects where you need to co-create with consumers to iteratively improve an idea

Top tips

  • It can be tempting to pack your survey full of questions but you need to be mindful of time. An online survey should take a maximum of 15 – 20 minutes to complete. Any longer than this and you risk seeing respondent fatigue or a significant proportion of people dropping off before the end. This will mean that fieldwork will take longer to complete.
  • Mix up the type of questions you use to keep the survey engaging for respondents

Online communities

Online communities are a qualitative technique for exploring a topic over a number of days, weeks or even months. Pre-screened respondents are invited to a secure online platform where they take part in a number of activities each day. This can involve discussing topics in groups, one on one or taking part in video, photo or audio tasks.

When to use an online community

  • To deepen understanding of a target audience
  • To explore content preferences and consumption
  • To test audience attitude and brand perceptions
  • To test new ideas with consumers – this can be anything from products and services to packaging or new marketing concepts. Online communities are particularly well suited to this. Many platforms enable consumers to mark-up concepts so you can gather in-depth feedback whilst also protecting the confidential nature of the stimulus, through functionality such watermarking or setting videos to self-destruct once they’ve been watched.  

When to avoid an online community

  • When you need to explore a topic with consumers on an individual level

Top tips

  • Do your research on the types of platforms available. Each offer different functionality so go back to your objectives to ensure you’re selecting most suitable for your project and what you want to achieve.

Digital depths

A digital depth is essentially an audio or video interview which takes places online. These tend to last about 30 – 90 minutes. A typical programme might involve 15 – 20 interviews per market, although this can vary based on your business objectives.

When to use a digital depth

  • To explore in-home brand or product usage or explore attitudes towards particular brands or products
  • To gain B2B market insights

When to avoid a digital depth

  • Projects where you require respondents to share and bounce ideas off each other
  • Projects that require large sample sizes as running 100s or 1000s of depths online or offline is just not feasible

Tops tips

  • One of the challenges levelled against online research is that it can be harder to build rapport online. To overcome this, consider pre-tasks that allow you to get to know the respondent beforehand and help them feel comfortable. This can be anything from asking them to complete a short diary task to some quickfire WhatsApp questions before you get started.
  • Be aware of market differences. In markets that have traditionally favoured face to face methods, using video as part of a digital depth is a must for engendering trust.  

Digital ethnographies

The objective of a digital ethnography is to view a consumer’s life as it happens. Ethnography emerged as an offline discipline, but developments in technology mean that it is now possible to conduct ethnography online. From 360 cameras to smart home technology right through to the humble old smartphone, there are a number of tools you can use. And in some cases, these new approaches can uncover insights that you just would not gather when there’s a researcher in the room.

When to use digital ethnographies

  • To explore in-home brand or product usage or to explore attitudes towards particular brands or products
  • To identify unmet needs
  • To understand cultural differences between markets

When not to use digital ethnographies

  • Projects where you need respondents to bounce ideas off one another

Top tips

  • Not everyone will be suited to take part in digital ethnography. You need respondents who are open and willing to let you into their lives. One way of doing this is to include a casting phase as part of the project to identify the right people to participate.

Online focus groups

An online focus group is where a group of pre-screened respondents are invited to join a private online platform for a few hours. Like an in-person focus group, a moderator will guide the discussion, which can be either text or video based.

When to use an online focus group

  • To test attitudes and responses to concepts and ideas at a high level
  • To sense check ideas quickly with consumers

When not to use an online focus group

  • To obtain detailed inputs from respondents at a one-to-one level

Top tips

  • If you are used to running focus groups in person, don’t automatically assume that online focus groups are the next best thing. Depending on what you’re trying to achieve, another methodology may better meet your objectives.
  • If you’re running a video based online focus group make sure that you’re able to see all of the participants at once. This will help in interpreting non-verbal cues and bringing people into the conversation.

Questions to ask yourself when designing your online research approach

As you can see, there are a number of different online methodologies available to use, each with their own distinct use cases, benefits and drawbacks. To help you design the best research approach, we’d recommend asking yourself 5 key questions:

  1. What am I trying to achieve? Some clients come to us dead set on using a particular methodology. We’d advise against this. By starting with your business objectives and then considering which methodology best allows you to meet these, you’ll be in a much stronger position to design an effective research approach.
  2. Do I need to collect quantitative or qualitative data? If you’re looking to conduct research at scale, you’ll need to opt for an online survey whereas if you want to explore a topic qualitatively, there are a number of different options available.
  3. What depth of insight do I require? Are you sense checking an idea or do you need to explore needs in real detail? This will have implications for the methodology you choose.
  4. Do I need to speak to consumers one-on-one or in a group? Different methodologies and platforms facilitate a different type of engagement with respondents. Think carefully about which you require up front.
  5. Which markets am I exploring? Not every methodology will work in every market. There are cultural factors and connectivity issues you’ll need to consider. For instance, if you’re conducting research in India outside metros or tier 1 cities, you’ll need to make sure your approach is mobile optimised but doesn’t require too much bandwidth. Or if looking to test concepts in China, you’ll need to do so in a one-on-one setting due to the Chinese habit to moderate answers and avoid causing offence in group environments. Our guide to conducting online research in Asia, can help you consider the best approach to take in each market. Remember, if you’re running a multi-market project, you don’t need to use the same methodology in every country. Harnessing an approach that will ensure you can answer your business objectives is more important than consistency of approach.
Stay ahead

Get regular insights

Keep up to date with the latest insights from our research as well as all our company news in our free monthly newsletter.

Setting your online research study up for success

So you’ve decided on the best methodology for your study. The next step is to get into the nitty gritty of setting up the project. Here there are a number of key things to consider.

  • Simplicity. When you’re writing your questionnaire or discussion guide, focus on making it as easy as possible to understand. Cut out the jargon to ensure a consistent understanding amongst respondents.
  • Flexibility. For online qualitative projects, it’s worth building in an element of flexibility into your discussion guide. This will allow you to delve into emerging insights during the course of the research.  
  • Cultural nuances. We talked about the importance of culture when selecting your methodology. It’s crucial to bear this in mind throughout the entire research project. When you’re writing a questionnaire, preparing a discussion guide or designing specific tasks, consider how cultural differences might affect participants’ responses. For instance, in Japan people tend to avoid giving negative feedback so you use a traditional 5 point Likert scale here you’ll find that most people will answer somewhere in the middle, making it hard to ascertain the trend. To get a clearer go / no-go result, you’ll want to consider a 4 point scale in this market.
  • Testing and training. Some online qualitative research approaches rely on external technology platforms. Make sure you build in a comprehensive testing phase to ensure these work across different browsers and in different markets, particularly if you’re using a tool for the first time. You should also consider preparing training materials for respondents. What might seem intuitive to us as researchers, may be not be as straight forward for the person taking part in the research.

How to moderate online research

Moderation is arguably even more important online than it is offline. So what should you bear in mind?

  1. Don’t forget the basics. Just because the research is taking place online doesn’t mean that you can forget everything you’ve learnt about great moderation. Dress appropriately and try and mimic face-to-face interactions – even something as simple as waving hello at the start of an interview can help to build rapport and set someone at ease.
  2. Read the (virtual) room. Sometimes what people choose not to say can be as revealing as what they do.
  3. Do your homework. Earlier in this guide, we mentioned the value that pre-tasks can bring. Even a short WhatsApp exchange can be useful in helping you to understand more about a respondent and how they feel about certain topics so you can get the most out of them in the session.
  4. Curate the conversation. Unlike a focus group or an interview which might only last 90 minutes, online communities take place over a longer period and therefore require ongoing moderation. Moderators should think about how they curate the conversation by connecting individual respondents up to foster group discussion.
  5. Conduct research in the local language. If you’re running a project in another country, use local language moderators who’ll be able to understand and draw out local nuances.

Analysing online research projects

Online market research is often considered to deliver more bang for your buck than a face-to-face approach because of the sheer amount of data you can collect. This is particularly true of online communities due to their longitudinal nature. As such, you need to carefully structure your analysis, ensuring you’re focusing on the ‘so what’ for your stakeholders.

There are also cultural considerations to take into account when it comes to analysis. You’ll need to bear in mind that consumers in different markets can answer questions differently when you’re interpreting the data. For instance, when asking about interest levels or purchase intent, the figures you see in ASEAN developing countries tend to be higher than in other markets. Even within markets, there can be regional differences to take account of. In Vietnam, for instance, there are cultural differences between consumers in the two major cities – Hanoi in the North and Ho Chi Minh in the South. Those in the North tend to favour products from well-known brands, while those in the South are more open to trying new things, and this is reflected in the data you see from these areas.

Considering harnessing online research in your own organisation?

This is where we’re in our element. Find out about some of our online methodologies like online surveys or online communities or get in touch to discuss a project.

In the world of market research, we can only get so far by relying on hard, numerical data.

Hard metrics like (generated from quantitative research) are extremely useful and should form a core part of any business strategy. But they only tell part of the overall story.

To dig deeper and gain a fuller picture of why our customers behave the way they do, it’s important to consider supplementing quantitative research with a more  qualitative approach. Qualitative research is based on conversational and open-ended communication and aims to dive a little deeper than quantitative metrics and explore the why behind customer’s actions.

If you want to get the most out of your research, you should be using both approaches. In this guide, we’ll take a look at what qualitative research is, what makes it so useful, and how you can employ it in your own work.

How is qualitative research different from quantitative research?

Quantitative research:

  • Is more data-based, relying on hard data points and objective measurements
  • It uses statistics and numerical data to identify trends and patterns
  • Allows you to quickly establish what’s happening, and look at possible causes 

Quantitative studies are extremely valuable. They allow us to gain a reliable, accurate understanding of what’s happening in our market  and amongst our customers, and make clear-headed decisions that influence the bigger picture. But quantitative data alone isn’t enough.

Qualitative research is more human-focused. It’s less concerned with numbers and figures, and more focused on what customers have to say. It can take the form of interviews, focus groups or online communities  and its goal is to dig into the more intangible and subjective reasons why customers behave the way they do.

Why is qualitative research useful?

Qualitative research is useful because it helps us dive into the human factors driving our customers’ actions. People are complex and often unpredictable, and our behaviour can’t really be boiled down into a series of metrics..

For example, we might know that sales for one product are outperforming another. But why is this happening? Our hard metrics can show us the overall trend and might allow us to pinpoint certain glaring patterns, but they don’t tell us what’s going on in our customers’ minds.

For this, we need qualitative studies. We need to gain insight into the microtrends that lie beneath bigger patterns. 

The benefits don’t end there, though. Qualitative research means getting to know your customers and their motivations better. Here’s how that helps:

  • It can help you to understand customer needs, generating new ideas for products and services. 
  • It can provide valuable feedback on your existing offering. Using qualitative research you can explore pain points and barriers to use, helping you understand how to improve your current products and services.
  • It can be a useful input to your marketing. By truly understanding your audience, you can take a more personalized approach, speaking their language and talking to your customers in a way they can really relate to. It can also provide useful input to campaign or content development. By understanding customer needs you can create marketing content that solves specific problems for your audience and delivers real value in response to the challenges they face and the pain points they grapple with.
Stay ahead

Get regular insights

Keep up to date with the latest insights from our research as well as all our company news in our free monthly newsletter.

Qualitative research methods 

Qualitative research is made up of a range of different methods and techniques. Each has its own use cases, and the best approaches will combine several methodologies based on your customers and your goals. Here are some of the main methods:

  • Focus groups. This is where you bring a small number of customers (usually less than 15) together in a group to discuss a particular issue. By tapping into the power of group dynamics, we’re able to uncover rich insights around attitudes and behaviours, and explore underlying motivations, need states and perceptions.
  • One-to-one, in-depth interviews. Here, researchers speak to customers directly, in a one-to-one setting. It’s a good way to get truly in-depth on a topic, delving into the participant’s opinions and gaining valuable feedback and insight. In depth interviews can be carried in person, on the phone or online. 
  • Expert interviews. Similar to in-depth interviews, expert interviews involve speaking to industry experts to build a rich understanding of the market and where it’s heading. This approach can help you explore the impact of emerging trends to help future proof your business.
  • Ethnography. This is where researchers immerse themselves in customers’ worlds to understand more about their day to day lives and the role that brands and products play. Ethnography can take different forms, from visiting consumers and accompanying them as they go about their day, to mobile self-ethnography where consumers complete video tasks to show us how they live. 
  • Online communities. This is where groups of consumers are brought together over a series of days on an online platform to explore specific issues. Consumers then complete individual or group tasks, enabling the researcher to uncover rich insights. Like mobile self-ethnography, online communities can involve photo and video tasks and are a great way of bringing an audience to life for key stakeholders. What’s more, as online communities consumers over a longer time period than an in-depth interview or a focus group, they allow you to explore complex or sensitive issues and uncover deep insights into attitudes and values to inform your decision-making.

Traditionally qualitative research was done according to the grounded theory method. This is a framework for research that involves collecting qualitative data through the above methods and then using that data to form a theory or hypothesis. However, it’s easy to underestimate the sheer amount of data you can collect through qualitative research and this is particularly true of online methods such as online communities. As such, it’s often not feasible to use the grounded theory method. At Kadence we take a different and more structured approach, exploring hypotheses with key stakeholders and designing the research so that we can test these. This means that the research is tightly focused on the areas that matter most to stakeholders, ensuring that the insights we uncover are actionable.  

Some examples of qualitative research questions you might ask:

  • How important is corporate responsibility to our customers?
  • What are the main reasons people use social media?
  • Why do people want to work for our organization?
  • How do adult males feel about hair loss?
  • What are the key motivations for  undertaking a weight loss programme?

Qualitative research is essential if you want to truly understand your customers and improve your product or service to deliver what they want and need. It goes hand in hand with more quantitative methods of research and helps add context, explanation, and depth to the more numerical and data-based metrics.

At Kadence, we can help you get the most out of qualitative research, to better understand your customers and market on all levels. To find out how, get in touch with us.

In the present circumstances, many of our clients are asking us questions about online research, both in the short and the long-term, in addition to more traditional techniques. Across our offices we’re experienced in a range of methodologies including online communities, video depth interviews and digital ethnographies. In this short video our Global Head of Qualitative Research explains some of the options available and how they work.

Stay ahead

Get regular insights

Keep up to date with the latest insights from our research as well as all our company news in our free monthly newsletter.

Imagine you’re a digital marketer for an online retailer specialising in fitness gear. You’ve just launched a new line of eco-friendly yoga mats, and you’re tasked with maximising sales through your website. You test two different product page versions to see which drives more purchases. 

Version A features a prominent “Limited Time Offer” banner at the top, while Version B includes a series of customer testimonials right beneath the product title. The results of this A/B test could significantly affect your sales figures and offer deeper insights into what motivates your customers to buy.

Such is the power of A/B testing, a method companies of all sizes use to make data-driven decisions that refine user experiences and improve conversion rates. 

A/B testing provides a data-driven solution to optimise website effectiveness without the guesswork. By comparing two versions of a page or element directly against each other, brands can see which changes produce positive outcomes and which ones do not, leading to better business results and a deeper understanding of customer behaviour.

Whether you’re looking to increase conversion rates, enhance user engagement, or drive more sales, effective A/B testing is the key to achieving your goals precisely and confidently.

A/B testing, or split testing, is a method in which two versions of a webpage or app are compared to determine which performs better. Imagine you’re at the helm of a ship; A/B testing gives you the navigational tools to steer more accurately toward your desired destination—increased sales, more sign-ups, or any other business goal. It involves showing the original version (A) and a modified version (B), where a single element may differ, such as the colour of a call-to-action button or the layout of a landing page, to similar visitors simultaneously. The version that outperforms the other in achieving a predetermined goal is then used moving forward.

›

The Importance of A/B testing and ROI

The compelling advantage of A/B testing is its direct contribution to enhancing business metrics and boosting return on investment (ROI). 

A recent study by eConsultancy shows that companies that use A/B testing are twice as likely to report significant improvements in conversion rates. 

Online retailers frequently use A/B testing to optimise website leads and increase conversion rates. This includes split testing product pages and online advertisements, such as Google Shopping Ads. By A/B testing different product page layouts, retailers can identify a version that increases their sales, impacting annual revenue. Similarly, SaaS providers test and optimise their landing pages through A/B testing to find the version that increases user sign-ups, directly improving their bottom line.

A/B testing is less about guessing and more about evidence-based decision-making, ensuring every change to your interface is a strategic enhancement, not just a cosmetic tweak.

Preparing for A/B Testing

1. Setting Objectives

Before launching an A/B test, defining clear, measurable objectives is critical. These objectives should be specific, quantifiable, and aligned with broader business goals. Common goals include increasing conversion rates, reducing bounce rates, or boosting the average order value. The clarity of these objectives determines the test’s focus and, ultimately, its success.

2. Identifying Key Elements to Test

Choosing the right elements on your website for A/B testing can significantly affect the outcome. High-impact elements often include:

  • CTAs: Testing variations in the text, color, or size of buttons to see which drives more clicks.
  • Layouts: Comparing different arrangements of elements on a page to determine which layout keeps visitors engaged longer.
  • Content: Tweaking headlines, product descriptions, or the length of informational content to optimise readability and conversion.
  • Images and Videos: Assessing different images or video styles to see which leads to higher engagement or sales.

3. Understanding Your Audience

Effective A/B testing requires a deep understanding of your target audience. Knowing who your users are, what they value, and how they interact with your website can guide what you test and how you interpret the data from those tests.

Data Analytics Snapshots:

Utilising tools like Google Analytics, heatmaps, or session recordings can provide insights into user behaviour. Heatmaps, for example, can show where users are most likely to click, how far they scroll, and which parts of your site draw the most attention. These tools can highlight areas of the site that are performing well or underperforming, guiding where to focus your testing efforts.

Importance of Audience Insights:

Understanding user behaviour through these tools helps tailor the A/B testing efforts to meet your audience’s needs and preferences, leading to more successful outcomes. For instance, if heatmaps show that users frequently abandon a long signup form, testing shorter versions or different layouts of the form could reduce bounce rates and increase conversions.

These preparatory steps—setting objectives, identifying key elements, and understanding the audience—create a strong foundation for successful A/B testing. By meticulously planning and aligning tests with strategic business goals, companies can ensure that their efforts lead to valuable, actionable insights that drive growth and improvement.

Designing A/B Tests

Developing Hypotheses

A well-crafted hypothesis is the cornerstone of any successful A/B test. It sets the stage for what you’re testing and predicts the outcome. A strong hypothesis is based on data-driven insights and clearly states what change is being tested, why, and its expected impact.

Guidance on Formulating Hypotheses:

  • Start with Data: Analyze your current data to identify trends and areas for improvement. For instance, if data shows a high exit rate from a checkout page, you might hypothesise that simplifying the page could retain more visitors.
  • Be Specific: A hypothesis should clearly state the expected change. For example, “Changing the CTA button from green to red will increase click-through rates by 5%,” rather than “Changing the CTA button colour will make it more noticeable.”
  • Link to Business Goals: Ensure the hypothesis aligns with broader business objectives, enhancing its relevance and priority.

Examples:

  • Good Hypothesis: “Adding customer testimonials to the product page will increase conversions by 10% because trust signals boost buyer confidence.”
  • Poor Hypothesis: “Changing things on the product page will improve it.”

Creating Variations

Once you have a solid hypothesis, the next step is to create the variations that will be tested. This involves tweaking one or more elements on your webpage based on your hypothesis.

Instructions for Creating Variations:

  • Single Variable at a Time: To understand what changes affect outcomes, modify only one variable per test. If testing a CTA button, change the color or the text, but not both simultaneously.
  • Use Design Tools: Utilise web design tools to create these variations. Ensure that the changes remain true to your brand’s style and are visually appealing.
  • Preview and Test Internally: Before going live, preview variations internally to catch potential issues.

Choosing the Right Tools

Selecting the appropriate tools is crucial for effectively running A/B tests. The right tool can simplify testing, provide accurate data, and help interpret results effectively.

By following these steps—developing a strong hypothesis, creating thoughtful variations, and choosing the right tools—you can design effective A/B tests that lead to meaningful insights and significant improvements in website performance. This strategic approach ensures that each test is set up for success, contributing to better user experiences and increased business outcomes.

Implementing A/B Tests

Effective implementation of A/B tests is critical to achieving reliable results that can inform strategic decisions. 

Test Setup and Configuration

Setting up an A/B test properly ensures that the data you collect is accurate and that the test runs smoothly without affecting the user experience negatively.

Step-by-step Guide on Setting Up Tests:

  • Define Your Control and Variation: Start by identifying your control version (the current version) and the variation that includes the changes based on your hypothesis.
  • Choose the Type of Test: Decide whether you need a simple A/B test or a more complex split URL test. Split URL testing is useful when major changes are tested, as it redirects visitors to a different URL.
  • Set Up the Test in Your Chosen Tool: Using a platform like Google Optimise, create your experiment by setting up the control and variations. Input the URLs for each and define the percentage of traffic directed to each version.
  • Implement Tracking: Ensure that your analytics tracking is correctly set up to measure results from each test version. This may involve configuring goals in Google Analytics or custom-tracking events.

Interactive Checklists or Setup Diagrams:

A checklist can help ensure all steps are followed, such as:

  • Define control and variation
  • Choose testing type
  • Configure the test in the tool
  • Set traffic allocation
  • Implement tracking codes

Best Practices for Running Tests

Once your test is live, managing it effectively is key to obtaining useful data.

Tips for Managing and Monitoring A/B Tests:

  • Monitor Performance Regularly: Check the performance of your test at regular intervals to ensure there are no unexpected issues.
  • Allow Sufficient Run Time: Let the test run long enough to reach statistical significance, usually until the results stabilise. You have enough data to make a confident decision.
  • Be Prepared to Iterate: Depending on the results, be prepared to make further adjustments and rerun the test. Optimisation is an ongoing process.

Visual Dos and Don’ts Infographics

To help visualise best practices, create an infographic that highlights the dos and don’ts:

  • Do: Test one change at a time, ensure tests are statistically significant, and use clear success metrics.
  • Don’t Change multiple elements at once, end tests prematurely, and ignore variations in user behaviour.

Statistical Significance and Sample Size

Understanding these concepts is crucial for interpreting A/B test results accurately.

Explanation of Key Statistical Concepts:

  • Statistical Significance: This measures whether the outcome of your test is likely due to the changes made rather than random chance. Typically, a result is considered statistically significant if the probability of the result occurring by chance is less than 5%.
  • Sample Size: The number of users you need in your test to reliably detect a difference between versions. A sample size that is too small may not accurately reflect the broader audience.

Graphs and Calculators:

  • Provide a graph showing how increasing sample size reduces the margin of error, enhancing confidence in the results.
  • Link to or embed a sample size calculator, allowing users to input their data (like baseline conversion rate and expected improvement) to determine how long to run their tests.

By following these guidelines and utilising the right tools and methodologies, you can implement A/B tests that provide valuable insights into user behavior and preferences, enabling data-driven decision-making that boosts user engagement and business performance.

Analyzing Test Results

Once your A/B test has concluded, the next crucial step is analyzing the results. This phase is about interpreting the data collected, understanding the statistical relevance of the findings, and making informed decisions based on the test outcomes.

Interpreting Data

Interpreting the results of an A/B test involves more than just identifying which variation performed better. It requires a detailed analysis to understand why certain outcomes occurred and how they can inform future business decisions.

How to Read Test Results:

  • Conversion Rates: Compare the conversion rates of each variation against the control. Look not only at which had the highest rate but also consider the context of the changes made.
  • Segmented Results: Break down the data by different demographics, device types, or user behaviours to see if there are significant differences in how certain groups reacted to the variations.
  • Consistency Over Time: Evaluate how the results varied over the course of the test to identify any patterns that could influence your interpretation, such as a weekend vs. weekday performance.

Statistical Analysis

A deeper dive into the statistical analysis will confirm whether the observed differences in your A/B test results are statistically significant and not just due to random chance.

Understanding Statistical Significance and Other Metrics:

  • P-value: This metric helps determine the significance of your results. A p-value less than 0.05 typically indicates that the differences are statistically significant.
  • Confidence Interval: This range estimates where the true conversion rate lies with a certain level of confidence, usually 95%.
  • Lift: This is the percentage increase or decrease in the performance metric you are testing for, calculated from the baseline of the control group.

Making Informed Decisions

With the data interpreted and the statistical analysis complete, the final step is to decide how to act on the insights gained from your A/B test.

Guidelines on How to Act on Test Results:

  • Implement Winning Variations: If one variation significantly outperforms the control, consider implementing it across the site.
  • Further Testing: If results are inconclusive or the lift is minimal, running additional tests with adjusted variables or targeting a different user segment may be beneficial.
  • Scale or Pivot: Depending on the impact of the changes tested, decide whether to scale these changes up to affect more of your business or to pivot and try a different approach entirely.

Decision Trees or Flowcharts:

Create a decision tree or flowchart that outlines the decision-making process following an A/B test. This could include nodes that consider whether the test was statistically significant, whether the results align with business goals, and what follow-up actions (like further testing, full implementation, or abandonment of the change) should be taken based on different scenarios.

By thoroughly analyzing A/B test results through data interpretation, statistical analysis, and strategic decision-making, organisations can ensure that they are making informed decisions that will enhance their website’s user experience and improve overall business performance. This data-driven approach minimises risks associated with website changes and ensures that resources are invested in modifications that provide real value.

Beyond Basic A/B Testing

Once you have mastered basic A/B testing, you can explore more sophisticated techniques that offer deeper insights and potentially greater improvements in user experience and conversion rates. This section delves into advanced testing strategies and the importance of ongoing optimisation through iterative testing.

Advanced Testing Techniques

Advanced testing methods allow you to explore more complex hypotheses about user behaviour and website performance, often involving multiple variables or entire user journeys.

Multivariate Testing (MVT):

  • Overview: Unlike A/B testing, which tests one variable at a time, multivariate testing allows you to test multiple variables simultaneously to see which combination produces the best outcome.
  • Application: For example, you might test different versions of an image, headline, and button on a landing page all at once to determine the best combination of elements.
  • Benefits: This approach can significantly speed up the testing process and is particularly useful for optimising pages with multiple elements of interest.

Multipage Testing:

  • Overview: Also known as “funnel testing,” this technique involves testing variations across multiple pages that make up a user journey or funnel.
  • Application: You might test variations of both the product and checkout pages to see which combination leads to higher conversion rates.
  • Benefits: Multipage testing helps ensure consistency in messaging and user experience across multiple stages of the user journey, which can improve overall conversion rates.

Continuous Improvement and Iteration

The goal of A/B testing is not just to find a winning variation but to continually refine and enhance your website based on user feedback and behaviour.

Importance of Ongoing Optimisation:

  • Iterative Process: Optimisation is an ongoing process that involves continually testing and refining website elements based on user data and business objectives.
  • Learning from Each Test: Each test provides valuable insights into whether a variation wins. These insights can inform future tests, leading to better user experiences and higher conversion rates.

Iterative Testing Strategies:

  • Start with Broad Tests: Begin with broader tests to identify which elements have the most significant impact on user behaviour.
  • Refine and Repeat: Use the insights gained to refine your hypotheses and test more specific variations.
  • Expand Testing: Once you’ve optimised major elements, expand your testing to less prominent components that could still affect user experience and conversions.

Timelines and Case Studies:

  • Timeline Example: Show a timeline that outlines an annual testing strategy, with phases for broad testing, refinement, and expansion.
  • Case Study: Present a case study of a company that implemented continuous testing. Highlight how iterative testing helped them achieve a significant, sustained increase in conversion rates over time. For instance, a tech company could use iterative testing to fine-tune its sign-up process, resulting in a 50% increase in user registrations over a year.

By advancing beyond basic A/B testing and embracing more complex and continuous testing strategies, companies can optimise their websites more effectively and foster a culture of data-driven decision-making. This approach leads to improvements that align with user preferences and business goals, ensuring sustained growth and a competitive edge in the market.

Common Pitfalls and How to Avoid Them

A/B testing is a powerful tool for website optimisation, but common pitfalls can undermine its effectiveness. This section explores typical errors that occur during the testing process and provides strategies to ensure the validity and reliability of your tests.

List of Common Mistakes

Identifying Errors and Solutions:

  • Testing Too Many Changes at Once: It can make determining which change affected the outcome difficult.
    • Solution: Focus on testing one change at a time or use multivariate testing for simultaneous changes and analyze the impact of each element separately.
  • Not Allowing Enough Time for the Test to Run: Ending a test too soon can lead to conclusions that aren’t statistically significant.
    • Solution: Ensure each test runs long enough to collect adequate data, reaching statistical significance before making decisions.
  • Testing Without a Clear Hypothesis: Starting tests without a clear, data-backed hypothesis leads to unclear outcomes.
    • Solution: Develop a precise hypothesis for each test based on thorough data analysis and clear business objectives.
  • Ignoring User Segmentation: Different segments may react differently to the same change.
    • Solution: Segment your audience and analyze how different groups respond to each variation.

Visuals of Pitfalls vs. Best Practices:

  • Create side-by-side infographics showing examples of these mistakes versus best practices. For example, visually compare the outcome of a test that changed multiple elements simultaneously against one that tested a single change.

Ensuring Validity and Reliability

Maintaining the integrity of your A/B tests is crucial for obtaining reliable, actionable insights.

Tips on Maintaining Test Integrity:

  • Use Proper Randomisation: Ensure that the distribution of users between the control and test groups is random to avoid selection bias.
    • Tool Tip: Utilise tools that automatically handle randomisation to avoid manual errors.
  • Control External Factors: Holidays, marketing campaigns, or significant news events can skew test results.
    • Solution: Monitor external factors, adjust the testing period, or filter the data to account for anomalies.
  • Ensure Consistent Test Conditions: Changes in the testing environment or platform during the test can invalidate results.
    • Solution: Keep the testing conditions consistent throughout the test period and verify configuration settings regularly.
  • Validate Test Setup Before Going Live: A misconfigured test can lead to incorrect data interpretation.
    • Solution: Run a smaller pilot test or use a checklist to ensure every test element is correctly set up before full deployment.

Troubleshooting Guide with Graphic Aids:

  • Develop a troubleshooting guide that includes common scenarios where A/B test integrity might be compromised. Include flowcharts or decision trees that help identify and resolve issues such as data discrepancies, unexpected user behaviour, or sudden changes in conversion rates.
  • Example Graphic Aid: A flowchart that helps determine actions when test results seem inconsistent with historical data or benchmarks. Steps might include checking configuration settings, reviewing segmentation criteria, or extending the test duration.

By understanding and avoiding these common pitfalls and maintaining rigorous standards for validity and reliability, organisations can ensure that their A/B testing efforts lead to meaningful improvements and robust data-driven decisions. This approach not only enhances the effectiveness of current tests but also builds a foundation for future testing strategies that are even more successful.

A/B Testing Case Studies

A/B testing has proven to be a critical tool for businesses aiming to optimise their online presence based on data-driven decisions. Here, we delve into some specific real-life case studies from different industries, highlighting the successes and lessons from A/B testing.

Success Stories

E-commerce: Humana

  • Overview: Humana, a well-known health insurance company, conducted an A/B test to increase click-through rates on one of their primary campaign landing pages. They tested the simplicity and message of their banner and CTA.
  • Changes Tested: The original banner had a lot of information and a standard “Shop Medicare Plans” button. The test variation simplified the message and changed the button text to “Get Started Now.”
  • Results: The variation led to a 433% increase in click-through rates to the insurance plans page.

B2B: SAP

  • Overview: SAP, a leader in enterprise application software, tested the copy of their CTA on a product page. The hypothesis was that a more action-oriented CTA would increase engagement.
  • Changes Tested: The original CTA read “Learn more,” which was changed to “See it in action” in the variation.
  • Results: This simple change in wording resulted in a 32% increase in clicks.

.

Digital Media: The Guardian

  • Overview: The Guardian tested different wordings for their support and donation CTAs to determine which would more effectively encourage readers to contribute financially.
  • Results: The test revealed that a direct ask for contributions using emotive language resulted in a higher click-through rate than a more generic request for support.
  • Lesson: This A/B test highlighted the importance of emotional resonance in messaging, especially for non-profit or cause-based initiatives.

Travel Industry: Expedia

  • Overview: Expedia conducted A/B testing to optimise hotel booking conversions on their site by altering the display of discount offers.
  • Changes Tested: They tested the visibility and presentation of savings messages (e.g., showing a percentage off versus a specific dollar amount saved).
  • Results: Showing the amount of money saved led to a slight decrease in conversion rates, contrary to expectations.
  • Lesson: The test underscored the potential for “over-optimising” to backfire and the need to balance how offers are presented to avoid overwhelming customers.

Final Checklist of A/B Testing Steps

To help ensure your A/B testing journey is structured and effective, here is a visual checklist encapsulating the process:

  1. Define Objectives: Clearly state what you aim to achieve.
  2. Formulate Hypotheses: Base your assumptions on data and prior insights.
  3. Select the Testing Tool: Choose a platform that suits your scale and complexity needs.
  4. Design the Test: Create variations based precisely on your hypotheses.
  5. Run the Test: Ensure the test is long enough to gather meaningful data.
  6. Analyze Results: Use statistical analysis to interpret the outcomes.
  7. Implement Changes: Apply successful variations or further refine and test.
  8. Repeat: Use insights gained to continuously improve further testing.

Regardless of the outcome, every test is a step forward in understanding your users better and refining your digital offerings to meet their needs more effectively. The journey of optimisation is continuous, and each effort builds upon the last, opening new doors to innovation and growth.

Harness the power of A/B testing to start making informed decisions that propel your business forward. Your next breakthrough could be just one test away.