We recently had an internal brainstorm around the topic of luxury. One of my colleagues mentioned that they thought that Calvin Klein as a fashion brand was no longer considered luxury but was at the height of luxury in the 90s. Alongside this, if you search for luxury brands in Google you get a list of the top brands by brand value in 2017 which includes the likes of Mulberry. If you looked at this 20 years ago, it would arguably have been different and would have included brands like Galliano or Lacroix that are less top of mind today.

Here at Kadence this got us thinking – what makes a brand ‘luxury’ and why do brands that consumers perceive as luxurious change over the years?

Looking first into what makes a brand ‘luxurious’ – we believe there are three main drivers:

  • High price: this doesn’t necessarily mean a high overall price, but a high price compared to other brands within that sector. For example, categories such as soap includes brands such as Molton Brown and L’Occitane that could be considered luxurious but may ‘only’ cost £20 compared to other categories like shoes or watches. As a side note, the quality also obviously arguably need to at least match expectations, especially if customers are spending large amounts.
  • Limited supply: luxury doesn’t have to be expensive but it arguably shouldn’t be easily obtainable. Sometimes a luxury brand can be one that not many people know about (like Delvaux or Serapian handbags) – adding to the feeling that you’re special and no one else really has an item like you have. Another example is the long waiting lists for Hermes Birkin bags – by limiting supply, they increase desirability.
  • Endorsement by celebrities: for some brands it is an A-list celebrity, for others a football player, but an endorsement by a celebrity can make items more appealing for consumers and potentially more ‘luxurious’ in their eyes. Brands do have to be careful though – Juicy Couture were propelled to fame by the likes of Paris Hilton, but that also put plenty of people off the brand too…

But we don’t think it stops there. Other elements can also come into play:

  • The packaging: brands are doing some really nice things with packaging nowadays. In Fall 2014, Chanel released a supermarket collection of bags – the packaging that they came in represented a supermarket carrier bag and the tag represented a barcode, all in-keeping with the overall theme of that season. And more broadly, when you buy an item of clothing from Net-a-porter they will put it in a box with a ribbon, and Christian Louboutin shoes come with a lovely red fabric bag. This makes the unboxing and unwrapping experience for the customer more memorable and special and can add that ‘luxury feel’.
  • Personalisation: customisation of a product is a great way to make the customer feel special and one of a kind. The fact that you know no one else has a product like yours makes the product a bit more luxurious. Kate Spade and Louis Vuitton are great examples of brands that have done personalisation well – allowing you to monogram your handbags or add a variety of ‘patches’ to them.

Why do brands become more or less luxurious? Michael Kors is a great example of how trying to reach the masses can arguably make you appear less luxurious – they released their ‘diffusion line’ MICHAEL Michael Kors which priced all goods to match the upper end of high street prices to cater for consumers shopping in high-street shops as well as larger department stores. This was very successful for them in terms of sales at first, but they have now slipped down to a more “casual luxury” brand forfeiting sales overall. In June 2017, Michael Kors announced that they been battling declining same-store sales for the past seven quarters with continuing decline into 2018. It was also announced that the retailer would shut more than 100 full-price retail stores in the next two years, and that their share prices are at their lowest in more than five years. In some ways they could be seen as suffering as a victim to their own success.

Brands will need to think carefully about each of these themes as the luxury sector continues to grow and develop, and these elements will differ depending on the product and category, which is where research comes in – we can actually speak to customers first-hand and unpick which elements are most important to them, uncovering brand perceptions and share of the market.

Nowadays people have very hectic and busy lifestyles – trying to balance work, home and a social life is increasingly challenging. However, companies are becoming more attuned to this and starting to adapt their products to suit.

Lack of time is typically the main reason for lack of participation in sport or gym attendance. However, it remains hugely popular, with the number of people taking part every week reaching around 16 million and the number of fitness centre members in the UK being the highest ever.

Maybe this is because gyms are increasing the number of classes or reducing their length to be more flexible. For example, some gyms are now promoting 30 minute classes, which are easier to squeeze into your lunch break. Some offices even have gyms that you can pop to whenever in the working day to help cater for busy lifestyles. Also having exercise studios entirely dedicated to one activity (such as yoga) means there are more classes for people to choose from.

Participation in high-intensity exercise classes and activities, like spin and boxercise, continue to attract large numbers of people, potentially rivalling more traditional sports such as football, netball and hockey. Outdoor activities such as bootcamp and Parkrun are also increasing in popularity which are seen as a highly sociable way to exercise. Maybe this is people’s way of ticking both the ‘exercise’ and ‘socialise’ boxes on their to-do lists?

More established activities like yoga and pilates are still gaining in popularity as well. These classes are expected to provide both physical and mental benefits – such as increased muscle strength and tone, improved athletic performance, stress relief and encouraging relaxation. People can therefore achieve a healthier lifestyle in a more compact way, which saves time relative to completing several separate activities to achieve the same benefits.

As well as exercise companies, nutrition companies are also adapting to people’s hectic lifestyles. Nutrition is becoming more of a focus and healthy eating is a main element of having a healthy lifestyle. We have therefore seen the rise in trends such as The Body Coach, Joe Wicks, who promises that you’ll be able to lose weight, despite eating more food and spending less time at the gym. He also has a 15 minute meals cookbook that caters for those lacking time in the evenings and suggests quick exercises you can do in the comfort of your own home.

Adapting to hectic lifestyles has also paved the way for food companies such as HelloFresh or Gousto. Their USP is that they deliver fresh ingredients and healthy recipes straight to your doorstep. This means you can get back from a busy day at work and have your dinner all bought and planned out for you – all you have to do is follow the recipe. The next step on from this is Deliveroo that actually delivers healthy food options straight to the doorstep of either your home or office ready for consumption! So now when you get home from work late and cooking is the last thing on your mind, instead of the traditional Chinese or Indian takeaways you can now get Pho or Sushi delivered to your door!

So having a healthy lifestyle doesn’t appear to be a fad or a trend, it’s the way things are nowadays. We have already seen intersections between health and exercise but this also begs the question – where will this go? What’s next for healthy lifestyles?

Maybe exercise and retail companies will start partnering up. For example, introducing grocery stores at the gym to save having to do a food shop later that evening. Or they could pair up in a totally different way. It may be counter-intuitive but BeerYoga is something that I stumbled upon recently where attendees can drink beer whilst doing yoga. Now this ticks all the boxes – social, mental and physical. Will that mean that WinePilates or SushiSpin emerge as trends as well? It’s fair to say some of these may seem slightly odd now but many fads do before they become accepted into society.

There’s a great book by Columbia Business School associate professor, William Duggan, called Strategic Intuition. The book posits that intuition is ‘the selective recombination of previous elements into a new whole.’

One of Duggan’s examples of intuitive thinkers is one of Napoleon’s early campaigns. When ordered to re-take the port of Toulon from the British invaders via frontal assault ‘with the sword and bayonet’, Napoleon suggests an alternative strategy; to take the smaller fort of L’Agiuilelette, which overlooks the port of Toulon. Against the received wisdom of his peers and commanders, Napoleon goes ahead with his plan, takes the fort, and in doing so terrifies the British into leaving Toulon and sets his path for Emperor of Europe.

What’s interesting here – and why Duggan raises this example from history – is how Napoleon came to his plan, by bringing together abstract parts of his memory and experience: his reading of the contour maps on the area of Toulon; his knowledge of how best to deploy light cannon and his understanding of past British defeats. The contour maps showed him that the fort of L’Aiguilette occupied high ground over Toulon; from his light cannon experience he knew he could take the cannon up to the fort and deploy them overlooking Toulon and the British fleet; and his understanding of past British defeats at Yorktown and the Siege of Boston taught him that the British would never again risk being cut off from their navy.

None of those thoughts – contour maps; light cannon; British defeats – were taught to Napoleon together. Instead, as Duggan argues, it is the ‘selective recombination of previous elements into a whole.’

Have you ever had an idea flash into your mind? A random thought disconnected from what you’re trying to concentrate on – that’s strategic intuition. And it’s proof that our brains are non-linear. Try as we might, we struggle to focus on a single thought for a long period of time. Rather our brains are adept at working in our subconscious and delivering fresh ideas and insights at a moment’s notice.

You can’t help but think of multiple things at once, or just as likely thinking multiple things about one idea at once. In contrast, however, Word, PowerPoint and Excel, by their very nature, are linear. Word splits information over different pages, PowerPoint chunks information in slides and Excel breaks then across tabs. And all are subject to the limitations of screen size. This linear function is in direct odds to the brain’s non-linear thinking, forcing you to work to their restrictions.

By chunking information into different pages, slides or tabs it also forces the brain to change its functioning.  When all the information is displayed at once, the brain can focus on analysis and connecting information. We can tap into the very strength of our brain – making random, subconscious associations. However, when chunked over different pages, slides and tabs the brain must first remember all the information it has been exposed to before it can then begin to analyse and connect it. This exerts increased cognitive load on the brain and causes a significant break down in your brain’s ability to create those connections.

So, what if we moved away from linear formats and embraced our brain’s capacity for non-linear thinking, for sparks of insight. This is where the power of Post-It notes – or record cards, or just scraps of paper – come in. 

When first planning or thinking of an idea or concept, a good idea is to plot your thoughts on Post-It notes. Each one holding one thought. And filling your table, or wall, or desk with them. The beauty of this is that it embraces our non-linear brain. A random thought or idea can be jotted down and placed to the side, not distracting your attention by needling your mind but also addressed and captured quickly during it’s fleeting appearance. Overall, Post-It planning seems to help in three main ways:

Making connections

Post-It notes and record cards allow for the creation of non-linear narratives. With the use of Post-It notes, or record cards and a box of pins, you can map out an entire concept visually, highlighting interconnecting thoughts and relationships – celebrating the very non-linear thinking our brains champion and computers cannot copy. As an individual activity working with Post-Its allows us to re-arrange ideas as we go. Once we have captured all our thinking on multiple Post-Its we can then begin to rearrange them over and over again, in different orders and ways until we are happy with our outcome.

A free-form structure

By filling a wall with Post-It notes we are avoiding a linear path through the information, rather we are creating a free-form structure. Every time you look at the wall, or return to the room, you can look at the Post-It notes in a different order. And perhaps draw out new meaning or sense from it. It also means you’re not enforcing a structure on others, they too can create their own path through the information – a very effective element when developing ideas and concepts with others.

Fostering collaboration

When working together, cards and post-it notes invite displayed thinking. By committing our thoughts to paper and then arranging them on a wall we can easily invite others to see our thinking; but just as easily others can begin to add to, edit or rationalize our thoughts, so that together we can create a shared cognition about an idea and together create a common understanding. This shared activity fosters creativity, especially as any person can re-arrange cards.

So, before you next fire up your laptop ask yourself, would I be more creative if I used Post-Its and embraced my brain’s non-linear thinking?

Amy Lo explores her personal experiences growing up across two vastly different continents and how these have shaped her insight career. 

When I was 12 my Dad announced I was to leave my home in Taiwan to attend boarding school in England. The resulting 10 years were to shape me in a way neither he, nor I could ever have imagined. 

Growing up across two continents that are so vastly different in terms of culture, climate and consumption meant living in a state of perpetual adaptation, seeking ways to adjust to the environment around me, both at school and then back at home during school holidays. 

I think this constant need to adapt to my surroundings is the reason I first started to closely observe the people around me, their behavior, their motivations, the things that made them similar and the things that made them different.

Skip to the present and as it turns out my fascination with people, their stories, backgrounds and culture has influenced me in my choice of career. As a qualitative market researcher it is my job to investigate the beliefs, perceptions and essential truths behind people’s behavior – and establish patterns amongst them.

I love this career for the opportunity it gives me to gain insight into our respondent’s lives and, of course to deliver Insight Worth Sharing to my clients. 

There is also a lot of variety; during my first role as a Graduate Insight Executive in Taiwan I spent time with a wide variety of respondents from tech-savvy consumers aiming to optimise a mobile-friendly home page for Yahoo to new mums sharing all about their nappy usage. One weekend we would be speaking to HNWIs about luxury holidays and the following weekend, accompanying Chinese teenagers on their hunt for the perfect pair of jeans! 

During a recent project in my current role at Kadence International in London I found myself face to face with my two ‘home’ nations. The study, for a luxury technology brand, involved investigating some of London and Shanghai’s wealthiest individuals and uncovered some vast and fascinating differences in priorities, preferences and behaviours of the Chinese elite against their UK counterparts. 

This study motivated me to understand more about today’s Asian consumer. How can brands adapt their approach to suit this vast and lucrative market? And, how can we as researchers select the best methodologies in order to gather the richest, most valuable insights?

Growing up across two continents that are so vastly different in terms of culture, climate and consumption meant living in a state of perpetual adaptation. I love this career for the opportunity it gives me to gain insight into our respondent’s lives and, of course to deliver Insight Worth Sharing to my clients. With over 700 million Internet users and a little shy of 600 million smartphone users in China (as of 2016), the future of online qual is extremely exciting.

In true millennial style, I started my investigations through my own social network. My friends from Asia were always posting in feeds, reviewing the latest products they have tried. I observed a willingness to share allegiances to particular brands, which doesn’t represent brand loyalty per se, simply that they are not afraid to share their opinions. Many of my female friends have their own blogs, discussing their views on the latest trends in clothes and make up and my feed is regularly inundated with ‘outfit of the day’ posts with links that take you to web shops where you can make a quick purchase from the endorsed brand or seller. 

Surprisingly for China, a country where censorship is widespread, opinions and voices on the Internet are loud and plentiful. Unlike Western countries, there is little trust in traditional media sources such as TV, press or radio. Instead, word of mouth is an increasingly powerful tool, as people use social media platforms to personally share information and opinions with friends and family. 

This trend has been identified by brands in China, who have made it their priority to create intelligent, comprehensive digital campaigns to facilitate the spread of their products or services. This is also why brands are carefully monitoring their e-reputation. Product reviews on the web have a growing influence on people’s decision making. Brands understand the need to nurture advocates within each and every social circle to build credibility and customer proximity. 

Back to my professional experience, working closely with a wide range of Chinese audiences both in Asia and in the UK, I have learnt that I most enjoy using methodologies that give me longer and closer contact with my audience, these allow me to really get to know each and every one of their stories, background and culture. 

Market research online communities offer a highly effective way for UK researchers to gather insight from Chinese audiences. Logistically simple (no working around time differences); methodologically effective (tap into natural online behaviours to provide a truthful engagement with our target audience) and financially efficient (no expensive flights and hotels!). 

Chinese audiences can often be more comfortable providing their opinions via the Internet particularly with certain more sensitive or divisive topics where they can retain a sense of anonymity. With online research methods, there are fewer concerns about their voices or faces being identified – and therefore a greater willingness to share.

With over 700 million Internet users and a little shy of 600 million smartphone users in China (as of 2016), the future of online qual is extremely exciting for me. Mobile devices are the main mode of Internet access and instant messaging is the top online activity in China. Apps such as WeChat are used on a daily basis, just as you and I use WhatsApp to keep in touch with friends and family. WeChat has evolved from a pure instant messaging app to (quoting the FT in April 2016) an app that is a phone, messenger, video conference, ecommerce platform and gaming console, not to mention noodle delivery service, for a nation of people in love with their smartphones. 

Some companies are already using WeChat as a data collection tool for short quantitative surveys, tapping into its mass user base and taking full advantage of its ability to provide instant responses.

And given that the app is already in most people’s pockets means we can largely conduct many of the conventional qualitative methods through WeChat as well. We’re already gaining insights through both interaction and observation, from in-depth interviews to accompanied shopping, to digital diary logging. It’s amazing – but we’re able to follow the steps of Chinese respondents through the lenses of their smartphones from the comfort of their chairs in London. 

The casual nature, accessibility and users’ familiarity with WeChat helps encourage user interaction, engagement and participation, thereby improving our capability to obtain accurate and honest insights. 

The opportunity to use social media platforms for qualitative research is not completely unique to the Chinese market. We know some have been doing focus groups on WhatsApp, and some are using Facebook as a research tool. There is no reason why something similar cannot become a more prevalent research method in the West, provided we have a similar multifunctioning social media platform and the same abundance of users already familiar with the platform.

Personally, I find the possibility of conducting focus groups and in-depth interviews from my iPhone a very exciting prospect. With social media platforms such as WeChat, in a click of a button, I’m in touch with a group of people 5000 miles away, tapping into every aspect and every minute of their lives and uncovering trends through my very own device. I can do this whilst on the go and, when something I see on the street suddenly inspires me, I no longer have to wait until Monday. I can simply pop a question to my group and wait 5 seconds to see what they have to say. 

Looking back, whilst my 12 year old self may have resented my Dad’s decision to send me away from Taiwan to the UK, in hindsight, it was the best decision he ever made.

Imagine you’re a digital marketer for an online retailer specialising in fitness gear. You’ve just launched a new line of eco-friendly yoga mats, and you’re tasked with maximising sales through your website. You test two different product page versions to see which drives more purchases. 

Version A features a prominent “Limited Time Offer” banner at the top, while Version B includes a series of customer testimonials right beneath the product title. The results of this A/B test could significantly affect your sales figures and offer deeper insights into what motivates your customers to buy.

Such is the power of A/B testing, a method companies of all sizes use to make data-driven decisions that refine user experiences and improve conversion rates. 

A/B testing provides a data-driven solution to optimise website effectiveness without the guesswork. By comparing two versions of a page or element directly against each other, brands can see which changes produce positive outcomes and which ones do not, leading to better business results and a deeper understanding of customer behaviour.

Whether you’re looking to increase conversion rates, enhance user engagement, or drive more sales, effective A/B testing is the key to achieving your goals precisely and confidently.

A/B testing, or split testing, is a method in which two versions of a webpage or app are compared to determine which performs better. Imagine you’re at the helm of a ship; A/B testing gives you the navigational tools to steer more accurately toward your desired destination—increased sales, more sign-ups, or any other business goal. It involves showing the original version (A) and a modified version (B), where a single element may differ, such as the colour of a call-to-action button or the layout of a landing page, to similar visitors simultaneously. The version that outperforms the other in achieving a predetermined goal is then used moving forward.

›

The Importance of A/B testing and ROI

The compelling advantage of A/B testing is its direct contribution to enhancing business metrics and boosting return on investment (ROI). 

A recent study by eConsultancy shows that companies that use A/B testing are twice as likely to report significant improvements in conversion rates. 

Online retailers frequently use A/B testing to optimise website leads and increase conversion rates. This includes split testing product pages and online advertisements, such as Google Shopping Ads. By A/B testing different product page layouts, retailers can identify a version that increases their sales, impacting annual revenue. Similarly, SaaS providers test and optimise their landing pages through A/B testing to find the version that increases user sign-ups, directly improving their bottom line.

A/B testing is less about guessing and more about evidence-based decision-making, ensuring every change to your interface is a strategic enhancement, not just a cosmetic tweak.

Preparing for A/B Testing

1. Setting Objectives

Before launching an A/B test, defining clear, measurable objectives is critical. These objectives should be specific, quantifiable, and aligned with broader business goals. Common goals include increasing conversion rates, reducing bounce rates, or boosting the average order value. The clarity of these objectives determines the test’s focus and, ultimately, its success.

2. Identifying Key Elements to Test

Choosing the right elements on your website for A/B testing can significantly affect the outcome. High-impact elements often include:

  • CTAs: Testing variations in the text, color, or size of buttons to see which drives more clicks.
  • Layouts: Comparing different arrangements of elements on a page to determine which layout keeps visitors engaged longer.
  • Content: Tweaking headlines, product descriptions, or the length of informational content to optimise readability and conversion.
  • Images and Videos: Assessing different images or video styles to see which leads to higher engagement or sales.

3. Understanding Your Audience

Effective A/B testing requires a deep understanding of your target audience. Knowing who your users are, what they value, and how they interact with your website can guide what you test and how you interpret the data from those tests.

Data Analytics Snapshots:

Utilising tools like Google Analytics, heatmaps, or session recordings can provide insights into user behaviour. Heatmaps, for example, can show where users are most likely to click, how far they scroll, and which parts of your site draw the most attention. These tools can highlight areas of the site that are performing well or underperforming, guiding where to focus your testing efforts.

Importance of Audience Insights:

Understanding user behaviour through these tools helps tailor the A/B testing efforts to meet your audience’s needs and preferences, leading to more successful outcomes. For instance, if heatmaps show that users frequently abandon a long signup form, testing shorter versions or different layouts of the form could reduce bounce rates and increase conversions.

These preparatory steps—setting objectives, identifying key elements, and understanding the audience—create a strong foundation for successful A/B testing. By meticulously planning and aligning tests with strategic business goals, companies can ensure that their efforts lead to valuable, actionable insights that drive growth and improvement.

Designing A/B Tests

Developing Hypotheses

A well-crafted hypothesis is the cornerstone of any successful A/B test. It sets the stage for what you’re testing and predicts the outcome. A strong hypothesis is based on data-driven insights and clearly states what change is being tested, why, and its expected impact.

Guidance on Formulating Hypotheses:

  • Start with Data: Analyze your current data to identify trends and areas for improvement. For instance, if data shows a high exit rate from a checkout page, you might hypothesise that simplifying the page could retain more visitors.
  • Be Specific: A hypothesis should clearly state the expected change. For example, “Changing the CTA button from green to red will increase click-through rates by 5%,” rather than “Changing the CTA button colour will make it more noticeable.”
  • Link to Business Goals: Ensure the hypothesis aligns with broader business objectives, enhancing its relevance and priority.

Examples:

  • Good Hypothesis: “Adding customer testimonials to the product page will increase conversions by 10% because trust signals boost buyer confidence.”
  • Poor Hypothesis: “Changing things on the product page will improve it.”

Creating Variations

Once you have a solid hypothesis, the next step is to create the variations that will be tested. This involves tweaking one or more elements on your webpage based on your hypothesis.

Instructions for Creating Variations:

  • Single Variable at a Time: To understand what changes affect outcomes, modify only one variable per test. If testing a CTA button, change the color or the text, but not both simultaneously.
  • Use Design Tools: Utilise web design tools to create these variations. Ensure that the changes remain true to your brand’s style and are visually appealing.
  • Preview and Test Internally: Before going live, preview variations internally to catch potential issues.

Choosing the Right Tools

Selecting the appropriate tools is crucial for effectively running A/B tests. The right tool can simplify testing, provide accurate data, and help interpret results effectively.

By following these steps—developing a strong hypothesis, creating thoughtful variations, and choosing the right tools—you can design effective A/B tests that lead to meaningful insights and significant improvements in website performance. This strategic approach ensures that each test is set up for success, contributing to better user experiences and increased business outcomes.

Implementing A/B Tests

Effective implementation of A/B tests is critical to achieving reliable results that can inform strategic decisions. 

Test Setup and Configuration

Setting up an A/B test properly ensures that the data you collect is accurate and that the test runs smoothly without affecting the user experience negatively.

Step-by-step Guide on Setting Up Tests:

  • Define Your Control and Variation: Start by identifying your control version (the current version) and the variation that includes the changes based on your hypothesis.
  • Choose the Type of Test: Decide whether you need a simple A/B test or a more complex split URL test. Split URL testing is useful when major changes are tested, as it redirects visitors to a different URL.
  • Set Up the Test in Your Chosen Tool: Using a platform like Google Optimise, create your experiment by setting up the control and variations. Input the URLs for each and define the percentage of traffic directed to each version.
  • Implement Tracking: Ensure that your analytics tracking is correctly set up to measure results from each test version. This may involve configuring goals in Google Analytics or custom-tracking events.

Interactive Checklists or Setup Diagrams:

A checklist can help ensure all steps are followed, such as:

  • Define control and variation
  • Choose testing type
  • Configure the test in the tool
  • Set traffic allocation
  • Implement tracking codes

Best Practices for Running Tests

Once your test is live, managing it effectively is key to obtaining useful data.

Tips for Managing and Monitoring A/B Tests:

  • Monitor Performance Regularly: Check the performance of your test at regular intervals to ensure there are no unexpected issues.
  • Allow Sufficient Run Time: Let the test run long enough to reach statistical significance, usually until the results stabilise. You have enough data to make a confident decision.
  • Be Prepared to Iterate: Depending on the results, be prepared to make further adjustments and rerun the test. Optimisation is an ongoing process.

Visual Dos and Don’ts Infographics

To help visualise best practices, create an infographic that highlights the dos and don’ts:

  • Do: Test one change at a time, ensure tests are statistically significant, and use clear success metrics.
  • Don’t Change multiple elements at once, end tests prematurely, and ignore variations in user behaviour.

Statistical Significance and Sample Size

Understanding these concepts is crucial for interpreting A/B test results accurately.

Explanation of Key Statistical Concepts:

  • Statistical Significance: This measures whether the outcome of your test is likely due to the changes made rather than random chance. Typically, a result is considered statistically significant if the probability of the result occurring by chance is less than 5%.
  • Sample Size: The number of users you need in your test to reliably detect a difference between versions. A sample size that is too small may not accurately reflect the broader audience.

Graphs and Calculators:

  • Provide a graph showing how increasing sample size reduces the margin of error, enhancing confidence in the results.
  • Link to or embed a sample size calculator, allowing users to input their data (like baseline conversion rate and expected improvement) to determine how long to run their tests.

By following these guidelines and utilising the right tools and methodologies, you can implement A/B tests that provide valuable insights into user behavior and preferences, enabling data-driven decision-making that boosts user engagement and business performance.

Analyzing Test Results

Once your A/B test has concluded, the next crucial step is analyzing the results. This phase is about interpreting the data collected, understanding the statistical relevance of the findings, and making informed decisions based on the test outcomes.

Interpreting Data

Interpreting the results of an A/B test involves more than just identifying which variation performed better. It requires a detailed analysis to understand why certain outcomes occurred and how they can inform future business decisions.

How to Read Test Results:

  • Conversion Rates: Compare the conversion rates of each variation against the control. Look not only at which had the highest rate but also consider the context of the changes made.
  • Segmented Results: Break down the data by different demographics, device types, or user behaviours to see if there are significant differences in how certain groups reacted to the variations.
  • Consistency Over Time: Evaluate how the results varied over the course of the test to identify any patterns that could influence your interpretation, such as a weekend vs. weekday performance.

Statistical Analysis

A deeper dive into the statistical analysis will confirm whether the observed differences in your A/B test results are statistically significant and not just due to random chance.

Understanding Statistical Significance and Other Metrics:

  • P-value: This metric helps determine the significance of your results. A p-value less than 0.05 typically indicates that the differences are statistically significant.
  • Confidence Interval: This range estimates where the true conversion rate lies with a certain level of confidence, usually 95%.
  • Lift: This is the percentage increase or decrease in the performance metric you are testing for, calculated from the baseline of the control group.

Making Informed Decisions

With the data interpreted and the statistical analysis complete, the final step is to decide how to act on the insights gained from your A/B test.

Guidelines on How to Act on Test Results:

  • Implement Winning Variations: If one variation significantly outperforms the control, consider implementing it across the site.
  • Further Testing: If results are inconclusive or the lift is minimal, running additional tests with adjusted variables or targeting a different user segment may be beneficial.
  • Scale or Pivot: Depending on the impact of the changes tested, decide whether to scale these changes up to affect more of your business or to pivot and try a different approach entirely.

Decision Trees or Flowcharts:

Create a decision tree or flowchart that outlines the decision-making process following an A/B test. This could include nodes that consider whether the test was statistically significant, whether the results align with business goals, and what follow-up actions (like further testing, full implementation, or abandonment of the change) should be taken based on different scenarios.

By thoroughly analyzing A/B test results through data interpretation, statistical analysis, and strategic decision-making, organisations can ensure that they are making informed decisions that will enhance their website’s user experience and improve overall business performance. This data-driven approach minimises risks associated with website changes and ensures that resources are invested in modifications that provide real value.

Beyond Basic A/B Testing

Once you have mastered basic A/B testing, you can explore more sophisticated techniques that offer deeper insights and potentially greater improvements in user experience and conversion rates. This section delves into advanced testing strategies and the importance of ongoing optimisation through iterative testing.

Advanced Testing Techniques

Advanced testing methods allow you to explore more complex hypotheses about user behaviour and website performance, often involving multiple variables or entire user journeys.

Multivariate Testing (MVT):

  • Overview: Unlike A/B testing, which tests one variable at a time, multivariate testing allows you to test multiple variables simultaneously to see which combination produces the best outcome.
  • Application: For example, you might test different versions of an image, headline, and button on a landing page all at once to determine the best combination of elements.
  • Benefits: This approach can significantly speed up the testing process and is particularly useful for optimising pages with multiple elements of interest.

Multipage Testing:

  • Overview: Also known as “funnel testing,” this technique involves testing variations across multiple pages that make up a user journey or funnel.
  • Application: You might test variations of both the product and checkout pages to see which combination leads to higher conversion rates.
  • Benefits: Multipage testing helps ensure consistency in messaging and user experience across multiple stages of the user journey, which can improve overall conversion rates.

Continuous Improvement and Iteration

The goal of A/B testing is not just to find a winning variation but to continually refine and enhance your website based on user feedback and behaviour.

Importance of Ongoing Optimisation:

  • Iterative Process: Optimisation is an ongoing process that involves continually testing and refining website elements based on user data and business objectives.
  • Learning from Each Test: Each test provides valuable insights into whether a variation wins. These insights can inform future tests, leading to better user experiences and higher conversion rates.

Iterative Testing Strategies:

  • Start with Broad Tests: Begin with broader tests to identify which elements have the most significant impact on user behaviour.
  • Refine and Repeat: Use the insights gained to refine your hypotheses and test more specific variations.
  • Expand Testing: Once you’ve optimised major elements, expand your testing to less prominent components that could still affect user experience and conversions.

Timelines and Case Studies:

  • Timeline Example: Show a timeline that outlines an annual testing strategy, with phases for broad testing, refinement, and expansion.
  • Case Study: Present a case study of a company that implemented continuous testing. Highlight how iterative testing helped them achieve a significant, sustained increase in conversion rates over time. For instance, a tech company could use iterative testing to fine-tune its sign-up process, resulting in a 50% increase in user registrations over a year.

By advancing beyond basic A/B testing and embracing more complex and continuous testing strategies, companies can optimise their websites more effectively and foster a culture of data-driven decision-making. This approach leads to improvements that align with user preferences and business goals, ensuring sustained growth and a competitive edge in the market.

Common Pitfalls and How to Avoid Them

A/B testing is a powerful tool for website optimisation, but common pitfalls can undermine its effectiveness. This section explores typical errors that occur during the testing process and provides strategies to ensure the validity and reliability of your tests.

List of Common Mistakes

Identifying Errors and Solutions:

  • Testing Too Many Changes at Once: It can make determining which change affected the outcome difficult.
    • Solution: Focus on testing one change at a time or use multivariate testing for simultaneous changes and analyze the impact of each element separately.
  • Not Allowing Enough Time for the Test to Run: Ending a test too soon can lead to conclusions that aren’t statistically significant.
    • Solution: Ensure each test runs long enough to collect adequate data, reaching statistical significance before making decisions.
  • Testing Without a Clear Hypothesis: Starting tests without a clear, data-backed hypothesis leads to unclear outcomes.
    • Solution: Develop a precise hypothesis for each test based on thorough data analysis and clear business objectives.
  • Ignoring User Segmentation: Different segments may react differently to the same change.
    • Solution: Segment your audience and analyze how different groups respond to each variation.

Visuals of Pitfalls vs. Best Practices:

  • Create side-by-side infographics showing examples of these mistakes versus best practices. For example, visually compare the outcome of a test that changed multiple elements simultaneously against one that tested a single change.

Ensuring Validity and Reliability

Maintaining the integrity of your A/B tests is crucial for obtaining reliable, actionable insights.

Tips on Maintaining Test Integrity:

  • Use Proper Randomisation: Ensure that the distribution of users between the control and test groups is random to avoid selection bias.
    • Tool Tip: Utilise tools that automatically handle randomisation to avoid manual errors.
  • Control External Factors: Holidays, marketing campaigns, or significant news events can skew test results.
    • Solution: Monitor external factors, adjust the testing period, or filter the data to account for anomalies.
  • Ensure Consistent Test Conditions: Changes in the testing environment or platform during the test can invalidate results.
    • Solution: Keep the testing conditions consistent throughout the test period and verify configuration settings regularly.
  • Validate Test Setup Before Going Live: A misconfigured test can lead to incorrect data interpretation.
    • Solution: Run a smaller pilot test or use a checklist to ensure every test element is correctly set up before full deployment.

Troubleshooting Guide with Graphic Aids:

  • Develop a troubleshooting guide that includes common scenarios where A/B test integrity might be compromised. Include flowcharts or decision trees that help identify and resolve issues such as data discrepancies, unexpected user behaviour, or sudden changes in conversion rates.
  • Example Graphic Aid: A flowchart that helps determine actions when test results seem inconsistent with historical data or benchmarks. Steps might include checking configuration settings, reviewing segmentation criteria, or extending the test duration.

By understanding and avoiding these common pitfalls and maintaining rigorous standards for validity and reliability, organisations can ensure that their A/B testing efforts lead to meaningful improvements and robust data-driven decisions. This approach not only enhances the effectiveness of current tests but also builds a foundation for future testing strategies that are even more successful.

A/B Testing Case Studies

A/B testing has proven to be a critical tool for businesses aiming to optimise their online presence based on data-driven decisions. Here, we delve into some specific real-life case studies from different industries, highlighting the successes and lessons from A/B testing.

Success Stories

E-commerce: Humana

  • Overview: Humana, a well-known health insurance company, conducted an A/B test to increase click-through rates on one of their primary campaign landing pages. They tested the simplicity and message of their banner and CTA.
  • Changes Tested: The original banner had a lot of information and a standard “Shop Medicare Plans” button. The test variation simplified the message and changed the button text to “Get Started Now.”
  • Results: The variation led to a 433% increase in click-through rates to the insurance plans page.

B2B: SAP

  • Overview: SAP, a leader in enterprise application software, tested the copy of their CTA on a product page. The hypothesis was that a more action-oriented CTA would increase engagement.
  • Changes Tested: The original CTA read “Learn more,” which was changed to “See it in action” in the variation.
  • Results: This simple change in wording resulted in a 32% increase in clicks.

.

Digital Media: The Guardian

  • Overview: The Guardian tested different wordings for their support and donation CTAs to determine which would more effectively encourage readers to contribute financially.
  • Results: The test revealed that a direct ask for contributions using emotive language resulted in a higher click-through rate than a more generic request for support.
  • Lesson: This A/B test highlighted the importance of emotional resonance in messaging, especially for non-profit or cause-based initiatives.

Travel Industry: Expedia

  • Overview: Expedia conducted A/B testing to optimise hotel booking conversions on their site by altering the display of discount offers.
  • Changes Tested: They tested the visibility and presentation of savings messages (e.g., showing a percentage off versus a specific dollar amount saved).
  • Results: Showing the amount of money saved led to a slight decrease in conversion rates, contrary to expectations.
  • Lesson: The test underscored the potential for “over-optimising” to backfire and the need to balance how offers are presented to avoid overwhelming customers.

Final Checklist of A/B Testing Steps

To help ensure your A/B testing journey is structured and effective, here is a visual checklist encapsulating the process:

  1. Define Objectives: Clearly state what you aim to achieve.
  2. Formulate Hypotheses: Base your assumptions on data and prior insights.
  3. Select the Testing Tool: Choose a platform that suits your scale and complexity needs.
  4. Design the Test: Create variations based precisely on your hypotheses.
  5. Run the Test: Ensure the test is long enough to gather meaningful data.
  6. Analyze Results: Use statistical analysis to interpret the outcomes.
  7. Implement Changes: Apply successful variations or further refine and test.
  8. Repeat: Use insights gained to continuously improve further testing.

Regardless of the outcome, every test is a step forward in understanding your users better and refining your digital offerings to meet their needs more effectively. The journey of optimisation is continuous, and each effort builds upon the last, opening new doors to innovation and growth.

Harness the power of A/B testing to start making informed decisions that propel your business forward. Your next breakthrough could be just one test away.