A successful product is one that not only looks great but also solves a real problem. Ticking off both boxes requires understanding your customers’ motivations, goals, and behaviours—and user research is the best method for gaining those insights.
User studies are conducted in person traditionally, but it’s possible to accomplish the same goals in an online environment.
Remote user studies can provide rich data. They’re ideal if you work with a global audience, need alternatives due to lockdown or distancing requirements, or face budget and time constraints.
This article will examine how to conduct user studies in an online environment. We’ll discuss the benefits of usability studies, compare lab studies vs. online studies, and share essential research methods and tools.
What is a User Study?
User research is the study of target customers’ needs and their behaviours in achieving them. The aim is to uncover insights that will assist in designing products that best meet user expectations.
There are two broad categories of user research:
● Quantitative: What’s happening; measurable, numerical results (ex: how many people clicked a button?).
● Qualitative: Why it’s happening; motivations behind the behaviour (ex: why didn’t some people click a button?).
Researchers usually start with qualitative research to discover customer needs and motivations and test their initial designs using quantitative measures.
There are also two basic approaches to user research:
● Attitudinal: Listening to users’ words (ex: interviews, surveys).
● Behavioural: Watching their actions (ex: card sorting, usability testing).
The best user research applies quantitative and qualitative research and attitudinal and behavioural approaches.
The Benefits of Usability Testing
Used correctly, user research should lead to a product, service, or website that better meets the specific needs of your customers.
When people feel like you “get them,” they’re likely to:
● Buy more quickly the first time.
● Spend more money.
● Make more repeat purchases.
● Remain customers for longer.
● Tell others about your product, service, or website.
Increasing customer acquisition, retention, loyalty, and referrals will positively impact your bottom line. User studies can also improve ROI by minimizing design and development costs and reducing support calls.
The more complex your product, service, or website, the greater the risk of skipping user research.
Unfortunately, many companies bypass user studies. They don’t want to invest time or money, or believe they already know what their customers want.
Basing design on unvalidated assumptions can lead to wasting time, resources, and money on a product, service, or website that flops—and you won’t know if you’ve missed the mark until after launch.
Get regular insights
Keep up to date with the latest insights from our research as well as all our company news in our free monthly newsletter.
How to Conduct User Studies
While user studies are most effective and efficient when performed early in the development process, it’s never too late to conduct this type of research.
To begin, set concrete goals. What are you trying to learn? What do you want to understand about your users? What problem are you trying to solve? Write a research brief that asks straightforward questions that lead to definitive, measurable answers.
Next, choose your research method or methods wisely (more on the most common methodologies next). Don’t simply pick what you know best. Take time to consider which methods are most applicable to your specific project.
After completing the user studies, share your findings. The results are only helpful if clearly communicated with key stakeholders, including product or website designers and developers, marketing managers, and C-suite leaders.
It’s vital that these colleagues understand, believe in, and know how to act on the data.
Finally, remember that user research should never end. It’s important to continue learning from your users. The marketplace and technology evolve, so must your product, service, or website.
Research Methods
The most common user study methods include interviews, focus groups, surveys, card sorting, A/B testing, and usability testing. Let’s look at the pros and cons of each research methodology.
● Interviews
Asking non-leading, open-ended questions with a user to learn about their attitudes, desires, and experiences.
Pros: Provides the most detailed information, can see non-verbal cues
Cons: Expensive and time consuming to conduct and analyse
● Focus Groups
Interviewing several people at the same time.
Pros: Increased reach, reduces time and expenses
Cons: Potential for moderator bias, loudest users can influence others, difficult to analyse
● Surveys
Gathering quantitative and qualitative information from users via (typically anonymous) questionnaires.
Pros: Easy, inexpensive, most expansive reach
Cons: Low response rates, poorly worded questions can skew results, limited ability for follow up
● Card Sorting
Asking users to categorize a set of terms to understand how they organize information.
Pros: Quick, simple, and inexpensive to perform
Cons: Provides limited information; analysis can take time
● A/B Testing
Showing two different concepts to an equal number of users to determine which better accomplished a specific goal.
Pros: Conclusive answers to specific questions
Cons: Limited use cases, limited data, time-consuming
● Usability Testing
Observing users perform predefined tasks.
Pros: Measures actual behaviours (not perceived preferences)
Cons: Expensive and difficult to run
Online Studies vs. Lab Studies
Many types of user research can be conducted remotely. In fact, online user research has key advantages over traditional testing.
● Reach: Access to an unlimited geographic area allows for a larger pool of participants.
● Environment: Users are in their space using their hardware and software, which creates a greater comfort level.
● Cost: Eliminating travel expenses and reducing logistical challenges makes remote studies quicker and less expensive to complete.
● Neutrality: Less potential for bias introduced by a lab setting and/or a moderator’s body language.
● Flexibility: Online research bypasses potential barriers, such as lack of transportation or a global pandemic.
Despite these myriad benefits, there are reasons that online user studies may not be feasible or preferred.
● If information security is paramount, it’s generally more challenging to maintain control over online proprietary information (and user data).
● If your user base is mainly rural or lower-income, you may find it challenging to find participants who have reliable high-speed internet connections.
● If your study depends heavily on tracking a user’s movements or interaction with a product, it may not be possible in a virtual setting.
Moderated vs. Unmoderated User Studies
Remote research is divided into two categories—moderated and unmoderated.
In moderated sessions, a facilitator speaks directly to participants and guides them through questions and/or tasks.
This type of qualitative research provides the most in-depth insights into precisely what users think and do and why. They are also more expensive and limited to the availability of a trained moderator.
Unmoderated studies are conducted online at the user’s convenience. Participants follow on-screen instructions and are encouraged to speak their thoughts aloud, which are recorded.
This type of research doesn’t allow for explanation or follow-up questions, which can create confusion and limit the quality of feedback. On the upside, you can run unlimited sessions at all times using a variety of online technology solutions at a lower cost per participant.
Recruiting for Online User Studies
Whether you choose moderated or unmoderated studies, it’s important to ensure that participants fit your user base or target audience. The right users make all the difference in the quality and usability of your results.
Generally, the most reliable recruitment will happen using your database of customers. If that doesn’t exist or doesn’t produce enough participants, however, you can also try:
● A recruitment agency or panel company can find specific participants but will be costly.
● User testing software companies that specialize in recruiting, which will generate a list of participants who applied for your project through a project board or email blast.
● Posting on social media outlets like Reddit, Craigslist, and Linkedin. In this case, it’s important to screen potential candidates because almost anyone could see and apply.
● Asking family and friends is a low-cost solution, but could be create issues of bias and mismatches with your actual user base.
How to Choose the Best Research Method
Some types of user research (interviews, usability studies) are easier to recreate online than others (focus groups, card sorting).
The ideal situation is to combine insights from multiple types of user research methods and testing rounds. However, it can be time and cost prohibitive to implement several methods in the real world.
Generally, in-person moderated studies are the best choice if your study requires hands-on participation and has a lot of room for confusion. To conduct these studies online, it’s imperative to consider all potential challenges and to thoroughly test the study before recruiting actual participants.
If you’re most concerned with asking open-ended questions that elicit a great depth of insights, a moderated study conducted online will be a great choice.
If your budget is tight or you’re most concerned with getting a larger quantity of responses, a remote unmoderated study is probably the best option.
Best Practices for Moderated Remote Studies
Moving from moderating in-person research to remote research can feel like a big shift, but by and large, a lot of the practices stay the same.
Here are some best practices for conducting remote moderated testing.
Study design
Shifting from in-person to remote moderated research requires thoughtful preparation. Start with the problem you want to solve or the hypothesis you want to test, and create questions or tasks that address it.
Qualitative methods can generate a lot of information. When every question or task has a clear purpose, you’ll waste less time, reduce participant fatigue, and avoid “analysis paralysis.”
Knowing exactly what you’ll ask also helps identify where you’ll need tools to support your online research (read more about virtual research tools below).
Also, decide who will observe sessions. Allowing developers, project managers, marketers, and others to witness (and even engage with) the research can increase the likelihood that the results will be understood and put to use.
Make sure observers know in advance of the research session what’s expected of them. Should they mute themselves? What types of questions, if any, can they ask and when? How should they communicate with you or each other during the session?
Session management
Before jumping into your research, create a welcoming atmosphere. Let participants know that you appreciate their time and value their input, and review confidentiality measures.
Disclose upfront whether there are observers and if they will also ask questions. Share the session length, the types of questions or tasks they can expect, the desired answer complexity.
Confirm that users have—and know how to use—anything necessary for the session. Let them know what to do in case of technical difficulties.
Ask permission to record video and/or audio, and both the moderator and user should turn off any potential distractions.
During the session, observers should take notes on any insights they have about what the participant says or does
End the session by thanking participants for their time and insights, and letting them know how they’ll receive any compensation.
Stop the recording when participants leave or leave it on if your team plans to stay and compare notes about the session.
Data analysis
After all of your remote research sessions, gather all observers’ notes and work together to distill the findings. Look for and discuss patterns and themes and how the team will apply the information to your product, service, or website.
Any tools you’ve used may also produce data, which may need to be aggregated before analysis and compilation with your team’s findings.
Best Practices for Remote Unmoderated Studies
Instead of having a human direct the study, unmoderated research relies on a software application to instruct users, ask questions, and record their actions or answers.
By and large, remote unmoderated studies share the same best practices as those above for moderated research. However, there are a few unique considerations to keep in mind.
Study design
Start with your problem or hypothesis and create questions or tasks that address it. Beware that clarity is paramount without a moderator offering explanations or answering questions.
Make instructions explicit, so users know exactly what to do. If you’re asking participants to record themselves, include clear triggers so they know when they should start and stop. Be specific with open-ended questions to avoid rambling responses unrelated to your goals.
Run a trial session to uncover any ambiguous instructions, questions, or potential problems with the study design or technology. Replicate actual testing situations by using real participants with their equipment.
Finally, a 10 to 15 per cent drop-off rate is typical in unmoderated studies, so plan to recruit more participants than you need.
Session management
Lower your drop-off rate by recording or writing a warm welcome that thanks participants for their time and insights.
Also, be clear up front with instructions and expectations. Let the user know exactly what they will be doing and how long the session will take.
Before they log off, include a final thank-you message and any information about compensation or follow-up they may receive.
Data analysis
Unmoderated studies can accumulate a lot of data and typically require extra manpower to analyse.
Quantitative data is straightforward. The testing tool will automatically collect and generate data visualizations for metrics like success rate, task time, and ratings.
For qualitative data, however, you’ll need to review interview questions and session recordings to take notes on user behaviour and identify positive and negative reactions.
Unmoderated testing tools with robust video analysis features can help by aggregating, exporting, sharing, and visualizing any notes you add to recordings and creating compilations of important moments.
Tools for Remote User Studies
Conducting user studies online necessarily requires software or apps. There are many options available (at a wide range of price points) for every type of research task.
● User Recruitment (Respondent, Ethnio)
● Video Conferencing/Recording (Zoom, Google Meet, Skype, GoTo Meeting)
● Note-Taking (ConfirmKit, Pear Note)
● Transcription (Rev, Reduct, Otter.ai, Trint)
● Surveys (Alchemer, Typeform, Survey Monkey, Survey Legend, YesInsights, Survicate)
● Usability Testing (Lookback, PingPong, UserTesting, Loop11, Userbrain, Userlytics, UsabilityHub, UserZoom, FocusVision, Qualtrics, InVision)
● A/B Testing (Optimizely, VWO)
Conclusion
User research is a crucial method for validating assumptions about your users’ needs and experiences. Done well, remote user testing can provide rich data that will help you understand how your target audience interacts with your product, service, or website.
Unfortunately, user studies are not “one size fits all.” There are many methods, each with distinct advantages and disadvantages. The choice depends on your goals for the type of data you want to gather.
Conducting user studies in the online environment isn’t always the best option. Still, it can be incredibly helpful if you’re working with a global audience, are under budget or time constraints, or face limitations due to the COVID pandemic.
Senior Marketing Executive
Sales & Marketing
General Manager PR -Internal Communications & Government Affairs
Vital Strategies
Customer Intelligence Director