Imagine launching a new drag-and-drop feature for your app, only to find that users barely notice or engage with it. This is a common scenario faced by UI/UX designers at Eleken agency, where testing several design options becomes essential to understand what truly drives user interaction and satisfaction.
To navigate this challenge, choosing the right experimental design can make all the difference. Two primary methodologies, between-subjects design and within-subjects design, offer distinct approaches to structuring user research. Each method comes with its own set of advantages, considerations, and use cases. The crucial question becomes: how do you decide which design best suits your research needs?
In this guide, we’ll break down the key aspects of between-subjects and within-subjects designs. We’ll explore their definitions, practical applications, pros and cons, and offer real-life scenarios to illustrate how each design can be effectively applied in UX research. By the end, you’ll be equipped with the knowledge to choose the optimal approach for insightful and impactful user testing.
What is a between-subjects design?
Let’s pick up where we left off—your app’s new drag-and-drop feature. Despite meticulous design and engineering, initial results show minimal user interaction. Determined to boost engagement, you craft an additional screen that subtly nudges users to try the feature. But will this new approach resonate with them? The only way to find out is through user testing. However, before diving in, it's essential to choose the right experimental setup.
Two key options stand out: between-subjects design and within-subjects design. Each one offers a unique lens through which to view user behavior, and the choice between them can significantly impact the insights gained.
Picture this: we take a group of testers and split them into two. The first group interacts with the app as it currently stands. The second group experiences the new design, crafted to encourage engagement. This setup is an example of a between groups design.
A between-subjects experimental design is a research method where participants are divided into separate groups, with each group experiencing only one version of the variable being tested.
By using this approach, each participant is exposed to only one version of the app. This ensures user feedback is based solely on their interaction with the version they tested, free from comparisons or biases from seeing both.
For our challenge with the new feature, if the second group, in statistically significant numbers, shows higher interaction with the feature compared to the first, we know the redesign is effective. If not, it’s back to the drawing board for further adjustments.
Now, let’s consider the other option: using a within-subjects design. In this approach, we have the same group of testers interact with both versions of the app. First, they use the app as it currently stands. Afterward, they try the redesigned version with the nudge prompting them to engage with the drag-and-drop feature.
A within-subjects experimental design is a research method where the same participants experience all versions of the variable being tested.
The main advantage here is that each participant provides feedback on both versions, allowing for direct comparisons within the same user base. This reduces variability due to individual differences, as the same users are effectively their own control group.
In the context of our client’s challenge, this approach could help reveal specific user preferences or subtle improvements. For instance, if participants report finding the new version more engaging after trying both, we can confidently attribute their feedback to their firsthand comparison. This insight might be harder to capture in a between-subjects design, where feedback comes from different groups.
However, we need to be mindful of potential learning effects. Once users are familiar with the original version, their experience with the new version might be influenced by what they already know. To address this, we can randomize the order of the versions or add breaks between tests to ensure more accurate results.
Practical examples of between-subjects design
Here are some examples that highlight the applications of a between subject experiment in UX research process:
1. Testing two onboarding flows
Imagine you're designing an onboarding process for a productivity app and want to determine which flow helps users understand core features more effectively. Using a between-subjects design, you could split users into two groups: one group experiences the original onboarding flow, while the other tries a new, streamlined version.
By comparing metrics like time-to-completion and task success rate between the two groups, you can identify which flow provides a better first-time experience without the influence of previous exposure.
2. Evaluating different pricing page designs
If you’re considering two different designs for a SaaS pricing page, a between-subjects design allows you to assess how each version impacts user decisions. One group of participants interacts with a version that highlights a freemium option, while the other group sees a version that emphasizes annual plans.
By comparing conversion rates and user engagement between the groups, you can gain insights into which layout better supports your business goals.
3. Fixing to-do list UI for task order clarity
Another real-life story from our clients: their users are struggling to understand the order of tasks in the to-do list feature, leading to confusion and missed priorities. To address this, you might decide to test two design variations: one version with numbered tasks and another using bold color coding to indicate priority levels.
Using a between-subjects design, split participants into two groups: one interacts with the numbered tasks version, while the other sees the color-coded priority version. By observing task completion rates and collecting feedback on ease of use, you can determine which UI change more effectively improves task clarity and helps users understand task order.
But in all those cases, a within-subjects study design can work just as well, offering its own set of advantages. So how do you know which method to choose for your specific project?
Between subjects vs within subjects design: How to decide?
When choosing between a between-subjects design and a within-subjects design, it's essential to consider the specific needs and constraints of your study. Some independent variables can dictate which study design is best.
For example, if you want to compare user behavior between distinct groups (e.g., younger vs. older users or newcomers vs. experienced users), a between-subjects design is the natural choice. A single participant can't be both a new and an experienced user at the same time, so separate groups are needed for each category.user at the same time, so separate groups are needed for each category.
However, if your variables don’t help you with the choice, consider the pros and cons of each method and make your choice based on your specific situation.
Advantages of between-subjects design
Using a between-subjects design for usability testing offers several advantages, making it a popular choice for evaluating distinct versions of a product. Here’s why it could be the right approach for your UX research:
1. No carryover or learning effects
One of the biggest strengths of a between-subjects design is that it eliminates carryover effects. Since participants only see one version of the app, there’s no risk of them becoming familiar with the interface or learning features that could influence their response to another version. This is crucial when testing elements that need a fresh perspective.
2. Simplified participant experience
Between-subjects design tends to be simpler for participants. They only need to evaluate one version, reducing cognitive load and potential confusion. This streamlined approach can lead to more straightforward and genuine feedback, as users aren’t tasked with comparing multiple versions. Additionally, it can be easier to manage for you, especially when using remote user testing software, as participants can complete tasks independently with minimal oversight.
Advantages of within-subjects design
A within-subjects design brings a different set of strengths to user testing, making it valuable for specific research goals. Here’s why this approach could be the best fit for your UX study:
1. Fewer participants needed
One of the primary benefits of a within-subjects design is its efficiency in terms of participant numbers. Since each participant tests all versions of the app, you need fewer participants to gather comprehensive data. This can be particularly useful when working with a limited testing pool or when participant recruitment is challenging. If you’re looking to recruit testers efficiently, check out our guide on how to find beta testers for tips and strategies.
2. Minimum of noisy data
A major advantage of within-subjects design is that it helps minimize noisy data. Since each participant tests all versions, their unique traits—like mood, background knowledge, or energy level—affect all conditions equally. This consistency makes it easier to detect genuine differences between versions, as any individual bias is evenly distributed.
Between-Subjects: Pros
Within-Subjects: Pros
- No carryover or learning effects
- Fewer participants needed
- Simplified participant experience
- Minimum of noisy data
Understanding mixed study designs
In some cases, a research project can benefit from using a mixed study design, which incorporates both within-subjects and between-subjects elements. This hybrid approach can provide more nuanced insights by combining the strengths of both methods.
How does a mixed study design work?
A mixed study design involves at least two independent variables—one analyzed within subjects and the other between subjects. For example, imagine you're running a user experience study for a car-rental app. You want to compare user interactions on two different sites (Site A and Site B) and examine how age influences performance.
- Within-subject variable: Each participant interacts with both Site A and Site B. This allows for direct comparison of user behavior across the two sites, ensuring that individual differences are minimized because the same users experience both conditions.
- Between-subject variable: Age is a between-subjects variable, with participants divided into groups (e.g., under 30 and over 30). A single participant can only belong to one age group, so comparisons between age groups must involve different sets of users.
By using a mixed design, researchers can explore how the different variables interact. For instance, does the performance difference between Site A and Site B vary by age group? This type of study offers a comprehensive view of how multiple factors influence user behavior.
How to use a between-subjects design
Implementing a between-subjects design involves these key steps:
1. Define your research question and hypothesis:
- Start by clearly outlining what you want to investigate. Formulate a hypothesis that states your expected outcome and identifies the variables you plan to study.
2. Identify independent and dependent variables:
- Determine the independent variable (the one you manipulate) and the dependent variable (the outcome you measure). For instance, if you’re testing two versions of an app interface, the independent variable is the version of the interface, and the dependent variable could be user engagement or task completion rate.
3. Determine levels of the independent variable:
- Establish the different conditions or levels of your independent variable (e.g., Version A and Version B of the app interface).
4. Set Sample size and selection criteria:
- Decide on the number of participants you need to achieve statistically meaningful results. Set criteria for participant selection to ensure they represent the user base of your product or service.
5. Randomize participant assignment:
- Randomly assign participants to different groups to reduce selection bias and ensure comparability. This step is crucial for making the groups representative and minimizing external factors that could skew the results.
6. Conduct the experiment:
- Administer the test by having each group interact with only one version of the independent variable. Make sure that the testing conditions are consistent for all groups to maintain the integrity of the data.
7. Collect and analyze data:
- Gather feedback and metrics from participants. Analyze the results to determine if there are statistically significant differences between the groups, which will help you test your hypothesis and draw conclusions.
For more guidance, you may want to refer to our UX research plan template.
Now that you know how to set up a between-subjects study, it’s time to think about the tools that can make your research smoother and more effective. At Eleken, we have a few favorites that help us get the job done, and we'd love to share them with you.
Tools for conducting user testing
At Eleken, we use versatile platforms to streamline our user testing processes. Here are some key tools to consider:
1. Lookback for moderated tests
At Eleken, Lookback is our go-to tool for moderated user tests. It supports both real-time observation and engagement, making it ideal for capturing qualitative feedback. Researchers can conduct interviews and record user interactions, providing context-rich insights that complement both between-subjects and within-subjects designs.
- Free trial: 60-day free trial
- Pricing: starts at $25 per month (billed annually)
2. Maze for unmoderated tests
Maze is Eleken's go-to tool for unmoderated tests. This platform lets you create and run usability tests and share them with participants through a simple link. Maze helps you collect useful insights quickly and generate detailed reports.
It connects easily with design tools like Figma, InVision, Marvel, and Sketch, so you can import prototypes directly. Besides unmoderated usability tests, Maze offers other research methods like card sorting, tree testing, 5-second testing, and A/B testing. For more insights into the topic, read our post on A/B testing for SaaS.
- Freemium: allows 1 study / month
- Pricing: $99 per month or $1,188 per year
Choosing the right study design for your UX research
Deciding between a between-subjects, within-subjects, or mixed study design depends on your research goals, the variables involved, and your available resources.
- Between-subjects designs are ideal when you need to test different groups, like comparing newcomers and experienced users, ensuring that results aren’t influenced by prior exposure.
- Within-subjects designs offer efficiency with fewer participants and more consistent data, making them great for direct comparisons within the same user group.
- Mixed designs combine elements of both and can provide deeper insights by examining how different factors interact.
Remember, the key to impactful UX research is choosing the design that best suits your project’s needs. And once your study is set up, leveraging the right tools, like Lookback for moderated tests and Maze for unmoderated tests, can streamline the process and maximize the quality of your findings.
.png)
