|Magnolia A/B/n Testing is currently available as a private Beta only. If you would like to participate in our Beta Program, please fill in this form.|
Magnolia offers native A/B/n testing to help you make decisions about your content based on real data from your existing traffic. Here, we explain what A/B/n Testing is, how results are calculated with Magnolia’s A/B/n Testing app, and a brief overview of what to expect using the app.
The goal of testing is to determine which version of a page performs better among customers to increase your online conversion rates.
A/B testing is a testing strategy in which two versions of a page are tested against each other.
A/B/n testing is an extension of A/B testing in which multiple variants of a page are compared against each other.
Before designing a test, it’s considered good practice to identify a problem you want to solve and define a hypothesis. Existing problems are generally identified using analytics for your site.
Problem: Form completion for a free trial is very low despite high traffic on the page.
Hypothesis: By reducing the number of form fields from 15 to 5, we will increase the number of forms completed.
A/B Test: Variant A uses the original 15 field form, variant B uses a new, shorter form.
The resulting conversion rates show that variant B performs better. This page can be selected as the winner.
Your A/B/n test results are calculated by taking the user events from within your test (e.g., click rates) on a daily basis. These events are registered to a database and cleaned in a data pipeline to make the resulting content more usable for displaying results in the dashboard.
The test results displayed in the A/B/n Testing app rely on the Magnolia Analytics dashboard.
While collecting data, we do not store user-specific records, but rather test totals. When using the A/B/n Testing app, it is not possible to identify individual users/site visitors from the test results.
Magnolia provides a marketer-friendly app to enable you to quickly create and manage your tests. The Testing app gives you a single location where you can create and see all your tests.
The Testing app allows you to:
Create tests using pages from the Pages app.
Add a goal for your tests.
Allocate an audience segment such as North America or EMEA. You do not have to narrow the audience if the test is meant for all users.
Currently, site traffic is evenly distributed.
Edit, abort, and delete tests.
View test results.
Replace the original page with a winning variant.
A conversion happens when a site visitor completes a specific action. Some examples of conversions are a user clicking a button or visiting a specific page. This specified conversion acts as the basis to compare variants via the targeted uplift.
The target improvement (%) relative to the base conversion. Uplift involves testing sites to check for increased conversions and an improved user experience.
The likelihood that the uplift reported by the test will be correct. A confidence level of 90% means that there is a probability of 90% that the A/B/n test results are reliable and represent the true population parameter.
The true confidence rate percentage of the A/B/n test. This is displayed in the Results tab.
The range of values in relation to your chosen confidence level and conversion rate that predict the true mean of the parameter is within the range. For example, when we use the phrase "plus or minus a few X", the range of that plus or minor margin of error is the confidence interval.
Variants are different versions of the original page (or base variant). They are used to determine which structure, design, or visual might be more beneficial to site visitors. The control (base) variant is typically compared against slightly different variants of itself. With A/B/n Testing at Magnolia, you can have as many variations of the original page as required for your test.
A segment is a portion of site visitors who meet specified criteria. You should create a segment when you know your audience well and you want to routinely target content to them. For A/B/n Testing with Magnolia, this is typically a demographic region such as North America or Europe, Middle East, and Africa. Segments are made using Magnolia Personalization.
Results are considered significant when at least one of your variants has met the confidence levels based on the base conversion and uplift. This means that, based on preconfigured confidence levels, the variant has met the uplift target based on the initial base conversion rate.