A/B/n Testing

Magnolia A/B/n Testing is currently available as a Beta only.

Magnolia offers native A/B/n Testing to help you make decisions about your content based on real data from your existing traffic. Here, we explain what A/B/n Testing is, how results are calculated with Magnolia’s A/B/n Testing app, and a brief overview of what to expect using the app.

For details on how to use the A/B/n Testing app, see the A/B/n Testing app page.

What is A/B/n Testing?

A/B testing is a testing strategy in which two versions of a page are tested against each other.

The goal of testing is to determine which version of a page performs better among customers to increase your online conversion rates.

image

A/B/n testing is an extension of A/B testing in which multiple variants of a page are compared against each other.

image

Start with a problem

Before designing a test, it’s good practice to identify a problem you want to solve and define a hypothesis. Existing problems are generally identified using analytics for your site.

image

Example:

  • Problem: Form completion for a free trial is very low despite high traffic on the page.

  • Hypothesis: By reducing the number of form fields from 15 to 5, we will increase the number of forms completed.

  • A/B Test: Variant A uses the original 15 field form, variant B uses a new, shorter form.

  • The resulting conversion rates show that variant B performs better. This page can be selected as the winner.

How results are calculated

The Magnolia algorithm is built on the Frequentist approach, which optimizes traffic volumes -even when traffic is low- as well as conversion rates and may reduce the number of observations required for a successful experiment by 50% or more. Predictions are made using data from the current experiment only.

Your A/B/n test results are calculated by taking the user events from within your test (such as click rates) on a daily basis. These events are registered to a database and processed in a data pipeline to make the resulting content more usable for displaying results in the dashboard. Note that results can take up to 24 hours to become available.

Magnolia Analytics is used to display and visualize the results of A/B/n Testing results.

GDPR Compliance

While collecting data, we do not store user-specific records, but rather test totals. When using the A/B/n Testing app, it is not possible to link back to any personally identifiable information from the test results.

Dedicated Testing app in Magnolia

Magnolia provides a marketer-friendly app to enable you to quickly create and manage your tests. The Testing app gives you a single location where you can create and see all your tests.

abn testing app

The Testing app allows you to:

  • Create tests using pages from the Pages app.

  • Add a goal for your tests.

  • Allocate an audience segment such as North America or EMEA. You do not have to narrow the audience if the test is meant for all users.

  • Currently, site traffic is evenly distributed across all variants.

  • Edit, abort, and delete tests.

  • View test results.

  • Replace the original page with a winning variant.

To create a test, follow the instructions on the A/B/n Testing app page.

Useful terms

Base conversion

A conversion happens when a site visitor completes a specific action. Some examples of conversions are a user clicking a button or visiting a specific page. This specified conversion acts as the basis to compare variants via the targeted uplift.

Uplift

The target improvement (%) relative to the base conversion. Uplift involves testing sites to check for increased conversions and an improved user experience.

Confidence level

The likelihood that the uplift reported by the test will be correct. A confidence level of 90% means that there is a probability of 90% that the A/B/n test results are reliable and represent the true population parameter.

Confidence rate

The true confidence rate percentage of the A/B/n test. This is displayed in the Results tab.

Confidence interval

The range of values in relation to your chosen confidence level and conversion rate that predict the true mean of the parameter is within the range. For example, when we use the phrase "plus or minus a few X", the range of that plus or minor margin of error is the confidence interval.

Base variant

The original page selected in the test. This becomes Variant A.

Variants

In A/B/n testing, variants are different versions of the original page (or base variant). They are used to determine which structure, design, or visual might be more beneficial to site visitors. The control (base) variant is typically compared against slightly different variants of itself. With A/B/n Testing at Magnolia, you can have as many variations of the original page as required for your test.

The term variant is also used in the context of Personalization, where a variant is an alternative content element that replaces the original element in personalized content delivery.

Segments

A segment is a portion of site visitors who meet specified criteria. You should create a segment when you know your audience well and you want to routinely target content to them. For A/B/n Testing with Magnolia, this is typically a demographic region such as North America or Europe, Middle East, and Africa. Segments are made using Magnolia Personalization.

Significant results

Results are considered significant when at least one of your variants has met the confidence levels based on the base conversion and uplift. This means that, based on preconfigured confidence levels, the variant has met the uplift target based on the initial base conversion rate.

dev days event sign up
dev days event sign up
Feedback