---
title: "A/B Testing: How to Run Tests That Give Real Answers (With Examples)"
url: https://lucidbrands.agency/resources/website-design-and-cro/ab-testing
description: "What A/B testing is, how it works, a step by step process, how much traffic and time you need, statistical significance in plain English, test ideas, common mistakes, tools and what it costs."
lang: en
---

Image: Lucid Brands Home (https://lucidbrands.agency/img/logo.svg)

©Lucid Brands 2026

Close article: https://lucidbrands.agency/resources/website-design-and-cro

Image: Website Design & CRO resource article: A/B Testing: How to Run Tests That Mean Something (https://lucidbrands.agency/media/resources/art-ab-testing.jpg)

# A/B Testing: How to Run Tests That Mean Something

A/B testing (also called split testing) compares two versions of a web page, email or ad by showing each version to a random half of your audience and measuring which one gets more conversions. Version A is the current version (the control) and version B changes one thing (the variant). A test only means something if it runs long enough, with enough visitors, to reach statistical significance, usually 95% confidence.

## What is A/B testing?

Instead of guessing which headline, button or offer works better, you let real visitors decide. The method comes from randomized experiments in science: because visitors are assigned at random, the only systematic difference between the two groups is the change you made.

Large companies test constantly. Salesforce reports that Google ran 17,523 live traffic experiments in 2019, which resulted in 3,620 launches. Streaming services, retailers and software companies are known for testing everything from artwork to pricing pages. You do not need that scale, but the principle is the same.

## How A/B testing works

1. You create a variant of a page or email with one change.
2. A testing tool splits traffic randomly: half see A, half see B.
3. The tool records conversions for each version.
4. After enough visitors, a statistical test shows whether the difference is likely real or just chance.
5. If B wins with confidence, you roll it out. If not, you keep A and learn from the result.

## A/B vs multivariate vs split URL testing

| Test type | What it compares | Traffic needed | Best for |
| --- | --- | --- | --- |
| A/B test | Two versions with one change | Lowest | Most businesses; clear answers |
| A/B/n test | Three or more versions of one element | Higher | Testing several headlines at once |
| Multivariate test | Several elements changed in combinations | Highest | Very high traffic sites |
| Split URL test | Two different pages on separate URLs | Similar to A/B | Full redesigns or different page layouts |

## Do you have enough traffic to test?

This is the question most guides skip. Small differences need a lot of visitors to detect.

## A worked example

A generic landing page converts at 3%. You want to know if a new headline can lift that to 3.6% (a 20% relative increase). At 95% confidence and the usual 80% statistical power, a standard sample size calculator puts the requirement in the range of roughly 14,000 visitors per version, so about 28,000 in total.

- With 1,000 visitors a week to that page, the test would take more than six months. Not practical.
- With 10,000 visitors a week, it takes about three weeks. Practical.
- If you only care about big wins (say 3% to 4.5%), you need far fewer visitors.

Always run your own numbers in a free sample size calculator before starting. The inputs are your current conversion rate, the smallest lift worth detecting, confidence level and power.

## If your traffic is low

- Test bigger changes (a new offer or page layout, not a button color).
- Test higher traffic places, like the homepage or a paid ad landing page.
- Test in email, where you control the audience size.
- Use qualitative research: session recordings, heatmaps, customer interviews and user tests.
- Make improvements based on best practices and measure before and after, accepting less certainty.

## How to run an A/B test in eight steps

1. **Research the problem.** Use analytics, heatmaps, recordings and customer feedback to find where people drop off. See conversion rate optimization (https://lucidbrands.agency/resources/website-design-and-cro/conversion-rate-optimization).
2. **Write a hypothesis.** "Because [evidence], we believe changing [element] for [audience] will increase [metric]."
3. **Pick one primary metric.** Usually form submissions, purchases or calls. Make sure conversion tracking (https://lucidbrands.agency/resources/analytics-and-reporting/conversion-tracking) works first.
4. **Calculate sample size and duration** before launch. Plan for at least one full week, ideally two, to cover weekday and weekend behavior.
5. **Build the variant** and check it on desktop and mobile.
6. **Run quality checks:** both versions load quickly, tracking fires and traffic splits evenly.
7. **Let it run** to the planned sample size without stopping early.
8. **Analyze and document** the result, including what you learned if it lost.

## Hypothesis template

| Field | Example |
| --- | --- |
| Evidence | Recordings show visitors scroll past the form without noticing it |
| Change | Move the quote form above the fold and shorten it from 8 fields to 4 |
| Audience | Mobile visitors from paid search |
| Primary metric | Quote form submissions |
| Expected result | More submissions with similar lead quality |
| Sample size and duration | Planned before launch |

## Statistical significance in plain English

Statistical significance tells you how unlikely your result would be if there were actually no difference between A and B.

- Statsig explains the common rule: a p value below 0.05, which corresponds to 95% confidence.
- 95% confidence means that if there were truly no difference, you would see a result this extreme less than 5% of the time.
- It does not mean there is a 95% chance B is better, and it does not tell you how big the lift will be in the long run.

Look at the confidence interval too. "B increased conversions by 10%, with a likely range of 2% to 18%" is more useful than a single number.

## A/B test ideas

| Area | Ideas to test |
| --- | --- |
| Headlines | Benefit vs feature, specific number vs general claim, question vs statement |
| Calls to action | Button text ("Get my quote" vs "Submit"), placement, a second CTA lower on the page. See call to action examples (https://lucidbrands.agency/resources/website-design-and-cro/call-to-action-examples). |
| Forms | Fewer fields, multi step vs single step, phone field optional |
| Offers | Free estimate vs discount, guarantee wording, urgency |
| Social proof | Reviews near the CTA, star ratings, logos, case study snippets |
| Pricing pages | Showing prices vs "contact us," plan order, annual vs monthly default |
| Images | Real team photos vs stock, product in use vs on white |
| Page layout | Long vs short page, video at the top vs lower |
| Email | Subject lines, sender name, send time, one CTA vs several. See email marketing best practices (https://lucidbrands.agency/resources/email-marketing/email-marketing-best-practices). |
| Ads | Headlines, descriptions and images using built in ad platform experiments |

Start with the pages that matter most, such as your main landing page (https://lucidbrands.agency/resources/website-design-and-cro/landing-page).

## Testing emails and ads

Email and ads are often the easiest places to start, because the tools are built in and you control the audience.

- **Email:** most email platforms let you send two subject lines to a small share of your list, then send the winner to the rest. Test one thing at a time: subject line, sender name, offer or main call to action. Judge on clicks or conversions, not only opens, since opens are less reliable than they used to be.
- **Google Ads and Microsoft Ads:** use the built in experiments feature to split traffic between campaign versions, such as a new bidding strategy or landing page.
- **Meta Ads:** use the A/B test feature in Ads Manager rather than running two ads in the same ad set, where the platform will favor one early.

The same rules apply: one change, a clear metric, enough volume and patience.

## Common A/B testing mistakes

| Mistake | Why it is a problem | Fix |
| --- | --- | --- |
| Stopping early ("peeking") | Early results swing wildly; stopping at the first good day inflates false wins | Set sample size in advance and wait |
| Testing too many variants | Each extra comparison raises the chance of a false winner | Limit variants or adjust the analysis |
| Sample ratio mismatch | If the split is not close to 50/50, something is broken (tracking, redirects, bots) | Check traffic split daily; investigate gaps |
| Changing the test mid run | Mixes two experiments into one | Start a new test instead |
| Running tests during unusual periods | Holidays and promotions skew behavior | Avoid or account for them |
| Measuring the wrong metric | More clicks can mean fewer sales | Use the business outcome as the primary metric |
| Ignoring speed | A slow variant loses for reasons unrelated to the idea. See website speed (https://lucidbrands.agency/resources/website-design-and-cro/website-speed). | Check load time for both versions |
| Not documenting losers | You repeat failed ideas | Keep a test log |

## What are the disadvantages of A/B testing?

It needs enough traffic, takes time, only tests one idea at a time and can lead teams to chase small wins instead of fixing big problems. It also tells you what happened, not always why. Pair it with qualitative research.

## A/B testing tools

The ranking pages mention tools such as VWO, Optimizely, Crazy Egg, Omniconvert, Kameleoon and Statsig, plus built in testing in email platforms, ad platforms and some website builders. When choosing:

- Does it work with your website platform?
- Does it slow the page or cause flicker?
- Does it connect to Google Analytics 4 (https://lucidbrands.agency/resources/analytics-and-reporting/google-analytics-4-guide) and your conversion events?
- Does it report confidence intervals and warn about sample ratio problems?
- Is pricing based on traffic, and does it fit your volume?

## What A/B testing costs

Costs come from three places:

| Cost | Notes |
| --- | --- |
| Tool | Ranges from free built in features (email and ad platforms) to paid testing platforms priced by traffic. Check current pricing. |
| People | Research, design, development and analysis time for each test |
| Opportunity | Visitors who see a losing variant during the test |

Is A/B testing worth it? For pages with enough traffic and a clear conversion goal, yes. A single winning test on a high traffic page can pay for months of work. For low traffic sites, research led improvements usually pay off faster.

## FAQ

## What is A/B testing in performance marketing?

It means testing ad creative, landing pages, audiences or emails against each other with randomly split traffic to find which version produces more conversions at a lower cost.

## Does Netflix use A/B testing?

Netflix is widely known for testing many parts of its service, including the artwork shown for titles. It is a common example of large scale experimentation.

## How much does A/B testing cost?

Email and ad platforms often include testing for free. Dedicated website testing tools are usually priced by traffic. The bigger cost is the time to research, build and analyze tests.

## How long should an A/B test run?

Until it reaches the sample size you calculated before launch, and at least one full week, ideally two, so it covers different days of the week.

## Run tests that actually improve results

Lucid Brands' Design and Development team builds landing pages and variants, sets up tracking and helps you test changes that matter. If your ads send the traffic, Media Management starts at $500 a month. Start with a free website audit (https://lucidbrands.agency/website-audit) or contact us (https://lucidbrands.agency/contact).

Your path to lucidity

## Get Started With a Free Website Audit

Not sure where to begin? Start with a free website audit. Our team analyzes your website's SEO, speed, content and user experience to identify what to improve first.

Explore More

## Related Resources

### Website Copy: How to Write Pages That Convert

How to write website copy that turns visitors into leads: a page by page guide for home, about, service and contact pages, headline formulas, before and after examples, fill in templates and a review checklist.

Read More
https://lucidbrands.agency/resources/website-design-and-cro/website-copy

### ADA Compliance for Websites

Does your website need to be ADA compliant? What the ADA and DOJ say, why WCAG 2.1 and 2.2 AA are the working standard, a practical checklist, how to test in Chrome, why overlays fall short and what affects the cost.

Read More
https://lucidbrands.agency/resources/website-design-and-cro/ada-compliance-website

### Call to Action: Meaning, Tips and 60 Examples

What a call to action is, how to write one that gets clicked, button design and placement tips, plus 60 CTA examples for websites, ads, email, social and more.

Read More
https://lucidbrands.agency/resources/website-design-and-cro/call-to-action-examples

All Website Design & CRO Resources (https://lucidbrands.agency/resources/website-design-and-cro)

## Privacy Choices

We measure visits and personalize ads as you browse. Your choice below decides whether that keeps happening, and you can change it any time. Our website provider counts visits without cookies.

Privacy policy (https://lucidbrands.agency/privacy-policy)

## Structured data

```json
[
  {
    "@context": "https://schema.org",
    "@type": "BlogPosting",
    "@id": "https://lucidbrands.agency/resources/website-design-and-cro/ab-testing#article",
    "url": "https://lucidbrands.agency/resources/website-design-and-cro/ab-testing",
    "headline": "A/B Testing: How to Run Tests That Mean Something",
    "description": "What A/B testing is, how it works, a step by step process, how much traffic and time you need, statistical significance in plain English, test ideas, common mistakes, tools and what it costs.",
    "keywords": "ab testing",
    "datePublished": "2026-11-14",
    "dateModified": "2026-11-14",
    "mainEntityOfPage": "https://lucidbrands.agency/resources/website-design-and-cro/ab-testing",
    "articleSection": "Website Design & CRO",
    "author": {
      "@type": "Organization",
      "name": "Lucid Brands Team",
      "url": "https://lucidbrands.agency/about"
    },
    "publisher": {
      "@id": "https://lucidbrands.agency/#organization"
    },
    "isPartOf": {
      "@id": "https://lucidbrands.agency/resources/website-design-and-cro#webpage"
    }
  },
  {
    "@context": "https://schema.org",
    "@type": "FAQPage",
    "mainEntity": [
      {
        "@type": "Question",
        "name": "What is A/B testing in performance marketing?",
        "acceptedAnswer": {
          "@type": "Answer",
          "text": "It means testing ad creative, landing pages, audiences or emails against each other with randomly split traffic to find which version produces more conversions at a lower cost."
        }
      },
      {
        "@type": "Question",
        "name": "Does Netflix use A/B testing?",
        "acceptedAnswer": {
          "@type": "Answer",
          "text": "Netflix is widely known for testing many parts of its service, including the artwork shown for titles. It is a common example of large scale experimentation."
        }
      },
      {
        "@type": "Question",
        "name": "How much does A/B testing cost?",
        "acceptedAnswer": {
          "@type": "Answer",
          "text": "Email and ad platforms often include testing for free. Dedicated website testing tools are usually priced by traffic. The bigger cost is the time to research, build and analyze tests."
        }
      },
      {
        "@type": "Question",
        "name": "How long should an A/B test run?",
        "acceptedAnswer": {
          "@type": "Answer",
          "text": "Until it reaches the sample size you calculated before launch, and at least one full week, ideally two, so it covers different days of the week."
        }
      }
    ]
  },
  {
    "@context": "https://schema.org",
    "@type": "BreadcrumbList",
    "itemListElement": [
      {
        "@type": "ListItem",
        "position": 1,
        "name": "Home",
        "item": "https://lucidbrands.agency/"
      },
      {
        "@type": "ListItem",
        "position": 2,
        "name": "Resources",
        "item": "https://lucidbrands.agency/resources"
      },
      {
        "@type": "ListItem",
        "position": 3,
        "name": "Website Design & CRO",
        "item": "https://lucidbrands.agency/resources/website-design-and-cro"
      },
      {
        "@type": "ListItem",
        "position": 4,
        "name": "A/B Testing: How to Run Tests That Mean Something",
        "item": "https://lucidbrands.agency/resources/website-design-and-cro/ab-testing"
      }
    ]
  }
]
```