A/B test your video

Updated Oct 31, 2026 · 8 min read · Pro, Business and Agency plans

This article shows you how to show different versions of one video to different visitors, and find out which one gets more plays, clicks, viewers who stay, or sales.

Available on: Pro, Business and Agency plans. Funnels have their own A/B test step: see A/B test your offer.

How it works

You test one thing at a time, with 2 to 4 versions. Version A is what your video has now.

You can testCompared by
Thumbnail: a frame, the short loop, or an image you uploadPlay rate: how many visitors who saw it pressed play
How it starts: plays muted, or waits for a clickPlay rate
Button start time: when a button appearsClick rate: how many people who played clicked that button
Button textClick rate
Start volume: how loud the sound starts (10% to 100%)Viewers who stayed 1 minute after the sound came on
Pause and return: Smart Pause, Welcome back and the pause screen on (A) or off (B)Viewers who played and watched half of the video or more; the card also shows the average watch time of each version. See Pause the video when viewers leave and welcome them back

When your thank-you page counts sales (see Count sales on your thank-you page), the test is compared by sales once every version has at least 10 sales. Until then it is compared by plays or clicks as in the table, and the card shows each version's sales next to them. (A single early sale used to switch the whole test to sales and hide a clear play or click winner for weeks.)

  • Each visitor gets one version and keeps it on their device: coming back shows them the same version.
  • You choose how visitors are split, for example 50/50 or 70/30.
  • Your editor's preview always shows version A.

Start a test

  1. Go to Videos and open your video.
  2. Click the Behavior tab.
  3. Under A/B test, click Set up a test.
  4. In What to test, choose Thumbnail, How it starts, Button start time, Button text, Start volume or Pause and return.
  5. For a button, choose the Button.
  6. In Versions, choose A and B, A, B, C or A, B, C, D, and set each version: its thumbnail (A frame, Short loop or Upload an image), its time, its text, or its start volume (with Hear it to hear a sample at that volume).
  7. Choose the Split.
  8. Click Start the test.

Testing two images of your own? Version A is always the video's own thumbnail. Upload your first image in the Look tab under Thumbnail (Image), then set up the test and upload the second image as version B. The set-up shows both images, A's from the Look tab and B's under Version B: thumbnail, with Choose another image to replace B's.

Each version must differ from A. While the test runs, the A/B test section shows what is tested and since when, and its versions and split can't be changed: changing them would mix two versions under one letter. To test something else, click Stop and start a new test: the current test ends (its results stay in Analytics) and a new one opens with the same versions, ready to edit.

Test the start volume

A start volume test shows each version's start volume to its share of viewers: for example A at 70% (what the video has now, set under Sound in the Behavior tab) against B at 100%. Each version needs its own volume, different from A's.

Only viewers who turn the sound on on computers and Android are counted. iPhone and iPad can't change the volume, so their viewers would hear the same volume in every version: they are left out. Viewers whose own volume is remembered from an earlier visit (see Set the start volume and level the sound) are left out too, because they don't hear the version's volume.

The results card shows, for each version:

ColumnMeans
Viewers with sound onViewers who clicked play, or turned the sound on after a muted start
Stayed 1 minute after the sound came onOf those, how many watched at least a minute from then on (for a video under a minute: to its last second). The versions are compared by this until every version has 10 sales, then by sales
Left in the first 15 secondsOf those, how many watched less than 15 seconds
Turned the volume down / upThe share who turned it down (or muted it), and who turned it up. A version that starts at 100% shows "—" for up: its viewers can't turn it up
SalesSales counted for the version (as in every test)

Each version needs at least 100 viewers with the sound on before the versions are compared. Until then the card says how many more viewers with the sound on are needed, for example "Too early to tell: 78 more viewers with the sound on."

Read the result

Open the video's Analytics. The A/B test card shows, for each version, its share, the visitors, the plays, clicks or sales, and the rate. The figures cover the whole test, from its first day, and are updated with the rest of your Analytics (within a few minutes).

  • Too early to tell until each version has at least 100 visitors and 10 who played, clicked or bought.
  • Then the card shows which version leads and its chance to be the best, for example "B leads, 92% likely best. We decide once a week from day 14, so a lucky streak can't pick the winner. Next check: Nov 9.", and a sentence like "B gets clicks from about 50% more people than A."
  • B is the winner appears only at a weekly check, from day 14 of the test, when B is more than 99% likely to be the best version: "Checked on Nov 9: B is more than 99% likely to be the best version."

Why we wait for the weekly check

A running test's numbers move every day, and early leads are often luck. If a card named a winner whenever a version looked 95% sure, a test of two identical versions would name one most of the time, simply because the card is looked at again and again. So PlayerMagick checks once a week, from day 14 (days 14, 21, 28 and so on, counted from the day the test started), and names a winner only when it is more than 99% sure. Between checks you see the leader and its chance, never a winner.

  • About the same: from day 28, when the versions are so close that keeping either costs almost nothing, the card says so. Keep the one you prefer.
  • No clear winner: after 12 weeks without a winner the card stops checking. The versions are too close to call with this much traffic: keep the one you prefer.
  • A check counts the whole days before it, so the figures it used don't change as the day goes on.
  • When sales take over from plays or clicks (10 sales per version), a winner found on plays or clicks no longer stands: the next checks look at the sales.
  • Tests that ended before this rule keep the result they had.

Keep a version for everyone

When a weekly check names a winner, the A/B test card shows Keep B for everyone. Click it and confirm. Its thumbnail, start, time, text, start volume or pause and return setting becomes the video's own setting, and the test ends. The results stay in Analytics.

When the versions do about the same, or after 12 weeks without a winner, the card shows Keep A for everyone, Keep B for everyone and so on: keep the one you prefer.

Before there is a winner you can still decide for yourself: click Keep a version anyway, choose the version, read the warning and click Keep B anyway. Early leads are often luck, so do this only when you have another reason to choose.

To end a test and keep version A, click Stop the test in the Behavior tab.

Good to know

  • Start a new test after the last one ended: Start a new test. Each test has its own results.
  • Changing the video's own setting that is being tested (for example its thumbnail in the Look tab) changes version A from then on.
  • Stopping the test and starting a new one is the way to change a version: a running test keeps its versions and split.
  • A sale is counted for a version when the visitor clicked the video's button after playing for at least about 15 seconds (see Show a call-to-action button during the video) and watched at least 10 seconds of the video.
  • Sales are reported by the visitor's browser, as for funnels: someone determined could send fake plays and sales for one version. PlayerMagick limits them (10 new sales an hour per internet connection, a real play first) and waits for 10 sales per version before it compares by sales, so one fake sale can't decide a test. For a big decision, check the result against your checkout's own sales report.

Troubleshooting

  • The test shows no visitors. Statistics may be paused for your account (Analytics then says "Stats are paused"); the test counts again from the date shown.
  • I always see the same version. Each device keeps its version. Use another browser or a private window to see another one.

Was this article helpful?