A holdout test that can’t fool you.

Every server-side tracking tool says it makes your ads cheaper. Some publish uplift figures averaged across their customers. None of that tells you whether it works on your account.

A holdout test can. Run half your spend with server-side conversions and half without, and compare what each half actually bought. It sounds simple. There are about five ways to get it wrong, and most of them make the tool look better than it is.

The one rule: score it on your numbers

Never judge the test on the ad platform's own reporting. The side you feed server-side conversions to will always report more conversions, because you've sent it more. That's the feed working, not the ads working.

Score both sides on the same thing, counted the same way, outside the platform: real cost per real outcome, from your payments, your database or your CRM.

Keep the two sides properly apart

This is where most tests quietly fail. A server feed doesn't teach one campaign. It teaches the pixel, or the dataset, behind every campaign that uses it. Run both sides on one pixel and the control side learns from the test side's data, and the test means nothing.

• Side A gets the pixel plus the server feed, on its own dataset.
• Side B gets the pixel only, on a separate dataset. It's the control: exactly what you run today.
• Split by region, with regions that don't overlap, so one person can't be in both.
• Same budget share, same creative, same offer on both sides, for the whole test.

Regions are never identical. So record both sides for a couple of weeks before you switch the feed on. What you're testing is whether the ratio between them moves, not whether one region beats another.

Lock the card before day one

Write four things down, date them, and don't change them once the test starts.

1. The metric. Real cost per real outcome, scored on your backend. One metric, not a menu you choose from afterwards.
2. The length. At least four weeks. The first two don't count, while the platform is still learning from the new data.
3. The threshold. What counts as a win, in numbers, before you've seen any.
4. The timestamp. Locked and dated, and shared with whoever has to believe the result.

The card is what stops everyone, including you, from moving the goalposts after seeing the results.

How many conversions you need

With few conversions, chance swamps everything. A test can only prove a lift bigger than the noise. The table shows roughly the smallest lift you can prove, for a given number of conversions on each side.

The maths, for anyone who wants to check it: treat each side's conversions as a count, compare them as a ratio, and ask for 80% power at 5% significance. The smallest detectable lift is then about e^(2.8 × √(2 ÷ n)) − 1, where n is the conversions on each side. It assumes both sides get equal spend and the conversions are independent. Real accounts are messier, so treat the table as a floor.

Two things follow. A small account can only prove a big lift, so run longer. Or measure an earlier step with more volume, such as leads or add-to-baskets, alongside the sale.

Five ways to fool yourself

1. Scoring on platform numbers. The fed side always looks better there. Use your backend.
2. One shared pixel. The control learns from the test. Separate datasets, separate regions.
3. Peeking and stopping early. Check the test daily and stop the first time it looks good, and you'll "find" wins that are noise. Run the full length on the card.
4. Changing things mid-test. A new creative, a budget shift or a sale on one side breaks the comparison. If you must change something, change it on both sides on the same day.
5. Counting the learning phase. The first weeks measure the platform adjusting, not the steady state.

Publish it either way

A proof page that only shows wins isn't proof. If you're an agency running this for a client, agree up front that the result gets reported whatever it says. A null result is still worth having: it tells you where not to spend.