Designing a fair test

A Scowtt pilot is only as convincing as the test behind it. This page covers what makes an account a good fit for a test, what to expect over time, and the practices that keep the result down to Scowtt's signal and nothing else.

How Scowtt tests

• A 50/50 test is the default. A test campaign optimizing on Scowtt's signal runs alongside your existing campaign, ideally as an in-platform experiment, at every scale where that's possible.
• A geo test is an option for larger accounts, using matched regions when splitting a campaign isn't practical.
• Every test has a control. Scowtt doesn't run before-and-after comparisons: results compared across different periods are harder to trust and take longer to read. If an account is too small to split, a test isn't the right next step yet.

See How a Scowtt pilot is structured and the setup guides for Google Ads and Meta.

Is your account a good fit?

The figures below are for the Scowtt test arm, after the budget is split. Accounts that meet the recommended level see results sooner and larger. Accounts near the minimum still benefit, but expect a longer learning period and a smaller lift.

What counts as an activity signal

The kind of data in your CRM matters as much as how much there is. Activity signals carry the most predictive value:

More types of activity make a richer model, but also need more volume. An account with around 50 outcomes a month works best with its 3 to 5 most predictive activities; much larger accounts can support 10 to 20. See What data Scowtt needs.

How long from lead to outcome

Scowtt needs time to score a lead, send its value to the platform, and influence bidding before the outcome happens.

How campaigns are optimized

• Each campaign, or each portfolio bid strategy, is optimized on its own. With a portfolio bid strategy, campaigns in the portfolio are combined, so it's their total volume that counts.
• One model per platform. Optimizing across Google and Meta together in a single model isn't available yet.
• Location-based campaigns can be combined. If your lead volume is split across separate campaigns per location, Scowtt can set up one test campaign covering all locations.

When a test may not be the right fit yet

• The account is too small to split into a 50/50 test with a control.
• Lead volume is too close to outcome volume (for example, 50 leads and 50 purchases), leaving no room for the model to tell leads apart.
• Leads convert in under 3 days, and there's no earlier step to optimize toward.
• The CRM holds no activity data, only acquisition details such as device, geography or keyword.
• The business is pure e-commerce with direct attribution. Scowtt adds the most where there's a lead or nurture stage between the click and the sale.

If any of these apply, your Scowtt team will talk through what would need to change first.

What to expect over time

Performance in the first few weeks shouldn't be used to judge the test: the model and the ad platforms need time to ramp up.

See Reading early results before revenue matures.

Choosing campaigns

• Exclude brand and remarketing campaigns. They don't add to new lead volume, and brand traffic converts anyway, which hides lift.
• Enough volume. Use campaigns that meet the volumes above, so the test reaches a confident answer in a reasonable time.
• Representative of your business. Choose campaigns that reflect your core business, so results carry over when you expand.
• No big changes planned. Avoid campaigns with a rebrand, landing page overhaul or large promotion planned during the test.

Keeping the arms equal

• Match everything except the signal. Budgets, targeting, creative, landing pages and geography should be the same in both arms.
• Use comparable bid strategies. Keep both arms on similar strategies (for example, both value-based), so the test compares the signal, not the strategy.
• Agree changes together. Neither arm should get manual changes unless both teams agree.

How long to run

• Allow for the learning period before reading results.
• Run long enough to capture the full time from lead to your primary KPI, plus enough matured volume to make a confident call.
• Don't end early. Avoid stopping on immature or noisy data, and agree the readout date up front.

Budget and pacing

• Give the test arm room. It needs enough budget to respond to the new signal; a budget-capped campaign can't show the impact.
• Watch pacing daily. If one arm under- or over-delivers, adjust targets such as target ROAS, and record the change.

Measurement

• Check tracking before launch. Confirm conversion tracking, click-ID capture (GCLID, GBRAID and WBRAID) and CRM attribution are working. See Capturing click IDs on your forms.
• Lock definitions before launch. Fix KPI definitions and reporting sources in advance. See Choosing success criteria and Reporting during your pilot.
• Keep a change log for both arms throughout the test.

Check-ins and the readout

• Hold regular check-ins, for example every week or two, on pacing and health, separate from the final readout.
• Treat early data as directional. Make the final call at matched maturity.
• The final readout covers your primary KPI against the target, a full-funnel comparison, lead quality, what we learned, and a clear recommendation: expand, adjust or stop. See Reading your Scowtt vs BAU pilot readout.

At a glance