Testing can seem like a magic pill that will unlock all of the secrets of the paid media universe.
Experiment frameworks are commonplace these days. When performance is down, it’s easy to fall back on a test-and-iterate approach. It feels like the right move.
But though it’s tempting, testing isn’t always practical or effective. It can actually be counterproductive.
Let’s dig in.
When To Test On Paid Social
Testing is productive when you have two highly distinct creative options or audiences, and plenty of time or a large budget.
When you have two creative directions and want to minimize subjectivity, it’s a good time to test. Pressure testing your assumptions is important when you have a bold idea or point of view, but it’s less effective when you don’t have an existing hypothesis.
Split tests using different creative elements, such as visuals, copy, or calls to action, gather data-driven insights on what performs best. Built-in split testing tools offered in Meta and TikTok make it straightforward to implement holdout tests.
Additionally, testing makes sense when you have two vastly different audiences on which you want to test the same creative in parallel. This approach can help you make data-informed decisions and better understand your target consumer.
When Not To Test On Paid Social
In either of the aforementioned scenarios, it’s crucial that you have sufficient time to run a test. If your test reaches only a small sample size (say 5,000 users), it’s unlikely to be statistically significant. Be sure that the results of your test cannot be explained by coincidence. The larger the sample size, the more conclusive the results (typically). Use this tool to determine the necessary sample size for your test.
If you plan to run a test campaign for only a few days or weeks, it’s probably not worth it at low spend levels because your test will be stuck in learning mode. If you want fast results that are statistically significant, you’ll need a larger budget.
Other challenges to testing are when there are external factors at play, or your options aren’t highly differentiated.
Let’s go back to high school statistics. A/B testing is a type of regression analysis, so you first must control for any outside variables that could be influencing the results.
For example, say you’re testing two creative options to advertise a set of bed sheets. One option emphasizes that the sheets keep you warm in the winter, and one option emphasizes that they keep you cool in the summer.
If you’re A/B testing the creative in December, the “keep you warm” option is probably going to win. But that doesn’t necessarily mean that your audience prefers that option overall—they’re just reacting to cold temperatures.
What we’re saying is that your tests won’t be accurate if your business is seasonal or dynamic. If the price of your goods is changing week-to-week—or something else is fluctuating in a way that directly affects demand for your goods—you cannot confidently draw any conclusions from A/B testing.
Your tests are also unlikely to be statistically significant if your creative or audience options aren’t distinct. The novelty effect or selection effect could be at play—not to mention local maximums, which is what occurs when you reach peak performance within a narrow set of solutions. Additionally, if two ads aren’t highly differentiated, there probably won’t be a significant variance in performance.
How To Approach Paid Social Tests
Sometimes the conditions are imperfect for a split test, but don’t let perfect be the enemy of good. In many cases, it’s OK to be directionally correct. For instance, if you test two ad sets against each other in a Meta experiment targeting different geos, the ads in each ad set may not get identical delivery. That doesn’t mean the results will be worthless.
We recommend testing when it makes sense, but don’t blindly follow the “always be testing” mantra just because it seems like a safe move. Testing should not be used to replace common sense. You can still be in a learning mindset and track what has worked and what hasn’t even if you’re not running split tests.
Making decisions based on proven tactics, common sense, and creative instincts rather than relying exclusively on tests can help achieve better short- and long-term results.
Curious to learn more about paid social testing? Contact us at [email protected] for a consultation.
