How to test WeatherAds using an A/B split test in AdWords or Facebook



This guide will walk you through a very easy way of setting up an A/B split test to measure the effectiveness of running weather targeting on your AdWords or Facebook campaigns. The best time to set this up is right at the start of your free 15 day WeatherAds trial. Doing so will give you concrete metrics on the impact that the weather targeting has had on your campaign performance, by the time your trial period elapses.  We recommend letting your A/B split test for 10-15 days as this will allow enough time for the test to produce conclusive results.


AdWords: How to Set Up An Easy A/B Split test for your AdWords Campaign


Setting up a split test for your AdWords campaign(s) is incredibly quick and easy. There are a number of variables you can test, but for the purposes of this guide we’ll show you how to set up a test to determine what performance lift you obtain from weather targeting your campaigns (weather targeted variant vs. non weather targeted variant). The method uses AdWords Experiments. Simply follow the steps below:


1. Log into your AdWords dashboard & click on the campaign you wish to run an A/B split test on.


2. Click on ‘drafts’ in the upper right hand corner, select ‘new’, and name your draft.



3. Hit ‘Create’ and then select ‘Run an experiment’



4. Name your experiment and allocate a traffic split between the original and the test variant. (if you’re unsure if weather targeting will make a difference, try a 50-50 split). If your product is highly weather sensitive and you want the bulk of your impressions to be weather targeted, you can choose a higher percentage in favor of weather targeting, e.g. 80-20).



5. Navigate back to WeatherAds and import one of variants in for weather targeting (we recommend importing the test variant). Create your weather targeting rules and set live.



AdWords effectively treats the variants as independent campaigns, complete with their own metrics. However, the two variants will not compete against each another, instead sharing the same traffic and budget between themselves. Crucially, the weather targeting that runs on one variant will not affect the other, as you can see from the screenshots below:  

Weather targeted variant - you can see the original geo (UK) has been automatically broken down into more granular locations by WeatherAds, and weather based bid modification are being actioned.
Original variant (non-weather targeted) The original geo (UK) has not been broken down into more granular locations, hence this campaign variant is not being weather targeted. 


6. Once your experiment has some history, AdWords will show you a report that compares the results from both campaigns, so that you can easily identify the best performer. It will look something like this:



NOTE: If you decide to set up a split test on a campaign that is already running weather targeting in WeatherAds (or has been in the past), when you to duplicate it using experiments, make sure to set the geos back to normal and make sure there are no geo bid modifiers on the control variant. If you’re starting with a new campaign (one that’s not been weather targeted yet) you won’t have to do this.


NOTE: Any manual changes you make to your original campaign will not be reflected in the variant campaign. We do not recommend you make manual optimizations to your campaign during the test period, however, If you do decide to make them, remember to make them across both variants, otherwise your A/B test results will be skewed! 


Other Variables To Test Using This Method


In the above example we’ve used the most obvious test – weather targeted v.s non weather targeted variant. However, you can also use this method to test other variables also, for example:

  • Different weather conditions (e.g. cold temperature vs rain)

  • Different amounts of a condition (e.g. moderate rain vs heavy rain)

  • Different weather-based rules (e.g. +25% vs +50% bid adjustments)

  • Weather based creative vs. generic creative

  • Weather based bid adjustments vs. switching ads on/ off based on weather

  • Different time-frames (e.g. past vs current weather)



Facebook: How to set up an A/B split test in Facebook to measure WeatherAds ROI


Like AdWords Experiments, Facebook also provides the ability to run a split test. Here's their help article on it. Below is a step by step guide to setting up your A/B Split Test in Facebook to determine the value of running weather targeting on your campaign:


1. Log into your Facebook Ads dashboard, and click on the green ‘Create’ button.



2. Pick your marketing objective, and then tick ‘create split test’ checkbox



3. Under the ‘What do you want to test?’ dropdown, select ‘creative’



4. Fill in your audience & budget as you normally would. Pick a start and end date for your split test. We recommend running for it around 10 days.



5. Upload your creative into Ad Set 'A' and create your ad. When you’re happy with it click ‘continue to Ad B’



6. Your ad should be copied over into Ad Set B. Now you have to make a minor change to your second ad because Facebook won’t allow you to test identical ads. However, the most minor change between the two creative variants is enough. We recommend changing one character in the 'headline' text (e.g. changing a full stop to a dash) or even just putting an extra space at the end of your headline, as this won't be visible to the viewer. The aim here is to make a change which is so innocuous it will not impact the results. When you’ve done that click ‘confirm’.



7. Voila! Your split test has been created. You can navigate to your ad sets and rename them so you know which one is the weather targeted variant.



8. Head over to your WeatherAds dashboard, go to import campaigns, and import in one of your newly created ad sets for weather targeting. Do not import the other variant, which will be your control group. Create your weather targeting rules for your ad set and set it live. Congratulations! Your test has been set up. Now all you have to do it sit back and wait for the results of your test to roll in.



As you can see, the weather targeting will be applied to one of your ad sets but not the other:

Weather targeted variant - you can see the locations are being added and removed based on localized weather. 
Control Variant (non-weather targeted) Locations have not been broken down by WeatherAds.


9. Let your split test run and then check the results to see which variant performed better – the weather targeted variant or the generic variant. You can run your results through an A/B significance test to determine if they are valid.



NOTE: Whilst an alternative method to run an A/B split test in Facebook is to simply duplicate the ad set, and add weather targeting to one but not the other – you may run into issues (for example internal competition between the 2 ad sets) caused by both of your ad sets targeting exactly the same audience and exactly the same locations. With the A/B split test method described above, Facebook splits your audience into 2 groups, ensuring no overlap, whilst your geo targets remain identical across your 2 variants. This is by far the fairest way of running a weather targeting split test.