Multi-Variant Testing of Online Media

Online media budgets have been growing exceptionally year-to-year, some analysts have indicated that one day online media will surpass good old TV. This really doesn't surprise me; you can purchase a target audience online. With TV you can only hope that out of the 10 Million eyes that viewed your spot some of them were influenced and may purchase your product. You can’t track them to sale, cart, or even a KPI. And that is only the beginning. Within online media you can continually optimize message, creative and calls to action to drive more actions. This is where multi-variant testing comes into play. If you are trying to assess if banner A is performing better than banner b you can run a split test to see which one wins. If you want to know if placement a with banner a and message a with call to action a performs better than placement a with banner a and message b with call to action a then you need to start building a multi-variant testing matrix. This is easier said than done. We were just discussing a test where the banner had four calls to action and we wanted to assess which combination was more effectually driving actions. Well this became a production nightmare. To produce every permutation we needed to create 75 different combinations. We then needed to run them in the same place with a comparable weighting, which meant that we needed to run them all at 1.25% of the time to ensure a proper distribution. This is all well and good but to get a significant sample we will need to run this placement for a month to get any actionable data.

Is this worth the time? Should we be doing this? Well those are really the two major questions. If the results display that one combination converts consumers at a significantly higher rate and by running that "winning" banner we offset the cost of producing 75 permutation then I would say yes, if the difference is negligible then you have learned not to perform that test again.

So, what I am saying is that no matter the outcome the results will benefit the company, good bad or indifferent. I also believe that the tests need to be controlled because if you have to create too many permutation you never be able to get an adequate sample and you will be in testing mode for years before you have actionable results.

Just test within your means.