Online media budgets have been growing exceptionally year-to-year, some analysts have indicated that one day online media will surpass good old TV. This really doesn't surprise me; you can purchase a target audience online. With TV you can only hope that out of the 10 Million eyes that viewed your spot some of them were influenced and may purchase your product. You can’t track them to sale, cart, or even a KPI. And that is only the beginning. Within online media you can continually optimize message, creative and calls to action to drive more actions. This is where multi-variant testing comes into play. If you are trying to assess if banner A is performing better than banner b you can run a split test to see which one wins. If you want to know if placement a with banner a and message a with call to action a performs better than placement a with banner a and message b with call to action a then you need to start building a multi-variant testing matrix. This is easier said than done. We were just discussing a test where the banner had four calls to action and we wanted to assess which combination was more effectually driving actions. Well this became a production nightmare. To produce every permutation we needed to create 75 different combinations. We then needed to run them in the same place with a comparable weighting, which meant that we needed to run them all at 1.25% of the time to ensure a proper distribution. This is all well and good but to get a significant sample we will need to run this placement for a month to get any actionable data.
Is this worth the time? Should we be doing this? Well those are really the two major questions. If the results display that one combination converts consumers at a significantly higher rate and by running that "winning" banner we offset the cost of producing 75 permutation then I would say yes, if the difference is negligible then you have learned not to perform that test again.
So, what I am saying is that no matter the outcome the results will benefit the company, good bad or indifferent. I also believe that the tests need to be controlled because if you have to create too many permutation you never be able to get an adequate sample and you will be in testing mode for years before you have actionable results.
Just test within your means.
iTunes 101: Using the Album Art widget
-
Here's a cool new feature in iTunes 10 that we haven't mentioned yet: when
listening to any song in your library, you can double click on the album art
win...
25 minutes ago
Posts
3 comments:
Hey Chris, a comment running 75 creatives, each 1.25% of the time, for a month to get actionable data...
Am I understanding you correctly - you're changing just the call to action text, and you have 75 different possible versions?
Assuming what you're really doing is changing the call to action text plus changing several other variables such as graphic image, font size, CTA placement within the banner, etc., then you might consider performing a regression on response rates across each of your ~75 combinations.
Regression (ANOVA) is part and parcel of multivariate testing (aka multivariable aka multi-variant) and tells you which of those overall variables (CTA vs. image, for example) was influencial on the response (banner CTR).
Even with smaller sample sizes you'll get the benefit of seeing which variables were the "levers" that influence response. And understanding these levers early on in the test (say 2 weeks into it) will enable you to pare down your list of ~75 and concentrate testing efforts on fewer, more meaningful combinations. The more iterative pruning you can do, the faster you will converge on the optimal creative combination.
Hi Chris,
Great to hear more and more analysts are starting to use multi-variate tests. I agree with Dave: If you use specialized software you can make use of some great statistics that take care of the fact that you don't have to test all 75 iterations (or treatments) to get an answer on how well the different creatives (attributes) perform and what the interactions are.
There are several multi-variate test suppliers like Memetrics, SiteSpect, Optimost, Offermatica to name but a few.
My experience with multi-variate testing (about 2 years) is that you also need sound experimental design to get the data about how well a banner, landing page or AdWord will perform.
Hi Chris,
Great to hear more and more analysts are starting to use multi-variate tests. I agree with Dave: If you use specialized software you can make use of some great statistics that take care of the fact that you don't have to test all 75 iterations (or treatments) to get an answer on how well the different creatives (attributes) perform and what the interactions are.
There are several multi-variate test suppliers like Memetrics, SiteSpect, Optimost, Offermatica to name but a few.
Another comment I would like to make is that in my experience with multi-variate testing (about 2 years) is that you also need sound experimental design to get the data you want about a banner, page or AdWord. Because a lot of my clients come up with not a multi-variate test but a multiple A/B test in one experiment.
Post a Comment