test and learn

11月 132014

The online dating website OKCupid recently posted the results of some real life experiments that they had performed on their user community. The experiments included three tests to evaluate the influence of certain user profile changes in compatibility matching (manipulating compatibility scores, suppressing user profile photos and the impact of rating scales and photos – the cool versus pretty test). All websites perform experiments, but at what risk?

OKCupid CEO Christian Rudder posted on the company’s blog: “If you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.” He goes on to say “OKCupid doesn’t really know what it’s doing. Neither does any other website…Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out.”

Like OKCupid, more and more data-centric companies are using rapid experimentation approaches (also called test-and-learn) to better understand how consumers react to their products and services.
This not only influences product development cycles, but drives more effective marketing and contact strategies.

But we already test-and-learn!
– the marketers shout. Not nearly enough, if at all!

A recent survey by WhichTestWon.com highlighted that more than 25% of online direct-to-consumer marketers aren’t doing any testing of their site or campaigns. In B2B organizations, more than 50% of marketers are not testing. With web and mobile becoming an important and integrated component of many organizations’ channel strategies, you can’t afford to not continually (and quickly) evaluate those strategies. To give a sense of scale, web giants (Amazon, eBay, Google, etc.) may run hundreds of tests per day. While that likely doesn’t make sense for your organization, there’s a happy medium between not enough and too much.

You have the data you need to target - use it!

You have the data you need to target - use it!

As consumers become more aware, knowledgeable and self-empowered, it becomes critical for the business to engage with the consumer from their point of view:

What do they want or need?
What’s the most effective way to engage with them?
How do I entice them into my store or website?
How do I get them to come back?

There’s no magic bullet or secret sauce. But with all of the data available to marketers today, you no longer have to stab in the dark to find out what your customers will respond to. Not taking advantage of insight around what customers respond to (or don’t!) seems almost criminal.

For one organization, their marketing group uses an iterative test-and-learn process in their marketing campaigns. As the campaigns are designed and executed, results are used to continually adjust the campaigns going forward. Marketing automation tools allow them to create multiple variants of a campaign, and data analysis drives a closed-loop measurement process. Creativity still remains an essential component of marketing, but data provides the connection between creativity and meaning. So while you might not be able to “Amazonify” your test-and-learn capability tomorrow, it’s never too late to get started.

Want a "real world" example of how Marketing Automation can make a difference in these scenarios? Check out this story about Staples - the office superstore giant that also operates the third largest e-commerce site in the world. Let us know what you think.


tags: experimentation, innovation, marketing automation, test-and-learn
3月 272012

Having the time to catch up with what's happening on the marketing front is a real privilege. While I'm giving a presentation later this afternoon on the need for real-time engagement, what I really love about the Marketing Association's Smarter Data conference is listening to some real visionaries.

Getting better doesn't need to be hard. It just means doing things ... better. Campbell Brown should know, too - he's the Marketing Director from GrabOne, a company that's grown from 6 staff to 120 since July 2010 and achieved an over 70% market share. Not a bad trajectory!

Daily deals aren't new - back in Australia, we have Catch of the Day. What is new is how effectively GrabOne has managed to not only corner their market but to start expanding into a variety of verticals. Critically though, they didn't get there overnight - they got there by focusing on the foundations. Return on investment. Customer engagement. And, testing and learning what works.

A key message of Campbell's presentation was that everything should be driven by return. Regardless of whether you're trying to engage via social media, electronic direct mail, or online, if you don't know your return from each of these channels, you're lost at sea. This is even more important given how quickly the social media landscape continues to change; paraphrasing Campbell, "adapt or die".

So what does Campbell credit for his success? Some simple lessons:

  • Have a robust ability to do A / B testing
  • Don't overcomplicate your segmentation
  • Don't underestimate the importance of change management
  • Make ROI transparent - don't rely on "branding campaigns"

Everything GrabOne does is based on exhaustive A / B testing. They know what works and more importantly, they drive to know why. That's a critical element in driving outcomes and yet, it's so rare.

Given GrabOne's success, there's an interesting question to ask: How well do you understand the return on investment you get from your marketing spend?

tags: A/B, analytics, economic return, GrabOne, success, test and learn, value