“Fools say they learn by experience. I prefer to profit by others’ experience.”

-seemingly good advice from Otto Von Bismarck, 1870-something.  

It’s easy to get scared by the sight of blood, to freeze with your fellow witnesses in the silent moment after impact. To instinctually look away in horror as Matt Samuels slowly lifts his face from the pavement and gushes a pint of blood per second through each nostril. Matt Samuels! The perpetual team captain and acknowledged owner of the most expensive bike, roller blades, and scooter on the block! If Matt can’t execute a full-twisting, no-hands backflip with a superman landing off his back porch, no one can.

And so the lesson goes: “Fools say they learn by experience. I prefer to profit by others’ experience.” Matt never tried that bike trick again. And the rest of us, we never even dared attempt a feat of such obvious (if retrospective) unfeasibility.

Today, however, as I sit at my desk and contemplate how our clients can achieve widespread recognition, passionate brand loyalty, and seemingly unfeasible feats of profitability in 2011, I wonder, “What if that giant gust of wind hadn’t hit Matt head-on upon lift-off? Or, even more intriguingly, what if Matt’s twin brother Mark had attempted the exact same trick at the exact same moment, but off the front porch? Would the same gust of wind that slammed Matt facedown onto the driveway have simultaneously launched Mark into the most glorious feat of BMX flight known to Pasadena Park Elementary School?!”

Alas, we’ll never know. We relied on experience to declare the trick impossible and to deem ourselves wise for learning from Matt’s failure.

Now, I know better. In 2011, I won’t let the brands I work with heed the same misguided wisdom and learn from experience—their own or others’. The problem with experience, whether you read about it in an industry journal, hear about it from peers who’ve “been there, done that,” or even witness it yourself from your fourth-grade neighbor’s back porch, is that you never have the full story.

We all muddle through life making decisions based on incomplete information. We consider the limited information we have and take a chance. We’ll never solve for every variable, and it would be futile to try. The goal needn’t be to eliminate every doubt, but rather to make an educated guess.  Unfortunately, the typical experiences to which we look to form our educated guesses provide, well, crappy information.

For most decisions, we can tolerate some crappy info. For the brands we work with as advertisers, however, we’ve got to try to cut the crap.

First, there are the confounding variables lurking in dark corners, tempting you to claim direct causation in cases of hidden correlation. Consider the new social shopping experience launched on your brand’s website—the one that quadrupled sales. Did you account for the potential lurking variable? The server upgrades made to accommodate the interactive experience probably also made the check-out process faster, reducing response time and eliminating many of the browser time-outs that caused shoppers to abandon their order mid-check out. Which factor really contributed to the skyrocketing sales?

Then, of course, there are the inherent biases that trick you into mistaking the cause for the effect. You produce a spectacularly creative (in your humble opinion) video that aims to convince the world that your brand’s product is cool. The video goes viral. You’ve successfully made the brand cool. Then again, you could be mistaking the cause for the effect. Maybe the video went viral precisely because everyone already thought the product was cool.

Among other inevitable sources of crappy information, there’s also the universe’s ultimate jab at learning from experience: randomness. When you do something just one time, there’s a reasonable chance the outcome you experienced was a random occurrence, rather than a meaningful repeatable event. Or, it was the result of so many different simultaneous factors that you might as well call it random.

Without actively controlling for this crap, we can’t even trust our own experiences to lead the way. Would you really look to others’ experience, then, as a superior source of information? No way. The best way we can help brands profit from experience is to ignore it. Instead, in 2011 let’s commit to test and learn for ourselves. A good experiment needs just three basic elements: replication, control, and randomization. To prove you needn’t have a doctorate in statistics to conduct a solid test, I turn to The Cartoon Guide to Statistics (Gonick & Smith 183) to explain these three principles:

Replication: The same treatments (your ad, website, press release) are assigned to different experimental units (consumers). Without replication, it’s impossible to assess natural variability and measurement error.

Control: Refers to any method that accounts for and reduces natural variability. (Create a control group of consumers that won’t see your ad. Compare their results with the exposed groups.)

Randomization: The essential step in all statistics! Treatments must be assigned randomly to experimental units. Flip a coin. (Or download the Coin Flip app for your iPhone. In digital media, this is even easier. Google Website Optimizer and other user-friendly, and often free, tools will randomize the treatment assignments for you.)

By establishing the simple principles above in your testing, you’ll create a framework for decision-making far more advanced than anything experience will provide. All you need now is the will to apply testing in your day-to-day workflow and the courage to accept that even testing won’t provide you 100% certainty of the right decision. But if you can look at the results and establish, say, even 70% confidence in your conclusion, you’ll be in a far better position to make an educated guess than anyone relying purely on experience. The key is to move forward, take action, and keep on testing and learning. While we might not achieve academic-grade experimental design, we can certainly learn from deliberate, controlled tests that allow us to quickly act upon real options to optimize results—be it sales, cost-savings, customer satisfaction, or the sickest BMX bike trick you’ve ever seen.

- Emily Hill is the Director of Analytics at Studiocom’s Atlanta office. Follow her on Twitter @emilyhilltweets.

Works Cited

Gonick, Larry, and Woollcott Smith.  The Cartoon Guide to Statistics.  New York: HarperCollins 1993.  Print.

“Good-bye Elliott.” Shirtoid.  http://shirtoid.com/9273/good-bye-elliot/ (7 Jul 2011).