What counts as science?
It’s certainly not for me to say. I’m neither a historian of science, nor a metaphysician, nor any type of arbiter of what counts as “scientific”. I did pretty well in high school, but. I abandoned the computer club (despite getting good, easy grades in Computer Science) because I didn’t have the requisite level of interest level in programming.
At one point, my ideas and track record passed enough muster (formally submitted to a national research body) in my chosen field (Political Science, with a shifting emphasis on modern political philosophy and policy studies) to be awarded a 2-year doctoral (SSHRCC) research grant that supplemented my small teaching income. That helped me eat “more Kraft Dinner with the fancy Dijon ketchup”… until the two years were up.
That was a few years ago. In any case, I’ve always admired those who undertake research studies using the most reliable methodology feasible.
In our case, we can be much nimbler. No grants required. No Ph.D. needed.
But we do need models of reliability – heroes of methodology, even – to look up to. There are real scientists, and they have to prove themselves not only to science talk show hosts, but more importantly, their expert peers. They do have Ph.D.’s. They undertake well-funded studies, collect data, and attempt to determine whether their findings are statistically valid (or valid in some other way).
(Unfortunately, the academic world – particularly in the discipline of psychology – is currently roiling in serious turmoil as it’s come to light that study after study, touting “discovery” after “discovery,” cannot be reproduced. The primary issue – one that you would think would be obvious to statisticians and scholars in most other disciplines, you would think – seems to behave been ridiculously small sample sizes. In other cases, the problem is that subjects are participating in simulations and are making decisions in a sterile environment that doesn’t reflect real-world interaction, motivation, fear, incentives, etc. And in still other cases, scholars cooked the books. They wanted to report outsized effects for fear of not being published when submitting papers representing the true, uninteresting data.)
Holding out for a hero (or two)
There is hope. Take Steve Horvath, an aging researcher (“in both senses of the word,” Horvath quips). He and his team developed a tool called GrimAge and tested it on over 7,000 people. It measures DNA methylation. It was proposed that the level of DNA methylation would correlate strongly with lifespan.
Horvath told Quirks and Quarks host Bob Macdonald: “We are, of course, statisticians and we can calculate statistical significance levels associated with these prediction tools. And it’s so significant, that it’s really more likely that planet Earth will be hit by an asteroid tomorrow than that this predictor doesn’t work.”
I love it. Dr. Horvath is confident in his science. But, now, he has mathematical proof to back it up, doesn’t he?
That doesn’t mean his method will predict outcomes for every single human. But it’s good enough that it should have many real-world applications. For example, he claims they’re working on determining a person’s smoking history just from this DNA analysis.
Adopting better habits is often associated with better outcomes. That sounds suspiciously like it could be cause and effect.
Taleb: “Just as the slick fellow in a Ferrari looks richer than the rumpled centimillionaire, scientism looks more scientific than real science.”
And: “True intellect should not appear to be intellectual.”
(Nassim Nicholas Taleb, Skin in the Game: Hidden Asymmetries in Daily Life (Random House, 2018) p. 160.)
The “Real” Scientists of PPC
As someone who has literally worn a lab coat to a PPC marketing panel, I have a lot to answer for. I’ll tell you what, I get really nervous every time I speak on that panel – the others are wearing lab coats too, but they write software as a “fun side hustle,” and/or have science degrees, and they’re tall. (Also, lack of gender balance: noted.) But fortunately, as pro athletes say, things go fine when I try to “stay within myself.” So I usually present results from some experiments I’ve run to test a feature or tactic. I would like to say that attendees are keenly attuned to whether the experiment design is sound, and whether the narrative and data presented might be missing some key bit of context. But more often than not, half the room is looking for a quick win or even a shiny object. They aren’t going to go back and begin rigorously testing their marketing hypotheses. That’s their choice.
Using a similar opportunity to present “results” as the junior gold mining companies who have unscrupulously “salted” early samples with gold in order to falsify assay results, to and pump up their worthless stock so they can dump some (surely the most inflated value of all of these, all time, was Bre-X) – I’m convinced that some presenters at industry conferences simply swap in fake numbers to make their case studies and claims appear more dramatic. I mean, what’s stopping them (other than their consciences, obviously)? No one’s checking. No one’s logging in (that would be private client data!). So don’t believe everything you hear.
Experiments are greatest when you design and run them yourself. Even then, if you’re testing a slick new tactic or feature, the experiment could be skewed towards letting that thing win. It’s always worth asking yourself what everyone’s motivations are.
And by all means, don’t listen to me just because I dress up in a lab coat once in a while.
Read Part 4: Ad Creative for PPC: Let the Data Speak
