Did Your Last Test Form a Hypothesis, or Validate One?

Did your last "test" help form a new hypothesis, or validate an existing one?

Most of us would say validate. That's usually the problem.

I just wrapped Inside The Box by David Epstein and there's a lot to unpack in it, but one piece that stuck with me is HARKing: Hypothesizing After the Results are Known. It's an academic term for a habit that describes half the "learnings" I've read, "uncovered", or heard from folks throughout my career. You go looking through the data, you find something, and you write the hypothesis afterward like you had it the whole time. The book's image for it is shooting at a gun range and then drawing the circle around the bullet hole.

Generators and validators

The way I've started framing it for my team is hypothesis generators versus hypothesis validators.

A backward read of six months of campaign data is a generator. It's useful and it tells you what to ask next, but it can't tell you what's true, because you picked the metric, the window and the cut after you'd already seen the data.

A pre-designed test is a validator: metric, threshold and duration locked before launch, delivery split evenly instead of routed by an algorithm that already favors the people who were going to convert anyway.

Most of what we call a "learning" is the first one being treated like the second.

What happened to clinical trials

Epstein's example is what happened to clinical trials. The NHLBI funded thirty large cardiovascular trials between 1970 and 1999, and more than half found a benefit. Then in 2000 researchers had to preregister their hypothesis and analysis plan before collecting any data. Of the twenty-five trials published after that, two found a benefit.

Same drugs. The only thing that changed was the requirement to name the target in advance.

Slow down in exactly one place

So yes, this is an argument for slowing down, but only in one place. Waiting has a real cost and I'm not asking anyone to sit on a signal for a quarter. Either you set the test up properly at the start, or you didn't, and then you owe the output a few minutes of honest thought about what it actually is before you spend against it.

If it's cheap and reversible, like shifting a small percentage of weight toward the creative theme that looks strong, act on the generator today. Being wrong costs almost nothing.

If it's expensive or sticky, like a new platform or an annual production budget, that read needs to become a validator first, because you're about to spend against a pattern that has a real chance of being noise.

The same rule holds when I'm building. A prompt change that "looked better" on the one example I happened to be staring at is a generator. Running it against a fixed bench of seventy cases, with the scoring decided before the run, is a validator. I've been burned enough times by the first one that the bench now exists before the change does. (The same instinct is behind if you haven't seen the code do the right thing, it doesn't work.)

My team is sick of hearing me ask it, but when someone brings up test results now, the first question is whether it's a generator or a validator.