Sign in Subscribe

Assumption Testing

The Dos and Don'ts of Hypothesis Testing

I see many product managers struggling with how to design and run good product experiments. It’s hard to know what to test and for how long. Few of us are trained to think like scientists. The tools are hard to use. We often overgeneralize our conclusions. It’s great

Why You Can (And Should) Experiment When Building Enterprise Products

Why You Can (And Should) Experiment When Building Enterprise Products

Every time I speak about hypothesis testing, I get a series of questions about how to run experiments from people working on enterprise products. The assumption behind these questions is that hypothesis testing only works with consumer Internet products. And the assumption behind that belief is that the only way

Run Experiments Before You Write Code

The Lean Startup has a flaw. It’s a simple one. It advocates the feedback loop: Build -> Measure -> Learn. I agree wholeheartedly with this loop. The flaw is in where you should start. I prefer: Learn -> Build -> Measure It’s a subtle

Don't Rely On Confidence Alone

You find yourself in a conference room with your product team. The white board is covered with sketches. You have an idea that everyone is excited about. You mapped it out. You have a plan. Your engineering team is anxious to get started. This is what makes building products fun

How to Estimate the Expected Impact of a Product Change

How to Estimate the Expected Impact of a Product Change

Your design team is excited about a new signup process. They are convinced it will increase registrations. Your sales team keeps asking for the same feature over and over again. They think it’s responsible for lost sales since your biggest competitor features it prominently. Your product manager wants to

What to Do When You Don't Have Enough Traffic To A/B Test

You’ve read the Lean Startup. You’ve drank the Kool-Aid. You’re ready to start experimenting. There’s only one problem. You don’t have any web traffic. If the only tool in your toolbox is A/B testing, then this might be a problem. Fortunately, there are dozens

The 5 Components of a Good Hypothesis

Update: I've since revised this hypothesis format. You can find the most current version in this article: * How to Improve Your Experiment Design (And Build Trust in Your Product Experiments) “My hypothesis is …” These words are becoming more common everyday. Product teams are starting to talk like scientists.