🧪 Running Experiments

= De-risking decision-making

Before we talk about experiments, I need you:

I know folks reading the SBO Hub blog enjoy thinking about:

🧠 types of strategy,

🤑 developing pricing strategy,

⚙️ why the GTM org needs RevOps,

😎 building the BizOps talent machine,

….and of course using AI.

When I wrote about this last one I did say I think it will be the one post which will age very very quickly. So today I want to test the hypothesis, and I need your help.

In true SBO Hub fashion, we go data-driven.

If you are reading this, and you are in Strategy & BizOps, now is your time to shine! Please head to the link and tell us about your preferred AI tools and use cases. No naming and shaming, just the stuff that you actually love.

At the end of this you get a cool bonus: what did I use to fill out this form?

Spoiler alert, yes, it involves an AI tool 😄.

The form will be open until April 24th 27th - I am extending the deadline because I am OCD and I need a nice round number of responses. Who likes something that ends in an ‘8?’

Once you complete it, why not share this with your BizOps peers? Or your COO? Thank you and see you around for the results!

---

And now, onto the Experiments!

When things are moving fast, and opinions are flying right, left, and center, it is easy to trip up. Sometimes it is clear you did. The worst is when it isn’t. And you keep tripping up.

Usually in such an environment founders will hear me say “Let’s slow down, to speed up.” But oh dear the blasphemy - the S-word! “Slow down? Are you mad?” I then have to remind people they need to slow then - yet again - and ask themselves “ What does she mean?” before they judge. Critical thinking goes out of the door when you are stressed. Fact. So, here is what I mean: “slow down” doesn’t mean don’t execute quickly. I believe that more often that not, that’s what they hear. They hear, let’s stop executing. But in reality, what I mean is, “Let’s de-risk this decision”, at least to some extent. What that in turn means in practice is to run an experiment or two. Such that when you have the results form the experiment, you remove some risk (or in other words uncertainty) and you know that your execution actually makes progress towards your Goal.

There is a rule of thumb in business that de-risking is about improving the Progress to risk ratio.

P / r = the higher the ratio, the more de-risked the decision making

Progress (P) of course relates to strides made in the direction of your Goal. And risk (r) is about wasted time, resource, or reputation¹. De-risking is important (when done well) because if your company is to become a “revenue machine”, you need to become a learning machine first. The company needs to be an instrument for discovering your big revenue growth levers. That’s Progress.

Let’s imagine you got your Vision of the world, and your Strategy (on paper at least). You have a meticulous Plan for execution with prioritised bets. Your BizOps team has the capability to set, monitor, and report insight on the system’s performance (I am calling the company a system, because that’s what it is: a system for generating revenue). But no Strategy and Plan go as planned. It is no secret. So you have to be prepared to go a level deeper and get early signal about what works. The next level is the day to day Execution: the cadence, framework, and learning from Experiments, which will drive your top line decision-making.

The only question that matters is…

How are we going to grow this business?

Before running even a single experiment you need to know where to focus. At Uber we used to do a lot of analysis before spending even a minute on experiments. This gave us conviction we are spending our precious time on the valuable things to learn.

How we did it? Depending on your context, you might want to amend some of the below, but I’d generally go about it in this fashion:

1. Assemble a highly visible experimentation team, cross functionally between product, marketing, sales (if relevant), and of course strategy/ops.

2. Use the prioritised bets to determine what area to test in first.

   1. If you need more granularity, you can map your growth model to find points of leverage for the most impactful work

   2. choose the bottle necks that would double or triple your business

3. Prioritise hypotheses for testing, focusing on the upside ($$$).

   1. For costly experiments, find MVP style tests that are much cheaper²

4. Run your experiment.

5. Data, insight, learn.

6. Incorporate your learnings back into your plans and strategy.

7. Celebrate your team!

For the experiments themselves, we had assembled and iterated on a framework itself. One might say, the framework was also an experiment. 😉

Framework for Experimentation:

The below slide summarises this all. You start with:

• The Hypothesis: A statement, often written in the spirit of “We believe that [XYZ] is NOT [TRUE].” Let’s unpack it:

  • A good assumption contains a belief about your customer that’s rooted in insights you gained from your data or customer conversations.

  • Make it a null hypothesis to avoid confirmation bias.

  • XYZ stands for the customer segment you are testing.

  • TRUE stands for what is the insight gained from data, or i.e. what is the current context.

• The Experiment: Think about what you want to test. “To test this, we will change [ABC].”

  • ABC stands for what solutions are you going to launch, add, subtract, otherwise change.

  • Keep it at one variable at a time. For example if you are changing pricing, test price levels separate from messaging or packaging or something else.

• The Prediction: Then move on to what you think you will observe in the data. “We predict that [KPI] will move by [XX] units in [UP/DOWN] direction.”

  • Self-explanatory. But keep it simple, and keep it at orders of magnitude not single per-centage points.

• The Business Effect: And finally… “If we are right, we will do [KLM].” Think about what would you change about how you operate your business? If you can’t think of anything you’d do differently, it’s probably not worth running the experiment.

  • KLM stands for how will the results of the test change the way you run your business.

Final word

If you want to keep increasing your P/r ratio, you have to keep iterating. Above is my Iteration Loop diagram. Nothing ground-breaking. But it shows how experiments help push your Progress forward (in 2), and keep eliminating risks, or unknowns (in 3). It is up to you of course to have set the experiments in ways that you can do 2, 3, and ultimately improve your decision-making.

It’s the scientific way of slowing down to speed up.

1. Early stage founders usually don’t care about reputation - not that I can tell. They say they do, but they are ok putting logos on their web site that don’t belong there, and accepting a slap on the wrist when found out. But what they do care about is risk in the form of time and resource. So even if you are doing this for the efficiency of your time and resource, you still need to learn.

2. For example, a favorite example actually, don’t launch a Partner motion out of the blue. You can’t expect your motion to work form the get go if you have no resource. But you can test the appetite from Partners, by setting up landing pages, or propositions, or business model explainers.