Sat June 21 2014

The Danger of Binary Outcomes and How to Avoid Them

A common theme in running a startup or building a product is optimal time allocation. Since time is the only truly immutable resource, the most precious and scarce resources are your:

  1. development (engineering/design) cycles
  2. sales/biz dev cycles
  3. marketing (user/customer acquisition/retention/virality) cycles

Each of these cycles are iterative processes with inputs and outputs, and the outputs tell you whether you’re on the right track — whether you should commit more time and resources to the effort.

The most risky cycles are those with binary outcomes because they are “all-or-nothing” results. They can screw you over because they can zero out after a large time investment on your part. Your goal is to derisk these binary outcomes as much as possible to avoid wasted time allocated. Don’t build past what you need to build to answer your questions. Kevin Systrom and Mike Krieger call this the Sink or Swim School of Engineering.

The way you prevent this is by subdividing the big binary outcome into smaller outcomes to make sure you’re directionally correct. Use iteration to prove out your hypotheses.

Examples of Binary Outcomes

How many times have you heard or told yourself something like the following?

  • Development: “We just need to launch our iPhone app and we’ll achieve instant viral growth.”
  • Sales: “We just need to sign Nike and we’ll be golden.”
  • Marketing: “We just need 100,000 users before we can launch.”

These are all hypotheses with uncertain results. The reason you commit cycles to them is to try to prove or disprove these hypotheses.

The problem with all of these hypotheses is that they are binary outcomes — they are all-or-nothing. In many cases, they are Catch-22’s. They 1) are vaguely defined, and 2) have long cycle times before you get any information back confirming or disproving the hypothesis.

Here’s a way to visually understand this:

binaryoutcomes

Let’s take for example the development cycle binary outcome: “We just need to launch our iPhone app and we’ll get viral growth. There is a huge market need for this app.” The two possible outcomes here are:

  1. True. Expected Value: There will be viral growth when we launch our iPhone app.
  2. False. Expected Value: There is little to no traction.

Each of the two outcomes have a probability and expected value. The first outcome, “True,” is the confirmation you’d like to see, and the second outcome, “False,” has an expected value of 0.

Cycle Time: 3 months: Let’s say that you want to “stay in stealth mode” and not get any feedback except from your closest friends until you launch. It will take 3 months to develop the product and launch it.

The main problem here is that it’s a binary outcome. If after the 3 months, you realize that there is little to no traction for the iPhone app, you will have wasted 3 months of dev cycles building something for which there is a low probability of success.

It would be optimal to realize that there is no light at the end of the tunnel before allocating 3 months worth of dev cycles to an outcome with an probable expected value of 0.

Solution: Interim Proof Points

The solution is to introduce interim proof points with short cycle times so that you get the information faster. Break down the binary decision tree into smaller interim proofpoints that can give you conviction that you should be allocating dev cycles.

In this example, let’s say that you introduce 3 week dev cycles to make sure you’re moving in the right direction.

Now the decision tree begins to look like this, subdivided:

binarycorrect

Before you commit to a dev cycle, make sure it matters. Make sure you have conviction. When you get information in quicker time intervals, you get a better sense of whether you are allocating your cycles in the right way. You want to get quick feedback about whether it is worth your time to pursue it.

Examples of Interim Proof Points

  1. User Acquisition: “A user will convert through the sign up funnel and download our app.”
  2. User Retention: “A user will return at least once a week to use our app.”
  3. Viral Loops Potential: “There will be a viral coefficient greater than 1.”

Now say for example that you find Interim Proof Point #2 to be incorrect:

binaryreallocation

You can make sure you don’t waste dev cycles trying to get to proof point #3 if it’s iterative and dependent on proof point #2 being true.

Splitting up the long binary cycles into shorter iteration cycles can help a lot to make sure you’re allocating your time optimally.

Here’s an example: foursquare. I don’t have any specific metrics around foursquare’s decision to shift their focus from check-ins to Explore. However, I imagine the decision was difficult for the foursquare team because their check-in metrics were still good, just not growing as fast as their Explore metrics. They made the decision to focus on the higher growth opportunity rather than stick with the safer option that might have resulted in an outside competitor snatching up the Explore opportunity from them.

Here’s another example: Zynga. Back when Myspace was still dominant and Facebook was still the emerging contender, developers committed most of their resources to building on the Myspace platform. Zynga did the same, but the difference was that they tested relentlessly, and discovered that Facebook provided massive virality opportunities. The Zynga team therefore decided to reallocate their cycles to building on Facebook.

This was a radical decision, since the Facebook opportunity was still small. However, much like the pattern in the Innovator’s Dilemma (Clay Christensen), Zynga jumped to a separate value curve with an exponential growth rate rather than stayed on the larger, but slow growth Myspace market. By the time others realized the potential of Facebook, Zynga had already entrenched themselves.

The key point here is that Zynga: 1) tested relentlessly, and 2) had extremely short dev cycles to be able to realize the disruptive opportunity in Facebook. If they had long dev cycles, it’s quite possible they would miss those early signs.

Shorten Your Iteration Cycles!

So the goal is to continue to create shorter and shorter iteration cycles to make decisions faster and more optimally. You should be getting feedback and confirmation you’re moving in the right direction on the order of days, or even hours.

Don’t Leave Your Fate Up To Randomness

To summarize, your goal is to make random binary outcomes deterministic rather than random. When you have a binary decision tree, it is hard to break down the specific ways that you can control your fate. Instead of having one big probability, you should break out a single binary decision tree into multiple branches.

Don’t get stuck wandering the forest. This framework helps you avoid the sunk cost problem. It’s not productive to work on the small things.

Subdivide the big binary outcome into smaller outcomes to make sure you’re directionally correct.

Thanks to Andrew Chen, Robert Goldberg, and Vish Makhijani for helping me crystallize my thoughts in this post.

Browse by Category