Monkeys Crashing

In our last article, we argued that one should always design for failure. Everywhere.

The problem is, how do you enforce it?

At heart, this is a larger problem with desire vs. incentives.

• You desire your people to perform certain actions and act in certain ways, for example, design your applications for failure.
• You incentivize your people towards deliverables, without a focus on the underlying design that enables your higher-level desires.

This is a broader problem than product design; it is an issue in marketing, sales, finance and every aspect of the business.

Does the following sales problem sound familiar?

• Your sales staff are paid on commission, say, 10% of their revenue, with a $2MM "nut", or sales target.
• You need profitability.
• Your sales team continually hits or exceeds targets, yet you cannot figure out why you are missing your profit goals.

I have seen this at many companies, time and again, and the problem is the same. Let's assume you are targeting gross margins of 50%, which means that for every $2MM your sales people sell, you should make $1MM in gross profit. Sounds great.

Your sales staff, however, have a different perspective. Your top salesperson has worked hard on this $100,000 deal. Now, at the very end, the customer is pushing back hard. They want another 20% off, for a final price of $80,000. Sure, your salesperson could push back and prove value, dismantle the competition, etc. But that would take 4 more weeks of really hard effort, time that could be used to close the next deal. So your salesperson pushes back on you, convinces you that without the discount, the deal is dead, and you give in. After all, it is $80,000 in revenue.

And where did your profit go? Your costs still are $50,000, so instead of 50% in gross margin, you have 30/80 = 37.5% in gross margin, which most certainly is not 50%. Your deal revenue fell 20%, but your deal profit fell 40% Further, as these stories repeat themselves, eventually you become used to discounting ever further, providing cheaper prices and a cheaper brand rather than better product.

The problem in the sales case is that you have incentivized your sales staff to prioritize revenue over profit at all costs. Your incentives do not match the behaviour you want.

Returning to our product problem, how do you encourage, or incentivize, your brilliant product staff to actually design for failure? You probably measure your staff on some mixture of feature release speed and availability. While availability is an effect of Ephemeral Design (or Design for Failure), as a metric it has a fundamental proxy problem:

It can take quite some time for design to impact availability, during which the design becomes baked into your product and extraordinarily expensive to fix.

Thus, what you really want is to set up a system that makes it impossible to do anything but design correctly.

Netflix solved this problem a few years ago with their (funny-named) ChaosMonkey, a.k.a Simian Army, since there are many "monkeys" in the "army".

ChaosMonkey is a system that lives inside Netflix's infrastructure. It goes out and randomly kills, suddenly and without warning, services. It is the equivalent of putting a bunch of real monkeys in your data centre and having them randomly pull power plugs.

For most businesses, this would be anathema. I can think of almost zero companies with whom I have worked who would even consider this (at least when I first engaged with them). The risks seem absurd. What if this is the one service that does not recover well? What if you lose a crucial "in-flight" transaction?

That is exactly the point.

When your product people know that their work can disappear without warning, any time of day or night, they build completely differently. They have a strong incentive to "Design for Failure".

It raises the bar to an extremely high level, so high that your thinking has to change. You need to evolve.

Fundamentally, this is my earlier argument behind killing your SLAs.

When you no longer have any SLA wiggle room, when you offer 100% availability at all times and accept clearly defined penalties for missing 100%, you raise the bar high enough that you are forced to redesign how you do everything - not just product, but sales, marketing and finance, as well.

I love the Beatles's music, but one of the great critiques of "Imagine" is that a world without nations, ethnicities, families... is a world with nothing. We cannot love everyone equally; we require differences. Von Clausewitz said, "he who defends everything, defends nothing." The "Imagine" corollary might be, "he who loves everything, loves nothing."

"Four 9s" creates a subconscious "Imagine" world. 100% availability as a target recognize that it is nearly impossible to achieve it... and forces us to figure out where we can focus, where the highest return will be, and how to change everything so that the weaker parts get the most loving care, or get tossed and replaced entirely.

Summary

Incentives and the height of the bar matter greatly in how you, your employees and your business as a whole operate. While demanding incentives and uncertainty create greater levels of risk, they also drive better choices and greater creativity, which leads to better results overall... for those willing to face the challenge.

Are you designed for failure across all of your products, services and divisions? Which of your incentives are driving best behaviour... and which are driving away your best employees? Ask us to determine them for you.