Don't Break Your Customers

Published: by

Anyone who does Web-scale or information technology over the past two years knows containers. The primary reason is the success of docker in making not-so-new containers easy to build, deploy, manage and use.

Personally, I think Docker containers are great. They provide a far more efficient level of isolation than VM virtualization, without sacrificing manageability.

Docker itself, however, is a young company, and every now and then young companies, whose products are moving very quickly, make silly mistakes.

The number one rule in product is, Don't Break Your Customers. If you are releasing a product or service that is an upgrade, never break what worked before without very clear warning, and avoid such "breaking changes" until major releases. Release 1.2 should add features and usability and bugfixes to 1.1, but should not break functionality of 1.1. Release 2.0 can be backwards-incompatible with 1.x, but should have lots of advance warning.

However, the number two rule is, Don't Assume You Know Everything Your Customers Do. Indeed, this is the basis for Steve Blank's Customer Development Methodology, and from it Eric Ries' Lean Startup. I always liked Steve Blank's phrasing, "there are no answers inside the building." Everything you think you know is nothing more than an assumption, until tested in the real world.

What does all of this have to do with Docker?

Docker containers are made by running "images", snapshots of the state of the filesystem. Images, in turns, are composed of lots of overlapping layers of images.

A very convenient method for passing information around is "labels", which function like Gmail tags. Labels are just arbitrary key=value pairs. You can have "foo=bar" or "jeff=him" or even "com.google.some.label=why are you not here?"

Labels can be applied both to images and to containers. For example, if my image C is built on top of B which is built on top of A, then running a container from C actually is running a container from C on top of B on top of A.

Up until the most recent release, 1.8, I could get the combination of labels for a container and all the image layers by asking the container for its labels. Let's assume that each of the layers had the following labels:

  • A: foo=bar
  • B: abc=def
  • C: me=you

We run the container based on image C and add a label:

  • container: help=someone

If we ask the container, "tell me your labels," we would get a combined list:

  • foo=bar
  • abc=def
  • me=you
  • help=someone

Unsurprisingly, many container management products made use of this convenience.

Lo and behold, in 1.8, asking the container, "tell me your labels," would return just the labels for the container. If you wanted all of the others, you have to go through the following process:

  1. Ask the container for its labels; get help=someone
  2. Ask the container what its image is; get C
  3. Ask C for its labels; get me=you
  4. Ask C what image it is based upon; get B
  5. Ask B for its labels; get abc=def
  6. Ask B what image it is based upon; get A
  7. Ask A for its labels; get foo=bar
  8. Ask A what image it is based upon; get none.

A small change, undocumented and unwarned, caused every single program that uses labels to go from 1 clean step to 8 steps... and that is just if it is 3 layers. Most images are based on 10 or more layers. Causing programs to require an additional order of magnitude of work is usually considered, in technical parlance, bad.

The original issue is listed here, for those inclined to read technology discussions. What comes across from the discussion is:

  1. Customers are very upset.
  2. The people at Docker clearly did not understand the impact.
  3. They made the change anyways.

They violated Rule #1 because they violated Rule #2. They thought they understood the ways in which people would use their product, what was rational and what was not, but were stuck "inside the building."

Of course, being stuck with constant backwards compatibility is not a great strategy either. It leads to slow, bloated software that eventually competitors without baggage rush past.

Summary

  1. Don't break your customers.
  2. Don't assume that you know your customers, and how they use your products, better than they do.
  3. Ask! Ask customers and the market, "we are planning on doing this; does that matter?"
  4. If after all of that, breaking changes really are necessary, save them for major releases, and have lots of advance warning.
  5. Document the changes and publicize them well in advance.

 

If you break your customers, partners, integrators or OEMs, or regularly have strained product-based relationships with them, your process needs help. Ask us to help you.