Software Engineering and Human Nature

This morning, Adrian Colyer posted his morning paper on a "functional programming."

Most readers  of this blog are not deep into different programming paradigms, so I will give a very short layman's overview here. For those who are comfortable, jump ahead a few paragraphs. (For the real experts, please do not nitpick on the details; the point is only to give an overview, not to debate the fine points.)

When software engineers write software, they use programming languages, like Ruby or Java or JavaScript or Go or C or C++ or Haskell or Erlang or (you get the picture), and "paradigms", or styles and methods of programming.

At a very high level (we are in the upper stratosphere), two alternate styles of programming exist.

1. Imperative: Imperative programming involves changing the state of various containers or variables, in order to get an outcome. Most languages follow this paradigm, including Ruby/Java/JavaScript/Perl/Basic/C++/etc.
2. Functional: Functional programming does not let you change variables. Once you say "x" is equal to 25, you can never change "x" to anything else. You can assign a different variable, like "y", to 40, but "x" is done.

I will try to illustrate with an extremely simple example. I won't use any code, just logic. Let us say I have the word "atomic" (very innovative). I want to write code to figure out how many letters are in that word, ignoring for the moment that just about every language under the sun already has some built in method for returning a word's length.

1. Imperative: Create a variable, like in math, call it "x", set it to 0. Cycle through each letter of the word, each time incrementing "x" by 1. When there are no more letter left, "x" holds to the length of the word: 6.
2. Functional: Create a function, call it "wordlength". Call "wordlength" and pass it your word ("atomic"). Your function will ask two questions: is it at the end of the word? return 0. Is it not? lop off the first letter of the word, and add 1 to the result of calling "wordlength" again with the shortened word. In the end, "wordlength" will get called 6 times, each time adding the previous result, which will add up to 6.

(For those who know some math or programming but want to learn functional, the best introduction is the classic, "The Little Schemer".)

Imperative programming is far easier to understand, learn and write. Functional programming while harder to grasp, is far easier to write safely and test, and is more likely to be robust in planned and unplanned situations.

As the Adrian's paper notes, the ability to mutate state, to change those variables in Imperative, is the source of significant amounts of "unnecessary complexity". The 2003 Northeast US Blackout was caused, to some extent, by two different parts of code waiting for the same information to change, a situation that simply cannot exist in functional.

So with all of the benefits of functional, why is the overwhelming majority of programmers and projects imperative? Universities teach C and C++ and Java; startups write in JavaScript and Ruby; enterprises use C# and Java.

The answer, as always, lies in human nature.

In some key ways, functional runs against the grain of human nature.

1. State reflects the real world: Is functional cleaner, and purer, and safer? Yes. But the real world is not stateless and immutable. People have state: they are walking or running or sitting or eating or sleeping; cars have state: they are parked or driving at 112 kmh or old or new or dirty; businesses have state: they are growing or flat or shrinking or moving or selling or retrenching. The entire purpose of writing software is to solve real world problems. The easiest way to do that is to visualize the real-world problems and reflect their flows in software. Just as the real world has state, software representations of the real world have state. There is actually very little in the real world that is "functional", purely inputs and outputs, with no state at all.
2. We educate against functional: In order to program functionally, we need to "recurse", or call a function from within itself. For example, in our word length count example, we had to call "wordlength" with our 6-letter word, which in turn lopped off one letter and called "wordlength" again, which in turn will lop off one more letter and call "wordlength" again, over and over again. University courses, books and online tutorials all teach against recursion. Most courses and books even teach how to eliminate recursion from your programs because it is supposed to be terribly inefficient compared to the alternative of mutating state. Of course, in software languages built for recursion, the software processor (or compiler) takes care of the efficiencies for you. Nonetheless, the mindset of "recursion is bad" - which, in some circumstances, it is - has seeped into the mindset of the majority of software engineers and educators.
3. Recursion is hard: Look at the two examples above. Which is easier to understand and write out or even think about: looping through all of the letters one by one and keeping a running tally, or calling some function that lops off a letter and calls itself while adding one to the result? Recursion makes your head spin. Once implemented, functional is far easier to manage, understand, change and test. But getting that first step done requires far more intellectual horsepower and effort.

The net result of all of the above is that there are:

• Far more people who can write imperative than functional. On StackOverflow, a site for asking and answering development questions, 819,000 people follow Java, 810,000 follow JavaScript... and 36,000 people follow Haskell, Erlang and Scheme combined. That is one and a half orders of magnitude more.
• There are far more jobs in imperative than in functional.
• There are far more tools, libraries and frameworks to simplify your life in imperative than in functional. Testing tools, monitoring tools, integrated development environments (IDEs), application servers, the list goes on.

As great as the benefit of functional are - and they are really valuable - the supply of labour, the availability of tools, and the availability of jobs all feed back into each other to keep the non-functional style dominant.

Imperative grew because it was easier to grasp and manage, in the early days of expensive memory and CPU, seemed more efficient, and because it reflected the real world. This in turn created newer generations of languages that followed the same model, taking it even deeper with concepts like object-oriented programming that represented the real world even better.

In the end, it is technical idealism bumping up against real-world human pragmatism.

Will it change? That depends on the use cases. There are distinct advantages to the functional style, especially in a distributed and multi-processor world. Where those advantages carry sufficient weight, functional languages win. Heroku, the platform-as-a-service provider purchase by Salesforce in 2012 for over $200MM, wrote one of their key components, and one of the few they did not open-source, the "routing mesh", in Erlang.

For functional to gain serious traction, displacing other languages like Java or Ruby or JavaScript, there will have to be situations wherein the pain of mutability is clearly in excess, and clear upfront its benefits, to override the complexity of a functional mindset and how it does not easily represent the real world we live in.

Summary

Technical solutions, even those that have distinct advantages over existing solutions and styles, are just as subject to market forces as the next smartphone or health food store. They must offer sufficient benefit over solutions currently in place to warrant customers shifting away from all of their existing benefits and comforts. This pattern holds true even for engineers, who normally are quite ready to learn and adopt new technologies.

When existing benefits include not only comfort, but a closer representation of the world they know and live in, the advantages must be even greater.

When you are deciding what solutions to offer, whether programming languages and styles for your software engineering team or services and products to your market, ask yourself:

1. Are the advantages real?
2. Are they sufficiently better to change current behaviour?
3. Are they sufficiently complex to resist that behaviour change?

The best mousetrap doesn't always win, and "best" is always in the mind of the user. Ask us to help you evaluate the potential of your internal and external mousetrap.