Complexity is Not What it Used to Be

This was originally published in May 2012.

Those who cannot remember the past are condemned to repeat it. George Santayana

The time was 1847. The place was the Vienna General Hospital. New mothers in the doctors’ wards had been dying of puerperal fever with an extremely high mortality rate – three times that in the midwives’ ward. It was a mystery. It could not be explained. But Ignaz Semmelweis had been observing this for years. Had studied the situation and made some interesting connections.

The situation was very puzzling. There were two clinics in the hospital Semmelweis oversaw. Clinic one was the teaching service for medical students; clinic two was where only midwives worked. Why was the presence of doctors apparently killing new mothers? Women were coming to the hospital’s maternity ward for the benefits of the child care they would receive. But the high mortality rates had them try to avoid coming on a day when they would be admitted to the first clinic. In fact, many preferred to have their births in the street, then come to the clinic for the benefits. Surprisingly, the mortality rate for those giving birth in the street was significantly lower than for those giving birth in the doctors’ clinic. It was a mystery, and nothing could explain it.

Until Semmelweis figured out the connection. And proved it – lowering the mortality rate from the 10-35% it had been to 1%. The connection was doctors working on cadavers (it was a teaching hospital) and then going to do their rounds with patients. The solution was Semmelweis instituting the practice of hand disinfection with a chlorinated lime solution he created. The results were dramatic. But his theory was incomplete. He could not explain why it worked. The existence of germs had not been postulated yet, let alone detected.

His theories were scorned. Administrators of hospitals thought the suggested disinfection process would take too much time. Doctors were not eager to admit that they had caused so many deaths. It was not until years after Semmelweis’ death that his theories were accepted – after Pasteur could demonstrate the existence of germs. For more on Semmelweis, see Wikipedia.

How does this relate to us? I would suggest the knowledge of germs to doctors is like the knowledge of flow to software developers. It is not all there is (other things cause disease than germs) but it is pretty important to know. Things that often appear complex and unknowable, are, in fact, complicated but unknown. BTW: I am not suggesting that software development as a whole is not complex, just that not all of it is complex.

I have been doing some form of agile consciously for over 20 years. I have been doing agile practices at one time or another for over 3 decades Unfortunately, that “one time or another” was hit or miss. I did it when I intuited a solution, but that was relatively rare. I am a big believer in understanding why what you are doing works – see an old blog of mine – Smart People, XP and Scrum – Is There a Pattern?

It seems the software industry has hit a crisis in the adoption of Agile. It is almost to be expected that when you hear about a large organization successfully adopting Scrum for several teams working individually, you learn they can’t quite get it to work well across teams. Why is this? Well, it’s a mystery for some. Not for others.

Since 2005 we (Net Objectives) have been helping clients who have been encountering cross-team challenges in their development methods (IT and products). Many of the insights we’ve had have come from looking at the theories of Flow and how they apply to software development. This is one reason I am so passionate about the need to understand that Scrum itself, is a manifestation of Flow and Lean thinking. By not being consciously aware of this, many Scrum practitioners can’t extend it as needed, or abandon it for better methods when available.

I have seen some development groups (75-150 folks) transform themselves almost overnight by attending to flow. I have also been somewhat mystified by much of the Agile consulting community’s resistance to many of these ideas – having once been thrown off a discussion group for insisting they were a better alternative than (still) popular methods of team collaboration.

Why the difference between “necessary” and “sufficient” is even more important in complex systems

First, let’s get clear. All organizations are complex. So if you’re doing software dev you are in a complex system. Aspects of it also have the possibility for chaotic events (such as the Martian Lander disaster).

But let’s consider something necessary to consider in even simple situations. Continue reading “Why the difference between “necessary” and “sufficient” is even more important in complex systems”