Most everyone has heard about complexity in software development by now. The takeaway many get is that it’s impossible to make predictions. But this is an over-reaction. Complexity means is that you can’t count on your predictions. This means we must always use feedback to see if our expected improvements really occur. When they don’t we should look to see why they didn’t.
While I agree that complexity means that we can’t predict what’s going to happen when we make a change. But not having certainty does not mean there aren’t forces we have to attend to. It’s like driving. While complex, driving on the correct side of the road makes what happens both more predictable and effective.
What we must be aware of is that we may not get what we expect to happen. This is not failing but rather an opportunity to learn about learning the relationships between the people and actions in our organization. Some we don’t see well and others don’t behave the way we expect. But by considering what should happen contrasted with what does, we learn about these relationships when we get unexpected behavior. We can then take action based on this new knowledge.
It’s not failing when we learn fast in this way. It’s how we get improvement in complex systems and those that can go off course because of small misunderstandings and mistakes. The key, of course, is feedback, continual learning and creating a model of how things work we are always suspicious of. If all you do is inspect and adapt without creating a model of what’s happening you lose the opportunity of increasing your understanding.
In my FLEX book I am building out such a model. Check it out at https://portal.netobjectives.com/pages/books/going-beyond-lean-and-agile/creating-a-roadmap-for-improvement/#ActionsThatHelp
Systems thinking tells us to attend to the relationships between the components of a system more than the components themselves. The challenge is as the number of components increase the number of relationships between them increases exponentially. And, when these relationships have to do with human behavior they are only somewhat predictable. But this complexity does not mean that we cannot make reasonably accurate predictions about the affects of change in a system. Nor that we can’t learn how to make better predictions when we are wrong. While it is true that one cannot have certainty on what affects a change will make, there is a fair amount of predictably available.
The approach to take is to make predictions based on the laws of development present and on the known relationships and current state of the organization. We can make a change, but know we may be missing some relationships and possibly be misinterpreting others. We therefore consider the change an experiment. One that we think will work. If we don’t get an improvement we have likely found out about a relationship we weren’t aware of or a relationship that we have to be careful of. In either event we’ve learned something and our next “experiments” should be better.
This was originally published in May 2012.
Those who cannot remember the past are condemned to repeat it. George Santayana
The time was 1847. The place was the Vienna General Hospital. New mothers in the doctors’ wards had been dying of puerperal fever with an extremely high mortality rate – three times that in the midwives’ ward. It was a mystery. It could not be explained. But Ignaz Semmelweis had been observing this for years. Had studied the situation and made some interesting connections.
The situation was very puzzling. There were two clinics in the hospital Semmelweis oversaw. Clinic one was the teaching service for medical students; clinic two was where only midwives worked. Why was the presence of doctors apparently killing new mothers? Women were coming to the hospital’s maternity ward for the benefits of the child care they would receive. But the high mortality rates had them try to avoid coming on a day when they would be admitted to the first clinic. In fact, many preferred to have their births in the street, then come to the clinic for the benefits. Surprisingly, the mortality rate for those giving birth in the street was significantly lower than for those giving birth in the doctors’ clinic. It was a mystery, and nothing could explain it.
Until Semmelweis figured out the connection. And proved it – lowering the mortality rate from the 10-35% it had been to 1%. The connection was doctors working on cadavers (it was a teaching hospital) and then going to do their rounds with patients. The solution was Semmelweis instituting the practice of hand disinfection with a chlorinated lime solution he created. The results were dramatic. But his theory was incomplete. He could not explain why it worked. The existence of germs had not been postulated yet, let alone detected.
His theories were scorned. Administrators of hospitals thought the suggested disinfection process would take too much time. Doctors were not eager to admit that they had caused so many deaths. It was not until years after Semmelweis’ death that his theories were accepted – after Pasteur could demonstrate the existence of germs. For more on Semmelweis, see Wikipedia.
How does this relate to us? I would suggest the knowledge of germs to doctors is like the knowledge of flow to software developers. It is not all there is (other things cause disease than germs) but it is pretty important to know. Things that often appear complex and unknowable, are, in fact, complicated but unknown. BTW: I am not suggesting that software development as a whole is not complex, just that not all of it is complex.
I have been doing some form of agile consciously for over 20 years. I have been doing agile practices at one time or another for over 3 decades Unfortunately, that “one time or another” was hit or miss. I did it when I intuited a solution, but that was relatively rare. I am a big believer in understanding why what you are doing works – see an old blog of mine – Smart People, XP and Scrum – Is There a Pattern?
It seems the software industry has hit a crisis in the adoption of Agile. It is almost to be expected that when you hear about a large organization successfully adopting Scrum for several teams working individually, you learn they can’t quite get it to work well across teams. Why is this? Well, it’s a mystery for some. Not for others.
Since 2005 we (Net Objectives) have been helping clients who have been encountering cross-team challenges in their development methods (IT and products). Many of the insights we’ve had have come from looking at the theories of Flow and how they apply to software development. This is one reason I am so passionate about the need to understand that Scrum itself, is a manifestation of Flow and Lean thinking. By not being consciously aware of this, many Scrum practitioners can’t extend it as needed, or abandon it for better methods when available.
I have seen some development groups (75-150 folks) transform themselves almost overnight by attending to flow. I have also been somewhat mystified by much of the Agile consulting community’s resistance to many of these ideas – having once been thrown off a discussion group for insisting they were a better alternative than (still) popular methods of team collaboration.
First, let’s get clear. All organizations are complex. So if you’re doing software dev you are in a complex system. Aspects of it also have the possibility for chaotic events (such as the Martian Lander disaster).
But let’s consider something necessary to consider in even simple situations. Continue reading “Why the difference between “necessary” and “sufficient” is even more important in complex systems”