Challenging Why (not if) Scrum Fails

This was originally published in May 2009

Virtually 2 years ago I wrote a blog called Challenging Why (not if) Scrum Works.  Basically, I was looking to see why Scrum worked so I would be able to best take advantage of Scrum. I believe Scrum works very much due to the structure of the team, the iterative nature of development and the proper context within which the team works.  In this prior blog, I reported my experience with teams that were co-located, had all team members work together and worked on only one project compared with those who were not co-located, had team members of different roles report to different managers so they were not always working together and these same people were typically on 3-5 projects at once. The co-located teams were three times more productive than the other ones – even though the people, domain and customers were virtually the same. I thought this was a great insight for two reasons.  First, it meant if you couldn’t deliver (or even build) in increments, there were things you could do to improve your development methods.  Second, if you could do incremental development, these practices were some of the first things to implement.

Continue reading “Challenging Why (not if) Scrum Fails”

Going Beyond Design Patterns

This was originally published August 2012

In our Design Patterns Thinking  course we quote Christopher Alexander’s Timeless Way of Building – the book that inspired the software development community to create design patterns. Most everyone familiar with design patterns is familiar with Alexander’s phrase “patterns are solutions to recurring problems in a context.” However, my favorite quote of Alexander is 4 pages from the end of his 549 page book –”At this final stage, the patterns are no longer important: the patterns have taught you to be receptive to what is real.”

For those new to patterns, these two quotes may seem contradictory, but to readers of his book and attendees at our course, the typical response is “yes, I already knew that.” Design patterns are solutions to a design problem in the way recipes are solutions to a “what do I want to eat?” problem. Learning design patterns as solutions to recurring problems in a context in order to be a great designer is like learning how to be a great chef by learning recipes. While useful to a degree, it is not very effective to become a master.

We need to remind ourselves that patterns are examples of solutions given a particular set of issues to be dealt with. The Gang of Four’s set of 23 patterns provides a meta-pattern that can teach us how to handle variations in a problem domain. Our Design Pattern Repository lists patterns by what they encapsulate. This is an extremely important insight. In both Agile and non-Agile teams, requirements will change – the biggest differences are: rate of change, size of change, and how we prepare our code for the change.

Most new requirements change in how a concept is to be implemented more as opposed to a totally new concept emerging. Code is often written as “this is the way it’ll work” and therefore is not well prepared for when a new, different way comes along. When we truly understand the thought process of patterns, they will provide us with a different way of looking at the world. I discussed this in my article Can Patterns Be Harmful? as well as my book Design Patterns Explained: A New Perspective on Object-Oriented Design. In other words, patterns don’t just provide us with good solutions, they can provide us with a thought process in which to achieve quality solutions to unique situations for which no current pattern exists.

The focus on patterns as solutions is unfortunate. The insistence that patterns are discovered, instead of created, obscures the fact that there is a thought process underneath these patterns. This thought process is worth discovering.  One aspect of this thought process is how to refactor code using the knowledge of patterns – something called “refactoring to patterns.” I introduced this concept in our first design patterns class in 1999 (as well as my aforementioned book). It provides a basis for refactoring quality code as needed using the thought process of patterns. You can learn more about this in the Refactor To The Open Closed chapter in our Essential Skills for the Agile Developer: A Guide to Better Programming and Design book. This is also discussed in the Avoid Over and Under Design chapter the same book. If you prefer webinars, watch Avoiding Over and Under Design in Agile Projects.

If you want to learn more, the best place is to take one of our Design Patterns classes. We currently have one scheduled in Seattle – Design Patterns for the Agile Developer, October 30- November 1. If you are in another city and have 4 or more folks interested, please contact Mike Shalloway (mike.shalloway@netobjectives.com) and we’ll see if we can schedule a course near you.

 

Complexity is Not What it Used to Be

This was originally published in May 2012.

Those who cannot remember the past are condemned to repeat it. George Santayana

The time was 1847. The place was the Vienna General Hospital. New mothers in the doctors’ wards had been dying of puerperal fever with an extremely high mortality rate – three times that in the midwives’ ward. It was a mystery. It could not be explained. But Ignaz Semmelweis had been observing this for years. Had studied the situation and made some interesting connections.

The situation was very puzzling. There were two clinics in the hospital Semmelweis oversaw. Clinic one was the teaching service for medical students; clinic two was where only midwives worked. Why was the presence of doctors apparently killing new mothers? Women were coming to the hospital’s maternity ward for the benefits of the child care they would receive. But the high mortality rates had them try to avoid coming on a day when they would be admitted to the first clinic. In fact, many preferred to have their births in the street, then come to the clinic for the benefits. Surprisingly, the mortality rate for those giving birth in the street was significantly lower than for those giving birth in the doctors’ clinic. It was a mystery, and nothing could explain it.

Until Semmelweis figured out the connection. And proved it – lowering the mortality rate from the 10-35% it had been to 1%. The connection was doctors working on cadavers (it was a teaching hospital) and then going to do their rounds with patients. The solution was Semmelweis instituting the practice of hand disinfection with a chlorinated lime solution he created. The results were dramatic. But his theory was incomplete. He could not explain why it worked. The existence of germs had not been postulated yet, let alone detected.

His theories were scorned. Administrators of hospitals thought the suggested disinfection process would take too much time. Doctors were not eager to admit that they had caused so many deaths. It was not until years after Semmelweis’ death that his theories were accepted – after Pasteur could demonstrate the existence of germs. For more on Semmelweis, see Wikipedia.

How does this relate to us? I would suggest the knowledge of germs to doctors is like the knowledge of flow to software developers. It is not all there is (other things cause disease than germs) but it is pretty important to know. Things that often appear complex and unknowable, are, in fact, complicated but unknown. BTW: I am not suggesting that software development as a whole is not complex, just that not all of it is complex.

I have been doing some form of agile consciously for over 20 years. I have been doing agile practices at one time or another for over 3 decades Unfortunately, that “one time or another” was hit or miss. I did it when I intuited a solution, but that was relatively rare. I am a big believer in understanding why what you are doing works – see an old blog of mine – Smart People, XP and Scrum – Is There a Pattern?

It seems the software industry has hit a crisis in the adoption of Agile. It is almost to be expected that when you hear about a large organization successfully adopting Scrum for several teams working individually, you learn they can’t quite get it to work well across teams. Why is this? Well, it’s a mystery for some. Not for others.

Since 2005 we (Net Objectives) have been helping clients who have been encountering cross-team challenges in their development methods (IT and products). Many of the insights we’ve had have come from looking at the theories of Flow and how they apply to software development. This is one reason I am so passionate about the need to understand that Scrum itself, is a manifestation of Flow and Lean thinking. By not being consciously aware of this, many Scrum practitioners can’t extend it as needed, or abandon it for better methods when available.

I have seen some development groups (75-150 folks) transform themselves almost overnight by attending to flow. I have also been somewhat mystified by much of the Agile consulting community’s resistance to many of these ideas – having once been thrown off a discussion group for insisting they were a better alternative than (still) popular methods of team collaboration.

Challenging why (not if) Scrum works

This was originally published in May 2007. Minor edits made. I have left the # of years the same to keep context.

I have repeatedly heard that “Scrum succeeds largely because the people doing the work define how to do the work” (see From The Top by Ken Schwaber for his full text). However, I do not think that is even a third of why Scrum works – and be clear, I am certainly not questioning that Scrum works – I just want to get at the why it works.

Continue reading “Challenging why (not if) Scrum works”

Shalloway’s Law and Shalloway’s Principle

This blog was first written in August 2007

A few years ago somebody came up to me in one of my classes and said, “you’re the CEO of a successful company, co-author of a successful book (Design Patterns Explained: A New Perspective on Object-Oriented Design), … you ought to have something named after yourself.” So I, of course, immediately like this guy and think – hmm, what should that be?  I came up with this:

Shalloway’s Law:

“When N things need to change and N>1, Shalloway will find at most N-1 of these things.”

Hey, I didn’t say it was complimentary. It’s a law! I have to follow it. That’s the problem. BTW: They didn’t ask me about gravity either when I was born! I really would like to break that law at times too!

Eventually I came up with Shalloway’s Principle:

“Avoid situations w

Hey, I didn’t say it was complimentary. It’s a law! I have to follow it. That’s the problem. BTW: They didn’t ask me about gravity either when I was born! I really would like to break that law at times too!

So I came up with Shalloway’s Principle:

“Avoid situations where Shalloway’s Law applies”

Shaloway’s law applied when “N > 1″ and Shalloway has to find all of the things involved. In other words, avoid redundancy (make N=1) or make it so Shalloway doesn’t have to find the things. For example, I may have redundancy in my interfaces. But I also have a cool “to do list generator” (some people call it a compiler) that when I change a method’s interface it tells me what I need to update. Better not to have redundancy at all, but if you do, make sure Shalloway’s law does not apply.

For an in depth view of this, see Shalloway’s Law and Shalloway’s Principle from Essential Skills for the Agile Developer: A Guide to Better Programming and Design.

Specifying Exceptions in TDD

Exceptions in software represent a mechanism for raising an alarm when something goes wrong. They are used when there is a potential problem that cannot be detected by the compiler, linker, or other automated aspect of the development process, and thus may potentially make it into the released product.

When an exception is declared in the code, it is basically a way of saying, “We hope this never happens, but if it does at least we’ll be made aware of it.” Getting the exception is bad news, but the problem that caused it is made visible so we can deal with it. Not getting the exception is good news. Everything is fine.

In TDD, however, this is logically reversed. Continue reading “Specifying Exceptions in TDD”

One important aspect of systems-thinking that has long been ignored by the Agile community is that a system is not its components. Rather, it must be recognized that these interact in a way that creates the behavior.

But the nature of these interactions are very complex – meaning that they can’t be predicted. Unforeseen events occur, unexpected interactions and sometimes small, even obscure events happen that cause huge side effects. This is a reflection that product development (creating the unknown) by a group of human beings (by nature unknowable) comprise what is known as a complex system.

Many people have gotten caught up in the theory of complexity going further than what I need is necessary for pragmatic effect. While it is true that complex systems can’t be predicted, there are many patterns of behavior exhibited by them. In the same way engineers were quite effective in building magnificent edifices (e.g., the Pyramids) without a full understanding of the science underneath the methods used, it is possible to adjust the behavior of complex systems without understanding the full nature of the principles involved.

We mostly need to know that 1) our changes may produce unexpected behavior and 2) we are embedded in a system where small errors can cause big, undesirable affects (the essence of Chaos Theory). However, instead of giving up and saying we can’t predict things, we can move forward with an knowing our understanding is always incomplete. We do, of course, need quick feedback, both of our actions in our work and in any actions we take towards its improvement.

Both agility of development and improvement of our methods is required.

 

The need to teach the principles that drive practices with the practices

This is an except from Al Shalloway’s upcoming book: Going Beyond Lean and Agile: Introducing FLEX – FLow for Enterprise Transformation. It is from a particularly important section called Teaching and Adoption.

I have never liked the common Scrum/SAFe approach of teaching select practices that are expected to be used as is. Their justification is that you need to understand how to use these practices before going beyond them. I have observed that while people need a set starting point they also need to understand why things are working. This creates learning opportunities from the start and enables a gradual improvement, or even, transcendence of the practice.

The martial arts model of “Shu (follow) Ha (break with) Ri (transcend)” is often used to justify this approach. But several several flaws exist in this logic. In the martial arts you are trying disengage your mind at first, not so in knowledge work In addition, no guidance on how to move from following (“Shu”) to breaking with (“Ha”) is provided. What’s worse, is that by defining Scrumbut in the way it has been, the belief that people who don’t “follow the rules” are somehow bad and get poor results.

The worst flaw, however, is more insidious. It’s the loss of the opportunity to learn while using. This would be to provide the underlying model of the practice. People can ‘follow’ the practice while seeing how it applies the underlying principles involved. This enables them to learn how to use these principles and tailor the practices for their own situation. They never break with the underlying principle but will break with the practice and possibly even transcend it completely. But not the principle.

Let me give you an example from sailing. Fledgling sailors are told to look at a flag or ribbon on the mast to see which direction the wind is going. But they are also told that they can learn to see the wind in the water if they look upwind and attend to the ripples on the waves. Newbies can immediately use the flag on the mast. But by looking at the more advanced practice of looking at the ripples on the waves they can learn to see the wind before it hits the sail – a very useful thing. So very quickly they learn to see the wind as it hits the sail and before it hits the sail. In addition, they eventually notice side ripples on the main ripples. These are from swirls in the wind. This tells them even more. They learn through a combination of using the basic practice, trying more expert practices, falling back to the basic one when needed and understanding what is happening much more. They never transcend the principles – only the practice that they started with. And they don’t have to do a sudden “break” with the practice, but can do it over time.

Learning with this combination of practices and principles is what sets Net Objectives training as different from frameworks. All frameworks are a combination of practices and principles. But the framework is defined by some core set of practices (e.g., in Scrum you must have time-boxing and cross-functional teams). The dangers of this are twofold. The most obvious one is that the practices prescribed may not fit your organization. But the other, more insidious one, is that doing this prevents you from learning as quickly as you would otherwise.

Specifying Workflows in TDD, Part 2: How

Specifying a workflow in #TDD means writing a test that says, “When entity A is called upon to accomplish a task, it must interact with entity B is the following way.” This is done when the interaction in question is part of a required workflow, and not simply an implementation decision that should be changeable.

The best way to accomplish this is to create a mock of entity B. Such a mock would “log” how it is used by entity A at test time, and then allow the test to examine the log and compare it to what should have happened. There are many ways to accomplish this. Continue reading “Specifying Workflows in TDD, Part 2: How”

Specifying Workflows in TDD, Part 1: Why

The term “workflow” in this sense is meant to indicate the way two system elements interact when performing a task. Often these interactions themselves are not exposed outside the system, and so only their resulting behavior should be tested. This is as true in TDD as it is in traditional testing.

Most workflows are implementation details that developers should be able to change so long as the right behavior is still achieved. If the developers have a better idea, or technology improves, these changes should not break tests. Continue reading “Specifying Workflows in TDD, Part 1: Why”