Why Scrum Works and How This Tells Us When It Won’t

This was originally posted in August 2012


We hear that Scrum works because it allows teams to self-organize. I believe, however, that Scrum works because it mandates a method that manages queues implicitly. Self-organization may get you there, and will certainly be useful in sustaining improvements achieved with Scrum’s implicit queue management. But it is not the real reason Scrum works. Knowing why Scrum works at the team level provides insights into why it often doesn’t work well at the enterprise level.

Queue management at the team level is different than queue management at the organization level. Understanding the need and value of Lean flow is essential to make the leap from team to enterprise.  Understanding how Scrum works at the team can help you see how to extend your thinking to make it work at the enterprise – or to jump to a Lean method more suited to enterprise wide product management. If you don’t see how simple Scrum methods will work across the organization, I’d suggest trusting your instincts and learning more before starting.

Background story

This section tells how I got here. It is not essential to get to the bottom-line of the blog (feel free to just jump there). But, if you are interested in learning how I got here, please read on.

I’ve been introspective about development since about 1984. Before then, I was interested in how to improve – learning new methods, etc. But I can specifically remember the insights that had me start asking why things worked, not just what I could do to do better. For a while, this was a personal endeavor. But when design patterns came out, I started asking publicly why they worked. I challenged the notion they had to be discovered. I was interested in what people were thinking that had them write code that later was considered to be good enough to be a pattern. In other words, I wasn’t as interested in learning what the good designs of others were, I wanted to know what they were thinking so that I could think that way as well. This eventually culminated in my book Design Patterns Explained: A New Perspective on Object-Oriented Design. This book is still relevant and one of the few that deals with the true issue of what patterns are. For those who want a shorter treatise, I recommend my article Can Patterns Be Harmful or my more recent blog Going Beyond Design Patterns.

When XP came out I continued my questioning – why did it work? BTW: I knew that it did, I just wondered why. I questioned why we needed to do all twelve practices to get value when they all seemed individually useful. I knew they worked because I had used them all individually at some point in my career with great value. I thought XP was brilliant for both putting them together and for giving us a framework for using them. For whatever reason, neither of these continued probes got me any points with thought leaders. I greatly appreciated Rebecca Wirfs-Brock’s understanding one day when she told me I was just a model-builder but it made people uncomfortable.

When Scrum came out, we (Net Objectives) started teaching it. For the first few years of the 21st century we were one of the biggest Scrum training organizations around. And, of course, I started asking “why did Scrum work?” After a few years of this I finally wrote a blog – Challenging why (not if) Scrum works – on this. I had clearly come to a different conclusion than the party-line. Again, my questioning was not appreciated and led to the first time I was thrown off the Scrum Development users’ group (the second was when I had the audacity to challenge if Scrum-of-Scrums can work in dysfunctional organizations).

I believe challenging doctrine and questioning why something works or is something that should even be done is both useful and the epitome of Agile. I find it both ironic and unfortunate that it is not the least bit appreciated by some Agile leaders.

The bottom line

In early 2007, I wrote a blog Challenging why (not if) Scrum works that described our realization that cross-functional teams doing waterfall out-performed siloed teams doing waterfall by 3-to-1. Others have told me it can be as high as 10-to-1. I had been suspecting for quite some time before this that it wasn’t self-organization that was making Scrum work but that it was the cross-functional teams making things work better. Cross-functional teams result in shorter queues, fewer hand-offs and significantly less time from making a mistake until it is discovered. This literally lowers the amount of work that needs to be done (see Real Tenets of Lean: Avoid Creating Waste By Eliminating Delays on our Lightning Webinars page).

Of course, self-organizing teams are important and result in sustainability of improvement. I wasn’t (and am still not) questioning the value of self-organization. Rather I am saying that knowing how to do things the right way is often more useful than having to figure it out for yourself – and that self-organization doesn’t always result in this understanding.

So why do cross-functional teams (and therefore Scrum) work? I would suggest it is because having cross-functional teams implicitly manages queues. By having shorter queues, errors are detected quickly. Scrum mandates this with cross-functional teams, but never explains why they work. Kanban provides a deep understanding of lean-flow. This, coupled with explicitly discussing work-flow policies provides the basis for continual improvement.

It is ironic that explicit policies are a great boon to self-organization but have been vilified by many in the Scrum community. Without a theory on why a group’s implicit workflow works, self-improvement often flounders. We see this in many Scrum teams – they have a great start only to languish later. This happens because self-organization requires principles to organize around once the easy gains are achieved.

Scrum can achieve great success at the team level because implicit knowledge is sufficient and is easily shared across team members. Unfortunately, implicit knowledge at the team level is difficult to share across teams. This impedes creating a common vision (or at least does not help create one). Different teams attempt to accomplish different things and this causes thrashing to take place when they attempt to integrate their work. Scrum-of-Scrums attempts to bridge this, but since there is not a shared approach across the teams, it often does not help align the teams. In any event, a higher view is required, one which the individual teams cannot provide. This is similar to the way separate fire-fighting teams may not ever see the big picture of fighting the forest fire. Someone well away from the front must be in charge and give objectives to the teams. This is the same reason that teams participating in the Scrum-of-Scrums may never be able to see the bigger picture required to achieve the high-level result desired.

Besides all of this lies the fact that queue and work-flow management at the team level is considerably different than at the enterprise level. In a team, it is easy to see when individual (local) optimizations that hurt team (global) results occur. Time-boxing is extremely useful for this. Developers may have coded all the stories, but if they haven’t been tested at the end of the sprint they’ve not completed their global challenge. Many companies see this on projects requiring multiple teams – all of the teams have completed their sprint’s stories but they don’t have completed functional code across the teams. Scrum will tell us of this impediment, but doesn’t help us see how to fix the situation.

It is often said that that’s not the point of Scrum – telling us how to fix impediments. The purpose of Scrum is to reveal them in 30 days. Personally, I’d rather use a method that 1) predicts the impediments that will happen, 2) exposes them when they do, and 3) tells us how to fix them. Kanban and an understanding of Lean-flow do just this. I don’t know why re-inventing the wheel seems to be an attitude so prevalent in the Agile world.

Challenges at the enterprise level are much more complicated (even complex) than at the team level. They do not reveal their secrets as easily as team challenges. Some science (i.e., lean-flow) is necessary.

Using Lean-Flow to avoid challenges met by organizations using Scrum

After several years of success at the team level using Scrum, we started running into a series of challenges we could not solve with the insights/methods Scrum provided. These included:

  • Handling the cross-team technical and business dependencies of multiple teams
  • Creating enterprise wide architectures
  • Managing the workload of people who are required by several teams (this occurs due to any number of reasons but includes folks who have needed subject matter expertise, critical skills or rare knowledge of legacy code).
  • Coordinating multiple stakeholders with multiple teams
  • Aligning groups with disparate motivations

Does this mean my investment in Scrum has been wasted

In a word – no. In two words – absolutely not. Scrum provides many insights that help get teams started down the Agile path. This does not, btw, imply that it is always the best place to start. But what has been learned is definitely of value. It also doesn’t mean you must abandon Scrum and go to something else. But you likely do want to extend your use of Scrum and possibly change your mindset about Agile as well.

The key is to realize that Scrum does not provide the insights required to help manage enterprise queues. These are much more complicated because of the organizational dynamics involved.

What to do next

Go to our Team-Agility / Scrum Support page. A wealth on how to do Scrum and/or Kanban better.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.