Why Shu Ha Ri and Scrum Can Make for a Dangerous Combination

This was originally published May, 2017

Note: This blog assumes the reader understands the basic roles and practices of Scrum.

Scrum suggests that the way to improve a team’s workflow and the organization within which it works is to remove impediments to its core roles (product owner, team, Scrum Master) and practices (cross-functional teams, daily standups, and using time-boxing for work, demos and building backlogs). It takes an inspect and adapt approach that requires little understanding of the underlying laws of software development other than an acknowledgement that reducing the time for feedback is essential and that small batches are better than large ones.

Continue reading “Why Shu Ha Ri and Scrum Can Make for a Dangerous Combination”

Smart People, XP and Scrum – Is there a pattern?

This blog was originally written January 2010.There is a division in the agile community about whether one should rely on people or focus on people supported by systemic thinking (no one I know of suggests systems alone are enough). This debate is often the people over process vs. people and process (or as Don Reinertsen would say people times process). I’ve been in the agile community for some time and have seen some interesting things that I think shed some light on this debate. This long-time perspective has enabled me to see an interesting pattern. This blog will discuss the pattern of what happens when smart people do not have the proper understanding of what they are doing.

I’ll start with what I consider to be my most embarrassing moment in my career. It was in 1984, 14 years into my development career. I contracted to build what would be the software system to power the touch controlled information kiosks at Vancouver Expo ’86. At the time, this was very avant garde. I was essentially in charge of rewriting a Basic language prototype into C for both improved performance and features. Since I was experienced in both languages, I remember thinking it’ll be easy – it’s just a re-write.

There were two main components of the application. Mine was the user component that defined how the system should work. Basically you entered events on a timeline that the system would run when the screen was touch. The other was a run-time component that ran the pseudo code mine compiled. I sub-contracted someone else to do the executable program because mine looked to be the more complex beast. At the time, I had a reputation for functioning code extremely quickly (and yes, I intentionally did not use the word maintainable).

It only took me a few week to get the basics of the system up and running – everyone was pleased. I was confident of success because, given this was a rewrite I figured the customer would know what was needed and I would just be adding functionality. Unfortunately, after they started using the system for a while bad things started to happen. It seemed every time they wanted a new input feature (e.g., specifying when a new event, like touching the screen, or starting audio) I would put it in quickly and it would work, but a couple of days later I would find out that I broke something that had been functioning. The problem was that I had tightly coupled code and was not following Shalloway’s principle. Up to this time I had studied how I could code better (e.g., structured programming, etc.) but hadn’t done a study on what caused errors (e.g., tight coupling, lack of encapsulation). BTW – this is not the embarrassing part yet.

The next few weeks followed this pattern – 1) get a customer request, 2) get the request working, 3) be told by the customer the next day or two that something else was no longer working, 4) fix the new bug. This extra bug fixing work was taking a considerable amount of time. It was clear that we were in serious trouble. Now, with what I know today, I would have concerned myself with writing better code (stopping errors instead of fixing them). But what I did back then was recognize that I was causing bugs because I just wasn’t finding all the coupled cases (I was unaware of Shalloway’s Law at the time – in fact, it was this experience that inspired Shalloway’s law). I figured if there were just a way I could tell I was about to commit an error, I could continue programming fast. The problem of having to type something in several places didn’t bother me. At the time I could type about 100 wpm (not my highest speed, but still pretty fast).

I thought the answer to my problems was detecting errors quickly, and (mostly) effortlessly. So here’s what I did. I spent a day essentially writing the equivalent of a UI Test runner and sub-contracted someone to run the tests for me. While I could re-run the test cases automatically, I needed someone to set them up and check the results against good cases. I had basically instituted semi-automatic acceptance testing in 1984 (still not the embarrassing moment – this was actually pretty cool).

From this point on we zoomed along. My quick coding style was no longer holding us back. I’d make a change, give it to my tester and within 15 minutes he’d tell me what I unintentionally broke by forgetting to change something that was coupled to my fix.. I fixed it almost immediately because I knew it was something I just changed. Bottom line, we got our system out in very good time. We even became a real product whereas we were originally only supposed to be a tactical solution for the Expo. The strategic product was being built in parallel with a longer timeframe and 30 people (compared to our 4). However, our product ended up being better so they released both.

So what’s embarrassing about “inventing” automated acceptance testing in 1984 and building a product for my client within budget and time while exceeding functionality initially envisioned and with a high quality? It was that I didn’t do automated acceptance test again until 2000 when I read about XP.

This episode was one reason I knew XP would work the moment I heard about it. I had done an iterative, close customer, automated acceptance test, continuous build (there was only me! ) project 16 years earlier. Only now I had 16 years of experience in considering what made for good programming.

This was why I immediately questioned why XP worked (not if, I was clear that it did). I remember this not being very well received. At the time, Kent Beck and Ron Jeffries (two of the originators of XP) pretty much insisted that you had to do all of the twelve practices of XP or you’d lose its power. There was also little in the way of explaining how to code.

Yes, I know about the four rules of writing simple code:

  1. The system (code plus tests) clearly communicates everything that needs to be communicated at the current instant in its development. This means that it runs every existing test, and that the source code clearly reveals the intention behind it to anyone who reads it.
  2. The system contains no duplicate code, unless that would violate (1).
  3. The system contains the minimum number of classes possible without violating (1) or (2).
  4. The system contains the minimum number of methods possible, consistent with (1) (2) and (3).

The problem with this definition is that it is practiced based. It also is stated in a way that is understandable to someone who already understands these practices (that is, has intuited the principles underneath them) but will cause great misunderstanding for those that don’t have this intuitive sense.

Of course, Kent, Ron and Ward (the third originator of XP) are all brilliant developers and had the necessary intuition. Unfortunately, most of the people getting excited about XP didn’t. I remember talking to several of my associates about XP and said that without the proper understanding of what was underneath XP (something no one wanted to talk about at the time) there would be serious problems for any undertaking it. I even gave a time frame – 6 months. Now be clear, I though XP was brilliant. I just said it was dangerous without the key understanding of it. Sure enough, while many people had great success, many others had great problems with code poorly written (ironically, mostly in the test code).

Those of you who know me know I’ve said pretty much the same thing about Scrum. I’ve written on why it works and why it doesn’t. Ironically, here, as in the XP case, my comments/concerns were pretty much ignored by the Scrum community. Today we have many (most?) Scrum teams practicing what the Scrum community calls “Scrum-but” (that is, we do Scrum, but …”). I wrote a blog on this as well The 5-whys of Lean as the answer to the but of Scrum. Even Ken Schwaber, Scrum’s co-creator and biggest evangelist, has said, I estimate that 75% of those organizations using Scrum will not succeed in getting the benefits that they hope for from it.

So what is the pattern of these three things?

  • My not doing automated acceptance testing for 16 years
  • XP teams running into code problems after a few months
  • The prevalence of Scrum-But and the general lack of success by many companies undertaking Scrum

I would suggest it is counting on smart people to find the right thing to do is not always a winning strategy. That giving people understanding of the principles and rules underneath programming and development will make them much better. I admit this begs the question that I am a “smart” person. But, I do think I qualify – summa cum laude, 2 masters degrees (one from MIT), successful author, have run a successful business for 11 years (and still going), … I’m not trying to toot my own horn here. In fact, I’m saying, how could someone as smart as me do something as stupid as not use automated acceptance testing for 16 years (isn’t that embarrassing?).

Well, my answer is that relying on practices even if you are smart is insufficient. You must learn why those practices work. Of course, this makes sense only if you believe there are rules underneath what we do. Many in the agile community don’t believe this (I’ll be writing a blog on this next week). Bottom line for me is, get the best people you possibly can. Then, make sure they study their methods, as explicitly as possible, so they can create solid, support systems and understanding of what they do. You will get a much greater return from their efforts if you do so.

In my case, the understanding would have had me look to see where I could apply automated acceptance testing effectively. Years later, I now understand that one key aspect of automated acceptance testing is to eliminate the added work that comes from the delay between code and test. I clearly knew this at some level in 1984. But not at a deep enough, or consciously high enough, level to take advantage of it on a regular basis.

XP has been around long enough that people have finally gotten to the why it works. In Scrum’s case I believe we find people doing Scrum But because their lack of understanding of the principles underneath Scrum prevents them from effectively changing the given practices. They often think they are doing the right thing, when in fact, it is not effective.

This is why, at Net Objectives, all of our training and consulting starts with why things work. If this makes sense to you, and you think you can use some help in doing this, please send me an email alshall AT netobjectives.com to see if we can help.

If you want more information on what we now consider to be useful principles and guidelines for coding better, check out these resources pages (you’ll have to register to get access to some of these):

Challenging Why (not if) Scrum Fails

This was originally published in May 2009

Virtually 2 years ago I wrote a blog called Challenging Why (not if) Scrum Works.  Basically, I was looking to see why Scrum worked so I would be able to best take advantage of Scrum. I believe Scrum works very much due to the structure of the team, the iterative nature of development and the proper context within which the team works.  In this prior blog, I reported my experience with teams that were co-located, had all team members work together and worked on only one project compared with those who were not co-located, had team members of different roles report to different managers so they were not always working together and these same people were typically on 3-5 projects at once. The co-located teams were three times more productive than the other ones – even though the people, domain and customers were virtually the same. I thought this was a great insight for two reasons.  First, it meant if you couldn’t deliver (or even build) in increments, there were things you could do to improve your development methods.  Second, if you could do incremental development, these practices were some of the first things to implement.

Continue reading “Challenging Why (not if) Scrum Fails”

Challenging why (not if) Scrum works

This was originally published in May 2007. Minor edits made. I have left the # of years the same to keep context.

I have repeatedly heard that “Scrum succeeds largely because the people doing the work define how to do the work” (see From The Top by Ken Schwaber for his full text). However, I do not think that is even a third of why Scrum works – and be clear, I am certainly not questioning that Scrum works – I just want to get at the why it works.

Continue reading “Challenging why (not if) Scrum works”

TDD “Good” Tests Part 3. There must be no other test that fails for this reason

When organizations adopt TDD as their development paradigm, early results can be quite good once the teams get over the initial learning curve. Code quality goes up, defect rate goes down, and the team gains confidence which allows them to be aggressive in pursuing business value.

But there is a negative trend that can emerge as the test suite grows in size over time. Continue reading “TDD “Good” Tests Part 3. There must be no other test that fails for this reason”

What I have Learned from Scrum

Here are some things I have learned from Scrum.

  • Cross functional teams are good. Just having them achieves a three to tenfold improvement over a group of people working on several projects at once. And they improve innovation by the team.
  • Time-boxing increases discipline, visibility and the ability to pivot.
  • Small batches are good and breaking work down into small pieces is essential.
  • Smaller release cycles improve most everything.
  • It is useful to have a team coach.
  • Do not expect people to figure out what they need to do just because you have put them in a framework.
  • Focus on learning the practices of a framework makes learning what you actually need to accomplish (flow) harder.
  • People like to be given a set of practices to use.
  • Defining a simple set of practices to use can lead to rigid dogma.
  • Take an approach that transitions you to the behaviors you need.
  • Approaches that work well in one context may not work well in another even though people them everywhere without noticing this.
  • And, just because you can put whatever you want into a framework, that doesn’t mean the framework is not prescriptive. In itself, the framework has things you must do.

Continue reading “What I have Learned from Scrum”

What I Think About Scrum

First of all, I like Scrum. I think it can be a great framework when used in the right place. But I also think it must be taught as a tool in your toolbox, not an end in and of itself. This means initial training of Scrum should include more of the flow model (eliminating delays in workflow, feedback and using information) on which it is based. Test-first methods should also be incorporated into this training. This combination allows for teams to avoid most of the pitfalls teams new to Scrum face. I also believe one should look to see if Scrum or Kanban is better for a particular team (or something in between). See first comment for how I do this.

See Why Agile Coaches Need to Know Both Scrum and Kanban.   

Continue reading “What I Think About Scrum”

Why Being Explicit in Workflow is Useful: Case Study, Scrum Based on Lean Flow

Scrum is a lightweight framework that encourages correct action through performing its ceremonies and practices. While that is great in theory, the lack of the “why” of the ceremonies often has them be not followed.

Consider the “sprint” in Scrum. According to the Scrum Guide, “The heart of Scrum is a Sprint, a time-box of one month or less during which a ‘Done,’ usable, and potentially releasable product increment is created.”

It implies the need to start and finish stories in the same iteration. In other words, one of its objectives is to have a short time between the beginning of a story until it is completed. Here is what this provides. Continue reading “Why Being Explicit in Workflow is Useful: Case Study, Scrum Based on Lean Flow”

Improve Your Scrum by Using Flow Thinking

Scrum is based on empirical process control, the pillars of which are transparency, inspection, and adaptation. This mostly means that instead of following a plan we observe how things are going and adjust accordingly. But this doesn’t mean that we can’t use theory. It’s just that when we do use theory we must verify it worked. Continue reading “Improve Your Scrum by Using Flow Thinking”

Nice to See Scrum Evolve

Recently, Scrum.org announced the integration of Kanban practices into Scrum and that “Observe Orient Decide Act (OODA) is the mindset of Scrum. This comes on top of the announcement of Scrum team training a few years ago. These are all good things. It’s nice to see the Scrum community catching up to the many non-certifying consultants who’ve been doing these things for 5-10 years. Continue reading “Nice to See Scrum Evolve”