Implementing Scrum: from shock therapy to emergent change

Tags: Agile Team Scrum

What I really like about Agile (and Scrum) is its revolutionary potential for organizational change. But change is difficult, and many Scrum implementations fail or don’t achieve their potential. How we can successfully implement Scrum? In this post, I will discuss one such approach, called Shock Therapy. Although it certainly has its share of success stories, I will present three arguments against this approach and provide a critical perspective. The bottom line is that it is doubtful if Shock Therapy can achieve sustainable change. I will round up with a few approaches that may help in achieving sustainable change, such as Scrum.org's Agility Path framework or other "emergent change" models.

Many teams start with Scrum by adopting parts of its framework, sometimes with the intention to move to ‘full’ Scrum eventually. They begin with what feels like most valuable to them at the time, like Daily Scrums and setting up a backlog. Proponents of an approach called ‘Shock Therapy’ (Sutherland et. al., 2009) argue that this approach is risky. They assume that Scrum is an ecosystem of strongly related parts. If teams cherry pick the parts they like, they will never experience the potential of Scrum. In fact, their partial implementation will likely result in decreased quality, increased re-work and resistance within and outside the team. From here, chances are that teams will abandon Scrum altogether stating that 'it did not work for them'. In reality, their problem lies with their implementation, not with Scrum itself.

The recipe provided by Shock Therapy approaches like Rapid Scrum is to 'shock' teams into Scrum by rigorously and rapidly implementing the Scrum framework without deviations. Because this is difficult for novice Scrum Masters, Shock Therapy should be applied by  experienced coaches. They are responsible for bootstrapping the Scrum process with the Scrum Team as a drill instructor. In addition to applying Scrum wholly, the coach sets the initial sprint length, determines a definition of done, requires the use of user stories and story points, replaces any digital Scrum Boards with a (traditional) whiteboard and requires teams to be present and focussed during all meetings. Teams are not allowed to bend these rules and the Scrum framework until they have worked through at least three sprints, demonstrated significant increases in sprint velocity and have a strong business case for their proposed changed. Sutherland et. al. (2009) present impressive results from Scrum Teams at Yahoo, JayWay and MySpace where sprint velocity increased by values between 240% to 500%.

The bottom line of Shock Therapy is that the basic rules are set in stone (at least for a while) and that teams and individuals are required to follow these rules. It is forced change, achieved through the application of ‘naked power’ (Loveday, 1984). Any resistance is quickly dealt with and defused by an experienced coach. The assumption is that this approach replaces old, unproductive behavior with more healthy, productive behavior. This mirrors similar techniques applied by the military to shock new recruits into effective fighting units during boot camps.

But does it result in sustained change?

Shock Therapy seems like a very powerful and effective approach at first glance. There is quite some evidence from the authors of this approach (Sutherland et. al., 2009) that successful application by an experienced coach can boost the sprint velocity of a team dramatically over the course of even a few sprints. I have not been able to find any data on how these teams and organizations faired though. Did the change sustain? Did these organizations become truly agile? Neither have I been able to find the hit/miss rate of this approach. The only thing I did find was a video post from the Atlassian team describing their mixed experience with Shock Therapy. In any case, I think there are other reasons why a 'Shock Therapy' approach is probably not the 'road to go' for most organizations.

Reason #1: it’s only about velocity

The first argument against Shock Therapy is that it assumes that an increase in team productivity (as indicated by the increased sprint velocity) is the most important goal of Scrum. This is self-evident in how Shock Therapy is 'sold' ('teams improve their velocity by X% after only a few sprints') and how teams are only allowed to make changes once they've reached a certain level of productivity. Although this is certainly important, and increasing productivity works as a powerful motivator to adopt Scrum, it does little justice to other goals of Scrum. For one, Scrum is also about learning, growing and improving as a team. It pushes teams to critically reflect on their own process periodically to see where it can be improved. This requires teams to develop modes of communication that foster giving feedback to each other, questioning the status quo and breaking down barriers to change. In terms of Argyris' learning models; model I values have to be replaced with model II values (Argyris & Schön, 1978). Second, Scrum is about customer orientation. Teams have to learn how to work together and communicate effectively with their product owner, to understand the business value of their work and to help product owners maximize product value. Furthermore, Scrum is about allowing teams to self-organize to solve complex, unpredictable problems. Teams have to experiment with types of leadership, learn how to make decisions, resolve problems and resolve conflicts more pragmatically in the face of uncertainty, unpredictability and complexity. Although Scrum is certainly about improving productivity, it is more about teams adopting the Agile principles. Shock therapy does not address this specifically, but the assumptions seems to be that the above will happen automatically when teams forcibly follow the Scrum framework to the letter. 

Reason #2: mixed messages

A second argument is that Shock Therapy is sending a mixed message when it comes to one of the core principles of Scrum; self-organizing teams. According to Scrum.org, Scrum 'is a simple framework for effective team collaboration on complex projects. Scrum provides a small set of rules that create just enough structure for teams to be able to focus their innovation on solving what might otherwise be an insurmountable challenge' (Scrum.org, 2013). Proponents of Shock Therapy argue that novice Scrum Masters are not skilled enough to implement Scrum fully. This example is used to make that point:

Novice Leadership and Team Discovery approaches at MySpace and Jayway allowed the teams to become distracted by new terminology, roles and artifacts. In the absence of strong, experienced leadership, most teams spent their formative months focused on aspects of the framework rather than on delivering value to the customer. They also under-emphasized or failed to implement critical elements of the Scrum Framework, which sets them up for limited success at best.” (Sutherland, 2009)

It seems contradictory that Scrum is about promoting self-organization, while Shock Therapy begins by blocking attempts at self-organization. Instead of cultivating and growing leadership within the team itself, an outside shock therapy coach moves in and forces a team to work according to his or her rules. This seems oddly out of place when considering one of the Agile principles: 'humans and interactions over processes and tools'. Although this "appeal to authority" is not a strong argument in itself, there is an important point to be made; if you force a process (Scrum) on a team, no matter how well this process works, are you not completely ignoring the dynamics that already exist within the team? Although these dynamics may not be productive, they are there for a reason. They are part of the history of the team and apparently have some value. As such, they should be treated with respect. A more prudent approach would be to help teams see why their old dynamics don’t work, and help them replace these dynamics with new ones. Just by doing this, they learn some key lessons that will help them become Agile.

Reason #3: organizational change without cultural change will fail

And this is where a final, deeper, argument against Shock Therapy can be made. Implementing Scrum is a form of organizational change. It is about moving teams or entire organizations from one state to a desired other state. This makes it no different from implementing a new organizational strategy, restructuring a department or attempting to reduce racism or bullying on the workfloor. These kinds of changes fail more often than not, as evidenced by dozens of studies.

In 2008, a study conducted by IBM among 1.500 change management executives worldwide revealed that 60% of change programmes fail to meet their objectives (Jørgensen, 2008). 70% of process re-engineering changes fail to meet their mark (Kotter, 1995), as do 80% of cultural changes associated with mergers and acquisitions (Smith, 2002) and up to 75% of all Total Quality Management initiatives (Spector & Beer, 1994). In a survey of over 1.500 change executes, McKinsey identified a failure rate of 70% for all kinds of organizations change (Fine et. al., 2008). Balogun and Hope Hailey (2004) report that 70% of all change programmes fail to meet their objectives. So, despite our best intentions, most organizational change efforts fail to deliver. Or worse, after a bit of experimentation, people simply revert to their old habits. Implementing Scrum is really no different, as evidenced by the statement of one of Scrum’s founders (Ken Schwaber) during an interview with AgileCollab (2008) estimated that "75% of those organizations using Scrum will not succeed in getting the benefits that they hope for from it.". For the purpose of this blog, I'm mostly interested in why the transformations failed to hit their intended mark, even though it can be argued that the definition of "failure" is very vague.

One could attempt to execute change by focussing on changing behaviour, like a Shock Therapy approach. After all, organizational change should obviously result in how we do our work and how we work together. But this is a superficial, hollow, change. One of the largest barriers to sustainable organizational change is its own culture, the shared set of mental assumptions that define 'how work is done here' (Deal & Kennedy, 1982). Many scholars consider it the primary source of organizational inertia (i.e. Schein, 1992). If change is not anchored in organizational culture, it will not sustain (Kotter, 1985). In the aforementioned study by IBM, changing mindsets, attitudes and corporate culture (the soft elements of change) are considered the primary challenges for succesful change.

As employees of organizations, all our behaviour is a manifestation of a deeper set of values ('how we work here'), even if we are usually not aware of them (Schein, 2004). This is our organizational culture, our mindset, and it pervades all our behaviour and determines how we interpret behaviour from others and react to it. If we truly value learning and transparency in our organization, we will spend time discussing things that went wrong and how we can improve them. If we don't, we will ignore such opportunities, find scapegoats or play politics. If we truly value customers in our organization, we will take them seriously in everything we do. If we don't, we will curse them regularly for their stupidity, ignore their (valid) concerns and blame them for their lack of direction. If we value and trust the opinions of others in our organization, decision making will be more participatory and democratic. If we don't, a single authoritarian manager will decide for all of us. Although we can change the behaviour in these examples, we should also change their underlying values. When those values change, behavior will change. This goes hand in hand. If a shortcut is taken and only behaviour is changed, there is a real risk that the underlying values are not changed. In that case, the change will not be sustained. Any new behavior will falter and revert after failures, when the 'all knowing coach' leaves or simply as a function of time. Although we did what we were told, we didn't really 'get it'. Shock therapy seems to assume that values will change as a result of changing behavior. But this assumption is highly unlikely if we consider empirical research done by organizational development professionals. The bottom line is that there is a big difference between doing Agile and being Agile. Shock therapy seems to focus on the former, while ignoring the latter. As a result, change will probably not be sustained.

But what's the alternative?

Although Shock Therapy seems like a quick way to implement Scum, and may be a practical approach when very rapid change is necessary, it is questionable if the resulting change persists. As Kotter (1995) noted in his review of hundreds of organizational change programmes: “The most general lesson to be learned from the more successful [changes, ed], is that the change process goes through a series of phases … that require a considerable length of time. Skipping steps creates only the illusion of speed and never produces a satisfying result”. Therefore, a more thorough, thoughtful and gradual approach to implementing Scrum will yield superior results. But it will take a lot of time and lot of effort to break through organizational inertia, resistance and culture. One example of such a gradual approach is the new Agility Path framework, recently presented by Scrum.org at the Scrum Day Europe 2013 (http://www.scrum.org/Agility-Path). This framework appears to have substantial overlap with a model from the field of Organizational Development (OD), focussed on treating change as an "emergent" phenemenon. This "emergent change approach" tries to move teams towards a defined vision (like Scrum) in incremental steps, while implementing strong feedback loops to constantly identify and tackle new bottlenecks that “emerge”. Every cycle involves the experimentation with new behavior and new practices (from Scrum or otherwise). This approach applies principles from Action Research models and Systems Theory to make organizational change - that is highly unpredictable - manageable and actively involves everyone in the change process. It also emphasizes the use of 'objective measure' (or 'metrics', in the IT world) to frequently evaluate the effect of interventions.

In my next post, I will describe this approach in more detail and derive a potential application for implementing Scrum. I will also compare to the Agility Path framework.

A note on statistics

I would be very interested in a more thorough empirical analysis of the success of Scrum implementations. What is the achieved effect? Did the effects sustain? What worked and what did not? This kind of objective data is hard to come by or not available at all, but it would allow the comparison of different models and approaches for change. It also makes debates on the best approach less sensitive to the latest management hypes. I would be very interested in helping with this, or findings ways to gather this kind of data.

References

AgileCollab (2008), Interview with Ken Schwaber, original site offline, but original quote retrieved from http://mattcallanan.blogspot.nl/2010/02/ken-schwaber-on-scrum.html (Accessed July 3);

Argyris, C. & D. Schön. (1978),Organisational Learning. Reading, MA. Addison-Wesley;

Balogun, J. & Hope Hailey, V. (2004) Exploring Strategic Change (2nd. ed.), London: Prentice Hall;

Deal T. E. & Kennedy, A. A. (1982). Corporate Cultures: The Rites and Rituals of Corporate Life, Harmondsworth, Penguin Books;

Jørgensen, H. H., Owen, L., & Neus, A. (2008). Making Change Work, IBM Global Services;

Kotter (1995), Why Transformation Efforts Fail. Harvard Business Review, March-April;

Loveday, L. (1984). Change strategies and the use of OD consultants: Part 1, Leadership and Organizational Development Journal 5(2): 3-5;

Fine, D., Hansen, M. A., & Roggenhofer, S. (2008). From Lean to Lasting: Making Operational Improvements Stick. The McKinsey Quarterly, November;

Schein, E. M. (2004).Organizational culture and leadership (3rd. ed.). Jossy-Bass;

Smith, M. E. (2002).Success rates for different types of organizational change, Performance Improvements, 41-1, pp. 26-33, January;

Spector, B., & Beer, M. (1994). Beyond TQM programs. Journal of Organizational Change Management, 7(2), pp. 63-70; 

Sutherland J., Downey, S. & Granvik, B. (2009), Shock Therapy: A bootstrap for hyper-productive Scrum, in Agile Conference, 2009. AGILE ‘09, August 24-28, 2009, Chicago (IL);

Christiaan Verwijs
Christiaan Verwijs

Scrum Master, Trainer, Developer & founder of Agilistic