Scrum implementations fail when they ignore organizational culture

Tags: Agile Team Scrum

When we implement Scrum, it's not just a matter of changing superficial behavior. Simply following the Scrum framework to the letter will not make us a perfect Scrum Team. For Scrum to truly work, we have to address and change core values about communication, leadership, change, failure, learning and teamwork that are deeply rooted in the culture of our organization. If we don't change these values, change will only be superficial and will not be sustained. In this post, I will describe the relevance of organisational culture when implementing Scrum (or any other change programme) and explain why organizational culture is often a major source of resistance. 

A product owner I know recently told me about his doubts on Scrum. Although his team did go through the motions, and value was delivered frequently, he did not feel that the team was very sensitive to his requests. New items that he added to the backlog often met with a cynical response by developers, pushing him into the defensive, and resulted in overly high estimates. When the team failed a sprint and an important deadline, the team heavily opposed spending a little overtime to get everything done. Despite the urgency of the project, the team suggested putting the leftover work in a new sprint (of a month). It’s not hard to understand why this product owner became highly skeptical of Scrum and felt that teams started using it as a shield.

Implementing Scrum is all about changing how a team does its work. It is not just a matter of changing behavior with the Scrum guide in hand. Even if a team dutifully does its Daily Standup, the sprint plannings, reviews and retrospectives it does not automatically make them a successful Scrum Team. It does not automatically result in higher quality software, sustained delivery of working software, a more customer oriented approach and a learning attitude within the team. Only 'walking the walk and talking the talk' is not going to vastly improve productivity, quality and customer satisfaction. A successful Scrum implementation requires that people change the attitudes, values and assumptions that underlie their behaviour and how they develop software. Otherwise, change will not sustain. And it's not just the teams that need convincing. The organization in which the work is done also has to change. Teams have to be given freedom to experiment, organizational barriers to Scrum have to be removed, management styles have to change and a learning attitude must be encouraged.

So, implementing Scrum is more about changing deep-seated values and assumptions and less about actually changing superficial behaviour. This is incredibly difficult, and mirrors closely the difficulty of other kinds of change programmes (the following is taken from a recent post). In 2008, a study conducted by IBM among 1.500 change management executives worldwide revealed a failure rate of 60% (Jørgensen, 2008). 70% of process re-engineering changes fail to hit their mark (Kotter, 1995), as do 80% of cultural changes associated with mergers and acquisitions (Smith, 2002) and up to 75% of all Total Quality Management initiatives (Spector & Beer, 1994). In a survey of over 1.500 change executes, McKinsey identified a failure rate of 70% for all kinds of organizations change (Fine et. al., 2008). Balogun and Hope Hailey (2004) report that 70% of all change programmes fail to hit their goals. So, despite our best intentions, most organizational change efforts don’t deliver what we expected. Or worse, after a bit of experimentation, people revert to their old habits. Implementing Scrum is really no different.

Organizational culture: a barrier to change

One of the largest barriers to organizational change is its culture; the shared set of mental assumptions that define 'how work is done here' (Deal & Kennedy, 1982). Many scholars consider it the primary source of organizational inertia (i.e. Schein, 1992). If change is not anchored in organizational culture, it will not sustain (Kotter, 1985). Schein (2004) considers organizational culture as a phenomenon that is largely subconscious:

Organization culture as an onion of layers

The visible part of organizational culture is exposed through behaviour and artifacts, like slang, jokes, management styles, dress code and furniture. Underneath this outer layer are the espoused values of a company. These are well-known (at least inside the organization) principles, rules of behaviour and norms, like a 'customer first' mantra. The lowest level of culture consists of all the deeply embedded basic assumptions and values that are shared by all employees and drive the higher levels of organizational culture and all behaviour. Most of these values are completely subconscious, and therefore very hard to identify. Together they shape our workplace and our work habits. This also goes the other way; work habits that serve a purpose (though not necessarily productive or effective) may feed back into organizational culture and influence values, espoused values or artifacts. In a sense, organizational culture can be considered as an ongoing sediment of work habits that 'worked' in the past, for one reason or another. Organizational culture is like a shoreline, continuously changed by the sand deposited and displaced by the waves. Some examples of core values, espoused values and behavior are shown in the table below:

Core value

Espoused value

Behavior

Failure is (always) bad

Failure is not an option

Shifting blame. Untransparant communication. Less risk-taking.

Knowledge is power

Use of scientific method

Intellectual work is rewarded. Education level determines salary.

Women and men are not equal

This is a man’s work

Glass ceiling for women. Masculine jokes.

Learning avoids future mistakes

When we make a mistake, we reflect on it

Critical incident analysis. Open feedback.

Don’t trust others

Believe in individuals over teams

Little focus on team work. Individual rewards.

Employees are incapable of making company-serving decisions

Strong management

Top down control. Hierarchical structure.

It is important to emphasize the continuous feedback between behaviour, espoused values and core values. If organizations punish individuals for their mistakes, it may be beneficial for employees to become less forthcoming with mistakes. Because fewer failures are reported, organizations may believe that punishing actually works. The core value is reinforced by the behaviour it causes. Although the behaviour is hardly productive, it does protect employees. This feedback loop implies that most behaviour in organizations served a purpose at some point in time, even if it was wholly unproductive, wasteful or ineffective. In a sense, the behavior has become entrenched. The difficulty lies in the fact that most people are hardly ever aware of what influences their behaviour in the workplace and how it came to be. 

And this is the root case of organizational inertia. Every organizational change attempts to disrupt this historically bound system of values and behaviours to some degree. No matter how well-intended the change is, if change is forced upon an organization and its employees without appreciating this historical system, it will be resisted. Senge (1990) observed correctly that 'people don't change, they resist being changed'. This resistance can be overt (e.g. protest, sabotage, unwillingness to cooperate) or passive (reverting to old habits, skepticism). Either way, change is not sustained. Worse, it may actually increase organizational inertia as people become disillusioned with change ('here we go again' or ‘these changes never work out, so why bother?’).

Which values are important for Scrum?

Although Scrum is only a framework for iteratively developing software, its approach and artifacts are based on a number of core values. If these core values are not aligned with organisational culture, resistance will certainly follow or the change programme will not achieve its goals. Below are some of the core values that I believe are prevalent in Scrum (and the 'perfect Scrum company'):

  • By discussing mistakes openly, we can learn from them;
  • Honest en constructive feedback is necessary for growth;
  • Growth (both personally and as a team) is important;
  • Professionals are capable of manageing their own work;
  • Teams are capable of manageing their own work;
  • Frequent communication is necessary for success;
  • Involvement of the customer is necessary;
  • Frequently delivered imperfect solutions are better than seldom delivered perfect solutions;
  • The customer is leading in deciding what to build;
  • The customer is expert in his/her domain;
  • Our job is to deliver business value to our customers;
  • Everyone can learn to perform new tasks and jobs;
  • Teams (and individuals) are intrinsically motivated to do a good job (and can thus be trusted);
  • As a team, we are more effective than as individuals;
  • Frequent change is a reality of software development;
  • The best decisions are those we make together, democratically;
  • Software development is very complex and requires an iterative approach;

At this very moment, a research project by Scrum.org trainer Dominik Maximini is aiming to identify core values that describe the 'perfect Scrum company'. For those interested in participating, the online survey can be found here.

Approaches to implementing Scrum

And this brings us to an important question; how can we successfully implement Scrum (or any other change programme) while taking organizational culture into account? Obviously, this is very difficult. It requires careful attention to both the behaviour that has to be changed and the organizational culture in which that and current behaviour is (to) be embedded.  

There are two broad categories of approaches on how to implement Scrum. The first category applies ‘Shock Therapy’ to teams or organizations by very rapidly and forcibly implementing Scrum (Sutherland, et. al., 2009). Although successes have been reported with this method (ibid), there is no data on the sustainability of these changes that I am aware of. It may be as Kotter (1995) noted in his review of hundreds of organizational change programmes: “The most general lesson to be learned from the more successful cases, is that the change process goes through a series of phases … that require a considerable length of time. Skipping steps creates only the illusion of speed and never produces a satisfying result”. In a recent post, I outlined the issues with this approach if we consider it from the perspective of organizational culture. In short: by forcing change, any effects will probably not sustain because employees don't truly buy-in.

A second category is more gradual. This includes ad-hoc implementations of Scrum that slowly grow towards maturity, but often without a conscious effort. But it also includes more guided change models, such as the Agility Path framework fromScrum.org. Agility Path strives to implement practices associated with Scrum in a stepwise fashion, while continuously monitoring important metrics. This approach is closely related to what is called the ‘emergent change approach’. This humanistic approach to organizational change is based on insights from systems theory, action research and organizational development (OD) and calls for organizational change programmes that heavily involve employees in order to incrementally change towards a common goal. For the sake of readability and length, I will present this approach in the second part of this post (which will be coming shortly).

References 

Balogun, J. & Hope Hailey, V. (2004) Exploring Strategic Change (2nd. ed.), London: Prentice Hall;

Deal T. E. & Kennedy, A. A. (1982). Corporate Cultures: The Rites and Rituals of Corporate Life, Harmondsworth, Penguin Books;

Fine, D., Hansen, M. A., & Roggenhofer, S. (2008). From Lean to Lasting: Making Operational Improvements Stick. The McKinsey Quarterly, November;

Jørgensen, H. H., Owen, L., & Neus, A. (2008). Making Change Work, IBM Global Services;

Kotter (1995), Why Transformation Efforts Fail. Harvard Business Review, March-April;

Loveday, L. (1984). Change strategies and the use of OD consultants: Part 1, Leadership and Organizational Development Journal 5(2): 3-5;

Schein, E. M. (2004). Organizational culture and leadership (3rd. ed.). Jossy-Bass;

Senge, P. M. (1990). The fifth discipline: The art & practice of the learning organization. New York: Doubleday;

Smith, M. E. (2002). Success rates for different types of organizational change, Performance Improvements, 41-1, pp. 26-33, January;

Spector, B., & Beer, M. (1994). Beyond TQM programs. Journal of Organizational Change Management, 7(2), pp. 63-70; 

Sutherland J., Downey, S. & Granvik, B. (2009), Shock Therapy: A bootstrap for hyper-productive Scrum, in Agile Conference, 2009. AGILE ‘09, August 24-28, 2009, Chicago (IL);

Christiaan Verwijs
Christiaan Verwijs

Scrum Master, Trainer, Developer & founder of Agilistic