A participative action research approach for implementing sustainable Scrum

Tags: Agile Team Scrum

In this article I present a participative action research approach for implementing Scrum. I have developed this approach during my search for an implementation strategy that fosters and encourages a democratic, participative change philosophy without losing focus and becoming too pragmatic. The approach is based on insights from action research, organizational development and appreciative inquiry and assumes that by involving employees in a focused manner, change will be more sustainable, effective, ethical and hopefully also more fun.

During the past few years, I’ve been in the fortunate position to assist and coach several organizations in their quest to become Agile and to improve the quality of their software development process. This includes the company where I work, NowOnline, but also companies where I’ve been hired to provide training and coaching.

My experience is that implementing Scrum is difficult and complicated. Initially, I assumed that Scrum could be sold to teams and management simply by emphasizing the benefits. I  assumed that my story, my enthusiasm and my examples would convince everyone to give it a try and go all the way (100% Scrum), right away. The reality was quite different.

Although I usually found some support, I also ran into resistance. Some of it was overt (skeptical responses, criticism) and some of it was passive (no desire for change, not attending meetings). In any case, Scrum implementations progressed at a slower rate than I would’ve liked. Teams often cherry-picked the parts that they liked, while ignoring or abandoning other parts. Call me naive, but this confused the hell out of me. I really wasn’t sure what the best strategy was here, judging from a change management perspective. Is it best to force change and make a team apply the Scrum framework to the letter? As my initial strategy, I found that this actually increased resistance, reduced effectiveness and polarized conflicts within teams between proponents and opponents. I found that a more patient, pragmatic and democratic approach was more successful. Following Peter Senge’s (1990) observation that ‘people don’t resist change, they resist being changed’, I allowed teams to pave their own road, while giving them advice where I could. If they wanted to cherry-pick some parts of Scrum, I would perhaps object, but it was ultimately their decision. But there was no clear vision of the future, so this approach can easily lead to a half-assed Scrum implementation that fails to achieve the expected potential. And if that doesn’t happen; the method is often blamed (and discarded, never to be tried again).

Giving teams this level of control over the change is difficult for two reasons. First, teams often choose to ignore certain parts of Scrum because they don’t see the need (yet). For example, most teams do not (initially) understand why the sprint burndown is important. So if it is their choice, they ignore it. Other teams prefer technical issues instead of functional requirements on the backlog, because that’s what they’re used to. Sometimes, daily standups are ignored because everyone considers meetings wasteful. Or the definition of ‘potentially shippable code’ is stretched to allow showing buggy code to a product owner with the promise of improving it later. And this is where the second reason becomes apparent; implementing Scrum is really like any other organizational change. And like any other organizational change, the primary source of resistance and inertia lies in organizational culture (see another post). And changing organizational culture is very difficult if you are a part of that very culture. Because teams are so accustomed to “the way we do things here” (organizational culture), they may be unable to see alternatives or seriously consider them. This is why having someone from the outside (like a coach) is so important. And this is also why this coach really does need to have considerable control over the change. He or she has to “shake” everyone and everything up.

I found myself wondering if there is a middle road here. A forceful approach where control resides with the coach obviously doesn’t work very well, but neither does an approach where control is mostly with the team. I realized that implementing Scrum is like any other organizational change. And I was aware of the existence of effective change models that are based on similar questions. After reading up on these models, I drafted a crude approach that (for me) combines the best of both worlds: a participative approach with a clear sense of direction and guidance. I will present this approach below. But I will begin with a bit of theory.

Organizational change, complexity and incremental change

The Action Research model, coined by Lewin (1951), provides us with a useful framework for organizational change. According to this model, organizational change is a form of complex problem solving and thus requires a reflective process with very frequent opportunities for feedback to identify new obstacles and avenues for change. Lewin (ibid) assumed that organizational behavior of any kind is the result of a more or less stable ‘status quo’ between forces that are driving for change and forces that are resisting. These forces can be political, technological, social, ethical or philosophical. The point here is that any form of organizational change (like implementing Scrum or becoming more Agile) requires careful attention to these forces.

Lewin (ibid) states that in order to initiate change, the status quo must be disrupted (unfrozen). This means that either resisting forces have to be decreased, driving forces have to be increased or the direction of forces has to be changed. Involving employees and teams is crucial here; when people are involved in decisions that affect their work, they are more likely to adopt new ways. Another way to generate energy for change is by collecting objective data to diagnose the current situation and objectify reasons for change.

Once sufficient energy for change has been generated, organizations can move towards a desired state, after which new work habits become part of the ‘current way of working’ (frozen). This is necessarily an incremental process, where every change is a small step towards some envisioned goal. Organizational change is a complex endeavor, so it will involve many steps as the interplay of forces continuously changes. It is no surprise that Action Research, like Scrum, has it’s roots in systems (feedback) theory.

Lewin’s three steps for change, iteratively applied

Action Research is applied in the field of Organizational Development (OD), which is a humanistic approach (Huse, 1982, Porras & Robertson, 1992) that attempts to improve individual and organizational effectiveness through the use of scientific principles and practices (French & Bell, 1973). In OD, Lewin’s model has been further refined:

  • Unfreezing: In the first stage, the organization is diagnosed and (preferably objective) data is collected with regards to a desired goal. This data is shared and discussed with the target of the change (the employees) and collectively translated into a short term joint action plan that solves an immediate problem (and is often part of a larger problem);
  • Move: The joint action plan is put into effect, perhaps through workshops, training or on-the-job coaching;
  • Freeze: If the transformation was successful, behavior and values have changed and embedded themselves in organizational culture. Data is again gathered and the process is repeated until the desired end state is achieved;

Organizational development provides us with a useful approach for implementing Scrum because it it calls for the participative, democratic approach that I value yet provides a basic framework for change and a goal to move towards (incrementally). As such, this approach is very suited for sustained, organizational change of the existing workforce. It is not suited for very rapid change (McLean, 2005) that is often accompanied by more drastic measures (firing people that don’t fit with the change, downsizing). Although I know of no implementation of Scrum that uses this approach (although I assume that many implementations go like this intuitively), I have attempted to translate OD principles into a more formalized method below.

Phase 1: Dreaming: defining a vision for the future

Organizational change without vision is a recipe for failure (Kotter, 1995). Simply put, a vision should present a desirable (and realistic) future workplace that makes it clear what the benefits of the change are and should act as an anchor. It answers questions like ‘How will we work together in the future?’, ‘How do customers perceive us?’ and ‘What makes us proud of our work?’. In our case, the vision for the future can describe a workplace where a team is working at the peak of it’s ability.

Vision can be formulated by analyzing the problems and identifying a scenario in which those problems have been resolved. The analysis of problems, however, tends to result in an ever increasing palette of deeper problems. Instead, applying parts of a process called ‘appreciative inquiry’ is more constructive. In appreciative inquiry (Cooperrider & Whitney, 2001) the focus is shifted from problem-analysis to collectively discussing and imagining a future of what might and should be given what’s already present.

Applied to implementing Scrum, appreciative inquiry can be used to help a team imagine a realistic future scenario in which they work at the best of their ability and are effective and efficient as the team that they already are. Given this future, the team can identify why they will be successful in that future:

Envisioning (or dreaming up) a bright future for the team

This exercise generates creativity, positive energy and a desire for change. It also sets some goalposts for change, but does not too narrowly define what will be changed and how. As with complex software development, a plan that is too detailed is bound to fail. But most importantly, by identifying reasons for their future success, a team is essentially identifying avenues (or opportunities) for improvement in the now. 

Phase 2: Design: Where does Scrum fit in?

Because we’re implementing Scrum, it is important for a team to determine how they can benefit from the various Scrum practices and concepts. Instead of telling a team to follow the framework, it’s more constructive to help them understand why Scrum can work for them. After all, Scrum has been developed to make teams more efficient and remove bottlenecks that most software development teams experience. If teams feel that they are participating in and designing the change, it will help cement the changes more firmly.

A useful exercise is to map with a team how Scrum practices and concepts can contribute to the identified avenues for improvement:

Mapping Scrum to opportunities for change (the identified opportunities are on the left, Scrum practices are on the right)

Take a team that wishes to increase involvement of their customer to help align expectations. Several Scrum practices are useful here. The customer can appoint a product owner that becomes part of the Scrum cycle. Doing a joint sprint planning and review sessions is also useful. Setting up and sharing a product backlog may help the customer to get a sense of what the team is working on.

This exercise is useful because it shifts attention away from resisting change to identifying benefits of change. It creates a roadmap for change. This approach is also more likely to generate questions as to the purpose of certain Scrum practices or concepts. If the ‘Daily standup’ is not mapped to any of the opportunities on the left, a coach can ask a team to imagine what a daily standup may be useful for. Practices that remain unmapped, even after repeated attempts, apparently do appear beneficial to the team (yet). Forcing teams to adopt these practices will certainly result in resistance. A more useful approach is to revisit this board frequently throughout the change process to determine if the team’s perspective has changed. By experimenting with Scrum, they may start seeing the benefits of a Daily Scrum at a future point in time.

Phase 3: Unfreezing: Setting up a joint action plan

Trying to change everything at once is not a good idea and should be cautioned against. Instead, it’s better to think of organizational change as a series of experiments with new behavior and practices in order to move towards the envisioned future. Change that is too rapid and too all-encompassing upsets organizational culture and causes resistance. It is also too costly when the change fails altogether. This is where action research comes in. By approaching organizational change as a series of experiments, we allow ourselves to check our assumptions and 'fail as early as possible'. Essentially, every experiment is focused on testing a number of assumptions (or hypothesis) as to what will bring a team closer to the envisioned future. 

In action research, teams actively participate in the change. During the first stage (unfreezing), the team sets up a joint action plan that describes what will be experimented with during the upcoming iteration. In the first session, the team decides on the length for every experiment-cycle. Depending on the time and the self-confidence of the team, the length can be adjusted from one week to one month or more. The action plan requires three questions to be answered:

What practices are we experimenting with as a team?

For the action plan, teams select a few Scrum practices that are most likely to move them towards their envisioned future vision. They can either cherry-pick these practices or decide to tackle one of the identified opportunities for improvement. So, if a team wishes to involve customers in the process, they may decide to help the customer appoint a product owner, set up a shared product backlog and do sprint planning and review sessions with the customer. If a team wants to improve inter-team communication, they can start doing daily standups and set up a digital or analogue) scrum board.

How are we implementing the practices as a team?

Once a basic action plan has been defined, the individual actions can be translated into concrete steps. If a team is going to do daily standups, they need to decide when and where this meeting is going to be held and what happens when someone is late. In other cases, teams may require additional training. If, for example, a team wishes to start writing user stories to capture functionality, but they have little experience, a good user story workshop is a good idea. Or a whiteboard and post its have to be purchased if teams wish to set up a scrum board (or a digital equivalent, like GreenHopper or Rally).

How are we going to determine if the experiment is successful as a team?

Gathering (more or less) objective data is important to help a team determine the success of the experiment. This data can be collected throughout the sprint or during the experiment review, with surveys, metrics, interviews or other measurements:

Identifying useful metrics to measure the success of an experiment

The result of this phase is not a formal document with 15 signatures, but a shared understanding of what the team is going to experiment with, how to perform the experiment and how to ascertain it’s success.

Phase 4: Transition (moving)

The action plan is implemented. This requires coaching from an Agile coach who can help the team stay focused. Because a team is obviously doing real work during the experiment, a coach should help the team to implement the action plan without affecting their current work too much. The coach can also help the team collect the data they require to determine the success of the change at the end of the iteration.

Phase 5: Review the experiment (Learn)

At the end of the iteration, the team reconvenes to share experiences and evaluate the success of the experiment. They do this with the data collected during the sprint. This is preferably done during a retrospective. Elementary questions are: “What worked?” and “What can be improved?”. 

Based on the retrospective, the envisioned future (phase 1) and the mapping of Scrum practices to opportunities (phase 2) can be reviewed and updated. An Agile coach should be on hand to counsel teams where necessary. When the review is done, and there is still ground to cover with respect to the envisioned future, the cycle repeats.

How fast can Scrum be implemented with this approach?

Scrum can be implemented rapidly with this approach. I would say that a few months is possible. It all depends on the level of understanding in the team as to why change is necessary and how Scrum can help and the time a team has to experiment with new practices. If there is resistance (in or outside the team), the change may require more time. I don’t think this approach is useful when very rapid change is necessary, for example to avert imminent financial problems or loss of customers. If Scrum has to be implemented in a manner of weeks, it may be more beneficial to apply a shock therapy approach. I do have serious doubts as to the sustainability and ethics of very rapid organizational change, so I’m certainly no fan of this. I think it’s always best to involve employees closely in these kinds of changes, instead of forcing a new way of working.

Concluding thoughts

In this article I have presented a participative action research approach for implementing Scrum. I have developed this approach during my search for an implementation strategy that fosters and encourages a democratic, participative change philosophy without losing focus and becoming too pragmatic. The approach is based on insights from the fields of action research, organizational development and appreciative inquiry. Like all organizational change, employee participation is key to success. Change is more likely to succeed if people can participate in the decisions that affect their work. The key principle of the approach presented is that teams should design their own incremental implementation strategy for Scrum, based on a bright envisioned future team. This humanistic approach assumes that by involving employees, change will be more sustainable, more effective and more ethical. Another important principle is that organizational change (like implementing Scrum) is a complex endeavour and requires an approach that provides frequent feedback to allow us to 'fail as early as possible'. This avoids costly time-consuming mistakes.

And this, to me, seems to fit very well with the principles of Scrum itself. I certainly am going to continue applying this approach and I am very much looking forward to hear success stories (or failure stories, which are equally educational).

The participative action research model in one picture

References

Cooperrider, D.L. & Whitney, D (2001). A positive revolution in change. In Cooperrider, D. L. Sorenson, P., Whitney, D. & Yeager, T. (eds.) Appreciative Inquiry: An Emerging Direction for Organization Development (9-29). Champaign, IL: Stipes;

French W. L. & Cecil Bell, C. (1973). Organization development: behavioral science interventions for organization improvement. Englewood Cliffs, N.J.: Prentice-Hall. p. 18;

Huse, E. F. (1982). Management. St Paul: West Publishers;

Kotter (1995), Why Transformation Efforts Fail. Harvard Business Review, March-April;

Lewin, K. (1951). Field Theory in Social Science. London: Harper Row;

Porras, J. L., Robertson, P. J. (1992). Organizational Development: Theory, Practice, and Research. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of Industrial and Organizational Psychology. (2nd ed., Vol. 3), Palo Alto, California: Consulting Psychologists Press.

McLean, G. N. (2005). Organization Development Principles, Processes, Performance, San Francisco: Berrett-Koehler Publishers;

Senge, P. M. (1990). The fifth discipline: The art & practice of the learning organization. New York: Doubleday;

Christiaan Verwijs
Christiaan Verwijs

Scrum Master, Trainer, Developer & founder of Agilistic