Why I don't believe in Evidence-Based Management (2 of 6): Origins and EBM applied to software development

Tags: Agile

Evidence-Based Management (EBM) is an emerging movement that aims to improve the quality of decision-making by urging managers to use 'the best available evidence' to support their decisions. Recently, Scrum.org has picked up on EBM and is now actively promoting it as the next logical step in improving organizational agility within the context of software development. This series discusses a number of objections that I have against EBM. In my previous post, I introduced the series and summarized my objections. In this post, I will clarify what Evidence-Based Management is and how it is being applied to software development.

Evidence-Based Management urges managers to improve the quality of their decisions by seeking out “the best available evidence” to support their decisions (Pfeffer & Sutton, 2006). This requires that managers classify the quality of evidence for compelling claims and seek out the best sources of evidence. Preferably, this evidence should be objective, like scientific or local research, instead of subjective like personal experience, best practices, management hypes and uncontrolled research.

The origins of Evidence-Based Management

Evidence-Based Management has its roots in Evidence-Based Medicine. This movement was born in the early 1990’s (Sackett et .al. 1996) out of the observation that many medical practitioners delivered health care in ignorance of published research. This meant that patients didn’t benefit from new insights and (often) kept getting outdated, pointless or even harmful medical treatment. This is why, in Evidence-Based Medicine, research evidence from a variety of fields is evaluated and included in the diagnosis and management of individual patients (Greenhalgh, 2010). Because not all evidence is of equal quality, evidence should be ranked according to one of the available taxonomies (such as GRADE, Guyatt, 2010). For example, evidence that is derived from randomized, controlled design experiments (RCD’s) is of much higher quality than non-random, uncontrolled, case- or observational studies. So, when making decisions about treatment and diagnosis, a practitioner should make use of the vast body of knowledge available in the scientific fields of medicine, biology and chemistry instead of trusting solely on their experience, intuition or peer opinions. This requires significant education and training, as emphasized in the Sicily statement of Evidence-Based Practice (Dawes et al. 2005).

Similar Evidence-Based Practices are more recently being applied in management (e.g. Pfeffer & Sutton, 2006). Proponents argue that managers mostly base decisions on intuitions, hearsay, expertise and gut feeling. This makes decisions susceptible to ‘snake-oil solutions’, management hypes and ‘best practices’. And since being a manager is not a profession, there are no ethical guidelines, no requirements in terms of skills and scientific training (Rousseau, 2007). For example, only 3.9% of dutch HR professionals even read scientific literature or journals (Sanders, Riemsdijk & Groen, 2008). So how can they even know what the best way is to motivate employees, how successful downsizing is, or if the MBTI still holds up to scientific scrutiny? That is why proponents of EBM argue that managers should make better use of scientific research done in the fields of the social and political sciences (Rousseau, 2007; 2005). In practical terms, this means that managers should learn how to translate a practical issue into a testable question, systematically search and retrieve evidence, judge the trustworthiness and relevance of the evidence, incorporate the evidence in the decision-making process and evaluate the outcome (CEBMa, 2013). If scientific evidence (‘E’ evidence) is lacking, local evidence can be gathered in one’s organization instead, even though this evidence is considered weak (‘e’ evidence) (Rousseau, 2005).

Evidence-Based Management in software development

Recently, principles of EBM are being applied to software development. The assumption is that software development is rarely measured based on outcomes (Schwaber, 2014; Scrum.org, 2014b). If outcomes, like the value delivered, would be measured and correlated with practices in use, we would be able to identify which practices work and which don’t (ibid). This prevents the abundance of ‘silver bullet’ solutions, just-in-time experts and the latest management hypes from influencing decisions. Evidence can support managers in making decisions about questions like ‘Should we adopt pair-coding practices?’, ‘Can we reach more customers with functionality X or Y’? ‘What is the optimal team size?’, ‘Should we invest in this or that training?’, ‘What is the optimal length of our sprints?’, ‘Which architecture should we choose?’, ‘Should we outsource testing or development of software?’.

This is why Scrum.org is now actively promoting the adoption of Evidence-Based Management for decisions concerning software development (Schwaber, 2014; Scrum.org, 2014a, 2014b). The goal is to elevate ‘conversations from preferences and opinions to logic and insight’ (Scrum.org, 2014b) through ‘more rational, fact-based decisions’. Scrum.org makes a strong distinction between so-called ‘direct’ and ‘circumstantial’ evidence, and touts scientific practices and empiricism as enablers. The former is (supposedly) irrefutable and based on objective facts, while the latter is subjective and refutable (Scrum.org, 2014a). A good decision is based on the ‘best available evidence’, which obviously should include as much direct evidence as possible, supported by available circumstantial evidence. A decision that is solely based on circumstantial evidence isn’t a ‘fact-based’ decision, and presumably of lower quality. But how does one apply these principles in an actual organization? 

To help managers apply EBM, Scrum.org has introduced a framework to apply EBM. Key part of this framework is the 'Agility Index'. This index is a score that represents ‘a single organization’s health in overall agility’ (Scrum.org, 2014b) and is composed of metric scores in three key areas (‘Key Value Areas’, KVA): ‘Time to Market’, ‘Current Value’ and ‘Ability to Innovate’. A set of eleven metrics (‘Key Value Measures’, KVM) is proposed to measure these three areas that are considered to be unambiguous (Scrum.org, 2014b) sources of objective data. ‘Time to Market’ is measured through release frequency, release stabilization and cycle time. ‘Ability to Innovate’ is measured through the installed version index, the usage index, innovation rate and defects. Finally, ‘Current Value’ is measured through revenue per employee, product cost ration, employee satisfaction and customer satisfaction.

By measuring these metrics (or a subset), it is possible to produce charts and pictures that can be presented as ‘snapshots’ of organizational agility:

Image courtesy of Scrum.org (2014b)

Scrum.org suggests that managers use the metrics to periodically identify ‘Key Value Areas’ that should be improved (preferably one at a time), and conduct experiments to see what works. These experiments generally consist of applying new or different practices, like ‘pair coding’, ‘test-driven development’ or Scrum. Scrum.org maintains a list of several hundreds of these practices that can be readily applied. By periodically reviewing the metrics within a so-called ‘learning loop’, the effect of experimental practices can be evaluated and decisions can be made on how to proceed. 

This certainly sounds promising, particularly because managers are provided with a practical framework to apply EBM within their organizations. I applaud Scrum.org for pushing people making management-decisions to increase the quality of decision-making by using more objective data. But I am highly skeptical as well. My gripe is not with the general approach of continuous improvements based on information from a variety of sources. But there are several objections that can be raised against this particular implementation and EBM in general, which I will discuss in upcoming posts. However, there are two observations that I would to share here. The first is that this implementation focuses exclusively on local evidence (from the own organization), while EBM focuses primarily on the use of evidence from the body of scientific literature (Rousseau, 2005). Scrum.org (2014a) supports this decision with the assumption that ‘there is no widely spread and accepted external evidence on the effectiveness of organizational practices in software organizations’, but provides no evidence to support this rather unlikely claim. The second observation is that Scrum.org does not provide evidence for the effectiveness of its own framework, the metrics used, their unambiguousness and the suggested practices other than expert writings. Without such objective evidence, how is this framework any different from the latest management hype? Shouldn't the EBM-aware manager respond to this framework with a healthy dose of skepticism?

In my next post I will explore the epistemological issues that plague EBM in general. I will argue that the distinction between ‘direct’ and ‘circumstantial’ evidence is highly problematic. Even in the most tightly controlled scientific studies in the social sciences it’s impossible to achieve ‘irrefutable’ or ‘unambiguous’ proof of an assertion, let alone with local research in one’s own organisation. And if there is no such thing as ‘irrefutable’ proof or strong evidence, what value does EBM offer in the first place?

References

Center for Evidence-Based Management (CEBMa) (2013). A definition of evidence-based management. Retrieved August 8, 2014 from http://www.cebma.org/a-definition-of-evidence-based-management;

Dawes, M., Summerskill. W., Glasziou, P., Cartabellotta, A., Martin, J., Hopayian, K., Porzsolt., F, Burls, A. & Osborne, J. (2005). Sicily statement on evidence-based practice. BMC Medical Education, 5(1). Retrieved August 8, 2014 from http://www.biomedcentral.com/1472-6920/5/1;

Greenhalgh, T. (2010). How to read a paper: the basic of evidence-based medicine. John-Wiley & Sons, No 19, 201;

Guyatt, G. H., Oxman, A. D., Holger, J., Schünemann, P. T. & Knottnerus, A. (2010). GRADE Guidelines: A new series of articles in the Journal of Clinical Epidemiology, Journal of clinical epidemiology, Vol. 64, Nr. 4, pp. 380-382;

Pfeffer, J. & Sutton, R. I. (2006). Hard Facts, Dangerous Half-Trutsh and Total Nonsense: Profiting from Evidence-Based Medicine. Boston, MA: Harvard Business School Press.

Rousseau, D. M. (2005). Is there such a thing as ‘Evidence Based Management?’. Academy of Management Review, Vol. 31, No. 2, pp. 256–269;

Rousseau, D. M. (2007). Educating managers from an evidence-based perspective. Academy of Management Learning & Education, Vol. 6, No. 1, pp. 84-101;

Sackett, D. L., Rosenberg, W. M., Gray, J.A., Haynes, R. B. & Richardson, W. S. (1996). Evidence Based Medicine: What is is and what it isn’t. BMJ, 312, pp. 71-71;

Sanders, K., Riemsdijk, M. & Groena, B. (2008). The gap between research and practice: a replication study of the HR professionals’ beliefs about effective human resource practices. International Journal of Human Resource Management, Vol.  19, No. 10;

Scrum.org (2014a). Empirical management explored: Evidence-Based Management for Software Organizations. Retrieved August 9 from https://www.scrum.org/Portals/0/Documents/Community%20Work/Empirical-Management-Explored.pdf

Scrum.org (2014b). Evidence Based Management Guide. Retrieved September 6 from http://www.ebmgt.org/portals/agilitypath/Documents/EBMgtGuidev1_CWT.pdf;

Schwaber, K. (2014), Evidence-Based Management. Retrieved August 8, 2014 from http://kenschwaber.wordpress.com/2014/02/16/evidence-based-management;

Christiaan Verwijs
Christiaan Verwijs

Scrum Master, Trainer, Developer & founder of Agilistic