Software development projects often fail because they ignore the human factor. Our software attempts to replace human-based decision-making in business processes with (strict) software-based decision-making under the assumption that this will increase quality, reduce costs and prevent mistakes. Although this may work in theory, the reality of work is often very different. Because it is not longer possible to 'bend the rules', users don't feel supported by the software but rather have to 'fight' it. As a result, we shouldn't be surprised about resistance against the software. In this post, I will argue that we're better off learning more about human-based decision-making, and writing software that guides users, rather than forcing them.
At NowOnline, we frequently have these kinds of conversations with our customers when we are talking about a potential project:
Me: "So, what problem are you running into in your business?"
Customer: "We need to automate process X. Employees make a lot of mistakes, lose money and waste time."
Me: "So, what is your proposed solution?"
Customer: "We need to enforce process X in a software tool, so that mistakes are avoided and money and time are not wasted."
Me: "Why are your employees not following process X right now?"
Customer: "Because they lack discipline."
I understand where this customer is coming from. There is a process that is important to the business. It involves a number of decisions based on information (my definition of a 'business process' within the context of software development). Mistakes are being made, money is lost and time is wasted. So things need to be improved. At first glance, the best approach is to automate this procedure, enforce the relevant business rules and train employees to use the software. Instead of manually performing the procedure, the software now takes care of that. Employees become responsible for entering data, and triggering decisions, while the software makes the actual decisions and enforces the business rules.
But is this really going to solve the customer's problem? I seriously doubt it. Although the solution looks great on paper, it rarely works well in practice. You'll probably end up with software that works according to procedures and protocols, but is also very user-unfriendly. It will restrict the flexibility that users had before. The 'exceptions to the rule' that could be dealt with by 'bending the process', have now become problematic. And if users cannot deal with these exceptions, they will probably either mis-use the software (with creative work-arounds) or resist against using the software in general. Which is all very understandeable.
As software developers and analists, we sometimes believe that the purpose of software development is to automate business processes in order to eliminate human factors (intuition, emotion, politics, principles, etc). We assume that by doing so, we help our customers improve their businesses. If a business process works according to the 'rules', they will be more efficient, quality will improve and the amount of time will be reduced. This is probably also why our customers hire us in the first place. They also believe that decision-making should be devoid by human factors as much as possible.
Although this approach is very common, I strongly believe that it is one of the key reasons why many software implementations fail miserably. Although these projects do a wonderful job of clarifying and enforcing a business process, they do a lousy job of supporting and helping employees actually do their job. So in the end, they fail because they don't match up with the day-to-day reality of work.
Some real-life examples
Let me give you some examples to illustrate this mis-match, taken from projects that we did for NowOnline:
- In the past, we have developed software to generate contracts for employees (the business process). For important internal reasons, the start date of a contract could not be set before today. This made sense from a business perspective and was agreed on by all parties. But after going live, our support desk began to notice a sharp increase in support requests to 'change the start date'. Obviously, it was necessary to set the start date before today at least a few times a week. Although our software worked according to procedure, the procedure obviously required more flexibility. The interesting thing is that everyone actually agreed that the procedure should be this strict, even though exceptions occured frequently and resulted in frustrations with the end users, delays and a heavy load on our support desk;
- One of our customers asked us to develop an environment to track the status of project proposals (the business process). A number of business rules governed the decisions on which status was allowed for a project. For example; a project proposal could not be considered 'accepted' when it had not gone through a strict 'testing' phase. After we roled out the software, we quickly learned that employees ran into situations where this was actually useful for a variety of reasons, even though it was against procedure. When the employees had to follow procedure, using the software became tedious and frustrating. Users obviously resisted. We eventually removed the business rule;
- We recently developed an application that asked users to enter valid phonenumbers. Our customer wanted to make sure that the number was valid, so we added validation logic to block invalid numbers. We quickly learned that users did not like this validation. Instead of helping them, it was actively blocking their work by preventing exotic phonenumbers (different notations, international numbers, voip numbers, internal numbers, etc). We initially tried to expand the logic, but we quickly learned that it was easier to just drop the validation altogether. This did not result in an increased number of errors;
- At NowOnline, we use JIRA's greenhopper as our digital Scrum Board. JIRA is quite extensive, and allows you to set up complicated workflows for items on the backlog. Initially, this is exactly what we did. We devised a 'best practice' workflow, implemented and enforced it. We added sensible rules such as 'An item can only go to Done when it has been Tested' or 'An item can't go back to the backlog if it's in Done'. Although this looked great on paper, it really didn't work. Throughout the sprints, we often ran into exceptions to these rules. We knew the rules, but sometimes had to 'bend' them a little bit to speed up our work. But JIRA didn't allow this, so we often had to perform quite a number of clicks. We ended up removing the rules enforcing the workflow;
Now, one could argue that users should simply follow procedure, even if they don't like it. The procedures and business rules are there for a good reason, after all. They are intended to prevent illegal scenarios, avoid mistakes and improve quality. That's a fair point. But these examples are intended to examplify the mis-match that often exists between the theory and the practice of day-to-day work. Even before software entered the workplace, this was already the case. We all knew the required procedures, but we also knew that we sometimes had to bend them a little for the 'exceptions to the rule'.
The root of the problem: assumptions about rationality
The key problem, to me, seems to be our assumption that business processes can and should be governed by strict business rules in order to allow a business to operate at the peak of it's performance. A business process is considered of high quality when it is devoid of human factors such as intuition, politics, emotion, principles and maybe even ethics. In other words; we assume that a business process should be completely rational. It should be a logical sequence of causes and effects (or IF-THEN's), based on the information at hand. We also assume that we, ourselves, can understand these processes from a purely rational perspective. By analyzing the process and by applying a reductionist approach, we can identify increasingly smaller steps, decisions and pieces of information, which can then be automated. This essentially allows us to write software that turns users into 'data entry drones' and makes decisions based on the provided information for them. I believe this is a fallacy.
Businesses are made up of people that make decisions. Economists and psychologists have long since known that human decision-making is far from rational. Instead, humans rely on quick heuristics to make decisions based on partial information (Simon, 1991), like 'gut feeling' or 'majority vs minority'. One could argue that this is undesirable because it can result in abuse, errors and mistakes. Although this may be true to some extent, I consider this a very cynical view of human decision-making. Businesses have been succesful before the advent of software, when all decision-making was necessarily done by humans. Thankfully, I am not alone in this. Several authors have pointed out the quality, speed and effectiveness of human decision-making, such as Gigerenzer (1999, 2007), Dane, Rockman & Pratt (2012) and Hyppänen (2013) and many . My point here is that decision-making in businesses is not rational because they are made up of humans. If current decision-making processes are not rational, how we can assume to understand them from a rational perspective? We will just end up writing a perfect procedure on paper that doesn't fit well with the day-to-day reality of work.
For me, the core of the problem is this: we often write software that is intended to replace human-based decision-making with software-based decision-making. But by doing so, we ignore a lot of factors in that decision-making that are a reality of work that can't be ignored, such a politics, ethics and intuition. We end up writing perfect software for a perfect process, but only on paper. We forget that this software has to be used by humans in the unpredictable reality of the workplace. And so, we act surprised when users resist the software or find creative work-arounds that we did not intend. So I ask again; do the users have to be trained to fit with the software, or should the software be adapted to fit with the users? I am strongly in favor of the latter.
A solution: software to guide and support human-based decisions
As software developers and analists, we should learn more about humans and how they go about their work. Our software should not limit the creativity and resourcefulness of users by making decisions for them. We should not be too skeptical about the ability of humans to do their work effectively and make high-quality decisions. Instead, we should learn how we can support them in their own (heuristical) decision-making. Our software should inform users about the process and the rules, but it should not enforce them too strongly. Essentially, our software should be more about guiding, coaching and educating, rather than about removing the 'human element'. And maybe, this will more often result in the conclusion that software may not actually be the best solution for the problem at hand. But more on that in a future blog!
I can easily summarize this post with one of the Agile principles:
(Software should be) about humans and interactions, and less about processes and tools.
Dane, E., Rockmann, K. W., & Pratt, M. G. (2012). When should I trust my gut? Linking domain expertise to intuitive decision-making effectiveness. Organizational Behavior and Human Decision Processes, 119-2, pp. 187-194;__
Gigerenzer, G. & Todd, P. M. (1999). Simple Heuristics That Make Us Smart. Oxford Press;
Gigerenzer, G. (2007). Gut Feelings: The Intelligence of the Unconscious. Penguin Books;
Hyppänen, O. (2013). Decision Makers' Use of Intuition at the Front End of Innovation, Aalto University (link);
Simon, Herbert (1991). Bounded Rationality and Organizational Learning. Organization Science, 2-1, pp. 125-134;