Avoiding software development failure
Why do so many IT projects fail?
It is a fact of life that as many as 80-90% of IT projects run over budget or are terminated prematurely. And even those that reach some sort of completion often fall far short of meeting user expectations and business performance goals.
The Standish Group reports that around 30% of IT projects will be cancelled. Around 52% of projects will cost 189% of their original estimates. Only 16% are completed on-time and on-budget. In larger companies, the news is even worse: only 9% of their projects come in on-time and on-budget.
What is the problem?
Common problems nominated in failed projects are:
- lack of user involvement in the design
- lack of management involvement
- lack of a clear scope
The single major reason for the failure of most projects is simple:
These projects fail because they have attempted to automate an ill-defined process
The method used to develop in-house IT systems consists of "paving the cowpath": taking an existing process and automating it. That's fine if you have a nice straight, mapped, cowpath.
When business analysts from IT turn up at a user department, they are often faced with users who have only a vague idea of the actual process which is in place. Different users will have different ideas of the same process. Process steps will be ill-defined and often ignored by those operating the process.
Inevitably, the analysts will come back with a user requirements statement which represents what they have been told, plus perhaps what they themselves imagine the user needs. With no other input but their own (faulty) recollection of the process, users are often more than happy to sign off on this.
None of this is the fault of the IT methodology, or the analysts themselves, or even the users interviewed. The fault is in the gap between operations and IT.
Hammer's "Business Process Re-Engineering" tries to solve the problem by ignoring the users (and the cowpath) altogether and building roads where IT thinks they should go. Unfortunately (as Hammer admits), this often fails.
Alternative
We have been trialling a new approach with our clients which comes from our 20 years experience of watching failed implementations. It's very simple:
Before you start, document the processes that you want the system to automate
... in other words, map the cowpath.
In implementing a new computer system the essential first step is often overlooked - to document the business processes and procedures as they were before the planning for the implementation begins.
Why should you commit resources to documenting our procedures as they are now, when you are about to change them?
Because without a clearly defined starting point, the way ahead, and the business goals, are much more difficult to define - and ultimately to verify. You can't automate or change it if you don't know what it is.Most IT development methodologies include allowance for defining the process, and in most projects an attempt is made to achieve this. But the fundamental flaw in this methodology is that the process definition is traditionally part of the IT methodology and not part of the business methodology.
For process definition to be effective, there must be a clear division between the business stakeholders and the IT stakeholders. In short, the best way to achieve a successful implementation is to document the processes before the IT analysts get involved. It is vital that the processes be documented in a way that can be readily understood by users and management alike.
Process improvement
The most critical area in implementation is that delicate transition between the way it was done before, and the way it will be done on the new system. This process needs to be fully understood and properly controlled with full involvement of both business and IT personnel, not just IT alone.
For maximum understanding and exchange of ideas in this transition there must be a large area of commonality of understanding and terminology between how the business sees and defines its processes, and how the IT sees and defines the processes. This leaves the business free to optimise and agree on processes without having to be involved in IT's approach to the problem. At the same time it leaves IT free to define and specify the businesses needs in their own terms without confusing the issue and cutting the business out of the loop.
Don't think that defining processes in this way locks you into a particular way of doing things. A well-defined and well-documented set of processes is a necessary basis for process improvement whether this takes place prior to IT involvement or during the system implementation. Well defined processes free a business from abstract constraints and pave the way for process improvement.
The consequences of not defining business processes prior to system implementation are:
- Incomplete software specifications. Without defined processes, the IT software requirements specifications that are developed do not accurately reflect business needs. The amount of time allocated for business analysts to specify the software is greatly underestimated and, if there are no written procedures, often falls far short of that required to adequately and accurately specify the system.
- Residual manual processes. Projects begun without process definition eventually lurch into a state of mild chaos where the software is not understood and not readily taken up by the business. This leads to the emergence of a large number of business processes and sub-processes which cannot be performed on the system in its initial state and which remain as "manual" processes. You can hear the analysts say, "You didn't say you wanted to do that!", to which the user's response is, "You didn't ask!". If the processes had been documented in the first place, there would be no question.
- Inadequate software testing. Without documented business procedures, there is no clear basis for testing the system and no way of gauging the benefits of implementation. Even the most sophisticated software testing methodologies and tools require a detailed definition of what is expected of the software to be given before their sophistication can be fully exploited. Without process definition this is not possible to provide. Often software testing is left to users who are not able to fully exercise the system. In short, how do you test a system if you have no clear idea of what is expected of it? Regression testing is not performed at each release and functionality formerly present in the software is not verified and can be lost.
The result of all this is a system that is not fulfilling business needs, and which is falling short of expectations.
Documentation and training
It takes little imagination to see how a lack of process definition can both dramatically increase the time it takes to document the system, and dramatically reduce the quality of the end result. Both of these deficiencies have a major contribution to project failure. Without proper process definition, the task of developing user documentation for the system becomes an odyssey of discovery and rediscovery - a process which cannot be planned and which presents us with no visible "end result" milestone.
Without process definition it is almost impossible to analyse staff competency requirements and to plan and implement effective training programs. How can you develop training courses when you don't know who should attend and what each person's specific training needs are. In this situation, the process of user training cannot but fail to meet expectations and fail to provide the needed impetus to user acceptance and uptake of the new system.
Conclusions
How do you avoid failure in a software implementation project? The key is to have in place, before you start the system specification phase, a clearly defined and accurately documented set of business processes and procedures. If you have these, you will have:
- a precise appreciation of business needs and expectations; this will enable you to see the way ahead
- a firm basis for the development of the software specification; this ensures that the software will come as close as possible to meeting the business needs sooner, and ensure a smooth transition for the business
- an ideal platform for continuous improvement of business processes, either during the implementation, or after
- an accurate outline of the project on which you can base your implementation plan and monitor project progress
- a readily adaptable set of parameters on which to base system testing
- a vastly improved appreciation of business needs and expectations against which you can gauge the success of the project
Who should be involved?
Who should have the responsibility for documenting the processes before analysis begins? Most operational areas are too busy doing what they do to step back and document it fully. And yet they are the key people who should be providing input. The answer is to involve a professional technical writer who can work with users (methods like flowcharting sessions were made for this) and provide a clear and concise map of the land before IT arrives.
If you would like to discuss our approach further, please contact us and talk to Phil Cohen or Bill Sayer.
This article may be reproduced only with the permission of HCi (email HCi ). Copyright HCi, 2001.
