People involved in software projects would say that software development is about understanding complexity.
What is complexity?
Complexity is easy to define at a high level, but people get vague when pressed for a precise definition. Let’s see if we can quantify the problem, then come back and define it.
Assume we need to develop a project of 50,000 lines of code 1 (scenario A), this project would take 5,000 man days to develop assuming 10 production lines of code per day 2. Even if we are generous and assume 50 production lines of code per day per developer, the project would still take 1,000 days to develop.
Now take that developed code and have a single developer re-enter it (scenario B). The developer is likely to be able to code at least 1,000 lines of code per day and it would only take about 50 days to enter the entire program.
97.5% of a project is about solving problems, not writing code
In scenario A we don’t know which lines of code to write, time is expended in figuring out:
- what the actual requirements are
- how to break the requirements into high level design
- how to translate the high level design into programming languages and libraries
- where the code defects are
While it might seem startling that 95% of our time is spend figuring out which lines of code to write, a little reflection will convince us that this is the case. We mistakenly believe that projects fail because there is not enough time to write the code, but really, the problem is that we don’t know what lines of code to write.
Projects do not fail because their is insufficient time to write code
Where does Complexity Come From?
Now that we have quantified how much time is spent solving complexity, where does it come from? Complexity comes from team resources not understanding how to solve the problems they face. When confronted with a problem the team must think about how to solve it.
Complexity is not a issue if enough time has been budgeted in the project to solve it. Complexity is a problem when you need to solve issues and that time was not budgeted. Projects run late mainly because of two reasons:
- Insufficient time
- You did not use formal estimation techniques and underestimate the project
- You use formal estimates and executives push for a delivery date independent of the problems to be solved in the project
- Insufficient requirements
- You estimated sufficient time for your team’s skill level to solve the problems but you did not have all the requirements
Insufficient time
It is pretty clear why insufficient time causes projects to fail, but here are some specific statistics.
If executives reject formal business estimates:
Rejection of estimates for business reasons lead to productivity down 15% and quality down 21%
If executives push for project deadlines that are too agressive:
Excessive schedule pressure leads to productivity down 16% and quality down 23%
Insufficient Requirements
The second situation happens when you underestimate requirement complexity. On a regular basis projects are launched where only a fraction of the requirements are gathered. Once the project starts there is a mad dash to capture the missing requirements, i.e scope creep (see Shift Happens).
Project plans are often based on the initial set of requirements and the project deadline is not adjusted when you discover that there are missing requirements.
Failure to estimate requirement changes to schedule lead to productivity down 16% and quality down 22%
Why We Underestimate Requirement Complexity
Human beings are wired to underestimate complexity. For example, suppose that a customer says that they want a car, we all know what a car is, correct?
Look at the following cars:
Clearly these cars are not the same, and the cost and time required to build them is not the same. Abstractly, we all understand what a car is and we can discuss this in general. But when we start drilling into the details, there are many questions to be answered, such as:
- How fast does the car need to go?
- How many people does it need to seat?
- What kind of fuel economy do you want?
- What safety features are required?
In the rush to get projects started, we skimp on the requirements process and many questions are not answered. But at some point the developers need that information so that they can solve their problems and the requirements need to be gathered. Quite often projects run out of time simply because they never budgeted time to gather enough requirements.
Skill and Complexity
There is a clear trade off between the time required to solve a complex problem and the skill level of the team. A problem is complex when you have never seen it before, but once you have solved it, then you always see the solution.
Just like the maze shown here, the first time might take a fair amount of time to figure out the solution. Afterwards you will go directly to the solution without dead ends.
When gathering requirements, the better the analyst understands the domain, the less likely they are to miss requirements or have inconsistent requirements. This eliminates the likelihood of inconsistent requirements and increases the probability that enough requirements are captured to start the project.
When developers have experience with the domain then they take much less time to break the requirements into high level designs. When they have experience with the language being used and the libraries they can translate that high level design into a low level design and code much quicker.
Solving Complexity
So successful software projects recognize that complexity amounts to the time required for the team to figure out anything for which they are not subject matter experts.
- For the analysts
- This is about understanding the subject matter and making sure to gather enough consistent requirements
- For the developers
- This is about understanding the subject matter for high level design, and understanding the language and libraries for implementation
- For the project managers
- This is about understanding how capable the team is and making sure that sufficient time is budgeted for them to solve all problems
- For the IT executives
- This is about understanding that any given project team needs sufficient time to solve problems and not caving into executive pressure to arbitrarily shorten projects
References
- Jones, Capers.SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008.
| Reference: | How to avoid underestimating complexity from our JCG partner Dalip Mahal at the Accelerated Development blog. |
