Why Do Some Software Projects Never Finish? November 28, 2019

Software projects run behind schedule for all kinds of reasons. Unfortunately when that happens, it is often too late in the game. Common questions that are asked during this stage are "how much more budget should be allocated?" and "how much more time?"

There are two classes of slowdown. One is as simple as needing more time due to sheer amount of work. The other is categorically a non-survivable disaster. In this article, we'll examine how architectural considerations could make or break a project.

First let's start with an ideal, albeit naïve scenario - a 4-page document that tells four unrelated stories. Each page takes two hours to write, the total project time is 8 hours. At the 3-page mark, we know exactly that another 2 hours is needed to complete the project.

A complex software project with many dynamic features cannot be planned out in a linear fashion. As shown in the above diagram, the steps, or "iterations" in a project have varying amount of gain given the same amount of time. Typically, the initial iteration takes longer, but it lays the critical foundation so that the subsequent iterations can be carried out more efficiently. An experienced developer would plan out each stage so that the scope in each stage is progressively smaller. The sum of all scopes add to the total scope. The scope size over time fits on a geometric curve.

In many cases, instead of the planned reduction of scope over time, the chart also shows the uncontrolled progress over time, due to architectural limitations.

If the developers are "lucky", the productivity curve should almost converge to the target scope. With unplanned features, also known as a "scope creep", the gap to the finish line grows. Depending on the nature of the scope creep, the delay could be merely adding the extra time for developing the extra features.

It is also likely, however, that the last stretch becomes too wide to close. This is a situation that is difficult to recognize and very costly, if not impossible to recover from. Therefore we must prevent such "infinity stretch" with extra care.

The line between "good enough" and "we'll never finish" is on a sliding scale. Once the threshold of aborting the project is reached, the client panics to savage the project, but at the same time unwilling to dial back the clock to an earlier bail point. This is why many projects continue to fail after many change of hands.

A healthy project trajectory is determined early on. The following diagram shows a plan that is ideal as well as realistic:

In addition to deciding How Much should be done before the first milestone, it is also crucial to define What must be accomplished. Furthermore, the relations between the early and subsequent stages must be figured out.

A software product has many features, many of which can be grouped and generalized. If the first step in development concentrates on factoring commonalities and creating parallel structures, the later stages can take place at a linear pace. It's common to develop abstraction layers such as interfaces, wrappers and plugin systems at this phase.

The exact abstraction and implementation take time to settle. It is a back and forth negotiation process. Therefore, our general approach is to include both steps in the same iteration. By the time the Sub-phase 1 is finished, the rest of the development becomes straightforward, much like building a hotel with the basic wiring and plumbing and one room fully furnished - the remaining rooms are simply blanks to fill.

In conclusion, and abstractly speaking, the more complex a software project, the more it gravitates toward a geometric regression curve. Therefore, more upfront engineering effort is needed to escape the curve into a linear pattern. The sooner the transition happens, the higher chance the project can finish on time. Many projects are still in the geometric curve, or flat lined too late. In either case, such projects will never be completed.