Change is a fact of life. Nowhere is this more obvious in a medium where the very thing we make is completely intangible and malleable: software development. There is almost nothing that is impossible in software–there are only limited resources.

This malleability has been one of the reasons for the enormous pace of change we’ve seen in computers in the last half century, but at the same time it’s been a stumbling block to true engineering practices.

Regardless of philosophy, change will happen in all software projects to some degree. There are two extremes of development that most organizations fall between:

• A single coder hacks together whatever the boss needs changed today. (Also fixes bugs from yesterday’s fixes.)
• NASA’s awesomely rigid development practices for the Space Shuttle software (see the Manager’s Handbook for Software Development on the Book list) resulting in extremely high-quality, virtually bug-free software (at a very high price).

For most of us, neither extreme is appropriate. Deciding the right balance of change control is a hard problem, but it’s not impossible. Still, some organizations manage to do so badly at it that it deserves to be studied and dissected.

The Census

As an example, consider the 2010 Census, a project that has been covered in the press and on tech blogs over the last few months. The following is from the second paragraph of the report on the GAO’s study of this massive program:

In October 2007, GAO reported that changes to requirements had been a contributing factor to both cost increases and schedule delays experienced by the FDCA [Field Data Collection Automation] program. Increases in the number of requirements led to the need for additional work and staffing. In addition, an initial underestimate of the contract costs caused both cost and schedule revisions. In response to the cost and schedule changes, the Bureau decided to delay certain system functionality, which increased the likelihood that the systems testing…would not be as comprehensive as planned.[…] Without effective management of these and other key risks, the FDCA program faced an increased probability that the system would not be delivered on schedule and within budget or perform as expected. Accordingly, GAO recommended that the FDCA project team strengthen its risk management activities, including risk identification and oversight.

The Bureau has recently made efforts to further define the requirements for the FDCA program, and it has estimated that the revised requirements will result in significant cost increases.

As with many government projects, there is enough failure to go around. The key idea I want to focus on is a failure on the part of the government agency and on the part of the contractor implementing the technology to adequately plan for change.

Before that, though, I want to say that I have no knowledge of their specific management or technical practices, other than what is publicly reported. For all I know, all parties performed extremely well and still failed. (However, from friends and acquaintances in government and contractor positions around D.C., I think it’s fair to say that most organizations here are behind the curve in software best practices, to put it politely.)

With that (admittedly harsh) assumption, it’s apparent that the agency failed to adequately plan, think through requirements, and communicate effectively to the contracted company. On the developer’s side, it’s apparent they didn’t have effective mechanisms for handling changes.

Before a project is undertaken, a set of fundamental principles must be agreed upon:

1. The client will change their minds. There is no such thing as “written in stone.”
2. Making changes is hard or impossible.

These two principles are fundamentally in conflict. On the one hand, the developer must know that changes are going to occur: sometimes large changes. They can plan for that from the code level up to their organization and processes.

On the other hand, the client, must realize that they can’t just suggest a change at the end of the project and expect it to be thrown in and work correctly.

The tension between these two principles implies certain practices and expectations on all parties, which in an ideal world would lead to a successful outcome more often.

Developer Responsibilities

The developer, knowing that the client does not understand the full set of requirements up front, designs their system in a way that can easily handle change. This means modularity, this means testability. From details as small as designing methods to operate on Streams instead of filenames, to as large as easily reusable components.

The first step is a change in mind set: an acceptance that change will occur, and that it’s ok. When I first started development, frequent changes really frustrated me. Sometimes they still do, but I’m getting better (I hope). Once the mind is malleable, practices can be implemented to handle it effectively.

A few coding practices that encourage changes:

• Test-driven development (or at least full unit testing). In this day and age, with how easy it is to accomplish this, is there a good reason not to do it? I’m not convinced by many arguments against unit testing. A full suite of unit tests gives you confidence in your code, and the ability to safely refactor even large portions of your program. Full TDD I could go either way.
• Constant refactoring to fit the best design possible (within reason. Obviously, there is no “best” design). Software, like everything else in the universe, obeys the law of entropy. Without constant maintenance, it will degrade. Designs degrade as you add on to them. Classes degrade as you stuff them full of things. Don’t stand for it–take the time to make things neat. Get some good books on refactoring if you don’t know how to start.
• Highly Modular. Apply all your OO-design skills here. It matters. Cohesion and coupling really do matter.
• Small, frequent iterations. Most software systems these days are too big and too complex to understand at once. Iterative development is the key to success.
• Strong source-control practices. Does this even need to be discussed? If your organization doesn’t have good source control, you’re doing software wrong.

Other practices on the developer’s part read like a list of Extreme programming fundamentals:

• Communication – The key to any relationship, whether personal or corporate, is communication. The right amount should be discovered and adhered to. It should probably be slightly more than you’re comfortable with (since many developers would rather not talk to anyone ;).
• Simplicity – I believe that the single hardest thing to do in developing software is to manage complexity. It’s requires more brainpower and creativity than all other activities. Coding is easy. Coding so that it is easy to understand a year from now is difficult. If the design is needlessly complex up front, then change is that much harder to implement.  Always remember that software complexity increases exponentially.
• Courage – Stand up for correct principles. Don’t allow politics to interfere with what needs to be done. Don’t be afraid to change the design if it needs to happen. Change coding practices as needed. Never be afraid of the truth–just deal with it.
• Respect – Both sides need to remember that the other is expert in their domain. Humility is key.
• Responsibility – this is something I fear is lacking much in government. Everybody needs to take responsibility for their part. This implies accountability. This can be shared responsibility at some levels, or the-buck-stops-here responsibility at higher levels. If no one is responsible, nobody cares if it fails. Courage is a prerequisite to this.

This mind set needs to encompass not just pure software development, but also processes at the organizational level. This can take many forms, but a few ideas:

• Personal relationships between clients and developers
• Flexible team structure and members
• An easy way to submit and discuss change requests
• A culture that accepts changes
• A good understanding of the problem domain

Jeremy Miller (The Shade Tree Developer) talks a lot about processes and practices and it is a very good read.

Unfortunately, there exists an attitude at many companies that it’s better to milk the clients for as much money as possible rather than do the hard work of getting a good process.

Client Responsibilities

I think my introduction and the discussion about developer responsibilities may imply that because software is so ephemeral, it is therefore easy to change it. After all, it’s not a physical object that has to be broken down and rebuilt–what’s so hard? To anyone who has worked on non-trivial projects, this is laughable. Let’s clarify:

It is easy to program. It’s painfully, mind-bendingly, insanely difficult to design software. So difficult, in fact, that no one can do it well. The sooner everybody understand that, the sooner we can get started with real work. Good design is thought-work, iterated over and over, added to experience and analysis.

Some details are easy to change. Others are fundamental to the structure of the building. By way of metaphor, changing your mind about a house you’re building to suggest a stone facade instead of brick is relatively easy. Deciding you would like to have ten stories added onto your house changes the game significantly. Now you have to build a completely new foundation.

I believe the customer’s responsibility is to gain knowledge and insight into the development practice so they understand why things are difficult. Computers are so fundamental to our culture, we can no longer afford to have companies and agencies run by people who don’t understand them to some degree. The exact degree is debatable, but it’s certainly higher than where we’re at now. In my own situation, I think that when I started my boss did not realize how seemingly-trivial changes could be phenomenally difficult to implement, and therefore bug-prone–all because of design decisions made earlier. I now have him understanding that there is no such thing as a simple change.  (Of course, I still occasionally underpromise and overdeliver–I have to maintain my miracle-worker image after all. 🙂

A client who wants software done and doesn’t want to end up paying an extra $3,000,000,000 for software should understand that critical requirements can’t be left until the product is being delivered. “We didn’t think of it before” is not a good excuse.

A client who understands this will realize how critical initial requirements are and ensure they’re communicated clearly. Most of all, they don’t wait around twirling their thumbs, only to reject the system when it’s done because it doesn’t meet their requirements. They will insist on periodic feedback and demos to make sure things are going the right way.

The responsibility for huge failures is on everybody’s shoulders. I believe that it can often be traced back to an unwillingness to face up to the truth, lack of responsibility, and misunderstanding the nature of the project.

Large projects aren’t doomed to fail

There is a fundamental truth, however, of all projects: there exists at least one change that is too large to do. Maybe the Census required that change–I don’t know. I do know that if both sides follow the best practices they can, the number of must-fail changes can be minimized.

That said, here’s why I refuse to believe the census project wasn’t sorely mismanaged from the outset by all sides: FedEx, UPS, Amazon, Wal-Mart, Microsoft, and dozens (hundreds?) of other companies have built enormous and complex systems for managing large databases and mobile platforms involving thousands of partners. It is ridiculous to me that they could not have built a system for our government quicker, cheaper, and less epic failure than the one they got. If the Census is fundamentally more complex than the problems they deal with, someone please let me know.

Change is part of everything now

The ability to adapt to change is critical, not just for software developers, but for everyone in our society. Technology, events, trends, and money all flow so quickly now that those who can adapt quickly will succeed.

It’s not just software companies that need to adopt sound development practices. There will always be companies that build out our infrastructure (Microsoft, Google, Cisco, etc.), but in the end every company will be a software development company. Disregarding good practices because “we’re not a software” does not work anymore. It’s not true. You are a software company. We all will be.

Idealism or…

If a lot of this seems idealistic, that’s ok. I wish the world were like Star Trek as much as the next geek, but I realize that there are a lot of motivation$ out there. The world is what it is, but if we aren’t trying to improve it by working towards a standard, then what’s the point? We can improve it. There is no excuse for such catastrophic wastes of time, money, and effort.