AND I love the Discovery Channel commercial.
Change is a fact of life. Nowhere is this more obvious in a medium where the very thing we make is completely intangible and malleable: software development. There is almost nothing that is impossible in software–there are only limited resources.
This malleability has been one of the reasons for the enormous pace of change we’ve seen in computers in the last half century, but at the same time it’s been a stumbling block to true engineering practices.
Regardless of philosophy, change will happen in all software projects to some degree. There are two extremes of development that most organizations fall between:
- A single coder hacks together whatever the boss needs changed today. (Also fixes bugs from yesterday’s fixes.)
- NASA’s awesomely rigid development practices for the Space Shuttle software (see the Manager’s Handbook for Software Development on the Book list) resulting in extremely high-quality, virtually bug-free software (at a very high price).
For most of us, neither extreme is appropriate. Deciding the right balance of change control is a hard problem, but it’s not impossible. Still, some organizations manage to do so badly at it that it deserves to be studied and dissected.
As an example, consider the 2010 Census, a project that has been covered in the press and on tech blogs over the last few months. The following is from the second paragraph of the report on the GAO’s study of this massive program:
In October 2007, GAO reported that changes to requirements had been a contributing factor to both cost increases and schedule delays experienced by the FDCA [Field Data Collection Automation] program. Increases in the number of requirements led to the need for additional work and staffing. In addition, an initial underestimate of the contract costs caused both cost and schedule revisions. In response to the cost and schedule changes, the Bureau decided to delay certain system functionality, which increased the likelihood that the systems testing…would not be as comprehensive as planned.[…] Without effective management of these and other key risks, the FDCA program faced an increased probability that the system would not be delivered on schedule and within budget or perform as expected. Accordingly, GAO recommended that the FDCA project team strengthen its risk management activities, including risk identification and oversight.
The Bureau has recently made efforts to further define the requirements for the FDCA program, and it has estimated that the revised requirements will result in significant cost increases.
As with many government projects, there is enough failure to go around. The key idea I want to focus on is a failure on the part of the government agency and on the part of the contractor implementing the technology to adequately plan for change.
Before that, though, I want to say that I have no knowledge of their specific management or technical practices, other than what is publicly reported. For all I know, all parties performed extremely well and still failed. (However, from friends and acquaintances in government and contractor positions around D.C., I think it’s fair to say that most organizations here are behind the curve in software best practices, to put it politely.)
With that (admittedly harsh) assumption, it’s apparent that the agency failed to adequately plan, think through requirements, and communicate effectively to the contracted company. On the developer’s side, it’s apparent they didn’t have effective mechanisms for handling changes.
Before a project is undertaken, a set of fundamental principles must be agreed upon:
- The client will change their minds. There is no such thing as “written in stone.”
- Making changes is hard or impossible.
These two principles are fundamentally in conflict. On the one hand, the developer must know that changes are going to occur: sometimes large changes. They can plan for that from the code level up to their organization and processes.
On the other hand, the client, must realize that they can’t just suggest a change at the end of the project and expect it to be thrown in and work correctly.
The tension between these two principles implies certain practices and expectations on all parties, which in an ideal world would lead to a successful outcome more often.
The developer, knowing that the client does not understand the full set of requirements up front, designs their system in a way that can easily handle change. This means modularity, this means testability. From details as small as designing methods to operate on Streams instead of filenames, to as large as easily reusable components.
The first step is a change in mind set: an acceptance that change will occur, and that it’s ok. When I first started development, frequent changes really frustrated me. Sometimes they still do, but I’m getting better (I hope). Once the mind is malleable, practices can be implemented to handle it effectively.
A few coding practices that encourage changes:
- Test-driven development (or at least full unit testing). In this day and age, with how easy it is to accomplish this, is there a good reason not to do it? I’m not convinced by many arguments against unit testing. A full suite of unit tests gives you confidence in your code, and the ability to safely refactor even large portions of your program. Full TDD I could go either way.
- Constant refactoring to fit the best design possible (within reason. Obviously, there is no “best” design). Software, like everything else in the universe, obeys the law of entropy. Without constant maintenance, it will degrade. Designs degrade as you add on to them. Classes degrade as you stuff them full of things. Don’t stand for it–take the time to make things neat. Get some good books on refactoring if you don’t know how to start.
- Highly Modular. Apply all your OO-design skills here. It matters. Cohesion and coupling really do matter.
- Small, frequent iterations. Most software systems these days are too big and too complex to understand at once. Iterative development is the key to success.
- Strong source-control practices. Does this even need to be discussed? If your organization doesn’t have good source control, you’re doing software wrong.
- Communication – The key to any relationship, whether personal or corporate, is communication. The right amount should be discovered and adhered to. It should probably be slightly more than you’re comfortable with (since many developers would rather not talk to anyone ;).
- Simplicity – I believe that the single hardest thing to do in developing software is to manage complexity. It’s requires more brainpower and creativity than all other activities. Coding is easy. Coding so that it is easy to understand a year from now is difficult. If the design is needlessly complex up front, then change is that much harder to implement. Always remember that software complexity increases exponentially.
- Courage – Stand up for correct principles. Don’t allow politics to interfere with what needs to be done. Don’t be afraid to change the design if it needs to happen. Change coding practices as needed. Never be afraid of the truth–just deal with it.
- Respect – Both sides need to remember that the other is expert in their domain. Humility is key.
- Responsibility – this is something I fear is lacking much in government. Everybody needs to take responsibility for their part. This implies accountability. This can be shared responsibility at some levels, or the-buck-stops-here responsibility at higher levels. If no one is responsible, nobody cares if it fails. Courage is a prerequisite to this.
This mind set needs to encompass not just pure software development, but also processes at the organizational level. This can take many forms, but a few ideas:
- Personal relationships between clients and developers
- Flexible team structure and members
- An easy way to submit and discuss change requests
- A culture that accepts changes
- A good understanding of the problem domain
Unfortunately, there exists an attitude at many companies that it’s better to milk the clients for as much money as possible rather than do the hard work of getting a good process.
I think my introduction and the discussion about developer responsibilities may imply that because software is so ephemeral, it is therefore easy to change it. After all, it’s not a physical object that has to be broken down and rebuilt–what’s so hard? To anyone who has worked on non-trivial projects, this is laughable. Let’s clarify:
It is easy to program. It’s painfully, mind-bendingly, insanely difficult to design software. So difficult, in fact, that no one can do it well. The sooner everybody understand that, the sooner we can get started with real work. Good design is thought-work, iterated over and over, added to experience and analysis.
Some details are easy to change. Others are fundamental to the structure of the building. By way of metaphor, changing your mind about a house you’re building to suggest a stone facade instead of brick is relatively easy. Deciding you would like to have ten stories added onto your house changes the game significantly. Now you have to build a completely new foundation.
I believe the customer’s responsibility is to gain knowledge and insight into the development practice so they understand why things are difficult. Computers are so fundamental to our culture, we can no longer afford to have companies and agencies run by people who don’t understand them to some degree. The exact degree is debatable, but it’s certainly higher than where we’re at now. In my own situation, I think that when I started my boss did not realize how seemingly-trivial changes could be phenomenally difficult to implement, and therefore bug-prone–all because of design decisions made earlier. I now have him understanding that there is no such thing as a simple change. (Of course, I still occasionally underpromise and overdeliver–I have to maintain my miracle-worker image after all. 🙂
A client who wants software done and doesn’t want to end up paying an extra $3,000,000,000 for software should understand that critical requirements can’t be left until the product is being delivered. “We didn’t think of it before” is not a good excuse.
A client who understands this will realize how critical initial requirements are and ensure they’re communicated clearly. Most of all, they don’t wait around twirling their thumbs, only to reject the system when it’s done because it doesn’t meet their requirements. They will insist on periodic feedback and demos to make sure things are going the right way.
The responsibility for huge failures is on everybody’s shoulders. I believe that it can often be traced back to an unwillingness to face up to the truth, lack of responsibility, and misunderstanding the nature of the project.
Large projects aren’t doomed to fail
There is a fundamental truth, however, of all projects: there exists at least one change that is too large to do. Maybe the Census required that change–I don’t know. I do know that if both sides follow the best practices they can, the number of must-fail changes can be minimized.
That said, here’s why I refuse to believe the census project wasn’t sorely mismanaged from the outset by all sides: FedEx, UPS, Amazon, Wal-Mart, Microsoft, and dozens (hundreds?) of other companies have built enormous and complex systems for managing large databases and mobile platforms involving thousands of partners. It is ridiculous to me that they could not have built a system for our government quicker, cheaper, and less epic failure than the one they got. If the Census is fundamentally more complex than the problems they deal with, someone please let me know.
Change is part of everything now
The ability to adapt to change is critical, not just for software developers, but for everyone in our society. Technology, events, trends, and money all flow so quickly now that those who can adapt quickly will succeed.
It’s not just software companies that need to adopt sound development practices. There will always be companies that build out our infrastructure (Microsoft, Google, Cisco, etc.), but in the end every company will be a software development company. Disregarding good practices because “we’re not a software” does not work anymore. It’s not true. You are a software company. We all will be.
If a lot of this seems idealistic, that’s ok. I wish the world were like Star Trek as much as the next geek, but I realize that there are a lot of motivation$ out there. The world is what it is, but if we aren’t trying to improve it by working towards a standard, then what’s the point? We can improve it. There is no excuse for such catastrophic wastes of time, money, and effort.
In the future, when you have your first humanoid robot servant and you decide to have some fun with it, and you tell it to do something stupid and arbitrary like go stand in the corner for no reason, put itself in embarrassing poses, or anything generally “abusive”, will you feel bad afterwards? Should you? Why?
I think I would, perhaps depending on the severity and arbitrariness. I asked my wife. She said definitely yes, and that regardless of whether the machine can “feel” embarrassment or frustration, it reveals a character deficiency in yourself. She compared it to abuse against animals.
I think it may also depend on how humanized the robot is. You would feel bad doing it to Data (ignoring his free will), but maybe not to a mute, grotesque car welder.
Thankfully, South Korea has come up with a robot code of ethics.
Leticia has been interested in getting us involved more in the community so she’s been hunting around for local service opportunities. She finally found a great program in our area that I thought I’d share for anybody looking for volunteer experiences in the DC metro region.
Fairfax City runs a division called Volunteer Fairfax that provides volunteers for numerous events through the year. They also organize team-building service activities for your company, up to thousands of people.
Volunteer Fairfax runs a specific program called Volunteers for Change. This program provides hundreds of weekend and evening volunteer opportunities, organized over the web. It’s perfect for people who can’t commit to regular hours every week or month, but want to do something when they can.
Events from their sample calendar:
- Dinner prep at Ronald McDonald house
- Bowling with the mentally/physically disabled
- Bag groceries
- Thrift store sorting
- theater ushers
- special city events
- medical supply sorting
How it works:
I saw this book when I bought Programming WPF a few weeks ago and it looked promising enough to buy. I’ve been doing unit testing in C# for a few years now, but I thought there were always things to learn and maybe I’d pick up a few new ideas.
It is easy to contrast this book with Beck’s Test Driven Development: By Example, and the two books definitely have a very different feel.
Beck’s book has a very evangelical feel to it, and it’s main purpose is to teach a mind set more than technical details. I believe this is important–maybe of first importance–but once you understand that, the rest of the book is a little simplistic for more experienced developers.
Pragmatic Unit Testing, on the other hand, focuses much more on the practical aspects (hmmm….I highly suspect that’s where the title comes from….) of unit testing. I liked the ideas on how to use categories and attributes to segregate tests that take too long to run on a regular basis. I also liked the section on singletons and getting around time-dependencies. The DateTime.Now problem is something I’ve had to deal with quite a bit in our server-side software that has a lot of time-dependant behavior. (In most cases, the problems were solved with refactoring the time into a function parameter.)
There are also good discussions of more mundane issues like how to deploy NUnit, where to put tests in a project, team practices, GUIs, threading, and C#-specific issues.
The discussion about mock objects (a very basic introduction) is also quite clear and understandable–more so than many resources I’ve seen on the web, which often assume you already know all about them.
Something I don’t like: the acronyms (BICEP, CORRECT, A TRIP). They kind of bug me. I like the ideas behind the acronyms and I think it’s more important and effective (for me, anyway) to internalize the principles of testing rather than remembering specific acronyms and the words they go with. YMMV.
I will probably only read Test Drive Development: By Example once, but I will definitely come back to Pragmatic Unit Testing occasionally to refresh my ideas.