Tag Archives: government

Adapting to Changes

Change is a fact of life. Nowhere is this more obvious in a medium where the very thing we make is completely intangible and malleable: software development. There is almost nothing that is impossible in software–there are only limited resources.

This malleability has been one of the reasons for the enormous pace of change we’ve seen in computers in the last half century, but at the same time it’s been a stumbling block to true engineering practices.

Regardless of philosophy, change will happen in all software projects to some degree. There are two extremes of development that most organizations fall between:

  • A single coder hacks together whatever the boss needs changed today. (Also fixes bugs from yesterday’s fixes.)
  • NASA’s awesomely rigid development practices for the Space Shuttle software (see the Manager’s Handbook for Software Development on the Book list) resulting in extremely high-quality, virtually bug-free software (at a very high price).

For most of us, neither extreme is appropriate. Deciding the right balance of change control is a hard problem, but it’s not impossible. Still, some organizations manage to do so badly at it that it deserves to be studied and dissected.

The Census

As an example, consider the 2010 Census, a project that has been covered in the press and on tech blogs over the last few months. The following is from the second paragraph of the report on the GAO’s study of this massive program:

In October 2007, GAO reported that changes to requirements had been a contributing factor to both cost increases and schedule delays experienced by the FDCA [Field Data Collection Automation] program. Increases in the number of requirements led to the need for additional work and staffing. In addition, an initial underestimate of the contract costs caused both cost and schedule revisions. In response to the cost and schedule changes, the Bureau decided to delay certain system functionality, which increased the likelihood that the systems testing…would not be as comprehensive as planned.[…] Without effective management of these and other key risks, the FDCA program faced an increased probability that the system would not be delivered on schedule and within budget or perform as expected. Accordingly, GAO recommended that the FDCA project team strengthen its risk management activities, including risk identification and oversight.

The Bureau has recently made efforts to further define the requirements for the FDCA program, and it has estimated that the revised requirements will result in significant cost increases.

As with many government projects, there is enough failure to go around. The key idea I want to focus on is a failure on the part of the government agency and on the part of the contractor implementing the technology to adequately plan for change.

Before that, though, I want to say that I have no knowledge of their specific management or technical practices, other than what is publicly reported. For all I know, all parties performed extremely well and still failed. (However, from friends and acquaintances in government and contractor positions around D.C., I think it’s fair to say that most organizations here are behind the curve in software best practices, to put it politely.)

With that (admittedly harsh) assumption, it’s apparent that the agency failed to adequately plan, think through requirements, and communicate effectively to the contracted company. On the developer’s side, it’s apparent they didn’t have effective mechanisms for handling changes.

Before a project is undertaken, a set of fundamental principles must be agreed upon:

  1. The client will change their minds. There is no such thing as “written in stone.”
  2. Making changes is hard or impossible.

These two principles are fundamentally in conflict. On the one hand, the developer must know that changes are going to occur: sometimes large changes. They can plan for that from the code level up to their organization and processes.

On the other hand, the client, must realize that they can’t just suggest a change at the end of the project and expect it to be thrown in and work correctly.

The tension between these two principles implies certain practices and expectations on all parties, which in an ideal world would lead to a successful outcome more often.

Developer Responsibilities

The developer, knowing that the client does not understand the full set of requirements up front, designs their system in a way that can easily handle change. This means modularity, this means testability. From details as small as designing methods to operate on Streams instead of filenames, to as large as easily reusable components.

The first step is a change in mind set: an acceptance that change will occur, and that it’s ok. When I first started development, frequent changes really frustrated me. Sometimes they still do, but I’m getting better (I hope). Once the mind is malleable, practices can be implemented to handle it effectively.

A few coding practices that encourage changes:

  • Test-driven development (or at least full unit testing). In this day and age, with how easy it is to accomplish this, is there a good reason not to do it? I’m not convinced by many arguments against unit testing. A full suite of unit tests gives you confidence in your code, and the ability to safely refactor even large portions of your program. Full TDD I could go either way.
  • Constant refactoring to fit the best design possible (within reason. Obviously, there is no “best” design). Software, like everything else in the universe, obeys the law of entropy. Without constant maintenance, it will degrade. Designs degrade as you add on to them. Classes degrade as you stuff them full of things. Don’t stand for it–take the time to make things neat. Get some good books on refactoring if you don’t know how to start.
  • Highly Modular. Apply all your OO-design skills here. It matters. Cohesion and coupling really do matter.
  • Small, frequent iterations. Most software systems these days are too big and too complex to understand at once. Iterative development is the key to success.
  • Strong source-control practices. Does this even need to be discussed? If your organization doesn’t have good source control, you’re doing software wrong.

Other practices on the developer’s part read like a list of Extreme programming fundamentals:

  • Communication – The key to any relationship, whether personal or corporate, is communication. The right amount should be discovered and adhered to. It should probably be slightly more than you’re comfortable with (since many developers would rather not talk to anyone ;).
  • Simplicity – I believe that the single hardest thing to do in developing software is to manage complexity. It’s requires more brainpower and creativity than all other activities. Coding is easy. Coding so that it is easy to understand a year from now is difficult. If the design is needlessly complex up front, then change is that much harder to implement.  Always remember that software complexity increases exponentially.
  • Courage – Stand up for correct principles. Don’t allow politics to interfere with what needs to be done. Don’t be afraid to change the design if it needs to happen. Change coding practices as needed. Never be afraid of the truth–just deal with it.
  • Respect – Both sides need to remember that the other is expert in their domain. Humility is key.
  • Responsibility – this is something I fear is lacking much in government. Everybody needs to take responsibility for their part. This implies accountability. This can be shared responsibility at some levels, or the-buck-stops-here responsibility at higher levels. If no one is responsible, nobody cares if it fails. Courage is a prerequisite to this.

This mind set needs to encompass not just pure software development, but also processes at the organizational level. This can take many forms, but a few ideas:

  • Personal relationships between clients and developers
  • Flexible team structure and members
  • An easy way to submit and discuss change requests
  • A culture that accepts changes
  • A good understanding of the problem domain

Jeremy Miller (The Shade Tree Developer) talks a lot about processes and practices and it is a very good read.

Unfortunately, there exists an attitude at many companies that it’s better to milk the clients for as much money as possible rather than do the hard work of getting a good process.

Client Responsibilities

I think my introduction and the discussion about developer responsibilities may imply that because software is so ephemeral, it is therefore easy to change it. After all, it’s not a physical object that has to be broken down and rebuilt–what’s so hard? To anyone who has worked on non-trivial projects, this is laughable. Let’s clarify:

It is easy to program. It’s painfully, mind-bendingly, insanely difficult to design software. So difficult, in fact, that no one can do it well. The sooner everybody understand that, the sooner we can get started with real work. Good design is thought-work, iterated over and over, added to experience and analysis.

Some details are easy to change. Others are fundamental to the structure of the building. By way of metaphor, changing your mind about a house you’re building to suggest a stone facade instead of brick is relatively easy. Deciding you would like to have ten stories added onto your house changes the game significantly. Now you have to build a completely new foundation.

I believe the customer’s responsibility is to gain knowledge and insight into the development practice so they understand why things are difficult. Computers are so fundamental to our culture, we can no longer afford to have companies and agencies run by people who don’t understand them to some degree. The exact degree is debatable, but it’s certainly higher than where we’re at now. In my own situation, I think that when I started my boss did not realize how seemingly-trivial changes could be phenomenally difficult to implement, and therefore bug-prone–all because of design decisions made earlier. I now have him understanding that there is no such thing as a simple change.  (Of course, I still occasionally underpromise and overdeliver–I have to maintain my miracle-worker image after all. 🙂

A client who wants software done and doesn’t want to end up paying an extra $3,000,000,000 for software should understand that critical requirements can’t be left until the product is being delivered. “We didn’t think of it before” is not a good excuse.

A client who understands this will realize how critical initial requirements are and ensure they’re communicated clearly. Most of all, they don’t wait around twirling their thumbs, only to reject the system when it’s done because it doesn’t meet their requirements. They will insist on periodic feedback and demos to make sure things are going the right way.

The responsibility for huge failures is on everybody’s shoulders. I believe that it can often be traced back to an unwillingness to face up to the truth, lack of responsibility, and misunderstanding the nature of the project.

Large projects aren’t doomed to fail

There is a fundamental truth, however, of all projects: there exists at least one change that is too large to do. Maybe the Census required that change–I don’t know. I do know that if both sides follow the best practices they can, the number of must-fail changes can be minimized.

That said, here’s why I refuse to believe the census project wasn’t sorely mismanaged from the outset by all sides: FedEx, UPS, Amazon, Wal-Mart, Microsoft, and dozens (hundreds?) of other companies have built enormous and complex systems for managing large databases and mobile platforms involving thousands of partners. It is ridiculous to me that they could not have built a system for our government quicker, cheaper, and less epic failure than the one they got. If the Census is fundamentally more complex than the problems they deal with, someone please let me know.

Change is part of everything now

The ability to adapt to change is critical, not just for software developers, but for everyone in our society. Technology, events, trends, and money all flow so quickly now that those who can adapt quickly will succeed.

It’s not just software companies that need to adopt sound development practices. There will always be companies that build out our infrastructure (Microsoft, Google, Cisco, etc.), but in the end every company will be a software development company. Disregarding good practices because “we’re not a software” does not work anymore. It’s not true. You are a software company. We all will be.

Idealism or…

If a lot of this seems idealistic, that’s ok. I wish the world were like Star Trek as much as the next geek, but I realize that there are a lot of motivation$ out there. The world is what it is, but if we aren’t trying to improve it by working towards a standard, then what’s the point? We can improve it. There is no excuse for such catastrophic wastes of time, money, and effort.

The power of the blog to motivate corporate, societal, and government change

This is an issue that has been discussed many times previously–so many that I won’t even bother to link to those discussions. By now it’s well-understood that blogs carry a power stronger than most in the media initially assumed possible.

Not just blogs, but the entire “Web 2.0” phenomenon–MySpace, YouTube–the whole rotten bunch. 🙂 Would Patricia Dunn have stepped down as chair of HP were it not for the constant pounding brought on by the likes of Scoble? Maybe, maybe not. In some sectors, blogs are becoming as well-regarded, if not more, than traditional publishing. Maybe this is limited to the computer industry. Maybe I just read too many blogs. 🙂

Still, it seems that the nature of debate and information dissemination has changed. No longer are we fed what mainstream publishers tell us–even if it’s of better quality. We are now free to choose what and how we read–for good or bad.

We’ve already seen the effects on the corporations. Companies simply can’t get away with anything anymore. Somebody, somewhere, will jump on it.

Areas where I think it will get more interesting:

  1. entertainment – RIAA, MPAA, I’m talking about you. You have ZERO friends among bloggers. All of the bad things you’ve done in courts to innocent people, all of your extortion is shouted from the rooftops by people like those at TechDirt.  You can’t win this war. For now, the audience isn’t very general, but news spreads, and it’s spreading faster and further. Sooner or later, you will lose the PR battle completely–in the meantime, unless your companies drastically change how they do business, your business will be swept out from under you, relegated to the dustbin of irrelevance.
  2. corporations – Microsoft already can’t do anything without the blogosphere lighting up. In some ways, they’ve chosen to embrace this–witness the very high-quality set of developer blogs they host. On the other hand, they’re like any other large company–they have secrets and tactics they would rather not be public debate-fodder. Corporations will be forced to open the windows and let the light shine in on what they’re doing. 
  3. government – imagine if honest, whistle-blowing (or even dishonest whistle-blowing!) staffers ratted on all the corruption in Washington. Imagine if every backroom deal was publicized in embarrassing detail. I don’t think we’re anywhere close to that yet, but there are signs that things are beginning to emerge. Look at the hilarity on YouTube about Senator Ted Stevens’ gaffe about the Internet’s tubes. How long as CSPAN been broadcasting, again? Our elected officials say dumb things about topics they don’t understand all the time–but now we can hear about it over and over again.

Overall, I think blogging will lead to more accountability of traditional structures of society. However, even with these possibilities, there are potential pitfalls:

  1. Overcrowded Medium — occurs when there are WAYYYYYYY too many people broadcasting that not enough people are listening. If everybody in the world blogged, who would read them?
  2. Loss of accountability – if there is accountability for things that are written online, than anything goes. The Internet is already the source of much bad information–it can become much worse if most of it is partisan, subjective, opinionated blather. Still, I’m not convinced it will really be worse than the status quo. The media now is far from infallible. Maybe part of me just wants to keep faith in people’s ability to reason. 🙂
  3. Undercrowded Debates – Broadcast media is a finite resource therefore it maintains its quality mostly by the fact that it has  to judge some things more worthy of discussion than others. Those topics are what people hear about. The Internet, on the other hand, is an unlimited resource. Anybody can have a blog on anything and most do. 🙂 This means that people themselves must choose what they follow, leading to some topics having far fewer meaningful discussions than others. For example, blogs about software and computers comprise a fairly large and active community. Politics has a large community. But what about small-interest, high-importance communities and topics? Where are the scientist blogs about global warming? I’m sure there are some, but is that kind of community ever going to gain a large enough population to affect societal opinion?
  4. Lack of participation – related to Undercrowded Debates, this means people don’t participate in all the areas that are pertinent to their lives. For example, how many of the US Internet users follow blogs discussing network neutrality? This is certainly an issue that could affect all of us, but from what I can tell it’s mostly debated on tech blogs, while the rest of the country misrepresents the entire issue. It works the other way around–I don’t read any political blogs at the moment. What issues am I missing out on? It’s too easy to become part of a niche community on the Internet and ignore the community as a whole.

Some of these problems stem from the anonymity of the Internet, others from the exponential increases in information available to us. Perhaps there are technologies in the pipeline that will solve these issues for us someday. They certainly aren’t going away.

tags: , , , ,

Alexandria, Virginia has RSS Feeds available

My city of Alexandria, VA has a couple of RSS feeds available. They’re not much quite yet (DASH bus service notifications and city press releases), but there is the promise of more.

 What I’d like to see next? Daily crime reports and history and recreation news.

Alexandria also has a very impressive collection of online GIS applications that give you multiple views of the city.

It’s nice to see towns becoming more technologically adept, especially with their communications options. I already get some city news via e-mail, but I would prefer RSS.

Raising Taxes versus Lowering Spending

Every few months, our Senators and Congress-men and women debate taxes and whether or not to raise, lower, keep them steady, or blame others for doing so.

The main thing that I notice during this ridiculous political fights is that nobody ever talks about the most basic financial wisdom: Spend Less Than You Earn. Why doesn’t this apply to governments as well as individuals? Most people in Congress are very wealthy and presumably understand basic financial principles of wealth-building quite well. So why can’t they all exercise restraint and simply spend less? Raising taxes to cover budget shortfalls makes me very nervous because there is no amount of money that cannot be overspent! Raising taxes is not a real solution, just as earning more money isn’t a solution to your bad spending habits.