Tag Archives: adaptability

Adapting to Changes

Change is a fact of life. Nowhere is this more obvious in a medium where the very thing we make is completely intangible and malleable: software development. There is almost nothing that is impossible in software–there are only limited resources.

This malleability has been one of the reasons for the enormous pace of change we’ve seen in computers in the last half century, but at the same time it’s been a stumbling block to true engineering practices.

Regardless of philosophy, change will happen in all software projects to some degree. There are two extremes of development that most organizations fall between:

  • A single coder hacks together whatever the boss needs changed today. (Also fixes bugs from yesterday’s fixes.)
  • NASA’s awesomely rigid development practices for the Space Shuttle software (see the Manager’s Handbook for Software Development on the Book list) resulting in extremely high-quality, virtually bug-free software (at a very high price).

For most of us, neither extreme is appropriate. Deciding the right balance of change control is a hard problem, but it’s not impossible. Still, some organizations manage to do so badly at it that it deserves to be studied and dissected.

The Census

As an example, consider the 2010 Census, a project that has been covered in the press and on tech blogs over the last few months. The following is from the second paragraph of the report on the GAO’s study of this massive program:

In October 2007, GAO reported that changes to requirements had been a contributing factor to both cost increases and schedule delays experienced by the FDCA [Field Data Collection Automation] program. Increases in the number of requirements led to the need for additional work and staffing. In addition, an initial underestimate of the contract costs caused both cost and schedule revisions. In response to the cost and schedule changes, the Bureau decided to delay certain system functionality, which increased the likelihood that the systems testing…would not be as comprehensive as planned.[…] Without effective management of these and other key risks, the FDCA program faced an increased probability that the system would not be delivered on schedule and within budget or perform as expected. Accordingly, GAO recommended that the FDCA project team strengthen its risk management activities, including risk identification and oversight.

The Bureau has recently made efforts to further define the requirements for the FDCA program, and it has estimated that the revised requirements will result in significant cost increases.

As with many government projects, there is enough failure to go around. The key idea I want to focus on is a failure on the part of the government agency and on the part of the contractor implementing the technology to adequately plan for change.

Before that, though, I want to say that I have no knowledge of their specific management or technical practices, other than what is publicly reported. For all I know, all parties performed extremely well and still failed. (However, from friends and acquaintances in government and contractor positions around D.C., I think it’s fair to say that most organizations here are behind the curve in software best practices, to put it politely.)

With that (admittedly harsh) assumption, it’s apparent that the agency failed to adequately plan, think through requirements, and communicate effectively to the contracted company. On the developer’s side, it’s apparent they didn’t have effective mechanisms for handling changes.

Before a project is undertaken, a set of fundamental principles must be agreed upon:

  1. The client will change their minds. There is no such thing as “written in stone.”
  2. Making changes is hard or impossible.

These two principles are fundamentally in conflict. On the one hand, the developer must know that changes are going to occur: sometimes large changes. They can plan for that from the code level up to their organization and processes.

On the other hand, the client, must realize that they can’t just suggest a change at the end of the project and expect it to be thrown in and work correctly.

The tension between these two principles implies certain practices and expectations on all parties, which in an ideal world would lead to a successful outcome more often.

Developer Responsibilities

The developer, knowing that the client does not understand the full set of requirements up front, designs their system in a way that can easily handle change. This means modularity, this means testability. From details as small as designing methods to operate on Streams instead of filenames, to as large as easily reusable components.

The first step is a change in mind set: an acceptance that change will occur, and that it’s ok. When I first started development, frequent changes really frustrated me. Sometimes they still do, but I’m getting better (I hope). Once the mind is malleable, practices can be implemented to handle it effectively.

A few coding practices that encourage changes:

  • Test-driven development (or at least full unit testing). In this day and age, with how easy it is to accomplish this, is there a good reason not to do it? I’m not convinced by many arguments against unit testing. A full suite of unit tests gives you confidence in your code, and the ability to safely refactor even large portions of your program. Full TDD I could go either way.
  • Constant refactoring to fit the best design possible (within reason. Obviously, there is no “best” design). Software, like everything else in the universe, obeys the law of entropy. Without constant maintenance, it will degrade. Designs degrade as you add on to them. Classes degrade as you stuff them full of things. Don’t stand for it–take the time to make things neat. Get some good books on refactoring if you don’t know how to start.
  • Highly Modular. Apply all your OO-design skills here. It matters. Cohesion and coupling really do matter.
  • Small, frequent iterations. Most software systems these days are too big and too complex to understand at once. Iterative development is the key to success.
  • Strong source-control practices. Does this even need to be discussed? If your organization doesn’t have good source control, you’re doing software wrong.

Other practices on the developer’s part read like a list of Extreme programming fundamentals:

  • Communication – The key to any relationship, whether personal or corporate, is communication. The right amount should be discovered and adhered to. It should probably be slightly more than you’re comfortable with (since many developers would rather not talk to anyone ;).
  • Simplicity – I believe that the single hardest thing to do in developing software is to manage complexity. It’s requires more brainpower and creativity than all other activities. Coding is easy. Coding so that it is easy to understand a year from now is difficult. If the design is needlessly complex up front, then change is that much harder to implement.  Always remember that software complexity increases exponentially.
  • Courage – Stand up for correct principles. Don’t allow politics to interfere with what needs to be done. Don’t be afraid to change the design if it needs to happen. Change coding practices as needed. Never be afraid of the truth–just deal with it.
  • Respect – Both sides need to remember that the other is expert in their domain. Humility is key.
  • Responsibility – this is something I fear is lacking much in government. Everybody needs to take responsibility for their part. This implies accountability. This can be shared responsibility at some levels, or the-buck-stops-here responsibility at higher levels. If no one is responsible, nobody cares if it fails. Courage is a prerequisite to this.

This mind set needs to encompass not just pure software development, but also processes at the organizational level. This can take many forms, but a few ideas:

  • Personal relationships between clients and developers
  • Flexible team structure and members
  • An easy way to submit and discuss change requests
  • A culture that accepts changes
  • A good understanding of the problem domain

Jeremy Miller (The Shade Tree Developer) talks a lot about processes and practices and it is a very good read.

Unfortunately, there exists an attitude at many companies that it’s better to milk the clients for as much money as possible rather than do the hard work of getting a good process.

Client Responsibilities

I think my introduction and the discussion about developer responsibilities may imply that because software is so ephemeral, it is therefore easy to change it. After all, it’s not a physical object that has to be broken down and rebuilt–what’s so hard? To anyone who has worked on non-trivial projects, this is laughable. Let’s clarify:

It is easy to program. It’s painfully, mind-bendingly, insanely difficult to design software. So difficult, in fact, that no one can do it well. The sooner everybody understand that, the sooner we can get started with real work. Good design is thought-work, iterated over and over, added to experience and analysis.

Some details are easy to change. Others are fundamental to the structure of the building. By way of metaphor, changing your mind about a house you’re building to suggest a stone facade instead of brick is relatively easy. Deciding you would like to have ten stories added onto your house changes the game significantly. Now you have to build a completely new foundation.

I believe the customer’s responsibility is to gain knowledge and insight into the development practice so they understand why things are difficult. Computers are so fundamental to our culture, we can no longer afford to have companies and agencies run by people who don’t understand them to some degree. The exact degree is debatable, but it’s certainly higher than where we’re at now. In my own situation, I think that when I started my boss did not realize how seemingly-trivial changes could be phenomenally difficult to implement, and therefore bug-prone–all because of design decisions made earlier. I now have him understanding that there is no such thing as a simple change.  (Of course, I still occasionally underpromise and overdeliver–I have to maintain my miracle-worker image after all. 🙂

A client who wants software done and doesn’t want to end up paying an extra $3,000,000,000 for software should understand that critical requirements can’t be left until the product is being delivered. “We didn’t think of it before” is not a good excuse.

A client who understands this will realize how critical initial requirements are and ensure they’re communicated clearly. Most of all, they don’t wait around twirling their thumbs, only to reject the system when it’s done because it doesn’t meet their requirements. They will insist on periodic feedback and demos to make sure things are going the right way.

The responsibility for huge failures is on everybody’s shoulders. I believe that it can often be traced back to an unwillingness to face up to the truth, lack of responsibility, and misunderstanding the nature of the project.

Large projects aren’t doomed to fail

There is a fundamental truth, however, of all projects: there exists at least one change that is too large to do. Maybe the Census required that change–I don’t know. I do know that if both sides follow the best practices they can, the number of must-fail changes can be minimized.

That said, here’s why I refuse to believe the census project wasn’t sorely mismanaged from the outset by all sides: FedEx, UPS, Amazon, Wal-Mart, Microsoft, and dozens (hundreds?) of other companies have built enormous and complex systems for managing large databases and mobile platforms involving thousands of partners. It is ridiculous to me that they could not have built a system for our government quicker, cheaper, and less epic failure than the one they got. If the Census is fundamentally more complex than the problems they deal with, someone please let me know.

Change is part of everything now

The ability to adapt to change is critical, not just for software developers, but for everyone in our society. Technology, events, trends, and money all flow so quickly now that those who can adapt quickly will succeed.

It’s not just software companies that need to adopt sound development practices. There will always be companies that build out our infrastructure (Microsoft, Google, Cisco, etc.), but in the end every company will be a software development company. Disregarding good practices because “we’re not a software” does not work anymore. It’s not true. You are a software company. We all will be.

Idealism or…

If a lot of this seems idealistic, that’s ok. I wish the world were like Star Trek as much as the next geek, but I realize that there are a lot of motivation$ out there. The world is what it is, but if we aren’t trying to improve it by working towards a standard, then what’s the point? We can improve it. There is no excuse for such catastrophic wastes of time, money, and effort.


Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order:

Top 5 Attributes of Highly Effective Programmers

What attributes can contribute to a highly successful software developer versus the ordinary run-of-the-mill kind? I don’t believe the attributes listed here are the end-all, be-all list, nor do I believe you have to be born with them. Nearly all things in life can be learned, and these attributes are no exception.

Like this article? Check out my new book, Writing High-Performance .NET Code.

Humility

Humility is first because it implies all the other attributes, or at least enables them. There are a lot of misunderstandings of what humility is and sometimes it’s easier to explain it by describing what humility isn’t:

  • humility isn’t letting people walk all over you
  • humility isn’t suppressing your opinions
  • humility isn’t thinking you’re a crappy programmer

C.S. Lewis said it best, in the literary guise of a devil trying to subvert humanity:

Let him think of [humility] not as self-forgetfulness but as a certain kind of opinion (namely, a low opinion) of his own talents and character. . . Fix in his mind the idea that humility consists in trying to believe those talents to be less valuable than he believes them to be. . . By this method thousands of humans have been brought to think that humility means pretty women trying to believe they are ugly and clever men trying to believe they are fools.*

Ok, so we realize humility isn’t pretending to be worse than you are and it’s not timidity. So what is it?

Simply put, humility is an understanding that the world doesn’t begin and end with you. It’s accepting that you don’t know everything there is to know about WPF, or Perl, or Linux. It’s an acknowledgment of the fact that, even if you’re an expert in some particular area, there is still much to learn. In fact, there is far more to learn than you could possibly do in a lifetime, and that’s ok.

Once you start assuming you’re the expert and final word on something, you’ve stopped growing, stopped learning, and stopped progressing. Pride can make you obsolete faster than you can say “Java”.

The fact is that even if you’re humble, you’re probably pretty smart. If you work in a small organization with few programmers (or any organization with few good programmers), chances are you’re more intelligent than the majority of them when it comes to computers. (If you are smarter than all of them about everything, then either you failed the humility test or you need to get out of that company fast). Since you happen to know more about computers and how software works than most people, and since everybody’s life increasingly revolves around using computers, this will give you the illusion that you are smarter than others about everything. This is usually a mistake.

Take Sales and Marketing for example. I have about 50 Dilbert strips hung in my office. I would guess half of them make fun of Sales or Marketing in some way. It’s easy, it’s fun, and it’s often richly deserved!

But if they didn’t sell your software, you wouldn’t get paid. You need them as much as they need you. If someone asked you to go sell your software, how would you do it? Do you even like talking to people? As clueless as they are about the realities of software development, they have skills you don’t.

There are some industries where extreme ego will get you places. I do not believe this to be the usual case in software, at least in companies run by people who understand software. Ego is not enough–results matter. If you have a big ego, you’d better be able to back it up. Unfortunately, the problem with egos is that they grow–eventually you won’t be able to keep up with it, and people will see through you.

The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague. – Dijkstra

Without humility, you will make mistakes. Actually, even with humility you’ll make mistakes, but you’ll realize it sooner. By assuming that you know how to solve a problem immediately, you may take steps to short-circuit the development process. You may think you understand the software so well that you can easily fix a bug with just a few tweaks…and yet, you didn’t realize that this other function over here is now broken. A humble programmer will first say “I don’t know the right way to solve this yet” and take the time to do the analysis.

Finally, humble people are a lot more pleasant to work with. They don’t make their superiority an issue. They don’t always have the “right answer” (meaning everybody else is wrong, of course). You can do pair programming with a humble person, you can do code reviews with a humble person, you can instruct a humble person.

Leave your ego out of programming.

It’s not at all important to get it right the first time. It’s vitally important to get it right the last time. – Andrew Hunt and David Thomas

Perhaps most importantly, a humble programmer can instruct him/herself…

Love of Learning

If you’re new to this whole programming thing, I hate to break it to you: school has just begun. Whatever you thought of your BS/MS in Computer Science you worked so hard at–it was just the beginning. It will get you your first job or two, but that’s it. If you can’t learn as you go, from now until you retire, you’re dead weight. Sure, you’ll be able to find a job working somewhere with your pet language for the rest of your life–COBOL and Fortran are still out there after all, but if you really want to progress you’re going to have to learn.

Learning is not compulsory. Neither is survival. – W. Edwards Deming

This means reading. A lot. If you don’t like reading, I suggest you start–get into Harry Potter, fantasy, science fiction, historical fiction, whatever. Something. Just read. Then get some technical books. Start with my list of essential developer books. They’re not as exciting as Harry Potter, but they’re not bad either.

A lot material is online, but for high quality, authoritative prose on fundamental subjects, a good book beats all.

Reading isn’t enough, though. You have to practice. You have to write your own test projects. You have to force yourself to push your boundaries. You can start by typing in code sample, but then you need to change them in ways unique to you. You should have personal projects and hobbies that expand your skills. Be writing your own tools, or “fun” programs. Write a game. Do what it takes to learn new things!

The type of programs you write have a big bearing on how well you learn new material. It has to be something that interests you, or you won’t keep it up. In my case, I’m developing software related to my LEGO hobby. In the past, I’ve written tools for word puzzles, system utilities, multimedia plug-ins, and more. They all started out of a need I had, or a desire to learn something new and useful to me personally.

Another aspect of this love of learning is related to debugging software. An effective developer is not satisfied with a problem until they understand how it works, why it happens, and the details of how to fix it. The details matter–understanding why a bug occurs is just as important as knowing what to do to fix it.

Learn from mistakes. I have seen programmers who make a mistake, have the correct solution pointed out to them, say, “huh, wonder why it didn’t work for me”, and go on their merry way, none-the-wiser. Once their code is working “the way it should” they’re done. They don’t care why or how–it works and that’s enough. It’s not enough! Understand the mistake,what fixed it, and why.

Good judgement comes from experience, and experience comes from bad judgement. – Fred Brooks

Obviously, some balance has to be struck here. You cannot learn everything–it simply isn’t possible. Our profession is becoming increasingly specialized because there is simply too much out there. I also think that in some respects, you need to love learning just for the sake of learning.

Detail-orientedness

Developing software with today’s technologies is all about the details. Maybe in 100 years, software will progress to the point where it can write itself, be fully component-pluggable, self-documenting, self-testing, and then…there won’t be any programmers. But until that comes along (if ever), get used to paying attention to a lot of details.

To illustrate: pick a feature of any software product, and try to think of all the work that would have to be done to change it in some way. For anything non-trivial, you could probably come up with a list of a hundred discrete tasks: modifying the UI (which includes graphics, text, localization, events, customization, etc.), unit tests, algorithms, interaction with related components, and on and on, each discrete step being broken into sub-steps.

I have always found that plans are useless, but planning is indispensable. – Dwight Eisenhower

Here’s a problem, though: few humans can keep every single task in their head, especially over time. Thankfully, detail-orientedness does not necessarily mean being able to mentally track each and every detail. It means that you develop a mental pattern to deal with them. For example, the steps of changing a piece of software could be:

  1. Thoroughly understand what the code is doing and why
  2. Look for any and all dependencies and interactions with this code
  3. Have a well-thought-out mental picture of how it fits together.
  4. Examine the consequences of changing the feature.
  5. Update all related code that needs to (and repeat this cycle for those components)
  6. Update auxiliary pieces that might depend on this code (build system, installer, tests, documentation, etc.)
  7. Test and repeat.

An example: I find that as I’m working on a chunk of code, I realize there are several things I need to do after I’m finished with my immediate task. If I don’t do them, the software will break. If I try to remember all of them, one will surely slip by the wayside. I have a few choices here:

  1. Defer until later, while trying to remember them all
  2. Do them immediately
  3. Defer until later, after writing  them down

Each of these might be useful in different circumstances. Well…maybe not #1. I think that’s doomed to failure from the start and creates bad habits. If the secondary tasks  are short, easy, and well-understood, just do them immediately and get back to your primary task.

However, if you know they’ll require a lot of work, write them down. I prefer a sturdy engineering notebookin nearly all cases, but text files, Outlook tasks, notes, OneNote, bug tracking systems, and other methods can all work together to enable this.

The more experience you have, the easier you’ll be able to track the details you need to worry about. You’ll also analyze them quicker, but you will always need some way of keeping track of what you need to do next. There are simply too many details. Effective organization is a key ability of any good software engineer.

Another aspect of paying attention to details is critical thinking. Critical thinking implies a healthy skepticism about everything you do. It is particularly important as you examine the details of your implementation, designs, or plans. It’s the ability to pull out of those details what is important, what is correct, or on the other hand, what is garbage and should be thrown out. It also guides when you should use well-known methods of development, and when you need to come up with a novel solution to a hard problem.

Adaptability

“Enjoying success requires the ability to adapt. Only by being open to change will you have a true opportunity to get the most from your talent.” – Nolan Ryan

Change happens. Get used to it. This is a hard one for me, to tell the truth. I really, really like having a plan and following it, adapting it to my needs, not those of others.

The fact is, in software development, the project you end up writing will not be the one you started. This can be frustrating if you don’t know how to handle it.

To become adaptive first requires a change in mind set. This mind set says that change is inevitable, it’s ok, and you’re ready for it. How do you become ready for it? This is a whole other topic in itself, and I will probably devote a separate essay for it.

Other than the shift in mind set, start using techniques and technologies that enable easy change. Things like unit testing, code coverage, and refactoring all enable easier modification of code.

In war as in life, it is often necessary when some cherished scheme has failed, to take up the best alternative open, and if so, it is folly not to work for it with all your might. – Winston Churchill

For me, the first step in changing my mind set is to not get frustrated every time things change (“But you specifically said we were NOT going to implement the feature to work this way!”).

Passion

I think passion is up there with humility in importance. It’s so fundamental, that without it, the others don’t matter.

Anyone can dabble, but once you’ve made that commitment, your blood has that particular thing in it, and it’s very hard for people to stop you. – Bill Cosby

It’s also the hardest to develop. I’m not sure if it’s innate or not. In my own case, I think my passion developed at a very early age. It’s been there as long as I remember, even if I had periods of not doing much with it.

I’ve interviewed dozens of prospective developers at my current job, and this is the one thing I see consistently lacking. So many of them are in it just for another job. If that’s all you want, just a job to pay the bills, so be it. (Of course, I have to ask, if that’s the case, why are you reading this article?)

One person with passion is better than forty people merely interested. – E. M. Forster

There’s a world of difference between someone who just programs and someone who loves to program. Someone who just programs will probably not be familiar with the latest tools, practices, techniques, or technologies making their way down the pipeline. They won’t think about programming outside of business hours. On the weekends, they do their best to forget about computers. They have no personal projects, no favorite technologies, no blogs they like to read, and no drive to excel. They have a hard time learning new things and can be a large burden on an effective development team.

Ok, that’s maybe a bit of exaggeration, but by listing the counterpoints, it’s easier to see symptoms of someone who does have passion:

  • Thinks and breaths technology
  • Reads blogs about programming
  • Reads books about programming
  • Writes a blog about programming
  • Has personal projects
  • These personal projects are more important than the boring stuff at work
  • Keeps up with latest technologies for their interests
  • Pushes for implementation of the latest technologies (not blindly, of course)
  • Goes deep in technical problems.
  • Not content with merely coding to spec.
  • Needs an outlet of creativity, whether it be professional (software design) or personal (music, model building, LEGO building, art, etc.)
  • Thinks of the world in terms of Star Trek

Just kidding on the last one…

…(maybe)

That’s my list. It’s taken a few months to write this, and I hope it’s genuinely useful to someone, especially new, young software engineers just getting started. This is a hard industry, but it should also be fun. Learning these attributes, changing your mind set, and consciously deciding to become the engineer and programmer you want to be are the first steps. And also part of every step thereafter.

Nobody is born with any of these–they are developed, practiced, and honed to perfection over a lifetime. There is no better time to start than now.

* The Screwtape Letters, C.S. Lewis


Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order: