One of the key requirements for being able to reliably update software is the confidence that the changes you are making are safe. The amount of confidence required increases with the complexity of the system.
In my day job I work on a real-time messaging system that can have very, very little downtime. As the service grows and sees more traffic, the amount of downtime shrinks. We start to worry now if upgrades take longer than 5 minutes. (It’s almost to the point where we’ll need redundant systems in order to do maintenance).
To upgrade this software, I have to have an awful lot of confidence in the code changes made. Sometimes that confidence varies.
What to do to gain confidence:
1. Consistent process
Having a set of rules to follow is a fundamental requirement of good software engineering. I’m not going to discuss what the process should be, but you should have one that works well.
Why is this important?
Programmers like order. We like well-defined problems where we can see the end from the beginning. We don’t like haziness, indeterminism, or too many choices.
A process nails down the unknowns–it tells you very specifically what the next thing to do is. A good process leaves no room for doubt.
A good development, testing, and deployment process is the first step to building confidence in what you’re doing.
For my messaging system example, here’s a short summary of what our upgrade process is. We didn’t just come up with it–it evolved as our business grew and the requirements grew with it.
- All unit and integration tests pass (we have about 2200 automated tests for this that can run in about 4 minutes)
- We’ve run it on staging server for a few weeks with no issues.
- All new features have been tested on the staging server
- Formal change document has been submitted and approved.
- Formal test results documented.
- Previous version backed up.
- Perform upgrade
- Monitor system (including custom automated monitoring tools)
At any point, I know where we are in the process and what needs to be done next. Sure, there may be details within these steps that require thought and creativity, but the process guides it all and makes us more confident that we’re not performing ad-hoc operations.
2. Unit Testing
There are other types of testing, but it all starts at the unit level, with simple tests that exercise your code line by line, function by function, feature by feature. I recently wrote a few thoughts about unit testing. Unit testing is where you can see the overall wellness of your code–you want that green bar!
Without unit testing, how do you know the code you’re writing is doing what it should? do you just run it and push it through its paces? This is highly inefficient for most types of code. You’ll run out of steam before you start getting close to edge cases.
The fact is that automated unit tests are a baseline for confidence in your code. You need to be able demonstrate time and again that your code performs well.
This all presupposes that you are writing good unit tests. If you’re not sure, start studying. I don’t buy the arguments about lulling developers into a false sense of security–sure, that can happen, but having good developers who understand this is a prerequisite.
If you’re not unit testing–what is your basis for confidence in your code?
3. Code Coverage
Code coverage goes hand-in-hand with unit testing as a good way to automatically discover what areas of your program are in need of more testing. I’ve found that one of the biggest barriers to unit testing a large C++ application we have is that we have no way of easily measuring test coverage. If we had time, we could definitely to the analysis ourselves, or we could spend a lot of money to get a C++ instrumentation profiler, but these are slow and very tedious to use in my experience.
In .Net, use the tools to your advantage.
The psychological benefits of seeing 75-, 90-, 95-, even 100-percent coverage are immense. You know that every line of the program has at least been touched.
Of course, most code coverage tools analyze line coverage, not path coverage. Combine complexity analysis with code coverage to determine which functionality should probably have better testing. There are plenty of free and commercial tools that will give you cyclomatic complexity, among other metrics.
Use other analysis tools like FxCop to make sure your other ducks are in a row. It can find easy-to-overlook problems like not validating arguments of public methods, which can then lead to more unit tests and more coverage to achieve.
4. Automation
Take yourself out of the equation as much as possible. The point of a process is to be repeatable–it’s like automating yourself. Not only should unit testing be automated (thankfully, most testing frameworks handle this easily), but so should coverage and quality analyses.
What about deployment? Automate it. Documentation generation? CD master creation? Web upload? E-mail notification? Automate them all. Production builds should be invoked with a single command.
Working on boring, repeatable code? Automate it with code-gen.
The bottom line is: Don’t waste your brain cells on stuff that is highly repeatable, especially when it is prone to mistakes.
5. Code Review
Last week, a rather serious bug was discovered in some of our software (not released yet, thankfully, but close). The bug was mine, and I knew exactly what the problem was, but instead of designing a solution by myself, I brought a co-worker into the discussion just to bounce ideas off of. He had great suggestions, and made me think of things I might not necessarily have thought of on my own. We both went over the code and came to a solution that was simple and acceptable to both of us. The confidence level was much higher with this than it would have been otherwise.
This story is repeated daily by programmers throughout the world. Code review is a practice based on the simple notion that there is no one person smart enough to get it correct the first time.
Even if you’re working alone, which I often do, it pays huge dividends to regularly review your code with an eye for finding trouble. If you see any weakness at all, don’t ignore it–fix it. If you’re reviewing your own code, it’s a good idea to wait a bit after the time you wrote it. This gives your brain a chance to forget a little bit about it. Then, if you find you can’t understand it anymore, it’s either too complicated, or (if it fundamentally really is complicated) you need better comments.
Reviewing with other people has more benefit, however. Not everybody thinks the same way about problems. People have different experience, different expertise and focus, and you can’t take advantage of that if you don’t let them teach you. Even if the other people have less expertise than you, it is still beneficial (assuming they have some basic competency that they can bring to the discussion).
Once you let other people tear into your code (nicely, I hope), your confidence can be higher because you can add the confidence other people have in it (once your problems are corrected, of course!)
6. Repeatable Experiences
In the end, one of the best ways to increase your confidence in yourself, your code, and your practices is to have the evidence of repeated experiences behind you. You’re always learning, and that learning contributes to improvements in processes, testing, and your personal coding practices. Once you learn what works, especially during tricky upgrades, you can go into the next trial with increased confidence that you’re doing something right.
Have any other ideas on increasing confidence? Leave them in the comments!
Technorati Tags: process,confidence,unit testing