Therefore, you need to know the size of the data you’re going to serialize. There are a few ways to do this:

1. Measure the position you’re at, write the data, measure the new position, subtract, and that’s your length.
2. If you want to write the length first (which is usually better), you can write a dummy value, such as 0, then writing the data, then backing up in the stream, and writing the real value.
3. If you can’t back up the stream (very possible in some situations, or undesirable in others), you can measure the amount of data before you write. However, now you have to maintain that code in addition to the actual serialization.
4. My solution presented here, avoids having to maintain separate code by writing the data to a null stream which does not write any data, but keeps track of how much data was “written.”

class NullStream : System.IO.Stream
{
    public override bool CanRead { get { return false; } }

    public override bool CanSeek { get { return false; } }

    public override bool CanWrite {get { return true; } }

    public override void Flush() { /*do nothing*/ }

    public override long Length { get { return Position; } }

    private long _position = 0;
    public override long Position
    {
        get
        {
            return _position;
        }
        set
        {
            _position = value;
        }
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        throw new InvalidOperationException();
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        Position += count;
    }

    public override long Seek(long offset, System.IO.SeekOrigin origin)
    {
        throw new InvalidOperationException();
    }

    public override void SetLength(long value)
    {
        throw new InvalidOperationException();
    }
}

You can use it like this:

long GetDataSize()
{
    using (NullStream stream = new NullStream())
    {
        if (SaveData(stream))
        {
            return stream.Position;
        }
    }
    return 0;
}

There is a downside to something like this: you’re still essentially doing a lot of the work of serialization. Sure, you’re not writing out the bytes anywhere, but if, say, you need to encode a string as bytes before writing to the stream, that’s still going to happen.

Still, this technique made sense in my case, maybe it will work for you.

Here’s how it’s going to work:

1. Leave a comment on this post describing a project you’d like to build with C# 4.

2. I’ll pick two people from those comments at random.

(Make sure you enter your e-mail address where asked—it won’t be published to the blog, but I need it to contact you.)

I’ll leave the comments open for a while and I’ll update this post with the closing date.

Feel free to share a link to this blog post, tweet it, etc. If I get a lot of responses I may give away more.

Thanks to all those have already bought it!

UPDATE 18 May: I am going to close comments on Saturday morning (22 May) and pick the winners then. Thanks for commenting!

UPDATE 22 May: Comments are closed.

Well, it’s finally out! Amazon no longer lists the book as available for pre-sale, and it should be shipping to purchasers today or tomorrow. If you’re a B&N shopper, you can also order it there, or grab it in stores within a few days.

From the product description:

Real Solutions for C# 4.0 Programmers

Need fast, robust, efficient code solutions for Microsoft C# 4.0? This book delivers exactly what you’re looking for. You’ll find more than 200 solutions, best-practice techniques, and tested code samples for everything from classes to exceptions, networking to XML, LINQ to Silverlight. Completely up-to-date, this book fully reflects major language enhancements introduced with the new C# 4.0 and .NET 4.0. When time is of the essence, turn here first: Get answers you can trust and code you can use, right now!

Beginning with the language essentials and moving on to solving common problems using the .NET Framework, C# 4.0 How-To addresses a wide range of general programming problems and algorithms. Along the way is clear, concise coverage of a broad spectrum of C# techniques that will help developers of all levels become more proficient with C# and the most popular .NET tools.

Fast, Reliable, and Easy to Use!

• Write more elegant, efficient, and reusable code
• Take advantage of real-world tips and best-practices advice
• Create more effective classes, interfaces, and types
• Master powerful data handling techniques using collections, serialization, databases, and XML
• Implement more effective user interfaces with both WPF and WinForms
• Construct Web-based and media-rich applications with ASP.NET and Silverlight
• Make the most of delegates, events, and anonymous methods
• Leverage advanced C# features ranging from reflection to asynchronous programming
• Harness the power of regular expressions
• Interact effectively with Windows and underlying hardware
• Master the best reusable patterns for designing complex programs

I’ll be doing a book giveaway at some point as well, once I get my own shipment. Stay tuned!

Get it from Amazon

Get it from Barnes and Noble

My first book, C# 4.0 How-To is now shipping! If you like tips you can use, check it out!

From everything I’ve read, people who have suffered through the WPF learning curve have this to say, more or less:

yeah, it was really tough going for a few months. But now I can create awesome apps in a fraction of the time it would take with older technologies.

So with that in mind, I really do want to learn WPF. I have a number of C# references, weighty tomes that bend my shelves, but the main book I use is Programming WPFby Chris Sells and Ian Griffiths. I really like this book—it goes in deep. However, I realized that reading through it cover to cover and doing all the sample apps wasn’t going to work—it gets boring, no matter how good the book is. So here is my recommendation on how to learn WPF (and it probably applies to any programming technology):

1. Start reading the book, do the code, type stuff in, copy it, tweak it. Do this for as long as you can.
2. Once step 1 becomes boring, STOP. It is not productive to force yourself through the whole thing like this.
3. Find a sample project in your target technology. I used Family.Show. There are plenty out there.
4. Think of a project YOU find interesting that would be good in [WPF|other]. Start doing this. Even if you don’t know where to start at all.
5. While getting started, every step will be a challenge. Figure it out step-by-step, going back to the book and online resources.

You might be tempted to skip steps 1-2. I think this is a bad idea. You need at least some foundational understanding. Only when you can’t take it any more and you’re in danger of quitting, move on.

This has worked well for me in learning WPF. I decided to implement a game (if it ever gets into a polished state, I’ll share it).

Don’t underestimate the challenges in step 4, though. I had to think about how to even start, going back to the book numerous times, reading large sections. I looked up articles online about patterns and WPF, user controls, and more. Many seemingly-small steps in just displaying windows took hours to figure out. Figuring out data binding (really figuring it out in the context of my app) took hours. The point of doing your own project isn’t because it’s easier than following the book—it’s because it’s fun and you have more motivation to learn.

My first book, C# 4.0 How-To is now shipping! If you like tips you can use, check it out!

Here’s a brief run-down of my initial experience with it.

Installation

There is no installation file—everything is packaged into a zip. After running, I was greeted by a project selection screen, in which I created a new project and added some assemblies.

Analysis

Once you have all the assemblies you want to analyze selected, you can run the analysis, which generates both an HTML report with graphics, and an interactive report that you can use to drill down into almost any detail of your code. Indeed, it’s almost overwhelming the amount of detail present in this tool.

One graph you see almost immediately is Abstractness Vs. Instability.

This is a good high-level overview of your entire project at the assembly level. Basically, what this means is that assemblies that are too abstract and unstable are potentially useless and should be culled, while assemblies that are concrete and stable can be hard to maintain. Instability is defined in the help docs in terms of coupling (internal and external), while abstractness is the ratio of abstract types to total types in an assembly.

This is followed by the dependency graph:

After these graphics come lots of reports that dig into your code for all sorts of conditions.

For example, the first one in my report was “Quick summary of methods to refactor".” That seems pretty vague, until you learn how they determine this. All the reports in NDepend are built off of a SQL-like query language called CQL (Code Query Language). The syntax for this is extremely easy. The query and result for this report are:

With very little work on my part, I instantly have a checklist of items I need to look at to improve code quality and maintainability.

There are tons of other reports: methods that are too complex, methods that are poorly commented, have too many parameters, to many local variables, or classes with too many methods, etc. And of course, you can create your own (which I demonstrate below).

Interactive Visualization

All of these reports are put into the HTML report. But as I said, you can use the interactive visualizer to drill down further into your code.

The first thing you’re likely to see is a group of boxes looking like this:

These boxes show the relative sizes of your code from the assembly level down to the methods. Holding the mouse over a box will bring up more information about the method. You can also change the metric you’re measuring by—say to cyclomatic complexity.

Another view, perhaps the most useful of all is the CQL Queries view. In this, you can see the results from all of hundreds of code queries, as well as create your own. For instance, I can see all the types with poor cohesion in my codebase:

In this view, the CQL queries are selected in the bottom-right, and the results show up on the left. The metrics view highlights the affected methods.

Creating a query

Early in the development of my project, I named quite a few classes starting with a LB prefix. I’ve changed some of them, but I think there are still a few lying around and I want to change them as well. So I’ll create CQL query to return all the types that begin with “LB.”

   1: // <Name>Types beginning with LB</Name>

   2: WARN IF Count > 0 IN SELECT TYPES WHERE 

   3:  NameLike "LB" AND     

   4:  !IsGeneratedByCompiler AND 

   5:  !IsInFrameworkAssembly     

That’s it! You can see the results to the right. It’s ridiculously easy to create your own queries to examine nearly any aspect of your code. And that’s if the hundreds of included queries don’t do it for you. In many ways, the queries are similar to the analysis FxCop does, but I think CQL seems generally more powerful (while lacking some of the cool things FxCop has).

VS and Reflector Add-ins

NDepend has a couple of extras that enable integration of Visual Studio (2005 and 2008) and NDepend and Reflector. When you right-click on an item in VS, you will have some additional options available:

Clicking on the submenu gives you options to directly run queries in NDepend. Very cool stuff.

Summary and where to get more info

If you are at all interested in code metrics, and how good your code is behaving, how maintainable it is, you need this tool. It’s now going to be a standard part of my toolbox for evaluating the quality of my code and what parts need attention.

If you’re using NDepend for personal and non-commercial reasons, you can download it for free. It doesn’t have all the features, but it has more than enough. Professional use does require a license.

One of the things I was particularly impressed with was the amount of help content available. There are tons of tutorials for every part of the program.

I’m going to keep playing with this and I’m sure I’ll mention some more things as I discover them. For now, NDepend is very cool—it’s actually fun to play with, and it gives you good information for what to work on.

Links:

• NDepend homepage
• NDepend metrics explanation

Half way through the training, while discussing something tangentially related to my problems, I realized the technical solution to the specific issue I was facing.

Sometimes you just need to ignore the problem for a while, and come back to it from a different angle.

A computer that records and correlates conversations real-time must have a phenomenal ability (compared to our current technology) to not just syntactically parse the content, but also construct semantic models of it. If a computer is going to respond intelligently to you, it has to understand you. This is far beyond our current technology, but we’re moving there. In 20 years who knows where this will be. In 100, we can’t even imagine it. 400 years is nearly beyond contemplation.

The philosophy of computer understanding, and human-computer interaction specifically is incredibly interesting. I was led to think a lot about this while reading Robert Glass’s Software Creativity 2.0. This book is about the design and construction of software, but it has a deep philosophical undercurrent running throughout that kept me richly engaged. Much of the book is presented as conflicts between opposing forces:

• Discipline versus Flexibility
• Formal Methods versus Heuristics
• Optimizing versus satisficing
• Quantitative versus qualitative reasoning
• Process versus product
• Intellectual versus clerical
• Theory versus practice
• Industry versus academe
• Fun versus getting serious

Too often, neither one of these sides is “right”—they are just part of the problem (or the solution). While the book was written from the perspective of software construction, I think you can twist the intention just a little and consider them as attributes of software itself, not just how to write it, but how software must function. Most of those titles can be broken up into a dichotomy of Thinking versus Doing.

Thinking: Flexibility, Heuristics, Satisficing, Qualitative, Process, Intellectual, Theory, Academe

Doing: Discipline, Formal Methods, Optimizing, Quantitative, Product, Clerical, Practice, Industry

Computers are wonderful at the doing, not so much at the thinking. Much of thinking is synthesizing information, recognizing patterns, and highlighting the important points so that we can understand it. As humans, we have to do this or we are overwhelmed and have no comprehension. A computer has no such requirement—all information is available to it, yet it has no capability to synthesize, apply experience and perhaps (seemingly) unrelated principles to the situation. In this respect, the computer’s advantage in quantity is far outweighed by its lack of understanding. It has all the context in the world, but no way to apply it.

A good benchmark for a reasonable AI on the level I’m dreaming about is a program that can synthesize a complex set of documents (be they text, audio, or video) and produce a comprehensible summary that is not just selected excerpts from each. This functionality implies an ability to understand and comprehend on many levels. To do this will mean a much deeper understanding of the problems facing us in computer science, as represented in the list above.

You can start to think of these attributes/actions as mutually beneficial and dependent, influencing one another, recursively, being distinct (at  first), and then morphing into a spiral, both being inputs to the other. Quantitative reasoning leads to qualitative analysis which leads back to qualitative measures, etc.

It made me think of Douglas R. Hofstadter’s opus Godel, Escher, Bach: An Eternal Golden Braid. This is a fascinating book that, if you can get through it (I admit I struggled through parts), wants you to think of consciousness as the attempted resolution of a very high-order strange loop.

The Strange Loop phenomenon occurs whenever, by moving upwards (or downwards) through the levels of some hierarchical system, we unexpectedly find ourselves right back where we started.

In the book, he discusses how this pattern appears in many areas, most notably music, the works of Escher, and in philosophy, as well as consciousness.

My belief is that the explanations of “emergent” phenomena in our brains—for instance, ideas, hopes, images, analogies, and finally consciousness and free will—are based on a kind of Strange Loop, an interaction between levels in which the top level reaches back down towards the bottom level and influences it, while at the same time being itself determined by the bottom level. In other words, a self-reinforcing “resonance” between different levels… The self comes into being at the moment it has the power to reflect itself.

I can’t help but think that this idea of a strange loop, combined with Glass’s attributes of software creativity are what will lead to more intelligent computers.

I had been planning on writing a blog entry on the apparently sad state of our CS industry these days, and the complete lack of qualified American resumes that come across my desk, when we actually got a decent one today.

Still, there is much to be said about the poor quality of education. At some points, we’ve gone through dozens of candidates that had such weak skills that I’m surprised they graduated from a reputable institution.

Then I noticed on the Google blog today a new initiative to partner with CS programs around the country. I applaud this effort and all like it. Serious companies like Google, Microsoft, Yahoo, Amazon, Adobe, Apple, and everybody else who builds amazing software need to get involved and lay down the expectations.

Does anyone know if Microsoft has a similar comprehensive program? I know that they have MSDNAA, but that seems more like giving software in a marketing campaign than setting the agenda. I do see smaller efforts with robotics that are great, but a more general push is needed.

I consider myself lucky for having gone through a fantastic computer science education at BYU. I found it even much better than my graduate program.

The thought leaders need to start insisting on higher standards, and we need to shame schools that churn out useless bodies whose jobs will soon be outsourced.

I can see some pushback from academic circles because they won’t want big business telling them how to teach, but the reality is that when schools are doing such a poor job of preparing people, they need to change and listen to those who are going to be hiring.

We need to stop complaining and start changing the situation.

1. The opportunity to work with people smarter than me. The chance to meet some of the people I admire in the software community.
2. The projects and technology under development always inspire me. Almost every event I’ve gone to has had me come away wanting to look into some other cool technology and thinking of the ways it can change the world.
3. A real career path as a software engineer.
4. Chance to change projects whenever I want. During my interviews, many people were quite open with me: they get bored with a project eventually and want to switch after two years or so. Microsoft’s culture easily allows this.
5. Compete with Google. Google needs some real competition. Just as Firefox lit a fire under the IE team, MS needs to light a fire under Google.
6. The benefits are awesome. They truly treat you well.
7. They are extremely open on telecommuting.
8. How many companies can you work at where your stuff affects so many people? There aren’t that many…
9. The challenge. I love challenges. I love learning new things, and working hard to solve problems. Challenges are how you grow.
10. The area. Beautiful country. Cheaper than DC. The rain.

What will be more interesting is to compare this list with what I come up with in a year.

If you are following the instructions to build log4cxx 0.10 in Visual Studio 2005, and you have the Windows Platform SDK v6.0 installed, you may get errors compiling multicast.c in the apr project.

I found the solution, and it’s pretty easy. Open up multicast.c and edit the lines:

136: #if MCAST_JOIN_SOURCE_GROUP

148: #if MCAST_JOIN_SOURCE_GROUP

to be, instead:

136: #if defined(group_source_req)

148: #if defined(group_source_req)

e voilà! now it compiles.

Change is a fact of life. Nowhere is this more obvious in a medium where the very thing we make is completely intangible and malleable: software development. There is almost nothing that is impossible in software–there are only limited resources.

This malleability has been one of the reasons for the enormous pace of change we’ve seen in computers in the last half century, but at the same time it’s been a stumbling block to true engineering practices.

Regardless of philosophy, change will happen in all software projects to some degree. There are two extremes of development that most organizations fall between:

• A single coder hacks together whatever the boss needs changed today. (Also fixes bugs from yesterday’s fixes.)
• NASA’s awesomely rigid development practices for the Space Shuttle software (see the Manager’s Handbook for Software Development on the Book list) resulting in extremely high-quality, virtually bug-free software (at a very high price).

For most of us, neither extreme is appropriate. Deciding the right balance of change control is a hard problem, but it’s not impossible. Still, some organizations manage to do so badly at it that it deserves to be studied and dissected.

The Census

As an example, consider the 2010 Census, a project that has been covered in the press and on tech blogs over the last few months. The following is from the second paragraph of the report on the GAO’s study of this massive program:

In October 2007, GAO reported that changes to requirements had been a contributing factor to both cost increases and schedule delays experienced by the FDCA [Field Data Collection Automation] program. Increases in the number of requirements led to the need for additional work and staffing. In addition, an initial underestimate of the contract costs caused both cost and schedule revisions. In response to the cost and schedule changes, the Bureau decided to delay certain system functionality, which increased the likelihood that the systems testing…would not be as comprehensive as planned.[...] Without effective management of these and other key risks, the FDCA program faced an increased probability that the system would not be delivered on schedule and within budget or perform as expected. Accordingly, GAO recommended that the FDCA project team strengthen its risk management activities, including risk identification and oversight.

The Bureau has recently made efforts to further define the requirements for the FDCA program, and it has estimated that the revised requirements will result in significant cost increases.

As with many government projects, there is enough failure to go around. The key idea I want to focus on is a failure on the part of the government agency and on the part of the contractor implementing the technology to adequately plan for change.

Before that, though, I want to say that I have no knowledge of their specific management or technical practices, other than what is publicly reported. For all I know, all parties performed extremely well and still failed. (However, from friends and acquaintances in government and contractor positions around D.C., I think it’s fair to say that most organizations here are behind the curve in software best practices, to put it politely.)

With that (admittedly harsh) assumption, it’s apparent that the agency failed to adequately plan, think through requirements, and communicate effectively to the contracted company. On the developer’s side, it’s apparent they didn’t have effective mechanisms for handling changes.

Before a project is undertaken, a set of fundamental principles must be agreed upon:

1. The client will change their minds. There is no such thing as “written in stone.”
2. Making changes is hard or impossible.

These two principles are fundamentally in conflict. On the one hand, the developer must know that changes are going to occur: sometimes large changes. They can plan for that from the code level up to their organization and processes.

On the other hand, the client, must realize that they can’t just suggest a change at the end of the project and expect it to be thrown in and work correctly.

The tension between these two principles implies certain practices and expectations on all parties, which in an ideal world would lead to a successful outcome more often.

Developer Responsibilities

The developer, knowing that the client does not understand the full set of requirements up front, designs their system in a way that can easily handle change. This means modularity, this means testability. From details as small as designing methods to operate on Streams instead of filenames, to as large as easily reusable components.

The first step is a change in mind set: an acceptance that change will occur, and that it’s ok. When I first started development, frequent changes really frustrated me. Sometimes they still do, but I’m getting better (I hope). Once the mind is malleable, practices can be implemented to handle it effectively.

A few coding practices that encourage changes:

• Test-driven development (or at least full unit testing). In this day and age, with how easy it is to accomplish this, is there a good reason not to do it? I’m not convinced by many arguments against unit testing. A full suite of unit tests gives you confidence in your code, and the ability to safely refactor even large portions of your program. Full TDD I could go either way.
• Constant refactoring to fit the best design possible (within reason. Obviously, there is no “best” design). Software, like everything else in the universe, obeys the law of entropy. Without constant maintenance, it will degrade. Designs degrade as you add on to them. Classes degrade as you stuff them full of things. Don’t stand for it–take the time to make things neat. Get some good books on refactoring if you don’t know how to start.
• Highly Modular. Apply all your OO-design skills here. It matters. Cohesion and coupling really do matter.
• Small, frequent iterations. Most software systems these days are too big and too complex to understand at once. Iterative development is the key to success.
• Strong source-control practices. Does this even need to be discussed? If your organization doesn’t have good source control, you’re doing software wrong.

Other practices on the developer’s part read like a list of Extreme programming fundamentals:

• Communication – The key to any relationship, whether personal or corporate, is communication. The right amount should be discovered and adhered to. It should probably be slightly more than you’re comfortable with (since many developers would rather not talk to anyone .
• Simplicity – I believe that the single hardest thing to do in developing software is to manage complexity. It’s requires more brainpower and creativity than all other activities. Coding is easy. Coding so that it is easy to understand a year from now is difficult. If the design is needlessly complex up front, then change is that much harder to implement.  Always remember that software complexity increases exponentially.
• Courage – Stand up for correct principles. Don’t allow politics to interfere with what needs to be done. Don’t be afraid to change the design if it needs to happen. Change coding practices as needed. Never be afraid of the truth–just deal with it.
• Respect – Both sides need to remember that the other is expert in their domain. Humility is key.
• Responsibility – this is something I fear is lacking much in government. Everybody needs to take responsibility for their part. This implies accountability. This can be shared responsibility at some levels, or the-buck-stops-here responsibility at higher levels. If no one is responsible, nobody cares if it fails. Courage is a prerequisite to this.

This mind set needs to encompass not just pure software development, but also processes at the organizational level. This can take many forms, but a few ideas:

• Personal relationships between clients and developers
• Flexible team structure and members
• An easy way to submit and discuss change requests
• A culture that accepts changes
• A good understanding of the problem domain

Jeremy Miller (The Shade Tree Developer) talks a lot about processes and practices and it is a very good read.

Unfortunately, there exists an attitude at many companies that it’s better to milk the clients for as much money as possible rather than do the hard work of getting a good process.

Client Responsibilities

I think my introduction and the discussion about developer responsibilities may imply that because software is so ephemeral, it is therefore easy to change it. After all, it’s not a physical object that has to be broken down and rebuilt–what’s so hard? To anyone who has worked on non-trivial projects, this is laughable. Let’s clarify:

It is easy to program. It’s painfully, mind-bendingly, insanely difficult to design software. So difficult, in fact, that no one can do it well. The sooner everybody understand that, the sooner we can get started with real work. Good design is thought-work, iterated over and over, added to experience and analysis.

Some details are easy to change. Others are fundamental to the structure of the building. By way of metaphor, changing your mind about a house you’re building to suggest a stone facade instead of brick is relatively easy. Deciding you would like to have ten stories added onto your house changes the game significantly. Now you have to build a completely new foundation.

I believe the customer’s responsibility is to gain knowledge and insight into the development practice so they understand why things are difficult. Computers are so fundamental to our culture, we can no longer afford to have companies and agencies run by people who don’t understand them to some degree. The exact degree is debatable, but it’s certainly higher than where we’re at now. In my own situation, I think that when I started my boss did not realize how seemingly-trivial changes could be phenomenally difficult to implement, and therefore bug-prone–all because of design decisions made earlier. I now have him understanding that there is no such thing as a simple change.  (Of course, I still occasionally underpromise and overdeliver–I have to maintain my miracle-worker image after all.

A client who wants software done and doesn’t want to end up paying an extra $3,000,000,000 for software should understand that critical requirements can’t be left until the product is being delivered. “We didn’t think of it before” is not a good excuse.

A client who understands this will realize how critical initial requirements are and ensure they’re communicated clearly. Most of all, they don’t wait around twirling their thumbs, only to reject the system when it’s done because it doesn’t meet their requirements. They will insist on periodic feedback and demos to make sure things are going the right way.

The responsibility for huge failures is on everybody’s shoulders. I believe that it can often be traced back to an unwillingness to face up to the truth, lack of responsibility, and misunderstanding the nature of the project.

Large projects aren’t doomed to fail

There is a fundamental truth, however, of all projects: there exists at least one change that is too large to do. Maybe the Census required that change–I don’t know. I do know that if both sides follow the best practices they can, the number of must-fail changes can be minimized.

That said, here’s why I refuse to believe the census project wasn’t sorely mismanaged from the outset by all sides: FedEx, UPS, Amazon, Wal-Mart, Microsoft, and dozens (hundreds?) of other companies have built enormous and complex systems for managing large databases and mobile platforms involving thousands of partners. It is ridiculous to me that they could not have built a system for our government quicker, cheaper, and less epic failure than the one they got. If the Census is fundamentally more complex than the problems they deal with, someone please let me know.

Change is part of everything now

The ability to adapt to change is critical, not just for software developers, but for everyone in our society. Technology, events, trends, and money all flow so quickly now that those who can adapt quickly will succeed.

It’s not just software companies that need to adopt sound development practices. There will always be companies that build out our infrastructure (Microsoft, Google, Cisco, etc.), but in the end every company will be a software development company. Disregarding good practices because “we’re not a software” does not work anymore. It’s not true. You are a software company. We all will be.

Idealism or…

If a lot of this seems idealistic, that’s ok. I wish the world were like Star Trek as much as the next geek, but I realize that there are a lot of motivation$ out there. The world is what it is, but if we aren’t trying to improve it by working towards a standard, then what’s the point? We can improve it. There is no excuse for such catastrophic wastes of time, money, and effort.

It is easy to contrast this book with Beck’s Test Driven Development: By Example, and the two books definitely have a very different feel.

Beck’s book has a very evangelical feel to it, and it’s main purpose is to teach a mind set more than technical details. I believe this is important–maybe of first importance–but once you understand that, the rest of the book is a little simplistic for more experienced developers.

Pragmatic Unit Testing, on the other hand, focuses much more on the practical aspects (hmmm….I highly suspect that’s where the title comes from….) of unit testing. I liked the ideas on how to use categories and attributes to segregate tests that take too long to run on a regular basis. I also liked the section on singletons and getting around time-dependencies. The DateTime.Now problem is something I’ve had to deal with quite a bit in our server-side software that has a lot of time-dependant behavior. (In most cases, the problems were solved with refactoring the time into a function parameter.)

There are also good discussions of more mundane issues like how to deploy NUnit, where to put tests in a project, team practices, GUIs, threading, and C#-specific issues.

The discussion about mock objects (a very basic introduction) is also quite clear and understandable–more so than many resources I’ve seen on the web, which often assume you already know all about them.

Something I don’t like: the acronyms (BICEP, CORRECT, A TRIP). They kind of bug me. I like the ideas behind the acronyms and I think it’s more important and effective (for me, anyway) to internalize the principles of testing rather than remembering specific acronyms and the words they go with. YMMV.

Last Word…

I will probably only read Test Drive Development: By Example once,  but I will definitely come back to Pragmatic Unit Testing occasionally to refresh my ideas.

Using just these links as a guide, we quickly (less than a day) put up a data source over https for our customers to download GIS data to Google Earth.

KML reference and tutorial – KML is the XML language used to describe features that can be displayed in Google Earth and Google Maps.

Sample code to generate KML from a web service or web page (it’s VB.Net).

How to implement a custom authentication provider for IIS in .Net. Very useful if you need to authenticate your KML-generating web page and you don’t want to use Active Directory.

(click for larger image)

QBASIC is not included in Windows anymore, but you can still get it.

(click for larger image)

I had also tried modifying the including GORILLAS.BAS and NIBBLES.BAS, but I was still a little too new at this.

When I was 14 I was getting into C/C++ via Borland C++ 3.1 in a big way and spent hours coding prank programs. I had two that I remember:

First was a program called Camels that displayed “I Love Camels!!!” in a vertical, colorful scrolling sine wave down the screen. It trapped Ctrl-Break/Ctrl-C so you couldn’t break out of it. If you hit Ctrl-K, it brought up a password screen that allowed you to exit if you knew the password. Then I put it on a lab at school, set AUTOEXEC.BAT to run it, and modified CONFIG.SYS with “switches /n” to disallow the user hitting F5 to skip processing of AUTOEXEC.BAT. This stunt kind of got me in trouble–the day after school ended, I got a call from my computer science teacher that he couldn’t access the computer and if I wanted a grade I had better get over there and remove that program because he couldn’t get onto the computer. So I had to bike a few miles to school (my parents were out of town) and remove it. Why didn’t the instructor just use a boot disk? No idea… By the way, I got an A.

One of my first Windows programs was something called “Chucky” (why? I don’t know…). Chucky liked to eat….hard disk space. He would startup with no Window, run in the background, and every few minutes it would add a few thousand lines of text to  the file C:\Windows\Chucky.txt. It was probably something like “I am Chucky, I am hungry.”

I even eventually convinced my parents to get me Turbo C++ so I could build Windows programs (suing OWL).

When I was in college, I got Visual C++ 6 and thought a fun program would be a desktop utility that occasionally changed your Outlook signature to include a random quotation. You could build up a little database of quotes you liked, and the program would change it on a regular schedule. A friend of mine and I stayed up for nearly 3 days straight working on it. I did most of the programming–he was thinking of new ideas, ways to do things. It was great fun.

These important formational periods are what got me excited about programing. The learning that goes on during a 72-hour hacking session is something that can’t be duplicated in a classroom. The glee at creating pranks is not matched (often) by homework assignments. Sometimes when I’m feeling the drudgery of the current code I work on, I need to remember the excitement I felt back then.

I also need to find something equivalently exciting to work on. One of the things I’m going to do to “get the magic back” (so to speak) is to make sure I’m always experimenting with the latest and great .Net stuff coming out. I need to finally get into WPF, and I’ve even got a fun project to apply it to.

Sample Database

First, let’s conceptualize a very simple database consisting of user information (name, date of birth), and e-mails. A user can have more than one e-mail.

We want to track all changes to the FirstName, LastName, and birthdate fields. In addition we want to track when e-mails are added or removed from a user. As we’ll see, these aims are accomplished using two different methods.

My implementation is done in SQL Server 2000 and C#, but any database that supports triggers can be used.

Changes in a Single Table

With this method we want to track the changes to all fields of a table. In our example, we want to know when FirstName, LastName, and birthdate change values in the UserData table.

To accomplish this we need another table to track the history. This table is going to have the exact same fields as UserData, plus a few extra for the change tracking.

Table: UserDataChanges

Now the automated part–adding a trigger to populate this automatically:

CREATE TRIGGER UserDataChangeTrigger ON UserData FOR UPDATE, INSERT

AS    

    IF (UPDATE (FirstName) OR UPDATE(LastName) OR UPDATE(birthdate))    

    BEGIN     

        INSERT UserDataChanges 

            (ChangeTime, ChangeUser, ID, FirstName, LastName, birthdate)

            (SELECT GetUtcDate(), user, ID, FirstName,LastName,birthdate 

                FROM inserted)     

    END     

This trigger will insert a new row into the UserDataChanges table whenever a row in the UserData table is updated or inserted. The IF (UPDATE(FirstName)…. ) is not strictly required in this scenario, but in other cases I did not want a change recorded when certain fields were updated (i.e., you have a field that tracks the last change time of that row, or the number of orders, or any other field that can change frequently and isn’t important to track–you don’t want to create too much noise in this or it will not be useful). The GetUtcDate() and user are SQL Server functions that retrieve the current UTC time and the username of the process that caused the change–very useful for tracking responsibility. The inserted table is created by the server for use by the trigger and contains all the new values.

Changes in a Foreign Key Table

The UserEmails has to be handled differently because there can be multiple e-mails for each user and we can assume they can be added, or removed at will (Remove + Add = Update, so I won’t consider direct updates here).

The solution I landed on was to have a generic event log table that stores manual log entries as well as “special” entries denoting adding or removing e-mails.

Table: UserEventLog

This table can be used for both adding text notes to a user and, by using the EventType field, special events. In our example, we have two events we need to track:

(In code, I’ve made these enumerations)

Next we add a trigger on the UserEmails table:

CREATE TRIGGER UserEmails_EmailAddedTrigger

ON UserEmails

FOR INSERT

AS

 BEGIN

     INSERT UserEventLog(ID, EventTime, EventType, ChangeUser, Notes)

        (SELECT ID, GetUtcDate(), 1, user, '{'+email+'}' FROM inserted)

 END

The value 1 stands for EmailAdded. I’ve added braces around the actual e-mail address to set it apart from regular notes (we’ll see how to integrate everything later).

To handle the deletion of e-mails add another trigger:

CREATE TRIGGER UserEmails_EmailRemovedTrigger

ON UserEmails

FOR DELETE

AS

 BEGIN

     INSERT UserEventLog(ID, EventTime, EventType, ChangeUser, Notes )

        (SELECT ID, GetUtcDate(), 2, user, '{'+email+'}' FROM deleted)

 END

The only things different: FOR DELETE (instead of INSERT), changed the EventType to 2 (EmailRemoved), and the values are taken from the SQL Server-supplied deleted table.

That’s enough to get a pretty good change-tracking system in place, but you’ll still have to build a UI to display it effectively.

Displaying the Changes in the UI

With the above work done, you end up with two types of entities: changes and events. While it would be possible to integrate all functionality into a single event/change table using a lot more logic in the SQL Trigger code, I’m personally more comfortable with the change logic being in my application code. I think this way the database is kept more “pure” and open to changes down the line.

That means we will need to integrate these two types of entities into a single list, ordered by date/time. I’m going to assume the existence of two classes or structs that represent each of these entities. They’ll be called UserChange and UserEvent. I’ll also assume that the lists of each of these are already sorted by time, since that’s trivial to do in a SQL query.

Given that, we need a function that takes both of these lists and produces a sorted, combined list with an easy-to-understand list.

How the function works:

1. Go through both lists, and pick whichever one is next, time-wise.
2. Translate the object into a string/list-view representation of that object.
3. If it’s a UserChange object, compare it to the previous one to figure out what changed.
4. Sort the list in reverse order to put newer items at the top.

Here’s the C# code which I’ve adapted from our production system. Don’t get hung up on the details:

private void FillLog(IList<UserEvent> events, IList<UserChange> changes)

{

    List<ListViewItem> tempItems = new List<ListViewItem>();

    int currentEventIdx = 0;

    int currentChangeIdx = 0;

    eventLogListView1.Items.Clear();

    while (currentEventIdx < events.Count

    || currentChangeIdx < changes.Count)

    {

    UserChange currentChange = null;

    UserChange prevChange = null;

    UserEvent currentEvent = null;

    DateTime changeTime = DateTime.MaxValue;

    DateTime eventTime = DateTime.MaxValue;

    if (currentChangeIdx < changes.Count)

    {

        currentChange = changes[currentChangeIdx];

        changeTime = currentChange.ChangeDate;

        if (currentChangeIdx > 0)

        {

        prevChange = changes[currentChangeIdx - 1];

        }

    }

    if (currentEventIdx < events.Count)

    {

        currentEvent = events[currentEventIdx];

        eventTime = currentEvent.EventDate;

    }

    string dateStr;

    string userStr;

    string eventTypeStr="";

    string notesStr;

    if (changeTime < eventTime)

    {

        dateStr = Utils.FormatDateTime(changeTime);

        userStr = currentChange.UserName;

        notesStr = GetChangeString(currentChange, prevChange);

        currentChangeIdx++;

    }

    else

    {

        dateStr = Utils.FormatDate(eventTime);

        userStr = currentEvent.UserName;

        notesStr = currentEvent.Notes;

        eventTypeStr = currentEvent.EventType.ToString();

        currentEventIdx++;

    }

    if (notesStr.Length > 0)

    {

        ListViewItem item = new ListViewItem(dateStr);

        item.SubItems.Add(userStr);

        item.SubItems.Add(eventTypeStr);

        item.SubItems.Add(notesStr);

        item.ToolTipText = notesStr;

        item.BackColor = (tempItems.Count % 2 == 0) ? 

            Color.Wheat : Color.White;

        tempItems.Add(item);

    }

    }//end while

    eventLogListView1.BeginUpdate();

    for (int i = tempItems.Count - 1; i >= 0; i--)

    {

    eventLogListView1.Items.Add(tempItems[i]);

    }

    eventLogListView1.AutoResizeColumn(0, 

        ColumnHeaderAutoResizeStyle.ColumnContent);

    eventLogListView1.AutoResizeColumn(1, 

        ColumnHeaderAutoResizeStyle.ColumnContent);

    eventLogListView1.AutoResizeColumn(2, 

        ColumnHeaderAutoResizeStyle.ColumnContent);

    eventLogListView1.Columns[3].Width = eventLogListView1.Width - 

    (eventLogListView1.Columns[0].Width +

    eventLogListView1.Columns[1].Width +

    eventLogListView1.Columns[2].Width +10);

    eventLogListView1.EndUpdate();

}

Now we need to define GetChangeString, which figures out the differences in successive UserChange objects and displays only pertinent information.

private string GetChangeString(

    BuoyDataChange currentChange, 

    BuoyDataChange prevChange)

{

    StringBuilder sb = new StringBuilder();

    if (prevChange == null)

    {

        CompareAndAdd(sb, "First Name", 

            null, currentChange.FirstName);

        CompareAndAdd(sb, "Last Name", 

            null, currentChange.LastName);

        CompareAndAdd(sb, "Birth Date", 

            null, currentChange.BirthDate);

    }

    else

    {

        CompareAndAdd(sb, "First Name", 

            prevChange.FirstName, currentChange.FirstName);

        CompareAndAdd(sb, "Last Name", 

            prevChange.LastName, currentChange.LastName);

        CompareAndAdd(sb, "Birth Date", 

            prevChange.BirthDate, currentChange.BirthDate);

    }

    return sb.ToString();

}

And one last helper function which compares two objects and if different appends the change to a StringBuilder object.

private void CompareAndAdd(StringBuilder sb, string field, 

    object oldVal, object newVal)

   {

       if (oldVal == null && newVal == null)

           return;

       if (oldVal == null || !oldVal.Equals(newVal))

       {

           if (sb.Length > 0)

           {

               sb.Append(", ");

           }

           sb.AppendFormat("{0}:{1} -> {2}", field, oldVal, newVal);

       }

   }

In this way you can end up with an automated system that displays all changes in an easy-to-understand format.

Here’s a sample of what our system looks like (click to enlarge):

Other ways to accomplish this? Better ways? Please leave a comment!

Problem: Visual Studio 2005 solutions no longer open when you double-click them in Windows Vista. In fact, when you double-click nothing happens.

Solution: Change them to open with Visual Studio 2005 directly instead of the vslauncher.exe (which opens up the solution with the correct version of Visual Studio if you have more than one).

Caveat: Only makes sense if  you use only Visual Studio 2005.

How-to:

1. Right-click on a solution file.
2. Choose “Open With…”
3. Choose “Browse…”
4. Browse to file “C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\devenv.exe” (or wherever you installed Visual Studio)
5. Click “Open” button
6. Check “Always use the selected program to open this kind of file”

Now your solutions will load Visual Studio, bring up the UAC prompt, and it all works great.

Found via here and here.

My first book, C# 4.0 How-To is now shipping! If you like tips you can use, check it out!

Definition

Any discussion of time-wasting is profitless unless you define what wasting time is. My definition is:

Wasting time is doing anything that does not contribute to my goals.

That is a very broad definition, but it is very useful. It presupposes a goal-oriented mind set and I don’t want to get too far down that path here. If you’re really interested in a goal-focused system, I highly encourage you to read The 7 Habits of Highly Effective People, which is a principles-based approach to effectiveness and goal-setting is a huge part of it.

Whatever your personal system, much of the corporate world and building software specifically revolves around goals (aka “milestones”, “targets”).

This definition includes taking breaks and eating lunch, but let’s not be silly–we’re not talking about that and I’m not going to water down the definition nor am I going to spend pages talking pointlessly about exceptions to it. We’re all intelligent people here and can understand the important principles.

Many Small Goals Are Better Than One Large One

Software development processes have been undergoing evolution since the beginning and lately the whole agile process seems to be taking over. Whatever the process, many companies are finding more success in breaking down large projects into tiny goal-driven chunks, sometimes lasting as little as a week.

This same principles can be applied to ourselves at both large and fine-grained levels. It is definitely good and desirable to have the overall vision of our project in mind, but  this doesn’t often help us get the work done. Some of my most productive days are when I break down a huge task into tiny subtasks and set a goal for each one (“I will have this done by 11am today, then I will wrap up this other small one by 4pm.”)

An example: I’m currently writing some code to move a huge amount of data around in our production database. We’re going to be rolling out a major update that requires some fundamental changes to how things operate. This task is so large and daunting that I get a headache just thinking about it and so I could put it off, just spinning my wheels until I decide to face the inevitable. Instead, I’ve broken it into several smaller tasks that are each easily managed and understood.

(In my example, the After column actually contains about 30 items, depending on how far I want to break it down…it could be more.)

Now, instead of being overwhelmed by the complexity of a large task (and thus doing nothing), I can easily handle each of the sub-steps efficiently. I’ve changed a two-week task into many hour-long tasks.

The psychological effect of  too-hard/too-complex/too-much is devastating. You can’t handle something like that–no one can, and so you won’t–you’ll just end up wasting time fretting about it. Break it up for your sanity and happiness, as well as productivity.

Motivation is Crucial

Often, a key to not wasting time is having sufficient motivation. Motivation can come in all sorts of ways–the key is to figure out what motivates you and then set yourself up to succeed by using that motivation as a carrot to pull you forward. It can be a good idea to share your motivations with managers so they understand what drives you.

Motivation can often begin with picking good goals. If your goals are unrealistic, you are almost guaranteed to fail in some way. Despair feeds on itself and will sink your productivity and cause you to engage in anything but work. Not only will you avoid the drudgery of work, but you won’t take steps to improve yourself or change the situation. This cycle must be broken immediately.

If your projects are just not that interesting this can be a challenge. Everybody has tasks they don’t particularly like, but if the majority of your time is spent doing stuff you get no pleasure out of, you are doing a huge disservice to yourself and your future. Eventually, you’ll become wasted and useless to both yourself and your employer. Fix the situation–get a new project, get a new job, find side projects to do that you do enjoy as rewards for getting through the drudgery–anything to avoid becoming the shell of a person you once were.

Maybe you don’t necessarily need a new job right now–maybe you just need to fix the situation at your current job, get some enjoyable hobbies at home, spend more time with the family. The needed changes aren’t always drastic–but figure them out so you don’t spend every day wallowing in a mire not doing anything useful.

Eliminate Distractions Now

I don’t think I’ve answered my office phone in about a year. Not that many people call it in the first place, e-mail being highly-preferred around here, but I like to say I stand on principle. You can read a lot about creating the right environment for highly-skilled software developers in the fabulous book Peopleware: Productive Projects and Teams.

I’ve also stuck to the practice of keeping my e-mail inbox cleared. I delete almost everything I receive unless I need to store it for later or act on it.

In my entry about working an 8-hour day, I talked about various time-wasting activities that I’ve observed, such as  wandering the halls, micro-managing, too many words, poor inbox management skills, and more.

The goal of eliminating distractions is not to completely choke off any aspect of fun, diversion, and other social aspects of working in an office–those are good things. The goal is eliminate the things that don’t help us in our jobs and that aren’t really all that enjoyable to us anyway, all things considered.

One way of eliminating your distractions is to go out of your way ahead of time to manage them so they don’t come up later. Some ideas:

• If you need somebody to do a task, anticipate questions, concerns, or problems they may have. Try to address these in an e-mail or in person (or both) quickly so that the person can expect it and won’t come to you later with problems.
• Shut  the office door, turn off the phone, close e-mail. Don’t let people find a way to distract you.
• Shut down your feed-reader, disable pop-up notifications from it. Set aside time during the day to review feeds and news.
• Have a clean desk. Keep only things you’re actively working on visible.
• Set a schedule or a signal to your peers and supervisors of times when you are busy and should not be bothered. Be assertive and enforce it.
• Focus on one project at a time. Everybody has a million things to do.

Plan Weekly, Daily, Hourly

Finally, bringing it around full circle back to goals: plan as much as you can to the extent it makes sense. That’s a weasel sentence, I know, but there’s no way around it. In general, though, I think we could all do with more planning.

Effective planning combines all the above principles into a coherent framework for your work week.

Every week, you have certain meetings, tasks to be completed, issues to be researched, people to be spoken to.

Every day, a certain subset of those must be done.

Every hour, you must pick a task to work on.

My plan is to take 10-20 minutes every day to plan the day’s activities, set min-goals, while at the same time strategizing to eliminate distractions. Every Monday morning I take an additional 10-20 minutes to review and set the major goals for the week, ensure meetings are scheduled, projects are given the correct priority, I know my tasks and responsibilities, and that I have enough dead-time left unscheduled because things always come up (we operate quickly-growing 24/7/365 services–there’s no avoiding issues).

Every time you finish a task, there should be another one waiting, whether you decide to tackle it right away or take a break and do something else. As long as you have a plan, it’s ok.

There are tons of other resources out there–I’ll just link to some in the forums of Steve Pavlina.

Now I should stop wasting time and get back to work…

My first book, C# 4.0 How-To is now shipping! If you like tips you can use, check it out!

Geek Softworks

It uses WordPress, but it’s not a blog–it’s for the software I write. So far, only a few products are up, including DiskSlicer (a new version!), Windows Media Top 10 Plugin, and Word Count for Windows Live Writer.

I also setup forums for those projects. The site is still pretty small, but it’s functional and it will grow.

Thanks for looking!

It’s gotten a few hundred downloads (thanks!), but the single review is actually spam. Unfortunately, I can’t remove it or even report the review from my account since it’s my own software.

Plea for help: Can a helpful reader out there please report this review as spam and request it to be removed? Also, could someone out there who does use Live Writer write a real review of the plug-in?

If you have further ideas to improve it, please let me know.