Tag Archives: programming

log4cxx + VS2005 + Windows SDK v6.0 = compile error

If you are following the instructions to build log4cxx 0.10 in Visual Studio 2005, and you have the Windows Platform SDK v6.0 installed, you may get errors compiling multicast.c in the apr project.

I found the solution, and it’s pretty easy. Open up multicast.c and edit the lines:

136: #if MCAST_JOIN_SOURCE_GROUP

148: #if MCAST_JOIN_SOURCE_GROUP

to be, instead:

136: #if defined(group_source_req)

148: #if defined(group_source_req)

 

e voilĂ ! now it compiles.

Adapting to Changes

Change is a fact of life. Nowhere is this more obvious in a medium where the very thing we make is completely intangible and malleable: software development. There is almost nothing that is impossible in software–there are only limited resources.

This malleability has been one of the reasons for the enormous pace of change we’ve seen in computers in the last half century, but at the same time it’s been a stumbling block to true engineering practices.

Regardless of philosophy, change will happen in all software projects to some degree. There are two extremes of development that most organizations fall between:

  • A single coder hacks together whatever the boss needs changed today. (Also fixes bugs from yesterday’s fixes.)
  • NASA’s awesomely rigid development practices for the Space Shuttle software (see the Manager’s Handbook for Software Development on the Book list) resulting in extremely high-quality, virtually bug-free software (at a very high price).

For most of us, neither extreme is appropriate. Deciding the right balance of change control is a hard problem, but it’s not impossible. Still, some organizations manage to do so badly at it that it deserves to be studied and dissected.

The Census

As an example, consider the 2010 Census, a project that has been covered in the press and on tech blogs over the last few months. The following is from the second paragraph of the report on the GAO’s study of this massive program:

In October 2007, GAO reported that changes to requirements had been a contributing factor to both cost increases and schedule delays experienced by the FDCA [Field Data Collection Automation] program. Increases in the number of requirements led to the need for additional work and staffing. In addition, an initial underestimate of the contract costs caused both cost and schedule revisions. In response to the cost and schedule changes, the Bureau decided to delay certain system functionality, which increased the likelihood that the systems testing…would not be as comprehensive as planned.[…] Without effective management of these and other key risks, the FDCA program faced an increased probability that the system would not be delivered on schedule and within budget or perform as expected. Accordingly, GAO recommended that the FDCA project team strengthen its risk management activities, including risk identification and oversight.

The Bureau has recently made efforts to further define the requirements for the FDCA program, and it has estimated that the revised requirements will result in significant cost increases.

As with many government projects, there is enough failure to go around. The key idea I want to focus on is a failure on the part of the government agency and on the part of the contractor implementing the technology to adequately plan for change.

Before that, though, I want to say that I have no knowledge of their specific management or technical practices, other than what is publicly reported. For all I know, all parties performed extremely well and still failed. (However, from friends and acquaintances in government and contractor positions around D.C., I think it’s fair to say that most organizations here are behind the curve in software best practices, to put it politely.)

With that (admittedly harsh) assumption, it’s apparent that the agency failed to adequately plan, think through requirements, and communicate effectively to the contracted company. On the developer’s side, it’s apparent they didn’t have effective mechanisms for handling changes.

Before a project is undertaken, a set of fundamental principles must be agreed upon:

  1. The client will change their minds. There is no such thing as “written in stone.”
  2. Making changes is hard or impossible.

These two principles are fundamentally in conflict. On the one hand, the developer must know that changes are going to occur: sometimes large changes. They can plan for that from the code level up to their organization and processes.

On the other hand, the client, must realize that they can’t just suggest a change at the end of the project and expect it to be thrown in and work correctly.

The tension between these two principles implies certain practices and expectations on all parties, which in an ideal world would lead to a successful outcome more often.

Developer Responsibilities

The developer, knowing that the client does not understand the full set of requirements up front, designs their system in a way that can easily handle change. This means modularity, this means testability. From details as small as designing methods to operate on Streams instead of filenames, to as large as easily reusable components.

The first step is a change in mind set: an acceptance that change will occur, and that it’s ok. When I first started development, frequent changes really frustrated me. Sometimes they still do, but I’m getting better (I hope). Once the mind is malleable, practices can be implemented to handle it effectively.

A few coding practices that encourage changes:

  • Test-driven development (or at least full unit testing). In this day and age, with how easy it is to accomplish this, is there a good reason not to do it? I’m not convinced by many arguments against unit testing. A full suite of unit tests gives you confidence in your code, and the ability to safely refactor even large portions of your program. Full TDD I could go either way.
  • Constant refactoring to fit the best design possible (within reason. Obviously, there is no “best” design). Software, like everything else in the universe, obeys the law of entropy. Without constant maintenance, it will degrade. Designs degrade as you add on to them. Classes degrade as you stuff them full of things. Don’t stand for it–take the time to make things neat. Get some good books on refactoring if you don’t know how to start.
  • Highly Modular. Apply all your OO-design skills here. It matters. Cohesion and coupling really do matter.
  • Small, frequent iterations. Most software systems these days are too big and too complex to understand at once. Iterative development is the key to success.
  • Strong source-control practices. Does this even need to be discussed? If your organization doesn’t have good source control, you’re doing software wrong.

Other practices on the developer’s part read like a list of Extreme programming fundamentals:

  • Communication – The key to any relationship, whether personal or corporate, is communication. The right amount should be discovered and adhered to. It should probably be slightly more than you’re comfortable with (since many developers would rather not talk to anyone ;).
  • Simplicity – I believe that the single hardest thing to do in developing software is to manage complexity. It’s requires more brainpower and creativity than all other activities. Coding is easy. Coding so that it is easy to understand a year from now is difficult. If the design is needlessly complex up front, then change is that much harder to implement.  Always remember that software complexity increases exponentially.
  • Courage – Stand up for correct principles. Don’t allow politics to interfere with what needs to be done. Don’t be afraid to change the design if it needs to happen. Change coding practices as needed. Never be afraid of the truth–just deal with it.
  • Respect – Both sides need to remember that the other is expert in their domain. Humility is key.
  • Responsibility – this is something I fear is lacking much in government. Everybody needs to take responsibility for their part. This implies accountability. This can be shared responsibility at some levels, or the-buck-stops-here responsibility at higher levels. If no one is responsible, nobody cares if it fails. Courage is a prerequisite to this.

This mind set needs to encompass not just pure software development, but also processes at the organizational level. This can take many forms, but a few ideas:

  • Personal relationships between clients and developers
  • Flexible team structure and members
  • An easy way to submit and discuss change requests
  • A culture that accepts changes
  • A good understanding of the problem domain

Jeremy Miller (The Shade Tree Developer) talks a lot about processes and practices and it is a very good read.

Unfortunately, there exists an attitude at many companies that it’s better to milk the clients for as much money as possible rather than do the hard work of getting a good process.

Client Responsibilities

I think my introduction and the discussion about developer responsibilities may imply that because software is so ephemeral, it is therefore easy to change it. After all, it’s not a physical object that has to be broken down and rebuilt–what’s so hard? To anyone who has worked on non-trivial projects, this is laughable. Let’s clarify:

It is easy to program. It’s painfully, mind-bendingly, insanely difficult to design software. So difficult, in fact, that no one can do it well. The sooner everybody understand that, the sooner we can get started with real work. Good design is thought-work, iterated over and over, added to experience and analysis.

Some details are easy to change. Others are fundamental to the structure of the building. By way of metaphor, changing your mind about a house you’re building to suggest a stone facade instead of brick is relatively easy. Deciding you would like to have ten stories added onto your house changes the game significantly. Now you have to build a completely new foundation.

I believe the customer’s responsibility is to gain knowledge and insight into the development practice so they understand why things are difficult. Computers are so fundamental to our culture, we can no longer afford to have companies and agencies run by people who don’t understand them to some degree. The exact degree is debatable, but it’s certainly higher than where we’re at now. In my own situation, I think that when I started my boss did not realize how seemingly-trivial changes could be phenomenally difficult to implement, and therefore bug-prone–all because of design decisions made earlier. I now have him understanding that there is no such thing as a simple change.  (Of course, I still occasionally underpromise and overdeliver–I have to maintain my miracle-worker image after all. 🙂

A client who wants software done and doesn’t want to end up paying an extra $3,000,000,000 for software should understand that critical requirements can’t be left until the product is being delivered. “We didn’t think of it before” is not a good excuse.

A client who understands this will realize how critical initial requirements are and ensure they’re communicated clearly. Most of all, they don’t wait around twirling their thumbs, only to reject the system when it’s done because it doesn’t meet their requirements. They will insist on periodic feedback and demos to make sure things are going the right way.

The responsibility for huge failures is on everybody’s shoulders. I believe that it can often be traced back to an unwillingness to face up to the truth, lack of responsibility, and misunderstanding the nature of the project.

Large projects aren’t doomed to fail

There is a fundamental truth, however, of all projects: there exists at least one change that is too large to do. Maybe the Census required that change–I don’t know. I do know that if both sides follow the best practices they can, the number of must-fail changes can be minimized.

That said, here’s why I refuse to believe the census project wasn’t sorely mismanaged from the outset by all sides: FedEx, UPS, Amazon, Wal-Mart, Microsoft, and dozens (hundreds?) of other companies have built enormous and complex systems for managing large databases and mobile platforms involving thousands of partners. It is ridiculous to me that they could not have built a system for our government quicker, cheaper, and less epic failure than the one they got. If the Census is fundamentally more complex than the problems they deal with, someone please let me know.

Change is part of everything now

The ability to adapt to change is critical, not just for software developers, but for everyone in our society. Technology, events, trends, and money all flow so quickly now that those who can adapt quickly will succeed.

It’s not just software companies that need to adopt sound development practices. There will always be companies that build out our infrastructure (Microsoft, Google, Cisco, etc.), but in the end every company will be a software development company. Disregarding good practices because “we’re not a software” does not work anymore. It’s not true. You are a software company. We all will be.

Idealism or…

If a lot of this seems idealistic, that’s ok. I wish the world were like Star Trek as much as the next geek, but I realize that there are a lot of motivation$ out there. The world is what it is, but if we aren’t trying to improve it by working towards a standard, then what’s the point? We can improve it. There is no excuse for such catastrophic wastes of time, money, and effort.

Review: Pragmatic Unit Testing in C# with NUnit, 2nd Ed.

I saw this book when I bought Programming WPF a few weeks ago and it looked promising enough to buy. I’ve been doing unit testing in C# for a few years now, but I thought there were always things to learn and maybe I’d pick up a few new ideas.

It is easy to contrast this book with Beck’s Test Driven Development: By Example, and the two books definitely have a very different feel.

Beck’s book has a very evangelical feel to it, and it’s main purpose is to teach a mind set more than technical details. I believe this is important–maybe of first importance–but once you understand that, the rest of the book is a little simplistic for more experienced developers.

Pragmatic Unit Testing, on the other hand, focuses much more on the practical aspects (hmmm….I highly suspect that’s where the title comes from….) of unit testing. I liked the ideas on how to use categories and attributes to segregate tests that take too long to run on a regular basis. I also liked the section on singletons and getting around time-dependencies. The DateTime.Now problem is something I’ve had to deal with quite a bit in our server-side software that has a lot of time-dependant behavior. (In most cases, the problems were solved with refactoring the time into a function parameter.)

There are also good discussions of more mundane issues like how to deploy NUnit, where to put tests in a project, team practices, GUIs, threading, and C#-specific issues.

The discussion about mock objects (a very basic introduction) is also quite clear and understandable–more so than many resources I’ve seen on the web, which often assume you already know all about them.

Something I don’t like: the acronyms (BICEP, CORRECT, A TRIP). They kind of bug me. I like the ideas behind the acronyms and I think it’s more important and effective (for me, anyway) to internalize the principles of testing rather than remembering specific acronyms and the words they go with. YMMV.

Last Word…

I will probably only read Test Drive Development: By Example once,  but I will definitely come back to Pragmatic Unit Testing occasionally to refresh my ideas.

Custom data source for Google Earth

Using just these links as a guide, we quickly (less than a day) put up a data source over https for our customers to download GIS data to Google Earth.

KML reference and tutorial – KML is the XML language used to describe features that can be displayed in Google Earth and Google Maps.

Sample code to generate KML from a web service or web page (it’s VB.Net).

How to implement a custom authentication provider for IIS in .Net. Very useful if you need to authenticate your KML-generating web page and you don’t want to use Active Directory.

Tracking database changes using triggers

Tracking changes in database tables is an incredibly useful feature–especially for operational data that can change often. Having recently had to implement this feature, I thought I’d share some of the techniques I learned.

Sample Database

First, let’s conceptualize a very simple database consisting of user information (name, date of birth), and e-mails. A user can have more than one e-mail.

 

Table: UserData

Field Type
ID int (PK, identitiy)
FirstName varchar
LastName varchar
birthdate date

Table: UserEmails

Field Type
UserID int (FK)
email varchar

 

 

We want to track all changes to the FirstName, LastName, and birthdate fields. In addition we want to track when e-mails are added or removed from a user. As we’ll see, these aims are accomplished using two different methods.

My implementation is done in SQL Server 2000 and C#, but any database that supports triggers can be used.

Changes in a Single Table

With this method we want to track the changes to all fields of a table. In our example, we want to know when FirstName, LastName, and birthdate change values in the UserData table.

To accomplish this we need another table to track the history. This table is going to have the exact same fields as UserData, plus a few extra for the change tracking.

Table: UserDataChanges

Field Type
ChangeID int (PK, identity)
ChangeTime datetime
ChangeUser varchar
ID int (FK)
FirstName varchar
LastName varchar
birthdate date

Now the automated part–adding a trigger to populate this automatically:

CREATE TRIGGER UserDataChangeTrigger ON UserData FOR UPDATE, INSERT
AS    
    IF (UPDATE (FirstName) OR UPDATE(LastName) OR UPDATE(birthdate))    
    BEGIN     
        INSERT UserDataChanges 
            (ChangeTime, ChangeUser, ID, FirstName, LastName, birthdate)
            (SELECT GetUtcDate(), user, ID, FirstName,LastName,birthdate 
                FROM inserted)     
    END     

This trigger will insert a new row into the UserDataChanges table whenever a row in the UserData table is updated or inserted. The IF (UPDATE(FirstName)…. ) is not strictly required in this scenario, but in other cases I did not want a change recorded when certain fields were updated (i.e., you have a field that tracks the last change time of that row, or the number of orders, or any other field that can change frequently and isn’t important to track–you don’t want to create too much noise in this or it will not be useful). The GetUtcDate() and user are SQL Server functions that retrieve the current UTC time and the username of the process that caused the change–very useful for tracking responsibility. The inserted table is created by the server for use by the trigger and contains all the new values.

Changes in a Foreign Key Table

The UserEmails has to be handled differently because there can be multiple e-mails for each user and we can assume they can be added, or removed at will (Remove + Add = Update, so I won’t consider direct updates here).

The solution I landed on was to have a generic event log table that stores manual log entries as well as “special” entries denoting adding or removing e-mails.

Table: UserEventLog

Field Type
EventID int (PK, identity)
ID int (FK)
EventTime datetime
EventType int
ChangeUser varchar
Notes varchar

This table can be used for both adding text notes to a user and, by using the EventType field, special events. In our example, we have two events we need to track:

 

Event Value
EmailAdded 1
EmailRemoved 2

(In code, I’ve made these enumerations)

Next we add a trigger on the UserEmails table:

CREATE TRIGGER UserEmails_EmailAddedTrigger
ON UserEmails
FOR INSERT
AS
 BEGIN
     INSERT UserEventLog(ID, EventTime, EventType, ChangeUser, Notes)
        (SELECT ID, GetUtcDate(), 1, user, '{'+email+'}' FROM inserted)
 END

The value 1 stands for EmailAdded. I’ve added braces around the actual e-mail address to set it apart from regular notes (we’ll see how to integrate everything later).

To handle the deletion of e-mails add another trigger:

CREATE TRIGGER UserEmails_EmailRemovedTrigger
ON UserEmails
FOR DELETE
AS
 BEGIN
     INSERT UserEventLog(ID, EventTime, EventType, ChangeUser, Notes )
        (SELECT ID, GetUtcDate(), 2, user, '{'+email+'}' FROM deleted)
 END

The only things different: FOR DELETE (instead of INSERT), changed the EventType to 2 (EmailRemoved), and the values are taken from the SQL Server-supplied deleted table.

That’s enough to get a pretty good change-tracking system in place, but you’ll still have to build a UI to display it effectively.

Displaying the Changes in the UI

With the above work done, you end up with two types of entities: changes and events. While it would be possible to integrate all functionality into a single event/change table using a lot more logic in the SQL Trigger code, I’m personally more comfortable with the change logic being in my application code. I think this way the database is kept more “pure” and open to changes down the line.

That means we will need to integrate these two types of entities into a single list, ordered by date/time. I’m going to assume the existence of two classes or structs that represent each of these entities. They’ll be called UserChange and UserEvent. I’ll also assume that the lists of each of these are already sorted by time, since that’s trivial to do in a SQL query.

Given that, we need a function that takes both of these lists and produces a sorted, combined list with an easy-to-understand list.

How the function works:

  1. Go through both lists, and pick whichever one is next, time-wise.
  2. Translate the object into a string/list-view representation of that object.
  3. If it’s a UserChange object, compare it to the previous one to figure out what changed.
  4. Sort the list in reverse order to put newer items at the top.

Here’s the C# code which I’ve adapted from our production system. Don’t get hung up on the details:

 

private void FillLog(IList<UserEvent> events, IList<UserChange> changes)
{
    List<ListViewItem> tempItems = new List<ListViewItem>();
 
    int currentEventIdx = 0;
    int currentChangeIdx = 0;
    eventLogListView1.Items.Clear();
 
    while (currentEventIdx < events.Count
    || currentChangeIdx < changes.Count)
    {
    UserChange currentChange = null;
    UserChange prevChange = null;
    UserEvent currentEvent = null;
 
    DateTime changeTime = DateTime.MaxValue;
    DateTime eventTime = DateTime.MaxValue;
 
    if (currentChangeIdx < changes.Count)
    {
        currentChange = changes[currentChangeIdx];
        changeTime = currentChange.ChangeDate;
        if (currentChangeIdx > 0)
        {
        prevChange = changes[currentChangeIdx - 1];
        }
 
    }
 
    if (currentEventIdx < events.Count)
    {
        currentEvent = events[currentEventIdx];
        eventTime = currentEvent.EventDate;
    }
    string dateStr;
    string userStr;
    string eventTypeStr="";
    string notesStr;
 
    if (changeTime < eventTime)
    {
        dateStr = Utils.FormatDateTime(changeTime);
        userStr = currentChange.UserName;
        notesStr = GetChangeString(currentChange, prevChange);
        currentChangeIdx++;
    }
    else
    {
        dateStr = Utils.FormatDate(eventTime);
        userStr = currentEvent.UserName;
        notesStr = currentEvent.Notes;
        eventTypeStr = currentEvent.EventType.ToString();
        currentEventIdx++;
    }
 
    if (notesStr.Length > 0)
    {
        ListViewItem item = new ListViewItem(dateStr);
        item.SubItems.Add(userStr);
        item.SubItems.Add(eventTypeStr);
        item.SubItems.Add(notesStr);
        item.ToolTipText = notesStr;
        item.BackColor = (tempItems.Count % 2 == 0) ? 
            Color.Wheat : Color.White;
        tempItems.Add(item);
 
    }
 
    }//end while
    eventLogListView1.BeginUpdate();
    for (int i = tempItems.Count - 1; i >= 0; i--)
    {
    eventLogListView1.Items.Add(tempItems[i]);
    }
 
    eventLogListView1.AutoResizeColumn(0, 
        ColumnHeaderAutoResizeStyle.ColumnContent);
    eventLogListView1.AutoResizeColumn(1, 
        ColumnHeaderAutoResizeStyle.ColumnContent);
    eventLogListView1.AutoResizeColumn(2, 
        ColumnHeaderAutoResizeStyle.ColumnContent);
    eventLogListView1.Columns[3].Width = eventLogListView1.Width - 
    (eventLogListView1.Columns[0].Width +
    eventLogListView1.Columns[1].Width +
    eventLogListView1.Columns[2].Width +10);
 
    eventLogListView1.EndUpdate();
}

Now we need to define GetChangeString, which figures out the differences in successive UserChange objects and displays only pertinent information.

 

private string GetChangeString(
    BuoyDataChange currentChange, 
    BuoyDataChange prevChange)
{
    StringBuilder sb = new StringBuilder();
 
    if (prevChange == null)
    {
        CompareAndAdd(sb, "First Name", 
            null, currentChange.FirstName);
        CompareAndAdd(sb, "Last Name", 
            null, currentChange.LastName);
        CompareAndAdd(sb, "Birth Date", 
            null, currentChange.BirthDate);
    }
    else
    {
        CompareAndAdd(sb, "First Name", 
            prevChange.FirstName, currentChange.FirstName);
        CompareAndAdd(sb, "Last Name", 
            prevChange.LastName, currentChange.LastName);
        CompareAndAdd(sb, "Birth Date", 
            prevChange.BirthDate, currentChange.BirthDate);
    }
    return sb.ToString();
}

And one last helper function which compares two objects and if different appends the change to a StringBuilder object.

 

private void CompareAndAdd(StringBuilder sb, string field, 
    object oldVal, object newVal)
   {
       if (oldVal == null && newVal == null)
           return;
 
       if (oldVal == null || !oldVal.Equals(newVal))
       {
           if (sb.Length > 0)
           {
               sb.Append(", ");
           }
           sb.AppendFormat("{0}:{1} -> {2}", field, oldVal, newVal);
       }
   }

In this way you can end up with an automated system that displays all changes in an easy-to-understand format.

Here’s a sample of what our system looks like (click to enlarge):

Change log screenshot

Other ways to accomplish this? Better ways? Please leave a comment!

kick it on DotNetKicks.com

Opening Visual Studio solutions from Explorer in Vista

You’ve installed Visual Studio 2005 on Vista and dutifully changed it to run as administrator, like you’re supposed to. And then…

Problem: Visual Studio 2005 solutions no longer open when you double-click them in Windows Vista. In fact, when you double-click nothing happens.

Solution: Change them to open with Visual Studio 2005 directly instead of the vslauncher.exe (which opens up the solution with the correct version of Visual Studio if you have more than one).

Caveat: Only makes sense if  you use only Visual Studio 2005.

How-to:

  1. Right-click on a solution file.
  2. Choose “Open With…”
  3. Choose “Browse…”
  4. Browse to file “C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\devenv.exe” (or wherever you installed Visual Studio)
  5. Click “Open” button
  6. Check “Always use the selected program to open this kind of file”

openwith

Now your solutions will load Visual Studio, bring up the UAC prompt, and it all works great.

Found via here and here.

4 Principles of Not Wasting Time

There are so many postings out there on all sorts of blogs about how not to waste time that I’m not sure I can contribute something very meaningful (certainly not new), but since it’s something I’ve been thinking about, I might as well spill some ideas about it.

Like this? Please check out my latest book, Writing High-Performance .NET Code.

Definition

Any discussion of time-wasting is profitless unless you define what wasting time is. My definition is:

Wasting time is doing anything that does not contribute to my goals.

That is a very broad definition, but it is very useful. It presupposes a goal-oriented mind set and I don’t want to get too far down that path here. If you’re really interested in a goal-focused system, I highly encourage you to read The 7 Habits of Highly Effective People, which is a principles-based approach to effectiveness and goal-setting is a huge part of it.

Whatever your personal system, much of the corporate world and building software specifically revolves around goals (aka “milestones”, “targets”).

This definition includes taking breaks and eating lunch, but let’s not be silly–we’re not talking about that and I’m not going to water down the definition nor am I going to spend pages talking pointlessly about exceptions to it. We’re all intelligent people here and can understand the important principles.

Many Small Goals Are Better Than One Large One

Software development processes have been undergoing evolution since the beginning and lately the whole agile process seems to be taking over. Whatever the process, many companies are finding more success in breaking down large projects into tiny goal-driven chunks, sometimes lasting as little as a week.

This same principles can be applied to ourselves at both large and fine-grained levels. It is definitely good and desirable to have the overall vision of our project in mind, but  this doesn’t often help us get the work done. Some of my most productive days are when I break down a huge task into tiny subtasks and set a goal for each one (“I will have this done by 11am today, then I will wrap up this other small one by 4pm.”)

An example: I’m currently writing some code to move a huge amount of data around in our production database. We’re going to be rolling out a major update that requires some fundamental changes to how things operate. This task is so large and daunting that I get a headache just thinking about it and so I could put it off, just spinning my wheels until I decide to face the inevitable. Instead, I’ve broken it into several smaller tasks that are each easily managed and understood.

Before After
  • Convert database to new format
  • Export SQL script of new tables, triggers, indexes, etc. directly from SQL Server
  • Aggregate data from Table1 into NewTableX
  • Move data from Table2 to NewTableY
  • Move data from Table3 to NewTableZ
  • Verify moved data
  • drop old tables
  • drop old columns
  • etc.

(In my example, the After column actually contains about 30 items, depending on how far I want to break it down…it could be more.)

Now, instead of being overwhelmed by the complexity of a large task (and thus doing nothing), I can easily handle each of the sub-steps efficiently. I’ve changed a two-week task into many hour-long tasks.

The psychological effect of  too-hard/too-complex/too-much is devastating. You can’t handle something like that–no one can, and so you won’t–you’ll just end up wasting time fretting about it. Break it up for your sanity and happiness, as well as productivity.

Motivation is Crucial

Often, a key to not wasting time is having sufficient motivation. Motivation can come in all sorts of ways–the key is to figure out what motivates you and then set yourself up to succeed by using that motivation as a carrot to pull you forward. It can be a good idea to share your motivations with managers so they understand what drives you.

Motivation can often begin with picking good goals. If your goals are unrealistic, you are almost guaranteed to fail in some way. Despair feeds on itself and will sink your productivity and cause you to engage in anything but work. Not only will you avoid the drudgery of work, but you won’t take steps to improve yourself or change the situation. This cycle must be broken immediately.

If your projects are just not that interesting this can be a challenge. Everybody has tasks they don’t particularly like, but if the majority of your time is spent doing stuff you get no pleasure out of, you are doing a huge disservice to yourself and your future. Eventually, you’ll become wasted and useless to both yourself and your employer. Fix the situation–get a new project, get a new job, find side projects to do that you do enjoy as rewards for getting through the drudgery–anything to avoid becoming the shell of a person you once were.

Maybe you don’t necessarily need a new job right now–maybe you just need to fix the situation at your current job, get some enjoyable hobbies at home, spend more time with the family. The needed changes aren’t always drastic–but figure them out so you don’t spend every day wallowing in a mire not doing anything useful.

Eliminate Distractions Now

I don’t think I’ve answered my office phone in about a year. Not that many people call it in the first place, e-mail being highly-preferred around here, but I like to say I stand on principle. You can read a lot about creating the right environment for highly-skilled software developers in the fabulous book Peopleware: Productive Projects and Teams.

I’ve also stuck to the practice of keeping my e-mail inbox cleared. I delete almost everything I receive unless I need to store it for later or act on it.

In my entry about working an 8-hour day, I talked about various time-wasting activities that I’ve observed, such as  wandering the halls, micro-managing, too many words, poor inbox management skills, and more.

The goal of eliminating distractions is not to completely choke off any aspect of fun, diversion, and other social aspects of working in an office–those are good things. The goal is eliminate the things that don’t help us in our jobs and that aren’t really all that enjoyable to us anyway, all things considered.

One way of eliminating your distractions is to go out of your way ahead of time to manage them so they don’t come up later. Some ideas:

  • If you need somebody to do a task, anticipate questions, concerns, or problems they may have. Try to address these in an e-mail or in person (or both) quickly so that the person can expect it and won’t come to you later with problems.
  • Shut  the office door, turn off the phone, close e-mail. Don’t let people find a way to distract you.
  • Shut down your feed-reader, disable pop-up notifications from it. Set aside time during the day to review feeds and news.
  • Have a clean desk. Keep only things you’re actively working on visible.
  • Set a schedule or a signal to your peers and supervisors of times when you are busy and should not be bothered. Be assertive and enforce it.
  • Focus on one project at a time. Everybody has a million things to do.

Plan Weekly, Daily, Hourly

Finally, bringing it around full circle back to goals: plan as much as you can to the extent it makes sense. That’s a weasel sentence, I know, but there’s no way around it. In general, though, I think we could all do with more planning.

Effective planning combines all the above principles into a coherent framework for your work week.

Every week, you have certain meetings, tasks to be completed, issues to be researched, people to be spoken to.

Every day, a certain subset of those must be done.

Every hour, you must pick a task to work on.

My plan is to take 10-20 minutes every day to plan the day’s activities, set min-goals, while at the same time strategizing to eliminate distractions. Every Monday morning I take an additional 10-20 minutes to review and set the major goals for the week, ensure meetings are scheduled, projects are given the correct priority, I know my tasks and responsibilities, and that I have enough dead-time left unscheduled because things always come up (we operate quickly-growing 24/7/365 services–there’s no avoiding issues).

Every time you finish a task, there should be another one waiting, whether you decide to tackle it right away or take a break and do something else. As long as you have a plan, it’s ok.

There are tons of other resources out there–I’ll just link to some in the forums of Steve Pavlina.

Now I should stop wasting time and get back to work… 😉

Announcing: GeekSoftworks.com

I’ve setup a new domain for a front page for my software hobbies and what will eventually be my “store front”:

Geek Softworks

It uses WordPress, but it’s not a blog–it’s for the software I write. So far, only a few products are up, including DiskSlicer (a new version!), Windows Media Top 10 Plugin, and Word Count for Windows Live Writer.

I also setup forums for those projects. The site is still pretty small, but it’s functional and it will grow.

Thanks for looking!

In this universe we obey the law of commutativity

This kind of thing has happened to be a few times now, so I thought I’d share the fun.

In one of our pieces of software we have a process that looks like this:

void MyThread()
{
    while (true)
    {
        DoFunctionA();
        DoFunctionB();
        SleepFor10Seconds();
    }
    
}

While FunctionA and FunctionB are conceptually similar, they interact with completely different systems.

We had a problem with FunctionA the other day–it was taking 120 seconds to do its thing instead of the normal 10 (or less) because a remote server was down. This caused problems for FunctionB because it wasn’t running as often as it should have so things were getting backed up in the system. Oops.

Now, the solution is to split these two functions into two independent threads so they don’t interfere with each other, and I’ve been meaning to do this for a while, so that’s what I proposed.

Response back: “That’s a good idea, but before we do something complicated like that, can we just put FunctionB first?

Um, no.

The time we want to minimize is the time between running FunctionB, which is TimeSleep + TimeA. Putting FunctionB first makes it TimeA + TimeSleep. Last I checked, those were actually equivalent.

How to work an 8-hour day

One of the things I decided when I started working was that I was not going to be one of those guys who worked 12 hours a day for a company (if I ever become an entrepreneur, all bets are off since I’m working for myself). So far, I’ve been pretty successful, and I’ve noticed a few things that may help others. Some of these are more observations than practices.

1. Understand reality: Work Load

The first thing to realize is that the amount of work to do will always exceed the time available. This is why effective management is important. If you are being ineffectively managed, it may be difficult to force a change in some of the following areas.

Just because there is always more work to do, this is no reason to kill yourself trying to get it all done. Or even overexert yourself (except in rare instances). Don’t misunderstand: I’m not saying you have an excuse to slack. Giving all of your time and attention to your employer/projects/job is the baseline here. I am saying that just because you have a lot of work is not a valid reason to work 12 hour days.

Unless you enjoy it…in which case you’re reading the wrong article. If you’re a workaholic, sacrificing your health, family, and free time to get ahead, knock yourself out. You can stop reading now.

There are always special circumstances, though. If you operate in an environment that runs 24/7 services and Something Bad happens–well, then you need to fix it if takes you 24 hours. Hopefully, you’ll get corresponding time off in return. If it’s the last week before release of a project and a major bug comes up–get it done.

I’m talking about normal days, normal work.

If you are working 12 hour days and you don’t like it, then change. No excuses. Change the job or change jobs.

2. Don’t waste time

I knew someone who always complained about having so much work to do, working 60-70 hour weeks, always been pressured, etc. I know he did have quite a bit of pressure to do a number of things, but I eventually stopped feeling empathy for him because I constantly saw him talking to so many people (friends, not direct co-workers) during work.

If you’re so busy working long hours, why do you waste so much time? Why is it that the people who complain loudest about being overworked mostly create the situation themselves?

Don’t wander the halls looking for distractions–chances are you’ll find them.

There is a balance to strike here between enjoyable work environment, having friends, being part of a team, and actually getting things done. There is also something to be said for the difficulty of concentrating on challenging topics for a long time at a stretch. Breaks are definitely needed.

There are trade offs to everything. If you spend an hour or two every day reading blogs instead of working on your projects, you’ll pay for it in time later. If this is a trade off you’re willing to make, so be it, but make sure you understand your priorities and the consequences of your decisions.

I know someone else who used 100 words to say something when 4 would do. This made meetings long, tortuous affairs (unless strong ground-rules were created, endorsed, and enforced). Even simple questions were avoided because nobody wanted to sit there and endure a longer-than necessary answer about the history of the universe. Don’t waste people’s times, and don’t stand for people wasting yours.

Be wise, what can I say more?

3. Don’t micro-manage

The topic of micro-management is something that could take up other blog posts and books dedicated to the subject, but I just want to focus on aspects of it related to time.

This is an inverse of wasting time on trivialities. I once had another boss who spent unbelievable amounts of time responding to every trivial e-mail (and he insisted on being CC’d on every topic of course) and he also complained about working extremely long weeks. And he worked all weekend. He checked on the status of things constantly and was usually the first to spot potential problems. (There were other reasons for this, too: he was usually a main point-of-contact for customers and he had more domain knowledge of everything we were doing). He was very, very smart and usually not wrong, but he did spend a lot of time doing things that were arguably someone else’s job. We never felt invested in these aspects of work, though, because he just did them.

If you find yourself with your fingers in every aspect of your organization, you have a problem, and need to take some time out for consideration:

  • Why do you feel the need to be connected to everything going on? Is it because you have to feel in control, or do you not trust others to do a good job?
  • Do you not trust your employees? If not, why not? Do they really do a bad job?
  • If they do a bad job, why? Are they fundamentally lazy, unqualified, dumb? If so, why are they still working there? Why are you paying them and doing their job anyway?
  • Would better training help?
  • Maybe expectations aren’t clear. Instead of you picking up the slack, review the expectations you have for them and go over deficiencies. Make them responsible and give them ownership. Don’t undermine their jobs by taking it away. Check on them in the future to make sure things are improved, but don’t involve yourself day-to-day.
  • Is it necessary to involve yourself in every discussion or can you just ignore until it reaches a level you need to get into? Resist the urge to comment on things you weren’t asked about.

If nothing else, micro-management always breeds resentment.

 

4. Set boundaries

I have tried to make it clear to my bosses over the years that when I’m working on a task, I am not to be bothered on a whim. When I’m concentrating, don’t bug me. This is a somewhat loose rule for me, because it really depends on what I’m working on. However, it is more true than not.

The context switching that can happen with interruptions is dangerous for productivity. Eric Gunnerson wrote about “flow state” years ago. Flow state is a zone you get in where your brain is completely engaged–you’re firing on all cylinders, fully committed and involved–pick your metaphor. It’s hard to get into and easy to leave.

I have a nice pair of Bose QuietComfort 2 Noise Cancelling Headphones. These are essential for me now. The whole topic of listening to music while programming is a separate one, with some debate, but I find it usually beneficial. Sometimes I put them on to block sound without listening to music. This also minimizes casual interruptions. I rarely listen to music while trying to create a new solution for something–only when I’m coding something I understand.

Also make sure to enforce a work/life boundary. I have made it clear in my job that I am not be called unless it’s an emergency that needs to be fixed RIGHT NOW. I ignore IMs and e-mails (unless I’m in the mood to answer something, or it’s trivial, or I know it would make somebody happy to know the answer to). If there’s a critical problem, I’ll get a phone call.

I also communicate vacations and just how far out of reach of civilization I will be well ahead of time.

5. Limit Meetings

There is an abundance of resources out there for effective meeting management. I’m more in favor of limiting the occurrences of meetings in the first place. How well you can do this depends a lot on your organization and the projects you’re working on.

Pointless meetings are a waste of everyone’s time. They can be demoralizing, energy-sucking vortexes. Recovering from a bad meeting can take further attention away from actually getting things done.

In my case, we have mercifully few meetings–only when the entire team needs to give input, we need to brainstorm about a tricky problem, or general status updates (yearly or quarterly).

Much of what goes on in meetings can be substituted by effective individual planning, e-mail, issue tracking, or planning better in the first place.

6. Know your tasks

Spinning your wheels is death. Don’t be wasting your time unconsciously (If I’m going to waste time, I’d rather do it consciously with Desktop Tower Defense).

If you’re stuck on a task, get help, change gears, do something else. Make sure you have a process to detect when you and others are stuck so that you don’t waste time.

Always know what you’re supposed to be working on. In addition to have issue-tracking software, make sure priorities are communicated clearly and often. Once you finish something, you should know what the next task is, whether  you do it then or not. Whenever I have a problem of knowing my priorities, I ask my manager. It’s his responsibility to know and set them. I don’t have to worry about it.

There is usually a list of “nice-to-haves” that you can work on once high-priority stuff is done. Don’t let that list get lost, but track it the same way you do regular issues.

7. Effective Information Management

This means all sorts of data: e-mail, documents, data, scheduling, task lists. Simplify, automate, streamline. I won’t go into specific techniques–you can find them elsewhere.

Just a few simple points that I use:

  • Don’t read what you don’t need to
  • Don’t respond when you don’t need to
  • File away or delete ASAP

It’s easy to get buried in information these days–and consequently it’s easy to think it’s at all important. Don’t fall into that trap.

8. External Motivations

A little obvious, this one, but important to ponder. None of this will work if you don’t have strong outside motivations. You have to really enjoy being outside of work–and I don’t think the satisfaction of doing nothing is enough. You’ll get bored of that eventually. Sitting in front of a TV every night “unwinding” for three hours is not a motivational experience (quite the opposite, usually).

Hobbies, projects, sports, exercise, cooking, family, church, books, culture, service are all part of life and should be enjoyed.

I usually schedule my evenings to some degree–whether it’s working out,  moving ahead on personal programming projects, building my latest LEGO model, watching a movie with Leticia, or reading a book–I’ve got a long list of things I enjoy doing. If I don’t follow the schedule exactly, or can’t fit everything in, I don’t sweat it–as long as I’m enjoying myself, that’s what matters.

Work isn’t everything–in fact, it isn’t even the greatest part of anything. Be well-rounded.

* Exceptions

Unfortunately, not every job can be forced into a neat 8-hour work day. My wife, for example, works for a news wire service. She works 8-hour days most of the time, but quite often she ends up staying late to take care of things. Unfortunately, when your job depends so much on other people, it can be a bit hard to plan days, and it’s very possible that work comes up right when you leave, or there is too much work to fit in in a day. However, this is unsustainable in the long run. Either more efficient processes have to be introduced, the work load decreased, or more people hired.

A word or two about politeness as well: it’s difficult in some environments to enforce better behaviors without being considered rude, grouchy, or holier-than-thou. Sometimes firmness has to give way to politeness. The carrot usually works better than the stick anyway in the long term.