Category Archives: Code

Never make assumptions about performance

The importance of measuring performance changes is a topic that has been covered by others smarter and more experienced than me, but I have a recent simple tale.

I’ve simplified the code quite a bit in order to demonstrate the issue. Suppose I have a wrapper around an image (it has many more attributes):

   1: class Picture
   2: {
   3:     Image _image;
   4:     string _path;
   5: 
   6:     public Image Photo
   7:     {
   8:         get
   9:         {
  10:             if (_image==null && !string.IsNullOrEmpty(_path))
  11:             {
  12:                 _image = Bitmap.FromFile(_path);
  13:             }
  14:             return _image;
  15:         }
  16:     }
  17: 
  18: }

I had this and a view that loading about 2,700 of these into a customized ListView control at program startup. On a cold start (where none of the pictures were in the disk’s cache), it would take 27 seconds. Unacceptable.

What to do?

My first thought was to load the pictures asynchronously. I wrapped Bitmap.FromFile() into a function and called it asynchronously. When it was done, it fired an event that percolated up to the top.

Well, I spent about 30 minutes implementing that and ran it–horrible. The list showed up immediately, but it was unusable. The problem? Dumping 2,700 items into the ThreadPool queue is a problem. It doesn’t create 2,700 threads, but it causes enough problems to not be a viable option.

Asynchronicity is still the answer, though. But it’s at a different level. Instead of loading the individual images asynchronously, I skipped loading the images when creating the list control and instead launched a thread to load all the images and update them when done. The list loads in under a second, and the pictures show up little by little after that.

Measure, measure, measure. And pay attention.

Technorati Tags: ,,


Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order:

Tip: Easily Automating use of WaitCursor

This is really simple and probably common, but it’s a useful tip anyway.

Say you need to set a form’s cursor to the wait cursor while you accomplish something.

You would do something like this:

this.Cursor = Cursors.WaitCursor;
 
//do something
 
this.Cursor = Cursors.Default;

Of course, what if “do something” throws an exception? Then your cursor won’t be set back to the default.

So let’s wrap it in try-finally.

try
{
    this.Cursor = Cursors.WaitCursor;
    //do something
}
finally
{
    this.Cursor = Cursors.Default;
}

Better, but that’s a lot of wrapping around a simple operation. Notice, that the try-finally is exactly the pattern that using is. Why not wrap this functionality into a simple to use class that handles the cursor reset automatically?

internal class WaitCursor : IDisposable
{
    private Control _control = null;
    public WaitCursor(Control control)
    {
        _control = control;
        _control.Cursor = Cursors.WaitCursor;
    }
    public void Dispose()
    {
        _control.Cursor = Cursors.Default;
    }
}

And to use it in a form is simple:

using (new WaitCursor(this))
{
    //do work
}

Of course, you could easily add functionality like restoring the previous cursor instead of always the default, handling exceptions, make it thread-aware so it uses Invoke if necessary, better error-handling, etc.

Technorati Tags: ,,,


Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order:

.Net Reflector

Lutz Roeder’s .Net Reflector has been discussed on many blogs before, but I want to give it an additional plug. I recently had to emulate some C# serial-port code in our C++ app. The .Net SerialPort class is great, easy-to-use, and works well. Unfortunately, we’re using a C++ serial port library that does not support all the possible features. Fortunately, we have the source code and can easily extend it. Unfortunately, I’m not too familiar with serial port programming, and the .Net functionality does not all obviously map to the Win32 API in every respect.

Enter Reflector. It was trivial to poke into the .Net assemblies and see what the SerialPort class was doing under the covers and then use the correct Win32 functionality in our app.

There are also a ton of plugins available.

Technorati Tags: ,,,


Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order:

Solving "Unexpected Store Error" in Exchange

Getting a weird COM Exception with the cryptic ID 0x8055001E?

We’ve been struggling with this problem for over a year now, and we finally have a solution.

We have some critical code that is contacting Exchange server via COM Interop and CDOEX.DLL to read some inboxes and process e-mails. About once a month or so, we get this error:

System.Runtime.InteropServices.COMException (0x8055001E): Unexpected
store error: %1!d! (0x%1!8.8x!)
   at ADODB.RecordsetClass.Open(Object Source, Object
ActiveConnection, CursorTypeEnum CursorType, LockTypeEnum LockType,
Int32 Options)
   at MessageService.Exchange.ExchangeClient.Connect(String folderUrl,
String userId, String password, Boolean useHttp)

After this point, restarting our software does not help. The only recourse is to restart the Exchange store completely. Did I mention that our software needs to run 24/7/365 with no downtime (a few minutes here and there are acceptable)?

So about once a month, I get a message on my phone, I log into the server, reboot Exchange, and all is well.

Searching on Google revealed nothing at all. Until recently.

I now believe the problem was we were checking two e-mail accounts back-to-back, in a loop like this (highly simplified):

while (running)
{
    CheckAccount1();
    CheckACcount2();
    Thread.Sleep(60000);
}

Apparently, there is some bug in the CDO COM components’ code that will cause errors if you reconnect too fast. Occasionally, the Exchange code must have completed so quickly that it didn’t provide enough time for the COM components to clean up properly before the next solution attempt. Solution?

while (running)
{
    CheckAccount1();
    Thread.Sleep(5000);
    CheckAccount2();
    Thread.Sleep(60000);
}

We implemented that change on a staging server that was also experiencing this problem and haven’t had a single reoccurrence since. The fix will be going into production very soon. No more 2AM alerts!


Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order:

Easily Unit Testing Event Handlers

In C#, If you need to unit test a class that fires an event in certain circumstances (perhaps even asynchronously), you need to handle a little more than just running some code and doing the assertion. You have to make sure your unit test waits for the event to be fired. Here’s one naive way of doing it, a WRONG way:

   1: private bool statsUpdated = false;
   2: private ManualResetEvent statsUpdatedEvent = new ManualResetEvent(false);
   3:
   4: [Test]
   5: public void CheckStats()
   6: {
   7:     BrickDatabase db = new BrickDatabase(tempFolder, maxCacheAge);
   8:
   9:     statsUpdated = false;
  10:     statsUpdatedEvent.Reset();
  11:
  12:     db.InventoryStatsUpdated += new EventHandler(db_InventoryStatsUpdated);
  13:     db.DoSomethingThatFiresEvent();
  14:
  15:     statsUpdatedEvent.WaitOne();
  16:
  17:     Assert.IsTrue(statsUpdated);
  18: }
  19:
  20: void db_InventoryStatsUpdated(object sender, EventArgs e)
  21: {
  22:     statsUpdated = true;
  23:     statsUpdatedEvent.Set();
  24: }

There are a number of things wrong with this:

  1. The class variables. More complex unit test class. Have to coordinate these variables across multiple functions.
  2. Since they are class variables, you will want to reuse them, but you’d better remember to reset the event and the boolean every time!
  3. Have to have two functions to do something really, really simple.
  4. The WaitOne() does not have a timeout, so if the wait is ever satisfied then statsUpdated is guaranteed to be true.

Here’s a better way of doing it, using anonymous methods in C# 2.0:

   1: [Test]
   2: public void CheckStats()
   3: {
   4:     BrickDatabase db = new BrickDatabase(tempFolder, maxCacheAge);
   5:     bool statsUpdated = false;
   6:     ManualResetEvent statsUpdatedEvent = new ManualResetEvent(false);
   7:
   8:     db.InventoryStatsUpdated += delegate
   9:     {
  10:         statsUpdated = true;
  11:         statsUpdatedEvent.Set();
  12:     };
  13:
  14:     db.DoSomethingThatFiresEvent();
  15:
  16:     statsUpdatedEvent.WaitOne(5000,false);
  17:
  18:     Assert.IsTrue(statsUpdated);
  19: }

Improvements?

  1. The event is just part of the method. Since the event handler is an anonymous delegate, it can access the enclosing method’s local variables.
  2. Added 5,000ms timeout to the WaitOne() function to prevent hanging of unit tests.

Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order:

DiskSlicer 1.2 Out

I made some minor updates to DiskSlicer. Mostly, some minor bug fixes, but also I added the much-requested ability to delete files directly from the program.

Go get it and enjoy!


Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order:

Difference between ConfigurationSettings and ConfigurationManager

If you upgraded a project from .Net 1.0/1.1 to .Net 2.0, and it used application configuration files, you will soon come across the compiler warning message

‘System.Configuration.ConfigurationSettings.AppSettings’ is obsolete: ‘This method is obsolete, it has been replaced by System.Configuration!System.Configuration.ConfigurationManager.AppSettings’   

You have to add a reference to the System.Configuration assembly, but once you do, you can just do a search and replace on ConfigurationSettings.AppSettings and replace it with ConfigurationManager.AppSettings and the functionality will remain the same.

The advantage of using ConfigurationManager is laid out in the MSDN docs, but briefly:

  1. Access sections other than appConfig, including connectionStrings, and your own custom sections
  2. Can read and write the configuration.
  3. It can have separate settings for the application and the users
  4. It’s extensible

Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order:

Instant Searching and Filtering in .Net – Part 4

This is the final part of my series on instant searching and filtering using C#. The only further issue that I wanted to cover was efficiently using a ListView when the items  will change so often.

ListViews already have the concept of a virtual mode, where the consumer of the class must supply the items that are displayed.

When the VirtualMode property is set to true, you also need to set the VirtualListSize property to tell the control how many items it needs to think it has.

With this simple starting point, we can have something like the following:

 

   1: _listControl.VirtualMode = true;
   2: _listControl.RetrieveVirtualItem += 
   3:     new RetrieveVirtualItemEventHandler(list_RetrieveVirtualItem);

and list_RetrieveVirtualItem could look like this:

 

   1: void list_RetrieveVirtualItem(object sender, RetrieveVirtualItemEventArgs e)
   2: {
   3:     e.Item = new ListViewItem();
   4:     e.Item.Text = "Item Text Here";
   5: }

Using this, it would be fairly easy to hook up our filtered results and away we go. But we have a few problems:

  • We don’t want to generate a new ListViewItem every time the list control asks us for one. This is extremely wasteful–thousands of objects would be springing to life and dying constantly, which is a very inefficient use of the garbage collector.
  • On the other hand, we don’t want to cache a ListViewItem object for every single line of text that was indexed. Only a few of them will be on the screen at a time, and we’ve already cached the text, so we should use that.

Therefore, we need a way to efficiently cache a limited number of ListViewItem objects and use them as necessary, updating their Text and Tag properties as necessary.

With that introduction, let’s get into the code.

Our own ListView

First things first: there’s a bug in the .Net framework that impacts virtual mode on ListView. So let’s derive our own to resolve it:

 

   1: class QuickList : System.Windows.Forms.ListView
   2: {
   3:     /// <summary>
   4:     /// This is needed to fix a bug in the framework
   5:     /// </summary>
   6:     public new int VirtualListSize
   7:     {
   8:         get
   9:         {
  10:             return base.VirtualListSize;
  11:         }
  12:         set
  13:         {
  14:             if (Items.Count > 0)
  15:             {
  16:                 EnsureVisible(0);
  17:             }
  18:             base.VirtualListSize = value;
  19:         }
  20:     }
  21: }
 

That’s all there is to that. If you would like more information about the bug, Shahar Prish wrote up an excellent description and workarounds.

With that out of the way, most of the action we’re concerned with will take place in a separate class, which I’ll call…

QuickListController

This controller class is responsible for coordinating the filter text, filtered results, and their display in the list control. In my own use of these concepts, I’ve further factored out the filtering mechanism to be outside of this controller, but in this sample it’s better to combine it all for ease of understanding.

So how will this work? We want a cache of only the items that will be displayed. Unfortunately, the ListView control is going to ask us for item 30,775 for example, if we’re scrolled down a huge list. We don’t want to cache 30,775 items when only the surrounding 20 items are displayed. Fortunately, it’s easy to track a minimal number of items if we know the first visible index

Here are the fields and properties we’ll need:

 

   1: class QuickListController
   2: {
   3:     private int _topIndex = -1;
   4:     private int _bottomIndex = -1;
   5:     private IIndexer<string> _indexer;
   6:     private IList<string> _allLines;
   7:     private IList<string> _currentResults;
   8:     private QuickList _listControl;
   9:     List<ListViewItem> _itemCache = new List<ListViewItem>();
  10:     private string _filter;
  11:     
  12:     #region Properties
  13:     public IList<string> CurrentResults
  14:     {
  15:         get
  16:         {
  17:             return _currentResults;
  18:         }
  19:     }
  20:     public string FilterText
  21:     {
  22:         get
  23:         {
  24:             return _filter;
  25:         }
  26:         set
  27:         {
  28:             _filter = value;
  29:         }
  30:     }
  31:     public IIndexer<string> Indexer
  32:     {
  33:         get
  34:         {
  35:             return _indexer;
  36:         }
  37:         set
  38:         {
  39:             _indexer = value;
  40:         }
  41:     }
  42:     public IList<string> AllLines
  43:     {
  44:         get
  45:         {
  46:             return _allLines;
  47:         }
  48:         set
  49:         {
  50:             _allLines = value;
  51:         }
  52:     }
  53:     #endregion
  54:     //methods, etc.
  55: }

Ok, lots of boring code. We track the top and bottom indexes of the visible items, we store our indexer, a list of all the results (for when no text is entered–no point in searching for nothing). _currentResults will point to either _allLines or the actual results. _itemCache is a list of ListViewItem objects that will serve as our cache. And, of course, we need the text we want to filter on.

You’ll notice that the filtered results are now strings instead of bools. In previous articles, we didn’t care about the actual results–just a count of them. Now, we need to display them, so the filtered items are the same as the keys–the lines of the text in a file.

Constructor

Construction is likewise straightforward:

   1: public QuickListController(QuickList list)
   2: {
   3:     _listControl = list;
   4:     _listControl.VirtualMode = true;
   5:     _listControl.CacheVirtualItems += 
   6:         new CacheVirtualItemsEventHandler(list_CacheVirtualItems);
   7:     _listControl.RetrieveVirtualItem += 
   8:         new RetrieveVirtualItemEventHandler(list_RetrieveVirtualItem);
   9: }

Our controller requires an instance of a QuickList object, which it then sets to virtual mode and listens for two events. The first, CacheVirtualItems, is triggered when the ListView needs a new range of objects. It gives us an opportunity to see what the new top and bottom will be and update our cache accordingly.

The RetrieveVirtualItem event is called when the ListView needs to get the item to display. As we’ll see below, handling both of these events is fairly straightforward.

CacheVirtualItems

 

   1: void list_CacheVirtualItems(object sender, CacheVirtualItemsEventArgs e)
   2: {
   3:     _topIndex = e.StartIndex;
   4:     _bottomIndex = e.EndIndex;
   5:  
   6:     int sizeNeeded = _bottomIndex - _topIndex + 2;//extra 2
   7:     int toCreate = sizeNeeded - _itemCache.Count;
   8:     if (toCreate > 0)
   9:     {
  10:         if (_itemCache.Capacity < sizeNeeded)
  11:         {
  12:             _itemCache.Capacity += toCreate;
  13:         }
  14:         for (int i = 0; i < toCreate; i++)
  15:         {
  16:             _itemCache.Add(new ListViewItem());
  17:         }
  18:     }
  19: }

We save the top and bottom indexes (we actually never use the bottom index), and we then calculate how many total items we need, how many new ones we need to create, and then create new ones if necessary.

You’ll notice that there is no mechanism for removing items from the cache. As we know, another name for cache with a bad (no) removal policy is a memory leak, but since the size of this cache is limited by screen characteristics, and in the most dire circumstance shouldn’t be more than a few hundred, we’ll let it slide (just this once).

RetrieveVirtualItem

 

   1: void list_RetrieveVirtualItem(object sender, RetrieveVirtualItemEventArgs e)
   2: {
   3:     if (e.ItemIndex < _topIndex)
   4:     {
   5:         //should never get here
   6:         Debug.Assert(false);
   7:         _topIndex = e.ItemIndex;
   8:     }
   9:     
  10:     e.Item = GetListViewItem(e.ItemIndex, _topIndex);
  11: }

You’ll notice that if block shouldn’t ever be called, but during debugging I did have an instance where that was the case. It may have been a bug that has since been fixed, but I left it in for safety.

The actual work is done another function:

GetListViewItem

 

   1: private ListViewItem GetListViewItem(int itemIndex, int topIndex)
   2: {
   3:     if (itemIndex < 0 || topIndex < 0)
   4:     {
   5:         return null;
   6:     }
   7:     int cacheIndex = itemIndex - topIndex;
   8:     
   9:     Debug.Assert(cacheIndex < _itemCache.Count);
  10:     ListViewItem item = _itemCache[cacheIndex];
  11:     item.Text = _currentResults[itemIndex];
  12:     //could set tag if you have more complex objects
  13:     //item.Tag = _currentResults[itemIndex];
  14:     return item;
  15: }

Now that you understand how this is working, this function is really simple. Again, we do a simple sanity check. We calculate the index into our cache, which is simply the offset from the top index. After retrieving the pre-created item, we set its text (you could also set the item’s tag, color, or any other property here).

Now, some of you are wondering, “but wait, where do you set the VirtualListSize property?” Yes, that is rather important, isn’t it? How is the ListView control supposed to know how many items to ask for? For that matter, where is the actual filtering done?

The Filter

We have one more function, this one a public interface to the class’s consumer:

 

   1: public void RefreshList()
   2: {
   3:     if (string.IsNullOrEmpty(_filter))
   4:     {
   5:         _currentResults = _allLines;
   6:     }
   7:     else
   8:     {
   9:         _currentResults = _indexer.Lookup(_filter);
  10:     }
  11:     _listControl.VirtualListSize = _currentResults.Count;
  12: }

If the filter text is empty, we set our current results to the full set of lines. Otherwise, run the filter. Then we set our control’s virtual size to the number of results.

Conclusion

If you’ve struggled this far, congratulations. If you find anything in any of these articles useful, please leave a comment and let me know! If you find bugs or problems with my approach, I’d love to hear about better ways of accomplishing things.

The project download contains a GUI that hooks up these classes to an easy-to-use interface.

Also, if you’ve found this useful, please Buy me a Lego!

Download the project. (run the GuiSearch project–I changed TestSearch to a library file)

Download Les Misérables for testing.

 

Technorati Tags: , , , ,

Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order:

Tip: Mouse back and forward work in Visual Studio 2005 too

You know how you can use the extra mouse buttons to move back and forward in Internet Explorer?

The same shortcuts work in Visual Studio. Suppose you right-click on a function call, and select Go To Definition. Once you’re done looking at the definition, hit the back button on your mouse: You’re taken right back to where you came from!

(The keyboard equivalents are Ctrl+ – and Ctrl+Shift+ -)

Technorati Tags:


Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order:

Easily Getting Last Two Digits of Year

This is an easy one, but some people don’t know it: if you need the last two digits of a year, say 2007, it’s very easy to get, without converting to a string and getting the last two characters:

 

int lastTwo = year % 100;

Now that you know, it’s obvious, right?

And if you want the last three digits? number % 1000. In general:

int lastN = number % (pow(10, digitsRequired));

What about first N digits? similarly easy:

 

int firstTwo = number / 100;
int firstN = number / pow(10,digitsRequired);

 
Now  you know, and knowing if half the battle.

Check out my latest book, the essential, in-depth guide to performance for all .NET developers:

Writing High-Performance.NET Code, 2nd Edition by Ben Watson. Available for pre-order: