2 minute read

Micro-Benchmarking

Micro-optimising has a bad reputation although I’d argue that knowledge of such things can help you write better code. We should also make the distinction clear between this and micro-benchmarking, on the other hand, which is a little risky but a lot safer if you know how to do it right.

Now, if you’re making that mental link with premature optimization right now, off the back of a 44 year old (mis)quote from Donald Knuth, there are lots of good, more recent and balanced arguments on the subject out there now; and plenty of tools that help you do it better. I think some of the confusion comes from a misunderstanding of where it sits within a much broader range of knowledge and tools for performance. Remember, he did say that ‘we should not pass up our opportunities in that critical 3%’, which sounds about right in my experience.

Common Pitfalls

There’s loads of pitfalls going old-school with System.Diagnostics.Stopwatch such as:

  • not enough iterations of operations that take a negligible amount of time
  • forgetting to disable compiler optimizations
  • inadvertently measuring code not connected to what you are testing
  • failing to ‘warm-up’ for account for JIT costs and processor caching
  • forgetting to separate setup code from code under test
  • and so on…

Enter Adam Sitniks’ BenchmarkDotNet. This deals with all the problems above and more.

BenchmarkDotNet

It’s available as a NuGet package:

And has some excellent documentation:

You have a choice of configuration methods via objects, attributes or fluent. Things you can configure include:

  • Compare RyuJIT (default since .NET46 for x64 and since Core 2.0 for x86) and Legacy JIT
  • Compare x86 with x64
  • Compare Core with full framework (aka Clr)
  • JIT inlining (and tail calls, which can be confusingly similar to inlining in 64-bit applications in my experience)
  • You can even test the difference between Server GC and Workstation GC from my last tip

A Very Simple Example

For many scenarios, it is fine to just fire it up with the standard settings. Here’s an example of where I used it to get some comparisons between DateTime, DateTimeOffset  and NodaTime.

  • [ClrJob, CoreJob]: I used the attribute approach to configuration, decorating the class to make BenchmarkDotNet run the tests on .NET full framework and also Core.
  • [Benchmark]: used to decorate each method I wanted to benchmark
[ClrJob, CoreJob]
public class DateTimeBenchmark {
    private DateTime dateNow;
    private DateTimeOffset dateOffset;
    private ZonedDateTime nowInIsoUtc;

    [Benchmark]
    public void DateTime_Now() {
        dateNow = DateTime.Now;
    }

    [Benchmark]
    public void DateTime_Utc() {
        dateNow = DateTime.UtcNow;
    }

    [Benchmark]
    public void DateTimeOffset_Now() {
        dateOffset = DateTimeOffset.Now;
    }

    [Benchmark]
    public void DateTimeOffset_Utc() {
        dateOffset = DateTimeOffset.UtcNow;
    }

    [Benchmark]
    public void NodaTime_ZonedDateTime() {
        nowInIsoUtc = SystemClock.Instance.GetCurrentInstant().InUtc();
    }
}

A call to get things started:

static void Main(string[] args)
{
    var summary = BenchmarkRunner.Run<DateTimeBenchmark>();
}

Note if you want to try this code, you’ll need to install the NuGet packages for BenchmarkDotNet and NodaTime.

Output:

Mark of the web

Obviously, not a substitute for understanding the underlying implementation details of the DateTime  class in the Base Class Library (BCL); but a quick and easy to initially identify problem areas. In fact, this was just a small part of a larger explanation I gave to a colleague around ISO 8601, time zones, daylight saving and the pitfalls of DateTime.Now .

One Thing That Caught Me Out

One gotcha is, if you are testing Core and full framework, make sure you create a new Core console application and edit your csproj file, switching out <TargetFramework>  for e.g.

TargetFrameworks&gt;netcoreapp2.1;net46&lt;/TargetFrameworks&gt;

Comments