How to Use BenchmarkDotNet: 6 Simple Performance-Boosting Tips to Get Started

03/05/2024

BenchmarkDotNet C# / .NET / dotnet CodeProject Programming Software Engineering Tutorial .NET Benchmark C#code coding CSharp DotNet dotnet core performance

As software engineers, we are always striving for high performance and efficiency in our code. Whether it's optimizing algorithms or fine-tuning data structures, every decision we make can have a significant impact on the overall performance of our applications. One powerful way that can help us accurately measure the performance of our code is a process called benchmarking and we'll look at how to use BenchmarkDotNet with our C# code.

In this article, we'll explore the ins and outs of using BenchmarkDotNet to benchmark C# code efficiently. We'll be starting with the setup and I'll guide you through examples of how to write benchmarks and run them as well. By the end, you'll be able to write and run benchmarks on your C# code using BenchmarkDotNet effectively.

Let's jump into it!

What's In This Article: How to use BenchmarkDotNet

Remember to check out these platforms:

// FIXME: social media icons coming back soon!

1 - Installing and Setting Up BenchmarkDotNet

The first step is to install the BenchmarkDotNet package in your project. You can do this by opening the NuGet Package Manager in Visual Studio and searching for "BenchmarkDotNet." Select the latest version and click on the Install button to add it to your project. There are no other external dependencies or anything fancy that you need to do to get going.

As you'll see in the following sections, it's purely code configuration after the NuGet package is installed! It's worth mentioning that BenchmarkDotNet automatically performs optimizations during benchmarking C# code, such as JIT warm-up and running benchmarks in a random order, to provide accurate and unbiased results. But this all happens without you having to do anything special.

Keep in mind, that just like with writing tests, you'll very likely want to have your benchmark code in a dedicated project. This will allow you to release your core code separately from your benchmarks. There are probably not a lot of great reasons to deploy your benchmark code with your service or ship your benchmark code to your customers.

2 - Getting Started With Benchmark Methods in BenchmarkDotNet

When it comes to writing benchmark methods using the BenchmarkDotNet framework, there are a few things we need to configure properly. In this section, I'll guide you on how to write effective benchmark methods that accurately measure the performance of your C# code. This video on getting started with BenchmarkDotNet is a helpful guide as well:

Structure Your Benchmark Code

It's important to structure your benchmark code properly to ensure accurate and efficient benchmarks. Follow these guidelines:

Create a separate class for benchmarks: Start by creating a dedicated class for your benchmarks. This keeps your benchmark code organized and separate from the rest of your application code.
Apply the correct [XXXRunJob] attribute: Select the appropriate type of benchmark job you'd like to run by marking your benchmark class with one of the following attributes [ShortRunJob], [MediumRunJob], [LongRunJob], or [VeryLongRunJob].
Apply the [MemoryDiagnoser] attribute: To enable memory measurements during the benchmark, apply the [MemoryDiagnoser] attribute to your benchmark class. This attribute allows you to gather memory-related information along with execution time. If you're strictly concerned about runtime and not memory, you can omit this.

Note that your class cannot be sealed. I'd recommend just sticking to a standard public class definition with the appropriate attributes from above. You can have your benchmark class inherit from something else though, so if you find there's a benefit to using some inheritance here for reusability, that might be a viable option.

Writing Benchmark Methods

Benchmark methods are where you define the code that you want to measure. Here are some tips for writing benchmark methods using BenchmarkDotNet:

Apply the [Benchmark] attribute: Each benchmark method needs the [Benchmark] attribute. This attribute tells BenchmarkDotNet that this method should be treated as a benchmark.
Avoid setup costs in benchmark methods: Benchmark methods should focus on measuring the performance of the code itself, not the cost of initializing your benchmark scenario.
Avoid allocations and overuse of memory: Benchmark methods should not be concerned with the overhead caused by memory allocations. Minimize allocations and reduce memory usage within your benchmark methods to get accurate performance measurements.

Note that BenchmarkDotNet handles all of the warmup for you - You don't need to go out of your way to manually code in doing iterations ahead of time to get things to a steady state before benchmarking C# code.

Example Benchmark Method:

Here's an example of a benchmark method in BenchmarkDotNet:

[Benchmark]
public void SimpleMethodBenchmark()
{
    for (int i = 0; i < 1000; i++)
    {
        // Execute the code to be measured
        MyClass.SimpleMethod();
    }
}

In this example, the [Benchmark] attribute is applied to the SimpleMethodBenchmark method, indicating that it should be treated as a benchmark. Suppose we were using an instance method instead of a static method as illustrated. In that case, we'd want to instantiate the class OUTSIDE of the benchmark method -- especially if we need to create, configure, and pass in dependencies. Minimize (read: eliminate) the amount of work done in the method that isn't what you are trying to benchmark.

Remember, if you're interested in memory in addition to the runtime characteristics, make sure to apply the [MemoryDiagnoser] attribute to the benchmark class -- not the method that is the benchmark.

3 - Running Benchmarks with BenchmarkDotNet

When it comes to running benchmarks using BenchmarkDotNet in C#, there are a few important considerations to keep in mind. In this section, I'll explain how to run benchmarks, discuss different scenarios and options, and provide code examples to help you get started. You can follow along with this video on how to run your BenchmarkDotNet benchmarks as well:

Running Benchmarks With BenchmarkRunner

The BenchmarkRunner class provides a very simple mechanism for running all of the Benchmarks that you provide it from a type, list of types, or an assembly. The benefit of this is in the simplicity as you can just run the executable and it will go immediately run all of the benchmarks that you've configured the code to run.

You can check out some example code on GitHub or below:

using BenchmarkDotNet.Running;

var assembly = typeof(Benchmarking.BenchmarkDotNet.BenchmarkBaseClass.Benchmarks).Assembly;

BenchmarkRunner.Run(
    assembly,
    args: args);

In the code above, we are simply specifying an assembly for one of our benchmarks. However, you could use Assembly.GetExecutingAssembly or find other ways to list the types you're interested in.

Running Benchmarks With BenchmarkSwitcher

The BenchmarkSwitcher class is very similar - but the behavior is different in that you can filter which benchmarks you'd like to run. There are some slight API differences in that the BenchmarkRunner allows you to specify a collection of assemblies where the BenchmarkRunner (at the time of writing) does not.

Here's a code example, or you can check out the GitHub page:

using BenchmarkDotNet.Running;

var assembly = typeof(Benchmarking.BenchmarkDotNet.BenchmarkBaseClass.Benchmarks).Assembly;


BenchmarkSwitcher
    // used to load all benchmarks from an assembly
    .FromAssembly(assembly)
    // OR... if there are multiple assemblies,
    // you can use this instead
    //.FromAssemblies
    // OR... if you'd rather specify the benchmark
    // types directly, you can use this
    ///.FromTypes(new[]
    ///{
    ///    typeof(MyBenchmark1),
    ///    typeof(MyBenchmark2),
    ///})
    .Run(args);

As you will note in the code comments above, there are several ways that we can call the BenchmarkSwitcher. The primary difference between these two methods is that the switcher will allow you to provide user input or filter on the commandline.

4 - Customizing Benchmark Execution

BenchmarkDotNet provides various options to customize the execution of your benchmarks, allowing you to fine-tune the benchmarking process according to your needs. Two important options to consider are the iteration count and warm-up iterations.

Configuring Parameters for Benchmarks

If we want to have variety in our benchmark runs, we can use the [Params] attribute on a public field. This is akin to using the xUnit Theory for parameterized tests, if you're familiar with that. For each field you have marked with this attribute, you'll essentially be building a matrix of benchmark scenarios to go run.

Let's check out some example code:

[MemoryDiagnoser]
[ShortRunJob]
public class OurBenchmarks
{
    List<int>? _list;

    [Params(1_000, 10_000, 100_000, 1_000_000)]
    public int ListSize;

    [GlobalSetup]
    public void Setup()
    {
        _list = new List<int>();
        for (int i = 0; i < ListSize; i++)
        {
            _list.Add(i);
        }
    }

    [Benchmark]
    public void OurBenchmark()
    {
        _list!.Sort();
    }
}

In the code above, we have a ListSize field which is marked with [Params]. This means that in our [GlobalSetup] method, we're able to get a new value for each variation of the benchmarking matrix we want to run. In this case, since there's only one parameter, there will only be a benchmark for each value of ListSize -- so 4 different benchmarks based on the 4 different values specified.

Adjusting Iteration Count Per Benchmark

BenchmarkDotNet allows you to control the number of iterations for each benchmark method. By default, each benchmark is executed a reasonable number of times to obtain reliable measurements. However, you can adjust the iteration count by applying the [IterationCount] attribute to individual benchmark methods, specifying the desired number of iterations.

[Benchmark]
[IterationCount(10)] // Custom iteration count
public void MyBenchmarkMethod()
{
    // Benchmark code here
}

I should note that in practice, I have not personally had to configure the iteration count manually.

Custom Warm-up Iterations Per Benchmark

Warm-up iterations are executed before the actual benchmark starts, allowing the JIT compiler and CPU caches to warm up. This helps eliminate any performance inconsistencies caused by the initial compilation of the benchmark method. You can customize the number of warm-up iterations using the [WarmupCount] attribute.

[Benchmark]
[WarmupCount(5)] // Custom warm-up count
public void MyBenchmarkMethod()
{
    // Benchmark code here
}

Like the iteration count, I have also not had an issue with the default warmup. I'd advise that you leave these as defaults unless you know what you're doing to tune these accordingly.

5: Analyzing and Interpreting Benchmark Results from BenchmarkDotNet

Benchmarking is an important part of the software development process when it comes to optimizing performance. After running benchmarks using BenchmarkDotNet, it's important to be able to accurately analyze and interpret the results. In this section, I'll guide you through the process of analyzing benchmark results and identifying potential performance improvements.

Establishing a Baseline Benchmark in BenchmarkDotNet

When analyzing benchmark results, several key metrics can provide valuable insights into the performance of your code. But what can be challenging is understanding what you're trying to compare against. If you simply have two implementations you are trying to compare against, it may not feel that challenging. However, when you're comparing many you likely want to establish a baseline to see your improvements (or regressions).

Here's how we can make a benchmark method as a "baseline":

[Benchmark(Baseline = true)]
public void Baseline()
{
    // Code under benchmark
}

Interpreting Benchmark Results

Once you have the benchmark results and understand the key metrics, it's important to interpret the findings to identify potential performance improvements. Here are a few examples of how to interpret the benchmark results:

Identifying Bottlenecks: Look for outliers or significantly higher execution times in certain benchmarks. This could indicate potential bottlenecks in your code that need to be optimized.
Comparing Performance: Compare the mean or median execution times of different benchmark methods or different versions of your code. This can help you identify which approach or version is more efficient.
Spotting Variability: Pay attention to the standard deviation and percentiles to identify any significant variability in the benchmark results. This can help you identify areas where performance optimizations could make a difference.

I find that I am spending most of my time looking at the median/mean to see what stands out as faster and slower. If you need to dive deeper into the statistical analysis and you just need a primer, I have found even Wikipedia provided a reasonable starting point.

6: Optimizing C# Code with BenchmarkDotNet

As software engineers, we constantly strive to write code that is not only functional but also performs efficiently. BenchmarkDotNet is a powerful tool that helps us measure and compare the performance of our code. By analyzing the results provided by BenchmarkDotNet, we can identify areas where our code may be underperforming and optimize it to achieve better results. In this section, I'll share some techniques for optimizing C# code based on the results we'd gather from running our benchmarks. You can check out this video on measuring and tuning iterator performance for a practical example:

Identifying Performance Bottlenecks

One of the first steps in optimizing C# code is identifying the performance bottlenecks. BenchmarkDotNet allows us to measure the execution time of our code and compare different implementations. By analyzing the benchmark results, we can pinpoint the areas that are taking up the most time.

Let's consider an example where we have a loop that performs a computation on a large array. We can use BenchmarkDotNet to measure the execution time of different implementations of this loop and identify any potential bottlenecks.

[ShortRunJob]
public class ArrayComputation
{
    private readonly int[] array = new int[1000000];

    [GlobalSetup]
    public void Setup()
    {
        // TODO: decide how you want to fill the array :)
    }

    [Benchmark]
    public void LoopWithMultipleOperations()
    {
        for (int i = 0; i < array.Length; i++)
        {
            array[i] += 1;
            array[i] *= 2;
            array[i] -= 1;
        }
    }

    [Benchmark]
    public void LoopWithSingleOperation()
    {
        for (int i = 0; i < array.Length; i++)
        {
            array[i] = (array[i] + 1) * 2 - 1;
        }
    }
}

In this example, we have two benchmark methods LoopWithMultipleOperations and LoopWithSingleOperation. The first method performs multiple operations on each element of the array, while the second method combines the operations into a single computation. By comparing the execution times of these two methods using BenchmarkDotNet, we can determine which implementation is more efficient.

Recall from the earlier sections to we could parameterize this for different sizes! Sometimes this is necessary to see if we have different behaviors under different scenarios, so it's worth exploring beyond just the surface.

Optimizing Loops and Reducing Memory Allocations

Loops are often an area where we can optimize our code for better performance. Inefficient loops can result in unnecessary memory allocations or redundant computations. BenchmarkDotNet can help us identify such issues and guide us in optimizing our code.

Consider the following example where we have a loop that concatenates strings, and we're making sure to use a [MemoryDiagnoser] as well:

[MemoryDiagnoser]
[ShortRunJob]
public class StringConcatenation
{
    private readonly string[] strings = new string[1000];

    [GlobalSetup]
    public void Setup()
    {
        // TODO: decide how you want to fill the array :)
    }

    [Benchmark]
    public string ConcatenateStrings()
    {
        string result = "";
        for (int i = 0; i < strings.Length; i++)
        {
            result += strings[i];
        }

        return result;
    }

    [Benchmark]
    public string StringBuilderConcatenation()
    {
        StringBuilder stringBuilder = new StringBuilder();
        for (int i = 0; i < strings.Length; i++)
        {
            stringBuilder.Append(strings[i]);
        }

        return stringBuilder.ToString();
    }
}

In this example, we have two benchmark methods: ConcatenateStrings and StringBuilderConcatenation. The first method uses string concatenation inside a loop, which can result in frequent memory allocations and poor performance. The second method uses a StringBuilder to efficiently concatenate the strings. By comparing the execution times of these two methods using BenchmarkDotNet, we can observe the performance difference and validate the effectiveness of using a StringBuilder for string concatenation.

Now You Know How to Use BenchmarkDotNet!

BenchmarkDotNet is a valuable tool for accurately and efficiently benchmarking C# code. Throughout this article, we explored tips for using BenchmarkDotNet effectively. By leveraging these, you can accurately measure and optimize the performance of your C# code. Improved performance can lead to more efficient applications, better user experiences, and overall enhanced software quality!

Remember, benchmarking is an iterative process, and there are other resources and tools available that can further assist you in optimizing your C# code. Consider exploring profiling tools, performance counters, and other performance analysis techniques to gain deeper insights into your application's performance. If you found this useful and you're looking for more learning opportunities, consider subscribing to my free weekly software engineering newsletter and check out my free videos on YouTube!

Table of Contents