.Net microbenchmarking

In a recent discussion on StackOverflow about performance of arrays vs. lists and for vs. foreach, Jon Skeet created a little microbenchmarking framework and posted his detailed findings on his blog.

First, my local results of the benchmark. I compiled them as Release under VS 2008 SP1. Running without debugging on a Q6600@2.40GHz, .NET 3.5 SP1. They pretty exactly match Jon’s numbers so I won’t duplicate them here, but you can find them in the page’s source.


============ Doubles ============
============ double[] ============
For 1,00
ForHoistLength 1,00
ForEach 1,00
IEnumerableForEach 8,28
Enumerable.Sum 8,27

============ List ============
For 2,00
ForHoistLength 1,43
ForEach 6,02
IEnumerableForEach 14,03
Enumerable.Sum 14,04

============ Ints ============
============ int[] ============
For 1,00
ForHoistLength 2,06
ForEach 1,38
IEnumerableForEach 15,46
Enumerable.Sum 16,06

============ List ============
For 2,84
ForHoistLength 3,53
ForEach 4,86
IEnumerableForEach 26,33
Enumerable.Sum 26,33

Out of interest how the differences are when there is more to do in the loop’s body, I added a test suite that formatted the sum as string and appended it to a StringBuilder:


var sbArraySuite = TestSuite.Create("build Strings from Array", doubleArray, String.Join("", doubleArray.Select(d => String.Format("{0:0.00}\n", d)).ToArray()))
    .Add(input =>
    {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < input.Length; i++)
            sb.AppendFormat("{0:0.00}\n", input[i]);
        return sb.ToString();
    }, "For")
    .Add(input =>
    {
        StringBuilder sb = new StringBuilder(); int length = input.Length;
        for (int i = 0; i < length; i++)
            sb.AppendFormat("{0:0.00}\n", input[i]);
        return sb.ToString();
    }, "ForHoistLength")
    .Add(input =>
    {
        StringBuilder sb = new StringBuilder();
        foreach (double d in input)
            sb.AppendFormat("{0:0.00}\n", d);
        return sb.ToString();
    }, "ForEach")
    .Add(IEnumerableForEachToString)
    .RunTests();

    ...

    static string IEnumerableForEachToString(IEnumerable<double> input)
    {
        StringBuilder sb = new StringBuilder();
        foreach (double d in input)
        {
            sb.AppendFormat("{0:0.00}\n", d);
        }
        return sb.ToString();
    }

Which gives these results:

============ build Strings from Array ============
For                        1,01
ForHoistLength             1,01
ForEach                    1,01
IEnumerableForEachToString 1,03

============ build Strings from List ============
For                        1,00
ForHoistLength             1,00
ForEach                    1,02
IEnumerableForEachToString 1,05

Suddenly the relative differences between for/foreach and array/list collapse to 1-5%. Which confirms my intuition that algorithms usually matter much more and such optimisations are quite useless. At least we all had good fun on a saturday ;-)

Leave a Reply