dd if=/dev/zero of=/dev/vg/lv

Since I had ten logical volumes, I also ran ten instances of dd in parallel. They were on a RAID and I was decommissioning the server, so I didn’t really care about performance. Speaking of which, I like to spy on running processes, call my a techno-voyeur if you want!

Anyways, vmstat was telling me that the system was chugging along nicely, reading(sic!) and writing approximately 16MB/s each. Something was clearly wrong. Note the “bi” and “bo” columns, denoting kB/s read and written:

david@hetz:~$ vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
[...]
 0 12   6416 270676 1601220   9628    0    0 16628 16553 4481 7638  1 21 26 52

I had forgotten to specify the block size which dd should use to make the transfer. And if the program is writing the zeroes one by one into the target file, the destination has to be read before it is modified, since the kernel cannot (and should not!) guess that the rest of the block (which is all we’re talking about at this stage) will be overwritten too.

To test my hypotheses, I restarted the processes and told dd to use nice 1MB sized blocks. I had ten threads to keep the hardware busy, so that shouldn’t make any problems.

dd if=/dev/zero of=/dev/vg/lv bs=1M

Really, in this configuration, the system stabilized around a nice 55MB/s writes. The kernel was able to recognize that the 1MB sized writes would cover complete blocks and that their content would be overwritten. No need to load them beforehand:

david@hetz:~$ vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
[...]
 0 12   6416  16308 1749688   5312    0    0     0 55644  551 1576  0 23  1 76

While I was waiting for the last process to finish, I noticed that throughput had risen to 60MB/s:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
[...]
 1  3   6416  15336 1750492   6088    0    0      4 62892  545  232  0 28 32 39

To summarize: having only one process running is ~10% faster than having ten processes running and using a non-trivial blocksize is more than three times faster than specifying none at all.

First, my local results of the benchmark. I compiled them as Release under VS 2008 SP1. Running without debugging on a Q6600@2.40GHz, .NET 3.5 SP1. They pretty exactly match Jon’s numbers so I won’t duplicate them here, but you can find them in the page’s source.

============ Doubles ============
============ double[] ============
For 1,00
ForHoistLength 1,00
ForEach 1,00
IEnumerableForEach 8,28
Enumerable.Sum 8,27

============ List ============
For 2,00
ForHoistLength 1,43
ForEach 6,02
IEnumerableForEach 14,03
Enumerable.Sum 14,04

============ Ints ============
============ int[] ============
For 1,00
ForHoistLength 2,06
ForEach 1,38
IEnumerableForEach 15,46
Enumerable.Sum 16,06

============ List ============
For 2,84
ForHoistLength 3,53
ForEach 4,86
IEnumerableForEach 26,33
Enumerable.Sum 26,33

Out of interest how the differences are when there is more to do in the loop’s body, I added a test suite that formatted the sum as string and appended it to a StringBuilder:

var sbArraySuite = TestSuite.Create("build Strings from Array", doubleArray, String.Join("", doubleArray.Select(d => String.Format("{0:0.00}\n", d)).ToArray()))
    .Add(input =>
    {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < input.Length; i++)
            sb.AppendFormat("{0:0.00}\n", input[i]);
        return sb.ToString();
    }, "For")
    .Add(input =>
    {
        StringBuilder sb = new StringBuilder(); int length = input.Length;
        for (int i = 0; i < length; i++)
            sb.AppendFormat("{0:0.00}\n", input[i]);
        return sb.ToString();
    }, "ForHoistLength")
    .Add(input =>
    {
        StringBuilder sb = new StringBuilder();
        foreach (double d in input)
            sb.AppendFormat("{0:0.00}\n", d);
        return sb.ToString();
    }, "ForEach")
    .Add(IEnumerableForEachToString)
    .RunTests();

    ...

    static string IEnumerableForEachToString(IEnumerable<double> input)
    {
        StringBuilder sb = new StringBuilder();
        foreach (double d in input)
        {
            sb.AppendFormat("{0:0.00}\n", d);
        }
        return sb.ToString();
    }

Which gives these results:

============ build Strings from Array ============
For                        1,01
ForHoistLength             1,01
ForEach                    1,01
IEnumerableForEachToString 1,03

============ build Strings from List ============
For                        1,00
ForHoistLength             1,00
ForEach                    1,02
IEnumerableForEachToString 1,05

Suddenly the relative differences between for/foreach and array/list collapse to 1-5%. Which confirms my intuition that algorithms usually matter much more and such optimisations are quite useless. At least we all had good fun on a saturday