Add `benchmark crud-modes`

[PhrozenByte]

/kind enhancement

Related: #9165 and #9166

Following the discussion in #9104 it's quite hard to properly benchmark Borg's various encryption modes right now. First, it's hard to benchmark multiple modes for comparison, because the user is required to manually create and later delete test repos using the designated encryption modes. Second, it's hard to use Borg's benchmark results to draw conclusions, see #9165 and #9166.

I thus like to suggest adding a new benchmark crud-modes command (name is open for debate, didn't find a better one…) that is basically a wrapper for benchmark crud to create and later delete test repos with different encryption modes. Besides making benchmarking easier, it also promotes actually comparing the performance of different encryption modes, and not deciding based on a feeling what might be best.

This command should be added to both Borg 1.4 and Borg 2. The command should benchmark all available encryption modes after another. AFAIK there's no benefit in benchmarking both repokey and keyfile, so Borg should just benchmark one. Basically, benchmark crud-mode should do the following (assuming Borg 1.4; with Borg 2 the commands and encryption modes are different):

export BORG_PASSPHRASE=mypassphrase
for e in none authenticated authenticated-blake2 repokey repokey-blake2; do
    borg init -e $e ./$e

    echo "$e $(printf '%.0s=' $(seq $(( 64 - ${#e} - 1 ))))"
    mkdir ./bench_data
    borg benchmark crud ./$e ./bench_data

    borg delete ./$e
    rmdir bench_data
done

The results shown by benchmark crud-modes are identical to that of benchmark crud, with the only exception being that the results are grouped by encryption mode. Additionally, it might be reasonable to also print a summary:

Including a summary

I'm not entirely sure yet whether benchmark crud-modes should also analyse the results similar to what I did manually in #9104 (comment) and #9104 (comment), but I tend to "yes". I don't necessarily mean drawing conclusions, that's a bit hard to implement, but e.g. calculating and printing averages for various commands and samples. For example, benchmark crud-modes could include a summary section that prints the averages over all samples per command and encryption mode. Such summary section could look like the following (using results from #9104 and including just two encryption modes). WDYT?

summary ========================================================
C-*-*           466.94 MB/s with none encryption (average of all samples: 1.87s)
R-*-*           376.14 MB/s with none encryption (average of all samples: 1.64s)
U-*-*         4,543.96 MB/s with none encryption (average of all samples: 0.41s)
D-*-*        28,203.73 MB/s with none encryption (average of all samples: 0.05s)
C-*-*           487.28 MB/s with authenticated-blake2 encryption (average of all samples: 1.67s)
R-*-*           495.45 MB/s with authenticated-blake2 encryption (average of all samples: 1.22s)
U-*-*         3,672.36 MB/s with authenticated-blake2 encryption (average of all samples: 0.44s)
D-*-*        10,657.19 MB/s with authenticated-blake2 encryption (average of all samples: 0.10s)

Support of --json-lines

Similar to the suggested benchmark crud --json-lines (see #9165), the new benchmark crud-modes command should also support the --json-lines option that yields the same output as benchmark crud --json-lines, but additionally includes an encryption key and includes the mode used in its id key. Here's an example line with --json-lines:

{"id": "authenticated_C-Z-BIG", "encryption": "authenticated", "command": "create1", "sample": "Z-BIG", "sample_count": 10, "sample_size": 100000000, "sample_random": false, "time": 4.460000, "io": 224320000}

With --json-lines the summary section should be omitted.

[ThomasWaldmann]

Not sure if that is really needed, the cpu benchmark already compares the different crypto, hashing and compression algorithms and the rest of the borg code behaves similarly.

Minor differences detected in the crud tests can also come from other influences (OS or other app activity), the crypto/hashing is usually only a small part of the time measured when running on relatively modern CPUs.

[PhrozenByte]

That's basically exactly the reason why I'm suggesting it 😄 benchmark cpu gives a crypto/hashing-focused view on things. Reality is wildly different, precisely because Borg is doing a whole lot of other things.

Before doing the benchmark comparisons in #9104 (comment) and #9104 (comment) I naturally had some expectations (just based on theory for my Borg 1.4 benchmarks, but for Borg 2 I based my expectations on benchmark cpu). However, I greatly overestimated the effects. Such overestimates can easily sway decisions:

• BLAKE2b was supposed to be ~32% faster than SHA256 according to benchmark cpu. This would definitely yield me to use BLAKE2b. However, in reality it's "just" ~10% faster - because, of course, Borg is doing way more things, reducing the impact. Now also considering that this decision is more or less final, and that newer hardware likely gains SHA256 hardware acceleration removing this difference, I would choose SHA256 instead.

• CHACHA20-POLY1305 was supposed to be a whopping ~75% slower than AES256-OCB. In reality it yields more or less the same performance. No idea why btw…

• I had virtually no idea what the comparison of aes-256-ctr-blake2b with aes-ocb (i.e. basically Borg 1.4 vs. Borg 2) would mean in practice.

Minor differences detected in the crud tests can also come from other influences (OS or other app activity)

That's absolutely true.

That's why I was benchmarking on a clean system: Fresh Arch Linux booted from external media, ram disk as storage, nothing else but the base system running, and giving the system a five minute cooldown phase between each benchmark. I learned my lesson from #9005 (comment) 😆

---

Just to be clear: This is no must-have. One can indeed do the same with some wrapper script, including both creating a summary and support of --json-lines.

However, the reason why I still feel like that this is a beneficial addition is because we're making it rather hard for users to make an informed decision about what encryption mode to use (which I suspect is why people want to keep none mode, #9104). I wanted to tell people in the docs how to make this decision right. I started writing a section (#9103 resp. its still planned follow-up), but threw it away because I felt like that it's getting way too complex. IMHO this should be a single command.

But as I said earlier in #9104 (comment), I can absolutely understand that this is no priority since it creates an one-time benefit for users at best. So, absolutely no hard feelings if you think it's not worth the effort.