Speedreader's Digest: Hashing Data on Linux using AF_ALG

(I seem to have written this in the tone of an advertisment for household cleaning products. Sorry.)

Do you need to quickly hash (e.g. SHA-1) content on Linux? Want to avoid linking against bloated crypto libraries? There's no need to roll your own; use the Linux kernel's AF_ALG functionality! It's fast, supports many hash algorithms, and plumbs easily into existing network or disk I/O pipelines.

Look at these benchmark results, comparing io_uring kdigest and openssl performance:

kdigest

FILE SIZE	512 bytes	4096	65536	1M	16M	32M
md5	0.0010984 +-1.44%	0.0008265 +-2.65%	0.0009556 +-1.52%	0.0024098 +-0.35%	0.0266533 +-0.06%	0.0521402 +-0.19%
sha1	0.0011012 +-1.59%	0.0009430 +-2.21%	0.0009173 +-1.89%	0.0019097 +-1.29%	0.0186466 +-0.18%	0.0361400 +-0.13%
sha224	0.0010983 +-1.29%	0.0010970 +-1.72%	0.0010350 +-1.32%	0.0036209 +-0.42%	0.0425834 +-0.06%	0.0841893 +-0.06%
sha256	0.0010996 +-1.34%	0.0011159 +-1.74%	0.0010299 +-1.09%	0.0036085 +-0.40%	0.0426466 +-0.12%	0.0840805 +-0.03%
sha384	0.0011094 +-1.58%	0.0011204 +-1.48%	0.0009555 +-2.96%	0.0027775 +-0.58%	0.0296736 +-0.27%	0.058271 +-0.27%
sha512	0.0010909 +-1.71%	0.0010763 +-3.21%	0.0009746 +-1.77%	0.0027719 +-0.74%	0.0297744 +-0.26%	0.058239 +-0.25%

openssl 3.1.4-3.2

FILE SIZE	512 bytes	4096	65536	1M	16M	32M
md5	0.0039263 +-0.81%	0.0029834 +-1.69%	0.0030833 +-1.28%	0.0044167 +-0.87%	0.0286969 +-0.20%	0.0536009 +-0.16%
sha1	0.0039302 +-1.16%	0.0029809 +-1.62%	0.0030169 +-0.99%	0.0040051 +-1.04%	0.0220672 +-0.29%	0.0414711 +-0.19%
sha224	0.0039211 +-0.99%	0.0039417 +-0.95%	0.0031360 +-1.43%	0.0055392 +-0.69%	0.0408525 +-0.18%	0.078564 +-0.20%
sha256	0.0039277 +-0.60%	0.0039653 +-0.97%	0.0031659 +-1.26%	0.0055284 +-0.76%	0.0408774 +-0.11%	0.0788206 +-0.10%
sha384	0.0039370 +-1.13%	0.0039494 +-0.87%	0.0030840 +-1.26%	0.0047742 +-0.81%	0.029442 +-0.35%	0.056091 +-0.24%
sha512	0.0039456 +-1.02%	0.0039779 +-1.09%	0.0030739 +-1.26%	0.0047586 +-0.55%	0.0294435 +-0.20%	0.056350 +-0.31%

Benchmark System

    Linux Kernel: openSUSE Tumbleweed 6.11.0-1-default
    CPU: Intel(R) Xeon(R) CPU E3-1260L v5 @ 2.90GHz
    Thread(s) per core:   2
    Core(s) per socket:   4
    Socket(s):            1
    RAM: 64GB

Benchmark Script

for size in $((32 * 1024 * 1024)) $((16 * 1024 * 1024)) $((1024 * 1024)) \
            $((64 * 1024)) $((4 * 1024)) 512; do
    dd if=/dev/urandom of="${size}.data" bs="$size" count=1 || break
    echo "==== hashing file of size $size ===="
    for i in md5 sha1 sha224 sha256 sha384 sha512; do
        # prime cache
        cat "${size}.data" > /dev/null
        perf stat --null -r 5 --table \
             openssl "$i" "${size}.data" \
             >/dev/null 2>openssl.${size}.${i}.perf
        perf stat --null -r 5 --table \
             ~/liburing/examples/kdigest "$i" "${size}.data" \
             >/dev/null 2>kdigest.${size}.${i}.perf
    done
done