(I seem to have written this in the tone of an advertisment for household cleaning products. Sorry.)
Do you need to quickly hash (e.g. SHA-1) content on Linux? Want to avoid linking against bloated crypto libraries? There's no need to roll your own; use the Linux kernel's AF_ALG functionality! It's fast, supports many hash algorithms, and plumbs easily into existing network or disk I/O pipelines.
Look at these benchmark results, comparing io_uring kdigest and openssl performance:
kdigest
| FILE SIZE | 512 bytes | 4096 | 65536 | 1M | 16M | 32M |
|---|---|---|---|---|---|---|
| md5 | 0.0010984 +-1.44% |
0.0008265 +-2.65% |
0.0009556 +-1.52% |
0.0024098 +-0.35% |
0.0266533 +-0.06% |
0.0521402 +-0.19% |
| sha1 | 0.0011012 +-1.59% |
0.0009430 +-2.21% |
0.0009173 +-1.89% |
0.0019097 +-1.29% |
0.0186466 +-0.18% |
0.0361400 +-0.13% |
| sha224 | 0.0010983 +-1.29% |
0.0010970 +-1.72% |
0.0010350 +-1.32% |
0.0036209 +-0.42% |
0.0425834 +-0.06% |
0.0841893 +-0.06% |
| sha256 | 0.0010996 +-1.34% |
0.0011159 +-1.74% |
0.0010299 +-1.09% |
0.0036085 +-0.40% |
0.0426466 +-0.12% |
0.0840805 +-0.03% |
| sha384 | 0.0011094 +-1.58% |
0.0011204 +-1.48% |
0.0009555 +-2.96% |
0.0027775 +-0.58% |
0.0296736 +-0.27% |
0.058271 +-0.27% |
| sha512 | 0.0010909 +-1.71% |
0.0010763 +-3.21% |
0.0009746 +-1.77% |
0.0027719 +-0.74% |
0.0297744 +-0.26% |
0.058239 +-0.25% |
openssl 3.1.4-3.2
| FILE SIZE | 512 bytes | 4096 | 65536 | 1M | 16M | 32M |
|---|---|---|---|---|---|---|
| md5 | 0.0039263 +-0.81% |
0.0029834 +-1.69% |
0.0030833 +-1.28% |
0.0044167 +-0.87% |
0.0286969 +-0.20% |
0.0536009 +-0.16% |
| sha1 | 0.0039302 +-1.16% |
0.0029809 +-1.62% |
0.0030169 +-0.99% |
0.0040051 +-1.04% |
0.0220672 +-0.29% |
0.0414711 +-0.19% |
| sha224 | 0.0039211 +-0.99% |
0.0039417 +-0.95% |
0.0031360 +-1.43% |
0.0055392 +-0.69% |
0.0408525 +-0.18% |
0.078564 +-0.20% |
| sha256 | 0.0039277 +-0.60% |
0.0039653 +-0.97% |
0.0031659 +-1.26% |
0.0055284 +-0.76% |
0.0408774 +-0.11% |
0.0788206 +-0.10% |
| sha384 | 0.0039370 +-1.13% |
0.0039494 +-0.87% |
0.0030840 +-1.26% |
0.0047742 +-0.81% |
0.029442 +-0.35% |
0.056091 +-0.24% |
| sha512 | 0.0039456 +-1.02% |
0.0039779 +-1.09% |
0.0030739 +-1.26% |
0.0047586 +-0.55% |
0.0294435 +-0.20% |
0.056350 +-0.31% |
Benchmark System
Linux Kernel: openSUSE Tumbleweed 6.11.0-1-default
CPU: Intel(R) Xeon(R) CPU E3-1260L v5 @ 2.90GHz
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
RAM: 64GB
Benchmark Script
for size in $((32 * 1024 * 1024)) $((16 * 1024 * 1024)) $((1024 * 1024)) \
$((64 * 1024)) $((4 * 1024)) 512; do
dd if=/dev/urandom of="${size}.data" bs="$size" count=1 || break
echo "==== hashing file of size $size ===="
for i in md5 sha1 sha224 sha256 sha384 sha512; do
# prime cache
cat "${size}.data" > /dev/null
perf stat --null -r 5 --table \
openssl "$i" "${size}.data" \
>/dev/null 2>openssl.${size}.${i}.perf
perf stat --null -r 5 --table \
~/liburing/examples/kdigest "$i" "${size}.data" \
>/dev/null 2>kdigest.${size}.${i}.perf
done
done
