Simple Hashing with AF_ALG

My previous post demonstrated how the Linux kernel's AF_ALG socket type can be used with io_uring for fast hashing.

I wasn't very happy with the complexity of the io_uring based implementation, so set out to write something much simpler with plain syscalls. What came out was just over 100 lines of boring, uncomplicated C. Better still, use of splice() for copy-offload sees it perform very similar to the io_uring based implementation on my systems.

I've published the BSD-3-Clause licensed source at https://github.com/ddiss/splice-digest, with the main snippets below:

#define SPLICE_MAX (1024 * 1024)
...
int main(int argc, char *argv[])
{
...
        struct sockaddr_alg sa = {
                .salg_family = AF_ALG,
                .salg_type = "hash",
        };
...
        infd = open(infile, O_RDONLY);
...
        sfd = socket(AF_ALG, SOCK_SEQPACKET, 0);
...
        memcpy(sa.salg_name, alg, alg_len + 1);
        if (bind(sfd, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
...
        }

        outfd = accept(sfd, NULL, 0);
...
        if (pipe(pipefds) < 0)
                err(-1, "pipe");
...
        for (insize = st.st_size; insize; insize -= got) {
                size_t tryl = (insize < SPLICE_MAX ? insize : SPLICE_MAX);

                l = splice(infd, NULL, pipefds[1], NULL, tryl, 0);
...
                got = splice(pipefds[0], NULL, outfd, NULL, l, SPLICE_F_MORE);
...
        }

        fprintf(stdout, "Spliced %s(%s): ", alg, infile);
        print_hash_result(outfd);
        putc('\n', stdout);
}

Hopefully this convinces a few people to drop those external crypto libraries in favour of Linux AF_ALG.

Update 2026-05-01: The recent copy.fail (CVE-2026-31431) bug is alarming, but this article hopefully serves as a reminder that there are legitimate use cases for AF_ALG.

Speedreader's Digest: Hashing Data on Linux using AF_ALG

(I seem to have written this in the tone of an advertisment for household cleaning products. Sorry. This follow-up post documents a simpler way to use AF_ALG)

Do you need to quickly hash (e.g. SHA-1) content on Linux? Want to avoid linking against bloated crypto libraries? There's no need to roll your own; use the Linux kernel's AF_ALG functionality! It's fast, supports many hash algorithms, and plumbs easily into existing network or disk I/O pipelines.

Look at these benchmark results, comparing io_uring kdigest and openssl performance:

kdigest

FILE SIZE	512 bytes	4096	65536	1M	16M	32M
md5	0.0010984 +-1.44%	0.0008265 +-2.65%	0.0009556 +-1.52%	0.0024098 +-0.35%	0.0266533 +-0.06%	0.0521402 +-0.19%
sha1	0.0011012 +-1.59%	0.0009430 +-2.21%	0.0009173 +-1.89%	0.0019097 +-1.29%	0.0186466 +-0.18%	0.0361400 +-0.13%
sha224	0.0010983 +-1.29%	0.0010970 +-1.72%	0.0010350 +-1.32%	0.0036209 +-0.42%	0.0425834 +-0.06%	0.0841893 +-0.06%
sha256	0.0010996 +-1.34%	0.0011159 +-1.74%	0.0010299 +-1.09%	0.0036085 +-0.40%	0.0426466 +-0.12%	0.0840805 +-0.03%
sha384	0.0011094 +-1.58%	0.0011204 +-1.48%	0.0009555 +-2.96%	0.0027775 +-0.58%	0.0296736 +-0.27%	0.058271 +-0.27%
sha512	0.0010909 +-1.71%	0.0010763 +-3.21%	0.0009746 +-1.77%	0.0027719 +-0.74%	0.0297744 +-0.26%	0.058239 +-0.25%

openssl 3.1.4-3.2

FILE SIZE	512 bytes	4096	65536	1M	16M	32M
md5	0.0039263 +-0.81%	0.0029834 +-1.69%	0.0030833 +-1.28%	0.0044167 +-0.87%	0.0286969 +-0.20%	0.0536009 +-0.16%
sha1	0.0039302 +-1.16%	0.0029809 +-1.62%	0.0030169 +-0.99%	0.0040051 +-1.04%	0.0220672 +-0.29%	0.0414711 +-0.19%
sha224	0.0039211 +-0.99%	0.0039417 +-0.95%	0.0031360 +-1.43%	0.0055392 +-0.69%	0.0408525 +-0.18%	0.078564 +-0.20%
sha256	0.0039277 +-0.60%	0.0039653 +-0.97%	0.0031659 +-1.26%	0.0055284 +-0.76%	0.0408774 +-0.11%	0.0788206 +-0.10%
sha384	0.0039370 +-1.13%	0.0039494 +-0.87%	0.0030840 +-1.26%	0.0047742 +-0.81%	0.029442 +-0.35%	0.056091 +-0.24%
sha512	0.0039456 +-1.02%	0.0039779 +-1.09%	0.0030739 +-1.26%	0.0047586 +-0.55%	0.0294435 +-0.20%	0.056350 +-0.31%

Benchmark System

    Linux Kernel: openSUSE Tumbleweed 6.11.0-1-default
    CPU: Intel(R) Xeon(R) CPU E3-1260L v5 @ 2.90GHz
    Thread(s) per core:   2
    Core(s) per socket:   4
    Socket(s):            1
    RAM: 64GB

Benchmark Script

for size in $((32 * 1024 * 1024)) $((16 * 1024 * 1024)) $((1024 * 1024)) \
            $((64 * 1024)) $((4 * 1024)) 512; do
    dd if=/dev/urandom of="${size}.data" bs="$size" count=1 || break
    echo "==== hashing file of size $size ===="
    for i in md5 sha1 sha224 sha256 sha384 sha512; do
        # prime cache
        cat "${size}.data" > /dev/null
        perf stat --null -r 5 --table \
             openssl "$i" "${size}.data" \
             >/dev/null 2>openssl.${size}.${i}.perf
        perf stat --null -r 5 --table \
             ~/liburing/examples/kdigest "$i" "${size}.data" \
             >/dev/null 2>kdigest.${size}.${i}.perf
    done
done

Multipath Failover Simulation with QEMU

While working on a Ceph OSD multipath issue, I came across a helpful post from Dan Horák on how to simulate a multipath device under QEMU.

qemu-kvm ... -device virtio-scsi-pci,id=scsi \
  -drive if=none,id=hda,file=<path>,cache=none,format=raw,serial=MPIO \
  -device scsi-hd,drive=hda \
  -drive if=none,id=hdb,file=<path>,cache=none,format=raw,serial=MPIO \
  -device scsi-hd,drive=hdb"

• <path> should be replaced with a file or device path (the same for each)
• serial= specifies the SCSI logical unit serial number

This attaches two virtual SCSI devices to the VM, both of which are backed by the same file and share the same SCSI logical unit identifier.
Once booted, the SCSI devices for each corresponding path appear as sda and sdb, which are then detected as multipath enabled and subsequently mapped as dm-0:

         Starting Device-Mapper Multipath Device Controller...
[  OK  ] Started Device-Mapper Multipath Device Controller.
...
[    1.329668] device-mapper: multipath service-time: version 0.3.0 loaded
...
rapido1:/# multipath -ll
0QEMU_QEMU_HARDDISK_MPIO dm-0 QEMU,QEMU HARDDISK
size=2.0G features='1 retain_attached_hw_handler' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 0:0:0:0 sda 8:0  active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 0:0:1:0 sdb 8:16 active ready running

QEMU additionally allows for virtual device hot(un)plug at runtime, which can be done from the QEMU monitor CLI (accessed via ctrl-a c) using the drive_del command. This can be used to trigger a multipath failover event:

rapido1:/# mkfs.xfs /dev/dm-0
meta-data=/dev/dm-0              isize=256    agcount=4, agsize=131072 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0, sparse=0
data     =                       bsize=4096   blocks=524288, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
rapido1:/# mount /dev/dm-0 /mnt/
[   96.846919] XFS (dm-0): Mounting V4 Filesystem
[   96.851383] XFS (dm-0): Ending clean mount

rapido1:/# QEMU 2.6.2 monitor - type 'help' for more information
(qemu) drive_del hda
(qemu) 

rapido1:/# echo io-to-trigger-path-failure > /mnt/failover-trigger
[  190.926579] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[  190.926588] sd 0:0:0:0: [sda] tag#0 Sense Key : 0x2 [current] 
[  190.926589] sd 0:0:0:0: [sda] tag#0 ASC=0x3a ASCQ=0x0 
[  190.926590] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 00 02 00 00 01 00
[  190.926591] blk_update_request: I/O error, dev sda, sector 2
[  190.926597] device-mapper: multipath: Failing path 8:0.

rapido1:/# multipath -ll
0QEMU_QEMU_HARDDISK_MPIO dm-0 QEMU,QEMU HARDDISK
size=2.0G features='1 retain_attached_hw_handler' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=0 status=enabled
| `- 0:0:0:0 sda 8:0  failed faulty running
`-+- policy='service-time 0' prio=1 status=active
  `- 0:0:1:0 sdb 8:16 active ready  running

The above procedure demonstrates cable-pull simulation while the broken path is used by the mounted dm-0 device. The subsequent I/O failure triggers multipath failover to the remaining good path.

I've added this functionality to Rapido (pull-request) so that multipath failover can be performed in a couple of minutes directly from kernel source. I encourage you to give it a try for yourself!

Rapido: Quick Kernel Testing From Source (Video)

I presented a short talk at the 2017 openSUSE Conference on Linux kernel testing using Rapido.

There were many other interesting talks during the conference, all of which can be viewed on the oSC 2017 media site.
A video of my presentation is embedded below.

Many thanks to the organisers and sponsors for putting on a great event.

Rapido: A Glorified Wrapper for Dracut and QEMU

Introduction

I've blogged a few of times about how Dracut and QEMU can be combined to greatly improve Linux kernel dev/test turnaround.

• My first post covered the basics of building the kernel, running dracut, and booting the resultant image with qemu-kvm.
• A later post took a closer look at network configuration, and focused on bridging VMs with the hypervisor.
• Finally, my third post looked at how this technique could be combined with Ceph, to provide a similarly efficient workflow for Ceph development.

In bringing this series to a conclusion, I'd like to introduce the newly released Rapido project. Rapido combines all of the procedures and techniques described in the articles above into a handful of scripts, which can be used to test specific Linux kernel functionality, standalone or alongside other technologies such as Ceph.

Usage - Standalone Linux VM

The following procedure was tested on openSUSE Leap 42.3 and SLES 12SP3, but should work fine on many other Linux distributions.

Step 1: Checkout and Build

Checkout the Linux kernel and Rapido source repositories:

~/> cd ~
~/> git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
~/> git clone https://github.com/rapido-linux/rapido.git

Build the kernel (using a config provided with the Rapido source):

~/> cp rapido/kernel/vanilla_config linux/.config
~/> cd linux
~/linux/> make -j6
~/linux/> make modules
~/linux/> INSTALL_MOD_PATH=./mods make modules_install

Step 2: Configuration 

Install Rapido dependencies: dracut and qemu.

Create a master rapido.conf configuration file using the example template:

~/linux/> cd ~/rapido
~/rapido/> cp rapido.conf.example rapido.conf
~/rapido/> vi rapido.conf

• set KERNEL_SRC="/home/<user>/linux"
• set KERNEL_INSTALL_MOD_PATH="${KERNEL_SRC}/mods"
• the remaining options can be left as is for now
  

Step 3: Image Generation 

Generate a minimal Linux VM image which includes binaries, libraries and kernel modules for filesystem testing:

~/rapido/> ./cut_fstests_local.sh
...
 dracut: *** Creating initramfs image file 'initrds/myinitrd' done ***
~/rapido/> ls -lah initrds/myinitrd
-rw-r--r-- 1 ddiss users 30M Dec 13 18:17 initrds/myinitrd

Step 4 - Boot!

 ~/rapido/> ./vm.sh
+ mount -t btrfs /dev/zram1 /mnt/scratch
[    3.542927] BTRFS info (device zram1): disk space caching is enabled
...
btrfs filesystem mounted at /mnt/test and /mnt/scratch
rapido1:/#  

In a whopping four seconds, or thereabouts, the VM should have booted to a rapido:/# bash prompt. Leaving you with two zram backed Btrfs filesystems mounted at /mnt/test and /mnt/scratch.

Everything, including the VM's root filesystem, is in memory, so any changes will not persist across reboot. Use the rapido.conf QEMU_EXTRA_ARGS parameter if you wish to add persistent storage to a VM.

Once you're done playing around, you can shutdown:

rapido1:/# shutdown
[  267.304313] sysrq: SysRq : sysrq: Power Off
rapido1:/# [  268.168447] ACPI: Preparing to enter system sleep state S5
[  268.169493] reboot: Power down
+ exit 0

Step 5: Network Configuration

The fstests_local VM above is networkless, so doesn't require bridge network configuration. For VMs that do (e.g. CephFS client below) edit rapido.conf:

• set TAP_USER="<user>"
• set MAC_ADDR1 to a valid MAC address, e.g. "b8:ac:24:45:c5:01"
• set MAC_ADDR2 to a valid MAC address, e.g. "b8:ac:24:45:c5:02"

Configure the isolated bridge and tap network devices. This must be done as root:

~/rapido/> sudo tools/br_setup.sh
~/rapido/> ip addr show br0
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    ...
    inet 192.168.155.1/24 scope global br0

Usage - Ceph vstart.sh cluster and CephFS client VM

This usage guide builds on the previous standalone Linux VM procedure, but this time adds Ceph to the mix. If you're not interested in Ceph (how could you not be!) then feel free to skip to the next section.

Step I - Checkout and Build

We already have a clone of the Rapido and Linux kernel repositories. All that's needed for CephFS testing is a Ceph build:

~/> git clone https://github.com/ceph/ceph
~/> cd ceph
<install Ceph build dependencies>
~/ceph/> ./do_cmake.sh -DWITH_MANPAGE=0 -DWITH_OPENLDAP=0 -DWITH_FUSE=0 -DWITH_NSS=0 -DWITH_LTTNG=0
~/ceph/> cd build
~/ceph/build/> make -j4 

Step II - Start a vstart.sh Ceph "cluster"

Once Ceph has finished compiling, vstart.sh can be run with the following parameters to configure and locally start three OSDs, one monitor process, and one MDS.

~/ceph/build/> OSD=3 MON=1 RGW=0 MDS=1 ../src/vstart.sh -i 192.168.155.1 -n
...
~/ceph/build/> bin/ceph -c status
...
     health HEALTH_OK
     monmap e2: 1 mons at {a=192.168.155.1:40160/0}
            election epoch 4, quorum 0 a
      fsmap e5: 1/1/1 up {0=a=up:active}
        mgr no daemons active 
     osdmap e10: 3 osds: 3 up, 3 in

Step III - Rapido configuration

Edit rapido.conf, the master Rapido configuration file:

~/ceph/build/> cd ~/rapido
~/rapido/> vi rapido.conf

• set CEPH_SRC="/home/<user>/ceph/src"
• KERNEL_SRC and network parameters were configured earlier

Step IV - Image Generation

The cut_cephfs.sh script generates a VM image with the Ceph configuration and keyring from the vstart.sh cluster, as well as the CephFS kernel module.

~/rapido/> ./cut_cephfs.sh

...
 dracut: *** Creating initramfs image file 'initrds/myinitrd' done ***

Step V - Boot!

Booting the newly generated image should bring you to a shell prompt, with the vstart.sh provisioned CephFS filesystem mounted under /mnt/cephfs:

~/rapido/> ./vm.sh

...

+ mount -t ceph 192.168.155.1:40160:/ /mnt/cephfs -o name=admin,secret=...
[    3.492742] libceph: mon0 192.168.155.1:40160 session established
...
rapido1:/# df -h /mnt/cephfs
Filesystem             Size  Used Avail Use% Mounted on
192.168.155.1:40160:/  1.3T  611G  699G  47% /mnt/cephfs

CephFS is a clustered filesystem, in which case testing from multiple clients is also of interest. From another window, boot a second VM:

~/rapido/> ./vm.sh

 

 

Further Use Cases

Rapido ships with a bunch of scripts for testing different kernel components:

• cut_cephfs.sh (shown above)
• Image: includes Ceph config, credentials and CephFS kernel module
• Boot: mounts CephFS filesystem
• cut_cifs.sh
• Image: includes CIFS (SMB client) kernel module
• Boot: mounts share using details and credentials specified in rapido.conf
• cut_dropbear.sh
• Image: includes dropbear SSH server
• Boot: starts an SSH server with SSH_AUTHORIZED_KEY
• cut_fstests_cephfs.sh
• Image: includes xfstests and CephFS kernel client
• Boot: mounts CephFS filesystem and runs FSTESTS_AUTORUN_CMD
• cut_fstests_local.sh (shown above)
• Image: includes xfstests and local Btrfs and XFS dependencies
• Boot: provisions local xfstest zram devices. Runs FSTESTS_AUTORUN_CMD
• cut_lio_local.sh
• Image: includes LIO, loopback dev and dm-delay kernel modules
• Boot: provisions an iSCSI target, with three LUs exposed
• cut_lio_rbd.sh
• Image: includes LIO and Ceph RBD kernel modules
• Boot: provisions an iSCSI target backed by CEPH_RBD_IMAGE, using target_core_rbd
• cut_qemu_rbd.sh
• Image: CEPH_RBD_IMAGE is attached to the VM using qemu-block-rbd
• Boot: runs shell only
• cut_rbd.sh
• Image: includes Ceph config, credentials and Ceph RBD kernel module
• Boot: maps CEPH_RBD_IMAGE using the RBD kernel clien
• cut_samba_cephfs.sh
• Image: includes Ceph vstart config, credentials and libcephfs from CEPH_SRC, and additionally pulls in Samba from a (pre compiled) SAMBA_SRC
• Boot: configures smb.conf with a CephFS backed share and starts Samba
• cut_samba_local.sh
• Image: includes local kernel filesystem utils, and pulls in Samba from SAMBA_SRC
• Boot: configures smb.conf with a zram backed share and starts Samba
• cut_tcmu_rbd_loop.sh
• Image: includes Ceph config, librados, librbd, and pulls in tcmu-runner from TCMU_RUNNER_SRC
• Boot: starts tcmu-runner and configures a tcmu+rbd backstore exposing CEPH_RBD_IMAGE via the LIO loopback fabric
• cut_usb_rbd.sh (see https://github.com/ddiss/rbd-usb)
• Image: usb_f_mass_storage, zram, dm-crypt, and RBD_USB_SRC
• Boot: starts the conf-fs.sh script from RBD_USB_SRC

 

 

Conclusion

• Dracut and QEMU can be combined for super-fast Linux kernel testing and development.
• Rapido is mostly just a glorified wrapper around these utilities, but does provide some useful tools for automated testing of specific Linux kernel functionality.

If you run into any problems, or wish to provide any kind of feedback (always appreciated), please feel free to leave a message below, or raise a ticket in the Rapido issue tracker.

Update 20170106:

• Add cut_tcmu_rbd_loop.sh details and fix the example CEPH_SRC path. 

 Update 20180312:

• Use KERNEL_INSTALL_MOD_PATH instead of an ugly symlink
• Update Github links to refer to new project URL
• Remove old brctl and tunctl dependencies
• Split network setup into a separate section, as fstests_local VMs are now networkless
• Add cut_samba_cephfs.sh and cut_samba_local.sh details

Linux USB Gadget Application Testing

Developing a USB gadget application that runs on Linux?
Following a recent Ceph USB gateway project, I was looking at ways to test a Linux USB device without the need to fiddle with cables, or deal with slow embedded board boot times.

Ideally USB gadget testing could be performed by running the USB device code within a virtual machine, and attaching the VM's virtual USB device port to an emulated USB host controller on the hypervisor system.

I was unfortunately unable to find support for virtual USB device ports in QEMU, so I abandoned the above architecture, and discovered dummy_hcd.ko instead.


The dummy_hcd Linux kernel module is an excellent utility for USB device testing from within a standalone system or VM.

dummy_hcd.ko offers the following features:

• Re-route USB device traffic back to the local system
• Effectively providing device loopback functionality
• USB high-speed and super-speed connection simulation

It can be enabled via the USB_DUMMY_HCD kernel config parameter. Once the module is loaded, no further configuration is required.

Ceph USB Storage Gateway

Last week was Hackweek, a week full of fun and innovation at SUSE. I decided to use the time to work on a USB storage gateway for Ceph.

The concept is simple - create a USB device that, when configured and connected, exposes remote Ceph RADOS Block Device (RBD) images for access as USB mass storage, allowing for:

• Ceph storage usage by almost any system with a USB port
  • Including dumb systems such as TVs, MP3 players and mobile phones
• Boot from RBD images
  • Many systems are capable of booting from a USB mass storage device
• Minimal configuration
  • Network, Ceph credentials and image details should be all that's needed for configuration

Hardware

I already own a Cubietruck, which has the following desirable characteristics for this project:

• Works with a mainline Linux Kernel
• Is reasonably small and portable
• Supports power and data transfer via a single mini-USB port
• Performs relatively well
• Dual-core 1GHz processor and 2GB RAM
• Gigabit network adapter and WiFi 802.11 b/g/n

Possible alternatives worth evaluation include C.H.I.P (smaller and cheaper), NanoPi2, and UP (faster). I should take this opportunity to mention that I do gladly accept hardware donations!

Base System

I decided on using openSUSE Tumbleweed as the base operating system for this project. An openSUSE Tubleweed ARM port for the Cubietruck is available for download at:
http://download.opensuse.org/ports/armv7hl/factory/images/openSUSE-Tumbleweed-ARM-JeOS-cubietruck.armv7l-Current.raw.xz

Installation is as straightforward as copying the image to an SD card and booting - I documented the installation procedure on the openSUSE Wiki.
Releases prior to Build350 exhibit boot issues due to the U-Boot device-tree path. However, this has been fixed in recent builds.

Kernel

The Linux kernel currently shipped with the openSUSE image does not include support for acting as a USB mass storage gadget, nor does it include Ceph RBD support. In order to obtain these features, and also reduce the size of the base image, I built a mainline Linux kernel (4.4-rc4) using a minimal custom kernel configuration:

~/> sudo zypper install --no-recommends git-core gcc make ncurses-devel bc
~/> git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
~/> cd linux
~/linux/> wget https://raw.githubusercontent.com/ddiss/ceph_usb_gateway/master/.config
          # or `make sunxi_defconfig menuconfig`
          #     ->enable Ceph, sunxi and USB gadget modules
~/linux/> make oldconfig
~/linux/> make -j2 zImage dtbs modules
~/linux/> sudo make install modules_install
~/linux/> sudo cp arch/arm/boot/zImage /boot/zImage-$(make kernelrelease)
~/linux/> sudo cp arch/arm/boot/dts/sun7i-a20-cubietruck.dtb /boot/dtb-4.3.0-2/
~/linux/> sudo cp arch/arm/boot/dts/sun7i-a20-cubietruck.dtb /boot/dtb/

This build procedure takes a long time to complete. Cross compilation could be used to improve build times.
I plan on publishing a USB gadget enabled ARM kernel on the Open Build Service in the future, which would allow for simple installation via zypper - watch this space!

Ceph RADOS Block Device (RBD) mapping

To again save space, I avoided installation of user-space Ceph packages by using the bare kernel sysfs interface for RBD image mapping.
The Ceph RBD kernel module must be loaded prior to use:

# modprobe rbd

Ceph RADOS block devices can be mapped using the following command:

# echo -n "${MON_IP}:6789 name=${AUTH_NAME},secret=${AUTH_SECRET} " \

          "${CEPH_POOL} ${CEPH_IMG} -" > /sys/bus/rbd/add

$MON_IP can be obtained from ceph.conf. Similarly, the $AUTH_NAME and $AUTH_SECRET credentials can be retrieved from a regular Ceph keyring file.
$CEPH_POOL and $CEPH_IMG correspond to the location of the RBD image.

A locally mapped RBD block device can be subsequently removed via:

# echo -n "${DEV_ID}" > /sys/bus/rbd/remove

$DEV_ID can be determined from the numeric suffix assigned to the /dev/rbdX device path.

Images can't be provisioned without the Ceph user-space utilities installed, so should be performed on a separate system (e.g. an OSD) prior to mapping on the Cubietruck. E.g. To provision a 10GB image:

# rbd create --size=10240 --pool ${CEPH_POOL} ${CEPH_IMG}

With my Cubietruck connected to the network via the ethernet adapter, I observed streaming read (/dev/rbd -> /dev/null) throughput at ~37MB/s, and the same value for streaming writes (/dev/zero -> /dev/rbd). Performance appears to be constrained by limitations of the Cubietruck hardware.

USB Mass Storage Gadget

The Linux kernel mass storage gadget module is configured via configfs. A device can be exposed as a USB mass storage device with the following procedure:

# modprobe sunxi configfs libcomposite usb_f_mass_storage

# mount -t configfs configfs /sys/kernel/config
# cd /sys/kernel/config/usb_gadget/
# mkdir -p ceph
# cd ceph

# mkdir -p strings/0x409
# echo "fedcba9876543210" > strings/0x409/serialnumber
# echo "openSUSE" > strings/0x409/manufacturer
# echo "Ceph USB Drive" > strings/0x409/product

# mkdir -p functions/mass_storage.usb0
# echo 1 > functions/mass_storage.usb0/stall
# echo 0 > functions/mass_storage.usb0/lun.0/cdrom
# echo 0 > functions/mass_storage.usb0/lun.0/ro
# echo 0 > functions/mass_storage.usb0/lun.0/nofua
# echo "$DEV" > functions/mass_storage.usb0/lun.0/file

# mkdir -p configs/c.1/strings/0x409
# echo "Config 1: mass-storage" > configs/c.1/strings/0x409/configuration
# echo 250 > configs/c.1/MaxPower
# ln -s functions/mass_storage.usb0 configs/c.1/

# ls /sys/class/udc > UDC

$DEV corresponds to a /dev/X device path, which should be a locally mapped RBD device path. The module can however also use local files as backing for USB mass storage.

Boot-Time Automation

By default, Cubietruck boots when the board is connected to a USB host via the mini-USB connection.
With RBD image mapping and USB mass storage exposure now working, the process can be run automatically on boot via a simple script: rbd_usb_gw.sh
Furthermore, a systemd service can be added:

[Unit]
Wants=network-online.target
After=network-online.target

[Service]
# XXX assume that rbd_usb_gw.sh is present in /bin
ExecStart=/bin/rbd_usb_gw.sh %i
Type=oneshot
RemainAfterExit=yes

Finally, this service can be triggered by Wicked when the network interface comes online, with the following entry added to /etc/sysconfig/network/config:

POST_UP_SCRIPT="systemd:rbd-mapper@.service"

Boot Performance Optimisation

A significant reduction in boot time can be achieved by running everything from initramfs, rather than booting to the full Linux distribution.
Generating a minimal initramfs image, with support for mapping and exposing RBD images is straightforward, thanks to the Dracut utility:

# dracut --no-compress  \
         --kver "`uname -r" \
         --install "ps rmdir dd vim grep find df modinfo" \
         --add-drivers "rbd musb_hdrc sunxi configfs" \
         --no-hostonly --no-hostonly-cmdline \
         --modules "bash base network ifcfg" \
         --include /bin/rbd_usb_gw.sh /lib/dracut/hooks/emergency/02_rbd_usb_gw.sh \
         myinitrd

The rbd_usb_gw.sh script is installed into the initramfs image as a Dracut emergency hook, which sees it executed as soon as initramfs has booted.

To ensure that the network is up prior to the launch of rbd_usb_gw.sh, the kernel DHCP client (CONFIG_IP_PNP_DHCP) can be used by appending ip=dhcp to the boot-time kernel parameters. This can be set from the U-Boot bootloader prompt:

=> setenv append 'ip=dhcp'

=> boot

The new initramfs image must be committed to the boot partition via:

# cp myinitrd /boot/

# rm /boot/initrd

# sudo ln -s /boot/myinitrd /boot/initrd

Note: In order to boot back to the full Linux distribution, you will have to mount the /boot partition and revert the /boot/initrd symlink to its previous target.

Future Improvements

• Support configuration of the device without requiring console access
• Run an embedded web-server, or expose a configuration filesystem via USB 
• Install the operating system onto on-board NAND storage,
• Further improve boot times
• Avoid U-Boot device probes
• Experiment with the new f_tcm USB gadget module
• Expose RBD images via USB and iSCSI

Credits

Many thanks to:

• My employer, SUSE Linux, for encouraging me to work on projects like this during Hackweek.
• The linux-sunxi community, for their excellent contributions to the mainline Linux kernel.
• Colleagues Dirk, Bernhard, Alex and Andreas for their help in bringing up openSUSE Tumbleweed on my Cubietruck board.

Edits

• 2025-04-02: fix architecture image background

QEMU/KVM Bridged Network with TAP interfaces

In my previous post, Rapid Linux Kernel Dev/Test with QEMU, KVM and Dracut, I described how build and boot a Linux kernel quickly, making use of port forwarding between hypervisor and guest VM for virtual network traffic.

This post describes how to plumb the Linux VM directly into a hypervisor network, through the use of a bridge.

Start by creating a bridge on the hypervisor system:

> sudo ip link add br0 type bridge

Clear the IP address on the network interface that you'll be bridging (e.g. eth0).
Note: This will disable network traffic on eth0!

> sudo ip addr flush dev eth0

Add the interface to the bridge:

> sudo ip link set eth0 master br0

Next up, create a TAP interface:

> sudo ip tuntap add dev tap0 mode tap user $(whoami)

The user parameter ensures that the current user will be able to connect to the TAP interface.

Add the TAP interface to the bridge:

> sudo ip link set tap0 master br0

Make sure everything is up:

> sudo ip link set dev br0 up
> sudo ip link set dev tap0 up

The TAP interface is now ready for use. Assuming that a DHCP server is available on the bridged network, the VM can now obtain an IP address during boot via:

> qemu-kvm -kernel arch/x86/boot/bzImage \
           -initrd initramfs \
           -device e1000,netdev=network0,mac=52:55:00:d1:55:01 \
           -netdev tap,id=network0,ifname=tap0,script=no,downscript=no \
           -append "ip=dhcp rd.shell=1 console=ttyS0" -nographic

The MAC address is explicitly specified, so care should be taken to ensure its uniqueness.

The DHCP server response details are printed alongside network interface configuration. E.g.

[    3.792570] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[    3.796085] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    3.812083] Sending DHCP requests ., OK
[    4.824174] IP-Config: Got DHCP answer from 10.155.0.42, my address is 10.155.0.1
[    4.825119] IP-Config: Complete:
[    4.825476]      device=eth0, hwaddr=52:55:00:d1:55:01, ipaddr=10.155.0.1, mask=255.255.0.0, gw=10.155.0.254
[    4.826546]      host=rocksolid-sles, domain=suse.de, nis-domain=suse.de
...

Didn't get an IP address? There are a few things to check:

• Confirm that the kernel is built with boot-time DHCP client (CONFIG_IP_PNP_DHCP=y) and E1000 network driver (CONFIG_E1000=y) support.
• Check the -device and -netdev arguments specify a valid e1000 TAP interface.
• Ensure that ip=dhcp is provided as a kernel boot parameter, and that the DHCP server is up and running.

Happy hacking

 Update 20161223:

• Use 'ip' instead of 'brctl' to manipulate the bridge device - thanks Yagamy Light!

Update 20171220:

• Use 'ip tuntap' instead of 'tunctl' to create the TAP interface - thanks Johannes!

Update 20200930:

• For performance reasons, I strongly recommend using virtio network adapters instead of e1000.

Rapid Linux Kernel Dev/Test with QEMU, KVM and Dracut

Update 2018-01-02: See my post on Rapido for a more automated approach to the procedure outlined below.

Inspired by Stefan Hajnoczi's excellent blog post, I recently set about constructing an environment for rapid testing of Linux kernel changes, particularly focused on the LIO iSCSI target. Such an environment would help me in number of ways:

• Faster dev / test turnaround.
• A modified kernel can be compiled and booted in a matter of seconds.
• Improved resource utilisation.
• No need to boot external test hosts or heavyweight VMs.
•  Simplified and speedier debugging.
• GDB and Wireshark can be plumbed directly into QEMU/KVM.

My requirements were slightly different to Stefan's, in that:

• I'd prefer to be lazy and use Dracut for initramfs generation.
• I need a working network connection between VM and hypervisor system
• The VM will act as the iSCSI target, the hypervisor as the initiator.

Starting with the Linux kernel, the first step is to build a bzimage:

~/> git clone \
        git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

hack, hack, hack.

~/linux/> make menuconfig

Set CONFIG_IP_PNP_DHCP=y and CONFIG_E1000=y to enable IP address assignment on boot.

~/linux/> make -j6
~/linux/> make modules
~/linux/> INSTALL_MOD_PATH=./mods make modules_install
~/linux/> sudo ln -s $PWD/mods/lib/modules/$(make kernelrelease) \
                     /lib/modules/$(make kernelrelease)

This leaves us with a compressed kernel image file at arch/x86/boot/bzimage, and corresponding modules installed under mods/lib/module/$(make kernelrelease), where $(make kernelrelease) evaluates to 4.1.0-rc7+ in this example. The /lib/modules/4.1.0-rc7+ symlink allows Dracut to locate the modules.

The next step is to generate an initial RAM filesystem, or initramfs, which includes a minimal set of user-space utilities, and kernel modules needed for testing:

~/linux/> dracut --kver "$(make kernelrelease)" \
                 --add-drivers "iscsi_target_mod target_core_mod" \
                 --add-drivers "target_core_file target_core_iblock" \
                 --add-drivers "configfs" \
                 --install "ps grep netstat" \
                 --no-hostonly --no-hostonly-cmdline \
                 --modules "bash base shutdown network ifcfg" initramfs
...
*** Creating image file done ***

We now have an initramfs file in the current directory, with the following contents:

• LIO kernel modules obtained from /lib/module/4.1.0-rc7, as directed via the --kver and --add-drivers parameters.
• User-space shell, boot and network helpers, as directed via the --modules parameter.

We're now ready to use QEMU/KVM to boot our test kernel and initramfs:

~/linux/> qemu-kvm -kernel arch/x86/boot/bzImage \
                   -initrd initramfs \
                   -device e1000,netdev=network0 \
                   -netdev user,id=network0 \
                   -redir tcp:51550::3260 \
                   -append "ip=dhcp rd.shell=1 console=ttyS0" \
                   -nographic

This boots the test environment, with the kernel and initramfs previously generated:

[    3.216596] dracut Warning: dracut: FATAL: No or empty root= argument
[    3.217998] dracut Warning: dracut: Refusing to continue
...
Dropping to debug shell.

dracut:/#

From the dracut shell, confirm that the QEMU DHCP server assigned the VM an IP address:

dracut:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
...

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
...
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0

Port 3260 (iSCSI) on this interface is forwarded to/from port 51550 on the hypervisor, as configured via the qemu-kvm -redir parameter.

Now onto LIO iSCSI target setup. First off load the appropriate kernel modules:

dracut:/# modprobe iscsi_target_mod
dracut:/# cat /proc/modules
iscsi_target_mod 246669 0 - Live 0xffffffffa006a000
target_core_mod 289004 1 iscsi_target_mod, Live 0xffffffffa000b000
configfs 22407 3 iscsi_target_mod,target_core_mod, Live 0xffffffffa0000000

LIO configuration requires a mounted configfs filesystem:

dracut:/# mount -t configfs configfs /sys/kernel/config/
dracut:/# cat /sys/kernel/config/target/version 
Target Engine Core ConfigFS Infrastructure v4.1.0 on Linux/x86_64 on 4.1.0-rc1+

An iSCSI target can be provisioned by manipulating corresponding configfs entries. I used the lio_dump output on an existing setup as reference:

dracut:/# mkdir /sys/kernel/config/target/iscsi

dracut:/# echo -n 0 > /sys/kernel/config/target/iscsi/discovery_auth/enforce_discovery_auth
dracut:/# mkdir -p /sys/kernel/config/target/iscsi/<iscsi_iqn>/tpgt_1/np/10.0.2.15:3260
...

Finally, we're ready to connect to the LIO target using the local hypervisor port that forwards to the VM's virtual network adapter:

~/linux/> iscsiadm --mode discovery \
                   --type sendtargets \
                   --portal 127.0.0.1:51550
10.0.2.15:3260,1 iqn.2015-04.suse.arch:5eca2313-028d-435c-9131-53a5ab256a83

It works!

There are a few things that can be adjusted:

• Port forwarding to the VM network is a bit fiddly - I'm now using a bridge/TAP configuration instead.
• When dropping into the emergency boot shell, Dracut executes scripts carried under /lib/dracut/hooks/emergency/. This means that a custom script can be triggered on boot via:

  ~/linux/> dracut -i runme.sh /lib/dracut/hooks/emergency/02-runme.sh ...
  

• It should be possible to have Dracut pull the kernel modules in from the temporary directory, but I wasn't able to get this working:

  ~/linux/> INSTALL_MOD_PATH=./mods make modules_install
  ~/linux/> dracut --kver "$(make kernelrelease)" --kmoddir ./mods/lib/...
  

• Boot time and initramfs file IO performance can be improved by disabling compression. This is done by specifying the --no-compress Dracut parameter.

Update 20150722:

• Don't install kernel modules as root, set up a /lib/modules symlink for Dracut instead.
• Link to bridge/TAP networking post.
• Describe boot script usage.

Update 20150813:

• Use $(make kernelrelease) rather than a hard-coded 4.1.0-rc7+ kernel version string - thanks Aurélien!

Update 20150908: Describe initramfs --no-compress optimisation.
Update 20180102: Link to the Rapido project.

Azure File Service IO with Elasto on Linux

In an earlier post I described the basics of the Microsoft Azure File Service, and how it can be used on Linux with the cifs.ko kernel client.

Since that time I've been hacking away on the Elasto cloud storage client, to the point that it now (with version 0.6.0) supports Azure File Service share provisioning as well as file and directory IO.

To play with Elasto yourself:

• Install the packages
• Download your Azure PublishSettings credentials
• Run

  elasto_cli -s Azure_PublishSettings_File -u afs://

Keep in mind that Elasto is still far from mature, so don't be surprised if it corrupts your data or causes a fire.
With the warning out of the way, I'd like to thank:

• My employer SUSE Linux, for supporting my Elasto development efforts during Hack Week.
• Samba Experience conference organisers, for giving me the chance to talk about the project.
• Kdenlive developers, for writing great video editing software.

Accénts & Ümlauts - A Custom Keyboard Layout on Linux

As a native English speaker living in Germany, I need to be able to reach the full Germanic alphabet without using long key combinations or (gasp) resorting to a German keyboard layout.

Accents and umlauts on US keyboards aren't only useful for expats. They're also enjoyed (or abused) by a number of English speaking subcultures:

• Hipsters: "This is such a naïve café."
• Metal heads: "Did you hear Spın̈al Tap are touring with Motörhead this year?"
• Teenage gamers: "über pwnage!"

The standard US system keyboard layout can be enhanced to offer German characters via the following key mappings:

Key	Key + Shift	Key + AltGr (Right Alt)	Key + AltGr + Shift
e	E	é	É
u	U	ü	Ü
o	O	ö	Ö
a	A	ä	Ä
s	S	ß	ß
5	%	€

With openSUSE and some other Linux distributions, this can be configured by first defining the mappings in /usr/share/X11/xkb/symbols/us_de:

partial default alphanumeric_keys
xkb_symbols "basic" {
    name[Group1]= "US/ASCII";
    include "us"

    key <AD03> {[e,          E,           eacute,         Eacute]};
    key <AD07> {[u,          U,           udiaeresis,     Udiaeresis]};
    key <AD09> {[o,          O,           odiaeresis,     Odiaeresis]};
    key <AC01> {[a,          A,           adiaeresis,     Adiaeresis]};
    key <AC02> {[s,          S,           ssharp,         ssharp]};
    key <AE05> {[NoSymbol, NoSymbol,      EuroSign]};

    key <RALT> {type[Group1]="TWO_LEVEL",
                [ISO_Level3_Shift, ISO_Level3_Shift]};

    modifier_map Mod5   {<RALT>};
};

Secondly, specify the keyboard layout as the system default in /etc/X11/xorg.conf.d/00-keyboard.conf:

Section "InputClass"
        Identifier "system-keyboard"
        MatchIsKeyboard "on"
        Option "XkbLayout" "us_de"
EndSection

Once in place, the key mappings can be easily modified to suit specific tastes or languages - viel Spaß!

Edits

• 2025-04-02: drop IBus config override section. IBus doesn't appear to be around anymore.

Using the Azure File Service on Linux

Update 2025-11-13: This post is ancient, so should not be used as a reference.
The Microsoft Azure File Service is a new SMB shared-storage service offered on the Microsoft Azure public cloud.

The service allows for the instant provisioning of file shares for private access by cloud provisioned VMs using the SMB 2.1 protocol, and additionally supports public access via a new REST interface.

Update 2015-05-25, edit 2025: File shares can now also not be provisioned from Linux using Elasto.

Linux VMs deployed on Azure can make use of this service using the Linux Kernel CIFS client. The kernel client must be configured to support and use the SMB 2.1 protocol dialect:

• CONFIG_CIFS_SMB2 must be enabled in the kernel configuration at build time
• Use

  # zcat /proc/config.gz | grep CONFIG_CIFS_SMB2

  to check this on a running system.

• The vers=2.1 mount.cifs parameter must be provided at mount time.
• Furthermore, the Azure storage account and access key must be provided as username and password.

# mount.cifs -o vers=2.1,user=smb //smb.file.core.windows.net/share /share/
Password for smb@//smb.file.core.windows.net/share:  ******...
# df -h /share/
Filesystem                         Size  Used Avail Use% Mounted on
//smb.file.core.windows.net/share  5.0T     0  5.0T   0% /share

This feature will be supported with the upcoming release of SUSE Linux Enterprise Server 12, and future openSUSE releases.

Disclaimer: I work in the Labs department at SUSE.