Rapido: Quick Kernel Testing From Source (Video)

I presented a short talk at the 2017 openSUSE Conference on Linux kernel testing using Rapido.

There were many other interesting talks during the conference, all of which can be viewed on the oSC 2017 media site.
A video of my presentation is embedded below.

Many thanks to the organisers and sponsors for putting on a great event.

Rapid Ceph Kernel Module Testing with vstart.sh

Introduction

Ceph's vstart.sh utility is very useful for deploying and testing a mock cluster directly from the Ceph source repository. It can:

• Generate a cluster configuration file and authentication keys
• Provision and deploy a number of OSDs
• Backed by local disk, or memory using the --memstore parameter
• Deploy an arbitrary number of monitor, MDS or rados-gateway nodes

All services are deployed as the running user. I.e. root access is not needed.

Once deployed, the mock cluster can be used with any of the existing Ceph client utilities, or exercised with the unit tests in the Ceph src/test directory.

When developing or testing Linux kernel changes for CephFS or RBD, it's useful to also be able to use these kernel clients against a vstart.sh deployed Ceph cluster.

Test Environment Overview - image based on content by Sage Weil

The instructions below walk through configuration and deployment of all components needed to test Linux kernel RBD and CephFS modules against a mock Ceph cluster. The procedure was performed on openSUSE Leap 42.1, but should also be applicable for other Linux distributions.

Network Setup

First off, configure a bridge interface to connect the Ceph cluster with a kernel client VM network:

> sudo /sbin/brctl addbr br0
> sudo ip addr add 192.168.155.1/24 dev br0
> sudo ip link set dev br0 up

br0 will not be bridged with any physical adapters, just the kernel VM via a TAP interface which is configured with:

> sudo /sbin/tunctl -u $(whoami) -t tap0
> sudo /sbin/brctl addif br0 tap0
> sudo ip link set tap0 up

For more information on the bridge setup, see:
http://blog.elastocloud.org/2015/07/qemukvm-bridged-network-with-tap.html

Ceph Cluster Deployment

The Ceph cluster can now be deployed, with all nodes accepting traffic on the bridge network:

> cd $ceph_source_dir
<build Ceph>
> cd src
> OSD=3 MON=1 RGW=0 MDS=1 ./vstart.sh -i 192.168.155.1 -n --memstore

$ceph_source_dir should be replaced with the actual path. Be sure to specify the same IP address with -i as was assigned to the br0 interface.

More information about vstart.sh usage can be found at:
 http://docs.ceph.com/docs/hammer/dev/dev_cluster_deployement/

Kernel VM Deployment

Build a kernel:

> cd $kernel_source_dir
> make menuconfig 

$kernel_source_dir should be replaced with the actual path. Ensure CONFIG_BLK_DEV_RBD=m, CONFIG_CEPH_FS=y, CONFIG_CEPH_LIB=y, CONFIG_E1000=y and CONFIG_IP_PNP=y are set in the kernel config. A sample can be found here.

> make
> INSTALL_MOD_PATH=./mods make modules_install

Create a link to the modules directory ./mods, so that Dracut can find them:

> sudo ln -s $PWD/mods/lib/modules/$(make kernelrelease) \
                /lib/modules/$(make kernelrelease)

Generate an initramfs with Dracut. This image will be used as the test VM.

> export CEPH_SRC=$ceph_source_dir/src
> dracut --no-compress  --kver "$(cat include/config/kernel.release)" \
        --install "tail blockdev ps rmdir resize dd vim grep find df sha256sum \
                   strace mkfs.xfs /lib64/libkeyutils.so.1" \
        --include "$CEPH_SRC/mount.ceph" "/sbin/mount.ceph" \
        --include "$CEPH_SRC/ceph.conf" "/etc/ceph/ceph.conf" \
        --add-drivers "rbd" \
        --no-hostonly --no-hostonly-cmdline \
        --modules "bash base network ifcfg" \
        --force myinitrd

Boot the kernel and initramfs directly using QEMU/KVM:

> qemu-kvm -smp cpus=2 -m 512 \
        -kernel arch/x86/boot/bzImage -initrd myinitrd \
        -device e1000,netdev=network1,mac=b8:ac:6f:31:45:70 \
        -netdev tap,id=network1,script=no,downscript=no,ifname=tap0 \
        -append "ip=192.168.155.2:::255.255.255.0:myhostname \
                rd.shell=1 console=ttyS0 rd.lvm=0 rd.luks=0" \
        -nographic

This should bring up a Dracut debug shell in the VM, with a network configuration matching the values parsed in via the ip= kernel parameter.

dracut:/# ip a
...
2: eth0: ... mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:ac:6f:31:45:70 brd ff:ff:ff:ff:ff:ff
    inet 192.168.155.2/24 brd 192.168.155.255 scope global eth0

For more information on kernel setup, see:
http://blog.elastocloud.org/2015/06/rapid-linux-kernel-devtest-with-qemu.html

RBD Image Provisioning

An RBD volume can be provisioned using the regular Ceph utilities in the Ceph source directory:

> cd $ceph_source_dir/src
> ./rados lspools
rbd
...

By default, an rbd pool is created by vstart.sh, which can be used for RBD images:

> ./rbd create --image-format 1 --size 1024 1g_vstart_img
> ./rbd ls -l
NAME           SIZE PARENT FMT PROT LOCK
1g_vstart_img 1024M          1

Note: "--image-format 1" is specified to ensure that the kernel supports all features of the provisioned RBD image.

Kernel RBD Usage

From the Dracut shell, the newly provisioned 1g_vstart_img image can be mapped locally using the sysfs filesystem:

dracut:/# modprobe rbd
[    9.031056] rbd: loaded
dracut:/# echo -n "192.168.155.1:6789 name=admin,secret=AQBPiuhd9389dh28djASE32Ceiojc234AF345w== rbd 1g_vstart_img -" > /sys/bus/rbd/add
[  347.743272] libceph: mon0 192.168.155.1:6789 session established
[  347.744284] libceph: client4121 fsid 234b432f-a895-43d2-23fd-9127a1837b32
[  347.749516] rbd: rbd0: added with size 0x40000000

Note: The monitor address and admin credentials can be retrieved from the ceph.conf and keyring files respectively, located in the Ceph source directory.

The /dev/rbd0 mapped image can now be used like any other block device:

dracut:/# mkfs.xfs /dev/rbd0 
...
dracut:/# mkdir -p /mnt/rbdfs
dracut:/# mount /dev/rbd0 /mnt/rbdfs
[  415.841757] XFS (rbd0): Mounting V4 Filesystem
[  415.917595] XFS (rbd0): Ending clean mount
dracut:/# df -h /mnt/rbdfs
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd0      1014M   33M  982M   4% /mnt/rbdfs

Kernel CephFS Usage

vstart.sh already goes to the effort of deploying a filesystem:

> cd $ceph_source_dir/src
> ./ceph fs ls
> name: cephfs_a, metadata pool: cephfs_metadata_a, data pools: [cephfs_data_a ]

All that's left is to mount it from the kernel VM using the mount.ceph binary that was copied into the initramfs:

dracut:/# mkdir -p /mnt/mycephfs
dracut:/# mount.ceph 192.168.155.1:6789:/ /mnt/mycephfs \
                -o name=admin,secret=AQBPiuhd9389dh28djASE32Ceiojc234AF345w==
[  723.103153] libceph: mon0 192.168.155.1:6789 session established
[  723.184978] libceph: client4122 fsid 234b432f-a895-43d2-23fd-9127a1837b32

dracut:/# df -h /mnt/mycephfs/
Filesystem            Size  Used Avail Use% Mounted on
192.168.155.1:6789:/  3.0G  4.0M  3.0G   1% /mnt/mycephfs

Note: The monitor address and admin credentials can be retrieved from the ceph.conf and keyring files respectively, located in the Ceph source directory.

Cleanup

Unmount CephFS:

dracut:/# umount /mnt/mycephfs

Unmount the RBD image:

dracut:/# umount /dev/rbd0
[ 1592.592510] XFS (rbd0): Unmounting Filesystem

Unmap the RBD image (0 is derived from /dev/rbdX):

dracut:/# echo -n 0 > /sys/bus/rbd/remove

Power-off the VM:

dracut:/# echo 1 > /proc/sys/kernel/sysrq && echo o > /proc/sysrq-trigger
[ 1766.387417] sysrq: SysRq : Power Off
dracut:/# [ 1766.811686] ACPI: Preparing to enter system sleep state S5
[ 1766.812217] reboot: Power down

Shutdown the Ceph cluster:

> cd $ceph_source_dir/src
> ./stop.sh

Conclusion

A mock Ceph cluster can be deployed from source in a matter of seconds using the vstart.sh utility.
Likewise, a kernel can be booted directly from source alongside a throwaway VM and connected to the mock Ceph cluster in a couple of minutes with Dracut and QEMU/KVM.

This environment is ideal for rapid development and integration testing of Ceph user-space and kernel components, including RBD and CephFS.

Rapid Linux Kernel Dev/Test with QEMU, KVM and Dracut

Update 2018-01-02: See my post on Rapido for a more automated approach to the procedure outlined below.

Inspired by Stefan Hajnoczi's excellent blog post, I recently set about constructing an environment for rapid testing of Linux kernel changes, particularly focused on the LIO iSCSI target. Such an environment would help me in number of ways:

• Faster dev / test turnaround.
• A modified kernel can be compiled and booted in a matter of seconds.
• Improved resource utilisation.
• No need to boot external test hosts or heavyweight VMs.
•  Simplified and speedier debugging.
• GDB and Wireshark can be plumbed directly into QEMU/KVM.

My requirements were slightly different to Stefan's, in that:

• I'd prefer to be lazy and use Dracut for initramfs generation.
• I need a working network connection between VM and hypervisor system
• The VM will act as the iSCSI target, the hypervisor as the initiator.

Starting with the Linux kernel, the first step is to build a bzimage:

~/> git clone \
        git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

hack, hack, hack.

~/linux/> make menuconfig

Set CONFIG_IP_PNP_DHCP=y and CONFIG_E1000=y to enable IP address assignment on boot.

~/linux/> make -j6
~/linux/> make modules
~/linux/> INSTALL_MOD_PATH=./mods make modules_install
~/linux/> sudo ln -s $PWD/mods/lib/modules/$(make kernelrelease) \
                     /lib/modules/$(make kernelrelease)

This leaves us with a compressed kernel image file at arch/x86/boot/bzimage, and corresponding modules installed under mods/lib/module/$(make kernelrelease), where $(make kernelrelease) evaluates to 4.1.0-rc7+ in this example. The /lib/modules/4.1.0-rc7+ symlink allows Dracut to locate the modules.

The next step is to generate an initial RAM filesystem, or initramfs, which includes a minimal set of user-space utilities, and kernel modules needed for testing:

~/linux/> dracut --kver "$(make kernelrelease)" \
                 --add-drivers "iscsi_target_mod target_core_mod" \
                 --add-drivers "target_core_file target_core_iblock" \
                 --add-drivers "configfs" \
                 --install "ps grep netstat" \
                 --no-hostonly --no-hostonly-cmdline \
                 --modules "bash base shutdown network ifcfg" initramfs
...
*** Creating image file done ***

We now have an initramfs file in the current directory, with the following contents:

• LIO kernel modules obtained from /lib/module/4.1.0-rc7, as directed via the --kver and --add-drivers parameters.
• User-space shell, boot and network helpers, as directed via the --modules parameter.

We're now ready to use QEMU/KVM to boot our test kernel and initramfs:

~/linux/> qemu-kvm -kernel arch/x86/boot/bzImage \
                   -initrd initramfs \
                   -device e1000,netdev=network0 \
                   -netdev user,id=network0 \
                   -redir tcp:51550::3260 \
                   -append "ip=dhcp rd.shell=1 console=ttyS0" \
                   -nographic

This boots the test environment, with the kernel and initramfs previously generated:

[    3.216596] dracut Warning: dracut: FATAL: No or empty root= argument
[    3.217998] dracut Warning: dracut: Refusing to continue
...
Dropping to debug shell.

dracut:/#

From the dracut shell, confirm that the QEMU DHCP server assigned the VM an IP address:

dracut:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
...

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
...
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0

Port 3260 (iSCSI) on this interface is forwarded to/from port 51550 on the hypervisor, as configured via the qemu-kvm -redir parameter.

Now onto LIO iSCSI target setup. First off load the appropriate kernel modules:

dracut:/# modprobe iscsi_target_mod
dracut:/# cat /proc/modules
iscsi_target_mod 246669 0 - Live 0xffffffffa006a000
target_core_mod 289004 1 iscsi_target_mod, Live 0xffffffffa000b000
configfs 22407 3 iscsi_target_mod,target_core_mod, Live 0xffffffffa0000000

LIO configuration requires a mounted configfs filesystem:

dracut:/# mount -t configfs configfs /sys/kernel/config/
dracut:/# cat /sys/kernel/config/target/version 
Target Engine Core ConfigFS Infrastructure v4.1.0 on Linux/x86_64 on 4.1.0-rc1+

An iSCSI target can be provisioned by manipulating corresponding configfs entries. I used the lio_dump output on an existing setup as reference:

dracut:/# mkdir /sys/kernel/config/target/iscsi

dracut:/# echo -n 0 > /sys/kernel/config/target/iscsi/discovery_auth/enforce_discovery_auth
dracut:/# mkdir -p /sys/kernel/config/target/iscsi/<iscsi_iqn>/tpgt_1/np/10.0.2.15:3260
...

Finally, we're ready to connect to the LIO target using the local hypervisor port that forwards to the VM's virtual network adapter:

~/linux/> iscsiadm --mode discovery \
                   --type sendtargets \
                   --portal 127.0.0.1:51550
10.0.2.15:3260,1 iqn.2015-04.suse.arch:5eca2313-028d-435c-9131-53a5ab256a83

It works!

There are a few things that can be adjusted:

• Port forwarding to the VM network is a bit fiddly - I'm now using a bridge/TAP configuration instead.
• When dropping into the emergency boot shell, Dracut executes scripts carried under /lib/dracut/hooks/emergency/. This means that a custom script can be triggered on boot via:

  ~/linux/> dracut -i runme.sh /lib/dracut/hooks/emergency/02-runme.sh ...
  

• It should be possible to have Dracut pull the kernel modules in from the temporary directory, but I wasn't able to get this working:

  ~/linux/> INSTALL_MOD_PATH=./mods make modules_install
  ~/linux/> dracut --kver "$(make kernelrelease)" --kmoddir ./mods/lib/...
  

• Boot time and initramfs file IO performance can be improved by disabling compression. This is done by specifying the --no-compress Dracut parameter.

Update 20150722:

• Don't install kernel modules as root, set up a /lib/modules symlink for Dracut instead.
• Link to bridge/TAP networking post.
• Describe boot script usage.

Update 20150813:

• Use $(make kernelrelease) rather than a hard-coded 4.1.0-rc7+ kernel version string - thanks Aurélien!

Update 20150908: Describe initramfs --no-compress optimisation.
Update 20180102: Link to the Rapido project.

Using the Azure File Service on Linux

Update 2025-11-13: This post is ancient, so should not be used as a reference.
The Microsoft Azure File Service is a new SMB shared-storage service offered on the Microsoft Azure public cloud.

The service allows for the instant provisioning of file shares for private access by cloud provisioned VMs using the SMB 2.1 protocol, and additionally supports public access via a new REST interface.

Update 2015-05-25, edit 2025: File shares can now also not be provisioned from Linux using Elasto.

Linux VMs deployed on Azure can make use of this service using the Linux Kernel CIFS client. The kernel client must be configured to support and use the SMB 2.1 protocol dialect:

• CONFIG_CIFS_SMB2 must be enabled in the kernel configuration at build time
• Use

  # zcat /proc/config.gz | grep CONFIG_CIFS_SMB2

  to check this on a running system.

• The vers=2.1 mount.cifs parameter must be provided at mount time.
• Furthermore, the Azure storage account and access key must be provided as username and password.

# mount.cifs -o vers=2.1,user=smb //smb.file.core.windows.net/share /share/
Password for smb@//smb.file.core.windows.net/share:  ******...
# df -h /share/
Filesystem                         Size  Used Avail Use% Mounted on
//smb.file.core.windows.net/share  5.0T     0  5.0T   0% /share

This feature will be supported with the upcoming release of SUSE Linux Enterprise Server 12, and future openSUSE releases.

Disclaimer: I work in the Labs department at SUSE.