There were many other interesting talks during the conference, all of which can be viewed on the oSC 2017 media site.
A video of my presentation is embedded below.
Showing posts with label KVM. Show all posts
Showing posts with label KVM. Show all posts
Friday, June 9, 2017
Rapido: Quick Kernel Testing From Source (Video)
There were many other interesting talks during the conference, all of which can be viewed on the oSC 2017 media site.
A video of my presentation is embedded below.
Tuesday, December 13, 2016
Rapido: A Glorified Wrapper for Dracut and QEMU
Introduction
I've blogged a few of times about how Dracut and QEMU can be combined to greatly improve Linux kernel dev/test turnaround.
- My first post covered the basics of building the kernel, running dracut, and booting the resultant image with qemu-kvm.
- A later post took a closer look at network configuration, and focused on bridging VMs with the hypervisor.
- Finally, my third post looked at how this technique could be combined with Ceph, to provide a similarly efficient workflow for Ceph development.
Usage - Standalone Linux VM
The following procedure was tested on openSUSE Leap 42.3 and SLES 12SP3, but should work fine on many other Linux distributions.
Step 1: Checkout and Build
Checkout the Linux kernel and Rapido source repositories:
~/> cd ~ ~/> git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/> git clone https://github.com/rapido-linux/rapido.git
Build the kernel (using a config provided with the Rapido source):
~/> cp rapido/kernel/vanilla_config linux/.config ~/> cd linux ~/linux/> make -j6 ~/linux/> make modules ~/linux/> INSTALL_MOD_PATH=./mods make modules_install
Step 2: Configuration
Install Rapido dependencies: dracut and qemu.
Create a master rapido.conf configuration file using the example template:
~/linux/> cd ~/rapido ~/rapido/> cp rapido.conf.example rapido.conf ~/rapido/> vi rapido.conf
- set KERNEL_SRC="/home/<user>/linux"
- set KERNEL_INSTALL_MOD_PATH="${KERNEL_SRC}/mods"
- the remaining options can be left as is for now
Step 3: Image Generation
Generate a minimal Linux VM image which includes binaries, libraries and kernel modules for filesystem testing:
~/rapido/> ./cut_fstests_local.sh ... dracut: *** Creating initramfs image file 'initrds/myinitrd' done *** ~/rapido/> ls -lah initrds/myinitrd -rw-r--r-- 1 ddiss users 30M Dec 13 18:17 initrds/myinitrd
Step 4 - Boot!
~/rapido/> ./vm.sh + mount -t btrfs /dev/zram1 /mnt/scratch [ 3.542927] BTRFS info (device zram1): disk space caching is enabled ... btrfs filesystem mounted at /mnt/test and /mnt/scratch rapido1:/#
In a whopping four seconds, or thereabouts, the VM should have booted to a rapido:/# bash prompt. Leaving you with two zram backed Btrfs filesystems mounted at /mnt/test and /mnt/scratch.
Everything, including the VM's root filesystem, is in memory, so any changes will not persist across reboot. Use the rapido.conf QEMU_EXTRA_ARGS parameter if you wish to add persistent storage to a VM.
Once you're done playing around, you can shutdown:
rapido1:/# shutdown [ 267.304313] sysrq: SysRq : sysrq: Power Off rapido1:/# [ 268.168447] ACPI: Preparing to enter system sleep state S5 [ 268.169493] reboot: Power down + exit 0
Step 5: Network Configuration
The fstests_local VM above is networkless, so doesn't require bridge network configuration. For VMs that do (e.g. CephFS client below) edit rapido.conf:
- set TAP_USER="<user>"
- set MAC_ADDR1 to a valid MAC address, e.g. "b8:ac:24:45:c5:01"
- set MAC_ADDR2 to a valid MAC address, e.g. "b8:ac:24:45:c5:02"
Configure the isolated bridge and tap network devices. This must be done as root:
~/rapido/> sudo tools/br_setup.sh
~/rapido/> ip addr show br0
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
...
inet 192.168.155.1/24 scope global br0
Usage - Ceph vstart.sh cluster and CephFS client VM
This usage guide builds on the previous standalone Linux VM procedure, but this time adds Ceph to the mix. If you're not interested in Ceph (how could you not be!) then feel free to skip to the next section.Step I - Checkout and Build
We already have a clone of the Rapido and Linux kernel repositories. All that's needed for CephFS testing is a Ceph build:
~/> git clone https://github.com/ceph/ceph ~/> cd ceph <install Ceph build dependencies> ~/ceph/> ./do_cmake.sh -DWITH_MANPAGE=0 -DWITH_OPENLDAP=0 -DWITH_FUSE=0 -DWITH_NSS=0 -DWITH_LTTNG=0 ~/ceph/> cd build ~/ceph/build/> make -j4
Step II - Start a vstart.sh Ceph "cluster"
Once Ceph has finished compiling, vstart.sh can be run with the following parameters to configure and locally start three OSDs, one monitor process, and one MDS.
~/ceph/build/> OSD=3 MON=1 RGW=0 MDS=1 ../src/vstart.sh -i 192.168.155.1 -n
...
~/ceph/build/> bin/ceph -c status
...
health HEALTH_OK
monmap e2: 1 mons at {a=192.168.155.1:40160/0}
election epoch 4, quorum 0 a
fsmap e5: 1/1/1 up {0=a=up:active}
mgr no daemons active
osdmap e10: 3 osds: 3 up, 3 in
Step III - Rapido configuration
Edit rapido.conf, the master Rapido configuration file:
~/ceph/build/> cd ~/rapido ~/rapido/> vi rapido.conf
- set CEPH_SRC="/home/<user>/ceph/src"
- KERNEL_SRC and network parameters were configured earlier
Step IV - Image Generation
The cut_cephfs.sh script generates a VM image with the Ceph configuration and keyring from the vstart.sh cluster, as well as the CephFS kernel module.
~/rapido/> ./cut_cephfs.sh
... dracut: *** Creating initramfs image file 'initrds/myinitrd' done ***
Step V - Boot!
Booting the newly generated image should bring you to a shell prompt, with the vstart.sh provisioned CephFS filesystem mounted under /mnt/cephfs:
~/rapido/> ./vm.sh
...
+ mount -t ceph 192.168.155.1:40160:/ /mnt/cephfs -o name=admin,secret=... [ 3.492742] libceph: mon0 192.168.155.1:40160 session established ... rapido1:/# df -h /mnt/cephfs Filesystem Size Used Avail Use% Mounted on 192.168.155.1:40160:/ 1.3T 611G 699G 47% /mnt/cephfsCephFS is a clustered filesystem, in which case testing from multiple clients is also of interest. From another window, boot a second VM:
~/rapido/> ./vm.sh
Further Use Cases
Rapido ships with a bunch of scripts for testing different kernel components:
- cut_cephfs.sh (shown above)
- Image: includes Ceph config, credentials and CephFS kernel module
- Boot: mounts CephFS filesystem
- cut_cifs.sh
- Image: includes CIFS (SMB client) kernel module
- Boot: mounts share using details and credentials specified in rapido.conf
- cut_dropbear.sh
- Image: includes dropbear SSH server
- Boot: starts an SSH server with SSH_AUTHORIZED_KEY
- cut_fstests_cephfs.sh
- Image: includes xfstests and CephFS kernel client
- Boot: mounts CephFS filesystem and runs FSTESTS_AUTORUN_CMD
- cut_fstests_local.sh (shown above)
- Image: includes xfstests and local Btrfs and XFS dependencies
- Boot: provisions local xfstest zram devices. Runs FSTESTS_AUTORUN_CMD
- cut_lio_local.sh
- Image: includes LIO, loopback dev and dm-delay kernel modules
- Boot: provisions an iSCSI target, with three LUs exposed
- cut_lio_rbd.sh
- Image: includes LIO and Ceph RBD kernel modules
- Boot: provisions an iSCSI target backed by CEPH_RBD_IMAGE, using target_core_rbd
- cut_qemu_rbd.sh
- Image: CEPH_RBD_IMAGE is attached to the VM using qemu-block-rbd
- Boot: runs shell only
- cut_rbd.sh
- Image: includes Ceph config, credentials and Ceph RBD kernel module
- Boot: maps CEPH_RBD_IMAGE using the RBD kernel clien
- cut_samba_cephfs.sh
- Image: includes Ceph vstart config, credentials and libcephfs from CEPH_SRC, and additionally pulls in Samba from a (pre compiled) SAMBA_SRC
- Boot: configures smb.conf with a CephFS backed share and starts Samba
- cut_samba_local.sh
- Image: includes local kernel filesystem utils, and pulls in Samba from SAMBA_SRC
- Boot: configures smb.conf with a zram backed share and starts Samba
- cut_tcmu_rbd_loop.sh
- Image: includes Ceph config, librados, librbd, and pulls in tcmu-runner from TCMU_RUNNER_SRC
- Boot: starts tcmu-runner and configures a tcmu+rbd backstore exposing CEPH_RBD_IMAGE via the LIO loopback fabric
- cut_usb_rbd.sh (see https://github.com/ddiss/rbd-usb)
- Image: usb_f_mass_storage, zram, dm-crypt, and RBD_USB_SRC
- Boot: starts the conf-fs.sh script from RBD_USB_SRC
Conclusion
- Dracut and QEMU can be combined for super-fast Linux kernel testing and development.
- Rapido is mostly just a glorified wrapper around these utilities, but does provide some useful tools for automated testing of specific Linux kernel functionality.
If you run into any problems, or wish to provide any kind of feedback (always appreciated), please feel free to leave a message below, or raise a ticket in the Rapido issue tracker.
Update 20170106:
- Add cut_tcmu_rbd_loop.sh details and fix the example CEPH_SRC path.
- Use KERNEL_INSTALL_MOD_PATH instead of an ugly symlink
- Update Github links to refer to new project URL
- Remove old brctl and tunctl dependencies
- Split network setup into a separate section, as fstests_local VMs are now networkless
- Add cut_samba_cephfs.sh and cut_samba_local.sh details
Tuesday, June 28, 2016
Linux USB Gadget Application Testing
Developing a USB gadget application that runs on Linux?
Following a recent Ceph USB gateway project, I was looking at ways to test a Linux USB device without the need to fiddle with cables, or deal with slow embedded board boot times.
Ideally USB gadget testing could be performed by running the USB device code within a virtual machine, and attaching the VM's virtual USB device port to an emulated USB host controller on the hypervisor system.
I was unfortunately unable to find support for virtual USB device ports in QEMU, so I abandoned the above architecture, and discovered dummy_hcd.ko instead.
The dummy_hcd Linux kernel module is an excellent utility for USB device testing from within a standalone system or VM.
dummy_hcd.ko offers the following features:
Following a recent Ceph USB gateway project, I was looking at ways to test a Linux USB device without the need to fiddle with cables, or deal with slow embedded board boot times.
Ideally USB gadget testing could be performed by running the USB device code within a virtual machine, and attaching the VM's virtual USB device port to an emulated USB host controller on the hypervisor system.
The dummy_hcd Linux kernel module is an excellent utility for USB device testing from within a standalone system or VM.
dummy_hcd.ko offers the following features:
- Re-route USB device traffic back to the local system
- Effectively providing device loopback functionality
- USB high-speed and super-speed connection simulation
Tuesday, May 10, 2016
Rapid Ceph Kernel Module Testing with vstart.sh
Introduction
Ceph's vstart.sh utility is very useful for deploying and testing a mock cluster directly from the Ceph source repository. It can:- Generate a cluster configuration file and authentication keys
- Provision and deploy a number of OSDs
- Backed by local disk, or memory using the --memstore parameter
- Deploy an arbitrary number of monitor, MDS or rados-gateway nodes
Once deployed, the mock cluster can be used with any of the existing Ceph client utilities, or exercised with the unit tests in the Ceph src/test directory.
When developing or testing Linux kernel changes for CephFS or RBD, it's useful to also be able to use these kernel clients against a vstart.sh deployed Ceph cluster.
 |
| Test Environment Overview - image based on content by Sage Weil |
The instructions below walk through configuration and deployment of all components needed to test Linux kernel RBD and CephFS modules against a mock Ceph cluster. The procedure was performed on openSUSE Leap 42.1, but should also be applicable for other Linux distributions.
Network Setup
First off, configure a bridge interface to connect the Ceph cluster with a kernel client VM network:> sudo /sbin/brctl addbr br0 > sudo ip addr add 192.168.155.1/24 dev br0 > sudo ip link set dev br0 up
br0 will not be bridged with any physical adapters, just the kernel VM via a TAP interface which is configured with:
> sudo /sbin/tunctl -u $(whoami) -t tap0 > sudo /sbin/brctl addif br0 tap0 > sudo ip link set tap0 up
For more information on the bridge setup, see:
http://blog.elastocloud.org/2015/07/qemukvm-bridged-network-with-tap.html
Ceph Cluster Deployment
The Ceph cluster can now be deployed, with all nodes accepting traffic on the bridge network:> cd $ceph_source_dir <build Ceph> > cd src > OSD=3 MON=1 RGW=0 MDS=1 ./vstart.sh -i 192.168.155.1 -n --memstore
$ceph_source_dir should be replaced with the actual path. Be sure to specify the same IP address with -i as was assigned to the br0 interface.
More information about vstart.sh usage can be found at:
http://docs.ceph.com/docs/hammer/dev/dev_cluster_deployement/
Kernel VM Deployment
Build a kernel:> cd $kernel_source_dir > make menuconfig$kernel_source_dir should be replaced with the actual path. Ensure CONFIG_BLK_DEV_RBD=m, CONFIG_CEPH_FS=y, CONFIG_CEPH_LIB=y, CONFIG_E1000=y and CONFIG_IP_PNP=y are set in the kernel config. A sample can be found here.
> make > INSTALL_MOD_PATH=./mods make modules_install
Create a link to the modules directory ./mods, so that Dracut can find them:
> sudo ln -s $PWD/mods/lib/modules/$(make kernelrelease) \
/lib/modules/$(make kernelrelease)
Generate an initramfs with Dracut. This image will be used as the test VM.
> export CEPH_SRC=$ceph_source_dir/src
> dracut --no-compress --kver "$(cat include/config/kernel.release)" \
--install "tail blockdev ps rmdir resize dd vim grep find df sha256sum \
strace mkfs.xfs /lib64/libkeyutils.so.1" \
--include "$CEPH_SRC/mount.ceph" "/sbin/mount.ceph" \
--include "$CEPH_SRC/ceph.conf" "/etc/ceph/ceph.conf" \
--add-drivers "rbd" \
--no-hostonly --no-hostonly-cmdline \
--modules "bash base network ifcfg" \
--force myinitrd
Boot the kernel and initramfs directly using QEMU/KVM:
> qemu-kvm -smp cpus=2 -m 512 \
-kernel arch/x86/boot/bzImage -initrd myinitrd \
-device e1000,netdev=network1,mac=b8:ac:6f:31:45:70 \
-netdev tap,id=network1,script=no,downscript=no,ifname=tap0 \
-append "ip=192.168.155.2:::255.255.255.0:myhostname \
rd.shell=1 console=ttyS0 rd.lvm=0 rd.luks=0" \
-nographic
This should bring up a Dracut debug shell in the VM, with a network configuration matching the values parsed in via the ip= kernel parameter.
dracut:/# ip a
...
2: eth0: ... mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether b8:ac:6f:31:45:70 brd ff:ff:ff:ff:ff:ff
inet 192.168.155.2/24 brd 192.168.155.255 scope global eth0
For more information on kernel setup, see:
http://blog.elastocloud.org/2015/06/rapid-linux-kernel-devtest-with-qemu.html
RBD Image Provisioning
An RBD volume can be provisioned using the regular Ceph utilities in the Ceph source directory:> cd $ceph_source_dir/src > ./rados lspools rbd ...
By default, an rbd pool is created by vstart.sh, which can be used for RBD images:
> ./rbd create --image-format 1 --size 1024 1g_vstart_img > ./rbd ls -l NAME SIZE PARENT FMT PROT LOCK 1g_vstart_img 1024M 1
Note: "--image-format 1" is specified to ensure that the kernel supports all features of the provisioned RBD image.
Kernel RBD Usage
From the Dracut shell, the newly provisioned 1g_vstart_img image can be mapped locally using the sysfs filesystem:dracut:/# modprobe rbd [ 9.031056] rbd: loaded dracut:/# echo -n "192.168.155.1:6789 name=admin,secret=AQBPiuhd9389dh28djASE32Ceiojc234AF345w== rbd 1g_vstart_img -" > /sys/bus/rbd/add [ 347.743272] libceph: mon0 192.168.155.1:6789 session established [ 347.744284] libceph: client4121 fsid 234b432f-a895-43d2-23fd-9127a1837b32 [ 347.749516] rbd: rbd0: added with size 0x40000000
Note: The monitor address and admin credentials can be retrieved from the ceph.conf and keyring files respectively, located in the Ceph source directory.
The /dev/rbd0 mapped image can now be used like any other block device:
dracut:/# mkfs.xfs /dev/rbd0 ... dracut:/# mkdir -p /mnt/rbdfs dracut:/# mount /dev/rbd0 /mnt/rbdfs [ 415.841757] XFS (rbd0): Mounting V4 Filesystem [ 415.917595] XFS (rbd0): Ending clean mount dracut:/# df -h /mnt/rbdfs Filesystem Size Used Avail Use% Mounted on /dev/rbd0 1014M 33M 982M 4% /mnt/rbdfs
Kernel CephFS Usage
vstart.sh already goes to the effort of deploying a filesystem:> cd $ceph_source_dir/src > ./ceph fs ls > name: cephfs_a, metadata pool: cephfs_metadata_a, data pools: [cephfs_data_a ]All that's left is to mount it from the kernel VM using the mount.ceph binary that was copied into the initramfs:
dracut:/# mkdir -p /mnt/mycephfs
dracut:/# mount.ceph 192.168.155.1:6789:/ /mnt/mycephfs \
-o name=admin,secret=AQBPiuhd9389dh28djASE32Ceiojc234AF345w==
[ 723.103153] libceph: mon0 192.168.155.1:6789 session established
[ 723.184978] libceph: client4122 fsid 234b432f-a895-43d2-23fd-9127a1837b32
dracut:/# df -h /mnt/mycephfs/
Filesystem Size Used Avail Use% Mounted on
192.168.155.1:6789:/ 3.0G 4.0M 3.0G 1% /mnt/mycephfs
Note: The monitor address and admin credentials can be retrieved from the ceph.conf and keyring files respectively, located in the Ceph source directory.
Cleanup
Unmount CephFS:dracut:/# umount /mnt/mycephfsUnmount the RBD image:
dracut:/# umount /dev/rbd0 [ 1592.592510] XFS (rbd0): Unmounting FilesystemUnmap the RBD image (0 is derived from /dev/rbdX):
dracut:/# echo -n 0 > /sys/bus/rbd/removePower-off the VM:
dracut:/# echo 1 > /proc/sys/kernel/sysrq && echo o > /proc/sysrq-trigger [ 1766.387417] sysrq: SysRq : Power Off dracut:/# [ 1766.811686] ACPI: Preparing to enter system sleep state S5 [ 1766.812217] reboot: Power downShutdown the Ceph cluster:
> cd $ceph_source_dir/src > ./stop.sh
Conclusion
A mock Ceph cluster can be deployed from source in a matter of seconds using the vstart.sh utility.Likewise, a kernel can be booted directly from source alongside a throwaway VM and connected to the mock Ceph cluster in a couple of minutes with Dracut and QEMU/KVM.
This environment is ideal for rapid development and integration testing of Ceph user-space and kernel components, including RBD and CephFS.
Sunday, July 12, 2015
QEMU/KVM Bridged Network with TAP interfaces
In my previous post, Rapid Linux Kernel Dev/Test with QEMU, KVM and Dracut, I described how build and boot a Linux kernel quickly, making use of port forwarding between hypervisor and guest VM for virtual network traffic.
This post describes how to plumb the Linux VM directly into a hypervisor network, through the use of a bridge.
Start by creating a bridge on the hypervisor system:
Clear the IP address on the network interface that you'll be bridging (e.g. eth0).
Note: This will disable network traffic on eth0!
Next up, create a TAP interface:
Add the TAP interface to the bridge:
Make sure everything is up:
The TAP interface is now ready for use. Assuming that a DHCP server is available on the bridged network, the VM can now obtain an IP address during boot via:
The MAC address is explicitly specified, so care should be taken to ensure its uniqueness.
The DHCP server response details are printed alongside network interface configuration. E.g.
Didn't get an IP address? There are a few things to check:
Update 20161223:
This post describes how to plumb the Linux VM directly into a hypervisor network, through the use of a bridge.
Start by creating a bridge on the hypervisor system:
> sudo ip link add br0 type bridge
Clear the IP address on the network interface that you'll be bridging (e.g. eth0).
Note: This will disable network traffic on eth0!
> sudo ip addr flush dev eth0Add the interface to the bridge:
> sudo ip link set eth0 master br0
Next up, create a TAP interface:
> sudo ip tuntap add dev tap0 mode tap user $(whoami)The user parameter ensures that the current user will be able to connect to the TAP interface.
Add the TAP interface to the bridge:
> sudo ip link set tap0 master br0
Make sure everything is up:
> sudo ip link set dev br0 up > sudo ip link set dev tap0 up
The TAP interface is now ready for use. Assuming that a DHCP server is available on the bridged network, the VM can now obtain an IP address during boot via:
> qemu-kvm -kernel arch/x86/boot/bzImage \
-initrd initramfs \
-device e1000,netdev=network0,mac=52:55:00:d1:55:01 \
-netdev tap,id=network0,ifname=tap0,script=no,downscript=no \
-append "ip=dhcp rd.shell=1 console=ttyS0" -nographic
The MAC address is explicitly specified, so care should be taken to ensure its uniqueness.
The DHCP server response details are printed alongside network interface configuration. E.g.
[ 3.792570] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 3.796085] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 3.812083] Sending DHCP requests ., OK [ 4.824174] IP-Config: Got DHCP answer from 10.155.0.42, my address is 10.155.0.1 [ 4.825119] IP-Config: Complete: [ 4.825476] device=eth0, hwaddr=52:55:00:d1:55:01, ipaddr=10.155.0.1, mask=255.255.0.0, gw=10.155.0.254 [ 4.826546] host=rocksolid-sles, domain=suse.de, nis-domain=suse.de ...
Didn't get an IP address? There are a few things to check:
- Confirm that the kernel is built with boot-time DHCP client (CONFIG_IP_PNP_DHCP=y) and E1000 network driver (CONFIG_E1000=y) support.
- Check the -device and -netdev arguments specify a valid e1000 TAP interface.
- Ensure that ip=dhcp is provided as a kernel boot parameter, and that the DHCP server is up and running.
Update 20161223:
- Use 'ip' instead of 'brctl' to manipulate the bridge device - thanks Yagamy Light!
- Use 'ip tuntap' instead of 'tunctl' to create the TAP interface - thanks Johannes!
- For performance reasons, I strongly recommend using virtio network adapters instead of e1000.
Wednesday, June 10, 2015
Rapid Linux Kernel Dev/Test with QEMU, KVM and Dracut
Update 2018-01-02: See my post on Rapido for a more automated approach to the procedure outlined below.
Inspired by Stefan Hajnoczi's excellent blog post, I recently set about constructing an environment for rapid testing of Linux kernel changes, particularly focused on the LIO iSCSI target. Such an environment would help me in number of ways:
- Faster dev / test turnaround.
- A modified kernel can be compiled and booted in a matter of seconds.
- Improved resource utilisation.
- No need to boot external test hosts or heavyweight VMs.
- Simplified and speedier debugging.
My requirements were slightly different to Stefan's, in that:
- I'd prefer to be lazy and use Dracut for initramfs generation.
- I need a working network connection between VM and hypervisor system
- The VM will act as the iSCSI target, the hypervisor as the initiator.
Starting with the Linux kernel, the first step is to build a bzimage:
~/> git clone \
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
hack, hack, hack.
~/linux/> make menuconfig
Set CONFIG_IP_PNP_DHCP=y and CONFIG_E1000=y to enable IP address assignment on boot.
~/linux/> make -j6
~/linux/> make modules
~/linux/> INSTALL_MOD_PATH=./mods make modules_install
~/linux/> sudo ln -s $PWD/mods/lib/modules/$(make kernelrelease) \
/lib/modules/$(make kernelrelease)
This leaves us with a compressed kernel image file at arch/x86/boot/bzimage, and corresponding modules installed under mods/lib/module/$(make kernelrelease), where $(make kernelrelease) evaluates to 4.1.0-rc7+ in this example. The /lib/modules/4.1.0-rc7+ symlink allows Dracut to locate the modules.The next step is to generate an initial RAM filesystem, or initramfs, which includes a minimal set of user-space utilities, and kernel modules needed for testing:
~/linux/> dracut --kver "$(make kernelrelease)" \
--add-drivers "iscsi_target_mod target_core_mod" \
--add-drivers "target_core_file target_core_iblock" \
--add-drivers "configfs" \
--install "ps grep netstat" \
--no-hostonly --no-hostonly-cmdline \
--modules "bash base shutdown network ifcfg" initramfs
...
*** Creating image file done ***
We now have an initramfs file in the current directory, with the following contents:
- LIO kernel modules obtained from /lib/module/4.1.0-rc7, as directed via the --kver and --add-drivers parameters.
- User-space shell, boot and network helpers, as directed via the --modules parameter.
We're now ready to use QEMU/KVM to boot our test kernel and initramfs:
~/linux/> qemu-kvm -kernel arch/x86/boot/bzImage \
-initrd initramfs \
-device e1000,netdev=network0 \
-netdev user,id=network0 \
-redir tcp:51550::3260 \
-append "ip=dhcp rd.shell=1 console=ttyS0" \
-nographic
This boots the test environment, with the kernel and initramfs previously generated:
[ 3.216596] dracut Warning: dracut: FATAL: No or empty root= argument [ 3.217998] dracut Warning: dracut: Refusing to continue ... Dropping to debug shell. dracut:/#
From the dracut shell, confirm that the QEMU DHCP server assigned the VM an IP address:
dracut:/# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default ...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
...
inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
Port 3260 (iSCSI) on this interface is forwarded to/from port 51550 on the hypervisor, as configured via the qemu-kvm -redir parameter.
Now onto LIO iSCSI target setup. First off load the appropriate kernel modules:
dracut:/# modprobe iscsi_target_mod dracut:/# cat /proc/modules iscsi_target_mod 246669 0 - Live 0xffffffffa006a000 target_core_mod 289004 1 iscsi_target_mod, Live 0xffffffffa000b000 configfs 22407 3 iscsi_target_mod,target_core_mod, Live 0xffffffffa0000000
LIO configuration requires a mounted configfs filesystem:
dracut:/# mount -t configfs configfs /sys/kernel/config/ dracut:/# cat /sys/kernel/config/target/version Target Engine Core ConfigFS Infrastructure v4.1.0 on Linux/x86_64 on 4.1.0-rc1+
An iSCSI target can be provisioned by manipulating corresponding configfs entries. I used the lio_dump output on an existing setup as reference:
dracut:/# mkdir /sys/kernel/config/target/iscsi
dracut:/# echo -n 0 > /sys/kernel/config/target/iscsi/discovery_auth/enforce_discovery_auth dracut:/# mkdir -p /sys/kernel/config/target/iscsi/<iscsi_iqn>/tpgt_1/np/10.0.2.15:3260 ...
Finally, we're ready to connect to the LIO target using the local hypervisor port that forwards to the VM's virtual network adapter:
~/linux/> iscsiadm --mode discovery \
--type sendtargets \
--portal 127.0.0.1:51550
10.0.2.15:3260,1 iqn.2015-04.suse.arch:5eca2313-028d-435c-9131-53a5ab256a83
It works!
There are a few things that can be adjusted:
- Port forwarding to the VM network is a bit fiddly - I'm now using a bridge/TAP configuration instead.
- When dropping into the emergency boot shell, Dracut executes scripts carried under /lib/dracut/hooks/emergency/. This means that a custom script can be triggered on boot via:
~/linux/> dracut -i runme.sh /lib/dracut/hooks/emergency/02-runme.sh ...
- It should be possible to have Dracut pull the kernel modules in from the
temporary directory, but I wasn't able to get this working:
~/linux/> INSTALL_MOD_PATH=./mods make modules_install ~/linux/> dracut --kver "$(make kernelrelease)" --kmoddir ./mods/lib/...
- Boot time and initramfs file IO performance can be improved by disabling compression. This is done by specifying the --no-compress Dracut parameter.
Update 20150722:
- Don't install kernel modules as root, set up a /lib/modules symlink for Dracut instead.
- Link to bridge/TAP networking post.
- Describe boot script usage.
- Use $(make kernelrelease) rather than a hard-coded 4.1.0-rc7+ kernel version string - thanks Aurélien!
Update 20180102: Link to the Rapido project.
Subscribe to:
Posts (Atom)