Converting the Root Filesystem to ZFS on Fedora Linux

What This Document Covers

This document is a step-by-step guide on how to convert an existing installation of Fedora 25 that is not currently using ZFS to using ZFS for all primary filesystems (/, /boot, /var, etc). At the end of the document, the zpool will be expanded across a second storage device providing a mirror setup. This is done without any data loss and with minimal downtime.

While there "should" be no data loss, the operations this guide will recommend are inherently risky and a typo or disk failure while following this guide might destroy all existing data. Make a backup before proceeding.

This guide is written for Fedora 25, however, it should work identically in Fedora 26. Users of other distributions can use this guide as a general outline.

Why Use ZFS-on-Linux

The primary feature I wanted from switching to ZFS was the ability ZFS has to detect silent data corruption and self-heal when data corruption is detected (in mirror setups). When ZFS is used in a two-disk mirror, it also has the benefit of nearly doubling read speeds, which is generally unheard of in other RAID-1 setups. The transparent compression was also compelling; comparing before and after results on my system, the compression saves about 15GiB of storage space. ZFS has many other great features which are documented on the ZFS Wikipedia page.

Prerequisite Reading

Before attempting to follow this guide, an understanding of ZFS and the ZFS-on-Linux Project is required. Please see these resources:

Overview of Converting to ZFS-on-Linux

The rest of this document will be a step-by-step guide to converting an existing installation of Fedora 25 to use ZFS as the / and /boot filesystems. The high-level overview of the plan is to install ZFS-on-Linux, create a ZFS filesystem on a second storage device, use rsync to copy over all the data from the current operating system install, install GRUB2 on the second storage device, boot off the second storage device, and add the first storage device to the zpool as a mirror device. Easy.

Assumptions#Top

/dev/sda is the disk with the non-ZFS system install; /dev/sdb is a blank unused storage device. Both drives are the same size (250GiB), which is important, if planning on creating the mirrored storage pool as instructed at the end of this guide.

In this guide, I am using an MBR-style partition since my system doesn't use EFI. However, this guide will work for an EFI enabled system, although the path to the grub.cfg file will be different and an EFI System Partition would need to be created.

Initial Disk and ZFS Setup#Top

Ideally, we'd just give the entire block device /dev/sdb to ZFS, but since we need somewhere to install GRUB2 we'll create an MBR style partition table with one partition that starts 1MiB into the space and takes up all free space.

On /dev/sdb, create a partition table with one partition filling the entire drive. Ensure the first partition starts 1MiB into the drive, and set the partition ID to bf/Solaris, otherwise GRUB2 will throw errors at boot:

fdisk -l /dev/sdb
Disk /dev/sdb: 232.9 GiB, 250059350016 bytes, 488397168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe68f3500

Device     Boot Start       End   Sectors   Size Id Type
/dev/sdb1        2048 488397167 488395120 232.9G bf Solaris

Make a note of your current kernel version. Ensure when the kernel-headers and kernel-devel packages get installed they match your running kernel. Otherwise the ZFS DKMS module will fail to build:

uname -r
4.11.12-200.fc25.x86_64

Install the packages for ZFS-on-Linux. Pay special attention to ensure the kernel-headers and kernel-devel package versions match that of the installed kernel:

dnf install -y http://download.zfsonlinux.org/fedora/zfs-release.fc25.noarch.rpm
dnf install -y kernel-headers kernel-devel zfs zfs-dkms zfs-dracut

Double check that all kernel package versions are the same:

rpm -qa | grep kernel
kernel-core-4.11.12-200.fc25.x86_64
kernel-headers-4.11.12-200.fc25.x86_64
kernel-modules-4.11.12-200.fc25.x86_64
kernel-devel-4.11.12-200.fc25.x86_64
kernel-4.11.12-200.fc25.x86_64

Verify the ZFS DKMS module built and installed for the current kernel version:

dkms status
spl, 0.7.0, 4.11.12-200.fc25.x86_64, x86_64: installed
zfs, 0.7.0, 4.11.12-200.fc25.x86_64, x86_64: installed

ZFS DKMS Module Issues

If the spl or zfs module failed to build and/or install, the most common reason is either the kernel-headers or kernel-devel packages are not installed and/or the installed version does not match version of the current running kernel.

Update the kernel to the latest version, reboot to the latest version, and ensure all kernel packages are the same. Then rebuild the spl & zfs modules using dkms:

ZVER=$(rpm -q --qf "%{VERSION}\n" zfs)
for mod in spl zfs; do
   dkms add ${mod}/${ZVER}
   dkms build ${mod}/${ZVER}
   dkms install ${mod}/${ZVER}
done

Load the ZFS modules and confirm the ZFS modules loaded:

modprobe zfs
lsmod | grep zfs
zfs                  3395584  6
zunicode              331776  1 zfs
zavl                   16384  1 zfs
icp                   253952  1 zfs
zcommon                69632  1 zfs
znvpair                77824  2 zcommon,zfs
spl                   106496  4 znvpair,zcommon,zfs,icp

Zpool and Dataset Creation#Top

Zpool Creation & Properties

In ZFS-on-Linux, it is considered best practice to always add disks to the zpool using the device ID found under /dev/disk/by-id/. This is due to the fact that, in Linux, base device names such as /dev/sdb are not guaranteed to be the same across reboots. Having the device name change would worst-case destroy data in the zpool, best-case prevent the zpool from importing.

In the following steps, replace ata-Samsung_SSD_850_EVO_250GB_S21NNXAG927110B-part1 with the device ID of the second drive.

Determine the device ID of the /dev/sdb1 device:

find -L /dev/disk/by-id/ -samefile /dev/sdb1
/dev/disk/by-id/wwn-0x5002538d405cc6aa
/dev/disk/by-id/ata-Samsung_SSD_850_EVO_250GB_S21NNXAG927110B-part1

Since the drives being used in this guide are both SSD which use 4KiB sector sizes but which lie to the operating system and claim to use 512 byte sectors, use the option ashift=12 to force ZFS to align the filesystem to 4KiB. This will result in a little better performance at the expense of slightly less efficient storage of small files.

Create a ZFS storage pool named "tank" on the secondary storage device:

zpool create -o ashift=12 tank /dev/disk/by-id/ata-Samsung_SSD_850_EVO_250GB_S21NNXAG927110B-part1
zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank   232G   468K   232G         -     0%     0%  1.00x  ONLINE  -

Dataset Creation & Properties

When deciding which datasets to create and what ZFS features to enable, there are many choices to consider. In this example, only a basic group of datasets will be created. For real-world use on a multiuser system, setting a quota on tank/home would be strongly recommended to prevent one user from taking all available space in the pool. Other recommendations are to create separate datasets with quotas for tank/var, tank/var/log, and tank/var/log/audit, to prevent application logs from taking all available space in the pool.

Two other choices, are whether or not compression and deduplication should be enabled. In testing for the data which I am storing, deduplication doesn't provide any notable space saving, while taking a lot of system memory (8 GiB in testing). Compression, on the other hand, is essentially free. On my datasets, compression does not affect performance, and as of the writing of this guide, is currently saving 15GiB of space. On a 250GB SSD, that's a nice savings.

From previous testing, compression makes no difference on my /boot/ and /home/ directories.

My /home/ directory is using ecryptfs, which effectively makes /home/ incompressible.

Create the ZFS datasets with desired options:

zfs create -o compression=lz4 -o dedup=off tank/root
zfs create -o compression=off -o dedup=off -o quota=500M tank/root/boot
zfs create -o compression=lz4 -o dedup=off tank/root/tmp
zfs create -o compression=lz4 -o dedup=off tank/root/var
zfs create -o compression=off -o dedup=off tank/home

Next, create a ZFS ZVolume for swap. For the best performance, set the volume block size the same as the system's page size; for most x86_64 systems, this will be 4KiB. The command getconf PAGESIZE can be used to verify. The sync=always and primarycache=metadata options are used to prevent swap data from being cached in memory, which would defeat the purpose of swap. ZVOLs will appear in the /dev/zvol/tankname directory.

Create a 4GiB ZFS volume and configure it to be used as swap space:

getconf PAGESIZE
4096
zfs create -V 4G -b 4096 -o logbias=throughput -o sync=always -o primarycache=metadata -o com.sun:auto-snapshot=false tank/swap
mkswap /dev/zvol/tank/swap
Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
no label, UUID=e444defd-5710-4ace-a3d8-7ae7dfdb8302

Confirm the datasets were created successfully:

zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
tank             876K   225G   112K  /tank
tank/home         96K   225G    96K  /tank/home
tank/root        392K   225G   104K  /tank/root
tank/root/boot    96K   500M    96K  /tank/root/boot
tank/root/tmp     96K   225G    96K  /tank/root/tmp
tank/root/var     96K   225G    96K  /tank/root/var
tank/swap       4.25G   118G    60K  -

At this point, the ZFS storage pool and datasets have been created. ZFS is working. The next few steps will take us through copying over the installed operating system to the ZFS datasets.

Chroot Environment Setup#Top

ZFS Mountpoints

Examine the default mountpoints of the ZFS datasets:

zfs get mountpoint
NAME            PROPERTY    VALUE            SOURCE
tank            mountpoint  /tank            default
tank/home       mountpoint  /tank/home       default
tank/root       mountpoint  /tank/root       default
tank/root/boot  mountpoint  /tank/root/boot  default
tank/root/tmp   mountpoint  /tank/root/tmp   default
tank/root/var   mountpoint  /tank/root/var   default

Update the mountpoint for the dataset tank/home to mount inside the directory /tank/root/:

zfs set mountpoint=/tank/root/home tank/home

Examine the new default mountpoints:

zfs get mountpoint
NAME            PROPERTY    VALUE            SOURCE
tank            mountpoint  /tank            default
tank/home       mountpoint  /tank/root/home  local
tank/root       mountpoint  /tank/root       default
tank/root/boot  mountpoint  /tank/root/boot  default
tank/root/tmp   mountpoint  /tank/root/tmp   default
tank/root/var   mountpoint  /tank/root/var   default

Confirm the mountpoints are mounted:

zfs mount
tank                            /tank
tank/root                       /tank/root
tank/root/boot                  /tank/root/boot
tank/root/tmp                   /tank/root/tmp
tank/root/var                   /tank/root/var
tank/home                       /tank/root/home

UDEV Modification

There is a bug in the way which GRUB2 detects ZFS filesystems, we need to fix this bug now so that the udev rule we create will get copied over to the new ZFS root dataset in the next steps.

The commands grub2-install and grub2-probe attempt to find the symbolic links which should and do exist in the directory /dev/disk/by-id/ but GRUB2 looks in only the base of the /dev/ directory. I believe this is a bug, but every time I've ever reported a bug, I get called a n00b. So screw it, now I just find workarounds for other people's bugs.

Create a udev rule to accommodate bugginess (This rule is specific to SCSI devices. If using something else, a different rule would be needed. (/lib/udev/rules.d/60-persistent-storage.rules is a good reference)):

cat /etc/udev/rules.d/61-persistent-storage-scsi-zfs.rules
# SCSI devices
KERNEL=="sd*[!0-9]|sr*", ENV{ID_SERIAL}!="?*", IMPORT{program}="scsi_id --export --whitelisted -d $devnode", ENV{ID_BUS}="scsi"
KERNEL=="cciss*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}!="?*", IMPORT{program}="scsi_id --export --whitelisted -d $devnode", ENV{ID_BUS}="cciss"
KERNEL=="sd*|sr*|cciss*", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="?*", SYMLINK+="$env{ID_BUS}-$env{ID_SERIAL}"
KERNEL=="sd*|cciss*", ENV{DEVTYPE}=="partition", ENV{ID_SERIAL}=="?*", SYMLINK+="$env{ID_BUS}-$env{ID_SERIAL}-part%n"

Reload the udev rules:

udevadm control -R
udevadm trigger /dev/sdb

Data Propagation

The next few steps will copy all the data from the / filesystem to the ZFS datasets that were created earlier. Ideally this would be done with the root filesystem mounted read-only to prevent data from changing while being copied. Unfortunately in this scenario we only have one computer, two storage devices and no rescue media.

If LVM is in-use on /dev/sda, one option would be to create snapshots of the logical volumes mount and rsync the LVM snapshots. This is left as an exercise to the reader, since the writer didn't think to do that until well after the fact.

Turn off all unnecessary services, like database servers and user programs that are likely to write to the disk while the copy is in progress. Cross your fingers, and hope for the best.

The rsync commands used below have a boat-load of options being passed. Broken down, they do the following:

-W, --whole-file
Copy files whole (w/o delta-xfer algorithm).
-S, --sparse
Handle sparse files efficiently.
-H, --hard-links
Preserve hard links.
-A, --acls
Preserve ACLs (implies -p).
-X, --xattrs
Preserve extended attributes.
-h, --human-readable
Output numbers in a human-readable format.
-a, --archive
Archive mode; equals -rlptgoD (no -H,-A,-X).
-v, --verbose
Increase verbosity.
-x, --one-file-system
Don't cross filesystem boundaries.
--progress
Show progress during transfer.
--stats
Give some file-transfer stats.

All datasets except for tank/root need to be unmounted for the initial copy of data:

echo tank/root/{var,tmp,boot,home} | xargs -n1 zfs unmount
zfs mount
tank                            /tank
tank/root                       /tank/root

Copy all data from the root filesystem to the ZFS dataset tank/root:

rsync -WSHAXhavx --progress --stats / /tank/root/

Remount all ZFS datasets and confirm they mounted:

zfs mount -a
zfs mount
tank                            /tank
tank/root                       /tank/root
tank/root/boot                  /tank/root/boot
tank/home                       /tank/root/home
tank/root/tmp                   /tank/root/tmp
tank/root/var                   /tank/root/var

Use rsync to copy the data from all other filesystems to the ZFS datasets. (Note the trailing slash on the source directory, it is very important):

rsync -WSHAXhav --progress --stats /boot/ /tank/root/boot/
rsync -WSHAXhav --progress --stats /tmp/ /tank/root/tmp/
rsync -WSHAXhav --progress --stats /var/ /tank/root/var/
rsync -WSHAXhav --progress --stats /home/ /tank/root/home/

There should now be an exact copy of the data from the operating system to the new ZFS filesystem.

Entering the Chroot Environment

Prepare a chroot environment to interactively enter the ZFS filesystem so that the the boot-loader can be installed on /dev/sdb.

Bind mount the filesystems /dev/, /sys/, /run/ and /proc/ from the running system into the chroot environment:

mount --bind /dev/ /tank/root/dev/
mount --bind /proc/ /tank/root/proc/
mount --bind /sys/ /tank/root/sys/
mount --bind /run/ /tank/root/run/

Enter the chroot:

chroot /tank/root/ /bin/bash

Edit /etc/fstab to point to the new ZFS datasets. Pay special attention to the filesystem type column; remember to update it to zfs:

cat /etc/fstab
tank/root             /         zfs   defaults                          0  0
proc                  /proc     proc  rw,nosuid,nodev,noexec,hidepid=2  0  0
tank/home             /home     zfs   defaults                          0  0
tank/root/boot        /boot     zfs   defaults                          0  0
tank/root/tmp         /tmp      zfs   defaults                          0  0
tank/root/var         /var      zfs   defaults                          0  0
/tmp                  /var/tmp  none  bind                              0  0
/dev/zvol/tank/swap   swap      swap  defaults                          0  0

Check that GRUB2 can recognize ZFS filesystem and install the GRUB2 boot-loader:

grub2-probe /
zfs
grub2-install --modules=zfs /dev/sdb
Installing for i386-pc platform.
Installation finished. No error reported.

Update the GRUB2 configuration file /etc/default/grub for the new ZFS filesystem. Note the example below, and remove 'root=ZFS=' if it appears. GRUB_PRELOAD_MODULES="zfs" must be added so that GRUB2 can detect the zpool at boot:

cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="quiet net.ifnames=0"
GRUB_DISABLE_RECOVERY="true"
GRUB_PRELOAD_MODULES="zfs"

GRUB2 Boot Errors

When booting, if errors like those seen below appear, ensure the directive GRUB_PRELOAD_MODULES="zfs" exists in the file /etc/default/grub and rerun grub2-mkconfig.

error: compression algorithm 17 not supported
error: compression algorithm 37 not supported
error: compression algorithm 79 not supported
error: unknown device 66326293.
error: compression algorithm inherit not supported
error: unsupported embedded BP (type=47)

Regenerate the GRUB2 configuration file /boot/grub2/grub.cfg:

grub2-mkconfig > /boot/grub2/grub.cfg

Enable the ZFS target and related services:

systemctl enable zfs.target zfs-import-cache zfs-import-cache zfs-import-scan zfs-share zfs-mount

Solving Cache Issues

There appears to be an issue where the directory /etc/zfs/ and the file /etc/zfs/zpool.cache do not get created at pool creation time, but they do get generated inside the Initial RAM Filesystem by Dracut. This causes the system to fail to import the pool on the first reboot. This next step ensures the zpool.cache file is correct and the first reboot will succeed.

Ensure the /etc/zfs/ directory exists, and generate a new /etc/zfs/zpool.cache file:

[ ! -d /etc/zfs ] && mkdir /etc/zfs
zpool set cachefile=/etc/zfs/zpool.cache tank

Rebuild the Initial RAM Filesystem using Dracut:

dracut --force

Exit the chroot environment and unmount all non-ZFS filesystems from the chroot:

exit
umount /tank/root/{dev,proc,sys,run}

Default Mountpoints and the Boot Filesystem#Top

Set the datasets' mountpoints to legacy to be compatible with the systemd way of mounting filesystems:

zfs set mountpoint=legacy tank/root/boot
zfs set mountpoint=legacy tank/home
zfs set mountpoint=legacy tank/root/tmp
zfs set mountpoint=legacy tank/root/var

Set the mountpoint for the tank/root dataset to /:

zfs set mountpoint=/ tank/root
cannot mount '/': directory is not empty
property may be set but unable to remount filesystem

Set the boot filesystem property on the tank/root/boot dataset:

zpool set bootfs=tank/root/boot tank

Sync the ZFS zpool to ensure all changes have been committed to disk:

zpool sync tank

Export the zpool so that it will be importable on reboot:

zpool export tank

Confirm SELinux is still on and that people who turn it off are total n00bs:

getenforce
Enforcing

Reboot the system, and boot off of the /dev/sdb device:

reboot

If any errors occurred, it is likely an above step was either skipped or typo'd. Boot the system off /dev/sda, remount the datasets, enter the chroot environment, and double check all steps were completed properly.

Congratulations! If you have followed the steps in this guide this far, you now have a bootable ZFS filesystem and ZFS datasets for all primary filesystems. The next few steps in this guide will demonstrate how to attach the /dev/sda disk to be a member of the pool as a mirror device and show the performance of a mirror setup.

The Finishing Touch — Mirroring, Compression & Performance#Top

Now that ZFS is working correctly and the system is bootable a few final steps can be taken to get the full benefits of using ZFS.

Mirrored Storage Pool

In my own setup I chose to go with adding the /dev/sda disk into the zpool as a mirror device. I chose this because using a mirror disk in ZFS gives me a few huge benefits including, but not limited to: faster read (2x) speed, self-healing, and bragging rights.

First a new partition table must be created on /dev/sda, the partition created must be the same size or larger than /dev/sdb1 partition created earlier in this guide:

fdisk -l /dev/sda
Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x84b667c6

Device     Boot Start       End   Sectors   Size Id Type
/dev/sda1        2048 468862127 468860080 223.6G bf Solaris

Earlier in this guide we installed GRUB2 on /dev/sdb, by also installing GRUB2 on /dev/sda this will allow the system to boot should either storage device fail. Install the GRUB2 boot-loader onto /dev/sda:

grub2-install --modules=zfs /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.

Identify the device ID of the new created partition:

find -L /dev/disk/by-id/ -samefile /dev/sda1
/dev/disk/by-id/wwn-0x5001517bb2a26677-part1
/dev/disk/by-id/ata-INTEL_SSDSC2CT240A3_CVMP234400PF240DGN-part1

Add /dev/sda1 into the tank by 'attaching' the disk to the pool:

zpool attach tank /dev/disk/by-id/ata-Samsung_SSD_850_EVO_250GB_S21NNXAG927110B-part1 \
  /dev/disk/by-id/ata-INTEL_SSDSC2CT240A3_CVMP234400PF240DGN-part1

Verify the disk was attached and that there is now a mirror:

zpool status tank
  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 0h4m with 0 errors on Sun Aug  6 13:10:06 2017
config:

        NAME                                                     STATE     READ WRITE CKSUM
        tank                                                     ONLINE       0     0     0
          mirror-0                                               ONLINE       0     0     0
            ata-Samsung_SSD_850_EVO_250GB_S21NNXAG927110B-part1  ONLINE       0     0     0
            ata-INTEL_SSDSC2CT240A3_CVMP234400PF240DGN-part1     ONLINE       0     0     0

errors: No known data errors

After any layout change to the zpool, the zpool.cache file will have to be regenerated and the Initial RAM Filesystem rebuilt:

zpool set cachefile=/etc/zfs/zpool.cache tank
dracut --force

Mirrored Storage Pool Performance

Create a test file called readme (I crack me up). Use the /dev/urandom device instead of /dev/zero so that ZFS compression won't mess up the test results:

dd if=/dev/urandom of=readme count=128k bs=4096

Drop filesystem caches to ensure we are not measuring RAM speed. Before mirror creation:

echo 3 > /proc/sys/vm/drop_caches
pv -rb readme > /dev/null
 512MiB [502MiB/s]

After mirror creation:

echo 3 > /proc/sys/vm/drop_caches
pv -rb readme > /dev/null
 512MiB [1015MiB/s]

As shown in the results of the previous commands, adding a second device doubled the read speeds. Write speeds remain unaffected.

Compression Performance

As noted in the summary at the top of this guide, I stated that ZFS compression is currently saving me 15GiB of space.

The command zfs get compressratio can be used to display the current compression ratios:

zfs get compressratio
NAME             PROPERTY       VALUE  SOURCE
tank             compressratio  1.16x  -
tank/home        compressratio  1.00x  -
tank/root        compressratio  1.56x  -
tank/root/boot   compressratio  1.00x  -
tank/root/tmp    compressratio  2.14x  -
tank/root/var    compressratio  1.63x  -
tank/swap        compressratio  1.00x  -

Using the command du with the --apparent-size option shows the size a file claims to be:

du -hsxc --apparent-size /*
146M    /boot
24M     /etc
68G     /home
4.1G    /root
11M     /tmp
6.6G    /usr
33G     /var

Without the --apparent-size option, du shows the space on disk the file is actually taking:

du -hsxc /*
150M    /boot
16M     /etc
70G     /home    ## 4KiB sector size costs me 2GiB here.
4.1G    /root
291K    /tmp
4.7G    /usr     ## 2GiB savings here
20G     /var     ## ZFS's lz4 compression saves me 13GiB here

Comparing the output of the two du commands above, we can see on the directories /home and /boot, on which compression is disabled, that I'm actually using more disk space than the files alone are taking up. This is due to the use of the 4KiB sector size, but as we can see on the directories /etc, /tmp, /usr, and /var, compression is saving just over 15GiB of space.

Scrubbing#Top

Scrubbing is the method by which ZFS detects and corrects data errors. A ZFS scrub checks every block in the pool and compares it to the block's checksum value. The ZFS storage pool is not scrubbed automatically, and thus, it must be done manually or be scripted. To that end, I have written a small shell script which can be dropped into /etc/cron.weekly/. This script will automatically scrub the root zpool once a week and will complain about any errors over wall and syslog.

wget -O /etc/cron.weekly/zfs-scrub-and-report.sh \
    http://matthewheadlee.com/projects/zfs-as-root-filesystem-on-fedora-25/zfs-scrub-and-report.sh
chmod 700 /etc/cron.weekly/zfs-scrub-and-report.sh
/etc/cron.weekly/zfs-scrub-and-report.sh
Scrubbing has started.
Scrub complete, no errors detected.

Troubleshooting#Top

Sometimes during boot, the system will get dropped to the Dracut Emergency Shell. This generally seems to happen if there has been a physical change to the zpool, such as the addition of a drive. From my experience, it also seems to happen if the zpool wasn't properly exported.

The solution to both of these issues is while in the Dracut Emergency Shell. Use the command zfs set mountpoint= and mount all the datasets into the /sysroot/ directory. Than chroot /sysroot, and ensure /etc/zfs/ exists. Create a new zpool.cache file with zfs set cachefile=/etc/zfs/zpool.cache, then recreate the Dracut image dracut --force, and finally reboot. Step-by-step instructions:

Forcefully import the ZFS zpool:

zpool import -f tank

Set the ZFS mountpoints to /sysroot/ and mount everything:

zfs set mountpoint=/sysroot tank/root
zfs set mountpoint=/sysroot/boot tank/root/boot
zfs set mountpoint=/sysroot/tmp tank/root/tmp
zfs set mountpoint=/sysroot/var tank/root/var
zfs mount -a

Bind mount the filesystems /dev/, /sys/, /run/ and /proc/ from the Initial RAM Filesystem into the chroot environment:

mount --bind /dev/ /sysroot/dev/
mount --bind /proc/ /sysroot/proc/
mount --bind /sys/ /sysroot/sys/
mount --bind /run/ /sysroot/run/

Enter the chroot environment:

chroot /sysroot/ /bin/bash

Ensure the /etc/zfs/ directory exists, and generate a new /etc/zfs/zpool.cache file:

[ ! -d /etc/zfs ] && mkdir /etc/zfs
zpool set cachefile=/etc/zfs/zpool.cache tank

Rebuild the Initial RAM Filesystem using Dracut:

dracut --force

Exit the chroot environment and unmount all non-ZFS filesystems from the chroot environment:

exit
umount /sysroot/{dev,proc,sys,run}

Set the ZFS tank/root mountpoint back to /:

zfs set mountpoint=/ tank/root

Set the ZFS datasets' mountpoints back to legacy:

zfs set mountpoint=legacy tank/root/boot
zfs set mountpoint=legacy tank/root/tmp
zfs set mountpoint=legacy tank/root/var

Sync the ZFS zpool to ensure all changes have been committed to disk:

zpool sync tank

Export the zpool so that it will be importable on reboot, and then reboot:

zpool export tank
reboot -f

TO DO #Top

Copyright © 2017 Matthew Headlee. This work is licensed under an Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license.

Questions or Feedback? #Top

Contact me