Miscellaneous

On an EFI system, this is the correct way to reinstall GRUB2 as opposed to using grub2-install (But, this doesn't work post Fedora:32 due to grub patches):

export ZPOOL_VDEV_NAME_PATH=YES
dnf reinstall shim-* grub2-efi-* grub2-common

As of Fedora:33 and higher, GRUB2 has been stripped of all ZFS support. Thus, to boot ZFS, a custom stand-alone grub2.efi needs to be built. The easiest way to do this is from a Fedora:32 system.

Use a Fedora:32 container to generate a stand-alone ZFS-supporting grub2-efi binary with an embedded configuration (Note the --label tank option may be different on different systems):

podman container run -v /tmp/:/mnt/ --rm -ti fedora:32
dnf install -y 'grub2*'

Complete!
cat << 'EOF' > grub.cfg
search --no-floppy --label tank --set=dev
set prefix=($dev)/root/boot@/grub2
export $prefix
configfile $prefix/grub.cfg
EOF
grub2-mkstandalone -o /mnt/matthewcfg.efi --modules="zfs part_gpt all_video increment gzio " --format=x86_64-efi /boot/grub/grub.cfg=grub.cfg
ll -h /mnt/matthewcfg.efi
-rw-r--r--. 1 root root 7.2M Oct 20 20:54 /mnt/matthewcfg.efi
exit
sudo -i

mv /tmp/matthewcfg.efi /boot/efi/EFI/fedora/matthewcfg.efi
efibootmgr -c -d /dev/nvme0n1 -p 1 -L 'Matthew with Config' -l '\EFI\fedora\matthewcfg.efi'

To forward all logs from various Podman containers to a log file /var/log/containers/container_name.log, create a new file in /etc/rsyslog.d/ and restart rsyslog:

$EDITOR /etc/rsyslog.d/podman.conf
template(name="container-format" type="list") {
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    property(name="$!CONTAINER_NAME")
    constant(value=" ")
    property(name="$!CONTAINER_ID")
    constant(value=" ")
    property(name="$!MESSAGE")
}

template(name="name-log-by-container" type="list") {
    constant(value="/var/log/containers/")
    property(name="$!CONTAINER_NAME")
    constant(value=".log")
}

if($!CONTAINER_NAME != "") then {
    action(type="omfile" dynaFile="name-log-by-container" template="container-format")
    stop
}

Install the NVIDIA driver following the RPMFusion instructions.

After installing the NVIDIA driver through RPMFusion, copy the provided Xorg configuration file into place:

cp /usr/share/X11/xorg.conf.d/nvidia.conf /etc/X11/xorg.conf.d/nvidia.conf

Modify the nvidia.conf file, adding modesetting support to the embedded Intel card and set the NVIDIA card to be the primary GPU:

$EDITOR /etc/X11/xorg.conf.d/nvidia.conf
Section "OutputClass"
	Identifier "intel"
	MatchDriver "i915"
	Driver "modesetting"
EndSection

Section "OutputClass"
	Identifier "nvidia"
	MatchDriver "nvidia-drm"
	Driver "nvidia"
	Option "AllowEmptyInitialConfiguration"
	Option "SLI" "Auto"
	Option "BaseMosaic" "on"
	Option "PrimaryGPU" "yes"
EndSection

Section "ServerLayout"
	Identifier "layout"
	Option "AllowNVIDIAGPUScreens"
EndSection

Configure the system to keep the /dev/nvidia-uvm device file when the NVIDIA devices goes into power saving mode by enabling the nvidia-persistenced service:

systemctl enable nvidia-persistenced.service

Disable runtime power management on the NVIDIA device:

cp /lib/udev/rules.d/60-nvidia.rules /etc/udev/rules.d/60-nvidia.rules

$EDITOR /etc/udev/rules.d/60-nvidia.rules
 # Enable runtime PM for NVIDIA VGA/3D controller devices on driver bind
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="auto"
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="auto"
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="on"
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="on"

Reboot or restart Xorg.

Boot the system receiving the ZFS Snapshot from the rescue disk.

On the system receiving the ZFS Snapshot, remove all previous snapshots:

zpool import -Nf tank
zfs destroy -Rv 'tank@%'

From the sender, create a new snapshot, and send it:

zfs snapshot -r 'tank@p9snap'
zfs send -v --props --replicate --large-block --compressed tank@p9snap | ssh root@receiver 'zfs receive -F tank'

On the receiver, mount the freshly received tank, and chroot into it:

zfs set mountpoint=/sysroot tank/root
zfs set mountpoint=/sysroot/boot tank/root/boot
zfs set mountpoint=/sysroot/tmp tank/root/tmp
zfs set mountpoint=/sysroot/var tank/root/var
zfs mount -aO
mount --rbind /dev/ /sysroot/dev/
mount --rbind /proc/ /sysroot/proc/
mount --rbind /sys/ /sysroot/sys/
mount --rbind /run/ /sysroot/run/
chroot /sysroot/ /bin/bash

Set the system to do an SELinux relabel:

touch /.autorelabel

Edit /etc/fstab, and fix any UUIDs:

$EDITOR /etc/fstab
UUID="3AB8-F13E"     /boot/efi  vfat  defaults                          0  0

Correct the receiver's hostname:

echo receiver.xn0.org > /etc/hostname

If the system receiving the clone is tiny.xn0.org, add the zfs configuration limiting memory usage:

cp /root/tiny-modprobe.d-zfs.conf /etc/modprobe.d/zfs.conf

If the system receiving the clone is p50.xn0.org, copy the nvidia configuration files:

cp /root/nvidia-p50-x11.conf /etc/X11/xorg.conf.d/nvidia-p50-x11.conf
cp /root/Xsetup-p50 /etc/sddm/Xsetup

Depending on system UEFI setting, the PrimaryGPU option may need to be removed.

Rebuild the initial ram-filesystem for all kernels, rebuild the grub.cfg, and disable grub's pager:

zfsver=$(rpm -q --qf "%{VERSION}\n" zfs)
while read -r kernelver; do
  for module in spl zfs; do
    dkms add -k "${kernelver}" "${module}/${zfsver}"
    dkms build -k "${kernelver}" "${module}/${zfsver}"
    dkms install -k "${kernelver}" "${module}/${zfsver}"
  done
  initfilename="/boot/initramfs-${kernelver}.img"
  dracut --force "${initfilename}" "${kernelver}"
  if [[ "$(lsinitrd "${initfilename}" | grep -c -e spl.ko -e zfs.ko)" -ne 2 ]]; then
    echo "ERROR: failed to find spl or zfs modules in the "${initfilename}" initramfs."
    exit 1
  fi
done < <(rpm -q --qf "%{VERSION}-%{RELEASE}.%{ARCH}\n" kernel)

ZPOOL_VDEV_NAME_PATH=YES grub2-mkconfig > /boot/grub2/grub.cfg

sed -i '/^set pager/ s/1/0/' /boot/grub2/grub.cfg

Exit the chroot, unmount filesystems, and export tank:

exit
umount -l /sysroot/{dev,proc,sys,run}/
zfs unmount -a
zfs set mountpoint=/ tank/root
zfs set mountpoint=legacy tank/root/boot
zfs set mountpoint=legacy tank/root/tmp
zfs set mountpoint=legacy tank/root/var
zpool export tank

Reboot the receiver:

reboot

On the receiver, remove the snapshot p9snap:

zfs destroy -vr 'tank@p9snap'

On the sender, remove the snapshot p9snap:

zfs destroy -vr 'tank@p9snap'

Go in deep:

cd /usr/lib/systemd/user/sockets.target.wants
[mmh@p9 /usr/lib/systemd/user/sockets.target.wants]$

Yikes! A 53 character long prompt.

This step applies to bash

Configure bash to display only the last two directories of the path:

export PROMPT_DIRTRIM=2
[mmh@p9 .../user/sockets.target.wants]$

This step applies to zsh

zsh is a little more complicated to configure, but a also more flexible.

Configure zsh to trim the path:

export PS1="[%n@%m %(3~|…/%2~|%2~)]%(#.#.$) "
[mmh@p9 …/user/sockets.target.wants]$

A basic file, notice the IP address 211.149.136.164 appears twice, but with two different port numbers. There is are no duplicate lines in this file:

cat sortbug
67.207.248.25:54321
Same IP211.149.136.164:8000
122.228.218.163:8090
67.207.248.125:54321
Same IP211.149.136.164:8080

Piping sorted text to uniq, sort of works (pun intended), though the first two lines aren't sorted correctly:

sort -n sortbug | uniq
67.207.248.125:54321
Wrong position67.207.248.25:54321
122.228.218.163:8090
Same IP211.149.136.164:8000
Same IP211.149.136.164:8080

One would think running sort -un sortbug would give the same output as sort -n sortbug | uniq, however:

sort -un sortbug
67.207.248.25:54321
122.228.218.163:8090
211.149.136.164:8000

What the hell just happened? Two unique lines were just removed.

The info page for sort does explain this behavior, though the man has no mention of it:

Numeric sort uses what might be considered an unconventional method to compare strings representing floating point numbers. Rather than first converting each string to the C `double' type and then comparing those values, `sort' aligns the decimal-point characters in the two strings and compares the strings a character at a time. One benefit of using this approach is its speed. In practice this is much more efficient than performing the two corresponding string-to-double (or even string-to-integer) conversions and then comparing doubles. In addition, there is no corresponding loss of precision. Converting each string to `double' before comparison would limit precision to about 16 digits on most systems.

—(coreutils.info.gz (v8.25) sort invocation

The sort command has many problems sorting numbers with multiple decimal places in them. In the above example, notice the IP addresses 67.207.248.25 and 67.207.248.125 did not get sorted in the proper order.

Use sort to correctly sort IP addresses:

sort -t . -k1,1n -k2,2n -k3,3n -k4,4n sortbug
67.207.248.25:54321
67.207.248.125:54321
122.228.218.163:8090
211.149.136.164:8000
211.149.136.164:8080

Fear not! Your Windows License Key may be in the ACPI MSDM table:

xxd /sys/firmware/acpi/tables/MSDM
0000000: 4d53 444d 5500 0000 03df 4c45 4e4f 564f  MSDMU.....LENOVO
0000010: 4342 2d30 3120 2020 0100 0000 4143 5049  CB-01   ....ACPI
0000020: 0000 0400 0100 0000 0000 0000 0100 0000  ................
The Windows License begins here.0000030: 0000 0000 1d00 0000 3348 3837 4e4e 4e4e  ........3H8NN-NN
0000040: 4e4e 4e4e 4e4e 4e4e 4e4e 4e4e 4e4e 4e4e  NNN-NNNNN-NNNNN-
0000050: 4e4e 4243 44                             NNBCD

Often such files are found under the /proc/ and /sys/ directories. An example here:

ls /sys/module/iwlwifi/parameters/
11n_disable  antenna_coupling  d0i3_disable  debug       fw_restart   led_mode  power_level  swcrypto
amsdu_size   bt_coex_active    d0i3_timeout  fw_monitor  lar_disable  nvm_file  power_save   uapsd_disable

By using grep to search for null recursively through a directory, grep will print each file's name, and that file's content:

There is nothing inside the single-quotes.grep -R '' /sys/module/iwlwifi/parameters/
/sys/module/iwlwifi/parameters/nvm_file:(null)
/sys/module/iwlwifi/parameters/debug:0
/sys/module/iwlwifi/parameters/swcrypto:0
/sys/module/iwlwifi/parameters/power_save:N
/sys/module/iwlwifi/parameters/lar_disable:N
/sys/module/iwlwifi/parameters/d0i3_disable:Y
/sys/module/iwlwifi/parameters/led_mode:0
/sys/module/iwlwifi/parameters/fw_restart:Y
/sys/module/iwlwifi/parameters/bt_coex_active:N
/sys/module/iwlwifi/parameters/d0i3_timeout:1000
/sys/module/iwlwifi/parameters/11n_disable:0
/sys/module/iwlwifi/parameters/uapsd_disable:Y
/sys/module/iwlwifi/parameters/amsdu_size:0
/sys/module/iwlwifi/parameters/antenna_coupling:0
/sys/module/iwlwifi/parameters/fw_monitor:N
/sys/module/iwlwifi/parameters/power_level:0

And, finally. If there is a desire to get needlessly profligate, there's always the column command:

grep -R '' /sys/module/iwlwifi/parameters/ | column -s : -t
/sys/module/iwlwifi/parameters/disable_11ac      N
/sys/module/iwlwifi/parameters/power_save        N
/sys/module/iwlwifi/parameters/swcrypto          0
/sys/module/iwlwifi/parameters/power_level       0
/sys/module/iwlwifi/parameters/amsdu_size        0
/sys/module/iwlwifi/parameters/uapsd_disable     3
/sys/module/iwlwifi/parameters/d0i3_disable      Y
/sys/module/iwlwifi/parameters/lar_disable       N
/sys/module/iwlwifi/parameters/d0i3_timeout      1000
/sys/module/iwlwifi/parameters/11n_disable       0
/sys/module/iwlwifi/parameters/fw_restart        Y
/sys/module/iwlwifi/parameters/led_mode          0
/sys/module/iwlwifi/parameters/debug             0
/sys/module/iwlwifi/parameters/antenna_coupling  0
/sys/module/iwlwifi/parameters/bt_coex_active    Y
/sys/module/iwlwifi/parameters/fw_monitor        N
/sys/module/iwlwifi/parameters/nvm_file          (null)

Add the following functions and keybindings $HOME/.zshrc:

$EDITOR $HOME/.zshrc
setopt AUTO_PUSHD  ##AUTO_PUSHD is needed for alt+left to function intuitively.
function cdBack() {
  local sBackWD=~1
  print -P "\r\e[?7\e[0K${PS1}cd ${sBackWD/\~1/.}"
  unset sBackWD
  popd &> /dev/null
  zle reset-prompt
}
function cdUp() {
  print -P "\r\e[?7\e[0K${PS1}cd $(realpath ..)"
  pushd .. &> /dev/null
  zle reset-prompt
}
zle -N cdUp
zle -N cdBack
bindkey '^[[1;3A' cdUp   # Alt+Up-Arrow
bindkey '^[[1;3D' cdBack # Alt+Left-Arrow

Now CLI directory navigation is identical to GUI navigation:

pwd
/home/mmh/Desktop/
[mmh@host /home/mmh/Desktop/]$
press alt+up
[mmh@host /home/mmh/Desktop/]$ cd /home/mmh
press alt+up
[mmh@host /home/mmh/]$ cd /home
press alt+up
[mmh@host /home/]$ cd /
press alt+left
[mmh@host /]$ cd /home
press alt+left
[mmh@host /home/]$ cd /home/mmh
press alt+left
[mmh@host /home/mmh/]$ cd /home/mmh/Desktop
[mmh@host /home/mmh/Desktop/]$

One benefit to using zle reset-prompt is it will not foobar any already typed commands on the prompt:

[mmh@host /home/mmh/Desktop/]$
press alt+up
[mmh@host /home/mmh/]touch testfile press alt+left
[mmh@host /home/mmh/]$ cd /home/mmh/Desktop
[mmh@host /home/mmh/Desktop/]$ touch testfile

In the default bash or zsh configuration aliases do not expand past the first word. Generally desired behavior, however highly annoying when using sudo:

which ll
ll='ls -l --color=auto'
        /usr/bin/ls
sudo ll /root/
sudo: ll: command not found

There's a trick to get around this limitation though. By assigning sudo to an alias of itself with a trailing space, both the first and second words will be expanded:

The trailing space is critical.alias sudo='sudo '
sudo ll /root/
total 222K
drwx------.  6 root root    8 2018-Jan-15 04:23:09 PM .cache
drwx------. 10 root root   13 2018-Apr-14 01:49:29 PM .config
drwx------.  3 root root    3 2017-Feb-18 03:39:55 PM .dbus
drwxr-xr-x.  3 root root    3 2017-Feb-24 11:11:08 PM .local

Attach the backup disk to the virtual machine:

tank-backup-avirsh attach-disk kvm_c8_pod /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJPLP6X vde --serial 68UW8N0_VKJPLP6X --sourcetype block --cache none
tank-backup-bvirsh attach-disk kvm_c8_pod /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJNK77X vdf --serial 68UW8N0_VKJNK77X --sourcetype block --cache none

Import the backup tank and make a current set of snapshots:

zpool import tank-backup-a
zpool import tank-backup-b
sCurrSnap="tank@backup-time-$(date '+%Y%m%d')"
tank-backup-asPrevSnapA="$(zfs list -t snapshot -S name -o name -d 1 -H tank-backup-a | grep -om1 'backup-time-.*')"
tank-backup-bsPrevSnapB="$(zfs list -t snapshot -S name -o name -d 1 -H tank-backup-b | grep -om1 'backup-time-.*')"
zfs snapshot -r "${sCurrSnap}"

Transfer the snapshots to the backup disk:

tank-backup-azfs send --replicate --large-block --compressed --props -I "tank@${sPrevSnapA}" "${sCurrSnap}" | pv | zfs receive -F tank-backup-a

tank-backup-bzfs send --replicate --large-block --compressed --props -I "tank@${sPrevSnapB}" "${sCurrSnap}" | pv | zfs receive -F tank-backup-b

Export the backup tank:

zpool export tank-backup-a
zpool export tank-backup-b

Remove the backup disk from the virtual machine, power down the disk, and remove it from the kernel:

tank-backup-avirsh detach-disk kvm_c8_pod /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJPLP6X
tank-backup-ahdparm -Y /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJPLP6X
tank-backup-aecho 1 > /sys/block/$(basename $(realpath /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJPLP6X))/device/delete

tank-backup-bvirsh detach-disk kvm_c8_pod /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJNK77X
tank-backup-bhdparm -Y /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJNK77X
tank-backup-becho 1 > /sys/block/$(basename $(realpath /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJNK77X))/device/delete

The smartd service automatically detects the new disks and beings monitoring them. Stupidly, the smartd service flips-the-fuck-out when the drive is removed. Restart the smartd service:

systemctl restart smartd

Remove the backup disk.

Download buildhosts-unifi.sh and make it executable:

curl 'http://matthewheadlee.com/projects/miscellaneous/buildhosts-unifi.sh' > /usr/local/sbin/buildhosts-unifi.sh
chmod 700 /usr/local/sbin/buildhosts-unifi.sh

Run buildhosts-unifi.sh once to get the initial hosts file in-place:

buildhosts-unifi.sh

Create a new cron job to run buildhosts-unifi.sh once a week:

vi /etc/cron.d/buildhosts
# Run buildhosts every Thursday at 4:44AM
44 4 * * 4 root /usr/local/sbin/buildhosts-unifi.sh

Configure dnsmasq to use the DNS servers in listed order:

echo strict-order > /etc/dnsmasq.d/strict-order.conf
service dnsmasq restart

The command fc-match can be used to determine how the system will match a font name:

fc-match -a Monospace
DejaVuSansMono.ttf: "DejaVu Sans Mono" "Book"

Configure fontconfig to pick Noto Mono when applications request the monospace font:

$EDITOR ~/.config/fontconfig/fonts.conf
<?xml version='1.0'?>
<!DOCTYPE fontconfig SYSTEM 'fonts.dtd'>
<fontconfig>
 <dir>~/.fonts</dir>
 <alias>
  <family>monospace</family>
  <prefer>
    <family>Noto Mono</family>
  </prefer>
 </alias>
 <alias>
  <family>monospace</family>
  <prefer>
    <family>DejaVu Sans Mono</family>
  </prefer>
 </alias>
</fontconfig>

Confirm the configuration change was effective:

fc-match -a Monospace
NotoMono-Regular.ttf: "Noto Mono" "Regular"
DejaVuSansMono.ttf: "DejaVu Sans Mono" "Book"