Miscellaneous

Overview

These are miscellaneous projects and how-to guides which I find useful and hope that others will too.

Table of Contents

  1. Clone a system using ZFS
  2. Trimming Long Directory Paths in Prompt
  3. GNU sort's non-intuitive behavior
  4. Retrieve the Windows License Key from ACPI
  5. View Content of Many Small Files
  6. Browser-like Navigation in Z shell
  7. Shell Expansion of Aliases with sudo
  8. Backup of smb1 ZFS datasets
  9. buildhosts for Ubiquiti USG
  10. Ubiquiti USG use DNS in Order
  11. Override Default Font Selection
  12. buildhosts for DD-WRT
  13. Supercharger Parser v2
  14. Copyright
  15. Questions or Feedback?

Clone a system using ZFS πŸ“… 2019-09-01

Instructions to clone a system using zfs send and zfs recv

Boot the system receiving the ZFS Snapshot from the rescue disk.

On the system receiving the ZFS Snapshot, remove all previous snapshots:

zpool import -f tank
zfs destroy -Rv 'tank@%'

From the sender, create a new snapshot, and send it:

zfs snapshot -r 'tank@p9snap'
zfs send -v --props --replicate --large-block --compressed tank@p9snap | ssh root@receiver 'zfs receive -F tank'

On the receiver, mount the freshly received tank, and chroot into it:

zfs set mountpoint=/sysroot tank/root
zfs set mountpoint=/sysroot/boot tank/root/boot
zfs set mountpoint=/sysroot/tmp tank/root/tmp
zfs set mountpoint=/sysroot/var tank/root/var
zfs mount -a
mount --rbind /dev/ /sysroot/dev/
mount --rbind /proc/ /sysroot/proc/
mount --rbind /sys/ /sysroot/sys/
mount --rbind /run/ /sysroot/run/
chroot /sysroot/ /bin/bash

Set the system to do an SELinux relabel:

touch /.autorelabel

Edit /etc/fstab, and fix any UUIDs:

$EDITOR /etc/fstab
UUID="3AB8-F13E"     /boot/efi  vfat  defaults                          0  0

Correct the receiver's hostname:

echo receiver.xn0.org > /etc/hostname

If the system receiving the clone is tiny.xn0.org, add the zfs configuration limiting memory usage:

cp /root/tiny-modprobe.d-zfs.conf /etc/modprobe.d/zfs.conf

Rebuild the initial ram-filesystem, rebuild the grub.cfg, and disable grub's pager:

dracut --force /boot/initramfs-5.2.9-200.fc30.x86_64.img 5.2.9-200.fc30.x86_64

ZPOOL_VDEV_NAME_PATH=YES grub2-mkconfig > /boot/grub2/grub.cfg

sed -i '/^set pager/ s/1/0/' /boot/grub2/grub.cfg

Exit the chroot, unmount filesystems, and export tank:

exit
umount -l /sysroot/{dev,proc,sys,run}/
zfs set mountpoint=/ tank/root
zfs set mountpoint=legacy tank/root/boot
zfs set mountpoint=legacy tank/root/tmp
zfs set mountpoint=legacy tank/root/var
zpool export tank

Reboot the receiver:

reboot -f

On the receiver, remove the snapshot p9snap:

zfs destroy -vr 'tank@p9snap'

On the sender, remove the snapshot p9snap:

zfs destroy -vr 'tank@p9snap'

Trimming Long Directory Paths in Prompt πŸ“… 2017-01-18

A modification I like to make to my shell is to change the prompt to include the complete working directory path, and most of the time this isn't a problem. However, every now-and-then I find myself deep in the file system and the prompt starts taking up a large portion of my screen.

Go in deep:

cd /usr/lib/systemd/user/sockets.target.wants
[mmh@p9 /usr/lib/systemd/user/sockets.target.wants]$

Yikes! A 53 character long prompt.

Both bash and zsh provide solutions to shorten the prompt based on path depth.

This step applies to bash

Configure bash to display only the last two directories of the path:

export PROMPT_DIRTRIM=2
[mmh@p9 .../user/sockets.target.wants]$

This step applies to zsh

zsh is a little more complicated to configure, but a also more flexible.

Configure zsh to trim the path:

export PS1="[%n@%m %(3~|…/%2~|%2~)]%(#.#.$) "
[mmh@p9 …/user/sockets.target.wants]$

GNU sort's non-intuitive behavior πŸ“… 2013-03-12

When the sort options -n and -u are used together, unexpected side effects can occur. Notably, sort only considers uniqueness up to the first decimal place.

From the sort(1) manual page:

-n, --numeric-sort
compare according to string numerical value
-u, --unique
with -c, check for strict ordering; without -c, output only the first of an equal run

A basic file, notice the IP address 211.149.136.164 appears twice, but with two different port numbers. There is are no duplicate lines in this file:

cat sortbug
67.207.248.25:54321
Same IP211.149.136.164:8000
122.228.218.163:8090
67.207.248.125:54321
Same IP211.149.136.164:8080

Piping sorted text to uniq, sort of works (pun intended), though the first two lines aren't sorted correctly:

sort -n sortbug | uniq
67.207.248.125:54321
Wrong position67.207.248.25:54321
122.228.218.163:8090
Same IP211.149.136.164:8000
Same IP211.149.136.164:8080

One would think running sort -un sortbug would give the same output as sort -n sortbug | uniq, however:

sort -un sortbug
67.207.248.25:54321
122.228.218.163:8090
211.149.136.164:8000

What the hell just happened? Two unique lines were just removed.

The info page for sort does explain this behavior, though the man has no mention of it:

Numeric sort uses what might be considered an unconventional method to compare strings representing floating point numbers. Rather than first converting each string to the C `double' type and then comparing those values, `sort' aligns the decimal-point characters in the two strings and compares the strings a character at a time. One benefit of using this approach is its speed. In practice this is much more efficient than performing the two corresponding string-to-double (or even string-to-integer) conversions and then comparing doubles. In addition, there is no corresponding loss of precision. Converting each string to `double' before comparison would limit precision to about 16 digits on most systems.

β€”(coreutils.info.gz (v8.25) sort invocation

The sort command has many problems sorting numbers with multiple decimal places in them. In the above example, notice the IP addresses 67.207.248.25 and 67.207.248.125 did not get sorted in the proper order.

Use sort to correctly sort IP addresses:

sort -t . -k1,1n -k2,2n -k3,3n -k4,4n sortbug
67.207.248.25:54321
67.207.248.125:54321
122.228.218.163:8090
211.149.136.164:8000
211.149.136.164:8080

Retrieve the Windows License Key from ACPI πŸ“… 2015-08-13

Did you wipe your laptop, and install a non-approved operating system?

Did you pull all the stickers off your laptop, including the Windows license key?

Did you realize the mistake you've made by not running a Microsoft product?

Are you crying because you need your Windows key to get out of this depression?

Fear not! Your Windows License Key may be in the ACPI MSDM table:

xxd /sys/firmware/acpi/tables/MSDM
0000000: 4d53 444d 5500 0000 03df 4c45 4e4f 564f  MSDMU.....LENOVO
0000010: 4342 2d30 3120 2020 0100 0000 4143 5049  CB-01   ....ACPI
0000020: 0000 0400 0100 0000 0000 0000 0100 0000  ................
0000030: 0000 0000 1d00 0000 3348 3837 4e4e 4e4e  ........3H8NN-NN
0000040: 4e4e 4e4e 4e4e 4e4e 4e4e 4e4e 4e4e 4e4e  NNN-NNNNN-NNNNN-
0000050: 4e4e 4243 44                             NNBCD

Of course to any reasonable person, all the above questions are preposterous, and no one in their right mind would ever run Windows β€” Microsoft's spyware.

View Content of Many Small Files πŸ“… 2016-06-29

Assume there is a directory with a dozen files, each only a byte or so in size. How do you view the content with associated filename? A for-loop would work, but that's a lot of typing for very little benefit. The commands xargs and cat could do it, but that's just ridiculous.

Enter the magic of grep!

Often such files are found under the /proc/ and /sys/ directories. An example here:

ls /sys/module/iwlwifi/parameters/
11n_disable  antenna_coupling  d0i3_disable  debug       fw_restart   led_mode  power_level  swcrypto
amsdu_size   bt_coex_active    d0i3_timeout  fw_monitor  lar_disable  nvm_file  power_save   uapsd_disable

By using grep to search for null recursively through a directory, grep will print each file's name, and that file's content:

grep -R '' /sys/module/iwlwifi/parameters/
/sys/module/iwlwifi/parameters/nvm_file:(null)
/sys/module/iwlwifi/parameters/debug:0
/sys/module/iwlwifi/parameters/swcrypto:0
/sys/module/iwlwifi/parameters/power_save:N
/sys/module/iwlwifi/parameters/lar_disable:N
/sys/module/iwlwifi/parameters/d0i3_disable:Y
/sys/module/iwlwifi/parameters/led_mode:0
/sys/module/iwlwifi/parameters/fw_restart:Y
/sys/module/iwlwifi/parameters/bt_coex_active:N
/sys/module/iwlwifi/parameters/d0i3_timeout:1000
/sys/module/iwlwifi/parameters/11n_disable:0
/sys/module/iwlwifi/parameters/uapsd_disable:Y
/sys/module/iwlwifi/parameters/amsdu_size:0
/sys/module/iwlwifi/parameters/antenna_coupling:0
/sys/module/iwlwifi/parameters/fw_monitor:N
/sys/module/iwlwifi/parameters/power_level:0

And, finally. If there is a desire to get needlessly profligate, there's always the column command:

grep -R '' /sys/module/iwlwifi/parameters/ | column -s : -t
/sys/module/iwlwifi/parameters/disable_11ac      N
/sys/module/iwlwifi/parameters/power_save        N
/sys/module/iwlwifi/parameters/swcrypto          0
/sys/module/iwlwifi/parameters/power_level       0
/sys/module/iwlwifi/parameters/amsdu_size        0
/sys/module/iwlwifi/parameters/uapsd_disable     3
/sys/module/iwlwifi/parameters/d0i3_disable      Y
/sys/module/iwlwifi/parameters/lar_disable       N
/sys/module/iwlwifi/parameters/d0i3_timeout      1000
/sys/module/iwlwifi/parameters/11n_disable       0
/sys/module/iwlwifi/parameters/fw_restart        Y
/sys/module/iwlwifi/parameters/led_mode          0
/sys/module/iwlwifi/parameters/debug             0
/sys/module/iwlwifi/parameters/antenna_coupling  0
/sys/module/iwlwifi/parameters/bt_coex_active    Y
/sys/module/iwlwifi/parameters/fw_monitor        N
/sys/module/iwlwifi/parameters/nvm_file          (null)

Browser-like Navigation in Z shell πŸ“… 2017-07-25

In most web and file browsers the key combinations alt+left (back) and alt+up (parent directory) can be used for navigation. So why not in the shell? These following steps are specific for zsh, however it is possible to do the same in bash.

Add the following functions and keybindings $HOME/.zshrc:

$HOME/.zshrc
setopt AUTO_PUSHD  ##AUTO_PUSHD is needed for alt+left to function intuitively.
function cdBack() {
  local sBackWD=~1
  print -P "\r\e[?7\e[0K${PS1}cd ${sBackWD/\~1/.}"
  unset sBackWD
  popd &> /dev/null
  zle reset-prompt
}
function cdUp() {
  print -P "\r\e[?7\e[0K${PS1}cd $(realpath ..)"
  pushd .. &> /dev/null
  zle reset-prompt
}
zle -N cdUp
zle -N cdBack
bindkey '^[[1;3A' cdUp   # Alt+Up-Arrow
bindkey '^[[1;3D' cdBack # Alt+Left-Arrow

Now CLI directory navigation is identical to GUI navigation:

pwd
/home/mmh/Desktop/
[mmh@host /home/mmh/Desktop/]$
press alt+up
[mmh@host /home/mmh/Desktop/]$ cd /home/mmh
press alt+up
[mmh@host /home/mmh/]$ cd /home
press alt+up
[mmh@host /home/]$ cd /
press alt+left
[mmh@host /]$ cd /home
press alt+left
[mmh@host /home/]$ cd /home/mmh
press alt+left
[mmh@host /home/mmh/]$ cd /home/mmh/Desktop
[mmh@host /home/mmh/Desktop/]$

One benefit to using zle reset-prompt is it will not foobar any already typed commands on the prompt:

[mmh@host /home/mmh/Desktop/]$
press alt+up
[mmh@host /home/mmh/]touch testfile press alt+left
[mmh@host /home/mmh/]$ cd /home/mmh/Desktop
[mmh@host /home/mmh/Desktop/]$ touch testfile

Shell Expansion of Aliases with sudo πŸ“… 2018-07-14

In the default bash or zsh configuration aliases do not expand past the first word. Generally desired behavior, however highly annoying when using sudo:

which ll
ll='ls -l --color=auto'
        /usr/bin/ls
sudo ll /root/
sudo: ll: command not found

There's a trick to get around this limitation though. By assigning sudo to and alias of itself with a trailing space, both the first and second words will be expanded:

The trailing space is critical.alias sudo='sudo '
sudo ll /root/
total 222K
drwx------.  6 root root    8 2018-Jan-15 04:23:09 PM .cache
drwx------. 10 root root   13 2018-Apr-14 01:49:29 PM .config
drwx------.  3 root root    3 2017-Feb-18 03:39:55 PM .dbus
drwxr-xr-x.  3 root root    3 2017-Feb-24 11:11:08 PM .local

Backup of smb1 ZFS datasets πŸ“… 2019-01-27

These are the steps to do a recursive, incremental backup of the data stored on smb1/tank to the back up drives tank-backup-a and tank-backup-b.

The following commands are to be run on ssds9

Attach the backup disk to the virtual machine:

tank-backup-avirsh attach-disk kvm_smb1 /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJPLP6X vde --serial 68UW8N0_VKJPLP6X --sourcetype block --cache none
tank-backup-bvirsh attach-disk kvm_smb1 /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJNK77X vdf --serial 68UW8N0_VKJNK77X --sourcetype block --cache none

The following commands are to be run on smb1

Import the backup tank and make a current set of snapshots:

zpool import tank-backup-a
zpool import tank-backup-b
sCurrSnap="tank@backup-time-$(date '+%Y%m%d')"
tank-backup-asPrevSnapA="$(zfs list -t snapshot -S name -o name -d 1 -H | grep -m1 'backup-a@backup-time')"
tank-backup-bsPrevSnapB="$(zfs list -t snapshot -S name -o name -d 1 -H | grep -m1 'backup-b@backup-time')"
zfs snapshot -r "${sCurrSnap}"

Transfer the snapshots to the backup disk:

tank-backup-azfs send --replicate --large-block --compressed --props -I "${sPrevSnapA}" "${sCurrSnap}" | pv | zfs receive -F tank-backup-a

tank-backup-bzfs send --replicate --large-block --compressed --props -I "${sPrevSnapB}" "${sCurrSnap}" | pv | zfs receive -F tank-backup-b

Export the backup tank:

zpool export tank-backup-a
zpool export tank-backup-b

The following commands are to be run on ssds9

Remove the backup disk from the virtual machine, power down the disk, and remove it from the kernel:

virsh detach-disk kvm_smb1 /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJPLP6X
tank-backup-ahdparm -Y /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJPLP6X
tank-backup-aecho 1 > /sys/block/$(basename $(realpath /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJPLP6X))/device/delete
tank-backup-bhdparm -Y /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJNK77X
tank-backup-becho 1 > /sys/block/$(basename $(realpath /dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKJNK77X))/device/delete

The smartd service automatically detects the new disks and beings monitoring them. Stupidly, the smartd service flips-the-fuck-out when the drive is removed. Restart the smartd service:

systemctl restart smartd

Remove the backup disk.

Arguments Details

The zfs receive above uses the -F which does:

-F
Force a rollback of the file system to the most recent snapshot before performing the receive operation. If receiving an incremental replication stream (for example, one generated by zfs send -R [-i|-I]), destroy snapshots and file systems that do not exist on the sending side.

The zfs send command above has many options being passed, a brief overview of what they're accomplishing:

-v, --verbose
Print verbose information about the stream package generated. This information includes a per-second report of how much data has been sent.
-R, --replicate
Generate a replication stream package, which will replicate the specified file system, and all descendant file systems, up to the named snap‐ shot. When received, all properties, snapshots, descendant file systems, and clones are preserved. If the -i or -I flags are used in conjunction with the -R flag, an incremental replication stream is generated. The current values of proper‐ ties, and current snapshot and file system names are set when the stream is received. If the -F flag is specified when this stream is received, snapshots and file systems that do not exist on the sending side are destroyed.
-L, --large-block
Generate a stream which may contain blocks larger than 128KB. This flag has no effect if the large_blocks pool feature is disabled, or if the recordsize property of this filesystem has never been set above 128KB. The receiving system must have the large_blocks pool feature enabled as well. See zpool-features(5) for details on ZFS feature flags and the large_blocks feature.
-c, --compressed
Generate a more compact stream by using compressed WRITE records for blocks which are compressed on disk and in memory (see the compression property for details). If the lz4_compress feature is active on the sending system, then the receiving system must have that feature enabled as well. If the large_blocks feature is enabled on the sending system but the -L option is not supplied in conjunction with -c, then the data will be decompressed before sending so it can be split into smaller block sizes.
-p, --props
Include the dataset's properties in the stream. This flag is implicit when -R is specified. The receiving system must also support this feature.
-I snapshot
Generate a stream package that sends all intermediary snapshots from the first snapshot to the second snapshot. For example, -I @a fs@d is similar to -i @a fs@b; -i @b fs@c; -i @c fs@d. The incremental source may be specified as with the -i option.

buildhosts for Ubiquiti USG πŸ“… 2017-08-12

This script is written for the Ubiquiti USG.

This script downloads various anti-ad hosts files then merges them together, sorts in alphabetical order, and removes all duplicate entries.

Current host providers for this script are listed below, more can be added on request:

  1. http://winhelp2002.mvps.org/hosts.txt
  2. https://adaway.org/hosts.txt
  3. http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
  4. http://hosts-file.net/ad_servers.txt

A previous version of this script used to save all host entries to the file /etc/hosts, however this resulted in problems with the Ubiquiti USG due to the way it updates the file anytime a DHCP address was leased or renewed.

Now this script works by automatically configuring dnsmasq through the configuration file /etc/dnsmasq.d/buildhosts-blacklist.conf to read an additional hosts file: /etc/hosts-blacklist, where all blocked host entries are stored to.

Installation

The following commands are to be run on as root on the Ubiquiti USG

Download buildhosts-unifi.sh and make it executable:

curl 'http://matthewheadlee.com/projects/miscellaneous/buildhosts-unifi.sh' > /usr/local/sbin/buildhosts-unifi.sh
chmod 700 /usr/local/sbin/buildhosts-unifi.sh

Run buildhosts-unifi.sh once to get the initial hosts file in-place:

buildhosts-unifi.sh

Create a new cron job to run buildhosts-unifi.sh once a week:

vi /etc/cron.d/buildhosts
# Run buildhosts every Thursday at 4:44AM
44 4 * * 4 root /usr/local/sbin/buildhosts-unifi.sh

Force Ubiquiti USG to use DNS servers in listed order πŸ“… 2017-08-12

The dnsmasq service running on an Ubiquiti USG uses the name servers listed in /etc/resolv.conf in a round-robin fashion. When using split-views on an internal name server, causes the USG to sometimes resolve the internal address and other times the external address.

Configure dnsmasq to use the DNS servers in listed order:

echo strict-order > /etc/dnsmasq.d/strict-order.conf
service dnsmasq restart

Override Default Font Selection for Generic Font Names on Fedora πŸ“… 2019-01-08

The default font selected by GUI applications which utilize the fontconfig library can be changed by editing the ~/.config/fontconfig/fonts.conf configuration file. New "alias" elements can be added to this configuration file which will prepend to the system's default list.

The command fc-match can be used to determine how the system will match a font name:

fc-match -a Monospace
DejaVuSansMono.ttf: "DejaVu Sans Mono" "Book"

Configure fontconfig to pick Noto Mono when applications request the monospace font:

$EDITOR ~/.config/fontconfig/fonts.conf
<?xml version='1.0'?>
<!DOCTYPE fontconfig SYSTEM 'fonts.dtd'>
<fontconfig>
 <dir>~/.fonts</dir>
 <alias>
  <family>monospace</family>
  <prefer>
    <family>Noto Mono</family>
  </prefer>
 </alias>
 <alias>
  <family>monospace</family>
  <prefer>
    <family>DejaVu Sans Mono</family>
  </prefer>
 </alias>
</fontconfig>

Confirm the configuration change was effective:

fc-match -a Monospace
NotoMono-Regular.ttf: "Noto Mono" "Regular"
DejaVuSansMono.ttf: "DejaVu Sans Mono" "Book"

buildhosts for DD-WRT πŸ“… 2014-02-05

This script was written for routers running DD-WRT. This script gets various anti-ad hosts files, merges, sorts, and uniques, then installs.

Download

Supercharger Parser v2 πŸ“… 2014-08-01

This script grabs the locations of all Tesla chargers, including Superchargers and private chargers from teslamotors.com

Download

Questions or Feedback? #Top

Please feel free to reach out to me with any questions or feedback on the content presented here. My email, GPG key, and other contact methods can be found at the top of the page on my primary domain: http://www.matthewheadlee.com/