DRAFT: Pop!_OS 21.10: installation guide with btrfs-LVM-luks and auto snapshots with BTRBK
Please feel free to raise any comments or issues on the website’s Github repository. Pull requests are very much appreciated.
Table of Contents
Overview
I am exclusively using btrfs as my filesystem on all my Linux systems, see Why I (still) like btrfs. So, in this guide I will show how to install Pop!_OS 21.10 with the following structure:
- an unencrypted EFI partition for the systemd bootloader
- an unencrypted partition for the Pop!_OS recovery system
- an encrypted swap partition which works with hibernation
- a btrfs-LVM-inside-luks partition for the root filesystem
- the btrfs logical volume contains a subvolume
@for/and a subvolume@homefor/home. Note that the Pop!_OS installer does not create any subvolumes on btrfs, so we need to do this manually.
- the btrfs logical volume contains a subvolume
- automatic system snapshots and easy rollback similar to zsys using BTRBK which will regularly take (almost instant) snapshots of the system and (optionally) also on any apt operation
This setup works similarly well on other distributions, for which I also have installation guides with optional RAID1.
Step 0: General remarks
This tutorial is made with Pop!_OS 21.10 from https://system76.com/pop copied to an installation media (usually a USB Flash device but may be a DVD or the ISO file attached to a virtual machine hypervisor). Other versions of Pop!_OS and other distributions that use Systemd boot manager might also work, but sometimes require additional steps (see my other installation guides).
I strongly advise to try the following installation steps in a virtual machine first before doing anything like that on real hardware! For instance, you can spin up a virtual machine with 4 cores, 8 GB RAM, and a 64GB disk using e.g. the awesome quickemu project.
In the following, however, I outline the steps for my Dell Precision 7520 with a NVME drive which I use for the system files and another SSD which is used for btrfs backups.
Step 1a: Boot the install and perform a Clean Install with encryption
In previous installation guides I prepared the partitions manually; however, as I am basically using the same layout as the automatic POP!_OS installer with the only difference that I want to use btrfs instead of EXT4 as the filesystem, I simply perform the installation twice. The first one is the automatic Clean Install with encryption. When this finishes, do NOT Restart Device or Shut Down but instead right-click in the dock on the Install Pop!_OS app and select Quit.
If you want to see the structure of the installation keep reading, otherwise go to the next step to perform the second Installation.
Step 1b (optional): Understand the partition layout and installation structure
So, let’s open a terminal and have a look on the default partition layout:
sudo lsblk
# NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
# loop0 7:0 0 2.8G 1 loop /rofs
# sda 8:0 0 465.8G 0 disk
# └─sda1 8:1 0 465.8G 0 part
# sdb 8:16 1 3.7G 0 disk
# ├─sdb1 8:17 1 2.9G 0 part /cdrom
# ├─sdb2 8:18 1 4M 0 part
# └─sdb3 8:19 1 843.5M 0 part /var/crash
# nvme0n1 259:0 0 476.9G 0 disk
# ├─nvme0n1p1 259:1 0 498M 0 part
# ├─nvme0n1p2 259:2 0 4G 0 part
# └─nvme0n1p3 259:3 0 468.4G 0 part
# ├─nvme0n1p4 259:4 0 4G 0 part
sudo parted /dev/nvme0n1 unit MiB print
# Model: PM961 NVMe SAMSUNG 512GB (nvme)
# Disk /dev/nvme0n1: 488386MiB
# Sector size (logical/physical): 512B/512B
# Partition Table: gpt
# Disk Flags:
#
# Number Start End Size File system Name Flags
# 1 2.00MiB 500MiB 498MiB fat32 EFI boot, esp
# 2 500MiB 4596MiB 4096MiB fat32 recovery msftdata
# 3 4596MiB 484288MiB 479692MiB
# 4 484288MiB 488384MiB 4096MiB linux-swap(v1) swap
In my case sda is an internal SSD that I use for my backup strategy, whereas sdb is the flash drive that contains the installation files. So, for me the installation target device is called nvme0n1 and it has 4 partitions:
- a 498 MiB FAT32 EFI partition for the systemd bootloader
- a 4096 MiB FAT32 partition for the Pop!_OS recovery system
- a 479692MiB partition that contains the luks2 encrypted system files
- a 4096 MiB swap partition for (encrypted) swap use
Let’s have a closer look at the luks2-encrypted partition:
sudo cryptsetup luksDump /dev/nvme0n1p3
# LUKS header information
# Version: 2
# Epoch: 3
# Metadata area: 16384 [bytes]
# Keyslots area: 16744448 [bytes]
# UUID: e7e986dd-19b7-4535-98bf-8ba1760921f1
# Label: (no label)
# Subsystem: (no subsystem)
# Flags: (no flags)
#
# Data segments:
# 0: crypt
# offset: 16777216 [bytes]
# length: (whole device)
# cipher: aes-xts-plain64
# sector: 512 [bytes]
#
# Keyslots:
# 0: luks2
# Key: 512 bits
# Priority: normal
# Cipher: aes-xts-plain64
# Cipher key: 512 bits
# PBKDF: argon2i
# Time cost: 7
# Memory: 1048576
# Threads: 4
# Salt: 53 34 4d ce b8 9d 9b 3c f3 94 bc f1 b4 36 5e d1
# e9 fb d1 f2 1e 5b a4 fb 42 12 23 f6 80 ad 9f c9
# AF stripes: 4000
# AF hash: sha256
# Area offset:32768 [bytes]
# Area length:258048 [bytes]
# Digest ID: 0
# Tokens:
# Digests:
# 0: pbkdf2
# Hash: sha256
# Iterations: 126030
# Salt: 94 24 79 20 76 1b d3 72 6f 9f d9 0b 24 fe 92 db
# ac 16 47 67 29 ef 11 7c 56 2d 44 9e 31 ac e4 26
# Digest: b4 22 72 ae be 9c e1 82 2b d5 33 ae 01 db da a8
# 35 23 0b 85 55 ac 00 ee 2f 43 e2 de 73 10 21 13
So this basically uses the default options to encrypt a partition with luks (e.g. cryptsetup luksFormat /dev/nvme0n1p3). Let’s have a look what is inside the encrypted partition:
sudo cryptsetup luksOpen /dev/nvme0n1p3 cryptdata
# Enter passphrase for /dev/nvme0n1p3:
ls /dev/mapper
# control cryptdata data-root
sudo pvs
# PV VG Fmt Attr PSize PFree
# /dev/mapper/cryptdata data lvm2 a-- <468.43g 0
sudo vgs
# VG #PV #LV #SN Attr VSize VFree
# data 1 1 0 wz--n- <468.43g 0
sudo lvs
# LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
# root data -wi-a----- <468.43g
sudo lsblk /dev/mapper/data-root -f
# NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
# data-root ext4 1.0 05f227c9-b2d5-4a8e-a297-9eaa58b45906
By default POP!_OS uses Logical Volume Management (LVM), where our encrypted partition (called cryptdata) is a physical volume that contains a volume group called data. Inside the volume group there is a logical volume called root that contains our system files. This so-called root partition is formatted with ext4. The LVM is actually a bit of an overkill for my personal use case, but I’ll stick to it as otherwise the installer cannot access the luks2 partition.
Okay, let’s close everything:
sudo cryptsetup luksClose /dev/mapper/data-root
sudo cryptsetup luksClose /dev/mapper/cryptdata
ls /dev/mapper
# control
and do the second install.
Step 2: Install Pop!_OS using the Custom (Advanced) option
Now let’s open again the installer from the dock, select the region, language and keyboard layout. Then choose Custom (Advanced). You will see your partitioned hard disk:
- Click on the first partition, activate
Use partition, activateFormat, Use asBoot /boot/efi, Filesystem:fat32. - Click on the second partition, activate
Use partition, activateFormat, Use asCustomand enter/recovery, Filesystem:fat32. - Click on the third and largest partition. A
Decrypt This Partitiondialog opens, enter your luks password and hitDecrypt. A new line is displayedLVM data. Click on this partition, activateUse partition, activateFormat, Use asRoot (/), Filesystem:btrfs. - Click on the fourth partition, activate
Use partition, Use asSwap.
If you have other partitions, check their types and use; particularly, deactivate other EFI partitions.
Recheck everything (check the partitions where there is a black checkmark) and hit Erase and Install. Follow the steps to create a user account and to write the changes to the disk. Once the installer finishes do NOT Restart Device.
Step 3: Post-Installation steps
Open a terminal and switch to an interactive root session:
sudo -i
You might find maximizing the terminal window is helpful for working with the command-line.
Mount the btrfs top-level root filesystem
Let’s mount our root partition (the top-level btrfs volume always has root-id 5), but with mount options that optimize performance and durability on SSD or NVME drives:
cryptsetup luksOpen /dev/nvme0n1p3 cryptdata
# Enter passphrase for /dev/nvme0n1p3
mount -o subvolid=5,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async /dev/mapper/data-root /mnt
I have found that there is some general agreement to use the following mount options, namely:
ssd: use SSD specific options for optimal use on SSD and NVMEnoatime: prevent frequent disk writes by instructing the Linux kernel not to store the last access time of files and foldersspace_cache: allows btrfs to store free space cache on the disk to make caching of a block group much quickercommit=120: time interval in which data is written to the filesystem (value of 120 is taken from Manjaro’s minimal iso)compress=zstd: allows to specify the compression algorithm which we want to use. btrfs provides lzo, zstd and zlib compression algorithms; however, zstd has become the best performing candidate.discard=async: Btrfs Async Discard Support Looks To Be Ready For Linux 5.6
We will later also append these mount options to the fstab, but it is good practice to already make use of these optimizations for moving the system files into subvolumes.
Create btrfs subvolumes @ and @home
Now we will first create the subvolume @ and move all files and folders from the top-level filesystem into @. Note that as we use the optimized mount options like compression, these will be already applied during the moving process:
btrfs subvolume create /mnt/@
# Create subvolume '/mnt/@'
cd /mnt
ls | grep -v @ | xargs mv -t @ #move all files and folders to /mnt/@
ls -a /mnt
# . .. @
Now let’s create another subvolume called @home and move the user folder from /mnt/@/home/ into @home:
btrfs subvolume create /mnt/@home
# Create subvolume '/mnt/@home'
mv /mnt/@/home/* /mnt/@home/
ls -a /mnt/@/home
# . ..
ls -a /mnt/@home
# . .. wmutschl
btrfs subvolume list /mnt
# ID 264 gen 339 top level 5 path @
# ID 265 gen 340 top level 5 path @home
Changes to fstab
We need to adapt the fstab to
- mount
/to the@subvolume - mount
/hometo the@homesubvolume - make use of optimized btrfs mount options
So open it with a text editor, e.g.:
nano /mnt/@/etc/fstab
or use these sed commands
sed -i 's/btrfs defaults/btrfs defaults,subvol=@,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async/' /mnt/@/etc/fstab
echo "UUID=$(blkid -s UUID -o value /dev/mapper/data-root) /home btrfs defaults,subvol=@home,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0" >> /mnt/@/etc/fstab
Either way your fstab should look like this:
cat /mnt/@/etc/fstab
# PARTUUID=6b533522-0c33-4f44-890f-4be275c5b06f /boot/efi vfat umask=0077 0 0
# PARTUUID=45bb9da4-9571-40bc-8f20-468332234a62 /recovery vfat umask=0077 0 0
# /dev/mapper/cryptswap none swap defaults 0 0
# UUID=591dae2e-37ce-42c9-8ceb-5b124658ca6a / btrfs defaults,subvol=@,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0
# UUID=591dae2e-37ce-42c9-8ceb-5b124658ca6a /home btrfs defaults,subvol=@home,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0
Note that your PARTUUID and UUID numbers will be different. The last two lines for / and /home are the important ones.
Changes to crypttab
As we use discard=async, we need to add discard to the crypttab:
sed -i 's/luks/luks,discard/' /mnt/@/etc/crypttab
cat /mnt/@/etc/crypttab
# cryptdata UUID=c5b8099a-f035-47fb-939f-fa4ea770a403 none luks,discard
# cryptswap UUID=52de8233-c50b-4873-b586-9ab313d28b56 /dev/urandom swap,plain,offset=1024,cipher=aes-xts-plain64,size=512
Adjust configuration of kernelstub
We need to adjust some settings for the systemd boot manager and also make sure these settings are not overwritten if we install or update kernels and modules. Namely, we need to add rootflags=subvol=@ to the "user" kernel options of the kernelstub configuration file:
nano /mnt/@/etc/kernelstub/configuration
Here you need to add rootflags=subvol=@ to the "user" kernel options. That is, your configuration file should look like this:
cat /mnt/@/etc/kernelstub/configuration
# {
# "default": {
# "kernel_options": [
# "quiet",
# "splash"
# ],
# "esp_path": "/boot/efi",
# "setup_loader": false,
# "manage_mode": false,
# "force_update": false,
# "live_mode": false,
# "config_rev":3
# },
# "user": {
# "kernel_options": [
# "quiet",
# "loglevel=0",
# "systemd.show_status=false",
# "splash",
# "rootflags=subvol=@"
# ],
# "esp_path": "/boot/efi",
# "setup_loader": true,
# "manage_mode": true,
# "force_update": false,
# "live_mode": false,
# "config_rev":3
# }
# }
VERY IMPORTANTLY: Don’t forget the comma after "splash" (in the line above your added "rootflags=subvol=@" option) , otherwise you get errors when you later run update-initramfs (see below)!
Adjust configuration of systemd bootloader
We need to adjust some settings for the systemd boot manager, so let’s mount our EFI partition
mount /dev/nvme0n1p1 /mnt/@/boot/efi
Add rootflags=subvol=@ to last line of Pop_OS_current.conf either using a text editor or the following command
sed -i 's/splash/splash rootflags=subvol=@/' /mnt/@/boot/efi/loader/entries/Pop_OS-current.conf
cat /mnt/@/boot/efi/loader/entries/Pop_OS-current.conf
# title Pop!_OS
# linux /EFI/Pop_OS-UUID_of_data-root/vmlinuz.efi
# initrd /EFI/Pop_OS-UUID_of_data-root/initrd.img
# options root=UUID=UUID_of_data-root ro quiet loglevel=0 systemd.show_status=false splash rootflags=subvol=@
where UUID_of_data-root is the UUID of /dev/mapper/data-root.
Optionally, add a timeout to the systemd boot menu in order to easily access the recovery partition:
echo "timeout 2" >> /mnt/@/boot/efi/loader/loader.conf
cat /mnt/@/boot/efi/loader/loader.conf
# default Pop_OS-current
# timeout 2
Create a chroot environment and update initramfs
Now, let’s create a chroot environment, which enables you to work directly inside your newly installed OS, without actually rebooting. For this, unmount the top-level root filesystem from /mnt and remount the subvolume @ which we created for / to /mnt:
cd /
umount -l /mnt
mount -o defaults,subvol=@,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async /dev/mapper/data-root /mnt
Then the following commands will put us into our system using chroot:
for i in /dev /dev/pts /proc /sys /run; do mount -B $i /mnt$i; done
chroot /mnt
Cool, you are now inside your system and we can check whether our fstab mounts everything correctly:
mount -av
# /boot/efi : successfully mounted
# /recovery : successfully mounted
# none : ignored
# / : ignored
# /home : successfully mounted
Looks good! Now we need to update the initramfs to make it aware of our changes:
update-initramfs -c -k all
Note that if you run into errors like this:
update-initramfs: Generating /boot/initrd.img-5.11.0-7620-generic
kernelstub.Config : INFO Looking for configuration...
Traceback (most recent call last):
File "/usr/bin/kernelstub", line 244, in <module>
main()
File "/usr/bin/kernelstub", line 241, in main
kernelstub.main(args)
File "/usr/lib/python3/dist-packages/kernelstub/application.py", line 142, in main
config = Config.Config()
File "/usr/lib/python3/dist-packages/kernelstub/config.py", line 50, in __init__
self.config = self.load_config()
File "/usr/lib/python3/dist-packages/kernelstub/config.py", line 60, in load_config
self.config = json.load(config_file)
File "/usr/lib/python3.9/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 20 column 7 (char 363)
run-parts: /etc/initramfs/post-update.d//zz-kernelstub exited with return code 1
you probably forgot a comma after "splash" in the /etc/kernelstub/configuration file (see above).
Step 4: Reboot, some checks, and update system
Now, it is time to exit the chroot - cross your fingers - and reboot the system:
exit
# exit
reboot now
If all went well you should see a single passphrase prompt (YAY!), where you enter the luks passphrase and your system boots.
Now let’s click through the welcome screen and open a terminal to see whether everything is set up correctly:
sudo mount -av
# /boot/efi : already mounted
# /recovery : already mounted
# none : ignored
# / : ignored
# /home : already mounted
sudo mount -v | grep /dev/mapper
# /dev/mapper/data-root on / type btrfs (rw,noatime,compress=zstd:3,ssd,discard=async,space_cache,commit=120,subvolid=265,subvol=/@)
# /dev/mapper/data-root on /home type btrfs (rw,noatime,compress=zstd:3,ssd,discard=async,space_cache,commit=120,subvolid=266,subvol=/@home)
sudo swapon
# NAME TYPE SIZE USED PRIO
# /dev/dm-2 partition 4G 0B -2
sudo btrfs filesystem show /
# Label: none uuid: 591dae2e-37ce-42c9-8ceb-5b124658ca6a
# Total devices 1 FS bytes used 8.15GiB
# devid 1 size 468.43GiB used 10.02GiB path /dev/mapper/data-root
sudo btrfs subvolume list /
# ID 265 gen 82 top level 5 path @
# ID 266 gen 82 top level 5 path @home
If all look’s good, let’s update and upgrade the system:
sudo apt update
sudo apt upgrade
sudo apt dist-upgrade
sudo apt autoremove
sudo apt autoclean
flatpak update
If you installed on a SSD or NVME, enable fstrim.timer as both fstrim and discard=async mount option can peacefully co-exist:
sudo systemctl enable fstrim.timer
Again, for SSD trimming to work properly, it is important that you add discard to your crypttab (see above). Also check whether you find issue_discards=1 in /etc/lvm/lvm.conf (which should be correct by default):
sudo grep "issue_discards" /etc/lvm/lvm.conf
# # Configuration option devices/issue_discards.
# issue_discards = 1
Now reboot:
sudo reboot now
Step 5: automatic snapshots and backups with btrfs using BTRBK
Step 5a: preparations
First I create a mount point /btrfs_pool for the top-level root of my btrfs partition:
sudo mkdir /btrfs_pool
Next I add an entry to the fstab to mount this at boot time:
echo "UUID=$(blkid -s UUID -o value /dev/mapper/data-root) /btrfs_pool btrfs defaults,subvolid=5,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0" | sudo tee -a /etc/fstab
cat /etc/fstab
# PARTUUID=6b533522-0c33-4f44-890f-4be275c5b06f /boot/efi vfat umask=0077 0 0
# PARTUUID=45bb9da4-9571-40bc-8f20-468332234a62 /recovery vfat umask=0077 0 0
# /dev/mapper/cryptswap none swap defaults 0 0
# UUID=591dae2e-37ce-42c9-8ceb-5b124658ca6a / btrfs defaults,subvol=@,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0
# UUID=591dae2e-37ce-42c9-8ceb-5b124658ca6a /home btrfs defaults,subvol=@home,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0
# UUID=591dae2e-37ce-42c9-8ceb-5b124658ca6a /btrfs_pool btrfs defaults,subvolid=5,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0
The last line is added which mounts the top-level (subvolid=5, NOT subvol!) to /btrfs_pool. Let’s try the fstab and mount everything:
sudo mount -av
# /boot/efi : already mounted
# /recovery : already mounted
# none : ignored
# / : ignored
# /home : already mounted
# /btrfs_pool : successfully mounted
BTRBK needs a folder under the top-level where it stores the snapshots, let’s call this folder _btrbk_snap and create it:
sudo mkdir /btrfs_pool/_btrbk_snap
ls /btrfs_pool
# @ _btrbk_snap @home
Step 5b: install and configure BTRBK
Install BTRBK from the repos:
sudo apt install -y btrbk
Next I create the following configuration file:
mkdir -p $HOME/scripts
nano $HOME/scripts/precision-btrbk.conf
My configuration file (just for snapshots) looks like this:
transaction_log /var/log/btrbk.log
lockfile /var/lock/btrbk.lock
timestamp_format long
snapshot_dir _btrbk_snap
snapshot_preserve_min 3h
snapshot_preserve 6h 5d 3w 1m
volume /btrfs_pool
snapshot_create always
subvolume @
subvolume @home
This looks into /btrfs_pool and creates snapshots for the subvolumes @ and @home into the directory /btrfs_pool/_btrbk_snap. All snapshots are preserved for at least 3 hours, while the usual retention strategy is to keep 6 hourly, 5 daily, 3 weekly and 1 monthly snapshot. Let’s test this:
sudo btrbk -c $HOME/scripts/precision-btrbk.conf dryrun
# --------------------------------------------------------------------------------
# Backup Summary (btrbk command line client, version 0.27.1)
# Date: Mon Dec 13 21:10:45 2021
# Config: /home/wmutschl/scripts/precision-btrbk.conf
# Dryrun: YES
# Legend:
# === up-to-date subvolume (source snapshot)
# +++ created subvolume (source snapshot)
# --- deleted subvolume
# *** received subvolume (non-incremental)
# >>> received subvolume (incremental)
# --------------------------------------------------------------------------------
# /btrfs_pool/@
# +++ /btrfs_pool/_btrbk_snap/@.20211213T2110
# /btrfs_pool/@home
# +++ /btrfs_pool/_btrbk_snap/@home.20211213T2110
#
# NOTE: Dryrun was active, none of the operations above were actually executed!
If there was no error, let’s actually run this to create our first snapshots (this should take a fraction of a second) and see whether they are stored correctly:
sudo btrbk -c $HOME/scripts/precision-btrbk.conf run
# --------------------------------------------------------------------------------
# Backup Summary (btrbk command line client, version 0.27.1)
# Date: Mon Dec 13 21:10:59 2021
# Config: /home/wmutschl/scripts/precision-btrbk.conf
#
# Legend:
# === up-to-date subvolume (source snapshot)
# +++ created subvolume (source snapshot)
# --- deleted subvolume
# *** received subvolume (non-incremental)
# >>> received subvolume (incremental)
# --------------------------------------------------------------------------------
# /btrfs_pool/@
# +++ /btrfs_pool/_btrbk_snap/@.20211213T2110
#
# /btrfs_pool/@home
# +++ /btrfs_pool/_btrbk_snap/@home.20211213T2110
ls /btrfs_pool/_btrbk_snap
# @.20211213T2110 @home.20211213T2110
sudo btrfs subvolume list /
# ID 265 gen 153 top level 5 path @
# ID 266 gen 154 top level 5 path @home
# ID 269 gen 154 top level 5 path _btrbk_snap/@.20211213T2110
# ID 270 gen 154 top level 5 path _btrbk_snap/@home.20211213T2110
Step 5c: create systemd timer for BTRBK to run every hour
On servers that run constantly I usually use the crontab for automatic snapshots with BTRBK; however, on my laptop I use a systemd timer instead. First let’s adapt/create the btrbk timer: sudo nano /lib/systemd/system/btrbk.timer which looks like this:
[Unit]
Description=btrbk hourly snapshots and backup
[Timer]
OnCalendar=hourly
AccuracySec=10min
Persistent=true
[Install]
WantedBy=timers.target
Second let’s adapt the service sudo nano /lib/systemd/system/btrbk.service which looks like this:
[Unit]
Description=btrbk snapshots and backup
Documentation=man:btrbk(1)
[Service]
Type=oneshot
ExecStart=/usr/sbin/btrbk -c /home/wmutschl/scripts/precision-btrbk.conf run
Make sure the permissions are correct:
sudo chmod 644 /lib/systemd/system/btrbk.timer
sudo chmod 644 /lib/systemd/system/btrbk.service
and checkout if it works (tip: you can exit the log outputs by writing :q or hitting CTRL+C)):
sudo systemctl start btrbk.service
sudo systemctl status btrbk.service
# ○ btrbk.service - btrbk snapshots and backup
# Loaded: loaded (/lib/systemd/system/btrbk.service; static)
# Active: inactive (dead)
# Docs: man:btrbk(1)
#
# Dec 13 21:24:22 pop-os btrbk[6699]: --- deleted subvolume
# Dec 13 21:24:22 pop-os btrbk[6699]: *** received subvolume (non-incremental)
# Dec 13 21:24:22 pop-os btrbk[6699]: >>> received subvolume (incremental)
# Dec 13 21:24:22 pop-os btrbk[6699]: --------------------------------------------------------->
# Dec 13 21:24:22 pop-os btrbk[6699]: /btrfs_pool/@
# Dec 13 21:24:22 pop-os btrbk[6699]: +++ /btrfs_pool/_btrbk_snap/@.20211213T2124
# Dec 13 21:24:22 pop-os btrbk[6699]: /btrfs_pool/@home
# Dec 13 21:24:22 pop-os btrbk[6699]: +++ /btrfs_pool/_btrbk_snap/@home.20211213T2124
# Dec 13 21:24:22 pop-os systemd[1]: btrfs-btrbk-systemd.service: Deactivated successfully.
# Dec 13 21:24:22 pop-os systemd[1]: Finished btrbk snapshots and backup.
cat /var/log/btrbk.log
# 2021-12-13T21:24:22+0100 startup v0.27.1 - - - # btrbk command line client, version 0.27.1
# 2021-12-13T21:24:22+0100 snapshot starting /btrfs_pool/_btrbk_snap/@.20211213T2124 /btrfs_pool/@ - -
# 2021-12-13T21:24:22+0100 snapshot success /btrfs_pool/_btrbk_snap/@.20211213T2124 /btrfs_pool/@ - -
# 2021-12-13T21:24:22+0100 snapshot starting /btrfs_pool/_btrbk_snap/@home.20211213T2124 /btrfs_pool/@home - -
# 2021-12-13T21:24:22+0100 snapshot success /btrfs_pool/_btrbk_snap/@home.20211213T2124 /btrfs_pool/@home - -
# 2021-12-13T21:24:22+0100 finished success - - - -
ls /btrfs_pool/_btrbk_snap
# @.20211213T2110 @home.20211213T2110
# @.20211213T2124 @home.20211213T2124
Check if snapshots are created and if any errors occured. If all is well, then enable the timer:
sudo systemctl enable btrbk.timer
# Created symlink /etc/systemd/system/timers.target.wants/btrbk.timer → /lib/systemd/system/btrbk.timer.
sudo systemctl start btrbk.timer
sudo systemctl daemon-reload
sudo systemctl list-timers --all
You should see the following line (tip: you can exit by inserting :q or clicking CTRL+C):
NEXT LEFT LAST PASSED UNIT ACTIVATES
Mon 2021-12-13 22:00:00 CET 33min left n/a n/a btrbk.timer btrbk.service
Recheck the hourly timer after an hour (or 33min in my case) to make sure everything is working:
sudo systemctl list-timers --all
cat /var/log/btrbk.log
ls /btrfs_pool/_btrbk_snap
Make sure the snapshots are created without errors.
Step 5d (optional): Mount an encrypted backup disk as btrfs send/receive target
I use the internal SSD as a backup disk to receive the incremental btrfs snapshots. So let’s create a GPT table on it, create an encrypted partition and format it with the btrfs filesystem. I usually use GParted for this (after installing it); however, you can also use command-line tools like parted, e.g.:
sudo parted /dev/sda mklabel gpt
# Warning: The existing disk label on /dev/sda will be destroyed and all data on this disk will
# be lost. Do you want to continue?
# Yes/No? Yes
# Information: You may need to update /etc/fstab.
sudo parted /dev/sda mkpart primary 1MiB 100%
# Information: You may need to update /etc/fstab.
sudo parted /dev/sda name 1 BACKUP
sudo parted /dev/sda unit MiB print
# Model: ATA Samsung SSD 840 (scsi)
# Disk /dev/sda: 476940MiB
# Sector size (logical/physical): 512B/512B
# Partition Table: gpt
# Disk Flags:
# Number Start End Size File system Name Flags
# 1 1.00MiB 476940MiB 476939MiB BACKUP
sudo cryptsetup luksFormat /dev/sda1
# WARNING: Device /dev/sda1 already contains a 'crypto_LUKS' superblock signature.
# WARNING!
# ========
# This will overwrite data on /dev/sda1 irrevocably.
# Are you sure? (Type 'yes' in capital letters): YES
# Enter passphrase for /dev/sda1:
# Verify passphrase:
sudo cryptsetup luksOpen /dev/sda1 cryptbackup
# Enter passphrase for /dev/sda1:
sudo mkfs.btrfs /dev/mapper/cryptbackup
# btrfs-progs v5.10.1
# See http://btrfs.wiki.kernel.org for more information.
#
# Detected a SSD, turning off metadata duplication. Mkfs with -m dup if you want to force metadata duplication.
# Label: (null)
# UUID: 6f462de9-8148-4b61-b390-c1b9038cb367
# Node size: 16384
# Sector size: 4096
# Filesystem size: 465.75GiB
# Block group profiles:
# Data: single 8.00MiB
# Metadata: single 8.00MiB
# System: single 4.00MiB
# SSD detected: yes
# Incompat features: extref, skinny-metadata
# Runtime features:
# Checksum: crc32c
# Number of devices: 1
# Devices:
# ID SIZE PATH
# 1 465.75GiB /dev/mapper/cryptbackup
Let’s create a mount point /btrfs_backup for this disk and update the fstab:
sudo mkdir /btrfs_backup
Next I add an entry to the fstab to mount this at boot time:
echo "UUID=$(blkid -s UUID -o value /dev/mapper/cryptbackup) /btrfs_backup btrfs defaults,subvolid=5,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0" | sudo tee -a /etc/fstab
cat /etc/fstab
# PARTUUID=6b533522-0c33-4f44-890f-4be275c5b06f /boot/efi vfat umask=0077 0 0
# PARTUUID=45bb9da4-9571-40bc-8f20-468332234a62 /recovery vfat umask=0077 0 0
# /dev/mapper/cryptswap none swap defaults 0 0
# UUID=591dae2e-37ce-42c9-8ceb-5b124658ca6a / btrfs defaults,subvol=@,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0
# UUID=591dae2e-37ce-42c9-8ceb-5b124658ca6a /home btrfs defaults,subvol=@home,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0
# UUID=591dae2e-37ce-42c9-8ceb-5b124658ca6a /btrfs_pool btrfs defaults,subvolid=5,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0
# UUID=6f462de9-8148-4b61-b390-c1b9038cb367 /btrfs_backup btrfs defaults,subvolid=5,ssd,noatime,space_cache,commit=120,compress=zstd,discard=async 0 0
The last line is added which mounts the top-level (subvolid=5, NOT subvol!) to /btrfs_backup. Let’s try the fstab and mount everything:
sudo mount -av
# /boot/efi : already mounted
# /recovery : already mounted
# none : ignored
# / : ignored
# /home : already mounted
# /btrfs_pool : successfully mounted
# /btrfs_backup : successfully mounted
I want my system to automatically unlock my backup disk such that I need to type my luks passphrase only once (this step is optional, but recommended). So let’s create a key-file, secure it, and add it to our luks partition of the backup disk:
sudo mkdir /etc/luks
sudo dd if=/dev/urandom of=/etc/luks/boot_os.keyfile bs=4096 count=1
# 1+0 records in
# 1+0 records out
# 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000928939 s, 4.4 MB/s
sudo chmod u=rx,go-rwx /etc/luks
sudo chmod u=r,go-rwx /etc/luks/boot_os.keyfile
sudo cryptsetup luksAddKey /dev/sda1 /etc/luks/boot_os.keyfile
# Enter any existing passphrase:
Let’s restrict the pattern of keyfiles and avoid leaking key material for the initramfs hook:
echo "KEYFILE_PATTERN=/etc/luks/*.keyfile" | sudo tee -a /etc/cryptsetup-initramfs/conf-hook
echo "UMASK=0077" | sudo tee -a /etc/initramfs-tools/initramfs.conf
These commands will harden the security options in the intiramfs configuration file and hook (not sure if this is also needed for systemd bootloader?!?).
Next, add the keyfile to your crypttab:
echo "cryptbackup UUID=$(blkid -s UUID -o value /dev/sda1) /etc/luks/boot_os.keyfile luks,discard" | sudo tee -a /etc/crypttab
cat /etc/crypttab
# cryptdata UUID=c5b8099a-f035-47fb-939f-fa4ea770a403 none luks,discard
# cryptswap UUID=52de8233-c50b-4873-b586-9ab313d28b56 /dev/urandom swap,plain,offset=1024,cipher=aes-xts-plain64,size=512
# cryptbackup UUID=87278135-9aec-4da7-9fe4-fca1ecf2aeb7 /etc/luks/boot_os.keyfile luks,discard
and update the initramfs:
sudo update-initramfs -u -k all
BTRBK needs folders under the top-level of your backup disk where it stores the received snapshots, let’s call these folders @ and @home and create it:
sudo mkdir /btrfs_backup/@
sudo mkdir /btrfs_backup/@home
ls /btrfs_backup
# @ @home
We need to also change the configuration file used for BTRBK: nano $HOME/scripts/precision-btrbk.conf:
transaction_log /var/log/btrbk.log
lockfile /var/lock/btrbk.lock
timestamp_format long
snapshot_dir _btrbk_snap
snapshot_preserve_min 3h
snapshot_preserve 6h 5d 3w 1m
target_preserve_min 3h
target_preserve 24h 31d 52w
volume /btrfs_pool
snapshot_create always
target send-receive /btrfs_backup
subvolume @
subvolume @home
Let’s first run it in dry mode:
sudo btrbk -c $HOME/scripts/precision-btrbk.conf dryrun
If there was no error, run it:
sudo btrbk -c $HOME/scripts/precision-btrbk.conf run --progress
This will take a while, because the initial backup is transferred to your backup disk. All other snapshots will be sent and received incrementially.
Step 5e (optional): Automatic snapshots for any apt operation
TBA
Now, if you run sudo apt install|remove|upgrade|dist-upgrade, btrbk will create a snapshot of your system, but these won’t be sent to your backup disk right away, but only every hour.
FINISHED! CONGRATULATIONS AND THANKS FOR STICKING THROUGH!
Check out my Pop!_OS post-installation steps.
If you ever need to rollback your system, checkout my Recovery and system rollback.