Create (or Expand) a ZFS Mirror in FreeBSD
Here’s a quick guide to convert a system created by the FreeBSD installer from a single-drive stripe into a 2-drive mirror. The same procedure also works to expand an existing mirror by adding additional drives.
The single-drive system was originally partitioned using Auto (ZFS) in the installer and configuring these options:
|Pool Type/Disks:||stripe: 1 disk|
To add a new mirror drive to the system, first back up the data.
zpool status to examine the current ZFS pool. In this example, the zroot pool consists of the third partition (p3) of drive ada0 and it has been encrypted with GELI (.eli).
$ su Password: # zpool status pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 ada0p3.eli ONLINE 0 0 0 errors: No known data errors
swapinfo to examine the swap partition and notice that the swap uses the second partition (p2) of the same drive (ada0) and that it is also GELI-encrypted (.eli).
# swapinfo Device 1K-blocks Used Avail Capacity /dev/ada0p2.eli 2097152 0 2097152 0%
camcontrol to get the drive’s serial number for reference.
# geom disk list ada0 | grep ident ident: WD-WMATV0942547 # camcontrol identify ada0 | grep serial serial number WD-WMATV0942547
gpart show to display the complete partition layout for drive ada0.
# gpart show ada0 => 40 1953525088 ada0 GPT (932G) 40 1024 1 freebsd-boot (512K) 1064 984 - free - (492K) 2048 4194304 2 freebsd-swap (2.0G) 4196352 1949327360 3 freebsd-zfs (930G) 1953523712 1416 - free - (708K)
At this point, copy the serial number of the new drive to be added to the system from the drive label.
Insert the new drive into an empty bay and run
dmesg to find the device node assigned to it.
In this case, the new drive has been assigned ada1.
# dmesg [...] ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: <Hitachi HDS721010CLA332 JP4OA25C> ATA8-ACS SATA 2.x device ada1: Serial Number JP2921HQ0A6ZJA ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 953869MB (1953525168 512 byte sectors)
Delete any existing partitions on the new drive using
gpart destroy. The command will give an error if the drive doesn’t contain any partitions.
# gpart destroy -F ada1 ada1 destroyed # gpart destroy -F ada1 gpart: arg0 'ada1': Invalid argument
The easiest way to copy the partition layout from drive ada0 to drive ada1 is to use the
gpart backup and
gpart restore commands.
gpart backup to examine the partition layout on drive ada0.
# gpart backup ada0 GPT 128 1 freebsd-boot 40 1024 gptboot0 2 freebsd-swap 2048 4194304 swap0 3 freebsd-zfs 4196352 1949327360 zfs0
The last column in the output lists the automatically generated GPT labels for each of the three partitions on drive
ada0, which is why the installer appended a
0 to the end of each GPT label.
Since the new drive has been assigned the device node
ada1, run the command again and pipe the output into
sed to replace the last
0 on each line with a
1 to generate new GPT label names for the new drive.
# gpart backup ada0 | sed -r 's/(^[[:digit:]].*)0/\11/' GPT 128 1 freebsd-boot 40 1024 gptboot1 2 freebsd-swap 2048 4194304 swap1 3 freebsd-zfs 4196352 1949327360 zfs1
If the output from sed looks correct, run the last command again and pipe those results into
gpart restore to copy the partition layout to ada1.
# gpart backup ada0 | sed -r 's/(^[[:digit:]].*)0/\11/' | gpart restore -lF ada1
gpart show to see that both drives now have the same layout.
# gpart show -l => 40 1953525088 ada0 GPT (932G) 40 1024 1 gptboot0 (512K) 1064 984 - free - (492K) 2048 4194304 2 swap0 (2.0G) 4196352 1949327360 3 zfs0 (930G) 1953523712 1416 - free - (708K) => 40 1953525088 ada1 GPT (932G) 40 1024 1 gptboot1 (512K) 1064 984 - free - (492K) 2048 4194304 2 swap1 (2.0G) 4196352 1949327360 3 zfs1 (930G) 1953523712 1416 - free - (708K)
Since the root partition was GELI-encrypted during the installation, check the installer log to find out what options were used for the
geli init and
geli attach commands.
# grep geli /var/log/bsdinstall_log [...] DEBUG: zfs_create_boot: geli init -bg -e AES-XTS -J - -l 256 -s 4096 "ada0p3" [...] DEBUG: zfs_create_boot: geli attach -j - "ada0p3" [...]
geli init command for the new drive using slightly different options. Remove the
-J - option to make the command interactive and change ada0p3 to ada1p3 to match the new drive.
When requested, enter the same GELI password currently used to boot the system.
# geli init -bg -e AES-XTS -l 256 -s 4096 "ada1p3" Enter new passphrase: Reenter new passphrase: Metadata backup for provider ada1p3 can be found in /var/backups/ada1p3.eli and can be restored with the following command: # geli restore /var/backups/ada1p3.eli ada1p3
Mount the new GELI partition using the
geli attach command, enter the password when requested and view the active containers with
# geli attach "ada1p3" Enter passphrase: # geli status Name Status Components ada0p3.eli ACTIVE ada0p3 ada0p2.eli ACTIVE ada0p2 ada1p3.eli ACTIVE ada1p3
zpool attach to attach the new root partition to the existing stripe and convert it into a mirror. Write the boot code to the boot partition on the new drive as recommended by the output text.
# zpool status pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 ada0p3.eli ONLINE 0 0 0 errors: No known data errors # zpool attach zroot ada0p3.eli ada1p3.eli Make sure to wait until resilver is done before rebooting. If you boot from pool 'zroot', you may need to update boot code on newly attached disk 'ada1p3.eli'. Assuming you use GPT partitioning and 'da0' is your new boot disk you may use the following command: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1 partcode written to ada1p1 bootcode written to ada1
At this point, /etc/fstab contains an entry for the swap partition on drive ada0, which was added by the installer, but no entry for the swap partition on the newly added drive.
Edit the file or use
sed to add an entry for the new drive.
# cat /etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/ada0p2.eli none swap sw 0 0 # sed -n 's/ada0p2/ada1p2/p' /etc/fstab /dev/ada1p2.eli none swap sw 0 0 # sed -n 's/ada0p2/ada1p2/p' /etc/fstab >> /etc/fstab # cat /etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/ada0p2.eli none swap sw 0 0 /dev/ada1p2.eli none swap sw 0 0
Wait for the drive to finish resilvering and check the status of the mirror.
# zpool status pool: zroot state: ONLINE scan: resilvered 823M in 0 days 00:00:19 with 0 errors on Mon Mar 23 21:56:30 2020 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p3.eli ONLINE 0 0 0 ada1p3.eli ONLINE 0 0 0 errors: No known data errors
That’s it. The same procedure can be repeated to add more drives as needed, but drives are reliable enough that it’s hard to justify mirroring more than three.
# dmesg [...] # zpool attach zroot ada0p3.eli ada2p3.eli [...] # zpool status pool: zroot state: ONLINE scan: resilvered 823M in 0 days 00:00:16 with 0 errors on Mon Mar 23 22:27:10 2020 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p3.eli ONLINE 0 0 0 ada1p3.eli ONLINE 0 0 0 ada2p3.eli ONLINE 0 0 0 errors: No known data errors