Programmers' Pain
12Nov/120

Howto setup an encrypted RAID1 array using external USB 3.0 disks

RAID1 setup using Linux software raid and LUKS encryption

RAID1 setup using Linux software raid and LUKS encryption

After one of my last blog posts dealt with modifying my two Sharkoon SATA Quickport XT USB3.0 docking stations I did spend some time figuring out how I can use them to set up multiple RAID1 volumes that should be encrypted using the LUKS environment of recent Linux distributions. Also I was searching for a mechanism to clearly map every involved disk to an unique raid volume so that I don’t end up with a generic mount point like /mnt/raid1/ that would just show the content of the currently docked disks – instead every RAID1 volume should have it’s own dedicated mount point. The solution I’ve came up with uses UDEV rules to identify the disks, mdadm to map two disks to a single software-RAID1 volume, cryptsetup to encrypt the raid volume and some smaller, self-written helper scripts that do simplify the usage of those external RAID1 volumes. How it all works and how you can set up such an environment on your own Linux machine: just click ‘continue reading’ 🙂

Defining UDEV rules to get unique symbolic links for each disk

So the general idea is that each disk will have a dedicated UDEV rule which will result in a unique symlink in the /dev directory pointing to the actual block device the disk has been made available at by the Linux kernel. In addition to the /dev/sdd and /dev/sde mount points where my docking stations usually do show up each docked disk will have an unique symlink like /dev/raid1v1d1 and /dev/raid1v1d2 pointing to the actual /dev/sdd and /dev/sde block devices. raid1v1d1 stands for: raid1, volume1, disk1 in this example to give those symlinks a meaningful name without being forced to type tons of text. With using those symlinks I can set up multiple RAID1 volumes with mdadm that won’t collide with each other as if you would have used the /dev/sdX block devices directly since the real block devices will be shared by multiple disks:

Imagine you’d have set up two RAID1 volumes with mdadm that would directly use the /dev/sdX block devices – If you would put one disk of each RAID1 volume into the docking stations than the second disk you’ve inserted would be detected as an out-of-sync disk of the first disks RAID1 volume since both RAID1 volumes do share the same /dev/sdX block devices. The result would be that the RADI1 volume of the first docked disk will run with only one active member disk and the second disk would be handled as an out-of-sync disk although it isn’t even a member of the first disks RAID1 volume. Even worse: if you would trigger a resync of the RAID1 volume than you’d delete the contents of the second disk if you’d skip the ‘this disk contains metadata of another raid volume’ warning message! To eliminate that risk in general I’ll only use those unique symlinks defined by the following example UDEV rules:

1
2
3
4
5
6
7
8
9
10
11
#RAID1 Volume 1
SUBSYSTEMS=="usb", KERNEL=="sd?", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="1234511111", SYMLINK+="raid1v1d1"
SUBSYSTEMS=="usb", KERNEL=="sd?", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="1234522222", SYMLINK+="raid1v1d2"
SUBSYSTEMS=="usb", KERNEL=="sd?1", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="1234511111", SYMLINK+="raid1v1d1p1"
SUBSYSTEMS=="usb", KERNEL=="sd?1", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="1234522222", SYMLINK+="raid1v1d2p1"

#RAID1 Volume 2
SUBSYSTEMS=="usb", KERNEL=="sd?", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="1234533333", SYMLINK+="raid1v2d1"
SUBSYSTEMS=="usb", KERNEL=="sd?", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="1234544444", SYMLINK+="raid1v2d2"
SUBSYSTEMS=="usb", KERNEL=="sd?1", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="1234533333", SYMLINK+="raid1v2d1p1"
SUBSYSTEMS=="usb", KERNEL=="sd?1", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="1234544444", SYMLINK+="raid1v2d2p1"

Those example UDEV rules that I do store in the /etc/udev/rules.d/99-usbRaid1.rules file on my Linux machine are doing multiple things:

  • define raid1v#d# symlinks for in total four different disks that are identified via their serial numbers (ID_SERIAL_SHORT property).
  • define raid1v#d#p1 symlinks for the first partition for each of the four disks.
  • those rules will only be considered if the the new /dev/sdX block device is provided by the USB subsystem of the kernel since I’m using USB3.0 docking stations.

So whenever I dock one of those disks two additional symlinks (/dev/raid1v#d# and /dev/raid1v#d#p1) will be created if the new block device was provided by the USB subsystem and if the serial of the disk matches the one defined in the UDEV rules. You can use hdparm the get the serial number of a disk via its block device:

1
hdparm -I /dev/sdX | grep "Serial Number:"

Whenever you change existing UDEV rules or add new ones than you should restart the UDEV service of your Linux distribution and call udevadm to force UDEV to re-trigger all known UDEV rules for all existing block devices. As an alternative you could also re-connect your USB disks so that UDEV will consider your new rules as soon as the disk are available again.

1
2
service udev restart
udevadm trigger

If you haven’t set up your disks with a fresh partition table and an empty first partition yet I’d suggest to use parted, gparted or any other disk partition tool to do so. For all of my disks I did use a GPT partition table and a primary partition without any file system type since it will be used by mdadm own data format anyway. After you’re done with setting up the partitions and calling udevadm trigger once again you should find the following entries in your /dev directory assuming that you would have the disks with the exemplary 1234511111 and 1234522222 serial numbers up and running:

1
2
3
4
5
ls -la /dev/raid1*
lrwxrwxrwx 1 root root 23 Jul 13 17:05 /dev/raid1v1d1 -> sdd
lrwxrwxrwx 1 root root 23 Jul 13 17:05 /dev/raid1v1d1p1 -> sdd1
lrwxrwxrwx 1 root root 23 Jul 13 17:05 /dev/raid1v1d2 -> sde
lrwxrwxrwx 1 root root 23 Jul 13 17:05 /dev/raid1v1d2p1 -> sde1
Setting up RAID1 volumes using mdadm

So whenever you’re done with setting up the partitions you’re ready to call mdadm to create a new RAID1 volume using the two partitions of the two disks available via the symlinks defined by the UDEV rules:

1
2
3
mdadm --create /dev/md/raid1v1 --level=1 --raid-devices=2 /dev/raid1v1d1p1 /dev/raid1v1d2p1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/raid1v1 started.

What has happened here is that we’ve just created a new RAID1 volume that will be available via the /dev/md/raid1v1 block device following the same naming schema as the disk symlinks. As soon as the mdadm call returns to your Bash console prompt the new RAID1 volume is ready to be used. If you want to have a quick look at the current status of the new volume just have a look at the /proc filesystem:

1
2
3
4
5
6
7
cat /proc/mdstat
Personalities : [raid1]
md127 : active raid1 sde1[1] sdd1[0]
1465005888 blocks super 1.2 [2/2] [UU]
[>....................]  resync =  0.0% (183680/1465005888) finish=664.5min speed=36736K/sec

unused devices: <none>

The output of /proc/mdstat contains several interesting information that are worth to talk about in a bit more detail: The first thing you should notice is that your new /dev/md/raid1v1 block device is again just an alias for the real /dev/md127 block device provided by the Linux kernel. Again, we’re following the same principal as we did with the disks that each RAID1 volume will have an unique symlink so that we can mount it in a dedicated location. As for the disks it’s again uncritical that we reuse the real block devices like /dev/sdd, /dev/sde or /dev/md127 since we can only mount two disks at the same time.

The second thing is that although we explicitly specified the symlinks we’ve set up with our UDEV rules the Linux software raid implementation still uses the real block devices the disks were made available by the Linux kernel. This is again not a problem since we only use those symlinks to have an unique device name per disk and raid volume. This symlink-based approach will especially show its benefits if you would get another couple of docking stations to be able to have two RAID1 volumes running at the same time or if you’re thinking about mounting two disks directly into your PC: In both cases you’ll just don’t care which block devices the kernel will choose to give you access to your disks since you’ll only use those symlinks to access your RAID1 volumes. Also mdadm will use the same approach to give access to those volumes via /dev/md127, /dev/md126 and so forth but all you’ll care about are those /dev/md/raid1v# symlinks:

1
2
3
4
ls -la /dev/md/*
total 4
lrwxrwxrwx  1 root root    23 Jul 15 19:30 raid1v1 -> ../md127
lrwxrwxrwx  1 root root   23 Jul 15 19:30 /dev/md/raid1v1p1 -> ../md127p1

Also notice that mdadm will automatically provide you with additional symlinks for every partition on your raid volume. So whenever you’ve used gparted or a similar tool to create an empty partition on the /dev/md/raid1v1 block device you should see the same content as above in your /dev/md directory.

Encrypting the RAID1 volume using LUKS / cryptsetup

As written in the introduction of this post I also want to encrypt those RAID1 volumes using LUKS environment and its cryptsetup tool. Setting up an encrypted container isn’t really complicated if you’re happy with using the default values:

1
2
3
4
5
6
7
8
9
10
cryptsetup --verbose --verify-passphrase luksFormat /dev/md/raid1v1p1

WARNING!
========
This will overwrite data on /dev/md/raid1v1p1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter LUKS passphrase:
Verify passphrase:
Command successful.

I’ve added the –verify-passphrase argument on purpose since otherwise cryptsetup would just create the encrypted container without validating the password you’ve entered via the passphrase prompt. To be able to use your just created cryptsetup container on top of the mdadm RAID1 volume you have to open it first so that it’s unencrypted content will be available via another block device to be found in the /dev/mapper directory afterwards:

1
2
3
cryptsetup luksOpen /dev/md/raid1v1p1 luks-raid1v1

Enter passphrase for /dev/md/raidv1p1:

After you’ve opened the encrypted container using the password you’ve used while setting up the container you’ll find a new block device called /dev/mapper/luks-raid1v1 that you can format as any other disk partition with your favoured file system.

If you’ve followed the instructions of this blog post so far you should have a formatted block device by now that is ready to be mounted and used. Since triggering a single umount call isn’t enough to stop the LUKS block device and the RAID1 volume behind it you’d need to issue the following commands to stop your encrypted RAID1 volume again:

1
2
3
umount /dev/mapper/luks-raid1v1
cryptsetup luksClose luks-raid1v1
mdadm --stop /dev/md/raid1v1

The next time you want to use your RAID1 volume again you would have to trigger similar commands in a reversed order:

  1. mdadm to start the RAID1 volume again
  2. cryptsetup luksOpen to provide a decrypted block device in the /dev/mapper directory
  3. mount to make the RAID1 volume accessible

Since that’s kind of boring commands to type every time you want to use one of your RAID1 volumes and since there’s a chance to mess things up if you e.g. forget to stop the RAID1 volume before ejecting the disks from the docking stations I did create two simple helper scripts that can be used to automate those calls.

Helper scripts to simplify (un)mounting the encrypted RAID1 volumes

To be able to automated the (un)mounting process of your RAID1 volumes it makes sense to define a common base directory where each volume will be mounted using a dedicated subdirectory. I did choose the /mnt/usbRaid1 directory for that purpose. So beside initially creating this base directory you would have to create a new subdirectory for every newly created RAID1 volume:

1
2
3
mkdir -p /mnt/usbRaid1/raid1v1
touch /mnt/usbRaid1/raid1v1/not\ mounted\!
chmod 0444 /mnt/usbRaid1/raid1v1/not\ mounted\!

After creating the raid1v1 subdirectory inside the /mnt/usbRaid1 directory you’re ready to mount the RAID1 volume from this example. Before mounting it for the first time I’ve also created an empty “not mounted!” file and set its permissions to read-only so that you can instantly see if the RAID1 volume is mounted or not.

1
mount /dev/mapper/raid1v1 /mnt/usbRaid1/raid1v1

After mounting your RAID1 volume the “not mounted!” file isn’t visible any more since the /mnt/usbRaid1/raid1v1 directory now points to your encrypted RAID1 volume. At that point it’s pretty easy to simplify the (un)mounting process since we did use a consistent naming schema for all device, symlink and mount point names so that providing the volume name (raid1v1 in this example) to a Bash script is enough to trigger all actions inside the script. For that purpose I’ve created two Bash scripts called raid_open.sh <volume name> and raid_close.sh <volume name> that take care of opening a RAID1 volume and mounting it into the matching directory and closing the volume again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
#!/bin/sh

# raid_open.sh <raid volume>

# check user the script was called with
if [ "`whoami`" != "root" ]; then
  echo "ERROR: the script has the be called with root permissions!"
  exit 1
fi

# check number of given command line arguments
if [ $# -ne "1" ]; then
  echo "Usage: `basename $0` {raid}"
  exit 42
fi

RAID=$1
MD_DEVICE="/dev/md/$RAID"

# check that the given raid volume is available and in a clean state
if [ `mdadm --detail $MD_DEVICE | grep "State : clean" | wc -l` == "1" ]; then
  # open the partition on the raid volume and mount it
  cryptsetup luksOpen ${MD_DEVICE}p1 luks-$RAID
  if [ $? == "0" ]; then
    mount /dev/mapper/luks-$RAID /mnt/usbRaid1/$RAID
  else
    echo "ERROR: cryptsetup wasn't able to open the ${MD_DEVICE}p1 device!"
  fi
else
  # give some additional detail in case the raid volume isn't in a clean state
  MD_DEVICE=`readlink -f $MD_DEVICE`
  echo "ERROR: $MD_DEVICE (`readlink -f $MD_DEVICE`) either doesn't exists or is in an invalid state!"
  echo "please check the output of the /proc/mdstat content:"
  cat /proc/mdstat
fi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
#!/bin/sh

# raid_close.sh <raid volume>

# check user the script was called with
if [ "`whoami`" != "root" ]; then
  echo "ERROR: the script has the be called with root permissions!"
  exit 1
fi

# check number of given command line arguments
if [ $# -ne "1" ]; then
  echo "Usage: `basename $0` {raid}"
  exit 42
fi

RAID=$1
MD_DEVICE="/dev/md/$RAID"

# check if the given raid volumne is mounted
if [ `mount | grep $RAID | wc -l` == "1" ]; then
  umount /dev/mapper/luks-$RAID
  if [ $? == "0" ]; then
    # use cryptsetup to close the unmounted volumne
    cryptsetup luksClose luks-$RAID
    if [ $? == "0" ]; then
      # stop the underlying raid volume
      mdadm --stop /dev/md/$RAID
      if [ $? == "0" ]; then
        echo "Raid volume '$RAID' has been unmounted and stopped!"
      else
        echo "ERROR: mdadm wasn't able to stop the the '/dev/md/$1' raid volume!"
        echo "ERROR: check the following output of /proc/mdstat for further details!"
        cat /proc/mdstat 
      fi
    else
      echo "ERROR: cryptsetup wasn't able to close the 'luks-$1' volume!"
    fi
  else
    echo "ERROR: unmount failed for device $MD_DEVICE!"
  fi
else
  echo "ERROR: /dev/mapper/luks-$RAID isn't mounted!"
fi

So opening the raid1v1 encrypted RAID1 volume from this example is as simple as calling raid_open.sh raid1v1 which will internally call cryptsetup that will ask you for the password of the encrypted container. Closing an currently mounted RAID1 volume is also straight forward via calling raid_close.sh raid1v1. Both scripts provide an basic error handling and do print some additional information in case something goes wrong. Please not that the raid_open.sh script does use the example /mnt/usbRaid1 directory used to mount the RAID1 volumes into their matching subdirectories. If you want to use a different directory you need to change the script accordingly.

I do use my external RAID1 volumes in the described setup for quite a while now and I’m pretty happy with the usability of that solution. As usual there’s still ways to further improve this like e.g. using the automount daemon to trigger those scripts automatically combined with some sort of shared token for every RAID1 volume that you only have to authenticate once at e.g. the system startup so that you don’t have to type the LUKS containers password every time you want to mount a RAID1 volume. But compared to the usual mount and umount commands you’d trigger for normal disks the two scripts aren’t really a major drawback. Anyway, I just wanted to share my solution – maybe it’s considered being useful by some other Linux users out there 🙂

Tagged as:
Leave a comment
Comments (0) Trackbacks (0)

No comments yet.


Leave a comment

Connect with Facebook

No trackbacks yet.