Best order of RAID, LVM and LUKS

First, the order of LUKS and LVM depends on if you want to have different LUKS passwords or other settings for different LVs. If say, you need to set up different passwords for different LVs, you definitely need to put LUKS on top of LVM. On the other hand, if all LVs share the same password and settings like keylen, you'd want to have LUKS below LVM, so that you don't have to deal with the overhead of having more than one LUKS partition (think what you would have to do when you need to change the password).

Second, you almost always want RAID to be the lowest level, so that when one disk dies, it can be swapped easily and transparently. If you were to set up RAID on top of LVM, you'd have to replace a PV when one disk dies, that would be a big pain in the neck. Also RAID on top of LVM would totally defeat the flexibility of LVM. You then probably need to set up the second layer of LVM on top of RAID again!

Therefore, because in most cases people just need to use a single password, this would suffice:

RAID --> LUKS --> LVM --> ext4

In some cases, you might need to use LVM to combine multiple RAID devices into a large volume, then you can do:

RAID --> LVM --> LUKS (--> LVM) --> ext4

Theoretically the order shouldn't affect performance much, if every layer is set up correctly, and in practice I haven't seen this setup has a particularly bad performance. The most important thing is probably the alignment:

  1. make sure your partitions are 1MB aligned (very important for SSD);
  2. for RAID layer, pick chunk size wisely;
  3. for LVM, make sure you set --dataalignment to match the RAID chunk size (this might be helpful).

Also if on SSD, make sure you enable LUKS TRIM/DISCARD pass-through by adding rd.luks.options=discard to /etc/default/grub and discard to /etc/crypttab (These are what I do on Red Hat/Fedora Linux. Might be a little different on Debian.) LVM and RAID should automatically support discard if you use a new-ish kernel.

Of course, these are just general guidelines. If you have special needs, feel free to update your question or comment here.


If you want all RAID, LUKS and LVM I would recommend RAID -> LUKS -> LVM -> FS. RAID --> LVM --> LUKS --> LVM --> FS is no better than RAID -> LUKS -> LVM - to extend volumes just add more RAID -> LUKS devices to a volume group.

RAID --> LVM --> LUKS --> FS - encrypting only particular Logical volumes has a feature of not encrypting everything by default (may be seen as advantage or disadvantage) but it will make extending root FS easier.

Extending LUKS on top of logical volumes is a common source of problems when users extend/resize them in the wrong order. Having LUKS on the whole md RAID device will simplify resizing - add new md device, create LUKS on top of that, add device to /etc/crypttab (at least on Fedora and RHEL clones) and extend your volume group. If root FS is on the volume group, you will need to add another rd.luks.uuid entry to kernel cmdline (edit /etc/default/grub and regenerate grub.cfg.)

LUKS -> RAID is usually wrong - data will be encrypted multiple times, consuming more CPU cycles with no gain. There is also a chance of mistakenly replacing a failed disk with a new one without setting up LUKS when a disk fails.

Extending and shrinking:

When extending always go from the bottom of the stack, when shrinking from the top.

Example:

Extending RAID -> LVM -> LUKS -> FS (the first two step are optional if there is enough free space in the volume group):

  1. Add new disks and create md RAID.
  2. Add mdX device to volume group.
  3. Extend logical volume.
  4. Extend LUKS device.
  5. Extend FS.

Shrinking RAID -> LVM -> LUKS -> FS:

  1. Shrink FS.
  2. Shrink LUKS device.
  3. Shrink logical volume.