Why is my systemd journal not persistent across reboots?

The reason for this behavior is that the machine identifier in /etc/machine-id changes at every reboot. This starts a new logging directory under /var/log/journal. Old logs can be viewed with the following command:

journalctl --merge

I'm still looking into the cause of the changing machine-id. Linode support is aware of the problem. I will update this answer when I know more.

---

UPDATE -- The root cause of the problem is simply that Linode zeroed out the contents of /etc/machine-id from their filesystem images. The result is the following chain of events:

1. The kernel loads and mounts the root filesystem read-only
2. systemd, run from the initial ramdisk, tries to read /etc/machine-id from the root filesystem (the file exists but has zero contents)
3. systemd cannot read the machine identifier, but can also not write a new one since the root filesystem is mounted read-only
4. systemd mounts tmpfs on /etc/machine-id (Yes, apparently you can mount a filesystem onto a file)
5. systemd invokes systemd-machine-id-setup which generates a random machine-id and stores it in the now-volatile /etc/machine-id
6. The system boots with a volatile machine identifier

You can check if your system has a volatile, rather than a permanent machine-id by looking at the output of mount:

$ mount | grep machine-id
tmpfs on /etc/machine-id type tmpfs (ro,mode=755)

The problem is easy to fix: simply write a persistent machine-id to the real /etc/machine-id. This is easier said than done, however, because you cannot unmount tmpfs from /etc/machine-id on a running system. These are the steps I took to fix it on Linode:

1. cp /etc/machine-id /etc/machine-id.copy, then poweroff the system
2. In the Linode Manager, go to the tab Rescue and boot into rescue mode
3. Access the system via the Lish console
4. Mount the root filesystem: mount /dev/xvda /mnt
5. Move the copy created in step 1 to the real machine-id: mv /etc/machine-id.copy /etc/machine-id
6. Reboot

Such are the consequences of a missing machine-id at boot. I hope this will help a random passer-by in the future.