What hash algorithms can I use in preseed's passwd/user-password-crypted entry?

You can use anything which is supported in the /etc/shadow file. The string given in the preseed file is just put into /etc/shadow. To create a salted password to make it more difficult just use mkpasswd with the salt option (-S):

mkpasswd -m sha-512 -S $(pwgen -ns 16 1) mypassword
$6$bLyz7jpb8S8gOpkV$FkQSm9YZt6SaMQM7LPhjJw6DFF7uXW.3HDQO.H/HxB83AnFuOCBRhgCK9EkdjtG0AWduRcnc0fI/39BjmL8Ee1

In the command above the salt is generated by pwgen.

Looking at the appropriate part of the debian-installer source code we can see that it simply calls usermod USER --password=CRYPTED-PASSWORD inside the target chroot.

Further usermod's manpage susggests that the --password option accepts "The encrypted password, as returned by crypt(3)." and that "The password will be written in the local /etc/passwd or /etc/shadow file.". This suggests we can only use the crypted password formats described in the crypt(3) man page.

All hope is not lost however. From the aforementioned man page we learn that crypt actually includes a salt field in the crypted password string, the format being $hash_id$salt$hash. So at least in principle it should be resistant against rainbow tables.

Apart from rainbow table attacks we still have to consider brute-force attacks. If we look at the glibc implementation of crypt we see that it actually implements password stretching using multiple rounds of SHA-512 not entirely unlike but, unfortunately, not using a standard approach such as PBKDF2.

Furthermore we see that we can actually control the number of hash rounds applied by crypt using an additional field in the crypted password ($rounds=$). Looking at the mkpasswd(1) man page we find this exposed as the -R option. Using this feature we can significantly raise the default number of rounds of 5000 (see ROUNDS_DEFAULT in the source code) which on my machine takes less than a couple of milliseconds to calculate to, say, 10 million which takes a couple of seconds instead:

> mkpasswd -R 10000000 -m sha-512 mypassword
$6$rounds=10000000$Rq30Hdd.0LzWq3x$XRXHvd5MnIi5MD2H8Jtn5W0cjvq4siGtUgWUaETc4QZyvuR4iY0Af.DoNfj1E6SvoHaVotAEjIiOPS3GvwJjM0