How to generate certificate if private key is in HSM?

I've seen some HSMs come with their own support programs ready to use for a variety of things including key generation and cert or CSR creation, or make them available (for example on the vendor website); did you check for that? Assuming that isn't a (good) option:

You don't identify your HSM; there are thousands of models with at least hundreds of interfaces, although a significant fraction (not nearly all) use the "standard" (mostly) PKCS#11 interface.

OpenSSL has a fairly open-ended ENGINE API that redirects some subsets of cryptographic functions -- such as privatekey generation, signing and decryption -- to an implementation other than OpenSSL's normal one -- such as an HSM. OpenSSL comes with a few engines builtin -- at least by default; a particular build (such as the package for a Linux distribution) may omit the builtin engines, in which case you may need to do your own build. Other people can also write engine modules, including but not limited to a maker or supplier of a particular HSM model or line and including you.

1. If your HSM is one of those with a builtin engine, configure that engine as needed according to its documentation, and use the appropriate -engine id or -*engine id options to your openssl (sub)commands.

Note that req -new generates a certificate only with -x509, usually plus some related arguments, and a cert generated this way is selfsigned; otherwise req -new generates a Certificate Signing Request (CSR) that you then get a CA to "convert" into a CA-signed "real" certificate (the PKCS#10 CSR itself is always selfsigned). req can also generate a new privatekey instead of using an existing one, and this generation can be "in" the engine thus on the HSM.

2. If your HSM does not have a builtin engine but does have an engine from some other party, install it to your system. This is the case for PKCS#11. This may require changing your version of OpenSSL to one supported by the engine. Then proceed as in 1: configure the engine and use it.

3. If your HSM does not have any engine but its API provides the operations OpenSSL wants in an engine, you can write (and debug!) an engine module for it. Then proceed as in 1 using your engine. Also consider offering your engine module to the world, the community using this type of HSM, and/or the OpenSSL project.

4. If your HSM's capabilities don't fit into the engine API, or no engine module exists and you don't want to create one, you can instead write your own program, using some (perhaps much) of the code from openssl/apps/req.c, to generate a privatekey and/or use an existing one on your HSM, build the data structures for a certificate (or CSR) within OpenSSL in more or less the existing way, but then give X509_[REQ_]sign_ctx an EVP_PKEY (the polymorphic-in-C data structure used in OpenSSL for various types of asymmetric keys) that you have set up with custom methods that use your HSM's API (and some identification of the key on the HSM) to do the signing. Make sure any licensing on your program is compatible with the OpenSSL license (which is essentially BSD-advertising style).

Most HSMs have a PKCS#11 driver. You can use OpenSC's engine_pkcs11 to enable OpenSSL to leverage an HSM (or a smartcard). There's a document on how to use it to create a request.

You'll need to install and configure the PKCS#11 middleware (driver), compile the engine_pkcs11 module, and some adaptation of the instructions above are probably needed.