Standards for encrypting passwords in configuration files?
If you are running a server that needs a password for access to some remote service, then there's no good solution. Storing the password in a configuration file stored in a suitable location is probably making the best of a set of poor choices.
The choices are: have a user enter in the password at boot time (this is bad, because if your server gets rebooted, now the server is unreachable until a person can physically walk up to the server and enter the password); hardcode the password into the source code of the server code (this is much worse than putting it in a configuration file); or store the password in a configuration file (the least bad of all options). The main thing to watch out for with the configuration file, is to make sure it is stored outside of your web root and that its file permissions are locked down appropriately.
Encrypting the password that's stored in the configuration file doesn't help, because now where do you store the decryption key? You've just shifted the problem around. "It's turtles all the way down" is not an acceptable response.
On Windows, another option you might look at is storing the password in DPAPI. This helps a little bit for desktop applications, but is not very useful for unattended servers.
For more, read the following questions on this site: Store a password to avoid user interaction, where to store a key for encryption, How do open source projects handle secure artifacts?, How can I decrypt data with Java, without hard-coding the key?, Password in file .php, Where do I securely store the key for a system where the source is visible?.
P.S. In principle, you could use a TPM to store the password -- but in practice, this gets you into bleeding-edge issues, so it's probably not viable for most folks.
So the following was a bit too long for a comment...
Perhaps taking one step back and comparing benefits of preventative and detective controls might help. Preventative controls include encryption but you could also encode the password to make it less obvious. This approach is meant to protect the password from accidental sharing (a b32 encoding would produce less meaningful characters (b32 produces longer string than b64). Such an approach just increases the difficulty of memorizing the random sequence of numbers as well as the method that should be used to decode the string. Base32/64 encoding is a simple way of protecting passwords that do not require additional logic to be built/maintained.
The other approaches to preventative controls would likely use encryption. There are many different ways to protect the key. Without getting into the details or reiterating what D.W. already posted, you can layer detective controls to improve the security posture. For example, you can audit access to a file that contains the key. You can correlate events (such as restarting of server/service) with access to the key file. Any other access request (successful or not) to the key file could indicate abnormal activity.
To get to your questions, here's my take:
if you have to store password in a config file, I'd recommend at least encoding the password where possible. Encoding the password would reduce the chance that it's leaked in the event someone scrolls through the file, say with a vendor support rep watching. Encrypting the password is much safer, but that requires additional complexity.
How to deal with the fact that a key has to be hard coded or stored in another file. Well, separating the encryption key in another file increases the difficulty for someone to view the key. For example, you can use access control to limit access to the key file but still maintain a more open ACL for the config file. Similarly, you can implement auditing for access to the key file, which you can use to correlate back to events that require use of key. Hard coding the key may be fine if you limit access to the binary. Careful encoding of password can easily be detected by running a "strings" against the binary. You can encode/encrypt the hard coded password (i.e. require a separate function (perhaps with auditing) when to decode the password, but that increases the complexity for the developer and admin (i.e. how does one change the key without rebuilding/recompiling the binary?).
Should the encryption keys for password be human readable? It depends. There's only a limited number of ways to protect the key. An encryption key is usually seen as an alphanumeric string that's hard to commit to to memory. You can always encode/encrypt the key, but such methods doesn't deter someone that's smart enough to take a screen shot. However, you can use simple "keys" (more like passwords) as input to a key expansion function. In those instances, perhaps additional measures such as encoding adds some additional value relative to the cost of complexity.
If anything, a good approach is to implement multiple layers of controls. Preventative controls are more difficult while detective controls are generally easier to implement. Separating key files could simplify the overall architecture and implementation of controls. Regardless of whether preventative or detective controls are used, enabling some auditing functions is a must along with review of audit logs. This way, should the improbable happen (access to key), you can take corrective action.
Store the password in a (non-session) O/S user environment variable, and run the program as that user.
Advantages:
You will never accidentally check the password into source control because there's no file to check in.
If you screw up file permissions on your config file (which never happens right?), your passwords are not compromised
Can only be read by user or root (same as a config file, same as a private key protecting an encrypted file)
If you're encrypting the file, how are you securing your key?
Clear your envvars before starting new processes though, as they may be passed through
One advantage to an HSM is that while user or root can use the HSM to decrypt a value, they can never get the key inside it.