What is the best way to transmit password hash over unreliable TLS?

You can't.

There is no secure way to transmit it if you can't authenticate the server and establish a secure channel. If you can't authenticate the server then you can't be sure if you are talking to the server or the attacker. Hence, even if you use TLS to establish a secure channel, it doesn't protect you at all since you could be talking to the attacker while thinking that you are talking to the server.

The certificate is essential. Without it, or some knowledge of the public key of the server, you cannot be sure that you are really communicating to the server as an attacker could act as a man in the middle between you and the server.

TLS use Diffie-Hellman for the key exchange. If you look at the security section of the wikipedia page you will find the following attack which is exactly what you are vulnerable to.

In the original description, the Diffie–Hellman exchange by itself does not provide authentication of the communicating parties and is thus vulnerable to a man-in-the-middle attack. Mallory may establish two distinct key exchanges, one with Alice and the other with Bob, effectively masquerading as Alice to Bob, and vice versa, allowing her to decrypt, then re-encrypt, the messages passed between them.

Conclusion

You need to have a valid certificate to be able to authenticate the server or you will be vulnerable to man in the middle attack. An easy way to achieve that is to include the server certificate in your application which is also known as certificate pinning.


You should look into Certificate pinning.

This is effectively allowing you to trust your self-signed certificate, and that server certificate only from your client by a hard lookup of the public key of the certificate. So the chain of authority is not followed (which is where a self-signed certificate falls down) and instead the public key is validated by the client application to be trusted directly.

You should not look at hashing, or anything else on the application layer as it will be inherently susceptible to a MITM attack.


Bundle your self created root CA certificate with your application and configure it to reject all untrusted certificates. If you only use your own CA cert, this is arguably more secure than relying on the normal CAs, since CAs have been compromised in the past by attackers, and are subject to government authorities and politics as well.

Rolling your own authentication scheme is foolhardy, and is almost guaranteed to be less secure than simply using TLS with a bundled certificate.