openssl decryption in javascript in browser
As I was pointing out in a comment to your question, the vector of attack you're envisaging (compromised server) implies that the JavaScript is likely to be compromised too, in which case the JavaScript code running on the client shouldn't be trusted anyway. (It would be quite easy to make the JavaScript send the deciphered data back to the server with an asynchronous request in the background: again, since the server would be under the attacker's control, there wouldn't be any need for tricks to circumvent same-origin policies there.)
I would suggest going down the route of a standalone application (such as Java WebStart), perhaps signed (with a private key that's not held on the server).
If you're still willing to go ahead with this sort of architecture, avoid releasing the user's private key into the JavaScript at all cost. This could compromise the user's private key, not just the encrypted data.
When you use a private key in your browser for SSL/TLS client-certificate authentication, the private key isn't exposed to any code used by the server. It's used by the browser for the handshake, and the server gets the certificate (which is public), but the private key doesn't go anywhere near what the HTML+JS code can see. (In fact, in OSX with Safari, the private key is used by the underlying SSL/TLS library, and not even exposed to the user process.)
The JavaScript libraries for RSA that I've seen require direct use of the private key, that is, they need to be able to use the private exponent directly. That's clearly not good if you're in a situation you can't trust the server.
Being able to use a private key within the browser for RSA operations, without letting the script get hold of the private material itself would require tighter integration with the browser, in particular, some API to sign and decipher that would use these functions directly in the browser's security mechanism, without exposing the private key material (overall, a similar approach to what PKCS#11 offers to applications using it).
As far as I'm aware, the current Mozilla crypto JavaScript API doesn't provide functions to decipher/sign using the browsers (it's only for certificate request and key generation). There seems to be plans to do this, though:
- https://wiki.mozilla.org/Privacy/Features/DOMCryptAPISpec/Latest
- http://mozilla.ddahl.com/domcrypt/demos/demo.html
On the IE plaform, CAPICOM should have been of interest, but it seems to be deprecated nowadays.