How to roll my own security mechanism - avoid SSL

You're confusing SSL/TLS with its most common usage pattern, which is TLS in conjunction with X.509 certificates and a Public Key Infrastructure (PKI, RFC 5280).

While it's really important to secure a TLS connection by authenticating its server site (to prevent active MITM attacks), the TLS specification doesn't actually mandate to do it using X.509 certificates, even less using a PKI.

As @jathanism said, you can choose not to rely on a hierarchical PKI, by using a self-signed X.509 certificate. You will have to establish trust in the server certificate by some other means.

In addition, TLS can also be used with OpenPGP certificates, Kerberos or Pre-Shared Keys.

TLS itself has had very few known bugs as far as I know, e.g. there was the renegotiation bug (CVE-2009-3555), which has been fixed (more recently, there were also certain issues with certain block ciphers, which can be fixed by choosing better cipher suites, or upgrading to newer versions of TLS).

The issues you're talking about (and often discussed in the press) tend to be related to PKIs, i.e. how to establish trust in the remote party. How this should be done isn't dictated by the TLS specification itself.

If you want to try to find a better way, by all means do some research, but I'd suggest focussing on this part and leaving SSL/TLS and the cryptography side of things alone. You'll probably find that the biggest problems regarding PKIs and their alternatives are administrative, not so much technical ones.

The other aspect of authenticating the remote party is to check you're talking to the intended party (i.e. host name verification).

If you have control of both sides, use self-signed SSL certificates. Done. You're not trusting a root Certificate Authority, and therefore your crypto becomes almost impossible to crack without internal knowledge of the architecture.

I have decided to go with SSL as in the end it ended up being the simplest approach - the other idea just wasn't working well on the WCF side.

I have written this up on code-project to help others out having a similar problem.

http://www.codeproject.com/KB/WCF/WCFJsonRestHttpSecureRole.aspx

I'd still need to figure out how to trust only my own public key and no CA keys - looking at the as3cryptolib it might be possible with the CAStore property on the TLSConfig - http://code.google.com/p/as3crypto/source/browse/trunk/as3crypto/src/com/hurlant/crypto/tls/TLSConfig.as