Which protocols exist for end-to-end encrypted group chat?

Let me try to sum up what the landscape of end-to-end encrypted messaging protocols for group chat looks like:

• Protocols like PGP have been around for some time and offer "group messaging" by simply encrypting the content with a randomly generated symmetric key and then encrypting that key asymmetrically with the public keys of each of the recipients. These protocols only sends the encrypted content once but encrypts the encryption key to each of the members of the group. Note that similarly to PGP, this approach does not provide any perfect forward secrecy, deniability or conversation integrity (and thus no transcript consistency).

• OTR was introduced to address some of the shortcomings of PGP, improving on perfect forward secrecy, conversation integrity and deniability. Ian Goldberg, the author of OTR also wrote a paper on a multi-party variant of the protocol, named mpOTR. mpOTR was designed with the XMPP transport in mind and inherently synchronous in its design, meaning that each group member is expected to be online at any time to negotiate new keying material. The described protocol does not provide in-session perfect forward secrecy and has not been largely deployed. N+1Sec is a similar protocol with some improvements. Note that these protocols have a lot of algorithmic complexity and tend to scale badly, especially when you add latency into the mix.

• Then you have a whole class of protocols, that we simply call N times protocols because they are just sending each message . These protocols have the advantage of reusing an existing one-to-one protocol, which is really convenient when you already have a channel that gives you nice features such as asynchronous perfect forward secrecy. The group structure is not a cryptographic concept in this case, losing on the cryptographic guarantees but lowering algorithmic complexity. The Open Whisper Systems blog has a great post about why Signal does this instead of mpOTR-style messaging. This class of protocols violates your second requirement since they are what we call “client-side fan-out” where the client encrypts and sends out all of the different messages.

• There exists an optimisation on Signal's which was adopted by WhatsApp and that you can find in their whitepaper called Sender Keys that has “server-side fan-out”. It uses N times on setup, but after the first message, each member of the group can send a single messages to the group. This protocol has perfect forward secrecy by using a hash ratchet (but does not provide perfect future secrecy). Transcript consistency is enforced by the server-side (because server-side fan-out), but not from a cryptographic perspective.

These are the types of protocols that I've seen being implemented. There are challenges, both in usability and crypto research on how to combine asynchronosity with perfect future secrecy and transcript consistency in the group setting.

If you want a protocol that answers both of your requirements, I think something like the Sender Keys variant of the Signal protocol is what you're looking for.

Maybe Asynchronous Ratcheting Tree is what you are asking for.

You can also take a look at Messaging Layer Security:

Messaging Layer Security (MLS) is an IETF working group building a modern, efficient, secure group messaging protocol.