gpg-agent refuses SSH keys with ssh-add reporting "agent refused operation"

The answer was apparently to run:

echo UPDATESTARTUPTTY | gpg-connect-agent

I have no idea why the pinentry program worked fine for other uses such as decrypting files, but didn't work for ssh-add.

While this now works, it also makes a copy of the ssh private key that doesn't show up under gpg -Kv, and furthermore doesn't seem to allow you to change the passphrase on your private key (since you can't edit it with --edit-key). Basically I'm pretty unhappy with the way gpg-agent provides low visibility into where your secrets are being copied. If you hit this question because you hoped gpg-agent might be a better alternative to ssh-agent, then I'd encourage you to stick to ssh-agent instead of trying out my answer. The main reason to prefer gpg-agent is if you need to for smart-card use.

In my case, the problem was the pinentry program used. I was using pinentry-emacs. It seems it doesn't handle the dual text field entry window prompt that ssh-add triggers when used with gpg-agent.

Removing pinentry-emacs and installing the GTK pinentry solved the problem here.