How can I get GPG Agent to cache my password?

In addition to setting up the cache times in gpg-agent.conf, you also have to make sure GnuPG is actually interfacing the gpg-agent. GnuPG 2 and upwards generally does, but the GnuPG 1 branch does not. By default git is using the gpg binary, which (at the time of writing this answer) still is GnuPG 1, while GnuPG 2 is installed as gpg2 on most systems.

In the end, you have two possibilities:

• set up git to use gpg2 by changing the git configuration:

  git config --global gpg.program gpg2
  

• set up gpg(GnuPG 1) to use gpg-agent by adding use-agent to gpg.conf