Is there a keyboard shortcut to "unselect" in Windows Explorer?

You can use Ctrl+Arrow to move among selected items. Ctrl+Space will select/deselect. (Space alone will select but not deselect.)

As far as I know there isn't a keyboard shortcut for Invert Selection in Windows Explorer, and unfortunately Select All doesn't toggle the selection.

So if you really want an all keyboard solution:

Ctrl + A, Alt + E, I

If you're used to Emacs multi-key bindings then this shouldn't bother you. :)

I also found another quick solution: pressing F5 will refresh the folder and deselect single or multiple items.