Why my favicon isn't updating / changing?
After much fiddling, I have finally got it to work. Posting an answer here for anyone with this problem in the future.
If you're having this problem, follow these steps:
Go to your Magento Admin Page, to the favicon settings, located here:
System > Config > General > Design > HTML Head > Favicon Icon
Delete your current favicon by checking the box and clicking Save Config.
Visit your homepage, preferably from a browser without any of your site's cache. I used Google Chrome's Guest feature.
On your site, press Ctrl + U (or F12, you just want to view the site's code).
Find the line in the header that points to the favicon (should now be Magento's default favicon). For me, it was located in
/skin/frontend/base/default
Replace the favicon.ico with another image. (I used another .ico, not sure if other formats will be friendly here.)
Clear your browser's cache and refresh.