Windows Media Player 12 shows wrong cover art when playing
TL;DR: see 2 possible solutions at the bottom
As noted by Nithin Philipps, the problem lies with the fact that all these MP3 were in the same folder and shared the same Folder.jpg and AlbumArtSmall.jpg. (I found this independently by using "Process Monitor" to find all the file access done by Windows Media Player and saw that it looked for a file named "Folder.jpg" and if it didn't find one (because I deleted it!) it created it (from the embedded MP3 cover) after 15s!)
I also found a page advising to delete these files, create new empty ones (0 bytes) and deny-write them by ACL. This way, Windows Media Player falls back to using the embedded art. ( http://lionsphil.livejournal.com/57126.html )
I protected my "downloads" folder this way as MP3 bought on VirginMega end up there. MP3 bought on Amazon end in different folders ("Artist Name"/"Album name"), so I didn't need to do anything for them.
I also found that Windows Media Player doesn't create the folder.jpg if the MP3 are on a network share (I guess it's because it can't hide them with the system attribute?), so when I get around to sort my new MP3 on my NAS, the problem will go away anyway.
So there are indeed two solutions for this problem:
"Protect" the folder containing the MP3 by creating empty Folder.jpg and AlbumArtSmall.jpg, and then denying write on them with ACL.
Put your MP3 on a Network Share (maybe a FAT drive would also work?). Using a RAIDed NAS to protect them is a good idea anyway, hard-drive crashes aren't nice!