How to delete music as it's being played

Windows Media Player has an option to allow this. Go to Organize -> Options -> Library and check the option Delete files from computer when deleted from library

This option requires your music files to be part of Windows Media Player's library which by default means they need to be located in your user's Music folder (%UserProfile%\Music). If your music is stored elsewhere you can either move the files into your music folder, or add their location to WMP's library paths using Organize -> Manage Libraries -> Music.

With the option above enabled, you can now play your music from WMP's library and when you find a song you want to remove, just select it and press Delete. Playback will stop and the file will be deleted (to the recycle bin). To play the next song, simply press Enter. Alternatively, press CTRL+F before deleting the current song to start playing the next song, then press Delete

Note that WMP has a playlists feature that is visible as a side-bar (shown/hidden by the Play tab). This is separate from the library and unaffected by the obtion above. For the above option to work you need to select and delete the song from the library view, not the playlist view.

_{This method works in both Windows 7 and Windows 10.}

Well, good old Winamp has this feature. You can add your whole library, browse it through Winamp, and delete files directly from it.

To be honest though, I haven't used it in ages, so I don't know if the latest build (5.8) has it still.

I think you're doing a lot of unnecessary work by running your windows maximized. Resize your Media Player and Explorer windows so both are visible at the same time and not overlapping. Some versions of Windows have a "window snapping" feature that will let you easily dock one application to the left half of your screen and another to the right side.

When both programs are on-screen at the same time, you can process a folder quickly, without even needing to touch the mouse. In the Explorer window, press Enter to play the currently-selected file in WMP. Switch back to Explorer with Alt+Tab. Delete the song with Delete, or move to the next song with Down Arrow -> Enter. Use the left hand for Alt+Tab and the right hand for Enter, Delete, and Down Arrow. I've done this sort of thing before for sorting through photos, and I very quickly got into a rhythm (yay muscle memory) that let me burn through a large collection in no time. There's still some processing overhead for each file, but you should be able to do all of that while you're listening to the current file and deciding whether to keep it or not.

If it won't let you delete a file that's still playing, you may have to start the next one and then go back and delete it.