CoreOS: tcpdump mysteriously solves network issue (excessive number of sockets used)

First of all, thank you for very well written question!

As the level of detail you described is very high and you are already on gdb level, I assume my answer won't be much of use for you. Anyway, here's a try:

• Presumably you already tried something like ss -ae and lsof -n?
• Does dmesg return anything interesting when this happens?
• Do you use iptables on the server?
• If you set the promiscuous mode using some other way than tcpdump (say, ip link set [interface] promisc on), does this also fix the problem?
• Have you checked for any suspicious processes, files or other weird activity? Just thinking that maybe some uninvited nasty process is lurking in the shadows hiding itself, and goes silent whenever promiscuous mode is set?
• If you leave tcpdump running background, will this problem return?

I hope this helps.