Unknown malware, how to report it and whom to report it to?
If you can find the offending executable or dll, one thing you can do is to upload it at https://www.virustotal.com. Try even the svchost file if you feel it's suspicious.
It will show you how many antivirus engines detect it out of a huge list, and will also forward your sample, if it fails to be detected, to antivirus companies for further processing and signature development.
Many antivirus vendors will more than happily examine a file - as an example, this is Comodo's upload link. Microsoft will also take a look.
I've never actually submitted malware to an antivirus vendor, but I'm pretty certain if it is new malware you won't get analysis done there and then if you see what I mean - you'll probably get an "in database, does this" or "not in database, we'll take a look" type result, since automatically analysing malware is quite hard except for trivial code. I'm not sure even if it was a new piece of malware that you would get feedback, but the result would be an added detection signature and potentially patched exploits if they are unknown, so it is absolutely worth doing to be part of the solution, if you like.
In terms of exploring what it does in greater detail, windbg and process monitor are great companion tools to process explorer and autoruns is an excellent tool for examining startup entry points for changes.
The standard advice with rootkits is a clean install from trusted media since it is pretty hard to know, for certain, you've got rid of it, so if it comes back, that is my advice, along with quarantining the system from others on the network to be sure it doesn't begin exploiting those.
If it is a rootkit and a well written one it may be able to evade certain detection tools. Often, variants of malware are released after the exploit mechanism they used is patched, so you could be in that window and that may explain why a given malware detector doesn't work for the malware it is supposed to detect.
how to get some expert to analyze my system in order to find what kind of beast it is, so that it may be stopped before it goes out in the wild... if it already hasn't.
If you have it on your home system then by definition it is in the wild. So, too late for that. Unfortunatly for analyzing, you may have destroyed an important component in the MBR.
In my opinion the best way to analyze the malware on your system would be to disconnect it from the network. Use a cold boot technique to image the RAM. Power it down, remove the hard drive, and then image the hard drive drive. Image any flash on the motherboard, PCI cards, or anything else the provides persistant writable storage. Then analysis begins! Of course you would need to keep the system hardware available in an unused state in case the researcher wants to verify a theory.
Or you could do as @Ninefingers suggests, seems a lot simpler that way.