VSTS build agents - Can one computer run multiple build agents?

Yes you can run multiple agents in a single VM.

Make two directories say Agent1 and Agent2, extract the agent in each one of them and configure them with different names against your VSTS/TFS account.

It should work out of the box.

We run 4 agent jobs per machine concurrently with no issues. As mentioned above, should work out of the box. Just make sure you clean up directories. We have a script to do it every night