Cannot register a build agent in VSTS

To fix the issue, I had to go to Administrative Tools > Services, and change the VSTS Agent service account from Network service to Local system.

The service was then able to start and work as expected.

I found the answer. The authentication failure was a red herring. The actual problem is that it was waiting for user input, but powershell was not writing out the question and waiting for the answer. Here is the line in the log file:

[2017-07-11 14:03:20Z INFO Terminal] WRITE: Enter work folder (press enter for _work) > 
[2017-07-11 14:03:20Z INFO Terminal] READ LINE

In order to get round this problem, I made sure I specified all the required arguments and used --unattended to ensure it wasn't going to ask me anything else. In the end it was like this:

PS>.\config.cmd --url $url --auth 'PAT' --token $patKey --pool $poolId --agent $serverId --work '_work' --runasservice --unattended