VSTS Online Building Setup Projects?

To Build .msi file by VS installer project, you can use Build VS Installer task in marketplace.

You can specify to build .sln or .vdproj to generate .msi file(s) in Task-mode option.

You have to configure your own build agent to run the build since Visual Studio Installer Projects extension is not installed on the Hosted & Hosted 2017 Agent.

Make sure the VS Installer Projects extension is installed on your own build agent and then you can build the setup project either use command line task with "devenv" or use the "Build VS Installer" task.

And if you get the "8000000A" error like following: Please following the instruction here to configure your build agent: Solution: An error occurred while validating. HRESULT = '8000000A'.

Then you should be able to build the setup project successfully: