Reason to keep a GPU in a computer that is running Ubuntu Server

Graphics cards help with:

• CUDA/OpenCL style workloads (eg bitcoin mining)
• 3D rendering. You might be rendering stuff on demand.
• Most cards can help with video decoding but this applies to many IGPs too.
• Some cards have hardware to help with video encoding. The quality is usually a lot poorer and this is still under development. ffmpeg can use nvenc (obviously on a Nvidia card). I'm not sure if this applies to you.

Almost everything else will only go through the CPU. If you're not doing any of the above (in a way that uses the GPU) you might as well pull the card.

But hang on a second, you can also play games on Ubuntu. The AMD drivers are occasionally infuriating, but that's potentially an option. If you're planning on running a media centre, your onboard graphics might be enough but we use a low-end Nvidia card to help in ours.

Yes if the server generates or, for instance, re-sizes images. Software that uses, or could use, the existing graphic card for such tasks, is not unheard of.

Surely, this depends a lot on what kind of software, what kind of GPU, is the suitable driver available and things the like.