How to configure iGPU for xserver and nvidia GPU for CUDA work

I first installed NVIDIA drivers and CUDA packages following this guide. Except, after a reboot I ended up with /usr/lib/xorg/Xorg showing up in the output of nvidia-smi. This wasn't good, since I needed to have all of NVIDIA GPU RAM available to my work.

After some research I found a solution that solved my problem:

I created /etc/X11/xorg.conf with the following content:

Section "Device"
    Identifier      "intel"
    Driver          "intel"
    BusId           "PCI:0:2:0"
EndSection

Section "Screen"
    Identifier      "intel"
    Device          "intel"
EndSection

(if you try to do the same, make sure to check where your GPU is. Mine was on 00:02.0 which translates to PCI:0:2:0)

% lspci  | grep VGA
00:02.0 VGA compatible controller: Intel Corporation Device 3e92
01:00.0 VGA compatible controller: NVIDIA Corporation GP104 (rev a1)

After rebooting, xorg and other programs no longer appeared in the output of nvidia-smi. And I was able to use pytorch with CUDA-10.0.

Note, that I still have all the NVIDIA drivers installed, but they don't interfere.

---

update: for Ubuntu 20.04 some extra changes are needed for this to work. You will find the full details here.