Ubuntu 16.04, CUDA 8 - CUDA driver version is insufficient for CUDA runtime version

Running

sudo apt-get purge nvidia-*

and reinstalling the drivers using

sudo apt-get install nvidia-375

solved it. Just for the record, the first time I updated the drivers using the GUI (Additional Drivers tab in Software & Updates).

First, check "CUDA Toolkit and Compatible Driver Versions" from here, and make sure that your cuda toolkit version is compatible with your cuda-driver version， e.g. if your driver version is nvidia-390, your cuda version must lower than CUDA 9.1.
Then, back to this issue. This issue is caused by "your cuda-driver version doesn't match your cuda version, and your CUDA local version may also different from the CUDA runtime version(cuda version in some specific virtual environments)."
I met the same issue when I tried to run tensorflow-gpu under the environment of "tensorflow_gpuenv" created by conda, and tried to test whether the "gpu:0" device worked. My driver version is nvidia-390 and I've already install cuda 9.0, so it doesn't make sense that raising that weird issue. I finally found the reason that the cuda version in the conda virtual environment is cuda 9.2 which isn't compatible with nvidia-390. I solved the issue by following steps in ubuntu 18.04:

• check cuda driver version
  ~$ nvidia-smi or ~$ cat /proc/driver/nvidia/version
• check local cuda version
  ~$ nvcc --version or ~$ cat /usr/local/cuda/version.txt

• check local cudnn version
  ~$ cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2

  ---

• check cuda version in virtual environment
  ~$ conda list you can see something like these :
  cudatoolkit      9.2       0
  cudnn        7.3.1      cuda9.2_0
  you may find that the cuda version in virtual environment is different from the local cuda version, and isn't compatible with driver version nvidia-390.

So reinstall cuda in the virtual environment:

• reinstall cuda : ~$ conda install cudatoolkit=8.0
  (change the version number '8.0' to other version number which match your driver version, and your cudnn version will update automatically to match the new version cuda )

I have followed the instructions on this page, and it works for me.

https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=debnetwork

First, download installer for Linux Ubuntu 16.04 x86_64.

Next, follow these steps to install Linux Ubuntu:

1. sudo dpkg -i cuda-repo-ubuntu1604_9.2.148-1_amd64.deb

2. sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

3. sudo apt-get update

4. sudo apt-get install cuda