Solving the version conflicts between the Nvidia driver and CUDA toolkit

Navigating Nvidia GPU drivers and CUDA development software can be challenging. Upgrading CUDA versions or updating the Linux system may lead to issues such as GPU driver corruption. In such situations, we often encounter questions that require online searches for solutions, which can take time and effort.

Some questions related to Nvidia driver and CUDA failures include:

A) The following packages have unmet dependencies:
cuda-drivers-535 : Depends: nvidia-dkms-535 (>= 535.161.08)
Depends: nvidia-driver-535 (>= 535.161.08) but it is not going to be installed

B) UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g., changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.) Reboot after installing CUDA.

C) NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

After going through this time-consuming process, I realized that having a deeper understanding of the intricate relationship between CUDA and Nvidia drivers could have enabled me to resolve the driver corruption issue more swiftly. This realization underscores the importance of acquiring comprehensive knowledge about the interplay between software components and hardware drivers, which can greatly streamline troubleshooting processes and enhance system maintenance efficiency. In this post, I will try to clarify the concepts of GPU driver and CUDA version, and other related questions.

What is CUDA?
CUDA, short for Compute Unified Device Architecture, is a groundbreaking parallel computing platform and application programming interface (API) model developed by NVIDIA. This powerful technology extends the capabilities of NVIDIA GPUs (Graphics Processing Units) far beyond traditional graphics rendering, allowing them to perform a wide range of general-purpose processing tasks with remarkable efficiency.

Key Components of CUDA:
CUDA Toolkit: NVIDIA provides a comprehensive development environment through the CUDA Toolkit. This includes an array of tools and resources such as libraries, development tools, compilers (like nvcc), and runtime APIs, all designed to help developers build and optimize GPU-accelerated applications.

CUDA C/C++: CUDA extends the C and C++ programming languages with special keywords and constructs, enabling developers to write code that runs on both the CPU and the GPU. This dual capability allows for offloading computationally intensive and parallelizable sections of the code to the GPU, significantly boosting performance for various applications.

Runtime API: The CUDA runtime API offers developers a suite of functions to manage GPU resources. This includes device management, memory allocation on the GPU, launching kernels (which are parallel functions executed on the GPU), and synchronizing operations between the CPU and GPU. This API simplifies the development process by abstracting the complexities of direct GPU programming.

GPU Architecture: At the heart of CUDA's power is the parallel architecture of NVIDIA GPUs. These GPUs are equipped with thousands of cores capable of executing multiple computations simultaneously. CUDA leverages this massive parallelism to accelerate a broad spectrum of tasks, from scientific simulations and data analytics to image processing and deep learning.

CUDA transforms NVIDIA GPUs into versatile, high-performance computing engines that can handle a diverse range of computational tasks, making it an essential tool for developers seeking to harness the full potential of modern GPUs.

NVCC and NVIDIA-SMI: Key Tools in the CUDA Ecosystem
In the CUDA ecosystem, two critical command-line tools are nvcc, the NVIDIA CUDA Compiler, and nvidia-smi, the NVIDIA System Management Interface. Understanding their roles and how they interact with different versions of CUDA is essential for effectively managing and developing with NVIDIA GPUs.

NVCC (NVIDIA CUDA Compiler):
nvcc is the compiler specifically designed for CUDA applications. It allows developers to compile programs that utilize GPU acceleration, transforming CUDA code into executable binaries that run on NVIDIA GPUs. This tool is bundled with the CUDA Toolkit, providing a comprehensive environment for developing CUDA-accelerated software.

NVIDIA-SMI (NVIDIA System Management Interface):
nvidia-smi is a command-line utility provided by NVIDIA to monitor and manage GPU devices. It offers insights into GPU performance, memory usage, and other vital metrics, making it an indispensable tool for managing and optimizing GPU resources. This utility is installed alongside the GPU driver, ensuring it is readily available for system monitoring and management.
**
CUDA Versions and Compatibility**
CUDA includes two APIs: the runtime API and the driver API.

Runtime API: The version reported by nvcc corresponds to the CUDA runtime API. This API is included with the CUDA Toolkit Installer, which means nvcc reports the version of CUDA that was installed with this toolkit.

Driver API: The version displayed by nvidia-smi corresponds to the CUDA driver API. This API is installed as part of the GPU driver package, which is why nvidia-smi reflects the version of CUDA supported by the installed driver.

It's important to note that these two versions can differ. For instance, if nvcc and the driver are installed separately or different versions of CUDA are installed on the system, nvcc and nvidia-smi might report different CUDA versions.

Managing CUDA Installations
Driver and Runtime API Installations:

The driver API is typically installed with the GPU driver. This means that nvidia-smi is available as soon as the GPU driver is installed.
The runtime API and nvcc are included in the CUDA Toolkit, which can be installed independently of the GPU driver. This allows developers to work with CUDA even without a GPU, although it is mainly for coding and not for actual GPU execution.
Version Compatibility:

The CUDA driver API is generally backward compatible, meaning it supports older versions of CUDA that nvcc might report. This flexibility allows for the coexistence of multiple CUDA versions on a single machine, providing the option to choose the appropriate version for different projects.
It's essential to ensure that the driver API version is equal to or greater than the runtime API version to maintain compatibility and avoid potential conflicts.

The compatibility of CUDA version and GPU version can be found from table 3 in https://docs.nvidia.com/deploy/cuda-compatibility/index.html .

Install different CUDA versions
Here are all the CUDA versions for installation:
https://developer.nvidia.com/cuda-toolkit-archive
Let us use CUDA Toolkit 12.0 as an example:

Very Important for the last option of Installer Type: runfile (local)
If you chose other options like deb, it may reinstall the old driver, and uninstall your newer GPU driver. But runfile will give you an option during the installation to skip updating the GPU driver, so you may keep your newer drivers. This is very important for case you have already installed the GPU driver separately.

Install GPU Drivers

sudo apt search nvidia-driver
sudo apt install nvidia-driver-510
sudo reboot
Nvidia-smi

Multiple CUDA Version Switching
To begin with, you need to set up the CUDA environment variables for the actual version in use. Open the .bashrc file (vim ~/.bashrc) and add the following statements:

CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"

This indicates that when CUDA is required, the system will search in the /usr/local/cuda directory. However, the CUDA installations typically include version numbers, such as cuda-11.0. So, what should we do? Here comes the need to create symbolic links. The command for creating symbolic links is as follows:

sudo ln -s /usr/local/cuda-11.0/ /usr/local/cuda

After this is done, a cuda file will appear in the /usr/local/ directory, which points to the cuda-11.0 folder. Accessing this file is equivalent to accessing cuda-11.0. This can be seen in the figure below:

At this point, running nvcc --version will display:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Sun_Jan__9_22:14:01_CDT_2022
Cuda compilation tools, release 11.0, V11.0.218

For instance, if you need to set up a deep learning environment with Python 3.9.8 + TensorFlow 2.7.0 + CUDA 11.0, follow these steps:

First, create a Python environment with Python 3.9.8 using Anaconda:

conda create -n myenv python=3.9.8
conda activate myenv

Then, install TensorFlow 2.7.0 using pip:

pip install tensorflow==2.7.0

That's it! Since this Python 3.9.8 + TensorFlow 2.7.0 + CUDA 11.0 environment generally meets the requirements in the code, it is certainly compatible. We just need to ensure that the CUDA version matches the version required by the author.

Solve the driver and CUDA version problems
As we already know the relationship between Nvidia driver and CUDA, we may already know how to solve the above-mentioned problems.

If you do not want to bother to search over the internet, you can simply remove all Nvidia drivers and CUDA versions, and reinstall them by following the previous steps. Here is one way to get rid of all previous Nividia-related packages.

sudo apt-get remove --purge '^nvidia-.*'
sudo apt-get remove --purge '^libnvidia-.*'
sudo apt-get remove --purge '^cuda-.*'
Then run:
sudo apt-get install linux-headers-$(uname -r)

If you plan to upgrade your GPU for advanced AI computing. You may save money by selling used GPU online!