CUDA Stack Sanity Check for Scientific Python

Pain point

GPU frameworks install but fail at runtime with driver/runtime mismatch errors, missing libcuda, or no GPU

detected.

Repro/diagnostic steps

1. Collect: `nvidia-smi` output (driver + reported CUDA capability).
2. Identify framework build: PyTorch/TensorFlow CUDA build tag, and the install command used.
3. Run a minimal GPU probe script (framework-specific).

Root causes (common)

• NVIDIA driver too old for the CUDA runtime/toolkit used.
• Confusion between "driver supports CUDA X" and "toolkit installed is CUDA X".
• Multiple CUDA toolkits on PATH/LD_LIBRARY_PATH.
• Container run without GPU runtime flags.

Fix workflow

1. Ensure driver meets compatibility requirements per NVIDIA guidance.
2. Install the framework build that matches the supported CUDA runtime for your system.
3. Prefer prebuilt wheels/containers that bundle CUDA runtime when appropriate.
4. Validate with a smoke test (list devices; run one tensor op on GPU).

Expected result

• Framework reports GPU devices and executes a minimal operation on GPU without version errors.

References

• https://docs.nvidia.com/deploy/cuda-compatibility/
• https://pytorch.org/get-started/locally/
• https://www.tensorflow.org/install/pip
• https://stackoverflow.com/questions/3253257/cuda-driver-version-is-insufficient-for-cuda-runtime-version