LanceDB Main

Main

agentsllmsvector-databaselancedbgptopenaiAImultimodal-aiRAG-using-Groqmachine-learningembeddingsfine-tuningexamplesdeep-learninggpt-4-visionllama-indexragmultimodallangchainlancedb-recipes

alph-notebooks/lancedb-recipes / main.ipynb

Export

Run Notebooks

Contents

No cells yet

Add cells to see them here

Installing required libraries

[ ]

Requirement already satisfied: sentence-transformers in /usr/local/lib/python3.10/dist-packages (3.0.1)
Requirement already satisfied: lancedb in /usr/local/lib/python3.10/dist-packages (0.11.0)
Requirement already satisfied: groq in /usr/local/lib/python3.10/dist-packages (0.9.0)
Requirement already satisfied: transformers<5.0.0,>=4.34.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (4.42.4)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (4.66.4)
Requirement already satisfied: torch>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (2.3.1+cu121)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.25.2)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.3.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (1.13.1)
Requirement already satisfied: huggingface-hub>=0.15.1 in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (0.23.5)
Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from sentence-transformers) (9.4.0)
Requirement already satisfied: deprecation in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.1.0)
Requirement already satisfied: pylance==0.15.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (0.15.0)
Requirement already satisfied: ratelimiter~=1.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (1.2.0.post0)
Requirement already satisfied: requests>=2.31.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.31.0)
Requirement already satisfied: retry>=0.9.2 in /usr/local/lib/python3.10/dist-packages (from lancedb) (0.9.2)
Requirement already satisfied: pydantic>=1.10 in /usr/local/lib/python3.10/dist-packages (from lancedb) (2.8.2)
Requirement already satisfied: attrs>=21.3.0 in /usr/local/lib/python3.10/dist-packages (from lancedb) (23.2.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from lancedb) (24.1)
Requirement already satisfied: cachetools in /usr/local/lib/python3.10/dist-packages (from lancedb) (5.4.0)
Requirement already satisfied: overrides>=0.7 in /usr/local/lib/python3.10/dist-packages (from lancedb) (7.7.0)
Requirement already satisfied: pyarrow>=12 in /usr/local/lib/python3.10/dist-packages (from pylance==0.15.0->lancedb) (14.0.2)
Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from groq) (3.7.1)
Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from groq) (1.7.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from groq) (0.27.0)
Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from groq) (1.3.1)
Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from groq) (4.12.2)
Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->groq) (3.7)
Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->groq) (1.2.2)
Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->groq) (2024.7.4)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->groq) (1.0.5)
Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->groq) (0.14.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (3.15.4)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (2024.6.1)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence-transformers) (6.0.1)
Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (0.7.0)
Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.10/dist-packages (from pydantic>=1.10->lancedb) (2.20.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->lancedb) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.31.0->lancedb) (2.0.7)
Requirement already satisfied: decorator>=3.4.2 in /usr/local/lib/python3.10/dist-packages (from retry>=0.9.2->lancedb) (4.4.2)
Requirement already satisfied: py<2.0.0,>=1.4.26 in /usr/local/lib/python3.10/dist-packages (from retry>=0.9.2->lancedb) (1.11.0)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (1.13.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (3.1.4)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (2.20.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (12.1.105)
Requirement already satisfied: triton==2.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence-transformers) (2.3.1)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.11.0->sentence-transformers) (12.5.82)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.34.0->sentence-transformers) (2024.5.15)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.34.0->sentence-transformers) (0.4.3)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.34.0->sentence-transformers) (0.19.1)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers) (1.4.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence-transformers) (3.5.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.11.0->sentence-transformers) (2.1.5)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.11.0->sentence-transformers) (1.3.0)

Importing libraries

[ ]

Getting all links at certain depth

[ ]

Scrapping URL Content

[ ]

Storing URLs with scrapped data

[ ]

https://docs.nvidia.com/cuda/ 0
https://developer.nvidia.com/nvidia-video-codec-sdk 1
https://nvlabs.github.io/cub/ 1
https://nvidia.github.io/libcudacxx/ 1
https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html 1
https://nvidia.github.io/cccl/thrust/ 1
https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html 1
https://docs.nvidia.com/deploy/cuda-compatibility/index.html 1
https://docs.nvidia.com/cupti/index.html 1
https://docs.nvidia.com/gpudirect-storage/index.html 1
https://docs.nvidia.com/compute-sanitizer/index.html 1
https://docs.nvidia.com/nsight-systems/index.html 1
https://docs.nvidia.com/nsight-compute/index.html 1
https://docs.nvidia.com/nsight-visual-studio-edition/index.html 1
https://developer.nvidia.com/cuda-toolkit-archive 1
https://www.nvidia.com/en-us/about-nvidia/privacy-policy/ 1
https://www.nvidia.com/en-us/about-nvidia/privacy-center/ 1
https://www.nvidia.com/en-us/preferences/start/ 1
https://www.nvidia.com/en-us/about-nvidia/terms-of-service/ 1
https://www.nvidia.com/en-us/about-nvidia/accessibility/ 1
https://www.nvidia.com/en-us/about-nvidia/company-policies/ 1
https://www.nvidia.com/en-us/product-security/ 1
https://www.nvidia.com/en-us/contact/ 1

Chunking data and storing it's embeddings using clustering

[ ]

Storing chunks and embeddings of all URLs

[ ]

23
URL during Chunking : https://docs.nvidia.com/cuda/

/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: 
The secret `HF_TOKEN` does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
  warnings.warn(

Length of chuncked sentences : 127

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://developer.nvidia.com/nvidia-video-codec-sdk
Length of chuncked sentences : 114

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://nvlabs.github.io/cub/
Length of chuncked sentences : 1
Empty data returning
URL during Chunking : https://nvidia.github.io/libcudacxx/
Length of chuncked sentences : 1
Empty data returning
URL during Chunking : https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html
Length of chuncked sentences : 322

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://nvidia.github.io/cccl/thrust/
Length of chuncked sentences : 29

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html
Length of chuncked sentences : 368

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://docs.nvidia.com/deploy/cuda-compatibility/index.html
Length of chuncked sentences : 324

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://docs.nvidia.com/cupti/index.html
Length of chuncked sentences : 5

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://docs.nvidia.com/gpudirect-storage/index.html
Length of chuncked sentences : 2
Empty data returning
URL during Chunking : https://docs.nvidia.com/compute-sanitizer/index.html
Length of chuncked sentences : 1
Empty data returning
URL during Chunking : https://docs.nvidia.com/nsight-systems/index.html
Length of chuncked sentences : 9

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://docs.nvidia.com/nsight-compute/index.html
Length of chuncked sentences : 19

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://docs.nvidia.com/nsight-visual-studio-edition/index.html
Length of chuncked sentences : 18

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://developer.nvidia.com/cuda-toolkit-archive
Length of chuncked sentences : 7

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://www.nvidia.com/en-us/about-nvidia/privacy-policy/
Length of chuncked sentences : 93

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://www.nvidia.com/en-us/about-nvidia/privacy-center/
Length of chuncked sentences : 5

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://www.nvidia.com/en-us/preferences/start/
Length of chuncked sentences : 1
Empty data returning
URL during Chunking : https://www.nvidia.com/en-us/about-nvidia/terms-of-service/
Length of chuncked sentences : 145

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://www.nvidia.com/en-us/about-nvidia/accessibility/
Length of chuncked sentences : 5
URL during Chunking : https://www.nvidia.com/en-us/about-nvidia/company-policies/

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

Length of chuncked sentences : 7

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://www.nvidia.com/en-us/product-security/
Length of chuncked sentences : 8

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

URL during Chunking : https://www.nvidia.com/en-us/contact/
Length of chuncked sentences : 11

/usr/local/lib/python3.10/dist-packages/sklearn/cluster/_kmeans.py:1416: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  super()._check_params_vs_input(X, default_n_init=10)

[ ]

Creating Vector Database with lancedb and storing data

[ ]

Searching top results related to query

[ ]

[32]

[33]

['CUDA Compatibility CUDA Compatibility describes the use of new CUDA toolkit components on systems with older base installations. The NVIDIA® CUDA® Toolkit enables developers to build NVIDIA GPU accelerated compute applications for desktop computers, enterprise, and data centers to hyperscalers. It consists of the CUDA compiler toolchain including the CUDA runtime (cudart) and various CUDA libraries and tools. To build an application, a developer has to install only the CUDA Toolkit and necessary libraries required for linking. In order to run a CUDA application, the system should have a CUDA enabled GPU and an NVIDIA display driver that is compatible with the CUDA Toolkit that was used to build the application itself. Figure 1 Components of CUDA\uf0c1 Every CUDA toolkit also ships with an NVIDIA display driver package for convenience. This driver supports all the features introduced in that version of the CUDA Toolkit. The driver package includes both the user mode CUDA driver (libcuda. Typically, upgrading a CUDA Toolkit involves upgrading both the toolkit and the driver to get the bleeding edge toolkit and driver capabilities. Figure 2 CUDA Upgrade Path\uf0c1 But this is not always required. CUDA Compatibility guarantees allow for upgrading only certain components and that will be the focus of the rest of this document. We will see how the upgrade to a new CUDA Toolkit can be simplified to not always require a full system upgrade. From CUDA 11 onwards, applications compiled with a CUDA Toolkit release from within a CUDA major release family can run, with limited feature-set, on systems having at least the minimum required driver version as indicated below. This minimum required driver can be different from the driver packaged with the CUDA Toolkit but should belong to the same major release. Refer to the CUDA Toolkit Release Notes for the complete table. CUDA Toolkit Linux x86_64 Minimum Required Driver Version Windows Minimum Required Driver Version CUDA 12.39* CUDA 11.39 (Windows) as indicated, minor version compatibility is possible across the CUDA 11. While applications built against any of the older CUDA Toolkits always continued to function on newer drivers due to binary backward compatibility, before CUDA 11, applications built against newer CUDA Toolkit releases were not supported on older drivers without forward compatibility package. If you are using a new CUDA 10. Consequently, the minimum required driver version changed for every new CUDA Toolkit minor release until CUDA 11. Therefore, system administrators always have to upgrade drivers in order to support applications built against CUDA Toolkits from 10. CUDA Toolkit Linux x86_64 Minimum Required Driver Version Windows Minimum Required Driver Version CUDA 10.22 CUDA 10.96 CUDA 10.31 With minor version compatibility, upgrading to CUDA 11.02 that was shipped with CUDA 11.0, as shown below: Minimum required driver version guidance can be found in the CUDA Toolkit Release Notes. If either of these caveats are limiting, then a new CUDA driver from the same minor version of the toolkit that the application was built with or later is required. Limited feature set Sometimes features introduced in a CUDA Toolkit version may actually span both the toolkit and the driver. Refer to the CUDA Compatibility Developers Guide for more details. PTX Developers should refer to the CUDA Compatibility Developers Guide and PTX programming guide in the CUDA C++ Programming Guide for details on this limitation. As described, applications that directly rely only on the CUDA runtime can be deployed in the following two scenarios: CUDA driver that’s installed on the system is newer than the runtime.\r\nCUDA runtime is newer than the CUDA driver on the system but they are from the same major release of CUDA Toolkit. For example, each cuDNN version requires a certain version of cuBLAS. Figure 3 NVRTC supports minor version compatibility from CUDA 11. To support such scenarios, CUDA introduced a Forward Compatibility Upgrade path in CUDA 10. It’s mainly intended to support applications built on newer CUDA Toolkits to run on systems installed with an older NVIDIA Linux GPU driver from different major release families. This new forward-compatible upgrade path requires the use of a special package called “CUDA compat package”. The CUDA compat package is available in the local installers or the CUDA network repositories provided by NVIDIA as cuda-compat-12.5 it will be found in /usr/local/cuda-12. The cuda-compat package consists of the following files: libcuda.* - the CUDA Driver libnvidia-nvvm.* - JIT LTO ( CUDA 11.* -GPU debugging support for CUDA Driver (CUDA 11.8 and later only) These files should be kept together as the CUDA driver is dependent on the libnvidia-ptxjitcompiler. Example: CUDA Compatibility is installed and the application can now run successfully as shown below. In this example, the user sets LD_LIBRARY_PATH to include the files installed by the cuda-compat-12-1 package. Check the files installed under /usr/local/cuda/compat: The user can set LD_LIBRARY_PATH to include the files installed before running the CUDA 12.1 application: The cuda-compat package files can also be extracted from the appropriate datacenter driver ‘runfile’ installers (. Copy the four CUDA compatibility upgrade files, listed at the start of this section, into a user- or root-created directory. Note Symlinks under /usr/local/cuda/compat need to be created manually when using the runfile installer. CUDA forward compat packages should be used only in the following situations when forward compatibility is required across major releases. The CUDA compat package is named after the highest toolkit that it can support.5 application support, please install the cuda-compat package for 12.x toolkits, then the cuda-compat package is not required in most cases. Unlike the minor-version compatibility that is defined between CUDA runtime and CUDA driver, forward compatibility is defined between the kernel driver and the CUDA driver, and hence such restrictions do not apply. NVIDIA Kernel Mode Driver - Production Branch CUDA Forward Compatible Upgrade 470.02+ (CUDA 11.02+ (CUDA 12.03+ (CUDA 12.06+ (CUDA 12.14+ (CUDA 12.02+ (CUDA 12.xx) are not supported targets for CUDA Forward Compatibility. Examples of how to read this table: The CUDA 12-4 compat package is “C”ompatible with driver versions 470, 535. The CUDA “12-3” release is not-compatible (“X”) with driver version 550 as it was released prior to the driver. There are specific features in the CUDA driver that require kernel-mode support and will only work with a newer kernel mode driver. CUDA Forward Compatible Upgrade CUDA - OpenGL/Vulkan Interop cuMemMap* set of functionalities System Base Installation: 525 (>=. In addition to the CUDA driver and certain compiler components, there are other drivers in the system installation stack (for example, OpenCL) that remain on the old version. The forward-compatible upgrade path is for CUDA only. A well-written application should use following error codes to determine if CUDA Forward Compatible Upgrade is supported. CUDA_ERROR_SYSTEM_DRIVER_MISMATCH = 803. This error indicates that there is a mismatch between the versions of the display driver and the CUDA driver. CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE = 804. This error indicates that the system was upgraded to run with forward compatibility but the visible hardware detected by CUDA does not support this configuration. There are two models of deployment for the CUDA compat package. The user can set LD_LIBRARY_PATH to include the files installed before running the CUDA 11. In the cases where the module script cannot use CUDA compatible upgrade, a fallback path to the default system’s installed CUDA driver can provide a more consistent experience and this can be achieved using RPATH. The CUDA driver maintains backward compatibility to continue support of applications built on older toolkits. Using a compatible minor driver version, applications build on CUDA Toolkit 11 and newer are supported on any driver from within the corresponding major release. Using the CUDA Forward Compatibility package, system administrators can run applications built using a newer toolkit even when an older driver that does not satisfy the minimum required driver version is installed on the system. This forward compatibility allows the CUDA deployments in data centers and enterprises to benefit from the faster release cadence and the latest features and performance of CUDA Toolkit. CUDA compatibility helps users by: Faster upgrades to the latest CUDA releases: Enterprises or data centers with NVIDIA GPUs have complex workflows and require advance planning for NVIDIA driver rollouts. Not having to update the driver for newer CUDA releases can mean that new versions of the software can be made available faster to users without any delays. Faster upgrades of the CUDA libraries: Users can upgrade to the latest software libraries and applications built on top of CUDA (for example, math libraries or deep learning frameworks) without an upgrade to the entire CUDA Toolkit or driver. This is possible as these libraries and frameworks do not have a direct dependency on the CUDA runtime, compiler or driver. This section includes some FAQs related to CUDA compatibility. What is the difference between CUDA forward compatible upgrade and CUDA minor version compatibility? When should users use these features? Area CUDA Forward Compatible Upgrade CUDA Minor Version Compatibility Compatibility Across older drivers from different major release versions of CUDA. Across minor release versions of CUDA only. Between kernel driver and user mode CUDA driver. Between libraries or runtimes that link to the CUDA driver. When to use If you cannot upgrade the kernel driver but need to use the latest CUDA Toolkit. GPUs supported 11. All GPU products supported OS distributions supported Linux only Windows, Linux Features supported Some features such as (CUDA-GL interop, Power 9 ATS, cuMemMap APIs) are not supported. All existing CUDA features (from older minor releases) work. CUDA releases supported All CUDA releases supported through the lifetime of the datacenter driver branch. For example,\r\nR418 (CUDA 10.1) EOLs in March 2022 - so all CUDA versions released (including major releases) during this timeframe are supported.\r\nCompatibility is not supported across major CUDA releases. Users can also set up LD_LIBRARY_PATH with the new libraries from the cuda-compat-* package. Note For mobile compatibility information, see CUDA Upgradable Package for Jetson. Does CUDA forward compatible upgrades work intra-branch? Users can upgrade the kernel mode driver within the same branch. Sometimes this may require updating the cuda-compat* package. Which GPUs are supported by the driver? The CUDA compatible upgrade is meant to ease the management of large production systems for enterprise customers.\r\nIt’s important to note that HW support is defined by the kernel mode driver and as such, newer CUDA drivers on their own will not enable new HW support.5 - current latest Refer to CUDA Driver Lifecycle to find the latest supported driver. If we build our CUDA application using CUDA 11.0, can it continue to be used with newer NVIDIA drivers (such as CUDA 11.)? Or is it only the other way around? Drivers have always been backwards compatible with CUDA. This means that a CUDA 11. CUDA applications typically statically include all the libraries (for example cudart, CUDA math libraries such as cuBLAS, cuFFT) they need, so they should work on new drivers or CUDA Toolkit installations. In other words, since CUDA is backward compatible, existing CUDA applications can continue to be used with newer CUDA versions. What is the minimum CUDA 11.x driver that supports the CUDA minor version compatibility? The minimum driver version required is 450. What about new features introduced in minor releases of CUDA? How does a developer build an application using newer CUDA Toolkits (e.x) work on a system with a CUDA 11.0 driver (R450)? By using new CUDA versions, users can benefit from new CUDA programming model APIs, compiler optimizations and math library features. A subset of CUDA APIs don’t need a new driver and they can all be used without any driver dependencies. To use other CUDA APIs introduced in a minor release (that require a new driver), one would have to implement fallbacks or fail gracefully. This situation is not different from what is available today where developers use macros to compile out features based on CUDA versions. Users should refer to the CUDA headers and documentation for new CUDA APIs introduced in a release. Does CUDA compatibility work with containers? Yes. CUDA minor version compatibility and CUDA forward compatible upgrade both work when using either NGC Deep Learning Framework containers or using containers that are based on the official CUDA base images. The images include the CUDA compatible upgrade libraries and the NVIDIA Container Toolkit (nvidia-docker2) has logic to correctly load the required libraries',
, 'Installation Guides Programming Guides CUDA API References PTX Compiler API References Miscellaneous Tools White Papers Application Notes Compiler SDK Develop, Optimize and Deploy GPU-Accelerated Apps The NVIDIA® CUDA® Toolkit provides a development environment for creating high performance GPU-accelerated\r\napplications. With the CUDA Toolkit, you can develop, optimize, and deploy your applications on GPU-accelerated\r\nembedded systems, desktop workstations, enterprise data centers, cloud-based platforms and HPC supercomputers. The Release Notes for the CUDA Toolkit. The list of CUDA features by release. The CUDA Toolkit End User License Agreement applies to the NVIDIA CUDA Toolkit, the NVIDIA CUDA Samples, the NVIDIA Display Driver, NVIDIA Nsight tools (Visual Studio Edition), and the associated documentation on CUDA APIs, programming model and development tools. This guide provides the minimal first-steps instructions for installation and verifying CUDA on a standard system. This guide discusses how to install and check for correct operation of the CUDA Development Tools on Microsoft Windows systems. This guide discusses how to install and check for correct operation of the CUDA Development Tools on GNU/Linux systems. This guide provides a detailed discussion of the CUDA programming model and programming interface. The appendices include a list of all CUDA-enabled devices, detailed description of all extensions to the C++ language, listings of supported mathematical functions, C++ features supported in host and device code, details on texture fetching, technical specifications of various devices, and concludes by introducing the low-level driver API. This document shows how to write PTX that is ABI-compliant and interoperable with other CUDA code. This document shows how to inline PTX (parallel thread execution) assembly language statements into CUDA code. The CUDA math API. The cuBLAS library is an implementation of BLAS (Basic Linear Algebra Subprograms) on top of the NVIDIA CUDA runtime. The cuDLA API. The API reference for libcu++, the CUDA C++ standard library. The cuRAND library user guide. The cuSPARSE library user guide. NVRTC is a runtime compilation library for CUDA C++. It accepts CUDA C++ source code in character string form and creates handles that can be used to obtain the PTX. The PTX string generated by NVRTC can be loaded by cuModuleLoadData and cuModuleLoadDataEx, and linked with other modules by cuLinkAddData of the CUDA Driver API. The cuSOLVER library user guide. This guide shows how to compile a PTX program into GPU assembly code using APIs provided by the static PTX Compiler library. This document describes the demo applications shipped with the CUDA Demo Suite. This guide is intended to help users get started with using NVIDIA CUDA on Windows Subsystem for Linux (WSL 2). The guide covers installation and running CUDA applications and containers in this environment. This document describes CUDA Compatibility, including CUDA Enhanced Compatibility and CUDA Forward Compatible Upgrade. The CUDA Profiling Tools Interface (CUPTI) enables the creation of profiling and tracing tools that target CUDA applications. The CUDA debugger API. vGPUs that support CUDA. This is a reference document for nvcc, the CUDA compiler driver. CUDA-GDB is an extension to the x86-64 port of GDB, the GNU Project debugger. The application notes for cuobjdump, nvdisasm, and nvprune. High-level language front-ends, like the CUDA C compiler front-end, can generate NVVM IR',
, 'The API reference for CUPTI, the CUDA Profiling Tools Interface',
, 'CUDA Debugger Reference Release Information Copyright and License Notices NVIDIA® Nsight™ Visual Studio Edition is an application development environment which brings GPU computing into Microsoft Visual Studio. See the latest features and updates for this version of NVIDIA Nsight Visual Studio Edition. This chapter walks you through the system requirements for NVIDIA Nsight Visual Studio Edition, and the steps you’ll need to install and get started using the software. Additional resources for learning more about working with NVIDIA Nsight Visual Studio Edition. Find documentation for previous versions of NVIDIA Nsight Visual Studio Edition. This document is the End User License Agreement (EULA) for NVIDIA Nsight Visual Studio Edition. This document contains specific license terms and conditions for NVIDIA Nsight Visual Studio Edition',
, 'Previous releases of the CUDA Toolkit, GPU Computing SDK, documentation and developer drivers can be found using the links below. Latest ReleaseCUDA Toolkit 12.1 (July 2024), Versioned Online Documentation Archived Releases Learn more about the latest CUDA Toolkit and the CUDA Tools and Library Ecosystem']

Using llama-3.1-70b for results

[34]

[37]

('CUDA stands for Compute Unified Device Architecture. It is a development environment created by NVIDIA for creating '
 'high-performance applications that can run on NVIDIA GPUs. The CUDA Toolkit provides a set of tools, libraries, and '
 'programming models for developers to build and optimize applications that can take advantage of the massively '
 'parallel processing capabilities of NVIDIA GPUs.')

[ ]