8000 Eval bug: NVIDIA Jetson AGX Xavier CUDA Compatibility Issue with llama.cpp · Issue #13629 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]
Skip to content

Eval bug: NVIDIA Jetson AGX Xavier CUDA Compatibility Issue with llama.cpp #13629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
sangyuxiaowu opened this issue May 19, 2025 · 8 comments
Closed

Eval bug: NVIDIA Jetson AGX Xavier CUDA Compatibility Issue with llama.cpp #13629

sangyuxiaowu opened this issue May 19, 2025 · 8 comments
Labels
bug-unconfirmed

Comments

@sangyuxiaowu
Copy link

Name and Version

ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version
version: 0 (unknown) llama.cpp-b5415
built with cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 for aarch64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

Jetson AGX Xavier

Models

No response

Problem description & steps to reproduce

I'm experiencing a CUDA compatibility issue with the latest version of llama.cpp on my Jetson AGX Xavier device (Ubuntu 20.04). Details:

  • Device: Jetson AGX Xavier
  • OS: Ubuntu 20.04
  • CUDA Version: 12.2
  • Issue: While an older release of llama.cpp (b4835) works correctly with CUDA, the latest version fails to run after successful compilation
  • Error Message: ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version
  • Working Output (with older version): ggml_cuda_init: found 1 CUDA devices: Device 0: Xavier, compute capability 7.2, VMM: yes

NVCC:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:08:11_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

In addition: The current device operating system version NVIDIA provides up to 20.04 and Cuda 12.2

First Bad Commit

It can be determined that the previous version b4835 can be used normally

Relevant log output

agx@ubuntu:/work/llama.cpp-b5415/build/bin$ ./llama-cli -m /work/gpt/models/qwen2-1_5b-instruct-q5_k_m.gguf -ngl 20
ggml_cuda_init: failed to initialize CUDA: CUDA driver version is insufficient for CUDA runtime version
warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions
build: 0 (unknown) with cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 for aarch64-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
@CISC
Copy link
Collaborator
CISC commented May 19, 2025 •
edited
Loading

It sounds like you might not have the correct driver installed, can you provide the output of nvidia-smi?

@sangyuxiaowu
Copy link
Author

It sounds like you might not have the correct driver installed, can you provide the output of ?nvidia-smi

The Jetson device does not have this command, and it can be used correctly in the b4835 version.

@CISC
Copy link
Collaborator
CISC commented May 19, 2025

Well, what nvidia driver do you have then?

@sangyuxiaowu
Copy link
Author
sangyuxiaowu commented May 19, 2025 •
edited
Loading

cuda-toolkit-12-2

agx@ubuntu:~$ sudo nvidia-smi
[sudo] password for agx:
sudo: nvidia-smi: command not found
agx@ubuntu:~$ cat /proc/driver/nvidia/version
cat: /proc/driver/nvidia/version: No such file or directory
agx@ubuntu:~$ lsmod | grep nv
nvadsp                110592  1 snd_soc_tegra210_adsp
nvgpu                2510848  0
nvmap                 192512  2 nvgpu

Image

@JohannesGaessler
Copy link
Collaborator

Please do a git bisect to identify the exact commit that introduced the problem.

@sangyuxiaowu
Copy link
Author

Is this it?
b4835...b5233

@JohannesGaessler
Copy link
Collaborator

git bisect is a git utility that repeatedly gives you commits to test to identify which commit introduced a bug. It's just a more convenient way of doing a binary search.

@sangyuxiaowu
Copy link
Author

Sorry, I found the reason. It seems that it is an environmental problem.
Once 'apt update' caused some conflicts. It was found by the pinch method that the original version that can be compiled and used was not normal after recompiling, but the original compiled version is OK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug-unconfirmed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@CISC @sangyuxiaowu @JohannesGaessler
0