8000 undefined symbol: __nvJitLinkCreate_12_8, version libnvJitLink.so.12 · Issue #152783 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

undefined symbol: __nvJitLinkCreate_12_8, version libnvJitLink.so.12 #152783

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
FurkanGozukara opened this issue May 4, 2025 · 17 comments
Open

undefined symbol: __nvJitLinkCreate_12_8, version libnvJitLink.so.12 #152783

FurkanGozukara opened this issue May 4, 2025 · 17 comments
Assignees
@malfet @atalman
Labels
has workaround module: binaries Anything related to official binaries that we release to users needs reproduction Someone else needs to try reproducing the issue given the instructions. No action needed from user triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@FurkanGozukara
Copy link
FurkanGozukara commented May 4, 2025 •
edited by pytorch-bot bot
Loading

I am trying to use Torch 2.7 with CUDA 12.8 on Linux with Kohya trainer and I am getting this error

Exactly same installation and setup works on Windows

I tried Torch 2.7 official and latest Torch 2.8 nightly all CUDA 12.8 and same error


╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/Ubuntu/apps/kohya_ss/kohya_gui.py:12 in <module>                       │
│                                                                              │
│    11 from kohya_gui.textual_inversion_gui import ti_tab                     │
│ ❱  12 from kohya_gui.utilities import utilities_tab                          │
│    13 from kohya_gui.lora_gui import lora_tab                                │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/kohya_gui/utilities.py:6 in <module>              │
│                                                                              │
│    5 from .blip_caption_gui import gradio_blip_caption_gui_tab               │
│ ❱  6 from .blip2_caption_gui import gradio_blip2_caption_gui_tab             │
│    7 from .git_caption_gui import gradio_git_caption_gui_tab                 │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/kohya_gui/blip2_caption_gui.py:2 in <module>      │
│                                                                              │
│     1 from PIL import Image                                                  │
│ ❱   2 from transformers import Blip2Processor, Blip2ForConditionalGeneration │
│     3 import torch                                                           │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/__ │
│ init__.py:26 in <module>                                                     │
│                                                                              │
│     25 # Check the dependencies satisfy the minimal versions required.       │
│ ❱   26 from . import dependency_versions_check                               │
│     27 from .utils import (                                                  │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/de │
│ pendency_versions_check.py:16 in <module>                                    │
│                                                                              │
│   15 from .dependency_versions_table import deps                             │
│ ❱ 16 from .utils.versions import require_version, require_version_core       │
│   17                                                                         │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/ut │
│ ils/__init__.py:34 in <module>                                               │
│                                                                              │
│    33 )                                                                      │
│ ❱  34 from .generic import (                                                 │
│    35     ContextManagers,                                                   │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/ut │
│ ils/generic.py:462 in <module>                                               │
│                                                                              │
│   461 if is_torch_available():                                               │
│ ❱ 462     import torch.utils._pytree as _torch_pytree                        │
│   463                                                                        │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/__init__. │
│ py:418 in <module>                                                           │
│                                                                              │
│    417         _load_global_deps()                                           │
│ ❱  418     from torch._C import *  # noqa: F403                              │
│    419                                                                       │
╰──────────────────────────────────────────────────────────────────────────────╯
ImportError: 
/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/lib/../../nvi
dia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkCreate_12_8, 
version libnvJitLink.so.12

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @seemethere @malfet @osalpekar @atalman
@FurkanGozukara FurkanGozukara mentioned this issue May 4, 2025
Closed
@ngimel
Copy link
Collaborator
ngimel commented May 4, 2025

Looks like 12.8 should be included in the workaround https://github.com/pytorch/pytorch/pull/141063/files#diff-c8835eba8562819a44e70a30545213c96e68c588a2cfa673fa495b187b1ea8dfR324, cc @malfet, @atalman

@FurkanGozukara
Copy link
Author

Looks like 12.8 should be included in the workaround https://github.com/pytorch/pytorch/pull/141063/files#diff-c8835eba8562819a44e70a30545213c96e68c588a2cfa673fa495b187b1ea8dfR324, cc @malfet, @atalman

when can this be added to nightly? certainly my followers couldn't do this

@FurkanGozukara
Copy link
Author

@ngimel i tried but still error here

Image

@atalman atalman added this to the 2.7.1 milestone May 5, 2025
@zou3519 zou3519 added module: binaries Anything related to official binaries that we release to users triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module triage review and removed triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels May 6, 2025
@albanD albanD added triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module and removed triage review labels May 12, 2025
@malfet
Copy link
Contributor
malfet commented May 12, 2025

@FurkanGozukara can you please check if deleting LD_LIBRARY_PATH (or adding a correct location of it) will solve the problem?

@albanD
Copy link
Collaborator
albanD commented May 12, 2025

We should get a repro for this and make sure we have appropriate testing for releases

@FurkanGozukara did you try to update your global install to be 12.8 to see if that helps? Similar issues like this in the past have been due to mismatched global instances and removing these has helped as a workaround.

@FurkanGozukara
Copy link
Author

We should get a repro for this and make sure we have appropriate testing for releases

@FurkanGozukara did you try to update your global install to be 12.8 to see if that helps? Similar issues like this in the past have been due to mismatched global instances and removing these has helped as a workaround.

global install didnt fix sadly

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/Ubuntu/apps/kohya_ss/kohya_gui.py:12 in <module>                       │
│                                                                              │
│    11 from kohya_gui.textual_inversion_gui import ti_tab                     │
│ ❱  12 from kohya_gui.utilities import utilities_tab                          │
│    13 from kohya_gui.lora_gui import lora_tab                                │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/kohya_gui/utilities.py:6 in <module>              │
│                                                                              │
│    5 from .blip_caption_gui import gradio_blip_caption_gui_tab               │
│ ❱  6 from .blip2_caption_gui import gradio_blip2_caption_gui_tab             │
│    7 from .git_caption_gui import gradio_git_caption_gui_ta
8000
b                 │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/kohya_gui/blip2_caption_gui.py:2 in <module>      │
│                                                                              │
│     1 from PIL import Image                                                  │
│ ❱   2 from transformers import Blip2Processor, Blip2ForConditionalGeneration │
│     3 import torch                                                           │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/__ │
│ init__.py:26 in <module>                                                     │
│                                                                              │
│     25 # Check the dependencies satisfy the minimal versions required.       │
│ ❱   26 from . import dependency_versions_check                               │
│     27 from .utils import (                                                  │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/de │
│ pendency_versions_check.py:16 in <module>                                    │
│                                                                              │
│   15 from .dependency_versions_table import deps                             │
│ ❱ 16 from .utils.versions import require_version, require_version_core       │
│   17                                                                         │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/ut │
│ ils/__init__.py:34 in <module>                                               │
│                                                                              │
│    33 )                                                                      │
│ ❱  34 from .generic import (                                                 │
│    35     ContextManagers,                                                   │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/ut │
│ ils/generic.py:462 in <module>                                               │
│                                                                              │
│   461 if is_torch_available():                                               │
│ ❱ 462     import torch.utils._pytree as _torch_pytree                        │
│   463                                                                        │
│                                                                              │
│ /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/__init__. │
│ py:409 in <module>                                                           │
│                                                                              │
│    408         _load_global_deps()                                           │
│ ❱  409     from torch._C import *  # noqa: F403                              │
│    410                                                                       │
╰──────────────────────────────────────────────────────────────────────────────╯
ImportError: 
/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/torch/lib/../../nvi
dia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkCreate_12_8, 
version libnvJitLink.so.12

@FurkanGozukara
Copy link
Author

LD_LIBRARY_PATH

it was set like this

/usr/local/cuda-12.4/extras/CUPTI/lib64:/usr/local/cuda-12.4/lib64:/opt/thinlinc/lib64:/opt/thinlinc/lib

after i did unset it fixed the issue and now working

so what is the conclusion? and thank you

@ngimel
Copy link
Collaborator
ngimel commented May 13, 2025

THe conclusion then is you shouldn't have LD_LIBRARY_PATH point to 12.4 when you are trying to run with 12.8

@albanD
Copy link
Collaborator
albanD commented May 13, 2025

Conflicting LD_LIBRARY_PATH would cause this and we wouldn't be able to do anything about it on our end.
I guess this one can be closed then as "incorrect LD_LIBRARY_PATH setting from the user" @malfet ?

@FurkanGozukara
Copy link
Author

Conflicting LD_LIBRARY_PATH would cause this and we wouldn't be able to do anything about it on our end. I guess this one can be closed then as "incorrect LD_LIBRARY_PATH setting from the user" @malfet ?

isnt pytorch supposed to be standalone? i mean i dont use anything from /usr/local/cuda-12.4 and when i remove it , it works

@ngimel
Copy link
Collaborator
ngimel commented May 14, 2025

It's a bug in nvidia packaging, we cannot do anything about it, we are fixing most common cases by preloading libraries, but your LD_LIBRARY_PATH makes us find wrong libraries in this case.

@FurkanGozukara
Copy link
Author

It's a bug in nvidia packaging, we cannot do anything about it, we are fixing most common cases by preloading libraries, but your LD_LIBRARY_PATH makes us find wrong libraries in this case.

but removing it fixing it. so it is not depended to it actually. interesting. thanks for replies and solution

@malfet malfet reopened this May 15, 2025
@malfet
Copy link
Contributor
malfet commented May 15, 2025 •
edited
Loading

Reopening this one, as it is, as @ngimel correctly pointed out is just another manifestation of #140797 and we should add 12.8 to the same workaround

But I could not reproduce the problem with 12.8 wheels and 12.4 installed locally:

% docker run --rm -it nvidia/cuda:12.4.1-devel-ubuntu22.04 bash -c "apt update; apt install -y python3-pip; pip3 install torch numpy --index-url https://download.pytorch.org/whl/cu128; LD_LIBRARY_PATH=/usr/local/cuda/lib64 python3 -c 'import torch;print(torch.__version__); print(open(\"/proc/self/maps\").read())'"

@malfet malfet added the needs reproduction Someone else needs to try reproducing the issue given the instructions. No action needed from user label May 15, 2025
@malfet malfet removed this from the 2.7.1 milestone May 15, 2025
@malfet
Copy link
Contributor
malfet commented May 15, 2025

Actually #145582 removed 12.4 and 12.6 constraint, so it should have worked for 12.8 out of box
With that in mind, removing 2.7.1 milestone and adding needs reproduction

@FurkanGozukara
Copy link
Author

Actually #145582 removed 12.4 and 12.6 constraint, so it should have worked for 12.8 out of box With that in mind, removing 2.7.1 milestone and adding needs reproduction

I reproduced this multiple times

It happens with this repo : https://github.com/bmaltais/kohya_ss

@atalman
Copy link
Contributor
atalman commented May 15, 2025

Hi @FurkanGozukara could you please try to provide a minimal repro similar to command from @malfet ? :

% docker run --rm -it nvidia/cuda:12.4.1-devel-ubuntu22.04 bash -c "apt update; apt install -y python3-pip; pip3 install torch numpy --index-url https://download.pytorch.org/whl/cu128; LD_LIBRARY_PATH=/usr/local/cuda/lib64 python3 -c 'import torch;print(torch.__version__); print(open(\"/proc/self/maps\").read())'"

Series of commands to repro this could also work.

@FurkanGozukara
Copy link
Author

Hi @FurkanGozukara could you please try to provide a minimal repro similar to command from @malfet ? :

% docker run --rm -it nvidia/cuda:12.4.1-devel-ubuntu22.04 bash -c "apt update; apt install -y python3-pip; pip3 install torch numpy --index-url https://download.pytorch.org/whl/cu128; LD_LIBRARY_PATH=/usr/local/cuda/lib64 python3 -c 'import torch;print(torch.__version__); print(open(\"/proc/self/maps\").read())'"

Series of commands to repro this could also work.

sadly i don't know what triggering the error. but that repo is causing error not even before starting the app

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@malfet malfet

@atalman atalman

Labels
has workaround module: binaries Anything related to official binaries that we release to users needs reproduction Someone else needs to try reproducing the issue given the instructions. No action needed from user triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@malfet @zou3519 @albanD @atalman @ngimel @FurkanGozukara
0