CMake: update FindCUDAToolkit.cmake, use torch::nvtx3 if present, mod… #153339

whitesscott · 2025-05-11T05:39:52Z

CMake: update FindCUDAToolkit.cmake, use torch::nvtx3 if present, modify torch/init.py to include libnvToolsExt.so if cuda < 12.9

Fixes #152756

Cuda-12.9 removed libnvToolsExt.so.* and is now pure header nvtx3.

Modify torch/init.py to include libnvToolsExt.so only if cuda < 12.9

pytorch/cmake/Modules/FindCUDAToolkit.cmake is old enough to not have any reference to CUDA::nvtx3. I updated to the current version FindCUDAToolkit.cmake.

…ify torch/__init__.py to include libnvToolsExt.so if cuda < 12.9

pytorch-bot · 2025-05-11T05:39:56Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153339

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 58 New Failures, 1 Unrelated Failure

As of commit 81860e8 with merge base e4f2282 ():

NEW FAILURES - The following jobs have failed:

Lint / lintrunner-noclang / linux-job (gh)
>>> Lint for torch/__init__.py:
pull / cuda12.4-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-docs / build-docs-functorch-false (gh)
Process completed with exit code 2.
pull / linux-docs / build-docs-python-false (gh)
Process completed with exit code 2.
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.g4dn.12xlarge.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.g4dn.12xlarge.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda12.6-py3.10-gcc11 / test (default, 1, 5, linux.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda12.6-py3.10-gcc11 / test (default, 2, 5, linux.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda12.6-py3.10-gcc11 / test (default, 3, 5, linux.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda12.6-py3.10-gcc11 / test (default, 4, 5, linux.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda12.6-py3.10-gcc11 / test (default, 5, 5, linux.4xlarge.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda12.6-py3.10-gcc11-sm89 / test (default, 1, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda12.6-py3.10-gcc11-sm89 / test (default, 2, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda12.6-py3.10-gcc11-sm89 / test (default, 3, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda12.6-py3.10-gcc11-sm89 / test (default, 4, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-cuda12.6-py3.10-gcc11-sm89 / test (default, 5, 5, linux.g6.4xlarge.experimental.nvidia.gpu) (gh)
Process completed with exit code 1.
pull / linux-focal-py3_9-clang9-xla / test (xla, 1, 1, linux.12xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.13-clang10 / test (crossref, 1, 2, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.13-clang10 / test (crossref, 2, 2, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.13-clang10 / test (default, 1, 5, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.13-clang10 / test (default, 2, 5, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.13-clang10 / test (default, 3, 5, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.13-clang10 / test (default, 4, 5, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.13-clang10 / test (default, 5, 5, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 1, 3, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 2, 3, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.13-clang10 / test (dynamo_wrapped, 3, 3, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10 / test (crossref, 1, 2, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10 / test (crossref, 2, 2, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10 / test (default, 1, 5, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10 / test (default, 2, 5, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10 / test (default, 3, 5, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10 / test (default, 4, 5, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10 / test (default, 5, 5, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 1, 3, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 2, 3, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10 / test (dynamo_wrapped, 3, 3, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10-onnx / test (default, 1, 2, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-focal-py3.9-clang10-onnx / test (default, 2, 2, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.10-clang15-asan / test (default, 1, 6, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.10-clang15-asan / test (default, 3, 6, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.10-clang15-asan / test (default, 4, 6, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.10-clang15-asan / test (default, 5, 6, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.10-clang15-asan / test (default, 6, 6, linux.4xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (backwards_compat, 1, 1, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (default, 1, 5, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (default, 2, 5, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (default, 3, 5, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (default, 4, 5, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (default, 5, 5, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (distributed, 1, 2, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (distributed, 2, 2, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (docs_test, 1, 1, linux.2xlarge) (gh)
Process completed with exit code 1.
pull / linux-jammy-py3.9-gcc11 / test (jit_legacy, 1, 1, linux.2xlarge) (gh)
Process completed with exit code 1.


pull / linux-jammy-py3.9-gcc11 / test (numpy_2_x, 1, 1, linux.2xlarge) (gh)

Process completed with exit code 1.
pull / linux-jammy-xpu-2025.0-py3.9 / build (gh)

Process completed with exit code 1.


BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures


pull / linux-jammy-py3-clang12-executorch / test (executorch, 1, 1, linux.2xlarge) (gh) (trunk failure)

Process completed with exit code 1.


This comment was automatically generated by Dr. CI and updates every 15 minutes.

linux-foundation-easycla · 2025-05-11T05:39:56Z

The committers listed above are authorized under a signed CLA.

✅ login: whitesscott / name: scott white (69c4a83, 81860e8, c019802, 4fd746e, 5a6f3b7, d0bc8bf)

… for older Cuda nvtoolsext

whitesscott · 2025-05-11T07:08:51Z

@pytorchbot label "topic: not user facing"

Skylion007 · 2025-05-11T17:03:55Z

cmake/Dependencies.cmake

@@ -1018,22 +1018,24 @@ endif()

 # ---[ nvtx
 if(USE_SYSTEM_NVTX)
-  find_path(nvtx3_dir NAMES nvtx3 PATHS ${CUDA_INCLUDE_DIRS})
+  find_path(nvtx3_dir NAMES nvtx3 PATHS /usr/local/cuda/include ${CUDA_INCLUDE_DIRS})


Why is this hard coded? is there not some CMake variable for this?

mistaken testing artifact. Thank you for finding it.

…ed path /usr/local/cuda/include

…ction

…vtoolsext

…fied torch/__init__.py

CMake: update FindCUDAToolkit.cmake, use torch::nvtx3 if present, mod…

4fd746e

…ify torch/__init__.py to include libnvToolsExt.so if cuda < 12.9

pytorchbot added the open source label May 11, 2025

torch_global_deps: link torch::nvtx3 first, fall back to CUDA::nvtx3,…

c019802

… for older Cuda nvtoolsext

pytorch-bot bot added the topic: not user facing topic category label May 11, 2025

Skylion007 reviewed May 11, 2025

View reviewed changes

whitesscott added 4 commits May 11, 2025 13:55

torch_global_deps: remove inadvertently retained from testing hardcod…

d0bc8bf

…ed path /usr/local/cuda/include

caffe2/CMakeLists: link torch_global_deps against nvtx3 in correct se…

5a6f3b7

…ction

added missing elseif torch::nvtx3 first, fall back to CUDA::nvtx3 / n…

69c4a83

…vtoolsext

rollback cmake/Modules/FindCUDAToolkit.cmake. I should have ONLY modi…

81860e8

…fied torch/__init__.py

colesbury requested a review from malfet May 13, 2025 00:56

colesbury added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label May 13, 2025

Skylion007 requested review from eqy and ngimel May 14, 2025 15:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CMake: update FindCUDAToolkit.cmake, use torch::nvtx3 if present, mod… #153339

CMake: update FindCUDAToolkit.cmake, use torch::nvtx3 if present, mod… #153339

CMake: update FindCUDAToolkit.cmake, use torch::nvtx3 if present, mod… #153339

Are you sure you want to change the base?

CMake: update FindCUDAToolkit.cmake, use torch::nvtx3 if present, mod… #153339

Conversation

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/153339

❌ 58 New Failures, 1 Unrelated Failure

Choose a reason for hiding this comment

Choose a reason for hiding this comment