[ROCm][CI] upgrade CI to ROCm 6.3 #142152

jithunnair-amd · 2024-12-05T18:32:32Z

cc @jeffdaily @sunway513 @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

pytorch-bot · 2024-12-05T18:32:37Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/142152

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 13 New Failures, 2 Cancelled Jobs, 3 Unrelated Failures

As of commit 970a3dc with merge base 8d9ff9c ():

NEW FAILURES - The following jobs have failed:

periodic / linux-focal-cuda11.8-py3.10-gcc9-debug / test (default, 4, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh)
##[error]Response status code does not indicate success: 401 (Unauthorized).
periodic / linux-focal-cuda11.8-py3.10-gcc9-debug / test (default, 5, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh)
##[error]Response status code does not indicate success: 401 (Unauthorized).
periodic / linux-focal-cuda11.8-py3.10-gcc9-debug / test (default, 7, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh)
##[error]Response status code does not indicate success: 401 (Unauthorized).
periodic / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 1, 2, lf.linux.4xlarge) (gh)
inductor/test_aot_inductor_arrayref.py::AOTInductorTestABICompatibleCpuWithStackAllocation::test_while_loop_with_pytree_inputs_cpu_with_stack_allocation
periodic / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_AVX512, 2, 2, lf.linux.4xlarge) (gh)
##[error]Response status code does not indicate success: 401 (Unauthorized).
periodic / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 1, 2, lf.linux.4xlarge) (gh)
test_python_dispatch.py::TestPythonRegistration::test_returning_symint
periodic / linux-focal-cuda12.1-py3.10-gcc9 / test (nogpu_NO_AVX2, 2, 2, lf.linux.4xlarge) (gh)
/tmp/tmpoe97aer7/c255nrykvpjexf3lghkqpzhyao4awezxb4tgrxdiznd3d5kzayt6/cf4bdgos2hfmdgeb3fgg77fxuafdosh75b4z2ymqgvnzwjekd66e.cpp:702:54: error: no match for ‘operator=’ (operand types are ‘torch::aot_inductor::RAIIAtenTensorHandle’ and ‘std::remove_reference<torch::aot_inductor::ArrayRefTensor<float>&>::type’ {aka ‘torch::aot_inductor::ArrayRefTensor<float>’})
periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 1, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
test_python_dispatch.py::TestPythonRegistration::test_returning_symint
periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 3, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
##[error]Response status code does not indicate success: 401 (Unauthorized).
periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 4, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
inductor/test_aot_inductor_arrayref.py::AOTInductorTestABICompatibleCpuWithStackAllocation::test_while_loop_with_pytree_inputs_cpu_with_stack_allocation
periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (nogpu_AVX512, 1, 2, lf.linux.4xlarge) (gh)
inductor/test_aot_inductor_arrayref.py::AOTInductorTestABICompatibleCpuWithStackAllocation::test_while_loop_with_pytree_inputs_cpu_with_stack_allocation
periodic / linux-focal-cuda12.4-py3.10-gcc9 / test (nogpu_NO_AVX2, 2, 2, lf.linux.4xlarge) (gh)
/tmp/tmpz6m3ms69/c255nrykvpjexf3lghkqpzhyao4awezxb4tgrxdiznd3d5kzayt6/cdhit7w47un77dzdwketca2xojow4p2xi3elu7jd77zepfpaj5ri.cpp:702:54: error: no match for ‘operator=’ (operand types are ‘torch::aot_inductor::RAIIAtenTensorHandle’ and ‘std::remove_reference<torch::aot_inductor::ArrayRefTensor<float>&>::type’ {aka ‘torch::aot_inductor::ArrayRefTensor<float>’})
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 5, 5, lf.linux.4xlarge.nvidia.gpu) (gh)
'test/functorch/test_vmap.py::TestVmapOperatorsOpInfoCUDA::test_vmap_exhaustive_linalg_lu_factor_cuda_float32'

CANCELLED JOBS - The following jobs were cancelled. Please retry:

periodic / linux-focal-cuda11.8-py3.9-gcc9 / build (gh)
##[error]The operation was canceled.
periodic / linux-focal-cuda12.1-py3-gcc9-slow-gradcheck / build (gh)
##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

periodic / linux-focal-cuda11.8-py3.10-gcc9-debug / test (default, 2, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (trunk failure)
test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_linear_nt_dim_3_cuda
periodic / linux-focal-cuda11.8-py3.10-gcc9-debug / test (default, 3, 7, lf.linux.4xlarge.nvidia.gpu, oncall:debug-build) (gh) (trunk failure)
'test/profiler/test_profiler.py::TestProfiler::test_profile_all_threads'
periodic / linux-focal-cuda12.4-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, lf.linux.4xlarge.nvidia.gpu) (gh) (trunk failure)
Build completed, 1 test FAILED, 3229 total actions

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jeffdaily · 2024-12-05T22:20:33Z

@pytorchbot rebase

pytorchmergebot · 2024-12-05T22:22:07Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-12-05T22:22:09Z

Tried to rebase and push PR #142152, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

jeffdaily · 2024-12-06T00:33:53Z

@pytorchbot rebase -b main

pytorchmergebot · 2024-12-06T00:35:23Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2024-12-06T00:35:25Z

Successfully rebased rocm63_ci onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout rocm63_ci && git pull --rebase)

jeffdaily · 2024-12-16T18:17:39Z

@pytorchbot rebase

pytorchmergebot · 2024-12-16T18:19:17Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-12-16T18:19:20Z

Successfully rebased rocm63_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm63_ci && git pull --rebase)

jeffdaily · 2024-12-16T19:22:02Z

@pytorchbot rebase -b main

pytorchmergebot · 2024-12-16T19:23:28Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2024-12-16T19:23:30Z

Successfully rebased rocm63_ci onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout rocm63_ci && git pull --rebase)

jeffdaily · 2025-01-02T21:44:15Z

@pytorchbot rebase

pytorchmergebot · 2025-01-02T21:45:41Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-01-02T21:45:44Z

Successfully rebased rocm63_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm63_ci && git pull --rebase)

pruthvistony

LGTM

jeffdaily · 2025-01-09T17:12:26Z

@pytorchbot merge -f "ROCm-only changes to CI docker image and workflows; all ROCm tests passed; ignore all other failures as unrelated"

pytorchmergebot · 2025-01-09T17:14:00Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorch-bot bot added ciflow/rocm Trigger "default" config CI on ROCm module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Dec 5, 2024

pytorchbot added the open source label Dec 5, 2024

pytorchmergebot force-pushed the rocm63_ci branch from 1010f1c to 2f0fd66 Compare December 6, 2024 00:35

jeffdaily marked this pull request as ready for review December 11, 2024 00:02

jeffdaily requested review from a team and jeffdaily as code owners December 11, 2024 00:02

jeffdaily approved these changes Dec 11, 2024

View reviewed changes

jithunnair-amd added ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR ciflow/inductor-rocm Trigger "inductor" config CI on ROCm labels Dec 12, 2024

pytorchmergebot force-pushed the rocm63_ci branch from 9f157d0 to 00f5d8d Compare December 16, 2024 18:19

pytorchmergebot force-pushed the rocm63_ci branch from 00f5d8d to 4c4f4a1 Compare December 16, 2024 19:23

jithunnair-amd and others added 2 commits January 2, 2025 21:45

[ROCm][CI] upgrade CI to ROCm 6.3

f5e591e

reduce precision for one cpp test_api test

31d4d08

pytorchmergebot force-pushed the rocm63_ci branch from 4c4f4a1 to 31d4d08 Compare January 2, 2025 21:45

jeffdaily added 2 commits January 8, 2025 20:49

HIP runtime workaround, build custom 6.3 branch

8942057

fix clr build

970a3dc

pruthvistony approved these changes Jan 9, 2025

View reviewed changes

pytorchmergebot added the merging label Jan 9, 2025

pytorchmergebot closed this in 1365ae8 Jan 9, 2025

pytorchmergebot added Merged and removed merging labels Jan 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ROCm][CI] upgrade CI to ROCm 6.3 #142152

[ROCm][CI] upgrade CI to ROCm 6.3 #142152

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[ROCm][CI] upgrade CI to ROCm 6.3 #142152

[ROCm][CI] upgrade CI to ROCm 6.3 #142152

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/142152

❌ 13 New Failures, 2 Cancelled Jobs, 3 Unrelated Failures

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Merge started

Uh oh!

Uh oh!