8000 DISABLED AotInductorTest.BasicPackageLoaderTestCpu (build.bin.test_aoti_inference) · Issue #152891 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

DISABLED AotInductorTest.BasicPackageLoaderTestCpu (build.bin.test_aoti_inference) #152891

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pytorch-bot bot opened this issue May 6, 2025 · 12 comments
Open

DISABLED AotInductorTest.BasicPackageLoaderTestCpu (build.bin.test_aoti_inference) #152891

pytorch-bot bot opened this issue May 6, 2025 · 12 comments
Labels
high priority module: aotinductor aot inductor module: flaky-tests Problem is a flaky test in CI oncall: pt2 skipped Denotes a (flaky) test currently skipped in CI. triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@pytorch-bot
Copy link
pytorch-bot bot commented May 6, 2025 •
edited
Loading

Platforms: inductor

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

Over the past 3 hours, it has been determined flaky in 15 workflow(s) with 30 failures and 15 successes.

Debugging instructions (after clicking on the recent samples link):
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:

  1. Click on the workflow logs linked above
  2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
  3. Grep for AotInductorTest.BasicPackageLoaderTestCpu
  4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.
Sample error message 
unknown file
C++ exception with description "Error in dlopen: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /tmp/aioncA/data/aotinductor/model/c7h2yca3vsx4yicdfqxgycd7yal4vbvlzyz5zkddivrsf7pk3dij.wrapper.so)
Exception raised from DynamicLibrary at ../aten/src/ATen/DynamicLibrary.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9c (0x7faf5d59211c in /var/lib/jenkins/workspace/build/lib/libc10.so)
frame #1: <unknown function> + 0xdda719 (0x7faf749da719 in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
frame #2: torch::inductor::AOTIModelContainerRunner::AOTIModelContainerRunner(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) + 0x123 (0x7faf79a1b653 in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
frame #3: torch::inductor::AOTIModelContainerRunnerCpu::AOTIModelContainerRunnerCpu(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool) + 0x73 (0x7faf79a1d2e3 in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x5e1d38d (0x7faf79a1d38d in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
frame #5: torch::inductor::AOTIModelPackageLoader::AOTIModelPackageLoader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, unsigned long, signed char) + 0xe5a (0x7faf79a0cbda in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x34e70 (0x5635ded58e70 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #7: torch::aot_inductor::AotInductorTest_BasicPackageLoaderTestCpu_Test::TestBody() + 0x41 (0x5635ded59381 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #8: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) + 0x51 (0x5635dedaaa81 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #9: <unknown function> + 0x768b0 (0x5635ded9a8b0 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #10: testing::TestInfo::Run() + 0x40a (0x5635ded9adca in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #11: <unknown function> + 0x7aea9 (0x5635ded9eea9 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #12: testing::internal::UnitTestImpl::RunAllTests() + 0xf28 (0x5635deda02f8 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #13: testing::UnitTest::Run() + 0x93 (0x5635deda0ac3 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #14: main + 0x104 (0x5635ded53c04 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #15: __libc_start_main + 0xf3 (0x7faf5cd3b083 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #16: _start + 0x2e (0x5635ded558de in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
" thrown in the test body.
unknown file:0: C++ failure

Test file path: `` or test/run_test

ResponseTimeoutError: Response timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/main/test/ -1 (connected: true, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 0)
headers: {}

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @clee2000 @chauhang @penguinwu @desertfire @chenyang78 @yushangdi @benjaminglass1 @avikchaudhuri @gmagogsfm @zhxchen17 @tugsbayasgalan @angelayi @suo @ydwu4
@pytorch-bot pytorch-bot bot added module: flaky-tests Problem is a flaky test in CI skipped Denotes a (flaky) test currently skipped in CI. oncall: pt2 labels May 6, 2025
@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 6, 2025
Hello there! From the DISABLED prefix in this issue title, it looks like you are attempting to disabled a job in PyTorch CI. The information I have parsed is below:
  • Job name: AotInductorTest.BasicPackageLoaderTestCpu (build.bin.test_aoti_inference)
  • Credential: pytorch-bot[bot]

Within ~15 minutes, AotInductorTest.BasicPackageLoaderTestCpu (build.bin.test_aoti_inference) and all of its dependants will be disabled in PyTorch CI. Please verify that the job name looks correct. With great power comes great responsibility.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 6, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@masnesral masnesral added the module: aotinductor aot inductor label May 6, 2025
@masnesral masnesral added high priority triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module and removed oncall: export labels May 6, 2025
@masnesral
Copy link
Contributor

hi-pri per oncall runbook

@yushangdi yushangdi mentioned this issue May 7, 2025
Closed
@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 9, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 9, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 9, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 9, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 12, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 12, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 13, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 13, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 13, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

pytorchmergebot pushed a commit that referenced this issue May 14, 2025
@yushangdi @pytorchmergebot
Change aoti cpp tests to run serially within file (#152960)
b3dea0c 
Fixes #152674
#152889
#152888
#152891

`--dist=loadfile` ensures all tests in the same source file run in the same worker.

Tests like `FreeInactiveConstantBufferRuntimeConstantFoldingCuda` expect exclusive access to memory during test time to compute diffs (e.g., initMemory - updateMemory2 == DATASIZE).

With `-n 3`, tests run in separate processes, but CUDA device memory is shared — and cudaMemGetInfo() reads device-wide global state.

```
 python test/run_test.py --cpp --verbose -i cpp/test_aoti_inference -dist=loadfile
```
Pull Request resolved: #152960
Approved by: https://github.com/desertfire, https://github.com/cyyever
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
high priority module: aotinductor aot inductor module: flaky-tests Problem is a flaky test in CI oncall: pt2 skipped Denotes a (flaky) test currently skipped in CI. triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@masnesral
0