8000 DISABLED AotInductorTest.BasicTestCpu (build.bin.test_aoti_inference) · Issue #152889 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

DISABLED AotInductorTest.BasicTestCpu (build.bin.test_aoti_inference) #152889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pytorch-bot bot opened this issue May 6, 2025 · 13 comments
Closed

DISABLED AotInductorTest.BasicTestCpu (build.bin.test_aoti_inference) #152889

pytorch-bot bot opened this issue May 6, 2025 · 13 comments
Labels
high priority module: aotinductor aot inductor module: flaky-tests Problem is a flaky test in CI oncall: pt2 skipped Denotes a (flaky) test currently skipped in CI. triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@pytorch-bot
Copy link
pytorch-bot bot commented May 6, 2025 •
edited
Loading

Platforms: inductor

This test was disabled because it is failing in CI. See recent examples and the most recent trunk workflow logs.

Over the past 3 hours, it has been determined flaky in 12 workflow(s) with 24 failures and 12 successes.

Debugging instructions (after clicking on the recent samples link):
DO NOT ASSUME THINGS ARE OKAY IF THE CI IS GREEN. We now shield flaky tests from developers so CI will thus be green but it will be harder to parse the logs.
To find relevant log snippets:

  1. Click on the workflow logs linked above
  2. Click on the Test step of the job so that it is expanded. Otherwise, the grepping will not work.
  3. Grep for AotInductorTest.BasicTestCpu
  4. There should be several instances run (as flaky tests are rerun in CI) from which you can study the logs.
Sample error message 
unknown file
C++ exception with description "Error in dlopen: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /tmp/torchinductor_jenkins/cycbfqzvaesb562wvzdixz3vb5gax4oe6ys7kwovlnnc2ceecyrz/cdsq7wn66anyx7drzdhzrmx7gs47b3nhhszv3xrt7bvzf26h6qtx.wrapper.so)
Exception raised from DynamicLibrary at ../aten/src/ATen/DynamicLibrary.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9c (0x7f56886e111c in /var/lib/jenkins/workspace/build/lib/libc10.so)
frame #1: <unknown function> + 0xdda719 (0x7f569fbda719 in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
frame #2: torch::inductor::AOTIModelContainerRunner::AOTIModelContainerRunner(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) + 0x123 (0x7f56a4c1b653 in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
frame #3: torch::inductor::AOTIModelContainerRunnerCpu::AOTIModelContainerRunnerCpu(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool) + 0x73 (0x7f56a4c1d2e3 in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x331c9 (0x55d2030d61c9 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #5: torch::aot_inductor::AotInductorTest_BasicTestCpu_Test::TestBody() + 0x43 (0x55d2030d6713 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #6: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) + 0x51 (0x55d203129a81 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #7: <unknown function> + 0x768b0 (0x55d2031198b0 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #8: testing::TestInfo::Run() + 0x40a (0x55d203119dca in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #9: <unknown function> + 0x7aea9 (0x55d20311dea9 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #10: testing::internal::UnitTestImpl::RunAllTests() + 0xf28 (0x55d20311f2f8 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #11: testing::UnitTest::Run() + 0x93 (0x55d20311fac3 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #12: main + 0x104 (0x55d2030d2c04 in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
frame #13: __libc_start_main + 0xf3 (0x7f5687e8a083 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #14: _start + 0x2e (0x55d2030d48de in /var/lib/jenkins/workspace/build/bin/test_aoti_inference)
" thrown in the test body.
unknown file:0: C++ failure

Test file path: `` or test/run_test

ResponseTimeoutError: Response timeout for 5000ms, GET https://raw.githubusercontent.com/pytorch/pytorch/main/test/ -1 (connected: true, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 0)
headers: {}

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @clee2000 @chauhang @penguinwu @desertfire @chenyang78 @yushangdi @benjaminglass1 @avikchaudhuri @gmagogsfm @zhxchen17 @tugsbayasgalan @angelayi @suo @ydwu4
@pytorch-bot pytorch-bot bot added module: flaky-tests Problem is a flaky test in CI skipped Denotes a (flaky) test currently skipped in CI. oncall: pt2 labels May 6, 2025
@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 6, 2025
Hello there! From the DISABLED prefix in this issue title, it looks like you are attempting to disabled a job in PyTorch CI. The information I have parsed is below:
  • Job name: AotInductorTest.BasicTestCpu (build.bin.test_aoti_inference)
  • Credential: pytorch-bot[bot]

Within ~15 minutes, AotInductorTest.BasicTestCpu (build.bin.test_aoti_inference) and all of its dependants will be disabled in PyTorch CI. Please verify that the job name looks correct. With great power comes great responsibility.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 6, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@masnesral masnesral added the module: aotinductor aot inductor label May 6, 2025
@masnesral masnesral added high priority triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module and removed oncall: export labels May 6, 2025
@masnesral
Copy link
Contributor

hi-pri per oncall runbook

@yushangdi yushangdi mentioned this issue May 7, 2025
Closed
@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 9, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 9, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 9, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

8000
@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 9, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 12, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 12, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 12, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 13, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 13, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

@pytorch-bot PyTorch Bot
Copy link
Author
pytorch-bot bot commented May 13, 2025

Another case of trunk flakiness has been found here. The list of platforms [inductor] appears to contain all the recently affected platforms [inductor]. Either the change didn't propogate fast enough or disable bot might be broken.

pytorchmergebot pushed a commit that referenced this issue May 14, 2025
@yushangdi @pytorchmergebot
Change aoti cpp tests to run serially within file (#152960)
b3dea0c 
Fixes #152674
#152889
#152888
#152891

`--dist=loadfile` ensures all tests in the same source file run in the same worker.

Tests like `FreeInactiveConstantBufferRuntimeConstantFoldingCuda` expect exclusive access to memory during test time to compute diffs (e.g., initMemory - updateMemory2 == DATASIZE).

With `-n 3`, tests run in separate processes, but CUDA device memory is shared — and cudaMemGetInfo() reads device-wide global state.

```
 python test/run_test.py --cpp --verbose -i cpp/test_aoti_inference -dist=loadfile
```
Pull Request resolved: #152960
Approved by: https://github.com/desertfire, https://github.com/cyyever
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
high priority module: aotinductor aot inductor module: flaky-tests Problem is a flaky test in CI oncall: pt2 skipped Denotes a (flaky) test currently skipped in CI. triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@desertfire @masnesral
0