8000 gh-128515: Add BOLT build to CI by zanieb · Pull Request #128845 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-128515: Add BOLT build to CI #128845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jan 18, 2025
Merged

gh-128515: Add BOLT build to CI #128845

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
8699afc
Add BOLT support to reusable Ubuntu workflow
zanieb Jan 14, 2025
254b553
Add BOLT to build matrix
zanieb Jan 14, 2025
9b179cb
Install Clang
zanieb Jan 14, 2025
684ece4
Skip BOLT CI on aarch64
zanieb Jan 14, 2025
03be932
Ignore `test_unpickle_module_race` during BOLT profiling
zanieb Jan 14, 2025
01cb8d8
Skip perf trampoline tests when BOLT instrumented
zanieb Jan 15, 2025
5f9c410
Move `bolt` flags into alphabetical order
zanieb Jan 16, 2025
6aa007c
Merge branch 'main' into zb/bolt
corona10 Jan 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Ignore test_unpickle_module_race during BOLT profiling
  • Loading branch information
@zanieb
zanieb committed Jan 14, 2025
commit 03be9327c5e98a018dd5444b1bf2f8c79dc91698
5 changes: 4 additions & 1 deletion .github/workflows/reusable-ubuntu.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
run: echo "::add-matcher::.github/problem-matchers/gcc.json"
- name: Install dependencies
run: sudo ./.github/workflows/posix-deps-apt.sh
- name: Install Clang
- name: Install Clang and BOLT
if: ${{ fromJSON(inputs.bolt-optimizations) }}
run: |
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" ./llvm.sh 19
Expand Down Expand Up @@ -84,7 +84,10 @@ jobs:
key: ${{ github.job }}-${{ runner.os }}-${{ env.IMAGE_VERSION }}-${{ inputs.config_hash }}
- name: Configure CPython out-of-tree
working-directory: ${{ env.CPYTHON_BUILDDIR }}
# `test_unpickle_module_race` writes to the source directory, which is
# read-only during builds — so we exclude it from profiling with BOLT.
run: >-
PROFILE_TASK='-m test --pgo --ignore test_unpickle_module_race'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, why doesn't it raise any issues while we build PGO build?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there is a PGO build in CI

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The read-only build file system looks specific to this CI setup)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there is a PGO build in CI

FYI, At Github Action there is no CI for PGO and LTO.
But at build bot we run CI for the PGO and LTO.
https://buildbot.python.org/#/builders/378/builds/1554

Copy link
Me 8000 mber

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The read-only build file system looks specific to this CI setup)

Let me take a look more detail. :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/python/cpython/actions/runs/12776586124/job/35615597427

ERROR: test_unpickle_module_race (test.test_pickle.PyUnpicklerTests.test_unpickle_module_race)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/import_helper.py", line 48, in forget
    unlink(source + 'c')
    ~~~~~~^^^^^^^^^^^^^^
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/os_helper.py", line 345, in unlink
    _unlink(filename)
    ~~~~~~~^^^^^^^^^^
OSError: [Errno 30] Read-only file system: '/home/runner/work/cpython/cpython-ro-srcdir/Lib/locker.pyc'

I think it's fine to remove the test from the optimization suite. It seems likely for there to be some problems here due to the read-only setup. It's known that the tests require a writeable source directory

cpython/.github/workflows/reusable-ubuntu.yml

Lines 105 to 110 in ae31df3

- name: Remount sources writable for tests
# some tests write to srcdir, lack of pyc files slows down testing
run: sudo mount "$CPYTHON_RO_SRCDIR" -oremount,rw
- name: Tests
working-directory: ${{ env.CPYTHON_BUILDDIR }}
run: xvfb-run make ci

Copy link
Contributor Author
@zanieb zanieb Jan 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#29904 added the read-only out of tree buil 8000 ds.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(We could disable the read-only builds for the BOLT job but it seems more painful than it's worth)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay let’s exclude the test with current way and let’s pile the issue about the test suite problem. Thank you for the investigation:)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hugovk Do we have any reason to mount file system as the read-only?

Zanie found it was added in #29904, which gives this reason:

The Ubuntu test runner now configures and compiles CPython out of tree.
The source directory is a read-only bind mount to ensure that the build
cannot create or modify any files in the source tree.

../cpython-ro-srcdir/configure
--config-cache
--with-pydebug
Expand Down
Loading
0