8000 gh-111062: Reusable Ubuntu build that supports free-threaded mode as the conditional CI by NCLI · Pull Request #111452 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

gh-111062: Reusable Ubuntu build that supports free-threaded mode as the conditional CI #111452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Oct 30, 2023
Merged

gh-111062: Reusable Ubuntu build that supports free-threaded mode as the conditional CI #111452

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
6c35a89
initial draft of no-gil CI/CD workflow
NCLI Oct 29, 2023
c35d2ce
Moved to reusable ubuntu workflow.
NCLI Oct 29, 2023
0ad8938
Disabled additional test
NCLI Oct 29, 2023
05166b6
Fixed workflow reference
NCLI Oct 29, 2023
ce06c53
Fixed syntax issue for ubuntu_version.
NCLI Oct 29, 2023
3d0ab04
Fixed syntax issue for ubuntu_version.
NCLI Oct 29, 2023
f1f0ff3
Fixed syntax issue for ubuntu_version v3.
NCLI Oct 29, 2023
5fd16f5
Fixed syntax issue for ubuntu_version v4.
NCLI Oct 29, 2023
6f11e3a
Fixed test run condition.
NCLI Oct 29, 2023
e2d21b0
Fixed test run condition v2
NCLI Oct 29, 2023
badb818
Removed needs from reusable workflow
NCLI Oct 29, 2023
01af84a
Moved needs back to build.yml
NCLI Oct 29, 2023
1cb1f77
Fixed conditional
NCLI Oct 29, 2023
7949d22
Set ubuntu version
NCLI Oct 29, 2023
6d05b08
Reordered variables
NCLI Oct 29, 2023
fd24bcc
Merge branch 'main' into reusable_ubuntu_build
NCLI Oct 29, 2023
f57f1c3
Remove drequirement for ubuntu_version
NCLI Oct 29, 2023
3e19889
Ubuntu build test with GIL disabled - first pass
NCLI Oct 29, 2023
5cb40bb
Added changes to the reusable workflow
NCLI Oct 29, 2023
e611636
Reenabled all tests, fixed import
NCLI Oct 29, 2023
65e38f3
Evaluate input variable
NCLI Oct 29, 2023
3225b9b
Evaluate input variable
NCLI Oct 29, 2023
5dd7c5e
Merge branch 'main' into reusable_ubuntu_build_nogil
NCLI Oct 29, 2023
12c9def
Update .github/workflows/build.yml
NCLI Oct 29, 2023
8000
6d36072
Update .github/workflows/build.yml
NCLI Oct 29, 2023
adf483b
Update .github/workflows/build.yml
NCLI Oct 29, 2023
8cffab1
Update .github/workflows/build.yml
NCLI Oct 29, 2023
26ec9b9
Update .github/workflows/build.yml
corona10 Oct 29, 2023
acdbec2
Update .github/workflows/build.yml
corona10 Oct 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 21 additions & 58 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,70 +248,30 @@ jobs:

build_ubuntu:
name: 'Ubuntu'
runs-on: ubuntu-20.04
timeout-minutes: 60
needs: check_source
if: needs.check_source.outputs.run_tests == 'true'
env:
OPENSSL_VER: 3.0.11
PYTHONSTRICTEXTENSIONBUILD: 1
steps:
- uses: actions/checkout@v4
- name: Register gcc problem matcher
run: echo "::add-matcher::.github/problem-matchers/gcc.json"
- name: Install dependencies
run: sudo ./.github/workflows/posix-deps-apt.sh
- name: Configure OpenSSL env vars
run: |
echo "MULTISSL_DIR=${GITHUB_WORKSPACE}/multissl" >> $GITHUB_ENV
echo "OPENSSL_DIR=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}" >> $GITHUB_ENV
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV
- name: 'Restore OpenSSL build'
id: cache-openssl
uses: actions/cache@v3
with:
path: ./multissl/openssl/${{ env.OPENSSL_VER }}
key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }}
- name: Install OpenSSL
if: steps.cache-openssl.outputs.cache-hit != 'true'
run: python3 Tools/ssl/multissltests.py --steps=library --base-directory $MULTISSL_DIR --openssl $OPENSSL_VER --system Linux
- name: Add ccache to PATH
run: |
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
- name: Configure ccache action
uses: hendrikmuhs/ccache-action@v1.2
- name: Setup directory envs for out-of-tree builds
run: |
echo "CPYTHON_RO_SRCDIR=$(realpath -m ${GITHUB_WORKSPACE}/../cpython-ro-srcdir)" >> $GITHUB_ENV
echo "CPYTHON_BUILDDIR=$(realpath -m ${GITHUB_WORKSPACE}/../cpython-builddir)" >> $GITHUB_ENV
- name: Create directories for read-only out-of-tree builds
run: mkdir -p $CPYTHON_RO_SRCDIR $CPYTHON_BUILDDIR
- name: Bind mount sources read-only
run: sudo mount --bind -o ro $GITHUB_WORKSPACE $CPYTHON_RO_SRCDIR
- name: Restore config.cache
uses: actions/cache@v3
with:
path: ${{ env.CPYTHON_BUILDDIR }}/config.cache
key: ${{ github.job }}-${{ runner.os }}-${{ needs.check_source.outputs.config_hash }}
- name: Configure CPython out-of-tree
working-directory: ${{ env.CPYTHON_BUILDDIR }}
run: |
uses: ./.github/workflows/reusable-build-ubuntu.yml
with:
config_hash: ${{ needs.check_source.outputs.config_hash }}
options: |
../cpython-ro-srcdir/configure \
--config-cache \
--with-pydebug \
--with-openssl=$OPENSSL_DIR
- name: Build CPython out-of-tree
working-directory: ${{ env.CPYTHON_BUILDDIR }}
run: make -j4
- name: Display build info
working-directory: ${{ env.CPYTHON_BUILDDIR }}
run: make pythoninfo
- name: Remount sources writable for tests
# some tests write to srcdir, lack of pyc files slows down testing
run: sudo mount $CPYTHON_RO_SRCDIR -oremount,rw
- name: Tests
working-directory: ${{ env.CPYTHON_BUILDDIR }}
run: xvfb-run make test

build_ubuntu_free_threaded:
name: 'Ubuntu (free-threaded)'
needs: check_source
if: needs.check_source.outputs.run_tests == 'true' && contains(github.event.pull_request.labels.*.name, 'topic-free-threaded')
uses: ./.github/workflows/reusable-build-ubuntu.yml
with:
config_hash: ${{ needs.check_source.outputs.config_hash }}
options: |
../cpython-ro-srcdir/configure \
--config-cache \
--with-pydebug \
--with-openssl=$OPENSSL_DIR \
--disable-gil

build_ubuntu_ssltests:
name: 'Ubuntu SSL tests with OpenSSL'
Expand Down Expand Up @@ -582,6 +542,7 @@ jobs:
- build_win_arm64
- build_macos
- build_ubuntu
- build_ubuntu_free_threaded
- build_ubuntu_ssltests
- test_hypothesis
- build_asan
Expand All @@ -595,6 +556,7 @@ jobs:
with:
allowed-failures: >-
build_macos,
build_ubuntu_free_threaded,
build_ubuntu_ssltests,
build_win32,
build_win_arm64,
Expand All @@ -617,6 +579,7 @@ jobs:
build_win_arm64,
build_macos,
build_ubuntu,
build_ubuntu_free_threaded,
build_ubuntu_ssltests,
build_asan,
'
Expand Down
70 changes: 70 additions & 0 deletions .github/workflows/reusable-build-ubuntu.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
on:
workflow_call:
inputs:
config_hash:
required: true
type: string
options:
required: true
type: string

jobs:
build_ubuntu_reusable:
timeout-minutes: 60
runs-on: ubuntu-20.04
env:
OPENSSL_VER: 3.0.11
PYTHONSTRICTEXTENSIONBUILD: 1
steps:
- uses: actions/checkout@v4
- name: Register gcc problem matcher
run: echo "::add-matcher::.github/problem-matchers/gcc.json"
- name: Install dependencies
run: sudo ./.github/workflows/posix-deps-apt.sh
- name: Configure OpenSSL env vars
run: |
echo "MULTISSL_DIR=${GITHUB_WORKSPACE}/multissl" >> $GITHUB_ENV
echo "OPENSSL_DIR=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}" >> $GITHUB_ENV
echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV
- name: 'Restore OpenSSL build'
id: cache-openssl
uses: actions/cache@v3
with:
path: ./multissl/openssl/${{ env.OPENSSL_VER }}
key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }}
- name: Install OpenSSL
if: steps.cache-openssl.outputs.cache-hit != 'true'
run: python3 Tools/ssl/multissltests.py --steps=library --base-directory $MULTISSL_DIR --openssl $OPENSSL_VER --system Linux
- name: Add ccache to PATH
run: |
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
- name: Configure ccache action
uses: hendrikmuhs/ccache-action@v1.2
- name: Setup directory envs for out-of-tree builds
run: |
echo "CPYTHON_RO_SRCDIR=$(realpath -m ${GITHUB_WORKSPACE}/../cpython-ro-srcdir)" >> $GITHUB_ENV
echo "CPYTHON_BUILDDIR=$(realpath -m ${GITHUB_WORKSPACE}/../cpython-builddir)" >> $GITHUB_ENV
- name: Create directories for read-only out-of-tree builds
run: mkdir -p $CPYTHON_RO_SRCDIR $CPYTHON_BUILDDIR
- name: Bind mount sources read-only
run: sudo mount --bind -o ro $GITHUB_WORKSPACE $CPYTHON_RO_SRCDIR
- name: Restore config.cache
uses: actions/cache@v3
with:
path: ${{ env.CPYTHON_BUILDDIR }}/config.cache
key: ${{ github.job }}-${{ runner.os }}-${{ inputs.config_hash }}
- name: Configure CPython out-of-tree
working-directory: ${{ env.CPYTHON_BUILDDIR }}
run: ${{ inputs.options }}
- name: Build CPython out-of-tree
working-directory: ${{ env.CPYTHON_BUILDDIR }}
run: make -j4
- name: Display build info
working-directory: ${{ env.CPYTHON_BUILDDIR }}
run: make pythoninfo
- name: Remount sources writable for tests
# some tests write to srcdir, lack of pyc files slows down testing
run: sudo mount $CPYTHON_RO_SRCDIR -oremount,rw
- name: Tests
working-directory: ${{ env.CPYTHON_BUILDDIR }}
run: xvfb-run make test
0