8000 Parallel build race condition on AIX since python-2.7 · Issue #63720 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

Parallel build race condition on AIX since python-2.7 #63720

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
haubi mannequin opened this issue Nov 7, 2013 · 15 comments
Closed

Parallel build race condition on AIX since python-2.7 #63720

haubi mannequin opened this issue Nov 7, 2013 · 15 comments
Labels
3.9 only security fixes 3.10 only security fixes build The build process and cross-build type-bug An unexpected behavior, bug, or error

Comments

@haubi
Copy link
Mannequin
haubi mannequin commented Nov 7, 2013
BPO 19521
Nosy @loewis, @ericvw, @haubi, @skrah, @aixtools, @miss-islington, @kadler, @isidentical
PRs
  • bpo-19521: fix parallel build on AIX #682
  • bpo-19521: Fix parallel build race condition on AIX #21997
  • [3.9] bpo-19521: Fix parallel build race condition on AIX (GH-21997) #22001
  • [3.8] bpo-19521: Fix parallel build race condition on AIX (GH-21997) #22002
    		•
    Files
  • python-tip-aix-parallel.patch: Fix parallel build race condition on AIX
  • issue19521-parallel-build-race-on-aix.patch: patch for 3.4
  • aix-parallel-build-race-refresh.patch: Refreshed 2014 patch for tip of CPython
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2013-11-07.16:04:54.897>
labels = ['3.10', 'type-bug', '3.9', 'build']
title = 'Parallel build race condition on AIX since python-2.7'
updated_at = <Date 2020-09-10.12:22:11.359>
user = 'https://github.com/haubi'

    bugs.python.org fields:

    activity = <Date 2020-09-10.12:22:11.359>
actor = 'skrah'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2013-11-07.16:04:54.897>
creator = 'haubi'
dependencies = []
files = ['32534', '35479', '45665']
hgrepos = ['251']
issue_num = 19521
keywords = ['patch']
message_count = 15.0
messages = ['202357', '202379', '202407', '202435', '219742', '281827', '281833', '375626', '375627', '376059', '376063', '376067', '376069', '376070', '376681']
nosy_count = 9.0
nosy_names = ['loewis', 'ericvw', 'haubi', 'skrah', 'David.Edelsohn', 'Michael.Felt', 'miss-islington', 'kadler', 'BTaskaya']
pr_nums = ['682', '21997', '22001', '22002']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19521'
versions = ['Python 3.9', 'Python 3.10']
    @haubi
    Copy link
    Mannequin Author
    haubi mannequin commented Nov 7, 2013

    Since python-3.2, there is a race condition building in parallel on AIX:

    Consider these Makefile(.pre.in) rules:

    $(BUILDPYTHON): ...
    $(LINKCC) ... $(LINKFORSHARED) ...

    Modules/_testembed: ...
    $(LINKCC) ... $(LINKFORSHARED) ...

    Modules/_freeze_importlib: ...
    $(LINKCC) ...

    On AIX, the variables get these values:

    LINKCC = $(srcdir)/Modules/makexp_aix Modules/python.exp ...
    LINKFORSHARED = -Wl,-bE:Modules/python.exp ...

    Now $(BUILDPYTHON) and Modules/_testembed may run in parallel, causing Modules/python.exp to be created by two instances of makexp_aix eventually running at the same time.

    Attached patch fixes this problem for cpython tip (doubt supporting AIX 4.1 and earlier still is necessary).

    Thank you!

    @haubi haubi mannequin added the build The build process and cross-build label Nov 7, 2013
    @loewis
    Copy link
    Mannequin
    loewis mannequin commented Nov 7, 2013

    Wouldn't it be better if linking _testembed generated _testembed.exp instead of generating python.exp? I hope using $@.exp somehow could help. Hard-coding the name of the export file sounds like a flaw in the first place.

    @haubi
    Copy link
    Mannequin Author
    haubi mannequin commented Nov 8, 2013

    I'm unsure about the real purpose of _testembed, but given the name it does make sense to me to export the same symbols as $(BUILDPYTHON), thus reusing python.exp.

    @DavidEdelsohn
    Copy link
    Mannequin
    DavidEdelsohn mannequin commented Nov 8, 2013

    +1

    @haubi
    Copy link
    Mannequin Author
    haubi mannequin commented Jun 4, 2014

    Patch including configure update now.

    @ericvw ericvw mannequin added the 3.7 (EOL) end of life label Nov 27, 2016
    @ericvw
    Copy link
    Mannequin
    ericvw mannequin commented Nov 27, 2016

    I also have found this goes back since Python 2.7.

    I have refreshed the patched for the tip of CPython. What can I do to help push this forward?

    @ericvw ericvw mannequin changed the title parallel build race condition on AIX since python-3.2 Parallel build race condition on AIX since python-2.7 Nov 27, 2016
    @ericvw
    Copy link
    Mannequin
    ericvw mannequin commented Nov 27, 2016

    I may be able to simplify the build on AIX by removing ld_so_aix and python.exp entirely. Would this be a preferred solution if I am able to get something working? If so, should I create a separate issue to track the change?

    @skrah skrah mannequin added 3.10 only security fixes and removed 3.7 (EOL) end of life labels Aug 15, 2020
    @skrah
    Copy link
    Mannequin
    skrah mannequin commented Aug 18, 2020

    The original patch is a bit dated, do we still need the export symbol generation?

    https://www.ibm.com/support/knowledgecenter/en/SSGH3R_16.1.0/com.ibm.xlcpp161.aix.doc/proguide/compiling_shared_aix.html

    "Use the -G option to create a shared library from the generated object files, and to enable runtime linking with applications that support it."

    "If you do not specify a -bE option, all the global symbols are exported except for those symbols that have the hidden or internal visibility attribute."

    @DavidEdelsohn
    Copy link
    Mannequin
    DavidEdelsohn mannequin commented Aug 18, 2020

    Yes, export file generation still is required.

    Python does not need to utilize runtime linking. Using -G is a very bad choice and severely discouraged with severe performance and other penalties.

    @skrah
    Copy link
    Mannequin
    skrah mannequin commented Aug 29, 2020

    Okay, thanks. The -G option is of course attractive for Linux projects
    because it requires minimal changes in the build machinery.

    I've added support for libmpdec/libmpdec++ (next release) for AIX-style
    shared libraries (shr.o inside libmpdec.a). The AIX linker has a certain
    elegance, but it requires something like 150 lines of code changes and
    conditionals inside the Makefiles.

    I can confirm that -G is substantially slower, so I'm going to commit this
    patch soon.

    @skrah
    Copy link
    Mannequin
    skrah mannequin commented Aug 29, 2020

    I can't find the reason for:

        if test -z "$EXPORTSYMS"; then
        EXPORTSYMS="[Modules/python.exp](https://github.com/python/cpython/blob/main/Modules/python.exp)"
    fi

    Modules/python.exp is hardcoded elsewhere, and I'd rather set
    EXPORTSYMS unconditionally on AIX and unset it unconditionally
    for all other systems (which don't build with the patch if
    EXPORTSYMS happens to be defined).

    @skrah
    Copy link
    Mannequin
    skrah mannequin commented 8000 Aug 29, 2020

    New changeset e6dcd37 by Stefan Krah in branch 'master':
    bpo-19521: Fix parallel build race condition on AIX (GH-21997)
    e6dcd37

    @skrah
    Copy link
    Mannequin
    skrah mannequin commented Aug 29, 2020

    New changeset 88b86a9 by Miss Islington (bot) in branch '3.9':
    bpo-19521: Fix parallel build race condition on AIX (GH-22001)
    88b86a9

    @skrah
    Copy link
    Mannequin
    skrah mannequin commented Aug 29, 2020

    This is in master and 3.9.1. I'll not backport to 3.8 because a release candidate is imminent.

    @skrah skrah mannequin added the 3.9 only security fixes label Aug 29, 2020
    @skrah skrah mannequin closed this as completed Aug 29, 2020
    @skrah skrah mannequin added the type-bug An unexpected behavior, bug, or error label Aug 29, 2020
    @skrah
    Copy link
    Mannequin
    skrah mannequin commented Sep 10, 2020

    I have been asked to backport this to 3.8. There's a very small window
    of opportunity:

    3.8.6: Monday, 2020-09-21
    3.8.7rc1: Monday, 2020-11-02
    3.8.7: Monday, 2020-11-16 (final version!)

    Backporting procedures since 3.8 are unclear and a source of
    constant friction, so most core developers seem to have stopped
    doing any backports at all. It is not a battle I'll choose, but
    if you get a second core dev to review this trivial patch I'll
    commit it.

    There's a simple solution for 3.8: Do not use the parallel build,
    the regular build takes around 4 min.

    For the buildbots you can ask the operator for a custom command
    line.

    @skrah skrah mannequin reopened this Sep 10, 2020
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes 3.10 only security fixes build The build process and cross-build type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @kumaraditya303
    0