[CMAKE] Look for `Development.Module` instead of `Development` #129669

oraluben · 2024-06-27T15:37:24Z

Based on the cmake issue and manylinux issue, when building a python module, it should find the Development.Module module, not Development, which includes Development.Module and Development.Embed, and will expect the shared python library only. After this PR and before #124613, pytorch could be built with a static libpython (e.g. in manylinux).

pytorch-bot · 2024-06-27T15:37:27Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129669

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a92b687 with merge base 1e61cb8 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

colesbury · 2024-06-27T21:50:37Z

@malfet would you please review this?

oraluben · 2024-06-29T09:16:04Z

Is it possible to land this on main before 1/7 so we can backport it to 2.4? I've created a simple example that triggers this:

FROM quay.io/pypa/manylinux2014_x86_64

RUN git clone --recurse-submodules --shallow-submodules -j8 https://github.com/pytorch/pytorch.git /pytorch
RUN python3.10 -mvenv /venv

WORKDIR /pytorch

RUN /venv/bin/pip install ninja
RUN /venv/bin/pip install -r requirements.txt
RUN source /venv/bin/activate && python setup.py develop

gives:

...
508.7 [8039/8053] Linking CXX shared module functorch/functorch.so
508.7 FAILED: functorch/functorch.so
508.7 : && /opt/rh/devtoolset-10/root/usr/bin/c++ -fPIC -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DTMP_
USE_TSC_AS_TIMESTAMP -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -W
error=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unu
sed-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno$
error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -W$
rror=format -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG  -Wl,--no-as-needed -shared  -o functorch/functorch.so functorch/CMakeFiles/functorch.dir/csrc/dim/dim.cpp.o functor$
h/CMakeFiles/functorch.dir/csrc/dim/dim_opcode.c.o functorch/CMakeFiles/functorch.dir/csrc/init_dim_only.cpp.o  -Wl,-rpath,/pytorch/build/lib::  lib/libtorch.so  -ltorch_pyt$
on  -Wl,--no-as-needed,"/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  -pthread  lib/libc10.so && /opt/_internal/pipx/venvs/cmake/lib/python3.10/sit$
-packages/cmake/data/bin/cmake -E __run_co_compile --lwyu="ldd;-u;-r" --source=functorch/functorch.so && :
508.7 /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: cannot find -ltorch_python
508.7 collect2: error: ld returned 1 exit status

malfet · 2024-07-09T03:02:39Z

@pytorchbot merge -i

pytorchmergebot · 2024-07-09T03:04:20Z

Merge failed

Reason: This PR needs a release notes: label
If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Details for Dev Infra team

Raised by workflow job

oraluben · 2024-07-09T03:05:02Z

@pytorchbot label "topic: not user facing"

oraluben · 2024-07-09T03:09:54Z

@malfet Would you mind merge again?

oraluben · 2024-07-09T04:04:34Z

@pytorchmergebot merge

pytorchmergebot · 2024-07-09T04:06:09Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-07-09T04:06:16Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, linux.g5.4xlarge.nvidia.gpu)

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

oraluben · 2024-07-09T04:09:07Z

@pytorchbot help

pytorch-bot · 2024-07-09T04:09:09Z

❌ 🤖 pytorchbot command failed:

@pytorchbot: error: argument command: invalid choice: 'help' (choose from 'merge', 'revert', 'rebase', 'label', 'drci', 'cherry-pick', 'close')

usage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick,close} ...

Try @pytorchbot --help for more info.

oraluben · 2024-07-09T04:09:36Z

@pytorchbot --help

pytorch-bot · 2024-07-09T04:09:38Z

PyTorchBot Help

usage: @pytorchbot [-h] {merge,revert,rebase,label,drci,cherry-pick,close} ...

In order to invoke the bot on your PR, include a line that starts with
@pytorchbot anywhere in a comment. That line will form the command; no
multi-line commands are allowed. Some commands may be used on issues as specified below.

Example:
    Some extra context, blah blah, wow this PR looks awesome

    @pytorchbot merge

optional arguments:
  -h, --help            Show this help message and exit.

command:
  {merge,revert,rebase,label,drci,cherry-pick,close}
    merge               Merge a PR
    revert              Revert a PR
    rebase              Rebase a PR
    label               Add label to a PR
    drci                Update Dr. CI
    cherry-pick         Cherry pick a PR onto a release branch
    close               Close a PR

Merge

usage: @pytorchbot merge [-f MESSAGE | -i] [-ic] [-r [{viable/strict,main}]]

Merge an accepted PR, subject to the rules in .github/merge_rules.json.
By default, this will wait for all required checks (lint, pull) to succeed before merging.

optional arguments:
  -f MESSAGE, --force MESSAGE
                        Merge without checking anything. This requires a reason for auditting purpose, for example:
                        @pytorchbot merge -f 'Minor update to fix lint. Expecting all PR tests to pass'
                        
                        Please use `-f` as last resort, prefer `--ignore-current` to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.
  -i, --ignore-current  Merge while ignoring the currently failing jobs.  Behaves like -f if there are no pending jobs.
  -ic                   Old flag for --ignore-current. Deprecated in favor of -i.
  -r [{viable/strict,main}], --rebase [{viable/strict,main}]
                        Rebase the PR to re run checks before merging.  Accepts viable/strict or main as branch options and will default to viable/strict if not specified.

Revert

usage: @pytorchbot revert -m MESSAGE -c
                          {nosignal,ignoredsignal,landrace,weird,ghfirst}

Revert a merged PR. This requires that you are a Meta employee.

Example:
  @pytorchbot revert -m="This is breaking tests on trunk. hud.pytorch.org/" -c=nosignal

optional arguments:
  -m MESSAGE, --message MESSAGE
                        The reason you are reverting, will be put in the commit message. Must be longer than 3 words.
  -c {nosignal,ignoredsignal,landrace,weird,ghfirst}, --classification {nosignal,ignoredsignal,landrace,weird,ghfirst}
                        A machine-friendly classification of the revert reason.

Rebase

usage: @pytorchbot rebase [-s | -b BRANCH]

Rebase a PR. Rebasing defau
8000
lts to the stable viable/strict branch of pytorch.
Repeat contributor may use this command to rebase their PR.

optional arguments:
  -s, --stable          [DEPRECATED] Rebase onto viable/strict
  -b BRANCH, --branch BRANCH
                        Branch you would like to rebase to

Label

usage: @pytorchbot label labels [labels ...]

Adds label to a PR or Issue [Can be used on Issues]

positional arguments:
  labels  Labels to add to given Pull Request or Issue [Can be used on Issues]

Dr CI

usage: @pytorchbot drci 

Update Dr. CI. Updates the Dr. CI comment on the PR in case it's gotten out of sync with actual CI results.

cherry-pick

usage: @pytorchbot cherry-pick --onto ONTO [--fixes FIXES] -c
                               {regression,critical,fixnewfeature,docs,release}

Cherry pick a pull request onto a release branch for inclusion in a release

optional arguments:
  --onto ONTO           Branch you would like to cherry pick onto (Example: release/2.1)
  --fixes FIXES         Link to the issue that your PR fixes (Example: https://github.com/pytorch/pytorch/issues/110666)
  -c {regression,critical,fixnewfeature,docs,release}, --classification {regression,critical,fixnewfeature,docs,release}
                        A machine-friendly classification of the cherry-pick reason.

Close

usage: @pytorchbot close

Close a PR [Can be used on issues]

oraluben · 2024-07-09T04:10:14Z

@pytorchbot drci

oraluben · 2024-07-09T05:15:03Z

@pytorchbot rebase

pytorchmergebot · 2024-07-09T05:16:29Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-07-09T05:16:33Z

Successfully rebased update-cmake-find-python onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout update-cmake-find-python && git pull --rebase)

oraluben · 2024-07-09T08:23:14Z

@pytorchbot merge -i

pytorch-bot · 2024-07-09T08:23:17Z

-i flag is only allowed for users with write permissions

oraluben · 2024-07-09T08:23:36Z

@pytorchmergebot merge

pytorchmergebot · 2024-07-09T08:25:09Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…ch#129669) Based on the [cmake issue](https://gitlab.kitware.com/cmake/cmake/-/issues/23716) and [manylinux issue](pypa/manylinux#1347), when building a python module, it should find the `Development.Module` module, not `Development`, which includes `Development.Module` and `Development.Embed`, and will expect the shared python library only. After this PR and before pytorch#124613, pytorch could be built with a static libpython (e.g. in manylinux). Pull Request resolved: pytorch#129669 Approved by: https://github.com/malfet

pytorchbot added the open source label Jun 27, 2024

colesbury requested a review from malfet June 27, 2024 21:50

This was referenced Jun 28, 2024

[v.2.4.0] Release Tracker #128436

Closed

[CMAKE] Look for Development.Module instead of Development #129729

Merged

soulitzer added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jul 2, 2024

oraluben mentioned this pull request Jul 9, 2024

Failure to find python error for GCC 11 and Manylinux 2_28 #130278

Closed

malfet approved these changes Jul 9, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 9, 2024

pytorchmergebot added the merging label Jul 9, 2024

pytorchmergebot removed the merging label Jul 9, 2024

pytorch-bot bot added the topic: not user facing topic category label Jul 9, 2024

pytorchmergebot added the merging label Jul 9, 2024

pytorchmergebot removed the merging label Jul 9, 2024

[CMAKE] Look for Development.Module instead of Development

a92b687

pytorchmergebot force-pushed the update-cmake-find-python branch from b7f402e to a92b687 Compare July 9, 2024 05:16

pytorchmergebot added the merging label Jul 9, 2024

pytorchmergebot added the Merged label Jul 9, 2024

pytorchmergebot closed this in 953c647 Jul 9, 2024

pytorchmergebot removed the merging label Jul 9, 2024

oraluben deleted the update-cmake-find-python branch July 9, 2024 12:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CMAKE] Look for `Development.Module` instead of `Development` #129669

[CMAKE] Look for `Development.Module` instead of `Development` #129669

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[CMAKE] Look for Development.Module instead of Development #129669

[CMAKE] Look for Development.Module instead of Development #129669

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129669

✅ No Failures

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Merge failed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Merge started

Uh oh!

Merge failed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PyTorchBot Help

Merge

Revert

Rebase

Label

Dr CI

cherry-pick

Close

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Merge started

Uh oh!

Uh oh!

[CMAKE] Look for `Development.Module` instead of `Development` #129669

[CMAKE] Look for `Development.Module` instead of `Development` #129669