8000 Third party LICENSE collection error when build wheel package · Issue #126769 · pytorch/pytorch · GitHub
[go: up one dir, main page]
Skip to content

Third party LICENSE collection error when build wheel package #126769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Tracked by #130151
xiaoran007 opened this issue May 21, 2024 · 10 comments
Closed
Tracked by #130151

Third party LICENSE collection error when build wheel package #126769

xiaoran007 opened this issue May 21, 2024 · 10 comments
Assignees
@c-p-i-o
Labels
module: build Build system issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Milestone
2.4.0

Comments

@xiaoran007
Copy link
xiaoran007 commented May 21, 2024 •
edited by pytorch-bot bot
Loading

🐛 Describe the bug

When build wheel package use

python setup.py bdist_wheel

The third party LICENSE collection may not work, for example the LICENSE file in path

opentelemetry-cpp/tools/vcpkg/ports/sigslot

can not be identified by the identify_license() function in third_party/build_bundled.py.

License
The sigslot library has been placed in the public domain. This means that you are free to use it however you like.

The author takes no responsibility or liability of any kind for any use that you may make of this library.

If you screw up, it's your fault.

If the library screws up, you got it for free, so you should have tested it better - it's still your responsibility.

The contains in this LICENSE file can not be identified by any key words defined in identify_license() function.

Versions

tag: v2.1.0
os: ubuntu 20.04

cc @malfet @seemethere
@janeyx99 janeyx99 added module: build Build system issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels May 21, 2024
@malfet malfet added this to the 2.4.0 milestone May 22, 2024
@malfet
Copy link
Contributor
malfet commented May 22, 2024

Tentatively assigning to @c-p-i-o , who added the dependency in #122999

@ezyang just curios, is it used for anything yet? If not, then perhaps revert?

P.S. I wonder how it works with the nightly builds though?

@xiaoran007
Copy link
Author

I found this issue when building directly whl on my ubuntu machine (without using docker), or maybe I need to use docker (follow the ci script) to build it?
And if I set "Unknown license" as return value rather than value error, it works.

        if 'ApacheLicense' in txt:
            # Hmm, do we need to check the text?
            return 'Apache-2.0'
        .
        .
        .
        elif all([squeeze(m) in txt.lower() for m in bsd3_src_txt]):
            return 'BSD-Source-Code'
        elif any([squeeze(m) in txt.lower() for m in mit_txt]):
            return 'MIT'
        else:
            # A license may not be on this list, but is still a valid license.
            raise ValueError('unknown license')

@c-p-i-o
Copy link
Contributor
c-p-i-o commented May 22, 2024

We did attempt to fix this in build_bundled.py script (https://github.com/pytorch/pytorch/blob/main/third_party/build_bundled.py#L117)
Is this fix not working?

@c-p-i-o
Copy link
Contributor
c-p-i-o commented May 22, 2024

Tentatively assigning to @c-p-i-o , who added the dependency in #122999

@ezyang just curios, is it used for anything yet? If not, then perhaps revert?

We are waiting for some higher level abstractions for metrics to land. Once landed, for the OSS community, we plan on implementing those higher level abstractions via OTEL.

P.S. I wonder how it works with the nightly builds though?

Works on nightly because of (https://github.com/pytorch/pytorch/blob/main/third_party/build_bundled.py#L117)

@c-p-i-o
Copy link
Contributor
c-p-i-o commented May 22, 2024 •
edited
Loading

@xiaoran007 FWIW, the issue didn't repro for me on main.

python setup.py bdist_wheel

...
adding license file 'LICENSE'
adding license file 'NOTICE'
...
adding 'torch-2.4.0a0+gitc5d83bc.dist-info/LICENSE'
adding 'torch-2.4.0a0+gitc5d83bc.dist-info/METADATA'
adding 'torch-2.4.0a0+gitc5d83bc.dist-info/NOTICE'
adding 'torch-2.4.0a0+gitc5d83bc.dist-info/WHEEL'
adding 'torch-2.4.0a0+gitc5d83bc.dist-info/entry_points.txt'

I think what's happening here is that

  1. you might have checked out main. This fetches third-party/opentelemetry-cpp.

  2. Now, when you switch over to tag v2.1.0, you end up pulling an oldbuild_bundled.py from the tag but you still have third-party/opentelemetry-cpp in your directory left there.

  3. The old build_bundled.py script tries to iterate over the new third-party/opentelemetry-cpp library looking for LICENSE files and fails as it doesn't understand the weird sigslot file.

To confirm this, you'd need to do git status after you checkout v2.1.0 tag and you should see third-party/opentelemetry-cpp as a untracked files

Can you confirm this is what might be happening?

@xiaoran007
Copy link
Author

Yes, when I checkout to tag v2.1.0, the files are untracked:

HEAD detached at v2.1.0
Untracked files:
  (use "git add <file>..." to include in what will be committed)
        third_party/cpp-httplib/
        third_party/opentelemetry-cpp/

And I tried removing them and rebuilding the whl package now. It may takes a bit of time.

@xiaoran007
Copy link
Author

And I tried removing them and rebuilding the whl package now. It may takes a bit of time.

It works, thank you very much!

@c-p-i-o
Copy link
Contributor
c-p-i-o commented May 23, 2024

Thanks @xiaoran007!
I'm closing this issue for now - but re-open it if you run into any further issues.

@c-p-i-o c-p-i-o closed this as completed May 23, 2024
@abalib
Copy link
Contributor
abalib commented Jun 5, 2024

Yes, when I checkout to tag v2.1.0, the files are untracked:

HEAD detached at v2.1.0
Untracked files:
  (use "git add <file>..." to include in what will be committed)
        third_party/cpp-httplib/
        third_party/opentelemetry-cpp/

And I tried removing them and rebuilding the whl package now. It may takes a bit of time.

rm -rf third_party/opentelemetry-cpp after checking out the older tag builds.

@atalman atalman mentioned this issue Jul 5, 2024
Closed
29 tasks
@atalman
Copy link
Contributor
atalman commented Jul 9, 2024

Validated with 2.4:

024-07-09T16:55:55.7784566Z Copying torch.egg-info to build/bdist.linux-x86_64/wheel/torch-2.4.0+cu118-py3.9.egg-info
2024-07-09T16:55:55.7816182Z running install_scripts
2024-07-09T16:55:55.8448235Z adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
2024-07-09T16:55:55.8448910Z adding license file "NOTICE" (matched pattern "NOTICE*")
2024-07-09T16:55:55.8480003Z creating build/bdist.linux-x86_64/wheel/torch-2.4.0+cu118.dist-info/WHEEL
2024-07-09T16:55:55.8481246Z creating '/tmp/wheelhouse118/torch-2.4.0+cu118-cp39-cp39-linux_x86_64.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
2024-07-09T16:55:55.8652133Z adding 'functorch/_C.cpython-39-x86_64-linux-gnu.so'
2024-07-09T16:55:55.8657626Z adding 'functorch/__init__.py'
2024-07-09T16:55:55.8659457Z adding 'functorch/_src/__init__.py'
2024-07-09T16:55:55.8661390Z adding 'functorch/_src/aot_autograd/__init__.py'
2024-07-09T16:55:55.8662921Z adding 'functorch/_src/eager_transforms/__init__.py'
2024-07-09T16:55:55.8664457Z adding 'functorch/_src/make_functional/__init__.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@c-p-i-o c-p-i-o

Labels
module: build Build system issues triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
2.4.0
Development

No branches or pull requests
6 participants
@malfet @atalman @c-p-i-o @abalib @janeyx99 @xiaoran007
0