10000 BUG: Cannot install from source on macOS 12.1 with `ld: library not found for -lSystem` · Issue #20677 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

BUG: Cannot install from source on macOS 12.1 with ld: library not found for -lSystem #20677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
corneliusroemer opened this issue Dec 29, 2021 · 9 comments
Closed

BUG: Cannot install from source on macOS 12.1 with ld: library not found for -lSystem #20677

corneliusroemer opened this issue Dec 29, 2021 · 9 comments
Labels
00 - Bug 57 - Close? Issues which may be closable unless discussion continued

Comments

@corneliusroemer
Copy link
Contributor
corneliusroemer commented Dec 29, 2021 •
edited
Loading

Describe the issue:

I'm trying to contribute and thus build from source but I can't get it to work.

I get the following error:

ld: library not found for -lSystem
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

I created a conda environment using

conda env create -n numpydev

and this seems to be where ld was installed:

  - ld64                            530  hd2e7500_21         installed         
  - ld64_osx-64                     530  he8994da_21         installed         
  - ldid                          2.1.2  h6a69015_3          installed 

❯ which ld
/usr/local/Caskroom/mambaforge/base/envs/numpydev/bin/ld

❯ ld -v
@(#)PROGRAM:ld  PROJECT:ld64-530
BUILD 16:51:25 Apr  5 2021
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em (tvOS)
LTO support using: LLVM version 11.1.0 (static support for 27, runtime is 27)
TAPI support using: TAPI version 11.0.0 (tapi-1100.0.11)

❯ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 13.0.0 (clang-1300.0.29.30)
Target: x86_64-apple-darwin21.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Reproduce the code example:

python runtests.py -vv

Error message:

This is the full stack trace:

Full stack trace 
python runtests.py -vv
Building, see build.log...
Running from numpy source directory.
numpy/random/_bounded_integers.pxd.in has not changed
numpy/random/_philox.pyx has not changed
numpy/random/_bounded_integers.pyx.in has not changed
numpy/random/_sfc64.pyx has not changed
numpy/random/_mt19937.pyx has not changed
numpy/random/bit_generator.pyx has not changed
numpy/random/_bounded_integers.pyx has not changed
numpy/random/mtrand.pyx has not changed
numpy/random/_generator.pyx has not changed
numpy/random/_pcg64.pyx has not changed
numpy/random/_common.pyx has not changed
Cythonizing sources
INFO: blas_opt_info:
INFO: blas_armpl_info:
INFO: customize UnixCCompiler
INFO:   libraries armpl_lp64_mp not found in ['/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib', '/usr/local/lib', '/usr/lib']
INFO:   NOT AVAILABLE
INFO: 
INFO: blas_mkl_info:
INFO:   libraries  not found in ['']
INFO:   NOT AVAILABLE
INFO: 
INFO: blis_info:
INFO:   libraries blis not found in ['']
INFO:   NOT AVAILABLE
INFO: 
INFO: openblas_info:
INFO:   libraries  not found in ['']
INFO:   NOT AVAILABLE
INFO: 
INFO: accelerate_info:
INFO:   libraries accelerate not found in ['/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib', '/usr/local/lib', '/usr/lib']
INFO: Library accelerate was not found. Ignoring
INFO:   libraries veclib not found in ['/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib', '/usr/local/lib', '/usr/lib']
INFO: Library veclib was not found. Ignoring
INFO:   FOUND:
INFO:     extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
INFO:     extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
INFO:     define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
INFO: 
INFO:   FOUND:
INFO:     extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
INFO:     extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
INFO:     define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
INFO: 
INFO: lapack_opt_info:
INFO: lapack_armpl_info:
INFO:   libraries armpl_lp64_mp not found in ['/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib', '/usr/local/lib', '/usr/lib']
INFO:   NOT AVAILABLE
INFO: 
INFO: lapack_mkl_info:
INFO:   libraries  not found in ['']
INFO:   NOT AVAILABLE
INFO: 
INFO: openblas_lapack_info:
INFO:   libraries  not found in ['']
INFO:   NOT AVAILABLE
INFO: 
INFO: openblas_clapack_info:
INFO:   libraries  not found in ['']
INFO:   NOT AVAILABLE
INFO: 
INFO: flame_info:
INFO:   libraries  not found in ['']
INFO:   NOT AVAILABLE
INFO: 
INFO:   FOUND:
INFO:     extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
INFO:     extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
INFO:     define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
INFO: 
Warning: attempted relative import with no known parent package
/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
running build
running config_cc
INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
INFO: unifing config_fc, config, build_clib, 
8000
build_ext, build commands --fcompiler options
running build_src
INFO: build_src
INFO: building py_modules sources
INFO: building library "npymath" sources
ld: library not found for -lSystem
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
Traceback (most recent call last):
  File "/Users/cr/code/numpy/setup.py", line 455, in <module>
    setup_package()
  File "/Users/cr/code/numpy/setup.py", line 447, in setup_package
    setup(**metadata)
  File "/Users/cr/code/numpy/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib/python3.10/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib/python3.10/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib/python3.10/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/cr/code/numpy/numpy/distutils/command/build.py", line 62, in run
    old_build.run(self)
  File "/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib/python3.10/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib/python3.10/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/local/Caskroom/mambaforge/base/envs/numpydev/lib/python3.10/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/cr/code/numpy/numpy/distutils/command/build_src.py", line 144, in run
    self.build_sources()
  File "/Users/cr/code/numpy/numpy/distutils/command/build_src.py", line 155, in build_sources
    self.build_library_sources(*libname_info)
  File "/Users/cr/code/numpy/numpy/distutils/command/build_src.py", line 288, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "/Users/cr/code/numpy/numpy/distutils/command/build_src.py", line 378, in generate_sources
    source = func(extension, build_dir)
  File "/Users/cr/code/numpy/numpy/core/setup.py", line 702, in get_mathlib_info
    raise RuntimeError(
RuntimeError: Broken toolchain: cannot link a simple C program. 

Build failed!

NumPy/Python version information:

Not applicable
@corneliusroemer corneliusroemer mentioned this issue Dec 29, 2021
Merged
@mattip
Copy link
Member
mattip commented Dec 29, 2021

Apparently you need to do something to get -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib into your link command.

@corneliusroemer
Copy link
Contributor Author

Thanks for figuring this out. How do I do this? The link command seems to be auto-generated, doesn't it?

@mattip
Copy link
Member
mattip commented Dec 29, 2021

I think you can set LDFLAGS=-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib ? Please report back success/failure since you probably are not the only one with this problem. Does /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib even exist? I found that on a stackoverflow question.

@corneliusroemer
Copy link
Contributor Author
corneliusroemer commented Dec 29, 2021 •
edited
Loading

Thanks for the tip, I tried but it didn't help.

export LDFLAGS=-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
python runtests.py -vv

causes the same error, alas.

Couldn't find any other tips anywhere. The path/directory does exist.

@corneliusroemer corneliusroemer mentioned this issue Dec 29, 2021
Closed
@mattip
Copy link
Member
mattip commented Dec 29, 2021

Can you see the LDFLAGS being used by the link command in the build log?

@corneliusroemer
Copy link
Contributor Author

I don't think I can, the complete build log is contained above in the collapsible.

@rgommers
Copy link
Member

This works fine for me with the numpy-dev conda env. I assume this was a temporary problem, since it happened shortly after the macOS 12.1 release?

@mattip mattip added the 57 - Close? Issues which may be closable unless discussion continued label Feb 18, 2022
@rgommers rgommers mentioned this issue Jul 25, 2022
Merged
@rgommers
Copy link
Member

I have done another search, it is completely unclear what is injecting -lSystem into the build. It doesn't appear to be in numpy.distutils nor in distutils in the stdlib. The flag does not show up if I build from source on macOS in a conda-forge environment.

No one else commented, so I assume it's some specific build env component that is injecting this flag. I will close this issue. If anyone figures out where -lSystem comes from, please comment!

@mattip
Copy link
Member
mattip commented Aug 16, 2022

fwiw, I think it is the gfortran from gcc used in the f2py tests. This stackoverflow suggests setting LIBRARY_PATH in the environment, which seems to solve the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug 57 - Close? Issues which may be closable unless discussion continued
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rgommers @mattip @corneliusroemer
0