asv fails to run on windows when pkg-config and OpenBLAS are set up properly

running either

python runtests.py --bench-compare <commit1> <commit2>

or

spin bench --compare <commit1> <commit2>

will end up calling

cd benchmark; asv continuous --factor 1.05 --bench main <commit1> <commit2>

Then asv creates a virtualenv for each commit, and uses a vanilla pip install to build NumPy. While this can work on macOS and linux (and use the system OpenBLAS), it will fail to correctly use OpenBLAS on windows, since the build also needs to

copy the OpenBLAS DLL into the correct place
build the _distributor_init.py file. In CI this is done via additional post-install code:

Is there a way to, if OpenBLAS is found, to do this as part of the build?

Here is what the various CI invocations to do these steps look like:

numpy/.github/workflows/windows_meson.yml

Lines 63 to 72 in 374e73d

    
               - name: post-install 
        
                 run: | 
        
                   $numpy_path = "${env:installed_path}\numpy" 
        
                   $libs_path = "${numpy_path}\.libs" 
        
                   mkdir ${libs_path} 
        
                   $ob_path = "C:/opt/64/bin/" 
        
                   cp $ob_path/*.dll $libs_path 
        
                   # Write _distributor_init.py to load .libs DLLs. 
        
                   python -c "from tools import openblas_support; openblas_support.make_init(r'${numpy_path}')"

numpy/.github/workflows/windows_clangcl.yml

Lines 66 to 75 in 374e73d

    
               - name: post-install 
        
                 run: | 
        
                   $numpy_path = "${env:installed_path}\numpy" 
        
                   $libs_path = "${numpy_path}\.libs" 
        
                   mkdir ${libs_path} 
        
                   $ob_path = "C:/opt/64/bin/" 
        
                   cp $ob_path/*.dll $libs_path 
        
                   # Write _distributor_init.py to load .libs DLLs. 
        
                   python -c "from tools import openblas_support; openblas_support.make_init(r'${numpy_path}')"

(on azure this also shows the complicated command line needed)

numpy/azure-steps-windows.yml

Lines 38 to 48 in 374e73d

    
           - powershell: | 
        
               python -c "from tools import openblas_support; openblas_support.make_init('numpy')" 
        
               If ( Test-Path env:NPY_USE_BLAS_ILP64 ) { 
        
                   python -m pip install . -Csetup-args="--vsenv" -Csetup-args="-Duse-ilp64=true" -Csetup-args="-Dblas-symbol-suffix=64_" 
        
               } else { 
        
                   python -m pip install . -Csetup-args="--vsenv" 
        
               } 
        
               # copy from c:/opt/openblas/openblas_dll to numpy/.libs to ensure it can 
        
               # get loaded when numpy is imported (no RPATH on Windows) 
        
               $target = $(python -c "import sysconfig; print(sysconfig.get_path('platlib'))") 
        
               mkdir $target/numpy/.libs

numpy/tools/wheels/cibw_before_build.sh

Lines 39 to 56 in 374e73d

    
           PYTHONPATH=tools python -c "import openblas_support; openblas_support.make_init('numpy')" 
        
           mkdir -p /c/opt/32/lib/pkgconfig 
        
           mkdir -p /c/opt/64/lib/pkgconfig 
        
           target=$(python -c "import tools.openblas_support as obs; plat=obs.get_plat(); ilp64=obs.get_ilp64(); target=f'openblas_{plat}.zip'; obs.download_openblas(target, plat, ilp64);print(target)") 
        
           if [[ $PLATFORM == 'win-32' ]]; then 
        
               # 32-bit openBLAS 
        
               # Download 32 bit openBLAS and put it into c/opt/32/lib 
        
               unzip -o -d /c/opt/ $target 
        
               cp /c/opt/32/bin/*.dll /c/opt/openblas/openblas_dll 
        
           else 
        
               # 64-bit openBLAS 
        
               unzip -o -d /c/opt/ $target 
        
               if [[ -f /c/opt/64/lib/pkgconfig/openblas64.pc ]]; then 
        
                   # As of v0.3.23, the 64-bit interface has a openblas64.pc file, 
        
                   # but this is wrong. It should be openblas.pc 
        
                   cp /c/opt/64/lib/pkgconfig/openblas{64,}.pc 
        
               fi 
        
               cp /c/opt/64/bin/*.dll /c/opt/openblas/openblas_dll

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	- name: post-install
	run: \|
	$numpy_path = "${env:installed_path}\numpy"
	$libs_path = "${numpy_path}\.libs"
	mkdir ${libs_path}
	$ob_path = "C:/opt/64/bin/"
	cp $ob_path/*.dll $libs_path
	# Write _distributor_init.py to load .libs DLLs.
	python -c "from tools import openblas_support; openblas_support.make_init(r'${numpy_path}')"

	- powershell: \|
	python -c "from tools import openblas_support; openblas_support.make_init('numpy')"
	If ( Test-Path env:NPY_USE_BLAS_ILP64 ) {
	python -m pip install . -Csetup-args="--vsenv" -Csetup-args="-Duse-ilp64=true" -Csetup-args="-Dblas-symbol-suffix=64_"
	} else {
	python -m pip install . -Csetup-args="--vsenv"
	}
	# copy from c:/opt/openblas/openblas_dll to numpy/.libs to ensure it can
	# get loaded when numpy is imported (no RPATH on Windows)
	$target = $(python -c "import sysconfig; print(sysconfig.get_path('platlib'))")
	mkdir $target/numpy/.libs

	PYTHONPATH=tools python -c "import openblas_support; openblas_support.make_init('numpy')"
	mkdir -p /c/opt/32/lib/pkgconfig
	mkdir -p /c/opt/64/lib/pkgconfig
	target=$(python -c "import tools.openblas_support as obs; plat=obs.get_plat(); ilp64=obs.get_ilp64(); target=f'openblas_{plat}.zip'; obs.download_openblas(target, plat, ilp64);print(target)")
	if [[ $PLATFORM == 'win-32' ]]; then
	# 32-bit openBLAS
	# Download 32 bit openBLAS and put it into c/opt/32/lib
	unzip -o -d /c/opt/ $target
	cp /c/opt/32/bin/*.dll /c/opt/openblas/openblas_dll
	else
	# 64-bit openBLAS
	unzip -o -d /c/opt/ $target
	if [[ -f /c/opt/64/lib/pkgconfig/openblas64.pc ]]; then
	# As of v0.3.23, the 64-bit interface has a openblas64.pc file,
	# but this is wrong. It should be openblas.pc
	cp /c/opt/64/lib/pkgconfig/openblas{64,}.pc
	fi
	cp /c/opt/64/bin/*.dll /c/opt/openblas/openblas_dll

Uh oh!

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions