8000 [CI/CD] Azure Pipeline: issue with `_sqlite3` module · Issue #120159 · python/cpython · GitHub
[go: up one dir, main page]
Skip to content

[CI/CD] Azure Pipeline: issue with _sqlite3 module #120159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
picnixz opened this issue Jun 6, 2024 · 2 comments
Closed

[CI/CD] Azure Pipeline: issue with _sqlite3 module #120159

picnixz opened this issue Jun 6, 2024 · 2 comments
Labels
OS-windows type-bug An unexpected behavior, bug, or error

Comments

@picnixz
Copy link
Member
picnixz commented Jun 6, 2024 •
edited
Loading

Bug report

Bug description:

This is something I wanted to point out but I'm not sure whether it's a known bug or not (if so, feel free to close the issue). I couldn't find a related issue.

The Azure Pipeline is failing on Windows, similarly to #84395 but with a more explicit error. There are two failing tests, namely

  • test.test_ctypes.test_loading.LoaderTest.test_load_dll_with_flags
  • test.test_import.ImportTests.test_dll_dependency_import

Both of them fail because they cannot find the DLL related to the sqlite3 module. The first failure failure is actually very clear:

 Traceback (most recent call last):
  File "D:\a\1\b\layout-appx-win32\Lib\test\test_ctypes\test_loading.py", line 157, in test_load_dll_with_flags
    shutil.copy(os.path.join(os.path.dirname(src), "sqlite3" + ext),
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                os.path.join(tmp, "sqlite3" + ext))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\1\b\layout-appx-win32\Lib\shutil.py", line 428, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\1\b\layout-appx-win32\Lib\shutil.py", line 260, in copyfile
    with open(src, 'rb') as fsrc:
         ~~~~^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\a\\1\\b\\layout-appx-win32\\DLLs\\sqlite3.dll'

I think the issue is with the DLLs directory which is never created (AFAICT), as illustrated by the current code:

cpython/Lib/test/test_ctypes/test_loading.py

Lines 150 to 158 in fd104df

with os_helper.temp_dir() as tmp:
# We copy two files and load _sqlite3.dll (formerly .pyd),
# which has a dependency on sqlite3.dll. Then we test
# loading it in subprocesses to avoid it starting in memory
# for each test.
target = os.path.join(tmp, "_sqlite3.dll")
shutil.copy(src, target)
shutil.copy(os.path.join(os.path.dirname(src), "sqlite3" + ext),
os.path.join(tmp, "sqlite3" + ext))

There should be some

tmp2 = os.path.join(tmp, "DLLs")
os.mkdir(tmp2)

similar to what is being done in test.test_import.ImportTests.test_dll_dependency_import. Note that the source DLL should be present since it is generated during the build.

The second failure occurs with:

ERROR: test_dll_dependency_import (test.test_import.ImportTests.test_dll_dependency_import)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\1\b\layout-appx-win32\Lib\test\test_import\__init__.py", line 762, in test_dll_dependency_import
...
subprocess.CalledProcessError: Command '[
	'C:\\Users\\VssAdministrator\\AppData\\Local\\Temp\\test_python_w9okac52\\tmp7ps_2s1g\\python.exe', 
	'-Sc', 
	"import os;p = os.add_dll_directory('D:\\\\a\\\\1\\\\b\\\\layout-appx-win32\\\\DLLs');import _sqlite3;p.close"
]' returned non-zero exit status 1.

Here I think what occurs is that the files are not copied correctly (the DLLs directory is correctly created but I assume that the issue is due to the missing dll file, or perhaps it's moved incorrectly). Here is the code setting up the environment:

cpython/Lib/test/test_import/__init__.py

Lines 738 to 755 in fd104df

8000
from _winapi import GetModuleFileName
dllname = GetModuleFileName(sys.dllhandle)
pydname = importlib.util.find_spec("_sqlite3").origin
depname = os.path.join(
os.path.dirname(pydname),
"sqlite3{}.dll".format("_d" if "_d" in pydname else ""))
with os_helper.temp_dir() as tmp:
tmp2 = os.path.join(tmp, "DLLs")
os.mkdir(tmp2)
pyexe = os.path.join(tmp, os.path.basename(sys.executable))
shutil.copy(sys.executable, pyexe)
shutil.copy(dllname, tmp)
for f in glob.glob(os.path.join(glob.escape(sys.prefix), "vcruntime*.dll")):
shutil.copy(f, tmp)
shutil.copy(pydname, tmp2)

CPython versions tested on:

CPython main branch

Operating systems tested on:

Other
@picnixz picnixz added the type-bug An unexpected behavior, bug, or error label Jun 6, 2024
@picnixz picnixz changed the title [CI/CD] Azue Pipeline: issue with _sqlite3 module [CI/CD] Azure Pipeline: issue with _sqlite3 module Jun 6, 2024
@eryksun
Copy link
Contributor
eryksun commented Jun 6, 2024

@zooba, this issue with finding "sqlite3.dll" in the appx layout is resolved by #120133, right?

@zooba
Copy link
Member
zooba commented Jun 6, 2024

this issue with finding "sqlite3.dll" in the appx layout is resolved by #120133, right?

Yeah, it will. I noticed the error because my own test builds were failing the other day, but didn't bother copying the whole thing into the bug this time (serves me right...)

@zooba zooba closed this as not planned Won't fix, can't repro, duplicate, stale Jun 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
OS-windows type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@eryksun @hugovk @zooba @picnixz
0