8000 Drop LoadLibrary by amos402 · Pull Request #880 · pythonnet/pythonnet · GitHub
[go: up one dir, main page]

Skip to content

Drop LoadLibrary #880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 14 commits into from
Closed

Drop LoadLibrary #880

wants to merge 14 commits into from

Conversation

amos402
Copy link
Member
@amos402 amos402 commented Jun 13, 2019

What does this implement/fix? Explain your changes.

There are some limitations to dynamic load a library on some systems(e. g. Android).
LoadLibrary just called for once for getting the _PyObject_NextNotImplemented,
we can use another ways to get this function.

Does this close any currently open issues?

No

Any other comments?

Checklist

Check all those that are applicable and complete.

  • Make sure to include one or more tests for your change
  • If an enhancement PR, please create docs and at best an example
  • Add yourself to AUTHORS
  • Updated the CHANGELOG

@filmor
Copy link
Member
filmor commented Jun 13, 2019

I'll have to look a bit more into this. I thought that this LoadLibrary call there is not only pulling in that one symbol but that it serves as a "proxy" to enforce loading the library into memory s.t. the P/Invoke bindings work. Since your patch passes all tests (apart from the known failing one) I must have misunderstood something and will test this in our use-case to verify. Thank you already for your contribution, if this works out this will help a lot in simplifying the build process :)

@lostmsu
Copy link
Member
lostmsu commented Jun 13, 2019

@filmor @amos402 I made a single Python.Runtime.dll to work with any Python 3.x interpreter in my fork (could have included 2.x, but its lifetime is ending) via extensive use of LoadLibrary and related methods. Which I intended to upstream once 2.7 is dropped (or, for example, if we had a separate 2.x branch in maintenance mode).

8000

@filmor
Copy link
Member
filmor commented Jun 13, 2019

@lostmsu What do you mean by "extensive use"? Isn't it enough to load libpython*.so once at startup? Could you point me to that code?

@lostmsu
Copy link
Member
lostmsu commented Jun 13, 2019

@filmor well, the LoadLibrary is called once, but all the functions are obtained using GetProcAddress/dlsym.
See losttech@672258f

And also losttech@80aa07f

All of that happens once at startup too. But there are many functions.

There is probably a performance impact too, due to the use of delegates in place of PInvoke calls.

@filmor
Copy link
Member
filmor commented Sep 12, 2019

@lostmsu What's your conclusion on keeping LoadLibrary? Does the code using delegates run reasonably or not?

@@ -297,16 +297,20 @@ internal static void Initialize(bool initSigs = false)
IntPtr dllLocal = IntPtr.Zero;
var loader = LibraryLoader.Get(OperatingSystem);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like you should delete these lines too, and line 295 as well since you moved it below.

Copy link
Contributor
@benoithudson benoithudson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a quite reasonable patch, I think we should merge it.

I've had trouble in the past with this LoadLibrary/FreeLibrary screwing up the dlopen flags.

* Drop C module dependency when getting _PyObject_NextNotImplemented
* Exception details for SetNoSiteFlag
@codecov-io
Copy link
codecov-io commented Dec 1, 2019

Codecov Report

Merging #880 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #880   +/-   ##
=======================================
  Coverage   86.71%   86.71%           
=======================================
  Files           1        1           
  Lines         301      301           
=======================================
  Hits          261      261     
10000
      
  Misses         40       40
Flag Coverage Δ
#setup_linux 65.44% <ø> (ø) ⬆️
#setup_windows 71.42% <ø> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f56ebc...65e209e. Read the comment docs.

@amos402
Copy link
Member Author
amos402 commented Dec 1, 2019

I removed the dependency of zipimport. But still have no solution for SetNoSiteFlag, maybe we should add a docs for it(like please use -S when using static compile).

{
dllLocal = loader.Load(_PythonDll);
throw new Exception($"Cannot load {_PythonDll}");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you throw more informative exception? Win32Exception would automatically read last error on Windows, but I don't know what to do for *nix

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally it should throw inside from Load, the nullptr checking just for in case(if the loader implementation didn't throw an exception). I'm not going to insert any platform specific code here.

private static IntPtr Get_PyObject_NextNotImplemented()
{
IntPtr globals = PyDict_New();
if (PyDict_SetItemString(globals, "__builtins__", PyEval_GetBuiltins()) != 0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can PyEval_GetBuiltins() return an error?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the docs it doesn't look like it can (https://docs.python.org/3/c-api/reflection.html#c.PyEval_GetBuiltins).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The globals dict is needed, but we don't seem to be using the __builtins__ in the string. Do we really need this line?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you don't setup a __builtins__ or set object explicit, object may not be found.

Comment on lines 335 to 336
XDecref(globals);
throw new PythonException();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to construct an instance of PythonException before doing XDecref, otherwise last error could potentially be overwritten by XDecref with something new.

@@ -1925,25 +1946,24 @@ internal static IntPtr PyMem_Realloc(IntPtr ptr, long size)

internal static void SetNoSiteFlag()
{
if (_PythonDll == "__Internal")
{
throw new NotSupportedException("SetNoSiteFlag didn't support on static compile");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why should this not be supported? The symbol is just in the running DLL, so loader.Load(_PythonDll) has to return essentially the result of dlopen(NULL, ...).

Copy link
Member F438 Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it should be supported. Just a mistake according to my wrong tests. I will recover it back.

@Jeff17Robbins
Copy link
Contributor

I believe issues #891, #946, and #967 all point to a fundamental problem with this PR.

On certain Linux distributions (Debian for example), Python C extension libraries are deliberately not linked to libpython.

Yet .NET Core's [DllImport] does not load libpython with RTLD_GLOBAL.

So if this PR removes dllLocal = loader.Load(_PythonDll); from runtime.cs, where the salient line in LibraryLoader.cs is var res = dlopen(filename, RTLD_NOW | RTLD_GLOBAL);, won't this PR break Python.NET's "Embedding" mode on .NET Core?

As Victor Stinner put it in Python issue issue 21536:

...libpython must always be loaded with RTLD_GLOBAL.

Unless we can get [DllImport] to set the RTLD_GLOBAL flag, I'm afraid we need the explicit loader.Load, at least for Linux .NET Core embedding.

@amos402
Copy link
Member Author
amos402 commented Feb 13, 2020

@Jeff17Robbins This PR won't break "Embedding" mode on .NET Core, moreover it improve the compatibility since it didn't need call the dlopen.

@amos402 amos402 mentioned this pull request Feb 13, 2020
4 tasks
@amos402
Copy link
Member Author
amos402 commented Oct 9, 2020

Contained in #958

@amos402 amos402 closed this Oct 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants
0