8000 A way to override Python.NET .NET <-> Python wrapping for custom classes · Issue #823 · pythonnet/pythonnet · GitHub
[go: up one dir, main page]
Skip to content

A way to override Python.NET .NET <-> Python wrapping for custom classes #823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
lostmsu opened this issue Mar 5, 2019 · 3 comments
Closed

A way to override Python.NET .NET <-> Python wrapping for custom classes #823

lostmsu opened this issue Mar 5, 2019 · 3 comments

Comments

@lostmsu
Copy link
Member
lostmsu commented Mar 5, 2019

Environment

  • Pythonnet version: 2.3.0
  • Python version: 3.6
  • Operating System: Windows 10 x64

Details

I am trying to generate statically-typed wrappers for Python libraries, particularly TensorFlow (see Gradient).

Since I am automatically generating .NET-side types for wrappers, I can make their methods to use my custom converter written in C# to convert function arguments to PyObject instances. And then for any result values, I can convert PyObject instances back into my wrapper types.

Problem I am facing is when a user of my wrapper wants to inherit from one of the Python-derived classes, and override and/or extend its behavior.

An artificial sample:

// this part of my library is auto-generated
// represents tensorflow module
class tf {
  static dynamic tf = Py.Import("tensorflow");
  // set_default_session - made up method
  static void SetDefaultSession(Session session) => tf.set_default_session(session.underlyingSession);

  // made up method
  static void ComputeUsingDefaultSession() => tf.compute_using_default_session();
}

// represents tf.Tensor
class Tensor { PyObject underlyingTensor; }
// represents tf.Session
class Session {
  PyObject underlyingSession;
  virtual object Run(Tensor tensor) {
    var pyTensor = ConvertToPyObject(tensor); // simply does tensor.underlyingTensor here
    dynamic pySession = this.underlyingSession;
    var result = pySession.Run(pyTensor);
    return ConvertFromPyObject(result); // wraps any returned object into one of generated classes
  }
}

// this code is what user of my library wants to do:
class MyBetterSession : Session {
  override object Run(Tensor tensor) {
    ... here he writes custom code to run a Tensor ...
  }
}

// an attempts to use the above (functions made up):
var simpleSession = new Session();
tf.SetDefaultSession(simpleSession); // OK
tf.ComputeUsingDefaultSession(); // SUCCEEDS

var betterSession = new MyBetterSession();
tf.SetDefaultSession(betterSession); // OK
tf.ComputeUsingDefaultSession(); // FAILS

The last line will fail, because Python will attempt to call MyBetterSession.Run with Python's class tf.Tensor, but the method actually expects wrapped class Tensor. I need to somehow tell Python or rather Python.NET to invoke my ConvertFromPyObject on the argument, before trying to find a matching overload.

I looked into Python.NET source, and the corresponding objects are MethodBinding and MethodObject, however, neither seem to provide any extensibility.

Now I am considering several approaches to the problem, and I just wanted to discuss them with Python.NET team, as some of them involve modification of Python.NET.

  1. Do not change anything in Python.NET, and for every user class like MyBetterSession generate a pythonic wrapper with the same set of methods, but replacing all parameter and return types with PyObject. It would require one of the following:
  • System.Reflection.Emit, which is a very heavy dependency, and also not very pleasant to work with
  • Roslyn, which is much heavier, but moderately OK to work with

  1. I noticed, that while Python.NET supports representing any Python object as dynamic, it does not actually support passing dynamic objects (e.g. IDynamicMetaObjectProvider instances) back to Python. I mean, it would pass them, but if Python would try to access a dynamic attribute, it would not attempt to call TryGetMember. I could potentially implement something similar to ClassBase specifically for wrapping IDynamicMetaObjectProvider instances. Then it is much easier to implement its members, that would simply wrap PyObject arguments before forwarding them to an instance of MyBetterSession.

  2. Have Python.NET directly expose some low-level interface, that would enable hooking into Python.NET's marshaling and, possibly, also method binding processes.
@filmor
Copy link
Member
filmor commented Mar 6, 2019

It would be great to have a feature like this.

I'd be in favour of option 3), as redesigning this to be extensible would probably allow us to fix a ton of open bugs that are related to either too eager or incomplete conversions.

Regarding 2), I've implemented this in a hacky way from Python's side for classes deriving directly from DynamicObject here: #72. If one moves this to C#, it should be easy enough to extend it to IDynamicMetaObjectProvider, the only reason this hack only works for DynamicObject is because you can't inject base classes into Python.NET provided interfaces.

@lostmsu
Copy link
Member Author
lostmsu commented Mar 7, 2019

@filmor , can you link the bugs, that you think might benefit from the updated overload resolution?

@lostmsu lostmsu mentioned this issue Mar 12, 2019
Merged
4 tasks
@lostmsu lostmsu mentioned this issue Mar 28, 2019
Closed
4 tasks
@lostmsu
Copy link
Member Author
lostmsu commented May 5, 2020

Fixed in #1022

@lostmsu lostmsu closed this as completed May 5, 2020
@igiona igiona mentioned this issue Sep 16, 2021
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@filmor @lostmsu
0