8000 Best effort serialization by BadSingleton · Pull Request #1892 · pythonnet/pythonnet · GitHub
[go: up one dir, main page]

Skip to content

Best effort serialization #1892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

BadSingleton
Copy link
Contributor
@BadSingleton BadSingleton commented Jul 29, 2022

What does this implement/fix? Explain your changes.

This PR adds a "last line of defense, best effort serialization" to serialize types not marked as Serializable. Such objects, when derivable, are deserialized as derived classes with all (overridable) methods and properties overriden to throw a "Not Serialized" Exception. Fields are not initialized and may be null. Instances of classes that are not derivable are null.

Does this close any currently open issues?

No, but it is an issue we've encountered with Unity.

Any other comments?

This is a "best effort" to ensure no SerializationExceptions are thrown on domain unloads/reloads. This should help users deal with serialization issues as sometimes unserializable classes may be in compiled libraries and therefore unmodifiable.

Checklist

Check all those that are applicable and complete.

  • Make sure to include one or more tests for your change
  • If an enhancement PR, please create docs and at best an example
  • Ensure you have signed the .NET Foundation CLA
  • Add yourself to AUTHORS
  • Updated the CHANGELOG

This commit adds a "last line of defense, best effort serialization"
to serialize types not marked as Serializable. Such objects are
deserialized as derived classes with all methods and properties
overriden to throw a "Not Serialized" Exception.
Fields are not initialized and may be null.

Sealed classes and implemented interface methods are still a problem to
be solved.
plus guard rails for private classes

and review fixes
don't try to derive from sealed types
@BadSingleton
Copy link
Contributor Author
BadSingleton commented Jul 29, 2022

Also just a note that I'll soon be away for vacations until october, I just want to get that ball rolling and gather feedback in the meantime.

@lostmsu
Copy link
Member
lostmsu commented Jul 29, 2022

@BadSingleton is there a way to provide that functionality without bringing the NonSerializedTypeBuilder into Python.NET? E.g. can we allow user to override the serializer somehow, and you guys would be able to keep this code inside Unity or as a new subproject like pythonnet/unity-serializer?

@lostmsu
Copy link
Member
lostmsu commented Jul 29, 2022

Also, can you describe the approach in more details?

@BadSingleton
Copy link
Contributor Author

@lostmsu I think the serializer was meant to be overridable in the first implementations of the serialization. I think I recall @amos402 saying something about the need for users to implement their own. If it becomes overridable we probably can keep it outside of this repo (just need to make sure with some order of initialization things)

The idea was to ensure that the presence of a non-serializable object in the graph wouldn't throw an exception, therefore corrupting the whole shutdown process and causing crashes on domain (un)load. As sometimes those objects are outside of our control, we can't modify them and/or avoid easily to serialize them (and/or the users of our APIs may be passing a Unity object that is not serializable). After a few google searches I found out that serialization surrogates could be used to serialize objects not marked [Serializable]. Since these objects are not meant to be serialized, but they're in the object graph that is being serialized, we serialize the type, not the object. When deserializing, there are two possibilities:

  1. The object can be derived from, and we add _NotSerialized suffix to the class name. If it is, every method and property (get and set) are overridden by a simple function that throws a NotSerializedException (the inspiration for the exception message comes from PyQt. Those who knows, know the pain.). Fields are left null.
  2. The object cannot be derived from (private, internal, ...). Not much to do here, except returning a null reference.

When serializing a derived _NotSerialized class, we serialize the base class to avoid having _NotSerialized_NotSerialized_... looping.

@BadSingleton
Copy link
Contributor Author

Ignore the part of the previous comment about the serializer not being overridable, I misremembered the code, it is overridable.

@lostmsu
Copy link
Member
lostmsu commented Jul 29, 2022

It seems that this code can be abstracted out of this repo with:

  1. Having a public post stash and pre restore hooks
  2. Exposing a wrapper for PyCapsule and maybe PyMem (though I think it can be replaced with Marshal.AllocHGlobal)
  3. Replacing FormatterType property with FormatterFactory
  4. Exposing MaybeType

@BadSingleton
Copy link
Contributor Author

Worst case, MaybeType is easily re-implementable. I'm not sure about exposing it.

@larryPlayabl
Copy link

I am also experiencing this issue when trying to use this library with Unity3D. Here is the specific serialization error I get on shutdown. @BadSingleton, does your fix prevent this and allow you to initialize and shutdown the python engine multiple times without error?

Got exception during shutdown: 
Message:Type '__System_Action`1\[\[System_String\, mscorlib\, Version=4_0_0_0\, Culture=neutral\, PublicKeyToken=b77a5c561934e089\]\]Dispatcher' in Assembly '__Python_Runtime_Generated_Assembly0, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' is not marked as serializable.
  at System.Runtime.Serialization.FormatterServices.InternalGetSerializableMembers (System.RuntimeType type) [0x00045] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Runtime.Serialization.FormatterServices+<>c__DisplayClass9_0.<GetSerializableMembers>b__0 (System.Runtime.Serialization.MemberHolder _) [0x00000] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Collections.Concurrent.ConcurrentDictionary`2[TKey,TValue].GetOrAdd (TKey key, System.Func`2[T,TResult] valueFactory) [0x00034] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Runtime.Serialization.FormatterServices.GetSerializableMembers (System.Type type, System.Runtime.Serialization.StreamingContext context) [0x0005e] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitMemberInfo () [0x0003d] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize (System.Object obj, System.Runtime.Serialization.ISurrogateSelector surrogateSelector, System.Runtime.Serialization.StreamingContext context, System.Runtime.Serialization.Formatters.Binary.SerObjectInfoInit serObjectInfoInit, System.Runtime.Serialization.IFormatterConverter converter, System.Runtime.Serialization.Formatters.Binary.ObjectWriter objectWriter, System.Runtime.Serialization.SerializationBinder binder) [0x00158] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.Serialize (System.Object obj, System.Runtime.Serialization.ISurrogateSelector surrogateSelector, System.Runtime.Serialization.StreamingContext context, System.Runtime.Serialization.Formatters.Binary.SerObjectInfoInit serObjectInfoInit, System.Runtime.Serialization.IFormatterConverter converter, System.Runtime.Serialization.Formatters.Binary.ObjectWriter objectWriter, System.Runtime.Serialization.SerializationBinder binder) [0x00006] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write (System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo objectInfo, System.Runtime.Serialization.Formatters.Binary.NameInfo memberNameInfo, System.Runtime.Serialization.Formatters.Binary.NameInfo typeNameInfo) [0x00101] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize (System.Object graph, System.Runtime.Remoting.Messaging.Header[] inHeaders, System.Runtime.Serialization.Formatters.Binary.__BinaryWriter serWriter, System.Boolean fCheck) [0x001d3] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream serializationStream, System.Object graph, System.Runtime.Remoting.Messaging.Header[] headers, System.Boolean fCheck) [0x0006e] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream serializationStream, System.Object graph, System.Runtime.Remoting.Messaging.Header[] headers) [0x00000] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream serializationStream, System.Object graph) [0x00000] in <4c36328e5e3c40ecaf35733ee9461b26>:0 
  at Python.Runtime.RuntimeData.Stash () [0x00049] in <17d3e183cc45426f82a1a9207d7443a1>:0 
  at Python.Runtime.Runtime.Shutdown () [0x00030] in <17d3e183cc45426f82a1a9207d7443a1>:0 
  at Python.Runtime.PythonEngine.Shutdown () [0x00064] in <17d3e183cc45426f82a1a9207d7443a1>:0 

@BadSingleton
Copy link
Contributor Author

Hello, @larryPlayabl. It has been a while, but if my memory serves right, this looks like the exact error this fix was made for. For added convenience, Unity's Python Scripting package has already that fix since version 6.0.0

@BadSingleton BadSingleton mentioned this pull request Mar 7, 2024
5 tasks
@BadSingleton
Copy link
Contributor Author

Closing, superseded by #2336

@lostmsu lostmsu closed this May 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants
0