PythonEngine.Shutdown() / Runtime.Shutdown() slowness #2008

dclaudeone11 · 2022-11-16T14:07:31Z

Environment

Pythonnet version: 3.0.1
Python version: 3.9.12
Operating System: Windows or Linux
.NET Runtime: net6.0

Details

Describe what you were trying to get done.

We are embedding Python in a .NET app using pythonnet.

The call to PythonEngine.Shutdown() at the exit of this app lasts for more than 2 minutes.

Looking at the shutdown with a profiler shows that most of the time is spent in Runtime.TryCollectingGarbage().

Looks like the more memory is allocated on the C# side the more the shutdown (i.e. calls to GC.Collect()) is slow.

Minimal, Complete, and Verifiable example

using Python.Runtime;

namespace Test;

static class Program
{
    static void Main()
    {
        var mem = new List<Array>();
        Runtime.PythonDLL = @$"{Environment.GetFolderPath(Environment.SpecialFolder.UserProfile)}\AppData\Local\Programs\Python\Python39\python39.dll";
        PythonEngine.Initialize();
        var threadState = PythonEngine.BeginAllowThreads();

        using (Py.GIL())
        {
            using (var scope = Py.CreateScope())
            {
                scope.Set("a", 1);
                scope.Set("b", 2);
                scope.Exec("result = a + b");
                var result = scope.Get("result").As<int>();
                Console.WriteLine($"Sum = {result}");
            }
        }

        // allocate dotnet-managed memory unrelated to pythonnet
        var rnd = new Random();
        for (var i = 0; i < 100_000_000; ++i)
        {
            mem.Add(new byte[rnd.Next(1, 10)]);
        }

        PythonEngine.EndAllowThreads(threadState);
        Console.WriteLine("Shutdown start");
        var start = DateTime.Now;
        PythonEngine.Shutdown();
        var diff = DateTime.Now - start;
        Console.WriteLine($"Shutdown stop {diff.TotalSeconds}");
    }
}

Typical output:

Sum = 3
Shutdown start
Shutdown stop 61.6796983

The text was updated successfully, but these errors were encountered:

filmor · 2022-11-22T17:12:22Z

This would probably be helped by handling #1781.

If we are finally shutting down, we don't actually have to stash anything, so we also shouldn't try to clean up garbage, right @lostmsu @BadSingleton ?

fischja · 2023-02-16T23:17:42Z

Any updates on this @filmor ?

BadSingleton · 2023-02-17T14:14:58Z

Apologies for the late reply. Yes, if we're shutting down for the last time, then there is no need to stash anything. If I recall correctly, it used to be that way before when there was still multiple modes. But I'm not sure about garbage collection, I've had some issues in the past (crashing on shutdown?) if it wasn't run correctly.

filmor · 2023-02-20T18:29:10Z

@lostmsu @BadSingleton Now that issues like #2107 show up because BinaryFormatter.Serialize has been hard-deprecated in .NET 7, we need to do somethhing. How about making Shutdown() behave like it used to (no stashing) and adding a new ShutdownAllowRestart or something like that?

lostmsu · 2023-02-20T19:15:35Z

@filmor @BadSingleton do we even need to support restart after shutdown on .NET 7?

BadSingleton · 2023-02-20T20:03:12Z

[...] do we even need to support restart after shutdown on .NET 7?

I've been out of the .NET loop for a few months, why wouldn't we need to support it?

Otherwise, I don't think it has to be BinaryFormatter, as long as the behaviour is the same(?)

benoithudson · 2023-02-20T20:27:34Z

Unity is moving to .NET 7 next year supposedly (in large part to support Span). So, it would be best if it were still supported.
https://blog.unity.com/technology/unity-and-net-whats-next

For the original comment: do we actually need to call Shutdown on shutdown? If not, that would save a lot of time.

lostmsu · 2023-02-20T20:49:14Z

@benoithudson what is Unity on now? I thought it relies on AppDomain, and AppDomain is not supported in .NET Core/6+. Without AppDomain, I do not think one needs to call Shutdown at all. What should be done instead perhaps is ability to reload individual types.

nlogozzo · 2023-02-23T04:23:07Z

Now that issues like #2107 show up because BinaryFormatter.Serialize has been hard-deprecated in .NET 7, we need to do somethhing.

Hi, anyway I can work around this issue for now?

AlexGilmor · 2023-02-23T05:20:13Z

I faced the same problem (Pythonnet 3.0.1, .NET Framework 4.7.2). In my app I have a window with the python code editor. I call PythonEngine.Initialize() when the user opens it, and the PythonEngine.Shotdown() when closing the window.

After executing simple scripts, the window closes with a small but noticeable delay. When executing scripts that create more complex objects, for example, matplayotlib figures, the delay increases to tens or even hundreds of seconds. I can call PythonEngine.Shotdown() not when closing the editor window, but when closing the application. But for now this only shifts the problem and does not solve it.

Unfortunately, I wasn’t able to find the official position on this subject in the documentation. The fact that many people use Pythonnet suggest that there is a way of bypass, because without it the issue looks serious. @dclaudeone11 Did you manage to find a temporary solution?

Without AppDomain, I do not think one needs to call Shutdown at all

@lostmsu First of all, thank you and other contributors for such a useful library.
Could you please explain what exactly is considered as a right way to stop Pythonnet (or app that uses it) for today? If I do not call PythonEngine.Shotdown() at all, after closing the main window of the program, a working process remains, which is obviously unacceptable. Given the popularity of this thread in the last days, it seems that many more people would be grateful for your official commentary, not only me.

filmor · 2023-02-23T07:38:58Z

@AlexGilmor If you want to shut down Python, use Shutdown. The slowness is due to to multiple garbage collection rounds and (potentially contributing, but not the only factor) the runtime stashing that was introduced to allow for AppDomain switching.

@nlogozzo Since you are not actually using the stashing, you can try setting a no-op formatter:

public class NoopFormatter : IFormatter {
  public object Deserialize(Stream s) => throw new NotImplementedException();
  public void Serialize(Stream s, object o) {}

  public SerializationBinder Binder { get; set; }
  public StreamingContext Context { get; set; }
  public ISurrogateSelector SurrogateSelector { get; set; }
}

Python.Runtime.RuntimeData.FormatterType = typeof(NoopFormatter);

nlogozzo · 2023-02-23T15:24:24Z

@nlogozzo Since you are not actually using the stashing, you can try setting a no-op formatter:

That worked...thanks!

kikaragyozov · 2023-03-30T13:03:43Z

Environment

Pythonnet version: 3.0.1
Python version: 3.9.12
Operating System: Windows or Linux
.NET Runtime: net6.0

Details

Describe what you were trying to get done.

We are embedding Python in a .NET app using pythonnet.

The call to PythonEngine.Shutdown() at the exit of this app lasts for more than 2 minutes.

Looking at the shutdown with a profiler shows that most of the time is spent in Runtime.TryCollectingGarbage().

Looks like the more memory is allocated on the C# side the more the shutdown (i.e. calls to GC.Collect()) is slow.

Minimal, Complete, and Verifiable example

using Python.Runtime;

namespace Test;

static class Program
{
    static void Main()
    {
        var mem = new List<Array>();
        Runtime.PythonDLL = @$"{Environment.GetFolderPath(Environment.SpecialFolder.UserProfile)}\AppData\Local\Programs\Python\Python39\python39.dll";
        PythonEngine.Initialize();
        var threadState = PythonEngine.BeginAllowThreads();

        using (Py.GIL())
        {
            using (var scope = Py.CreateScope())
            {
                scope.Set("a", 1);
                scope.Set("b", 2);
                scope.Exec("result = a + b");
                var result = scope.Get("result").As<int>();
                Console.WriteLine($"Sum = {result}");
            }
        }

        // allocate dotnet-managed memory unrelated to pythonnet
        var rnd = new Random();
        for (var i = 0; i < 100_000_000; ++i)
        {
            mem.Add(new byte[rnd.Next(1, 10)]);
        }

        PythonEngine.EndAllowThreads(threadState);
        Console.WriteLine("Shutdown start");
        var start = DateTime.Now;
        PythonEngine.Shutdown();
        var diff = DateTime.Now - start;
        Console.WriteLine($"Shutdown stop {diff.TotalSeconds}");
    }
}

Typical output:

Sum = 3
Shutdown start
Shutdown stop 61.6796983

Your issue does not occur when the project is ran in .NET 7 using a Release configuration. When ran in Debug configuration, the shutdown does indeed slow down significantly - from 0.08 seconds up to 63 seconds.

public class NoopFormatter : IFormatter {
public object Deserialize(Stream s) => throw new NotImplementedException();
public void Serialize(Stream s, object o) {}

public SerializationBinder Binder { get; set; }
public StreamingContext Context { get; set; }
public ISurrogateSelector SurrogateSelector { get; set; }
}

Python.Runtime.RuntimeData.FormatterType = typeof(NoopFormatter);

This does not fix the issue when the code is ran in Debug configuration under .NET 7.

gitlsl · 2023-08-30T07:20:05Z

v3.0.2 same question, NoopFormatter has no effact

filmor mentioned this issue Feb 23, 2023

PythonEngine.Shutdown() Crash #2107

Closed

agoodwin mentioned this issue Sep 6, 2023

Random Access Violations on shutdown #1977

Open

Ledragon mentioned this issue May 19, 2025

Memory not released upon multiple Initialize/Shutdown #2594

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PythonEngine.Shutdown() / Runtime.Shutdown() slowness #2008

PythonEngine.Shutdown() / Runtime.Shutdown() slowness #2008

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Environment

Details

Uh oh!

Uh oh!

PythonEngine.Shutdown() / Runtime.Shutdown() slowness #2008

PythonEngine.Shutdown() / Runtime.Shutdown() slowness #2008

Comments

Environment

Details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Environment

Details

Uh oh!

Uh oh!