Discussion: should PowerShell implement assembly load contexts for module dependencies, and how? #12920

rjmholt · 2020-06-09T00:09:00Z

.NET Core 2 introduced the concept of the AssemblyLoadContext.

This offers the possibility of preventing assembly dependency conflicts between modules, and even the possibility of unloading assemblies.

This has been successfully implemented in at least one PowerShell module in the wild. There's a blog post detailing how to do this here.

However, the question remains as to whether PowerShell should attempt to provide this functionality itself.

While it may sound like it definitely should, there are a number of subtle considerations, and it may be that PowerShell won't be able to get them right for all modules, making module loading more complex and error prone.

Some things to consider:

Should PowerShell users be able to choose whether a module is loaded through an ALC?
Should module authors be able to choose the same?
Loading through an ALC can create type identity issues if a module exposes its identities' types in its own API. This is a legitimate thing to do (e.g. a YAML-handling module that emits YAML objects defined by its underlying library). How do we handle this? Are we responsible for detecting it ahead of time? (In many cases this will be impossible, since the exposed APIs on an object in PowerShell are determined by reflection at runtime)
Passing dependency-typed objects between commands from different modules is also a valid scenario. Is there special handling here?
Is it feasible (both technically and in terms of maintenance) to provide tooling to help module authors analyse and understand how assembly load contexts should work with their module?
Will using ALCs for modules affect PowerShell classes in terms of how they are currently implemented or how they should work?

Note: An RFC about this topic that was withdrawn can be found here: https://github.com/PowerShell/PowerShell-RFC/blob/master/X-Withdrawn/RFC0043-Loading-Module-Into-Isolated-AssemblyLoadContext.md
The biggest challenge is the type identity issue. The module dependency is another challenge -- how to handle a required module? Today, the required module is loaded once and shared by multiple modules that depend on it.

Some other places where ALCs have been discussed:

vexx32 · 2020-06-09T01:10:26Z

Should PowerShell users be able to choose whether a module is loaded through an ALC?

Nope. No way, no how. ALCs are a pretty advanced concept, and I very much doubt you could reliably code a module to work both in an isolated and the shared ALC. Letting users of a module turn that on or off on a whim is pretty much just going to break things, and not much else.

Should module authors be able to choose the same?

Yeah, sure, it may not be super useful for all modules, but it definitely could come in handy for some. Authors should be able to pick, 100%.

Is it feasible (both technically and in terms of maintenance) to provide tooling to help module authors analyse and understand how assembly load contexts should work with their module?

Feasible? Potentially. On the assumption that we do make it a feature, though, tooling and documentation will be necessary.

Don't know enough to comment on the other bits, I think, I'll leave that to the cleverer cookies. 🙂

rjmholt · 2020-06-09T01:35:38Z

Should PowerShell users be able to choose whether a module is loaded through an ALC?

Nope. No way, no how. ALCs are a pretty advanced concept... Letting users of a module turn that on or off on a whim is pretty much just going to break things, and not much else.

I definitely agree with this point. I just want to raise a counterpoint which is that trying to defuse a module dependency conflict is a user scenario.

If only module authors get to decide, then they must do the due diligence on what other modules they might conflict with. When a user uses a module and it conflicts with something, there’s no switch they can try to get them out of a bind (like there is with-SkipEditionCheck).

OTOH perhaps that’s exactly as it should be. Users find a conflict and circle back to authors ask them to evaluate ALC loading.

rjmholt · 2020-06-09T02:12:15Z

I feel like we've already had parts of this discussion elsewhere. Is there an issue I've forgotten about?

SeeminglyScience · 2020-06-09T03:25:51Z

#2083 was the big one. #5504 also looks related.

This would be a fantastic tool to allow module developers to take on dependencies without worrying about breaking the environment. One thing I'd really like to see is an easy way to allow public types from the modules assembly to be resolvable by PowerShell, while still "hiding" the dependencies. Maintaining the ability to strongly type variables, use the -is operator, read OutputType decorations etc.

Personally I would only want to see this surfaced as an option for the module author, definitely off by default.

rjmholt · 2020-06-09T05:22:12Z

#2083 was the big one. #5504 also looks related.

Ah! Thanks!

daxian-dbw · 2020-06-12T18:18:45Z

Note: An RFC about this topic that was withdrawn can be found here: https://github.com/PowerShell/PowerShell-RFC/blob/master/X-Withdrawn/RFC0043-Loading-Module-Into-Isolated-AssemblyLoadContext.md
The biggest challenge is the type identity issue. The module dependency is another challenge -- how to handle a required module? Today, the required module is loaded once and shared by multiple modules that depend on it.

I updated the issue description with the above content, just to call out that we have an RFC about this that was withdrawn at the time.
We can potentially revive that RFC as the discussion here progresses.

mattpwhite · 2020-06-14T17:01:48Z

I'm not sure that I have a specific suggestion here, but wanted to provide some input about how we've hacked around these issues.

We have a large set of internally developed modules, with a number of third party modules mixed in. Many of these have dependencies on .NET libraries. The types from those libraries are used directly in PS code (including as members of PS classes) as well as within binary modules. Module A might be a binary module that depends on types in an assembly that is loaded by module B, that kind of thing.
Our internal modules load their dependencies via a custom loader module. This module's responsibility is to do things like avoid locking the dependency on disk (either by making a "shadow copy" elsewhere or using Assembly.Load to load from a byte array). The loader also registers an AssemblyResovle handler that tries to smooth over version conflicts (dicey, but less bad than simply failing). Finally, the loader is provided with a litmus test type name, if it finds that type in an already loaded assembly, it returns that assembly rather than actually loading anything new.
At least with the internal modules, we are careful to always have a single module "own" the dependency. For instance, we have our Json.NET wrapper module, if other modules depend on Json.NET (or load assemblies that depend on Json.NET), they declare a dependency on the Json.NET wrapper module and let it take care of loading it first. Between this and the AssemblyResolve handler, things mostly work.
In cases where we have to deal with third party modules with dependencies on old versions of things, we try to arrange things such that our version gets loaded first (and the AssemblyResolve handler with it). This requires doing dirty things sometimes - patching the third party manifest/psm1 and/or not actually declaring all dependencies in the manifest. It's gross, but at least in 5.1, there are so many bugs with expressing module dependencies (mostly related to bad dependency graph building) that we're kinda used to it.

This is all terrible, and we'd absolutely welcome something better. On the other hand, we have been able to make it mostly work for our scenarios. While I agree that asking module authors to engage with and understand ALCs might not be a great idea, I also wouldn't want the PS runtime to impose some "standard" behavior that breaks scenarios that can be made to work now (albeit badly).

I'm not sure how you differentiate the scenario where you're loading some module because you just want access to commands it exports and the scenario where you're loading it because you want the side effect of making types in the assemblies it loads to be made available to you (either directly or to another dependency you're going to load). At least I can't see how a default behavior would know which of these you're looking for unless you had some way to explicitly indicate that. That might not mean fully exposing the gory details of ALCs, but you'd need something. Making the 80% case easy and the 20% case impossible is worse than the 100% case being hard.

JustinGrote · 2020-07-02T18:50:06Z

For my most common use cases (Dumb example: I want to use Json 10 and Powershell uses Json 11+), if I could just specify assemblies or nuget packages in the manifest that would load into my module-specific ALC context but the rest of my module stays in the main context, I'd be happy. As long as I follow a couple rules:

I don't leak any of those classes out of the module ALC/scope
I only export PSCustomObjects or non-inherited Powershell Class objects

Then I can make my modules more broadly compatible. A wider ability of Powershell to "auto-detect" this so that module authors don't have to even think about it would be great, but to at least get something out the door it's a good place to start.

rjmholt · 2020-07-02T19:21:39Z

A wider ability of Powershell to "auto-detect" this so that module authors don't have to even think about it would be great

If you mean autodetection of whether those rules are followed, we could never do it safely because it's unfortunately undecidable given current APIs. Something like WriteObject could emit any type of object at runtime and the particular object emitted could depend on any arbitrary program logic, so it can't be known statically.

I don't mean that as a way of saying everything is impossible or shouldn't be done, more just want to establish that no strong solution exists for some of the problems we face here.

iSazonov · 2020-11-23T06:03:54Z

I suggest to start at 7.2 milestone with most simple and obvious - add an experimental parameter to Import-Module cmdlet so that PowerShell users can load module dependencies in ALCs. This will open the way for broader experimentation and provide more specific and useful feedback.

SeeminglyScience · 2020-11-23T14:19:30Z

@iSazonov I'm not sure it makes sense as a Import-Module parameter. This would be something the module author would opt into, not the user. If the module author doesn't design specifically for isolated use, I think it's more likely to break than work.

JustinGrote · 2020-11-23T16:43:35Z

@SeeminglyScience I agree, I think at this point just make some helpful cmdlets to load assemblies into an ALC for the module author without them needing to know C#.

Something like Add-Type -Path MyAssembly -AssemblyContextName MyModuleContext and show a warning if any of the types imported into the context already exist in the main ALC and that they should not export these types outside the module or they will get weird errors.

SeeminglyScience · 2020-11-23T16:54:41Z

Yeah that's the right direction for sure. I think even that has it's obstacles though, especially around dependency resolution. I know PSES had to register some custom assembly resolution handlers, not sure how typical that is.

iSazonov · 2020-11-23T17:17:33Z

Our goal is to get working modules despite the presence of conflicting assembly versions. Simply loading an assembly using Add-Type, or say it should be the concern of the module author, won't get us a step further.
Before making any complex decisions, we first need to simply load the module and then its dependencies into ALC.
For this we need an experimental parameter. In the future, it can be changed or removed as unnecessary if PowerShell is already smart enough.

JustinGrote · 2020-11-23T17:45:01Z

@iSazonov agreed, that's the ideal, I'm just saying baby steps to at least get a friendly ALC context for a module developer would be great for 7.2 vs. a more ambitious full module resolution and all the edge cases that would probably push it to 7.3 or more.

SeeminglyScience · 2020-11-23T17:54:07Z

Our goal is to get working modules despite the presence of conflicting assembly versions. Simply loading an assembly using Add-Type, or say it should be the concern of the module author, won't get us a step further.

To clarify, I'm not saying it should be, I'm saying it has to be because of how they work. It's not the perfect solution it may have seemed like originally, it's incredibly finicky and needs to be actively designed around.

iSazonov · 2020-11-23T18:00:11Z

it's incredibly finicky

This is why we need the experimental parameter to load two modules and do live testing.

SeeminglyScience · 2020-11-23T18:05:35Z

I'm not saying it's unpredictable, I'm saying it needs to be explicitly designed around.

Stewartarmbrecht · 2021-01-18T15:26:21Z

While this is being worked out, what is the direction to typical PowerShell users who are unable to import two modules into the same session. In my case I am unable to import powershell-yaml and platyPS into the same session.

=====> PowerShell Integrated Console v2020.6.0 <=====
PS /home/codespace/workspace> Install-Module powershell-yaml
PS /home/codespace/workspace> Install-Module platyPS
PS /home/codespace/workspace> Import-Module powershell-yaml
PS /home/codespace/workspace> Import-Module platyPS
Import-Module: Assembly with same name is already loaded

Is the response simply that you can not use these two modules together? Or is the expectation that the user should log an issue with one of the module owners? Is this something that would be reasonable to expect a fix for from the module owner?

JustinGrote · 2021-01-18T16:03:58Z

@Stewartarmbrecht the current workaround I use is to "start-job" the other module in a new process and do my work with it there, then return the output back to its "parent" process. However if you need tight integration between two modules this gets ugly fast.

The only other option in terms of module author support would be to expand Powershell Standard to include some of the most common libraries, so that module authors use the same assembly versions and don't conflict. This is fraught with issues however especially if a module author requires a newer version of a module.

Stewartarmbrecht · 2021-01-18T16:36:43Z

Thanks @JustinGrote I'll work on running PlatyPS using Start-Job.

Jeff-Lewis · 2021-07-23T21:05:54Z

Besides an impact on memory, what are the drawbacks of having every PS module load its dependencies into it's own ALC by default? This is how npm works and it makes it easy to depend on a lot of external packages.

JustinGrote · 2021-07-23T21:10:37Z

@Jeff-Lewis the biggest one is module owner discipline. If you return an ALC type to the user, and the type is already loaded, you get the very confusing "cannot cast MyType to type MyType" error message. There would have to be extensive sanity checking here to make this usable to the "common" module author who's not a guru.

rmbolger · 2022-09-01T20:40:30Z

I don't really have any suggestions. But as a script-based module author, I'd just like to request that whatever solution eventually comes will work with script-based modules in addition to binary modules.

There doesn't seem to be any good workarounds at the moment that don't involve getting into esoteric (to me) .NET compilation stuff. My only option seems to be trying to match dependency versions with other modules that my users report conflicts with.

JustinGrote · 2022-09-01T20:45:52Z

@rmbolger Are you basically expecting Add-Type to automatically load into an ALC if it is in a module context? That may be extremely difficult.

rmbolger · 2022-09-01T20:58:01Z

I don't use Add-Type today. I use RequiredAssemblies in the module manifest and the conflicting module does as well. I think I would expect some sort of additional manifest property that would explicitly tell PS to load the assembly into an ALC. Though I'm still not sure exactly what that means in terms of using it in my code.

My module also works on both PS 5.1 and 6+. So I'd hope the manifest changes would remain backwards compatible which probably means they'd have to live in the PrivateData section?

microsoft-github-policy-service · 2023-11-16T03:04:26Z

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

microsoft-github-policy-service · 2023-11-16T03:04:26Z

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

microsoft-github-policy-service · 2023-11-16T03:04:26Z

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

microsoft-github-policy-service · 2023-11-23T11:59:25Z

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

rjmholt added Issue-Enhancement the issue is more of a feature request than a bug WG-Engine core PowerShell engine, interpreter, and runtime labels Jun 9, 2020

rjmholt mentioned this issue Jun 11, 2020

Import-Module : Could not load file or assembly System.Collections.Immutable #12941

Closed

SeeminglyScience mentioned this issue Jul 20, 2020

Question on common dependency binary cmdlet module #13213

Closed

rjmholt mentioned this issue Nov 18, 2020

YamlDotNet - Assembly with same name is already loaded PowerShell/platyPS#481

Closed

microsoft-github-policy-service bot added Resolution-No Activity Issue has had no activity for 6 months or more labels Nov 16, 2023

microsoft-github-policy-service bot closed this as completed Nov 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Discussion: should PowerShell implement assembly load contexts for module dependencies, and how? #12920

Discussion: should PowerShell implement assembly load contexts for module dependencies, and how? #12920

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Discussion: should PowerShell implement assembly load contexts for module dependencies, and how? #12920

Discussion: should PowerShell implement assembly load contexts for module dependencies, and how? #12920

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!