[DSR] - RHEL 9 x64 #17548

hanneshayashi · 2022-06-21T05:17:06Z

hanneshayashi · 2022-06-21T05:26:03Z

RHEL 9 uses OpenSSL 3, which is causing issues when using the current RHEL 8 release of PowerShell.
The following Dockerfile installs PowerShell and attempts to install a Module from the PowerShell Gallery:

FROM quay.io/centos/centos:stream9

RUN curl https://packages.microsoft.com/config/rhel/8/prod.repo | tee /etc/yum.repos.d/microsoft.repo \
  && dnf -y upgrade --refresh \
  && dnf -y install powershell \
  && pwsh -Command "Install-Module -Name AzTable -Force"

ENTRYPOINT ["/usr/bin/env"]

Note that it does not matter which module. They all fail with the following error message:

WARNING: Source Location 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' is not valid.
Install-Package: Package 'AzTable' failed to download.

With verbose output, we get a little more detail and it becomes clear where the issue is:

ERBOSE: Acquiring providers for assembly: /opt/microsoft/powershell/7/Modules/PackageManagement/coreclr/netstandard2.0/Microsoft.PackageManagement.ArchiverProviders.dll
VERBOSE: Acquiring providers for assembly: /opt/microsoft/powershell/7/Modules/PackageManagement/coreclr/netstandard2.0/Microsoft.PackageManagement.NuGetProvider.dll
VERBOSE: Acquiring providers for assembly: /opt/microsoft/powershell/7/Modules/PackageManagement/coreclr/netstandard2.0/Microsoft.PackageManagement.MetaProvider.PowerShell.dll
VERBOSE: Acquiring providers for assembly: /opt/microsoft/powershell/7/Modules/PackageManagement/coreclr/netstandard2.0/Microsoft.PackageManagement.CoreProviders.dll
VERBOSE: Using the provider 'PowerShellGet' for searching packages.
VERBOSE: The -Repository parameter was not specified.  PowerShellGet will use all of the registered repositories.
VERBOSE: Getting the provider object for the PackageManagement Provider 'NuGet'.
VERBOSE: The specified Location is 'https://www.powershellgallery.com/api/v2' and PackageManagementProvider is 'NuGet'.
VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzTable'' for ''.
VERBOSE: Total package yield:'1' for the specified package 'AzTable'.
VERBOSE: Performing the operation "Install-Module" on target "Version '2.1.0' of module 'AzTable'".
VERBOSE: The installation scope is specified to be 'CurrentUser'.
VERBOSE: The specified module will be installed in '/root/.local/share/powershell/Modules'.
VERBOSE: The specified Location is 'NuGet' and PackageManagementProvider is 'NuGet'.
VERBOSE: Downloading module 'AzTable' with version '2.1.0' from the repository 'https://www.powershellgallery.com/api/v2'.
VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzTable'' for ''.
VERBOSE: InstallPackage' - name='AzTable', version='2.1.0',destination='/tmp/734365082'
VERBOSE: DownloadPackage' - name='AzTable', version='2.1.0',destination='/tmp/734365082/AzTable.2.1.0/AzTable.2.1.0.nupkg', uri='https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0'
VERBOSE: Downloading 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0'.
VERBOSE: The SSL connection could not be established, see inner exception.
VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' for '2' more times
VERBOSE: The SSL connection could not be established, see inner exception.
VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' for '1' more times
VERBOSE: The SSL connection could not be established, see inner exception.
VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' for '0' more times
VERBOSE: Downloading package 'AzTable' failed, please make sure 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' is accessable.
WARNING: Source Location 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' is not valid.

This does not only affect PowerShell Gallery, but it is easy to reproduce this way.
Up until a few days ago, we used a workaround, were we locked openssl and openssl-libs to an older version:

# ...
&& dnf install -y 'dnf-command(versionlock)' \
&& dnf versionlock add openssl-libs-1:3.0.1-5.el9 openssl-1:3.0.1-5.el9 \
# ...

This version has since been removed from the DNF repo, which leaves us unable to build a container based on RHEL 9 / CentOS Stream 9.

jborean93 · 2022-06-21T11:10:31Z

I've just done some more testing and PowerShell supports OpenSSL 3 for me. It just uses the functionality in dotnet which has been in place since .NET 6 (what pwsh 7.2.x uses) dotnet/runtime#46526.

What is interested in that this works in the test container

Invoke-WebRequest https://httpbin.org/get

But this does not (what Install-Module is calling)

Invoke-WebRequest https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0
# Invoke-WebRequest: The remote certificate is invalid because of errors in the certificate chain: NotSignatureValid

I've also verified that the OpenSSL 3.x libs are loaded in the process and it's not picking up any other ones that might be in the path

PS /> get-content /proc/$pid/maps | select-string 'libssl*'

7f491121f000-7f491123d000 r--p 00000000 00:21 5069468                    /usr/lib64/libssl.so.3.0.1
7f491123d000-7f4911299000 r-xp 0001e000 00:21 5069468                    /usr/lib64/libssl.so.3.0.1
7f4911299000-7f49112b6000 r--p 0007a000 00:21 5069468                    /usr/lib64/libssl.so.3.0.1
7f49112b6000-7f49112c0000 r--p 00096000 00:21 5069468                    /usr/lib64/libssl.so.3.0.1
7f49112c0000-7f49112c4000 rw-p 000a0000 00:21 5069468                    /usr/lib64/libssl.so.3.0.1

What is interesting is that openssl s_client -connect www.powershellgallery.com:443 works just fine with the cert that the PowerShell gallery exposes which tells me that potentially support for OpenSSL 3.x in dotnet still has some kinks left to work through. The issue dotnet/runtime#67304 seems to be quite closely related, they do say that .NET 6 shouldn't be affected but maybe it still is?

hanneshayashi · 2022-06-21T11:34:04Z

Thank you for investigating! You are right that it seems like not all SSL connections are affected, but it is definitely not exclusive to Install-Module. I believe (although I can't verify it at the moment) Invoke-Command with a Windows host as a target is was also affected.

PowerShell seems to at least try to use the installed openssl, but it seems like newer versions are causing issues (hence our workaround by locking it to an older one).

jborean93 · 2022-06-21T11:52:12Z

Yea I’ll have to try and investigate it some more, the latest pwsh preview which is based ok dotnet 7 should be good but it also has problems. There’s something up with the cert that the PSGallery uses which is acting differently in OpenSSL 3. All this happens in dotnet and it’s not really controlled by PowerShell but it is definitely something that should be investigated and fixed if need be.

hanneshayashi · 2022-06-21T11:57:55Z

This still gives me the same error:

FROM quay.io/centos/centos:stream9

RUN curl https://packages.microsoft.com/config/rhel/8/prod.repo | tee /etc/yum.repos.d/microsoft.repo \
  && dnf -y upgrade --refresh \
  && dnf -y install powershell-preview \
  && pwsh-preview -Command "Install-Module -Name AzTable -Force -Verbose"

ENTRYPOINT ["/usr/bin/env"]

VERBOSE: Acquiring providers for assembly: /opt/microsoft/powershell/7-preview/Modules/PackageManagement/coreclr/netstandard2.0/Microsoft.PackageManagement.MetaProvider.PowerShell.dll
VERBOSE: Acquiring providers for assembly: /opt/microsoft/powershell/7-preview/Modules/PackageManagement/coreclr/netstandard2.0/Microsoft.PackageManagement.CoreProviders.dll
VERBOSE: Acquiring providers for assembly: /opt/microsoft/powershell/7-preview/Modules/PackageManagement/coreclr/netstandard2.0/Microsoft.PackageManagement.ArchiverProviders.dll
VERBOSE: Acquiring providers for assembly: /opt/microsoft/powershell/7-preview/Modules/PackageManagement/coreclr/netstandard2.0/Microsoft.PackageManagement.NuGetProvider.dll
VERBOSE: Using the provider 'PowerShellGet' for searching packages.
VERBOSE: The -Repository parameter was not specified.  PowerShellGet will use all of the registered repositories.
VERBOSE: Getting the provider object for the PackageManagement Provider 'NuGet'.
VERBOSE: The specified Location is 'https://www.powershellgallery.com/api/v2' and PackageManagementProvider is 'NuGet'.
VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzTable'' for ''.
VERBOSE: Total package yield:'1' for the specified package 'AzTable'.
VERBOSE: Performing the operation "Install-Module" on target "Version '2.1.0' of module 'AzTable'".
VERBOSE: The installation scope is specified to be 'CurrentUser'.
VERBOSE: The specified module will be installed in '/root/.local/share/powershell/Modules'.
VERBOSE: The specified Location is 'NuGet' and PackageManagementProvider is 'NuGet'.
VERBOSE: Downloading module 'AzTable' with version '2.1.0' from the repository 'https://www.powershellgallery.com/api/v2'.
VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='AzTable'' for ''.
VERBOSE: InstallPackage' - name='AzTable', version='2.1.0',destination='/tmp/1012252532'
VERBOSE: DownloadPackage' - name='AzTable', version='2.1.0',destination='/tmp/1012252532/AzTable.2.1.0/AzTable.2.1.0.nupkg', uri='https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0'
VERBOSE: Downloading 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0'.
VERBOSE: The SSL connection could not be established, see inner exception.
VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' for '2' more times
VERBOSE: The SSL connection could not be established, see inner exception.
VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' for '1' more times
VERBOSE: The SSL connection could not be established, see inner exception.
VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' for '0' more times
VERBOSE: Downloading package 'AzTable' failed, please make sure 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' is accessable.
WARNING: Source Location 'https://www.powershellgallery.com/api/v2/package/AzTable/2.1.0' is not valid.
Install-Package: Package 'AzTable' failed to download.

kasini3000 · 2022-06-22T03:45:33Z

mark , I will update the one-click install script as soon as the rpm package is available in package.microsoft.com.

one-click install powershell on linux
https://gitee.com/chuanjiao10/kasini3000_agent_linux

hanneshayashi · 2022-06-22T04:43:29Z

Just for the record: I just tested it with the new 7.2.5 release and still get the same error.

hanneshayashi · 2022-06-22T11:20:52Z

It seems like there were (are?) some more issues related to openssl and RHEL 9. For example dotnet/runtime#70343. It seems like this particular fix was merged into .Net 6.0.7. Not sure which version the current release is using, but I suppose it's probably a little older.

jborean93 · 2022-06-22T11:25:34Z

That does seem quite relevant, considering it was only merged 13 days ago I would be surprised if 7.2.x has been updated to 6.0.7 (6.0.6 is also the latest in the GH releases page).

hanneshayashi · 2022-06-22T11:43:23Z

I don't suppose there is any way to find out if this fixes the issue by trying to build PowerShell against this unreleased .NET version, is there? 😄

We are currently stuck without a working PowerShell on RHEL 9...

For what it's worth https://pkgs.dev.azure.com also recreates the error:

Invoke-WebRequest https://pkgs.dev.azure.com
# Invoke-WebRequest: The remote certificate is invalid because of errors in the certificate chain: NotSignatureValid

Which indicates that there are probably a lot more sites and services that don't work either.

Dockerfile for reference:

FROM quay.io/centos/centos:stream9

RUN curl https://packages.microsoft.com/config/rhel/8/prod.repo | tee /etc/yum.repos.d/microsoft.repo \
  && dnf -y upgrade --refresh \
  && dnf -y install powershell \
  && pwsh -Command "Invoke-WebRequest https://pkgs.dev.azure.com"

ENTRYPOINT ["/usr/bin/env"]

jborean93 · 2022-06-22T11:48:13Z

If you can get a compiled build of dotnet including that version you should be able to replace the dll's for it in pwsh itself. There's a lot in there so it's not just a simple thing to do unless you know what to change unfortunately.

hanneshayashi · 2022-06-23T05:19:33Z

It looks like v7.3.0-preview.5 fixes the problem, which would hopefully mean that the backport to .NET 6 also fixes it. In the meantime, I will have to see if we can work with the preview version for now.

hanneshayashi · 2022-07-13T05:02:15Z

.NET 6.0.7 was just released: https://github.com/dotnet/runtime/releases/tag/v6.0.7
Can we expect the next stable version of PowerShell to be based on that release?

GitHub

Release .NET 6.0.7 · dotnet/runtime

Release

hanneshayashi · 2022-08-12T05:01:50Z

7.2.6 seems to have resolved the OpenSSL issues.

lumarel · 2022-11-17T12:54:51Z

Is there anything missing now to get the package in the RHEL 9 prod (|insider-(slow|fast) repo?

lumarel · 2023-02-27T09:40:01Z

@jborean93 (Sorry for pinging)
Do you have any update on this topic? We are currently already installing the package via the GH artifact, and also semi-manually updating, it would greatly improve the experience, if the package would get published to the repo.

Kalimetouf · 2023-03-20T09:04:40Z

Hello,

Do we have an update on this, any ETA of a full PowerShell Support for RHEL9 distribs ?

Thank you !

wilvh · 2023-06-08T08:12:19Z

Hello,

Same question here. When will RedHat Enterprise Linux 9 be officially supported? We have a pending project that requires Powershell on this operating system.

Thanks

patrickschneider · 2023-06-13T06:31:44Z

Hello,

according to the Table at
https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-linux?view=powershell-7.3&preserve-view=true#red-hat-enterprise-linux-rhel
RHEL 9 is already a supported OS for PowerShell 7.2+

Best regards,
Patrick

wilvh · 2023-06-13T06:43:49Z

Hello,

Thanks for the information. However there doesn't seem to be a Microsoft repository for RHEL 9. The URL https://packages.microsoft.com/config/rhel/9/prod.repo doesn't exist and I don't know if I should use the RHEL 8 URL.

Edit:
The URL must be https://packages.microsoft.com/rhel/9/prod/ as mentioned in #19672 but there's no Powershell package in it.

Best regards,

patrickschneider · 2023-06-13T08:27:20Z

Hi,

yes, you're absolutely right. I used the RHEL 8 Repo URL on RHEL 9 and so far I did not run into issues. But it would be a much better and cleaner if the RHEL 9 Repo would contain the PowerShell Package. Even if the RPM-File is identical (e.g. I compared the sha256sum of the RPM-Package from the MS RHEL 7 Repo with the RPM-Package from the MS RHEL 8 Repo and they're identical), the proper way to be able to maintain the Package on RHEL 9 would be the use of the RHEL 9 Repo.

Best regards,
Patrick

lumarel · 2023-10-12T21:11:47Z

As far as I can see since 7.2.14 / 7.3.7 the packages are sent to the repo, thank you so much for making our lives easier!
I think this issue can be closed now.

theJasonHelmick · 2024-10-29T17:33:55Z

This package is now available at PMC. For more information see the PMC Readme:
https://packages.microsoft.com/docs/readme.txt

We are still working on the docker image pipeline. I will post an update when we have published
images to the MCR.

theJasonHelmick · 2025-01-07T16:20:01Z

The RHEL 9 distribution is released to PMC and MCR.

microsoft-github-policy-service · 2025-01-07T16:20:10Z

📣 Hey @hanneshayashi, how did we do? We would love to hear your feedback with the link below! 🗣️

🔗 https://aka.ms/PSRepoFeedback

Microsoft Forms

hanneshayashi added Distribution-Request Request for a new distribution support Needs-Triage The issue is new and needs to be triaged by a work group. labels Jun 21, 2022

hanneshayashi changed the title ~~Distribution Support Request~~ Distribution Support Request - RHEL 9 Jun 21, 2022

HeyItsGilbert mentioned this issue Nov 15, 2023

No CentOS 9 repository on packages.microsoft.com microsoft/linux-package-repositories#91

Closed

theJasonHelmick added this to Distribution Requests Feb 29, 2024

theJasonHelmick moved this to Requests in Distribution Requests Feb 29, 2024

sdwheeler moved this from Requests to Backlog in Distribution Requests Mar 4, 2024

sdwheeler moved this from Backlog to Requests in Distribution Requests Mar 4, 2024

theJasonHelmick changed the title ~~Distribution Support Request - RHEL 9~~ [DSR] - RHEL 9 x64 Mar 7, 2024

theJasonHelmick moved this from Requests to Planned in Distribution Requests Mar 7, 2024

This was referenced Mar 7, 2024

[DSR] - RHEL9 #19672

Closed

[DSR] - RHEL9 -x64 #19748

Closed

theJasonHelmick added the KeepOpen The bot will ignore these and not auto-close label Sep 16, 2024

similar-issues-ai bot mentioned this issue Sep 23, 2024

[DSR] - Debian 12 x64 #24334

Closed

14 tasks

theJasonHelmick closed this as completed Jan 7, 2025

github-project-automation bot moved this from Planned to Done in Distribution Requests Jan 7, 2025

microsoft-github-policy-service bot removed Needs-Triage The issue is new and needs to be triaged by a work group. KeepOpen The bot will ignore these and not auto-close labels Jan 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[DSR] - RHEL 9 x64 #17548

[DSR] - RHEL 9 x64 #17548

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[DSR] - RHEL 9 x64 #17548

[DSR] - RHEL 9 x64 #17548

Comments

Name of the Distribution

Version of the Distribution

Package Types

Processor Architecture

.NET Core Support

Distribution Requirements

Exemption Justification

Progress

For PowerShell Team ONLY

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!