8000 Restructure the package build to simplify signing and packaging stages by adityapatwardhan · Pull Request #19321 · PowerShell/PowerShell · GitHub
[go: up one dir, main page]

Skip to content

Restructure the package build to simplify signing and packaging stages #19321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 39 commits into from
Mar 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
4e5c3e2
Only build assemblies
adityapatwardhan Feb 13, 2023
24384ea
Remove signfiles stage
adityapatwardhan Feb 13, 2023
39a45e3
Add signing for files
adityapatwardhan Feb 16, 2023
a590b70
Fix matrix name
adityapatwardhan Feb 16, 2023
c131544
Fix template call
adityapatwardhan Feb 16, 2023
a93ca75
Fix deps
adityapatwardhan Feb 16, 2023
c3d75fa
Remove macos signing dep
adityapatwardhan Feb 16, 2023
63201b0
Fix condition
adityapatwardhan Feb 16, 2023
76d7d9f
fix typo
adityapatwardhan Feb 16, 2023
8e8ce80
fix typo 2
adityapatwardhan Feb 16, 2023
27d8668
Update variables to be runtime variables
adityapatwardhan Feb 16, 2023
5fe78cf
Fix download
adityapatwardhan Feb 16, 2023
03bb412
Fix quotes
adityapatwardhan Feb 16, 2023
d22142c
Add packaging and package signing
adityapatwardhan Feb 16, 2023
5334f9d
Remove mac deps
adityapatwardhan Feb 16, 2023
16efa63
Re-order nuget
adityapatwardhan Feb 16, 2023
a2dfeca
Re-order steps
adityapatwardhan Feb 16, 2023
0fc5184
Fix paths
adityapatwardhan Feb 16, 2023
dab9714
Add self checkout
adityapatwardhan Feb 16, 2023
d8f3e19
Fix indent
adityapatwardhan Feb 16, 2023
99444c0
fix mariner package name and paths
adityapatwardhan Feb 16, 2023
3b3cf26
fix display names
adityapatwardhan Feb 17, 2023
01a1db0
Fix paths for downloaded files
adityapatwardhan Feb 17, 2023
223c209
Fix errors
adityapatwardhan Feb 17, 2023
c5f6153
Expand symbols zip
adityapatwardhan Feb 17, 2023
47fc47a
Fix misc issues
adityapatwardhan Feb 17, 2023
5a7afef
Add variables to signing job
adityapatwardhan Feb 17, 2023
7fb66a6
Update pool and checkout paths
adityapatwardhan Feb 17, 2023
5c247e4
Fix the typo
adityapatwardhan Feb 22, 2023
88f9028
Fix path
adityapatwardhan Feb 22, 2023
ce8ebf3
Fix variable name
adityapatwardhan Feb 23, 2023
088cc0f
Fix quotes
adityapatwardhan Feb 23, 2023
a5bc420
Fix file name of signed zip file
adityapatwardhan Feb 23, 2023
c1a5d20
Fix upload logic
adityapatwardhan Feb 23, 2023
e63d45a
Fix typo
adityapatwardhan Feb 23, 2023
eca2e70
Fix nuget stage order
adityapatwardhan Feb 23, 2023
965d45d
Add windows desktop fxdependent
adityapatwardhan Mar 10, 2023
1eaef47
Re-order nuget stage
adityapatwardhan Mar 10, 2023
549df39
Add display names and re-order compliance
adityapatwardhan Mar 13, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
299 changes: 204 additions & 95 deletions tools/releaseBuild/azureDevOps/releaseBuild.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ variables:
value: spdx:2.2
- name: BUILDSECMON_OPT_IN
value: true
- group: PoolNames

stages:
- stage: prep
Expand All @@ -68,30 +69,6 @@ stages:
parameters:
buildArchitecture: arm64

- template: templates/mac-file-signing.yml
parameters:
buildArchitecture: x64

- template: templates/mac-file-signing.yml
parameters:
buildArchitecture: arm64

- template: templates/mac-package-build.yml
parameters:
buildArchitecture: x64

- template: templates/mac-package-build.yml
parameters:
buildArchitecture: arm64

- template: templates/mac-package-signing.yml
parameters:
buildArchitecture: x64

- template: templates/mac-package-signing.yml
parameters:
buildArchitecture: arm64

- stage: linux
dependsOn: ['prep']
jobs:
Expand All @@ -113,29 +90,6 @@ stages:
parameters:
buildName: alpine

- template: templates/linux-authenticode-sign.yml

- template: templates/linux-packaging.yml
parameters:
buildName: deb
parentJob: sign_linux_builds

- template: templates/linux-packaging.yml
parameters:
buildName: rpm
uploadDisplayName: Upload and Sign
parentJob: sign_linux_builds

- template: templates/linux-packaging.yml
parameters:
buildName: alpine
parentJob: sign_linux_builds

- template: templates/linux-packaging.yml
parameters:
buildName: fxdependent
parentJob: sign_linux_builds

- stage: windows
dependsOn: ['prep']
jobs:
Expand Down Expand Up @@ -168,74 +122,229 @@ stages:
parameters:
Architecture: fxdependentWinDesktop

- template: templates/windows-packaging.yml
parameters:
Architecture: x64
parentJob: build_windows_x64_release

- template: templates/windows-packaging.yml
parameters:
Architecture: x64
BuildConfiguration: minSize
parentJob: build_windows_x64_minSize

- template: templates/windows-packaging.yml
parameters:
Architecture: x86
parentJob: build_windows_x86_release

- template: templates/windows-packaging.yml
parameters:
Architecture: arm
parentJob: build_windows_arm_release

- template: templates/windows-packaging.yml
parameters:
Architecture: arm64
parentJob: build_windows_arm64_release

- template: templates/windows-packaging.yml
parameters:
Architecture: fxdependent
parentJob: build_windows_fxdependent_release
- stage: SignFiles
displayName: Sign files
dependsOn: ['windows', 'linux', 'macos']
jobs:
- template: templates/mac-file-signing.yml
parameters:
buildArchitecture: x64

- template: templates/mac-file-signing.yml
parameters:
buildArchitecture: arm64

- job: SignFilesWinLinux
pool:
name: $(windowsPool)
demands:
- ImageOverride -equals PSMMS2019-Secure
displayName: Sign files

variables:
- group: ESRP
- name: runCodesignValidationInjection
value: false
- name: NugetSecurityAnalysisWarningLevel
value: none
- name: repoFolder
value: PowerShell
- name: repoRoot
value: $(Agent.BuildDirectory)\$(repoFolder)
- name: complianceRepoFolder
value: compliance

strategy:
matrix:
linux-x64:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider dynamically producing the matrix, so we can move it into a single JSON in our repo as we need this matrix in more places.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can do that as part of the next iteration, i need to split the deb build and packaging.

runtime: linux-x64
unsignedBuildArtifactContainer: pwshLinuxBuild.tar.gz
unsignedBuildArtifactName: pwshLinuxBuild.tar.gz
signedBuildArtifactName: pwshLinuxBuild.tar.gz
signedArtifactContainer: authenticode-signed
linux-x64-Alpine:
runtime: linux-x64-Alpine
unsignedBuildArtifactContainer: pwshLinuxBuildAlpine.tar.gz
unsignedBuildArtifactName: pwshLinuxBuild.tar.gz
signedBuildArtifactName: pwshLinuxBuildAlpine.tar.gz
signedArtifactContainer: authenticode-signed
linux-arm32:
runtime: linux-arm32
unsignedBuildArtifactContainer: pwshLinuxBuildArm32.tar.gz
unsignedBuildArtifactName: pwshLinuxBuildArm32.tar.gz
signedBuildArtifactName: pwshLinuxBuildArm32.tar.gz
signedArtifactContainer: authenticode-signed
linux-arm64:
runtime: linux-arm64
unsignedBuildArtifactContainer: pwshLinuxBuildArm64.tar.gz
unsignedBuildArtifactName: pwshLinuxBuildArm64.tar.gz
signedBuildArtifactName: pwshLinuxBuildArm64.tar.gz
signedArtifactContainer: authenticode-signed
linux-fxd:
runtime: linux-fxd
unsignedBuildArtifactContainer: pwshLinuxBuildFxdependent.tar.gz
unsignedBuildArtifactName: pwshLinuxBuild.tar.gz
signedBuildArtifactName: pwshLinuxBuildFxdependent.tar.gz
signedArtifactContainer: authenticode-signed
linux-mariner:
runtime: linux-mariner
unsignedBuildArtifactContainer: pwshMarinerBuildAmd64.tar.gz
unsignedBuildArtifactName: pwshMarinerBuildAmd64.tar.gz
signedBuildArtifactName: pwshMarinerBuildAmd64.tar.gz
signedArtifactContainer: authenticode-signed
linux-minsize:
runtime: linux-minsize
unsignedBuildArtifactContainer: pwshLinuxBuildMinSize.tar.gz
unsignedBuildArtifactName: pwshLinuxBuildMinSize.tar.gz
signedBuildArtifactName: pwshLinuxBuildMinSize.tar.gz
signedArtifactContainer: authenticode-signed
win-x64:
runtime: win-x64
unsignedBuildArtifactContainer: results
unsignedBuildArtifactName: '**/*-symbols-win-x64.zip'
signedBuildArtifactName: '-symbols-win-x64-signed.zip'
signedArtifactContainer: results
win-x86:
runtime: win-x86
unsignedBuildArtifactContainer: results
unsignedBuildArtifactName: '**/*-symbols-win-x86.zip'
signedBuildArtifactName: '-symbols-win-x86-signed.zip'
signedArtifactContainer: results
win-arm32:
runtime: win-arm32
unsignedBuildArtifactContainer: results
unsignedBuildArtifactName: '**/*-symbols-win-arm32.zip'
signedBuildArtifactName: '-symbols-win-arm32-signed.zip'
signedArtifactContainer: results
win-arm64:
runtime: win-arm64
unsignedBuildArtifactContainer: results
unsignedBuildArtifactName: '**/*-symbols-win-arm64.zip'
signedBuildArtifactName: '-symbols-win-arm64-signed.zip'
signedArtifactContainer: results
win-x64-gc:
runtime: win-x64-gc
unsignedBuildArtifactContainer: results
unsignedBuildArtifactName: '**/*-symbols-win-x64-gc.zip'
signedBuildArtifactName: '-symbols-win-x64-gc-signed.zip'
signedArtifactContainer: results
win-fxdependent:
runtime: win-fxdependent
unsignedBuildArtifactContainer: results
unsignedBuildArtifactName: '**/*-symbols-win-fxdependent.zip'
signedBuildArtifactName: '-symbols-win-fxdependent-signed.zip'
signedArtifactContainer: results
win-fxdependentWinDesktop:
runtime: win-fxdependentWinDesktop
unsignedBuildArtifactContainer: results
unsignedBuildArtifactName: '**/*-symbols-win-fxdependentWinDesktop.zip'
signedBuildArtifactName: '-symbols-win-fxdependentWinDesktop-signed.zip'
signedArtifactContainer: results
steps:
- template: templates/sign-build-file.yml

- stage: mac_packaging
displayName: macOS packaging
dependsOn: ['SignFiles']
jobs:
- template: templates/mac-package-build.yml
parameters:
buildArchitecture: x64

- template: templates/windows-packaging.yml
parameters:
Architecture: fxdependentWinDesktop
parentJob: build_windows_fxdependentWinDesktop_release
- template: templates/mac-package-build.yml
parameters:
buildArchitecture: arm64

- template: templates/windows-package-signing.yml
parameters:
parentJobs:
- sign_windows_x64_release
- sign_windows_x64_minSize
- sign_windows_x86_release
- sign_windows_arm_release
- sign_windows_arm64_release
- sign_windows_fxdependent_release
- sign_windows_fxdependentWinDesktop_release
- stage: linux_packaging
displayName: Linux Packaging
dependsOn: ['SignFiles']
jobs:
- template: templates/linux-packaging.yml
parameters:
buildName: deb

- template: templates/linux-packaging.yml
parameters:
buildName: rpm
uploadDisplayName: Upload and Sign

- template: templates/linux-packaging.yml
parameters:
buildName: alpine

- template: templates/linux-packaging.yml
parameters:
buildName: fxdependent

- stage: win_packaging
displayName: Windows Packaging
dependsOn: ['SignFiles']
jobs:
- template: templates/windows-packaging.yml
parameters:
Architecture: x64
parentJob: build_windows_x64_release

- template: templates/windows-packaging.yml
parameters:
Architecture: x64
BuildConfiguration: minSize
parentJob: build_windows_x64_minSize

- template: templates/windows-packaging.yml
parameters:
Architecture: x86
parentJob: build_windows_x86_release

- template: templates/windows-packaging.yml
parameters:
Architecture: arm
parentJob: build_windows_arm_release

- template: templates/windows-packaging.yml
parameters:
Architecture: arm64
parentJob: build_windows_arm64_release

- template: templates/windows-packaging.yml
parameters:
Architecture: fxdependent
parentJob: build_windows_fxdependent_release

- template: templates/windows-packaging.yml
parameters:
Architecture: fxdependentWinDesktop
parentJob: build_windows_fxdependentWinDesktop_release

- stage: package_signing
displayName: Package Signing
dependsOn: ['mac_packaging', 'linux_packaging', 'win_packaging']
jobs:
- template: templates/windows-package-signing.yml

# This is done late so that we dont use resources before the big signing and packaging tasks.
- stage: compliance
dependsOn: ['windows']
dependsOn: ['package_signing']
jobs:
- template: templates/compliance.yml

- stage: nuget_and_json
dependsOn: ['windows','linux','macOS']
displayName: NuGet Packaging and Build Json
dependsOn: [package_signing]
jobs:
- template: templates/nuget.yml

- template: templates/json.yml

- stage: test_and_release_artifacts
displayName: Test and Release Artifacts
dependsOn: ['prep']
jobs:
- template: templates/testartifacts.yml

- job: release_json
displayName: Create and Upload release.json
pool:
name: PowerShell1ES
name: $(windowsPool)
demands:
- ImageOverride -equals PSMMS2019-Secure
steps:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
dependsOn:
${{ parameters.parentJobs }}
pool:
name: PowerShell1ES
name: $(windowsPool)
demands:
- ImageOverride -equals PSMMS2019-Secure

Expand Down
2 changes: 1 addition & 1 deletion tools/releaseBuild/azureDevOps/templates/json.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
${{ parameters.parentJobs }}
condition: succeeded()
pool:
name: PowerShell1ES
name: $(windowsPool)
demands:
- ImageOverride -equals PSMMS2019-Secure

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
parameters:
buildName: ''
uploadDisplayName: 'Upload'
parentJob: ''

jobs:
- job: pkg_${{ parameters.buildName }}
Expand All @@ -11,7 +10,6 @@ jobs:
name: PowerShell1ES
demands:
- ImageOverride -equals PSMMSUbuntu20.04-Secure
dependsOn: sign_linux_builds
variables:
- name: runCodesignValidationInjection
value: false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ parameters:
jobs:
- job: MacFileSigningJob_${{ parameters.buildArchitecture }}
displayName: macOS File signing ${{ parameters.buildArchitecture }}
dependsOn: build_macOS_${{ parameters.buildArchitecture } ACE5 }
condition: succeeded()
pool:
name: PowerShell1ES
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ parameters:
jobs:
- job: package_macOS_${{ parameters.buildArchitecture }}
displayName: Package macOS ${{ parameters.buildArchitecture }}
dependsOn: MacFileSigningJob_${{ parameters.buildArchitecture }}
condition: succeeded()
pool:
vmImage: macos-latest
Expand Down
Loading
0