8000 chore: [SVLS-6279] fips-flavored builds by apiarian-datadog · Pull Request #556 · DataDog/datadog-lambda-extension · GitHub
[go: up one dir, main page]

Skip to content

chore: [SVLS-6279] fips-flavored builds #556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 35 commits into from
Mar 20, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
4caefd7
chore: [SVLS-6279] fips-flavored builds
apiarian-datadog Feb 14, 2025
f2c87e1
chore: wire up go fips compile
apiarian-datadog Feb 14, 2025
aa060c2
chore: fips layer builds, too
apiarian-datadog Feb 14, 2025
d6e7d1e
chore: [SVLS-6279] itar checking script
apiarian-datadog Feb 19, 2025
1eb76f0
Revert "chore: [SVLS-6279] itar checking script"
apiarian-datadog Feb 20, 2025
b48b66f
chore: [SVLS-6279] make publish_layer(s).sh a bit more reusable
apiarian-datadog Feb 20, 2025
e669ec9
chore: [SVLS-6279] layer package
apiarian-datadog Feb 20, 2025
9313fa9
chore: [SVLS-6279] a new publish_govcloud_layers.sh script
apiarian-datadog Feb 21, 2025
c56a868
chore: [SVLS-6279] fix some typos
apiarian-datadog Feb 21, 2025
eac2416
chore: [SVLS-6279] we expect no goboring for non fips builds
apiarian-datadog Feb 21, 2025
7820a79
chore: [SVLS-6279] make sure we are fipsing correctly
apiarian-datadog Feb 21, 2025
6c6ae73
chore: [SVLS-6279] parametrize and adjust max layer sizes
apiarian-datadog Feb 21, 2025
c5f688c
chore: [SVLS-6279] fix build_private_image.sh
apiarian-datadog Feb 24, 2025
de115ec
chore: [SVLS-6279] rename WORKFLOW_LAYER_SUFFIX to PIPELINE_LAYER_SUFFIX
apiarian-datadog Feb 24, 2025
1471b5b
chore: [SVLS-6279] rename add_permissions to add_layer_version_permis…
apiarian-datadog Feb 24, 2025
b26c7c4
chore: [SVLS-6279] mb layer size for convenience
apiarian-datadog Feb 24, 2025
4fc95a3
Revert "chore: [SVLS-6279] mb layer size for convenience"
apiarian-datadog Feb 24, 2025
3706f68
chore: [SVLS-6279] fix layer size limits
apiarian-datadog Feb 24, 2025
6f93f75
chore: [SVLS-6279] fix typos
apiarian-datadog Feb 28, 2025
645e193
chore: [SVLS-6279] production artifacts expire in a day instead of an…
apiarian-datadog Feb 28, 2025
c6f971d
chore: fix bottlecap compilex
apiarian-datadog Feb 28, 2025
c1304e5
chore: remove role examples
apiarian-datadog Mar 3, 2025
59cb8bc
chore: we do not have a staging environment
apiarian-datadog Mar 3, 2025
354514c
chore: environments make more sense as objects instead of a list
apiarian-datadog Mar 3, 2025
4f640be
chore: simplify layer bundling
apiarian-datadog Mar 3, 2025
121165c
chore: get rid of most of the environment loops
apiarian-datadog Mar 3, 2025
16b0dc4
chore: do cargo fmt, check, and clippy only once
apiarian-datadog Mar 13, 2025
71c1f73
chore: we do not push fips layers from gitlab but we do check their size
apiarian-datadog Mar 13, 2025
4f8da9f
chore: separate layer signing from layer publishing
apiarian-datadog Mar 17, 2025
597c330
chore: fix typo
apiarian-datadog Mar 17, 2025
b4396e5
chore: fix lyaer size limits
apiarian-datadog Mar 17, 2025
85666ec
chore: address review comments
apiarian-datadog Mar 19, 2025
a83f4e8
chore: expand signed artifacts to 1 week
apiarian-datadog Mar 19, 2025
b46e0e8
chore: better VERSION handling for publishing layers
apiarian-datadog Mar 19, 2025
0e4b85d
chore: use the main branch now that datadog-agent#33799 is merged
apiarian-datadog Mar 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ variables:
AGENT_VERSION:
description: "Latest release version of the datadog-agent to tag the build with."
value: "7.61.0"
LAYER_SUFFIX:
PIPELINE_LAYER_SUFFIX:
description: "Suffix to be appended to the layer name (default empty)."
value: ""

Expand Down
8 changes: 6 additions & 2 deletions .gitlab/datasources/environments.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
environments:
- name: sandbox
sandbox:
external_id: sandbox-publish-externalid
role_to_assume: sandbox-layer-deployer
account: 425362996713
- name: prod
add_layer_version_permissions: 0
automatically_bump_version: 1
prod:
external_id: prod-publish-externalid
role_to_assume: dd-serverless-layer-deployer-role
account: 464622532012
add_layer_version_permissions: 1
automatically_bump_version: 0
73 changes: 68 additions & 5 deletions .gitlab/datasources/flavors.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,42 +2,105 @@ flavors:
- name: amd64
arch: amd64
alpine: 0
needs_code_checks: true
fips: 0
needs_layer_sign: true
needs_layer_publish: true
suffix: amd64
layer_name_base_suffix: ""
max_layer_compressed_size_mb: 23
max_layer_uncompressed_size_mb: 54

- name: arm64
arch: arm64
alpine: 0
needs_code_checks: true
fips: 0
needs_layer_sign: true
needs_layer_publish: true
suffix: arm64
layer_name_base_suffix: "-ARM"
max_layer_compressed_size_mb: 21
max_layer_uncompressed_size_mb: 50

- name: amd64, alpine
arch: amd64
alpine: 1
needs_code_checks: false
fips: 0
needs_layer_sign: false
needs_layer_publish: false
suffix: amd64-alpine

- name: arm64, alpine
arch: arm64
alpine: 1
needs_code_checks: false
fips: 0
needs_layer_sign: false
needs_layer_publish: false
suffix: arm64-alpine

- name: amd64, fips
arch: amd64
alpine: 0
fips: 1
needs_layer_sign: true
needs_layer_publish: false
suffix: amd64-fips
max_layer_compressed_size_mb: 24
max_layer_uncompressed_size_mb: 56

- name: arm64, fips
arch: arm64
alpine: 0
fips: 1
needs_layer_sign: true
needs_layer_publish: false
suffix: arm64-fips
max_layer_compressed_size_mb: 21
max_layer_uncompressed_size_mb: 52

- name: amd64, fips, alpine
arch: amd64
alpine: 1
fips: 1
needs_layer_sign: false
needs_layer_publish: false
suffix: amd64-alpine-fips

- name: arm64, fips, alpine
arch: arm64
alpine: 1
fips: 1
needs_layer_sign: false
needs_layer_publish: false
suffix: arm64-alpine-fips

# Unfortunately our mutli-arch images don't fit nicely into the flavors
# structure above.
# structure above. Since we are making multi-arch images, the suffixes here
# omit the "arch" component and only include the alpine/fips bits.
multi_arch_image_flavors:
- name: basic
alpine: 0
fips: 0
platform: linux/amd64,linux/arm64
dependency_names: [amd64, arm64]
suffix: ""

- name: alpine
alpine: 1
fips: 0
platform: linux/amd64,linux/arm64
dependency_names: ["amd64, alpine", "arm64, alpine"]
suffix: "-alpine"

- name: fips
alpine: 0
fips: 1
platform: linux/amd64,linux/arm64
dependency_names: ["amd64, fips", "arm64, fips"]
suffix: "-fips"

- name: fips, alpine
alpine: 1
fips: 1
platform: linux/amd64,linux/arm64
dependency_names: ["amd64, fips, alpine", "arm64, fips, alpine"]
suffix: "-alpine-fips"
28 changes: 9 additions & 19 deletions .gitlab/scripts/build_private_image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,30 +11,19 @@ DOCKER_TARGET_IMAGE="425362996713.dkr.ecr.us-east-1.amazonaws.com/self-monitorin
EXTENSION_DIR=".layers"
IMAGE_TAG="latest"

if [ -z "$ALPINE" ]; then
printf "[ERROR]: ALPINE not specified\n"
exit 1
else
printf "Alpine build requested: ${ALPINE}\n"
fi

printf "Authenticating Docker to ECR...\n"
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 425362996713.dkr.ecr.us-east-1.amazonaws.com

if [ "$ALPINE" = "0" ]; then
printf "Building image\n"
TARGET_IMAGE="Dockerfile.extension_image"
else
printf "Building image for alpine\n"
TARGET_IMAGE="Dockerfile.extension_image.alpine"
fi

# NOTE: this probably does not work the way that we expect it to, especially
# when suffixes are involved. This is a known bug but we don't really check
# anything other than the basic `self-monitoring-lambda-extension:latest` image
# in our self-monitoring, so it's not a thing we're going to fix right now.
LAYER_NAME="Datadog-Extension"
if [ -z "$LAYER_SUFFIX" ]; then
if [ -z "$PIPELINE_LAYER_SUFFIX" ]; then
printf "Building container images tagged without suffix\n"
else
printf "Building container images tagged with suffix: ${LAYER_SUFFIX}\n"
LAYER_NAME="${LAYER_NAME}-${LAYER_SUFFIX}"
printf "Building container images tagged with suffix: ${PIPELINE_LAYER_SUFFIX}\n"
LAYER_NAME="${LAYER_NAME}-${PIPELINE_LAYER_SUFFIX}"
fi

# Increment last version
Expand All @@ -44,7 +33,8 @@ printf "Tagging container image with version: $VERSION and latest\n"

docker buildx build \
--platform $PLATFORM \
-f ./images/${TARGET_IMAGE} \
-f ./images/Dockerfile.extension_image \
--build-arg SUFFIX=$SUFFIX \
--tag "$DOCKER_TARGET_IMAGE:${IMAGE_TAG}${SUFFIX}" \
--tag "$DOCKER_TARGET_IMAGE:${VERSION}${SUFFIX}" \
--push .
Expand Down
14 changes: 12 additions & 2 deletions .gitlab/scripts/check_layer_size.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,16 @@

set -e

if [ -z "$MAX_LAYER_COMPRESSED_SIZE_MB" ]; then
printf "[ERROR]: MAX_LAYER_COMPRESSED_SIZE_MB not specified\n"
exit 1
fi

if [ -z "$MAX_LAYER_UNCOMPRESSED_SIZE_MB" ]; then
printf "[ERROR]: MAX_LAYER_UNCOMPRESSED_SIZE_MB not specified\n"
exit 1
fi

validate_size() {
local max_size=$1
local file_size=$2
Expand All @@ -23,8 +33,8 @@ if [ -z "$LAYER_FILE" ]; then
exit 1
fi

MAX_LAYER_COMPRESSED_SIZE_KB=$(( 23 * 1024)) # 23 MB, amd64 is 22, while arm64 i 8000 s 20
MAX_LAYER_UNCOMPRESSED_SIZE_KB=$(( 54 * 1024 )) # 53 MB, amd is 53, while arm64 is 47
MAX_LAYER_COMPRESSED_SIZE_KB=$(( $MAX_LAYER_COMPRESSED_SIZE_MB * 1024))
MAX_LAYER_UNCOMPRESSED_SIZE_KB=$(( $MAX_LAYER_UNCOMPRESSED_SIZE_MB * 1024 ))

FILE=".layers"/$LAYER_FILE
FILE_SIZE=$(stat --printf="%s" "$FILE")
Expand Down
8 changes: 8 additions & 0 deletions .gitlab/scripts/compile_bottlecap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,13 @@ else
printf "Alpine compile requested: ${ALPINE}\n"
fi

if [ -z "$FIPS" ]; then
printf "[ERROR]: FIPS not specified\n"
exit 1
else
printf "Fips compile requested: ${FIPS}\n"
fi

if [ "$ALPINE" = "0" ]; then
COMPILE_IMAGE=Dockerfile.bottlecap.compile
else
Expand Down Expand Up @@ -58,6 +65,7 @@ docker_build() {
-t datadog/compile-bottlecap \
-f ./images/${file} \
--build-arg PLATFORM=$PLATFORM \
--build-arg FIPS="${FIPS}" \
. -o $BINARY_PATH

# Copy the compiled binary to the target directory with the expected name
Expand Down
19 changes: 18 additions & 1 deletion .gitlab/scripts/compile_go_agent.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@ else
printf "Alpine compile requested: ${ALPINE}\n"
fi

if [ -z "$FIPS" ]; then
printf "[ERROR]: FIPS not specified\n"
exit 1
else
printf "Fips compile requested: ${FIPS}\n"
fi

if [ -z "$CI_COMMIT_TAG" ]; then
# Running on dev
printf "Running on dev environment\n"
Expand All @@ -49,9 +56,18 @@ else
COMPILE_IMAGE=Dockerfile.go_agent.alpine.compile
fi

if [ -z "$SUFFIX" ]; then
printf "No suffix provided, using ${ARCHITECTURE}\n"
SUFFIX=$ARCHITECTURE
fi

# Allow override build tags
if [ -z "$BUILD_TAGS" ]; then
BUILD_TAGS="serverless otlp"
if [ "$FIPS" = "0" ]; then
BUILD_TAGS="serverless otlp"
else
BUILD_TAGS="serverless otlp serverlessfips"
fi
fi

# Allow override agent path
Expand Down Expand Up @@ -89,6 +105,7 @@ function docker_compile {
--build-arg EXTENSION_VERSION="${VERSION}" \
--build-arg AGENT_VERSION="${AGENT_VERSION}" \
--build-arg BUILD_TAGS="${BUILD_TAGS}" \
--build-arg FIPS="${FIPS}" \
. -o $BINARY_PATH

# Copy the compiled binary to the target directory with the expected name
Expand Down
Loading
Loading
0