refactor: rewrite pkg-builder with modular crate architecture#37
Open
refactor: rewrite pkg-builder with modular crate architecture#37
Conversation
Replace old workspace/ (types, debian, packager_deb, cli, pkg_builder) with new crates/ (config, tool, runtime, pipeline, cli). Key improvements: - Single config parse via PkgConfig::load() instead of double-parse - Arc<PkgConfig> + Workspace by reference instead of cloning BuildContext - 3 error types instead of 10+ - Distribution-specific logic consolidated into Distribution methods - Programmatic ScriptBuilder replaces include_str! shell templates - 94 tests including integration tests parsing all example TOMLs
- Rename cli binary from pkg-builder-new to pkg-builder to match CI expectations - Disable jammy_amd64_packaging job in CI workflow
Add Trixie variant to DebianCodename enum with codename parsing, example configs for all 12 language templates, and integration tests. Existing Debian wildcard matching in tool/pipeline code handles Trixie automatically.
Extract debcrafter wrapper from crates/tool into a dedicated crates/debcrafter crate. This ports debcrafter's binary generation logic as a library dependency (pinned to commit 8189263) so pkg-builder can generate debian/ directories without requiring a separate debcrafter binary installation. - Remove prebuilt debcrafter binaries from CI setup - Make debcrafter_version optional in config for backward compat - Add DebcrafterError type independent from ToolError
Use debcrafter's own Package::load() instead of the wrapper's toml 0.8 to deserialize .sps files, fixing "expected a borrowed string" parse error caused by toml 0.5 vs 0.8 incompatibility. Remove unused toml dependency from the wrapper crate. Also add trixie (Debian 13) CI job and reduce all distro matrices to only the c language for quick smoke testing.
Debcrafter's process_source internally appends {name}-{version}/ to
the dest dir before creating debian/. Passing build_files_dir (which
is already {name}-{version}) caused debian/ to be nested one level
too deep, leading to "No such file or directory" when the pipeline
tried to find debian/control. Pass build_artifacts_dir (the parent)
instead, matching the old code's behavior.
autopkgtest uses QEMU backend which requires KVM permissions. The image builder already used run_command_sudo but the test runner did not, causing "no permission to write /dev/kvm" on CI.
Changelog files had 'bookworm' instead of 'trixie' as the distribution, causing lintian distribution-and-changes-mismatch warnings when building with sbuild -d trixie. Also updated debian/rules comments to say 'debian' instead of 'bookworm' since the behavior is not release-specific.
Move packaging steps into a reusable workflow (.github/workflows/packaging.yml) called by per-distro matrix jobs in tests.yml. This gives nested UI grouping in GitHub Actions (e.g. bookworm / package (c)) instead of flat job names.
Move the language matrix from caller jobs into the reusable packaging workflow so GitHub Actions groups as distro / package (language) instead of distro (language) / package. Also add concurrency control to cancel in-progress runs on the same PR branch.
Adds compare-debs workflow that builds the bookworm/c hello-world package with both main and current branch, uploading artifacts for manual comparison. Temporarily disables tests workflow on PRs.
The main branch needs debcrafter_8189263 binary in PATH. Also use explicit glob patterns for artifact upload to avoid colon characters in .build log filenames.
The compare-debs workflow served its purpose during the refactor verification. Re-enable the pull_request trigger on the tests workflow and add workflow_dispatch for manual runs.
Add optional snapshot_date and snapshot_security_date fields to [build_env] config. When set, chroot creation and build tools use snapshot.debian.org instead of live mirrors, ensuring reproducible build environments across runs. - Validate date format (YYYYMMDD or YYYYMMDDTHHMMSSZ), reject Ubuntu - Include snapshot date in cache filename for separate chroot caching - Handle expired Valid-Until in snapshot archives (debootstrap + apt) - Wire snapshot repo URLs through sbuild, piuparts, and autopkgtest - Disable trixie packaging tests (temporarily broken)
Replace the hardcoded Rust runtime implementations with a declarative recipe-based Makefile generator. pkg-builder now reads pkg-builder.toml + recipe files and generates a fully expanded Makefile. Key changes: - New `crates/makefile/` crate: recipe parser, variable substitution, REPEAT/END block expansion, INCLUDE resolution, Makefile emitter - 12 pipeline recipes (4 distros × 3 source types) in `pipelines/` - 11 runtime recipes (go, rust, node, java, java-gradle, nim, dotnet-*, c, python) in `runtimes/` - Flatten config format: [package], [source], [build], [runtime], [testing], [tools] sections replacing deeply nested TOML structure - [runtime] uses serde flatten for arbitrary key-value pairs - Delete `crates/runtime/` entirely (replaced by recipe files) - Delete `crates/config/src/language.rs` (replaced by runtime.rs) - Wire `pkg-builder generate` command and `build` to generate+make - Convert all 48 example configs to new flat format - 194 tests passing (45 config + 42 parse_examples + 16 makefile unit + 52 generate_examples + 24 pipeline + 15 tool)
Remove the local debcrafter implementation and route all CLI commands through the generated Makefile, which installs and invokes the upstream debcrafter binary directly.
…cking Replace the script-runner style Makefile with a proper build graph where each pipeline phase produces a tracked output file, so Make skips already-completed steps. Add shell safety flags, immediate-assignment variables, conditional test target variables overridable at invocation time, and fix chroot tarball always rebuilding by using order-only phony prerequisites.
…ix commands Move [verify] section into pkg-builder.toml, removing the separate pkg-builder-verify.toml file. Add top-level clean subcommand, fix lintian overrides for bookworm/c hello-world, and remove redundant host-side lintian target since lintian runs inside sbuild.
Jammy (Ubuntu 22.04) support is being dropped. Python examples are removed as they are not being maintained going forward.
Use shallow clones with --depth=1, pin submodules to exact commits, strip .git metadata, and produce reproducible tarballs with deterministic ownership and timestamps. Also fix trixie hello-world package name.
Enable trixie and expand noble/bookworm CI to all supported languages, add README, build-all script, Apache-2.0 license to Cargo manifests, clean up runtime recipes, and update examples with current hashes and lintian overrides.
Replace flat docs/ files with a proper mdBook structure organized into getting-started, guides, and reference sections. Rewrites the stale config.md (which documented the old schema) and adds new pages for concepts, testing, verification, CLI reference, and recipe reference. Trims README to link to the docs site instead of duplicating content.
… update virtual package hashes
…utput HashMap iteration order is non-deterministic, causing runtime variables like BINARY_URL, BINARY_CHECKSUM, etc. to appear in different order between builds. Switch to BTreeMap for sorted, reproducible output.
…e generation Drop unsupported jammy distribution (no examples existed), unused python runtime recipe, and redundant build_package() in pipeline crate (CLI already handles generate + make). Fix virtual package tarball reproducibility by adding deterministic tar flags. Fix stale test assertions and missing struct fields across test configs.
Replace the preflight error-and-hint approach with an opt-in flag that automatically runs install commands for missing tools. CI uses the flag so builds no longer fail with misleading preflight warnings.
…overrides and hashes Add Standards-Version and Homepage to generated debian/control, simplify lintian overrides for resolved warnings, remove verbose tar extraction flag, and update package verification hashes.
- Rename `package` to `build` with `--with-tests` replacing `--run-*` flags - Move `lintian`, `piuparts`, `autopkgtest` under `test` subcommand - Remove `version` subcommand in favor of clap's built-in `--version`/`-V` - Make `--config` a global option (default: current directory) - Add `init` placeholder command for future setup wizard - Separate `env clean` (chroot removal) from `clean` (build artifacts) - Add `test-lintian` and `env-clean` Makefile targets - Update CI workflows, build-all.sh, and all documentation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…make all Replaces `build --with-tests` (which ran `make all` including piuparts) with separate conditional steps matching the main branch behavior. Both default to disabled since CI runners don't have these tools. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The find command was only removing .git files (-type f) but not .git directories in submodules, leaving git metadata in the source tarball. Also regenerates examples with latest Makefile changes.
…oducible builds The previous tar command included a directory entry whose output varied across tar/gzip versions, causing hash mismatches between local and CI.
Update actions/checkout v4 → v6 and actions/upload-pages-artifact v3 → v4.
…itecture Replace monolithic emitter and recipe modules with a structured pipeline: parser extracts build config, builder produces an IR build plan, and renderer emits the final Makefile. Also add rust to CI test matrix.
The tool crate (sbuild, lintian, piuparts wrappers) and most pipeline modules were dead code since the Makefile handles all build operations. The only live code path was hash verification, now inlined in the CLI.
Add interactive project setup wizard that generates pkg-builder.toml, .sss, and .sps files. All init/generation logic lives in the new pkg-builder-init library crate with enum-driven design (SourceType, Distribution, Runtime, RuntimeSetup) so consumers can use it programmatically. The CLI is a thin adapter that renders prompts based on lib-provided field descriptors. Also extracts verify_hashes into config crate and run_make into makefile crate.
Set up CI-driven release workflow that creates release PRs on merge to main, generates changelogs, and bumps pkg_builder version in examples.
…s to 0.3.1 Update docs to reflect the new init wizard, removal of pipeline/tool crates, and correct cargo install path. Revert crate versions from 0.4.0 to 0.3.1 so release-plz can handle the version bump.
…d CI improvements Replace SHA-1 with SHA-256 for package verification, escape TOML string interpolation, validate recipe names against path traversal, fix shell injection in CI workflows, add HTTP timeouts, guard against infinite variable substitution loops, and clean up dead code.
…files Reject shell metacharacters in package name/version/revision, source URLs, git tags, and submodule path/commit fields at config parse time. Also shell-quote all interpolated values in rendered Makefile commands as defense-in-depth.
Adds `pkg-builder update` for bumping packages to new upstream versions. Handles TOML rewriting (preserving comments), changelog generation, GitHub API version auto-detection, word-boundary-aware file substitution, and interactive patch testing with quilt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete rewrite of pkg-builder from the old
workspace/layout (types, debian, packager_deb, pkg_builder, cli) into a clean modularcrates/architecture with four focused library crates and a thin CLI.New crate architecture
config— singlePkgConfig::load()entry point, typed enums for Distribution/Architecture/Source, snapshot pinning, hash verification (verify_hashes)makefile— declarative Makefile generation via builder → IR → parser → renderer pipeline, recipe file parsing,run_makeexecutioninit— enum-driven project scaffolding library (SourceType,Distribution,Runtime,RuntimeSetup), tarball download/hashing, runtime auto-resolution (Go, Rust, Node, Nim), file generation for pkg-builder.toml/.sss/.spscli— thin adapter over the library crates, interactive prompts via dialoguerKey changes
pkg-builder.toml) replaces separate config + verify filesDistribution,Architecture,SourceType,Runtimewith methods instead of string matchingDistributionmethodspkg-builder initgenerates all project files, with CLI flags for non-interactive use and runtime auto-resolution for Go/Rust/Node/NimTest plan
cargo buildcompiles cleanlycargo test— all 130 tests pass (44 config, 31 parse_examples, 38 generate_examples, 17 makefile)pkg-builder initgenerates valid config files (tested with--source-type virtual --distribution bookworm --runtime none)