Skip to content

Release Pipeline Design: Build Once, Test, Promote

Status: approved 2026-11-19. Steps 0 to 4 are implemented (step 2 in two parts, see Versioning; steps 3 and 4 in part, see What shipped for steps 3 and 4); step 5 (dry run) and the build half of step 6 are implemented, see What shipped for steps 5 and 6; step 7 (the hotfix path) is implemented but switched off, see What shipped for step 7; step 8 (the live release) is built, rehearsed and switched off, see What shipped for step 8; the rest is not. The rollout is a list of small steps, each independently mergeable and reversible (see Rollout plan).

Summary

Today development and master each run their own maven-build, so production runs an image that was never the one tested on development. Nothing waits for the tests, nothing checks that a rollout worked, and nothing rolls back.

The proposal, in one paragraph: build once on development, test that exact build, mark it verified, and release by a manual promote-to-prod job that adds a production tag to the same image digest (no rebuild, no copy, because all environments share one registry repository per image). The job edits the production kustomization, checks the rollout, and only then fast-forwards master to the released commit and creates the release tag, so master is always exactly what production runs. master is built only for a hotfix, and then it builds and tests before deploying. Every deploy to either environment is followed by a verification job (a smoke test of the public hostnames, which also proves the new build is the one answering) and Flux notifications cover what HTTP cannot see. Automatic revert is built, reports only at first, and is switched on by a CI variable once two releases have verified cleanly. The release label, release tag and release notes are all generated by the promote job, and the version is fixed before the first promoted release.

Where we are (verified 2026-11-19)

Area Today
Branches development (staged work, deploys to dev) and master (what prod runs). Releases are an MR "development into master", which creates a merge commit; a master-only commit chore: bump version to X [skip ci] is added after the build. master allows force-push (protected branch setting)
Build maven-build runs on both branches: Maven package, then five images (web, rest, worker, auth, docs) built and pushed in a Docker-in-Docker sidecar. Tag is <project.version>-<branch slug>, for example 0.101.0-master or 0.102.0-dev.6916b050-development
Tests unit-tests and integration-tests run on development and feature branches only, never on master. They skip on --skip-test in the commit message, and on changes: rules (a docs-only push). They are interruptible, so a newer commit cancels an older pipeline's tests
Deploy update-infra-repo-dev and -prod need only maven-build. Dev therefore deploys before the tests finish and whatever their result. Deploy is a sed over every newTag: line in the environment's kustomization in the infrastructure repository (k3s-stack), pushed to main
Rollout Flux Kustomizations have wait: true; a 10 minute timeout on prod. No remediation, no rollback, no Flux alert or notification objects anywhere in k3s-stack. CI finishes green when the tag is pushed, not when the rollout finishes
Registry One repository per image. Cleanup policy: daily, keep 10 newest, delete tags older than 90 days matching ^\d+\.\d+\.\d+-dev\.[0-9a-f]+-development$, never delete tags matching ^\d+\.\d+\.\d+-master$
Supply chain Trivy container scan of the web image only. Images are built with --provenance=false. No SBOM, signatures, secret detection or dependency update automation in the pipeline
Version Computed from commit messages by dev-scripts/nextVersion.sh, written into the poms by mvn versions:set in every build, and also embedded in the jar (see Versioning)

Corrections to the brief this design was started from

  1. healthChecks omitting auth and docs is moot. With wait: true, Flux waits on every workload the Kustomization applies and ignores spec.healthChecks. So auth and docs are already waited for. (Confirm on dev in step 4 by breaking the docs image tag and watching the Kustomization stay not-ready; the missing-tag case in finding 5 is therefore caught by Flux, but only after the fact, and nothing tells anyone.) The real gap is that nothing reports the result.
  2. A promotion is a tag, not a copy. Dev and prod tags live in the same repository per image, so promoting is adding a second tag to the same manifest digest (crane tag). The "does buildx imagetools create rewrap the manifest" concern disappears: nothing is rewritten, and signatures and attestations (stored per digest) stay attached.
  3. master allows force-push. If master is going to mean "what production runs", that should be switched off.
  4. The nextVersion.sh bug is slightly different from "the footer can never match". The script loops for commit in $(git log --pretty=%s), which word-splits, so it tests individual words rather than subject lines, and it only reads subjects, never bodies or footers. BREAKING CHANGE: (two words) can never match, feat!: is not recognised, and a subject such as fix: handle feat: in titles is misread. Fix and test it before relying on the version for anything (step 0).
  5. Release notes list four images (docs missing) and claim linux/amd64, linux/arm64 while DOCKER_PLATFORMS defaults to arm64 only. Both are wrong today.
  6. AOT cache: the dev builds train it again (the earlier master-only change is reverted), so a promoted image carries it. The cache validates jar path, size, mtime, JDK and flags, not the image tag, so adding tags does not invalidate it. The .gitlab-ci.yml read for this document predates that revert; re-check the AOT_TRAINING line when step 1 touches the build job. Dev builds train it, so the promoted images carry it.

Branching and release model

Keep two long-lived branches, but change what master means. development is the trunk: all work merges there through short-lived branches and it is always the thing being built, tested and deployed to dev. master becomes the record of what production runs, and normally contains no commit that is not on development: a release moves master forward to a verified development commit (fast-forward, by the promote-to-prod job). The two branches then differ only by "development is ahead".

Why not collapse to one branch (pure trunk-based development)?

  • Pure trunk-based practice (DORA's capability, trunk-based.dev) deploys every merge to production behind feature flags, with progressive delivery and automated rollback on error rates. We have multi-tenant shops with real money flowing, no traffic-splitting layer, one replica of rest, worker and auth, a Cloudflare tunnel in front, and test coverage that is good but not a substitute for a human choosing when to ship. A human-chosen release moment is a feature here: continuous delivery, not continuous deployment.
  • A branch is a cheap, visible, protected pointer to "what prod runs", it gives hotfixes a natural base, and it keeps today's habits. The cost of two branches (drift, back-merges) is removed by the fast-forward rule and enforced by a check, rather than relying on discipline.
  • A single branch plus release tags and the same manual promote job is equally valid and is a small step from here: the job would simply stop moving master. It needs no migration beyond this one.

Where this deviates from "pure" best practice, deliberately:

Best practice Here Because
Deploy every merge to prod A human triggers each production release no progressive delivery or per-request routing, multi-tenant blast radius
Environment promotion by a PR to a deployment repo, approved per environment The promote-to-prod job edits the k3s-stack kustomization directly (as now) a two-person team; keep one automated path, add verification and a revert that starts report-only
Admission-time signature verification (Kyverno, policy controller) Verify at promotion time in CI one more cluster controller to run and upgrade; the same guarantee at this scale as long as only CI can write the deploy repo
Canary or blue/green (Flagger, Argo Rollouts) Rolling update, health gate, automatic revert no mesh or traffic split; web already keeps an old pod up during a rollout (maxUnavailable: 1 with two replicas)

The release trigger: a manual promote-to-prod job

A release is the manual job promote-to-prod, played on the development pipeline of a verified commit C. It:

  1. checks the preconditions (below) and resolves the five digests behind git-C;
  2. adds the release tag (X.Y.Z, computed by the fixed nextVersion.sh from the commits since the last release tag that is reachable from C) to each image digest with crane tag;
  3. edits the production kustomization in k3s-stack (digests, and BADGER_RELEASE_VERSION) and pushes it;
  4. runs verify-deploy against production (smoke test and build identity, below);
  5. on success: fast-forwards master to C, pushes the git tag X.Y.Z on C, and creates the GitLab release with notes generated from git log <previous release tag>..C and the digests actually promoted. On failure master and the tag do not move, and the revert job reports (or reverts, when enabled).

Moving master after verification makes it literally what production runs: a failed rollout never leaves it pointing at something that is not live. The push is made by the CI bot with -o ci.skip, and master is protected so that only the CI bot (and, for hotfixes, maintainers via merge request) can write to it, with force-push off.

The rejected alternative is Option A: releasing by a merge request from development into master and having the master pipeline decide to promote or build. It keeps a familiar review step but needs a rule to guess from a merge commit whether it matches something already built, which is where the "promoted an image that lacks a master-only hotfix" failure comes from. Option B has no such guess: the thing released is named by its commit.

Hotfix flow

When staged work on development must not go to prod yet:

  1. Branch from master (hotfix/...), fix, push. The branch pipeline runs unit and integration tests.
  2. Merge request into master, fast-forward only where possible, so master's commit is the one that was tested on the branch. The merge itself is the only human write to master.
  3. The master pipeline for that commit sees content that is not on development, so it builds the images and runs the full tests before it deploys, then deploys, verifies (with the same smoke test) and creates a patch release tag on the commit. The images carry git-<sha> and the OCI labels like any build.
  4. Emergency override: a manual deploy-hotfix-without-tests job exists in that pipeline, named for what it does. It deploys the built images without waiting for the tests, records tests: skipped (override by <user>) in the release notes and the k3s-stack commit message, and the smoke test, Flux alerts and (when enabled) the revert still run.
  5. A job then opens an MR master into development (the back-merge), and merges it automatically when it is clean, otherwise labels it for a human. It carries [skip ci] only when development has no other changes to build; normally development builds it like any other change.

How the pieces work with a machine-moved master:

  • Normal releases never put a commit on master that is not on development, and the bot only fast-forwards it, so master stays an ancestor of development.
  • A hotfix is the one commit that breaks that. Until the back-merge lands, master is not an ancestor of development. That is detected by the master-contained check: git merge-base --is-ancestor origin/master <development tip>. It runs on every development pipeline (red, with a message naming the missing commits) and is a precondition of promote-to-prod. So promote-to-prod refuses until the back-merge is done; that, plus the precondition below that master must be an ancestor of C so that fast-forward is possible, make the back-merge impossible to forget.
  • After the back-merge, development contains the hotfix and builds it; the next promotion carries it to production in a verified development build, as a fast-forward again.
  • A human merging staged development content straight into master is caught: the master pipeline sees a commit that is an ancestor of development and fails with "release with promote-to-prod".

Build-once artifact model

Tags, labels, digests

Per image, per build on development:

Tag Meaning Mutable? Retention
git-<full 40-char sha> Identity of the build. The only tag the promotion logic looks up. On a master build (a hotfix) the tag is git-<sha>-master, so the cleanup policy's keep pattern for -master tags protects it not moved by design (a retry of maven-build for the same commit rebuilds and moves it) follows its -dev. tag: deleted by the cleanup policy together with it
<version>-dev.<sha8>-development Today's human tag, kept for continuity and the dev kustomization no existing policy
verified-<full sha> Written by the last job of a successful dev pipeline: built, tested, deployed to dev and smoke tested written once same as git-
<X.Y.Z> (a release) Added at promotion; names the release written once kept forever (rollback targets)

Every image carries org.opencontainers.image.revision=<sha>, .source, .created, and org.opencontainers.image.version where known, so docker inspect answers "what is this". The git-<sha> tag is the lookup key because a rerun of a pipeline for the same commit gets a new <version>-dev.<sha8> tag only if the version moved, and the old scheme cannot be searched by SHA.

maven-build records the five digests in an artifact (images.env: WEB_IMAGE=, WEB_DIGEST=sha256:..., WEB_TAGS= and the same for REST, WORKER, AUTH, DOCS, plus IMAGES_COMMIT_SHA, IMAGES_PIPELINE_ID, IMAGES_VERSION, IMAGES_CREATED and IMAGES_PLATFORMS). The job takes each digest from BuildKit's --metadata-file and fails unless docker buildx imagetools inspect reports the same digest for both pushed tags. Both tags are pushed by the same build, so they are one manifest. Labels are set with --label on the final image (revision, created, version; source, vendor, title and description stay in the Dockerfiles, and the docs image takes source from the build command because its Dockerfile has no labels).

Metadata and attestations: a proportionate subset

Item Verdict Where
OCI labels (revision, source) Do now step 1
Immutable git-<sha> tag, deploy by digest Do now steps 1, 4, 8
SBOM (BuildKit --sbom=true, or syft) attached to the digest Do soon; makes "are we affected by CVE X" a query not an investigation step 10
Provenance (--provenance=mode=max) Do with the SBOM. It was switched off; confirm the GitLab registry and a single-platform index are happy with the attestation manifests before turning it on, and that retention does not orphan them step 10
cosign signature, keyless via GitLab's OIDC token Do soon; signed once at build, verified in the promote job against the identity of this project's pipeline on development or master step 10
Kyverno or policy-controller admission verification Defer; revisit if more people or systems can write to k3s-stack later
SLSA level claims, reproducible builds Not worth it at this scale n/a

Promotion mechanics

Tool

crane (go-containerregistry), pinned by version and checksum in the promote job, in a small image with git and curl. The job needs no Docker-in-Docker and none of the 7Gi pod: crane digest to resolve, crane tag repo@sha256:... <newtag> to promote, crane manifest to verify. regctl is equivalent; crane is chosen for crane tag being a single call and for being in the container ecosystem most often cited for this. Do not use buildx imagetools create (it can rewrap manifests); verify after tagging that crane digest repo:<newtag> equals the digest recorded at build, and fail otherwise.

Preconditions and failure modes (promote-to-prod for commit C)

The job is a script (build-scripts/promote-to-prod.sh) unit-tested with fixture repositories. It names the commit it releases, so there is nothing to infer from merge parents. It refuses unless all of these hold, and the failure is loud and names the remedy. The docs image is one of the five and follows the same rules.

# Check If it fails
1 C is on development, and the pipeline for C ran on development refuse
2 All five images have git-C and verified-C (written by the dev pipeline only when build, tests and the dev smoke test all passed) refuse: "C is not verified; fix development or wait for its pipeline"
3 The five digests all come from the same pipeline (recorded in images.env), not a mixed set refuse
4 master is an ancestor of C (master-contained, and so a fast-forward is possible) refuse: "master has commits not on development; finish the hotfix back-merge"
5 The production kustomization's current digests are readable and saved, for revert refuse
6 The registry, GitLab API or k3s-stack is unreachable refuse; there is no fallback to building
7 C has all five images but a registry cleanup removed one refuse and name the rebuild path: rerun the build for C (manual build-sha pipeline on development)

The master pipeline then has only two outcomes: a commit that is not on development is a hotfix (build, test, deploy, as above), and a commit that is on development fails with "release with promote-to-prod". Nothing builds silently, and a commit with master-only content can never be mistaken for a development-tested one because only promote-to-prod writes release tags and it only accepts development commits.

PROMOTE_DRY_RUN=true (the default until the switch step; the dry-run job also passes --dry-run) performs checks 1 to 7 and prints what it would have done, touching nothing.

Deploy mechanics (k3s-stack)

  • Both environments' kustomizations switch from newTag: "<tag>" to newTag plus digest (digest: sha256:...) per image, with the human tag kept in a comment. A digest is immutable, so "what is running" is unambiguous even if a tag is later moved or deleted. Check on dev that kustomize renders repo@sha256:... as expected before using it on prod.
  • The deploy script's blanket sed over every newTag: line becomes a per-image edit that fails if any of the five images is missing, so a missing badger-docs tag can no longer reach the cluster.
  • The deploy job saves the previous values it replaced (artifact previous-images.env) for rollback.
  • One serialised queue (resource_group: k3s-stack-main) stays, as the repository is shared.

Verification gate and rollback

A verify-deploy job runs after every environment deploy (dev and prod). It uses HTTP only: no cluster credential is given to CI, at least for now.

  1. Smoke test over HTTP, against the public hostnames, run from CI:
  2. the storefront home page, one collection, one product page and the basket respond 200 with expected markers;
  3. the REST API and the auth server's /.well-known / JWKS respond; the docs site responds;
  4. build identity: each of web, rest, auth and docs exposes the commit SHA it was built from (a response header on the app services and a build.txt file on the docs image). The test loops until the SHA equals the expected one on enough requests to cover both web replicas, which is what proves this deploy is what answers, not the old pods still serving during the rollout.
  5. Flux status is covered by alerts, not by CI. Alertmanager rules in k3s-stack (10-alerting-rules.yaml: FluxKustomizationNotReady after 5 minutes, from kube-state-metrics' gotk_resource_info, with FluxMetricsMissing after 15 minutes as its dead-man check, and DeploymentNotFullyAvailable) announce a stuck or failed reconcile, whether or not a pipeline is watching. The Flux Provider and Alert idea was dropped. Alerts are in place from the first promoted release.
  6. Result. On success on dev, mark-verified writes verified-<sha>. On failure the pipeline is red, the alert has fired, and the revert job runs.

The honest gap. The worker has no HTTP endpoint, so a worker that fails to start (or crash-loops after the HTTP services are healthy) is not seen by the smoke test; it is caught only by the Flux alert (a rollout that never becomes ready) and by whatever runtime alerting exists. Two ways to close it later, neither decided: an in-cluster read-only ServiceAccount token for the runner so CI can wait on the Flux Kustomization's Ready condition and its applied revision (the preferred later addition), or the worker writing its build SHA to Redis where the REST service's build-identity response reports it.

Rollback strategy: pipeline revert in k3s-stack, not progressive delivery. Kubernetes' own rolling update already keeps an old pod serving when the new one never becomes ready (web keeps one of two), so the typical bad release is a stalled rollout rather than an outage. The revert job restores the previous digests saved by the deploy job, pushes to k3s-stack, and runs the smoke test again; the previous release's images are retained forever. Flux Kustomizations have no built-in remediation (that is a HelmRelease feature), and Flagger or Argo Rollouts need a traffic-splitting layer we do not run; reconsider if we add one or the replica counts grow. A guard: forward-compatibility of data migrations (expand then contract; the system-update JSON files are applied at start-up) is the real limit on rollback.

Revert is report-only first. It always computes what it would revert and says so in the job log and the notification. It acts only when the CI variable AUTO_REVERT_PROD (and, separately, AUTO_REVERT_DEV) is true. Dev runs report-only for a week; prod stays report-only until two promoted releases have verified cleanly, then the variable is flipped. Verification and Flux alerts are live from release one.

Test gating

The aim is one unambiguous statement: commit C is verified if and only if, on C's tree, build, unit tests, integration tests and the dev smoke test all passed.

  • Deploy to dev stays fast (deploys right after the build, as today) but is annotated, not gated: the dev environment is where we find out, and holding it back by 7 minutes costs more than it saves (decision 2). What changes is that promotion is gated on verified-<sha>, and the dev deploy's k3s-stack commit message says the commit is not yet verified.
  • On development, the tests always run. Remove the changes: skip there (keep it on feature branches) and let --skip-test mean "deploy to dev, do not mark verified". A docs-only push after a code push then costs 7 minutes of test compute but never leaves a SHA ambiguous. Optimisation later: reuse a previous result when the git rev-parse <sha>:<path> tree hashes of the Maven build inputs (the .java-build-paths list) are identical to an already-verified commit, writing verified-<sha> without rerunning.
  • Auto-cancel can drop an older commit's tests; that is fine, because that commit simply never becomes verified and release picks a newer one. The master-contained and verified checks mean an abandoned SHA cannot be promoted by accident. The mark-verified job is interruptible: false, and needs the tests with optional: false, so a skipped test job (rules) blocks it rather than counting as a pass. A tiny script assert-job-results.sh reads the pipeline's jobs via the API and requires named jobs to be success.
  • master for a hotfix runs the tests before deploy (needs the tests on that path).
  • Compile once (later, step 11): each of build, unit and integration compiles the tree separately. Options in order of effort: share target/ classes through a job artifact; add Maven's build-cache extension backed by the existing S3 cache; or fold the unit run into the build job. This is a latency optimisation (the three run in parallel, so the wall-clock gain is limited by the integration job queueing for a burst node, 7+ minutes) and is deliberately not on the critical path of this design.

Versioning and release identity

Today the version number has three jobs mixed together: build identity (jar manifest, build-info, asset cache-busting), display (admin version string), and release name (the tag). The jar embeds @project.version@ in messages.properties (admin.badger.system.version), AssetUrlDialect (?v=), MenuGeneratorExtension (the menu.js key) and the build-info actuator, so a promoted image would report its development version. All of this is fixed before the first promoted release (step 2 of the rollout), not after:

  1. Build identity is the commit SHA, exposed as one build property. ?v= and the menu.js key use it: the bytes are identical in dev and prod, so identical cache-busting is correct, and an asset URL changes exactly when the build does. The same value is the build-identity response header used by the smoke test.
  2. Display is a release label from the environment variable BADGER_RELEASE_VERSION, set per environment in the kustomization (the file the deploy job already edits) and falling back to the build identity when unset (dev shows dev <sha8>; prod shows 0.103.0).
  3. Release name is a git tag on the promoted commit, created by promote-to-prod after a verified rollout, version computed by the fixed nextVersion.sh.
  4. The chore: bump version [skip ci] commit is dropped, with it the master-only content. Every build already rewrites the poms with versions:set, so the committed pom version is redundant. The checked-in poms keep a constant placeholder (0.0.0-SNAPSHOT). Before removing it, check that nothing outside CI relies on the committed version: devEnv.sh, the dev scripts, the IDE import, the Docker build contexts. The old finalise_version_update job then tags the released commit without committing (a small change to the existing release path, until the switch step retires it).
  5. Release notes are generated by promote-to-prod from git log <previous release tag>..C, for all five images, with the digests actually promoted, and the platforms read from the image manifest rather than hard-coded.
  6. nextVersion.sh: rewrite to read subject and body per commit (git log --format=%B with a delimiter), recognise type!: and BREAKING CHANGE: footers, honour feat:/feat(scope):, and cover it with a small test script. Pre-1.0 stays "breaking bumps the major" per the existing convention. It finds the last release tag with git tag --merged <commit> --sort=-v:refname, so hotfix tags on master count once back-merged.

The development image's own human tag stays <version>-dev.<sha8>-development for the dev kustomization; it is not a release name.

Security and supply-chain steps, in priority order

  1. Scan every image, not only web (Trivy over all five digests, in the verify stage of the dev pipeline). Report on all severities; fail mark-verified on fixable CRITICAL (then HIGH once the backlog is clear), with a reviewed allowlist file for accepted findings.
  2. Secret detection (GitLab's Jobs/Secret-Detection template, push protection if the plan allows) on every branch pipeline.
  3. Dependency updates: Renovate (or GitLab's dependency scanning plus a scheduled update MR) for Maven, npm, the Docker base images and the pinned CI images. Today pins are manual.
  4. Disable force-push on master, and restrict who and what can push master and k3s-stack main (the CI bot only, for deploys).
  5. SBOM, provenance and cosign signature as in the table above; verify signature and identity in the promote job.
  6. SAST (GitLab's template) as a non-blocking report first.
  7. Later: admission-time verification, if more than CI can reach the deploy repository.

Rollout plan

Each step is its own MR, ships to development first, and lists how to verify before anything touches master. Bold marks steps that touch the production release path.

# Step Depends on Risk Verify on development Monitor
0 No-regret fixes: rewrite and test nextVersion.sh; turn off force-push on master; correct the release-notes image list and platform text; extend the registry cleanup policy so git-* and verified-* tags follow their -dev. tag and release tags are kept none Low. Release notes and version arithmetic only Script test; compare its output with the existing tags over history Next release's version and notes
1 Immutable identity: add git-<sha> tag and OCI labels to all five images, write images.env with digests as an artifact. No behaviour change; old tags stay 0 Low crane ls / crane config on a dev build; digest in artifact equals registry digest Registry size (extra tags are free: same digest)
2 Versioning, all of it: SHA as build property (?v=, menu.js key, build-info, build-identity header and docs build.txt); BADGER_RELEASE_VERSION in both kustomizations and the admin label; constant placeholder pom version and no bump commit (after checking devEnv.sh and IDE reliance); the old finalise_version_update tags without committing; release-notes generator (tested, not yet called by a promote) 1 Medium: touches AssetUrlDialect, the admin version string and the existing release job Render tests; asset URLs and menu.js key behave the same on dev; admin shows the label; run the tag-only finalise on a scratch branch Admin footer, asset cache hit rate, the next legacy release's tag
3 Test gating on development: tests always run there, --skip-test and cancellation semantics as above, assert-job-results.sh, mark-verified writes verified-<sha> 1 Low: new jobs only. Dev pipelines get longer by the cost of the previously skipped test runs Push a code change, a docs-only change, a --skip-test change, and cancel one mid-run; verified- tags appear only when expected Runner queue time, pipeline duration
4 Dev verification: smoke test and build-identity check as a verify-deploy job; Flux alerts for dev (built as Alertmanager rules instead of a Flux Provider and Alert); dev deploy by digest; the revert job in report-only mode; confirm Flux waits on the docs workload; mark-verified also requires a green smoke test 2, 3 Medium for dev (a flaky smoke test can block verification). Dev is the proving ground Deliberately break a dev deploy (bad tag, failing readiness) and watch the report and the alert; a week of report-only Smoke-test flake rate, report-only "would revert" count
5 promote-to-prod in dry run: the manual job on the development pipeline, checks 1 to 7 and the notes generator, nothing written 3, 4 None on prod (report only) Run it on several real verified commits; unit-test the checks against fixture repositories (unverified, mixed set, master not an ancestor, missing image) Mismatch between its verdict and what is true
6 Prod verification and Flux alerts on the existing prod deploy: verify-deploy against production hostnames and Flux alerts for prod (built as Alertmanager rules, see below), with AUTO_REVERT_PROD absent (report only). Lands while the old master build still deploys 2, 4 Medium: the first prod smoke test; a failure only reports Run the same job against dev, then prod after a legacy release Prod verify results and alerts
7 Hotfix path and master protection: master pipeline for a commit not on development builds, tests before deploy, deploys, verifies and tags; the deploy-hotfix-without-tests override; the master-contained check on every development pipeline; the auto back-merge MR; master pipeline fails for staged content; restrict master pushes to the CI bot and maintainers via MR 2, 6 Medium: first change to the master pipeline's jobs Rehearse with a trivial hotfix branch off master in a scratch project or with prod deploy disabled; check the back-merge MR and the master-contained red/green Back-merge MRs merge; check goes green
8 The switch (built, inert; see What shipped for step 8 and the runbook): promote-to-prod goes live (PROMOTE_DRY_RUN=false): it tags by digest, deploys by digest, verifies, then fast-forwards master and tags and creates the release. The old master maven-build, update-infra-repo-prod and finalise_version_update for staged content are retired by RELEASE_PIPELINE=promote (the inverse of the proposed LEGACY_MASTER_RELEASE: unset restores the old path for two releases, then it is removed) 5, 6, 7 High (prod release path). Mitigated by the dry runs and the escape hatch A full rehearsal against dev (point the job at the dev kustomization with a variable); the first live release is watched end to end Reported prod digest equals dev's; release duration; master equals the released commit
9 Switch on AUTO_REVERT_PROD=true after two clean promoted releases (and AUTO_REVERT_DEV after the dev week) 8 Medium: an overzealous revert is a prod event Simulate a failure on dev first Revert count, flake rate
10 Supply chain: scan all images, secret detection, Renovate, SBOM and provenance, cosign sign at build and verify at promotion. Scan gating starts report-only 3 (scan gate), 8 (verify in promote) Low to medium Dev pipelines show each artefact; verification fails on an unsigned test image Scan time, finding counts
11 Optional: compile-once, reuse of a verified result by tree hash, the in-cluster ServiceAccount for Flux readiness checks (closes the worker gap), Kyverno 8 Varies Decide after step 9 n/a

Steps 0 to 5 do not change what production runs, except step 2, which changes the version string and the existing release job's tag step. Step 8 is the first that changes how production is released, so it is deliberately late and reversible by a single variable.

What shipped for steps 3 and 4

On development only; nothing on master or in the production kustomization changed.

  • Tests always run on development. unit-tests and integration-tests lost their changes: rule there; feature branches keep it. --skip-test still skips them and the commit is never verified. Each job still compiles the tree itself: compile-once stays step 11, so the build is not reused by the tests.
  • Dev deploy is annotated. update-infra-repo-dev still needs only maven-build and deploys at once; its k3s-stack commit message now ends (not yet verified: commit <sha8>, pipeline <id>). The production deploy job's message is unchanged.
  • verify-deploy-dev (stage verify, build-scripts/verify-deploy.sh) runs after the dev deploy is pushed. It waits until web, rest and auth each answer X-Badger-Build: <full sha> six requests in a row (up to 15 minutes), then checks the web home page and robots.txt, the REST health endpoint, the auth server's metadata and JWKS. Not done: collection, product and basket pages (the default shop has no stable seoNames to assert on; /basket is a 404 there).
  • Docs build identity is skipped unless a service token is configured. docs.bdgr-demo.co.uk is behind Cloudflare Access, so an anonymous request gets a login redirect. The script reports that as SKIP. Set the CI variables CF_ACCESS_CLIENT_ID and CF_ACCESS_CLIENT_SECRET (an Access service token allowed on that host) and it checks /build.txt, failing if it is wrong. Until then a stale docs pod is not caught.
  • mark-verified (build-scripts/mark-verified.sh, with assert-job-results.sh and a pinned crane, install-crane.sh) first requires unit-tests, integration-tests and verify-deploy-dev to be success in this pipeline through the API (a job that did not run is not a pass), then adds verified-<full sha> to the digests recorded in images.env with crane tag and reads it back. If the registry's git-<sha> tag no longer points at the tested digest it refuses. A rerun of the whole pipeline for a commit moves an existing verified- tag to the new build, with a warning in the log.
  • Not done in step 4: Flux alerts (later built as Alertmanager rules), dev deploy by digest, the report-only revert job and the break-it-deliberately check of the docs workload. These need changes in k3s-stack or a deliberate bad deploy to dev and remain for a follow-up.

What shipped for steps 5 and 6

On development only. Nothing on master, in the production kustomization or on the production cluster changed.

  • promote-to-prod is a manual job, dry run only (build-scripts/promote-to-prod.sh, stage finalise_version, allow_failure, so an unplayed job never blocks a pipeline). Play it on a development pipeline; set PROMOTE_SHA when playing to name another commit than the pipeline's. It runs checks 1 to 7 (commit on development; all five git-<sha> and verified-<sha> tags present and the same digest; one build, by the revision, created and version labels; master an ancestor; the production kustomization readable; a missing image names the rebuild path), then prints what the live job would do: the release version nextVersion.sh gives that commit, one crane tag <image>@<digest> <version> per image, a diff of the production kustomization (new tag plus digest per image, made on a throwaway copy), the BADGER_RELEASE_VERSION value, the verify-deploy call, the master fast-forward, the tag and the generated release notes. It exits 1 when the live job would refuse (the plan is still printed, marked not run).
  • It cannot change anything. The script contains no write command (only echo "would run: ..."), DRY_RUN=false is refused, and promote-to-prod.test.sh fails if crane is ever called with a non-read subcommand or the script gains an executable crane tag, git push or git tag. The job blanks GITLAB_API_TOKEN and DEPLOY_SSH_KEY (project variables reach every job); its only credential is the job token, which reads this project's registry and git and, because the project is on k3s-stack's job token allowlist, clones k3s-stack for check 5.
  • While master carries legacy bump commits, check 4 refuses. That is true today (master has commits development lacks) and is the check doing its job; it clears once a release is made by the new path and master only ever fast-forwards.
  • Production verification reuses build-scripts/verify-deploy.sh, now with EXPECTED_RELEASE (web, rest and auth must also answer X-Badger-Release: <name>; the header is new, set by BuildIdentityFilter from BADGER_RELEASE_VERSION) and a legacy mode with no EXPECTED_SHA for a production still running builds from before the identity header: the smoke test runs and build identity is printed as NOT CHECKED, never as a pass. Docs are public in prod, so DOCS_REQUIRED is on there and a missing or stale /build.txt fails. verify-deploy.test.sh tests it against a local fake of the four services.
  • Jobs. verify-deploy-prod (on master, after update-infra-repo-prod, allow_failure, report only) waits for production to answer with the commit just deployed; check-prod is a manual job on any development pipeline that runs the smoke test against production as it is now.
  • Flux alerts are live, as Alertmanager rules (not a Flux Provider and Slack secret, which were dropped): DeploymentNotFullyAvailable, FluxKustomizationNotReady and FluxMetricsMissing in k3s-stack 10-alerting-rules.yaml, alert only, nothing reverts. The empty BADGER_RELEASE_VERSION env entry that promote-to-prod needs is also already in the production deployment patches. Nothing is left to create.

What shipped for step 7

Built, tested with fixtures, and inert: nothing here changes what master does or what runs in production until a project CI variable is set.

  • The switch is RELEASE_PIPELINE, a project-level CI variable (Settings, CI/CD, Variables), unset today. Rollout step 8 flips it by setting RELEASE_PIPELINE=promote (no commit needed) at the same time as promote-to-prod goes live. Unset (or any other value), the master pipeline is the legacy one: the pipeline's job list for master is the same as before, the legacy update-infra-repo-prod, verify-deploy-prod, finalise_version_update and create_gitlab_release run, and none of the new master jobs exist. Setting it back (deleting it) is the escape hatch; step 8's LEGACY_MASTER_RELEASE can be this same variable inverted.
  • master-contained (build-scripts/master-contained.sh) runs on every development pipeline: git merge-base --is-ancestor origin/master <commit>, red with the commits master has and development lacks. promote-to-prod check 4 now calls the same script, so they cannot disagree (master-contained.test.sh asserts it). While RELEASE_PIPELINE is not promote the job is allow_failure (a warning): the legacy flow merges development into master with a merge commit development lacks.
  • Classifier (classify-master job, classify-master-commit.sh), first job of a master pipeline in promote mode: a commit not on development is a hotfix and maven-build proceeds (it needs the classifier, optionally, so the edge does not exist in legacy mode); a commit on development fails with "release with promote-to-prod" and nothing builds.
  • Hotfix chain (promote mode, master only): maven-build (images tagged git-<sha>-master as before) in parallel with unit-tests and integration-tests (their master rule is now on in promote mode) -> update-infra-repo-hotfix (needs all three, none optional: a test job that did not run fails it) -> verify-hotfix (the smoke test as a gate) -> finalise-hotfix (hotfix-release.sh: patch tag from nextVersion.sh, GitLab release with tests: passed, then the back-merge). The k3s-stack commit message carries hotfix, tests: passed.
  • Override: manual deploy-hotfix-without-tests -> verify-hotfix-override -> finalise-hotfix-override, with tests: skipped (override by <user>) in the k3s-stack commit message and the release notes (RELEASE_TESTS_NOTE in releaseNotes.sh). If the tests also pass later, the normal deploy finds production already at that tag and pushes nothing.
  • Back-merge (back-merge.sh, the last step of hotfix-release.sh): opens (or reuses) an MR master -> development, polls GitLab's detailed_merge_status, merges at once when mergeable, otherwise labels it back-merge-needs-human, exits 42, which the job maps to "passed with warnings" (allow_failure: exit_codes: 42). It merges rather than waiting for a pipeline because merge-request pipelines are off in this project and the hotfix was just tested. Needs GITLAB_API_TOKEN to be allowed to merge into development.
  • Not done in step 7: restricting pushes to master (a GitLab protected-branch setting, to be made with the switch), and "the master pipeline fails for staged content" is the classifier above.
  • Tested: master-contained.test.sh, back-merge.test.sh (back-merge against a fake GitLab API, hotfix-release.sh against a fixture repo and a bare remote), releaseNotes.test.sh. The CI wiring is checked by GitLab's linter and by the development pipeline; the promote-mode master jobs cannot run until the variable is set, so their first real run is a hotfix after the switch.

What shipped for step 8

The live release exists and is rehearsed. It is inert: nothing changes until the user sets the project CI variables below. Full order of steps, abort and revert: the Release Runbook.

  • Two switches, both deliberate. RELEASE_PIPELINE=promote (step 7's variable) and PROMOTE_DRY_RUN=false (new). The job promote-to-prod-live exists in a pipeline only when both are set; the script itself is live only when PROMOTE_DRY_RUN is exactly false, so any other value is a dry run. The old DRY_RUN variable is gone.
  • The dry-run job cannot be turned live. It runs promote-to-prod.sh --dry-run (GitLab project variables outrank a job's own variables:, so a project PROMOTE_DRY_RUN=false would otherwise have reached it), the script unsets GITLAB_API_TOKEN, PROMOTE_BOT_TOKEN and DEPLOY_SSH_KEY itself (the job's YAML blanking of them was also overridable by a project variable), and it never loads the write library.
  • Where the writes are. Every write is a function in build-scripts/promote-live.sh, sourced only on the live branch after checks 1 to 7 have all passed. promote-to-prod.sh and the pure helpers in promote-lib.sh contain none, and promote-to-prod.test.sh fails if a write appears there, if the library is loaded in a dry run (with a canary in its place), or if a dry run sees a credential.
  • Sequence, as in the design: guards, crane tag of the release version onto the five digests (read back), k3s-stack edit (digests, BADGER_RELEASE_VERSION in four patches, previous values saved as previous-images.env), verify-deploy.sh with EXPECTED_SHA, EXPECTED_RELEASE and DOCS_REQUIRED, and only then the master fast-forward (-o ci.skip), the tag and the GitLab release. A failed verification exits 3 with the previous digests and the exact git revert command; master and the tag do not move. A failure after verification exits 4 and a rerun finishes the rest.
  • Guards: PROMOTE_EXPECT_VERSION must equal the version the commit would get (typed in the play form, after reading the dry run), the four production patches must already carry the env entry, an existing release tag must point at the same digest, and the live job refuses unless RELEASE_PIPELINE=promote.
  • The write credential is PROMOTE_BOT_TOKEN, scoped in GitLab to the environment production-release, which only this job declares, so no other job holds it. It falls back to GITLAB_API_TOKEN. The job shares the k3s-stack-main resource group with the deploy jobs.
  • Bug found by the rehearsal and fixed: on a kustomization that already had a digest: line, the edit duplicated it, so a rerun pushed a second commit. It would have bitten the second release. Reruns are now no-ops.
  • Legacy path: step 7 had already put update-infra-repo-prod, verify-deploy-prod, finalise_version_update, create_gitlab_release and the master maven-build (via the classifier) behind RELEASE_PIPELINE. Nothing further was needed: with the variable promote they do not exist on master, unset they run as before.
  • Rehearsed by promote-live.test.sh (fixture repositories, fake registry, GitLab API and k3s-stack): success, verification failure, mixed digests, unreachable registry, rerun after partial failure, guards, master moving mid-release, release API failure, protected master. Not tested: the real GitLab protection rules, variable scoping and push options, Flux acting on the edit, the real production smoke test, crane tag against the real registry.

Decisions

Recorded 2026-11-19, with the reason each was chosen.

  1. Release trigger: Option B, a manual promote-to-prod job. The thing released is a named, verified commit, so there is no inference from a merge commit; and master moves only after a verified rollout, so it always means "what prod runs". The MR-driven Option A is the rejected alternative.
  2. Dev deploys immediately and is annotated, not gated. Dev is where we find out; holding it back by the length of the tests costs more than it saves. Promotion is gated on verified-<sha>.
  3. Versioning is fixed in full before the first promoted release (step 2, directly after the build-identity step). A promoted image must never report a dev version, and the bump commit must not exist when master is fast-forwarded.
  4. A manual deploy-hotfix-without-tests override exists on the hotfix path, named for what it does, recorded on the release; smoke test, alerts and revert still run. An emergency must not need a pipeline edit.
  5. Prod revert is alert-only first. Verification and Flux alerts are live from release one; the revert only reports until a CI variable is switched on after two clean releases (dev: after a week of report-only).
  6. HTTP only; no cluster credential for CI. Fewer secrets and a smaller blast radius. The gap is stated: the worker has no HTTP endpoint, so a worker that fails to start is seen only by the Flux alert. An in-cluster read-only ServiceAccount stays the later option.
  7. The docs image is promoted with the other four (same tags, same rules), since the deploy rewrites every tag and a missing docs tag would otherwise be an ImagePullBackOff.

References

  • DORA capabilities: trunk-based development, continuous delivery, deployment automation.
  • trunk-based.dev: short-lived branches, release from trunk or release branches, hotfix patterns.
  • Flux Kustomization spec.wait, spec.timeout, notification Provider and Alert.
  • OCI image-spec annotations; sigstore cosign keyless signing with CI OIDC identity.
  • The existing pieces this builds on: .gitlab-ci.yml, build-scripts/build.sh, dev-scripts/nextVersion.sh, Build & Runtime Footprint, and the cluster manifests in the infrastructure repository.