Release Pipeline Design: Build Once, Test, Promote¶
Status: approved 2026-11-19. Steps 0 to 4 are implemented (step 2 in two parts, see Versioning; steps 3 and 4 in part, see What shipped for steps 3 and 4); step 5 (dry run) and the build half of step 6 are implemented, see What shipped for steps 5 and 6; step 7 (the hotfix path) is implemented but switched off, see What shipped for step 7; step 8 (the live release) is built, rehearsed and switched off, see What shipped for step 8; the rest is not. The rollout is a list of small steps, each independently mergeable and reversible (see Rollout plan).
Summary¶
Today development and master each run their own maven-build, so production runs an image that
was never the one tested on development. Nothing waits for the tests, nothing checks that a rollout
worked, and nothing rolls back.
The proposal, in one paragraph: build once on development, test that exact build, mark it
verified, and release by a manual promote-to-prod job that adds a production tag to the same image
digest (no rebuild, no copy, because all environments share one registry repository per image).
The job edits the production kustomization, checks the rollout, and only then fast-forwards master
to the released commit and creates the release tag, so master is always exactly what production
runs. master is built only for a hotfix, and then it builds and tests before deploying. Every
deploy to either environment is followed by a verification job (a smoke test of the public
hostnames, which also proves the new build is the one answering) and Flux notifications cover what
HTTP cannot see. Automatic revert is built, reports only at first, and is switched on by a CI
variable once two releases have verified cleanly. The release label, release tag and release notes
are all generated by the promote job, and the version is fixed before the first promoted release.
Where we are (verified 2026-11-19)¶
| Area | Today |
|---|---|
| Branches | development (staged work, deploys to dev) and master (what prod runs). Releases are an MR "development into master", which creates a merge commit; a master-only commit chore: bump version to X [skip ci] is added after the build. master allows force-push (protected branch setting) |
| Build | maven-build runs on both branches: Maven package, then five images (web, rest, worker, auth, docs) built and pushed in a Docker-in-Docker sidecar. Tag is <project.version>-<branch slug>, for example 0.101.0-master or 0.102.0-dev.6916b050-development |
| Tests | unit-tests and integration-tests run on development and feature branches only, never on master. They skip on --skip-test in the commit message, and on changes: rules (a docs-only push). They are interruptible, so a newer commit cancels an older pipeline's tests |
| Deploy | update-infra-repo-dev and -prod need only maven-build. Dev therefore deploys before the tests finish and whatever their result. Deploy is a sed over every newTag: line in the environment's kustomization in the infrastructure repository (k3s-stack), pushed to main |
| Rollout | Flux Kustomizations have wait: true; a 10 minute timeout on prod. No remediation, no rollback, no Flux alert or notification objects anywhere in k3s-stack. CI finishes green when the tag is pushed, not when the rollout finishes |
| Registry | One repository per image. Cleanup policy: daily, keep 10 newest, delete tags older than 90 days matching ^\d+\.\d+\.\d+-dev\.[0-9a-f]+-development$, never delete tags matching ^\d+\.\d+\.\d+-master$ |
| Supply chain | Trivy container scan of the web image only. Images are built with --provenance=false. No SBOM, signatures, secret detection or dependency update automation in the pipeline |
| Version | Computed from commit messages by dev-scripts/nextVersion.sh, written into the poms by mvn versions:set in every build, and also embedded in the jar (see Versioning) |
Corrections to the brief this design was started from¶
healthChecksomitting auth and docs is moot. Withwait: true, Flux waits on every workload the Kustomization applies and ignoresspec.healthChecks. So auth and docs are already waited for. (Confirm on dev in step 4 by breaking the docs image tag and watching the Kustomization stay not-ready; the missing-tag case in finding 5 is therefore caught by Flux, but only after the fact, and nothing tells anyone.) The real gap is that nothing reports the result.- A promotion is a tag, not a copy. Dev and prod tags live in the same repository per image, so
promoting is adding a second tag to the same manifest digest (
crane tag). The "doesbuildx imagetools createrewrap the manifest" concern disappears: nothing is rewritten, and signatures and attestations (stored per digest) stay attached. masterallows force-push. Ifmasteris going to mean "what production runs", that should be switched off.- The
nextVersion.shbug is slightly different from "the footer can never match". The script loopsfor commit in $(git log --pretty=%s), which word-splits, so it tests individual words rather than subject lines, and it only reads subjects, never bodies or footers.BREAKING CHANGE:(two words) can never match,feat!:is not recognised, and a subject such asfix: handle feat: in titlesis misread. Fix and test it before relying on the version for anything (step 0). - Release notes list four images (docs missing) and claim
linux/amd64, linux/arm64whileDOCKER_PLATFORMSdefaults to arm64 only. Both are wrong today. - AOT cache: the dev builds train it again (the earlier master-only change is reverted), so a
promoted image carries it. The cache validates jar path, size, mtime, JDK and flags, not the image
tag, so adding tags does not invalidate it. The
.gitlab-ci.ymlread for this document predates that revert; re-check theAOT_TRAININGline when step 1 touches the build job. Dev builds train it, so the promoted images carry it.
Recommended model¶
Branching and release model¶
Keep two long-lived branches, but change what master means. development is the trunk: all work
merges there through short-lived branches and it is always the thing being built, tested and deployed
to dev. master becomes the record of what production runs, and normally contains no commit that is
not on development: a release moves master forward to a verified development commit
(fast-forward, by the promote-to-prod job). The two branches then differ only by "development is ahead".
Why not collapse to one branch (pure trunk-based development)?
- Pure trunk-based practice (DORA's capability, trunk-based.dev) deploys every merge to production behind feature flags, with progressive delivery and automated rollback on error rates. We have multi-tenant shops with real money flowing, no traffic-splitting layer, one replica of rest, worker and auth, a Cloudflare tunnel in front, and test coverage that is good but not a substitute for a human choosing when to ship. A human-chosen release moment is a feature here: continuous delivery, not continuous deployment.
- A branch is a cheap, visible, protected pointer to "what prod runs", it gives hotfixes a natural base, and it keeps today's habits. The cost of two branches (drift, back-merges) is removed by the fast-forward rule and enforced by a check, rather than relying on discipline.
- A single branch plus release tags and the same manual promote job is equally valid and is a small
step from here: the job would simply stop moving
master. It needs no migration beyond this one.
Where this deviates from "pure" best practice, deliberately:
| Best practice | Here | Because |
|---|---|---|
| Deploy every merge to prod | A human triggers each production release | no progressive delivery or per-request routing, multi-tenant blast radius |
| Environment promotion by a PR to a deployment repo, approved per environment | The promote-to-prod job edits the k3s-stack kustomization directly (as now) |
a two-person team; keep one automated path, add verification and a revert that starts report-only |
| Admission-time signature verification (Kyverno, policy controller) | Verify at promotion time in CI | one more cluster controller to run and upgrade; the same guarantee at this scale as long as only CI can write the deploy repo |
| Canary or blue/green (Flagger, Argo Rollouts) | Rolling update, health gate, automatic revert | no mesh or traffic split; web already keeps an old pod up during a rollout (maxUnavailable: 1 with two replicas) |
The release trigger: a manual promote-to-prod job¶
A release is the manual job promote-to-prod, played on the development pipeline of a verified
commit C. It:
- checks the preconditions (below) and resolves the five digests behind
git-C; - adds the release tag (
X.Y.Z, computed by the fixednextVersion.shfrom the commits since the last release tag that is reachable from C) to each image digest withcrane tag; - edits the production kustomization in k3s-stack (digests, and
BADGER_RELEASE_VERSION) and pushes it; - runs
verify-deployagainst production (smoke test and build identity, below); - on success: fast-forwards
masterto C, pushes the git tagX.Y.Zon C, and creates the GitLab release with notes generated fromgit log <previous release tag>..Cand the digests actually promoted. On failuremasterand the tag do not move, and the revert job reports (or reverts, when enabled).
Moving master after verification makes it literally what production runs: a failed rollout never
leaves it pointing at something that is not live. The push is made by the CI bot with
-o ci.skip, and master is protected so that only the CI bot (and, for hotfixes, maintainers via
merge request) can write to it, with force-push off.
The rejected alternative is Option A: releasing by a merge request from development into
master and having the master pipeline decide to promote or build. It keeps a familiar review
step but needs a rule to guess from a merge commit whether it matches something already built, which
is where the "promoted an image that lacks a master-only hotfix" failure comes from. Option B has no
such guess: the thing released is named by its commit.
Hotfix flow¶
When staged work on development must not go to prod yet:
- Branch from
master(hotfix/...), fix, push. The branch pipeline runs unit and integration tests. - Merge request into
master, fast-forward only where possible, so master's commit is the one that was tested on the branch. The merge itself is the only human write tomaster. - The
masterpipeline for that commit sees content that is not ondevelopment, so it builds the images and runs the full tests before it deploys, then deploys, verifies (with the same smoke test) and creates a patch release tag on the commit. The images carrygit-<sha>and the OCI labels like any build. - Emergency override: a manual
deploy-hotfix-without-testsjob exists in that pipeline, named for what it does. It deploys the built images without waiting for the tests, recordstests: skipped (override by <user>)in the release notes and the k3s-stack commit message, and the smoke test, Flux alerts and (when enabled) the revert still run. - A job then opens an MR
masterintodevelopment(the back-merge), and merges it automatically when it is clean, otherwise labels it for a human. It carries[skip ci]only when development has no other changes to build; normally development builds it like any other change.
How the pieces work with a machine-moved master:
- Normal releases never put a commit on
masterthat is not ondevelopment, and the bot only fast-forwards it, somasterstays an ancestor ofdevelopment. - A hotfix is the one commit that breaks that. Until the back-merge lands,
masteris not an ancestor ofdevelopment. That is detected by themaster-containedcheck:git merge-base --is-ancestor origin/master <development tip>. It runs on every development pipeline (red, with a message naming the missing commits) and is a precondition ofpromote-to-prod. Sopromote-to-prodrefuses until the back-merge is done; that, plus the precondition below thatmastermust be an ancestor of C so that fast-forward is possible, make the back-merge impossible to forget. - After the back-merge, development contains the hotfix and builds it; the next promotion carries it to production in a verified development build, as a fast-forward again.
- A human merging staged development content straight into
masteris caught: themasterpipeline sees a commit that is an ancestor ofdevelopmentand fails with "release with promote-to-prod".
Build-once artifact model¶
Tags, labels, digests¶
Per image, per build on development:
| Tag | Meaning | Mutable? | Retention |
|---|---|---|---|
git-<full 40-char sha> |
Identity of the build. The only tag the promotion logic looks up. On a master build (a hotfix) the tag is git-<sha>-master, so the cleanup policy's keep pattern for -master tags protects it |
not moved by design (a retry of maven-build for the same commit rebuilds and moves it) |
follows its -dev. tag: deleted by the cleanup policy together with it |
<version>-dev.<sha8>-development |
Today's human tag, kept for continuity and the dev kustomization | no | existing policy |
verified-<full sha> |
Written by the last job of a successful dev pipeline: built, tested, deployed to dev and smoke tested | written once | same as git- |
<X.Y.Z> (a release) |
Added at promotion; names the release | written once | kept forever (rollback targets) |
Every image carries org.opencontainers.image.revision=<sha>, .source, .created, and
org.opencontainers.image.version where known, so docker inspect answers "what is this".
The git-<sha> tag is the lookup key because a rerun of a pipeline for the same commit gets a new
<version>-dev.<sha8> tag only if the version moved, and the old scheme cannot be searched by SHA.
maven-build records the five digests in an artifact (images.env: WEB_IMAGE=, WEB_DIGEST=sha256:...,
WEB_TAGS= and the same for REST, WORKER, AUTH, DOCS, plus IMAGES_COMMIT_SHA,
IMAGES_PIPELINE_ID, IMAGES_VERSION, IMAGES_CREATED and IMAGES_PLATFORMS). The job takes each digest from
BuildKit's --metadata-file and fails unless docker buildx imagetools inspect reports the same
digest for both pushed tags. Both tags are pushed by the same build, so they are one manifest.
Labels are set with --label on the final image (revision, created, version; source,
vendor, title and description stay in the Dockerfiles, and the docs image takes source from
the build command because its Dockerfile has no labels).
Metadata and attestations: a proportionate subset¶
| Item | Verdict | Where |
|---|---|---|
| OCI labels (revision, source) | Do now | step 1 |
Immutable git-<sha> tag, deploy by digest |
Do now | steps 1, 4, 8 |
SBOM (BuildKit --sbom=true, or syft) attached to the digest |
Do soon; makes "are we affected by CVE X" a query not an investigation | step 10 |
Provenance (--provenance=mode=max) |
Do with the SBOM. It was switched off; confirm the GitLab registry and a single-platform index are happy with the attestation manifests before turning it on, and that retention does not orphan them | step 10 |
| cosign signature, keyless via GitLab's OIDC token | Do soon; signed once at build, verified in the promote job against the identity of this project's pipeline on development or master |
step 10 |
| Kyverno or policy-controller admission verification | Defer; revisit if more people or systems can write to k3s-stack | later |
| SLSA level claims, reproducible builds | Not worth it at this scale | n/a |
Promotion mechanics¶
Tool¶
crane (go-containerregistry), pinned by version and checksum in the promote job, in a small
image with git and curl. The job needs no Docker-in-Docker and none of the 7Gi pod:
crane digest to resolve, crane tag repo@sha256:... <newtag> to promote, crane manifest to
verify. regctl is equivalent; crane is chosen for crane tag being a single call and for
being in the container ecosystem most often cited for this. Do not use buildx imagetools create
(it can rewrap manifests); verify after tagging that
crane digest repo:<newtag> equals the digest recorded at build, and fail otherwise.
Preconditions and failure modes (promote-to-prod for commit C)¶
The job is a script (build-scripts/promote-to-prod.sh) unit-tested with fixture repositories. It
names the commit it releases, so there is nothing to infer from merge parents. It refuses unless
all of these hold, and the failure is loud and names the remedy. The docs image is one of the five
and follows the same rules.
| # | Check | If it fails |
|---|---|---|
| 1 | C is on development, and the pipeline for C ran on development |
refuse |
| 2 | All five images have git-C and verified-C (written by the dev pipeline only when build, tests and the dev smoke test all passed) |
refuse: "C is not verified; fix development or wait for its pipeline" |
| 3 | The five digests all come from the same pipeline (recorded in images.env), not a mixed set |
refuse |
| 4 | master is an ancestor of C (master-contained, and so a fast-forward is possible) |
refuse: "master has commits not on development; finish the hotfix back-merge" |
| 5 | The production kustomization's current digests are readable and saved, for revert | refuse |
| 6 | The registry, GitLab API or k3s-stack is unreachable | refuse; there is no fallback to building |
| 7 | C has all five images but a registry cleanup removed one | refuse and name the rebuild path: rerun the build for C (manual build-sha pipeline on development) |
The master pipeline then has only two outcomes: a commit that is not on development is a
hotfix (build, test, deploy, as above), and a commit that is on development fails with "release
with promote-to-prod". Nothing builds silently, and a commit with master-only content can never be
mistaken for a development-tested one because only promote-to-prod writes release tags and it only
accepts development commits.
PROMOTE_DRY_RUN=true (the default until the switch step; the dry-run job also passes --dry-run) performs checks 1 to 7 and prints what it would
have done, touching nothing.
Deploy mechanics (k3s-stack)¶
- Both environments' kustomizations switch from
newTag: "<tag>"tonewTagplus digest (digest: sha256:...) per image, with the human tag kept in a comment. A digest is immutable, so "what is running" is unambiguous even if a tag is later moved or deleted. Check on dev that kustomize rendersrepo@sha256:...as expected before using it on prod. - The deploy script's blanket
sedover everynewTag:line becomes a per-image edit that fails if any of the five images is missing, so a missingbadger-docstag can no longer reach the cluster. - The deploy job saves the previous values it replaced (artifact
previous-images.env) for rollback. - One serialised queue (
resource_group: k3s-stack-main) stays, as the repository is shared.
Verification gate and rollback¶
A verify-deploy job runs after every environment deploy (dev and prod). It uses HTTP only: no
cluster credential is given to CI, at least for now.
- Smoke test over HTTP, against the public hostnames, run from CI:
- the storefront home page, one collection, one product page and the basket respond 200 with expected markers;
- the REST API and the auth server's
/.well-known/ JWKS respond; the docs site responds; - build identity: each of web, rest, auth and docs exposes the commit SHA it was built from
(a response header on the app services and a
build.txtfile on the docs image). The test loops until the SHA equals the expected one on enough requests to cover both web replicas, which is what proves this deploy is what answers, not the old pods still serving during the rollout. - Flux status is covered by alerts, not by CI. Alertmanager rules in k3s-stack (
10-alerting-rules.yaml:FluxKustomizationNotReadyafter 5 minutes, from kube-state-metrics'gotk_resource_info, withFluxMetricsMissingafter 15 minutes as its dead-man check, andDeploymentNotFullyAvailable) announce a stuck or failed reconcile, whether or not a pipeline is watching. The Flux Provider and Alert idea was dropped. Alerts are in place from the first promoted release. - Result. On success on dev,
mark-verifiedwritesverified-<sha>. On failure the pipeline is red, the alert has fired, and the revert job runs.
The honest gap. The worker has no HTTP endpoint, so a worker that fails to start (or crash-loops
after the HTTP services are healthy) is not seen by the smoke test; it is caught only by the Flux
alert (a rollout that never becomes ready) and by whatever runtime alerting exists. Two ways to close it
later, neither decided: an in-cluster read-only ServiceAccount token for the runner so CI can wait on
the Flux Kustomization's Ready condition and its applied revision (the preferred later addition), or the
worker writing its build SHA to Redis where the REST service's build-identity response reports it.
Rollback strategy: pipeline revert in k3s-stack, not progressive delivery. Kubernetes' own rolling update already keeps an old pod serving when the new one never becomes ready (web keeps one of two), so the typical bad release is a stalled rollout rather than an outage. The revert job restores the previous digests saved by the deploy job, pushes to k3s-stack, and runs the smoke test again; the previous release's images are retained forever. Flux Kustomizations have no built-in remediation (that is a HelmRelease feature), and Flagger or Argo Rollouts need a traffic-splitting layer we do not run; reconsider if we add one or the replica counts grow. A guard: forward-compatibility of data migrations (expand then contract; the system-update JSON files are applied at start-up) is the real limit on rollback.
Revert is report-only first. It always computes what it would revert and says so in the job log and
the notification. It acts only when the CI variable AUTO_REVERT_PROD (and, separately, AUTO_REVERT_DEV) is true.
Dev runs report-only for a week; prod stays report-only until two promoted releases have verified
cleanly, then the variable is flipped. Verification and Flux alerts are live from release one.
Test gating¶
The aim is one unambiguous statement: commit C is verified if and only if, on C's tree, build, unit tests, integration tests and the dev smoke test all passed.
- Deploy to dev stays fast (deploys right after the build, as today) but is annotated, not
gated: the dev environment is where we find out, and holding it back by 7 minutes costs more than
it saves (decision 2). What changes is that promotion is gated on
verified-<sha>, and the dev deploy's k3s-stack commit message says the commit is not yet verified. - On
development, the tests always run. Remove thechanges:skip there (keep it on feature branches) and let--skip-testmean "deploy to dev, do not mark verified". A docs-only push after a code push then costs 7 minutes of test compute but never leaves a SHA ambiguous. Optimisation later: reuse a previous result when thegit rev-parse <sha>:<path>tree hashes of the Maven build inputs (the.java-build-pathslist) are identical to an already-verified commit, writingverified-<sha>without rerunning. - Auto-cancel can drop an older commit's tests; that is fine, because that commit simply never
becomes verified and release picks a newer one. The
master-containedandverifiedchecks mean an abandoned SHA cannot be promoted by accident. Themark-verifiedjob isinterruptible: false, andneedsthe tests withoptional: false, so a skipped test job (rules) blocks it rather than counting as a pass. A tiny scriptassert-job-results.shreads the pipeline's jobs via the API and requires named jobs to besuccess. masterfor a hotfix runs the tests before deploy (needsthe tests on that path).- Compile once (later, step 11): each of build, unit and integration compiles the tree separately.
Options in order of effort: share
target/classes through a job artifact; add Maven's build-cache extension backed by the existing S3 cache; or fold the unit run into the build job. This is a latency optimisation (the three run in parallel, so the wall-clock gain is limited by the integration job queueing for a burst node, 7+ minutes) and is deliberately not on the critical path of this design.
Versioning and release identity¶
Today the version number has three jobs mixed together: build identity (jar manifest, build-info,
asset cache-busting), display (admin version string), and release name (the tag). The jar embeds
@project.version@ in messages.properties (admin.badger.system.version), AssetUrlDialect
(?v=), MenuGeneratorExtension (the menu.js key) and the build-info actuator, so a promoted image
would report its development version. All of this is fixed before the first promoted release
(step 2 of the rollout), not after:
- Build identity is the commit SHA, exposed as one build property.
?v=and themenu.jskey use it: the bytes are identical in dev and prod, so identical cache-busting is correct, and an asset URL changes exactly when the build does. The same value is the build-identity response header used by the smoke test. - Display is a release label from the environment variable
BADGER_RELEASE_VERSION, set per environment in the kustomization (the file the deploy job already edits) and falling back to the build identity when unset (dev showsdev <sha8>; prod shows0.103.0). - Release name is a git tag on the promoted commit, created by
promote-to-prodafter a verified rollout, version computed by the fixednextVersion.sh. - The
chore: bump version [skip ci]commit is dropped, with it the master-only content. Every build already rewrites the poms withversions:set, so the committed pom version is redundant. The checked-in poms keep a constant placeholder (0.0.0-SNAPSHOT). Before removing it, check that nothing outside CI relies on the committed version:devEnv.sh, the dev scripts, the IDE import, the Docker build contexts. The oldfinalise_version_updatejob then tags the released commit without committing (a small change to the existing release path, until the switch step retires it). - Release notes are generated by
promote-to-prodfromgit log <previous release tag>..C, for all five images, with the digests actually promoted, and the platforms read from the image manifest rather than hard-coded. nextVersion.sh: rewrite to read subject and body per commit (git log --format=%Bwith a delimiter), recognisetype!:andBREAKING CHANGE:footers, honourfeat:/feat(scope):, and cover it with a small test script. Pre-1.0 stays "breaking bumps the major" per the existing convention. It finds the last release tag withgit tag --merged <commit> --sort=-v:refname, so hotfix tags onmastercount once back-merged.
The development image's own human tag stays <version>-dev.<sha8>-development for the dev
kustomization; it is not a release name.
Security and supply-chain steps, in priority order¶
- Scan every image, not only web (Trivy over all five digests, in the verify stage of the dev
pipeline). Report on all severities; fail
mark-verifiedon fixable CRITICAL (then HIGH once the backlog is clear), with a reviewed allowlist file for accepted findings. - Secret detection (GitLab's
Jobs/Secret-Detectiontemplate, push protection if the plan allows) on every branch pipeline. - Dependency updates: Renovate (or GitLab's dependency scanning plus a scheduled update MR) for Maven, npm, the Docker base images and the pinned CI images. Today pins are manual.
- Disable force-push on
master, and restrict who and what can pushmasterand k3s-stackmain(the CI bot only, for deploys). - SBOM, provenance and cosign signature as in the table above; verify signature and identity in the promote job.
- SAST (GitLab's template) as a non-blocking report first.
- Later: admission-time verification, if more than CI can reach the deploy repository.
Rollout plan¶
Each step is its own MR, ships to development first, and lists how to verify before anything touches
master. Bold marks steps that touch the production release path.
| # | Step | Depends on | Risk | Verify on development | Monitor |
|---|---|---|---|---|---|
| 0 | No-regret fixes: rewrite and test nextVersion.sh; turn off force-push on master; correct the release-notes image list and platform text; extend the registry cleanup policy so git-* and verified-* tags follow their -dev. tag and release tags are kept |
none | Low. Release notes and version arithmetic only | Script test; compare its output with the existing tags over history | Next release's version and notes |
| 1 | Immutable identity: add git-<sha> tag and OCI labels to all five images, write images.env with digests as an artifact. No behaviour change; old tags stay |
0 | Low | crane ls / crane config on a dev build; digest in artifact equals registry digest |
Registry size (extra tags are free: same digest) |
| 2 | Versioning, all of it: SHA as build property (?v=, menu.js key, build-info, build-identity header and docs build.txt); BADGER_RELEASE_VERSION in both kustomizations and the admin label; constant placeholder pom version and no bump commit (after checking devEnv.sh and IDE reliance); the old finalise_version_update tags without committing; release-notes generator (tested, not yet called by a promote) |
1 | Medium: touches AssetUrlDialect, the admin version string and the existing release job |
Render tests; asset URLs and menu.js key behave the same on dev; admin shows the label; run the tag-only finalise on a scratch branch |
Admin footer, asset cache hit rate, the next legacy release's tag |
| 3 | Test gating on development: tests always run there, --skip-test and cancellation semantics as above, assert-job-results.sh, mark-verified writes verified-<sha> |
1 | Low: new jobs only. Dev pipelines get longer by the cost of the previously skipped test runs | Push a code change, a docs-only change, a --skip-test change, and cancel one mid-run; verified- tags appear only when expected |
Runner queue time, pipeline duration |
| 4 | Dev verification: smoke test and build-identity check as a verify-deploy job; Flux alerts for dev (built as Alertmanager rules instead of a Flux Provider and Alert); dev deploy by digest; the revert job in report-only mode; confirm Flux waits on the docs workload; mark-verified also requires a green smoke test |
2, 3 | Medium for dev (a flaky smoke test can block verification). Dev is the proving ground | Deliberately break a dev deploy (bad tag, failing readiness) and watch the report and the alert; a week of report-only | Smoke-test flake rate, report-only "would revert" count |
| 5 | promote-to-prod in dry run: the manual job on the development pipeline, checks 1 to 7 and the notes generator, nothing written |
3, 4 | None on prod (report only) | Run it on several real verified commits; unit-test the checks against fixture repositories (unverified, mixed set, master not an ancestor, missing image) | Mismatch between its verdict and what is true |
| 6 | Prod verification and Flux alerts on the existing prod deploy: verify-deploy against production hostnames and Flux alerts for prod (built as Alertmanager rules, see below), with AUTO_REVERT_PROD absent (report only). Lands while the old master build still deploys |
2, 4 | Medium: the first prod smoke test; a failure only reports | Run the same job against dev, then prod after a legacy release | Prod verify results and alerts |
| 7 | Hotfix path and master protection: master pipeline for a commit not on development builds, tests before deploy, deploys, verifies and tags; the deploy-hotfix-without-tests override; the master-contained check on every development pipeline; the auto back-merge MR; master pipeline fails for staged content; restrict master pushes to the CI bot and maintainers via MR |
2, 6 | Medium: first change to the master pipeline's jobs | Rehearse with a trivial hotfix branch off master in a scratch project or with prod deploy disabled; check the back-merge MR and the master-contained red/green |
Back-merge MRs merge; check goes green |
| 8 | The switch (built, inert; see What shipped for step 8 and the runbook): promote-to-prod goes live (PROMOTE_DRY_RUN=false): it tags by digest, deploys by digest, verifies, then fast-forwards master and tags and creates the release. The old master maven-build, update-infra-repo-prod and finalise_version_update for staged content are retired by RELEASE_PIPELINE=promote (the inverse of the proposed LEGACY_MASTER_RELEASE: unset restores the old path for two releases, then it is removed) |
5, 6, 7 | High (prod release path). Mitigated by the dry runs and the escape hatch | A full rehearsal against dev (point the job at the dev kustomization with a variable); the first live release is watched end to end | Reported prod digest equals dev's; release duration; master equals the released commit |
| 9 | Switch on AUTO_REVERT_PROD=true after two clean promoted releases (and AUTO_REVERT_DEV after the dev week) |
8 | Medium: an overzealous revert is a prod event | Simulate a failure on dev first | Revert count, flake rate |
| 10 | Supply chain: scan all images, secret detection, Renovate, SBOM and provenance, cosign sign at build and verify at promotion. Scan gating starts report-only | 3 (scan gate), 8 (verify in promote) | Low to medium | Dev pipelines show each artefact; verification fails on an unsigned test image | Scan time, finding counts |
| 11 | Optional: compile-once, reuse of a verified result by tree hash, the in-cluster ServiceAccount for Flux readiness checks (closes the worker gap), Kyverno | 8 | Varies | Decide after step 9 | n/a |
Steps 0 to 5 do not change what production runs, except step 2, which changes the version string and the existing release job's tag step. Step 8 is the first that changes how production is released, so it is deliberately late and reversible by a single variable.
What shipped for steps 3 and 4¶
On development only; nothing on master or in the production kustomization changed.
- Tests always run on
development.unit-testsandintegration-testslost theirchanges:rule there; feature branches keep it.--skip-teststill skips them and the commit is never verified. Each job still compiles the tree itself: compile-once stays step 11, so the build is not reused by the tests. - Dev deploy is annotated.
update-infra-repo-devstill needs onlymaven-buildand deploys at once; its k3s-stack commit message now ends(not yet verified: commit <sha8>, pipeline <id>). The production deploy job's message is unchanged. verify-deploy-dev(stageverify,build-scripts/verify-deploy.sh) runs after the dev deploy is pushed. It waits until web, rest and auth each answerX-Badger-Build: <full sha>six requests in a row (up to 15 minutes), then checks the web home page androbots.txt, the REST health endpoint, the auth server's metadata and JWKS. Not done: collection, product and basket pages (the default shop has no stable seoNames to assert on;/basketis a 404 there).- Docs build identity is skipped unless a service token is configured.
docs.bdgr-demo.co.ukis behind Cloudflare Access, so an anonymous request gets a login redirect. The script reports that asSKIP. Set the CI variablesCF_ACCESS_CLIENT_IDandCF_ACCESS_CLIENT_SECRET(an Access service token allowed on that host) and it checks/build.txt, failing if it is wrong. Until then a stale docs pod is not caught. mark-verified(build-scripts/mark-verified.sh, withassert-job-results.shand a pinnedcrane,install-crane.sh) first requiresunit-tests,integration-testsandverify-deploy-devto besuccessin this pipeline through the API (a job that did not run is not a pass), then addsverified-<full sha>to the digests recorded inimages.envwithcrane tagand reads it back. If the registry'sgit-<sha>tag no longer points at the tested digest it refuses. A rerun of the whole pipeline for a commit moves an existingverified-tag to the new build, with a warning in the log.- Not done in step 4: Flux alerts (later built as Alertmanager rules), dev deploy by digest, the report-only revert job and the break-it-deliberately check of the docs workload. These need changes in k3s-stack or a deliberate bad deploy to dev and remain for a follow-up.
What shipped for steps 5 and 6¶
On development only. Nothing on master, in the production kustomization or on the production
cluster changed.
promote-to-prodis a manual job, dry run only (build-scripts/promote-to-prod.sh, stagefinalise_version,allow_failure, so an unplayed job never blocks a pipeline). Play it on adevelopmentpipeline; setPROMOTE_SHAwhen playing to name another commit than the pipeline's. It runs checks 1 to 7 (commit ondevelopment; all fivegit-<sha>andverified-<sha>tags present and the same digest; one build, by therevision,createdandversionlabels;masteran ancestor; the production kustomization readable; a missing image names the rebuild path), then prints what the live job would do: the release versionnextVersion.shgives that commit, onecrane tag <image>@<digest> <version>per image, a diff of the production kustomization (new tag plus digest per image, made on a throwaway copy), theBADGER_RELEASE_VERSIONvalue, theverify-deploycall, themasterfast-forward, the tag and the generated release notes. It exits 1 when the live job would refuse (the plan is still printed, marked not run).- It cannot change anything. The script contains no write command (only
echo "would run: ..."),DRY_RUN=falseis refused, andpromote-to-prod.test.shfails ifcraneis ever called with a non-read subcommand or the script gains an executablecrane tag,git pushorgit tag. The job blanksGITLAB_API_TOKENandDEPLOY_SSH_KEY(project variables reach every job); its only credential is the job token, which reads this project's registry and git and, because the project is on k3s-stack's job token allowlist, clones k3s-stack for check 5. - While
mastercarries legacy bump commits, check 4 refuses. That is true today (masterhas commitsdevelopmentlacks) and is the check doing its job; it clears once a release is made by the new path andmasteronly ever fast-forwards. - Production verification reuses
build-scripts/verify-deploy.sh, now withEXPECTED_RELEASE(web, rest and auth must also answerX-Badger-Release: <name>; the header is new, set byBuildIdentityFilterfromBADGER_RELEASE_VERSION) and a legacy mode with noEXPECTED_SHAfor a production still running builds from before the identity header: the smoke test runs and build identity is printed asNOT CHECKED, never as a pass. Docs are public in prod, soDOCS_REQUIREDis on there and a missing or stale/build.txtfails.verify-deploy.test.shtests it against a local fake of the four services. - Jobs.
verify-deploy-prod(onmaster, afterupdate-infra-repo-prod,allow_failure, report only) waits for production to answer with the commit just deployed;check-prodis a manual job on anydevelopmentpipeline that runs the smoke test against production as it is now. - Flux alerts are live, as Alertmanager rules (not a Flux Provider and Slack secret, which were
dropped):
DeploymentNotFullyAvailable,FluxKustomizationNotReadyandFluxMetricsMissingin k3s-stack10-alerting-rules.yaml, alert only, nothing reverts. The emptyBADGER_RELEASE_VERSIONenv entry thatpromote-to-prodneeds is also already in the production deployment patches. Nothing is left to create.
What shipped for step 7¶
Built, tested with fixtures, and inert: nothing here changes what master does or what runs in
production until a project CI variable is set.
- The switch is
RELEASE_PIPELINE, a project-level CI variable (Settings, CI/CD, Variables), unset today. Rollout step 8 flips it by settingRELEASE_PIPELINE=promote(no commit needed) at the same time aspromote-to-prodgoes live. Unset (or any other value), themasterpipeline is the legacy one: the pipeline's job list formasteris the same as before, the legacyupdate-infra-repo-prod,verify-deploy-prod,finalise_version_updateandcreate_gitlab_releaserun, and none of the newmasterjobs exist. Setting it back (deleting it) is the escape hatch; step 8'sLEGACY_MASTER_RELEASEcan be this same variable inverted. master-contained(build-scripts/master-contained.sh) runs on everydevelopmentpipeline:git merge-base --is-ancestor origin/master <commit>, red with the commits master has and development lacks.promote-to-prodcheck 4 now calls the same script, so they cannot disagree (master-contained.test.shasserts it). WhileRELEASE_PIPELINEis notpromotethe job isallow_failure(a warning): the legacy flow merges development into master with a merge commit development lacks.- Classifier (
classify-masterjob,classify-master-commit.sh), first job of amasterpipeline in promote mode: a commit not ondevelopmentis a hotfix andmaven-buildproceeds (itneedsthe classifier, optionally, so the edge does not exist in legacy mode); a commit ondevelopmentfails with "release with promote-to-prod" and nothing builds. - Hotfix chain (promote mode,
masteronly):maven-build(images taggedgit-<sha>-masteras before) in parallel withunit-testsandintegration-tests(theirmasterrule is now on in promote mode) ->update-infra-repo-hotfix(needs all three, none optional: a test job that did not run fails it) ->verify-hotfix(the smoke test as a gate) ->finalise-hotfix(hotfix-release.sh: patch tag fromnextVersion.sh, GitLab release withtests: passed, then the back-merge). The k3s-stack commit message carrieshotfix, tests: passed. - Override: manual
deploy-hotfix-without-tests->verify-hotfix-override->finalise-hotfix-override, withtests: skipped (override by <user>)in the k3s-stack commit message and the release notes (RELEASE_TESTS_NOTEinreleaseNotes.sh). If the tests also pass later, the normal deploy finds production already at that tag and pushes nothing. - Back-merge (
back-merge.sh, the last step ofhotfix-release.sh): opens (or reuses) an MRmaster->development, polls GitLab'sdetailed_merge_status, merges at once whenmergeable, otherwise labels itback-merge-needs-human, exits 42, which the job maps to "passed with warnings" (allow_failure: exit_codes: 42). It merges rather than waiting for a pipeline because merge-request pipelines are off in this project and the hotfix was just tested. NeedsGITLAB_API_TOKENto be allowed to merge intodevelopment. - Not done in step 7: restricting pushes to
master(a GitLab protected-branch setting, to be made with the switch), and "the master pipeline fails for staged content" is the classifier above. - Tested:
master-contained.test.sh,back-merge.test.sh(back-merge against a fake GitLab API,hotfix-release.shagainst a fixture repo and a bare remote),releaseNotes.test.sh. The CI wiring is checked by GitLab's linter and by thedevelopmentpipeline; the promote-modemasterjobs cannot run until the variable is set, so their first real run is a hotfix after the switch.
What shipped for step 8¶
The live release exists and is rehearsed. It is inert: nothing changes until the user sets the project CI variables below. Full order of steps, abort and revert: the Release Runbook.
- Two switches, both deliberate.
RELEASE_PIPELINE=promote(step 7's variable) andPROMOTE_DRY_RUN=false(new). The jobpromote-to-prod-liveexists in a pipeline only when both are set; the script itself is live only whenPROMOTE_DRY_RUNis exactlyfalse, so any other value is a dry run. The oldDRY_RUNvariable is gone. - The dry-run job cannot be turned live. It runs
promote-to-prod.sh --dry-run(GitLab project variables outrank a job's ownvariables:, so a projectPROMOTE_DRY_RUN=falsewould otherwise have reached it), the script unsetsGITLAB_API_TOKEN,PROMOTE_BOT_TOKENandDEPLOY_SSH_KEYitself (the job's YAML blanking of them was also overridable by a project variable), and it never loads the write library. - Where the writes are. Every write is a function in
build-scripts/promote-live.sh, sourced only on the live branch after checks 1 to 7 have all passed.promote-to-prod.shand the pure helpers inpromote-lib.shcontain none, andpromote-to-prod.test.shfails if a write appears there, if the library is loaded in a dry run (with a canary in its place), or if a dry run sees a credential. - Sequence, as in the design: guards,
crane tagof the release version onto the five digests (read back), k3s-stack edit (digests,BADGER_RELEASE_VERSIONin four patches, previous values saved asprevious-images.env),verify-deploy.shwithEXPECTED_SHA,EXPECTED_RELEASEandDOCS_REQUIRED, and only then themasterfast-forward (-o ci.skip), the tag and the GitLab release. A failed verification exits 3 with the previous digests and the exactgit revertcommand;masterand the tag do not move. A failure after verification exits 4 and a rerun finishes the rest. - Guards:
PROMOTE_EXPECT_VERSIONmust equal the version the commit would get (typed in the play form, after reading the dry run), the four production patches must already carry the env entry, an existing release tag must point at the same digest, and the live job refuses unlessRELEASE_PIPELINE=promote. - The write credential is
PROMOTE_BOT_TOKEN, scoped in GitLab to the environmentproduction-release, which only this job declares, so no other job holds it. It falls back toGITLAB_API_TOKEN. The job shares thek3s-stack-mainresource group with the deploy jobs. - Bug found by the rehearsal and fixed: on a kustomization that already had a
digest:line, the edit duplicated it, so a rerun pushed a second commit. It would have bitten the second release. Reruns are now no-ops. - Legacy path: step 7 had already put
update-infra-repo-prod,verify-deploy-prod,finalise_version_update,create_gitlab_releaseand the mastermaven-build(via the classifier) behindRELEASE_PIPELINE. Nothing further was needed: with the variablepromotethey do not exist onmaster, unset they run as before. - Rehearsed by
promote-live.test.sh(fixture repositories, fake registry, GitLab API and k3s-stack): success, verification failure, mixed digests, unreachable registry, rerun after partial failure, guards, master moving mid-release, release API failure, protected master. Not tested: the real GitLab protection rules, variable scoping and push options, Flux acting on the edit, the real production smoke test,crane tagagainst the real registry.
Decisions¶
Recorded 2026-11-19, with the reason each was chosen.
- Release trigger: Option B, a manual
promote-to-prodjob. The thing released is a named, verified commit, so there is no inference from a merge commit; andmastermoves only after a verified rollout, so it always means "what prod runs". The MR-driven Option A is the rejected alternative. - Dev deploys immediately and is annotated, not gated. Dev is where we find out; holding it back by
the length of the tests costs more than it saves. Promotion is gated on
verified-<sha>. - Versioning is fixed in full before the first promoted release (step 2, directly after
the build-identity step). A promoted image must never report a dev version, and the bump commit must
not exist when
masteris fast-forwarded. - A manual
deploy-hotfix-without-testsoverride exists on the hotfix path, named for what it does, recorded on the release; smoke test, alerts and revert still run. An emergency must not need a pipeline edit. - Prod revert is alert-only first. Verification and Flux alerts are live from release one; the revert only reports until a CI variable is switched on after two clean releases (dev: after a week of report-only).
- HTTP only; no cluster credential for CI. Fewer secrets and a smaller blast radius. The gap is stated: the worker has no HTTP endpoint, so a worker that fails to start is seen only by the Flux alert. An in-cluster read-only ServiceAccount stays the later option.
- The docs image is promoted with the other four (same tags, same rules), since the deploy rewrites
every tag and a missing docs tag would otherwise be an
ImagePullBackOff.
References¶
- DORA capabilities: trunk-based development, continuous delivery, deployment automation.
- trunk-based.dev: short-lived branches, release from trunk or release branches, hotfix patterns.
- Flux
Kustomizationspec.wait,spec.timeout, notificationProviderandAlert. - OCI image-spec annotations; sigstore cosign keyless signing with CI OIDC identity.
- The existing pieces this builds on:
.gitlab-ci.yml,build-scripts/build.sh,dev-scripts/nextVersion.sh, Build & Runtime Footprint, and the cluster manifests in the infrastructure repository.