Skip to content

Release Runbook: The First Real Release and Every One After

This is the checklist for releasing to production with promote-to-prod-live, and for aborting or undoing a release by hand. The design and reasoning are in Release Pipeline Design; the day-to-day summary is in the Release Transition Guide.

Nothing here is switched on until two project CI variables are set on purpose. Until then the master pipeline releases the old way and promote-to-prod is a dry run.

What the live job does

promote-to-prod-live is a manual job on a development pipeline. For one verified commit it:

  1. Checks the seven preconditions (the same checks as the dry run) and refuses with the reason if any fails.
  2. Guards. It refuses unless RELEASE_PIPELINE is promote, a bot token is present, PROMOTE_EXPECT_VERSION equals the version it computed, and all four production deployment patches already have a BADGER_RELEASE_VERSION env entry. Nothing has been written up to here.
  3. Tags the release version (for example 0.102.0) onto the five tested image digests (crane tag, nothing rebuilt), reading each back. A release tag that already points at a different digest aborts the job.
  4. Edits k3s-stack: the production kustomization (new newTag plus digest for the five images) and BADGER_RELEASE_VERSION in the web, rest, auth and worker patches, as one commit pushed to main. The previous values are saved as the job artifact previous-images.env.
  5. Verifies production over HTTP (verify-deploy.sh: smoke test, every service answers X-Badger-Build with the commit and X-Badger-Release with the version, docs /build.txt). It waits up to 15 minutes for the rollout.
  6. Only if verification passed: fast-forwards master to the commit (bot push with -o ci.skip, so no pipeline runs), pushes the git tag, and creates the GitLab release with generated notes.

If step 5 fails, master and the git tag do not move. The job turns red and prints the previous digests and the exact git revert command. Production may be in a half-rolled state until someone reverts or fixes forward: the revert job is report-only.

Every step is safe to repeat. Run the job again after a partial failure: tags already on the right digest are kept, a k3s-stack that already holds the release is not pushed again, and an existing tag or GitLab release is left alone.

Before the first release (one time)

Do these in order. None of them changes production by itself.

  1. k3s-stack prerequisites. The four production deployment patches (web, rest, auth, worker under flux/apps/badger-commerce/environments/production/) carry the env entry BADGER_RELEASE_VERSION with value: "". Both are already in k3s-stack main; confirm with a read. Also check that the alert rules exist in flux/infrastructure/monitoring/10-alerting-rules.yaml: DeploymentNotFullyAvailable (a Badger deployment short of ready replicas for 15 minutes, which covers the worker), FluxKustomizationNotReady (production or development Kustomization not Ready for 5 minutes, from kube-state-metrics' gotk_resource_info) and FluxMetricsMissing (that metric absent for 15 minutes, so the previous rule cannot go quiet by accident). They are alert only and nothing reverts. There is no Flux Provider and no Slack secret to create.
  2. The bot token. Create a project access token (or use the existing bot's) with the Developer role or higher and api and write_repository scopes, owned by the account that is allowed to push to master and to k3s-stack main. Add it as the project CI variable PROMOTE_BOT_TOKEN: masked (hidden if available), protected, and with environment scope production-release. Only promote-to-prod-live declares that environment, so no other job receives it. (If you skip the scope, the job falls back to the existing GITLAB_API_TOKEN, which every job can see: prefer the scoped one.) Reusing the existing GITLAB_API_TOKEN also works, provided its user may push to master (step 4) and push k3s-stack main, and the token has api and write_repository. Optionally add PROMOTE_BOT_USER (the token's username; any non-empty name works).
  3. k3s-stack write access. The same token's user must be allowed to push to k3s-stack main, as the deploy jobs' user already is. Confirm by looking at who pushed the last chore: update badger-commerce k3s ... images commit.
  4. Protect master (Settings, Repository, Protected branches): Allowed to push and merge: the bot user (and Maintainers for merge requests); Allow force push: off. The bot must be able to push to master because the job fast-forwards it. If the bot is not allowed, the job turns red after production is already updated and verified; rerun it once the permission is fixed.
  5. Back-merge token. The hotfix path's back-merge uses GITLAB_API_TOKEN, which must be allowed to open and merge merge requests into development. Check it with a dry visit to the token's role. Optionally create the project label back-merge-needs-human; GitLab creates it on first use otherwise.
  6. Dry run on the commit you intend to release (see below) and read it all.

Release day

  1. Flip the switch. In Settings, CI/CD, Variables set RELEASE_PIPELINE = promote and PROMOTE_DRY_RUN = false. GitLab decides which jobs a pipeline has when it is created, so promote-to-prod-live appears only in pipelines created after both are set. From now on master no longer releases the old way, and a master commit that is already on development fails with "release with promote-to-prod".
  2. Get a pipeline that has the live job. The next push to development does it; or use Run pipeline on development (it rebuilds, retests and re-verifies the tip: the same commit gets new images, so release from this pipeline's digests, not an older one's). Wait until it is fully green: that writes verified-<sha> on all five images, and master-contained must be green too.
  3. In that pipeline play promote-to-prod (the dry run) and read all of it: the seven checks, the version, the tags, the kustomization diff, the notes. It must end all checks passed. Note the version (for example 0.102.0).
  4. Play promote-to-prod-live from the same pipeline and, in the play form, add the variable PROMOTE_EXPECT_VERSION with that version.
  5. Watch the job log. Expect, in order: the checks, live: tagged ... five times, live: pushed k3s-stack <sha>, live: verifying production serves ..., live: verification passed, master fast-forwarded, pushed tag, created GitLab release, RELEASED <version>.
  6. In parallel watch: the Alertmanager alerts FluxKustomizationNotReady and DeploymentNotFullyAvailable (a rollout that never becomes ready), the web, rest, auth and worker pods rolling (kubectl -n badger-commerce get pods, or Grafana), the storefront and admin footer showing the new release label.
  7. Afterwards check: git ls-remote origin refs/heads/master equals the released commit; the tag <version> exists on it; the GitLab release lists five digests; the admin footer shows the version; the worker is running the new image (the smoke test cannot see it).

If it goes wrong

The job refused before writing anything

The log says REFUSED or LIVE FAIL with the reason, and nothing was changed. Fix the cause (an unverified image, master not contained, a wrong PROMOTE_EXPECT_VERSION, a missing env entry) and play it again.

Verification failed (exit 3, VERIFICATION FAILED)

Production was updated but did not verify; master and the git tag did not move.

  1. Decide: fix forward (if the failure is a transient check, play the job again: it re-verifies without redoing anything) or revert (below).
  2. Look at the Alertmanager alerts (FluxKustomizationNotReady, DeploymentNotFullyAvailable) and the pods: kubectl -n badger-commerce get pods, kubectl -n badger-commerce describe pod <name>, and the logs of the one that did not start. A worker that failed to start is not visible to the HTTP check.

Revert by hand

The job prints the exact command. It is a revert of the k3s-stack commit it pushed:

git clone [email protected]:badger-commerce/k3s-stack.git
cd k3s-stack
git revert --no-edit <the k3s-stack commit printed by the job>
git push origin main

Flux then rolls production back to the previous digests (saved in the artifact previous-images.env and printed in the log). The release tag left on the registry images is harmless. If other commits were pushed to k3s-stack after the release commit, git revert still works; resolve any conflict in kustomization.yaml by hand with the values from previous-images.env.

After a revert, master is still at the old release (it never moved), so nothing else needs undoing. To release the same commit again later, play the job again: it reuses the release tags and edits k3s-stack from the saved state.

The job went red after verification (exit 4)

Production is live on the new version and verified. Only the bookkeeping failed: master push refused (the bot lacks permission on the protected branch), master is no longer an ancestor (someone pushed to master during the release: look at git log origin/development..origin/master), a tag push, or the GitLab release. Fix the cause and play the job again: it repeats only what is missing.

Abort before or during a release

  • Before playing the live job: delete the project variable PROMOTE_DRY_RUN (or set it back to true). The job disappears from new pipelines (a pipeline already created keeps it, so also just do not play it) and the dry run is all that remains. To switch the whole new flow off, also delete RELEASE_PIPELINE; master then releases the old way again.
  • While the live job runs: cancel it only before the line live: pushed k3s-stack. After that production is changing; let the verification finish or revert as above. A cancelled job leaves tags on the images and, at most, the k3s-stack commit; both are safe to rerun over.

Going back to the old way

Delete RELEASE_PIPELINE. The master pipeline is then exactly the legacy one (build, deploy, version tag, release). Do this only if the new path is broken: the legacy path produces a different image than the one tested on development, and master will hold commits development lacks until they are merged back.

What the rehearsal covers, and what it cannot

build-scripts/promote-live.test.sh runs the real script against a fixture repository with a bare origin, a bare fake k3s-stack, a fake registry and a fake GitLab API: the full release, a failed verification (nothing moves; the printed revert restores the old files), a mixed-digest set, an unreachable registry, a rerun after partial failure (idempotent), a wrong expected version, a missing env entry, a release tag that already points elsewhere, master moving during the release, the GitLab release call failing, and a protected master refusing the bot. It cannot show: that GitLab's protected-branch rules, the variable scoping and the ci.skip push option behave as described on the real project, that Flux applies the digest edit as expected, or that the real HTTP verification passes. The first release is that test: do it when you can watch it.