Release Runbook: The First Real Release and Every One After¶
This is the checklist for releasing to production with promote-to-prod-live, and for aborting or
undoing a release by hand. The design and reasoning are in
Release Pipeline Design; the day-to-day summary is in the
Release Transition Guide.
Nothing here is switched on until two project CI variables are set on purpose. Until then the
master pipeline releases the old way and promote-to-prod is a dry run.
What the live job does¶
promote-to-prod-live is a manual job on a development pipeline. For one verified commit it:
- Checks the seven preconditions (the same checks as the dry run) and refuses with the reason if any fails.
- Guards. It refuses unless
RELEASE_PIPELINEispromote, a bot token is present,PROMOTE_EXPECT_VERSIONequals the version it computed, and all four production deployment patches already have aBADGER_RELEASE_VERSIONenv entry. Nothing has been written up to here. - Tags the release version (for example
0.102.0) onto the five tested image digests (crane tag, nothing rebuilt), reading each back. A release tag that already points at a different digest aborts the job. - Edits k3s-stack: the production kustomization (new
newTagplusdigestfor the five images) andBADGER_RELEASE_VERSIONin the web, rest, auth and worker patches, as one commit pushed tomain. The previous values are saved as the job artifactprevious-images.env. - Verifies production over HTTP (
verify-deploy.sh: smoke test, every service answersX-Badger-Buildwith the commit andX-Badger-Releasewith the version, docs/build.txt). It waits up to 15 minutes for the rollout. - Only if verification passed: fast-forwards
masterto the commit (bot push with-o ci.skip, so no pipeline runs), pushes the git tag, and creates the GitLab release with generated notes.
If step 5 fails, master and the git tag do not move. The job turns red and prints the previous
digests and the exact git revert command. Production may be in a half-rolled state until someone
reverts or fixes forward: the revert job is report-only.
Every step is safe to repeat. Run the job again after a partial failure: tags already on the right digest are kept, a k3s-stack that already holds the release is not pushed again, and an existing tag or GitLab release is left alone.
Before the first release (one time)¶
Do these in order. None of them changes production by itself.
- k3s-stack prerequisites. The four production deployment patches (
web,rest,auth,workerunderflux/apps/badger-commerce/environments/production/) carry the env entryBADGER_RELEASE_VERSIONwithvalue: "". Both are already in k3s-stackmain; confirm with a read. Also check that the alert rules exist influx/infrastructure/monitoring/10-alerting-rules.yaml:DeploymentNotFullyAvailable(a Badger deployment short of ready replicas for 15 minutes, which covers the worker),FluxKustomizationNotReady(production or development Kustomization not Ready for 5 minutes, from kube-state-metrics'gotk_resource_info) andFluxMetricsMissing(that metric absent for 15 minutes, so the previous rule cannot go quiet by accident). They are alert only and nothing reverts. There is no Flux Provider and no Slack secret to create. - The bot token. Create a project access token (or use the existing bot's) with the Developer role or higher and
apiandwrite_repositoryscopes, owned by the account that is allowed to push tomasterand to k3s-stackmain. Add it as the project CI variablePROMOTE_BOT_TOKEN: masked (hidden if available), protected, and with environment scopeproduction-release. Onlypromote-to-prod-livedeclares that environment, so no other job receives it. (If you skip the scope, the job falls back to the existingGITLAB_API_TOKEN, which every job can see: prefer the scoped one.) Reusing the existingGITLAB_API_TOKENalso works, provided its user may push tomaster(step 4) and push k3s-stackmain, and the token hasapiandwrite_repository. Optionally addPROMOTE_BOT_USER(the token's username; any non-empty name works). - k3s-stack write access. The same token's user must be allowed to push to k3s-stack
main, as the deploy jobs' user already is. Confirm by looking at who pushed the lastchore: update badger-commerce k3s ... imagescommit. - Protect
master(Settings, Repository, Protected branches): Allowed to push and merge: the bot user (and Maintainers for merge requests); Allow force push: off. The bot must be able to push tomasterbecause the job fast-forwards it. If the bot is not allowed, the job turns red after production is already updated and verified; rerun it once the permission is fixed. - Back-merge token. The hotfix path's back-merge uses
GITLAB_API_TOKEN, which must be allowed to open and merge merge requests intodevelopment. Check it with a dry visit to the token's role. Optionally create the project labelback-merge-needs-human; GitLab creates it on first use otherwise. - Dry run on the commit you intend to release (see below) and read it all.
Release day¶
- Flip the switch. In Settings, CI/CD, Variables set
RELEASE_PIPELINE=promoteandPROMOTE_DRY_RUN=false. GitLab decides which jobs a pipeline has when it is created, sopromote-to-prod-liveappears only in pipelines created after both are set. From now onmasterno longer releases the old way, and amastercommit that is already ondevelopmentfails with "release with promote-to-prod". - Get a pipeline that has the live job. The next push to
developmentdoes it; or use Run pipeline ondevelopment(it rebuilds, retests and re-verifies the tip: the same commit gets new images, so release from this pipeline's digests, not an older one's). Wait until it is fully green: that writesverified-<sha>on all five images, andmaster-containedmust be green too. - In that pipeline play
promote-to-prod(the dry run) and read all of it: the seven checks, the version, the tags, the kustomization diff, the notes. It must endall checks passed. Note the version (for example0.102.0). - Play
promote-to-prod-livefrom the same pipeline and, in the play form, add the variablePROMOTE_EXPECT_VERSIONwith that version. - Watch the job log. Expect, in order: the checks,
live: tagged ...five times,live: pushed k3s-stack <sha>,live: verifying production serves ...,live: verification passed,master fast-forwarded,pushed tag,created GitLab release,RELEASED <version>. - In parallel watch: the Alertmanager alerts
FluxKustomizationNotReadyandDeploymentNotFullyAvailable(a rollout that never becomes ready), the web, rest, auth and worker pods rolling (kubectl -n badger-commerce get pods, or Grafana), the storefront and admin footer showing the new release label. - Afterwards check:
git ls-remote origin refs/heads/masterequals the released commit; the tag<version>exists on it; the GitLab release lists five digests; the admin footer shows the version; the worker is running the new image (the smoke test cannot see it).
If it goes wrong¶
The job refused before writing anything¶
The log says REFUSED or LIVE FAIL with the reason, and nothing was changed. Fix the cause (an unverified image, master not contained, a wrong PROMOTE_EXPECT_VERSION, a missing env entry) and play it again.
Verification failed (exit 3, VERIFICATION FAILED)¶
Production was updated but did not verify; master and the git tag did not move.
- Decide: fix forward (if the failure is a transient check, play the job again: it re-verifies without redoing anything) or revert (below).
- Look at the Alertmanager alerts (
FluxKustomizationNotReady,DeploymentNotFullyAvailable) and the pods:kubectl -n badger-commerce get pods,kubectl -n badger-commerce describe pod <name>, and the logs of the one that did not start. A worker that failed to start is not visible to the HTTP check.
Revert by hand¶
The job prints the exact command. It is a revert of the k3s-stack commit it pushed:
git clone [email protected]:badger-commerce/k3s-stack.git
cd k3s-stack
git revert --no-edit <the k3s-stack commit printed by the job>
git push origin main
Flux then rolls production back to the previous digests (saved in the artifact previous-images.env and printed in the log). The release tag left on the registry images is harmless. If other commits were pushed to k3s-stack after the release commit, git revert still works; resolve any conflict in kustomization.yaml by hand with the values from previous-images.env.
After a revert, master is still at the old release (it never moved), so nothing else needs undoing. To release the same commit again later, play the job again: it reuses the release tags and edits k3s-stack from the saved state.
The job went red after verification (exit 4)¶
Production is live on the new version and verified. Only the bookkeeping failed: master push refused (the bot lacks permission on the protected branch), master is no longer an ancestor (someone pushed to master during the release: look at git log origin/development..origin/master), a tag push, or the GitLab release. Fix the cause and play the job again: it repeats only what is missing.
Abort before or during a release¶
- Before playing the live job: delete the project variable
PROMOTE_DRY_RUN(or set it back totrue). The job disappears from new pipelines (a pipeline already created keeps it, so also just do not play it) and the dry run is all that remains. To switch the whole new flow off, also deleteRELEASE_PIPELINE;masterthen releases the old way again. - While the live job runs: cancel it only before the line
live: pushed k3s-stack. After that production is changing; let the verification finish or revert as above. A cancelled job leaves tags on the images and, at most, the k3s-stack commit; both are safe to rerun over.
Going back to the old way¶
Delete RELEASE_PIPELINE. The master pipeline is then exactly the legacy one (build, deploy, version tag, release). Do this only if the new path is broken: the legacy path produces a different image than the one tested on development, and master will hold commits development lacks until they are merged back.
What the rehearsal covers, and what it cannot¶
build-scripts/promote-live.test.sh runs the real script against a fixture repository with a bare
origin, a bare fake k3s-stack, a fake registry and a fake GitLab API: the full release, a failed
verification (nothing moves; the printed revert restores the old files), a mixed-digest set, an
unreachable registry, a rerun after partial failure (idempotent), a wrong expected version, a missing
env entry, a release tag that already points elsewhere, master moving during the release, the GitLab
release call failing, and a protected master refusing the bot. It cannot show: that GitLab's
protected-branch rules, the variable scoping and the ci.skip push option behave as described on the
real project, that Flux applies the digest edit as expected, or that the real HTTP verification passes.
The first release is that test: do it when you can watch it.