Release Transition Guide: The New Way of Working¶
This is the short version for developers. It explains what changes day to day when releases move
from "rebuild master and deploy" to "promote the exact image that was already built and tested on
development". The reasoning, alternatives and every check are in
Release Pipeline Design; how version numbers work is in
Versioning.
Where we are¶
The change is rolled out in steps. Until the switch step, releases still work the old way, so nothing here changes what you do on release day yet.
| Part | State |
|---|---|
Every image carries its commit (git-<sha> tag, labels) and answers X-Badger-Build with it |
Live |
Tests gate development; images that pass are tagged verified-<sha>; a smoke test runs after the dev deploy |
Live |
promote-to-prod job on development |
Live as a dry run: it prints what it would do and changes nothing |
Prod verification after a deploy (verify-deploy-prod, manual check-prod) |
Live, report only |
| Alerts for failed rollouts (Alertmanager rules for a stuck deployment and a Flux Kustomization that is not Ready) | Live, alert only; no automatic revert |
Hotfix path on master (build, test, deploy, verify, tag, back-merge) |
Built, switched off until RELEASE_PIPELINE=promote is set (the switch step) |
master-contained check on development |
Live as a warning; becomes a hard failure with the switch |
Real releases by promote-to-prod-live |
Built and rehearsed, switched off: it does not exist until RELEASE_PIPELINE=promote and PROMOTE_DRY_RUN=false are both set. Step by step in the Release Runbook |
| Automatic revert | Not yet |
What changes¶
| Before | After the switch | |
|---|---|---|
| What production runs | A fresh build of master, never tested as that image |
The image development already built and tested, given a release tag |
| How you release | Merge development into master |
Play the manual promote-to-prod job on a verified development commit |
What master is |
A branch with its own commits (the version bump) | A pointer to what production runs. It only moves forward to a released development commit |
| Version bump commit | chore: bump version [skip ci] on master |
None. The poms keep 0.0.0-SNAPSHOT; the release is a git tag |
| Release name | Taken from the poms | Computed from commit messages and tagged on the released commit |
| Rollback | Manual | Manual at first; reported by alerts; automatic once enabled |
Day to day: nothing different¶
- Branch from
development, open a merge request, merge. Commit messages stay conventional (feat:,fix:,feat!:or aBREAKING CHANGE:footer), because they decide the next version. - Merging to
developmentbuilds the images once, runs unit and integration tests, deploys to dev and smoke-tests it. When all of that passes, the same images get averified-<sha>tag. - A red
developmentpipeline means that commit can't be released. Fix forward ondevelopment.
Don't merge into master by hand, and don't push to it. After the switch only the release job and
hotfix merge requests write to it.
Releasing¶
After the switch:
- Pick a commit on
developmentwhose pipeline is green (so it hasverified-<sha>on all five images: web, rest, worker, auth, docs). - In that pipeline, play
promote-to-prod. - The job refuses, with the reason and the fix, unless: the commit is on
development; all five images are verified and came from one pipeline;masteris an ancestor of the commit; the current prod image digests can be read (they are saved for a revert); and the registry, GitLab and the deploy repository are reachable. It never falls back to building. - It adds the release tag to the same image digests, sets the new image digests and
BADGER_RELEASE_VERSIONin the production kustomization, waits for the rollout, then runs the verification (smoke test, plus each service'sX-Badger-BuildandX-Badger-Releaseheaders and the public docs/build.txt). - Only if that passes does it fast-forward
master, push the version tag and create the GitLab release with notes generated from the commits since the last release.
If verification fails, master and the tag don't move. Today an alert tells you; once automatic
revert is switched on it also puts the previous digests back.
The live job is promote-to-prod-live; you give it the version the dry run printed
(PROMOTE_EXPECT_VERSION) so a release is never started by accident.
Trying it now: play promote-to-prod on any development pipeline. It is a dry run, so it prints
the plan (tags, kustomization diff, version, release notes) and changes nothing. The job shows red
if it would have refused; the pipeline stays green.
Hotfix when development has work you don't want in prod¶
- Branch
hotfix/...frommaster(what production runs), fix, and push. The branch pipeline runs the tests. - Merge request into
master, fast-forward where possible. - The
masterpipeline sees a commit that isn't ondevelopment, so for this case only it builds, runs the unit and integration tests, and only then deploys to production, runs the smoke test, tags a patch release and creates the GitLab release. - The same pipeline opens a back-merge merge request from
masterintodevelopmentand merges it straight away when GitLab reports it clean. If it has conflicts it is labelledback-merge-needs-humanand the job shows "passed with warnings": resolve and merge it. Until it lands,master-containedis red ondevelopmentandpromote-to-prodrefuses, so the hotfix can't be forgotten.
A merge of staged development content straight into master fails the master pipeline with
"release with promote-to-prod": nothing is built or deployed.
Emergency override. The manual job deploy-hotfix-without-tests (in the master pipeline, next
to the normal deploy) deploys the built images without waiting for the tests. It writes
tests: skipped (override by <user>) in the k3s-stack commit message and the release notes. The smoke
test (verify-hotfix-override), the alerts and the release tag still follow. Use it only when
production is down and the tests take too long.
Is it on yet? No. All of this runs only when the project CI variable RELEASE_PIPELINE is
promote (Settings, CI/CD, Variables). Unset, master keeps releasing as it does today. The
variable is set at the switch step, together with turning promote-to-prod live (PROMOTE_DRY_RUN=false).
The exact order, what to watch and how to abort or revert by hand are in the
Release Runbook.
Reading a version¶
- Build identity is the commit SHA.
curl -sI https://<host>/ | grep -i x-badger-buildon web, rest or auth shows which build answers; docs publish it at/build.txt. - Release name (
0.103.0) is shown in the admin footer and theX-Badger-Releaseheader. On dev it readsdev <sha8>. Until a release goes throughpromote-to-prod, prod's footer showsdev <sha8>too. - A release tag can be found on the git commit and on each image in the registry, next to the
git-<sha>andverified-<sha>tags for that same digest.
Things that still work the old way¶
devEnv.shand local builds don't depend on the committed version any more; the placeholder is expected.- Re-running a whole
developmentpipeline moves an existingverified-<sha>tag to the new build, and logs a warning. - A registry cleanup that removed an image for an old commit makes
promote-to-prodrefuse. Rebuild that commit through the manualbuild-shapipeline ondevelopment.