Skip to content

Release Transition Guide: The New Way of Working

This is the short version for developers. It explains what changes day to day when releases move from "rebuild master and deploy" to "promote the exact image that was already built and tested on development". The reasoning, alternatives and every check are in Release Pipeline Design; how version numbers work is in Versioning.

Where we are

The change is rolled out in steps. Until the switch step, releases still work the old way, so nothing here changes what you do on release day yet.

Part State
Every image carries its commit (git-<sha> tag, labels) and answers X-Badger-Build with it Live
Tests gate development; images that pass are tagged verified-<sha>; a smoke test runs after the dev deploy Live
promote-to-prod job on development Live as a dry run: it prints what it would do and changes nothing
Prod verification after a deploy (verify-deploy-prod, manual check-prod) Live, report only
Alerts for failed rollouts (Alertmanager rules for a stuck deployment and a Flux Kustomization that is not Ready) Live, alert only; no automatic revert
Hotfix path on master (build, test, deploy, verify, tag, back-merge) Built, switched off until RELEASE_PIPELINE=promote is set (the switch step)
master-contained check on development Live as a warning; becomes a hard failure with the switch
Real releases by promote-to-prod-live Built and rehearsed, switched off: it does not exist until RELEASE_PIPELINE=promote and PROMOTE_DRY_RUN=false are both set. Step by step in the Release Runbook
Automatic revert Not yet

What changes

Before After the switch
What production runs A fresh build of master, never tested as that image The image development already built and tested, given a release tag
How you release Merge development into master Play the manual promote-to-prod job on a verified development commit
What master is A branch with its own commits (the version bump) A pointer to what production runs. It only moves forward to a released development commit
Version bump commit chore: bump version [skip ci] on master None. The poms keep 0.0.0-SNAPSHOT; the release is a git tag
Release name Taken from the poms Computed from commit messages and tagged on the released commit
Rollback Manual Manual at first; reported by alerts; automatic once enabled

Day to day: nothing different

  1. Branch from development, open a merge request, merge. Commit messages stay conventional (feat:, fix:, feat!: or a BREAKING CHANGE: footer), because they decide the next version.
  2. Merging to development builds the images once, runs unit and integration tests, deploys to dev and smoke-tests it. When all of that passes, the same images get a verified-<sha> tag.
  3. A red development pipeline means that commit can't be released. Fix forward on development.

Don't merge into master by hand, and don't push to it. After the switch only the release job and hotfix merge requests write to it.

Releasing

After the switch:

  1. Pick a commit on development whose pipeline is green (so it has verified-<sha> on all five images: web, rest, worker, auth, docs).
  2. In that pipeline, play promote-to-prod.
  3. The job refuses, with the reason and the fix, unless: the commit is on development; all five images are verified and came from one pipeline; master is an ancestor of the commit; the current prod image digests can be read (they are saved for a revert); and the registry, GitLab and the deploy repository are reachable. It never falls back to building.
  4. It adds the release tag to the same image digests, sets the new image digests and BADGER_RELEASE_VERSION in the production kustomization, waits for the rollout, then runs the verification (smoke test, plus each service's X-Badger-Build and X-Badger-Release headers and the public docs /build.txt).
  5. Only if that passes does it fast-forward master, push the version tag and create the GitLab release with notes generated from the commits since the last release.

If verification fails, master and the tag don't move. Today an alert tells you; once automatic revert is switched on it also puts the previous digests back.

The live job is promote-to-prod-live; you give it the version the dry run printed (PROMOTE_EXPECT_VERSION) so a release is never started by accident.

Trying it now: play promote-to-prod on any development pipeline. It is a dry run, so it prints the plan (tags, kustomization diff, version, release notes) and changes nothing. The job shows red if it would have refused; the pipeline stays green.

Hotfix when development has work you don't want in prod

  1. Branch hotfix/... from master (what production runs), fix, and push. The branch pipeline runs the tests.
  2. Merge request into master, fast-forward where possible.
  3. The master pipeline sees a commit that isn't on development, so for this case only it builds, runs the unit and integration tests, and only then deploys to production, runs the smoke test, tags a patch release and creates the GitLab release.
  4. The same pipeline opens a back-merge merge request from master into development and merges it straight away when GitLab reports it clean. If it has conflicts it is labelled back-merge-needs-human and the job shows "passed with warnings": resolve and merge it. Until it lands, master-contained is red on development and promote-to-prod refuses, so the hotfix can't be forgotten.

A merge of staged development content straight into master fails the master pipeline with "release with promote-to-prod": nothing is built or deployed.

Emergency override. The manual job deploy-hotfix-without-tests (in the master pipeline, next to the normal deploy) deploys the built images without waiting for the tests. It writes tests: skipped (override by <user>) in the k3s-stack commit message and the release notes. The smoke test (verify-hotfix-override), the alerts and the release tag still follow. Use it only when production is down and the tests take too long.

Is it on yet? No. All of this runs only when the project CI variable RELEASE_PIPELINE is promote (Settings, CI/CD, Variables). Unset, master keeps releasing as it does today. The variable is set at the switch step, together with turning promote-to-prod live (PROMOTE_DRY_RUN=false). The exact order, what to watch and how to abort or revert by hand are in the Release Runbook.

Reading a version

  • Build identity is the commit SHA. curl -sI https://<host>/ | grep -i x-badger-build on web, rest or auth shows which build answers; docs publish it at /build.txt.
  • Release name (0.103.0) is shown in the admin footer and the X-Badger-Release header. On dev it reads dev <sha8>. Until a release goes through promote-to-prod, prod's footer shows dev <sha8> too.
  • A release tag can be found on the git commit and on each image in the registry, next to the git-<sha> and verified-<sha> tags for that same digest.

Things that still work the old way

  • devEnv.sh and local builds don't depend on the committed version any more; the placeholder is expected.
  • Re-running a whole development pipeline moves an existing verified-<sha> tag to the new build, and logs a warning.
  • A registry cleanup that removed an image for an old commit makes promote-to-prod refuse. Rebuild that commit through the manual build-sha pipeline on development.