Skip to content

Storefront Render Benchmark

dev-scripts/render-bench.sh measures what a change does to storefront rendering. It builds the web app from two git refs, puts the same load on each, and compares them. Use it for any change meant to make pages cheaper to render (templates, caches, extensions, dialects), and to check that one meant to change nothing else doesn't make pages slower.

Everything runs in Docker: the builds in the same Maven container as devEnv.sh, the web app and k6 in containers of their own. The benchmark has its own Mongo, RabbitMQ, Valkey and Typesense on an internal network (render-bench), so it never touches your badger-dev data and nothing it does can leave the machine.

Running it

./dev-scripts/render-bench.sh build before development   # any git ref
./dev-scripts/render-bench.sh build after WORKTREE       # this checkout, uncommitted edits included
./dev-scripts/render-bench.sh seed before                # first time only
./dev-scripts/render-bench.sh compare before after       # three rounds each, about 8 minutes a run

compare writes target/render-bench/report-before-vs-after.md. Each run's raw output (k6 summaries, counter snapshots, the web log and every page's HTML) stays in target/render-bench/results/<label>-r<round>/. Run ./dev-scripts/render-bench.sh down when you're finished, or down -v to delete the benchmark's database and builds too.

Before a real comparison, check the set-up with SHORT=1, which cuts every timed phase to about 10 seconds. Its figures measure nothing: with so little warm-up, two builds of the same code can differ by a factor of two.

What a run does

Each run is the same for both builds:

  1. Start web cold, on an empty Redis, with production's profile and garbage collector, limited to 2 CPUs and 1.5 GB (WEB_CPUS, WEB_MEMORY), close to one production pod.
  2. Request every page once and time it: the first request after a deploy, with nothing cached.
  3. Warm up for 45 seconds under mixed load. These requests aren't counted.
  4. Load each page on its own for 40 seconds with 8 virtual users: the requests per second a page sustains and its latency at that load.
  5. Request all the pages in turn at a fixed rate (RATE, default 35 per second) for 60 seconds. This is an open model: requests keep arriving whether or not the last one has finished, so its p50, p95 and p99 are the latency a visitor sees, and the most useful figures in the report.

Between steps 3 and 5 it records Redis commands, Mongo operations, CPU time, memory allocated, garbage collection pauses and the Thymeleaf cache counters, and reports each one per request.

compare alternates which build goes first in each round (A then B, B then A, ...). The report shows the median of the rounds with the range beside it, so one noisy round can't decide the result. Treat a change smaller than the ranges as no change.

Reading the report

  • Mixed load p50, p95, p99: the headline. Rendering work shows up most in p95 and p99.
  • Dropped requests: must be 0 for both builds. Any means the slower build couldn't keep up at RATE, and its latency figures are meaningless. Lower RATE and run again. A good RATE is about two thirds of what the slower build sustains.
  • CPU and memory allocated per request: CPU time is what a pod pays for. Allocation drives garbage collection, which drives p99.
  • Redis commands and Mongo operations per request: data access during rendering. A rise here is usually a cache that stopped working.
  • Same pages?: the first request's HTML from each build, with tokens, ids, times and whitespace evened out (dev-scripts/render-bench/normalise-html.py). A change that shouldn't change any page must show every page as identical. For a difference, diff the two files named there after running them through normalise-html.py.

Pages

dev-scripts/render-bench/pages.json lists the pages, each as a label, a host and a path. By default these are the home page, a collection page, a product page and the basket on two demo sites: lacamiseta-en (Nova, with collections) and halden (Depot, with a merchandising tree). seed creates the sites named by the hosts' first labels. To measure other pages, point PAGES at your own file before seeding; any demo site from dev-scripts/seedDemoSites.sh works.

Limits

  • Every request is a first visit. k6 keeps cookies, but nothing runs the page's JavaScript, so the header menu cookie a returning visitor's browser sets is never sent. The benchmark measures the full menu render, not the cached menu skeleton a repeat visitor gets.
  • One machine. Both builds run on the same laptop, one after the other, so compare them with each other, not with production. Close heavy applications while it runs. Results from different machines or different days don't compare.
  • No checkout or signed-in pages. Add them to pages.json only if they render for an anonymous visitor with a plain GET.