Comparing Change Request Models¶

Change requests are written by one call to an AI model, set per site as change-request-model. Before choosing a default (or changing the prompt), run the same requests through several models and compare cost per accepted post, counting the reviewer's time, not cost per call. The harness for that is a profile of the worker.

What it does¶

The eval set lives in the worker, under worker/src/main/resources/change-request-evals/:

For each run, the worker started with the change-request-eval profile:

1. refuses to start unless the dev profile is on, no production profile is (production, k3s, anything containing prod), and every tenant it would touch is named cr-eval-...;
2. deletes and rebuilds one tenant per reference site (cr-eval-hedgehog, cr-eval-rescue, cr-eval-law), the way a new shop is created, then adds the site's content and settings;
3. stores each case's photos as a change request's photos, through the media library;
4. sends each case to each model by calling the draft writer directly, with the same prompt, schema, checks and draft page writing as a real request (the requests themselves aren't stored, so the normal worker never picks them up);
5. writes a report, and leaves every draft page on its tenant to preview.

The worker exits when it has finished.

Running it¶

You need the dev stack's dependencies running and the jars built (./dev-scripts/devEnv.sh, after a build that includes the worker). Then:

./dev-scripts/change-requests/eval.sh                                   # every case, the default models
./dev-scripts/change-requests/eval.sh --models anthropic:claude-haiku-5-5,anthropic:claude-sonnet-5-5
./dev-scripts/change-requests/eval.sh --cases rescue                    # one site's cases
./dev-scripts/change-requests/eval.sh --cases "staff bio,law-update-01" # categories or case ids

Keys: the script reads ANTHROPIC_API_KEY and OPENAI_API_KEY from ~/devEnv.conf (or the environment) without printing them, and the eval tenants store them as their own keys. Without them, the platform keys set in your dev database are used. The run stops before calling anything if a provider in --models has no key, or if it would make more than 200 calls (badger.change-request-eval.max-calls).

Each eval tenant is served at https://cr-eval-<site>.bdgr.localhost, and its drafts show in its admin like any other draft page. The tenants are wiped at the start of the next run.

The report¶

Each run writes a folder named for when it ran:

results.csv columns, by group:

• What happened: valid (a draft was written), error (the reason code when not, for example invalid_json, refusal or invalid_content: MEDIA_FOREIGN@leadMediaId), served_model (a provider fallback can answer with another model), page_id and preview_url (a preview link to the draft, valid for 14 days).
• Automatic checks: recipe and recipe_match; media_used and foreign_media (photos that weren't the request's); facts_found of facts_expected, and missing_facts; unsupported_numbers and unsupported_names, the numbers and capitalised words in the copy that appear in neither the request, its photos' tags, nor the site's name, description and page titles. These are a plain word match, so treat them as things to look at: an invented name or figure shows up here, and so may an unusual heading word. Lengths (title_chars, meta_chars, excerpt_chars, prose_words, blocks) and over_limits.
• Cost: input_tokens (uncached), cached_input_tokens, cache_write_tokens, output_tokens, cost_usd (blank for a model with no price in ModelPricing), latency_ms (the whole step: building the prompt, the call and writing the draft).
• For the reviewer: accepted, voice_match and edit_minutes are left blank.

Cases are sent model by model within each case, so a model reads the site's rules and details from its prompt cache from the second request on that site; cached_input_tokens shows how much. A real site sending a request now and then gets fewer cache hits, so compare uncached input too.

Scoring¶

Open each draft from preview_url (or the tenant's admin) and fill in, per row:

• accepted: y if you'd approve it after editing, n if you'd reject it or send it back;
• voice_match: 1 (wrong voice) to 5 (sounds like the organisation);
• edit_minutes: how long the edits before approval took (or how long you spent before rejecting).

Then work out cost per accepted post, model by model:

python3 dev-scripts/change-requests/score.py dev-scripts/change-requests/out/<run>/results.csv --rate 30

which is (model cost of every run + reviewer minutes × hourly rate / 60) ÷ accepted drafts. The rate is in dollars an hour, like the model costs. A cheap model whose drafts take ten minutes to fix usually costs more per accepted post than a dearer one whose drafts need a glance.

Things to try with the results: the cheapest model by default, moving up a tier when the automatic checks fail or a reviewer rejects twice.

Photos¶

The shipped photos are plain colour placeholders: the case files say so ("placeholder": true), so the model is sent each photo's description instead of the picture, exactly as when a photo can't be fetched. That keeps the set small and free of anyone's pictures, and keeps runs repeatable.

To include real photos, copy change-request-evals/ somewhere, put the photos (JPEG, PNG, GIF or WebP, about 1024 pixels on the long side, under 5 MB) in its photos/ folder, point each case at its file with "placeholder": false, and run with --cases-dir. Use only photos you're allowed to use. With --tagging live the photos are also described by image tagging instead of the case's tags.

Adding a case¶

Add a JSON file to cases/:

{
  "id": "rescue-callout-05",
  "category": "call-out report",
  "site": "rescue",
  "expectedRecipe": "NEWS_POST",
  "sentAt": "2026-11-09T18:00:00Z",
  "request": "Call-out 47 this afternoon ...",
  "photos": [
    {"file": "placeholder-11.png", "placeholder": true, "title": "...", "description": "...",
     "altText": "...", "keywords": ["mountain rescue"]}
  ],
  "expectedFacts": ["47", "Blencathra"]
}

sentAt fixes when the request was sent, so "last Saturday" means the same day every run. Every expected fact must appear in the request or its photos' tags; EvalCaseLoaderTest checks that, and that every site and photo a case names exists.

Notes¶

• The harness is a Spring profile of the worker (ChangeRequestEvalRunner, package uk.co.kedos.badger.settbuilder.changerequests.eval). Its beans only exist under that profile, so mvn test and deployed workers never create them; its unit tests call no model.
• The OpenAI candidates are opt-in: add openai:gpt-6-luna, openai:gpt-6.1-terra and openai:gpt-6.1-sol to --models once an OpenAI key is set up. ModelPricing has their input and output list prices (per million tokens: Luna $0.10 / $0.50, Terra $1 / $5, Sol $2 / $10); cached input is priced as full input, so their cost_usd is an upper bound. Any other model without a price has a blank cost_usd.
• The report holds the requests and the generated copy. Keep it on your machine; the worker's log only has case ids, models, reason codes, tokens and costs.