Skip to content

Observability

Every Badger JVM service exposes Micrometer metrics over Actuator. In the k3s cluster vmagent scrapes them into VictoriaMetrics, Promtail ships the logs into Loki, and Grafana reads both.

This page covers what the applications expose and how to add to it. The dashboards and scrape configuration live in the k3s-stack repository under flux/infrastructure/monitoring/.

How a service gets scraped

Three things have to line up. Miss any one and the service is invisible.

  1. micrometer-registry-prometheus on the classpath. commerce-core already declares it, so every module that depends on commerce-core inherits it.
  2. prometheus exposed over HTTP, via management.endpoints.web.exposure.include.
  3. A management port the scraper can reach, declared to Kubernetes with prometheus.io/scrape, prometheus.io/port and prometheus.io/path annotations on the pod template — annotations on the Service are not enough, because discovery is pod-level.

web, rest and auth all run their management server on MANAGEMENT_PORT (9090 in the cluster), separate from the public application port, so actuator is never reachable from the internet.

badger-worker exports no metrics

The worker sets spring.main.web-application-type: none, so no HTTP server starts and Actuator has nowhere to publish. It shows up on the dashboards with replica counts, restarts, CPU, memory and logs — all from Kubernetes and Loki — and nothing JVM-level. Giving it metrics means giving it a management server, which is a deliberate change to a process that currently opens no ports.

What is exposed today

Area Metrics Useful because
HTTP http_server_requests_seconds_* tagged uri, method, status, outcome, exception Rate, errors and duration per endpoint
Logging logback_events_total{level} Error counts at the real level, counted in-process — no log-scraping regex involved
JVM jvm_memory_*, jvm_gc_*, jvm_threads_*, jvm_classes_*, process_cpu_usage Heap pressure, GC cost, thread saturation
Startup application_started_time_seconds, application_ready_time_seconds The number to watch when tuning image layering or CDS
Caches cache_gets_total{cache,result}, cache_size, cache_evictions_total Hit ratio per named cache
Redis lettuce_seconds_* Command rate and latency
MongoDB spring_data_repository_invocations_seconds_* tagged repository, method, state Per-repository latency and error rate
Messaging rabbitmq_* Publish/consume flow, rejects, unrouted messages
Web server jetty_threads_*, jetty_connections_* Thread-pool saturation
Scheduled work tasks_scheduled_execution_seconds_* Whether scheduled jobs run and how long they take
Per-tenant badger_performance_seconds* tagged method, siteId The performance monitor aspect
Browser errors badger_client_errors_total tagged siteId, kind; badger_client_errors_dropped_total tagged reason Script failures in shoppers' browsers, per shop (below)

The Mongo driver meters (mongodb_driver_pool_*, mongodb_driver_commands_seconds_*) come from Spring Boot's MongoMetricsAutoConfiguration, which contributes its command and pool listeners as MongoClientSettingsBuilderCustomizers. MongoConfiguration builds the client itself, so it applies those customizers explicitly (it used to discard the settings object entirely, and the meters never appeared). They sit below the Spring Data repository timer and include queries that don't go through a repository — e.g. @DBRef resolution. See Caching for the cache metrics that matter most.

Latency percentiles

Micrometer publishes timers as _count, _sum and _max only. Without histogram buckets, histogram_quantile() returns nothing at all — so a p95 panel or alert built on http_server_requests_seconds_bucket silently shows no data rather than failing loudly.

web, rest and auth therefore configure explicit boundaries:

management:
  metrics:
    distribution:
      slo:
        http.server.requests: 25ms,50ms,100ms,200ms,400ms,800ms,1s,2s,5s,10s
      minimum-expected-value:
        http.server.requests: 10ms
      maximum-expected-value:
        http.server.requests: 10s

These are SLO boundaries rather than percentiles-histogram: true on purpose. The full histogram generates roughly 40–70 buckets per series; across a couple of hundred endpoint/status/method combinations that is tens of thousands of extra series. Ten boundaries cost about 11 series per combination and give percentile estimates that are accurate in the range anyone acts on.

If you add a timer that needs percentiles, add its name here rather than turning on percentile histograms globally.

Adding a metric

Inject MeterRegistry and register a meter. Keep tag values bounded — a tag that can take user-supplied values (an order id, a URL, an email) multiplies series count without limit and is the usual cause of a monitoring system falling over.

@Service
@RequiredArgsConstructor
public class RefundService {

  private final MeterRegistry meterRegistry;

  public void refund(Order order) {
    // ...
    meterRegistry.counter("badger.refunds", "reason", reason.name()).increment();
  }
}

badger.refunds is exported as badger_refunds_total. Micrometer replaces dots with underscores and appends a suffix per meter type, so name meters in dotted form and read them back in underscored form.

The performance monitor aspect

PerformanceMonitor times every public method on *Service, *Repository, *Renderer and *Configurator classes, plus anything annotated @Monitored, and tags each timing with the method signature and the current tenant's siteId.

Those timings reach Grafana as badger_performance_seconds*, which is what drives the per-tenant panels. The admin screen at /system/performanceStatistics reads the same data out of Redis for a live view; Grafana keeps the history.

The aspect wraps hundreds of calls per storefront page, so it keeps one timer per (method, siteId) and reuses it: a call costs a map lookup, not a new meter id and registration. PerformanceMonitorService.clearPerformanceStatistics removes the timers and calls PerformanceMonitor.reset() so the next call registers fresh ones.

Because the aspect tags by siteId, this is the one place where tenant count drives cardinality. That is intentional and worth the cost, but it is a reason not to add further high-cardinality tags to the same meter.

Browser errors

The browser error beacon reports uncaught script errors and unhandled promise rejections from shoppers' browsers to POST /api/client-errors on the shop's own host, served by web. Each report is scrubbed of anything that could identify a person before it is counted or logged.

  • badger_client_errors_total{siteId, kind} counts reports per shop; kind is error or rejection. A rise after a release on one shop, or on every shop, is the thing to alert on.
  • badger_client_errors_dropped_total{reason} counts reports the endpoint threw away: invalid (not a report, or over 8 KB) and rate_limited (over the per-shop or per-pod cap).
  • Each counted report is one WARN line from ClientErrorServiceImpl, starting Client error, with a fingerprint, the shop, the release, the page type, the script and line, the scrubbed message and the first stack frames. In Loki, filter the web pods' logs with |= "Client error" (and |= "on site shop-a" for one shop); the fingerprint groups the same error.

The endpoint is a storefront endpoint, outside the REST API's OpenAPI specification: it has to be on the shop's own host so that the shop comes from the Host header rather than from the report. A global setting, client-error-beacon-enabled, switches it off for every shop.

Logs

Logback writes to stdout; Promtail tails the container logs and ships them to Loki with namespace, pod, container, node and app labels. Log lines are stored as plain text — no JSON parsing — so LogQL filters use line matchers (|~ "(?i)error") rather than label filters over parsed fields.

For counting errors, prefer logback_events_total{level="error"} over matching log text: it is counted in-process at the real level, so it cannot be fooled by the word "error" appearing in a stack trace or a URL.

No personal data in logs

A log line should say what happened and where, never exactly who it happened to, and never a secret.

Never log personal data or secrets, at any level, including debug. That means no emails, names, phone numbers, addresses, postcodes, IP addresses, Gift Aid or payment details (including gift card numbers and PINs), passwords, tokens, session IDs, API keys, or raw webhook/HTTP/queue payloads. Loki keeps log lines after the data they describe has been deleted, so a logged email survives a GDPR erasure. Don't log whole domain objects: Lombok's toString() prints every field.

Object IDs are fine, and every line about a person, order or payment should carry them (userId, orderId, donationId, order numbers, Stripe evt_…/pi_… IDs). They let a sysadmin investigate: take the ID from the log and look up the record in the admin tools, where access is controlled and the data is current. Tenant identifiers (siteId, catalogueId, the site name) are fine too: they name the business, not a person.