Observability¶
Every Badger JVM service exposes Micrometer metrics over Actuator. In the k3s cluster vmagent scrapes them into VictoriaMetrics, Promtail ships the logs into Loki, and Grafana reads both.
This page covers what the applications expose and how to add to it. The
dashboards and scrape configuration live in the k3s-stack repository under
flux/infrastructure/monitoring/.
How a service gets scraped¶
Three things have to line up. Miss any one and the service is invisible.
micrometer-registry-prometheuson the classpath.commerce-corealready declares it, so every module that depends oncommerce-coreinherits it.prometheusexposed over HTTP, viamanagement.endpoints.web.exposure.include.- A management port the scraper can reach, declared to Kubernetes with
prometheus.io/scrape,prometheus.io/portandprometheus.io/pathannotations on the pod template — annotations on the Service are not enough, because discovery is pod-level.
web, rest and auth all run their management server on MANAGEMENT_PORT
(9090 in the cluster), separate from the public application port, so actuator is
never reachable from the internet.
badger-worker exports no metrics
The worker sets spring.main.web-application-type: none, so no HTTP server
starts and Actuator has nowhere to publish. It shows up on the dashboards
with replica counts, restarts, CPU, memory and logs — all from Kubernetes and
Loki — and nothing JVM-level. Giving it metrics means giving it a management
server, which is a deliberate change to a process that currently opens no
ports.
What is exposed today¶
| Area | Metrics | Useful because |
|---|---|---|
| HTTP | http_server_requests_seconds_* tagged uri, method, status, outcome, exception |
Rate, errors and duration per endpoint |
| Logging | logback_events_total{level} |
Error counts at the real level, counted in-process — no log-scraping regex involved |
| JVM | jvm_memory_*, jvm_gc_*, jvm_threads_*, jvm_classes_*, process_cpu_usage |
Heap pressure, GC cost, thread saturation |
| Startup | application_started_time_seconds, application_ready_time_seconds |
The number to watch when tuning image layering or CDS |
| Caches | cache_gets_total{cache,result}, cache_size, cache_evictions_total |
Hit ratio per named cache |
| Redis | lettuce_seconds_* |
Command rate and latency |
| MongoDB | spring_data_repository_invocations_seconds_* tagged repository, method, state |
Per-repository latency and error rate |
| Messaging | rabbitmq_* |
Publish/consume flow, rejects, unrouted messages |
| Web server | jetty_threads_*, jetty_connections_* |
Thread-pool saturation |
| Scheduled work | tasks_scheduled_execution_seconds_* |
Whether scheduled jobs run and how long they take |
| Per-tenant | badger_performance_seconds* tagged method, siteId |
The performance monitor aspect |
| Browser errors | badger_client_errors_total tagged siteId, kind; badger_client_errors_dropped_total tagged reason |
Script failures in shoppers' browsers, per shop (below) |
The Mongo driver meters (mongodb_driver_pool_*,
mongodb_driver_commands_seconds_*) come from Spring Boot's
MongoMetricsAutoConfiguration, which contributes its command and pool
listeners as MongoClientSettingsBuilderCustomizers. MongoConfiguration
builds the client itself, so it applies those customizers explicitly (it used
to discard the settings object entirely, and the meters never appeared). They
sit below the Spring Data repository timer and include queries that don't go
through a repository — e.g. @DBRef resolution. See Caching for
the cache metrics that matter most.
Latency percentiles¶
Micrometer publishes timers as _count, _sum and _max only. Without
histogram buckets, histogram_quantile() returns nothing at all — so a p95 panel
or alert built on http_server_requests_seconds_bucket silently shows no data
rather than failing loudly.
web, rest and auth therefore configure explicit boundaries:
management:
metrics:
distribution:
slo:
http.server.requests: 25ms,50ms,100ms,200ms,400ms,800ms,1s,2s,5s,10s
minimum-expected-value:
http.server.requests: 10ms
maximum-expected-value:
http.server.requests: 10s
These are SLO boundaries rather than percentiles-histogram: true on purpose.
The full histogram generates roughly 40–70 buckets per series; across a couple of
hundred endpoint/status/method combinations that is tens of thousands of extra
series. Ten boundaries cost about 11 series per combination and give percentile
estimates that are accurate in the range anyone acts on.
If you add a timer that needs percentiles, add its name here rather than turning on percentile histograms globally.
Adding a metric¶
Inject MeterRegistry and register a meter. Keep tag values bounded — a tag that
can take user-supplied values (an order id, a URL, an email) multiplies series
count without limit and is the usual cause of a monitoring system falling over.
@Service
@RequiredArgsConstructor
public class RefundService {
private final MeterRegistry meterRegistry;
public void refund(Order order) {
// ...
meterRegistry.counter("badger.refunds", "reason", reason.name()).increment();
}
}
badger.refunds is exported as badger_refunds_total. Micrometer replaces dots
with underscores and appends a suffix per meter type, so name meters in dotted
form and read them back in underscored form.
The performance monitor aspect¶
PerformanceMonitor times every public method on *Service, *Repository,
*Renderer and *Configurator classes, plus anything annotated @Monitored,
and tags each timing with the method signature and the current tenant's
siteId.
Those timings reach Grafana as badger_performance_seconds*, which is what
drives the per-tenant panels. The admin screen at
/system/performanceStatistics reads the same data out of Redis for a live
view; Grafana keeps the history.
The aspect wraps hundreds of calls per storefront page, so it keeps one timer
per (method, siteId) and reuses it: a call costs a map lookup, not a new meter
id and registration. PerformanceMonitorService.clearPerformanceStatistics
removes the timers and calls PerformanceMonitor.reset() so the next call
registers fresh ones.
Because the aspect tags by siteId, this is the one place where tenant count
drives cardinality. That is intentional and worth the cost, but it is a reason
not to add further high-cardinality tags to the same meter.
Browser errors¶
The browser error beacon reports uncaught script errors and unhandled
promise rejections from shoppers' browsers to POST /api/client-errors on the shop's own host,
served by web. Each report is scrubbed of anything that could identify a person before it is
counted or logged.
badger_client_errors_total{siteId, kind}counts reports per shop;kindiserrororrejection. A rise after a release on one shop, or on every shop, is the thing to alert on.badger_client_errors_dropped_total{reason}counts reports the endpoint threw away:invalid(not a report, or over 8 KB) andrate_limited(over the per-shop or per-pod cap).- Each counted report is one
WARNline fromClientErrorServiceImpl, startingClient error, with a fingerprint, the shop, the release, the page type, the script and line, the scrubbed message and the first stack frames. In Loki, filter the web pods' logs with|= "Client error"(and|= "on site shop-a"for one shop); the fingerprint groups the same error.
The endpoint is a storefront endpoint, outside the REST API's OpenAPI specification: it has to be
on the shop's own host so that the shop comes from the Host header rather than from the report.
A global setting, client-error-beacon-enabled, switches it off for every shop.
Logs¶
Logback writes to stdout; Promtail tails the container logs and ships them to
Loki with namespace, pod, container, node and app labels. Log lines are
stored as plain text — no JSON parsing — so LogQL filters use line matchers
(|~ "(?i)error") rather than label filters over parsed fields.
For counting errors, prefer logback_events_total{level="error"} over matching
log text: it is counted in-process at the real level, so it cannot be fooled by
the word "error" appearing in a stack trace or a URL.
No personal data in logs¶
A log line should say what happened and where, never exactly who it happened to, and never a secret.
Never log personal data or secrets, at any level, including debug. That
means no emails, names, phone numbers, addresses, postcodes, IP addresses, Gift
Aid or payment details (including gift card numbers and PINs), passwords, tokens, session IDs, API keys, or raw
webhook/HTTP/queue payloads. Loki keeps log lines after the data they describe
has been deleted, so a logged email survives a GDPR erasure. Don't log whole
domain objects: Lombok's toString() prints every field.
Object IDs are fine, and every line about a person, order or payment should
carry them (userId, orderId, donationId, order numbers, Stripe
evt_…/pi_… IDs). They let a sysadmin investigate: take the ID from the log
and look up the record in the admin tools, where access is controlled and the
data is current. Tenant identifiers (siteId, catalogueId, the site name)
are fine too: they name the business, not a person.