Skip to content

Anonymous Visitors and Bots

Every storefront visitor gets a User, signed in or not. This page covers how anonymous ones are created, how bots are told apart, and how both are cleared away.

Creating a visitor

UserResolutionFilter gives a first-time visitor a transient user and a signed cookie. The user is saved (by the worker, see Visitor Identity) only if the response is a 2xx (handlers that redirect on a first request, such as QR direct checkout, save it early with UserResolutionFilter.persistDeferredUser). Requests to /error never save one, and nor do a bot's GET/HEAD requests or a GET/HEAD of a listing page (/c/**, /collection/**, /search): the visitor keeps the id in their cookie and is saved by their next request that is worth keeping, such as a product page or adding to the basket.

Until October 2026 every crawler hit that got a page created a user record. On DMRT that was about 1,300 a day, nearly all single visits, and a crawler walking Halden's facet combinations wrote one on every request.

Bot detection

BotDetectionService.isBot(request, siteContext) decides, in order:

  1. Cloudflare bot score: a CF-Bot-Score header below botScoreThreshold (default 30). The header only arrives with Cloudflare Bot Management, so most requests don't have it.
  2. Built-in pattern (AUTOMATED_USER_AGENT): words only automated clients use (bot, crawl, spider, siteaudit, externalhit, curl/, python-, headless, ...), a bare Mozilla/5.0, or no user agent. It applies on every site and a site setting can't turn it off.
  3. The site's list botUserAgentStrings (security settings): extra parts separated by :::, matched ignoring case. A site value replaces the default list, which is why the built-in pattern exists: a site set to Better Uptime Bot:::sitepulse would otherwise miss every search engine.

A visitor flagged as a bot is still served the page. The flag means:

  • reading pages saves no user at all; a request that posts saves one already marked for deletion (see below);
  • direct checkout links redirect to the product page instead of building a basket;
  • search clicks, impressions and experiments ignore the request.

Keep the pattern free of false positives. A real shopper flagged as a bot loses their basket when the hourly job deletes them. Add a word only if no browser sends it; for example bot is written (?<!cu)bot because Cubot phones put CUBOT in their user agent. To check a change, compare it against the user agents in pageViewHistory.pagesViewed.userAgent for real customers (ITEM_ADDED and ORDER_SUBMITTED events record no user agent, so ignore empty ones there). BotDetectionServiceTest holds examples of both kinds.

Clearing visitors away

Two worker jobs, both in worker/.../admin/:

Job Schedule Does
InactiveOrderAndUserCleaner (inactiveUsers) hourly at :30 Marks for deletion users with no login name and no email whose lastActiveDate is older than numberOfDaysToKeepInactiveUsers (global, default 30). Users with placed orders are kept, and given the contact details from their latest order.
MarkedForDeletionOrderAndUserCleaner (markedForDeletion) hourly on the hour Deletes marked users with their deletable orders and page history; a user with placed orders is obfuscated instead.

Bots skip the first job: the few that are saved are saved already marked, so they go within the hour. Anonymous visitors the detection misses stay for the full inactivity window, so the number of anonymous users on a busy site is roughly a month of undetected crawler visits. That is expected, not a stuck job. The jobs log Gertcha job <name> finished ... with counts each run.

When counting users in Mongo, the tenant is siteGroupId, not siteId.