Skip to content

Synthetic Traffic

Ranking by what sells, learned boosts and search experiments only show anything once a shop has real traffic: a thousand orders a month before learned ranking switches on, and thousands of visitors per arm before an experiment says anything. No demo shop has that. The synthetic traffic simulator fills the gap: simulated shoppers with known, hidden preferences browse a demo shop's real listings, and the report says whether learned ranking found what they like and whether an experiment told its arms apart.

It is Phase 4, part C, of docs/implementation-plan/search-experiments.md.

Safety

The simulator writes engagement counters and experiment counters for days that never happened, so it is fenced off from real shops:

  • Allow-list. A run is only queued, and only simulated, for a site whose id is in badger.synthetic-traffic.allowed-sites (comma-separated). It is empty by default, so nowhere. The dev profile (the local stack started by dev-scripts) sets halden in the web and worker application.yml; every cluster runs the production profile and never sets it. Both the web app (when queueing) and the worker (when running) check it.
  • Superadmin only. The page lives under /system (SUPERADMIN), and SyntheticTrafficService.queue checks the role again.
  • Nothing real is created. Orders exist in memory only: no order or user documents, no email, no payment, no stock change, no search-term analytics. What it writes: product engagement counters (productEngagementDaily), search experiment counters (searchExperimentDaily), the run document (syntheticTrafficRuns), learned ranking scores (as the nightly job would) and one site setting, below.
  • The orders gate. Learned ranking needs search-ranking-min-monthly-orders real orders, which simulated orders aren't. A run sets the superadmin-only site key search-ranking-gate-override (not editable, not visible) on its site, which waives that minimum (the entitlement still applies), and the report says so. Shop admins can't see or change the key (it is left out of the Settings pages and MCP, and the admin config API doesn't know it), and site transfer never carries it. Clear it from the same page.

Running it locally against Halden

  1. Start the stack and seed the demo shops: ./dev-scripts/devEnv.sh, then ./dev-scripts/seedDemoSites.sh halden. Halden is a trade supplier with about 210 products and a merchandising tree of 25 category pages under /c, at https://halden.bdgr.localhost. Make sure its search index is built (the category pages list products).
  2. The dev profile already allows halden. To allow another demo shop, set badger.synthetic-traffic.allowed-sites in your application-dev.yml for both the web app and the worker, and restart them.
  3. Sign in as a superadmin and open System > Synthetic traffic (https://bdgr.localhost/system/synthetic-traffic).

  4. Pick the shop and the traffic, and Queue run:

    Field Default Meaning
    Days 28 Past days to fill in, oldest first, ending yesterday (1 to 90)
    Shoppers a day 400 Each looks at one listing page
    Seed 42 The same seed, catalogue and listings give the same shoppers and preferences
    Hidden gems 10 Rarely seen products shoppers secretly love
    Refresh ranking every 7 days Recompute learned ranking as it goes; 0 only at the end
    Count through an experiment off Assign and count shoppers through a search experiment

  5. The worker picks the run up within 30 seconds. The page refreshes itself while a run is in progress; a 28 × 400 run takes a few minutes (listings are cached between refreshes, so it is a few hundred searches, not one per shopper).

  6. Open Report when it's done. Then browse Halden's categories: the gems should be near the top.

Running it again on the same days adds to the first run's counters rather than replacing them (the counters are daily $incs). To start clean, use a different shop or wait for the data to age out (engagement is kept 90 days).

How a run works

SyntheticTrafficSimulator (commerce-core, search.synthetic), run by the worker's SyntheticTrafficRunner:

  1. Traits. It reads the catalogue in its current Featured order and gives every product a hidden appeal (the chance a shopper who looks at it opens it, mostly 4% to 30%) and a hidden propensity (the chance an opened product is added to the basket, 8% to 30%). The hidden gems are drawn from the lower half of that order, the products shoppers rarely see, and get 45% to 65% appeal and 35% to 55% propensity (ShopperModel).
  2. Shoppers. Each simulated shopper picks a category of the primary merchandising tree (65%) or a search term (35%: the shop's top searches that found something, and words common to product names), and gets the real first page of that listing from MerchandisingQueryService, in Featured order, exactly as the storefront asks for it, with their experiment arm if they have one.
  3. Behaviour is a cascade click model: they look down the page from the top, open each product with its appeal, carry on looking with 40% after opening one and 88% after passing one (so the chance of even looking at a product falls with its position, the position bias learned ranking has to take out), and open at most three. Each opened product is added with its propensity; a shopper with something in the basket orders it 55% of the time, at the products' real prices.
  4. Recording. Impressions, clicks, adds and orders go through ProductEngagementService's …ForDay methods (the same buffer and bulk writes as real traffic, under the simulated day); an enrolled shopper's visit, clicks, adds and order through SearchExperimentService.recordForDay.
  5. Refreshing. A listing reflects the ranking as it is now, not as it was on the simulated day, so the run recomputes learned ranking (the nightly job's own code, LearnedRankingJob.refreshCatalogue) every refreshEveryDays days and after the last day; later days then see the improving order, which is how the gems get found. With 0, every day sees the order the shop had at the start.
  6. Experiment. With Count through an experiment, shoppers are assigned with SearchExperimentService.armFor, exactly as real visitors are. If the shop has a running synthetic experiment (left over from an earlier run) it is used and left running; if it has a real one, the run does not count through any experiment, so fake shoppers never reach a real experiment's results (the report note says so); otherwise the run creates one, marked synthetic: learned ranking (control) against units sold (treatment), starts it, and stops it at the end (its results are kept). Note the arms only differ once learned ranking is active, i.e. after the first refresh. If a run fails or its worker dies, the experiment it started is stopped too: straight away on a failure, or when the run is failed as stale (below).
  7. One run per shop. A shop can have one run queued or running at a time; this is enforced by a unique partial index (one_active_per_site_idx, on the run's active flag), so two superadmins queueing at once can't both succeed.

Reading the report

Number What it means
Shoppers, impressions, clicks, adds, orders, revenue What was simulated. Orders and revenue were never saved anywhere but the counters
Hidden gems, median Featured position The gems' median position across every category they're in (all their (gem, category) placements), before the run and after the final refresh. Lower is better: from the bottom half to the first page means learned ranking found them
Learned score vs hidden appeal Spearman rank correlation between each scored product's learned score for the shop's goal (search-ranking-objective) and its hidden appeal. 0 is no relation, 1 is ranking exactly by appeal. Clearly positive means the score is picking up what shoppers like; it can't reach 1, because the score also weighs propensity and price, and products seldom shown have little evidence. More traffic gives a stronger signal: 4 weeks of 400 shoppers a day on a 120-product catalogue gives about 0.35, 2,000 a day about 0.5 (SyntheticLearningTest)
vs hidden value The same against what the score actually estimates: appeal × propensity (× price for revenue and margin)
Learned ranking Whether the catalogue qualified at the end. Not qualified usually means Typesense isn't configured for the shop, or the shop isn't entitled
Experiment Visitors and revenue per visitor per arm, the lift, and the verdict from the experiment's own results (EARLY until both arms reach the planned sample size)

Each gem's own row gives its appeal and propensity, its median position before and after, and its learned score (unseen when it was never shown).

Code

  • commerce-core/.../search/synthetic/: SyntheticTrafficRun (document), SyntheticTrafficRunRepository (only the service uses it; atomic claim by findAndModify), SyntheticTrafficService(Impl) (allow-list, role check, queue, claim, progress, gate override), SyntheticTrafficSimulator (the run), ShopperModel (traits and click model), SyntheticStats (median, Spearman), SyntheticTrafficReport.
  • worker/.../search/SyntheticTrafficRunner: polls every 30 seconds (badger.search.synthetic-traffic.poll-ms, off with badger.search.synthetic-traffic.enabled=false), runs one run at a time on its own thread with the shop's tenant context, and fails runs whose worker went quiet for 30 minutes (the run beats before each learned ranking refresh and after every simulated day; a late completion of a run already failed as stale is ignored), stopping the synthetic experiment such a run left running.
  • web-admin: SyntheticTrafficController, templates/system/synthetic-traffic*.html.
  • Tests: ShopperModelTest, SyntheticStatsTest, SyntheticTrafficServiceImplTest (refusals, claim), SyntheticTrafficSimulatorTest (past days only, gate override, experiment), SyntheticLearningTest (with enough simulated traffic the real LearnedRankScorer recovers the hidden appeal and lifts the gems, no Mongo or Typesense), SyntheticTrafficRunnerTest, SyntheticTrafficRenderTest; the override in ProductEngagementServiceImplTest and LearnedRankingJobTest.