Search Experiments¶

A search experiment splits a share of the shop's visitors between two ways of ranking the Featured order of search and category pages, and compares what each earns per visitor. It answers "does ranking for margin (or turning off learned boosts, or ranking by conversion) actually make us money?" with a measurement rather than a hunch. It is Phase 4 of the plan in docs/implementation-plan/commerce-search-optimisation.md (design: docs/implementation-plan/search-experiments.md).

It builds on product engagement, ranking by what sells and learned boosts: the two arms are different combinations of those.

What an experiment compares¶

Each experiment has a control (usually the shop as it is) and a treatment (the change being tested). An arm is three choices:

The two arms must rank differently. An arm that ranks by what sells only takes effect where that is active for the shop (Enterprise, enough orders, switched on); elsewhere it ranks as units sold, and the admin says so on the form. A merchandiser's pins, boosts and buries apply in both arms.

Setting one up¶

In Search > Experiments (/admin/search/experiments), New experiment:

1. Name it and say what you expect (the hypothesis is for people reading the results later).
2. Choose the control and the treatment.
3. Choose visitors in the experiment (default 100%; the rest see the shop as normal and aren't counted), of those, in the treatment (default 50%) and planned visitors per arm (default 1,000).
4. Save the draft, then Start. A draft can be edited or deleted; a started experiment can only be stopped, and a stopped one is kept for its results and never restarted.

Only one experiment runs per shop. Starting a second is refused rather than stopping the first. Experiments need the same entitlement as ranking by what sells (merchandisingRules, Enterprise) and the Search Experiments setting (search-experiments-enabled, default on, Search settings). Without either nobody is put in an arm and nothing is counted, but experiments and their results can still be read; the admin shows a notice. There is no minimum number of orders, but there is a real cost to a test with little traffic: expect to wait weeks for a few hundred orders to show anything.

How visitors are split¶

A visitor is the shopper's user id: the BDGR_USR cookie id for a guest, the account id once signed in. Their arm is a fixed function of that id and the experiment's random salt (a hash, nothing is stored), so they see the same arm every time and two experiments split visitors independently. Bots are never put in an experiment.

A visitor is counted (as one visitor in their arm) the first time they see a Featured listing, a search results page or a category page on the default sort. A shopper who picks another sort sees that page ordered as they asked and it isn't counted. Their later listing views, clicks, adds to basket and orders are counted for their arm. An order counts for the arm the basket was built in, even if signing in moves the shopper to another arm before they pay.

Reading the results¶

Open an experiment from the list (running or stopped). The page shows:

• The verdict in plain English, based on revenue per visitor: too early, no clear difference, treatment better or control better.
• The two arms: visitors, listing views, click-through (clicks per listing view), search exit rate (the share of search results pages nothing was opened from), adds to basket, orders, conversion (orders per visitor), revenue and revenue per visitor.
• Treatment against control for conversion and revenue per visitor: the difference, its 95% range, the relative lift and its range, and a p-value. If the range includes zero there may be no difference at all.
• A chart of revenue per visitor so far, by day, for each arm.
• Warnings: an early notice until both arms have the planned visitors (don't stop on a lucky day: early gaps are very often chance), and a sample ratio mismatch warning when visitors split far from the planned share, which means something other than chance is sorting visitors and the comparison can't be trusted. A Synthetic badge marks an experiment run on simulated shoppers (see Synthetic Traffic).

Counts are saved every few seconds, so a very recent order may not show yet.

Things to know¶

• Visitor identity. A guest's id lasts as long as their BDGR_USR cookie (about 28 hours); after it expires a returning guest may be counted as a new visitor, in either arm. Signing in swaps a guest id for the account id, so a shopper can change arm once. Both are small and fall equally on both arms, but they mean visitors are a little over-counted and effects a little diluted.
• Exposure dedupe. "First time seen" is a short-lived hashed Redis key per experiment and visitor (30-day TTL, keyed with the experiment salt: no raw ids are stored anywhere). Each time an exposed visitor views a listing the key's expiry is pushed out another 30 days (they are still counted as a visitor only once), so only a visitor who stays away for 30 days is counted as a visitor again; a long experiment counts people, not three-day windows. While Redis is unreachable nothing is counted, for both arms.
• Clicks count from Featured listings only. A listing's sr reference carries whether the page was ranked by an experiment arm; a click from a page that wasn't (another sort, or a visitor not in the experiment) is still counted for product engagement but not for the experiment. References issued before this existed are treated as not Featured.
• Changes take up to 30 seconds to reach every node. Each web node remembers a site's running experiment for 30 seconds, so after a start or stop the other nodes keep assigning (or not assigning) arms for up to that long. The node that handled the change updates at once. Starting and stopping are conditional writes: if two people change an experiment at the same moment, the second gets a conflict and should reload.
• Demo sites and synthetic runs. While a synthetic traffic run is active on a demo shop, its synthetic experiment is a real running experiment: real visitors to that shop can be enrolled in it (their visits are counted in it alongside the simulated ones). A run never counts through a real experiment, and the synthetic one is stopped when the run ends, fails or is failed as stale.
• Featured sort only. The experiment changes, and counts, only the default sort of search and category pages. Other sorts, product pages found any other way, and the basket aren't part of it.
• Clicks need engagement tracking. Listing clicks and adds come from the same tracking as product engagement (product-engagement-tracking-enabled). Without it visitors, listing views, orders and revenue are still counted (so the verdict still works), but click-through, search exit and adds stay at zero.
• Revenue per visitor is slightly over-confident. The variance of revenue per visitor is built from each order's revenue squared, treating every order as a different visitor's spend. A visitor who places several orders in the experiment is really one larger spend, so the sum of squares is a little too small, the variance understated and the 95% range and p-value slightly too narrow (the verdict too keen). The effect is small unless repeat orders are common; the planned sample sizes and the too early rule do not remove it, so read a borderline verdict with that in mind. The per-visitor spend is not stored (nothing per visitor is), so it can't be corrected without storing more.
• Plain statistics, and no peeking. Conversion uses a two-proportion z-test and revenue per visitor Welch's test, both normal approximations that are fine at the planned sample sizes. There is no correction for looking repeatedly, so the verdict stays too early until both arms reach the planned visitors. Decide the planned size up front and read the result once it is in.
• Money is in minor units in REST and MCP, and formatted in the shop's currency in the admin.
• Privacy. Nothing per visitor is stored beyond the hashed exposure key; counters are daily totals per arm. Logs carry the site and experiment ids only.

Managing them¶

• Admin: Search > Experiments, as above. The roles that can use Learned boosts can use it.
• REST: GET and POST /v1/admin/search/experiments, GET, PUT (drafts) and DELETE (drafts) /v1/admin/search/experiments/{experimentId}, POST .../{experimentId}/start and .../stop, and GET .../{experimentId}/results. An unknown id is 404, a change the experiment's state forbids (not a draft, another running, not running) 409, and an invalid set-up 400. See the API reference.
• MCP: productRanking action experiments (the list; with id, one with its results); manageProductRanking actions createExperiment (name, hypothesis, the treatment from goal, learnedRanking and learnedBoosts; the control is the shop as it is; trafficPercent, treatmentPercent, visitorsPerArm), startExperiment and stopExperiment (id). See MCP.

Where the code is¶

commerce-core search/experiment: SearchExperimentService (managing, assignment, recording), ExperimentStatistics (the maths), SearchExperimentDailyStat (counters). The storefront hooks are in SearchClickTracker (web-mvc) and OrderServiceImpl. Admin: SearchExperimentsAdminRenderer and SearchExperimentsRestAdminController (web-admin); REST: SearchExperimentsAdminController; MCP: ProductRankingTools.

Tests¶

SearchExperimentServiceImplTest and ExperimentStatisticsTest (core), SearchExperimentsRenderTest (admin pages), SearchExperimentsAdminControllerTest (REST), ProductRankingToolsTest (MCP), SearchClickTrackerTest and OrderServiceSearchConversionTest (recording).