Skip to content

Product Engagement

How each product does in each listing: how often it is shown, where, how often shoppers open it, add it to a basket and buy it, and what it earns per listing slot. This is the measurement behind ranking by what sells, which orders listings by it on qualifying shops, and merchandisers can act on it directly.

It is an Enterprise feature: it records only on a site whose subscription grants merchandisingRules (while entitlement enforcement is off, every site does), with the product-engagement-tracking-enabled site setting on (Search settings, Record Product Engagement, default on).

What is counted

Per site, per day, per product (by SKU; a variant counts for its product) and per context:

Context Meaning
all Everything: the product's totals across every listing, and every product page view however the shopper got there
c:<nodeId> A category page (/c/..., or a filtered landing page under it) browsed without a search
q:<term> The results of a search for the term, on /search or inside a category (/c/...?q=)

A listing's activity counts twice, in its own context and in all.

Counter Counted when
impressions A listing page is rendered for a shopper: one per product on the page
positionSum, impressionBands The product's 1-based position across pages (31 on page 2 of 30), summed for the average and counted in bands 1–4, 5–12, 13–24, 25–48 and 49+
clicks, clickBands The product page is opened from a listing: once per listing page and product, at the position it was shown
views A product page view. In all, every view; in a listing's context, the views that came from it (equal to its clicks)
addToBaskets The product is added to a basket (in the listing it was added from, too)
orders, units, revenue An order is placed containing it: one order per product, its units, and its line totals in minor units
marginRevenue, margin Of those lines, the revenue whose cost price was known when the order was placed, and that revenue less its cost. Commercially sensitive: shown to admin and finance roles only, never through MCP

Reports derive the rates: click-through is clicks per impression, add-to-basket and conversion are adds and orders per view, and revenue per thousand impressions is what a listing slot earned. The position bands are there because shoppers click the first few products far more whatever they are: a product's click rate is only fair against what its positions usually earn, which the ranking will need (Phase 3 of the plan).

Not counted: requests flagged by bot detection (BotDetectionService), the filter sheet's live count previews (X-Badger-Count-Preview), which nobody sees, and listings with no products. Legacy /collection/... pages aren't counted yet.

How it works

The storefront end is SearchClickTracker (web-mvc), which also does search term click-through:

  1. SearchController and MerchandisingPageController call recordImpressions with the page's products and the first one's position, and put two things on the model: searchRef, an sr reference for the listing (SearchRefCodec: the listing's kind, its term or node id and the time, encrypted and authenticated with AES-GCM, bound to the site, valid 30 minutes), and listingPositions, a JSON map of each product's seoName to its position. The script in fragment/search-click-ref (included by every theme's searchResults.html and merchandisingNode.html) adds ?sr= and &sp=<position> to the product links and a hidden sr field to add-to-basket forms. It sends nothing itself. A reference has a kind: a /search term (q), a search inside a category (w) or a category browsed (c); only /search references count towards search term analytics.
  2. ProductController calls recordClick with the product's SKU: a view always, and with a valid reference a click in that listing at sp, once per listing page and product (a bounded, expiring in-memory set of references already seen; a reload is a view, not another click).
  3. BasketController calls recordAddToBasket. The reference is the posted sr, or the one on the same-host page that posted the form (its Referer). Added from a listing, the basket notes {p: productSkuId, c: context, at} in orderProperties.listingAttribution (at most 20 products, the latest listing winning a product).
  4. OrderServiceImpl.submitOrder, after a successful submit, calls ProductEngagementService.recordOrderPlaced: orders, units and revenue per product in all, and in the listing each was added from within 24 hours. The attribution is then removed from the order in memory and with a targeted $unset, so a placed order keeps no listing history.

ProductEngagementServiceImpl (commerce-core, search.engagement) buffers everything in memory by (site, day, context, product) and writes it every 10 seconds (badger.search-analytics.flush-interval-ms) and at shutdown as one unordered bulk of $inc upserts into productEngagementDaily, unique on (siteId, context, day, product), so several web nodes add to the same bucket. The buffer holds at most 50,000 entries between flushes; past that new ones are dropped (buffered ones still count). A failed write is retried on the next flush. A TTL index deletes each bucket 90 days after its day. Whether a site records (the setting and the entitlement, a database read) is remembered per site for a minute. Recording never fails a page, a basket or a checkout.

Privacy. Only SKUs, contexts and daily counts are stored. No user, session, basket id, IP or user agent. Search contexts use the search analytics normalisation, so a term that looks like personal data (an email address, a long run of digits) is never stored or given a reference. The basket's note is removed when the order is placed. Nothing new is logged beyond site and order ids.

Ranking eligibility

Ranking by what sells needs enough orders for each product's rates to mean something. ProductEngagementService.eligibility reports whether the site is entitled (the raw entitlement, ignoring the enforcement switch) and its orders in the last 30 days (placed and paid, not cancelled: OrderService.countSoldSince) against the global setting search-ranking-min-monthly-orders (default 1,000). Below it, units sold keep deciding the order.

Where to see it

  • Admin: Search > Product engagement (/admin/search/engagement, see the Search section).
  • REST: GET /v1/admin/search/product-engagement with optional context, from, to, sort (IMPRESSIONS, CLICKS, VIEWS, ADDS, ORDERS, REVENUE) and limit (default 25, max 100). Returns the listing's totals, its products, the two opportunity lists and the eligibility; 400 for an unknown context or a bad date.
  • MCP: siteTraffic action productEngagement, with group naming the listing: blank or all, a category (path, name or id, as the ranking tools take it) or search:<term>; optional from, to and limit (default 10, max 50). See MCP.

The two opportunity lists:

  • Viewed a lot, rarely bought: at least 20 views and a conversion under half the listing's average. A price, stock, size or photo problem, or the wrong product for the listing.
  • Sells well, rarely shown: at least 2 orders, a conversion at least 1.5 times the listing's average, and fewer impressions than the median seller. Candidates to boost.

Tests

  • ProductEngagementServiceImplTest: contexts and bands, clicks and views, basket attribution and orders, the 24-hour window, the gate and its cache, the buffer bound and retry, reports, opportunities and eligibility.
  • ProductEngagementRepositoryMongoRuntimeTest (needs MONGO_RUNTIME_URL): the upserts and aggregations against a real MongoDB.
  • SearchClickTrackerTest, SearchRefCodecTest, SearchClickRefRenderTest: references by listing kind, impressions skipping bots and count previews, clicks once per page and product, the script on every theme's search and category pages.
  • ProductEngagementAdminRendererTest, ProductEngagementRenderTest, ProductEngagementAdminControllerTest (REST), StatisticsToolsTest (MCP).