Skip to content

Site Data Transfer

BadgerCommerce provides built-in export and import functionality for transferring site configuration and catalogue data between environments. This feature is essential for copying data from production to development, creating site templates, or migrating between installations.

Key Features

Data Export

  • Background Export: Exports run asynchronously in the worker, not on the web request. The archive is streamed straight off a Mongo cursor to a temp file and uploaded to S3, so even very large sites export with a flat, bounded memory footprint (no out-of-memory failures).
  • Queue → poll → download: Requesting an export creates an ExportJob; the admin UI polls its status and downloads via a short-lived pre-signed S3 URL once it reports COMPLETED.
  • Selective Export: Only exports catalogue and configuration data - excludes PII and transactional data
  • Typed confirmation: Each action asks you to type the tenant's site ID before it runs (the server checks it too), so you cannot act on the wrong tenant by accident. It works the same however you signed in
  • Portable Format: Standard ZIP file with JSON data, easily inspectable and version-controllable

Data Import

  • Validation Preview: See what will be imported before committing
  • Conflict Resolution: Automatically replaces existing data in the target site
  • ID Remapping: All MongoDB IDs and cross-references are automatically remapped
  • Tenant Isolation: Imported data is fully owned by the target site

Privacy by Design

The export explicitly excludes all personally identifiable information (PII) and transactional data:

Exported (Safe) Excluded (PII/Transactional)
Products & Variants Orders
Collections Users/Customers
Pages (CMS) Donations
Taxonomies Gift Aid Declarations
Merchandising trees (category pages /c/**) Stripe payouts
Menus Gift Cards
Media metadata (including focal points) Subscriptions/Invoices
Delivery Options Fundraiser Requests
Promotions (automatic and code-entered) Analytics/Page Views
Stereotypes AI Conversations
Variant Groups (Colour, Size…) Sensitive config values
Menu choices (restaurant modifier groups)‡ Kitchen state of orders
Shoppable images (pictures with their products)¶
Site Text overrides Sales counts
Stock on hand per SKU Search term analytics
Where each store keeps each SKU (aisle, bay, shelf, walk sequence)†
Stock movements and stock counts (what staff changed at each store, and who)
Locations (stores and warehouses)† Table bookings and table holds§
Dining rooms (tables, combinations, services)§ Restaurant guest profiles§
Table service settings (service charge) Tip records, tip policies, workers' requests, void and comp approvals§
Live search synonyms Synonym suggestions and rejections
Search ranking rules (pins, boosts, buries per term) Cost prices (commercially sensitive)
Ranking goal schedules Product engagement and learned ranking scores
Generated lists of related products ("You may also like"): the target generates its own
Mailing lists (messaging topics)
Promotion uses (redemptions) and assigned customers
Gift Aid chaser emails sent and what donors did
An unpublished theme, design brief or CSS preview
Site Configuration* Event tickets
Terms of sale page and version settings Customer contacts, email consent records and contact-us enquiries
The shop's plan and billing: platform subscription, billing account, sign-up, billing events and entitlement¶

*Site configuration excludes sensitive values like API keys, tokens, and secrets. It also leaves behind keys that describe the source site itself: its Badger Sett organisation id (which routes subscription webhooks to every site carrying it) and the legal-entity sign-off (legal-entity-confirmed-at / -by), so the target site's owner confirms their own details. The whole auth-config group is site-bound too: the central-auth OAuth2 client id and secret, the auth portal URL, central-auth-enabled and the widget signing keys. The client id is registered against the source site's domain, so carrying it over breaks storefront login on the target. The ticketing-config group (the secret behind ticket QR codes) stays behind as well, so a copy never accepts the source site's tickets. The change-request-config group stays behind too, so an import never switches on change requests (and the AI draft writing they pay for) on another site; change requests themselves are never exported, and a draft page written for one arrives on the target without its link to the request. The prodigi-config group (the shop's own Prodigi account: API key, webhook secret, mode) stays behind too; see Print on Demand. Also left behind: the storage bucket overrides, the POS quick-pay SKU id and the superadmin's learned ranking gate override (search-ranking-gate-override), and the custom domain verification token (custom-domain-verification-token), and the sister sites in other currencies (alternate-sites; see Selling in Several Currencies), which name the source shop's other sites. The key that signs unsubscribe and confirmation links (messaging-link-signing-key) is also left behind, so a copy never honours the source shop's links; the target creates its own the first time it needs one. Email campaigns are not transferred: neither the campaigns, their sends and delivery events, the do-not-email list (including the key that fingerprints its addresses), nor the sending domain. Their audience and numbers belong to the shop they were written for, and a copied domain would point at the source shop's DNS and Mailtrap webhook. The settings that name the shop's own Mailtrap account (mailtrap-account-id, mailtrap-bulk-api-key), the webhook signing secret and the send rate Badger set (campaign-max-per-hour) are left behind too, and skipped on import. The sender name, address prefix, reply-to and tracking settings travel like other settings. These are skipped on import as well, so an export made before this rule cannot bring them in.

¶The shop's plan and what it pays belong to the shop that pays, not to its content: a copy would carry the source shop's plan, Stripe customer and subscription and any founding place onto another site. The target keeps its own plan, and the platform plan catalogue itself is shared by every site, so it is never part of an export. Enable central auth on the target site separately.

†Locations travel without their Stripe Terminal location and linked fulfiller, which belong to the source shop. The import merges them into the target by code, before stock; a location the target already has keeps its own Stripe Terminal location and fulfiller. It skips (with a warning) any location the target's plan can't hold. Their restaurant ordering settings (ordering hours, last orders, prep and accept times, delivery area) travel with them; a "we're busy" pause does not. Where each location keeps each SKU (catalogue/stock-placements.json: location code, SKU, aisle, bay, shelf and walk sequence) travels with the locations and their stock, and is saved over the target's by location code and SKU after the locations are in; places at a location the target doesn't have are skipped. See Locations.

§Each dining-in location's room (configuration/dining-rooms.json: areas, tables, combinations, services, pacing and the booking policy) replaces the target location's room, by location code; table ids travel unchanged. Table bookings never travel: they carry guests' names, contact details and allergies, and the source shop's evenings. Nor do the table holds that go with them, or the guest profiles matched to them. Table service's records stay with the site that made them: tip records, tip policy versions and workers' requests for their records are the employer's records under the Employment (Allocation of Tips) Act 2023, and the managers' approvals name its staff. Tabs are orders, which never travel.

¶Shoppable images are in catalogue/shoppable-images.json. The import replaces the target's images, gives each a new id and points it at the imported picture. Blocks name an image by its handle and pieces name products by SKU, both of which travel unchanged. An image whose picture or products didn't arrive is skipped with a warning. See Shoppable images.

‡Menu choices ("Choose a sauce", "Extras") are in catalogue/modifier-groups.json. The import replaces the target's groups, gives each a new id and points every menu item's choices at the new ids. A group the target refuses (say, one with no options) is skipped with a warning, and items lose it. An export made before groups travelled leaves the target's groups alone. See Restaurant ordering.

Storefront theme: the manifest records the source site's theme (sourceThemeName) and the import switches the target site to it, provided that theme is installed on the target database. If it isn't (say, a bespoke theme), the target keeps its current theme and the validation preview warns about it.

Site details travel in configuration/site.json: the source site's logo, product image path, extra theme stylesheets, social accounts, default locale and time zone, and its header/footer display options (navbarMode, hideSiteNameInNav, stickyFooter). They are applied through the same site update the tenant editor uses. Not carried: the site's id, name, domains, catalogue, mode and demo password, Stripe account, sign-in set-up and any other context property, which belong to the source site. An export made before site details were exported has no site.json; importing it leaves the target's own, and the result says so. A time zone the target can't read is skipped with a warning, and a blank one leaves the target's zone as it is.

Product image path. Media travel as metadata only: an imported media document keeps the source's relative URL (<sourceSiteId>/media/...), which only resolves against the source's image path. So the import copies the image path too. Within one environment every site uses the same image CDN and nothing changes. When it does change (an import across environments, or onto a site with its own path), the result warns: the target's own uploads go to its upload bucket and must be served from the new path too, or the image path should be set back under Details.

Merchandising trees travel with their nodes: the category pages, their backings (collection seoNames, taxonomy levels and attribute filters, SKUs, none of which change on import) and each page's facet order. Trees and nodes get new ids and every reference between them is remapped, including a product's chosen main category (primaryCategoryNodeId). The collections a category has replaced travel as collection seoNames, which don't change. A main category whose node didn't travel is dropped, and the product's home category is worked out from the tree instead. A bundle with no merchandising/ folder leaves the target's trees as they are.

Export File Structure

site-export-{siteId}-{timestamp}.zip
├── manifest.json              # Export metadata (version, timestamp, source site and theme)
├── catalogue/
│   ├── products.json          # All products with variants
│   ├── collections.json       # Product collections/categories
│   ├── media.json             # Media metadata (URLs, not binaries)
│   ├── variant-groups.json    # Colour / Size / … definitions the variants refer to
│   ├── stock.json             # Stock on hand per SKU and warehouse
│   └── stock-placements.json  # Where each location keeps each SKU (aisle, bay, shelf, sequence)
├── content/
│   ├── pages.json             # CMS pages
│   ├── menus-v2.json          # Navigation menus (current model)
│   └── menus.json             # Navigation menus (v1 model)
├── taxonomy/
│   └── taxonomies.json        # Product taxonomies
├── merchandising/
│   ├── trees.json             # Merchandising trees (one is primary: the /c/** pages)
│   └── nodes.json             # Their category pages: backing, facets, SEO, extensions
├── search/
│   ├── synonyms.json          # Live search synonyms (not suggestions or rejections)
│   ├── ranking-rules.json     # Pins, boosts and buries per search term
│   └── ranking-goal-schedules.json  # Dated ranking goals (categories by path)
└── configuration/
    ├── site.json                 # Site details: logo, image path, stylesheets, social accounts...
    ├── site-configurations.json  # Site settings (non-sensitive)
    ├── stereotypes.json          # Product/page stereotypes
    ├── delivery-options.json     # Shipping methods
    ├── site-text.json            # Site Text overrides (Admin > Settings > Site text)
    ├── messaging-topics.json     # Mailing lists (not who is subscribed to them)
    └── promotions.json           # Promotions and discounts (Jackson default typing)

How ID Remapping Works

When importing data from Site A into Site B, all identifiers are automatically remapped:

Field On Export On Import
catalogueId Source site's ID Replaced with target site's ID
siteId Source site's ID Replaced with target site's ID
MongoDB _id Original IDs New ObjectIds generated
Cross-references Original IDs Remapped to new IDs
skuId Original SKU IDs Kept as-is — SKU IDs are business identifiers, not database keys

Some references are remapped specially: a product's images and each variant's own images (media ids; a reference to media the export lacks is dropped), any extension's mediaId setting (the blog masthead image, which blog cards and og:image also use) on products, collections, pages and stereotypes, and the site's active taxonomy (taxonomy-active-id, which names a taxonomy by id). Collections, pages and other items that arrive without an id each get their own new one.

This ensures: - No ID collisions between sites - Complete data ownership by the target site - No lingering references to the source site

SKU ID Sequence

SKU IDs travel with their products rather than being regenerated, because they are referenced by customers, invoices and external systems. The target site's SKU ID generator, however, knows nothing about the values that just arrived — left alone, it would mint IDs that already belong to an imported product.

The import therefore advances the target site's skuId sequence past the highest numeric SKU ID it wrote. This is:

  • Conditional — the sequence is only ever moved forwards, so an import finishing late can never roll back an ID minted in the meantime
  • Idempotent — importing the same archive twice advances it once
  • Limited to numeric SKU IDs — hand-written ones like base-donation-mug were never minted by the generator and cannot collide with it

Order numbers use a separate sequence (orderPlaced) which is deliberately not touched, since orders are excluded from the transfer. A site whose order number sequence sits behind its existing orders — typically one restored from a database dump rather than an export — will mint duplicate order numbers; the admin order screen tolerates this and logs an error naming the colliding orders, but the sequence needs correcting by hand.

Usage

Accessing the Feature

  1. Navigate to System > Tenants
  2. Click on the tenant you want to export from or import into
  3. Find the Move data panel on the right side

Exporting Site Data

  1. Click Export Site Data
  2. Type the tenant's site ID when prompted
  3. The export is queued to the worker and the dialog shows progress while it generates in the background. Large catalogues may take a little while.
  4. When it completes, the download starts automatically (via a pre-signed S3 link). The file is named site-export-{siteId}-{timestamp}.zip

The export job is processed by the worker, so you can navigate away — the generated archive is stored in S3 and the download link remains valid for 24 hours.

Importing Site Data

  1. Click Import Site Data and select a ZIP file
  2. Type the tenant's site ID when prompted (you are asked once to check the file, and again before the import is written)
  3. Review the validation summary showing:
  4. Source site information
  5. Entity counts to be imported
  6. Warnings about data replacement
  7. Click Import Now to proceed
  8. Wait for the import to complete, then read the report under Move data (see The import report)

What the import replaces

Each part of the export replaces the target's version of it; a part the export doesn't contain leaves the target's own alone. A few parts need more than that:

  • Menus. When the export has menus (either model), the target's menus in both models are deleted first. The storefront shows the v2 menu named by default-menu-id and falls back to a v1 menu of that id, so an old v1 menu left on the target used to show through.
  • Settings. An export written by the platform holds every transferable setting of the source (the manifest says configurationComplete: true; older exports count as complete because they have an exportedAt). Importing one makes the target's transferable settings match it exactly: settings the target has and the export doesn't are removed. This matters most for a new tenant, whose seeded default-menu-id, landing page and redirect setting used to survive the import and point it at the wrong menu and home page. Secrets and site-bound settings are never removed. Hand-built bundles (the curated demo sites) say configurationComplete: false and are merged in.
  • Active taxonomy. If it names a taxonomy that isn't in the export it is not imported, and the result says so.

Caches

The import writes straight to the repositories, past the caches, and runs on whichever web pod took the admin request, as the system site. So once anything has been written, after success or a failure part-way, it flushes every cache of the target site on every pod (SiteCacheService.evictAll, see Caching). Before this, an import into a site on a multi-pod cluster could show the old home page and menus for up to 30 minutes, while the same import on a single local pod looked fine. Clear caches on the tenant page does the same flush by hand.

The import report

After an import the Move data panel shows what happened: the counts, the parts the export did not contain (the site kept its own), and warnings that need a look (a theme that isn't installed, a changed image path, a setting that could not be mapped, a part the manifest lists but whose file is missing or empty, a breadcrumb backfill that could not be queued). If the import fails part-way, the error names the step it stopped at; the steps before it were applied, their counts are shown, and the caches are still flushed and the search reindex still queued.

Once the import succeeds, a full search reindex is queued for the site (products, and pages and blog posts for content search), since the imported documents were never indexed one by one.

A breadcrumb backfill is queued for the site too. Collections and products are written in bulk, which skips the breadcrumb chain each one normally gets on save, so without it every imported category page would show only Home / Category until someone ran Breadcrumb Backfill by hand.

Use Cases

Development Environment Setup

Export production catalogue data to set up a realistic development environment without exposing customer data.

Site Templates

Create a "template site" with products, pages, and configuration, then export it for use as a starting point for new sites.

Staging/QA Testing

Copy catalogue data to a staging environment to test changes with real product data.

Disaster Recovery

Maintain offline backups of site configuration and catalogue data (note: for full backups including transactional data, use the S3 archive feature).

Multi-Environment Deployment

Promote catalogue changes from development through staging to production by exporting and importing.

Technical Details

Service Layer

The feature is implemented in the SiteTransferService interface with these operations:

public interface SiteTransferService {
    // Streams the ZIP straight into the provided stream off a Mongo cursor — driven by the worker,
    // which writes to a temp file and uploads it to S3. Memory-bounded regardless of catalogue size.
    Map<String, Long> writeSiteExport(SiteContext siteContext, OutputStream out) throws IOException;

    ImportValidationResult validateImport(InputStream zipFile);
    SiteImportResult importSiteData(SiteContext targetSite, InputStream zipFile);
}

Both directions are streaming and memory-bounded:

  • Export — the worker (ExportProcessingJob) picks up a pending SITE_TRANSFER_ZIP job, calls writeSiteExport to stream each entity collection (via a Mongo cursor + Jackson SequenceWriter) into a temp ZIP, then uploads it to S3 with RequestBody.fromFile.
  • Import — the uploaded ZIP is spooled to a temp file and each entry is deserialised with a Jackson MappingIterator, persisting in fixed-size batches. ID-remap tables hold only string pairs, so peak heap stays flat even for large catalogues.

Sensitive Data Filtering

Configuration keys are automatically filtered if they: - Are marked with ofSecretString() in ConfigKeyDefaults - Contain patterns like: apikey, apisecret, token, secret, password, credential, stripe, recaptcha

Import Order

Data is imported in dependency order to maintain referential integrity:

  1. Media (referenced by products)
  2. Taxonomies (referenced by products)
  3. Collections (referenced by products)
  4. Products
  5. Pages
  6. Menus
  7. Stereotypes
  8. Merchandising trees and nodes (after stereotypes, which a node can use)
  9. Delivery Options
  10. Promotions
  11. Site Configuration
  12. Theme
  13. Site details (site.json)
  14. Search synonyms (after site configuration, so content search is already on or off when they are pushed to search)

Search synonyms go through SearchSynonymService.importLive: the target's existing synonyms (live, suggested and rejected) are replaced, every imported synonym is re-validated, de-duplicated, given a new id and the target's catalogueId, saved as live, and then written to the target's search collections. If search can't be reached the import still succeeds, and a resync from Manage Commerce > Search Synonyms applies them later. An export made before synonyms existed has no search/synonyms.json, and importing it leaves the target's synonyms alone. See Search Synonyms.

Mailing lists (messaging topics) are merged by key through MessagingTopicService.importTopics: a list the target already has under the same key is updated with the export's name, description, channels, order and archived state, a list it lacks is added, and none of the target's own lists is ever deleted, because its consent records name them by key. Ids and the site are the target's. The people subscribed to a list are contacts and consent records, which never travel. An export without configuration/messaging-topics.json leaves the target's lists alone. See Customer Messaging and Consent.

Search ranking rules come next, through SearchRankingRuleService.importAll: the target's rules are replaced, with new ids, and a rule with no term or no products, or a second rule for a term, is skipped. They name products by SKU, which products keep, so nothing is remapped; a category's own ranking travels on its node in merchandising/nodes.json. An export without search/ranking-rules.json leaves the target's rules alone. See Pins, Boosts and Buries.

Ranking goal schedules follow, through RankingGoalService.replaceSchedules: the target's schedules are replaced, with new ids, and one with no name, goal or dates in order is skipped. Their categories are paths, which survive the transfer, so nothing is remapped; a category's own goal travels on its node. Learned scores and the product engagement behind them are derived from the source's traffic and orders, so they aren't exported, and neither are learned boosts: imported category rankings and search rules keep only the merchandiser's lists. See Ranking by What Sells.

Limitations

  • Media files: Only metadata is exported, not the actual binary files. Media URLs are preserved, so the files stay where the source site uploaded them; the image path is copied so they resolve (see Product image path above).
  • Product images: @DBRef image references are cleared during import; re-associate images in admin if needed.
  • Promotions: automatic and code promotions are both exported, with their rules, limits and exclusions. Their uses (promotionRedemption) are order history and stay behind, so usage counts start again at zero on the target. The customers a promotion is assigned to are left out (users don't transfer), and an import clears them from older exports too.
  • User data: No user accounts are transferred; users must be created separately on the target site.

Security Considerations

  • Export/import operations require admin authentication
  • Typed confirmation (the tenant's site ID) is required for both operations, and is validated on the server. There is no password re-prompt, so operators who sign in with a social login, Badger Sett or a passkey are not locked out
  • Sensitive configuration values are automatically stripped
  • All PII and transactional data is excluded by design
  • Audit logging captures export/import operations