Site Data Transfer¶
BadgerCommerce provides built-in export and import functionality for transferring site configuration and catalogue data between environments. This feature is essential for copying data from production to development, creating site templates, or migrating between installations.
Key Features¶
Data Export¶
- Background Export: Exports run asynchronously in the worker, not on the web request. The archive is streamed straight off a Mongo cursor to a temp file and uploaded to S3, so even very large sites export with a flat, bounded memory footprint (no out-of-memory failures).
- Queue → poll → download: Requesting an export creates an
ExportJob; the admin UI polls its status and downloads via a short-lived pre-signed S3 URL once it reportsCOMPLETED. - Selective Export: Only exports catalogue and configuration data - excludes PII and transactional data
- Typed confirmation: Each action asks you to type the tenant's site ID before it runs (the server checks it too), so you cannot act on the wrong tenant by accident. It works the same however you signed in
- Portable Format: Standard ZIP file with JSON data, easily inspectable and version-controllable
Data Import¶
- Validation Preview: See what will be imported before committing
- Conflict Resolution: Automatically replaces existing data in the target site
- ID Remapping: All MongoDB IDs and cross-references are automatically remapped
- Tenant Isolation: Imported data is fully owned by the target site
Privacy by Design¶
The export explicitly excludes all personally identifiable information (PII) and transactional data:
| Exported (Safe) | Excluded (PII/Transactional) |
|---|---|
| Products & Variants | Orders |
| Collections | Users/Customers |
| Pages (CMS) | Donations |
| Taxonomies | Gift Aid Declarations |
Merchandising trees (category pages /c/**) |
Stripe payouts |
| Menus | Gift Cards |
| Media metadata (including focal points) | Subscriptions/Invoices |
| Delivery Options | Fundraiser Requests |
| Promotions (automatic and code-entered) | Analytics/Page Views |
| Stereotypes | AI Conversations |
| Variant Groups (Colour, Size…) | Sensitive config values |
| Menu choices (restaurant modifier groups)‡ | Kitchen state of orders |
| Shoppable images (pictures with their products)¶ | |
| Site Text overrides | Sales counts |
| Stock on hand per SKU | Search term analytics |
| Where each store keeps each SKU (aisle, bay, shelf, walk sequence)† | |
| Stock movements and stock counts (what staff changed at each store, and who) | |
| Locations (stores and warehouses)† | Table bookings and table holds§ |
| Dining rooms (tables, combinations, services)§ | Restaurant guest profiles§ |
| Table service settings (service charge) | Tip records, tip policies, workers' requests, void and comp approvals§ |
| Live search synonyms | Synonym suggestions and rejections |
| Search ranking rules (pins, boosts, buries per term) | Cost prices (commercially sensitive) |
| Ranking goal schedules | Product engagement and learned ranking scores |
| Generated lists of related products ("You may also like"): the target generates its own | |
| Mailing lists (messaging topics) | |
| Promotion uses (redemptions) and assigned customers | |
| Gift Aid chaser emails sent and what donors did | |
| An unpublished theme, design brief or CSS preview | |
| Site Configuration* | Event tickets |
| Terms of sale page and version settings | Customer contacts, email consent records and contact-us enquiries |
| The shop's plan and billing: platform subscription, billing account, sign-up, billing events and entitlement¶ |
*Site configuration excludes sensitive values like API keys, tokens, and secrets. It also leaves
behind keys that describe the source site itself: its Badger Sett organisation id (which routes
subscription webhooks to every site carrying it) and the legal-entity sign-off
(legal-entity-confirmed-at / -by), so the target site's owner confirms their own details.
The whole auth-config group is site-bound too: the central-auth OAuth2 client id and secret, the
auth portal URL, central-auth-enabled and the widget signing keys. The
client id is registered against the source site's domain, so carrying it over breaks storefront
login on the target. The ticketing-config group (the secret behind ticket QR codes) stays
behind as well, so a copy never accepts the source site's tickets. The change-request-config
group stays behind too, so an import never switches on change requests (and
the AI draft writing they pay for) on another site; change requests themselves are never exported,
and a draft page written for one arrives on the target without its link to the request.
The prodigi-config group (the shop's own Prodigi account: API key, webhook secret, mode) stays
behind too; see Print on Demand.
Also left behind: the storage bucket overrides, the POS quick-pay SKU id and the superadmin's
learned ranking gate override (search-ranking-gate-override), and the custom domain verification token
(custom-domain-verification-token), and the sister sites in other currencies (alternate-sites; see
Selling in Several Currencies), which name the source shop's other sites. The key that signs unsubscribe and confirmation links
(messaging-link-signing-key) is also left behind, so a copy never honours the source shop's links; the
target creates its own the first time it needs one.
Email campaigns are not transferred: neither the campaigns, their sends and
delivery events, the do-not-email list (including the key that fingerprints its addresses), nor the
sending domain. Their audience and numbers belong to the shop they were written for, and a copied
domain would point at the source shop's DNS and Mailtrap webhook. The settings that name the shop's own
Mailtrap account (mailtrap-account-id, mailtrap-bulk-api-key), the webhook signing secret and the
send rate Badger set (campaign-max-per-hour) are left behind too, and skipped on import. The
sender name, address prefix, reply-to and tracking settings travel like other settings.
These are skipped on import as well, so an export made before this rule cannot bring them in.
¶The shop's plan and what it pays belong to the shop that pays, not to its content: a copy would carry the source shop's plan, Stripe customer and subscription and any founding place onto another site. The target keeps its own plan, and the platform plan catalogue itself is shared by every site, so it is never part of an export. Enable central auth on the target site separately.
†Locations travel without their Stripe Terminal location and linked fulfiller, which belong to the
source shop. The import merges them into the target by code, before stock; a location the target
already has keeps its own Stripe Terminal location and fulfiller. It skips (with a warning) any location the target's plan can't hold. Their restaurant ordering settings (ordering
hours, last orders, prep and accept times, delivery area) travel with them; a "we're busy" pause
does not. Where each location keeps each SKU (catalogue/stock-placements.json: location code,
SKU, aisle, bay, shelf and walk sequence) travels with the locations and their stock, and is saved
over the target's by location code and SKU after the locations are in; places at a location the
target doesn't have are skipped. See Locations.
§Each dining-in location's room (configuration/dining-rooms.json: areas, tables, combinations,
services, pacing and the booking policy) replaces the target location's room, by location code; table
ids travel unchanged. Table bookings never travel: they carry guests' names, contact details and
allergies, and the source shop's evenings. Nor do the table holds that go with them, or the guest profiles matched to them.
Table service's records stay with the site that made them: tip records, tip policy versions and
workers' requests for their records are the employer's records under the Employment (Allocation of
Tips) Act 2023, and the managers' approvals name its staff. Tabs are orders, which never travel.
¶Shoppable images are in catalogue/shoppable-images.json. The import replaces the target's
images, gives each a new id and points it at the imported picture. Blocks name an image by its
handle and pieces name products by SKU, both of which travel unchanged. An image whose picture or
products didn't arrive is skipped with a warning. See Shoppable images.
‡Menu choices ("Choose a sauce", "Extras") are in catalogue/modifier-groups.json. The import
replaces the target's groups, gives each a new id and points every menu item's choices at the new
ids. A group the target refuses (say, one with no options) is skipped with a warning, and items
lose it. An export made before groups travelled leaves the target's groups alone. See
Restaurant ordering.
Storefront theme: the manifest records the source site's theme (sourceThemeName) and the import
switches the target site to it, provided that theme is installed on the target database. If it isn't
(say, a bespoke theme), the target keeps its current theme and the validation preview warns about it.
Site details travel in configuration/site.json: the source site's logo, product image path,
extra theme stylesheets, social accounts, default locale and time zone, and its header/footer
display options (navbarMode, hideSiteNameInNav,
stickyFooter). They are applied through the same site update the tenant editor uses. Not
carried: the site's id, name, domains, catalogue, mode and demo password, Stripe account, sign-in
set-up and any other context property, which belong to the source site. An export made before
site details were exported has no site.json; importing it leaves the target's own, and the
result says so. A time zone the target can't read is skipped with a warning, and a
blank one leaves the target's zone as it is.
Product image path. Media travel as metadata only: an imported media document keeps the
source's relative URL (<sourceSiteId>/media/...), which only resolves against the source's
image path. So the import copies the image path too. Within one environment every site uses the
same image CDN and nothing changes. When it does change (an import across environments, or onto a
site with its own path), the result warns: the target's own uploads go to its upload bucket and
must be served from the new path too, or the image path should be set back under Details.
Merchandising trees travel with their nodes: the category pages, their backings (collection
seoNames, taxonomy levels and attribute filters, SKUs, none of which change on import) and each
page's facet order. Trees and nodes get new ids and every reference between them is remapped,
including a product's chosen main category (primaryCategoryNodeId). The collections a category
has replaced travel as collection seoNames, which don't change. A main category whose node
didn't travel is dropped, and the product's home category is worked out from the tree instead. A
bundle with no merchandising/ folder leaves the target's trees as they are.
Export File Structure¶
site-export-{siteId}-{timestamp}.zip
├── manifest.json # Export metadata (version, timestamp, source site and theme)
├── catalogue/
│ ├── products.json # All products with variants
│ ├── collections.json # Product collections/categories
│ ├── media.json # Media metadata (URLs, not binaries)
│ ├── variant-groups.json # Colour / Size / … definitions the variants refer to
│ ├── stock.json # Stock on hand per SKU and warehouse
│ └── stock-placements.json # Where each location keeps each SKU (aisle, bay, shelf, sequence)
├── content/
│ ├── pages.json # CMS pages
│ ├── menus-v2.json # Navigation menus (current model)
│ └── menus.json # Navigation menus (v1 model)
├── taxonomy/
│ └── taxonomies.json # Product taxonomies
├── merchandising/
│ ├── trees.json # Merchandising trees (one is primary: the /c/** pages)
│ └── nodes.json # Their category pages: backing, facets, SEO, extensions
├── search/
│ ├── synonyms.json # Live search synonyms (not suggestions or rejections)
│ ├── ranking-rules.json # Pins, boosts and buries per search term
│ └── ranking-goal-schedules.json # Dated ranking goals (categories by path)
└── configuration/
├── site.json # Site details: logo, image path, stylesheets, social accounts...
├── site-configurations.json # Site settings (non-sensitive)
├── stereotypes.json # Product/page stereotypes
├── delivery-options.json # Shipping methods
├── site-text.json # Site Text overrides (Admin > Settings > Site text)
├── messaging-topics.json # Mailing lists (not who is subscribed to them)
└── promotions.json # Promotions and discounts (Jackson default typing)
How ID Remapping Works¶
When importing data from Site A into Site B, all identifiers are automatically remapped:
| Field | On Export | On Import |
|---|---|---|
catalogueId |
Source site's ID | Replaced with target site's ID |
siteId |
Source site's ID | Replaced with target site's ID |
MongoDB _id |
Original IDs | New ObjectIds generated |
| Cross-references | Original IDs | Remapped to new IDs |
skuId |
Original SKU IDs | Kept as-is — SKU IDs are business identifiers, not database keys |
Some references are remapped specially: a product's images and each variant's own images (media ids; a reference to media the export lacks is dropped), any extension's mediaId
setting (the blog masthead image, which blog cards and og:image also use) on products, collections,
pages and stereotypes, and the site's active taxonomy (taxonomy-active-id, which names a
taxonomy by id). Collections, pages and other items that arrive
without an id each get their own new one.
This ensures: - No ID collisions between sites - Complete data ownership by the target site - No lingering references to the source site
SKU ID Sequence¶
SKU IDs travel with their products rather than being regenerated, because they are referenced by customers, invoices and external systems. The target site's SKU ID generator, however, knows nothing about the values that just arrived — left alone, it would mint IDs that already belong to an imported product.
The import therefore advances the target site's skuId sequence past the highest
numeric SKU ID it wrote. This is:
- Conditional — the sequence is only ever moved forwards, so an import finishing late can never roll back an ID minted in the meantime
- Idempotent — importing the same archive twice advances it once
- Limited to numeric SKU IDs — hand-written ones like
base-donation-mugwere never minted by the generator and cannot collide with it
Order numbers use a separate sequence (orderPlaced) which is deliberately not
touched, since orders are excluded from the transfer. A site whose order number sequence
sits behind its existing orders — typically one restored from a database dump rather than
an export — will mint duplicate order numbers; the admin order screen tolerates this and
logs an error naming the colliding orders, but the sequence needs correcting by hand.
Usage¶
Accessing the Feature¶
- Navigate to System > Tenants
- Click on the tenant you want to export from or import into
- Find the Move data panel on the right side
Exporting Site Data¶
- Click Export Site Data
- Type the tenant's site ID when prompted
- The export is queued to the worker and the dialog shows progress while it generates in the background. Large catalogues may take a little while.
- When it completes, the download starts automatically (via a pre-signed S3 link). The file is
named
site-export-{siteId}-{timestamp}.zip
The export job is processed by the worker, so you can navigate away — the generated archive is stored in S3 and the download link remains valid for 24 hours.
Importing Site Data¶
- Click Import Site Data and select a ZIP file
- Type the tenant's site ID when prompted (you are asked once to check the file, and again before the import is written)
- Review the validation summary showing:
- Source site information
- Entity counts to be imported
- Warnings about data replacement
- Click Import Now to proceed
- Wait for the import to complete, then read the report under Move data (see The import report)
What the import replaces¶
Each part of the export replaces the target's version of it; a part the export doesn't contain leaves the target's own alone. A few parts need more than that:
- Menus. When the export has menus (either model), the target's menus in both models are
deleted first. The storefront shows the v2 menu named by
default-menu-idand falls back to a v1 menu of that id, so an old v1 menu left on the target used to show through. - Settings. An export written by the platform holds every transferable setting of the source
(the manifest says
configurationComplete: true; older exports count as complete because they have anexportedAt). Importing one makes the target's transferable settings match it exactly: settings the target has and the export doesn't are removed. This matters most for a new tenant, whose seededdefault-menu-id, landing page and redirect setting used to survive the import and point it at the wrong menu and home page. Secrets and site-bound settings are never removed. Hand-built bundles (the curated demo sites) sayconfigurationComplete: falseand are merged in. - Active taxonomy. If it names a taxonomy that isn't in the export it is not imported, and the result says so.
Caches¶
The import writes straight to the repositories, past the caches, and runs on whichever web pod took
the admin request, as the system site. So once anything has been written, after success or a failure
part-way, it flushes every cache of the target site on every pod (SiteCacheService.evictAll, see
Caching). Before this, an import into
a site on a multi-pod cluster could show the old home page and menus for up to 30 minutes, while the
same import on a single local pod looked fine. Clear caches on the tenant page does the same
flush by hand.
The import report¶
After an import the Move data panel shows what happened: the counts, the parts the export did not contain (the site kept its own), and warnings that need a look (a theme that isn't installed, a changed image path, a setting that could not be mapped, a part the manifest lists but whose file is missing or empty, a breadcrumb backfill that could not be queued). If the import fails part-way, the error names the step it stopped at; the steps before it were applied, their counts are shown, and the caches are still flushed and the search reindex still queued.
Once the import succeeds, a full search reindex is queued for the site (products, and pages and blog posts for content search), since the imported documents were never indexed one by one.
A breadcrumb backfill is queued for the site too. Collections and products are written in bulk, which skips the breadcrumb chain each one normally gets on save, so without it every imported category page would show only Home / Category until someone ran Breadcrumb Backfill by hand.
Use Cases¶
Development Environment Setup¶
Export production catalogue data to set up a realistic development environment without exposing customer data.
Site Templates¶
Create a "template site" with products, pages, and configuration, then export it for use as a starting point for new sites.
Staging/QA Testing¶
Copy catalogue data to a staging environment to test changes with real product data.
Disaster Recovery¶
Maintain offline backups of site configuration and catalogue data (note: for full backups including transactional data, use the S3 archive feature).
Multi-Environment Deployment¶
Promote catalogue changes from development through staging to production by exporting and importing.
Technical Details¶
Service Layer¶
The feature is implemented in the SiteTransferService interface with these operations:
public interface SiteTransferService {
// Streams the ZIP straight into the provided stream off a Mongo cursor — driven by the worker,
// which writes to a temp file and uploads it to S3. Memory-bounded regardless of catalogue size.
Map<String, Long> writeSiteExport(SiteContext siteContext, OutputStream out) throws IOException;
ImportValidationResult validateImport(InputStream zipFile);
SiteImportResult importSiteData(SiteContext targetSite, InputStream zipFile);
}
Both directions are streaming and memory-bounded:
- Export — the worker (
ExportProcessingJob) picks up a pendingSITE_TRANSFER_ZIPjob, callswriteSiteExportto stream each entity collection (via a Mongo cursor + JacksonSequenceWriter) into a temp ZIP, then uploads it to S3 withRequestBody.fromFile. - Import — the uploaded ZIP is spooled to a temp file and each entry is deserialised with a
Jackson
MappingIterator, persisting in fixed-size batches. ID-remap tables hold only string pairs, so peak heap stays flat even for large catalogues.
Sensitive Data Filtering¶
Configuration keys are automatically filtered if they:
- Are marked with ofSecretString() in ConfigKeyDefaults
- Contain patterns like: apikey, apisecret, token, secret, password, credential, stripe, recaptcha
Import Order¶
Data is imported in dependency order to maintain referential integrity:
- Media (referenced by products)
- Taxonomies (referenced by products)
- Collections (referenced by products)
- Products
- Pages
- Menus
- Stereotypes
- Merchandising trees and nodes (after stereotypes, which a node can use)
- Delivery Options
- Promotions
- Site Configuration
- Theme
- Site details (
site.json) - Search synonyms (after site configuration, so content search is already on or off when they are pushed to search)
Search synonyms go through SearchSynonymService.importLive: the target's existing synonyms
(live, suggested and rejected) are replaced, every imported synonym is re-validated, de-duplicated,
given a new id and the target's catalogueId, saved as live, and then written to the target's
search collections. If search can't be reached the import still succeeds, and a resync from
Manage Commerce > Search Synonyms applies them later. An export made before synonyms existed has
no search/synonyms.json, and importing it leaves the target's synonyms alone. See
Search Synonyms.
Mailing lists (messaging topics) are merged by key through MessagingTopicService.importTopics: a list
the target already has under the same key is updated with the export's name, description, channels, order and
archived state, a list it lacks is added, and none of the target's own lists is ever deleted, because its consent
records name them by key. Ids and the site are the target's. The people subscribed to a list are contacts and
consent records, which never travel. An export without configuration/messaging-topics.json leaves the target's
lists alone. See Customer Messaging and Consent.
Search ranking rules come next, through SearchRankingRuleService.importAll: the target's rules
are replaced, with new ids, and a rule with no term or no products, or a second rule for a term, is
skipped. They name products by SKU, which products keep, so nothing is remapped; a category's own
ranking travels on its node in merchandising/nodes.json. An export without
search/ranking-rules.json leaves the target's rules alone. See
Pins, Boosts and Buries.
Ranking goal schedules follow, through RankingGoalService.replaceSchedules: the target's schedules
are replaced, with new ids, and one with no name, goal or dates in order is skipped. Their
categories are paths, which survive the transfer, so nothing is remapped; a category's own goal
travels on its node. Learned scores and the product engagement behind them are derived from the
source's traffic and orders, so they aren't exported, and neither are learned boosts: imported
category rankings and search rules keep only the merchandiser's lists. See Ranking by What Sells.
Limitations¶
- Media files: Only metadata is exported, not the actual binary files. Media URLs are preserved, so the files stay where the source site uploaded them; the image path is copied so they resolve (see Product image path above).
- Product images:
@DBRefimage references are cleared during import; re-associate images in admin if needed. - Promotions: automatic and code promotions are both exported, with their rules, limits and exclusions. Their uses (
promotionRedemption) are order history and stay behind, so usage counts start again at zero on the target. The customers a promotion is assigned to are left out (users don't transfer), and an import clears them from older exports too. - User data: No user accounts are transferred; users must be created separately on the target site.
Security Considerations¶
- Export/import operations require admin authentication
- Typed confirmation (the tenant's site ID) is required for both operations, and is validated on the server. There is no password re-prompt, so operators who sign in with a social login, Badger Sett or a passkey are not locked out
- Sensitive configuration values are automatically stripped
- All PII and transactional data is excluded by design
- Audit logging captures export/import operations
Related Features¶
- Multi-Tenancy Architecture - Understanding tenant isolation
- Site Customization - Configuring site settings
- Content Management - Managing pages and menus