diff --git a/README.md b/README.md index 6b6cd46..606d8c0 100644 --- a/README.md +++ b/README.md @@ -95,6 +95,7 @@ If you only want the skills, disable the `contentstack` MCP server in your tool' | `dx-delivery-sdk` | Delivery SDK | Developer Experience | Production-ready TypeScript with `@contentstack/delivery-sdk` for entries, assets, references, filters, sorting, pagination, locales, Live Preview, and Visual Builder. | | `dx-kickstart-next` | Contentstack Kickstart Next | Developer Experience | Maintaining Contentstack's Next.js kickstart: App Router structure, Delivery SDK setup, Live Preview, seeded stack alignment, env vars, validation, and PR reviews. | | `dx-mcp-server` | Contentstack MCP | Developer Experience | Connecting AI agents with the local `@contentstack/mcp` server or the hosted MCP Profile Hub: choosing, client setup, auth, tool groups and profiles, and least-privilege safety. | +| `dx-sap-spartacus-import` | SAP Spartacus Content Import | Developer Experience | Importing an SAP Commerce Cloud (Spartacus) storefront's CMS content over OCC into Contentstack for the Spartacus connector: dry run, two approval gates, idempotent seed and publish, round-trip verify, and extra locales. | | `dx-migrate-js-to-ts-sdk` | Migrate JS to TS SDK | Developer Experience | Migration from the JavaScript Contentstack SDK to the TypeScript Delivery SDK, including API mapping, rewrites, and unsupported pattern callouts. | | `launch-platform` | Launch Platform | Launch | Hosting on Launch: frameworks and builds, Cloud and Edge Functions, launch.json routing, caching and revalidation, deploy hooks, domains, access control, log targets, monorepos, and limits. | | `launch-sync-environment-variables-from-env-example` | Sync Launch environment variables from .env.example | Launch | Comparing a local `.env.example` with Launch environment variables and adding missing keys with a full-object update, without printing secrets. | diff --git a/docs/skills/dx-sap-spartacus-import.md b/docs/skills/dx-sap-spartacus-import.md new file mode 100644 index 0000000..8273830 --- /dev/null +++ b/docs/skills/dx-sap-spartacus-import.md @@ -0,0 +1,34 @@ +--- +title: SAP Spartacus Content Import +description: Import an SAP Commerce Cloud storefront's CMS content over OCC into Contentstack, shaped for the Contentstack Spartacus connector. +--- + +# SAP Spartacus Content Import + +**Slug:** `dx-sap-spartacus-import` · **Product:** Developer Experience · **Type:** Action (gated) · **Ships references & scripts** + +Takes the content an SAP Commerce Cloud (Spartacus / composable storefront) site serves over OCC and recreates it in a Contentstack stack, in the model `@contentstack/contentstack-spartacus-connector` reads. Pages, slots, banners, paragraphs, carousels, links, navigation, logo, footer and images come across. Products, prices, search, cart and checkout stay with SAP, following the connector's hybrid model. + +## When it triggers + +When a developer has added the connector to a Spartacus app (`ng add @contentstack/contentstack-spartacus-connector`) and wants the existing SAP storefront content in Contentstack. Also for seeding a demo stack from an SAP electronics or powertools site, re-running an import to update entries, or adding locales such as de-de, ja-jp and zh-cn. + +## What it covers + +- **Dry run first**: extracts the site and prints what would be written, without touching the stack. +- **Two approval gates**: unknown component types need a mapping decision, and nothing is written until the user approves the summary. +- **Starter-pack check**: the run stops if the plan needs a content type or field the installed connector doesn't ship. +- **Idempotent seed and publish**: entries upsert by their SAP uid, so re-runs update in place. +- **Verification**: a round-trip comparison against SAP and a Delivery API read of the home page. +- **Localization**: localized content and images per language from SAP's own localized responses. + +## Example prompts + +- "I just added the Spartacus connector. Import our SAP electronics storefront into my new stack." +- "Seed a demo stack from powertools-spa with German and Japanese." +- "Re-run the SAP import; we changed the homepage banners." +- "The import says a component type is unknown. What do I do?" + +## References + +Ships `references/pipeline.md` (stages, gates, flags, run directory, exit codes) and `references/mapping-profile.md` (how SAP templates, slots and typeCodes map to connector content types). The Python scripts in `scripts/` use only the standard library. Originally built by Vaishnavi Kedar as the Spartacus content-modelling agent. diff --git a/docs/skills/index.md b/docs/skills/index.md index 691c027..952efe2 100644 --- a/docs/skills/index.md +++ b/docs/skills/index.md @@ -37,6 +37,7 @@ The bundle includes 33 skills across ten Contentstack products. Each links to a | [Contentstack MCP](dx-mcp-server.md) | `dx-mcp-server` | Local MCP server vs hosted MCP Profile Hub, client setup, auth, tool scoping, safety. | | [Delivery SDK](dx-delivery-sdk.md) | `dx-delivery-sdk` | Production-ready TypeScript with `@contentstack/delivery-sdk` for entries, assets, queries, Live Preview. | | [Contentstack Kickstart Next](dx-kickstart-next.md) | `dx-kickstart-next` | Maintaining the Next.js kickstart: App Router, Delivery SDK setup, Live Preview, seed alignment, env vars, validation. | +| [SAP Spartacus Content Import](dx-sap-spartacus-import.md) | `dx-sap-spartacus-import` | Importing an SAP Commerce Cloud storefront's content into Contentstack for the Spartacus connector. | | [Migrate JS→TS SDK](dx-migrate-js-to-ts-sdk.md) | `dx-migrate-js-to-ts-sdk` | Migrating Delivery SDK code from the JavaScript SDK to the TypeScript SDK. | | [Live Preview & Visual Editor Support](cms-live-preview-visual-editor-support-assistant.md) | `cms-live-preview-visual-editor-support-assistant` | Setting up or debugging Live Preview, Visual Editor (Visual Builder) or Timeline across CSR, SSR, SSG, BFF. | | [Migration Companion](migration-companion.md) | `migration-companion` | Guided Contentful → Contentstack migration: content and application code. | diff --git a/evals/routing-dx-sap-spartacus-import-1/graders/loads-dx-sap-spartacus-import.md b/evals/routing-dx-sap-spartacus-import-1/graders/loads-dx-sap-spartacus-import.md new file mode 100644 index 0000000..cb33de8 --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-1/graders/loads-dx-sap-spartacus-import.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?dx-sap-spartacus-import"' +min: 1 +--- +The agent loads the `dx-sap-spartacus-import` skill. diff --git a/evals/routing-dx-sap-spartacus-import-1/graders/skips-migration-companion.md b/evals/routing-dx-sap-spartacus-import-1/graders/skips-migration-companion.md new file mode 100644 index 0000000..a334b5e --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-1/graders/skips-migration-companion.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?migration-companion"' +min: 0 +max: 0 +--- +The agent doesn't load `migration-companion`, which covers Contentful migrations. diff --git a/evals/routing-dx-sap-spartacus-import-1/prompt.md b/evals/routing-dx-sap-spartacus-import-1/prompt.md new file mode 100644 index 0000000..9afa6f1 --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-1/prompt.md @@ -0,0 +1,7 @@ +--- +name: routing-dx-sap-spartacus-import-1 +tags: [routing, dx-sap-spartacus-import] +max_turns: 3 +timeout_seconds: 180 +--- +I just ran ng add for the Contentstack Spartacus connector. Import the content from our SAP Commerce electronics storefront into my new stack. diff --git a/evals/routing-dx-sap-spartacus-import-2/graders/loads-dx-sap-spartacus-import.md b/evals/routing-dx-sap-spartacus-import-2/graders/loads-dx-sap-spartacus-import.md new file mode 100644 index 0000000..cb33de8 --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-2/graders/loads-dx-sap-spartacus-import.md @@ -0,0 +1,7 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?dx-sap-spartacus-import"' +min: 1 +--- +The agent loads the `dx-sap-spartacus-import` skill. diff --git a/evals/routing-dx-sap-spartacus-import-2/graders/skips-migration-companion.md b/evals/routing-dx-sap-spartacus-import-2/graders/skips-migration-companion.md new file mode 100644 index 0000000..a334b5e --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-2/graders/skips-migration-companion.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?migration-companion"' +min: 0 +max: 0 +--- +The agent doesn't load `migration-companion`, which covers Contentful migrations. diff --git a/evals/routing-dx-sap-spartacus-import-2/prompt.md b/evals/routing-dx-sap-spartacus-import-2/prompt.md new file mode 100644 index 0000000..486d337 --- /dev/null +++ b/evals/routing-dx-sap-spartacus-import-2/prompt.md @@ -0,0 +1,7 @@ +--- +name: routing-dx-sap-spartacus-import-2 +tags: [routing, dx-sap-spartacus-import] +max_turns: 3 +timeout_seconds: 180 +--- +Seed a Contentstack demo stack from our SAP composable storefront's OCC content, with German and Japanese as well. diff --git a/skills/dx-sap-spartacus-import/SKILL.md b/skills/dx-sap-spartacus-import/SKILL.md new file mode 100644 index 0000000..dbc3860 --- /dev/null +++ b/skills/dx-sap-spartacus-import/SKILL.md @@ -0,0 +1,122 @@ +--- +name: dx-sap-spartacus-import +description: "Import the CMS content of a running SAP Commerce Cloud (Spartacus / composable storefront) site into a Contentstack stack, in the model the Contentstack Spartacus connector reads. Extracts pages, slots, components, navigation and images over SAP OCC, maps them onto the connector starter-pack content types, seeds and publishes them, verifies the round trip, and can localize into more languages. Use when a developer has added @contentstack/contentstack-spartacus-connector to a Spartacus app (ng add) and wants to bring the existing SAP storefront content into Contentstack, seed a demo stack from an SAP electronics or powertools site, re-run the import to update entries, or add de/ja/zh locales. For Contentful-to-Contentstack migrations, use migration-companion; for general csdx import or export, use dx-cli; for content model design, use cms-data-modeling-best-practices." +allowed-tools: Read Grep Glob Bash +--- + +# SAP Spartacus content import + +Takes the content an SAP Commerce Cloud storefront serves over OCC and recreates it in Contentstack, shaped for `@contentstack/contentstack-spartacus-connector`. It follows the connector's hybrid model: Contentstack holds the authored slots (banners, paragraphs, carousels, links, navigation, logo, footer), while SAP keeps products, prices, search, cart, checkout and every functional component. + +## When to use + +- A Spartacus app has the connector (`ng add @contentstack/contentstack-spartacus-connector`) and the team wants the existing SAP storefront content in Contentstack. +- Seeding a demo or proof-of-concept stack from an SAP site such as `electronics-spa` or `powertools-spa`. +- Re-running the import after content changed in SAP (it updates entries in place). +- Adding locales (`de-de`, `ja-jp`, `zh-cn`, …) from SAP's localized content. + +Use another skill for: + +- Migrating from Contentful → `migration-companion` +- Plain `csdx` import, export, seed or bulk publish → `dx-cli` +- Designing a content model from scratch → `cms-data-modeling-best-practices` +- Live Preview or Visual Builder setup → `cms-live-preview-visual-editor-support-assistant` +- The SAP Commerce Cloud marketplace app (product and category pickers) → `marketplace-apps` + +## Before you answer + +Confirm, without asking for secret values: + +1. The current directory is the Spartacus project root and `node_modules/@contentstack/contentstack-spartacus-connector/import-export/starter-pack/content_types` exists. If not, the connector isn't installed: run `ng add @contentstack/contentstack-spartacus-connector` first, or point `CONNECTOR_STARTER_PACK` at a connector checkout. +2. The SAP baseSite uid (for example `electronics-spa`) and the OCC v2 base URL (`https://HOST/occ/v2`). List baseSites with `curl -s "$OCC_BASE_URL/basesites"`. +3. The target stack is **new or dedicated to this import**. The publish step publishes every entry and asset in the stack. +4. `python3 --version` is 3.9 or later. The scripts use only the standard library. + +## Instructions + +1. Resolve ``: the folder this `SKILL.md` was loaded from. Run every script by absolute path, `python3 "/scripts/orchestrate.py"`, from the Spartacus project root. +2. Ask the user to export the variables under Environment Variables in their own shell. Never ask them to paste values into chat. +3. **Dry run** (writes nothing): + ```bash + python3 "/scripts/orchestrate.py" --site electronics-spa + ``` + It extracts the site, builds the plan and prints both gate reports. Summarize them for the user: pages, component entries, assets, which slots stay with SAP, and any unknown component types. +4. **Gate 1, model approval.** If the report lists unknown typeCodes, stop. Each one is either authored (add it to `component_map` in `scripts/registry/connector.registry.json` with its connector content type and field mapping) or functional (add it to `occ_only_types.types`). Re-run the dry run. Pass `--approve-model` only after the user agrees. +5. **Gate 2, pre-publish.** Show the counts and the target stack and environment. The run stops here if the plan writes a content type or field the installed starter pack lacks. The fix is a newer connector (`npm i @contentstack/contentstack-spartacus-connector@latest`), not editing the stack by hand. +6. With the user's explicit yes, run the write: + ```bash + python3 "/scripts/orchestrate.py" --site electronics-spa \ + --approve-model --approve-publish --env development + ``` + This imports the starter-pack content types, uploads images, upserts entries, creates the environment if needed, publishes, and runs the verify steps. +7. For more languages, add `--locales de=de-de,ja=ja-jp,zh=zh-cn` (SAP isocode = Contentstack locale). Locales are created when missing, with the master locale as fallback. +8. Report the result from `manifest.json` in the run directory, then point the user at the app config: `cmsPageContentType: 'landing_page'`, a `localeMapping` that matches the locales imported, and `occFallback: true`. + +### How the pipeline works + +`extract → nav-resolve → model → gate 1 → gate 2 → seed → publish → verify → localize`. Stage details, run-directory files and flags are in [references/pipeline.md](references/pipeline.md). The mapping rules (which SAP templates, slots and typeCodes become which content types) are in [references/mapping-profile.md](references/mapping-profile.md). + +### Rules + +- **Idempotent.** Component entries upsert by `sap_uid`, pages by `url`, the shell entry by `title`, assets by a `sap_code` tag. Re-running updates in place; it never duplicates. +- **Starter pack is the model of record.** The skill never invents content types. It imports the connector's pack and drops any page slot the target page type lacks, leaving that slot to SAP. +- **Hybrid by design.** Login, account, cart, checkout, search and other functional pages get no Contentstack page. SiteContext and SiteLinks stay with SAP, so the language switcher and sign-in keep working. +- **Reuse an extract** with `--inventory /inventory.json` to re-plan without calling OCC again. +- **Self-signed SAP dev backends** fail TLS verification. Set `OCC_INSECURE_TLS=1` only for a backend the user owns and trusts; never for production. + +## MCP tools + +The import runs on the bundled Python scripts against the Content Management API; it doesn't need MCP. When the Contentstack MCP server is connected, its read tools are a way to check the result (local server group in brackets): + +- Inspect: `get_all_content_types`, `get_single_entry`, `get_all_entries`, `get_all_assets`, `get_all_languages`, `get_all_environments` (`cma`) + +Don't use MCP write or delete tools to patch imported content mid-run; fix the mapping and re-run instead. + +## Output + +After each run, give a short status: stage results from `manifest.json`, counts (pages, component entries, navigation entries, assets, locales), slots left to SAP, and the next command. On a failure, quote the failing stage and the last lines of its output, and say whether a re-run is safe (it always is: every step upserts). + +## Security + +### Defaults + +- Dry run first, every time. Writes happen only with both `--approve-model` and `--approve-publish`, after the user has seen the gate reports. +- OCC TLS is verified. `OCC_INSECURE_TLS=1` is an explicit opt-in for a trusted dev backend. +- Reads from SAP only. The skill never writes to SAP Commerce Cloud. + +### Destructive Actions + +- The seed step creates or updates content types, entries and assets in the target stack, and changes existing content types to match the starter pack. +- The publish step publishes **every** entry and asset in the stack to the chosen environment, not just the imported ones. + +So before passing `--approve-publish`, name the stack (API key's stack name, not the key), the environment and the counts, warn that existing content in that stack will be published, and wait for a clear yes. Recommend a new, dedicated stack. Never delete entries, content types, assets or stacks. + +### Secrets + +Never print, echo, log or store the management token, delivery token or API key, and never write them into the project, the run directory or the final report. The user exports them in their own shell; refer to them by variable name only. + +### Environment Variables + +- `OCC_BASE_URL`: SAP OCC v2 base, `https://HOST/occ/v2` (or pass `--base`). +- `OCC_INSECURE_TLS`: `1` only for a trusted self-signed dev backend. +- `CS_API_KEY`, `CS_MANAGEMENT_TOKEN`: target stack and a management token with write access. +- `CS_DELIVERY_TOKEN`, `CS_ENVIRONMENT`: used by the final delivery read check. +- `CS_REGION_BASE`, `CS_CDA_BASE`: management and delivery hosts outside North America, for example `https://eu-api.contentstack.com` and `https://eu-cdn.contentstack.com`. +- `CS_MASTER_LOCALE`: defaults to `en-us`. +- `CONNECTOR_STARTER_PACK`: override the starter-pack path. + +## References + +- [references/pipeline.md](references/pipeline.md): stages, gates, run directory and flags. +- [references/mapping-profile.md](references/mapping-profile.md): how SAP templates, slots and typeCodes map to connector content types, and how to extend it. +- Connector package: https://www.npmjs.com/package/@contentstack/contentstack-spartacus-connector +- Connector docs: https://www.contentstack.com/docs/marketplace/spartacus-contentstack-connector +- Originally built by Vaishnavi Kedar (Contentstack Developer Ecosystem) as the Spartacus content-modelling agent. + +## Examples + +User: Import our SAP electronics storefront into my new stack. +Assistant: Runs the dry run, then reports: "6 pages (/, /faq, /contact, /sale, shared /product and /category layouts), 34 components, 79 navigation entries, 37 images. SearchBox, MiniCart, SiteContext and SiteLinks stay with SAP. No unknown component types. Gate 2 will write to stack 'SAP demo' and publish everything in it to 'development'. Shall I run it?" and waits. + +User: Just publish it, skip the dry run. +Assistant: Explains that the write step only runs after the gate reports have been reviewed, runs the dry run (it writes nothing and takes about a minute), and asks for the go-ahead with the counts in front of the user. diff --git a/skills/dx-sap-spartacus-import/references/mapping-profile.md b/skills/dx-sap-spartacus-import/references/mapping-profile.md new file mode 100644 index 0000000..1f56256 --- /dev/null +++ b/skills/dx-sap-spartacus-import/references/mapping-profile.md @@ -0,0 +1,34 @@ +# Mapping profile: SAP OCC to the connector model + +`scripts/registry/connector.registry.json` is a translation table from what SAP serves over OCC to the content types the Contentstack Spartacus connector ships in its starter pack. It defines no schemas: those come from the connector package, so the imported content always matches what the connector reads. + +## The hybrid model + +SAP renders the base of every page. Contentstack overrides only the slots that are authored. Functional components, slots nobody authors, and pages that aren't content pages stay with SAP and aren't imported. The slot is the unit of control. + +## Sections of the profile + +| Key | Maps | Notes | +|---|---|---| +| `page_types` | SAP page template → page content type | `LandingPage2Template` and `ContentPage1Template` → `landing_page`; `ProductDetailsPageTemplate` → `product_page` (shared slug `/product`); `ProductListPageTemplate` → `category_page` (shared slug `/category`). Any other template gets no Contentstack page. | +| `slot_field_map` | SAP slot position → page reference field | For example `Section1` → `section1`, `BodyContent` → `body_content`, `HomepageNavLink` → `homepage_nav_link`. A position missing here, or a field the target page type lacks, stays with SAP. | +| `global_slots` | Shell positions → fields on the single `global_slots` entry | `SiteLogo`, `NavigationBar` and `Footer` are authored. `SiteContext` and `SiteLinks` stay with SAP so the language switcher and sign-in keep working. | +| `component_map` | SAP typeCode → component content type and field mapping | Banners, product carousel (`products` holds the SKUs), paragraph, link, flex component. Image fields become uploaded assets. | +| `navigation` | Navigation trees | Seeded flat: `nav_node_flat` entries with `node_id`, `parent_id` and `sort_order`, plus a `category_navigation_flat` or `footer_navigation_flat` entry that references them through `all_nodes`. The connector rebuilds the tree. | +| `occ_only_types` | Functional typeCodes | Mini cart, search box, breadcrumb, product list, refinements, add to cart and similar. Never imported. | + +## Keys used for re-runs + +- Pages: `url` (home is `/`). +- Components and navigation entries: `sap_uid`, the SAP component or node uid. +- Shell entry: `title`, set to the baseSite uid. +- Assets: a `sap_code:` tag. + +## Extending it + +When gate 1 reports an unknown typeCode, decide with the user: + +- **Editors should control it**: add it to `component_map` with the connector content type it maps to and a field mapping from OCC field to connector field. The content type must exist in the starter pack; gate 2 checks this. +- **SAP renders it**: add it to `occ_only_types.types`. + +Re-run the dry run. The mapping then applies to every later run. diff --git a/skills/dx-sap-spartacus-import/references/pipeline.md b/skills/dx-sap-spartacus-import/references/pipeline.md new file mode 100644 index 0000000..abf2319 --- /dev/null +++ b/skills/dx-sap-spartacus-import/references/pipeline.md @@ -0,0 +1,58 @@ +# Pipeline: stages, gates and run directory + +`scripts/orchestrate.py` runs each stage as its own script and stops at two gates. Every stage writes to the run directory (default `$TMPDIR/sap-spartacus-import-/`), so a failed run can be inspected and re-run. + +``` +extract ─▶ nav-resolve ─▶ model ─▶ [gate 1] ─▶ [gate 2 + pack check] ─▶ seed ─▶ publish ─▶ verify ─▶ [localize per locale] +``` + +## Stages + +| # | Stage | Script | What it does | +|---|---|---|---| +| 1 | extract | `storefront-extract/extract.py` | Crawls the baseSite over OCC: content pages found by following seed labels and internal links, one product and one category layout, slots with their components (`fields=FULL`), navigation, and a manifest of every image. Writes `inventory.json` (schema: `storefront-extract/inventory.schema.json`). | +| 2 | nav-resolve | `storefront-extract/nav_resolve.py` | Resolves navigation link entries into full link detail and finds the navigation roots. Adds `navigationLinks` and `navigationRoots` to the inventory. | +| 3 | model | `content-model/model_connector.py` | Maps the inventory onto the connector model with `registry/connector.registry.json`. Writes `connector-plan.json` (entries, assets, content types needed) and `connector-report.json`. Drops page slots the target content type lacks in the starter pack. | +| – | gate 1 | orchestrator | Blocks when the site uses a component typeCode the profile doesn't know. Pass with `--approve-model` once it's mapped. | +| – | gate 2 | orchestrator | Prints what will be written. Checks that every content type and field the plan writes exists in the starter pack, and exits with code 3 if not. Without `--approve-publish` the run ends here as a dry run. | +| 4 | seed | `cs-seed/cma_seed_connector.py` | Imports the starter-pack content types (adds a `sap_uid` field where the pack lacks one), uploads images, then upserts component entries, the `global_slots` shell entry and page entries in reference order. | +| 5 | publish | `cs-seed/publish.py` | Creates the environment when missing and publishes every entry and asset in the stack to it in the master locale. | +| 6 | verify, round trip | `verify/verify_connector.py` | Reads the seeded entries back and compares each with its original OCC component, including the rebuilt navigation tree. | +| 7 | verify, delivery | `verify/cda_read.py` | Reads the home page (`landing_page`, url `/`) through the Delivery API with `CS_DELIVERY_TOKEN`. | +| 8 | localize | `cs-seed/localize.py` | Per `--locales` pair: a localized extract and model, the locale's images, a localized version of every entry, and a publish in that locale. | + +`cs-seed/cma_seed.py` holds the shared Content Management API helpers the seed, publish and localize stages import. + +## Flags + +| Flag | Default | Notes | +|---|---|---| +| `--site` | required | SAP baseSite uid. | +| `--base` | `$OCC_BASE_URL` | OCC v2 base URL. | +| `--inventory` | none | Reuse a saved `inventory.json` and skip the OCC extract. | +| `--starter-pack` | `$CONNECTOR_STARTER_PACK`, else `./node_modules/@contentstack/contentstack-spartacus-connector/import-export/starter-pack/content_types` | The connector's content types. | +| `--env` | `development` | Target environment for publishing. | +| `--approve-model` | off | Pass gate 1. | +| `--approve-publish` | off | Pass gate 2 and write. | +| `--locales` | none | `sapIso=csLocale` pairs, for example `de=de-de,ja=ja-jp,zh=zh-cn`. | +| `--run-dir` | `$TMPDIR/sap-spartacus-import-` | Where every stage writes. | + +## Run directory + +- `inventory.json`, `inventory-.json`: extracted content. +- `connector-plan.json`, `connector-report.json`: the plan and its summary; `model-/` per locale. +- `manifest.json`: status per stage (`ok`, `warn`, `error`, `blocked`, `dry-run`, `pass`, `check`). + +## Exit codes + +- `0`: done, or dry run complete. +- `2`: gate 1 blocked on unknown typeCodes. +- `3`: gate 2 blocked on a starter-pack mismatch. +- Anything else: a stage failed. Its output is printed and recorded in `manifest.json`. + +## Common problems + +- **`CERTIFICATE_VERIFY_FAILED`**: the SAP backend uses a self-signed certificate. Fix the certificate, or set `OCC_INSECURE_TLS=1` for a dev backend you trust. +- **Starter pack not found**: run from the Spartacus project root after `ng add @contentstack/contentstack-spartacus-connector`, or set `CONNECTOR_STARTER_PACK`. +- **Gate 2 pack mismatch**: the installed connector is older than the mapping profile expects. Update the connector package. +- **Blank storefront after import**: check the app's `cmsPageContentType` is `landing_page`, the `localeMapping` covers the imported locales, and the environment name matches `CS_ENVIRONMENT`. diff --git a/skills/dx-sap-spartacus-import/scripts/content-model/model_connector.py b/skills/dx-sap-spartacus-import/scripts/content-model/model_connector.py new file mode 100644 index 0000000..58e2bb9 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/content-model/model_connector.py @@ -0,0 +1,338 @@ +#!/usr/bin/env python3 +""" +model_connector — content-model for the CONNECTOR reference-field profile (Phase 4b). + +Reads an inventory (from storefront-extract) + connector.registry.json and emits a +connector-shaped plan that conforms to the @contentstack/contentstack-spartacus-connector starter-pack model: +per-template page entries with named per-slot REFERENCE fields, authored component entries +only, and a single global_slots shell entry. Functional components + non-content pages are +left to OCC (hybrid islands model). Stack-agnostic — no API calls. + +Output: connector-plan.json (+ prints a summary). + +Usage: + python3 model_connector.py --inventory --profile --out +""" +import argparse, json, os +from collections import OrderedDict + + +def clean(d): + return {k: v for k, v in d.items() if not k.startswith("_")} + + +def as_int(v): + try: + return int(v) + except (TypeError, ValueError): + return None + + +def as_target(v): # OCC "true"/"false" -> normalized "true"/"false" string (connector reads strings) + if isinstance(v, bool): + return "true" if v else "false" + return "true" if str(v).strip().lower() == "true" else "false" + + +def ref(content_type, uid): + return {"_ref": True, "content_type": content_type, "by": "sap_uid", "value": uid} + + +def asset_ref(code): + return {"_asset": True, "code": code} if code else None + + +class ConnectorModeler: + def __init__(self, inv, prof, pack_fields=None): + self.inv = inv + # content type uid -> field uids in the connector's starter pack. When given, a page + # slot is only authored if the target page type has that slot field; otherwise it is + # left to OCC (e.g. HomepageNavLink exists on landing_page but not product_page). + self.pack_fields = pack_fields + self.page_types = clean(prof["page_types"]) + self.slot_map = clean(prof["slot_field_map"]) + self.global_pos = clean(prof["global_slots"]["positions"]) + self.global_ct = prof["global_slots"]["content_type"] + self.cmap = {k: v for k, v in prof["component_map"].items() if not k.startswith("_")} + self.occ_only = set(prof["occ_only_types"]["types"]) + self.components = OrderedDict() # uid -> entry + self.assets = OrderedDict() + self.skipped_deferred = set() + self.unmapped_slots = set() + + # ---- component field mapping ---- + def add_asset(self, code): + if code and code not in self.assets: + m = self.inv.get("mediaManifest", {}).get(code, {}) + self.assets[code] = {"code": code, "url": m.get("url"), + "mime": m.get("mime"), "alt_text": m.get("altText")} + + def map_component(self, c): + tc = c.get("typeCode") + spec = self.cmap.get(tc) + # Nav components are NOT regular slot components — build_nav() flattens + # them into the nav_node_flat pool + a flat nav component and wires them + # into the shell itself. Skip them here (both the legacy "deferred-nav" + # and the profile's "nav-pass" status) so process_page never wires the + # shell with a recursive nav type that build_nav then can't override. + if not spec or spec.get("status") in ("deferred-nav", "nav-pass"): + if spec: + self.skipped_deferred.add(tc) + return None # not authored (functional or nav) + uid = c.get("uid") + if not uid: + return None + if uid in self.components: + return uid + ct = spec["content_type"] + m = c.get("media") or {} + fields = {"sap_uid": uid} + if tc == "SimpleResponsiveBannerComponent": + for bp, fld in (("mobile", "media_mobile"), ("tablet", "media_tablet"), + ("desktop", "media_desktop"), ("widescreen", "media_widescreen")): + d = m.get(bp) + if isinstance(d, dict) and d.get("code"): + fields[fld] = asset_ref(d["code"]); self.add_asset(d["code"]) + d = m.get("desktop") if isinstance(m.get("desktop"), dict) else None + if d and d.get("code"): + fields["media"] = asset_ref(d["code"]) + fields["url_link"] = c.get("urlLink") + fields["title"] = c.get("name") or uid + elif tc == "SimpleBannerComponent": + if m.get("code"): + fields["media"] = asset_ref(m["code"]); self.add_asset(m["code"]) + fields["url_link"] = c.get("urlLink") + fields["title"] = c.get("name") or uid + elif tc == "ProductCarouselComponent": + fields["title"] = c.get("title") or c.get("name") or uid # rendered heading + fields["products"] = [x for x in (c.get("productCodes") or "").split() if x] + elif tc == "CMSParagraphComponent": + fields["title"] = c.get("name") or uid + fields["content"] = c.get("content") + elif tc == "CMSLinkComponent": + fields["title"] = c.get("name") or uid + fields["link_name"] = c.get("linkName") + fields["url"] = c.get("url") + fields["target"] = as_target(c.get("target")) + elif tc == "CMSFlexComponent": + fields["title"] = c.get("name") or uid + fields["flex_type"] = c.get("flexType") + else: + fields["title"] = c.get("name") or uid + entry = {"_content_type": ct, "_key": {"field": "sap_uid", "value": uid}} + entry.update({k: v for k, v in fields.items() if v is not None}) + self.components[uid] = entry + return uid + + # ---- one page (content page or layout sample) ---- + SHARED_SLUG = {"product_page": "/product", "category_page": "/category"} + + def process_page(self, label, p, global_fields, url_override=None): + tpl = p.get("template") + pt = self.page_types.get(tpl) + if not pt: + return None # non-content page -> OCC, no CS page + slot_fields = {} + for s in p.get("slots", []): + pos = s.get("position") + if pos not in self.global_pos and pos in self.slot_map and self.pack_fields is not None \ + and self.slot_map[pos] not in self.pack_fields.get(pt["content_type"], set()): + self.unmapped_slots.add(f"{pos} ({pt['content_type']} has no {self.slot_map[pos]})") + continue # target type lacks the slot -> OCC + if pos not in self.global_pos and pos not in self.slot_map: + self.unmapped_slots.add(pos) # authored page, unknown slot -> OCC + continue + comp_uids = [u for u in (self.map_component(c) for c in s.get("components", [])) if u] + if not comp_uids: + continue + if pos in self.global_pos: # shell -> global_slots (dedupe by uid) + fld = self.global_pos[pos] + bucket = global_fields.setdefault(fld, []) + seen = {r["value"] for r in bucket} + for u in comp_uids: + if u not in seen: + bucket.append(ref(self.components[u]["_content_type"], u)); seen.add(u) + elif pos in self.slot_map: # page slot + fld = self.slot_map[pos] + slot_fields[fld] = [ref(self.components[u]["_content_type"], u) for u in comp_uids] + else: + self.unmapped_slots.add(pos) # authored page, unknown slot -> OCC + url = url_override or ("/" if label == "homepage" else "/" + label) + page = {"_content_type": pt["content_type"], "_key": {"field": "url", "value": url}, + "title": p.get("title") or label, "url": url, + "page_type": p.get("typeCode"), "template": tpl} + page.update(slot_fields) + return page + + # ---- navigation: link leaves -> FLAT nav_node_flat pool -> flat nav components ---- + def build_nav(self, global_fields): + """Emit navigation as a FLAT adjacency list, not a recursive tree. + + Each menu becomes a pool of `nav_node_flat` entries — every node points at + its parent by the plain-text `parent_id` (a node id, NOT a reference) and + orders siblings via `sort_order` — plus a `category/footer_navigation_flat` + component that references the whole pool through `all_nodes`. The connector + rebuilds the tree in code, so the menu resolves in one shallow, constant + include chain regardless of depth (no Delivery API include-depth cap). + + Every entry is keyed by `sap_uid` (the connector seeder only tracks + sap_uid-keyed entries in its ref map); nav_node_flat also carries node_id == + sap_uid and parent_id == parent's sap_uid for the connector to read. + Emission order (link leaves -> nodes -> component) keeps every _ref target + created before its referrer. + """ + roots = self.inv.get("navigationRoots", {}) + links = self.inv.get("navigationLinks", {}) + tree = self.inv.get("navigation", {}) + nav_entries = [] + if not roots: + return nav_entries + # 1. link leaves actually used by the nav trees + used = set() + def collect(node): + for e in (node.get("entries") or []): + if e.get("itemType") == "CMSLinkComponent" and e.get("itemId") in links: + used.add(e["itemId"]) + for ch in (node.get("children") or []): + collect(ch) + for r in roots.values(): + if tree.get(r["root"]): + collect(tree[r["root"]]) + for lid in sorted(used): + ld = links[lid] + nav_entries.append({"_content_type": "cms_link_component", + "_key": {"field": "sap_uid", "value": lid}, "sap_uid": lid, + "title": ld.get("name") or ld.get("link_name") or lid, + "link_name": ld.get("link_name"), "url": ld.get("url"), + "target": as_target(ld.get("target"))}) + # 2. flatten a menu tree into nav_node_flat entries (parent_id text + + # sort_order); append in pre-order so parents precede children. + def flatten(node, parent_uid, order, collected): + uid = node.get("uid") + if not uid: + return + node_links = [e["itemId"] for e in (node.get("entries") or []) + if e.get("itemType") == "CMSLinkComponent" and e.get("itemId") in links] + # A SAP leaf node usually has NO title — its label lives on the linked + # cms_link_component. Fall back to the link's name so the storefront + # shows "Contact Us", not the node uid "ContactUsNavNode". + leaf_label = links[node_links[0]].get("link_name") if node_links else None + entry = {"_content_type": "nav_node_flat", + "_key": {"field": "sap_uid", "value": uid}, "sap_uid": uid, + "node_id": uid, "parent_id": parent_uid, "sort_order": order, + "title": node.get("title") or node.get("name") or leaf_label or uid} + if node_links: + entry["links"] = [ref("cms_link_component", lid) for lid in node_links] + nav_entries.append(entry) + collected.append(uid) + for i, ch in enumerate(node.get("children") or [], start=1): + flatten(ch, uid, i, collected) + # 3. flat nav components -> placed into the shell (navigation_bar / footer) + for cuid, r in roots.items(): + root_node = tree.get(r["root"]) + if not root_node: + continue + node_uids = [] + # the root is a pure container: its children are the top-level items + for i, ch in enumerate(root_node.get("children") or [], start=1): + flatten(ch, "", i, node_uids) + ct = ("category_navigation_flat" + if r["typeCode"] == "CategoryNavigationComponent" else "footer_navigation_flat") + nav_entries.append({"_content_type": ct, "_key": {"field": "sap_uid", "value": cuid}, + "sap_uid": cuid, "title": cuid, "wrap_after": as_int(r.get("wrapAfter")), + "all_nodes": [ref("nav_node_flat", nid) for nid in node_uids]}) + fld = self.global_pos.get(r.get("position")) + if fld: + bucket = global_fields.setdefault(fld, []) + if cuid not in {x["value"] for x in bucket}: + bucket.append(ref(ct, cuid)) + return nav_entries + + def run(self): + pages, global_fields = [], {} + for label, p in self.inv.get("contentPages", {}).items(): + pg = self.process_page(label, p, global_fields) + if pg: + pages.append(pg) + for label, p in self.inv.get("layoutSamples", {}).items(): # product/category pages + pt = self.page_types.get(p.get("template")) + if not pt: + continue + pg = self.process_page(label, p, global_fields, + url_override=self.SHARED_SLUG.get(pt["content_type"])) + if pg: + pages.append(pg) + nav_entries = self.build_nav(global_fields) # after pages so shell fields exist + + global_entry = None + if global_fields: + global_entry = {"_content_type": self.global_ct, + "_key": {"field": "title", "value": self.inv.get("site")}, + "title": self.inv.get("site")} + # Clear any shell field we no longer author (e.g. site_context/site_links now + # left to OCC) so an update removes stale references instead of keeping them. + ALL_SHELL = ["site_logo", "search_box", "mini_cart", "navigation_bar", + "site_context", "site_links", "header_links", "footer"] + for fld in ALL_SHELL: + global_entry.setdefault(fld, []) + global_entry.update(global_fields) + + components_out = list(self.components.values()) + nav_entries # nav ordered: links->nodes->components + used_cts = sorted({e["_content_type"] for e in components_out} | + {p["_content_type"] for p in pages} | + ({self.global_ct} if global_entry else set())) + from collections import Counter + nav_by_ct = Counter(e["_content_type"] for e in nav_entries) + plan = { + "site": self.inv.get("site"), "profile": "connector-reference-fields", + "content_types_needed": used_cts, + "assets": list(self.assets.values()), + "entries": {"components": components_out, + "global_slots": global_entry, "pages": pages}, + } + report = { + "site": self.inv.get("site"), + "authored_pages": [(p["url"], p["_content_type"]) for p in pages], + "component_entries": len(self.components), + "nav_entries": dict(nav_by_ct), + "assets": len(self.assets), + "global_slots_fields": sorted(global_fields.keys()), + "unmapped_slot_positions": sorted(self.unmapped_slots), + } + return plan, report + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--inventory", required=True) + ap.add_argument("--profile", required=True) + ap.add_argument("--out", required=True) + ap.add_argument("--starter-pack", default="", help="connector starter-pack content_types dir (drops slots a page type lacks)") + a = ap.parse_args() + inv = json.load(open(a.inventory)); prof = json.load(open(a.profile)) + os.makedirs(a.out, exist_ok=True) + pack_fields = None + if a.starter_pack: + pack_fields = {} + for f in os.listdir(a.starter_pack): + if f.endswith(".json") and f != "schema.json": + ct = json.load(open(os.path.join(a.starter_pack, f))) + if isinstance(ct, dict) and ct.get("uid"): + pack_fields[ct["uid"]] = {x["uid"] for x in ct.get("schema", [])} + plan, report = ConnectorModeler(inv, prof, pack_fields).run() + json.dump(plan, open(os.path.join(a.out, "connector-plan.json"), "w"), indent=2) + json.dump(report, open(os.path.join(a.out, "connector-report.json"), "w"), indent=2) + print(f"[model_connector] {report['site']} (profile: connector-reference-fields)") + print(f" authored pages: {len(report['authored_pages'])} -> {report['authored_pages']}") + print(f" component entries: {report['component_entries']} assets: {report['assets']}") + print(f" content types needed: {plan['content_types_needed']}") + print(f" global_slots fields: {report['global_slots_fields']}") + if report.get("nav_entries"): + print(f" nav entries: {report['nav_entries']}") + if report["unmapped_slot_positions"]: + print(f" unmapped slots -> OCC: {report['unmapped_slot_positions']}") + print(f" -> {a.out}/connector-plan.json") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed.py b/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed.py new file mode 100644 index 0000000..803f0c4 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed.py @@ -0,0 +1,346 @@ +#!/usr/bin/env python3 +""" +cma_seed — live Contentstack Management API executor for the seed plan. + +Applies a content-model output (content-types.json + entry-plan.json) to a real stack, +idempotently and in dependency order. English (master locale) only unless --locales given. + +Reads credentials from env: CS_API_KEY, CS_MANAGEMENT_TOKEN, CS_REGION_BASE +(default https://api.contentstack.io). Never logs the token. + +Stages (comma list, default all): globals,content_types,assets,entries + python3 cma_seed.py --model-dir DIR --site electronics-spa --stages globals,content_types + python3 cma_seed.py --model-dir DIR --site electronics-spa # full run + +Idempotent: content types & global fields by uid; assets by a `sap_code:` tag; +entries by their unique key field. Re-running updates in place. +""" +import argparse, json, mimetypes, os, ssl, sys, time, urllib.parse, urllib.request + +API = os.environ.get("CS_REGION_BASE", "https://api.contentstack.io").rstrip("/") +KEY = os.environ.get("CS_API_KEY") +TOKEN = os.environ.get("CS_MANAGEMENT_TOKEN") +MASTER = os.environ.get("CS_MASTER_LOCALE", "en-us") +OCC_HOST = (os.environ.get("OCC_BASE_URL") or "").split("/occ/")[0] + + +def occ_ssl_context(): + """TLS context for OCC calls. Verified by default; set OCC_INSECURE_TLS=1 only for a + dev backend with a self-signed certificate you trust.""" + ctx = ssl.create_default_context() + if os.environ.get("OCC_INSECURE_TLS") == "1": + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + return ctx + + +OCC_CTX = occ_ssl_context() + + +def die(msg): + sys.exit(f"[cma_seed] ERROR: {msg}") + + +def cma(method, path, body=None, raw=None, content_type=None, retry=3): + url = f"{API}/v3{path}" + data, ct = None, None + if raw is not None: + data, ct = raw, content_type + elif body is not None: + data, ct = json.dumps(body).encode(), "application/json" + req = urllib.request.Request(url, data=data, method=method) + req.add_header("api_key", KEY) + req.add_header("authorization", TOKEN) + if ct: + req.add_header("Content-Type", ct) + for attempt in range(retry): + try: + with urllib.request.urlopen(req, timeout=60) as r: + return r.status, json.loads(r.read().decode() or "{}") + except urllib.error.HTTPError as e: + payload = e.read().decode() + if e.code == 429 and attempt < retry - 1: # rate limited + time.sleep(1.5 * (attempt + 1)); continue + try: + return e.code, json.loads(payload) + except Exception: + return e.code, {"_raw": payload} + except Exception as e: + if attempt < retry - 1: + time.sleep(1); continue + return 0, {"_error": str(e)} + return 0, {"_error": "unreachable"} + + +# ---------------- schema normalization ---------------- +def ensure_title(ct): + """Contentstack requires a text field with uid 'title'. Inject one if absent and + point options.title at it. Other fields (page_label, slot_id, sap_name...) stay.""" + fields = ct.setdefault("schema", []) + if not any(f.get("uid") == "title" for f in fields): + fields.insert(0, {"display_name": "Title", "uid": "title", "data_type": "text", + "mandatory": True, "unique": False, + "field_metadata": {"_default": True}, "multiple": False}) + ct.setdefault("options", {}) + ct["options"]["title"] = "title" + ct["options"].setdefault("singleton", False) + # scrub any leftover _default on non-title fields (only title may be default) + for f in fields: + if f.get("uid") != "title": + fm = f.get("field_metadata") or {} + fm.pop("_default", None) + return ct + + +def html_rte_field(uid, display): + return {"display_name": display, "uid": uid, "data_type": "text", + "field_metadata": {"allow_rich_text": True, "rich_text_type": "advanced", + "description": "", "multiline": False, "version": 3}, + "multiple": False, "mandatory": False, "unique": False} + + +def normalize_content_type(ct): + """Make the registry-derived schema CMA-safe.""" + ct = json.loads(json.dumps(ct)) # deep copy + # rich_text.content -> HTML-based advanced RTE (stores HTML directly; round-trips clean) + if ct["uid"] == "rich_text": + ct["schema"] = [f for f in ct["schema"] if f.get("uid") != "content"] + ct["schema"].append(html_rte_field("content", "Content")) + # passthrough config: CS has no generic JSON field -> store JSON as multiline text + if ct["uid"] == "passthrough_component": + for f in ct["schema"]: + if f.get("uid") == "config": + f["data_type"] = "text" + f["field_metadata"] = {"description": "JSON-encoded verbatim OCC fields", + "multiline": True} + f.pop("enum", None) + for f in ct["schema"]: + if f.get("data_type") in ("reference", "global_field"): + f.setdefault("field_metadata", {}) + if f.get("data_type") == "reference": + f["field_metadata"].setdefault("ref_multiple", bool(f.get("field_metadata", {}).get("ref_multiple"))) + # CS select fields (enum) require a display_type + if f.get("enum"): + f["display_type"] = "dropdown" + f.setdefault("field_metadata", {}) + return ensure_title(ct) + + +# ---------------- idempotent upserts ---------------- +def upsert_global_field(gf): + uid = gf["uid"] + st, _ = cma("GET", f"/global_fields/{uid}") + body = {"global_field": gf} + if st == 200: + st2, r = cma("PUT", f"/global_fields/{uid}", body) + return "updated" if st2 == 200 else f"ERR {st2}:{r}" + st2, r = cma("POST", "/global_fields", body) + return "created" if st2 in (200, 201) else f"ERR {st2}:{json.dumps(r)[:300]}" + + +def upsert_content_type(ct): + uid = ct["uid"] + st, _ = cma("GET", f"/content_types/{uid}") + body = {"content_type": ct} + if st == 200: + st2, r = cma("PUT", f"/content_types/{uid}", body) + return "updated" if st2 == 200 else f"ERR {st2}:{json.dumps(r)[:400]}" + st2, r = cma("POST", "/content_types", body) + return "created" if st2 in (200, 201) else f"ERR {st2}:{json.dumps(r)[:400]}" + + +def download_occ(url): + full = url if url.startswith("http") else OCC_HOST + url + with urllib.request.urlopen(urllib.request.Request(full), context=OCC_CTX, timeout=60) as r: + return r.read() + + +def multipart(fields, filename, filedata, mime): + boundary = "----csseed" + str(len(filedata)) + out = [] + for k, v in fields.items(): + out.append(f"--{boundary}\r\nContent-Disposition: form-data; name=\"{k}\"\r\n\r\n{v}\r\n".encode()) + out.append((f"--{boundary}\r\nContent-Disposition: form-data; name=\"asset[upload]\"; " + f"filename=\"{filename}\"\r\nContent-Type: {mime}\r\n\r\n").encode()) + out.append(filedata) + out.append(f"\r\n--{boundary}--\r\n".encode()) + return b"".join(out), f"multipart/form-data; boundary={boundary}" + + +def asset_lookup(code): + """Idempotency key = asset title (= the media code; no length limit, unlike tags).""" + q = urllib.parse.quote(json.dumps({"title": code})) + st, r = cma("GET", f"/assets?query={q}&limit=1") + if st == 200 and r.get("assets"): + return r["assets"][0]["uid"] + return None + + +def upsert_asset(a, cache): + code = a["code"] + existing = asset_lookup(code) + if existing: + cache[code] = existing + return "exists" + try: + data = download_occ(a["url"]) + except Exception as e: + return f"ERR download:{e}" + fname = code.split("/")[-1].split("?")[0] or "asset" + mime = a.get("mime") or mimetypes.guess_type(fname)[0] or "application/octet-stream" + short_tag = "sapmedia_" + str(abs(hash(code)) % 10**10) # <=50 chars, stable within run + raw, ct = multipart({"asset[title]": code, "asset[tags]": short_tag}, fname, data, mime) + st2, r2 = cma("POST", "/assets", raw=raw, content_type=ct) + if st2 in (200, 201): + cache[code] = r2["asset"]["uid"] + return "created" + return f"ERR {st2}:{json.dumps(r2)[:200]}" + + +def resolve_value(v, key2uid, asset_cache): + """Turn plan placeholders (_ref/_asset/_rte_from_html) into CMA entry values.""" + if isinstance(v, dict): + if v.get("_asset"): + return asset_cache.get(v["code"]) + if v.get("_ref"): + uid = key2uid.get((v["content_type"], v["value"])) + return [{"uid": uid, "_content_type_uid": v["content_type"]}] if uid else None + if "_rte_from_html" in v: + return v["_rte_from_html"] or "" + return {k: resolve_value(x, key2uid, asset_cache) for k, x in v.items()} + if isinstance(v, list): + out = [] + for x in v: + rv = resolve_value(x, key2uid, asset_cache) + if isinstance(x, dict) and x.get("_ref"): + if rv: + out.extend(rv) # flatten single-ref arrays into the list + else: + out.append(rv) + return out + return v + + +def entry_payload(e, key2uid, asset_cache): + skip = {"_content_type", "_key", "_localized"} + body = {} + for k, v in e.items(): + if k in skip: + continue + body[k] = resolve_value(v, key2uid, asset_cache) + # passthrough config is a multiline-text field holding JSON + if e.get("_content_type") == "passthrough_component" and isinstance(body.get("config"), (dict, list)): + body["config"] = json.dumps(body["config"], ensure_ascii=False) + return {"entry": body} + + +def find_entry_uid(ct, key_field, key_value): + q = urllib.parse.quote(json.dumps({key_field: key_value})) + st, r = cma("GET", f"/content_types/{ct}/entries?query={q}&limit=1") + if st == 200 and r.get("entries"): + return r["entries"][0]["uid"] + return None + + +def upsert_entry(e, key2uid, asset_cache): + ct = e["_content_type"] + kf, kv = e["_key"]["field"], e["_key"]["value"] + body = entry_payload(e, key2uid, asset_cache) + existing = find_entry_uid(ct, kf, kv) + if existing: + st, r = cma("PUT", f"/content_types/{ct}/entries/{existing}", body) + uid = existing + status = "updated" if st == 200 else f"ERR {st}:{json.dumps(r)[:200]}" + else: + st, r = cma("POST", f"/content_types/{ct}/entries", body) + uid = r.get("entry", {}).get("uid") + status = "created" if st in (200, 201) else f"ERR {st}:{json.dumps(r)[:200]}" + if uid: + key2uid[(ct, kv)] = uid + return status + + +# ---------------- driver ---------------- +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--model-dir", required=True) + ap.add_argument("--site", required=True) + ap.add_argument("--stages", default="globals,content_types,assets,entries") + ap.add_argument("--limit-entries", type=int, default=0, help="cap entries per group (smoke test)") + a = ap.parse_args() + if not (KEY and TOKEN): + die("CS_API_KEY and CS_MANAGEMENT_TOKEN must be set in the environment") + stages = set(s.strip() for s in a.stages.split(",")) + + ct_doc = json.load(open(os.path.join(a.model_dir, "content-types.json"))) + ep = json.load(open(os.path.join(a.model_dir, "entry-plan.json"))) + + # topo order for content types (reuse seed.py logic) + sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + import seed + order = seed.topo_content_types(ct_doc["content_types"]) + by_uid = {c["uid"]: c for c in ct_doc["content_types"]} + + key2uid, asset_cache = {}, {} + + if "globals" in stages: + print("[globals]") + for gf in ct_doc.get("global_fields", []): + print(f" {gf['uid']}: {upsert_global_field(gf)}") + + if "content_types" in stages: + print("[content_types] (dependency order)") + for uid in order: + ct = normalize_content_type(by_uid[uid]) + print(f" {uid}: {upsert_content_type(ct)}") + + if "assets" in stages: + print(f"[assets] {len(ep['assets'])}") + ok = err = 0 + for asset in ep["assets"]: + r = upsert_asset(asset, asset_cache) + if r.startswith("ERR"): + err += 1; print(f" {asset['code']}: {r}") + else: + ok += 1 + print(f" assets: {ok} ok, {err} err, {len(asset_cache)} in cache") + + if "entries" in stages: + # assets must be resolvable for banner refs even if we skipped the stage this run + if not asset_cache and "assets" not in stages: + for asset in ep["assets"]: + uid = asset_lookup(asset["code"]) + if uid: + asset_cache[asset["code"]] = uid + # Nav components hold an all_nodes pool of nav_node_flat entries, which in + # turn reference `link` leaves (ordinary components). Refs resolve only + # against entries already created this run (key2uid), so seed in this + # order: non-nav components (incl. link leaves) -> nav_nodes -> nav + # components -> slots -> pages. + NAV_CTS = {"category_navigation_flat", "footer_navigation_flat"} + comps = ep["entries"].get("components", []) + seed_groups = [ + ("components", [e for e in comps if e["_content_type"] not in NAV_CTS]), + ("nav_nodes", ep["entries"].get("nav_nodes", [])), + ("nav_components", [e for e in comps if e["_content_type"] in NAV_CTS]), + ("slots", ep["entries"]["slots"]), + ("pages", ep["entries"]["pages"]), + ] + for group, items in seed_groups: + if a.limit_entries: + items = items[:a.limit_entries] + print(f"[entries:{group}] {len(items)}") + ok = err = 0 + for e in items: + r = upsert_entry(e, key2uid, asset_cache) + if r.startswith("ERR"): + err += 1 + if err <= 5: + print(f" {e['_key']['value']}: {r}") + else: + ok += 1 + print(f" {group}: {ok} ok, {err} err") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed_connector.py b/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed_connector.py new file mode 100644 index 0000000..34f971e --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/cs-seed/cma_seed_connector.py @@ -0,0 +1,190 @@ +#!/usr/bin/env python3 +""" +cma_seed_connector — live seeder for the CONNECTOR reference-field profile (Phase 4b). + +Imports the connector's shipped starter-pack content types into the stack, then applies a +connector-plan.json (from model_connector) idempotently: + content types (starter pack, +sap_uid idempotency field on components) + -> assets -> component entries (by sap_uid) -> global_slots (by title) -> pages (by url). + +Reuses low-level CMA helpers from cma_seed.py. Creds from env (CS_API_KEY, +CS_MANAGEMENT_TOKEN, CS_REGION_BASE). English/master locale only. + +Usage: + python3 cma_seed_connector.py --plan /connector-plan.json \ + --starter-pack /import-export/starter-pack/content_types \ + --stages types,assets,entries +""" +import argparse, json, os, sys, urllib.parse +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import cma_seed as low # reuse cma(), asset_lookup(), upsert_asset(), download_occ, multipart + +PAGE_TYPES = {"landing_page", "content_page", "product_page", "category_page"} +GLOBAL_TYPE = "global_slots" + +SAP_UID_FIELD = {"display_name": "SAP UID", "uid": "sap_uid", "data_type": "text", + "mandatory": False, "unique": True, "multiple": False, + "field_metadata": {"instruction": "Seed idempotency key = SAP component uid."}} + + +def load_starter_types(dir_): + out = {} + for fn in os.listdir(dir_): + if fn.endswith(".json"): + ct = json.load(open(os.path.join(dir_, fn))) + out[ct["uid"]] = ct + return out + + +def topo_types(starter): + """Order all starter content types so every reference target precedes its user.""" + ordered, seen = [], set() + def deps(ct): + d = set() + for f in ct.get("schema", []): + if f.get("data_type") == "reference": + for t in (f.get("reference_to") or []): + if t in starter and t != ct["uid"]: + d.add(t) + return d + def visit(uid, stack): + if uid in seen or uid in stack: + return + stack.add(uid) + for d in deps(starter[uid]): + visit(d, stack) + stack.discard(uid); seen.add(uid); ordered.append(uid) + for uid in starter: + visit(uid, set()) + return ordered + + +def import_content_types(needed, starter, verbose=True): + # Import the FULL shipped model (topo-ordered) so every reference_to resolves, + # even for types whose entries we defer (nav). Inject sap_uid on component types. + order = topo_types(starter) + for uid in order: + ct = json.loads(json.dumps(starter[uid])) + is_component = uid not in PAGE_TYPES and uid != GLOBAL_TYPE + if is_component and not any(f.get("uid") == "sap_uid" for f in ct["schema"]): + ct["schema"].append(dict(SAP_UID_FIELD)) + st, _ = low.cma("GET", f"/content_types/{uid}") + body = {"content_type": ct} + if st == 200: + st2, r = low.cma("PUT", f"/content_types/{uid}", body) + res = "updated" if st2 == 200 else f"ERR {st2}:{json.dumps(r)[:200]}" + else: + st2, r = low.cma("POST", "/content_types", body) + res = "created" if st2 in (200, 201) else f"ERR {st2}:{json.dumps(r)[:200]}" + if verbose: + print(f" {uid}: {res}") + + +def resolve(v, uid_map, asset_cache): + if isinstance(v, dict): + if v.get("_asset"): + return asset_cache.get(v["code"]) + if v.get("_ref"): + uid = uid_map.get(v["value"]) + return [{"uid": uid, "_content_type_uid": v["content_type"]}] if uid else None + return {k: resolve(x, uid_map, asset_cache) for k, x in v.items()} + if isinstance(v, list): + out = [] + for x in v: + rv = resolve(x, uid_map, asset_cache) + if isinstance(x, dict) and x.get("_ref"): + if rv: + out.extend(rv) + else: + out.append(rv) + return out + return v + + +def entry_body(e, uid_map, asset_cache): + return {"entry": {k: resolve(v, uid_map, asset_cache) + for k, v in e.items() if not k.startswith("_")}} + + +def find_uid(ct, field, value): + q = urllib.parse.quote(json.dumps({field: value})) + st, r = low.cma("GET", f"/content_types/{ct}/entries?query={q}&limit=1") + return (r.get("entries") or [{}])[0].get("uid") if st == 200 else None + + +def upsert_entry(e, uid_map, asset_cache): + ct = e["_content_type"]; kf, kv = e["_key"]["field"], e["_key"]["value"] + body = entry_body(e, uid_map, asset_cache) + existing = find_uid(ct, kf, kv) + if existing: + st, r = low.cma("PUT", f"/content_types/{ct}/entries/{existing}", body) + uid, res = existing, ("updated" if st == 200 else f"ERR {st}:{json.dumps(r)[:160]}") + else: + st, r = low.cma("POST", f"/content_types/{ct}/entries", body) + uid, res = r.get("entry", {}).get("uid"), ("created" if st in (200, 201) else f"ERR {st}:{json.dumps(r)[:160]}") + if uid and kf == "sap_uid": + uid_map[kv] = uid + return res, uid + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--plan", required=True) + ap.add_argument("--starter-pack", required=True) + ap.add_argument("--stages", default="types,assets,entries") + a = ap.parse_args() + if not (low.KEY and low.TOKEN): + low.die("CS_API_KEY and CS_MANAGEMENT_TOKEN must be set") + stages = set(s.strip() for s in a.stages.split(",")) + plan = json.load(open(a.plan)) + uid_map, asset_cache = {}, {} + + if "types" in stages: + print("[content types] (starter pack, +sap_uid on components)") + import_content_types(plan["content_types_needed"], load_starter_types(a.starter_pack)) + + if "assets" in stages: + print(f"[assets] {len(plan['assets'])}") + ok = err = 0 + for asset in plan["assets"]: + r = low.upsert_asset(asset, asset_cache) + ok += 1 if not r.startswith("ERR") else 0 + err += 1 if r.startswith("ERR") else 0 + if r.startswith("ERR"): + print(f" {asset['code']}: {r}") + print(f" assets: {ok} ok, {err} err") + + if "entries" in stages: + if not asset_cache: # resolve existing assets for banner refs + for asset in plan["assets"]: + u = low.asset_lookup(asset["code"]) + if u: + asset_cache[asset["code"]] = u + comps = plan["entries"]["components"] + print(f"[components] {len(comps)}") + ok = err = 0 + for e in comps: + res, _ = upsert_entry(e, uid_map, asset_cache) + ok += 1 if not res.startswith("ERR") else 0 + if res.startswith("ERR"): + err += 1 + if err <= 6: + print(f" {e['_key']['value']}: {res}") + print(f" components: {ok} ok, {err} err") + gs = plan["entries"].get("global_slots") + if gs: + res, _ = upsert_entry(gs, uid_map, asset_cache) + print(f"[global_slots] {res}") + pages = plan["entries"]["pages"] + print(f"[pages] {len(pages)}") + ok = err = 0 + for e in pages: + res, _ = upsert_entry(e, uid_map, asset_cache) + ok += 1 if not res.startswith("ERR") else 0 + if res.startswith("ERR"): + err += 1; print(f" {e['_key']['value']}: {res}") + print(f" pages: {ok} ok, {err} err") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/cs-seed/localize.py b/skills/dx-sap-spartacus-import/scripts/cs-seed/localize.py new file mode 100644 index 0000000..cd74b69 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/cs-seed/localize.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 +""" +localize — full per-locale localization from the FULLY LOCALIZED OCC response (i18n v2). + +Unlike a field-diff, this drives off a complete localized inventory (extract.py run with +lang=de/ja/zh + nav_resolve --lang), so pages, components, nav AND per-locale media are all +localized. For one target locale it: + 1. uploads the locale's images (different per language, e.g. ...DE...) as CS assets, + 2. localizes EVERY authored entry (components + pages + global_slots + nav) by writing a + locale variant (PUT ?locale) with localized scalar/text fields + localized asset refs; + reference fields are left to inherit from master so they resolve to localized targets, + 3. publishes each localized entry to the environment in that locale (?locale=, required — + else CS 141 "Localised entries can not be published from master locale"). + +The base (master en-us) entries must already be seeded. Matches entries by their key +(sap_uid / url / title). Idempotent. Creds from env (CS_API_KEY, CS_MANAGEMENT_TOKEN). + +Usage: + python3 localize.py --locale de-de --plan /connector-plan.json --env development +""" +import argparse, json, os, sys, urllib.parse +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import cma_seed as low # cma(), upsert_asset(), asset_lookup(), download_occ + + +_LOCALE_NAMES = {"de-de": "German - Germany", "ja-jp": "Japanese - Japan", + "zh-cn": "Chinese - China", "fr-fr": "French - France", + "es-es": "Spanish - Spain", "it-it": "Italian - Italy"} + + +def ensure_locale(code, master): + """Create the target locale (fallback = master) if the stack doesn't have it.""" + st, r = low.cma("GET", "/locales") + if code in {l["code"] for l in (r.get("locales") or [])}: + return "exists" + st2, r2 = low.cma("POST", "/locales", + {"locale": {"code": code, "fallback_locale": master, + "name": _LOCALE_NAMES.get(code, code)}}) + return "created" if st2 in (200, 201) else f"ERR {st2}:{json.dumps(r2)[:120]}" + + +def find_uid(ct, field, value, master): + q = urllib.parse.quote(json.dumps({field: value})) + st, r = low.cma("GET", f"/content_types/{ct}/entries?query={q}&locale={master}&limit=1") + es = r.get("entries") or [] + return es[0]["uid"] if es else None + + +def loc_fields(e, asset_cache, ref_uid): + """Full localized payload. IMPORTANT: a Contentstack localized entry is standalone — + it does NOT per-field-inherit from master, so we must send EVERY field, including + reference fields (resolved to the shared entry uids, which point to the localized + versions at query time) and localized asset uids.""" + def resolve(v): + if isinstance(v, dict): + if v.get("_asset"): + return asset_cache.get(v["code"]) + if v.get("_ref"): + uid = ref_uid(v["content_type"], v["value"]) + return [{"uid": uid, "_content_type_uid": v["content_type"]}] if uid else None + return {k: resolve(x) for k, x in v.items()} + if isinstance(v, list): + out = [] + for x in v: + rv = resolve(x) + if isinstance(x, dict) and x.get("_ref"): + if rv: + out.extend(rv) + elif rv is not None: + out.append(rv) + return out + return v + return {k: resolve(v) for k, v in e.items() if not k.startswith("_") and k != "sap_meta"} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--locale", required=True, help="target CS locale, e.g. de-de") + ap.add_argument("--plan", required=True, help="connector-plan.json from the LOCALIZED inventory") + ap.add_argument("--env", default="development") + ap.add_argument("--master", default="en-us") + a = ap.parse_args() + if not (low.KEY and low.TOKEN): + low.die("CS_API_KEY and CS_MANAGEMENT_TOKEN must be set") + loc = a.locale + plan = json.load(open(a.plan)) + print(f"[{loc}] locale: {ensure_locale(loc, a.master)}") + + # 1) localized assets (different image per language) -> code->uid, then PUBLISH them + # (unpublished assets are not delivered -> nested banner media resolves to null). + asset_cache = {} + ok = err = 0 + for asset in plan.get("assets", []): + r = low.upsert_asset(asset, asset_cache) # downloads localized bytes, uploads by title + ok += 1 if not r.startswith("ERR") else 0 + err += 1 if r.startswith("ERR") else 0 + pub = 0 + for aid in set(asset_cache.values()): + st, _ = low.cma("POST", f"/assets/{aid}/publish", + {"asset": {"environments": [a.env], "locales": [a.master, loc]}}) + pub += 1 if st in (200, 201) else 0 + print(f"[{loc}] assets: {ok} ok, {err} err, {len(asset_cache)} resolved, {pub} published") + + # resolve a reference target's entry uid by sap_uid (shared across locales), cached + ref_cache = {} + def ref_uid(ct, sap_uid): + key = (ct, sap_uid) + if key not in ref_cache: + ref_cache[key] = find_uid(ct, "sap_uid", sap_uid, a.master) + return ref_cache[key] + + def do(entries, label): + put_ok = pub_ok = miss = e_err = 0 + for e in entries: + ct = e["_content_type"]; kf = e["_key"]["field"]; kv = e["_key"]["value"] + uid = find_uid(ct, kf, kv, a.master) + if not uid: + miss += 1; continue + fields = loc_fields(e, asset_cache, ref_uid) + st, r = low.cma("PUT", f"/content_types/{ct}/entries/{uid}?locale={loc}", {"entry": fields}) + if st not in (200, 201): + e_err += 1 + if e_err <= 4: + print(f" {loc} {ct}/{kv}: PUT {st} {json.dumps(r)[:120]}") + continue + put_ok += 1 + st2, _ = low.cma("POST", f"/content_types/{ct}/entries/{uid}/publish?locale={loc}", + {"entry": {"environments": [a.env], "locales": [loc]}}) + pub_ok += 1 if st2 in (200, 201) else 0 + print(f" [{loc}] {label}: localized {put_ok}, published {pub_ok}, missing {miss}, err {e_err}") + + do(plan["entries"]["components"], "components+nav") + if plan["entries"].get("global_slots"): + do([plan["entries"]["global_slots"]], "global_slots") + do(plan["entries"]["pages"], "pages") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/cs-seed/publish.py b/skills/dx-sap-spartacus-import/scripts/cs-seed/publish.py new file mode 100644 index 0000000..892892d --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/cs-seed/publish.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +""" +publish — publish seeded entries + assets to a Contentstack environment (Phase 4b). + +Publishing makes entries readable by a delivery token (a storefront/CDA). Publishes every +entry of every content type on the stack, plus every asset, to the given environment/locale. +Idempotent (re-publishing is safe). Creds from env (CS_API_KEY, CS_MANAGEMENT_TOKEN). + +Usage: python3 publish.py --env development [--locale en-us] +""" +import argparse, json, os, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import cma_seed as low + + +def ensure_environment(name): + """Create the target environment if the stack doesn't have it yet (fresh-stack support).""" + st, r = low.cma("GET", "/environments") + existing = {e["name"] for e in (r.get("environments") or [])} + if name in existing: + return "exists" + st2, r2 = low.cma("POST", "/environments", + {"environment": {"name": name, "urls": [{"locale": "en-us", "url": "http://localhost:4200"}]}}) + return "created" if st2 in (200, 201) else f"ERR {st2}:{json.dumps(r2)[:160]}" + + +def list_all(path, key): + out, skip = [], 0 + while True: + st, r = low.cma("GET", f"{path}{'&' if '?' in path else '?'}limit=100&skip={skip}&include_count=true") + items = r.get(key) or [] + out += items; skip += len(items) + if len(items) < 100 or skip >= (r.get("count") or 0): + break + return out + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--env", required=True) + ap.add_argument("--locale", default="en-us") + a = ap.parse_args() + if not (low.KEY and low.TOKEN): + low.die("CS_API_KEY and CS_MANAGEMENT_TOKEN must be set") + + envs, loc = [a.env], [a.locale] + print(f"[environment '{a.env}'] {ensure_environment(a.env)}") + # assets first, then entries (components/shell) then pages last + assets = list_all("/assets", "assets") + print(f"[publish assets] {len(assets)}") + ok = err = 0 + for x in assets: + st, r = low.cma("POST", f"/assets/{x['uid']}/publish", + {"asset": {"environments": envs, "locales": loc}}) + ok += st in (200, 201); err += st not in (200, 201) + print(f" assets: {ok} ok, {err} err") + + cts = [c["uid"] for c in list_all("/content_types", "content_types")] + # order: components, then global_slots, then pages + page_types = {"landing_page", "content_page", "product_page", "category_page"} + order = ([c for c in cts if c not in page_types and c != "global_slots"] + + [c for c in cts if c == "global_slots"] + + [c for c in cts if c in page_types]) + for ct in order: + entries = list_all(f"/content_types/{ct}/entries", "entries") + if not entries: + continue + ok = err = 0 + for e in entries: + st, r = low.cma("POST", f"/content_types/{ct}/entries/{e['uid']}/publish", + {"entry": {"environments": envs, "locales": loc}}) + ok += st in (200, 201); err += st not in (200, 201) + if st not in (200, 201) and err <= 3: + print(f" {ct}/{e['uid']}: {st} {json.dumps(r)[:120]}") + print(f"[publish {ct}] {ok} ok, {err} err") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/orchestrate.py b/skills/dx-sap-spartacus-import/scripts/orchestrate.py new file mode 100644 index 0000000..87bffd0 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/orchestrate.py @@ -0,0 +1,289 @@ +#!/usr/bin/env python3 +""" +orchestrate — the content-modeling AGENT: one governed command that runs the whole +pipeline for a storefront, with two human gates. + + extract ──▶ nav-resolve ──▶ model ──▶ [GATE 1: model approval] + ──▶ [GATE 2: pre-publish] ──▶ seed ──▶ publish ──▶ verify + +It sequences the skills (each a standalone, tested script), enforces the gates, and +handles the one judgment call the skills don't: an UNKNOWN typeCode (Gate 1). Everything +is idempotent — re-running updates in place — and resumable via the run directory. + +GATES (fail-closed): the pipeline STOPS at each gate unless the matching approval flag is +passed, so nothing is created or published without an explicit human go. + Gate 1 model approval -> pass --approve-model (only blocks if unknown typeCodes found) + Gate 2 pre-publish -> pass --approve-publish (always blocks before any write) + +Default (no approval flags) = a full DRY RUN: extract→model + both gate reports, no writes. + +Usage (from the Spartacus project root, after `ng add @contentstack/contentstack-spartacus-connector`): + # dry run (safe): see the plan + gate reports, write nothing + python3 orchestrate.py --site electronics-spa + + # full governed run (requires CS_* env for seed/publish/verify) + python3 orchestrate.py --site electronics-spa --approve-model --approve-publish --env development + +Originally written by Vaishnavi Kedar (spartacus-content-modelling-agent); packaged as the +dx-sap-spartacus-import skill. +""" +import argparse, json, os, subprocess, sys, datetime, tempfile + +ROOT = os.path.dirname(os.path.abspath(__file__)) +SKILLS = ROOT +PROFILE = os.path.join(ROOT, "registry", "connector.registry.json") +# The connector's shipped starter-pack content types are the model of record. Default: the +# copy `ng add` installed in the current project; override with --starter-pack or +# CONNECTOR_STARTER_PACK (e.g. a connector checkout). +_INSTALLED = os.path.join(os.getcwd(), "node_modules", "@contentstack", "contentstack-spartacus-connector", + "import-export", "starter-pack", "content_types") +DEFAULT_STARTER = os.environ.get("CONNECTOR_STARTER_PACK", _INSTALLED) +DEFAULT_OCC = os.environ.get("OCC_BASE_URL") + +C = {"h": "\033[1m", "ok": "\033[32m", "warn": "\033[33m", "err": "\033[31m", + "gate": "\033[35m", "z": "\033[0m"} + + +def banner(txt, color="h"): + line = "─" * (len(txt) + 2) + print(f"{C[color]}┌{line}┐\n│ {txt} │\n└{line}┘{C['z']}") + + +def step(txt): + print(f"{C['h']}▶ {txt}{C['z']}") + + +OCC_BASE = None # set in main(); exported to every stage so asset downloads hit the same OCC + + +def run(script, args, env=None): + """Run a skill script; stream nothing, return (rc, output).""" + env = dict(env or os.environ) + if OCC_BASE: + env["OCC_BASE_URL"] = OCC_BASE + p = subprocess.run([sys.executable, os.path.join(SKILLS, script), *args], + capture_output=True, text=True, env=env) + out = (p.stdout or "") + (p.stderr or "") + return p.returncode, out + + +def known_typecodes(profile): + comp = {k for k in profile["component_map"] if not k.startswith("_")} + occ = set(profile["occ_only_types"]["types"]) + return comp | occ + + +def load_pack(dir_): + """uid -> content type JSON for every type in the starter pack (schema.json skipped).""" + pack = {} + for f in sorted(os.listdir(dir_)): + if f.endswith(".json") and f != "schema.json": + ct = json.load(open(os.path.join(dir_, f))) + if isinstance(ct, dict) and ct.get("uid"): + pack[ct["uid"]] = ct + return pack + + +def pack_mismatches(plan, pack): + """Every content type and field the plan writes must exist in the starter pack, or the + seed fails half-way. sap_uid is allowed: the seeder adds it when the pack lacks it.""" + problems = [] + for ct in plan.get("content_types_needed", []): + if ct not in pack: + problems.append(f"content type '{ct}' is not in the starter pack") + entries = plan.get("entries", {}) + groups = [entries.get("components", []), entries.get("pages", []), entries.get("nav", [])] + if entries.get("global_slots"): + groups.append([entries["global_slots"]]) + for group in groups: + for e in group: + ct = e.get("_content_type") + if ct not in pack: + continue + fields = {f["uid"] for f in pack[ct].get("schema", [])} | {"sap_uid", "title"} + for k in e: + if not k.startswith("_") and k not in fields: + problems.append(f"{ct}.{k} is not a field in the starter pack") + return sorted(set(problems)) + + +def main(): + global OCC_BASE + ap = argparse.ArgumentParser() + ap.add_argument("--site", required=True) + ap.add_argument("--inventory", default="", help="reuse a saved inventory.json instead of extracting") + ap.add_argument("--base", default=DEFAULT_OCC, help="OCC v2 base URL, e.g. https://HOST/occ/v2 (or set OCC_BASE_URL)") + ap.add_argument("--starter-pack", default=DEFAULT_STARTER, help="connector starter-pack content_types dir") + ap.add_argument("--env", default="development") + ap.add_argument("--approve-model", action="store_true", help="pass Gate 1") + ap.add_argument("--approve-publish", action="store_true", help="pass Gate 2 (enables writes)") + ap.add_argument("--locales", default="", help="optional i18n: sapIso=csLocale,... e.g. de=de-de,ja=ja-jp,zh=zh-cn") + ap.add_argument("--run-dir", default="") + a = ap.parse_args() + if not a.base and not a.inventory: + sys.exit("Set OCC_BASE_URL (or pass --base https://HOST/occ/v2).") + OCC_BASE = a.base + if not os.path.isdir(a.starter_pack): + sys.exit(f"Starter pack not found at {a.starter_pack}. Run from the Spartacus project root after " + "`ng add @contentstack/contentstack-spartacus-connector`, or set CONNECTOR_STARTER_PACK.") + locale_map = dict(p.split("=", 1) for p in a.locales.split(",") if "=" in p) if a.locales else {} + + run_dir = a.run_dir or os.path.join(tempfile.gettempdir(), f"sap-spartacus-import-{a.site}") + os.makedirs(run_dir, exist_ok=True) + inv_path = os.path.join(run_dir, "inventory.json") + profile = json.load(open(PROFILE)) + manifest = {"site": a.site, "started": datetime.datetime.now().isoformat(timespec="seconds"), + "stages": {}} + + def record(stage, status, note=""): + manifest["stages"][stage] = {"status": status, "note": note} + json.dump(manifest, open(os.path.join(run_dir, "manifest.json"), "w"), indent=2) + + banner(f"CONTENT-MODELING AGENT · {a.site} · profile connector-reference-fields") + print(f" run dir: {run_dir} env: {a.env} starter pack: {a.starter_pack}") + print(f" occ: {a.base} region: {os.environ.get('CS_REGION_BASE', 'https://api.contentstack.io')}") + print(f" approvals: model={'YES' if a.approve_model else 'no'} " + f"publish={'YES' if a.approve_publish else 'no'}" + f"{' (DRY RUN — no writes)' if not a.approve_publish else ''}\n") + + # 1. EXTRACT (or load fixture) + step("1/7 extract") + if a.inventory: + json.dump(json.load(open(a.inventory)), open(inv_path, "w")) + print(f" reused inventory {a.inventory} -> {inv_path}") + else: + rc, out = run("storefront-extract/extract.py", + ["--site", a.site, "--base", a.base, "--out", inv_path, "--quiet"]) + print(" " + out.strip().splitlines()[-1] if out.strip() else "") + if rc: + record("extract", "error", out[-300:]); sys.exit("extract failed") + record("extract", "ok") + + # 2. NAV RESOLVE + step("2/7 nav-resolve") + rc, out = run("storefront-extract/nav_resolve.py", ["--inventory", inv_path, "--site", a.site, "--base", a.base]) + print(" " + (out.strip().splitlines()[-1] if out.strip() else "")) + record("nav-resolve", "ok" if not rc else "warn") + + # 3. MODEL + step("3/7 model") + rc, out = run("content-model/model_connector.py", + ["--inventory", inv_path, "--profile", PROFILE, "--out", run_dir, "--starter-pack", a.starter_pack]) + print(" " + "\n ".join(out.strip().splitlines())) + if rc: + record("model", "error", out[-300:]); sys.exit("model failed") + record("model", "ok") + plan = json.load(open(os.path.join(run_dir, "connector-plan.json"))) + + # 4. GATE 1 — model approval (unknown typeCodes) + inv = json.load(open(inv_path)) + seen = set(inv.get("componentTypeInventory", {})) + unknown = sorted(seen - known_typecodes(profile)) + print() + banner("GATE 1 · MODEL APPROVAL", "gate") + if unknown: + print(f" {C['warn']}UNKNOWN typeCodes (not in profile): {unknown}{C['z']}") + print(" These need a human decision: add to component_map (authored) or occ_only_types (OCC).") + if not a.approve_model: + record("gate1", "blocked", f"unknown: {unknown}") + print(f"\n {C['err']}STOP{C['z']} — resolve the mapping, then re-run with --approve-model.") + sys.exit(2) + print(f" {C['ok']}approved (--approve-model){C['z']} — proceeding.") + else: + print(f" {C['ok']}no unknown typeCodes — all {len(seen)} classified.{C['z']}") + record("gate1", "passed", f"unknown={unknown}") + + # 5. GATE 2 — pre-publish summary + n_comp = len(plan["entries"]["components"]) + n_pages = len(plan["entries"]["pages"]) + print() + banner("GATE 2 · PRE-PUBLISH", "gate") + print(f" will UPSERT to stack + publish to '{a.env}':") + print(f" content types : {len(plan['content_types_needed'])}") + print(f" assets : {len(plan['assets'])}") + print(f" component entries: {n_comp} pages: {n_pages} global_slots: " + f"{'1' if plan['entries'].get('global_slots') else '0'}") + print(f" pages: {[p['_key']['value'] for p in plan['entries']['pages']]}") + if locale_map: + print(f" + localize into: {list(locale_map.values())} " + f"(fully-localized OCC extract per language, incl. per-locale images)") + mismatches = pack_mismatches(plan, load_pack(a.starter_pack)) + if mismatches: + print(f" {C['err']}starter pack mismatch ({len(mismatches)}){C['z']} — the plan writes types/fields the pack lacks:") + for m in mismatches[:25]: + print(f" - {m}") + print(" Update the connector (npm i @contentstack/contentstack-spartacus-connector@latest) or the " + "mapping profile before writing.") + record("gate2", "blocked", f"pack mismatch: {len(mismatches)}") + sys.exit(3) + print(f" {C['ok']}starter pack check: every planned type and field exists{C['z']}") + if not a.approve_publish: + record("gate2", "dry-run") + print(f"\n {C['warn']}DRY RUN complete{C['z']} — nothing written. " + f"Re-run with --approve-publish (and CS_* env) to seed + publish.") + sys.exit(0) + if not (os.environ.get("CS_API_KEY") and os.environ.get("CS_MANAGEMENT_TOKEN")): + record("gate2", "error", "no creds") + sys.exit(f"{C['err']}--approve-publish set but CS_API_KEY / CS_MANAGEMENT_TOKEN missing.{C['z']}") + print(f" {C['ok']}approved (--approve-publish){C['z']} — writing.") + record("gate2", "passed") + + plan_path = os.path.join(run_dir, "connector-plan.json") + # 6. SEED + step("4/7 seed (types + assets + entries)") + rc, out = run("cs-seed/cma_seed_connector.py", + ["--plan", plan_path, "--starter-pack", a.starter_pack, "--stages", "types,assets,entries"]) + tail = "\n ".join(out.strip().splitlines()[-6:]) + print(" " + tail) + record("seed", "ok" if not rc else "error", out[-300:]) + if rc: + sys.exit("seed failed") + + # 7. PUBLISH + step("5/7 publish") + rc, out = run("cs-seed/publish.py", ["--env", a.env, "--locale", "en-us"]) + print(" " + "\n ".join(out.strip().splitlines()[-4:])) + record("publish", "ok" if not rc else "error") + + # 8. VERIFY + step("6/7 verify — round-trip") + rc, out = run("verify/verify_connector.py", ["--inventory", inv_path, "--plan", plan_path]) + print(" " + "\n ".join(out.strip().splitlines()[-3:])) + rt_ok = rc == 0 + record("verify-roundtrip", "pass" if rt_ok else "check") + + step("7/7 verify — delivery read (landing page)") + venv = os.environ.copy(); venv["CS_ENVIRONMENT"] = a.env + rc, out = run("verify/cda_read.py", ["--content-type", "landing_page", "--url", "/"], env=venv) + print(" " + "\n ".join(out.strip().splitlines()[-2:])) + record("verify-cda", "pass" if rc == 0 else "check") + + # 9. LOCALIZE (optional) — full localized OCC extract per language, then localize+publish. + for sap_iso, cs_loc in locale_map.items(): + print() + banner(f"LOCALIZE · {sap_iso} → {cs_loc}", "gate") + linv = os.path.join(run_dir, f"inventory-{sap_iso}.json") + lmodel = os.path.join(run_dir, f"model-{sap_iso}") + # localized inventory (always live — i18n needs the localized OCC responses) + rc, out = run("storefront-extract/extract.py", + ["--site", a.site, "--base", a.base, "--langs", sap_iso, "--primary-only", "--out", linv, "--quiet"]) + if rc: + record(f"localize:{cs_loc}", "error", "extract failed"); continue + run("storefront-extract/nav_resolve.py", ["--inventory", linv, "--site", a.site, "--base", a.base, "--lang", sap_iso]) + rc, out = run("content-model/model_connector.py", ["--inventory", linv, "--profile", PROFILE, "--out", lmodel, "--starter-pack", a.starter_pack]) + if rc: + record(f"localize:{cs_loc}", "error", "model failed"); continue + rc, out = run("cs-seed/localize.py", + ["--locale", cs_loc, "--plan", os.path.join(lmodel, "connector-plan.json"), "--env", a.env]) + print(" " + "\n ".join(out.strip().splitlines()[-4:])) + record(f"localize:{cs_loc}", "ok" if not rc else "error", out[-200:]) + + print() + banner(f"DONE · {a.site} · round-trip {'PASS' if rt_ok else 'CHECK'}" + f"{' · +' + str(len(locale_map)) + ' locales' if locale_map else ''}", + "ok" if rt_ok else "warn") + print(f" manifest: {run_dir}/manifest.json") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/registry/connector.registry.json b/skills/dx-sap-spartacus-import/scripts/registry/connector.registry.json new file mode 100644 index 0000000..e73e536 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/registry/connector.registry.json @@ -0,0 +1,178 @@ +{ + "meta": { + "name": "sap-occ-to-connector-mapping-profile", + "version": "0.3.1", + "status": "draft", + "changelog": [ + "0.3.1 — un-deferred navigation (nav pass: nav_resolve.py + model_connector.build_nav seed nav_node tree + link leaves + category/footer_navigation_component into global_slots). Added product_page/category_page authoring from layoutSamples (shared slugs /product, /category). Refactor also stopped over-seeding components from non-authored pages.", + "0.3.0 — initial connector reference-field mapping profile." + ], + "profile": "connector-reference-fields", + "generated_from": [ + "electronics-spa", + "powertools-spa" + ], + "connector": "@contentstack/contentstack-spartacus-connector", + "content_types_source": "node_modules/@contentstack/contentstack-spartacus-connector/import-export/starter-pack/content_types/", + "philosophy": "Hybrid islands model (verified in connector code + CONTENT-MODEL.md). OCC renders the base of every page; Contentstack overrides ONLY authored slots. Functional components, unauthored slots, and non-content pages stay in OCC and are NOT seeded. Unit of control = the slot.", + "important": "This profile MAPS OCC inventory onto the connector's SHIPPED content types. It does NOT define content-type schemas — those are imported from the starter pack (via csdx or POSTing the JSON files). Idempotency: pages upsert by `url`; component entries upsert by `title` (set to the SAP component uid, which is unique).", + "differs_from_v1": "registry.json (v0.2.1) was a free-form model with a cms_content_slot entry type, a unified `banner`, and a passthrough catch-all. This profile instead conforms to the connector: named per-slot reference fields on per-template pages, one content type per SAP typecode, functional types left to OCC." + }, + "page_types": { + "_note": "OCC page template -> connector page content type. Only these templates are authored in Contentstack; every other template (Login/Account/Cart/Checkout/SearchResults/Error/StoreFinder/...) is left to OCC and produces NO Contentstack page.", + "_slug_rule": "url = '/' for the homepage label; otherwise '/' + page label. page_type <- page.typeCode; template <- page.template.", + "LandingPage2Template": { + "content_type": "landing_page", + "slug_field": "url" + }, + "ContentPage1Template": { + "content_type": "landing_page", + "slug_field": "url" + }, + "ProductDetailsPageTemplate": { + "content_type": "product_page", + "slug_field": "url" + }, + "ProductListPageTemplate": { + "content_type": "category_page", + "slug_field": "url" + }, + "_note_content_pages": "Content pages (ContentPage1Template) land on landing_page: the connector resolves every per-route page against cmsPageContentType (landing_page), and the starter pack carries body_content/side_content there (connector PR #35, DX-31800)." + }, + "slot_field_map": { + "_note": "SAP slot POSITION (from inventory slot.position) -> connector page reference-field uid. Reverse of the connector's SLOT_FIELD_TO_SAP_NAME. Positions not listed here are left to OCC. Shell positions (see global_slot_positions) are authored on the single global_slots entry, not on the page.", + "Section1": "section1", + "Section2": "section2", + "Section2A": "section2_a", + "Section2B": "section2_b", + "Section2C": "section2_c", + "Section3": "section3", + "Section4": "section4", + "Section5": "section5", + "BodyContent": "body_content", + "SideContent": "side_content", + "BottomHeaderSlot": "bottom_header_slot", + "ProductLeftRefinements": "product_left_refinements", + "ProductListSlot": "product_list_slot", + "ProductGridSlot": "product_grid_slot", + "SearchResultsListSlot": "search_results_list_slot", + "SearchResultsGridSlot": "search_results_grid_slot", + "Summary": "summary", + "UpSelling": "up_selling", + "CrossSelling": "cross_selling", + "Tabs": "tabs", + "PlaceholderContentSlot": "placeholder_content_slot", + "TopContent": "top_content", + "MiddleContent": "middle_content", + "BottomContent": "bottom_content", + "CenterRightContentSlot": "center_right_content_slot", + "CenterRightContent": "center_right_content", + "EmptyCartMiddleContent": "empty_cart_middle_content", + "LeftContentSlot": "left_content_slot", + "RightContentSlot": "right_content_slot", + "HomepageNavLink": "homepage_nav_link" + }, + "global_slots": { + "_note": "Author ONLY content-managed shell slots. SiteContext (theme+language switcher) and SiteLinks (login/register + utility links) are FUNCTIONAL — authoring them partially overrode OCC's full shell and dropped the language dropdown + sign-in/register. Leave those to OCC (occFallback). Author logo (branding), category nav, and footer nav only.", + "content_type": "global_slots", + "upsert_by": "title", + "positions": { + "SiteLogo": "site_logo", + "NavigationBar": "navigation_bar", + "Footer": "footer" + }, + "positions_left_to_occ": { + "SiteContext": "site_context (theme+language -> OCC)", + "SiteLinks": "site_links (login/register+links -> OCC)", + "SearchBox": "search_box (functional -> OCC)", + "MiniCart": "mini_cart (functional -> OCC)", + "HeaderLinks": "header_links (functional -> OCC)" + } + }, + "component_map": { + "_note": "OCC typeCode -> connector component content type + field mapping (connector field uid <- OCC source). Only authored/content-bearing types appear here. title <- component.name (entry title is set to component.uid for idempotency; name is kept where a display field exists).", + "SimpleResponsiveBannerComponent": { + "content_type": "simple_responsive_banner_component", + "assets": true, + "fields": { + "url_link": "urlLink", + "media_mobile": "media.mobile", + "media_tablet": "media.tablet", + "media_desktop": "media.desktop", + "media_widescreen": "media.widescreen", + "media": "media.desktop" + } + }, + "SimpleBannerComponent": { + "content_type": "simple_banner_component", + "assets": true, + "fields": { + "url_link": "urlLink", + "media": "media" + } + }, + "ProductCarouselComponent": { + "content_type": "product_carousel_component", + "fields": { + "title": "title", + "products": "productCodes[space-split]" + } + }, + "CMSParagraphComponent": { + "content_type": "cms_paragraph_component", + "fields": { + "content": "content" + } + }, + "CMSLinkComponent": { + "content_type": "cms_link_component", + "fields": { + "link_name": "linkName", + "url": "url", + "target": "target" + } + }, + "CMSFlexComponent": { + "content_type": "cms_flex_component", + "fields": { + "flex_type": "flexType" + } + }, + "CategoryNavigationComponent": { + "content_type": "category_navigation_component", + "status": "nav-pass", + "_note": "NOT a regular slot component. Handled by the nav pass: nav_resolve.py resolves link leaves from OCC, model_connector.build_nav emits the nav_node tree + cms_link_component leaves + this component, placed into global_slots.navigation_bar. map_component skips it (status != mapped)." + }, + "FooterNavigationComponent": { + "content_type": "footer_navigation_component", + "status": "nav-pass", + "_note": "see CategoryNavigationComponent; placed into global_slots.footer." + } + }, + "navigation": { + "_note": "Navigation is seeded flat: nav_node_flat entries (node_id/parent_id/sort_order) plus a category_navigation_flat or footer_navigation_flat component that references the pool via all_nodes; the connector rebuilds the tree. See model_connector.build_nav.", + "nav_node": "nav_node_flat", + "category": "category_navigation_flat", + "footer": "footer_navigation_flat", + "link_leaf": "cms_link_component" + }, + "occ_only_types": { + "_note": "Functional / commerce-driven components with no editorial content. Per the hybrid model these hydrate from OCC and are NOT seeded into Contentstack. Slots containing only these are left empty in CS (OCC fills them).", + "types": [ + "MiniCartComponent", + "SearchBoxComponent", + "BreadcrumbComponent", + "CMSSiteContextComponent", + "ProductRefinementComponent", + "SearchResultsListComponent", + "CartSuggestionComponent", + "JspIncludeComponent", + "MerchandisingCarouselComponent", + "BundleCarouselComponent", + "CMSProductListComponent", + "CMSTabParagraphContainer", + "ProductAddToCartComponent", + "ProductReferencesComponent" + ] + } +} diff --git a/skills/dx-sap-spartacus-import/scripts/storefront-extract/extract.py b/skills/dx-sap-spartacus-import/scripts/storefront-extract/extract.py new file mode 100644 index 0000000..d28a51f --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/storefront-extract/extract.py @@ -0,0 +1,345 @@ +#!/usr/bin/env python3 +""" +storefront-extract — Phase 2 of the SAP Spartacus -> Contentstack content-modeling agent. + +Crawls one SAP Commerce baseSite over the OCC API and emits a normalized "content +inventory" JSON that conforms to inventory.schema.json. That inventory is the single +contract every downstream skill (content-model, cs-seed, verify) consumes. + +Design facts this encodes (all validated live against the SAP backend): + * OCC /cms/pages is PAGE-BY-LABEL, not a bulk list -> content pages are found by a + guided BFS from known labels + internal urlLinks. + * fields=FULL inlines each component's fields, media (responsive variants) AND the + full navigationNode tree. + * Product/Category page LAYOUTS are fetched with pageType + code= (not pageLabelOrId). + * Media bytes download from host + media.url (the ?context= token is required). + * Content is per content-catalog + version; we read the published Online version. + +Deterministic: no wall-clock timestamps in the output, stable key order -> fixture-diffable. + +Usage: + python3 extract.py --site electronics-spa [--base URL] [--langs en,de] \ + [--out inventory.json] [--download-media DIR] [--quiet] +""" +import argparse, json, os, re, ssl, sys, urllib.parse, urllib.request +from collections import OrderedDict, deque + +DEFAULT_BASE = os.environ.get("OCC_BASE_URL") + + +def occ_ssl_context(): + """TLS context for OCC calls. Verified by default; set OCC_INSECURE_TLS=1 only for a + dev backend with a self-signed certificate you trust.""" + ctx = ssl.create_default_context() + if os.environ.get("OCC_INSECURE_TLS") == "1": + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + return ctx + +# Standard Spartacus content-page labels used to seed BFS discovery. +SEED_LABELS = [ + "homepage", "login", "register", "cart", "checkout", "orderConfirmation", + "search", "faq", "contact", "terms", "sale", "store-finder", "storefinder", + "wishlist", "notFound", "myaccount", "address-book", "update-profile", + "order-history", "my-coupons", "consents", "close-account", "payment-details", +] +# Component field names that carry localizable text (captured per language for i18n). +I18N_FIELDS = ("content", "title", "linkName", "name", "urlLink", "notice", "altText") +LABEL_RE = re.compile(r"^/([A-Za-z0-9\-_]+)/?$") + + +class Occ: + """Thin OCC client (TLS per occ_ssl_context).""" + def __init__(self, base, site): + self.base, self.site = base.rstrip("/"), site + self.host = base.split("/occ/")[0] + self.ctx = occ_ssl_context() + self.calls = 0 + + def get(self, path, **params): + self.calls += 1 + url = f"{self.base}/{self.site}/{path}?" + urllib.parse.urlencode(params) + req = urllib.request.Request(url, headers={"Accept": "application/json"}) + try: + with urllib.request.urlopen(req, context=self.ctx, timeout=40) as r: + return json.loads(r.read().decode()) + except Exception as e: + return {"__error__": str(e)} + + def download(self, media_url): + self.calls += 1 + url = media_url if media_url.startswith("http") else self.host + media_url + req = urllib.request.Request(url, headers={"Accept": "*/*"}) + with urllib.request.urlopen(req, context=self.ctx, timeout=60) as r: + return r.read() + + def page(self, page_type, key_field, key, lang): + return self.get("cms/pages", pageType=page_type, fields="FULL", lang=lang, + **{key_field: key}) + + +def sorted_json(o): + """Recursively order dict keys for deterministic, diff-stable output.""" + if isinstance(o, dict): + return OrderedDict((k, sorted_json(o[k])) for k in sorted(o)) + if isinstance(o, list): + return [sorted_json(v) for v in o] + return o + + +class Extractor: + def __init__(self, occ, langs, quiet=False): + self.occ = occ + self.langs = langs + self.primary = langs[0] + self.quiet = quiet + self.components_by_type = {} # typeCode -> [normalized component] + self.media = OrderedDict() # media code -> {url, mime, altText} + self.slots = OrderedDict() # slotId -> meta + self.nav_nodes = OrderedDict() # nodeUid -> raw navigationNode tree + self.warnings = [] + + def log(self, *a): + if not self.quiet: + print(*a, file=sys.stderr) + + # ---- media & link scanning ------------------------------------------------- + def record_media(self, m): + if not isinstance(m, dict): + return + if "code" in m and "url" in m: + self.media.setdefault(m["code"], { + "url": m.get("url"), "mime": m.get("mime"), "altText": m.get("altText")}) + else: + for v in m.values(): + if isinstance(v, dict): + self.record_media(v) + + def scan_labels(self, obj, out): + if isinstance(obj, dict): + for k, v in obj.items(): + if k in ("url", "urlLink", "linkTo") and isinstance(v, str): + mm = LABEL_RE.match(v.strip()) + if mm: + out.add(mm.group(1)) + self.scan_labels(v, out) + elif isinstance(obj, list): + for v in obj: + self.scan_labels(v, out) + + # ---- component normalization ---------------------------------------------- + DROP = ("uuid", "modifiedtime", "synchronizationBlocked") + + def normalize_component(self, c): + tc = c.get("typeCode", "") + if "media" in c: + self.record_media(c["media"]) + if "navigationNode" in c and isinstance(c["navigationNode"], dict): + nn = c["navigationNode"] + if nn.get("uid"): + self.nav_nodes.setdefault(nn["uid"], self.clean(nn)) + keep = self.clean(c) + self.components_by_type.setdefault(tc, []).append(keep) + return keep + + def clean(self, o): + if isinstance(o, dict): + return {k: self.clean(v) for k, v in o.items() if k not in self.DROP} + if isinstance(o, list): + return [self.clean(v) for v in o] + return o + + def normalize_page(self, label, raw): + slots = [] + for s in (raw.get("contentSlots") or {}).get("contentSlot", []): + sid = s.get("slotId") + self.slots.setdefault(sid, { + "position": s.get("position"), "shared": s.get("slotShared"), + "name": s.get("name"), "firstSeenOn": label}) + comps = (s.get("components") or {}).get("component", []) + slots.append({ + "slotId": sid, "position": s.get("position"), + "shared": s.get("slotShared"), + "components": [self.normalize_component(c) for c in comps]}) + return { + "label": raw.get("label") or label, "uid": raw.get("uid"), + "title": raw.get("title"), "template": raw.get("template"), + "typeCode": raw.get("typeCode"), "robotTag": raw.get("robotTag"), + "slots": slots} + + # ---- discovery ------------------------------------------------------------- + def discover_content_pages(self): + pages, visited = OrderedDict(), set() + frontier = deque(dict.fromkeys(SEED_LABELS)) + while frontier: + label = frontier.popleft() + if label in visited: + continue + visited.add(label) + raw = self.occ.page("ContentPage", "pageLabelOrId", label, self.primary) + if raw.get("__error__") or not raw.get("uid"): + continue + pages[label] = self.normalize_page(label, raw) + found = set() + self.scan_labels(raw, found) + for f in found: + if f not in visited: + frontier.append(f) + self.log(f" content pages: {len(pages)}") + return pages + + def sample_layout_pages(self): + out = OrderedDict() + # product code from search + s = self.occ.get("products/search", query="", pageSize="1", + fields="DEFAULT", lang=self.primary, curr="USD") + if not s.get("products"): + s = self.occ.get("products/search", query="a", pageSize="1", + fields="DEFAULT", lang=self.primary, curr="USD") + code = (s.get("products") or [{}])[0].get("code") + if code: + pp = self.occ.page("ProductPage", "code", code, self.primary) + if pp.get("uid"): + out["ProductPage"] = self.normalize_page(f"product:{code}", pp) + else: + self.warnings.append("no product code resolved for ProductPage layout") + # category code: try a handful + for cat in ("575", "576", "1", "brands", "1358"): + cp = self.occ.page("CategoryPage", "code", cat, self.primary) + if cp.get("uid"): + out[f"CategoryPage:{cat}"] = self.normalize_page(f"category:{cat}", cp) + break + else: + self.warnings.append("no category code resolved for CategoryPage layout") + self.log(f" layout samples: {list(out)}") + return out + + # ---- i18n ------------------------------------------------------------------ + def capture_i18n(self, page_labels): + i18n = OrderedDict() + for lang in self.langs[1:]: + per = OrderedDict() + for label in page_labels: + raw = self.occ.page("ContentPage", "pageLabelOrId", label, lang) + if raw.get("__error__"): + continue + for s in (raw.get("contentSlots") or {}).get("contentSlot", []): + for c in (s.get("components") or {}).get("component", []): + uid = c.get("uid") + if not uid: + continue + vals = {f: c[f] for f in I18N_FIELDS + if isinstance(c.get(f), str) and c[f]} + if vals: + per.setdefault(uid, {}).update(vals) + i18n[lang] = per + self.log(f" i18n[{lang}]: {len(per)} components") + return i18n + + # ---- media download (optional) -------------------------------------------- + def download_media(self, out_dir): + os.makedirs(out_dir, exist_ok=True) + ok = 0 + for code, m in self.media.items(): + if not m.get("url"): + continue + try: + data = self.occ.download(m["url"]) + fname = re.sub(r"[^A-Za-z0-9._-]", "_", code) + with open(os.path.join(out_dir, fname), "wb") as f: + f.write(data) + m["bytes"] = len(data) + ok += 1 + except Exception as e: + self.warnings.append(f"media download failed {code}: {e}") + self.log(f" media downloaded: {ok}/{len(self.media)}") + return ok + + # ---- assemble -------------------------------------------------------------- + def run(self, download_dir=None): + self.log(f"[extract] {self.occ.site} langs={self.langs}") + pages = self.discover_content_pages() + layouts = self.sample_layout_pages() + i18n = self.capture_i18n(list(pages)) + if download_dir: + self.download_media(download_dir) + + type_inventory = { + tc: {"instances": len(v), + "fields": sorted({k for c in v for k in c.keys()})} + for tc, v in sorted(self.components_by_type.items(), + key=lambda kv: -len(kv[1]))} + inv = { + "schemaVersion": "1.0", + "site": self.occ.site, + "source": self.occ.base, + "catalogVersion": "Online", + "primaryLanguage": self.primary, + "languages": self.langs, + "counts": { + "contentPages": len(pages), + "layoutSamples": len(layouts), + "uniqueSlots": len(self.slots), + "sharedSlots": sum(1 for s in self.slots.values() + if s["shared"] in (True, "true")), + "componentTypes": len(self.components_by_type), + "componentInstances": sum(len(v) for v in self.components_by_type.values()), + "navigationNodes": len(self.nav_nodes), + "mediaAssets": len(self.media), + }, + "componentTypeInventory": type_inventory, + "slots": self.slots, + "contentPages": pages, + "layoutSamples": layouts, + "navigation": self.nav_nodes, + "i18n": i18n, + "mediaManifest": self.media, + "warnings": self.warnings, + } + return inv + + +def resolve_languages(occ, arg): + if arg: + return [x.strip() for x in arg.split(",") if x.strip()] + d = occ.get("languages") + langs = [l["isocode"] for l in d.get("languages", []) if l.get("isocode")] + # put 'en' first if present for a stable primary + if "en" in langs: + langs = ["en"] + [l for l in langs if l != "en"] + return langs or ["en"] + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--site", required=True) + ap.add_argument("--base", default=DEFAULT_BASE, required=DEFAULT_BASE is None, + help="OCC v2 base URL, e.g. https://HOST/occ/v2 (or set OCC_BASE_URL)") + ap.add_argument("--langs", default="", help="comma list; default = site languages") + ap.add_argument("--primary-only", action="store_true", + help="crawl only the primary language (skip i18n)") + ap.add_argument("--out", default="inventory.json") + ap.add_argument("--download-media", default="", help="dir to save media bytes") + ap.add_argument("--quiet", action="store_true") + a = ap.parse_args() + + occ = Occ(a.base, a.site) + langs = resolve_languages(occ, a.langs) + if a.primary_only: + langs = langs[:1] + inv = Extractor(occ, langs, quiet=a.quiet).run(download_dir=a.download_media or None) + + with open(a.out, "w") as f: + json.dump(sorted_json(inv), f, indent=2) + + c = inv["counts"] + print(f"[extract] {a.site}: pages={c['contentPages']} layouts={c['layoutSamples']} " + f"types={c['componentTypes']} instances={c['componentInstances']} " + f"nav={c['navigationNodes']} media={c['mediaAssets']} " + f"| {occ.calls} OCC calls -> {a.out}") + if inv["warnings"]: + print(" warnings: " + "; ".join(inv["warnings"])) + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/storefront-extract/inventory.schema.json b/skills/dx-sap-spartacus-import/scripts/storefront-extract/inventory.schema.json new file mode 100644 index 0000000..8382d40 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/storefront-extract/inventory.schema.json @@ -0,0 +1,149 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "https://contentstack/sap-spartacus/inventory.schema.json", + "title": "Storefront Content Inventory", + "description": "Normalized output of storefront-extract. The single contract consumed by content-model, cs-seed, and verify. One file per baseSite.", + "type": "object", + "required": [ + "schemaVersion", "site", "primaryLanguage", "languages", "counts", + "componentTypeInventory", "slots", "contentPages", "navigation", + "i18n", "mediaManifest" + ], + "additionalProperties": true, + "properties": { + "schemaVersion": { "type": "string", "const": "1.0" }, + "site": { "type": "string", "description": "baseSite uid, e.g. electronics-spa" }, + "source": { "type": "string", "description": "OCC base url the inventory was read from" }, + "catalogVersion": { "type": "string", "description": "Online (published) unless overridden" }, + "primaryLanguage": { "type": "string" }, + "languages": { "type": "array", "items": { "type": "string" }, "minItems": 1 }, + + "counts": { + "type": "object", + "required": ["contentPages", "componentTypes", "componentInstances"], + "properties": { + "contentPages": { "type": "integer" }, + "layoutSamples": { "type": "integer" }, + "uniqueSlots": { "type": "integer" }, + "sharedSlots": { "type": "integer" }, + "componentTypes": { "type": "integer" }, + "componentInstances": { "type": "integer" }, + "navigationNodes": { "type": "integer" }, + "mediaAssets": { "type": "integer" } + } + }, + + "componentTypeInventory": { + "type": "object", + "description": "typeCode -> {instances, fields[]}. Used to diff against the registry to find unknown types (Gate 1).", + "additionalProperties": { + "type": "object", + "required": ["instances", "fields"], + "properties": { + "instances": { "type": "integer" }, + "fields": { "type": "array", "items": { "type": "string" } } + } + } + }, + + "slots": { + "type": "object", + "description": "slotId -> slot metadata. shared=true marks a global slot reused across pages.", + "additionalProperties": { + "type": "object", + "properties": { + "position": { "type": ["string", "null"] }, + "shared": { "type": ["boolean", "string", "null"] }, + "name": { "type": ["string", "null"] }, + "firstSeenOn": { "type": "string" } + } + } + }, + + "contentPages": { + "type": "object", + "description": "page label -> normalized page.", + "additionalProperties": { "$ref": "#/definitions/page" } + }, + + "layoutSamples": { + "type": "object", + "description": "One representative Product/Category page each (layout structure only; records stay in commerce).", + "additionalProperties": { "$ref": "#/definitions/page" } + }, + + "navigation": { + "type": "object", + "description": "navigationNode uid -> raw resolved node tree (children + entry references). Captured now; activated in registry v0.2.", + "additionalProperties": { "type": "object" } + }, + + "i18n": { + "type": "object", + "description": "language -> { componentUid -> { field -> localized string } } for non-primary languages.", + "additionalProperties": { + "type": "object", + "additionalProperties": { + "type": "object", + "additionalProperties": { "type": "string" } + } + } + }, + + "mediaManifest": { + "type": "object", + "description": "media code -> {url, mime, altText, bytes?}. url needs the host prefix and carries the ?context= download token.", + "additionalProperties": { + "type": "object", + "required": ["url"], + "properties": { + "url": { "type": ["string", "null"] }, + "mime": { "type": ["string", "null"] }, + "altText": { "type": ["string", "null"] }, + "bytes": { "type": "integer" } + } + } + }, + + "warnings": { "type": "array", "items": { "type": "string" } } + }, + + "definitions": { + "page": { + "type": "object", + "required": ["label", "template", "slots"], + "properties": { + "label": { "type": ["string", "null"] }, + "uid": { "type": ["string", "null"] }, + "title": { "type": ["string", "null"] }, + "template": { "type": ["string", "null"] }, + "typeCode": { "type": ["string", "null"] }, + "robotTag": { "type": ["string", "null"] }, + "slots": { + "type": "array", + "items": { + "type": "object", + "required": ["slotId", "components"], + "properties": { + "slotId": { "type": ["string", "null"] }, + "position": { "type": ["string", "null"] }, + "shared": { "type": ["boolean", "string", "null"] }, + "components": { + "type": "array", + "items": { + "type": "object", + "required": ["typeCode"], + "properties": { + "uid": { "type": ["string", "null"] }, + "typeCode": { "type": "string" } + }, + "additionalProperties": true + } + } + } + } + } + } + } + } +} diff --git a/skills/dx-sap-spartacus-import/scripts/storefront-extract/nav_resolve.py b/skills/dx-sap-spartacus-import/scripts/storefront-extract/nav_resolve.py new file mode 100644 index 0000000..593062d --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/storefront-extract/nav_resolve.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 +""" +nav_resolve — the deferred navigation resolve pass (enriches an inventory in place). + +storefront-extract captures the navigationNode trees, but each link leaf is only a +reference ({itemId, itemType: CMSLinkComponent}). This pass batch-resolves those leaves +from OCC (/cms/components) into full link detail, so the connector model can seed a +nav_node tree with real cms_link_component leaves. + +Adds two keys to the inventory: + navigationLinks : { itemId -> {link_name, url, target, external, name} } + navigationRoots : { componentUid -> rootNodeUid } (from CategoryNavigation/FooterNavigation) + +Usage: python3 nav_resolve.py --inventory --site electronics-spa [--base URL] +""" +import argparse, json, os, ssl, sys, urllib.parse, urllib.request + +DEFAULT_BASE = os.environ.get("OCC_BASE_URL") + + +def occ_ssl_context(): + """TLS context for OCC calls. Verified by default; set OCC_INSECURE_TLS=1 only for a + dev backend with a self-signed certificate you trust.""" + ctx = ssl.create_default_context() + if os.environ.get("OCC_INSECURE_TLS") == "1": + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + return ctx + + +CTX = occ_ssl_context() + + +def _get(url): + req = urllib.request.Request(url, headers={"Accept": "application/json"}) + with urllib.request.urlopen(req, context=CTX, timeout=40) as r: + return json.loads(r.read().decode()) + + +def occ_components(base, site, ids, lang="en"): + out = {} + for i in range(0, len(ids), 10): + chunk = ids[i:i + 10] + url = f"{base}/{site}/cms/components?" + urllib.parse.urlencode( + {"componentIds": ",".join(chunk), "lang": lang}) + try: + d = _get(url) + for c in (d.get("component") or d.get("components") or []): + out[c.get("uid")] = c + except Exception as e: + print(f" batch chunk failed: {e}", file=sys.stderr) + # single-GET fallback for anything the batch didn't return + missing = [i for i in ids if i not in out] + for iid in missing: + try: + c = _get(f"{base}/{site}/cms/components/{iid}?lang={lang}") + if c.get("uid"): + out[c["uid"]] = c + except Exception: + pass + return out + + +def collect_link_ids(node, ids): + for e in (node.get("entries") or []): + if isinstance(e, dict) and e.get("itemType") == "CMSLinkComponent" and e.get("itemId"): + ids.add(e["itemId"]) + for ch in (node.get("children") or []): + collect_link_ids(ch, ids) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--inventory", required=True) + ap.add_argument("--site", required=True) + ap.add_argument("--base", default=DEFAULT_BASE, required=DEFAULT_BASE is None, + help="OCC v2 base URL (or set OCC_BASE_URL)") + ap.add_argument("--lang", default="en", help="OCC language for localized nav labels") + a = ap.parse_args() + inv = json.load(open(a.inventory)) + + nav = inv.get("navigation", {}) + ids = set() + for node in nav.values(): + collect_link_ids(node, ids) + print(f"[nav_resolve] {a.site} (lang={a.lang}): {len(nav)} root nodes, {len(ids)} link leaves to resolve") + + resolved = occ_components(a.base, a.site, sorted(ids), lang=a.lang) + links = {} + for iid, c in resolved.items(): + links[iid] = {"link_name": c.get("linkName"), "url": c.get("url"), + "target": c.get("target"), "external": c.get("external"), + "name": c.get("name")} + inv["navigationLinks"] = links + + # component uid -> root node uid, from the nav components in the inventory + roots = {} + for p in inv.get("contentPages", {}).values(): + for s in p.get("slots", []): + for comp in s.get("components", []): + if comp.get("typeCode") in ("CategoryNavigationComponent", "FooterNavigationComponent"): + nn = comp.get("navigationNode") or {} + if nn.get("uid"): + roots[comp["uid"]] = {"root": nn["uid"], "typeCode": comp["typeCode"], + "position": s.get("position"), + "wrapAfter": comp.get("wrapAfter")} + inv["navigationRoots"] = roots + + json.dump(inv, open(a.inventory, "w"), indent=2) + print(f" resolved {len(links)}/{len(ids)} links; {len(roots)} nav components " + f"-> {sorted((r['typeCode'], r['position']) for r in roots.values())}") + print(f" enriched -> {a.inventory}") + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/verify/cda_read.py b/skills/dx-sap-spartacus-import/scripts/verify/cda_read.py new file mode 100644 index 0000000..29520d8 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/verify/cda_read.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +""" +cda_read — simulate the connector's delivery-time fetch (Phase 4b render pre-check). + +Uses the DELIVERY token against the Content Delivery API to fetch a page by its `url` slug +with slot references resolved (`include[]=`) — exactly what the connector's +ContentstackClientService.getPageBySlug does. Confirms the published content is readable by +a storefront and that slot references hydrate to real component entries. + +Read-only. Env: CS_API_KEY, CS_DELIVERY_TOKEN, CS_CDA_BASE (default https://cdn.contentstack.io), +CS_ENVIRONMENT (default development). + +Usage: python3 cda_read.py --content-type landing_page --url / [--preview] +""" +import argparse, json, os, sys, urllib.parse, urllib.request + +CDA = os.environ.get("CS_CDA_BASE", "https://cdn.contentstack.io").rstrip("/") +KEY = os.environ.get("CS_API_KEY") +DELIVERY = os.environ.get("CS_DELIVERY_TOKEN") +ENV = os.environ.get("CS_ENVIRONMENT", "development") + +SLOT_FIELDS = ["section1", "section2", "section2_a", "section2_b", "section2_c", + "section3", "section4", "section5", "body_content", "side_content", + "summary", "up_selling", "cross_selling", "tabs", "placeholder_content_slot", + "product_left_refinements", "product_list_slot", "product_grid_slot", + "search_results_list_slot", "search_results_grid_slot", + "top_content", "bottom_content", "header", "footer"] + + +def cda(path): + req = urllib.request.Request(f"{CDA}/v3{path}") + req.add_header("api_key", KEY) + req.add_header("access_token", DELIVERY) + with urllib.request.urlopen(req, timeout=40) as r: + return json.loads(r.read().decode()) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--content-type", default="landing_page") + ap.add_argument("--url", default="/") + a = ap.parse_args() + if not (KEY and DELIVERY): + sys.exit("set CS_API_KEY and CS_DELIVERY_TOKEN") + + params = [("environment", ENV), ("query", json.dumps({"url": a.url}))] + for f in SLOT_FIELDS: + params.append(("include[]", f)) + qs = urllib.parse.urlencode(params) + r = cda(f"/content_types/{a.content_type}/entries?{qs}") + entries = r.get("entries") or [] + print(f"[cda_read] {a.content_type} url={a.url} @ {CDA} env={ENV}") + if not entries: + print(" no entry found (published? correct env/token?)"); sys.exit(1) + e = entries[0] + print(f" title: {e.get('title')} template: {e.get('template')} page_type: {e.get('page_type')}") + total_refs = 0 + for f in SLOT_FIELDS: + v = e.get(f) + if isinstance(v, list) and v: + resolved = [x for x in v if isinstance(x, dict) and x.get("uid")] + total_refs += len(resolved) + types = sorted({x.get("_content_type_uid") or "?" for x in resolved}) + sample = resolved[0].get("title") if resolved else "" + print(f" {f:14s}: {len(resolved)} component(s) {types} e.g. \"{sample}\"") + print(f" RESULT: {'PASS — page reads with ' + str(total_refs) + ' resolved components' if total_refs else 'entry found but no references resolved (publish components too?)'}") + sys.exit(0 if total_refs else 1) + + +if __name__ == "__main__": + main() diff --git a/skills/dx-sap-spartacus-import/scripts/verify/verify_connector.py b/skills/dx-sap-spartacus-import/scripts/verify/verify_connector.py new file mode 100644 index 0000000..38b0d96 --- /dev/null +++ b/skills/dx-sap-spartacus-import/scripts/verify/verify_connector.py @@ -0,0 +1,282 @@ +#!/usr/bin/env python3 +""" +verify_connector — round-trip verification for the CONNECTOR profile (Phase 4b). + +Reads the seeded connector-model entries back from the stack, reverse-maps each authored +component to its original OCC shape, and diffs against the original inventory. Only the +AUTHORED components (those the connector profile seeds) are checked — functional components +are intentionally left to OCC and not in Contentstack. + +Read-only. Creds from env: CS_API_KEY, CS_MANAGEMENT_TOKEN, CS_REGION_BASE. + +Usage: python3 verify_connector.py --inventory --plan [--verbose] +""" +import argparse, json, os, re, sys, urllib.parse, urllib.request +from collections import defaultdict + +API = os.environ.get("CS_REGION_BASE", "https://api.contentstack.io").rstrip("/") +KEY, TOKEN = os.environ.get("CS_API_KEY"), os.environ.get("CS_MANAGEMENT_TOKEN") + +# connector content type -> OCC typeCode (subset we author) +CT_TO_TYPECODE = { + "simple_responsive_banner_component": "SimpleResponsiveBannerComponent", + "simple_banner_component": "SimpleBannerComponent", + "product_carousel_component": "ProductCarouselComponent", + "cms_paragraph_component": "CMSParagraphComponent", + "cms_link_component": "CMSLinkComponent", + "cms_flex_component": "CMSFlexComponent", +} + + +def cma(path): + req = urllib.request.Request(f"{API}/v3{path}") + req.add_header("api_key", KEY); req.add_header("authorization", TOKEN) + with urllib.request.urlopen(req, timeout=40) as r: + return json.loads(r.read().decode()) + + +def all_entries(ct): + out, skip = [], 0 + while True: + r = cma(f"/content_types/{ct}/entries?limit=100&skip={skip}&include_count=true") + es = r.get("entries") or [] + out += es; skip += len(es) + if len(es) < 100 or skip >= (r.get("count") or 0): + break + return out + + +def asset_titles(): + out, skip = {}, 0 + while True: + r = cma(f"/assets?limit=100&skip={skip}&include_count=true") + a = r.get("assets") or [] + for x in a: + out[x["uid"]] = x.get("title") + skip += len(a) + if len(a) < 100 or skip >= (r.get("count") or 0): + break + return out + + +def norm_html(s): + return re.sub(r"\s+", " ", (s or "").strip()) + + +def as_target(v): + return "true" if str(v).strip().lower() == "true" else "false" + + +def _uid(v): + if isinstance(v, list) and v: + return v[0] if isinstance(v[0], str) else v[0].get("uid") + return v if isinstance(v, str) else (v.get("uid") if isinstance(v, dict) else None) + + +def orig_proj(c): + tc = c["typeCode"] + m = c.get("media") or {} + if tc == "SimpleResponsiveBannerComponent": + codes = sorted(m[b]["code"] for b in ("mobile", "tablet", "desktop", "widescreen") + if isinstance(m.get(b), dict) and m[b].get("code")) + return {"media": codes, "url_link": c.get("urlLink") or ""} + if tc == "SimpleBannerComponent": + return {"media": [m["code"]] if m.get("code") else [], "url_link": c.get("urlLink") or ""} + if tc == "ProductCarouselComponent": + return {"title": c.get("title") or "", "products": sorted((c.get("productCodes") or "").split())} + if tc == "CMSParagraphComponent": + return {"content": norm_html(c.get("content"))} + if tc == "CMSLinkComponent": + return {"link_name": c.get("linkName") or "", "url": c.get("url") or "", + "target": as_target(c.get("target"))} + if tc == "CMSFlexComponent": + return {"flex_type": c.get("flexType") or ""} + return {} + + +def cs_proj(e, ct, assets): + if ct == "simple_responsive_banner_component": + codes = sorted(assets.get(_uid(e.get(f))) for f in + ("media_mobile", "media_tablet", "media_desktop", "media_widescreen") if e.get(f)) + return {"media": [c for c in codes if c], "url_link": e.get("url_link") or ""} + if ct == "simple_banner_component": + c = assets.get(_uid(e.get("media"))) + return {"media": [c] if c else [], "url_link": e.get("url_link") or ""} + if ct == "product_carousel_component": + return {"title": e.get("title") or "", "products": sorted(e.get("products") or [])} + if ct == "cms_paragraph_component": + return {"content": norm_html(e.get("content"))} + if ct == "cms_link_component": + return {"link_name": e.get("link_name") or "", "url": e.get("url") or "", + "target": as_target(e.get("target"))} + if ct == "cms_flex_component": + return {"flex_type": e.get("flex_type") or ""} + return {} + + +def flatten_source_tree(root): + """Flatten a source OCC navigationNode tree into + {node_uid: (parent_uid, order, link_itemId)}. The root node is a pure + container, so its children are the top level (parent '').""" + flat = {} + def walk(node, parent, order): + uid = node.get("uid") + if not uid: + return + entries = [e.get("itemId") for e in (node.get("entries") or []) + if e.get("itemType") == "CMSLinkComponent" and e.get("itemId")] + flat[uid] = (parent, order, entries[0] if entries else None) + for i, ch in enumerate(node.get("children") or [], start=1): + walk(ch, uid, i) + for i, ch in enumerate(root.get("children") or [], start=1): + walk(ch, "", i) + return flat + + +def verify_nav(inv, nav_orig, verbose): + """Read the seeded FLAT nav back and diff each menu against its source tree. + + For every CategoryNavigation/FooterNavigation component: fetch the flat + component, walk its all_nodes pool into nav_node_flat entries, reassemble the + tree from node_id/parent_id, and diff each node's (parent, sort_order, link) + against the original OCC navigationNode tree. Proves the flat model faithfully + represents the source hierarchy — the same reassembly the connector does at + runtime. Returns (checked, ok, issues).""" + FLAT_CT = ("category_navigation_flat", "footer_navigation_flat") + comps = {} + for ct in FLAT_CT: + for e in all_entries(ct): + if e.get("sap_uid"): + comps[e["sap_uid"]] = e + nodes_by_csuid = {e["uid"]: e for e in all_entries("nav_node_flat")} + link_sap_by_csuid = {e["uid"]: e.get("sap_uid") for e in all_entries("cms_link_component")} + + def node_link_sap(n): + u = _uid(n.get("links")) if n.get("links") else None + return link_sap_by_csuid.get(u) if u else None + + checked = ok = 0 + issues = [] + for cuid, c in nav_orig.items(): + checked += 1 + root = c.get("navigationNode") or {} + if not root.get("children"): + root = (inv.get("navigation") or {}).get(root.get("uid"), root) + src = flatten_source_tree(root) + cs = comps.get(cuid) + if not cs: + issues.append((cuid, ["flat component missing in stack"])); continue + got = {} + for r in cs.get("all_nodes") or []: + n = nodes_by_csuid.get(r.get("uid") if isinstance(r, dict) else r) + if n and n.get("node_id"): + got[n["node_id"]] = (n.get("parent_id") or "", n.get("sort_order"), node_link_sap(n)) + diffs = [] + for nid, (p, o, lk) in src.items(): + if nid not in got: + diffs.append(f"missing node {nid}"); continue + gp, go, glk = got[nid] + if gp != p: + diffs.append(f"{nid}: parent {gp!r}!={p!r}") + if go != o: + diffs.append(f"{nid}: sort {go}!={o}") + if glk != lk: + diffs.append(f"{nid}: link {glk}!={lk}") + extra = sorted(set(got) - set(src)) + if extra: + diffs.append(f"extra nodes {extra}") + if not diffs: + ok += 1 + else: + issues.append((cuid, diffs)) + if verbose: + print(f" nav {cuid}: source {len(src)} nodes, stack {len(got)} " + f"-> {'OK' if not diffs else diffs[:4]}") + return checked, ok, issues + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--inventory", required=True) + ap.add_argument("--plan", required=True) + ap.add_argument("--verbose", action="store_true") + a = ap.parse_args() + if not (KEY and TOKEN): + sys.exit("set CS_API_KEY and CS_MANAGEMENT_TOKEN") + inv = json.load(open(a.inventory)) + plan = json.load(open(a.plan)) + + # original components by uid + orig = {} + for p in inv["contentPages"].values(): + for s in p["slots"]: + for c in s["components"]: + if c.get("uid"): + orig.setdefault(c["uid"], c) + + # which uids we actually seeded (from the plan) + seeded_uids = {e["_key"]["value"] for e in plan["entries"]["components"]} + + assets = asset_titles() + # fetch seeded component entries by content type, index by sap_uid + cs_by_uid, cs_ct = {}, {} + for ct in CT_TO_TYPECODE: + for e in all_entries(ct): + u = e.get("sap_uid") + if u: + cs_by_uid[u] = e; cs_ct[u] = ct + + per = defaultdict(lambda: {"n": 0, "match": 0}) + mismatches, missing = [], [] + NAV_TYPES = {"CategoryNavigationComponent", "FooterNavigationComponent"} + for uid in seeded_uids: + c = orig.get(uid) + if not c: + continue + tc = c["typeCode"] + if tc in NAV_TYPES: # nav components verified separately (nav tree read), not here + continue + cs = cs_by_uid.get(uid) + if not cs: + missing.append(uid); continue + want, got = orig_proj(c), cs_proj(cs, cs_ct[uid], assets) + per[tc]["n"] += 1 + if want == got: + per[tc]["match"] += 1 + else: + mismatches.append((tc, uid, {k: (want.get(k), got.get(k)) + for k in set(want) | set(got) if want.get(k) != got.get(k)})) + + total = sum(r["n"] for r in per.values()) + matched = sum(r["match"] for r in per.values()) + print(f"[verify_connector] {inv['site']} @ {API}") + print(f" authored components checked: {total} matched: {matched} " + f"mismatched: {total - matched} missing-in-stack: {len(missing)}") + for tc in sorted(per): + print(f" {tc:34s} {per[tc]['match']}/{per[tc]['n']}") + if missing: + print(" MISSING:", missing[:10]) + if mismatches and a.verbose: + for tc, uid, d in mismatches[:15]: + print(f" {tc} {uid}: {json.dumps(d)[:200]}") + + # flat navigation: reassemble the seeded nav_node_flat pool and diff the tree + # against the source (skipped in the per-component loop above). + nav_orig = {uid: c for uid, c in orig.items() if c["typeCode"] in NAV_TYPES} + nav_checked, nav_ok, nav_issues = verify_nav(inv, nav_orig, a.verbose) + if nav_checked: + print(f" nav menus (flat) checked: {nav_checked} trees match: {nav_ok}") + for cuid, diffs in nav_issues: + print(f" NAV MISMATCH {cuid}: {diffs[:6]}") + + comp_ok = (matched == total and not missing and total > 0) + nav_pass = (nav_ok == nav_checked) + ok = comp_ok and nav_pass + print(" RESULT:", "PASS — connector model round-trips (components + nav)" if ok + else f"{total-matched} comp mismatch / {len(missing)} missing / " + f"{nav_checked-nav_ok} nav mismatch") + sys.exit(0 if ok else 1) + + +if __name__ == "__main__": + main()