[bdb-dd]

Fixes #107.

What

A normal pipeline run now keeps the documents of the dataset's other enabled pipelines, so a dataset can be filled by several pipelines. Engine switching (ADR 0025) already re-ingests several pipelines into one generation; normal runs now preserve them too.

• Ownership. Each stored document records the pipeline that wrote it, in a dedicated IndexDocument.pipeline attribute, separate from source metadata.
  • pgvector stores it in a new nullable kn_docs.pipeline column, added by an idempotent migration. Existing rows stay NULL.
  • A source field named ingestion_pipeline, or anything else, is ordinary metadata. It is never read as ownership, and fresh attribution never overwrites it.
• Carry forward. runIndexIngest writes the run's fresh documents into a new generation. It then streams the active generation and copies the documents owned by enabled siblings, with their stored vectors, chunks and phrases. No sibling source is fetched and nothing is embedded again. If a sibling has the same document id, the fresh document wins.
• Conditional promotion. A new optional port operation, IndexWriter.promoteIfActive(dataset, generation, expected), promotes only if the active generation is still the one the run read.
  • pgvector compares and promotes in one transaction, under the existing kn_active advisory lock.
  • A run that overlaps a sibling's promotion fails with "changed during ingestion; run the pipeline again", instead of dropping the sibling's newer documents. This holds across API instances.
• Lifecycle. The run snapshots the enabled, undeleted siblings in the dataset's scope when it is admitted, for API, operator and scheduled runs alike. A disabled or deleted pipeline's documents drop out at the next successful sibling run. A successful empty run on a multi-pipeline dataset promotes just its siblings. An all-failed run does not promote.
• Single-pipeline datasets run as before: no read-back or copy, and one cheap active-generation lookup on writers with conditional promotion.
• Writers without read or promoteIfActive refuse a multi-pipeline run before preparing or embedding anything, with an explicit unsupported error, rather than losing data. That includes Typesense today. Single-pipeline runs on them are unchanged.
• Engine switches.
  • Combined re-ingest attributes each pipeline and does not copy the previous generation.
  • Copy switches preserve the pipeline attribute.
  • Typesense stores no ownership (preservesPipeline: false), so a copy from Typesense yields unattributed documents.

Upgrade note

On an existing dataset with several pipelines, stored documents have no ownership yet. The first promotion after upgrade therefore drops the other sources' documents until each pipeline has run once. Every source must be available. The sequence, documented in docs/api/datasets.md and ADR 0025:

1. Pause schedules.
2. Run each enabled pipeline once, in sequence.
3. Re-enable schedules.

Datasets with one pipeline are not affected. Today such datasets already show only the last pipeline's documents, so this is not a regression. But it is destructive for whatever happens to be in the active generation.

Open questions

1. Rollout (decided): the documented maintenance rebuild. Pause schedules, run each enabled pipeline once in sequence, then re-enable schedules. No gated migration action.
2. Typesense. Multi-pipeline runs stay refused until Typesense has a safe conditional promotion. Is that acceptable?
3. Identity. Should document, chunk and phrase ids stay shared across pipelines, with fresh content winning a collision, or be namespaced per pipeline later?
4. Cleanup. Retention of generations left unpromoted by failed or stale runs. No collector is added here.
5. Scheduling. Sibling pipelines scheduled for the same slot can collide on one instance and need a retry. The scheduler is unchanged.
6. Embedding changes. Changing a dataset's embedding model should rebuild every source rather than carry vectors across embedding spaces. This needs a coordinated policy.

Tests

• Real-path regressions on the memory writer and on pgvector via Testcontainers, through IngestionService → indexIngestionBackend → runIndexIngest → writer:
  • run a then b, and both are searchable;
  • vectors are copied without an embed call;
  • disabled and deleted siblings, scope, and metadata turned off;
  • legacy rebuild, read and copy failure, and empty sources;
  • id collisions and missing writer capabilities;
  • combined engine-switch generations;
  • two service instances overlapping, and a retry.
• Database-source regressions (rowMapping → databaseSource → runIndexIngest):
  • a user column named ingestion_pipeline survives as metadata;
  • pre-upgrade metadata with that name is not trusted as ownership;
  • fresh metadata cannot attribute a document to a sibling.
• Writer contracts:
  • the optional conditional promotion: stale, initially empty, and concurrent;
  • the optional provenance round trip, including clearing on unattributed replacement.
    Both run on memory and pgvector, and are skipped only where an adapter declares no support.
• The pgvector migration is applied twice to a simulated old table.
• Sabotage legs, each turning its named tests red:
  • carry-forward off;
  • conditional promotion bypassed;
  • attribution off;
  • fresh and legacy metadata collisions;
  • pgvector persistence and migration;
  • copy-switch attribution;
  • Typesense extra writes;
  • the id-collision rule.
• tsc -b, tsc -p tsconfig.test.json, lint, format and boundaries are clean. Vitest over apps/api, packages/ingest, packages/providers-* and packages/port-knowledge: 1,854 passed, 12 skipped.

Draft, for review.

Independent verification

A separate verifier found three issues.

• The safeguards for all-failed runs and copy errors were not pinned by tests. Now they are.
• Changing a dataset's embedding model could mix two embedding spaces in one generation. Fixed: each document records its embedding model next to its pipeline (a nullable kn_docs.embedding_model column). A carry-forward over a sibling with a different or unknown model is refused with a 'run a combined rebuild' error. Copy switches re-embed when the models differ. This answers open question 6.
• The legacy rollout gate (open question 1) is decided: the documented maintenance rebuild.

[kentare]

The review on #125 asks whether this PR could follow ADR 0042's ownership rules as an interim step on today's tables (0042, Ingestion runs). I agree, and read the code against them. Take a dataset filled by two pipelines, A and B:

1. Disabling B drops B's documents at A's next run. A run takes a snapshot of its enabled siblings only (ingestion.ts:510-514), and the writer copies only theirs (index-writer.ts:190). In 0042, disabling a pipeline pauses it and keeps its documents.
2. Deleting B removes its documents as a side effect of A's next run. Removing a pipeline only writes a deleted version (ingestion.ts:448-471), and B's documents go when a sibling promotes without them. In 0042, deleting a pipeline starts a removal run of its own, which deletes only the documents no other pipeline owns, through the change brake.
3. After the upgrade, A's first run drops B's documents until B runs again. Existing rows get no owner (schema.ts:31), only attributed documents are copied, and nothing warns. 0042's rule is to refuse rather than drop: carry documents without an owner forward, or stop the run, until every pipeline has run once.

One more, which the description doesn't mention: with siblings, a successful run that lists zero documents now promotes (index-writer.ts:184), so an empty answer from a source deletes all of that pipeline's documents. 0042 deletes a listed document only after two complete listings without it, and stops a run that would change or delete more than 5 % and more than 50 of its pipeline's documents until someone approves it. Even a simple limit like that would catch all four cases.

Could this PR keep a disabled pipeline's documents, remove a deleted pipeline's documents only by an explicit removal, and refuse rather than drop when ownership is unknown? Then 0042 can point to it as the interim step.