Status: source-present for a single atomic partition; delivered-source GitHub qualification remains pending. The design target includes replayable projections and resumable public feeds. Cross-partition coverage and external-index publication are planned.
ChangeFeeds exposes committed document mutations to authorized readers and bounded scalar live queries, and provides a system outbox for in-partition projections. Actors are database clients, projection workers, and query clients. Current HTTP/.NET entry points are POST /v1/changes/read, POST /v1/query/live/start, POST /v1/query/live/read, and administrator projection operations under /v1/admin/projections/*; their contracts are in src/KeyLoad.Abstractions/ChangeFeeds.cs, implementation in src/KeyLoad.Core/ChangeFeeds.cs, ProjectionOutbox.cs, and src/KeyLoad.Query/LiveQueries.cs, and SDK methods in src/KeyLoad.Client/KeyLoadClient.cs. These are existing routes, not a proposal for additional endpoints.
| Surface | Current source | Target owner |
|---|---|---|
| Contracts | src/KeyLoad.Abstractions/ChangeFeeds.cs |
src/KeyLoad.Abstractions/Features/ChangeFeeds/ |
| Outbox and public feed | src/KeyLoad.Core/ProjectionOutbox.cs, ChangeFeeds.cs |
src/KeyLoad.Core/Features/ChangeFeeds/ |
| Scalar live query | src/KeyLoad.Query/Features/ChangeFeeds/LiveQueryExecutor.cs; public facade LiveQueries.cs |
Slice-local private owner borrowing the existing database/query engine; same single read cut |
| HTTP/.NET SDK | src/KeyLoad.Server/ApiEndpoints.cs, src/KeyLoad.Client/KeyLoadClient.cs |
Shared entry points; behavior remains in this slice |
| Tests | tests/KeyLoad.UnitTests/Features/ChangeFeeds/ feed/projection suites, LiveQueryTests.cs; tests/KeyLoad.RecoveryTests/ProjectionRecoveryTests.cs; tests/KeyLoad.IntegrationTests/ClusterTests.cs |
Unit feed/projection source is slice-local; remaining layout debt targets matching Features/ChangeFeeds/ folders |
| Durable design | ../design/change-feeds.md |
Detailed design reference; this file owns feature acceptance |
| Frontend | None | N/A: polling and query results are database/client contracts, with no independent UI |
| Official MCP | No implementation found | Required ClientApi caller surface; qualification is pending, not inferred from HTTP routes |
Current root-level source locations are migration debt under ADR-032. ChangeFeeds does not own the private redo journal, business event streams, message leases, replica placement, or external search-index files. A per-partition outbox is not a cross-partition CDC guarantee.
ADR-035 adds REQ-FEED-006, mapped to AC-FEED-004/005 and AC-MP-006/012: carry actual stored outbox byte length and owned lookup key alongside the decoded entry instead of copying/rereading its payload for quota subtraction during purge. One canonical point-reader decodes borrowed bytes and validates sequence/partition before yielding an owned entry. Existing feed/projection iteration reuses it; purge mutates only between point reads, not from a range callback. Producer, wire/signatures, filters, checkpoint pins, contiguous progress and exact response serialization budgets stay. Raw stored length cannot replace response size without a separate compatibility proof.
Purge passes min(head.Tail, ThroughSequence) as the reader's inclusive upper
position. It neither decodes nor validates a retained entry beyond that requested
cut; a no-op or already-reclaimed prefix reads no entries. Gaps and invalid
sequence/partition metadata inside the requested prefix still fail atomically.
Prepare the partition identity once per iterator, rather than hashing it for each
entry. This changes read work only, not committed purge outcomes or retention pins.
Worker owns only ProjectionOutbox.cs range-reader/purge regions and new Core/UnitTests Features/ChangeFeeds helpers/tests. Shared contracts, ChangeFeeds.cs, live queries, other producer/consumer methods, docs and CI remain lead-owned. Real-store tests first prove stored-byte reclamation across large/Unicode entries, pinned/partial purge, rollback on missing/wrong-sequence/partition records, reopen and a healthy subsequent producer. A corrupt next retained entry proves that no-op and prefix purge do not inspect it; restoring that real-store entry allows the next projection batch to consume it. GitHub executes TUnit/recovery/RF3; source and development builds do not establish reduced physical I/O.
Projection batch reads reuse their already-loaded outbox head and pass its tail
to the shared entry reader, avoiding a second metadata lookup. Per-entry byte
accounting uses the captured stored length only after a real-store test proves
that JsonDefaults round-trips a large Unicode outbox entry byte-for-byte. Exact
and one-byte-too-small batch limits retain their previous result/error semantics;
one single-entry batch performs exactly one lookup each for the principal,
consumer, head and outbox entry.
- Canonical mutations and their outbox entries are committed together. A failed batch exposes no partial entries; a retry with the same command does not append duplicates. Outbox positions are partition-local and monotonically increasing.
- Projection consumers persist an immutable filter/generation, contiguous checkpoint, signed batch token, replay receipt, and retention pin. Effects and checkpoint commit atomically in the same atomic partition. Failed effects do not advance the checkpoint; stale generation tokens are rejected.
- Public document feeds require
ChangesRead | DocumentsRead. Signed cursors bind the principal, policy/schema/visibility epochs, partition, collection, database incarnation, and position. Bounded pages project before/after images through row and field policy; deletion metadata has no after payload. - Scalar live queries capture a bounded initial Q1 snapshot and outbox cut together, then return bounded upsert/remove deltas. Unsupported ordering, top-k, aggregation, joins, graph, or ANN subscriptions are rejected by the current profile.
- Current source does not establish external-file atomic publication, automatic retention scheduling, multi-partition merge, or all event/topic/group feed semantics. Those remain planned and require their own contracts and evidence.
| Requirement | Measurable acceptance | Existing or planned evidence |
|---|---|---|
| REQ-FEED-001: publish canonical changes and projection effects atomically | AC-FEED-001 passes when a failed producer/effect batch changes neither canonical state nor visible outbox/checkpoint, and replay of the same command/batch returns its prior outcome without duplicate effects. | Existing ChangeFeedTests.EveryCanonicalMutationAndItsCommitShareThePartitionOutbox, FailedBatchAndSameCommandRetryCannotPublishPartialOrDuplicateChanges, FailedProjectionEffectsDoNotAdvanceTheCheckpointAndDuplicateDeliveryIsSafe; recovery ProjectionCrashRecoversEffectsOutboxReceiptOutcomeAndCheckpointTogether. |
| REQ-FEED-002: expose resumable, bounded, authorized document changes | AC-FEED-002 passes when a page returns only currently and historically visible, safely projected changes; its signed cursor resumes within retention, and byte/position bounds preserve the first undelivered entry. Stale policy/ACL/incarnation or reclaimed history returns an explicit error. | Existing ChangeFeedTests.FeedAndProjectionByteBoundariesPreserveTheFirstUndeliveredPosition, FeedResumesAnEmptyTailProjectsPiiAndRetainsDeletionMetadata, FeedNeverReturnsFormerOwnersDataAndAclChangesFenceOldCursors, FeedRevocationHistoryLossAndScopeMismatchAreExplicit. |
| REQ-FEED-003: make scalar live snapshot plus tail gap-free | AC-FEED-003 passes when a concurrent committed write is present either in the initial snapshot or a later delta, and replayed authorized deltas match the ordinary query oracle. Unsupported query profiles fail explicitly. | Existing LiveQueryTests.ScalarLiveDeltasMatchTheSharedQueryOracleAcrossSeededMutationsAndReplay, ConcurrentInitialSnapshotAndTailNeverLoseACommittedDocument, UnsupportedRankingIncompleteSnapshotsAndDifferentQueryCursorsAreRejected. |
| REQ-FEED-004: advance projection checkpoints contiguously under pinned retention | AC-FEED-004 passes when filtered positions advance contiguously, no checkpoint crosses a pending effect, and purge cannot pass the lowest active generation checkpoint. | Existing ChangeFeedTests.RetentionPinsProtectUnprocessedAndRebuildHistory, FilteredProjectionPositionsAdvanceContiguouslyWithoutPublishingForeignResources, AReleasedGenerationRejectsOldBatchesAndCachedCommandReceipts; planned real multi-partition coverage tests. |
| REQ-FEED-005: recover retained feed state without silent gaps | AC-FEED-005 passes when reopen/compaction and the declared RF3 snapshot-install path preserve outbox/checkpoint/receipt state or explicitly invalidate old cursors; history loss requires resynchronization. | Existing OutboxAndConsumerReceiptsRecoverAfterReopenAndCompaction, ProjectionRecoveryTests.ProjectionCrashRecoversEffectsOutboxReceiptOutcomeAndCheckpointTogether; current-source GitHub qualification and broader leader-loss cases remain pending. |
| REQ-FEED-006: bound projection read work without changing byte-budget behavior | AC-FEED-006 passes when persisted entry bytes equal JsonDefaults serialization after decode, exact and one-byte-short limits preserve success/rejection, and an isolated one-entry batch performs exactly four borrowed point lookups. |
ProjectionReadWorkTests.AcMp006ProjectionBatchReusesHeadAndStoredEntryBytes; real ZoneTree counters; TUnit execution remains GitHub Actions only. |
Empty-tail cursors are valid. Filtered partition positions may produce an empty page with HasMore; callers continue until caught up. A first item larger than the byte budget fails with BudgetExceeded without moving past it. Missing authorization fails before reading payloads. Principal revocation or policy/schema/row-visibility changes fence old cursors. A cursor behind a reclaimed prefix returns HistoryUnavailable, requiring a fresh snapshot. A deleted change preserves revision/deletion metadata and has no After payload; its Before image may be present only when historical and current row authorization allow it, and it is safely projected. Public readers never receive raw system-outbox entries or unrelated resource identities.
Related decisions: ADR-017, ADR-022, ADR-023, ADR-024, ADR-025, ADR-027, ADR-029, and ADR-030. Cluster routing follows the pending ADR-036. The operational distinction among redo journal, business streams, queue state, and outbox is in change-feed design.
TUnit unit, process-recovery, and Docker/Aspire RF3 cases execute through GitHub Actions only. The current source/test names above are traceability, not a passing result for the current checkout. Required external-index manifests, multi-partition semantics, official MCP calls, and delivered-source CI evidence remain pending. Tests use the real ZoneTree store and real cluster/client path; no mock-only acceptance is permitted.
sequenceDiagram
participant Writer
participant Partition as Atomic partition
participant Outbox
participant Reader
Writer->>Partition: Commit mutation and outbox entry
Partition-->>Outbox: Same durable apply frame
Reader->>Outbox: Read authorized bounded page
Outbox-->>Reader: Projected changes and signed cursor
Reader->>Outbox: Resume cursor or request live delta