Skip to content

The source API says where a ticket source's reading stopped, so an operator sees the checkpoint itself and not only that the round succeeded (#1212, ADR 0075) - #1455

Merged
WaylandYang merged 1 commit into
deeplethe:devfrom
rollroyces:feat/source-checkpoint-visibility
Oct 29, 2026

Conversation

@rollroyces

Copy link
Copy Markdown
Contributor

Why

A ticket source stops at its per-round budget all the time — 200 tickets or ten listing pages — and a round that stops there is a success: it consumed a prefix, and the next round continues from where it stopped (0072). The source list says "synced" and says how many documents the round added. It does not say where the reading stopped.

The checkpoint that says where lived on the source row with nothing carrying it out of the API, so an operator asking "did this source get everything?" had nothing to read: the run history says how many documents each round added, not how far into the provider's timeline the reading got. The answer was in a log line.

What changes

  • Source.sync_checkpoint and SourceView.sync_checkpoint: the update time a ticket source has consumed through — the time whose whole group was read. The list response carries it, and so do the create/update responses, so the two shapes of a source never disagree about the same row.
  • NULL means there is no checkpoint: the first round, or a kind that does not read by timestamp at all. It doubles as the applicability signal, so a client does not need to know the kinds to know whether the field means anything.
  • The raw timestamp, not a state the server computes. 0033 computes an RSS summary instead, because there the server can compute what applies. Here it cannot: whether the source has more to give is a question only the provider can answer, and the server does not ask it between rounds. A state word would have to guess.
  • web/src/api.ts types it. The interface does not read it yet — where to put it and what to call it is a separate change.
  • ADR 0075 records the shape; docs/design/sources.md records it in the design.

This does not change how a sync reads, or where the checkpoint is written — #1359 already put the checkpoint on the source and commits it with the successful run. This only lets it out of the API.

How it was checked

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace against PostgreSQL 16 / pgvector with UTOPIA_DATABASE_URL set and UTOPIA_TEST_REQUIRE_DB=1: 1797 passed, 1 failed, 6 ignored. The one failure is real_postgres_backup_and_restore_require_force_and_preserve_files_on_sql_failure, which panics with "pg_dump is required for database-backed CLI tests" on a machine without the PostgreSQL client tools; it fails identically on the unmodified base commit, so it is not this change.
  • pnpm typecheck (tsc --noEmit): 0 errors, run twice.
  • pnpm test (vitest): 49 files, 338 tests passed.
  • pnpm build (style-guard + tsc --noEmit + vite build): passed.
  • Two new regressions in crates/utopia-server/src/api/sources_checkpoint_tests.rs: the list carries the checkpoint and leaves it NULL for a kind that does not use one; the update response carries it too.

Before review

  • Every commit is signed off (git commit -s)
  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass (the one CLI failure needs pg_dump, which this machine lacks — it fails identically on the base commit)
  • For changes under web/: pnpm build and pnpm test pass
  • SQL under crates/utopia-store/ was tested with UTOPIA_DATABASE_URL set (those tests skip without it)
  • A change to the data model, the ontology contract or a public API has its ADR in docs/decisions/

…erator sees the checkpoint itself and not only that the round succeeded (deeplethe#1212, ADR 0075)

Signed-off-by: Royce Lam <roycelam@umich.edu>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the field, the NULL-means-no-checkpoint rule and ADR 0075 all read right. Merging.

@WaylandYang
WaylandYang merged commit 8dab158 into deeplethe:dev Oct 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants