fix(dataset): delete child chunks the index processor leaves behind

[shaba]

Extracted from #38519 so it can be reviewed independently. Refs #38518.

The leak

Nothing in clean_dataset_task deletes rows from child_chunks. The only dataset-wide delete lives in ParentChildIndexProcessor.clean(), and the entire block that performs it sits inside:

if dataset.indexing_technique == IndexTechniqueType.HIGH_QUALITY:
    ...
    if delete_child_chunks:
        # Use existing compound index: (tenant_id, dataset_id, ...)
        session.execute(
            delete(ChildChunk).where(
                ChildChunk.tenant_id == dataset.tenant_id, ChildChunk.dataset_id == dataset.id
            )
        )

A dataset whose indexing_technique is anything other than high quality therefore keeps every child chunk after it is deleted. ParagraphIndexProcessor.clean() and QAIndexProcessor.clean() never touch the table at all.

The same rows are stranded on a second path, one the task tolerates on purpose. clean_dataset_task wraps the index cleanup in its own try/except and continues deliberately:

except Exception:
    logger.exception(...)
    # Continue with document and segment deletion even if vector cleanup fails

Documents, segments, attachments and metadata are then deleted, and the child chunks that referenced them are not. So whenever the vector store is unreachable at delete time — the case that except exists for — the child chunks are orphaned even for a high-quality parent-child dataset.

Both leaks are silent. The dataset disappears from the UI, the rows stay in the table indefinitely, and because the dataset row is already gone there is nothing left to retry the cleanup.

The change

One delete in the task, after the index processor has had its turn:

session.execute(
    delete(ChildChunk).where(ChildChunk.tenant_id == tenant_id, ChildChunk.dataset_id == dataset_id)
)

The predicate matches both columns so it can use the existing compound index child_chunk_dataset_id_idx on (tenant_id, dataset_id, document_id, segment_id, index_node_id) — the same access path the parent-child processor already relies on for its own delete.

Placing it after the index cleanup makes it a no-op for the high-quality parent-child datasets that are already handled: those rows are gone by the time it runs.

Tests

Two cases in TestChildChunkCleanup, using the suite's existing SQLite session factory and fixtures:

• a child chunk that the index processor did not delete (the mocked processor stands in for both leak paths) is gone after the task runs — this fails on main
• another dataset's child chunks survive, covering the scoping of the predicate

tests/unit_tests/tasks/test_clean_dataset_task.py passes: 10 tests. ruff check and ruff format --check are clean.

[github-actions]

Pyrefly Type Coverage

Metric	Base	PR	Delta
Type coverage	61.14%	61.14%	+0.01%
Strict coverage	60.73%	60.74%	+0.01%
Typed symbols	41,940	41,954	+14
Untyped symbols	26,838	26,840	+2
Modules	3248	3248	0

[github-actions]

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-08-25 14:30:11.220967229 +0000
+++ /tmp/pyrefly_pr.txt	2026-08-25 14:29:56.941871092 +0000
@@ -9143,7 +9143,7 @@
 ERROR Argument `SimpleNamespace` is not assignable to parameter `account` with type `Account` in function `services.workflow_restore.apply_published_workflow_snapshot_to_draft` [bad-argument-type]
   --> tests/unit_tests/services/workflow/test_workflow_restore.py:70:17
 ERROR Argument `list[FromClause]` is not assignable to parameter `tables` with type `Sequence[Table] | None` in function `sqlalchemy.sql.schema.MetaData.create_all` [bad-argument-type]
-  --> tests/unit_tests/tasks/test_clean_dataset_task.py:91:56
+  --> tests/unit_tests/tasks/test_clean_dataset_task.py:93:56
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]
   --> tests/unit_tests/tasks/test_dataset_indexing_task.py:51:5
 ERROR Object of class `FunctionType` has no attribute `return_value` [missing-attribute]

[shaba]

The Web Full-Stack E2E failure here is unrelated to this change.

This PR touches two files, both Python: api/tasks/clean_dataset_task.py and its unit test. The failing scenario is a Playwright test of the Agent v2 Access Point UI — features/agent-v2/access-point.feature:69, "Web app access can be disabled and restored from Access Point" — which times out waiting for an In service badge to render. 56 of the 57 scenarios passed, and nothing in that path reads dataset cleanup code.

For comparison, #38519 changes the same task far more extensively and its run of the same E2E job passed a few minutes later, so the suite was exercising that scenario successfully at the time.

API Unit Tests and API Integration Tests both pass here. I do not have permission to re-run the job; a maintainer re-run should clear it.

[shaba]

Correction to my previous comment: I cited #38519 as having passed this same E2E job. That is no longer true — its next run failed the same scenario, so the comparison I drew there does not hold, and I withdraw it.

The conclusion is unchanged, but here is the actual evidence. features/agent-v2/access-point.feature:69 is failing intermittently across unrelated PRs right now. For example run 32860893991 on branch agent/sqlite-orm-app-pipelines — an ORM migration touching nothing in this area — fails on exactly the same step, same locator, same result line:

1) Web app access can be disabled and restored from Access Point # features/agent-v2/access-point.feature:69
     Then Agent v2 Web app access should be in service
         Locator: getByRole("region", { name: "Access Point" }).getByRole("article", { name: "Web app" }).first().getByText("In service")
         Timeout: 30000ms
57 scenarios (56 passed, 1 failed)

The same scenario also passed and then failed on an unchanged branch here, which is the signature of a flaky test rather than a regression.

This PR adds one delete statement to api/tasks/clean_dataset_task.py plus unit tests; no frontend code and no code path reachable from the Access Point UI. API Unit Tests and API Integration Tests pass.

[shaba]

Filed the E2E flake as #41263, with the sampling across recent CI runs and an analysis of where the race appears to be.