[saqibmanan]

Pending: covers open CivicDataLab/DataSpaceBackend#249 (head fa89bfa70595760581fca46aa4d93c06f4d54797). Skipped in CI until it merges. Safe to merge before it.

Source PR

CivicDataLab/DataSpaceBackend#249 — "feat(storage): move media to S3 and remove DVC versioning"

• base: dev, open, created 2026-10-19 (>72h old, no coverage yet)
• headRefOid: fa89bfa70595760581fca46aa4d93c06f4d54797

What actually changed (from the diff, not the title)

This is a large PR (40 files); the slice this adds coverage for is the one piece that's both unauthenticated and testable without a real backend deployment: the legacy media route.

• DataSpace/urls.py: the old static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) route (Django serving files/public/... off local disk, only ever live when DEBUG=True) is replaced by legacy_media.legacy_media_redirect at the same /public/<name> prefix.
• api/views/legacy_media.py (new): redirects a valid, existing public file's old link to its S3 URL; 404s with {"error": "Not found"} (JSON) for (a) a name that doesn't exist in public storage, and (b) a name whose posixpath.normpath differs from itself (path traversal) — rejected before storage is even consulted.
• api/utils/storage.py (new): public_storage()/private_storage() wrappers and the files/public//files/ legacy-prefix stripping used by the redirect.

Confirmed directly against the PR diff in DataSpaceBackend (DataSpace/urls.py, api/views/legacy_media.py, api/utils/storage.py), not from the PR title/description.

Category chosen: api + smoke

Pure unauthenticated HTTP status/content-type checks against a REST-ish Django view, no GraphQL and no UI — api fits. smoke is required alongside api/pending_pr so CI's pull_request marker filter (-m "smoke") actually selects it; api-only would be silently deselected (this repo has no job that ever selects regression either — SKILL.md §5). Never readonly, per §9: the gate opens on merge to dev, but the prod-deploy gate runs readonly tests, and this would fail every prod deploy until #249 actually ships.

Test added

tests/api/smoke/test_api_028_legacy_media_redirect.py — two unauthenticated REST checks against anon_api_client (no new fixtures needed):

1. test_missing_public_file_404s_as_json_not_django_html — GET /public/<missing-name> must 404 with Content-Type: application/json and body {"error": "Not found"}.
2. test_public_file_name_with_path_traversal_404s_as_json — GET /public/..%252f..%252fetc%252fpasswd (double-encoded, survives nginx to actually reach the Django view — confirmed separately that a single-encoded or bare ../ payload either gets blocked at nginx with a 400, or gets resolved by Django's own router before /public/, so neither reaches the view) must also 404 as JSON.

Both carry pytest.mark.pending_pr("DataSpaceBackend#249") (new file, via pytestmark), plus api and smoke.

Red proof — against dev, which still serves the pre-#249 code

Today's dev backend answers both cases with Django's HTML "technical 404" page, not JSON:

$ curl -sS -i https://dev.api.civicdataspace.in/public/does-not-exist-pr249-probe.png
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=utf-8
<!DOCTYPE html>... <title>Page not found at /public/does-not-exist-pr249-probe.png</title> ...

$ curl -sS -i "https://dev.api.civicdataspace.in/public/..%252f..%252fetc%252fpasswd"
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=utf-8
<!DOCTYPE html>... <title>Page not found at /public/..%2f..%2fetc%2fpasswd</title> ...

Pytest run with the pending_pr marker temporarily removed locally (reverted before commit, never pushed), against dev:

FAILED tests/api/smoke/test_api_028_legacy_media_redirect.py::test_missing_public_file_404s_as_json_not_django_html
AssertionError: expected the new legacy_media view's JSON 404 (Content-Type: application/json), got 'text/html; charset=utf-8'. This is the pre-#249 Django static()/debug-404 page, not the PR's view.

FAILED tests/api/smoke/test_api_028_legacy_media_redirect.py::test_public_file_name_with_path_traversal_404s_as_json
AssertionError: expected the new legacy_media view's JSON 404, got 'text/html; charset=utf-8' (body: <!DOCTYPE html> ...)

2 failed in 1.52s

Both fail at the intended content-type/body assertion, not a setup error — this is a genuine red, not a vacuous one.

Green — UNVERIFIED, not executed anywhere

UNVERIFIED: PR not deployed anywhere. DataSpaceBackend cannot be run in this sandbox (no Postgres, Redis, Elasticsearch, Keycloak or S3 available), so the post-#249 JSON-404 behaviour has only been read from the diff, never executed. Once #249 merges and is live on dev, re-run this file; both tests are expected to flip to passing with no changes needed here.

Skip-gating + collection proof

Marker present (as shipped), against dev:

tests/api/smoke/test_api_028_legacy_media_redirect.py::test_missing_public_file_404s_as_json_not_django_html SKIPPED
tests/api/smoke/test_api_028_legacy_media_redirect.py::test_public_file_name_with_path_traversal_404s_as_json SKIPPED
SKIPPED [2] ...: pending_pr DataSpaceBackend#249: not merged yet
2 skipped in 1.31s

Collected under both of CI's real marker filters:

$ pytest tests/api/smoke -m "smoke" --collect-only -q   # pull_request filter
...test_api_028_legacy_media_redirect.py (both tests)
56/129 tests collected (73 deselected)

$ pytest tests/api -m "smoke or functional" --collect-only -q   # dispatch filter
...test_api_028_legacy_media_redirect.py (both tests)
107/135 tests collected (28 deselected)

Gaps / out of scope

• The rest of #249 (S3 storage wiring for uploads, presigned private URLs, migrate_media_to_s3, DVC removal, resource-file SHA256 hashing on local_copy()) all need either an authenticated write (upload a file) or a live backend with real S3 credentials — neither is available in this sandbox. UNVERIFIED — not executed, no credentials in this environment. A human with TEST_EMAIL_1/TEST_PASSWORD_1 and a real backend deployment should extend this file (or add a sibling) to cover: a presigned private URL actually expiring after 5 minutes, an upload landing under the right <STORAGE_ENV>/public or <STORAGE_ENV>/private prefix, and the breaking-change migration steps in the PR body.
• This PR is also flagged by the backend PR's own description as having breaking deploy steps (new required env vars, a data migration script) — those are infra/ops concerns, not something a regression test in this repo can verify.

Housekeeping

• Branch test-sync/DataSpaceBackend-pr249, based on origin/CI (confirmed via recent merged test-repo PRs that CI, not main/dev, is where this repo's test work lands and where its workflows run — main only takes docs, dev runs days behind).
• Duplicate check (SKILL.md §8) run clean before writing anything: no existing test-sync/DataSpaceBackend-pr249 branch, no test-repo PR mentioning DataSpaceBackend#249 or #249.
• File numbering: test_api_028_* — confirmed free on origin/CI (git ls-tree showed 001–024, 027, nothing at 025–026/028+ before this; #224's and #420's pending PRs already claimed 025/026).
• Draft only — not requesting review, not merging.

---

Run context: scheduled cloud routine, no test credentials in this environment. Everything executed above against dev/public endpoints is real and was actually run — only the post-merge green and the rest of #249's authenticated/write surface are UNVERIFIED.

[saqibmanan]

consumer-smoke is red on this PR, but not because of this PR — the only change here is a new, currently-skipped file under tests/api/smoke/ (gated by pending_pr), and consumer-smoke doesn't even collect tests/api.

The failure is tests/consumer/smoke/test_components.py::test_TC_SEC_03_sort_dropdown, a pre-existing test this branch doesn't touch: '//div[contains(@class, 'sort') or .//select or .//button[contains(., 'Sort')]]' not satisfied by visibility_of_element_located. It failed at that exact assertion on every attempt including pytest's own built-in reruns (2 rerun in the summary), and the identical failure also reproduced on two sibling PRs from this same run (#201, #202) that touch entirely different files — so this isn't contention/flakiness, it's either the sort-dropdown locator or the live dev homepage no longer matching it.

No fix exists yet for this that I can port in. Leaving this for a human to look at test_components.py's sort-dropdown locator against the current dev homepage; not something a test-sync PR should patch blind.

---

Generated by Claude Code

[saqibmanan]

provider-smoke is also red on this PR (11 failed, 3 passed, 22 rerun), but again not because of this PR's change — the only file added here is under tests/api/smoke/, skipped by its pending_pr marker. Every failure is a selenium.common.exceptions.TimeoutException after a successful login, waiting on a data-dependent element (org dashboard Drafts tab, AI model editor Metadata tab, sidebar nav items) — exactly the "backend rate limit" hang pattern SKILL.md documents: login succeeds, then the page that needs data never renders because the backend is throttling.

This run fired api-smoke + consumer-smoke + provider-smoke back to back across three sibling PRs (#200, #201, #202) within about 20 minutes, which is well within what the skill says can exhaust DataSpaceBackend's 1000 POST/hour-per-IP limit from one CI runner pool. Probed https://dev.api.civicdataspace.in/api/graphql from outside the runner just now and got a clean 200, but the limit is per-IP and GitHub's runner IP isn't mine, so that's not conclusive either way.

Not re-running manually: the rate-limit window is fixed and clears up to 60 minutes after it opened, not 60 minutes after the last request, so an immediate re-run would likely just hit the same throttling again. Leaving this for a later CI run (or a human) once the window has cleared — not something this test-sync PR should chase.

---

Generated by Claude Code