Skip to content

feat: media deduplication - #296

Draft
itsoyou wants to merge 2 commits into
masterfrom
syk/scope-out-dedup
Draft

itsoyou wants to merge 2 commits into
masterfrom
syk/scope-out-dedup

Conversation

@itsoyou

@itsoyou itsoyou commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

SYN-150

Pull Request Checklist

  • In the description of your pull request:
    • Describe your change in a way that makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
    • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
  • Code style is correct (run the linters)

@itsoyou
itsoyou requested a review from a team as a code owner October 13, 2026 11:06
@itsoyou

itsoyou commented Oct 13, 2026

Copy link
Copy Markdown
Member Author

@famedly-agent review

@ghost

ghost commented Oct 13, 2026

Copy link
Copy Markdown

No reviewer covers this PR yet: 10 Python files.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 88f6284. Configure here.


with start_active_span("writing to main media repo"):
with f:
yield f, media_filepath

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shared thumbnails overwritten and deleted

High Severity

Thumbnails with a known sha256 are written in place with wb and always treated as owned. A second upload of the same content truncates the shared thumbnail, and a later spam-check or storage-provider failure deletes that file for every media ID still using it.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 88f6284. Configure here.

thumbnail_dir = self.filepaths.local_media_thumbnail_dir(sha256)

try:
os.remove(full_path)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete races with in-progress upload

High Severity

_remove_sha256_files_if_unused deletes the SHA-256 blob after a refcount check, but a concurrent upload already placed bytes at that path and has not inserted its DB row yet (quarantine and upload-limit checks run in between). The new media is stored in the database with no file on disk.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 88f6284. Configure here.

@codecov

codecov Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.99035% with 84 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.41%. Comparing base (04dc1d4) to head (5000b4f).

Files with missing lines Patch % Lines
synapse/media/media_repository.py 70.50% 34 Missing and 7 partials ⚠️
synapse/media/storage_provider.py 43.13% 29 Missing ⚠️
synapse/media/media_storage.py 87.61% 8 Missing and 6 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #296      +/-   ##
==========================================
- Coverage   81.22%   78.41%   -2.81%     
==========================================
  Files         510      509       -1     
  Lines       74055    74036      -19     
  Branches    11223    11244      +21     
==========================================
- Hits        60150    58056    -2094     
- Misses      10600    12607    +2007     
- Partials     3305     3373      +68     
Files with missing lines Coverage Δ
synapse/media/_base.py 70.49% <100.00%> (+0.09%) ⬆️
synapse/media/thumbnailer.py 78.17% <ø> (ø)
synapse/storage/databases/main/media_repository.py 86.56% <100.00%> (+0.38%) ⬆️
synapse/media/media_storage.py 76.32% <87.61%> (-3.50%) ⬇️
synapse/media/storage_provider.py 68.64% <43.13%> (-20.25%) ⬇️
synapse/media/media_repository.py 74.86% <70.50%> (-0.72%) ⬇️

... and 124 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 04dc1d4...5000b4f. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@itsoyou
itsoyou marked this pull request as draft October 14, 2026 08:33
@itsoyou
itsoyou force-pushed the syk/scope-out-dedup branch from 88f6284 to 5000b4f Compare October 26, 2026 12:40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant