Repository navigation
fix: count gallery posts and crossposts in the image duplicate scan - #89
Merged
Merged
Conversation
Gallery posts were not image posts to media_kind, so the scan neither compared with them nor counted them, and a post whose preview could not be read was dropped silently. Galleries now count as image posts (first 10 pictures, any picture matching), and an unreadable preview falls back to the full image. Co-Authored-By: Toldry <toldry@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXeu2v7gGePkS9NhaCgwDy
A crosspost's url is often the original post's permalink, with no preview, so crossposted images, galleries and videos were skipped and left out of the reply's image count. Co-Authored-By: Toldry <toldry@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXeu2v7gGePkS9NhaCgwDy
Toldry
changed the title
Co-Authored-By: Toldry <toldry@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXeu2v7gGePkS9NhaCgwDy
…ning Co-Authored-By: Toldry <toldry@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXeu2v7gGePkS9NhaCgwDy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Querent · project thread
Before: the reply under a crosspost to r/ShitAmericansEat said "I also looked through all 1 image posts made in r/ShitAmericansEat in the last 30 days", although the subreddit had 3 image posts in that period. On the VM, the 2 that were missed turned out to be crossposts. One was a crosspost of an image, whose
urlis the original post's permalink (/r/ShitAmericansSay/comments/...). The other was a crosspost of a gallery (url=reddit.com/gallery/<other id>,is_galleryNone).media_kindreturned None for both, so the scan never compared with them or counted them. Gallery posts in general were also skipped. A post whose preview could not be downloaded was dropped from the count silently, logged at DEBUG only.After: a crosspost's media is read from the original post (
crosspost_parent_list[0]), so crossposted images, galleries and videos are compared and counted. A gallery counts as an image post: its first 10 pictures are fingerprinted, and it matches when any of its pictures matches any picture of the source. A gallery source post is scanned too. When a preview cannot be read, the full image is tried. A scan that still cannot read some posts logs a WARNING saying how many.How:
media_field/media_urlread media fields from the crosspost parent when there is one, and the crosspost's own fields when the parent was deleted.gallery_image_urlsreadsmedia_metadata/gallery_data.fingerprint_postreturns a post's fingerprints, with the full-image fallback.best_matchcompares every pair of pictures. Unit tests cover galleries, crossposted images, galleries and videos, crossposts of text posts, crossposts whose original was deleted, the gallery cap, a gallery source post and the fallback.🤖 Generated with Claude Code
https://claude.ai/code/session_01PXeu2v7gGePkS9NhaCgwDy