diff --git a/Dockerfile b/Dockerfile index 2fa010bb..2ad10c7d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -43,11 +43,12 @@ WORKDIR /opt/explo/ # Copy entrypoint, binary, python helper COPY ./docker/start.sh /start.sh +COPY ./docker/explo-run.sh /explo-run.sh COPY --from=builder /app/explo . COPY src/downloader/youtube_music/search_ytmusic.py . -RUN chmod +x /start.sh ./explo +RUN chmod +x /start.sh /explo-run.sh ./explo ENV WEB_ADDR=":7288" diff --git a/TODO.md b/TODO.md new file mode 100644 index 00000000..494bcdd4 --- /dev/null +++ b/TODO.md @@ -0,0 +1,311 @@ +# Explo — Bug Tracking TODO + +> Persistent tracker for the code-review bug list. Keep this file updated across sessions — it is the single source of truth for what is open, in progress, and done. +> YouTube downloader items (`src/downloader/youtube.go` issues) were removed from this list at the owner's request and are intentionally not tracked here. + +**Created:** 2026-10-19 · **Source:** Full-codebase bug review (6 subsystem reviewers + adversarial verification pass, all findings re-checked against source) · **Branch at review:** `fix/album-dedupe` (`46d8f00`) + +## How to use this file across sessions + +1. **Start of session:** Read this file. Pick the highest-severity `open` item, change its Status to `in-progress`, and note the session/date. +2. **Before fixing:** Re-check the cited `file:line` — line numbers drift as code changes. Confirm the bug still exists before editing. +3. **Fix:** Make the smallest fix at the root cause. Add or update a test where the area has committed tests nearby. +4. **Verify:** Run `go vet ./...` and the relevant package tests, e.g. `go test ./src/client/...`. Note: some packages need localhost sockets (`httptest`), which may be blocked in sandboxed environments — record that instead of claiming a pass you did not see. +5. **End of session:** Update the checkbox, Status, PR/commit, and Verification fields for anything you touched — even if unfinished. Leave a one-line Handoff note so the next session can resume without re-investigating. +6. **Do not delete issues.** Move fixed items to `done` with the commit/PR that fixed them. If an issue turns out to be invalid, mark it `wontfix` with a one-line reason. + +Status values: `open` · `in-progress` · `blocked` · `done` · `wontfix` + +## Progress + +| Category | Total | Done | Open | +|---|---|---|---| +| App crashes | 5 | 5 | 0 | +| Data loss and corruption | 3 | 3 | 0 | +| Security | 2 | 2 | 0 | +| Playlists broken or wrong | 7 | 7 | 0 | +| Hangs and resource leaks | 4 | 4 | 0 | +| Wrong songs and metadata | 4 | 4 | 0 | +| Web UI and settings screens | 4 | 4 | 0 | +| Scheduling | 2 | 2 | 0 | +| **Total** | **31** | **31** | **0** | + +Update this table whenever an issue changes status. + +--- + +## App crashes + +- [x] **BUG-01 — Session race crashes the app** · Severity: High + - Location: `src/web/backend/auth/session.go:141` + - User impact: Using the UI in multiple tabs or clicking fast can crash the whole app. + - Detail: `Session.Get/Put/Delete` access `s.data`/`s.id` with no lock (only `Migrate` locks). Concurrent requests sharing one session cause a fatal `concurrent map writes` panic. + - Fix direction: Lock all session access (guard `Get`/`Put`/`Delete` and `writeCookieIfNecessary`'s read of `session.id`). + - Status: done · PR/commit: local branch `fix/bug-01-session-race` (c85aaaf, unpushed) · Verification: new -race regression test passes; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-02 — Lidarr album cache race crashes the app** · Severity: High + - Location: `src/downloader/lidarr.go:353` (also `:519`, panic at `:527`) + - User impact: Fetching several albums at once can crash the app instead of downloading them. + - Detail: `AlbumCache` map is read without `CacheMu` while `cacheAlbum` writes it under lock from concurrent goroutines. At `:519` a nil cache entry is then dereferenced at `:527`. + - Fix direction: `RLock` the reads and add a nil guard before use. + - Status: done · PR/commit: local branch `fix/bug-02-lidarr-cache-race` (88bac9b, unpushed) · Verification: new -race regression test passes; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-03 — Empty artist list panics the run** · Severity: High + - Location: `src/discovery/listenbrainz.go:393` + - User impact: One unusual track from ListenBrainz can crash the whole run so no playlist gets made. + - Detail: `recArtists[0]` is indexed unconditionally; ListenBrainz metadata with an empty/missing artists array panics. + - Fix direction: Guard `len(recArtists)` and degrade gracefully (skip or fall back to a joined/empty artist). + - Status: done · PR/commit: local branch `fix/bug-03-empty-artist-panic` (1b16855, unpushed) · Verification: new unit tests pass; full discovery package passes; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-04 — Config typo panics instead of erroring** · Severity: High + - Location: `src/discovery/discovery.go:21`, dereferenced at `src/main/main.go:198` + - User impact: One typo in your config crashes the app with no hint about what you typed wrong. + - Detail: `NewDiscoverer` returns nil for any `DISCOVERY_SERVICE` other than `listenbrainz`, and no config validation catches it. + - Fix direction: Validate the value in config loading, or return an error from `NewDiscoverer` and fail fast with the bad value in the message. + - Status: done · PR/commit: local branch `fix/bug-04-config-typo-panic` (8c7fbf1, unpushed) · Verification: new unit tests pass; go vet ./... clean; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-05 — Plex library panic** · Severity: Medium + - Location: `src/client/plex.go:381` + - User impact: Plex hiccups while creating a library can crash the run instead of showing an error. + - Detail: `AddLibrary()` indexes `MediaContainer.Library[0]` with no length check when Plex returns an empty container. + - Fix direction: Length-check and return a descriptive error. + - Status: done · PR/commit: local branch `fix/bug-05-plex-library-panic` (1d957aa, unpushed) · Verification: new unit tests pass; go vet + golangci-lint clean + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +## Data loss and corruption + +- [x] **BUG-06 — Custom playlists can be wiped by a race** · Severity: High + - Location: `src/web/backend/playlist/custom_playlists.go:107` + - User impact: Adding or removing playlists at the wrong moment can lose or wipe all your custom playlists. + - Detail: `saveCustomPlaylists` does an unlocked, non-atomic `os.WriteFile` while import/delete/refresh/cron all do unsynchronized load-modify-save. A reader mid-truncate gets a parse error, the loader returns nil, and the next save wipes `custom-playlists.json`. + - Fix direction: Add a mutex around load-modify-save and write via temp file + rename. + - Status: done · PR/commit: local branch `fix/bug-06-custom-playlists-race` (51474f7, unpushed) · Verification: new -race regression tests pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-07 — File move can erase an existing song** · Severity: Medium + - Location: `src/downloader/downloader.go:400` (see also `src/downloader/slskd_album.go:347`) + - User impact: Moving files can silently erase an existing song in your library. + - Detail: `moveFile` truncates the destination with `os.Create` before copying, with no same-file or existing-file check. If `src == dst` (or a sibling name collides with a library file), the source/existing file is zeroed while the track is still reported present. + - Fix direction: Refuse or skip when source and destination are the same file or the destination exists. + - Status: done · PR/commit: local branch `fix/bug-07-file-move-erases-song` (8b4d1c3, unpushed) · Verification: new unit tests pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-08 — Settings saves can be lost or corrupt the file** · Severity: Medium + - Location: `src/web/backend/settings/settings.go:124` + - User impact: Saving settings in quick succession can silently undo some of your changes, and a crash mid-save can corrupt the file. + - Detail: `UpdateEnvKeys` does an unsynchronized read-modify-write with a truncating `os.WriteFile`; concurrent handlers lose each other's updates, and a crash mid-write leaves a truncated `.env`. + - Fix direction: Lock the read-modify-write and write via temp file + rename. Related: BUG-06 (same pattern), BUG-10 (same file). + - Status: done · PR/commit: local branch `fix/bug-08-settings-save-race` (95d8455, unpushed) · Verification: new -race regression test + full settings package pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +## Security + +- [x] **BUG-09 — Settings value can inject extra config keys** · Severity: High + - Location: `src/web/backend/settings/settings.go:134` + - User impact: A value containing a line break can silently overwrite your other settings on the next start. + - Detail: `formatEnvValue` quotes only on `"$#?'`, never newline/CR, and `UpdateEnvKeys` writes `key=value` raw — a value like `x\nAPI_KEY=pwned` injects arbitrary extra `.env` lines. + - Fix direction: Reject or escape CR/LF in keys and values before writing. + - Status: done · PR/commit: local branch `fix/bug-09-env-injection` (3c0cd50, unpushed) · Verification: new unit tests pass; full settings package passes; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-10 — Admin API key shown in plain text** · Severity: Low + - Location: `src/web/frontend/src/lib/utils.js:5` (rendered at `src/web/frontend/src/components/Settings.jsx:888`) + - User impact: An admin API key shows in plain text on the settings screen where it should be hidden. + - Detail: `SENSITIVE_KEY_RE` matches `api_key` but not `APIKEY` (no underscore), so keys such as `ADMIN_SYSTEM_APIKEY` are not masked. + - Fix direction: Add `apikey` to the sensitive-key pattern and audit for other unmasked key spellings. + - Status: done · PR/commit: local branch `fix/bug-10-sensitive-key-mask` (65d43d2, unpushed) · Verification: Node probe over 15 sensitive + 5 plain spellings ALL OK; no JS test runner in repo so no committed test + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +## Playlists broken or wrong + +- [x] **BUG-11 — Playlists gain songs that never arrived** · Severity: High + - Location: `src/downloader/monitor.go:135` (error only logged at `:140`) + - User impact: Playlists can include songs that never actually landed in your library, and they never get retried. + - Detail: The track is marked `Present`, counted as success, and cleaned up *before* `MoveDownload` runs; if the move fails, the playlist still keeps the track. + - Fix direction: Set `Present` and count success only after `MoveDownload` succeeds. + - Status: done · PR/commit: local branch `fix/bug-11-move-before-present` (1287887, unpushed) · Verification: new stub tests pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-12 — One missing song can block a whole Subsonic playlist** · Severity: High + - Location: `src/client/subsonic.go:238` + - User impact: One song that could not be found can stop the entire playlist from being created in Navidrome/Subsonic. + - Detail: `CreatePlaylist` appends `&songId=` for every track without checking `track.Present` or an empty ID (unlike the Jellyfin/Emby formatters), so an absent track sends `&songId=` and the server can reject the whole create. + - Fix direction: Skip tracks that are not present or have an empty ID. + - Status: done · PR/commit: local branch `fix/bug-12-subsonic-missing-song` (a19fa89, unpushed) · Verification: new unit tests pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-13 — Jellyfin playlist creation fails without a username** · Severity: Medium + - Location: `src/client/jellyfin.go:236` + - User impact: Without a username set, playlist creation in Jellyfin always fails. + - Detail: With `SYSTEM_USERNAME` unset, `CreatePlaylist` sends the API key (`resolveAPIKey()`) as the Jellyfin `UserId`, which the server rejects with 400/404. + - Fix direction: Resolve the real user ID from the API, or omit the user where the API allows it; fail with a clear message otherwise. + - Status: done · PR/commit: local branch `fix/bug-13-jellyfin-userid` (39bff7b, unpushed) · Verification: new table tests pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-14 — “Replace playlist” duplicates playlists on Emby** · Severity: Medium + - Location: `src/client/emby.go:241` + - User impact: Choosing “replace playlist” on Emby quietly piles up duplicate playlists instead of replacing. + - Detail: `DeletePlaylist()` is a commented-out no-op returning nil, so callers believe the old playlist was deleted before creating the new one. + - Fix direction: Implement the Emby delete call, or make replace explicitly delete-then-create. + - Status: done · PR/commit: local branch `fix/bug-14-emby-delete-noop` (020c66d, unpushed) · Verification: go vet + golangci-lint clean; httptest delete tests committed — run on CI (sandbox blocks localhost binds) + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-15 — Playlist names with quotes break Jellyfin/Emby** · Severity: Medium + - Location: `src/client/jellyfin.go:243`, `src/client/emby.go:227` + - User impact: Playlist names containing quotes break creation with a server error. + - Detail: Playlist name/description are interpolated unescaped into JSON built with `fmt`, so a quote, backslash, or newline produces invalid JSON and the server returns 400. + - Fix direction: Build the bodies with `json.Marshal` instead of string formatting. + - Status: done · PR/commit: local branch `fix/bug-15-playlist-json-escape` (c27927f, unpushed) · Verification: new round-trip test passes; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-16 — MPD playlists double on every re-run** · Severity: Medium + - Location: `src/client/mpd.go:67` + - User impact: Re-running the same playlist in MPD adds every song again, doubling it each time. + - Detail: The `.m3u` is opened with `O_APPEND`, never truncated, and the handle is never closed. + - Fix direction: Open with `O_TRUNC` (create fresh like the other clients) and close the file. + - Status: done · PR/commit: local branch `fix/bug-16-mpd-append` (decf51d, unpushed) · Verification: new unit test passes; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-17 — Playlist card Run overwrites the wrong playlist** · Severity: Medium + - Location: `src/web/frontend/src/components/Settings.jsx:401` + - User impact: Running a playlist from its card can overwrite a different playlist than the one you clicked. + - Detail: The tracklist Run handler passes `replacePlaylists[playlist]` (the manual-run selector's state) instead of the opened tracklist's own flag. + - Fix direction: Use `openTracklist` when looking up the replace flag. + - Status: done · PR/commit: local branch `fix/bug-17-tracklist-replace-flag` (7045dcb, unpushed) · Verification: vite build passes; no JS test runner in repo so no committed test + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +## Hangs and resource leaks + +- [x] **BUG-18 — A trickling download hangs the run forever** · Severity: Medium + - Location: `src/downloader/monitor.go:161` (deadline check at `:167`) + - User impact: One very slow download can hang the whole run forever instead of timing out. + - Detail: The progress branch `continue`s before the `MaxDuration` check, so any download that moves at least a byte per poll never times out. + - Fix direction: Check the deadline before the progress `continue`, on every poll. + - Status: done · PR/commit: branch `feat/download-provider-fallback` (this commit) · Verification: `go vet ./...` clean; `go test ./src/downloader/ -run TestMonitorDownloads` passes including `-race`; new regression tests `TestMonitorDownloads_TricklingTrackHitsMaxDuration`, `TestMonitorDownloads_FallbackAfterAbandonsUnresolvedTracks`, `TestMonitorDownloads_FallbackAfterZeroWaitsForCompletion`. Full downloader package still fails only on pre-existing sandbox `httptest: bind: operation not permitted` (confirmed identical on untouched `dev`). + - Handoff: Fixed together with the provider-fallback port: the progress branch no longer bypasses `MaxDuration`, and the new `SLSKD_FALLBACK_AFTER`/`LIDARR_FALLBACK_AFTER` whole-provider budget (default 0 = disabled) abandons unresolved tracks (with `Cleanup`) so the next `DOWNLOAD_SERVICES` provider gets a turn. `dfaaedb`'s separate `SLSKD_MONITOR_MAX_RUNTIME` knob was not ported — its intent is now covered by the fixed `MaxDuration` plus `FallbackAfter`, avoiding two near-identical caps. + +- [x] **BUG-19 — One long log line wedges all future runs** · Severity: Medium + - Location: `src/web/backend/run/events.go:48` (wait at `:65`) + - User impact: One very long log line can wedge a run so every later run is refused as “already running”. + - Detail: `collectRunOutput` uses `bufio.Scanner` with the default 64KB line limit; an over-long line stops draining the pipe, the child blocks on write, and `cmd.Wait` never returns. + - Fix direction: Enlarge the scanner buffer and keep draining even after a scan error. + - Status: done · PR/commit: local branch `fix/bug-19-long-log-line` (63d5cf3, unpushed) · Verification: new unit tests + full run package pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-20 — Stalled or huge remote downloads can hang or OOM the app** · Severity: Medium + - Location: `src/util/http.go:94` (also `:106`, cover path at `:137`) + - User impact: One stalled or oversized download can hang the app or eat all its memory. + - Detail: `http.Get`/default client with no timeout, plus unbounded `io.ReadAll`, on remote responses (same pattern in playlist/apple-music fetching). + - Fix direction: Use a timeout client and cap reads with `io.LimitReader`. + - Status: done · PR/commit: local branch `fix/bug-20-http-timeouts` (8ac653e, unpushed) · Verification: new util unit tests pass; playlist package has no tests; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-21 — Rejected runs leak file handles** · Severity: Low + - Location: `src/web/backend/run/manual_run.go:95` + - User impact: Repeatedly trying to start a run while one is running slowly leaks system resources until the app can fail. + - Detail: The already-running branch closes `pw` a second time instead of closing the log file `lf` opened earlier, leaking one descriptor per rejected attempt. + - Fix direction: Close `lf` on that branch. + - Status: done · PR/commit: local branch `fix/bug-21-rejected-run-leak` (6f831aa, unpushed) · Verification: new FD-count regression test + full run package pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +## Wrong songs and metadata + +- [x] **BUG-22 — Every song tagged as track 1, disc 1** · Severity: Medium + - Location: `src/discovery/listenbrainz.go:542` + - User impact: Every song gets tagged as “track 1” of disc 1, scrambling album ordering in your player. + - Detail: Enrichment always takes `Media[0].Tracks[0]`, so track number, disc number, and release-track ID come from the first track of the first medium for every song. + - Fix direction: Select the track matching the recording/release track being enriched. + - Status: done · PR/commit: local branch `fix/bug-22-track-number` (ab7c991, unpushed) · Verification: new unit tests + full discovery package pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-23 — Playlist order is shuffled between runs** · Severity: Medium + - Location: `src/discovery/listenbrainz.go:357` + - User impact: Your playlist order changes randomly between runs instead of following the recommendation ranking. + - Detail: `getTracks` iterates the `Recordings` map, losing the recommendation score order from the API response. + - Fix direction: Iterate the ordered API slice, not the map. + - Status: done · PR/commit: local branch `fix/bug-23-playlist-order` (8228f08, unpushed) · Verification: new unit tests + full discovery package pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-24 — Bad metadata entries for some file types** · Severity: Medium + - Location: `src/util/metadata.go:45` (tag sets at `:99`, `:162`) + - User impact: Some songs fail tagging or get garbled tags depending on file type. + - Detail: Tag helpers check the value but not an empty key, and several tag sets leave fields empty — producing `=value` ffmpeg metadata entries for formats like m4a/wav and ape-family files. + - Fix direction: Skip empty keys (and empty values) when building ffmpeg metadata args. + - Status: done · PR/commit: local branch `fix/bug-24-metadata-empty-keys` (ae5f305, unpushed) · Verification: new unit tests + full util package pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-25 — Failed cover art can fail the song** · Severity: Medium + - Location: `src/util/http.go:136` (caller at `src/downloader/youtube.go:259`) + - User impact: When cover art fails to download the song can fail too, instead of just skipping the artwork. + - Detail: `DownloadCover` returns a destination path even when the fetch or write failed, so callers embed a cover from a file that does not exist and the track save fails. + - Fix direction: Return an error (or empty path) on failure and let callers proceed without cover art. + - Status: done · PR/commit: local branch `fix/bug-25-cover-failure` (5da4767, unpushed) · Verification: new failure-path unit test passes; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +## Web UI and settings screens + +- [x] **BUG-26 — Buttons break after a restart or a week left open (CSRF)** · Severity: Medium + - Location: `src/web/frontend/src/lib/api.js:1` (backend check: `src/web/backend/auth/middleware.go:119`) + - User impact: After a restart or a week left open, every button that changes something fails until you reload the page. + - Detail: The frontend caches the CSRF token forever in a module variable, but the backend mints a new token per session. Sessions are in-memory with a 7-day expiry, so any restart or expiry strands the open page with a stale token and every POST/DELETE returns `403 CSRF token mismatch`. Login/logout do not rotate the token, so they are not a trigger. + - Fix direction: In `apiFetch`, on 403 clear the cached token, re-fetch `/api/ui/csrf`, and retry the request once. + - Status: done · PR/commit: local branch `fix/bug-26-csrf-retry` (52f6bce, unpushed) · Verification: mocked-fetch Node probe ALL OK + vite build; no JS test runner in repo so no committed test + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-27 — Server hiccup leaves a permanent blank page** · Severity: Medium + - Location: `src/web/frontend/src/App.jsx:17`, `src/web/frontend/src/lib/api.js:54` + - User impact: If the server hiccups while the page loads you get a permanent blank page instead of an error or login screen. + - Detail: Startup fetches have no `catch` and `fetchConfig` does not check `res.ok`, so a 500/network failure throws and the view never leaves its initial state. + - Fix direction: Check `res.ok`, catch startup errors, and render an error/retry state. + - Status: done · PR/commit: local branch `fix/bug-27-startup-blank-page` (76d5ea6, unpushed) · Verification: vite build passes; no JS test runner in repo so no committed test + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-28 — Path-template settings page can never load** · Severity: Medium + - Location: `src/web/backend/routes.go:61` + - User impact: The path-template settings screen simply returns “not found” and can never be used. + - Detail: The pattern `api/ui/path-templates` lacks a leading `/`, so `ServeMux` parses `api` as a host pattern and normal requests 404. + - Fix direction: Register `/api/ui/path-templates`. + - Status: done · PR/commit: local branch `fix/bug-28-path-templates-route` (77a2b56, unpushed) · Verification: new mux-pattern test passes; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-29 — Wizard fields look editable but edits are ignored** · Severity: Medium + - Location: `src/web/frontend/src/components/Wizard.jsx:171` (backend values: `src/web/backend/settings/handlers.go:40`) + - User impact: Fields that should be locked by your Docker config look editable, but your edits are silently ignored. + - Detail: Step 2 checks `envSources[key] === '.env'`, but the backend only emits `'env'` or `'file'`, so the lock never engages. + - Fix direction: Compare against `'env'` (as Steps 1 and 3 already do). + - Status: done · PR/commit: local branch `fix/bug-29-wizard-env-lock` (65e2c31, unpushed) · Verification: vite build passes; no JS test runner in repo so no committed test + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +## Scheduling + +- [x] **BUG-30 — Legacy schedule setting disables all scheduling** · Severity: High + - Location: `docker/start.sh:66` + - User impact: If you set the old schedule option, none of your schedules run and music just never arrives. + - Detail: The legacy `CRON_SCHEDULE` branch runs `crond -f` in the foreground, blocking the later per-playlist schedules, `EXECUTE_ON_START`, and the final `crond` invocation. + - Fix direction: Don't run `crond` in the foreground in that branch; register the cron entry and let the normal startup path start the daemon. + - Status: done · PR/commit: local branch `fix/bug-30-legacy-cron-schedule` (526fb4b, unpushed) · Verification: sh -n clean + registration simulation OK; shell script, no committed test + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +- [x] **BUG-31 — Weekly playlists start refreshing daily** · Severity: Medium + - Location: `src/web/backend/playlist/jobs.go:45` + - User impact: A playlist set to refresh weekly starts refreshing every day once it is old enough. + - Detail: The cron closure gates on the `LastFetched` value captured at registration and never reloads it before the check (the fresh load happens only after the gate), so once a playlist is past its refresh interval it refreshes on every daily tick. + - Fix direction: Reload the playlist state before evaluating the refresh gate. + - Status: done · PR/commit: local branch `fix/bug-31-weekly-refresh` (872957f, unpushed) · Verification: new table tests pass; golangci-lint 0 issues + - Handoff: fixed locally, not pushed and no PR opened yet — review the branch, then push + open PR per issue + +--- + +## Session log + +Record one line per working session so progress survives across sessions. + +| Date | Session / author | Issues touched | Outcome / handoff | +|---|---|---|---| +| 2026-10-19 | Bug review | BUG-01 – BUG-31 created | Tracker created; all items open and unverified for fixes. | +| 2026-10-19 | Per-issue local fixes | BUG-01–17, BUG-19–31 fixed | All 30 remaining bugs fixed, each on its own local fix/bug-* branch with its own commit (see BUG entries). Nothing pushed; branches await review, then push + one PR per issue. | +| 2026-10-19 | Provider fallback port | BUG-18 done | Ported `FallbackAfter` from stale `b7122e3` onto `dev` (branch `feat/download-provider-fallback`, PR #25) and fixed BUG-18 deadline bypass. `feat/lidarr`/`feat/provider-fallback` remote branches deleted after the port landed on the new branch. | diff --git a/docker-compose.yaml b/docker-compose.yaml index 569716f4..2f3f5dc5 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -27,6 +27,10 @@ services: # Web UI credentials (optional) #- UI_USERNAME= #- UI_PASSWORD= + # Session cookie name. Defaults to explo_session_ from WEB_ADDR, so + # that instances on one host do not overwrite each other's login. Only set + # this if several instances share a host AND port behind a reverse proxy. + #- UI_COOKIE_NAME= # [Legacy] Schedules are managed through the web UI. # These can still be set here for backwards compatibility — they will take precedence over the UI. diff --git a/docker/explo-run.sh b/docker/explo-run.sh new file mode 100755 index 00000000..d03424a7 --- /dev/null +++ b/docker/explo-run.sh @@ -0,0 +1,46 @@ +#!/bin/sh +# Run one scheduled Explo job under a lock of its own. +# +# crond starts a job on schedule whether or not the last one finished, and a run +# can outlast its own interval -- downloads are slow. Two copies of the same job +# then race on the same playlist. The lock makes the second one stand down. +# +# The lock is per job, not global: a long weekly-exploration must not cancel +# weekly-jams, since skipping a different playlist entirely is worse than +# running the two at once. +# +# Usage: explo-run.sh [args...] + +set -u + +if [ "$#" -lt 2 ]; then + echo "[explo] usage: explo-run.sh [args...]" >&2 + exit 2 +fi + +job="$1" +shift + +lock="${EXPLO_LOCK_DIR:-/tmp}/explo-${job}.lock" + +# Without flock, run unlocked rather than not at all: an overlapping run is a +# lesser problem than a playlist that never gets built. +if ! command -v flock >/dev/null 2>&1; then + echo "[explo] $job: flock unavailable, running without an overlap guard" + exec "$@" +fi + +# Hold the lock on a file descriptor so it is released when this shell exits, +# however it exits. Taking it this way also keeps the exit status unambiguous: +# flock fails here only because the lock is held, never because the job did. +exec 9>"$lock" || { + echo "[explo] $job: couldn't open $lock, running without an overlap guard" + exec "$@" +} + +if ! flock -n 9; then + echo "[explo] $job: previous run still in progress, skipping this schedule" + exit 0 +fi + +exec "$@" diff --git a/docker/explo-run_test.sh b/docker/explo-run_test.sh new file mode 100755 index 00000000..abe844b9 --- /dev/null +++ b/docker/explo-run_test.sh @@ -0,0 +1,86 @@ +#!/bin/sh +# Tests for explo-run.sh. Run it directly: sh docker/explo-run_test.sh +# +# There is no shell test harness in CI (the pipeline runs golangci-lint only), +# so this is here to be run by hand and to document what the lock guarantees. + +set -u + +script="$(dirname "$0")/explo-run.sh" +lockdir="$(mktemp -d)" +export EXPLO_LOCK_DIR="$lockdir" + +failures=0 + +fail() { + echo "FAIL: $1" + failures=$((failures + 1)) +} + +pass() { + echo "ok: $1" +} + +# A job runs normally when nothing holds its lock. +out="$("$script" solo /bin/echo ran)" +if [ "$out" = "ran" ]; then + pass "runs the command when the lock is free" +else + fail "expected 'ran', got '$out'" +fi + +# A second copy of the SAME job stands down instead of racing the first. +"$script" overlap /bin/sleep 2 & +first=$! +sleep 0.3 + +out="$("$script" overlap /bin/echo ran)" +case "$out" in + *"skipping this schedule"*) + pass "a second copy of the same job skips" ;; + *) + fail "expected the second run to skip, got '$out'" ;; +esac + +# Skipping must look like success, or cron mails an error every schedule. +"$script" overlap /bin/echo ran >/dev/null +if [ $? -eq 0 ]; then + pass "a skipped run exits 0" +else + fail "a skipped run exited non-zero" +fi + +# A DIFFERENT job must not be blocked: skipping another playlist entirely is +# worse than two runs overlapping. +out="$("$script" other /bin/echo ran)" +if [ "$out" = "ran" ]; then + pass "a different job is not blocked" +else + fail "a different job was blocked, got '$out'" +fi + +wait "$first" + +# Once the first finishes, the job can run again. +out="$("$script" overlap /bin/echo ran)" +if [ "$out" = "ran" ]; then + pass "the lock is released when the run ends" +else + fail "lock was not released, got '$out'" +fi + +# The command's own exit status must reach cron unchanged. +"$script" status /bin/sh -c 'exit 3' +if [ $? -eq 3 ]; then + pass "the command's exit status is passed through" +else + fail "exit status was not passed through" +fi + +rm -rf "$lockdir" + +if [ "$failures" -ne 0 ]; then + echo "$failures failure(s)" + exit 1 +fi +echo "all passed" diff --git a/docker/start.sh b/docker/start.sh index 9561d63d..0febf346 100644 --- a/docker/start.sh +++ b/docker/start.sh @@ -61,16 +61,23 @@ if [ -f "$_cfg" ]; then fi -# $CRON_SHCEDULE was deprecated in v0.11.0, keeping this block for backwards compatibility +# $CRON_SHCEDULE was deprecated in v0.11.0, keeping this block for backwards compatibility. +# Only register the job here: running `crond -f` in the foreground would block +# everything below (per-playlist schedules, EXECUTE_ON_START, the final crond), +# so legacy users silently lost all scheduling. if [ -n "$CRON_SCHEDULE" ]; then - echo "$CRON_SCHEDULE cd /opt/explo && $RUNNER ./explo --config \"$_cfg\" >> /proc/1/fd/1 2>&1" >> /etc/crontabs/root + echo "$CRON_SCHEDULE cd /opt/explo && /explo-run.sh legacy $RUNNER ./explo --config \"$_cfg\" >> /proc/1/fd/1 2>&1" >> /etc/crontabs/root chmod 600 /etc/crontabs/root echo "[setup] Registered single CRON_SCHEDULE job: $CRON_SCHEDULE" - crond -f -l 2 fi # Loop over all *_SCHEDULE environment variables for var in $(env | grep "_SCHEDULE=" | cut -d= -f1); do + # CRON_SCHEDULE is registered by the legacy block above; registering it + # again here would run the same job twice. + if [ "$var" = "CRON_SCHEDULE" ]; then + continue + fi job="${var%_SCHEDULE}" # Job name (e.g WEEKLY_EXPLORATION) schedule="$(printenv "$var")" # Cron schedule flags_var="${job}_FLAGS" @@ -82,7 +89,9 @@ for var in $(env | grep "_SCHEDULE=" | cut -d= -f1); do fi # Default: just run explo if flags are empty - cmd="cd /opt/explo && $RUNNER ./explo --config \"$_cfg\" $flags >> /proc/1/fd/1 2>&1" + # Wrapped so a run that outlasts its own schedule does not get a second copy + # started on top of it. The lock is per job, so other playlists still run. + cmd="cd /opt/explo && /explo-run.sh \"$job\" $RUNNER ./explo --config \"$_cfg\" $flags >> /proc/1/fd/1 2>&1" echo "$schedule $cmd" >> /etc/crontabs/root echo "[setup] Registered job: $job" diff --git a/sample.env b/sample.env index f6f33587..d66d1209 100644 --- a/sample.env +++ b/sample.env @@ -1,6 +1,10 @@ # === Web UI === # Web cache limit in MB (cover art, backgrounds) (default: 500) # WEB_CACHE_MB=500 +# Kill a run started from the web UI after this many minutes. Without it a +# wedged run keeps the UI refusing new runs until it is stopped by hand. +# 0 disables the timeout. (default: 240) +# RUN_TIMEOUT=240 # === Discovery Config === @@ -52,6 +56,15 @@ LIBRARY_NAME= # KEEP_PERMISSIONS=true # Comma-separated list (no spaces) of download services, in priority order (supported: lidarr,slskd,youtube) (default: youtube) # DOWNLOAD_SERVICES=youtube +# Time budget (minutes) for a single download service before Explo gives up on it +# and lets the next service in DOWNLOAD_SERVICES try the remaining tracks. +# 0 (default) waits indefinitely. Example chain -- prefer full albums from Lidarr, +# fall back to slskd for anything it hasn't delivered in 15 minutes: +# DOWNLOAD_SERVICES=lidarr,slskd +# LIDARR_FALLBACK_AFTER=15 +# SLSKD_FALLBACK_AFTER=10 +# LIDARR_FALLBACK_AFTER=0 +# SLSKD_FALLBACK_AFTER=0 # Path templating, Options are Artist, Album, TrackName, TrackNumber, File, Ext (eg. "{{Artist}}/{{Album}}/{{File}}") # PATH_TEMPLATE="" @@ -85,6 +98,11 @@ LIBRARY_NAME= # SLSKD_MIGRATE_DOWNLOADS=false # Overwrite track metadata when moving file (default: false) # SLSKD_OVERWRITE_METADATA=false +# Download the whole release the matched file belongs to instead of just the +# recommended track. Only the recommended track is added to the playlist; the +# rest simply land in the library. Off by default -- it multiplies transfer +# volume per recommendation. (default: false) +# SLSKD_ALBUM_MODE=false # Directory where slskd downloads tracks (default: /slskd/) # PS! This is only needed on the binary version, in docker it's set through volume mapping # SLSKD_DIR=/slskd/ @@ -101,6 +119,31 @@ LIBRARY_NAME= # MIN_BIT_DEPTH=8 # Minimal Bitrate (default: 256) # MIN_BITRATE=256 +# Highest bitrate in kbps to accept, so a single track cannot arrive as a 138 MB +# WAV. Worked out from file size when a peer reports no bitrate, so it applies +# to files that describe themselves and files that do not. 0 disables the +# ceiling. (default: 0) +# MAX_BITRATE=0 +# Which end of the accepted range to reach for first. "none" leaves the order of +# EXTENSIONS deciding, as it always has. "smaller" takes the best lossy file and +# falls back to lossless only when there is none -- note it still prefers a 320 +# over a 256, since wanting a small file does not mean wanting a bad one. +# "larger" is the reverse. Ranking only; nothing is rejected. (default: none) +# SIZE_PREFERENCE=none +# Which release album mode takes when several match. "smaller" takes the one +# closest to the album's real track count, so a 21-track deluxe edition does +# not arrive in place of the 11-track album -- note this needs +# ENRICH_TRACK_METADATA, since that is where the real track count comes from, +# and falls back to "fuller" without it. "fuller" takes the most complete +# directory, as it always has. (default: smaller) +# RELEASE_PREFERENCE=smaller +# Trim a release down to the original album before downloading it. Extra discs a +# peer has flattened into one directory are dropped (a real double album is kept +# whole when ENRICH_TRACK_METADATA confirms it), and bonus tracks appended past +# the album's real length are dropped after them. The recommended track is always +# kept, even when it is itself a bonus track. Set to false to take releases +# exactly as the peer shares them. (default: true) +# PREFER_ORIGINAL_RELEASE=true # Comma-separated (without spaces) keywords to avoid, when filtering slskd results (default: live,remix,instrumental,extended,clean,acapella) # FILTER_LIST=live,remix,instrumental,extended,clean,acapella diff --git a/src/client/delete_playlist_test.go b/src/client/delete_playlist_test.go new file mode 100644 index 00000000..4cac74f6 --- /dev/null +++ b/src/client/delete_playlist_test.go @@ -0,0 +1,172 @@ +package client + +import ( + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + + "explo/src/config" + "explo/src/util" +) + +// recordingServer reports every request path it receives, so a test can assert +// that no request was made at all. +func recordingServer(t *testing.T, paths *[]string) *httptest.Server { + t.Helper() + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + *paths = append(*paths, r.URL.Path) + if _, err := w.Write([]byte(`{}`)); err != nil { + t.Errorf("fake server failed to write response: %v", err) + } + })) + t.Cleanup(server.Close) + + return server +} + +func testConfig(url, playlistID string) config.ClientConfig { + return config.ClientConfig{ + URL: url, + PlaylistID: playlistID, + PlaylistName: "Weekly Exploration", + Creds: config.Credentials{Headers: map[string]string{}}, + } +} + +// SearchPlaylist leaves PlaylistID empty when it finds nothing, and the URL +// that used to produce was "/playlists/" -- a delete aimed at the collection +// rather than at one playlist. Plex answered 403, which surfaced as a warning +// with a notification attached for what is really a no-op. +func TestPlexDeletePlaylist_SendsNothingWithoutAnID(t *testing.T) { + var paths []string + server := recordingServer(t, &paths) + + plex := NewPlex(testConfig(server.URL, ""), util.NewHttp(util.HttpClientConfig{Timeout: 5})) + + if err := plex.DeletePlaylist(); err != nil { + t.Errorf("DeletePlaylist() = %v, want nil -- nothing to delete is not a failure", err) + } + if len(paths) != 0 { + t.Errorf("requested %v, want no request at all", paths) + } +} + +// With an ID it must still delete that one playlist. +func TestPlexDeletePlaylist_DeletesTheFoundPlaylist(t *testing.T) { + var paths []string + server := recordingServer(t, &paths) + + plex := NewPlex(testConfig(server.URL, "12345"), util.NewHttp(util.HttpClientConfig{Timeout: 5})) + + if err := plex.DeletePlaylist(); err != nil { + t.Fatalf("DeletePlaylist() = %v, want nil", err) + } + if len(paths) != 1 || paths[0] != "/playlists/12345" { + t.Errorf("requested %v, want a single DELETE of /playlists/12345", paths) + } +} + +// Jellyfin has the same hole, and its bare URL is "/Items" -- every item on the +// server rather than one playlist. +func TestJellyfinDeletePlaylist_SendsNothingWithoutAnID(t *testing.T) { + var paths []string + server := recordingServer(t, &paths) + + jf := NewJellyfin(testConfig(server.URL, ""), util.NewHttp(util.HttpClientConfig{Timeout: 5})) + + if err := jf.DeletePlaylist(); err != nil { + t.Errorf("DeletePlaylist() = %v, want nil", err) + } + if len(paths) != 0 { + t.Errorf("requested %v, want no request at all", paths) + } +} + +func TestJellyfinDeletePlaylist_DeletesTheFoundPlaylist(t *testing.T) { + var paths []string + server := recordingServer(t, &paths) + + jf := NewJellyfin(testConfig(server.URL, "abc"), util.NewHttp(util.HttpClientConfig{Timeout: 5})) + + if err := jf.DeletePlaylist(); err != nil { + t.Fatalf("DeletePlaylist() = %v, want nil", err) + } + if len(paths) != 1 || paths[0] != "/Items/abc" { + t.Errorf("requested %v, want a single DELETE of /Items/abc", paths) + } +} + +// Emby used to delete nothing and report success, so "replace playlist" +// left the old playlist in place and created a duplicate. +func TestEmbyDeletePlaylist_DeletesTheFoundPlaylist(t *testing.T) { + var paths []string + server := recordingServer(t, &paths) + + emby := NewEmby(testConfig(server.URL, "abc"), util.NewHttp(util.HttpClientConfig{Timeout: 5})) + + if err := emby.DeletePlaylist(); err != nil { + t.Fatalf("DeletePlaylist() = %v, want nil", err) + } + if len(paths) != 1 || paths[0] != "/emby/Items/abc" { + t.Errorf("requested %v, want a single DELETE of /emby/Items/abc", paths) + } +} + +func TestEmbyDeletePlaylist_SendsNothingWithoutAnID(t *testing.T) { + var paths []string + server := recordingServer(t, &paths) + + emby := NewEmby(testConfig(server.URL, ""), util.NewHttp(util.HttpClientConfig{Timeout: 5})) + + if err := emby.DeletePlaylist(); err != nil { + t.Errorf("DeletePlaylist() = %v, want nil when there is no playlist", err) + } + if len(paths) != 0 { + t.Errorf("requested %v, want no request at all", paths) + } +} + +// Subsonic would send deletePlaylist with an empty id. +func TestSubsonicDeletePlaylist_SendsNothingWithoutAnID(t *testing.T) { + var paths []string + server := recordingServer(t, &paths) + + sub := NewSubsonic(testConfig(server.URL, ""), util.NewHttp(util.HttpClientConfig{Timeout: 5})) + + if err := sub.DeletePlaylist(); err != nil { + t.Errorf("DeletePlaylist() = %v, want nil", err) + } + if len(paths) != 0 { + t.Errorf("requested %v, want no request at all", paths) + } +} + +// MPD already skipped the delete, but reported it as an error, which reached +// the user as the same spurious warning. +func TestMPDDeletePlaylist_MissingPlaylistIsNotAnError(t *testing.T) { + mpd := NewMPD(testConfig("", "")) + + if err := mpd.DeletePlaylist(); err != nil { + t.Errorf("DeletePlaylist() = %v, want nil when there is no playlist", err) + } +} + +// A playlist that is there still gets removed. +func TestMPDDeletePlaylist_RemovesTheFile(t *testing.T) { + path := filepath.Join(t.TempDir(), "weekly.m3u") + if err := os.WriteFile(path, []byte("#EXTM3U\n"), 0o644); err != nil { + t.Fatalf("seeding playlist: %v", err) + } + + mpd := NewMPD(testConfig("", path)) + + if err := mpd.DeletePlaylist(); err != nil { + t.Fatalf("DeletePlaylist() = %v, want nil", err) + } + if _, err := os.Stat(path); !os.IsNotExist(err) { + t.Errorf("playlist file still present: %v", err) + } +} diff --git a/src/client/emby.go b/src/client/emby.go index e5267ada..b95722bb 100644 --- a/src/client/emby.go +++ b/src/client/emby.go @@ -2,6 +2,7 @@ package client import ( "bytes" + "encoding/json" "fmt" "log/slog" "strings" @@ -220,17 +221,26 @@ func (c *Emby) CreatePlaylist(tracks []*models.Track) error { return nil } +// Same class as the Jellyfin bodies: marshal so quotes, backslashes and +// newlines in playlist names cannot break the JSON (BUG-15). +func embyUpdatePlaylistBody(id, name, overview string) ([]byte, error) { + return json.Marshal(struct { + ID string `json:"Id"` + Name string `json:"Name"` + Overview string `json:"Overview"` + ProviderIDs map[string]string `json:"ProviderIds"` + }{id, name, overview, map[string]string{}}) +} + func (c *Emby) UpdatePlaylist() error { time.Sleep(5 * time.Second) // small buffer between playlist creation and updating, Emby doesn't update playlist otherwise reqParam := fmt.Sprintf("/emby/Items/%s", c.Cfg.PlaylistID) - payload := fmt.Appendf(nil, ` - { - "Id": "%s", - "Name": "%s", - "Overview": "%s", - "ProviderIds": {} - }`, c.Cfg.PlaylistID, c.Cfg.PlaylistName, c.Cfg.PlaylistDescr) // the additional field has to be added, otherwise Emby returns code 500 + // the additional field has to be added, otherwise Emby returns code 500 + payload, err := embyUpdatePlaylistBody(c.Cfg.PlaylistID, c.Cfg.PlaylistName, c.Cfg.PlaylistDescr) + if err != nil { + return fmt.Errorf("failed to marshal playlist body: %s", err.Error()) + } if _, err := c.HttpClient.MakeRequest("POST", c.Cfg.URL+reqParam, bytes.NewBuffer(payload), c.Cfg.Creds.Headers); err != nil { return err @@ -238,12 +248,22 @@ func (c *Emby) UpdatePlaylist() error { return nil } -func (c *Emby) DeletePlaylist() error { // Doesn't currently work due to a bug in Emby - /* reqParam := fmt.Sprintf("/emby/Items/Delete?Ids=%s", c.Cfg.PlaylistID) +func (c *Emby) DeletePlaylist() error { + // Without an ID this would address the collection rather than one + // playlist (see the Jellyfin guard). + if c.Cfg.PlaylistID == "" { + slog.Debug("no emby playlist to delete", "playlist", c.Cfg.PlaylistName) + return nil + } + + // This used to be a no-op returning nil, so callers that + // delete-then-create for "replace playlist" believed the old playlist + // was gone and silently piled up duplicates. + reqParam := fmt.Sprintf("/emby/Items/%s", c.Cfg.PlaylistID) - if _, err := util.MakeRequest("POST", c.Cfg.URL+reqParam, nil, c.Cfg.Creds.Headers); err != nil { - return err - } */ + if _, err := c.HttpClient.MakeRequest("DELETE", c.Cfg.URL+reqParam, nil, c.Cfg.Creds.Headers); err != nil { + return fmt.Errorf("deleteEmbyPlaylist(): %s", err.Error()) + } return nil } diff --git a/src/client/jellyfin.go b/src/client/jellyfin.go index ab4100c3..d760350c 100644 --- a/src/client/jellyfin.go +++ b/src/client/jellyfin.go @@ -225,29 +225,24 @@ func (c *Jellyfin) CreatePlaylist(tracks []*models.Track) error { if err != nil { return fmt.Errorf("failed to marshal track IDs: %s", err.Error()) } - var userID string isPublic := c.Cfg.PublicPlaylist - - if c.Cfg.Creds.User != "" { - userID, err = c.ResolveUserID() - if err != nil { - return err - } - } else { - userID = c.resolveAPIKey() + if c.Cfg.Creds.User == "" { isPublic = true } + // The API key is not a user ID: sending it as UserId made playlist + // creation fail whenever SYSTEM_USERNAME was unset (BUG-13). + userID, err := c.ResolveUserID() + if err != nil { + return err + } + queryParams := "/Playlists" - payload := fmt.Appendf(nil, ` - { - "Name": "%s", - "Ids": %s, - "MediaType": "Audio", - "UserId": "%s", - "IsPublic": %t - }`, c.Cfg.PlaylistName, songs, userID, isPublic) + payload, err := jfCreatePlaylistBody(c.Cfg.PlaylistName, songs, userID, isPublic) + if err != nil { + return fmt.Errorf("failed to marshal playlist body: %s", err.Error()) + } body, err := c.HttpClient.MakeRequest("POST", c.Cfg.URL+queryParams, bytes.NewReader(payload), c.Cfg.Creds.Headers) if err != nil { @@ -261,22 +256,42 @@ func (c *Jellyfin) CreatePlaylist(tracks []*models.Track) error { return nil } +// Playlist names and descriptions are user input: quoting a quote, +// backslash or newline into fmt-built JSON produced invalid bodies and +// the server answered 400 (BUG-15). Marshal instead. +func jfCreatePlaylistBody(name string, songs json.RawMessage, userID string, isPublic bool) ([]byte, error) { + return json.Marshal(struct { + Name string `json:"Name"` + IDs json.RawMessage `json:"Ids"` + MediaType string `json:"MediaType"` + UserID string `json:"UserId"` + IsPublic bool `json:"IsPublic"` + }{name, songs, "Audio", userID, isPublic}) +} + +func jfUpdatePlaylistBody(id, name, overview string, isPublic bool) ([]byte, error) { + return json.Marshal(struct { + ID string `json:"Id"` + Name string `json:"Name"` + Overview string `json:"Overview"` + IsPublic bool `json:"IsPublic"` + Genres []string `json:"Genres"` + Tags []string `json:"Tags"` + ProviderIDs map[string]string `json:"ProviderIds"` + }{id, name, overview, isPublic, []string{}, []string{}, map[string]string{}}) +} + func (c *Jellyfin) UpdatePlaylist() error { isPublic := c.Cfg.PublicPlaylist if c.Cfg.Creds.User == "" { isPublic = true } queryParams := fmt.Sprintf("/Items/%s", c.Cfg.PlaylistID) - payload := fmt.Appendf(nil, ` - { - "Id":"%s", - "Name":"%s", - "Overview":"%s", - "IsPublic":%t, - "Genres":[], - "Tags":[], - "ProviderIds":{} - }`, c.Cfg.PlaylistID, c.Cfg.PlaylistName, c.Cfg.PlaylistDescr, isPublic) // the additional fields have to be added, otherwise JF returns code 400 + // the additional fields have to be added, otherwise JF returns code 400 + payload, err := jfUpdatePlaylistBody(c.Cfg.PlaylistID, c.Cfg.PlaylistName, c.Cfg.PlaylistDescr, isPublic) + if err != nil { + return fmt.Errorf("failed to marshal playlist body: %s", err.Error()) + } if _, err := c.HttpClient.MakeRequest("POST", c.Cfg.URL+queryParams, bytes.NewBuffer(payload), c.Cfg.Creds.Headers); err != nil { return err @@ -285,6 +300,13 @@ func (c *Jellyfin) UpdatePlaylist() error { } func (c *Jellyfin) DeletePlaylist() error { + // Without an ID this would DELETE /Items, which is every item rather than + // one playlist. + if c.Cfg.PlaylistID == "" { + slog.Debug("no jellyfin playlist to delete", "playlist", c.Cfg.PlaylistName) + return nil + } + queryParams := fmt.Sprintf("/Items/%s", c.Cfg.PlaylistID) if _, err := c.HttpClient.MakeRequest("DELETE", c.Cfg.URL+queryParams, nil, c.Cfg.Creds.Headers); err != nil { @@ -324,13 +346,29 @@ func (c *Jellyfin) ResolveUserID() (string, error) { if err = util.ParseResp(body, &users); err != nil { return "", err } - for _, user := range users { - if strings.EqualFold(user.Name, c.Cfg.Creds.User) { - return user.ID, nil + return pickUserID(users, c.Cfg.Creds.User) +} + +// pickUserID selects the playlist owner. With no configured username it +// falls back to the server's only user; with several users and no name +// configured there is no safe guess, so it fails with guidance instead +// of sending the API key as a user ID. +func pickUserID(users []JFUser, username string) (string, error) { + if username != "" { + for _, user := range users { + if strings.EqualFold(user.Name, username) { + return user.ID, nil + } } + return "", fmt.Errorf("failed to find Jellyfin user %q", username) } - - return "", fmt.Errorf("failed to find Jellyfin user %q", c.Cfg.Creds.User) + if len(users) == 1 { + return users[0].ID, nil + } + if len(users) == 0 { + return "", fmt.Errorf("jellyfin returned no users; set SYSTEM_USERNAME to the playlist owner's username") + } + return "", fmt.Errorf("jellyfin has %d users and SYSTEM_USERNAME is not set; set SYSTEM_USERNAME to the playlist owner's username", len(users)) } // Check which API Key variable is used diff --git a/src/client/jellyfin_user_test.go b/src/client/jellyfin_user_test.go new file mode 100644 index 00000000..1d79cf1e --- /dev/null +++ b/src/client/jellyfin_user_test.go @@ -0,0 +1,41 @@ +package client + +import ( + "strings" + "testing" +) + +// BUG-13: with SYSTEM_USERNAME unset, CreatePlaylist sent the API key +// as the Jellyfin UserId and creation always failed. +func TestPickUserID(t *testing.T) { + users := []JFUser{{ID: "1", Name: "Alice"}, {ID: "2", Name: "Bob"}} + + tests := []struct { + name string + users []JFUser + username string + wantID string + wantErr string + }{ + {"named user, case-insensitive", users, "alice", "1", ""}, + {"named user missing", users, "Carol", "", "Carol"}, + {"no username, single user", users[:1], "", "1", ""}, + {"no username, many users", users, "", "", "SYSTEM_USERNAME"}, + {"no username, no users", nil, "", "", "SYSTEM_USERNAME"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + id, err := pickUserID(tt.users, tt.username) + if tt.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Fatalf("pickUserID = %q, %v; want an error mentioning %q", id, err, tt.wantErr) + } + return + } + if err != nil || id != tt.wantID { + t.Errorf("pickUserID = %q, %v; want %q, nil", id, err, tt.wantID) + } + }) + } +} diff --git a/src/client/mpd.go b/src/client/mpd.go index 4629c506..62714bd7 100644 --- a/src/client/mpd.go +++ b/src/client/mpd.go @@ -64,10 +64,17 @@ func (c *MPD) CheckRefreshState() bool { } func (c *MPD) CreatePlaylist(tracks []*models.Track) error { - f, err := os.OpenFile(c.Cfg.PlaylistDir+c.Cfg.PlaylistName+".m3u", os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0666) + // O_APPEND doubled the playlist on every re-run and the handle was + // never closed. Start fresh each time like the other clients. + f, err := os.OpenFile(c.Cfg.PlaylistDir+c.Cfg.PlaylistName+".m3u", os.O_TRUNC|os.O_WRONLY|os.O_CREATE, 0666) if err != nil { return err } + defer func() { + if err := f.Close(); err != nil { + slog.Warn("failed to close playlist file", "err", err.Error()) + } + }() for _, track := range tracks { if track.Present { @@ -100,7 +107,10 @@ func (c *MPD) DeletePlaylist() error { } return nil } - return fmt.Errorf("playlist not found") + // Nothing to delete is not a failure; the caller would only log a warning + // about a playlist that was never there. + slog.Debug("no mpd playlist to delete", "playlist", c.Cfg.PlaylistName) + return nil } func (c MPD) findTrack(name, path string) (string, error) { diff --git a/src/client/mpd_playlist_test.go b/src/client/mpd_playlist_test.go new file mode 100644 index 00000000..4e3b9505 --- /dev/null +++ b/src/client/mpd_playlist_test.go @@ -0,0 +1,39 @@ +package client + +import ( + "os" + "path/filepath" + "testing" + + "explo/src/models" +) + +// BUG-16: the .m3u was opened with O_APPEND and the handle never +// closed, so every re-run doubled the playlist. +func TestMPDCreatePlaylist_ReRunDoesNotDuplicate(t *testing.T) { + dir := t.TempDir() + cfg := testConfig("", "") + cfg.PlaylistDir = dir + string(os.PathSeparator) + cfg.PlaylistName = "weekly" + mpd := NewMPD(cfg) + + tracks := []*models.Track{ + {File: "a.flac", Present: true}, + {File: "b.flac", Present: true}, + {File: "missing.flac", Present: false}, + } + + for i := 0; i < 2; i++ { + if err := mpd.CreatePlaylist(tracks); err != nil { + t.Fatalf("CreatePlaylist run %d: %v", i, err) + } + } + + data, err := os.ReadFile(filepath.Join(dir, "weekly.m3u")) + if err != nil { + t.Fatal(err) + } + if string(data) != "a.flac\nb.flac\n" { + t.Errorf("playlist = %q, want each present track exactly once", data) + } +} diff --git a/src/client/playlist_body_test.go b/src/client/playlist_body_test.go new file mode 100644 index 00000000..fc3232e2 --- /dev/null +++ b/src/client/playlist_body_test.go @@ -0,0 +1,67 @@ +package client + +import ( + "encoding/json" + "testing" +) + +// BUG-15: playlist name/description were interpolated with fmt into +// hand-built JSON, so a quote, backslash or newline produced invalid +// JSON and the server answered 400. +func TestPlaylistBodies_HostileNamesRoundTrip(t *testing.T) { + name := `My "Best"\Playlist` + "\nHits" + descr := "line1\nline2 \"quoted\" \\ backslash" + + songs := json.RawMessage(`["a","b"]`) + + create, err := jfCreatePlaylistBody(name, songs, "user-1", true) + if err != nil { + t.Fatalf("jfCreatePlaylistBody: %v", err) + } + var decoded struct { + Name string `json:"Name"` + IDs json.RawMessage `json:"Ids"` + User string `json:"UserId"` + Media string `json:"MediaType"` + } + if err := json.Unmarshal(create, &decoded); err != nil { + t.Fatalf("create body is not valid JSON: %v\n%s", err, create) + } + if decoded.Name != name || decoded.User != "user-1" || decoded.Media != "Audio" || string(decoded.IDs) != string(songs) { + t.Errorf("create body = %s, want name/ids/user/media preserved", create) + } + + update, err := jfUpdatePlaylistBody("5", name, descr, false) + if err != nil { + t.Fatalf("jfUpdatePlaylistBody: %v", err) + } + var decodedUpdate struct { + ID string `json:"Id"` + Name string `json:"Name"` + Overview string `json:"Overview"` + Genres []string `json:"Genres"` + Tags []string `json:"Tags"` + } + if err := json.Unmarshal(update, &decodedUpdate); err != nil { + t.Fatalf("update body is not valid JSON: %v\n%s", err, update) + } + if decodedUpdate.Name != name || decodedUpdate.Overview != descr || decodedUpdate.Genres == nil || decodedUpdate.Tags == nil { + t.Errorf("update body = %s, want fields preserved with empty arrays, not null", update) + } + + emby, err := embyUpdatePlaylistBody("9", name, descr) + if err != nil { + t.Fatalf("embyUpdatePlaylistBody: %v", err) + } + var decodedEmby struct { + ID string `json:"Id"` + Name string `json:"Name"` + Overview string `json:"Overview"` + } + if err := json.Unmarshal(emby, &decodedEmby); err != nil { + t.Fatalf("emby body is not valid JSON: %v\n%s", err, emby) + } + if decodedEmby.Name != name || decodedEmby.Overview != descr { + t.Errorf("emby body = %s, want name/overview preserved", emby) + } +} diff --git a/src/client/plex.go b/src/client/plex.go index af1740a7..c77738fe 100644 --- a/src/client/plex.go +++ b/src/client/plex.go @@ -250,7 +250,15 @@ func (c *Plex) AddHeader() error { c.Cfg.Creds.Headers = make(map[string]string) c.Cfg.Creds.Headers["X-Plex-Client-Identifier"] = c.Cfg.ClientID - return nil + // Only bail out here when there is no key to install yet. The plex + // setup path calls AddHeader() again after GetAuth() fills in the + // token, but the admin client built in GetLibrary() gets a fresh + // Credentials{APIKey: ...} with a nil Headers map and is only called + // once -- returning early there left it with no X-Plex-Token, so + // every admin request 401'd. + if c.Cfg.Creds.APIKey == "" { + return nil + } } if c.Cfg.Creds.APIKey != "" { c.Cfg.Creds.Headers["X-Plex-Token"] = c.Cfg.Creds.APIKey @@ -370,9 +378,23 @@ func (c *Plex) AddLibrary() error { if err = util.ParseResp(body, &libraries); err != nil { return err } - c.LibraryID = libraries.MediaContainer.Library[0].Key + key, err := firstLibraryKey(libraries) + if err != nil { + return err + } + c.LibraryID = key return nil } + +// firstLibraryKey returns the created library's key. Plex can answer a +// library creation with an empty container (a hiccup, a race with another +// client); indexing [0] without a check panicked the run. +func firstLibraryKey(libraries Libraries) (string, error) { + if len(libraries.MediaContainer.Library) == 0 { + return "", fmt.Errorf("plex returned no library after creating one; check the Plex server and the configured library name and download directory") + } + return libraries.MediaContainer.Library[0].Key, nil +} func (c *Plex) RefreshLibrary() error { if c.AdminClient != nil { return c.AdminClient.refreshLibraryRequest() @@ -543,6 +565,14 @@ func (c *Plex) UpdatePlaylist() error { } func (c *Plex) DeletePlaylist() error { + // Without an ID this would DELETE the /playlists collection itself. Plex + // answers 403, but a delete aimed at every playlist is not a request worth + // making and finding nothing to delete is not a failure. + if c.Cfg.PlaylistID == "" { + slog.Debug("no plex playlist to delete", "playlist", c.Cfg.PlaylistName) + return nil + } + params := fmt.Sprintf("/playlists/%s", c.Cfg.PlaylistID) if _, err := c.HttpClient.MakeRequest("DELETE", c.Cfg.URL+params, nil, c.Cfg.Creds.Headers); err != nil { diff --git a/src/client/plex_library_test.go b/src/client/plex_library_test.go new file mode 100644 index 00000000..d5d6d1de --- /dev/null +++ b/src/client/plex_library_test.go @@ -0,0 +1,32 @@ +package client + +import ( + "encoding/json" + "testing" +) + +// BUG-05: Plex returning an empty MediaContainer after library creation +// panicked on Library[0]. +func TestFirstLibraryKey_EmptyContainerErrors(t *testing.T) { + var libs Libraries + if err := json.Unmarshal([]byte(`{"MediaContainer":{"size":0}}`), &libs); err != nil { + t.Fatal(err) + } + if _, err := firstLibraryKey(libs); err == nil { + t.Fatal("firstLibraryKey(empty) = nil error, want a descriptive failure instead of a panic") + } +} + +func TestFirstLibraryKey_ReturnsKey(t *testing.T) { + var libs Libraries + if err := json.Unmarshal([]byte(`{"MediaContainer":{"Directory":[{"title":"Music","key":"7"}]}}`), &libs); err != nil { + t.Fatal(err) + } + key, err := firstLibraryKey(libs) + if err != nil { + t.Fatalf("firstLibraryKey: %v", err) + } + if key != "7" { + t.Errorf("key = %q, want 7", key) + } +} diff --git a/src/client/subsonic.go b/src/client/subsonic.go index 04073a1f..310a9bd4 100644 --- a/src/client/subsonic.go +++ b/src/client/subsonic.go @@ -234,10 +234,25 @@ func (c *Subsonic) CheckRefreshState() bool { } } +// subsonicSongIDs returns the server song IDs to put in a playlist. +// Tracks that were never matched keep their discovery ID (or none) and +// Present=false; sending those as songId made the server reject the +// whole playlist creation over one missing song (BUG-12). +func subsonicSongIDs(tracks []*models.Track) []string { + ids := make([]string, 0, len(tracks)) + for _, track := range tracks { + if !track.Present || track.ID == "" { + continue + } + ids = append(ids, track.ID) + } + return ids +} + func (c *Subsonic) CreatePlaylist(tracks []*models.Track) error { var trackIDs strings.Builder - for _, track := range tracks { // build songID parameters - fmt.Fprintf(&trackIDs, "&songId=%s", track.ID) + for _, id := range subsonicSongIDs(tracks) { // build songID parameters + fmt.Fprintf(&trackIDs, "&songId=%s", id) } reqParam := fmt.Sprintf("createPlaylist?name=%s%s&f=json", url.QueryEscape(c.Cfg.PlaylistName), trackIDs.String()) @@ -289,6 +304,11 @@ func (c *Subsonic) UpdatePlaylist() error { } func (c *Subsonic) DeletePlaylist() error { + if c.Cfg.PlaylistID == "" { + slog.Debug("no subsonic playlist to delete", "playlist", c.Cfg.PlaylistName) + return nil + } + reqParam := fmt.Sprintf("deletePlaylist?id=%s&f=json", c.Cfg.PlaylistID) if _, err := c.subsonicRequest(reqParam); err != nil { diff --git a/src/client/subsonic_playlist_test.go b/src/client/subsonic_playlist_test.go new file mode 100644 index 00000000..3a8dccbb --- /dev/null +++ b/src/client/subsonic_playlist_test.go @@ -0,0 +1,33 @@ +package client + +import ( + "reflect" + "testing" + + "explo/src/models" +) + +// BUG-12: CreatePlaylist sent every track's ID as songId, including +// tracks never matched in Subsonic (Present=false, discovery ID or +// empty), and the server rejected the whole playlist over one song. +func TestSubsonicSongIDs_SkipsUnmatchedTracks(t *testing.T) { + tracks := []*models.Track{ + {ID: "song-1", Present: true}, + {ID: "mbid-not-a-song-id", Present: false}, + {ID: "", Present: true}, + {ID: "song-2", Present: true}, + } + + got := subsonicSongIDs(tracks) + want := []string{"song-1", "song-2"} + if !reflect.DeepEqual(got, want) { + t.Errorf("subsonicSongIDs = %v, want %v", got, want) + } +} + +func TestSubsonicSongIDs_NoneMatched(t *testing.T) { + tracks := []*models.Track{{ID: "x"}, {ID: ""}} + if got := subsonicSongIDs(tracks); len(got) != 0 { + t.Errorf("subsonicSongIDs = %v, want none", got) + } +} diff --git a/src/config/config.go b/src/config/config.go index 2205cdc1..86955478 100644 --- a/src/config/config.go +++ b/src/config/config.go @@ -48,9 +48,17 @@ type ServerConfig struct { Port string `env:"WEB_ADDR" env-default:":7288"` Username string `env:"UI_USERNAME"` Password string `env:"UI_PASSWORD"` + // Name of the session cookie. Empty means derive it from Port, so that + // instances on one host do not overwrite each other's session. Set it + // explicitly when several instances share a host and port behind a proxy. + CookieName string `env:"UI_COOKIE_NAME"` WebDataDir string `env:"WEB_DATA_PATH" env-default:"/opt/explo/config/"` WebEnvPath string `env:"WEB_ENV_PATH" env-default:"/opt/explo/.env"` CacheSizeMB int64 `env:"WEB_CACHE_MB" env-default:"500"` + // Kill a run started from the web UI after this many minutes. Without it a + // wedged run holds the "already running" guard until someone stops it by + // hand. 0 disables the timeout. + RunTimeout int `env:"RUN_TIMEOUT" env-default:"240"` ExploPath string } @@ -114,6 +122,10 @@ type Filters struct { Extensions []string `env:"EXTENSIONS" env-default:"flac,mp3"` // slskd MinBitDepth int `env:"MIN_BIT_DEPTH" env-default:"8"` MinBitRate int `env:"MIN_BITRATE" env-default:"256"` + // Ceiling in kbps. Zero disables it, so nothing changes on upgrade until + // someone sets one. Compared against a bitrate derived from file size when + // the peer reports none. + MaxBitRate int `env:"MAX_BITRATE" env-default:"0"` FilterList []string `env:"FILTER_LIST" env-default:"live,remix,instrumental,extended,clean,acapella"` } @@ -142,11 +154,30 @@ type Slskd struct { DownloadAttempts int `env:"SLSKD_DL_ATTEMPTS" env-default:"3"` // Max number of files to attempt downloading per track SlskdDir string `env:"SLSKD_DIR" env-default:"/slskd/"` MigrateDL bool `env:"SLSKD_MIGRATE_DOWNLOADS" env-default:"false"` // Move downloads from SlskdDir to DownloadDir - MigrateDLOld bool `env:"MIGRATE_DOWNLOADS" env-default:"false"` // Move downloads from SlskdDir to DownloadDir (Old) + MigrateDLOld bool `env:"MIGRATE_DOWNLOADS" env-default:"false"` // Move downloads from SlskdDir to DownloadDir (Old) RenameTrack bool `env:"RENAME_TRACK" env-default:"false"` // Rename track in {title}-{artist} format (will be deprecated) - PathTemplate string `env:"PATH_TEMPLATE"` + PathTemplate string `env:"PATH_TEMPLATE"` OverwriteMetadata bool `env:"SLSKD_OVERWRITE_METADATA" env-default:"false"` // overwrite metadata when migrating downloads KeepPermissions bool `env:"SLSKD_KEEP_PERMISSIONS" env-default:"true"` // keep original file permissions when migrating download + // Which end of the accepted quality range to reach for first: "none" leaves + // EXTENSIONS order deciding, as it always has; "smaller" takes the best + // lossy file and falls back to lossless only when there is none; "larger" + // is the reverse. + SizePreference string `env:"SIZE_PREFERENCE" env-default:"none"` + // Which release to reach for in album mode: "smaller" takes the one + // closest to the album's real track count, which needs + // ENRICH_TRACK_METADATA to know it; "fuller" takes the most complete + // one, as it always has. Unknown counts fall back to fuller either way. + ReleasePreference string `env:"RELEASE_PREFERENCE" env-default:"smaller"` + // Trim a candidate release down to the original album before downloading it: + // drop extra discs a peer has flattened into one directory, and drop bonus + // tracks appended past the album's real length. Needs ENRICH_TRACK_METADATA + // for the bonus-track half; disc trimming works without it. + PreferOriginalRelease bool `env:"PREFER_ORIGINAL_RELEASE" env-default:"true"` + // Download every track in the release the matched file sits in, rather + // than the single recommended track. Off by default: it multiplies how + // much is transferred per recommendation. + AlbumMode bool `env:"SLSKD_ALBUM_MODE" env-default:"false"` Timeout int `env:"SLSKD_TIMEOUT" env-default:"20"` Filters Filters MonitorConfig SlskdMon @@ -157,6 +188,9 @@ type SlskdMon struct { Interval int `env:"SLSKD_MONITOR_INTERVAL" env-default:"1"` // in minutes StallDuration int `env:"SLSKD_STALL_DURATION" env-default:"15"` // in minutes MaxDuration int `env:"SLSKD_MONITOR_MAX_DURATION" env-default:"120"` // in minutes + // Give up on this provider entirely after this long and let the next + // service in DOWNLOAD_SERVICES try the remaining tracks. 0 disables. + FallbackAfter int `env:"SLSKD_FALLBACK_AFTER" env-default:"0"` // in minutes } type Lidarr struct { @@ -177,6 +211,9 @@ type LidarrMon struct { Interval int `env:"LIDARR_MONITOR_INTERVAL" env-default:"1"` // in minutes StallDuration int `env:"LIDARR_STALL_DURATION" env-default:"20"` // in minutes MaxDuration int `env:"LIDARR_MONITOR_MAX_DURATION" env-default:"120"` // in minutes + // Give up on this provider entirely after this long and let the next + // service in DOWNLOAD_SERVICES try the remaining tracks. 0 disables. + FallbackAfter int `env:"LIDARR_FALLBACK_AFTER" env-default:"0"` // in minutes } type DiscoveryConfig struct { diff --git a/src/discovery/canonical.go b/src/discovery/canonical.go new file mode 100644 index 00000000..fb2fce4f --- /dev/null +++ b/src/discovery/canonical.go @@ -0,0 +1,98 @@ +package discovery + +import "strings" + +// The album's shape as its release group agrees it is. +// +// A recommendation carries whichever release MusicBrainz matched, which is not +// necessarily the album anyone thinks of as the album: a track resolving to a +// two-disc deluxe edition reports two discs and the deluxe's length, and album +// mode then treats a padded directory as correct. Asking the rest of the group +// what shape the album usually has answers that far more reliably than trusting +// the single matched release. +// +// This costs nothing extra. The enrichment request already asks for +// inc=media+releases and parses every release; until now all but one were +// discarded. + +// inflatedFormats report more media than the album has discs. A 2LP pressing of +// a single-CD album lists two media, which would drag the disc consensus up. +var inflatedFormats = []string{"vinyl", "cassette"} + +func inflatesDiscCount(rel MBRelease) bool { + for _, medium := range rel.Media { + for _, format := range inflatedFormats { + if strings.EqualFold(medium.Format, format) { + return true + } + } + } + return false +} + +// CanonicalReleaseShape reports the modal first-medium track count and the modal +// medium count across the releases sharing groupID, or zeroes when there is no +// usable consensus. Zero leaves every caller on the matched release's own +// numbers, which is what happens with enrichment switched off. +func CanonicalReleaseShape(releases []MBRelease, groupID string) (trackTotal, discTotal int) { + if groupID == "" { + return 0, 0 + } + + group := make([]MBRelease, 0, len(releases)) + for _, rel := range releases { + // Other release groups carry this recording too -- greatest hits, + // soundtracks -- and their lengths say nothing about this album. + if rel.ReleaseGroup.ID != groupID || len(rel.Media) == 0 || inflatesDiscCount(rel) { + continue + } + group = append(group, rel) + } + if len(group) == 0 { + return 0, 0 + } + + // Bootlegs and promos are poor evidence of the album's real shape, but they + // are better than nothing: a group with no official release at all still + // gets a consensus rather than falling back to the matched release. + official := make([]MBRelease, 0, len(group)) + for _, rel := range group { + if strings.EqualFold(rel.Status, "Official") { + official = append(official, rel) + } + } + if len(official) > 0 { + group = official + } + + tracks := make([]int, 0, len(group)) + discs := make([]int, 0, len(group)) + for _, rel := range group { + if count := rel.Media[0].TrackCount; count > 0 { + tracks = append(tracks, count) + } + discs = append(discs, len(rel.Media)) + } + + return modal(tracks), modal(discs) +} + +// modal returns the most common value, breaking ties toward the smaller one. +// +// The tie-break is the point of the whole exercise: when an album is shared +// equally as a 12-track original and a 20-track expanded edition, the original +// is the one being asked for. +func modal(values []int) int { + counts := make(map[int]int, len(values)) + for _, value := range values { + counts[value]++ + } + + var best, bestCount int + for value, count := range counts { + if count > bestCount || (count == bestCount && value < best) { + best, bestCount = value, count + } + } + return best +} diff --git a/src/discovery/canonical_test.go b/src/discovery/canonical_test.go new file mode 100644 index 00000000..799db2c3 --- /dev/null +++ b/src/discovery/canonical_test.go @@ -0,0 +1,184 @@ +package discovery + +import ( + "encoding/json" + "testing" +) + +const groupID = "group-1" + +// release builds one release in groupID. Each entry in mediaTracks is one +// medium's track count, so len(mediaTracks) is the disc count. +func release(status string, format string, mediaTracks ...int) MBRelease { + rel := MBRelease{Status: status} + rel.ReleaseGroup.ID = groupID + for _, count := range mediaTracks { + rel.Media = append(rel.Media, MBMedium{Format: format, TrackCount: count}) + } + return rel +} + +// The point of the whole exercise: the matched release may be a deluxe, and the +// group knows better. +func TestCanonicalReleaseShape_PrefersTheConsensusOverAnOutlier(t *testing.T) { + releases := []MBRelease{ + release("Official", "CD", 16), + release("Official", "CD", 16), + release("Official", "Digital Media", 16), + release("Official", "CD", 16, 23), // the two-disc deluxe + } + + tracks, discs := CanonicalReleaseShape(releases, groupID) + + if tracks != 16 || discs != 1 { + t.Errorf("CanonicalReleaseShape() = %d tracks, %d discs; want 16 and 1", tracks, discs) + } +} + +// A real double album has a group that agrees it is one. +func TestCanonicalReleaseShape_KeepsAGenuineDoubleAlbum(t *testing.T) { + releases := []MBRelease{ + release("Official", "CD", 13, 13), + release("Official", "CD", 13, 13), + release("Official", "CD", 26), + } + + if _, discs := CanonicalReleaseShape(releases, groupID); discs != 2 { + t.Errorf("disc consensus = %d, want 2", discs) + } +} + +// A recording appears on greatest hits and soundtracks too, and their lengths +// say nothing about this album. +func TestCanonicalReleaseShape_IgnoresOtherReleaseGroups(t *testing.T) { + other := release("Official", "CD", 40) + other.ReleaseGroup.ID = "group-2" + + releases := []MBRelease{ + release("Official", "CD", 11), + other, other, other, + } + + if tracks, _ := CanonicalReleaseShape(releases, groupID); tracks != 11 { + t.Errorf("track consensus = %d, want 11 from this group only", tracks) + } +} + +func TestCanonicalReleaseShape_IgnoresBootlegsWhenOfficialsExist(t *testing.T) { + releases := []MBRelease{ + release("Official", "CD", 12), + release("Bootleg", "CD", 30), + release("Bootleg", "CD", 30), + release("Bootleg", "CD", 30), + } + + if tracks, _ := CanonicalReleaseShape(releases, groupID); tracks != 12 { + t.Errorf("track consensus = %d, want the official 12 despite more bootlegs", tracks) + } +} + +// Poor evidence beats none: a group with nothing official still yields numbers +// rather than falling back to the matched release. +func TestCanonicalReleaseShape_UsesBootlegsWhenThatIsAllThereIs(t *testing.T) { + releases := []MBRelease{ + release("Bootleg", "CD", 9), + release("Bootleg", "CD", 9), + } + + if tracks, _ := CanonicalReleaseShape(releases, groupID); tracks != 9 { + t.Errorf("track consensus = %d, want 9", tracks) + } +} + +// A 2LP pressing of a single-CD album lists two media and would drag the disc +// consensus up to two. +func TestCanonicalReleaseShape_ExcludesVinylFromTheDiscCount(t *testing.T) { + releases := []MBRelease{ + release("Official", "CD", 14), + release("Official", "Vinyl", 7, 7), + release("Official", "Vinyl", 7, 7), + } + + tracks, discs := CanonicalReleaseShape(releases, groupID) + + if discs != 1 { + t.Errorf("disc consensus = %d, want 1; the vinyl pressings inflated it", discs) + } + if tracks != 14 { + t.Errorf("track consensus = %d, want the CD's 14", tracks) + } +} + +// The tie-break is the point: asked for an album shared equally as original and +// expanded, the original is the one wanted. +func TestCanonicalReleaseShape_BreaksTiesTowardTheSmaller(t *testing.T) { + releases := []MBRelease{ + release("Official", "CD", 12), + release("Official", "CD", 20), + } + + if tracks, _ := CanonicalReleaseShape(releases, groupID); tracks != 12 { + t.Errorf("track consensus = %d, want the smaller 12 on a tie", tracks) + } +} + +func TestCanonicalReleaseShape_ZeroWithoutUsableReleases(t *testing.T) { + cases := map[string]struct { + releases []MBRelease + group string + }{ + "no releases": {nil, groupID}, + "no group id": {[]MBRelease{release("Official", "CD", 12)}, ""}, + "no media": {[]MBRelease{release("Official", "CD")}, groupID}, + "all other group": {[]MBRelease{{Status: "Official"}}, groupID}, + } + + for name, tc := range cases { + tracks, discs := CanonicalReleaseShape(tc.releases, tc.group) + if tracks != 0 || discs != 0 { + t.Errorf("%s: got %d tracks, %d discs; want zeroes so callers fall back", + name, tracks, discs) + } + } +} + +// The consensus is read off a real MusicBrainz payload, so the struct tags have +// to survive. MBRelease and MBMedium were extracted from anonymous structs to +// make this helper testable, and a mistyped tag there would silently yield no +// consensus rather than an error. +func TestCanonicalReleaseShape_FromAMusicBrainzPayload(t *testing.T) { + const body = `{ + "id": "rec-1", + "releases": [ + {"id": "r1", "title": "When We All Fall Asleep", "status": "Official", + "release-group": {"id": "group-1", "primary-type": "Album"}, + "media": [{"position": 1, "format": "CD", "track-count": 14}]}, + {"id": "r2", "title": "When We All Fall Asleep", "status": "Official", + "release-group": {"id": "group-1", "primary-type": "Album"}, + "media": [{"position": 1, "format": "Digital Media", "track-count": 14}]}, + {"id": "r3", "title": "When We All Fall Asleep (Deluxe)", "status": "Official", + "release-group": {"id": "group-1", "primary-type": "Album"}, + "media": [{"position": 1, "format": "CD", "track-count": 14}, + {"position": 2, "format": "CD", "track-count": 8}]}, + {"id": "r4", "title": "Now That's What I Call Music", "status": "Official", + "release-group": {"id": "group-2", "primary-type": "Album"}, + "media": [{"position": 1, "format": "CD", "track-count": 42}]} + ] + }` + + var recording MBRecording + if err := json.Unmarshal([]byte(body), &recording); err != nil { + t.Fatalf("decoding the payload: %v", err) + } + if len(recording.Releases) != 4 { + t.Fatalf("decoded %d releases, want 4; the struct tags no longer match", + len(recording.Releases)) + } + + tracks, discs := CanonicalReleaseShape(recording.Releases, "group-1") + + if tracks != 14 || discs != 1 { + t.Errorf("CanonicalReleaseShape() = %d tracks, %d discs; want 14 and 1 "+ + "(the deluxe outvoted, the compilation ignored)", tracks, discs) + } +} diff --git a/src/discovery/discoverer_test.go b/src/discovery/discoverer_test.go new file mode 100644 index 00000000..04919eea --- /dev/null +++ b/src/discovery/discoverer_test.go @@ -0,0 +1,33 @@ +package discovery + +import ( + "strings" + "testing" + + cfg "explo/src/config" +) + +// BUG-04: a typo in DISCOVERY_SERVICE made NewDiscoverer return nil and +// the caller panicked dereferencing it, with no hint of the bad value. +func TestNewDiscoverer_UnknownServiceErrorsWithValue(t *testing.T) { + c, err := NewDiscoverer(cfg.DiscoveryConfig{Discovery: "listenbranz"}, nil) + if err == nil { + t.Fatal("NewDiscoverer with a typo = nil error, want a clear failure") + } + if c != nil { + t.Errorf("client = %v, want nil on error", c) + } + if !strings.Contains(err.Error(), "listenbranz") { + t.Errorf("error %q does not name the bad value", err) + } +} + +func TestNewDiscoverer_ListenBrainzOK(t *testing.T) { + c, err := NewDiscoverer(cfg.DiscoveryConfig{Discovery: "listenbrainz"}, nil) + if err != nil { + t.Fatalf("NewDiscoverer(listenbrainz): %v", err) + } + if c == nil || c.Discovery == nil { + t.Error("NewDiscoverer(listenbrainz) returned an incomplete client") + } +} diff --git a/src/discovery/discovery.go b/src/discovery/discovery.go index 1a1b2e99..ad8967ae 100644 --- a/src/discovery/discovery.go +++ b/src/discovery/discovery.go @@ -4,6 +4,7 @@ import ( cfg "explo/src/config" "explo/src/models" "explo/src/util" + "fmt" "log/slog" ) @@ -15,16 +16,16 @@ type Discovery interface { QueryTracks() ([]*models.Track, error) } -func NewDiscoverer(cfg cfg.DiscoveryConfig, httpClient *util.HttpClient) *DiscoverClient { +func NewDiscoverer(cfg cfg.DiscoveryConfig, httpClient *util.HttpClient) (*DiscoverClient, error) { c := &DiscoverClient{cfg: &cfg} switch cfg.Discovery { case "listenbrainz": c.Discovery = NewListenBrainz(cfg, httpClient) default: - return nil + return nil, fmt.Errorf("unknown DISCOVERY_SERVICE %q: supported values are: listenbrainz", cfg.Discovery) } - return c + return c, nil } func (c *DiscoverClient) Discover() ([]*models.Track, error) { diff --git a/src/discovery/listenbrainz.go b/src/discovery/listenbrainz.go index 993b5c2d..8a5eb4d5 100644 --- a/src/discovery/listenbrainz.go +++ b/src/discovery/listenbrainz.go @@ -149,36 +149,67 @@ type TopRecordings struct { } type MBRecording struct { - ID string `json:"id"` - Releases []struct { - ID string `json:"id"` - Title string `json:"title"` - Status string `json:"status"` - Country string `json:"country"` - Date string `json:"date"` - Year int `json:"year,omitempty"` - ArtistCredit []struct { - Name string `json:"name"` - Artist struct { - ID string `json:"id"` - SortName string `json:"sort-name"` - } `json:"artist"` - } `json:"artist-credit"` - ReleaseGroup struct { - ID string `json:"id"` - PrimaryType string `json:"primary-type"` - } `json:"release-group"` - Media []struct { - Position int `json:"position"` - Format string `json:"format"` - TrackCount int `json:"track-count"` - Tracks []struct { - ID string `json:"id"` - Position int `json:"position"` - Number string `json:"number"` - } `json:"tracks"` - } `json:"media"` - } `json:"releases"` + ID string `json:"id"` + Releases []MBRelease `json:"releases"` +} + +// MBRelease is one release carrying the recording. Named rather than anonymous +// so the release-group consensus can be computed by a helper that takes it. +type MBRelease struct { + ID string `json:"id"` + Title string `json:"title"` + Status string `json:"status"` + Country string `json:"country"` + Date string `json:"date"` + Year int `json:"year,omitempty"` + ArtistCredit []struct { + Name string `json:"name"` + Artist struct { + ID string `json:"id"` + SortName string `json:"sort-name"` + } `json:"artist"` + } `json:"artist-credit"` + ReleaseGroup struct { + ID string `json:"id"` + PrimaryType string `json:"primary-type"` + } `json:"release-group"` + Media []MBMedium `json:"media"` +} + +// MBMedium is one disc of a release. +type MBMedium struct { + Position int `json:"position"` + Format string `json:"format"` + TrackCount int `json:"track-count"` + Tracks []MBReleaseTrack `json:"tracks"` +} + +// MBReleaseTrack is one track on a disc. Its ID is the release-track +// entity; Recording.ID is the recording this track performance is of, +// which is what lets enrichment find the track being enriched rather +// than always taking Media[0].Tracks[0] (BUG-22). +type MBReleaseTrack struct { + ID string `json:"id"` + Position int `json:"position"` + Number string `json:"number"` + Recording struct { + ID string `json:"id"` + } `json:"recording"` +} + +// findReleaseTrack locates the track for recordingMBID inside a +// release's media, returning the disc and the track. It returns false +// when the recording is not listed, in which case callers keep the +// previous first-track behavior. +func findReleaseTrack(release MBRelease, recordingMBID string) (MBMedium, MBReleaseTrack, bool) { + for _, medium := range release.Media { + for _, track := range medium.Tracks { + if recordingMBID != "" && track.Recording.ID == recordingMBID { + return medium, track, true + } + } + } + return MBMedium{}, MBReleaseTrack{}, false } type TitleArgs struct { @@ -348,49 +379,76 @@ func (c *ListenBrainz) getTracks(mbids []string, singleArtist bool) ([]*models.T } tracks := make([]*models.Track, 0, len(recordings)) - for mbTrackID, recording := range recordings { - rec := recording.Recording - rel := recording.Release + // Iterate the requested MBIDs, not the response map: map order would + // shuffle the playlist away from the recommendation ranking on every + // run (BUG-23). Recordings the API did not return are dropped. + for _, mbTrackID := range orderedRecordingMBIDs(mbids, recordings) { + tracks = append(tracks, c.trackFromRecording(mbTrackID, recordings[mbTrackID], singleArtist)) + } - title := rec.Name - artist := recording.Artist.Name - mainArtist := recording.Artist.Name + return tracks, nil - recArtists := recording.Artist.Artists +} - if len(recArtists) > 1 { - mainArtist = recArtists[0].Name - if singleArtist { - var titleArgs TitleArgs - titleArgs.CleanTitle = rec.Name - for _, artist := range recArtists[1:] { +// trackFromRecording builds one track from ListenBrainz recording metadata. +// The artists list may be empty for unusual metadata; degrade to the credit +// name with no artist MBID instead of panicking on recArtists[0]. +func (c *ListenBrainz) trackFromRecording(mbTrackID string, recording Metadata, singleArtist bool) *models.Track { + rec := recording.Recording + rel := recording.Release - titleArgs.FeatArtists = append(titleArgs.FeatArtists, artist.Name) - } - title = c.buildTrackTitle(titleArgs) - artist = mainArtist + title := rec.Name + artist := recording.Artist.Name + mainArtist := recording.Artist.Name + + recArtists := recording.Artist.Artists + + artistID := "" + if len(recArtists) > 0 { + artistID = recArtists[0].MusicBrainzArtistID + } + + if len(recArtists) > 1 { + mainArtist = recArtists[0].Name + if singleArtist { + var titleArgs TitleArgs + titleArgs.CleanTitle = rec.Name + for _, artist := range recArtists[1:] { + titleArgs.FeatArtists = append(titleArgs.FeatArtists, artist.Name) } + title = c.buildTrackTitle(titleArgs) + artist = mainArtist } - - tracks = append(tracks, &models.Track{ - Album: recording.Release.Name, - Artist: artist, - AlbumArtist: rel.AlbumArtistName, - MainArtist: mainArtist, - CleanTitle: rec.Name, - Title: title, - Duration: rec.Length, - MusicBrainzTrackID: mbTrackID, - MusicBrainzAlbumID: rel.CaaReleaseMbid, - MusicBrainzReleaseGroupID: rel.ReleaseGroupMbid, - MusicBrainzArtistID: recArtists[0].MusicBrainzArtistID, - }) } - return tracks, nil + return &models.Track{ + Album: recording.Release.Name, + Artist: artist, + AlbumArtist: rel.AlbumArtistName, + MainArtist: mainArtist, + CleanTitle: rec.Name, + Title: title, + Duration: rec.Length, + MusicBrainzTrackID: mbTrackID, + MusicBrainzAlbumID: rel.CaaReleaseMbid, + MusicBrainzReleaseGroupID: rel.ReleaseGroupMbid, + MusicBrainzArtistID: artistID, + } +} +// orderedRecordingMBIDs intersects the requested ranking with the +// recordings the API actually returned, keeping request order. +func orderedRecordingMBIDs(mbids []string, recordings Recordings) []string { + ordered := make([]string, 0, len(recordings)) + for _, mbid := range mbids { + if _, ok := recordings[mbid]; ok { + ordered = append(ordered, mbid) + } + } + return ordered } + func (c *ListenBrainz) enrichTracks(tracks []*models.Track, singleArtist bool) ([]*models.Track, error) { mbids := make([]string, 0, len(tracks)) // wait time in s between MusicBrainz requests @@ -482,6 +540,8 @@ func (c *ListenBrainz) enrichTracks(tracks []*models.Track, singleArtist bool) ( discTotal := 0 mbReleaseTrackID := "" releaseType := "" + canonicalTrackTotal := 0 + canonicalDiscTotal := 0 var mbData *MBRecording var mbErr error @@ -526,17 +586,25 @@ func (c *ListenBrainz) enrichTracks(tracks []*models.Track, singleArtist bool) ( artistSort = bestRelease.ArtistCredit[0].Artist.SortName } + // Consensus over the whole group, not just the release the + // lookup landed on. Free: these releases are already parsed. + canonicalTrackTotal, canonicalDiscTotal = CanonicalReleaseShape(mbData.Releases, mbReleaseGroupID) + discTotal = len(bestRelease.Media) - if len(bestRelease.Media) > 0 { - firstMedia := bestRelease.Media[0] - media = firstMedia.Format - discNumber = firstMedia.Position - trackTotal = firstMedia.TrackCount - - if len(firstMedia.Tracks) > 0 { - firstTrack := firstMedia.Tracks[0] - trackNumber = firstTrack.Position - mbReleaseTrackID = firstTrack.ID + // Find the disc/track this recording is actually on. When + // the API omits recording links, fall back to the first + // track as before rather than leaving the numbers empty. + medium, mediumTrack, found := findReleaseTrack(bestRelease, track.MusicBrainzTrackID) + if !found && len(bestRelease.Media) > 0 && len(bestRelease.Media[0].Tracks) > 0 { + medium, mediumTrack = bestRelease.Media[0], bestRelease.Media[0].Tracks[0] + } + if found || len(bestRelease.Media) > 0 { + media = medium.Format + discNumber = medium.Position + trackTotal = medium.TrackCount + if len(medium.Tracks) > 0 { + trackNumber = mediumTrack.Position + mbReleaseTrackID = mediumTrack.ID } } } @@ -572,6 +640,8 @@ func (c *ListenBrainz) enrichTracks(tracks []*models.Track, singleArtist bool) ( TrackTotal: trackTotal, DiscNumber: discNumber, DiscTotal: discTotal, + CanonicalTrackTotal: canonicalTrackTotal, + CanonicalDiscTotal: canonicalDiscTotal, MusicBrainzTrackID: track.MusicBrainzTrackID, MusicBrainzAlbumID: recording.Release.CaaReleaseMbid, MusicBrainzReleaseGroupID: mbReleaseGroupID, diff --git a/src/discovery/listenbrainz_empty_artist_test.go b/src/discovery/listenbrainz_empty_artist_test.go new file mode 100644 index 00000000..a4e1b5fc --- /dev/null +++ b/src/discovery/listenbrainz_empty_artist_test.go @@ -0,0 +1,44 @@ +package discovery + +import ( + "encoding/json" + "testing" +) + +// BUG-03: ListenBrainz metadata with an empty artists array panicked on +// recArtists[0], crashing the whole run. +func TestTrackFromRecording_EmptyArtistsDoesNotPanic(t *testing.T) { + var md Metadata + raw := `{"artist":{"name":"Solo Credit","artists":[]},"recording":{"name":"Song","length":123},"release":{"name":"Album"}}` + if err := json.Unmarshal([]byte(raw), &md); err != nil { + t.Fatal(err) + } + + c := &ListenBrainz{} + track := c.trackFromRecording("mbid-1", md, false) + + if track.MusicBrainzArtistID != "" { + t.Errorf("MusicBrainzArtistID = %q, want empty for missing artists", track.MusicBrainzArtistID) + } + if track.MainArtist != "Solo Credit" || track.Title != "Song" { + t.Errorf("track = %+v, want credit-name fallback and title preserved", track) + } +} + +func TestTrackFromRecording_WithArtistsKeepsArtistID(t *testing.T) { + var md Metadata + raw := `{"artist":{"name":"A feat. B","artists":[{"artist_mbid":"a1","name":"A"},{"artist_mbid":"b2","name":"B"}]},"recording":{"name":"Song","length":1},"release":{"name":"Album"}}` + if err := json.Unmarshal([]byte(raw), &md); err != nil { + t.Fatal(err) + } + + c := &ListenBrainz{} + track := c.trackFromRecording("mbid-2", md, false) + + if track.MusicBrainzArtistID != "a1" { + t.Errorf("MusicBrainzArtistID = %q, want a1", track.MusicBrainzArtistID) + } + if track.MainArtist != "A" { + t.Errorf("MainArtist = %q, want A", track.MainArtist) + } +} diff --git a/src/discovery/recording_order_test.go b/src/discovery/recording_order_test.go new file mode 100644 index 00000000..8282b4ca --- /dev/null +++ b/src/discovery/recording_order_test.go @@ -0,0 +1,28 @@ +package discovery + +import ( + "reflect" + "testing" +) + +// BUG-23: getTracks iterated the Recordings response map, so playlist +// order changed randomly between runs instead of following the ranking. +func TestOrderedRecordingMBIDs_KeepsRequestOrder(t *testing.T) { + mk := func() Recordings { + return Recordings{"c": {}, "a": {}, "b": {}} + } + got := orderedRecordingMBIDs([]string{"a", "b", "c"}, mk()) + if want := []string{"a", "b", "c"}; !reflect.DeepEqual(got, want) { + t.Errorf("ordered = %v, want %v", got, want) + } +} + +func TestOrderedRecordingMBIDs_DropsMissingRecordings(t *testing.T) { + got := orderedRecordingMBIDs( + []string{"a", "missing", "b"}, + Recordings{"b": {}, "a": {}}, + ) + if want := []string{"a", "b"}; !reflect.DeepEqual(got, want) { + t.Errorf("ordered = %v, want %v", got, want) + } +} diff --git a/src/discovery/release_track_test.go b/src/discovery/release_track_test.go new file mode 100644 index 00000000..da7b63cd --- /dev/null +++ b/src/discovery/release_track_test.go @@ -0,0 +1,52 @@ +package discovery + +import ( + "encoding/json" + "testing" +) + +// BUG-22: enrichment always took Media[0].Tracks[0], so every song was +// tagged track 1 of disc 1. +func parseRelease(t *testing.T, raw string) MBRelease { + t.Helper() + var rel MBRelease + if err := json.Unmarshal([]byte(raw), &rel); err != nil { + t.Fatal(err) + } + return rel +} + +const twoDiscRelease = `{"media":[ + {"position":1,"format":"CD","track-count":2,"tracks":[ + {"id":"rt-a1","position":1,"number":"1","recording":{"id":"rec-a"}}, + {"id":"rt-a2","position":2,"number":"2","recording":{"id":"rec-b"}}]}, + {"position":2,"format":"CD","track-count":1,"tracks":[ + {"id":"rt-b1","position":1,"number":"1","recording":{"id":"rec-c"}}]} +]}` + +func TestFindReleaseTrack_FindsSecondDiscRecording(t *testing.T) { + rel := parseRelease(t, twoDiscRelease) + + medium, track, found := findReleaseTrack(rel, "rec-c") + if !found { + t.Fatal("recording rec-c not found") + } + if medium.Position != 2 || track.Position != 1 || track.ID != "rt-b1" { + t.Errorf("got disc %d track %+v, want disc 2 rt-b1", medium.Position, track) + } +} + +func TestFindReleaseTrack_UnknownRecordingReportsAbsent(t *testing.T) { + rel := parseRelease(t, twoDiscRelease) + + if _, _, found := findReleaseTrack(rel, "rec-missing"); found { + t.Error("unknown recording reported as found") + } +} + +func TestFindReleaseTrack_EmptyMedia(t *testing.T) { + rel := parseRelease(t, `{"media":[]}`) + if _, _, found := findReleaseTrack(rel, "rec-a"); found { + t.Error("recording found in empty media") + } +} diff --git a/src/downloader/downloader.go b/src/downloader/downloader.go index 71199483..5c65fd8d 100644 --- a/src/downloader/downloader.go +++ b/src/downloader/downloader.go @@ -33,6 +33,30 @@ type Downloader interface { Monitor } +// albumMigrator is implemented by downloaders that fetch a whole release and +// so have files to migrate beyond the one the monitor tracks. It is optional: +// a downloader that only ever grabs a single file simply does not implement it. +// +// MoveAlbumSiblings returns how many files are still in flight. The monitor +// calls it again while that is above zero, because the recommended track +// normally finishes ahead of the rest of its release. +type albumMigrator interface { + MoveAlbumSiblings(trackDir, destDir string, track *models.Track) int +} + +// albumMigrators returns the registered downloaders that migrate whole +// releases. Implementations are responsible for deciding whether they have +// anything to move for a given track. +func (c *DownloadClient) albumMigrators() []albumMigrator { + var migrators []albumMigrator + for _, d := range c.Downloaders { + if m, ok := d.(albumMigrator); ok { + migrators = append(migrators, m) + } + } + return migrators +} + // get download services from config and append them to DownloadClient func NewDownloader(cfg *cfg.DownloadConfig, httpClient *util.HttpClient, filterLocal bool) (*DownloadClient, error) { var downloader []Downloader @@ -306,12 +330,15 @@ func tempAudioFile(path string) string { } -func moveTrack(srcFile, destDir string, track *models.Track, pathTemplate string, keepPerms bool) error { +// moveTrack moves srcFile into the library and returns the directory it landed +// in. The source directory is removed only once it is empty, so album siblings +// still downloading next to it are left alone. +func moveTrack(srcFile, destDir string, track *models.Track, pathTemplate string, keepPerms bool) (string, error) { var dstFile string if pathTemplate != "" { relativePath, err := buildTrackPath(pathTemplate, track) if err != nil { - return err + return "", err } if track.File == "." || track.File == string(filepath.Separator) { @@ -326,29 +353,56 @@ func moveTrack(srcFile, destDir string, track *models.Track, pathTemplate string dstFile = filepath.Join(destDir, relativePath) } else { if err := os.MkdirAll(destDir, os.ModePerm); err != nil { - return err + return "", err } dstFile = filepath.Join(destDir, track.File) } if err := moveFile(srcFile, dstFile, keepPerms); err != nil { - return err + return "", err + } + + if err := removeDirIfEmpty(filepath.Dir(srcFile)); err != nil { + return "", err } - srcDir := filepath.Dir(srcFile) - isEmpty, err := isDirEmpty(srcDir) + return filepath.Dir(dstFile), nil +} + +// removeDirIfEmpty cleans up a download directory once nothing is left in it. +func removeDirIfEmpty(dir string) error { + isEmpty, err := isDirEmpty(dir) if err != nil { return fmt.Errorf("couldn't check if directory is empty: %s", err.Error()) - } else if isEmpty { - if err = os.Remove(srcDir); err != nil { - return fmt.Errorf("failed to remove empty directory: %s", err.Error()) - } } - - return nil + if !isEmpty { + return nil + } + if err = os.Remove(dir); err != nil { + return fmt.Errorf("failed to remove empty directory: %s", err.Error()) + } + return nil } func moveFile(srcFile, dstFile string, keepPermissions bool) error { + // Never truncate a file that is already where it belongs, and never + // erase an existing destination: os.Create below would zero the + // source itself when src == dst, or silently overwrite a library + // song whose name collides with the incoming file. + srcAbs, srcErr := filepath.Abs(srcFile) + dstAbs, dstErr := filepath.Abs(dstFile) + if srcErr == nil && dstErr == nil && srcAbs == dstAbs { + return nil + } + if srcInfo, err := os.Stat(srcFile); err == nil { + if dstInfo, err := os.Stat(dstFile); err == nil { + if os.SameFile(srcInfo, dstInfo) { + return nil + } + return fmt.Errorf("destination already exists, refusing to overwrite: %s", dstFile) + } + } + in, err := os.Open(srcFile) if err != nil { return fmt.Errorf("couldn't open source file: %w", err) diff --git a/src/downloader/lidarr.go b/src/downloader/lidarr.go index bc2ac774..f3a2b332 100644 --- a/src/downloader/lidarr.go +++ b/src/downloader/lidarr.go @@ -221,6 +221,7 @@ func (c *Lidarr) GetConf() (MonitorConfig, error) { CheckInterval: time.Duration(c.Cfg.MonitorConfig.Interval) * time.Minute, StallDuration: time.Duration(c.Cfg.MonitorConfig.StallDuration) * time.Minute, MaxDuration: time.Duration(c.Cfg.MonitorConfig.MaxDuration) * time.Minute, + FallbackAfter: time.Duration(c.Cfg.MonitorConfig.FallbackAfter) * time.Minute, MigrateDownload: c.Cfg.MigrateDL, ToDir: c.DownloadDir, FromDir: c.Cfg.LidarrDir, @@ -350,7 +351,7 @@ func (c *Lidarr) addNewAlbum(track *models.Track) error { body, err := c.HttpClient.MakeRequest("POST", queryURL, bytes.NewReader(payloadBody), c.Headers) if err != nil { if strings.Contains(err.Error(), "got 409") || strings.Contains(string(body), "has already been added") { - if _, ok := c.AlbumCache[c.cacheKey(track)]; ok { + if _, ok := c.cachedAlbum(c.cacheKey(track)); ok { slog.Debug("album already in Lidarr", "album", track.MusicBrainzReleaseGroupID) return nil } @@ -516,7 +517,11 @@ func (c *Lidarr) GetDownloadStatus(tracks []*models.Track) (map[string]FileStatu } - cache := c.AlbumCache[c.cacheKey(albumTracks[0])] + cache, ok := c.cachedAlbum(c.cacheKey(albumTracks[0])) + if !ok || cache == nil { + slog.Debug("[lidarr] album not cached, skipping history import", "album", albumTracks[0].Album) + continue + } for _, r := range history.Records { if r.Data.ImportedPath == "" { continue @@ -755,6 +760,17 @@ func (c *Lidarr) cacheKey(track *models.Track) string { return track.MainArtist + "|" + track.Album } +// cachedAlbum reads the album cache under its lock. All AlbumCache reads +// must go through here (or hold CacheMu) — cacheAlbum writes the map from +// concurrent goroutines. +func (c *Lidarr) cachedAlbum(key string) (*AlbumMetadata, bool) { + c.CacheMu.RLock() + defer c.CacheMu.RUnlock() + + a, ok := c.AlbumCache[key] + return a, ok +} + func (c *Lidarr) cacheAlbum(track *models.Track) { key := c.cacheKey(track) c.CacheMu.Lock() @@ -792,25 +808,25 @@ func (c *Lidarr) populateFromCache(track *models.Track) bool { } -func (c *Lidarr) MoveDownload(srcDir, destDir, albumPath string, track *models.Track) error { +func (c *Lidarr) MoveDownload(srcDir, destDir, albumPath string, track *models.Track) (string, error) { key := c.cacheKey(track) c.CacheMu.RLock() album := c.AlbumCache[key] if album == nil { c.CacheMu.RUnlock() - return fmt.Errorf("album not cached") + return "", fmt.Errorf("album not cached") } if album.Moved { c.CacheMu.RUnlock() - return nil + return "", nil } albumFiles := maps.Clone(album.Files) albumMeta := *album c.CacheMu.RUnlock() if len(albumFiles) == 0 { - return fmt.Errorf("no downloaded files found for album") + return "", fmt.Errorf("no downloaded files found for album") } slog.Info("moving album", "artist", track.MainArtist, "album", track.Album, "id", track.AlbumID) @@ -826,15 +842,15 @@ func (c *Lidarr) MoveDownload(srcDir, destDir, albumPath string, track *models.T albumTrack := c.addMetadatafromCache(albumMeta, file) - if err := moveTrack(srcFile, destDir, &albumTrack, c.Cfg.PathTemplate, c.Cfg.KeepPermissions); err != nil { - return fmt.Errorf("failed to move track: %w", err) + if _, err := moveTrack(srcFile, destDir, &albumTrack, c.Cfg.PathTemplate, c.Cfg.KeepPermissions); err != nil { + return "", fmt.Errorf("failed to move track: %w", err) } } c.CacheMu.Lock() album.Moved = true c.CacheMu.Unlock() - return nil + return "", nil } diff --git a/src/downloader/lidarr_cache_race_test.go b/src/downloader/lidarr_cache_race_test.go new file mode 100644 index 00000000..2cc629a2 --- /dev/null +++ b/src/downloader/lidarr_cache_race_test.go @@ -0,0 +1,45 @@ +package downloader + +import ( + "sync" + "testing" + + "explo/src/models" +) + +// BUG-02: AlbumCache was read without CacheMu while cacheAlbum wrote it +// from concurrent goroutines, and a missing entry was dereferenced. Run +// with -race. +func TestLidarrAlbumCache_ConcurrentAccess(t *testing.T) { + c := &Lidarr{AlbumCache: make(map[string]*AlbumMetadata)} + newTrack := func() *models.Track { + return &models.Track{MainArtist: "Artist", Album: "Album", AlbumID: "1", MainArtistID: "2"} + } + + var wg sync.WaitGroup + for i := 0; i < 8; i++ { + wg.Add(1) + go func() { + defer wg.Done() + // Each goroutine works on its own track, as concurrent album + // fetches do; they share only the cache map under test. + track := newTrack() + for j := 0; j < 200; j++ { + c.cacheAlbum(track) + if _, ok := c.cachedAlbum(c.cacheKey(track)); !ok { + t.Error("cached album disappeared") + return + } + c.populateFromCache(track) + } + }() + } + wg.Wait() + + if _, ok := c.cachedAlbum(c.cacheKey(newTrack())); !ok { + t.Fatal("album not cached after concurrent writes") + } + if _, ok := c.cachedAlbum("missing|key"); ok { + t.Error("missing key reported as cached") + } +} diff --git a/src/downloader/monitor.go b/src/downloader/monitor.go index 00ac2d28..1e19b292 100644 --- a/src/downloader/monitor.go +++ b/src/downloader/monitor.go @@ -5,14 +5,22 @@ import ( "explo/src/models" "fmt" "log/slog" + "path/filepath" "strings" "time" ) +// maxPollFailures is how many consecutive status polls may fail before the +// monitor gives up. Anything less than the whole run is worth surviving: the +// downloads themselves keep going, and only the monitor loses sight of them. +const maxPollFailures = 3 + type Monitor interface { GetDownloadStatus([]*models.Track) (map[string]FileStatus, error) GetConf() (MonitorConfig, error) - MoveDownload(string, string, string, *models.Track) error + // MoveDownload migrates a finished track into the library and returns the + // directory it landed in, so album siblings can follow it there. + MoveDownload(string, string, string, *models.Track) (string, error) Cleanup(models.Track, string) error } @@ -20,6 +28,11 @@ type MonitorConfig struct { CheckInterval time.Duration StallDuration time.Duration MaxDuration time.Duration + // FallbackAfter bounds the whole monitor phase for one provider. Once it + // elapses, any track still unresolved is abandoned so the next provider in + // DOWNLOAD_SERVICES can attempt it. Zero means wait indefinitely, which is + // the historical behaviour. + FallbackAfter time.Duration MigrateDownload bool FromDir string ToDir string @@ -37,10 +50,27 @@ type FileStatus struct { QueueID string `json:"queueID"` } +// pendingAlbum is a release whose recommended track has already been migrated +// but whose remaining files are still downloading. The monitor revisits these +// each tick, because the recommended track is queued first and so normally +// finishes well ahead of its siblings. +type pendingAlbum struct { + track *models.Track + trackDir string + destDir string + // remaining is the count the migrator last reported. Progress is measured + // against this rather than against the migrator's own bookkeeping, so the + // deadline depends only on what the interface returns. + remaining int + startedAt time.Time + lastChange time.Time +} + func (c *DownloadClient) MonitorDownloads(tracks []*models.Track, m Monitor) error { var successDownloads int progressMap := make(map[string]*DownloadMonitor) + pendingAlbums := make(map[string]*pendingAlbum) monCfg, err := m.GetConf() if err != nil { return err @@ -50,11 +80,27 @@ func (c *DownloadClient) MonitorDownloads(tracks []*models.Track, m Monitor) err defer ticker.Stop() + var pollFailures int + + phaseStart := time.Now() + for range ticker.C { statuses, err := m.GetDownloadStatus(tracks) if err != nil { - return fmt.Errorf("[%s/monitor] error fetching download status: %s", monCfg.Service, err.Error()) + // A failed poll used to end monitoring for every track at once, so + // one blip -- a restarted service, a dropped connection -- cost the + // whole playlist, since tracks that never reach Present are dropped + // before it is built. Ride out a few and try again on the next tick. + pollFailures++ + if pollFailures >= maxPollFailures { + return fmt.Errorf("[%s/monitor] error fetching download status %d times in a row, giving up: %s", + monCfg.Service, pollFailures, err.Error()) + } + slog.Warn("[monitor] couldn't fetch download status, retrying", + "service", monCfg.Service, "attempt", pollFailures, "context", err.Error()) + continue } + pollFailures = 0 slog.Debug("fetched download queue", "size", len(statuses)) currentTime := time.Now().Local() @@ -90,20 +136,36 @@ func (c *DownloadClient) MonitorDownloads(tracks []*models.Track, m Monitor) err stallTime := currentTime.Sub(tracker.LastUpdated) monitoredTime := currentTime.Sub(tracker.StartedAt) + expired := monCfg.MaxDuration > 0 && monitoredTime > monCfg.MaxDuration - if (fileStatus.BytesRemaining == 0 && fileStatus.BytesTransferred != 0) || fileStatus.PercentComplete == 100 || strings.Contains(fileStatus.State, "Succeeded") { + if (fileStatus.BytesRemaining == 0 && fileStatus.BytesTransferred != 0) || fileStatus.PercentComplete == 100 || strings.Contains(fileStatus.State, "Succeeded") { track.File = fileStatus.Filename - track.Present = true slog.Info("[monitor] file downloaded successfully", "service", monCfg.Service, "file", track.File) var filePath string track.File, filePath = parsePath(track.File) if monCfg.MigrateDownload { - if err = m.MoveDownload(monCfg.FromDir, monCfg.ToDir, filePath, track); err != nil { - slog.Error("error while moving file", "err", err) - } else { - slog.Info("track moved successfully", "service", monCfg.Service) + destDir, err := m.MoveDownload(monCfg.FromDir, monCfg.ToDir, filePath, track) + if err != nil { + // The song never landed in the library, so it must + // not be marked Present or counted: before this, + // the playlist kept a track that never arrived. + slog.Error("error while moving file, skipping track", "err", err) + tracker.Skipped = true + if err = m.Cleanup(*track, fileStatus.QueueID); err != nil { + slog.Debug("cleanup failed", logging.RuntimeAttr(err.Error())) + } + continue } + slog.Info("track moved successfully", "service", monCfg.Service) + c.trackAlbum(pendingAlbums, key, &pendingAlbum{ + track: track, + trackDir: filepath.Join(monCfg.FromDir, filePath), + destDir: destDir, + startedAt: currentTime, + lastChange: currentTime, + }, monCfg) } + track.Present = true delete(progressMap, key) successDownloads += 1 if err = m.Cleanup(*track, fileStatus.QueueID); err != nil { @@ -111,7 +173,7 @@ func (c *DownloadClient) MonitorDownloads(tracks []*models.Track, m Monitor) err } continue - } else if fileStatus.BytesTransferred > tracker.LastBytesTransferred { + } else if fileStatus.BytesTransferred > tracker.LastBytesTransferred && !expired { tracker.LastBytesTransferred = fileStatus.BytesTransferred tracker.LastUpdated = currentTime slog.Info("[monitor] progress updated", "service", monCfg.Service, "title", track.CleanTitle, "bytes transferred", fileStatus.BytesTransferred) @@ -119,7 +181,7 @@ func (c *DownloadClient) MonitorDownloads(tracks []*models.Track, m Monitor) err } else if fileStatus.State == "Errored" || stallTime > monCfg.StallDuration || - monitoredTime > monCfg.MaxDuration { + expired { switch { case fileStatus.State == "Errored": @@ -148,15 +210,54 @@ func (c *DownloadClient) MonitorDownloads(tracks []*models.Track, m Monitor) err continue } } - // Exit condition: all tracks have been processed or skipped - if tracksProcessed(tracks, progressMap) { + c.migratePendingAlbums(pendingAlbums, monCfg, time.Now().Local()) + + // Exit condition: all tracks have been processed or skipped, and + // every release has finished being migrated + if tracksProcessed(tracks, progressMap) && len(pendingAlbums) == 0 { slog.Info("[monitor] Finished", "service", monCfg.Service, "downloaded files", successDownloads, "total tracks", len(tracks)) return nil } + + // Budget for this provider exhausted: abandon whatever is still in + // flight so the next service in DOWNLOAD_SERVICES gets a turn. + if monCfg.FallbackAfter > 0 && time.Since(phaseStart) > monCfg.FallbackAfter { + abandoned := abandonUnresolved(tracks, statuses, progressMap, m) + slog.Info("[monitor] fallback threshold reached, moving on to the next download service", + "service", monCfg.Service, "threshold", monCfg.FallbackAfter, + "downloaded files", successDownloads, "unresolved tracks", abandoned) + return nil + } } return nil } +// abandonUnresolved cancels any still-running download for tracks this provider +// did not finish, so it does not keep downloading in the background after we +// have moved on. Returns how many tracks were left unresolved. +func abandonUnresolved(tracks []*models.Track, statuses map[string]FileStatus, progressMap map[string]*DownloadMonitor, m Monitor) int { + var unresolved int + for _, track := range tracks { + if track.Present { + continue + } + key := fmt.Sprintf("%s|%s", track.ID, track.CleanTitle) + if tracker, exists := progressMap[key]; exists && tracker.Skipped { + continue + } + unresolved++ + + fileStatus, exists := statuses[track.ID] + if !exists { + continue + } + if err := m.Cleanup(*track, fileStatus.QueueID); err != nil { + slog.Debug("cleanup failed", logging.RuntimeAttr(err.Error())) + } + } + return unresolved +} + // Checks if all tracks are processed (either downloaded or skipped) func tracksProcessed(tracks []*models.Track, progressMap map[string]*DownloadMonitor) bool { for _, track := range tracks { @@ -168,4 +269,59 @@ func tracksProcessed(tracks []*models.Track, progressMap map[string]*DownloadMon } } return true -} \ No newline at end of file +} +// trackAlbum records a release for later migration, migrating whatever has +// already finished. Releases with nothing left over are never recorded, so a +// single-track download costs nothing. +func (c *DownloadClient) trackAlbum(pending map[string]*pendingAlbum, key string, album *pendingAlbum, monCfg MonitorConfig) { + if remaining := c.migrateAlbum(album); remaining > 0 { + album.remaining = remaining + pending[key] = album + slog.Info("[monitor] waiting for the rest of the release", "service", monCfg.Service, + "album", album.track.Album, "files", remaining) + } +} + +// migrateAlbum moves whatever of one release has finished, returning how many +// of its files are still in flight. +func (c *DownloadClient) migrateAlbum(album *pendingAlbum) int { + var remaining int + for _, m := range c.albumMigrators() { + remaining += m.MoveAlbumSiblings(album.trackDir, album.destDir, album.track) + } + return remaining +} + +// migratePendingAlbums revisits every release still waiting on files. A release +// that stops making progress is abandoned on the same deadline a stalled track +// is, and one that keeps trickling in is abandoned at the same maximum a track +// is, so the monitor cannot be held open by a peer that has gone away or slowed +// to a crawl. +func (c *DownloadClient) migratePendingAlbums(pending map[string]*pendingAlbum, monCfg MonitorConfig, now time.Time) { + for key, album := range pending { + before := album.remaining + remaining := c.migrateAlbum(album) + album.remaining = remaining + + if remaining == 0 { + delete(pending, key) + if err := removeDirIfEmpty(album.trackDir); err != nil { + slog.Debug("couldn't clean up release directory", "context", err.Error()) + } + slog.Info("[monitor] release fully migrated", "service", monCfg.Service, "album", album.track.Album) + continue + } + + if remaining < before { + album.lastChange = now + } + + stalled := now.Sub(album.lastChange) > monCfg.StallDuration + overdue := monCfg.MaxDuration > 0 && now.Sub(album.startedAt) > monCfg.MaxDuration + if stalled || overdue { + delete(pending, key) + slog.Warn("[monitor] giving up on the rest of the release", "service", monCfg.Service, + "album", album.track.Album, "files left in place", remaining, "stalled", stalled) + } + } +} diff --git a/src/downloader/monitor_album_test.go b/src/downloader/monitor_album_test.go new file mode 100644 index 00000000..4ea04215 --- /dev/null +++ b/src/downloader/monitor_album_test.go @@ -0,0 +1,280 @@ +package downloader + +import ( + "os" + "path/filepath" + "testing" + "time" + + cfg "explo/src/config" + "explo/src/models" +) + +// fakeMigrator stands in for a downloader that fetches whole releases, draining +// a fixed number of files per pass so a test can control how progress unfolds. +type fakeMigrator struct { + remaining int + perPass int + calls int + lastDest string + lastSource string +} + +func (f *fakeMigrator) QueryTrack(*models.Track) error { return nil } +func (f *fakeMigrator) GetTrack(*models.Track) error { return nil } + +func (f *fakeMigrator) GetDownloadStatus([]*models.Track) (map[string]FileStatus, error) { + return nil, nil +} +func (f *fakeMigrator) GetConf() (MonitorConfig, error) { return MonitorConfig{}, nil } +func (f *fakeMigrator) Cleanup(models.Track, string) error { return nil } + +// MoveDownload moves the recommended track for real, as a downloader would, so +// the monitor has a landing directory to hand the rest of the release. +func (f *fakeMigrator) MoveDownload(srcDir, destDir, trackPath string, track *models.Track) (string, error) { + return moveTrack(filepath.Join(srcDir, trackPath, track.File), destDir, track, "", false) +} + +func (f *fakeMigrator) MoveAlbumSiblings(trackDir, destDir string, track *models.Track) int { + f.calls++ + f.lastSource, f.lastDest = trackDir, destDir + + f.remaining -= f.perPass + if f.remaining < 0 { + f.remaining = 0 + } + return f.remaining +} + +func newClient(t *testing.T, m *fakeMigrator) *DownloadClient { + t.Helper() + + return &DownloadClient{ + Cfg: &cfg.DownloadConfig{}, + Downloaders: []Downloader{m}, + } +} + +// A release whose files keep arriving must be revisited until none are left, +// and its download directory cleaned up once it is done. +func TestMigratePendingAlbums_RetriesUntilComplete(t *testing.T) { + migrator := &fakeMigrator{remaining: 3, perPass: 1} + client := newClient(t, migrator) + + trackDir := t.TempDir() + monCfg := MonitorConfig{StallDuration: time.Hour, MaxDuration: 24 * time.Hour} + + pending := map[string]*pendingAlbum{} + client.trackAlbum(pending, "key", &pendingAlbum{ + track: &models.Track{Album: "When We All Fall Asleep"}, + trackDir: trackDir, + destDir: t.TempDir(), + startedAt: time.Now(), + lastChange: time.Now(), + }, monCfg) + + if len(pending) != 1 { + t.Fatal("release with outstanding files was not recorded for retry") + } + + for i := 0; i < 3; i++ { + client.migratePendingAlbums(pending, monCfg, time.Now()) + } + + if len(pending) != 0 { + t.Errorf("release still pending after every file arrived (%d passes)", migrator.calls) + } + if _, err := os.Stat(trackDir); !os.IsNotExist(err) { + t.Errorf("emptied release directory was not cleaned up: %v", err) + } +} + +// Progress must reset the clock, or a slow release is abandoned mid-transfer. +func TestMigratePendingAlbums_ProgressExtendsTheDeadline(t *testing.T) { + migrator := &fakeMigrator{remaining: 10, perPass: 1} + client := newClient(t, migrator) + + start := time.Now() + monCfg := MonitorConfig{StallDuration: time.Minute, MaxDuration: 24 * time.Hour} + + album := &pendingAlbum{ + track: &models.Track{Album: "When We All Fall Asleep"}, + trackDir: t.TempDir(), + destDir: t.TempDir(), + startedAt: start, + lastChange: start, + } + pending := map[string]*pendingAlbum{} + client.trackAlbum(pending, "key", album, monCfg) + + // Well past the deadline, but a file lands on every pass. + client.migratePendingAlbums(pending, monCfg, start.Add(10*time.Minute)) + + if len(pending) != 1 { + t.Fatal("a release still receiving files was abandoned") + } + if !album.lastChange.After(start) { + t.Error("progress did not reset the deadline") + } +} + +// A peer that goes away must not hold the monitor open forever. +func TestMigratePendingAlbums_GivesUpWhenStalled(t *testing.T) { + migrator := &fakeMigrator{remaining: 5, perPass: 0} + client := newClient(t, migrator) + + start := time.Now() + trackDir := t.TempDir() + if err := os.WriteFile(filepath.Join(trackDir, "half.flac"), []byte("x"), 0o644); err != nil { + t.Fatalf("seeding stalled file: %v", err) + } + + monCfg := MonitorConfig{StallDuration: time.Minute, MaxDuration: 24 * time.Hour} + + pending := map[string]*pendingAlbum{} + client.trackAlbum(pending, "key", &pendingAlbum{ + track: &models.Track{Album: "When We All Fall Asleep"}, + trackDir: trackDir, + destDir: t.TempDir(), + startedAt: start, + lastChange: start, + }, monCfg) + + client.migratePendingAlbums(pending, monCfg, start.Add(30*time.Second)) + if len(pending) != 1 { + t.Fatal("release abandoned before the deadline") + } + + client.migratePendingAlbums(pending, monCfg, start.Add(2*time.Minute)) + if len(pending) != 0 { + t.Error("a stalled release was never abandoned; the monitor would never finish") + } + if _, err := os.Stat(filepath.Join(trackDir, "half.flac")); err != nil { + t.Errorf("abandoned files should be left in place, not deleted: %v", err) + } +} + +// A release that keeps trickling in resets the stall clock every pass, so only +// the overall cap stops it from holding the monitor open indefinitely. +func TestMigratePendingAlbums_GivesUpPastTheMaxDuration(t *testing.T) { + migrator := &fakeMigrator{remaining: 100, perPass: 1} + client := newClient(t, migrator) + + start := time.Now() + monCfg := MonitorConfig{StallDuration: time.Hour, MaxDuration: time.Hour} + + pending := map[string]*pendingAlbum{} + client.trackAlbum(pending, "key", &pendingAlbum{ + track: &models.Track{Album: "When We All Fall Asleep"}, + trackDir: t.TempDir(), + destDir: t.TempDir(), + startedAt: start, + lastChange: start, + }, monCfg) + + client.migratePendingAlbums(pending, monCfg, start.Add(30*time.Minute)) + if len(pending) != 1 { + t.Fatal("release abandoned before the max duration") + } + + // Still making progress, but past the cap. + client.migratePendingAlbums(pending, monCfg, start.Add(2*time.Hour)) + if len(pending) != 0 { + t.Error("a release still trickling in was never abandoned at the max duration") + } +} + +// A single-track download has no siblings, so it must never be recorded. +func TestTrackAlbum_IgnoresReleasesWithNothingLeft(t *testing.T) { + client := newClient(t, &fakeMigrator{remaining: 0, perPass: 0}) + + pending := map[string]*pendingAlbum{} + client.trackAlbum(pending, "key", &pendingAlbum{ + track: &models.Track{Album: "Single"}, + trackDir: t.TempDir(), + destDir: t.TempDir(), + }, MonitorConfig{}) + + if len(pending) != 0 { + t.Error("a release with no outstanding files was recorded for retry") + } +} + +// monitorStub drives MonitorDownloads: the tracked file is reported complete +// immediately, while the release behind it still has files in flight. +type monitorStub struct { + fakeMigrator + cfg MonitorConfig +} + +func (m *monitorStub) GetConf() (MonitorConfig, error) { return m.cfg, nil } + +func (m *monitorStub) GetDownloadStatus(tracks []*models.Track) (map[string]FileStatus, error) { + statuses := make(map[string]FileStatus, len(tracks)) + for _, track := range tracks { + statuses[track.ID] = FileStatus{ID: "1", Filename: track.File, State: "Completed, Succeeded", PercentComplete: 100} + } + return statuses, nil +} + +// The regression in full: MonitorDownloads must not finish while a release +// still has files coming. Before this, it migrated once at the moment the +// recommended track landed and then returned, stranding the rest of the album. +func TestMonitorDownloads_WaitsForTheRestOfTheRelease(t *testing.T) { + fromDir := t.TempDir() + toDir := t.TempDir() + + releaseDir := filepath.Join(fromDir, "release") + if err := os.MkdirAll(releaseDir, 0o755); err != nil { + t.Fatalf("creating release dir: %v", err) + } + if err := os.WriteFile(filepath.Join(releaseDir, "primary.flac"), []byte("primary"), 0o644); err != nil { + t.Fatalf("seeding primary: %v", err) + } + + stub := &monitorStub{ + fakeMigrator: fakeMigrator{remaining: 3, perPass: 1}, + cfg: MonitorConfig{ + CheckInterval: time.Millisecond, + StallDuration: time.Hour, + MaxDuration: 24 * time.Hour, + MigrateDownload: true, + FromDir: fromDir, + ToDir: toDir, + Service: "slskd", + }, + } + + client := &DownloadClient{ + Cfg: &cfg.DownloadConfig{}, + Downloaders: []Downloader{stub}, + } + + track := &models.Track{ + ID: "track-1", + Album: "When We All Fall Asleep", + CleanTitle: "Bad Guy", + File: `release\primary.flac`, + } + + done := make(chan error, 1) + go func() { done <- client.MonitorDownloads([]*models.Track{track}, stub) }() + + select { + case err := <-done: + if err != nil { + t.Fatalf("MonitorDownloads() = %v, want nil", err) + } + case <-time.After(5 * time.Second): + t.Fatal("MonitorDownloads never returned") + } + + // One pass drains one file, so it must have come back for the other two + // rather than migrating once and returning. + if stub.calls < 3 { + t.Errorf("migrator called %d times; the monitor stopped before the release finished", stub.calls) + } + if stub.remaining != 0 { + t.Errorf("%d files were left behind", stub.remaining) + } +} diff --git a/src/downloader/monitor_fallback_test.go b/src/downloader/monitor_fallback_test.go new file mode 100644 index 00000000..71d7402d --- /dev/null +++ b/src/downloader/monitor_fallback_test.go @@ -0,0 +1,115 @@ +package downloader + +import ( + "testing" + "time" + + cfg "explo/src/config" + "explo/src/models" +) + +// fallbackStub always reports progress but never completes, unless +// completeAfter is set, in which case it completes after that many polls. +type fallbackStub struct { + conf MonitorConfig + completeAfter int + polls int + cleanups int +} + +func (s *fallbackStub) GetConf() (MonitorConfig, error) { return s.conf, nil } +func (s *fallbackStub) Cleanup(models.Track, string) error { + s.cleanups++ + return nil +} +func (s *fallbackStub) MoveDownload(string, string, string, *models.Track) (string, error) { + return "", nil +} + +func (s *fallbackStub) GetDownloadStatus(tracks []*models.Track) (map[string]FileStatus, error) { + s.polls++ + statuses := make(map[string]FileStatus, len(tracks)) + for _, track := range tracks { + if s.completeAfter > 0 && s.polls > s.completeAfter { + statuses[track.ID] = FileStatus{ID: "1", Filename: track.File, State: "Completed, Succeeded", PercentComplete: 100, QueueID: "q1"} + continue + } + statuses[track.ID] = FileStatus{ID: "1", Filename: track.File, State: "InProgress", BytesTransferred: s.polls, BytesRemaining: 100, QueueID: "q1"} + } + return statuses, nil +} + +func runFallbackMonitor(t *testing.T, stub *fallbackStub, track *models.Track) error { + t.Helper() + client := &DownloadClient{Cfg: &cfg.DownloadConfig{}} + done := make(chan error, 1) + go func() { done <- client.MonitorDownloads([]*models.Track{track}, stub) }() + select { + case err := <-done: + return err + case <-time.After(5 * time.Second): + t.Fatal("MonitorDownloads never returned") + return nil + } +} + +func TestMonitorDownloads_FallbackAfterAbandonsUnresolvedTracks(t *testing.T) { + stub := &fallbackStub{conf: MonitorConfig{ + CheckInterval: time.Millisecond, + StallDuration: time.Hour, + MaxDuration: 24 * time.Hour, + FallbackAfter: 20 * time.Millisecond, + Service: "lidarr", + }} + track := &models.Track{ID: "track-1", File: "song.flac"} + + if err := runFallbackMonitor(t, stub, track); err != nil { + t.Fatalf("MonitorDownloads() = %v, want nil so the next provider can try", err) + } + if track.Present { + t.Error("unfinished track was marked present") + } + if stub.cleanups == 0 { + t.Error("abandoned track was never cleaned up; the provider would keep downloading in the background") + } +} + +func TestMonitorDownloads_FallbackAfterZeroWaitsForCompletion(t *testing.T) { + stub := &fallbackStub{completeAfter: 3, conf: MonitorConfig{ + CheckInterval: time.Millisecond, + StallDuration: time.Hour, + MaxDuration: 24 * time.Hour, + FallbackAfter: 0, + Service: "lidarr", + }} + track := &models.Track{ID: "track-1", File: "song.flac"} + + if err := runFallbackMonitor(t, stub, track); err != nil { + t.Fatalf("MonitorDownloads() = %v, want nil", err) + } + if !track.Present { + t.Error("completed track was not marked present with fallback disabled") + } +} + +// BUG-18: a trickling download moves bytes every poll, so the progress +// branch must not bypass the MaxDuration deadline. +func TestMonitorDownloads_TricklingTrackHitsMaxDuration(t *testing.T) { + stub := &fallbackStub{conf: MonitorConfig{ + CheckInterval: time.Millisecond, + StallDuration: time.Hour, + MaxDuration: 20 * time.Millisecond, + Service: "slskd", + }} + track := &models.Track{ID: "track-1", File: "song.flac"} + + if err := runFallbackMonitor(t, stub, track); err != nil { + t.Fatalf("MonitorDownloads() = %v, want nil", err) + } + if track.Present { + t.Error("trickling track was marked present after MaxDuration") + } + if stub.cleanups == 0 { + t.Error("timed-out track was never cleaned up") + } +} diff --git a/src/downloader/monitor_poll_test.go b/src/downloader/monitor_poll_test.go new file mode 100644 index 00000000..49963c44 --- /dev/null +++ b/src/downloader/monitor_poll_test.go @@ -0,0 +1,189 @@ +package downloader + +import ( + "fmt" + "net/http" + "testing" + "time" + + cfg "explo/src/config" + "explo/src/models" +) + +// pollStub drives MonitorDownloads with a scripted sequence of poll outcomes, +// so a test can interleave failures with successes. +type pollStub struct { + cfg MonitorConfig + // results is consumed one entry per poll; the last entry repeats. + results []pollResult + polls int +} + +type pollResult struct { + err error + complete bool +} + +func (p *pollStub) QueryTrack(*models.Track) error { return nil } +func (p *pollStub) GetTrack(*models.Track) error { return nil } +func (p *pollStub) GetConf() (MonitorConfig, error) { return p.cfg, nil } +func (p *pollStub) Cleanup(models.Track, string) error { return nil } +func (p *pollStub) MoveDownload(string, string, string, *models.Track) (string, error) { + return "", nil +} + +func (p *pollStub) GetDownloadStatus(tracks []*models.Track) (map[string]FileStatus, error) { + result := p.results[min(p.polls, len(p.results)-1)] + p.polls++ + + if result.err != nil { + return nil, result.err + } + + statuses := make(map[string]FileStatus, len(tracks)) + for _, track := range tracks { + if result.complete { + statuses[track.ID] = FileStatus{ID: "1", Filename: track.File, State: "Completed, Succeeded", PercentComplete: 100} + continue + } + statuses[track.ID] = FileStatus{ID: "1", Filename: track.File, State: "InProgress", BytesTransferred: p.polls, BytesRemaining: 100} + } + return statuses, nil +} + +func runMonitor(t *testing.T, stub *pollStub, track *models.Track) error { + t.Helper() + + client := &DownloadClient{Cfg: &cfg.DownloadConfig{}} + + done := make(chan error, 1) + go func() { done <- client.MonitorDownloads([]*models.Track{track}, stub) }() + + select { + case err := <-done: + return err + case <-time.After(5 * time.Second): + t.Fatal("MonitorDownloads never returned") + return nil + } +} + +func monitorCfg() MonitorConfig { + return MonitorConfig{ + CheckInterval: time.Millisecond, + StallDuration: time.Hour, + MaxDuration: 24 * time.Hour, + Service: "slskd", + } +} + +// A blip must not cost the playlist. Tracks that never reach Present are +// dropped before the playlist is built, so aborting the monitor on one failed +// poll used to throw away every download still in flight. +func TestMonitorDownloads_SurvivesATransientPollFailure(t *testing.T) { + stub := &pollStub{ + cfg: monitorCfg(), + results: []pollResult{ + {err: fmt.Errorf("connection refused")}, + {complete: true}, + }, + } + + track := &models.Track{ID: "track-1", File: "song.flac"} + + if err := runMonitor(t, stub, track); err != nil { + t.Fatalf("MonitorDownloads() = %v, want nil after recovering", err) + } + if !track.Present { + t.Error("track was abandoned after a single failed poll; it would be dropped from the playlist") + } +} + +// Recovery must reset the budget, or a long run accumulates unrelated blips +// until it trips the limit. +func TestMonitorDownloads_ResetsTheFailureBudgetOnSuccess(t *testing.T) { + stub := &pollStub{ + cfg: monitorCfg(), + results: []pollResult{ + {err: fmt.Errorf("blip")}, + {err: fmt.Errorf("blip")}, + {}, // recovered, still downloading + {err: fmt.Errorf("blip")}, + {err: fmt.Errorf("blip")}, + {complete: true}, + }, + } + + track := &models.Track{ID: "track-1", File: "song.flac"} + + if err := runMonitor(t, stub, track); err != nil { + t.Fatalf("MonitorDownloads() = %v, want nil -- four failures spread around a success is not four in a row", err) + } + if !track.Present { + t.Error("track never completed") + } +} + +// A service that is genuinely gone must still end the run rather than spin. +func TestMonitorDownloads_GivesUpOnRepeatedPollFailures(t *testing.T) { + stub := &pollStub{ + cfg: monitorCfg(), + results: []pollResult{{err: fmt.Errorf("connection refused")}}, + } + + err := runMonitor(t, stub, &models.Track{ID: "track-1", File: "song.flac"}) + if err == nil { + t.Fatal("MonitorDownloads() = nil, want an error once the service stays unreachable") + } + if stub.polls != maxPollFailures { + t.Errorf("polled %d times, want %d before giving up", stub.polls, maxPollFailures) + } +} + +// An empty status map means slskd has not surfaced the transfers yet. Treating +// that as an error aborted the whole run one minute after queueing. +func TestMonitorDownloads_EmptyStatusIsNotAFailure(t *testing.T) { + stub := &pollStub{ + cfg: monitorCfg(), + results: []pollResult{{}, {complete: true}}, + } + + // No tracks: GetDownloadStatus returns an empty map, exactly as slskd does + // before any transfer appears. + track := &models.Track{ID: "track-1", File: "song.flac"} + if err := runMonitor(t, stub, track); err != nil { + t.Fatalf("MonitorDownloads() = %v, want nil", err) + } + if !track.Present { + t.Error("track never completed") + } +} + +// The slskd side of the same bug: an empty transfer list is a normal state a +// minute after queueing, not a failure to report upward. +func TestSlskdGetDownloadStatus_EmptyListIsNotAnError(t *testing.T) { + client := albumClient(t, func(w http.ResponseWriter, r *http.Request) { + if _, err := w.Write([]byte(`[]`)); err != nil { + t.Errorf("fake slskd failed to write response: %v", err) + } + }) + + statuses, err := client.GetDownloadStatus([]*models.Track{{File: "song.flac", MainArtistID: "peer1"}}) + if err != nil { + t.Fatalf("GetDownloadStatus() = %v, want nil -- nothing queued yet is not a failure", err) + } + if len(statuses) != 0 { + t.Errorf("got %d statuses, want none", len(statuses)) + } +} + +// A genuinely broken request must still be reported. +func TestSlskdGetDownloadStatus_ReportsRealFailures(t *testing.T) { + client := albumClient(t, func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusInternalServerError) + }) + + if _, err := client.GetDownloadStatus([]*models.Track{{File: "song.flac"}}); err == nil { + t.Error("GetDownloadStatus() = nil, want an error when slskd fails") + } +} diff --git a/src/downloader/monitor_present_test.go b/src/downloader/monitor_present_test.go new file mode 100644 index 00000000..301bfe5b --- /dev/null +++ b/src/downloader/monitor_present_test.go @@ -0,0 +1,84 @@ +package downloader + +import ( + "fmt" + "testing" + "time" + + cfg "explo/src/config" + "explo/src/models" +) + +// moveStub completes every download immediately; its MoveDownload can be +// made to fail. +type moveStub struct { + conf MonitorConfig + moveErr error +} + +func (s *moveStub) GetConf() (MonitorConfig, error) { return s.conf, nil } +func (s *moveStub) Cleanup(models.Track, string) error { return nil } +func (s *moveStub) MoveDownload(string, string, string, *models.Track) (string, error) { + if s.moveErr != nil { + return "", s.moveErr + } + return "/library/album", nil +} + +func (s *moveStub) GetDownloadStatus(tracks []*models.Track) (map[string]FileStatus, error) { + statuses := make(map[string]FileStatus, len(tracks)) + for _, track := range tracks { + statuses[track.ID] = FileStatus{ID: "1", Filename: track.File, State: "Completed, Succeeded", PercentComplete: 100, QueueID: "q1"} + } + return statuses, nil +} + +func runMoveMonitor(t *testing.T, stub *moveStub, track *models.Track) { + t.Helper() + client := &DownloadClient{Cfg: &cfg.DownloadConfig{}} + done := make(chan error, 1) + go func() { done <- client.MonitorDownloads([]*models.Track{track}, stub) }() + select { + case err := <-done: + if err != nil { + t.Fatalf("MonitorDownloads() = %v, want nil", err) + } + case <-time.After(5 * time.Second): + t.Fatal("MonitorDownloads never returned") + } +} + +func migrateCfg() MonitorConfig { + return MonitorConfig{ + CheckInterval: time.Millisecond, + StallDuration: time.Hour, + MaxDuration: 24 * time.Hour, + MigrateDownload: true, + Service: "lidarr", + } +} + +// BUG-11: the track was marked Present and counted before MoveDownload +// ran, so a failed move still put a song that never arrived into the +// playlist. +func TestMonitorDownloads_FailedMoveDoesNotMarkPresent(t *testing.T) { + stub := &moveStub{conf: migrateCfg(), moveErr: fmt.Errorf("disk full")} + track := &models.Track{ID: "track-1", File: "song.flac"} + + runMoveMonitor(t, stub, track) + + if track.Present { + t.Error("track marked Present even though the move failed; it would enter the playlist without existing") + } +} + +func TestMonitorDownloads_SuccessfulMoveMarksPresent(t *testing.T) { + stub := &moveStub{conf: migrateCfg()} + track := &models.Track{ID: "track-1", File: "song.flac"} + + runMoveMonitor(t, stub, track) + + if !track.Present { + t.Error("track not Present after a successful move") + } +} diff --git a/src/downloader/movefile_test.go b/src/downloader/movefile_test.go new file mode 100644 index 00000000..4baadb4b --- /dev/null +++ b/src/downloader/movefile_test.go @@ -0,0 +1,77 @@ +package downloader + +import ( + "os" + "path/filepath" + "testing" +) + +// BUG-07: moveFile truncated the destination with os.Create before +// copying, so src == dst zeroed the only copy and a name collision +// silently erased an existing library song. + +func writeTestFile(t *testing.T, path, content string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0644); err != nil { + t.Fatal(err) + } +} + +func readTestFile(t *testing.T, path string) string { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + return string(data) +} + +func TestMoveFile_SameSourceAndDestinationKeepsFile(t *testing.T) { + path := filepath.Join(t.TempDir(), "song.flac") + writeTestFile(t, path, "audio") + + if err := moveFile(path, path, false); err != nil { + t.Fatalf("moveFile(same, same) = %v, want nil (already in place)", err) + } + if got := readTestFile(t, path); got != "audio" { + t.Errorf("file content = %q, want %q — the file was truncated", got, "audio") + } +} + +func TestMoveFile_ExistingDestinationIsNotErased(t *testing.T) { + dir := t.TempDir() + src := filepath.Join(dir, "downloads", "song.flac") + dst := filepath.Join(dir, "library", "song.flac") + writeTestFile(t, src, "new download") + writeTestFile(t, dst, "existing library song") + + if err := moveFile(src, dst, false); err == nil { + t.Fatal("moveFile onto an existing destination = nil, want refusal") + } + if got := readTestFile(t, dst); got != "existing library song" { + t.Errorf("destination = %q, want the existing song untouched", got) + } + if got := readTestFile(t, src); got != "new download" { + t.Errorf("source = %q, want the download kept after refusal", got) + } +} + +func TestMoveFile_NormalMoveStillWorks(t *testing.T) { + dir := t.TempDir() + src := filepath.Join(dir, "downloads", "song.flac") + dst := filepath.Join(dir, "library", "song.flac") + writeTestFile(t, src, "audio") + + if err := moveFile(src, dst, false); err != nil { + t.Fatalf("moveFile normal: %v", err) + } + if got := readTestFile(t, dst); got != "audio" { + t.Errorf("destination = %q, want audio", got) + } + if _, err := os.Stat(src); !os.IsNotExist(err) { + t.Errorf("source still present after move: %v", err) + } +} diff --git a/src/downloader/original_release.go b/src/downloader/original_release.go new file mode 100644 index 00000000..c50f96ba --- /dev/null +++ b/src/downloader/original_release.go @@ -0,0 +1,331 @@ +package downloader + +import ( + "log/slog" + "path" + "regexp" + "slices" + "strconv" + "strings" + + "explo/src/models" +) + +// Preferring the original release is a trim, not a rejection. A peer sharing a +// six-disc anthology usually has the album we want sitting inside it, so +// reducing the candidate to that album beats skipping the peer and beats +// downloading all six discs. +// +// The trim runs before scoring, which is the point: a 39-file box set and a +// 16-file pressing of the same album are not comparable until both have been +// reduced to the album, and fullerReleaseScore would otherwise hand the box set +// the win on bulk alone. + +// discTrackPattern matches a flattened "disc-track" prefix such as 01-04 or +// 1-09. The leading boundary rejects a digit, so a year like 1975-04 cannot be +// read as disc 19 or disc 5. +var discTrackPattern = regexp.MustCompile(`(?:^|[^0-9])([0-9]{1,2})-([0-9]{2})(?:[^0-9]|$)`) + +// packedDiscTrackPattern matches the four-digit DDTT form, as in "0101 - Colors". +// The disc half is capped at 09 so that a leading year (1999, 2015) is not read +// as disc 19 or disc 20. +var packedDiscTrackPattern = regexp.MustCompile(`^(0[1-9])([0-9]{2})(?:[^0-9]|$)`) + +// leadingTrackPattern matches a plain track number: "02 - Sign of the Times". +var leadingTrackPattern = regexp.MustCompile(`^([0-9]{1,2})(?:[^0-9]|$)`) + +// underscoreTrackPattern matches the track number in an underscore-joined name, +// as in "Hayley Williams_Ego Death_18_Parachute". Limited to one or two digits +// so an embedded year cannot match, and tried after the disc-aware patterns so +// that "..._01-04_Chocolate" is read as disc 1 rather than as track 1. +var underscoreTrackPattern = regexp.MustCompile(`_([0-9]{1,2})_`) + +// yearPattern finds a release year in a directory name. +var yearPattern = regexp.MustCompile(`(?:^|[^0-9])((?:19|20)[0-9]{2})(?:[^0-9]|$)`) + +// originalYearPenalty is how much a directory loses for naming a year later +// than the album's original release. Deliberately small: it only separates +// releases that already survived the trim as equals, which is the "then +// earliest" half of preferring the original. +const originalYearPenalty = 8 + +// fileNumbering is what a peer's filename says about where the file sits in the +// release. disc is 0 when the name carries no disc marker at all, which is the +// normal case for a single-disc directory. +type fileNumbering struct { + disc int + track int + ok bool +} + +// parseNumbering reads the disc and track position out of a filename. Peers +// name files every way imaginable, so this recognises the three forms that +// actually show up and gives up on anything else rather than guessing. +func parseNumbering(name string) fileNumbering { + base := path.Base(normalizePeerPath(name)) + base = strings.TrimSuffix(base, path.Ext(base)) + + if m := packedDiscTrackPattern.FindStringSubmatch(base); m != nil { + disc, _ := strconv.Atoi(m[1]) + trackNo, _ := strconv.Atoi(m[2]) + if trackNo > 0 { + return fileNumbering{disc: disc, track: trackNo, ok: true} + } + } + + if m := discTrackPattern.FindStringSubmatch(base); m != nil { + disc, _ := strconv.Atoi(m[1]) + trackNo, _ := strconv.Atoi(m[2]) + if disc > 0 && trackNo > 0 { + return fileNumbering{disc: disc, track: trackNo, ok: true} + } + } + + if m := leadingTrackPattern.FindStringSubmatch(base); m != nil { + trackNo, _ := strconv.Atoi(m[1]) + if trackNo > 0 { + return fileNumbering{track: trackNo, ok: true} + } + } + + if m := underscoreTrackPattern.FindStringSubmatch(base); m != nil { + trackNo, _ := strconv.Atoi(m[1]) + if trackNo > 0 { + return fileNumbering{track: trackNo, ok: true} + } + } + + return fileNumbering{} +} + +// minParsedShare is how much of a directory must yield a usable track number +// before the trim will act on it. Below this the directory is named in some way +// we do not understand, and trimming would be cutting at random -- a peer +// sharing a literal "{track-number:02d} - Cowgirl.mp3" is not hypothetical. +const minParsedShare = 2.0 / 3.0 + +// allowedDiscs is how many discs the release may legitimately span. +// +// The release group's consensus outranks the matched release. A recommendation +// that resolved to a two-disc deluxe reports DiscTotal 2, and trusting that +// would keep the padded directory whole -- the exact failure this feature +// exists to prevent. What the album's other editions agree on is the better +// answer, and a genuine double album has a group that agrees on two. +// +// With no consensus the matched release is used, and with neither the +// assumption is one disc: a peer flattening several discs into a single folder +// is far more common than an unenriched genuine double album. +func allowedDiscs(track models.Track) int { + if track.CanonicalDiscTotal > 0 { + return track.CanonicalDiscTotal + } + if track.DiscTotal > 1 { + return track.DiscTotal + } + return 1 +} + +// expectedAlbumLength is how many tracks the original album holds, preferring +// the group's consensus over the matched release for the same reason. +func expectedAlbumLength(track models.Track) int { + if track.CanonicalTrackTotal > 0 { + return track.CanonicalTrackTotal + } + return track.TrackTotal +} + +// trimToOriginal reduces a candidate release to the original album: extra discs +// first, then bonus tracks appended past the album's real length. +// +// The primary is retained unconditionally. Trimming away the recommended track +// would lose the playlist entry entirely, which is a worse outcome than +// downloading one file more than intended -- and a recommendation that is +// itself a bonus track is exactly the case that would otherwise do it. +func trimToOriginal(dir *peerDir, track models.Track) { + if dir.primary == nil || len(dir.files) < 2 { + return + } + + numbering := make([]fileNumbering, len(dir.files)) + var parsed int + for i, file := range dir.files { + numbering[i] = parseNumbering(string(file.Name)) + if numbering[i].ok { + parsed++ + } + } + if float64(parsed) < float64(len(dir.files))*minParsedShare { + slog.Debug("not trimming release, numbering unreadable", "dir", dir.dir, + "parsed", parsed, "files", len(dir.files)) + return + } + // Numbering that reads cleanly but repeats itself is describing something + // other than one release -- two editions side by side, or a misparse. Either + // way the positions cannot say which files are the album, and cutting on + // them would drop the wrong ones. Leaving the release whole is the safe + // failure; this is the only place where a wrong answer is worse than none. + if dup, ok := duplicatePosition(numbering); ok { + slog.Debug("not trimming release, numbering repeats", "dir", dir.dir, + "disc", dup.disc, "track", dup.track) + return + } + + primaryName := dir.primary.Name + keep := make([]bool, len(dir.files)) + for i := range keep { + keep[i] = true + } + + discs := trimExtraDiscs(dir, numbering, keep, track) + + // The bonus-track pass is skipped whenever more than one disc survives: + // TrackTotal counts the first medium only, so measuring a kept two-disc + // release against it would amputate the second disc for being there. + if discs <= 1 { + trimBonusTracks(dir, numbering, keep, track) + } + + kept := make([]File, 0, len(dir.files)) + for i, file := range dir.files { + if keep[i] || file.Name == primaryName { + kept = append(kept, file) + } + } + if len(kept) == len(dir.files) { + return + } + + // Info, not Debug: this changes what gets downloaded, so it belongs in the + // run log beside "album release selected" rather than being invisible. + slog.Info("trimmed release to the original album", + "dir", dir.dir, "from", len(dir.files), "to", len(kept), "discs kept", discs) + + dir.files = kept + // dir.primary points into the old backing array, so it has to be re-seated + // or every later use of it reads a file that is no longer in the release. + dir.primary = nil + for i := range dir.files { + if dir.files[i].Name == primaryName { + dir.primary = &dir.files[i] + break + } + } +} + +// duplicatePosition reports the first (disc, track) position claimed by more +// than one file, if any. +func duplicatePosition(numbering []fileNumbering) (fileNumbering, bool) { + type position struct{ disc, track int } + + seen := make(map[position]bool, len(numbering)) + for _, n := range numbering { + if !n.ok { + continue + } + at := position{n.disc, n.track} + if seen[at] { + return n, true + } + seen[at] = true + } + return fileNumbering{}, false +} + +// trimExtraDiscs drops discs beyond what the release may legitimately span, +// keeping the lowest-numbered ones. It reports how many discs survive. +func trimExtraDiscs(dir *peerDir, numbering []fileNumbering, keep []bool, track models.Track) int { + present := make([]int, 0, 4) + for _, n := range numbering { + if n.ok && n.disc > 0 && !slices.Contains(present, n.disc) { + present = append(present, n.disc) + } + } + if len(present) <= 1 { + return len(present) + } + + slices.Sort(present) + allowed := allowedDiscs(track) + if len(present) <= allowed { + return len(present) + } + + wanted := present[:allowed] + for i, n := range numbering { + if n.ok && n.disc > 0 && !slices.Contains(wanted, n.disc) { + keep[i] = false + } + } + return allowed +} + +// trimBonusTracks drops whatever sits past the album's real track count. +// +// Bonus material is appended after the album by near-universal convention, so +// the lowest TrackTotal positions are the album. Files with no readable track +// number sort last and are therefore the first to go. +func trimBonusTracks(dir *peerDir, numbering []fileNumbering, keep []bool, track models.Track) { + expected := expectedAlbumLength(track) + if expected <= 0 { + return + } + + order := make([]int, 0, len(dir.files)) + for i := range dir.files { + if keep[i] { + order = append(order, i) + } + } + if len(order) <= expected { + return + } + + slices.SortStableFunc(order, func(a, b int) int { + na, nb := numbering[a], numbering[b] + switch { + case na.ok != nb.ok: + if na.ok { + return -1 + } + return 1 + case na.track != nb.track: + return na.track - nb.track + default: + return 0 + } + }) + + for _, i := range order[expected:] { + keep[i] = false + } +} + +// laterThanOriginal reports whether a directory names a release year after the +// album's original one -- a remaster or reissue carrying the same tracklist, +// which the trim cannot tell apart from the original by file count alone. +// +// Only the final path segment is read: the parent folders belong to the peer's +// library layout, and a year in one of those says nothing about this release. +func laterThanOriginal(dir string, track models.Track) bool { + if track.OriginalYear <= 0 { + return false + } + + matches := yearPattern.FindAllStringSubmatch(path.Base(normalizePeerPath(dir)), -1) + if len(matches) == 0 { + return false + } + + earliest := 0 + for _, m := range matches { + year, err := strconv.Atoi(m[1]) + if err != nil { + continue + } + if earliest == 0 || year < earliest { + earliest = year + } + } + + return earliest > track.OriginalYear +} diff --git a/src/downloader/original_release_test.go b/src/downloader/original_release_test.go new file mode 100644 index 00000000..3e42d280 --- /dev/null +++ b/src/downloader/original_release_test.go @@ -0,0 +1,386 @@ +package downloader + +import ( + "strings" + "testing" + + cfg "explo/src/config" + "explo/src/models" +) + +// numberedDir builds a candidate release from filenames, with the primary +// pointed at whichever file names the recommended track. +func numberedDir(primaryTitle string, names ...string) peerDir { + dir := peerDir{dir: "Billie Eilish - When We All Fall Asleep"} + for _, name := range names { + dir.files = append(dir.files, File{ + Name: name, + Extension: "flac", + BitRate: 900, + }) + } + for i := range dir.files { + if strings.Contains(string(dir.files[i].Name), primaryTitle) { + dir.primary = &dir.files[i] + break + } + } + return dir +} + +func fileNames(dir peerDir) []string { + names := make([]string, 0, len(dir.files)) + for _, f := range dir.files { + names = append(names, string(f.Name)) + } + return names +} + +func TestParseNumbering(t *testing.T) { + cases := []struct { + name string + disc int + track int + ok bool + }{ + // The four shapes that actually turned up in production logs. + {"The 1975_The 1975_01-04_Chocolate.mp3", 1, 4, true}, + {"1-09 In My Room.flac", 1, 9, true}, + {"0101 - Colors.flac", 1, 1, true}, + {"02 - Sign of the Times.mp3", 0, 2, true}, + + {"Hayley Williams_Ego Death_18_Parachute.flac", 0, 18, true}, + {"05-gracie_abrams-look_at_my_life.mp3", 0, 5, true}, + + // A year must never be read as a disc marker. + {"1975-04 Chocolate.mp3", 0, 0, false}, + {"2015 - Badlands.flac", 0, 0, false}, + + // Nothing parseable: a peer whose naming template never expanded. + {"{track-number:02d} - Cowgirl.mp3", 0, 0, false}, + {"Sundown-HUNTR.flac", 0, 0, false}, + } + + for _, tc := range cases { + got := parseNumbering(tc.name) + if got.ok != tc.ok || got.disc != tc.disc || got.track != tc.track { + t.Errorf("parseNumbering(%q) = disc %d track %d ok %v, want disc %d track %d ok %v", + tc.name, got.disc, got.track, got.ok, tc.disc, tc.track, tc.ok) + } + } +} + +// The case that prompted this: a peer flattening several discs into one folder, +// which then outscored every honest copy of the album on file count alone. +func TestTrimToOriginal_DropsFlattenedExtraDiscs(t *testing.T) { + dir := numberedDir("Bad Guy", + "01-01 Bury A Friend.flac", "01-02 Bad Guy.flac", "01-03 Xanny.flac", + "02-01 Demo Take.flac", "02-02 Alternate Mix.flac", + ) + + trimToOriginal(&dir, models.Track{CleanTitle: "Bad Guy", TrackTotal: 3, DiscTotal: 1}) + + if len(dir.files) != 3 { + t.Fatalf("kept %d files %v, want the 3 on disc one", len(dir.files), fileNames(dir)) + } + for _, name := range fileNames(dir) { + if strings.HasPrefix(name, "02-") { + t.Errorf("kept %q from the second disc", name) + } + } +} + +// A real double album is not bloat. MusicBrainz is the arbiter, and when it +// says two discs, both stay. +func TestTrimToOriginal_KeepsARealDoubleAlbum(t *testing.T) { + dir := numberedDir("Bad Guy", + "01-01 Bury A Friend.flac", "01-02 Bad Guy.flac", + "02-01 Xanny.flac", "02-02 Ilomilo.flac", + ) + + trimToOriginal(&dir, models.Track{CleanTitle: "Bad Guy", TrackTotal: 2, DiscTotal: 2}) + + if len(dir.files) != 4 { + t.Fatalf("kept %d files %v, want all 4 of a genuine 2-disc release", + len(dir.files), fileNames(dir)) + } +} + +// With no MusicBrainz answer the assumption is one disc: a peer flattening +// discs together is far more common than an unenriched double album. +func TestTrimToOriginal_TrimsToDiscOneWhenMusicBrainzIsSilent(t *testing.T) { + dir := numberedDir("Bad Guy", + "01-01 Bury A Friend.flac", "01-02 Bad Guy.flac", + "02-01 Xanny.flac", "02-02 Ilomilo.flac", + ) + + trimToOriginal(&dir, models.Track{CleanTitle: "Bad Guy"}) + + if len(dir.files) != 2 { + t.Fatalf("kept %d files %v, want disc one only", len(dir.files), fileNames(dir)) + } +} + +// TrackTotal counts the first medium only, so measuring a legitimately kept +// two-disc release against it would amputate the second disc for existing. +func TestTrimToOriginal_SkipsTheBonusPassOnAKeptDoubleAlbum(t *testing.T) { + dir := numberedDir("Bad Guy", + "01-01 Bury A Friend.flac", "01-02 Bad Guy.flac", "01-03 Xanny.flac", + "02-01 Ilomilo.flac", "02-02 8.flac", "02-03 My Strange Addiction.flac", + ) + + trimToOriginal(&dir, models.Track{CleanTitle: "Bad Guy", TrackTotal: 3, DiscTotal: 2}) + + if len(dir.files) != 6 { + t.Fatalf("kept %d files %v, want all 6: TrackTotal describes disc one, not the release", + len(dir.files), fileNames(dir)) + } +} + +// The Halsey case: a single-disc anthology padded past the album's real length. +func TestTrimToOriginal_DropsBonusTracks(t *testing.T) { + dir := numberedDir("Bad Guy", + "01 Bury A Friend.flac", "02 Bad Guy.flac", "03 Xanny.flac", + "04 Bonus Demo.flac", "05 Bonus Live Take.flac", + ) + + trimToOriginal(&dir, models.Track{CleanTitle: "Bad Guy", TrackTotal: 3, DiscTotal: 1}) + + if len(dir.files) != 3 { + t.Fatalf("kept %d files %v, want the 3-track album", len(dir.files), fileNames(dir)) + } + for _, name := range fileNames(dir) { + if strings.Contains(name, "Bonus") { + t.Errorf("kept bonus material %q", name) + } + } +} + +// Trimming away the recommended track would lose the playlist entry, which is +// worse than downloading one file more than intended. +func TestTrimToOriginal_AlwaysKeepsThePrimary(t *testing.T) { + dir := numberedDir("Bad Guy", + "01 Bury A Friend.flac", "02 Xanny.flac", "03 Ilomilo.flac", + "04 Bonus Demo.flac", "05 Bad Guy.flac", + ) + + trimToOriginal(&dir, models.Track{CleanTitle: "Bad Guy", TrackTotal: 3, DiscTotal: 1}) + + var found bool + for _, name := range fileNames(dir) { + if strings.Contains(name, "Bad Guy") { + found = true + } + } + if !found { + t.Fatalf("trimmed away the recommended track; kept %v", fileNames(dir)) + } + if dir.primary == nil || !strings.Contains(string(dir.primary.Name), "Bad Guy") { + t.Errorf("primary = %v, want it still pointing at the recommended track", dir.primary) + } +} + +// dir.primary points into the files slice. Trimming reallocates it, so a +// pointer left behind would read a file no longer in the release. +func TestTrimToOriginal_ReseatsThePrimaryPointer(t *testing.T) { + dir := numberedDir("Bad Guy", + "01-01 Bury A Friend.flac", "01-02 Bad Guy.flac", + "02-01 Demo Take.flac", + ) + + trimToOriginal(&dir, models.Track{CleanTitle: "Bad Guy", TrackTotal: 2, DiscTotal: 1}) + + if dir.primary == nil { + t.Fatal("primary = nil after trimming") + } + var inSlice bool + for i := range dir.files { + if &dir.files[i] == dir.primary { + inSlice = true + } + } + if !inSlice { + t.Error("primary points outside the trimmed release; it was not re-seated") + } +} + +// Cutting a directory we cannot read the numbering of would be cutting at +// random. A peer sharing an unexpanded naming template is not hypothetical. +func TestTrimToOriginal_SkipsUnparseableDirectories(t *testing.T) { + dir := numberedDir("Bad Guy", + "Bury A Friend.flac", "Bad Guy.flac", "Xanny.flac", + "{track-number:02d} - Ilomilo.flac", "Goodbye.flac", + ) + + trimToOriginal(&dir, models.Track{CleanTitle: "Bad Guy", TrackTotal: 3, DiscTotal: 1}) + + if len(dir.files) != 5 { + t.Fatalf("kept %d files %v, want all 5 left alone", len(dir.files), fileNames(dir)) + } +} + +func TestLaterThanOriginal(t *testing.T) { + cases := []struct { + dir string + year int + want bool + }{ + {"Halsey/2023 - Badlands (Remastered)", 2015, true}, + {"Halsey/2015 - Badlands", 2015, false}, + {"Halsey/Badlands", 2015, false}, + {"Halsey/2015 - Badlands (Decade Edition Anthology)", 2015, false}, + {"Halsey/2023 - Badlands", 0, false}, + // A year in a parent folder describes the peer's library, not this + // release, so only the final segment is read. + {"@@x/2023 Rips/Halsey/2015 - Badlands", 2015, false}, + } + + for _, tc := range cases { + got := laterThanOriginal(tc.dir, models.Track{OriginalYear: tc.year}) + if got != tc.want { + t.Errorf("laterThanOriginal(%q, %d) = %v, want %v", tc.dir, tc.year, got, tc.want) + } + } +} + +// originalClient mirrors albumClient with the trim switched on. +func originalClient(t *testing.T, prefer bool) *Slskd { + t.Helper() + + client := NewSlskd(cfg.Slskd{ + URL: "http://localhost", + APIKey: "test-key", + Timeout: 5, + AlbumMode: true, + PreferOriginalRelease: prefer, + Filters: cfg.Filters{ + Extensions: []string{"flac", "mp3"}, + MinBitRate: 256, + MinBitDepth: 8, + FilterList: []string{"live", "remix"}, + }, + }, "/data/") + client.AddHeader() + + return client +} + +func flattenedTrack() models.Track { + track := albumTrack() + track.TrackTotal = 3 + track.DiscTotal = 1 + return track +} + +func flattenedResults() SearchResults { + return albumResults(peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, []string{ + "01-01 Bury A Friend.flac", "01-02 Bad Guy.flac", "01-03 Xanny.flac", + "02-01 Demo Take.flac", "02-02 Alternate Mix.flac", + }}) +} + +func TestCollectAlbumFiles_TrimsAFlattenedReleaseToTheAlbum(t *testing.T) { + files, err := originalClient(t, true).CollectAlbumFiles(flattenedTrack(), flattenedResults()) + if err != nil { + t.Fatalf("CollectAlbumFiles() = %v, want nil", err) + } + if len(files) != 3 { + t.Fatalf("queued %d files, want the 3 on disc one", len(files)) + } + if !strings.Contains(string(files[0].Name), "Bad Guy") { + t.Errorf("files[0] = %q, want the recommended track first", files[0].Name) + } +} + +func TestCollectAlbumFiles_LeavesReleasesAloneWhenDisabled(t *testing.T) { + files, err := originalClient(t, false).CollectAlbumFiles(flattenedTrack(), flattenedResults()) + if err != nil { + t.Fatalf("CollectAlbumFiles() = %v, want nil", err) + } + if len(files) != 5 { + t.Fatalf("queued %d files with the setting off, want all 5 untouched", len(files)) + } +} + +// The hole PR #19 left open. ListenBrainz resolved the recording to a two-disc +// deluxe, so DiscTotal says 2 and the flattened directory was kept whole -- the +// exact failure the feature exists to prevent. The group's consensus knows the +// album is single-disc. +func TestTrimToOriginal_ConsensusBeatsADeluxeMatch(t *testing.T) { + dir := numberedDir("Bad Guy", + "01-01 Bury A Friend.flac", "01-02 Bad Guy.flac", "01-03 Xanny.flac", + "02-01 Demo Take.flac", "02-02 Alternate Mix.flac", + ) + + trimToOriginal(&dir, models.Track{ + CleanTitle: "Bad Guy", + TrackTotal: 23, // the deluxe's length + DiscTotal: 2, // the deluxe's disc count + CanonicalTrackTotal: 3, + CanonicalDiscTotal: 1, + }) + + if len(dir.files) != 3 { + t.Fatalf("kept %d files %v, want the 3 on disc one", len(dir.files), fileNames(dir)) + } +} + +// A genuine double album has a group that agrees it is one, so consensus +// winning must not amputate it. +func TestTrimToOriginal_ConsensusKeepsAGenuineDoubleAlbum(t *testing.T) { + dir := numberedDir("Bad Guy", + "01-01 Bury A Friend.flac", "01-02 Bad Guy.flac", + "02-01 Xanny.flac", "02-02 Ilomilo.flac", + ) + + trimToOriginal(&dir, models.Track{ + CleanTitle: "Bad Guy", + CanonicalTrackTotal: 2, + CanonicalDiscTotal: 2, + }) + + if len(dir.files) != 4 { + t.Fatalf("kept %d files %v, want all 4", len(dir.files), fileNames(dir)) + } +} + +// Numbering that reads cleanly but repeats is describing something other than +// one release. Cutting on it would drop the wrong files. +func TestTrimToOriginal_SkipsWhenNumberingRepeats(t *testing.T) { + dir := numberedDir("Bad Guy", + "01 Bury A Friend.flac", "02 Bad Guy.flac", "03 Xanny.flac", + "01 Bury A Friend (Remaster).flac", "02 Bad Guy (Remaster).flac", + ) + + trimToOriginal(&dir, models.Track{CleanTitle: "Bad Guy", CanonicalTrackTotal: 3, CanonicalDiscTotal: 1}) + + if len(dir.files) != 5 { + t.Fatalf("kept %d files %v, want all 5 left alone", len(dir.files), fileNames(dir)) + } +} + +func TestExpectedAlbumLength_PrefersTheConsensus(t *testing.T) { + got := expectedAlbumLength(models.Track{TrackTotal: 23, CanonicalTrackTotal: 16}) + if got != 16 { + t.Errorf("expectedAlbumLength() = %d, want the consensus 16", got) + } + if got := expectedAlbumLength(models.Track{TrackTotal: 23}); got != 23 { + t.Errorf("without a consensus = %d, want the matched release's 23", got) + } +} + +// Everything above must be inert with enrichment off, leaving PR #19's +// behaviour byte for byte. +func TestTrimToOriginal_UnchangedWithoutAConsensus(t *testing.T) { + dir := numberedDir("Bad Guy", + "01-01 Bury A Friend.flac", "01-02 Bad Guy.flac", + "02-01 Xanny.flac", "02-02 Ilomilo.flac", + ) + + trimToOriginal(&dir, models.Track{CleanTitle: "Bad Guy", TrackTotal: 2, DiscTotal: 2}) + + if len(dir.files) != 4 { + t.Fatalf("kept %d files %v, want DiscTotal still honoured with no consensus", + len(dir.files), fileNames(dir)) + } +} diff --git a/src/downloader/preference.go b/src/downloader/preference.go new file mode 100644 index 00000000..21261e8c --- /dev/null +++ b/src/downloader/preference.go @@ -0,0 +1,273 @@ +package downloader + +import ( + "cmp" + + "explo/src/models" + "slices" + "strings" +) + +// Size preferences. Ranking, not filtering: everything inside the configured +// quality range stays eligible, the preference only decides what is reached +// for first. +const ( + // PreferNone leaves ordering to the order of EXTENSIONS, which is how + // Explo has always chosen, so an upgrade changes nothing. + PreferNone = "none" + // PreferSmaller takes the best lossy file available and only falls back to + // lossless when there is no lossy candidate at all. Note that this is not + // "smallest": within the lossy band it still wants the highest bitrate, + // which is the whole point -- a 320 beats a 256 beats a FLAC. + PreferSmaller = "smaller" + // PreferLarger is the reverse, and reduces to plain highest-bitrate-first. + PreferLarger = "larger" +) + +// normalisePreference tolerates casing and blank values so a hand-edited .env +// cannot break a run over a capital letter. +func normalisePreference(pref string) string { + switch strings.ToLower(strings.TrimSpace(pref)) { + case PreferSmaller: + return PreferSmaller + case PreferLarger: + return PreferLarger + default: + return PreferNone + } +} + +// bandRank groups a file for the preference. Lower sorts first. +// +// The two settings are not mirror images. "Larger" is satisfied by sorting on +// bitrate alone, while "smaller" needs the band, because wanting a small file +// does not mean wanting a bad one. +func bandRank(file File, pref string) int { + switch pref { + case PreferSmaller: + if isLossless(file) { + return 1 + } + return 0 + case PreferLarger: + if isLossless(file) { + return 0 + } + return 1 + default: + return 0 + } +} + +// compareFiles orders two candidates under a preference. Returns a negative +// number when a should be tried before b. +func (c Slskd) compareFiles(a, b File, pref string) int { + if pref == PreferNone { + // Preserve the historic behaviour exactly: the position of a file's + // extension in EXTENSIONS is the preference, and nothing else is. + return cmp.Compare(c.extensionRank(a), c.extensionRank(b)) + } + + if r := cmp.Compare(bandRank(a, pref), bandRank(b, pref)); r != 0 { + return r + } + + // Within a band, better is always better -- for both settings. + rateA, rateB := effectiveBitRate(a), effectiveBitRate(b) + + // A file nobody described is eligible but never preferred: we cannot say + // it is the best of its band without knowing what it is. + switch { + case rateA == 0 && rateB == 0: + return 0 + case rateA == 0: + return 1 + case rateB == 0: + return -1 + } + + return cmp.Compare(rateB, rateA) +} + +// extensionRank is a file's position in the configured extension list. +// Anything unlisted sorts last, though the caller has already filtered those +// out. +func (c Slskd) extensionRank(file File) int { + if i := slices.Index(c.Cfg.Filters.Extensions, file.Extension); i >= 0 { + return i + } + return len(c.Cfg.Filters.Extensions) +} + +// sortByPreference orders candidates in place. The sort is stable, so files +// that rank equally keep the order the search returned them in. +func (c Slskd) sortByPreference(files []File) { + pref := normalisePreference(c.Cfg.SizePreference) + slices.SortStableFunc(files, func(a, b File) int { + return c.compareFiles(a, b, pref) + }) +} + +// releaseBitRate summarises a release's quality as the median of its files' +// bitrates. The median rather than the mean, so that one 128 kbps bonus track +// cannot drag down an otherwise clean release. Files of unknown bitrate are +// left out; a release where nothing is known returns 0. +func releaseBitRate(files []File) int { + rates := make([]int, 0, len(files)) + for _, file := range files { + if rate := effectiveBitRate(file); rate > 0 { + rates = append(rates, rate) + } + } + + if len(rates) == 0 { + return 0 + } + + slices.Sort(rates) + return rates[len(rates)/2] +} + +// preferenceScore is what a release's quality contributes to its ranking. +// +// Deliberately smaller than either name-matching term in scoreDir: quality +// breaks a tie between releases that match the recommendation equally well and +// is never a reason to take a worse match. A wrong-album FLAC must still lose +// to a right-album MP3. +const preferenceScore = 20 + +// scorePreference rewards a release for sitting in the preferred band. It +// returns 0 for PreferNone, leaving release scoring exactly as it was. +func scorePreference(files []File, pref string) int { + if pref == PreferNone { + return 0 + } + + rate := releaseBitRate(files) + if rate == 0 { + return 0 + } + + // One file is enough to characterise the release: a directory of FLACs is + // lossless whatever its median bitrate works out to. + lossless := false + for _, file := range files { + if isLossless(file) { + lossless = true + break + } + } + + wantLossless := pref == PreferLarger + if lossless == wantLossless { + return preferenceScore + } + return 0 +} + +// Release preferences. Where SizePreference is about how good each file is, +// this is about how much of the release to take. +const ( + // PreferFullerRelease takes the most complete directory: the historic + // default, still the fallback whenever the real track count is unknown. + PreferFullerRelease = "fuller" + // PreferSmallerRelease reaches for the release closest to the album's real + // track count, which is what keeps a 21-track deluxe edition from being + // downloaded in place of the 11-track album. + // + // Deliberately "closest" and not "fewest": a directory holding only the + // recommended track would win every time under fewest, and album mode + // would quietly stop fetching albums at all. + PreferSmallerRelease = "smaller" +) + +// maxReleaseSizeScore caps every size term. Below the album (100) and artist +// (50) terms in scoreDir, so being the right size can decide between releases +// that match equally well but can never win against a better match. +const maxReleaseSizeScore = 40 + +// releaseSizeMissPenalty is how much each track of difference from the +// expected count costs, so a release ten tracks out scores nothing. +const releaseSizeMissPenalty = 4 + +func normaliseReleasePreference(pref string) string { + if strings.EqualFold(strings.TrimSpace(pref), PreferSmallerRelease) { + return PreferSmallerRelease + } + return PreferFullerRelease +} + +// expectedTrackCount reports how many files a correct release should hold, and +// whether that is knowable at all. +// +// The count comes from MusicBrainz during enrichment and covers the first +// medium only. A multi-disc release therefore legitimately holds more files +// than that, so a closeness score would punish the complete release for being +// complete -- better to admit we cannot tell. +// +// Which disc count decides that is the subtle part. The matched release may be +// a two-disc deluxe of a single-disc album, and bailing on its DiscTotal would +// hand the padded directory the fuller-release score it should have lost. The +// release group's consensus is asked first and the matched release only when +// there is none, so a genuine double album still bails while a deluxe-edition +// mismatch does not. +func expectedTrackCount(track models.Track) (int, bool) { + expected := expectedAlbumLength(track) + if expected <= 0 { + return 0, false + } + if discs := track.CanonicalDiscTotal; discs > 0 { + if discs > 1 { + return 0, false + } + return expected, true + } + if track.DiscTotal > 1 { + return 0, false + } + return expected, true +} + +// oversizedReleaseCap bounds the strict selection pass: a candidate holding +// more than this multiple of the expected track count is a compilation dump, +// not the album (BUG-35: a 64-file folder won a 10-track album). Deluxe +// padding stays well under it, and anything bigger still wins when nothing +// better exists -- the cap only decides between candidates, never eligibility +// outright. +const oversizedReleaseCap = 2 + +// oversizedDir reports whether dir holds far more files than the album should +// have. Without a usable expected count it never fires: same unknown-count +// cases as releaseSizeScore, same refusal to guess. +func oversizedDir(dir peerDir, track models.Track) bool { + expected, ok := expectedTrackCount(track) + if !ok { + return false + } + return len(dir.files) > oversizedReleaseCap*expected +} + +// releaseSizeScore rewards a release for being the size it ought to be. +// +// Without a usable expected count -- enrichment off, the lookup failed, or a +// multi-disc release -- it falls back to preferring the fuller release rather +// than guessing. Guessing is what would reopen the single-track hole: only +// when the real track count is known can a lone file be recognised as either a +// genuine single or a fragment of an album. +func releaseSizeScore(dir peerDir, track models.Track, pref string) int { + if pref != PreferSmallerRelease { + return fullerReleaseScore(dir) + } + + expected, ok := expectedTrackCount(track) + if !ok { + return fullerReleaseScore(dir) + } + + diff := len(dir.files) - expected + if diff < 0 { + diff = -diff + } + + return max(0, maxReleaseSizeScore-diff*releaseSizeMissPenalty) +} diff --git a/src/downloader/preference_test.go b/src/downloader/preference_test.go new file mode 100644 index 00000000..91b70daa --- /dev/null +++ b/src/downloader/preference_test.go @@ -0,0 +1,476 @@ +package downloader + +import ( + "testing" + + cfg "explo/src/config" + "explo/src/models" +) + +func prefClient(pref string) Slskd { + return Slskd{Cfg: cfg.Slskd{ + DownloadAttempts: 10, + SizePreference: pref, + Filters: cfg.Filters{ + Extensions: []string{"flac", "mp3"}, + MinBitDepth: 8, + }, + }} +} + +func names(files []File) []string { + out := make([]string, len(files)) + for i, f := range files { + out[i] = f.Name + } + return out +} + +func equal(a, b []string) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} + +// The candidates every ordering test starts from, deliberately not in any +// meaningful order. +func candidates() []File { + return []File{ + {Name: "mp3-256", Extension: "mp3", BitRate: 256}, + {Name: "flac-cd", Extension: "flac", BitRate: 1000}, + {Name: "mp3-320", Extension: "mp3", BitRate: 320}, + {Name: "flac-hi", Extension: "flac", BitRate: 4608}, + } +} + +func TestSortByPreference(t *testing.T) { + tests := []struct { + name string + pref string + want []string + }{ + { + // The point of "smaller": the best lossy file, not the smallest + // file. 320 beats 256, and both beat any FLAC. + name: "smaller takes the best lossy first", + pref: PreferSmaller, + want: []string{"mp3-320", "mp3-256", "flac-hi", "flac-cd"}, + }, + { + name: "larger takes the best lossless first", + pref: PreferLarger, + want: []string{"flac-hi", "flac-cd", "mp3-320", "mp3-256"}, + }, + { + // Unchanged behaviour: EXTENSIONS is flac,mp3, so FLAC first, and + // within a format the order the search returned them in. + name: "none follows the extension order", + pref: PreferNone, + want: []string{"flac-cd", "flac-hi", "mp3-256", "mp3-320"}, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + client := prefClient(tt.pref) + files := candidates() + client.sortByPreference(files) + + if got := names(files); !equal(got, tt.want) { + t.Errorf("order = %v, want %v", got, tt.want) + } + }) + } +} + +// The preference ranks, it does not filter: when the preferred band is empty +// the other one must still be reached rather than the track being lost. +func TestSortByPreference_FallsBackWhenTheBandIsEmpty(t *testing.T) { + client := prefClient(PreferSmaller) + files := []File{ + {Name: "flac-cd", Extension: "flac", BitRate: 1000}, + {Name: "flac-hi", Extension: "flac", BitRate: 4608}, + } + + client.sortByPreference(files) + + if len(files) != 2 { + t.Fatalf("kept %d files, want both -- preference must not drop candidates", len(files)) + } + if files[0].Name != "flac-hi" { + t.Errorf("first = %s, want the best of the only band available", files[0].Name) + } +} + +// A file nobody described is still eligible, but it cannot be called the best +// of its band. +func TestSortByPreference_UnknownBitrateRanksLastInItsBand(t *testing.T) { + client := prefClient(PreferSmaller) + files := []File{ + {Name: "mp3-unknown", Extension: "mp3"}, + {Name: "mp3-256", Extension: "mp3", BitRate: 256}, + {Name: "flac-cd", Extension: "flac", BitRate: 1000}, + } + + client.sortByPreference(files) + + want := []string{"mp3-256", "mp3-unknown", "flac-cd"} + if got := names(files); !equal(got, want) { + t.Errorf("order = %v, want %v", got, want) + } +} + +// filterFiles caps at DownloadAttempts, so the ordering decides which +// candidates survive, not just what order they are tried in. +func TestFilterFiles_PreferenceDecidesWhichSurviveTheCap(t *testing.T) { + client := prefClient(PreferSmaller) + client.Cfg.DownloadAttempts = 1 + + got, err := client.filterFiles(candidates()) + if err != nil { + t.Fatalf("filterFiles() = %v, want nil", err) + } + if len(got) != 1 || got[0].Name != "mp3-320" { + t.Errorf("kept %v, want just mp3-320", names(got)) + } +} + +func TestReleaseBitRate_MedianIgnoresAnOutlier(t *testing.T) { + files := []File{ + {BitRate: 320}, {BitRate: 320}, {BitRate: 320}, {BitRate: 320}, + {BitRate: 96}, // a bonus track nobody cares about + } + + if got := releaseBitRate(files); got != 320 { + t.Errorf("releaseBitRate() = %d, want 320 -- one bad track must not sink the release", got) + } +} + +func TestReleaseBitRate_UnknownWhenNothingIsDescribed(t *testing.T) { + if got := releaseBitRate([]File{{}, {}}); got != 0 { + t.Errorf("releaseBitRate() = %d, want 0", got) + } +} + +func albumDir(ext string, rate int, names ...string) peerDir { + dir := peerDir{dir: "Billie Eilish - When We All Fall Asleep"} + for _, n := range names { + dir.files = append(dir.files, File{Name: n, Extension: ext, BitRate: rate}) + } + return dir +} + +// Quality is a tiebreak between equally good matches. It must never be a +// reason to take a release that matches the recommendation worse. +func TestScoreDirWithPreference_QualityNeverBeatsAMatch(t *testing.T) { + track := models.Track{ + CleanTitle: "Bad Guy", + MainArtist: "Billie Eilish", + Album: "When We All Fall Asleep", + } + + client := prefClient(PreferLarger) + + right := albumDir("mp3", 320, "01 Bad Guy.mp3") + right.dir = "Billie Eilish - When We All Fall Asleep" + + wrong := albumDir("flac", 1000, "01 Bad Guy.flac") + wrong.dir = "Some Other Compilation" + + rightScore := client.scoreDirWithPreference(right, track) + wrongScore := client.scoreDirWithPreference(wrong, track) + + if wrongScore >= rightScore { + t.Errorf("wrong-album FLAC scored %d against right-album MP3 %d; quality outranked the match", + wrongScore, rightScore) + } +} + +// Between two releases that match equally well, the preference decides. +func TestScoreDirWithPreference_BreaksATieBetweenEqualMatches(t *testing.T) { + track := models.Track{ + CleanTitle: "Bad Guy", + MainArtist: "Billie Eilish", + Album: "When We All Fall Asleep", + } + + lossy := albumDir("mp3", 320, "01 Bad Guy.mp3") + lossless := albumDir("flac", 1000, "01 Bad Guy.flac") + + smaller := prefClient(PreferSmaller) + if smaller.scoreDirWithPreference(lossy, track) <= smaller.scoreDirWithPreference(lossless, track) { + t.Error("prefer smaller did not favour the lossy release") + } + + larger := prefClient(PreferLarger) + if larger.scoreDirWithPreference(lossless, track) <= larger.scoreDirWithPreference(lossy, track) { + t.Error("prefer larger did not favour the lossless release") + } +} + +// The default has to leave release scoring exactly as it was. +func TestScoreDirWithPreference_NoneChangesNothing(t *testing.T) { + track := models.Track{ + CleanTitle: "Bad Guy", + MainArtist: "Billie Eilish", + Album: "When We All Fall Asleep", + } + + client := prefClient(PreferNone) + for _, dir := range []peerDir{ + albumDir("mp3", 320, "01 Bad Guy.mp3"), + albumDir("flac", 1000, "01 Bad Guy.flac"), + } { + if got, want := client.scoreDirWithPreference(dir, track), scoreDir(dir, track); got != want { + t.Errorf("with no preference score = %d, want the unmodified %d", got, want) + } + } +} + +func TestNormalisePreference(t *testing.T) { + cases := map[string]string{ + "smaller": PreferSmaller, + "LARGER": PreferLarger, + " none ": PreferNone, + "": PreferNone, + "nonsense": PreferNone, + } + for in, want := range cases { + if got := normalisePreference(in); got != want { + t.Errorf("normalisePreference(%q) = %q, want %q", in, got, want) + } + } +} + +// releaseDir builds a candidate release of n files whose directory names both +// the artist and the album, so only the size term varies between cases. +func releaseDir(n int) peerDir { + dir := peerDir{dir: "Billie Eilish - When We All Fall Asleep"} + for i := 0; i < n; i++ { + dir.files = append(dir.files, File{ + Name: "track.mp3", + Extension: "mp3", + BitRate: 320, + }) + } + return dir +} + +func albumTrackOf(total, discs int) models.Track { + return models.Track{ + CleanTitle: "Bad Guy", + MainArtist: "Billie Eilish", + Album: "When We All Fall Asleep", + TrackTotal: total, + DiscTotal: discs, + } +} + +func TestReleaseSizeScore_PrefersTheRealAlbumLength(t *testing.T) { + track := albumTrackOf(11, 1) + + exact := releaseSizeScore(releaseDir(11), track, PreferSmallerRelease) + deluxe := releaseSizeScore(releaseDir(21), track, PreferSmallerRelease) + + if exact <= deluxe { + t.Errorf("11-track release scored %d against a 21-track deluxe %d; the deluxe would win", + exact, deluxe) + } + if exact != maxReleaseSizeScore { + t.Errorf("an exact match scored %d, want the full %d", exact, maxReleaseSizeScore) + } +} + +// The hole this design exists to close: under a naive "fewest tracks wins", +// a directory holding only the recommended track beats every real album and +// album mode stops fetching albums. +func TestReleaseSizeScore_ASingleFileLosesToTheRealAlbum(t *testing.T) { + track := albumTrackOf(12, 1) + + lone := releaseSizeScore(releaseDir(1), track, PreferSmallerRelease) + album := releaseSizeScore(releaseDir(12), track, PreferSmallerRelease) + + if lone >= album { + t.Errorf("a 1-file directory scored %d against the 12-track album %d; album mode would "+ + "quietly stop downloading albums", lone, album) + } +} + +// A real single is not a degenerate case: when the release genuinely has one +// track, one file is the right answer. +func TestReleaseSizeScore_ASingleFileWinsForARealSingle(t *testing.T) { + track := albumTrackOf(1, 1) + + lone := releaseSizeScore(releaseDir(1), track, PreferSmallerRelease) + padded := releaseSizeScore(releaseDir(9), track, PreferSmallerRelease) + + if lone <= padded { + t.Errorf("one file scored %d for a one-track single, against %d for nine", lone, padded) + } +} + +func TestReleaseSizeScore_FallsBackWithoutACanonicalCount(t *testing.T) { + cases := map[string]models.Track{ + // Enrichment is off or the MusicBrainz lookup failed. + "no track total": albumTrackOf(0, 1), + // TrackTotal counts the first medium only, so a two-disc release + // legitimately holds more files than it. Scoring closeness would + // punish the complete release for being complete. + "multi disc": albumTrackOf(11, 2), + } + + for name, track := range cases { + t.Run(name, func(t *testing.T) { + dir := releaseDir(21) + got := releaseSizeScore(dir, track, PreferSmallerRelease) + want := fullerReleaseScore(dir) + + if got != want { + t.Errorf("score = %d, want the fuller-release fallback %d", got, want) + } + }) + } +} + +// The default must leave release scoring exactly as it was. +func TestReleaseSizeScore_FullerIsUnchanged(t *testing.T) { + track := albumTrackOf(11, 1) + + for _, n := range []int{1, 11, 21, 60} { + dir := releaseDir(n) + if got, want := releaseSizeScore(dir, track, PreferFullerRelease), min(n, 40); got != want { + t.Errorf("fuller score for %d files = %d, want %d", n, got, want) + } + } +} + +// A 64-file folder is not a 10-track album, however well the folder is +// named. Anything at or under twice the expected count stays eligible -- +// deluxe padding included. +func TestOversizedDir_FlagsADump(t *testing.T) { + track := albumTrackOf(10, 1) + + if !oversizedDir(releaseDir(64), track) { + t.Error("64 files for a 10-track album is not flagged oversized") + } + if oversizedDir(releaseDir(10), track) { + t.Error("the exact 10-track release is flagged oversized") + } + if oversizedDir(releaseDir(20), track) { + t.Error("a 20-track deluxe for a 10-track album is flagged oversized; the cap must leave deluxe padding alone") + } +} + +// Same unknown-count cases as the scoring fallback: no expected count, no +// verdict. +func TestOversizedDir_NeverFiresWithoutAnExpectedCount(t *testing.T) { + cases := map[string]models.Track{ + "no track total": albumTrackOf(0, 1), + "multi disc": albumTrackOf(11, 2), + } + + for name, track := range cases { + t.Run(name, func(t *testing.T) { + if oversizedDir(releaseDir(64), track) { + t.Errorf("64 files flagged oversized with no usable expected count (%s)", name) + } + }) + } +} + +// The cap is what stops a right-sized wrong album beating a wrong-sized right +// one. Without it the size term could outrank the album name. +func TestReleaseSizeScore_NeverExceedsTheCap(t *testing.T) { + track := albumTrackOf(11, 1) + + for _, n := range []int{0, 1, 11, 21, 200} { + if got := releaseSizeScore(releaseDir(n), track, PreferSmallerRelease); got > maxReleaseSizeScore { + t.Errorf("score for %d files = %d, above the cap of %d", n, got, maxReleaseSizeScore) + } + } +} + +// End to end through the scorer the album flow actually calls. +func TestScoreDirWithPreference_RightSizedWrongAlbumStillLoses(t *testing.T) { + track := albumTrackOf(11, 1) + + client := prefClient(PreferNone) + client.Cfg.ReleasePreference = PreferSmallerRelease + + right := releaseDir(21) // the deluxe, but the correct album + + wrong := releaseDir(11) // exactly the right length, wrong album entirely + wrong.dir = "Some Other Compilation" + + if client.scoreDirWithPreference(wrong, track) >= client.scoreDirWithPreference(right, track) { + t.Error("a right-sized wrong album beat a wrong-sized right album") + } +} + +func TestScoreDirWithPreference_SmallerPicksTheStandardEdition(t *testing.T) { + track := albumTrackOf(11, 1) + + client := prefClient(PreferNone) + client.Cfg.ReleasePreference = PreferSmallerRelease + + standard := releaseDir(11) + deluxe := releaseDir(21) + + if client.scoreDirWithPreference(standard, track) <= client.scoreDirWithPreference(deluxe, track) { + t.Error("the deluxe edition beat the standard album") + } + + // And the default still does the opposite. + client.Cfg.ReleasePreference = PreferFullerRelease + if client.scoreDirWithPreference(deluxe, track) <= client.scoreDirWithPreference(standard, track) { + t.Error("with the default preference the fuller release did not win") + } +} + +func TestNormaliseReleasePreference(t *testing.T) { + cases := map[string]string{ + "smaller": PreferSmallerRelease, + "SMALLER": PreferSmallerRelease, + " fuller ": PreferFullerRelease, + "": PreferFullerRelease, + "nonsense": PreferFullerRelease, + } + for in, want := range cases { + if got := normaliseReleasePreference(in); got != want { + t.Errorf("normaliseReleasePreference(%q) = %q, want %q", in, got, want) + } + } +} + +// The scorer has the same deluxe-match problem as the trim: bailing on the +// matched release's DiscTotal hands a padded directory the fuller-release score. +func TestExpectedTrackCount_ConsensusBeatsADeluxeMatch(t *testing.T) { + expected, ok := expectedTrackCount(models.Track{ + TrackTotal: 23, DiscTotal: 2, + CanonicalTrackTotal: 16, CanonicalDiscTotal: 1, + }) + + if !ok { + t.Fatal("expectedTrackCount() gave up despite a single-disc consensus") + } + if expected != 16 { + t.Errorf("expected = %d, want the consensus 16", expected) + } +} + +// A genuine double album must still bail: its first-medium count cannot measure +// a complete two-disc directory. +func TestExpectedTrackCount_StillBailsOnARealDoubleAlbum(t *testing.T) { + if _, ok := expectedTrackCount(models.Track{ + TrackTotal: 13, DiscTotal: 2, + CanonicalTrackTotal: 13, CanonicalDiscTotal: 2, + }); ok { + t.Error("expectedTrackCount() answered for a 2-disc consensus; the complete release would be punished") + } +} diff --git a/src/downloader/quality.go b/src/downloader/quality.go new file mode 100644 index 00000000..bbbefb46 --- /dev/null +++ b/src/downloader/quality.go @@ -0,0 +1,75 @@ +package downloader + +import "strings" + +// losslessExtensions are the formats that carry no lossy compression. The +// split matters because it is the line people actually think in when they talk +// about a "small" or a "big" file, far more than any particular bitrate. +var losslessExtensions = map[string]struct{}{ + "flac": {}, + "wav": {}, + "aiff": {}, + "aif": {}, + "alac": {}, + "ape": {}, + "wv": {}, +} + +// isLossless reports whether a file's format is lossless. +func isLossless(file File) bool { + _, ok := losslessExtensions[strings.ToLower(strings.TrimPrefix(file.Extension, "."))] + return ok +} + +// effectiveBitRate reports a file's bitrate in kbps. +// +// Soulseek peers often report no bitrate at all, which is why the existing +// quality floors are written to skip a file rather than judge it when BitRate +// is 0. A ceiling cannot be written that way round: skipping the files nobody +// described would let through exactly the ones a ceiling exists to exclude. So +// when the peer says nothing, derive the bitrate from how big the file is and +// how long it plays. +// +// The reported value is preferred where there is one, so files that already +// describe themselves are judged exactly as they were before. +// +// Returns 0 when neither is available, which callers must read as "unknown" +// rather than "zero": an unknown file is never rejected by the ceiling. +func effectiveBitRate(file File) int { + if file.BitRate > 0 { + return file.BitRate + } + + if file.Size > 0 && file.Length > 0 { + return file.Size * 8 / file.Length / 1000 + } + + return 0 +} + +// withinQualityRange reports whether a file sits inside the configured floor +// and ceiling. A file of unknown bitrate passes: it may well be fine, and +// dropping everything undescribed would throw away most of what Soulseek +// offers. +func (c Slskd) withinQualityRange(file File) bool { + if file.BitDepth > 0 && file.BitDepth < c.Cfg.Filters.MinBitDepth { + return false + } + + bitRate := effectiveBitRate(file) + if bitRate == 0 { + return true + } + + if c.Cfg.Filters.MinBitRate > 0 && bitRate < c.Cfg.Filters.MinBitRate { + return false + } + + // Zero disables the ceiling, so an upgrade changes nothing until someone + // sets one. + if c.Cfg.Filters.MaxBitRate > 0 && bitRate > c.Cfg.Filters.MaxBitRate { + return false + } + + return true +} diff --git a/src/downloader/quality_test.go b/src/downloader/quality_test.go new file mode 100644 index 00000000..b0c79345 --- /dev/null +++ b/src/downloader/quality_test.go @@ -0,0 +1,228 @@ +package downloader + +import ( + "testing" + + cfg "explo/src/config" + "explo/src/models" +) + +func TestEffectiveBitRate(t *testing.T) { + tests := []struct { + name string + file File + want int + }{ + { + // A file that describes itself is judged exactly as it was before. + name: "reported bitrate wins", + file: File{BitRate: 320, Size: 99_999_999, Length: 200}, + want: 320, + }, + { + // 210 s of 320 kbps is about 8.4 MB. + name: "derived from size and length when unreported", + file: File{Size: 8_400_000, Length: 210}, + want: 320, + }, + { + name: "derived for a CD flac", + file: File{Size: 26_000_000, Length: 210}, + want: 990, + }, + { + name: "unknown without a length", + file: File{Size: 8_400_000}, + want: 0, + }, + { + name: "unknown without a size", + file: File{Length: 210}, + want: 0, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := effectiveBitRate(tt.file) + // Derivation is integer division, so allow a kbps either way. + if got < tt.want-2 || got > tt.want+2 { + t.Errorf("effectiveBitRate(%+v) = %d, want about %d", tt.file, got, tt.want) + } + }) + } +} + +func TestIsLossless(t *testing.T) { + for _, ext := range []string{"flac", "FLAC", ".flac", "wav", "aiff", "alac"} { + if !isLossless(File{Extension: ext}) { + t.Errorf("isLossless(%q) = false, want true", ext) + } + } + for _, ext := range []string{"mp3", "m4a", "aac", "ogg", ""} { + if isLossless(File{Extension: ext}) { + t.Errorf("isLossless(%q) = true, want false", ext) + } + } +} + +func rangeClient(minRate, maxRate int) Slskd { + return Slskd{Cfg: cfg.Slskd{ + DownloadAttempts: 10, + Filters: cfg.Filters{ + Extensions: []string{"flac", "mp3"}, + MinBitRate: minRate, + MaxBitRate: maxRate, + MinBitDepth: 8, + }, + }} +} + +func TestWithinQualityRange(t *testing.T) { + tests := []struct { + name string + client Slskd + file File + wantIn bool + }{ + {"inside the range", rangeClient(192, 320), File{BitRate: 256}, true}, + {"below the floor", rangeClient(192, 320), File{BitRate: 128}, false}, + {"above the ceiling", rangeClient(192, 320), File{BitRate: 1000}, false}, + {"at the ceiling", rangeClient(192, 320), File{BitRate: 320}, true}, + {"at the floor", rangeClient(192, 320), File{BitRate: 192}, true}, + { + // The default: no ceiling configured, so nothing changes for + // anyone who has not set one. + name: "a zero ceiling rejects nothing", + client: rangeClient(192, 0), + file: File{BitRate: 4608}, + wantIn: true, + }, + { + // Most of Soulseek reports nothing. Dropping all of it would leave + // almost no candidates. + name: "unknown bitrate passes the ceiling", + client: rangeClient(192, 320), + file: File{}, + wantIn: true, + }, + { + // Derived from size, so the ceiling reaches files the old + // reported-only checks skipped entirely. + name: "a big unreported file is caught by derivation", + client: rangeClient(192, 320), + file: File{Size: 26_000_000, Length: 210}, + wantIn: false, + }, + {"bit depth below the floor", rangeClient(0, 0), File{BitDepth: 4, BitRate: 256}, false}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := tt.client.withinQualityRange(tt.file); got != tt.wantIn { + t.Errorf("withinQualityRange(%+v) = %v, want %v", tt.file, got, tt.wantIn) + } + }) + } +} + +// filterFiles is the single-track path. +func TestFilterFiles_AppliesTheCeiling(t *testing.T) { + client := rangeClient(192, 320) + + files := []File{ + {Name: "big.flac", Extension: "flac", BitRate: 1000}, + {Name: "ok.mp3", Extension: "mp3", BitRate: 320}, + } + + got, err := client.filterFiles(files) + if err != nil { + t.Fatalf("filterFiles() = %v, want nil", err) + } + if len(got) != 1 || string(got[0].Name) != "ok.mp3" { + t.Errorf("kept %v, want only ok.mp3", got) + } +} + +func TestFilterFiles_NoCeilingKeepsEverything(t *testing.T) { + client := rangeClient(192, 0) + + files := []File{ + {Name: "big.flac", Extension: "flac", BitRate: 4608}, + {Name: "ok.mp3", Extension: "mp3", BitRate: 320}, + } + + got, err := client.filterFiles(files) + if err != nil { + t.Fatalf("filterFiles() = %v, want nil", err) + } + if len(got) != 2 { + t.Errorf("kept %d files, want both", len(got)) + } +} + +// The album path exempts the recommended track from the quality floors, on the +// grounds that a mediocre entry beats none. The ceiling is a different +// question -- it is about transfer volume -- so the primary must not be exempt +// from it. +func TestQualityFiltered_CeilingAppliesToThePrimaryToo(t *testing.T) { + client := rangeClient(192, 320) + + primary := File{Name: "01 primary.flac", Extension: "flac", BitRate: 1000} + dir := peerDir{ + files: []File{primary, {Name: "02 sibling.mp3", Extension: "mp3", BitRate: 320}}, + primary: &primary, + } + + kept := client.qualityFiltered(dir) + for _, f := range kept { + if string(f.Name) == "01 primary.flac" { + t.Error("the primary was kept despite being over the ceiling") + } + } +} + +// The floor exemption for the primary has to survive, or album mode loses the +// playlist entry it exists to produce. +func TestQualityFiltered_PrimaryStillExemptFromTheFloor(t *testing.T) { + client := rangeClient(320, 0) + + primary := File{Name: "01 primary.mp3", Extension: "mp3", BitRate: 128} + dir := peerDir{ + files: []File{primary, {Name: "02 sibling.mp3", Extension: "mp3", BitRate: 128}}, + primary: &primary, + } + + kept := client.qualityFiltered(dir) + if len(kept) != 1 || string(kept[0].Name) != "01 primary.mp3" { + t.Errorf("kept %v, want just the primary -- it is exempt from the floor, siblings are not", kept) + } +} + +// Guard against the album flow silently ignoring the new setting. +func TestCollectAlbumFiles_RespectsTheCeiling(t *testing.T) { + client := rangeClient(0, 320) + client.Cfg.AlbumMode = true + + results := albumResults(peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, + []string{"01 Bury A Friend.flac", "02 Bad Guy.flac"}}) + // albumResults leaves BitRate unset, so give the files a size that derives + // well above the ceiling. + for i := range results { + for j := range results[i].Files { + results[i].Files[j].Size = 26_000_000 + results[i].Files[j].Length = 210 + } + } + + track := models.Track{ + CleanTitle: "Bad Guy", + Artist: "Billie Eilish", + MainArtist: "Billie Eilish", + Album: "When We All Fall Asleep", + } + + if _, err := client.CollectAlbumFiles(track, results); err == nil { + t.Error("CollectAlbumFiles() = nil error, want a failure when every file is over the ceiling") + } +} diff --git a/src/downloader/slskd.go b/src/downloader/slskd.go index 6ca3d8b7..82286100 100644 --- a/src/downloader/slskd.go +++ b/src/downloader/slskd.go @@ -100,6 +100,8 @@ type Slskd struct { HttpClient *util.HttpClient DownloadDir string Cfg config.Slskd + // releases stops tracks from one album each downloading it in album mode. + releases *releaseRegistry } type SearchPayload struct { @@ -109,7 +111,8 @@ type SearchPayload struct { func NewSlskd(cfg config.Slskd, downloadDir string) *Slskd { return &Slskd{Cfg: cfg, HttpClient: util.NewHttp(util.HttpClientConfig{Timeout: cfg.Timeout}), - DownloadDir: downloadDir} + DownloadDir: downloadDir, + releases: newReleaseRegistry()} } func (c *Slskd) AddHeader() { @@ -125,6 +128,7 @@ func (c *Slskd) GetConf() (MonitorConfig, error) { CheckInterval: time.Duration(c.Cfg.MonitorConfig.Interval) * time.Minute, StallDuration: time.Duration(c.Cfg.MonitorConfig.StallDuration) * time.Minute, MaxDuration: time.Duration(c.Cfg.MonitorConfig.MaxDuration) * time.Minute, + FallbackAfter: time.Duration(c.Cfg.MonitorConfig.FallbackAfter) * time.Minute, MigrateDownload: c.Cfg.MigrateDL, ToDir: c.DownloadDir, FromDir: c.Cfg.SlskdDir, @@ -135,9 +139,21 @@ func (c *Slskd) GetConf() (MonitorConfig, error) { var errNoRes = errors.New("no results found for query") func (c *Slskd) QueryTrack(track *models.Track) error { + if c.Cfg.AlbumMode { + return c.queryAlbum(track) + } + return c.search(track, false) +} + +// search looks for a release when albumMode is set and for the lone track +// otherwise, recording the search on the track for GetTrack to read. +func (c *Slskd) search(track *models.Track, albumMode bool) error { wildcardSearch := false trackDetails := fmt.Sprintf("%s - %s", track.CleanTitle, track.Artist) + if albumMode { + trackDetails = albumSearchTerm(*track) + } retry: ID, err := c.searchTrack(trackDetails) @@ -156,7 +172,11 @@ func (c *Slskd) QueryTrack(track *models.Track) error { if errors.Is(err, errNoRes) && !wildcardSearch { cleanup() wildcardSearch = true - trackDetails = fmt.Sprintf("%s - %s", track.CleanTitle, wildcardArtist(track.Artist)) + if albumMode && strings.TrimSpace(track.Album) != "" { + trackDetails = fmt.Sprintf("%s - %s", wildcardArtist(track.MainArtist), track.Album) + } else { + trackDetails = fmt.Sprintf("%s - %s", track.CleanTitle, wildcardArtist(track.Artist)) + } slog.Debug("no result found with artist full name, trying with wildcard", "query", trackDetails) goto retry } @@ -177,6 +197,20 @@ func (c *Slskd) QueryTrack(track *models.Track) error { } func (c *Slskd) GetTrack(track *models.Track) error { + if c.Cfg.AlbumMode { + switch state := c.releases.state(track); { + case state != nil && state.role == roleShared: + // Already queued as part of another track's release. + return nil + case state != nil && state.role == roleSingle: + return c.getSingle(track) + } + return c.getAlbum(track) + } + return c.getSingle(track) +} + +func (c *Slskd) getSingle(track *models.Track) error { results, err := c.searchResults(track.ID) if err != nil { return err @@ -319,30 +353,28 @@ func (c Slskd) CollectFiles(track models.Track, searchResults SearchResults) ([] func (c Slskd) filterFiles(files []File) ([]File, error) { var filtered []File - for _, ext := range c.Cfg.Filters.Extensions { - for _, file := range files { - if file.Extension != ext { - continue - } - - if file.BitRate > 0 && file.BitRate < c.Cfg.Filters.MinBitRate { - continue - } - - if file.BitDepth > 0 && file.BitDepth < c.Cfg.Filters.MinBitDepth { - continue - } - - filtered = append(filtered, file) - if len(filtered) >= c.Cfg.DownloadAttempts { - return filtered, nil - } + for _, file := range files { + if !slices.Contains(c.Cfg.Filters.Extensions, file.Extension) { + continue } + if !c.withinQualityRange(file) { + continue + } + filtered = append(filtered, file) } if len(filtered) == 0 { return nil, fmt.Errorf("no files found that match filters") } + + // Collect first and order afterwards. The old shape looped over the + // extensions outside the files, which made the extension list double as a + // preference; that is now only one of the orderings on offer. + c.sortByPreference(filtered) + + if len(filtered) > c.Cfg.DownloadAttempts { + filtered = filtered[:c.Cfg.DownloadAttempts] + } return filtered, nil } @@ -415,10 +447,11 @@ func (c *Slskd) GetDownloadStatus(tracks []*models.Track) (map[string]FileStatus } } } - if len(fileStatuses) != 0 { - return fileStatuses, nil - } - return nil, fmt.Errorf("no files found to monitor") + // An empty result is not a failure. slskd may not have surfaced the + // transfers yet, and reporting that as an error used to abort monitoring + // for every track at once. The monitor decides on its own how long to wait + // for a file to appear. + return fileStatuses, nil } func (c Slskd) deleteDownload(user, ID string) error { @@ -438,8 +471,11 @@ func (c Slskd) deleteDownload(user, ID string) error { } func (c *Slskd) Cleanup(track models.Track, fileID string) error { - if err := c.deleteSearch(track.ID); err != nil { - slog.Debug("failed to delete search request", logging.RuntimeAttr(err.Error())) + // A track shared from another's release has no search of its own. + if !c.releases.isSharedID(track.ID) { + if err := c.deleteSearch(track.ID); err != nil { + slog.Debug("failed to delete search request", logging.RuntimeAttr(err.Error())) + } } if err := c.deleteDownload(track.MainArtistID, fileID); err != nil { slog.Debug("failed to delete download", logging.RuntimeAttr(err.Error())) @@ -470,6 +506,11 @@ func wildcardArtist(artist string) string { } // different failure states slskd has (format is "Completed,Rejected", "Errored,Cancelled" etc..) +// errorState is the single state normalize collapses every slskd failure into. +// Shared so that code reacting to a failed transfer cannot drift from the code +// that labels one. +const errorState = "Errored" + var failureStates = map[string]struct{} { "Aborted": {}, "TimedOut": {}, @@ -486,14 +527,14 @@ func normalize(state string) string{ p = strings.TrimSpace(p) if _, ok := failureStates[p]; ok { slog.Debug("[slskd] download failed", "status", state) - return "Errored" + return errorState } } return state } -func (c *Slskd) MoveDownload(srcDir, destDir, trackPath string, track *models.Track) error { +func (c *Slskd) MoveDownload(srcDir, destDir, trackPath string, track *models.Track) (string, error) { trackDir := filepath.Join(srcDir, trackPath) if c.Cfg.RenameTrack { // Rename file to {title}-{artist} format @@ -506,9 +547,10 @@ func (c *Slskd) MoveDownload(srcDir, destDir, trackPath string, track *models.Tr slog.Warn("problem overwriting metadata", "msg", err.Error()) } } - if err := moveTrack(srcFile, destDir, track, c.Cfg.PathTemplate, c.Cfg.KeepPermissions); err != nil { - return fmt.Errorf("failed to move track: %w", err) - } + landedIn, err := moveTrack(srcFile, destDir, track, c.Cfg.PathTemplate, c.Cfg.KeepPermissions) + if err != nil { + return "", fmt.Errorf("failed to move track: %w", err) + } - return nil + return landedIn, nil } \ No newline at end of file diff --git a/src/downloader/slskd_album.go b/src/downloader/slskd_album.go new file mode 100644 index 00000000..c7ea80b4 --- /dev/null +++ b/src/downloader/slskd_album.go @@ -0,0 +1,390 @@ +package downloader + +import ( + "bytes" + "encoding/json" + "fmt" + "log/slog" + "path" + "path/filepath" + "slices" + "strings" + + "explo/src/models" + "explo/src/util" +) + +// Soulseek peers report Windows-style paths. Normalising once means the rest of +// this file can treat them as ordinary slash-separated paths. +func normalizePeerPath(p string) string { + return strings.ReplaceAll(p, `\`, "/") +} + +// peerDir is one candidate release: the files a single peer shares from a +// single directory. +type peerDir struct { + username string + dir string + files []File + // primary is the file matching the recommended track. A directory without + // one is not a candidate, since the recommendation is what we came for. + primary *File +} + +// groupByDirectory buckets search results by peer and parent directory, keeping +// only files whose extension is allowed. Quality filtering happens later, per +// candidate, so that one bad file does not disqualify a whole release. +func (c Slskd) groupByDirectory(results SearchResults) []peerDir { + index := make(map[string]*peerDir) + var order []string + + for _, result := range results { + if result.FileCount == 0 || !result.HasFreeUploadSlot { + continue + } + for _, file := range result.Files { + file.Extension = resolveExtension(file) + if !slices.Contains(c.Cfg.Filters.Extensions, file.Extension) { + continue + } + + dir := path.Dir(normalizePeerPath(string(file.Name))) + key := result.Username + "\x00" + dir + + if _, ok := index[key]; !ok { + index[key] = &peerDir{username: result.Username, dir: dir} + order = append(order, key) + } + file.Username = result.Username + index[key].files = append(index[key].files, file) + } + } + + dirs := make([]peerDir, 0, len(order)) + for _, key := range order { + dirs = append(dirs, *index[key]) + } + return dirs +} + +// resolveExtension prefers the extension in the filename over the one the peer +// reports, matching what CollectFiles does for single tracks. +func resolveExtension(file File) string { + nameExt := util.AlnumOnly(strings.TrimPrefix(strings.ToLower(path.Ext(normalizePeerPath(string(file.Name)))), ".")) + if nameExt != "" { + return nameExt + } + return strings.TrimPrefix(strings.ToLower(file.Extension), ".") +} + +// findPrimary locates the recommended track inside a candidate directory. +// Without it the directory may well be the right album, but we would be +// downloading it without the track that prompted the recommendation. +func findPrimary(dir *peerDir, track models.Track) bool { + sanitizedTitle := util.AlnumOnly(track.CleanTitle) + if sanitizedTitle == "" { + return false + } + + for i := range dir.files { + name := util.AlnumOnly(path.Base(normalizePeerPath(string(dir.files[i].Name)))) + if !containsLower(name, sanitizedTitle) { + continue + } + // Duration is only meaningful for the recommended track; sibling tracks + // legitimately differ, which is why this check is not applied to them. + if track.Duration > 0 && dir.files[i].Length > 0 && + util.Abs(track.Duration/1000-dir.files[i].Length) > 10 { + continue + } + dir.primary = &dir.files[i] + return true + } + return false +} + +// scoreDir ranks a candidate release. A directory naming the album beats one +// that merely happens to contain the track, and among equals the fuller +// release wins. +func scoreDir(dir peerDir, track models.Track) int { + var score int + + sanitizedDir := util.AlnumOnly(dir.dir) + if album := util.AlnumOnly(track.Album); album != "" && containsLower(sanitizedDir, album) { + score += 100 + } + if artist := util.AlnumOnly(track.MainArtist); artist != "" && containsLower(sanitizedDir, artist) { + score += 50 + } + + score += fullerReleaseScore(dir) + + return score +} + +// fullerReleaseScore is the historic size term: more tracks means a more +// complete release. Capped so that file count can never outweigh actually +// matching the album. +func fullerReleaseScore(dir peerDir) int { + return min(len(dir.files), maxReleaseSizeScore) +} + +// scoreDirWithPreference adds the size preference to a release's score. Kept +// separate so the matching terms above stay readable, and weighted well below +// them: quality decides between equally good matches and never overrides one. +func (c Slskd) scoreDirWithPreference(dir peerDir, track models.Track) int { + score := scoreDir(dir, track) + scorePreference(dir.files, normalisePreference(c.Cfg.SizePreference)) + + // Swap the historic size term for the configured one. Both are capped the + // same way, so the balance against the name terms is unchanged. + score -= fullerReleaseScore(dir) + score += releaseSizeScore(dir, track, normaliseReleasePreference(c.Cfg.ReleasePreference)) + + // Trimming equalises tracklists, so a reissue and the original become + // indistinguishable by file count. The year is what still separates them. + if c.Cfg.PreferOriginalRelease && laterThanOriginal(dir.dir, track) { + score -= originalYearPenalty + } + + return score +} + +// qualityFiltered drops files failing the bitrate and bit-depth floors. Unlike +// filterFiles it does not cap the count -- the whole release is the point -- +// and it leaves the primary in place even if it fails, since a playlist entry +// with a mediocre file beats no entry at all. +func (c Slskd) qualityFiltered(dir peerDir) []File { + kept := make([]File, 0, len(dir.files)) + + for _, file := range dir.files { + // The ceiling applies to every file including the primary: it is about + // how much gets transferred, not how good the track is, and a single + // oversized primary is exactly what it exists to stop. + if c.Cfg.Filters.MaxBitRate > 0 { + if rate := effectiveBitRate(file); rate > 0 && rate > c.Cfg.Filters.MaxBitRate { + continue + } + } + + isPrimary := dir.primary != nil && file.Name == dir.primary.Name + if !isPrimary && !c.withinQualityRange(file) { + continue + } + + kept = append(kept, file) + } + return kept +} + +// CollectAlbumFiles picks the release to download. It returns the files to +// queue with the recommended track first, so callers can treat files[0] as the +// one that belongs in the playlist. +func (c Slskd) CollectAlbumFiles(track models.Track, results SearchResults) ([]File, error) { + candidates := c.groupByDirectory(results) + + var best *peerDir + var bestScore int + var bestFull *peerDir + var bestFullScore int + + for i := range candidates { + dir := &candidates[i] + + // A keyword hit on the directory rejects the whole release -- this is + // how "live" or "remix" pressings get excluded in album mode. + if ContainsKeyword(track, dir.dir, c.Cfg.Filters.FilterList) { + continue + } + if !findPrimary(dir, track) { + continue + } + // Before scoring, not after: a release still carrying its bonus discs + // would otherwise outscore the plain album on file count and win the + // comparison it should have lost. + if c.Cfg.PreferOriginalRelease { + trimToOriginal(dir, track) + } + score := c.scoreDirWithPreference(*dir, track) + if best == nil || score > bestScore { + best, bestScore = dir, score + } + // A lone file in an album-named folder is a stray, not a release -- + // the one-song download in place of the album -- and a directory + // holding far more than the album should is a dump, not the album + // either. Both stay eligible so a track with nothing better still + // downloads, but a right-sized release outranks them regardless of + // score. + if len(dir.files) > 1 && !oversizedDir(*dir, track) && (bestFull == nil || score > bestFullScore) { + bestFull, bestFullScore = dir, score + } + } + if bestFull != nil { + best, bestScore = bestFull, bestFullScore + } + + if best == nil { + return nil, fmt.Errorf("no release found containing %s - %s", track.MainArtist, track.CleanTitle) + } + + files := c.qualityFiltered(*best) + if len(files) == 0 { + return nil, fmt.Errorf("no files passed filters in %s", best.dir) + } + + // Primary first: queueAlbumDownload and the monitor both rely on it. + primaryName := best.primary.Name + slices.SortStableFunc(files, func(a, b File) int { + switch { + case a.Name == primaryName: + return -1 + case b.Name == primaryName: + return 1 + default: + return 0 + } + }) + + slog.Info("album release selected", + "dir", best.dir, "user", best.username, "tracks", len(files), "score", bestScore) + + return files, nil +} + +// queueAlbumDownload queues an entire release in a single request. slskd takes +// an array of files per peer, so the whole directory goes in one call. +func (c Slskd) queueAlbumDownload(files []File, track *models.Track) error { + if len(files) == 0 { + return fmt.Errorf("no files to queue for %s - %s", track.CleanTitle, track.Artist) + } + + primary := files[0] + + payload := make([]DownloadPayload, 0, len(files)) + for _, file := range files { + payload = append(payload, DownloadPayload{Filename: string(file.Name), Size: file.Size}) + } + + body, err := json.Marshal(payload) + if err != nil { + return fmt.Errorf("failed to marshal payload: %s", err.Error()) + } + + reqParams := fmt.Sprintf("/api/v0/transfers/downloads/%s", primary.Username) + if _, err := c.HttpClient.MakeRequest("POST", c.Cfg.URL+reqParams, bytes.NewBuffer(body), c.Headers); err != nil { + return fmt.Errorf("couldn't queue release for %s - %s: %s", track.CleanTitle, track.Artist, err.Error()) + } + + // The monitor and the playlist follow the primary; the rest ride along. + track.MainArtistID = primary.Username + track.Size = primary.Size + track.File = string(primary.Name) + + track.AlbumFiles = track.AlbumFiles[:0] + for _, file := range files[1:] { + track.AlbumFiles = append(track.AlbumFiles, string(file.Name)) + } + + slog.Info("queued release", + "track", track.CleanTitle, "artist", track.MainArtist, "files", len(files)) + + return nil +} + +// albumSearchTerm is what we ask slskd for in album mode. Falling back to the +// track keeps releases with no album metadata working. +func albumSearchTerm(track models.Track) string { + if strings.TrimSpace(track.Album) == "" { + return fmt.Sprintf("%s - %s", track.CleanTitle, track.Artist) + } + return fmt.Sprintf("%s - %s", track.MainArtist, track.Album) +} + +// siblingStates reports the transfer state slskd has for each file it is +// downloading from a peer, keyed by the peer's path for that file. +func (c Slskd) siblingStates(username string) (map[string]string, error) { + body, err := c.HttpClient.MakeRequest("GET", c.Cfg.URL+"/api/v0/transfers/downloads", nil, c.Headers) + if err != nil { + return nil, err + } + + var statuses DownloadStatus + if err := util.ParseResp(body, &statuses); err != nil { + return nil, err + } + + states := make(map[string]string) + for _, status := range statuses { + if status.Username != username { + continue + } + for _, dir := range status.Directories { + for _, file := range dir.Files { + states[string(file.Name)] = normalize(file.State) + } + } + } + return states, nil +} + +// MoveAlbumSiblings migrates whatever of the release has finished downloading +// and reports how many files are still in flight, so callers can come back for +// the rest. Doing this once was not enough: the recommended track is queued +// first and so usually finishes first, which left most of the album skipped and +// never looked at again. +// +// A sibling still transferring is left where it is -- copying a partial file +// would put a truncated track in the library. Migrated files are dropped from +// track.AlbumFiles, and so are ones slskd has given up on, which makes repeated +// calls idempotent and lets the count reach zero. +func (c Slskd) MoveAlbumSiblings(trackDir, destDir string, track *models.Track) int { + if !c.Cfg.AlbumMode || len(track.AlbumFiles) == 0 { + return 0 + } + + states, err := c.siblingStates(track.MainArtistID) + if err != nil { + // Nothing is dropped on a failed lookup: the files are still there and + // the next attempt can still find them. + slog.Warn("couldn't check release download states, will retry", + "album", track.Album, "context", err.Error()) + return len(track.AlbumFiles) + } + + var moved, failed int + pending := make([]string, 0, len(track.AlbumFiles)) + + for _, peerPath := range track.AlbumFiles { + state := states[peerPath] + name := path.Base(normalizePeerPath(peerPath)) + + switch { + case strings.Contains(state, "Succeeded"): + if err := moveFile(filepath.Join(trackDir, name), filepath.Join(destDir, name), c.Cfg.KeepPermissions); err != nil { + // Keep it pending: slskd can report a transfer complete a moment + // before the file is readable. + slog.Debug("album track not ready to move yet", "file", name, "context", err.Error()) + pending = append(pending, peerPath) + continue + } + moved++ + + case state == errorState: + // slskd has given up on this one, so waiting for it would only hold + // the release open until the deadline. + slog.Debug("album track failed to download, giving up on it", "file", name) + failed++ + + default: + pending = append(pending, peerPath) + } + } + + track.AlbumFiles = pending + + if moved > 0 || failed > 0 { + slog.Info("migrated release", "album", track.Album, + "moved", moved, "failed", failed, "still downloading", len(pending)) + } + + return len(pending) +} diff --git a/src/downloader/slskd_album_test.go b/src/downloader/slskd_album_test.go new file mode 100644 index 00000000..b4d6d6be --- /dev/null +++ b/src/downloader/slskd_album_test.go @@ -0,0 +1,574 @@ +package downloader + +import ( + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" + + cfg "explo/src/config" + "explo/src/models" +) + +func albumClient(t *testing.T, handler http.HandlerFunc) *Slskd { + t.Helper() + + server := httptest.NewServer(handler) + t.Cleanup(server.Close) + + client := NewSlskd(cfg.Slskd{ + URL: server.URL, + APIKey: "test-key", + Timeout: 5, + AlbumMode: true, + Filters: cfg.Filters{ + Extensions: []string{"flac", "mp3"}, + MinBitRate: 256, + MinBitDepth: 8, + FilterList: []string{"live", "remix"}, + }, + }, "/data/") + client.AddHeader() + + return client +} + +// peer describes one peer sharing one directory. +type peer struct { + user string + dir string + names []string +} + +// albumResults builds search results without naming SearchResults' anonymous +// element type, so new fields on it cannot break these tests. +func albumResults(peers ...peer) SearchResults { + results := make(SearchResults, len(peers)) + for i, p := range peers { + results[i].Username = p.user + results[i].HasFreeUploadSlot = true + results[i].FileCount = len(p.names) + for _, name := range p.names { + results[i].Files = append(results[i].Files, File{ + Name: p.dir + `\` + name, + Extension: "flac", + BitRate: 900, + BitDepth: 16, + Size: 1000, + Length: 200, + }) + } + } + return results +} + +func albumTrack() models.Track { + return models.Track{ + CleanTitle: "Bad Guy", + Title: "Bad Guy", + Artist: "Billie Eilish", + MainArtist: "Billie Eilish", + Album: "When We All Fall Asleep", + Duration: 200_000, + } +} + +// The whole point of album mode: every track in the release is returned, not +// just the one that was recommended. +func TestCollectAlbumFiles_ReturnsWholeRelease(t *testing.T) { + client := albumClient(t, nil) + + results := albumResults(peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, []string{"01 Bury A Friend.flac", "02 Bad Guy.flac", "03 Xanny.flac"}}) + + files, err := client.CollectAlbumFiles(albumTrack(), results) + if err != nil { + t.Fatalf("CollectAlbumFiles() = %v, want nil", err) + } + if len(files) != 3 { + t.Fatalf("got %d files, want the whole 3-track release", len(files)) + } + if !strings.Contains(string(files[0].Name), "Bad Guy") { + t.Errorf("files[0] = %q, want the recommended track first", files[0].Name) + } +} + +// A directory that happens to contain the track but is not the album must lose +// to the one that actually names the album. +func TestCollectAlbumFiles_PrefersTheMatchingAlbum(t *testing.T) { + client := albumClient(t, nil) + + results := albumResults( + peer{"peer1", `@@x\Various\Party Mix 2019`, + []string{"04 Bad Guy.flac", "05 Something Else.flac", "06 Another.flac", "07 More.flac"}}, + peer{"peer2", `@@y\Billie Eilish\When We All Fall Asleep`, + []string{"01 Bury A Friend.flac", "02 Bad Guy.flac"}}, + ) + + files, err := client.CollectAlbumFiles(albumTrack(), results) + if err != nil { + t.Fatalf("CollectAlbumFiles() = %v, want nil", err) + } + if files[0].Username != "peer2" { + t.Errorf("selected peer %q, want peer2 -- the compilation has more files but is the wrong release", files[0].Username) + } +} + +// A release with no copy of the recommended track is useless here, however +// well it otherwise matches. +func TestCollectAlbumFiles_RequiresTheRecommendedTrack(t *testing.T) { + client := albumClient(t, nil) + + results := albumResults(peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, + []string{"01 Bury A Friend.flac", "03 Xanny.flac"}}) + + _, err := client.CollectAlbumFiles(albumTrack(), results) + if err == nil { + t.Fatal("CollectAlbumFiles() = nil, want an error when the recommended track is absent") + } + if !strings.Contains(err.Error(), "no release found") { + t.Errorf("error = %q, want it to say no release was found", err) + } +} + +// FILTER_LIST applies to the directory in album mode, so a live pressing is +// rejected as a whole rather than track by track. +func TestCollectAlbumFiles_RejectsFilteredRelease(t *testing.T) { + client := albumClient(t, nil) + + results := albumResults(peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep (Live)`, []string{"01 Bury A Friend.flac", "02 Bad Guy.flac"}}) + + if _, err := client.CollectAlbumFiles(albumTrack(), results); err == nil { + t.Fatal("CollectAlbumFiles() = nil, want the live release to be filtered out") + } +} + +// Sibling tracks have their own durations, so the recommended track's runtime +// must not be used to exclude them. +func TestCollectAlbumFiles_DoesNotApplyTrackDurationToSiblings(t *testing.T) { + client := albumClient(t, nil) + + results := albumResults(peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, []string{"01 Bury A Friend.flac", "02 Bad Guy.flac"}}) + + // A 6-minute sibling against a 200s recommended track. + results[0].Files[0].Length = 360 + + files, err := client.CollectAlbumFiles(albumTrack(), results) + if err != nil { + t.Fatalf("CollectAlbumFiles() = %v, want nil", err) + } + if len(files) != 2 { + t.Errorf("got %d files, want 2 -- the longer sibling was wrongly excluded", len(files)) + } +} + +// selectionClient builds the same album-mode client as albumClient but +// without the httptest server: release selection never touches the network, +// so these tests also run where binding a local port is forbidden. +func selectionClient() *Slskd { + client := NewSlskd(cfg.Slskd{ + URL: "http://localhost:5030", + APIKey: "test-key", + Timeout: 5, + AlbumMode: true, + Filters: cfg.Filters{ + Extensions: []string{"flac", "mp3"}, + MinBitRate: 256, + MinBitDepth: 8, + FilterList: []string{"live", "remix"}, + }, + }, "/data/") + client.AddHeader() + + return client +} + +// A lone file in an album-named folder is a stray, not the album: the fuller +// release must win even when the stray scores higher on name match alone. +func TestCollectAlbumFiles_PrefersFullReleaseOverSingleFileStray(t *testing.T) { + client := selectionClient() + + // The stray outscores the real thing on name match alone (artist and + // album in the folder vs album only), so only the full-release + // preference can save the correct pick. + results := albumResults( + peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, + []string{"02 Bad Guy.flac"}}, + peer{"peer2", `@@y\Music\When We All Fall Asleep`, + []string{"01 Bury A Friend.flac", "02 Bad Guy.flac", "03 Xanny.flac"}}, + ) + + files, err := client.CollectAlbumFiles(albumTrack(), results) + if err != nil { + t.Fatalf("CollectAlbumFiles() = %v, want nil", err) + } + if files[0].Username != "peer2" { + t.Errorf("selected peer %q, want peer2 -- the single-file stray won over the full release", files[0].Username) + } + if len(files) != 3 { + t.Errorf("got %d files, want the whole 3-track release", len(files)) + } +} + +// A 64-file folder outscores the real 10-track album on the fuller-release +// size term, so the cap has to exclude it from the strict pass. The track +// carries the enriched count, the way a real recommendation does. +func TestCollectAlbumFiles_SkipsAnOversizedDump(t *testing.T) { + client := selectionClient() + track := albumTrack() + track.TrackTotal = 10 + + dump := []string{"01 Bad Guy.flac"} + for i := 2; i <= 64; i++ { + dump = append(dump, "Dump Track.flac") + } + correct := []string{"01 Bury A Friend.flac", "02 Bad Guy.flac", "03 Xanny.flac", + "04 Everything I Wanted.flac", "05 Ocean Eyes.flac", "06 Lovely.flac", + "07 You Should See Me.flac", "08 All The Good Girls.flac", + "09 Wish You Were Gay.flac", "10 Ilomilo.flac"} + + results := albumResults( + peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, dump}, + peer{"peer2", `@@y\Billie Eilish\When We All Fall Asleep`, correct}, + ) + + files, err := client.CollectAlbumFiles(track, results) + if err != nil { + t.Fatalf("CollectAlbumFiles() = %v, want nil", err) + } + if files[0].Username != "peer2" { + t.Errorf("selected peer %q, want peer2 -- the 64-file dump won over the 10-track album", files[0].Username) + } + if len(files) != 10 { + t.Errorf("got %d files, want the 10-track album", len(files)) + } +} + +// The dump stays eligible too: with nothing right-sized available, the big +// folder still beats no music. +func TestCollectAlbumFiles_FallsBackToADumpWhenNothingElse(t *testing.T) { + client := selectionClient() + track := albumTrack() + track.TrackTotal = 10 + + dump := []string{"01 Bad Guy.flac"} + for i := 2; i <= 64; i++ { + dump = append(dump, "Dump Track.flac") + } + + results := albumResults(peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, dump}) + + files, err := client.CollectAlbumFiles(track, results) + if err != nil { + t.Fatalf("CollectAlbumFiles() = %v, want nil", err) + } + if len(files) != 64 { + t.Errorf("got %d files, want the 64-file dump as a last resort", len(files)) + } +} + +// The stray stays eligible: with nothing fuller available, one song still +// beats no song. +func TestCollectAlbumFiles_FallsBackToSingleFileWhenNothingElse(t *testing.T) { + client := selectionClient() + + results := albumResults(peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, + []string{"02 Bad Guy.flac"}}) + + files, err := client.CollectAlbumFiles(albumTrack(), results) + if err != nil { + t.Fatalf("CollectAlbumFiles() = %v, want nil", err) + } + if len(files) != 1 { + t.Errorf("got %d files, want the lone stray as a last resort", len(files)) + } +} + +// One request per release, carrying every file, and the track left pointing at +// the recommended one so the monitor and the playlist still work. +func TestQueueAlbumDownload_QueuesEveryFileInOneRequest(t *testing.T) { + var gotPayload []DownloadPayload + var gotPath string + var requests int + + client := albumClient(t, func(w http.ResponseWriter, r *http.Request) { + requests++ + gotPath = r.URL.Path + body, err := io.ReadAll(r.Body) + if err != nil { + t.Errorf("reading request body: %v", err) + } + if err := json.Unmarshal(body, &gotPayload); err != nil { + t.Errorf("unmarshaling request body: %v", err) + } + if _, err := w.Write([]byte(`{}`)); err != nil { + t.Errorf("fake slskd failed to write response: %v", err) + } + }) + + results := albumResults(peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, []string{"01 Bury A Friend.flac", "02 Bad Guy.flac", "03 Xanny.flac"}}) + + files, err := client.CollectAlbumFiles(albumTrack(), results) + if err != nil { + t.Fatalf("CollectAlbumFiles() = %v, want nil", err) + } + + track := albumTrack() + if err := client.queueAlbumDownload(files, &track); err != nil { + t.Fatalf("queueAlbumDownload() = %v, want nil", err) + } + + if requests != 1 { + t.Errorf("made %d requests, want 1 for the whole release", requests) + } + if gotPath != "/api/v0/transfers/downloads/peer1" { + t.Errorf("posted to %q, want the peer's download endpoint", gotPath) + } + if len(gotPayload) != 3 { + t.Errorf("queued %d files, want 3", len(gotPayload)) + } + if !strings.Contains(track.File, "Bad Guy") { + t.Errorf("track.File = %q, want the recommended track -- the monitor follows this", track.File) + } + if len(track.AlbumFiles) != 2 { + t.Errorf("AlbumFiles has %d entries, want the 2 siblings", len(track.AlbumFiles)) + } + for _, sibling := range track.AlbumFiles { + if sibling == track.File { + t.Error("the recommended track is also listed as a sibling; it would be migrated twice") + } + } +} + +// Album mode searches for the release. Without an album name there is nothing +// to search for, so it must fall back to the track. +func TestAlbumSearchTerm(t *testing.T) { + track := albumTrack() + if got, want := albumSearchTerm(track), "Billie Eilish - When We All Fall Asleep"; got != want { + t.Errorf("albumSearchTerm() = %q, want %q", got, want) + } + + track.Album = " " + if got, want := albumSearchTerm(track), "Bad Guy - Billie Eilish"; got != want { + t.Errorf("with no album, albumSearchTerm() = %q, want the track fallback %q", got, want) + } +} + +// Peers report Windows paths; grouping by directory has to survive that or +// every file looks like it lives in its own release. +func TestNormalizePeerPathGrouping(t *testing.T) { + client := albumClient(t, nil) + + results := albumResults(peer{"peer1", `@@x\Billie Eilish\When We All Fall Asleep`, []string{"01 Bury A Friend.flac", "02 Bad Guy.flac"}}) + + dirs := client.groupByDirectory(results) + if len(dirs) != 1 { + t.Fatalf("grouped into %d directories, want 1 -- backslash paths were not normalised", len(dirs)) + } + if len(dirs[0].files) != 2 { + t.Errorf("directory holds %d files, want 2", len(dirs[0].files)) + } +} + +// downloadsResponse fakes GET /api/v0/transfers/downloads: one peer, one +// directory, and whatever per-file transfer states the test needs. +func downloadsResponse(t *testing.T, user string, states map[string]string) string { + t.Helper() + + files := make([]map[string]any, 0, len(states)) + for name, state := range states { + files = append(files, map[string]any{"filename": name, "state": state}) + } + + body, err := json.Marshal([]map[string]any{{ + "username": user, + "directories": []map[string]any{{"files": files}}, + }}) + if err != nil { + t.Fatalf("building fake downloads response: %v", err) + } + return string(body) +} + +// A sibling slskd has finished belongs in the library; one still transferring +// must be left alone, because copying it would put a truncated track there. +func TestMoveAlbumSiblings_SkipsUnfinishedTransfers(t *testing.T) { + const ( + peerDirectory = `@@x\Billie Eilish\When We All Fall Asleep` + done = peerDirectory + `\01 Bury A Friend.flac` + pending = peerDirectory + `\03 Xanny.flac` + ) + + client := albumClient(t, func(w http.ResponseWriter, r *http.Request) { + if _, err := w.Write([]byte(downloadsResponse(t, "peer1", map[string]string{ + done: "Completed, Succeeded", + pending: "InProgress", + }))); err != nil { + t.Errorf("fake slskd failed to write response: %v", err) + } + }) + + srcDir := t.TempDir() + destDir := t.TempDir() + for _, name := range []string{"01 Bury A Friend.flac", "03 Xanny.flac"} { + if err := os.WriteFile(filepath.Join(srcDir, name), []byte(name), 0o644); err != nil { + t.Fatalf("seeding %s: %v", name, err) + } + } + + track := albumTrack() + track.MainArtistID = "peer1" // queueAlbumDownload stashes the peer here + track.AlbumFiles = []string{done, pending} + + client.MoveAlbumSiblings(srcDir, destDir, &track) + + if _, err := os.Stat(filepath.Join(destDir, "01 Bury A Friend.flac")); err != nil { + t.Errorf("finished sibling was not migrated: %v", err) + } + if _, err := os.Stat(filepath.Join(srcDir, "01 Bury A Friend.flac")); !os.IsNotExist(err) { + t.Error("finished sibling was copied but the original was left behind") + } + if _, err := os.Stat(filepath.Join(destDir, "03 Xanny.flac")); !os.IsNotExist(err) { + t.Error("a still-transferring sibling was migrated; the library would get a truncated file") + } + if _, err := os.Stat(filepath.Join(srcDir, "03 Xanny.flac")); err != nil { + t.Errorf("still-transferring sibling should be left in place: %v", err) + } +} + +// Album mode is opt-in, so the migration must be inert when it is off even if +// a track somehow carries siblings. +func TestMoveAlbumSiblings_NoopWhenAlbumModeIsOff(t *testing.T) { + client := albumClient(t, func(w http.ResponseWriter, r *http.Request) { + t.Error("slskd was queried with album mode off") + }) + client.Cfg.AlbumMode = false + + srcDir := t.TempDir() + destDir := t.TempDir() + if err := os.WriteFile(filepath.Join(srcDir, "01 Bury A Friend.flac"), []byte("x"), 0o644); err != nil { + t.Fatalf("seeding sibling: %v", err) + } + + track := albumTrack() + track.MainArtistID = "peer1" + track.AlbumFiles = []string{`@@x\dir\01 Bury A Friend.flac`} + + client.MoveAlbumSiblings(srcDir, destDir, &track) + + if _, err := os.Stat(filepath.Join(srcDir, "01 Bury A Friend.flac")); err != nil { + t.Errorf("sibling was migrated with album mode off: %v", err) + } +} + +// *Slskd must satisfy the optional interface DownloadClient looks for, or the +// monitor's release migration silently never fires. +func TestSlskdImplementsAlbumMigrator(t *testing.T) { + var _ albumMigrator = &Slskd{} +} + +// The bug this guards: the recommended track is queued first and normally +// finishes first, so the siblings are still transferring at the moment the +// release is first migrated. They have to be picked up on a later pass instead +// of being skipped once and abandoned. +func TestMoveAlbumSiblings_PicksUpSiblingsThatFinishLater(t *testing.T) { + const ( + peerDirectory = `@@x\Billie Eilish\When We All Fall Asleep` + early = peerDirectory + `\01 Bury A Friend.flac` + late = peerDirectory + `\03 Xanny.flac` + ) + + // The state the slow sibling reports, flipped between attempts. + lateState := "InProgress" + + client := albumClient(t, func(w http.ResponseWriter, r *http.Request) { + if _, err := w.Write([]byte(downloadsResponse(t, "peer1", map[string]string{ + early: "Completed, Succeeded", + late: lateState, + }))); err != nil { + t.Errorf("fake slskd failed to write response: %v", err) + } + }) + + srcDir := t.TempDir() + destDir := t.TempDir() + for _, name := range []string{"01 Bury A Friend.flac", "03 Xanny.flac"} { + if err := os.WriteFile(filepath.Join(srcDir, name), []byte(name), 0o644); err != nil { + t.Fatalf("seeding %s: %v", name, err) + } + } + + track := albumTrack() + track.MainArtistID = "peer1" + track.AlbumFiles = []string{early, late} + + if remaining := client.MoveAlbumSiblings(srcDir, destDir, &track); remaining != 1 { + t.Fatalf("first pass left %d files pending, want 1", remaining) + } + if _, err := os.Stat(filepath.Join(destDir, "03 Xanny.flac")); !os.IsNotExist(err) { + t.Fatal("the unfinished sibling was migrated on the first pass") + } + + lateState = "Completed, Succeeded" + + if remaining := client.MoveAlbumSiblings(srcDir, destDir, &track); remaining != 0 { + t.Errorf("second pass left %d files pending, want 0", remaining) + } + if _, err := os.Stat(filepath.Join(destDir, "03 Xanny.flac")); err != nil { + t.Errorf("the sibling that finished later was never migrated: %v", err) + } +} + +// Already-migrated files must not be retried: the source is gone, so a second +// attempt would fail forever and hold the release open until the deadline. +func TestMoveAlbumSiblings_ForgetsMigratedFiles(t *testing.T) { + const done = `@@x\dir\01 Bury A Friend.flac` + + client := albumClient(t, func(w http.ResponseWriter, r *http.Request) { + if _, err := w.Write([]byte(downloadsResponse(t, "peer1", map[string]string{ + done: "Completed, Succeeded", + }))); err != nil { + t.Errorf("fake slskd failed to write response: %v", err) + } + }) + + srcDir := t.TempDir() + if err := os.WriteFile(filepath.Join(srcDir, "01 Bury A Friend.flac"), []byte("x"), 0o644); err != nil { + t.Fatalf("seeding sibling: %v", err) + } + + track := albumTrack() + track.MainArtistID = "peer1" + track.AlbumFiles = []string{done} + + client.MoveAlbumSiblings(srcDir, t.TempDir(), &track) + + if len(track.AlbumFiles) != 0 { + t.Errorf("AlbumFiles still holds %v after migration", track.AlbumFiles) + } +} + +// A sibling slskd has given up on must not keep the release open, or the +// monitor waits out the full deadline for a file that is never coming. +func TestMoveAlbumSiblings_DropsFailedSiblings(t *testing.T) { + const failed = `@@x\dir\03 Xanny.flac` + + client := albumClient(t, func(w http.ResponseWriter, r *http.Request) { + if _, err := w.Write([]byte(downloadsResponse(t, "peer1", map[string]string{ + failed: "Completed, Errored", + }))); err != nil { + t.Errorf("fake slskd failed to write response: %v", err) + } + }) + + track := albumTrack() + track.MainArtistID = "peer1" + track.AlbumFiles = []string{failed} + + if remaining := client.MoveAlbumSiblings(t.TempDir(), t.TempDir(), &track); remaining != 0 { + t.Errorf("a failed sibling left %d files pending, want 0", remaining) + } +} diff --git a/src/downloader/slskd_shared_release.go b/src/downloader/slskd_shared_release.go new file mode 100644 index 00000000..f6167351 --- /dev/null +++ b/src/downloader/slskd_shared_release.go @@ -0,0 +1,258 @@ +package downloader + +import ( + "fmt" + "log/slog" + "path" + "slices" + "strings" + "sync" + + "explo/src/models" + "explo/src/util" +) + +// In album mode every recommended track used to fetch its own copy of its +// album. A playlist carrying three songs from one record therefore downloaded +// that record three times, usually from three different peers, and the copies +// all landed in the library. releaseRegistry makes the first track of an album +// the one that downloads it; the others wait for that release and take their +// own file out of it. + +// trackRole is how a track is being fetched within a run. +type trackRole int + +const ( + // roleAlbum is the default: the track searches for and queues its own release. + roleAlbum trackRole = iota + // roleLeader is roleAlbum for a track whose release others are waiting on. + roleLeader + // roleShared tracks were found in a release another track queued, so + // there is nothing left to download for them. + roleShared + // roleSingle tracks share an album with a queued release that lacks them, + // so they fall back to a single-track download. + roleSingle +) + +type trackState struct { + role trackRole + key string + release *sharedRelease +} + +// sharedRelease is one release queued on behalf of every track from its album. +type sharedRelease struct { + done chan struct{} + leader *models.Track + // files is what the leader queued, primary first. It stays nil if the + // leader failed, which tells waiting tracks to try the album themselves. + files []File + // claimed holds the files already spoken for, so two tracks can never be + // handed the same file. + claimed map[string]bool +} + +type releaseRegistry struct { + mu sync.Mutex + releases map[string]*sharedRelease + tracks map[*models.Track]*trackState + // sharedIDs are the IDs handed to shared tracks. They name no slskd + // search, so Cleanup must not try to delete one. + sharedIDs map[string]bool +} + +func newReleaseRegistry() *releaseRegistry { + return &releaseRegistry{ + releases: make(map[string]*sharedRelease), + tracks: make(map[*models.Track]*trackState), + sharedIDs: make(map[string]bool), + } +} + +// albumKey identifies the album a track belongs to. The release group is +// preferred because it is the same for every edition of an album, which the +// album name is not. Tracks with no album cannot share a release. +func albumKey(track models.Track) string { + if track.MusicBrainzReleaseGroupID != "" { + return "rg:" + track.MusicBrainzReleaseGroupID + } + artist := strings.ToLower(util.AlnumOnly(track.MainArtist)) + album := strings.ToLower(util.AlnumOnly(track.Album)) + if artist == "" || album == "" { + return "" + } + return "name:" + artist + "\x00" + album +} + +// claim returns the release for key, creating it if this is the first track +// to ask. The creator is the leader and must resolve it with publish or fail. +func (r *releaseRegistry) claim(key string, track *models.Track) (*sharedRelease, bool) { + r.mu.Lock() + defer r.mu.Unlock() + + if rel, ok := r.releases[key]; ok { + return rel, false + } + + rel := &sharedRelease{ + done: make(chan struct{}), + leader: track, + claimed: make(map[string]bool), + } + r.releases[key] = rel + r.tracks[track] = &trackState{role: roleLeader, key: key, release: rel} + return rel, true +} + +// publish hands the queued release to every track waiting on it. +func (r *releaseRegistry) publish(state *trackState, files []File) { + r.mu.Lock() + defer r.mu.Unlock() + + state.release.files = files + state.release.claimed[files[0].Name] = true + close(state.release.done) +} + +// fail releases the claim so that a waiting track can lead the album instead. +// Failing is specific to the leader -- its track may be missing from every +// release found -- so another track from the album may well succeed. +func (r *releaseRegistry) fail(state *trackState) { + r.mu.Lock() + defer r.mu.Unlock() + + if r.releases[state.key] == state.release { + delete(r.releases, state.key) + } + state.role = roleAlbum + close(state.release.done) +} + +func (r *releaseRegistry) state(track *models.Track) *trackState { + if r == nil { + return nil + } + r.mu.Lock() + defer r.mu.Unlock() + return r.tracks[track] +} + +func (r *releaseRegistry) setRole(track *models.Track, role trackRole) { + r.mu.Lock() + defer r.mu.Unlock() + r.tracks[track] = &trackState{role: role} +} + +func (r *releaseRegistry) isSharedID(id string) bool { + if r == nil { + return false + } + r.mu.Lock() + defer r.mu.Unlock() + return r.sharedIDs[id] +} + +// share looks for track inside a published release. If it is there, the track +// follows that file instead of downloading anything: it is taken out of the +// leader's siblings so the monitor migrates it as a track in its own right, +// with its own metadata and path. +func (r *releaseRegistry) share(rel *sharedRelease, track *models.Track) bool { + r.mu.Lock() + defer r.mu.Unlock() + + candidates := peerDir{username: rel.files[0].Username} + for _, file := range rel.files { + if !rel.claimed[file.Name] { + candidates.files = append(candidates.files, file) + } + } + if !findPrimary(&candidates, *track) { + return false + } + file := *candidates.primary + rel.claimed[file.Name] = true + + rel.leader.AlbumFiles = slices.DeleteFunc(rel.leader.AlbumFiles, func(name string) bool { + return name == string(file.Name) + }) + + // The monitor looks statuses up by ID, so each shared track needs one of + // its own even though it has no search behind it. + track.ID = fmt.Sprintf("%s:shared:%d", rel.leader.ID, len(rel.claimed)) + r.sharedIDs[track.ID] = true + r.tracks[track] = &trackState{role: roleShared} + + track.MainArtistID = file.Username + track.Size = file.Size + track.File = string(file.Name) + + slog.Info("track found in a release already queued, not downloading it again", + "track", track.CleanTitle, "album", track.Album, "file", path.Base(normalizePeerPath(track.File))) + return true +} + +// queryAlbum is QueryTrack in album mode. The first track of an album searches +// for it; the rest wait to see what it queued. +func (c *Slskd) queryAlbum(track *models.Track) error { + key := albumKey(*track) + if c.releases == nil || key == "" { + return c.search(track, true) + } + + for { + rel, leader := c.releases.claim(key, track) + if leader { + if err := c.search(track, true); err != nil { + c.releases.fail(c.releases.state(track)) + return err + } + return nil + } + + <-rel.done + if rel.files == nil { + // The leader found nothing. Try to lead the album ourselves. + continue + } + if c.releases.share(rel, track) { + return nil + } + + slog.Info("track missing from the release queued for its album, downloading it alone", + "track", track.CleanTitle, "album", track.Album) + c.releases.setRole(track, roleSingle) + return c.search(track, false) + } +} + +// getAlbum is GetTrack in album mode. A leader resolves its release whatever +// happens, since other tracks are blocked until it does. +func (c *Slskd) getAlbum(track *models.Track) error { + state := c.releases.state(track) + + files, err := c.queueAlbum(track) + if state != nil && state.role == roleLeader { + if err != nil { + c.releases.fail(state) + } else { + c.releases.publish(state, files) + } + } + return err +} + +func (c *Slskd) queueAlbum(track *models.Track) ([]File, error) { + results, err := c.searchResults(track.ID) + if err != nil { + return nil, err + } + files, err := c.CollectAlbumFiles(*track, results) + if err != nil { + return nil, err + } + if err := c.queueAlbumDownload(files, track); err != nil { + return nil, err + } + return files, nil +} diff --git a/src/downloader/slskd_shared_release_test.go b/src/downloader/slskd_shared_release_test.go new file mode 100644 index 00000000..8b25cfe8 --- /dev/null +++ b/src/downloader/slskd_shared_release_test.go @@ -0,0 +1,281 @@ +package downloader + +import ( + "encoding/json" + "fmt" + "io" + "net/http" + "slices" + "strings" + "sync" + "testing" + + "explo/src/models" + + "golang.org/x/sync/errgroup" +) + +// fakeSlskd answers searches from a fixed catalogue and records what was +// searched for and queued. +type fakeSlskd struct { + t *testing.T + catalog func(query string) SearchResults + + mu sync.Mutex + searches []string + byID map[string]string + queued [][]DownloadPayload + deleted []string +} + +func (f *fakeSlskd) handler(w http.ResponseWriter, r *http.Request) { + f.mu.Lock() + defer f.mu.Unlock() + + var resp any = struct{}{} + p := r.URL.Path + + switch { + case r.Method == http.MethodPost && p == "/api/v0/searches": + var body SearchPayload + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + f.t.Errorf("decoding search: %v", err) + } + f.searches = append(f.searches, body.SearchText) + id := fmt.Sprintf("search-%d", len(f.searches)) + f.byID[id] = body.SearchText + resp = Search{ID: id} + + case r.Method == http.MethodGet && strings.HasSuffix(p, "/responses"): + id := strings.TrimSuffix(strings.TrimPrefix(p, "/api/v0/searches/"), "/responses") + resp = f.catalog(f.byID[id]) + + case r.Method == http.MethodGet && strings.HasPrefix(p, "/api/v0/searches/"): + id := strings.TrimPrefix(p, "/api/v0/searches/") + count := 0 + for _, result := range f.catalog(f.byID[id]) { + count += len(result.Files) + } + resp = Search{ID: id, IsComplete: true, FileCount: count} + + case r.Method == http.MethodDelete && strings.HasPrefix(p, "/api/v0/searches/"): + f.deleted = append(f.deleted, strings.TrimPrefix(p, "/api/v0/searches/")) + + case r.Method == http.MethodPost && strings.HasPrefix(p, "/api/v0/transfers/downloads/"): + body, _ := io.ReadAll(r.Body) + var payload []DownloadPayload + if err := json.Unmarshal(body, &payload); err != nil { + f.t.Errorf("decoding queue request: %v", err) + } + f.queued = append(f.queued, payload) + } + + if err := json.NewEncoder(w).Encode(resp); err != nil { + f.t.Errorf("fake slskd failed to write response: %v", err) + } +} + +func newFakeSlskd(t *testing.T, catalog func(string) SearchResults) (*fakeSlskd, *Slskd) { + t.Helper() + fake := &fakeSlskd{t: t, catalog: catalog, byID: make(map[string]string)} + client := albumClient(t, fake.handler) + return fake, client +} + +var rioFiles = []string{ + "01 Rio.flac", "02 My Own Way.flac", "03 Lonely in Your Nightboat.flac", + "04 Hungry Like the Wolf.flac", "05 Hold Back the Rain.flac", "08 Save a Prayer.flac", +} + +// rioCatalog shares Rio from two peers, the way the same album usually turns +// up more than once on Soulseek. +func rioCatalog(query string) SearchResults { + if strings.Contains(query, "Rio") { + return albumResults( + peer{"peer1", `@@a\Duran Duran\Rio`, rioFiles}, + peer{"peer2", `@@b\Music\Duran Duran - Rio (1982)`, rioFiles}, + ) + } + return nil +} + +func rioTrack(title string) *models.Track { + return &models.Track{ + CleanTitle: title, + Title: title, + Artist: "Duran Duran", + MainArtist: "Duran Duran", + Album: "Rio", + MusicBrainzReleaseGroupID: "rio-rg", + Duration: 200_000, + } +} + +// runTracks drives tracks the way StartDownload does, concurrently. +func runTracks(client *Slskd, tracks ...*models.Track) { + var g errgroup.Group + g.SetLimit(3) + for _, track := range tracks { + g.Go(func() error { + if err := client.QueryTrack(track); err != nil { + return nil + } + _ = client.GetTrack(track) + return nil + }) + } + _ = g.Wait() +} + +// The reported bug: three songs from Rio in one playlist downloaded Rio three +// times. One release must be queued, and each song must follow its own file +// in it. +func TestAlbumMode_DownloadsEachAlbumOnce(t *testing.T) { + fake, client := newFakeSlskd(t, rioCatalog) + + tracks := []*models.Track{rioTrack("Rio"), rioTrack("Hungry Like the Wolf"), rioTrack("Save a Prayer")} + runTracks(client, tracks...) + + if len(fake.searches) != 1 { + t.Errorf("searched %d times (%q), want once for the album", len(fake.searches), fake.searches) + } + if len(fake.queued) != 1 { + t.Fatalf("queued %d releases, want 1", len(fake.queued)) + } + if len(fake.queued[0]) != len(rioFiles) { + t.Errorf("queued %d files, want the whole %d-track release", len(fake.queued[0]), len(rioFiles)) + } + + ids := make(map[string]bool) + var leader *models.Track + for _, track := range tracks { + if track.ID == "" { + t.Errorf("%s has no ID; the monitor would never look at it", track.CleanTitle) + } + if ids[track.ID] { + t.Errorf("%s shares ID %q with another track; the monitor would confuse them", track.CleanTitle, track.ID) + } + ids[track.ID] = true + + if !strings.Contains(track.File, track.CleanTitle) { + t.Errorf("%s follows %q, want its own file", track.CleanTitle, track.File) + } + if len(track.AlbumFiles) > 0 { + leader = track + } + } + + if leader == nil { + t.Fatal("no track kept the release's siblings") + } + for _, track := range tracks { + if slices.Contains(leader.AlbumFiles, track.File) { + t.Errorf("%s is still a sibling of the release; it would be migrated twice", track.CleanTitle) + } + } + if want := len(rioFiles) - len(tracks); len(leader.AlbumFiles) != want { + t.Errorf("release has %d siblings left, want %d", len(leader.AlbumFiles), want) + } +} + +// A song the queued release lacks still has to be downloaded, on its own +// rather than as yet another copy of the album. +func TestAlbumMode_TrackMissingFromReleaseDownloadsAlone(t *testing.T) { + bonus := "Rio (Night Version)" + fake, client := newFakeSlskd(t, func(query string) SearchResults { + if strings.Contains(query, bonus) { + return albumResults(peer{"peer3", `@@c\Duran Duran\Singles`, []string{"Rio (Night Version).flac"}}) + } + return rioCatalog(query) + }) + client.Cfg.DownloadAttempts = 1 + + runTracks(client, rioTrack("Hungry Like the Wolf")) + single := rioTrack(bonus) + runTracks(client, single) + + if got := fake.searches[len(fake.searches)-1]; got != bonus+" - Duran Duran" { + t.Errorf("last search = %q, want a single-track search", got) + } + if len(fake.queued) != 2 || len(fake.queued[1]) != 1 { + t.Fatalf("queued %v, want the release then one file", fake.queued) + } + if !strings.Contains(single.File, "Night Version") { + t.Errorf("%s follows %q, want the single file", single.CleanTitle, single.File) + } +} + +// If the first track of an album finds no release with itself in it, the +// album is not written off: the next track tries it. +func TestAlbumMode_AnotherTrackLeadsWhenTheFirstFails(t *testing.T) { + fake, client := newFakeSlskd(t, rioCatalog) + + unfindable := rioTrack("Not On The Album") + runTracks(client, unfindable) + found := rioTrack("Save a Prayer") + runTracks(client, found) + + if len(fake.queued) != 1 { + t.Fatalf("queued %d releases, want 1 from the second track", len(fake.queued)) + } + if !strings.Contains(found.File, "Save a Prayer") { + t.Errorf("track follows %q, want its own file", found.File) + } +} + +func TestAlbumMode_DifferentAlbumsAreNotShared(t *testing.T) { + fake, client := newFakeSlskd(t, func(query string) SearchResults { + if strings.Contains(query, "Seven and the Ragged Tiger") { + return albumResults(peer{"peer4", `@@d\Duran Duran\Seven and the Ragged Tiger`, []string{"01 The Reflex.flac", "02 New Moon on Monday.flac"}}) + } + return rioCatalog(query) + }) + + other := rioTrack("The Reflex") + other.Album, other.MusicBrainzReleaseGroupID = "Seven and the Ragged Tiger", "seven-rg" + runTracks(client, rioTrack("Rio"), other) + + if len(fake.queued) != 2 { + t.Errorf("queued %d releases, want one per album", len(fake.queued)) + } +} + +// Shared tracks carry an ID that names no search, so cleaning up after one +// must not ask slskd to delete it. +func TestCleanup_SkipsSearchForSharedTracks(t *testing.T) { + fake, client := newFakeSlskd(t, rioCatalog) + + leader, shared := rioTrack("Rio"), rioTrack("Save a Prayer") + runTracks(client, leader) + runTracks(client, shared) + + if !client.releases.isSharedID(shared.ID) { + t.Fatalf("%s was not shared from the release", shared.CleanTitle) + } + fake.deleted = nil + if err := client.Cleanup(*shared, "file-id"); err != nil { + t.Fatalf("Cleanup() = %v", err) + } + if len(fake.deleted) != 0 { + t.Errorf("deleted searches %q for a shared track, want none", fake.deleted) + } +} + +func TestAlbumKey(t *testing.T) { + a, b := rioTrack("Rio"), rioTrack("Rio") + b.Album = "Rio (2009 Collector's Edition)" + if albumKey(*a) != albumKey(*b) { + t.Error("editions of one release group got different keys") + } + + a.MusicBrainzReleaseGroupID, b.MusicBrainzReleaseGroupID = "", "" + b.Album = "RIO" + if albumKey(*a) != albumKey(*b) { + t.Error("album names differing only in case got different keys") + } + + a.Album = "" + if albumKey(*a) != "" { + t.Error("a track with no album got a key; it would be grouped with others") + } +} diff --git a/src/downloader/youtube.go b/src/downloader/youtube.go index f09795d3..d47c9c17 100644 --- a/src/downloader/youtube.go +++ b/src/downloader/youtube.go @@ -258,15 +258,18 @@ func saveVideo(c Youtube, track *models.Track, stream *goutubedl.DownloadResult) if track.CoverPath == "" { _, track.CoverPath = util.DownloadCover(track.CoverURL, c.Cfg.CoversDir) } - streams = append(streams, ffmpeg.Input(track.CoverPath)) - opts = ffmpeg.KwArgs{ - "c:v": "mjpeg", - "disposition:v:0": "attached_pic", - "metadata:s:v": []string{"title=Album cover", "comment=Cover (front)"}, - "metadata": metadata, - "loglevel": "error", + if track.CoverPath != "" { + streams = append(streams, ffmpeg.Input(track.CoverPath)) + opts = ffmpeg.KwArgs{ + "c:v": "mjpeg", + "disposition:v:0": "attached_pic", + "metadata:s:v": []string{"title=Album cover", "comment=Cover (front)"}, + "metadata": metadata, + "loglevel": "error", + } } - } else { + } + if opts == nil { opts = ffmpeg.KwArgs{ "map": "0:a", "metadata": metadata, @@ -323,7 +326,7 @@ func (c *Youtube) Cleanup(track models.Track, ID string) error { return nil } -func (c *Youtube) MoveDownload(srcDir, destDir, trackPath string, track *models.Track) error { +func (c *Youtube) MoveDownload(srcDir, destDir, trackPath string, track *models.Track) (string, error) { - return nil + return "", nil } diff --git a/src/main/main.go b/src/main/main.go index 19f9823f..462045f2 100644 --- a/src/main/main.go +++ b/src/main/main.go @@ -194,8 +194,12 @@ func main() { } } } else { - disc := discovery.NewDiscoverer(cfg.DiscoveryCfg, httpClient) - tracks, err = disc.Discover() + disc, discErr := discovery.NewDiscoverer(cfg.DiscoveryCfg, httpClient) + if discErr != nil { + err = discErr + } else { + tracks, err = disc.Discover() + } } if err != nil { diff --git a/src/models/types.go b/src/models/types.go index 27999573..0779968f 100644 --- a/src/models/types.go +++ b/src/models/types.go @@ -18,6 +18,7 @@ type Track struct { CleanTitle string // Title as returned by LB Title string // Title as built in listenbrainz.go File string // File name + AlbumFiles []string // Sibling files queued with File when album mode is on Size int // File size Present bool // is track present in the system or not Duration int // Track duration in milliseconds (not available for every track) @@ -32,6 +33,12 @@ type Track struct { TrackTotal int // Total tracks in media DiscNumber int // Disc/media position DiscTotal int // Total discs/media + // Canonical* describe the album as its release group agrees it is, rather + // than as the one matched release happens to be. A recommendation resolving + // to a deluxe edition still reports the standard album's shape here, which + // is what album mode needs to recognise a padded release. + CanonicalTrackTotal int // Modal first-medium track count across the release group + CanonicalDiscTotal int // Modal medium count across the release group MusicBrainzReleaseGroupID string MusicBrainzAlbumArtistID string MusicBrainzTrackID string diff --git a/src/util/cover_test.go b/src/util/cover_test.go new file mode 100644 index 00000000..f41e47f5 --- /dev/null +++ b/src/util/cover_test.go @@ -0,0 +1,20 @@ +package util + +import ( + "testing" +) + +// BUG-25: DownloadCover returned the destination path even when the +// fetch failed, so callers embedded a cover from a nonexistent file +// and the track save failed with the artwork. +func TestDownloadCover_FailedFetchReturnsEmptyPaths(t *testing.T) { + api, path := DownloadCover("http://127.0.0.1:1/cover.jpg", t.TempDir()) + if api != "" || path != "" { + t.Errorf("DownloadCover(failed) = %q, %q; want empty paths so callers proceed without cover art", api, path) + } + + api, path = DownloadCover("", t.TempDir()) + if api != "" || path != "" { + t.Errorf("DownloadCover(empty url) = %q, %q; want empty paths", api, path) + } +} diff --git a/src/util/http.go b/src/util/http.go index 588e3b04..0ed72b6e 100644 --- a/src/util/http.go +++ b/src/util/http.go @@ -15,6 +15,32 @@ import ( "explo/src/logging" ) +// Bounds for ad-hoc remote fetches: a stalled server must not hang the +// app forever and an oversized response must not be read into memory +// without limit (BUG-20). +const ( + maxAPIResponseBytes = 32 << 20 + maxDownloadBytes = 64 << 20 + maxCoverBytes = 16 << 20 +) + +// remoteHTTPClient replaces http.Get/DefaultClient, which have no +// timeout at all, for file and cover downloads. +var remoteHTTPClient = &http.Client{Timeout: 60 * time.Second} + +// readLimited reads at most max bytes; one extra byte is read so an +// over-limit response is an error, not a silent truncation. +func readLimited(r io.Reader, max int64) ([]byte, error) { + data, err := io.ReadAll(io.LimitReader(r, max+1)) + if err != nil { + return nil, err + } + if int64(len(data)) > max { + return nil, fmt.Errorf("response exceeds %d bytes", max) + } + return data, nil +} + type HttpClientConfig struct { Timeout int } @@ -57,7 +83,7 @@ func (c *HttpClient) MakeRequest(method, url string, payload io.Reader, headers } }() - body, err := io.ReadAll(resp.Body) + body, err := readLimited(resp.Body, maxAPIResponseBytes) if err != nil { return nil, fmt.Errorf("failed to read response body: %s", err.Error()) } @@ -91,7 +117,7 @@ func DownloadFile(url, destPath string) (string, error) { if err := os.MkdirAll(filepath.Dir(destPath), 0755); err != nil { return "", fmt.Errorf("mkdir: %w", err) } - resp, err := http.Get(url) //nolint:noctx + resp, err := remoteHTTPClient.Get(url) if err != nil { return "", fmt.Errorf("get: %w", err) } @@ -103,7 +129,7 @@ func DownloadFile(url, destPath string) (string, error) { if resp.StatusCode != http.StatusOK { return "", fmt.Errorf("status %d from %s", resp.StatusCode, url) } - data, err := io.ReadAll(resp.Body) + data, err := readLimited(resp.Body, maxDownloadBytes) if err != nil { return "", fmt.Errorf("read body: %w", err) } @@ -134,7 +160,7 @@ func DownloadCover(url, coversDir string) (string, string) { } destPath := filepath.Join(coversDir, id+".jpg") if _, err := os.Stat(destPath); os.IsNotExist(err) { - resp, err := http.Get(url) //nolint:noctx + resp, err := remoteHTTPClient.Get(url) if err == nil { func() { defer func() { @@ -143,7 +169,7 @@ func DownloadCover(url, coversDir string) (string, string) { } }() if resp.StatusCode == http.StatusOK { - if data, err := io.ReadAll(resp.Body); err == nil { + if data, err := readLimited(resp.Body, maxCoverBytes); err == nil { if err := os.WriteFile(destPath, data, 0644); err != nil { slog.Error("failed writing cover", "path", destPath, "err", err.Error()) } @@ -152,6 +178,12 @@ func DownloadCover(url, coversDir string) (string, string) { }() } } + // A failed fetch or write used to return the destination path anyway, + // so callers embedded a cover from a file that does not exist and the + // track save failed with it (BUG-25). Only hand out paths that exist. + if _, err := os.Stat(destPath); err != nil { + return "", "" + } apiURL := fmt.Sprintf("/api/covers/%s.jpg", id) return apiURL, destPath } diff --git a/src/util/http_limit_test.go b/src/util/http_limit_test.go new file mode 100644 index 00000000..6f879d73 --- /dev/null +++ b/src/util/http_limit_test.go @@ -0,0 +1,31 @@ +package util + +import ( + "strings" + "testing" +) + +// BUG-20: remote responses were read with unbounded io.ReadAll on +// clients with no timeout, so one stalled or huge response could hang +// or OOM the app. +func TestReadLimited(t *testing.T) { + data, err := readLimited(strings.NewReader("hello"), 10) + if err != nil || string(data) != "hello" { + t.Errorf("readLimited small = %q, %v; want hello, nil", data, err) + } + + if _, err := readLimited(strings.NewReader(strings.Repeat("x", 100)), 10); err == nil { + t.Error("readLimited over-limit = nil error, want rejection instead of truncation") + } + + data, err = readLimited(strings.NewReader(strings.Repeat("x", 10)), 10) + if err != nil || len(data) != 10 { + t.Errorf("readLimited at-limit = %d bytes, %v; want 10, nil", len(data), err) + } +} + +func TestRemoteHTTPClientHasTimeout(t *testing.T) { + if remoteHTTPClient.Timeout <= 0 { + t.Error("remoteHTTPClient has no timeout; a stalled server would hang forever") + } +} diff --git a/src/util/metadata.go b/src/util/metadata.go index e5286ae0..e53e2d8f 100644 --- a/src/util/metadata.go +++ b/src/util/metadata.go @@ -42,15 +42,18 @@ func Abs(x int) int { return x } +// An empty key produces a "=value" ffmpeg metadata entry, which fails +// tagging or writes garbled tags depending on the container (BUG-24): +// tag sets leave fields empty for formats that have no name for them. func addStringTag(metadata []string, key string, value string) []string { - if value != "" { + if key != "" && value != "" { metadata = append(metadata, key+"="+value) } return metadata } func addIntTag(metadata []string, key string, value int) []string { - if value != 0 { + if key != "" && value != 0 { metadata = append(metadata, fmt.Sprintf("%s=%d", key, value)) } return metadata diff --git a/src/util/metadata_test.go b/src/util/metadata_test.go new file mode 100644 index 00000000..57fceb40 --- /dev/null +++ b/src/util/metadata_test.go @@ -0,0 +1,65 @@ +package util + +import ( + "strings" + "testing" + + "explo/src/models" +) + +// BUG-24: tag sets leave some fields empty per format, producing +// "=value" ffmpeg metadata entries for e.g. m4a/wav and ape files. +func fullTrack() models.Track { + return models.Track{ + File: "song.m4a", + Title: "Song", + Artist: "Artist", + Album: "Album", + AlbumArtist: "Album Artist", + ArtistSort: "Artist, A", + OriginalDate: "2020-01-01", + Genres: "Rock", + Media: "CD", + ReleaseType: "album", + ReleaseStatus: "official", + MusicBrainzReleaseGroupID: "rg", + MusicBrainzAlbumArtistID: "aa", + MusicBrainzTrackID: "t", + MusicBrainzAlbumID: "a", + MusicBrainzReleaseTrackID: "rt", + MusicBrainzArtistID: "ar", + OriginalYear: 2020, + TrackNumber: 3, + TrackTotal: 12, + DiscNumber: 1, + DiscTotal: 2, + ISRCs: []string{"US123"}, + } +} + +func TestBuildffmpegMetadata_HasNoEmptyKeys(t *testing.T) { + for _, file := range []string{"song.m4a", "song.wav", "song.ape", "song.mp3", "song.flac", "song.opus"} { + track := fullTrack() + track.File = file + for _, entry := range BuildffmpegMetadata(track) { + if strings.HasPrefix(entry, "=") { + t.Errorf("%s: entry %q has an empty key", file, entry) + } + if !strings.Contains(entry, "=") { + t.Errorf("%s: entry %q is not key=value", file, entry) + } + } + } +} + +func TestAddStringTag_SkipsEmptyKey(t *testing.T) { + if got := addStringTag(nil, "", "value"); len(got) != 0 { + t.Errorf("addStringTag empty key = %v, want nothing", got) + } + if got := addIntTag(nil, "", 3); len(got) != 0 { + t.Errorf("addIntTag empty key = %v, want nothing", got) + } + if got := addStringTag(nil, "title", "Song"); len(got) != 1 { + t.Errorf("addStringTag normal = %v, want one entry", got) + } +} diff --git a/src/web/backend/app/app.go b/src/web/backend/app/app.go index a302987e..add8f818 100644 --- a/src/web/backend/app/app.go +++ b/src/web/backend/app/app.go @@ -1,8 +1,11 @@ package app +import "time" type Config struct { - WebEnvPath string - WebDataDir string - ExploPath string -} \ No newline at end of file + WebEnvPath string + WebDataDir string + ExploPath string + // RunTimeout kills a run that has outlasted it. Zero means no timeout. + RunTimeout time.Duration +} diff --git a/src/web/backend/auth/cookiename.go b/src/web/backend/auth/cookiename.go new file mode 100644 index 00000000..17ecabd8 --- /dev/null +++ b/src/web/backend/auth/cookiename.go @@ -0,0 +1,65 @@ +package auth + +import ( + "net" + "strings" +) + +// cookieNamePrefix is shared by every instance; what follows distinguishes them. +const cookieNamePrefix = "explo_session" + +// DefaultCookieName derives a session cookie name from the address the server +// listens on. +// +// Cookies are scoped by host, not by port (RFC 6265 section 8.5), so several +// Explo instances on one machine -- the usual unraid setup, one container per +// port -- would otherwise all write a cookie called "session" for that host and +// overwrite each other. Sessions live in each process's own memory, so an ID +// minted by one instance is meaningless to the next, and the user is silently +// logged out of whichever instance they visited first. +// +// Including the port gives each instance its own cookie. Deployments that +// cannot be told apart this way (several instances behind one reverse proxy on +// the same host and port) should set UI_COOKIE_NAME explicitly. +func DefaultCookieName(addr string) string { + port := portFromAddr(addr) + if port == "" { + return cookieNamePrefix + } + + return cookieNamePrefix + "_" + port +} + +// portFromAddr pulls the port out of a listen address. It accepts the forms +// Go's net package does -- ":7288", "0.0.0.0:7288", "[::]:7288" -- and returns +// "" for anything it cannot read, so a malformed WEB_ADDR degrades to the +// shared name rather than producing a nonsense cookie. +func portFromAddr(addr string) string { + addr = strings.TrimSpace(addr) + if addr == "" { + return "" + } + + _, port, err := net.SplitHostPort(addr) + if err != nil { + return "" + } + + return sanitizeCookieName(port) +} + +// sanitizeCookieName keeps the name a valid cookie token. A port is normally +// digits, but WEB_ADDR may name a service instead ("http"), so anything outside +// the safe set is dropped rather than trusted into a Set-Cookie header. +func sanitizeCookieName(s string) string { + var b strings.Builder + + for _, r := range s { + switch { + case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '_', r == '-': + b.WriteRune(r) + } + } + + return b.String() +} diff --git a/src/web/backend/auth/middleware.go b/src/web/backend/auth/middleware.go index 7f719a14..5c64bac9 100644 --- a/src/web/backend/auth/middleware.go +++ b/src/web/backend/auth/middleware.go @@ -68,10 +68,14 @@ func (s *InMemorySessionStore) read(id string) (*Session, error) { } func (s *InMemorySessionStore) write(session *Session) error { + return s.writeWithID(session, session.ID()) +} + +func (s *InMemorySessionStore) writeWithID(session *Session, id string) error { s.mu.Lock() defer s.mu.Unlock() - s.sessions[session.id] = session + s.sessions[id] = session return nil } @@ -146,7 +150,7 @@ func writeCookieIfNecessary(w *sessionResponseWriter) { cookie := &http.Cookie{ Name: w.sessionManager.cookieName, - Value: session.id, + Value: session.ID(), HttpOnly: true, Path: "/", SameSite: http.SameSiteLaxMode, diff --git a/src/web/backend/auth/session.go b/src/web/backend/auth/session.go index dc6c5ec4..3cfa6795 100644 --- a/src/web/backend/auth/session.go +++ b/src/web/backend/auth/session.go @@ -13,7 +13,7 @@ import ( type Session struct { createdAt time.Time - mu sync.Mutex + mu sync.RWMutex id string data map[string]any } @@ -71,9 +71,9 @@ func (m *SessionManager) gc(d time.Duration) { func (m *SessionManager) validate(session *Session) bool { if time.Since(session.createdAt) > m.absoluteExpiration { - + // Delete the session from the store - err := m.store.destroy(session.id) + err := m.store.destroy(session.ID()) if err != nil { panic(err) } @@ -113,16 +113,16 @@ func (m *SessionManager) start(r *http.Request) (*Session, *http.Request) { func (m *SessionManager) Migrate(session *Session) error { session.mu.Lock() - defer session.mu.Unlock() + oldID := session.id + session.id = generateToken() + newID := session.id + session.mu.Unlock() - err := m.store.destroy(session.id) - if err != nil { + if err := m.store.destroy(oldID); err != nil { return err } - session.id = generateToken() - - return m.store.write(session) + return m.store.writeWithID(session, newID) } @@ -138,15 +138,31 @@ func generateToken() string { return base64.RawURLEncoding.EncodeToString(id) } +func (s *Session) ID() string { + s.mu.RLock() + defer s.mu.RUnlock() + + return s.id +} + func (s *Session) Get(key string) any { + s.mu.RLock() + defer s.mu.RUnlock() + return s.data[key] } func (s *Session) Put(key string, value any) { + s.mu.Lock() + defer s.mu.Unlock() + s.data[key] = value } func (s *Session) Delete(key string) { + s.mu.Lock() + defer s.mu.Unlock() + delete(s.data, key) } diff --git a/src/web/backend/auth/session_race_test.go b/src/web/backend/auth/session_race_test.go new file mode 100644 index 00000000..cdb9437f --- /dev/null +++ b/src/web/backend/auth/session_race_test.go @@ -0,0 +1,42 @@ +package auth + +import ( + "sync" + "testing" + "time" +) + +// BUG-01: concurrent requests sharing one session (multiple tabs, fast +// clicking) raced on Session.data and Session.id, causing a fatal +// "concurrent map writes" panic. Run with -race. +func TestSession_ConcurrentAccess(t *testing.T) { + manager := NewSessionManager(NewInMemorySessionStore(), time.Hour, 7*24*time.Hour, "explo_session_test") + session := newSession() + if err := manager.store.write(session); err != nil { + t.Fatalf("writing session: %v", err) + } + + var wg sync.WaitGroup + for i := 0; i < 8; i++ { + wg.Add(1) + go func(n int) { + defer wg.Done() + for j := 0; j < 500; j++ { + session.Put("key", n) + _ = session.Get("key") + session.Delete("key") + _ = session.Get("csrf_token") + } + }(i) + } + wg.Wait() + + // Session must still be usable after concurrent access. + session.Put("authenticated", true) + if got, _ := session.Get("authenticated").(bool); !got { + t.Error("session data lost after concurrent access") + } + if session.ID() == "" { + t.Error("session ID empty after concurrent access") + } +} diff --git a/src/web/backend/auth/session_test.go b/src/web/backend/auth/session_test.go new file mode 100644 index 00000000..5f463941 --- /dev/null +++ b/src/web/backend/auth/session_test.go @@ -0,0 +1,145 @@ +package auth + +import ( + "net/http" + "net/http/cookiejar" + "net/http/httptest" + "testing" + "time" +) + +// instance stands in for one Explo container: its own session manager, its own +// in-memory store, and its own listening port. +type instance struct { + manager *SessionManager + server *httptest.Server +} + +func newInstance(t *testing.T, cookieName string) *instance { + t.Helper() + + manager := NewSessionManager(NewInMemorySessionStore(), time.Hour, 7*24*time.Hour, cookieName) + + handler := manager.Handle(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + session := manager.GetSession(r) + if r.URL.Path == "/login" { + session.Put("authenticated", true) + w.WriteHeader(http.StatusOK) + return + } + if authenticated, _ := session.Get("authenticated").(bool); !authenticated { + w.WriteHeader(http.StatusUnauthorized) + return + } + w.WriteHeader(http.StatusOK) + })) + + server := httptest.NewServer(handler) + t.Cleanup(server.Close) + + return &instance{manager: manager, server: server} +} + +// closeBody releases a response body, reporting rather than swallowing a +// failure to do so. +func closeBody(t *testing.T, resp *http.Response) { + t.Helper() + + if err := resp.Body.Close(); err != nil { + t.Errorf("closing response body: %v", err) + } +} + +// get drives one request from the shared browser jar. +func get(t *testing.T, client *http.Client, url string) int { + t.Helper() + + resp, err := client.Get(url) + if err != nil { + t.Fatalf("GET %s: %v", url, err) + } + defer closeBody(t, resp) + + return resp.StatusCode +} + +// Two Explo instances on one unraid host are two ports on the same hostname. +// Cookies are not scoped by port (RFC 6265 section 8.5), so a fixed cookie +// name means the second instance overwrites the first's session ID, and +// neither store recognises the other's, logging the user out. +func TestSessionCookie_InstancesDoNotClobberEachOther(t *testing.T) { + first := newInstance(t, DefaultCookieName(":7288")) + second := newInstance(t, DefaultCookieName(":7289")) + + jar, err := cookiejar.New(nil) + if err != nil { + t.Fatalf("creating cookie jar: %v", err) + } + client := &http.Client{Jar: jar} + + if code := get(t, client, first.server.URL+"/login"); code != http.StatusOK { + t.Fatalf("logging into the first instance = %d, want 200", code) + } + + // Merely visiting the second instance must not disturb the first. + if code := get(t, client, second.server.URL+"/"); code != http.StatusUnauthorized { + t.Fatalf("second instance = %d, want 401 -- it shares no session with the first", code) + } + + if code := get(t, client, first.server.URL+"/"); code != http.StatusOK { + t.Errorf("first instance = %d, want 200 -- the second instance clobbered its session cookie", code) + } +} + +// The whole point of the default: two addresses must not produce one name. +func TestDefaultCookieName_DiffersPerAddress(t *testing.T) { + if a, b := DefaultCookieName(":7288"), DefaultCookieName(":7289"); a == b { + t.Errorf("both addresses produced %q; instances on one host would share a cookie", a) + } +} + +// WEB_ADDR is free-form, so the derivation has to cope with every listen +// address Go accepts, and degrade rather than emit a broken cookie name. +func TestDefaultCookieName_AddressForms(t *testing.T) { + tests := []struct { + name string + addr string + want string + }{ + {"port only", ":7288", "explo_session_7288"}, + {"host and port", "0.0.0.0:7288", "explo_session_7288"}, + {"ipv6", "[::]:7288", "explo_session_7288"}, + {"named service", ":http", "explo_session_http"}, + {"no port falls back to the shared name", "0.0.0.0", "explo_session"}, + {"empty falls back to the shared name", "", "explo_session"}, + {"unsafe characters are dropped", ":72 88;x", "explo_session_7288x"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := DefaultCookieName(tt.addr); got != tt.want { + t.Errorf("DefaultCookieName(%q) = %q, want %q", tt.addr, got, tt.want) + } + }) + } +} + +// A derived name is only useful if the session actually round-trips under it. +func TestSessionCookie_UsesTheConfiguredName(t *testing.T) { + const name = "explo_session_7288" + + instance := newInstance(t, name) + + resp, err := http.Get(instance.server.URL + "/login") + if err != nil { + t.Fatalf("GET /login: %v", err) + } + defer closeBody(t, resp) + + for _, cookie := range resp.Cookies() { + if cookie.Name == name { + return + } + } + t.Errorf("no %q cookie in %v", name, resp.Cookies()) +} diff --git a/src/web/backend/defs/defs.go b/src/web/backend/defs/defs.go index 6a3c289d..c72a44a7 100644 --- a/src/web/backend/defs/defs.go +++ b/src/web/backend/defs/defs.go @@ -193,7 +193,10 @@ var AllConfigKeys = []string{ "SYSTEM_USERNAME", "SYSTEM_PASSWORD", "PLAYLIST_DIR", "SLEEP", "PUBLIC_PLAYLIST", "DOWNLOAD_DIR", "USE_SUBDIRECTORY", "PATH_TEMPLATE", "ENRICH_TRACK_METADATA", "DOWNLOAD_SERVICES", "YOUTUBE_API_KEY", "TRACK_EXTENSION", "FILTER_LIST", - "SLSKD_URL", "SLSKD_API_KEY", "SLSKD_MIGRATE_DOWNLOADS", + "MIN_BITRATE", "MAX_BITRATE", "SIZE_PREFERENCE", "RELEASE_PREFERENCE", + "PREFER_ORIGINAL_RELEASE", + "SLSKD_URL", "SLSKD_API_KEY", "SLSKD_MIGRATE_DOWNLOADS", "SLSKD_ALBUM_MODE", + "SLSKD_FALLBACK_AFTER", "LIDARR_FALLBACK_AFTER", "LIDARR_URL", "LIDARR_API_KEY", "LIDARR_MIGRATE_DOWNLOADS", "LIDARR_ROOT_FOLDER", "WIZARD_COMPLETE", "EXTENSIONS", "LISTENBRAINZ_USER_TOKEN", } diff --git a/src/web/backend/playlist/apple_music.go b/src/web/backend/playlist/apple_music.go index e928418f..e84c6488 100644 --- a/src/web/backend/playlist/apple_music.go +++ b/src/web/backend/playlist/apple_music.go @@ -3,7 +3,6 @@ package playlist import ( "encoding/json" "fmt" - "io" "log/slog" "net/http" "strings" @@ -64,7 +63,7 @@ func fetchAppleMusicPlaylist(pageURL string) (string, string, []PlaylistTrack, e req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") req.Header.Set("Accept-Language", "en-US,en;q=0.9") - resp, err := http.DefaultClient.Do(req) + resp, err := remoteFetchClient.Do(req) if err != nil { return "", "", nil, fmt.Errorf("apple music: fetch failed: %w", err) } @@ -77,7 +76,7 @@ func fetchAppleMusicPlaylist(pageURL string) (string, string, []PlaylistTrack, e return "", "", nil, fmt.Errorf("apple music: page returned %d", resp.StatusCode) } - body, err := io.ReadAll(resp.Body) + body, err := readFetchBody(resp.Body) if err != nil { return "", "", nil, fmt.Errorf("apple music: failed to read body: %w", err) } diff --git a/src/web/backend/playlist/custom_playlists.go b/src/web/backend/playlist/custom_playlists.go index da85ca06..4a6b3dc0 100644 --- a/src/web/backend/playlist/custom_playlists.go +++ b/src/web/backend/playlist/custom_playlists.go @@ -2,12 +2,14 @@ package playlist import ( "encoding/json" + "errors" "fmt" "log/slog" "os" "path/filepath" "regexp" "strings" + "sync" "time" "explo/src/discovery" @@ -48,17 +50,15 @@ func GetCustomPlaylist(cfgDir, id string) *CustomPlaylist { // MarkCustomPlaylistArtworkUploaded sets ArtworkUploaded=true and persists. func MarkCustomPlaylistArtworkUploaded(cfgDir, id string) error { - playlists := loadCustomPlaylists(cfgDir) - for i := range playlists { - if playlists[i].ID == id { - if playlists[i].ArtworkUploaded { - return nil + return updateCustomPlaylists(cfgDir, func(playlists []CustomPlaylist) ([]CustomPlaylist, error) { + for i := range playlists { + if playlists[i].ID == id { + playlists[i].ArtworkUploaded = true + break } - playlists[i].ArtworkUploaded = true - return saveCustomPlaylists(cfgDir, playlists) } - } - return nil + return playlists, nil + }) } var lbMBIDRe = regexp.MustCompile(`[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}`) @@ -91,7 +91,23 @@ func customPlaylistsPath(cfgDir string) string { return filepath.Join(cfgDir, "custom-playlists.json") } +// customPlaylistsMu serializes every load-modify-save of +// custom-playlists.json. Without it, two writers (import, delete, refresh, +// cron) could each load the same list and the later save silently dropped +// the other's change; a reader mid-write could also parse a truncated file +// as empty and the next save then wiped every playlist. +var customPlaylistsMu sync.Mutex + +var errPlaylistNotFound = errors.New("playlist not found") + func loadCustomPlaylists(cfgDir string) []CustomPlaylist { + customPlaylistsMu.Lock() + defer customPlaylistsMu.Unlock() + + return loadCustomPlaylistsUnlocked(cfgDir) +} + +func loadCustomPlaylistsUnlocked(cfgDir string) []CustomPlaylist { data, err := os.ReadFile(customPlaylistsPath(cfgDir)) if err != nil { return nil @@ -105,11 +121,47 @@ func loadCustomPlaylists(cfgDir string) []CustomPlaylist { } func saveCustomPlaylists(cfgDir string, playlists []CustomPlaylist) error { + customPlaylistsMu.Lock() + defer customPlaylistsMu.Unlock() + + return saveCustomPlaylistsUnlocked(cfgDir, playlists) +} + +// saveCustomPlaylistsUnlocked writes via a temp file in the same directory +// plus rename, so a reader never observes a half-written file. +func saveCustomPlaylistsUnlocked(cfgDir string, playlists []CustomPlaylist) error { raw, err := json.MarshalIndent(playlists, "", " ") if err != nil { return err } - return os.WriteFile(customPlaylistsPath(cfgDir), raw, 0644) + tmp, err := os.CreateTemp(cfgDir, "custom-playlists-*.tmp") + if err != nil { + return err + } + tmpName := tmp.Name() + defer func() { _ = os.Remove(tmpName) }() + if _, err := tmp.Write(raw); err != nil { + _ = tmp.Close() + return err + } + if err := tmp.Close(); err != nil { + return err + } + return os.Rename(tmpName, customPlaylistsPath(cfgDir)) +} + +// updateCustomPlaylists runs fn against the freshly loaded list and saves +// the result, holding the lock across the whole load-modify-save. If fn +// returns an error, nothing is saved and the error is returned. +func updateCustomPlaylists(cfgDir string, fn func([]CustomPlaylist) ([]CustomPlaylist, error)) error { + customPlaylistsMu.Lock() + defer customPlaylistsMu.Unlock() + + updated, err := fn(loadCustomPlaylistsUnlocked(cfgDir)) + if err != nil { + return err + } + return saveCustomPlaylistsUnlocked(cfgDir, updated) } // FetchResult is the uniform return type for fetching playlist data from any source. diff --git a/src/web/backend/playlist/custom_playlists_test.go b/src/web/backend/playlist/custom_playlists_test.go new file mode 100644 index 00000000..e3763ed1 --- /dev/null +++ b/src/web/backend/playlist/custom_playlists_test.go @@ -0,0 +1,63 @@ +package playlist + +import ( + "fmt" + "sync" + "testing" +) + +// BUG-06: unsynchronized load-modify-save of custom-playlists.json let two +// writers drop each other's playlists, and a non-atomic write could be +// parsed as empty mid-truncate, wiping the file on the next save. +func TestUpdateCustomPlaylists_ConcurrentAppendsAllSurvive(t *testing.T) { + dir := t.TempDir() + const writers = 16 + + var wg sync.WaitGroup + for i := 0; i < writers; i++ { + wg.Add(1) + go func(n int) { + defer wg.Done() + id := fmt.Sprintf("custom-%d", n) + err := updateCustomPlaylists(dir, func(cur []CustomPlaylist) ([]CustomPlaylist, error) { + return append(cur, CustomPlaylist{ID: id, Name: id}), nil + }) + if err != nil { + t.Errorf("update %s: %v", id, err) + } + // Readers must never observe a torn/empty file mid-write. + _ = loadCustomPlaylists(dir) + }(i) + } + wg.Wait() + + got := loadCustomPlaylists(dir) + if len(got) != writers { + t.Fatalf("saved %d playlists, want %d — a concurrent writer's append was lost", len(got), writers) + } + seen := make(map[string]bool, writers) + for _, p := range got { + seen[p.ID] = true + } + for i := 0; i < writers; i++ { + if !seen[fmt.Sprintf("custom-%d", i)] { + t.Errorf("playlist custom-%d missing after concurrent saves", i) + } + } +} + +func TestUpdateCustomPlaylists_ErrorSkipsSave(t *testing.T) { + dir := t.TempDir() + if err := saveCustomPlaylists(dir, []CustomPlaylist{{ID: "keep", Name: "keep"}}); err != nil { + t.Fatalf("seed save: %v", err) + } + if err := updateCustomPlaylists(dir, func(cur []CustomPlaylist) ([]CustomPlaylist, error) { + return nil, errPlaylistNotFound + }); err != errPlaylistNotFound { + t.Fatalf("update error = %v, want errPlaylistNotFound", err) + } + got := loadCustomPlaylists(dir) + if len(got) != 1 || got[0].ID != "keep" { + t.Errorf("list = %v, want the seeded playlist untouched after a failed update", got) + } +} diff --git a/src/web/backend/playlist/handlers.go b/src/web/backend/playlist/handlers.go index 49ff8faa..91f0208a 100644 --- a/src/web/backend/playlist/handlers.go +++ b/src/web/backend/playlist/handlers.go @@ -148,8 +148,10 @@ func (p *Playlist) HandleImportCustomPlaylist(w http.ResponseWriter, r *http.Req ColorIndex: len(existing), LastFetched: time.Now().UTC(), } - existing = append(existing, cp) - if err := saveCustomPlaylists(p.cfg.WebDataDir, existing); err != nil { + if err := updateCustomPlaylists(p.cfg.WebDataDir, func(current []CustomPlaylist) ([]CustomPlaylist, error) { + cp.ColorIndex = len(current) + return append(current, cp), nil + }); err != nil { slog.Error("custom-playlists: failed to save metadata", "err", err) http.Error(w, "failed to save playlist metadata: "+err.Error(), http.StatusInternalServerError) return @@ -235,8 +237,15 @@ func (p *Playlist) HandleRefreshCustomPlaylist(w http.ResponseWriter, r *http.Re } go downloadAndCacheCovers(p.cfg.WebDataDir, id, tracks) - playlists[idx].LastFetched = time.Now().UTC() - if err := saveCustomPlaylists(p.cfg.WebDataDir, playlists); err != nil { + if err := updateCustomPlaylists(p.cfg.WebDataDir, func(current []CustomPlaylist) ([]CustomPlaylist, error) { + for i := range current { + if current[i].ID == id { + current[i].LastFetched = time.Now().UTC() + break + } + } + return current, nil + }); err != nil { slog.Warn("custom-playlists: failed to update last_fetched after refresh", "err", err) } @@ -260,24 +269,28 @@ func (p *Playlist) HandleDeleteCustomPlaylist(w http.ResponseWriter, r *http.Req deleteTracks := r.URL.Query().Get("delete_tracks") == "true" slog.Info("custom-playlists: delete request", "id", id, "delete_tracks", deleteTracks) - existing := loadCustomPlaylists(p.cfg.WebDataDir) - filtered := existing[:0] found := false var deletedName string - for _, p := range existing { - if p.ID == id { - found = true - deletedName = p.Name - } else { - filtered = append(filtered, p) + err := updateCustomPlaylists(p.cfg.WebDataDir, func(existing []CustomPlaylist) ([]CustomPlaylist, error) { + filtered := make([]CustomPlaylist, 0, len(existing)) + for _, p := range existing { + if p.ID == id { + found = true + deletedName = p.Name + } else { + filtered = append(filtered, p) + } } - } - if !found { + if !found { + return nil, errPlaylistNotFound + } + return filtered, nil + }) + if err == errPlaylistNotFound { http.Error(w, "playlist not found", http.StatusNotFound) return } - - if err := saveCustomPlaylists(p.cfg.WebDataDir, filtered); err != nil { + if err != nil { http.Error(w, "failed to save: "+err.Error(), http.StatusInternalServerError) return } diff --git a/src/web/backend/playlist/jobs.go b/src/web/backend/playlist/jobs.go index abf444a3..ab22eeb7 100644 --- a/src/web/backend/playlist/jobs.go +++ b/src/web/backend/playlist/jobs.go @@ -10,6 +10,18 @@ import ( "github.com/go-co-op/gocron/v2" ) +// dueForRefresh reports whether a playlist whose data was last fetched +// at lastFetched is due again under a refreshDays cadence. A zero or +// negative cadence refreshes on every tick; the cron schedule itself +// sets the tick rate, so a weekly list is refreshed by its daily job +// exactly when seven days have passed since the last fetch. +func dueForRefresh(lastFetched time.Time, refreshDays int, now time.Time) bool { + if refreshDays <= 0 { + return true + } + return !now.Before(lastFetched.Add(time.Duration(refreshDays) * 24 * time.Hour)) +} + // RegisterCustomPlaylistRefresh registers a cache-refresh job for each custom playlist // using its stored schedule. Falls back to daily at 4 AM if no schedule is set. func (p *Playlist) RegisterCustomPlaylistRefresh(j *jobs.Jobs) error { @@ -42,7 +54,15 @@ func (p *Playlist) RegisterCustomPlaylistRefresh(j *jobs.Jobs) error { _, err := j.Scheduler.NewJob( gocron.CronJob(schedule, false), gocron.NewTask(func() { - if time.Since(plist.LastFetched) < time.Duration(plist.RefreshDays)*24*time.Hour { + // Reload the current state before gating: plist is the + // copy captured at registration, so gating on its + // LastFetched refreshed every run once the playlist was + // past its interval (daily refresh of a weekly list). + lastFetched := plist.LastFetched + if fresh := GetCustomPlaylist(p.cfg.WebDataDir, plist.ID); fresh != nil { + lastFetched = fresh.LastFetched + } + if !dueForRefresh(lastFetched, plist.RefreshDays, time.Now()) { return } slog.Info("custom-playlists: refreshing", "id", plist.ID, "name", plist.Name, "source", plist.Source) @@ -52,14 +72,15 @@ func (p *Playlist) RegisterCustomPlaylistRefresh(j *jobs.Jobs) error { return } writePrefetchCache(p.cfg.WebDataDir, plist.ID, result.Tracks) - playlists := loadCustomPlaylists(p.cfg.WebDataDir) - for i, pl := range playlists { - if pl.ID == plist.ID { - playlists[i].LastFetched = time.Now().UTC() - break + if err := updateCustomPlaylists(p.cfg.WebDataDir, func(playlists []CustomPlaylist) ([]CustomPlaylist, error) { + for i, pl := range playlists { + if pl.ID == plist.ID { + playlists[i].LastFetched = time.Now().UTC() + break + } } - } - if err := saveCustomPlaylists(p.cfg.WebDataDir, playlists); err != nil { + return playlists, nil + }); err != nil { slog.Error("custom-playlists: failed to save after refresh", "err", err) } }), diff --git a/src/web/backend/playlist/playlist.go b/src/web/backend/playlist/playlist.go index d37b285a..ad73a13a 100644 --- a/src/web/backend/playlist/playlist.go +++ b/src/web/backend/playlist/playlist.go @@ -19,10 +19,29 @@ import ( "os" "path/filepath" "strings" + "time" ) const lbAPIBase = "https://api.listenbrainz.org/1" +// remoteFetchClient bounds playlist/cover fetches: http.Get and +// DefaultClient have no timeout, and their responses were read into +// memory without a cap (BUG-20). +var remoteFetchClient = &http.Client{Timeout: 30 * time.Second} + +const maxFetchBytes = 16 << 20 + +func readFetchBody(r io.Reader) ([]byte, error) { + data, err := io.ReadAll(io.LimitReader(r, maxFetchBytes+1)) + if err != nil { + return nil, err + } + if int64(len(data)) > maxFetchBytes { + return nil, fmt.Errorf("response exceeds %d bytes", maxFetchBytes) + } + return data, nil +} + // PlaylistTrack represents a single track fetched from any playlist source. // Replaces the raw [][4]string{title, artist, album, coverURL} pattern // with named fields and adds MainArtist for accurate matching in music servers. @@ -150,7 +169,7 @@ func WritePlaylistCache(cfgPath, playlist string, tracks []*models.Track, added } func lbGet(url string) ([]byte, error) { - resp, err := http.Get(url) //nolint:noctx + resp, err := remoteFetchClient.Get(url) if err != nil { return nil, err } @@ -163,7 +182,7 @@ func lbGet(url string) ([]byte, error) { if resp.StatusCode != http.StatusOK { return nil, fmt.Errorf("LB returned %d", resp.StatusCode) } - return io.ReadAll(resp.Body) + return readFetchBody(resp.Body) } type cachedPrefetchTrack struct { @@ -252,14 +271,14 @@ func randomLocalCoverHiRes(coversDir string) string { return "/api/covers/" + bgFile } // Download hi-res from Cover Art Archive using the release MBID - resp, err := http.Get("https://coverartarchive.org/release/" + mbid + "/front-1200") //nolint:noctx + resp, err := remoteFetchClient.Get("https://coverartarchive.org/release/" + mbid + "/front-1200") if err != nil || resp.StatusCode != http.StatusOK { if resp != nil { resp.Body.Close() // nolint:errcheck } continue } - data, err := io.ReadAll(resp.Body) + data, err := readFetchBody(resp.Body) resp.Body.Close() // nolint:errcheck if err != nil { continue @@ -296,14 +315,14 @@ func fetchSitewideCovers(coversDir string) string { } url := "https://coverartarchive.org/release/" + rel.ReleaseMbid + "/front-1200" - dlResp, err := http.Get(url) //nolint:noctx + dlResp, err := remoteFetchClient.Get(url) if err != nil || dlResp.StatusCode != http.StatusOK { if dlResp != nil { dlResp.Body.Close() // nolint:errcheck } continue } - data, err := io.ReadAll(dlResp.Body) + data, err := readFetchBody(dlResp.Body) dlResp.Body.Close() //nolint:errcheck if err != nil { continue diff --git a/src/web/backend/playlist/refresh_gate_test.go b/src/web/backend/playlist/refresh_gate_test.go new file mode 100644 index 00000000..83f266bc --- /dev/null +++ b/src/web/backend/playlist/refresh_gate_test.go @@ -0,0 +1,36 @@ +package playlist + +import ( + "testing" + "time" +) + +// BUG-31: the refresh gate used LastFetched captured at registration, +// so once a weekly playlist was past its interval it refreshed on +// every daily tick. The gate must evaluate the freshly loaded state. +func TestDueForRefresh(t *testing.T) { + now := time.Now() + weekAgo := now.Add(-7 * 24 * time.Hour) + sixDaysAgo := now.Add(-6 * 24 * time.Hour) + + tests := []struct { + name string + lastFetched time.Time + days int + want bool + }{ + {"weekly, fetched a week ago: due", weekAgo, 7, true}, + {"weekly, fetched six days ago: not due", sixDaysAgo, 7, false}, + {"weekly, fetched just now: not due", now, 7, false}, + {"daily, fetched yesterday: due", now.Add(-24 * time.Hour), 1, true}, + {"zero cadence refreshes every tick", sixDaysAgo, 0, true}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := dueForRefresh(tt.lastFetched, tt.days, now); got != tt.want { + t.Errorf("dueForRefresh(%v, %d) = %v, want %v", tt.lastFetched, tt.days, got, tt.want) + } + }) + } +} diff --git a/src/web/backend/routes.go b/src/web/backend/routes.go index ec75b4bf..651129ad 100644 --- a/src/web/backend/routes.go +++ b/src/web/backend/routes.go @@ -54,10 +54,14 @@ func (s *Server) registerSettingRoutes() { s.mux.Handle("POST /api/ui/config/path-template", s.auth(s.settings.HandleSavePathTemplate)) s.mux.Handle("POST /api/ui/config/enrich-metadata", s.auth(s.settings.HandleSaveEnrichMetadata)) s.mux.Handle("POST /api/ui/config/replace-playlist", s.auth(s.settings.HandleSaveReplacePlaylist)) + s.mux.Handle("POST /api/ui/config/local-only", s.auth(s.settings.HandleSaveLocalOnly)) s.mux.Handle("POST /api/ui/config/clean-downloads", s.auth(s.settings.HandleSaveCleanDownloads)) - // Path template presets: GET list, POST add; DELETE per name under prefix - s.mux.Handle("api/ui/path-templates", s.auth(s.settings.HandlePathTemplates)) + // Path template presets: GET list, POST add; DELETE per name under prefix. + // The pattern needs its leading slash: without it ServeMux parses + // "api" as a host pattern and normal requests never reach the + // handler, so the settings screen could never load presets. + s.mux.Handle("/api/ui/path-templates", s.auth(s.settings.HandlePathTemplates)) s.mux.Handle("DELETE /api/ui/path-templates/", s.auth(s.settings.HandleDeletePathTemplate)) // Wizard steps (POST) — require auth diff --git a/src/web/backend/routes_test.go b/src/web/backend/routes_test.go new file mode 100644 index 00000000..940817f3 --- /dev/null +++ b/src/web/backend/routes_test.go @@ -0,0 +1,25 @@ +package backend + +import ( + "net/http" + "testing" + + "explo/src/config" +) + +// BUG-28: the pattern lacked its leading slash, so ServeMux parsed +// "api" as a host pattern and requests fell through to the SPA +// fallback instead of the presets handler. +func TestRoute_PathTemplatesRegistered(t *testing.T) { + for _, method := range []string{http.MethodGet, http.MethodPost} { + req, err := http.NewRequest(method, "/api/ui/path-templates", nil) + if err != nil { + t.Fatal(err) + } + s := NewServer(config.ServerConfig{}) + _, pattern := s.mux.Handler(req) + if pattern != "/api/ui/path-templates" { + t.Errorf("%s matched pattern %q, want /api/ui/path-templates", method, pattern) + } + } +} diff --git a/src/web/backend/run/events.go b/src/web/backend/run/events.go index 3f958bbd..d0acfd09 100644 --- a/src/web/backend/run/events.go +++ b/src/web/backend/run/events.go @@ -8,9 +8,55 @@ import ( "path/filepath" "bufio" "os/exec" + "strings" ) +// maxRunLineBytes caps how much of one output line is kept. Lines beyond +// it are drained and truncated rather than stopping the reader. +const maxRunLineBytes = 1 << 20 + +// scanRunLines emits each line from r until EOF. Unlike bufio.Scanner +// with its 64KB limit, a single very long line cannot stop draining: +// before this, an over-long line wedged the child on a full pipe, +// cmd.Wait never returned, and every later run was refused as already +// running. +func scanRunLines(r io.Reader, emit func(string)) error { + reader := bufio.NewReaderSize(r, 64*1024) + for { + var buf []byte + truncated := false + for { + frag, err := reader.ReadSlice('\n') + if len(buf)+len(frag) <= maxRunLineBytes { + buf = append(buf, frag...) + } else { + if len(buf) < maxRunLineBytes { + buf = append(buf, frag[:maxRunLineBytes-len(buf)]...) + } + truncated = true + } + if err == bufio.ErrBufferFull { + continue + } + if len(frag) > 0 || len(buf) > 0 { + line := strings.TrimRight(string(buf), "\r\n") + if truncated { + line += "...[truncated]" + } + emit(line) + } + if err == io.EOF { + return nil + } + if err != nil { + return err + } + break + } + } +} + func (mr *ManualRun) appendRunLog(line string) { event := runEvent{data: line} @@ -45,9 +91,7 @@ func (mr *ManualRun) collectRunOutput(cmd *exec.Cmd, pr *os.File, lf *os.File) { }() } - scanner := bufio.NewScanner(pr) - for scanner.Scan() { - line := scanner.Text() + if err := scanRunLines(pr, func(line string) { // Echo to stdout so runs show up in docker logs. _, _ = fmt.Fprintln(os.Stdout, line) if lf != nil { @@ -56,8 +100,7 @@ func (mr *ManualRun) collectRunOutput(cmd *exec.Cmd, pr *os.File, lf *os.File) { } } mr.appendRunLog(line) - } - if err := scanner.Err(); err != nil { + }); err != nil { mr.appendRunLog("failed to read run output: " + err.Error()) } diff --git a/src/web/backend/run/events_test.go b/src/web/backend/run/events_test.go new file mode 100644 index 00000000..0d2e8514 --- /dev/null +++ b/src/web/backend/run/events_test.go @@ -0,0 +1,54 @@ +package run + +import ( + "strings" + "testing" +) + +// BUG-19: bufio.Scanner's 64KB line limit stopped draining run output, +// so one long line blocked the child on a full pipe, cmd.Wait never +// returned, and every later run was refused as already running. +func collectLines(t *testing.T, input string) []string { + t.Helper() + var lines []string + if err := scanRunLines(strings.NewReader(input), func(line string) { + lines = append(lines, line) + }); err != nil { + t.Fatalf("scanRunLines: %v", err) + } + return lines +} + +func TestScanRunLines_LongLineDoesNotStopDraining(t *testing.T) { + long := strings.Repeat("x", 200*1024) // 3x the old 64KB scanner limit + lines := collectLines(t, "first\n"+long+"\nafter\n") + + if len(lines) != 3 { + t.Fatalf("got %d lines, want 3 — draining stopped at the long line", len(lines)) + } + if lines[0] != "first" || lines[2] != "after" { + t.Errorf("lines around the long one = %q, %q", lines[0], lines[2]) + } + if lines[1] != long { + t.Errorf("long line kept %d bytes, want it intact under the cap", len(lines[1])) + } +} + +func TestScanRunLines_HugeLineIsTruncatedButDrainingContinues(t *testing.T) { + huge := strings.Repeat("y", maxRunLineBytes+5000) + lines := collectLines(t, huge+"\nnext") + + if len(lines) != 2 || lines[1] != "next" { + t.Fatalf("lines = %d entries, want the truncated line plus 'next'", len(lines)) + } + if !strings.HasSuffix(lines[0], "...[truncated]") { + t.Error("huge line was not marked truncated") + } +} + +func TestScanRunLines_PreservesEmptyLinesAndPartialTail(t *testing.T) { + lines := collectLines(t, "a\n\nb") + if len(lines) != 3 || lines[0] != "a" || lines[1] != "" || lines[2] != "b" { + t.Errorf("lines = %q, want [a '' b]", lines) + } +} diff --git a/src/web/backend/run/manual_run.go b/src/web/backend/run/manual_run.go index 7dacdc30..87ba0981 100644 --- a/src/web/backend/run/manual_run.go +++ b/src/web/backend/run/manual_run.go @@ -1,6 +1,7 @@ package run import ( + "time" "fmt" "errors" "log/slog" @@ -53,7 +54,10 @@ func newManualRunState() manualRunState { } func (mr *ManualRun) startRun(args []string) error { - ctx, cancel := context.WithCancel(context.Background()) + // A run that wedges holds the "already running" guard below, so every later + // run is refused until someone stops it by hand. The timeout kills the + // child, which lets cmd.Wait return and clears the guard on its own. + ctx, cancel := runContext(mr.cfg.RunTimeout) cmd := exec.CommandContext(ctx, mr.cfg.ExploPath, args...) // Strip WEB_UI from env so the child process runs normally, not as web server. env := make([]string, 0, len(os.Environ())) @@ -89,8 +93,8 @@ func (mr *ManualRun) startRun(args []string) error { slog.Warn("failed to close file writer", "err", err.Error()) } if lf != nil { - if err := pw.Close(); err != nil { - slog.Warn("failed to close file writer", "err", err.Error()) + if err := lf.Close(); err != nil { + slog.Warn("failed to close run log", "err", err.Error()) } } return errRunAlreadyStarted @@ -176,4 +180,11 @@ func buildArgs(playlist, downloadMode, WebEnvPath string, replacePlaylist bool) args = append(args, "--replace-playlist=false") } return args -} \ No newline at end of file +} +// runContext bounds a run, or leaves it unbounded when the timeout is zero. +func runContext(timeout time.Duration) (context.Context, context.CancelFunc) { + if timeout <= 0 { + return context.WithCancel(context.Background()) + } + return context.WithTimeout(context.Background(), timeout) +} diff --git a/src/web/backend/run/manual_run_test.go b/src/web/backend/run/manual_run_test.go new file mode 100644 index 00000000..e0de5679 --- /dev/null +++ b/src/web/backend/run/manual_run_test.go @@ -0,0 +1,157 @@ +package run + +import ( + "context" + "os" + "testing" + "time" + + "explo/src/web/backend/app" +) + +// Zero means a run is never killed on a timer, only by the stop endpoint. +func TestRunContext_ZeroLeavesTheRunUnbounded(t *testing.T) { + ctx, cancel := runContext(0) + defer cancel() + + if _, ok := ctx.Deadline(); ok { + t.Error("a zero timeout produced a deadline; runs would be killed unexpectedly") + } + + select { + case <-ctx.Done(): + t.Error("context was already done") + default: + } +} + +// A configured timeout has to actually bound the run. +func TestRunContext_AppliesTheTimeout(t *testing.T) { + ctx, cancel := runContext(30 * time.Millisecond) + defer cancel() + + if _, ok := ctx.Deadline(); !ok { + t.Fatal("no deadline was set, so a wedged run would never be killed") + } + + select { + case <-ctx.Done(): + if ctx.Err() != context.DeadlineExceeded { + t.Errorf("ctx.Err() = %v, want DeadlineExceeded", ctx.Err()) + } + case <-time.After(2 * time.Second): + t.Error("the deadline never fired") + } +} + +// The guard that refuses a second run must clear itself when a wedged run is +// killed, or the timeout buys nothing: every later run is still refused. +func TestStartRun_TimeoutReleasesTheAlreadyRunningGuard(t *testing.T) { + mr := NewManualRun(app.Config{ + // A command that would otherwise run far longer than the test. + ExploPath: "/bin/sleep", + RunTimeout: 50 * time.Millisecond, + }) + + if err := mr.startRun([]string{"30"}); err != nil { + t.Fatalf("startRun() = %v, want nil", err) + } + if !mr.currentRunStatus().Running { + t.Fatal("run did not register as running") + } + + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) { + if !mr.currentRunStatus().Running { + return + } + time.Sleep(5 * time.Millisecond) + } + + t.Error("the run was still marked running long after its timeout; later runs would keep being refused") +} + +// Back-to-back runs must be possible once the first has been cut short. +func TestStartRun_SecondRunIsAcceptedAfterATimeout(t *testing.T) { + mr := NewManualRun(app.Config{ + ExploPath: "/bin/sleep", + RunTimeout: 50 * time.Millisecond, + }) + + if err := mr.startRun([]string{"30"}); err != nil { + t.Fatalf("first startRun() = %v, want nil", err) + } + + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) && mr.currentRunStatus().Running { + time.Sleep(5 * time.Millisecond) + } + + if err := mr.startRun([]string{"0"}); err != nil { + t.Errorf("second startRun() = %v, want nil once the first was killed", err) + } +} + +// BUG-21: the already-running branch closed pw a second time instead of +// the log file, leaking one descriptor per rejected attempt. +func TestStartRun_RejectedRunsDoNotLeakFileHandles(t *testing.T) { + mr := NewManualRun(app.Config{ + ExploPath: "/bin/sleep", + RunTimeout: 5 * time.Second, + WebDataDir: t.TempDir(), + }) + + if err := mr.startRun([]string{"30"}); err != nil { + t.Fatalf("startRun() = %v, want nil", err) + } + t.Cleanup(func() { + mr.state.mu.Lock() + cancel := mr.state.cancel + mr.state.mu.Unlock() + if cancel != nil { + cancel() + } + }) + + countFDs := func() int { + entries, err := os.ReadDir("/dev/fd") + if err != nil { + t.Skipf("/dev/fd unavailable: %v", err) + } + return len(entries) + } + + before := countFDs() + for i := 0; i < 5; i++ { + if err := mr.startRun([]string{"30"}); err != errRunAlreadyStarted { + t.Fatalf("rejected startRun() = %v, want errRunAlreadyStarted", err) + } + } + if after := countFDs(); after > before { + t.Errorf("open FDs grew %d -> %d over 5 rejected runs; each rejection leaks a handle", before, after) + } +} + +// While a run really is in flight, a second must still be refused. +func TestStartRun_RefusesAConcurrentRun(t *testing.T) { + mr := NewManualRun(app.Config{ + ExploPath: "/bin/sleep", + RunTimeout: 5 * time.Second, + }) + + if err := mr.startRun([]string{"30"}); err != nil { + t.Fatalf("startRun() = %v, want nil", err) + } + t.Cleanup(func() { + mr.state.mu.Lock() + cancel := mr.state.cancel + mr.state.mu.Unlock() + if cancel != nil { + cancel() + } + }) + + if err := mr.startRun([]string{"30"}); err != errRunAlreadyStarted { + t.Errorf("second startRun() = %v, want errRunAlreadyStarted", err) + } +} diff --git a/src/web/backend/server.go b/src/web/backend/server.go index 9da7747e..9e28a41e 100644 --- a/src/web/backend/server.go +++ b/src/web/backend/server.go @@ -39,11 +39,19 @@ type Server struct { } func NewServer(cfg config.ServerConfig) *Server { + // Cookies are not scoped by port, so several instances on one host would + // share -- and overwrite -- a fixed cookie name. Default to a name derived + // from the listen address; UI_COOKIE_NAME overrides it. + cookieName := cfg.CookieName + if cookieName == "" { + cookieName = auth.DefaultCookieName(cfg.Port) + } + sessionManager := auth.NewSessionManager( auth.NewInMemorySessionStore(), 1*time.Hour, 7*(24*time.Hour), - "session", + cookieName, ) authStore := auth.NewAuthStore( @@ -55,6 +63,7 @@ func NewServer(cfg config.ServerConfig) *Server { WebEnvPath: cfg.WebEnvPath, WebDataDir: cfg.WebDataDir, ExploPath: cfg.ExploPath, + RunTimeout: time.Duration(cfg.RunTimeout) * time.Minute, } settings := settings.NewSettings(webCfg) diff --git a/src/web/backend/settings/env_injection_test.go b/src/web/backend/settings/env_injection_test.go new file mode 100644 index 00000000..d25d6e25 --- /dev/null +++ b/src/web/backend/settings/env_injection_test.go @@ -0,0 +1,67 @@ +package settings + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "explo/src/web/backend/app" +) + +// BUG-09: a settings value containing a line break was written raw into +// .env, injecting arbitrary extra keys on the next start. +func TestUpdateEnvKeys_RejectsLineBreakInjection(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, ".env") + original := "EXISTING=1\n" + if err := os.WriteFile(path, []byte(original), 0600); err != nil { + t.Fatal(err) + } + s := &Settings{cfg: app.Config{WebEnvPath: path}} + + for name, updates := range map[string]map[string]string{ + "value newline": {"NOTE": "x\nAPI_KEY=pwned"}, + "value CR": {"NOTE": "x\rAPI_KEY=pwned"}, + "key newline": {"NOTE\nAPI_KEY": "pwned"}, + "key equals": {"NOTE=API_KEY": "pwned"}, + "empty key": {"": "pwned"}, + } { + t.Run(name, func(t *testing.T) { + if err := s.UpdateEnvKeys(updates, nil); err == nil { + t.Errorf("UpdateEnvKeys(%v) = nil, want rejection", updates) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if string(data) != original { + t.Errorf("env file changed after rejected update: %q", data) + } + }) + } +} + +func TestUpdateEnvKeys_StillWritesNormalValues(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, ".env") + if err := os.WriteFile(path, []byte("EXISTING=1\n"), 0600); err != nil { + t.Fatal(err) + } + s := &Settings{cfg: app.Config{WebEnvPath: path}} + + if err := s.UpdateEnvKeys(map[string]string{"NOTE": "hello world #1", "NEW_KEY": "abc"}, nil); err != nil { + t.Fatalf("UpdateEnvKeys normal values: %v", err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + got := s.ParseEnvText(string(data)) + if got["NOTE"] != "hello world #1" || got["NEW_KEY"] != "abc" || got["EXISTING"] != "1" { + t.Errorf("parsed env = %v, want NOTE/NEW_KEY written and EXISTING preserved; raw:\n%s", got, data) + } + if strings.Contains(string(data), "pwned") { + t.Error("injected content present") + } +} diff --git a/src/web/backend/settings/env_save_race_test.go b/src/web/backend/settings/env_save_race_test.go new file mode 100644 index 00000000..44962915 --- /dev/null +++ b/src/web/backend/settings/env_save_race_test.go @@ -0,0 +1,63 @@ +package settings + +import ( + "fmt" + "os" + "path/filepath" + "sync" + "testing" + + "explo/src/web/backend/app" +) + +// BUG-08: concurrent UpdateEnvKeys calls each read the same starting +// .env and the later write silently undid the earlier update; a crash +// mid-write also left a truncated file. +func TestUpdateEnvKeys_ConcurrentUpdatesAllSurvive(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, ".env") + if err := os.WriteFile(path, []byte("BASE=1\n"), 0600); err != nil { + t.Fatal(err) + } + s := &Settings{cfg: app.Config{WebEnvPath: path}} + + const writers = 12 + var wg sync.WaitGroup + for i := 0; i < writers; i++ { + wg.Add(1) + go func(n int) { + defer wg.Done() + key := fmt.Sprintf("KEY_%d", n) + if err := s.UpdateEnvKeys(map[string]string{key: fmt.Sprint(n)}, nil); err != nil { + t.Errorf("UpdateEnvKeys(%s): %v", key, err) + } + }(i) + } + wg.Wait() + + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + got := s.ParseEnvText(string(data)) + if got["BASE"] != "1" { + t.Errorf("BASE = %q, want 1", got["BASE"]) + } + for i := 0; i < writers; i++ { + key := fmt.Sprintf("KEY_%d", i) + if got[key] != fmt.Sprint(i) { + t.Errorf("%s = %q, want %d — a concurrent update was lost", key, got[key], i) + } + } + // Atomic rename must not litter temp files. + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatal(err) + } + if len(entries) != 1 || entries[0].Name() != ".env" { + t.Errorf("dir entries = %v, want only .env", entries) + } + if info, err := os.Stat(path); err != nil || info.Mode().Perm() != 0600 { + t.Errorf(".env mode = %v, err = %v; want 0600", info.Mode().Perm(), err) + } +} diff --git a/src/web/backend/settings/handlers.go b/src/web/backend/settings/handlers.go index ff02eead..5114f0ce 100644 --- a/src/web/backend/settings/handlers.go +++ b/src/web/backend/settings/handlers.go @@ -8,6 +8,7 @@ import ( "net/http" "net/url" "os" + "strconv" "strings" "syscall" "time" @@ -72,7 +73,10 @@ func (s *Settings) HandleSaveConfig(w http.ResponseWriter, r *http.Request) { http.Error(w, "invalid .env content: "+err.Error(), http.StatusBadRequest) return } - if err := os.WriteFile(s.cfg.WebEnvPath, data, 0600); err != nil { + envFileMu.Lock() + err = writeEnvFileAtomic(s.cfg.WebEnvPath, data) + envFileMu.Unlock() + if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } @@ -81,7 +85,10 @@ func (s *Settings) HandleSaveConfig(w http.ResponseWriter, r *http.Request) { // handleResetConfig resets all settings and restarts the container. func (s *Settings) HandleResetConfig(w http.ResponseWriter, r *http.Request) { - if err := os.WriteFile(s.cfg.WebEnvPath, web.SampleEnv, 0600); err != nil { + envFileMu.Lock() + err := writeEnvFileAtomic(s.cfg.WebEnvPath, web.SampleEnv) + envFileMu.Unlock() + if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } @@ -128,7 +135,7 @@ func (s *Settings) HandleSaveSchedule(w http.ResponseWriter, r *http.Request) { currentFlags = s.ParseEnvText(string(data))[envPrefix+"_FLAGS"] } mergeFlags := func(base string) string { - injected := []string{"--replace-playlist=false", "--clean-downloads"} + injected := []string{"--replace-playlist=false", "--clean-downloads", localOnlyFlag} var extras []string for _, f := range injected { if strings.Contains(currentFlags, f) { @@ -213,63 +220,108 @@ func (s *Settings) HandleSaveEnrichMetadata(w http.ResponseWriter, r *http.Reque w.WriteHeader(http.StatusOK) } -// HandleSaveReplacePlaylist injects or removes --replace-playlist=false from a playlist's FLAGS env var. -func (s *Settings) HandleSaveReplacePlaylist(w http.ResponseWriter, r *http.Request) { - var body struct { - ID string `json:"id"` - Name string `json:"name"` - Replace bool `json:"replace"` +// localOnlyFlag builds the playlist from tracks already in the library instead +// of downloading. It is the same flag the docs suggest putting in *_FLAGS by +// hand; the UI just writes it for you. +const localOnlyFlag = "--download-mode=skip" + +// playlistFlagsKey resolves which env var holds a playlist's flags, and what +// those flags are when the file says nothing, for built-in and custom +// playlists alike. +func playlistFlagsKey(id, name string) (flagsKey, defaultFlags string, err error) { + if def, ok := defs.PlaylistDefs[id]; ok { + return def.EnvPrefix + "_FLAGS", def.DefaultFlags, nil } - if err := json.NewDecoder(r.Body).Decode(&body); err != nil { - http.Error(w, "invalid JSON: "+err.Error(), http.StatusBadRequest) - return + if defs.CustomIDRe.MatchString(id) { + return util.CustomEnvPrefix(name) + "_FLAGS", "--playlist " + id, nil } + return "", "", fmt.Errorf("unknown playlist name") +} - var envPrefix string - var defaultFlags string - if def, ok := defs.PlaylistDefs[body.ID]; ok { - envPrefix = def.EnvPrefix - defaultFlags = def.DefaultFlags - } else if defs.CustomIDRe.MatchString(body.ID) { - envPrefix = util.CustomEnvPrefix(body.Name) - defaultFlags = "--playlist " + body.ID - } else { - http.Error(w, "unknown playlist name", http.StatusBadRequest) - return +// setPlaylistFlag adds or removes one flag in a playlist's FLAGS env var, +// leaving every other flag as it found it. Adding a flag that is already there +// and removing one that is not are both no-ops, so a repeated request cannot +// duplicate or mangle anything. +func (s *Settings) setPlaylistFlag(id, name, flag string, enabled bool) error { + flagsKey, defaultFlags, err := playlistFlagsKey(id, name) + if err != nil { + return err } - flagsKey := envPrefix + "_FLAGS" - const replaceFlag = "--replace-playlist=false" - data, err := os.ReadFile(s.cfg.WebEnvPath) if err != nil && !os.IsNotExist(err) { - http.Error(w, err.Error(), http.StatusInternalServerError) - return + return err } - current := s.ParseEnvText(string(data)) - currentFlags := current[flagsKey] + + currentFlags := s.ParseEnvText(string(data))[flagsKey] if currentFlags == "" { currentFlags = defaultFlags } - hasFlag := strings.Contains(currentFlags, replaceFlag) newFlags := currentFlags - if !body.Replace && !hasFlag { - newFlags = strings.TrimSpace(currentFlags + " " + replaceFlag) - } else if body.Replace && hasFlag { - newFlags = strings.TrimSpace(strings.ReplaceAll(currentFlags, replaceFlag, "")) + switch hasFlag := strings.Contains(currentFlags, flag); { + case enabled && !hasFlag: + newFlags = strings.TrimSpace(currentFlags + " " + flag) + case !enabled && hasFlag: + newFlags = strings.TrimSpace(strings.ReplaceAll(currentFlags, flag, "")) for strings.Contains(newFlags, " ") { newFlags = strings.ReplaceAll(newFlags, " ", " ") } } - if err := s.UpdateEnvKeys(map[string]string{flagsKey: newFlags}, web.SampleEnv); err != nil { - http.Error(w, err.Error(), http.StatusInternalServerError) + return s.UpdateEnvKeys(map[string]string{flagsKey: newFlags}, web.SampleEnv) +} + +// HandleSaveReplacePlaylist injects or removes --replace-playlist=false from a playlist's FLAGS env var. +func (s *Settings) HandleSaveReplacePlaylist(w http.ResponseWriter, r *http.Request) { + var body struct { + ID string `json:"id"` + Name string `json:"name"` + Replace bool `json:"replace"` + } + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, "invalid JSON: "+err.Error(), http.StatusBadRequest) + return + } + + // The flag is the negative: present means "do not replace". + if err := s.setPlaylistFlag(body.ID, body.Name, "--replace-playlist=false", !body.Replace); err != nil { + http.Error(w, err.Error(), statusFor(err)) return } w.WriteHeader(http.StatusOK) } +// HandleSaveLocalOnly injects or removes --download-mode=skip from one +// playlist's FLAGS env var, so a playlist can be built from the library alone +// while the others keep downloading. +func (s *Settings) HandleSaveLocalOnly(w http.ResponseWriter, r *http.Request) { + var body struct { + ID string `json:"id"` + Name string `json:"name"` + LocalOnly bool `json:"local_only"` + } + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, "invalid JSON: "+err.Error(), http.StatusBadRequest) + return + } + + if err := s.setPlaylistFlag(body.ID, body.Name, localOnlyFlag, body.LocalOnly); err != nil { + http.Error(w, err.Error(), statusFor(err)) + return + } + w.WriteHeader(http.StatusOK) +} + +// statusFor keeps an unrecognised playlist a client error while anything else +// stays a server error. +func statusFor(err error) int { + if err != nil && err.Error() == "unknown playlist name" { + return http.StatusBadRequest + } + return http.StatusInternalServerError +} + // HandleSaveCleanDownloads injects or removes --clean-downloads from every playlist's FLAGS env var. func (s *Settings) HandleSaveCleanDownloads(w http.ResponseWriter, r *http.Request) { var body struct { @@ -429,19 +481,29 @@ func (s *Settings) HandleWizardStep2(w http.ResponseWriter, r *http.Request) { // handleWizardStep3 saves downloader configuration. func (s *Settings) HandleWizardStep3(w http.ResponseWriter, r *http.Request) { var body struct { - DownloadDir string `json:"download_dir"` - UseSubdirectory bool `json:"use_subdirectory"` - SlskdMigrateDL bool `json:"slskd_migrate_downloads"` - DownloadServices []string `json:"download_services"` - YoutubeAPIKey string `json:"youtube_api_key"` - TrackExtension string `json:"track_extension"` // yt-dlp - FilterList string `json:"filter_list"` - SlskdURL string `json:"slskd_url"` - SlskdAPIKey string `json:"slskd_api_key"` - LidarrURL string `json:"lidarr_url"` - LidarrAPIKey string `json:"lidarr_api_key"` - LidarrMigrateDL bool `json:"lidarr_migrate_downloads"` - Extensions string `json:"extensions"` // slskd + DownloadDir string `json:"download_dir"` + UseSubdirectory bool `json:"use_subdirectory"` + SlskdMigrateDL bool `json:"slskd_migrate_downloads"` + DownloadServices []string `json:"download_services"` + YoutubeAPIKey string `json:"youtube_api_key"` + TrackExtension string `json:"track_extension"` // yt-dlp + FilterList string `json:"filter_list"` + SlskdURL string `json:"slskd_url"` + SlskdAPIKey string `json:"slskd_api_key"` + SlskdAlbumMode bool `json:"slskd_album_mode"` + LidarrURL string `json:"lidarr_url"` + LidarrAPIKey string `json:"lidarr_api_key"` + LidarrMigrateDL bool `json:"lidarr_migrate_downloads"` + LidarrRootFolder string `json:"lidarr_root_folder"` + Extensions string `json:"extensions"` // slskd + MinBitRate int `json:"min_bitrate"` + MaxBitRate int `json:"max_bitrate"` + SizePreference string `json:"size_preference"` + ReleasePreference string `json:"release_preference"` + // A pointer because this one defaults to on: a plain bool cannot tell + // "the client cleared it" from "the client did not send it", and would + // quietly disable the feature on every payload that omits the key. + PreferOriginalRelease *bool `json:"prefer_original_release"` } if err := json.NewDecoder(r.Body).Decode(&body); err != nil { http.Error(w, "invalid JSON: "+err.Error(), http.StatusBadRequest) @@ -465,6 +527,21 @@ func (s *Settings) HandleWizardStep3(w http.ResponseWriter, r *http.Request) { if body.LidarrMigrateDL { lidarrMigrateDL = "true" } + albumMode := "false" + if body.SlskdAlbumMode { + albumMode = "true" + } + // An empty value would blank the key rather than record the default. + if body.SizePreference == "" { + body.SizePreference = "none" + } + if body.ReleasePreference == "" { + body.ReleasePreference = "smaller" + } + preferOriginal := "true" + if body.PreferOriginalRelease != nil && !*body.PreferOriginalRelease { + preferOriginal = "false" + } updates := map[string]string{ "DOWNLOAD_DIR": body.DownloadDir, "USE_SUBDIRECTORY": useSubdir, @@ -475,11 +552,20 @@ func (s *Settings) HandleWizardStep3(w http.ResponseWriter, r *http.Request) { "FILTER_LIST": body.FilterList, "SLSKD_URL": body.SlskdURL, "SLSKD_API_KEY": body.SlskdAPIKey, + "SLSKD_ALBUM_MODE": albumMode, "LIDARR_URL": body.LidarrURL, "LIDARR_API_KEY": body.LidarrAPIKey, "LIDARR_MIGRATE_DOWNLOADS": lidarrMigrateDL, + "LIDARR_ROOT_FOLDER": body.LidarrRootFolder, "EXTENSIONS": body.Extensions, // slskd - "WIZARD_COMPLETE": "true", + "MIN_BITRATE": strconv.Itoa(body.MinBitRate), + "MAX_BITRATE": strconv.Itoa(body.MaxBitRate), + "SIZE_PREFERENCE": body.SizePreference, + "RELEASE_PREFERENCE": body.ReleasePreference, + + "PREFER_ORIGINAL_RELEASE": preferOriginal, + + "WIZARD_COMPLETE": "true", } if err := s.UpdateEnvKeys(updates, web.SampleEnv); err != nil { diff --git a/src/web/backend/settings/playlist_flags_test.go b/src/web/backend/settings/playlist_flags_test.go new file mode 100644 index 00000000..dda86e4d --- /dev/null +++ b/src/web/backend/settings/playlist_flags_test.go @@ -0,0 +1,199 @@ +package settings + +import ( + "bytes" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" + + "explo/src/web/backend/app" +) + +// flagsSettings points a Settings at a throwaway .env seeded with the given +// contents. +func flagsSettings(t *testing.T, env string) (*Settings, string) { + t.Helper() + + envPath := filepath.Join(t.TempDir(), ".env") + if env != "" { + if err := os.WriteFile(envPath, []byte(env), 0o644); err != nil { + t.Fatalf("seeding env: %v", err) + } + } + + return NewSettings(app.Config{WebEnvPath: envPath}), envPath +} + +// flagsFor reads back one playlist's FLAGS value. +func flagsFor(t *testing.T, s *Settings, envPath, key string) string { + t.Helper() + + data, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading env: %v", err) + } + return s.ParseEnvText(string(data))[key] +} + +func postJSON(t *testing.T, handler http.HandlerFunc, body map[string]any) *httptest.ResponseRecorder { + t.Helper() + + payload, err := json.Marshal(body) + if err != nil { + t.Fatalf("marshaling body: %v", err) + } + + rec := httptest.NewRecorder() + handler(rec, httptest.NewRequest(http.MethodPost, "/", bytes.NewReader(payload))) + return rec +} + +// Turning the toggle on has to write the flag the runner actually reads. +func TestSaveLocalOnly_AddsAndRemovesTheFlag(t *testing.T) { + s, envPath := flagsSettings(t, "DAILY_JAMS_FLAGS=--playlist daily-jams\n") + + if rec := postJSON(t, s.HandleSaveLocalOnly, map[string]any{ + "id": "daily-jams", "local_only": true, + }); rec.Code != http.StatusOK { + t.Fatalf("enabling = %d (%s), want 200", rec.Code, rec.Body.String()) + } + + flags := flagsFor(t, s, envPath, "DAILY_JAMS_FLAGS") + if !strings.Contains(flags, "--download-mode=skip") { + t.Errorf("flags = %q, want the local-only flag", flags) + } + if !strings.Contains(flags, "--playlist daily-jams") { + t.Errorf("flags = %q, lost the existing flags", flags) + } + + if rec := postJSON(t, s.HandleSaveLocalOnly, map[string]any{ + "id": "daily-jams", "local_only": false, + }); rec.Code != http.StatusOK { + t.Fatalf("disabling = %d, want 200", rec.Code) + } + + flags = flagsFor(t, s, envPath, "DAILY_JAMS_FLAGS") + if strings.Contains(flags, "--download-mode=skip") { + t.Errorf("flags = %q, want the local-only flag gone", flags) + } + if !strings.Contains(flags, "--playlist daily-jams") { + t.Errorf("flags = %q, lost the existing flags on removal", flags) + } + if strings.Contains(flags, " ") { + t.Errorf("flags = %q, removal left a double space", flags) + } +} + +// The UI can fire the same request twice; neither direction may compound. +func TestSaveLocalOnly_IsIdempotent(t *testing.T) { + s, envPath := flagsSettings(t, "DAILY_JAMS_FLAGS=--playlist daily-jams\n") + + for i := 0; i < 3; i++ { + postJSON(t, s.HandleSaveLocalOnly, map[string]any{"id": "daily-jams", "local_only": true}) + } + if got := strings.Count(flagsFor(t, s, envPath, "DAILY_JAMS_FLAGS"), "--download-mode=skip"); got != 1 { + t.Errorf("flag appears %d times after repeated enables, want 1", got) + } + + for i := 0; i < 3; i++ { + postJSON(t, s.HandleSaveLocalOnly, map[string]any{"id": "daily-jams", "local_only": false}) + } + if flags := flagsFor(t, s, envPath, "DAILY_JAMS_FLAGS"); strings.Contains(flags, "--download-mode=skip") { + t.Errorf("flags = %q, want the flag gone after repeated disables", flags) + } +} + +// Setting one playlist to local-only must leave the others downloading -- the +// whole point is that it is per playlist. +func TestSaveLocalOnly_LeavesOtherPlaylistsAlone(t *testing.T) { + s, envPath := flagsSettings(t, "DAILY_JAMS_FLAGS=--playlist daily-jams\nWEEKLY_JAMS_FLAGS=--playlist weekly-jams\n") + + postJSON(t, s.HandleSaveLocalOnly, map[string]any{"id": "daily-jams", "local_only": true}) + + if flags := flagsFor(t, s, envPath, "WEEKLY_JAMS_FLAGS"); strings.Contains(flags, "--download-mode=skip") { + t.Errorf("weekly-jams = %q, want it untouched", flags) + } +} + +// It must not disturb the toggle that sits next to it in the same menu. +func TestSaveLocalOnly_PreservesTheReplaceFlag(t *testing.T) { + s, envPath := flagsSettings(t, "DAILY_JAMS_FLAGS=--playlist daily-jams --replace-playlist=false\n") + + postJSON(t, s.HandleSaveLocalOnly, map[string]any{"id": "daily-jams", "local_only": true}) + + flags := flagsFor(t, s, envPath, "DAILY_JAMS_FLAGS") + if !strings.Contains(flags, "--replace-playlist=false") { + t.Errorf("flags = %q, lost the replace setting", flags) + } + if !strings.Contains(flags, "--download-mode=skip") { + t.Errorf("flags = %q, want the local-only flag", flags) + } +} + +// Saving a schedule rewrites FLAGS wholesale from a default and re-adds only +// the flags it knows about. Leaving the new one off that list would silently +// turn local-only back off the next time the schedule was edited. +func TestSaveSchedule_PreservesLocalOnly(t *testing.T) { + s, envPath := flagsSettings(t, "DAILY_JAMS_FLAGS=--playlist daily-jams --download-mode=skip\n") + + rec := postJSON(t, s.HandleSaveSchedule, map[string]any{ + "id": "daily-jams", "enabled": true, "day": -1, "hour": 3, "minute": 30, + }) + if rec.Code != http.StatusOK { + t.Fatalf("saving schedule = %d (%s), want 200", rec.Code, rec.Body.String()) + } + + if flags := flagsFor(t, s, envPath, "DAILY_JAMS_FLAGS"); !strings.Contains(flags, "--download-mode=skip") { + t.Errorf("flags = %q, editing the schedule dropped local-only", flags) + } +} + +// Custom playlists get their env prefix from their name, not their id. +func TestSaveLocalOnly_SupportsCustomPlaylists(t *testing.T) { + s, envPath := flagsSettings(t, "") + + rec := postJSON(t, s.HandleSaveLocalOnly, map[string]any{ + "id": "custom-abc123", "name": "Road Trip", "local_only": true, + }) + if rec.Code != http.StatusOK { + t.Fatalf("custom playlist = %d (%s), want 200", rec.Code, rec.Body.String()) + } + + data, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading env: %v", err) + } + if !bytes.Contains(data, []byte("--download-mode=skip")) { + t.Errorf("custom playlist flags were not written:\n%s", data) + } +} + +// An unrecognised playlist is the caller's mistake, not a server fault. +func TestSaveLocalOnly_RejectsAnUnknownPlaylist(t *testing.T) { + s, _ := flagsSettings(t, "") + + rec := postJSON(t, s.HandleSaveLocalOnly, map[string]any{"id": "nonsense", "local_only": true}) + if rec.Code != http.StatusBadRequest { + t.Errorf("unknown playlist = %d, want 400", rec.Code) + } +} + +// The refactor that introduced the shared helper must not have changed how the +// neighbouring toggle behaves. +func TestSaveReplacePlaylist_StillWorks(t *testing.T) { + s, envPath := flagsSettings(t, "DAILY_JAMS_FLAGS=--playlist daily-jams\n") + + postJSON(t, s.HandleSaveReplacePlaylist, map[string]any{"id": "daily-jams", "replace": false}) + if flags := flagsFor(t, s, envPath, "DAILY_JAMS_FLAGS"); !strings.Contains(flags, "--replace-playlist=false") { + t.Errorf("flags = %q, want the replace flag added when replace is off", flags) + } + + postJSON(t, s.HandleSaveReplacePlaylist, map[string]any{"id": "daily-jams", "replace": true}) + if flags := flagsFor(t, s, envPath, "DAILY_JAMS_FLAGS"); strings.Contains(flags, "--replace-playlist=false") { + t.Errorf("flags = %q, want the replace flag gone when replace is on", flags) + } +} diff --git a/src/web/backend/settings/settings.go b/src/web/backend/settings/settings.go index 4159d758..1aa49cdf 100644 --- a/src/web/backend/settings/settings.go +++ b/src/web/backend/settings/settings.go @@ -3,7 +3,9 @@ package settings import ( "fmt" "os" + "path/filepath" "strings" + "sync" "explo/src/web/backend/app" ) @@ -68,9 +70,64 @@ func validateEnvText(text string) error { return nil } +// envFileMu serializes every read-modify-write of the .env file. Without +// it, two settings saves in quick succession each read the same starting +// content and the later write silently undid the earlier one's changes. +var envFileMu sync.Mutex + +// writeEnvFileAtomic writes via a temp file in the same directory plus +// rename, so a crash mid-save (or a concurrent reader) never sees a +// truncated .env. +func writeEnvFileAtomic(path string, data []byte) error { + tmp, err := os.CreateTemp(filepath.Dir(path), ".env-*.tmp") + if err != nil { + return err + } + tmpName := tmp.Name() + defer func() { _ = os.Remove(tmpName) }() + if _, err := tmp.Write(data); err != nil { + _ = tmp.Close() + return err + } + if err := tmp.Close(); err != nil { + return err + } + return os.Rename(tmpName, path) +} + +// validateEnvUpdate rejects keys and values that could inject extra .env +// lines. A value containing CR/LF would otherwise be written raw as +// key=value and the remainder parsed as additional keys on next start. +func validateEnvUpdate(key, value string) error { + if key == "" { + return fmt.Errorf("env key must not be empty") + } + for i, r := range key { + switch { + case r == '_' || ('A' <= r && r <= 'Z') || ('a' <= r && r <= 'z'): + case '0' <= r && r <= '9' && i > 0: + default: + return fmt.Errorf("invalid env key %q", key) + } + } + if strings.ContainsAny(value, "\r\n") { + return fmt.Errorf("env value for %s must not contain line breaks", key) + } + return nil +} + // updateEnvKeys reads the env file (falling back to fallback if missing), updates the // given key=value pairs in-place preserving comments, and writes the result back. func (s *Settings) UpdateEnvKeys(updates map[string]string, fallback []byte) error { + for k, v := range updates { + if err := validateEnvUpdate(k, v); err != nil { + return err + } + } + + envFileMu.Lock() + defer envFileMu.Unlock() + path := s.cfg.WebEnvPath data, err := os.ReadFile(path) if os.IsNotExist(err) { @@ -121,7 +178,7 @@ func (s *Settings) UpdateEnvKeys(updates map[string]string, fallback []byte) err prevBlank = blank } - return os.WriteFile(path, []byte(strings.Join(out, "\n")+"\n"), 0600) + return writeEnvFileAtomic(path, []byte(strings.Join(out, "\n")+"\n")) } // Check for special chars in env vars that might need quoting diff --git a/src/web/backend/settings/wizard_test.go b/src/web/backend/settings/wizard_test.go new file mode 100644 index 00000000..7075e791 --- /dev/null +++ b/src/web/backend/settings/wizard_test.go @@ -0,0 +1,276 @@ +package settings + +import ( + "bytes" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "explo/src/web/backend/app" + "explo/src/web/backend/defs" +) + +// newSettings points a Settings at a throwaway .env. +func newSettings(t *testing.T) (*Settings, string) { + t.Helper() + + envPath := filepath.Join(t.TempDir(), ".env") + + return NewSettings(app.Config{WebEnvPath: envPath}), envPath +} + +// postStep3 drives the wizard's downloader step with the given body. +func postStep3(t *testing.T, s *Settings, body map[string]any) *httptest.ResponseRecorder { + t.Helper() + + payload, err := json.Marshal(body) + if err != nil { + t.Fatalf("marshaling body: %v", err) + } + + req := httptest.NewRequest(http.MethodPost, "/api/ui/wizard/step3", bytes.NewReader(payload)) + rec := httptest.NewRecorder() + s.HandleWizardStep3(rec, req) + + return rec +} + +// The toggle is only useful if it reaches the .env the downloader reads. +func TestWizardStep3_PersistsAlbumMode(t *testing.T) { + tests := []struct { + name string + on bool + want string + }{ + {"enabled", true, "SLSKD_ALBUM_MODE=true"}, + {"disabled", false, "SLSKD_ALBUM_MODE=false"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + s, envPath := newSettings(t) + + rec := postStep3(t, s, map[string]any{ + "download_services": []string{"slskd"}, + "slskd_album_mode": tt.on, + }) + if rec.Code != http.StatusOK { + t.Fatalf("step3 = %d (%s), want 200", rec.Code, rec.Body.String()) + } + + written, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading written env: %v", err) + } + if !bytes.Contains(written, []byte(tt.want)) { + t.Errorf("written env does not contain %q:\n%s", tt.want, written) + } + }) + } +} + +// AllConfigKeys is what /api/config reads back. Leaving the key out of it is +// silent: the wizard writes the toggle, then shows it as off next time it is +// opened, because the value never reaches the frontend. +func TestAlbumModeIsReadableByTheUI(t *testing.T) { + if !slices.Contains(defs.AllConfigKeys, "SLSKD_ALBUM_MODE") { + t.Error("SLSKD_ALBUM_MODE missing from AllConfigKeys; the wizard toggle would always read back as off") + } +} + +// The bitrate range is only useful if the wizard writes what the downloader +// reads. +func TestWizardStep3_PersistsTheBitrateRange(t *testing.T) { + s, envPath := newSettings(t) + + rec := postStep3(t, s, map[string]any{ + "download_services": []string{"slskd"}, + "extensions": "flac,mp3", + "min_bitrate": 192, + "max_bitrate": 320, + }) + if rec.Code != http.StatusOK { + t.Fatalf("step3 = %d (%s), want 200", rec.Code, rec.Body.String()) + } + + written, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading written env: %v", err) + } + for _, want := range []string{"MIN_BITRATE=192", "MAX_BITRATE=320"} { + if !bytes.Contains(written, []byte(want)) { + t.Errorf("written env does not contain %q:\n%s", want, written) + } + } +} + +// A zero ceiling is a real setting -- it means "no limit" -- so it has to be +// written rather than treated as absent. +func TestWizardStep3_WritesAZeroCeiling(t *testing.T) { + s, envPath := newSettings(t) + + postStep3(t, s, map[string]any{ + "download_services": []string{"slskd"}, + "min_bitrate": 256, + "max_bitrate": 0, + }) + + written, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading written env: %v", err) + } + if !bytes.Contains(written, []byte("MAX_BITRATE=0")) { + t.Errorf("a zero ceiling was not written:\n%s", written) + } +} + +// AllConfigKeys is what /api/config returns. A key missing from it is written +// and then never read back, so the control silently resets to its default on +// every reload with nothing to explain why. +func TestBitrateRangeIsReadableByTheUI(t *testing.T) { + for _, key := range []string{"MIN_BITRATE", "MAX_BITRATE"} { + if !slices.Contains(defs.AllConfigKeys, key) { + t.Errorf("%s missing from AllConfigKeys; the control would always read back as its default", key) + } + } +} + +// The preference is only useful if the wizard writes what the downloader reads. +func TestWizardStep3_PersistsTheSizePreference(t *testing.T) { + s, envPath := newSettings(t) + + postStep3(t, s, map[string]any{ + "download_services": []string{"slskd"}, + "size_preference": "smaller", + }) + + written, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading written env: %v", err) + } + if !bytes.Contains(written, []byte("SIZE_PREFERENCE=smaller")) { + t.Errorf("written env does not record the preference:\n%s", written) + } +} + +// An absent value must record the default rather than blanking the key, which +// is what UpdateEnvKeys does with an empty string. +func TestWizardStep3_DefaultsTheSizePreference(t *testing.T) { + s, envPath := newSettings(t) + + postStep3(t, s, map[string]any{"download_services": []string{"slskd"}}) + + written, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading written env: %v", err) + } + assertEnvValue(t, written, "SIZE_PREFERENCE", "none") +} + +func TestSizePreferenceIsReadableByTheUI(t *testing.T) { + if !slices.Contains(defs.AllConfigKeys, "SIZE_PREFERENCE") { + t.Error("SIZE_PREFERENCE missing from AllConfigKeys; the control would always read back as its default") + } +} + +// envValue reads a key's written value, ignoring commented lines. sample.env +// documents every key as a comment carrying its default, so a plain substring +// search for "KEY=default" matches that comment and passes whether or not the +// wizard wrote anything at all. +func envValue(written []byte, key string) string { + for _, line := range strings.Split(string(written), "\n") { + line = strings.TrimSpace(line) + if strings.HasPrefix(line, "#") { + continue + } + if name, value, ok := strings.Cut(line, "="); ok && strings.TrimSpace(name) == key { + return strings.TrimSpace(value) + } + } + return "" +} + +func assertEnvValue(t *testing.T, written []byte, key, want string) { + t.Helper() + + if got := envValue(written, key); got != want { + t.Errorf("%s = %q, want %q\n%s", key, got, want, written) + } +} + +func TestWizardStep3_PersistsTheReleasePreference(t *testing.T) { + s, envPath := newSettings(t) + + postStep3(t, s, map[string]any{ + "download_services": []string{"slskd"}, + "release_preference": "smaller", + }) + + written, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading written env: %v", err) + } + if !bytes.Contains(written, []byte("RELEASE_PREFERENCE=smaller")) { + t.Errorf("written env does not record the release preference:\n%s", written) + } +} + +// An absent value must record the default rather than blanking the key. +func TestWizardStep3_DefaultsTheReleasePreference(t *testing.T) { + s, envPath := newSettings(t) + + postStep3(t, s, map[string]any{"download_services": []string{"slskd"}}) + + written, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading written env: %v", err) + } + assertEnvValue(t, written, "RELEASE_PREFERENCE", "smaller") +} + +func TestReleasePreferenceIsReadableByTheUI(t *testing.T) { + if !slices.Contains(defs.AllConfigKeys, "RELEASE_PREFERENCE") { + t.Error("RELEASE_PREFERENCE missing from AllConfigKeys; the control would always read back as its default") + } +} + +func TestWizardStep3_PersistsPreferOriginalRelease(t *testing.T) { + s, envPath := newSettings(t) + + postStep3(t, s, map[string]any{ + "download_services": []string{"slskd"}, + "prefer_original_release": false, + }) + + written, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading written env: %v", err) + } + assertEnvValue(t, written, "PREFER_ORIGINAL_RELEASE", "false") +} + +// This one defaults to on, so an omitted key must not read as "off". A plain +// bool on the request body cannot tell the two apart, which is why the field is +// a pointer. +func TestWizardStep3_DefaultsPreferOriginalReleaseToOn(t *testing.T) { + s, envPath := newSettings(t) + + postStep3(t, s, map[string]any{"download_services": []string{"slskd"}}) + + written, err := os.ReadFile(envPath) + if err != nil { + t.Fatalf("reading written env: %v", err) + } + assertEnvValue(t, written, "PREFER_ORIGINAL_RELEASE", "true") +} + +func TestPreferOriginalReleaseIsReadableByTheUI(t *testing.T) { + if !slices.Contains(defs.AllConfigKeys, "PREFER_ORIGINAL_RELEASE") { + t.Error("PREFER_ORIGINAL_RELEASE missing from AllConfigKeys; the toggle would always read back as its default") + } +} diff --git a/src/web/frontend/src/App.jsx b/src/web/frontend/src/App.jsx index 0d69946c..dd2837f7 100644 --- a/src/web/frontend/src/App.jsx +++ b/src/web/frontend/src/App.jsx @@ -13,6 +13,7 @@ export default function App() { const [bgLoaded, setBgLoaded] = useState(false) const [fadingOut, setFadingOut] = useState(false) const [authDisabled, setAuthDisabled] = useState(false) + const [loadError, setLoadError] = useState(null) useEffect(() => { Promise.all([ @@ -26,6 +27,11 @@ export default function App() { } else { setView('login') } + }).catch((e) => { + // A server hiccup during startup used to leave the page blank + // forever: the view never left its initial null state. + setLoadError(e?.message || 'failed to reach the server') + setView('error') }) fetchBackgroundArt().then((artUrl) => { if (artUrl) setBgUrl(artUrl) @@ -33,7 +39,14 @@ export default function App() { }, []) async function handleLoginSuccess({ fromLogin = false } = {}) { - const { values, sources } = await fetchConfig() + let values, sources + try { + ({ values, sources } = await fetchConfig()) + } catch (e) { + setLoadError(e?.message || 'failed to load configuration') + setView('error') + return + } setConfig(values) setEnvSources(sources || {}) const nextView = values.WIZARD_COMPLETE === 'true' ? 'settings' : 'wizard' @@ -48,6 +61,23 @@ export default function App() { } } + if (view === 'error') { + return ( +
+
+

Couldn't load Explo

+

{loadError}

+ +
+
+ ) + } + if (view === null) return
if (view === 'login') { diff --git a/src/web/frontend/src/components/Settings.jsx b/src/web/frontend/src/components/Settings.jsx index 2908642f..74685958 100644 --- a/src/web/frontend/src/components/Settings.jsx +++ b/src/web/frontend/src/components/Settings.jsx @@ -15,7 +15,8 @@ import { fetchConfig, fetchConfigRaw, saveConfig, resetConfig, saveSchedule, startRun, stopRun, fetchRunStatus, fetchLogs, fetchCustomPlaylists, deleteCustomPlaylist, savePathTemplate, saveEnrichMetadata, - saveReplacePlaylist, saveCleanDownloads, + saveReplacePlaylist, + saveLocalOnly, saveCleanDownloads, fetchPathTemplatePresets, addPathTemplatePreset, deletePathTemplatePreset, } from '../lib/api' import { parseSlogLine, cronToFields, highlightEnv } from '../lib/utils' @@ -203,6 +204,7 @@ function CustomPlaylistsSection({ function HomeSection() { const [schedules, setSchedules] = useState(null) const [replacePlaylists, setReplacePlaylists] = useState({}) + const [localOnlyPlaylists, setLocalOnlyPlaylists] = useState({}) const [scheduleSaveStatus, setScheduleSaveStatus] = useState({}) const [lbUser, setLbUser] = useState('') const [openTracklist, setOpenTracklist] = useState(null) @@ -229,10 +231,14 @@ function HomeSection() { setCustomPlaylists(customList) const rp = {} + const lo = {} for (const p of PLAYLISTS) { - rp[p.value] = !(values[p.flagsKey] || '').includes('--replace-playlist=false') + const flags = values[p.flagsKey] || '' + rp[p.value] = !flags.includes('--replace-playlist=false') + lo[p.value] = flags.includes('--download-mode=skip') } setReplacePlaylists(rp) + setLocalOnlyPlaylists(lo) const s = {} for (const p of PLAYLISTS) { @@ -324,6 +330,14 @@ function HomeSection() { setReplacePlaylists(prev => ({ ...prev, [id]: !next })) ) }, + localOnly: localOnlyPlaylists[id] ?? false, + onLocalOnlyToggle: () => { + const next = !(localOnlyPlaylists[id] ?? false) + setLocalOnlyPlaylists(prev => ({ ...prev, [id]: next })) + saveLocalOnly(id, s.name, next).catch(() => + setLocalOnlyPlaylists(prev => ({ ...prev, [id]: !next })) + ) + }, onDayChange: day => setSchedules(prev => ({ ...prev, [id]: { ...prev[id], day } })), @@ -384,7 +398,7 @@ function HomeSection() { playlist={openTracklist} refreshTick={refreshTick} onRun={async () => { - await startRun(openTracklist, 'normal', replacePlaylists[playlist] ?? true) + await startRun(openTracklist, 'normal', replacePlaylists[openTracklist] ?? true) setRunning(true) setStatus('running…') setLogEntries([]) diff --git a/src/web/frontend/src/components/Wizard.jsx b/src/web/frontend/src/components/Wizard.jsx index 56421f23..8e956f7e 100644 --- a/src/web/frontend/src/components/Wizard.jsx +++ b/src/web/frontend/src/components/Wizard.jsx @@ -10,6 +10,7 @@ import { useState } from 'react' import { wizardStep1, wizardStep2, wizardStep3, prefetchPlaylists } from '../lib/api' +import QualityRange from './ui/QualityRange' import { ToggleRow } from './ui/Toggle' import { DirInput } from './ui/DirInput' import { TextField } from './ui/common' @@ -167,7 +168,7 @@ function Step2({ fields, setField, envSources, onBack, onNext, saving }) { sleepMinutes, publicPlaylist, } = fields; - const isLocked = (key) => envSources[key] === ".env"; + const isLocked = (key) => envSources[key] === "env"; const urlPlaceholder = () => { const ports = { jellyfin: '8096', emby: '8096', plex: '32400', subsonic: '4533' } @@ -496,11 +497,17 @@ function Step3({ fields, setField, envSources, onBack, onFinish, saving }) { filterList, slskdUrl, slskdApiKey, + slskdAlbumMode, lidarrUrl, lidarrApiKey, lidarrMigrateDownloads, lidarrRootFolder, extensions, + minBitrate, + maxBitrate, + sizePreference, + releasePreference, + preferOriginalRelease, } = fields; const isLocked = (key) => envSources[key] === "env"; @@ -606,11 +613,67 @@ function Step3({ fields, setField, envSources, onBack, onFinish, saving }) { setField('slskdApiKey', e.target.value)} autoComplete="off" spellCheck={false} disabled={isLocked('SLSKD_API_KEY')} /> - - setField('extensions', e.target.value)} - placeholder="flac,mp3" autoComplete="off" spellCheck={false} disabled={isLocked('EXTENSIONS')} /> - + { + setField('extensions', next.extensions) + setField('minBitrate', next.minBitrate) + setField('maxBitrate', next.maxBitrate) + setField('sizePreference', next.sizePreference) + }} + /> + setField("slskdAlbumMode", v)} + disabled={isLocked("SLSKD_ALBUM_MODE")} + name="Download full albums" + desc="Fetch the whole release each recommendation belongs to. Only the recommended track is added to the playlist; the rest land in your library. Downloads much more per run." + /> + +
+

Which release to take

+

+ Smaller picks the edition closest to the album's real length, so a 21-track + deluxe does not arrive in place of the 11-track album. Needs Enrich track + metadata, since that is where the real length comes from; without it this + falls back to taking the fuller release. +

+
+ {[ + { value: 'fuller', label: 'Fuller release' }, + { value: 'smaller', label: 'Smaller release' }, + ].map(opt => { + const active = (releasePreference || 'smaller') === opt.value + return ( + + ) + })} +
+
+ setField("preferOriginalRelease", v)} + disabled={isLocked("PREFER_ORIGINAL_RELEASE")} + name="Prefer the original album" + desc="Trim a release back to the original album: drop extra discs a peer has flattened into one folder, then drop bonus tracks past the album's real length. A genuine double album is kept whole, and the recommended track is never trimmed away." + /> + {/* Show keyword exclusion when YouTube isn't enabled — otherwise it lives in the YouTube section */} { if (!menuOpen) { setConfirmDelete(false); setDeleteTracksChecked(false); return } @@ -707,6 +710,40 @@ export function PlaylistCard({ )}
)} + {onLocalOnlyToggle && ( +
+ + {localOnlyInfoOpen && ( + + Builds this playlist from tracks already in your library instead of + downloading, so it may come out shorter. Applies to scheduled runs; + the Run button follows its own download mode. + + )} +
+ )} {onDelete && !confirmDelete && ( + ) + })} + +

+ {PREFERENCE_HINTS[sizePreference || 'none']} +

+ + + ) +} diff --git a/src/web/frontend/src/lib/api.js b/src/web/frontend/src/lib/api.js index 0331c4fb..eabd1b88 100644 --- a/src/web/frontend/src/lib/api.js +++ b/src/web/frontend/src/lib/api.js @@ -11,19 +11,31 @@ async function ensureCSRF() { async function apiFetch(url, options = {}) { const method = (options.method || 'GET').toUpperCase() + const mutating = ['POST', 'PUT', 'PATCH', 'DELETE'].includes(method) const headers = new Headers(options.headers || {}) - if (['POST', 'PUT', 'PATCH', 'DELETE'].includes(method)) { - const token = await ensureCSRF() - headers.set('X-CSRF-Token', token) - } - - return fetch(url, { + const send = () => fetch(url, { credentials: 'include', ...options, headers, }) + + if (!mutating) return send() + + headers.set('X-CSRF-Token', await ensureCSRF()) + let res = await send() + + // The cached token goes stale when the backend restarts or the session + // expires (sessions are in-memory): drop it, fetch a fresh one, and + // retry the request once instead of failing every button until reload. + if (res.status === 403) { + csrfToken = null + headers.set('X-CSRF-Token', await ensureCSRF()) + res = await send() + } + + return res } export async function checkAuth() { @@ -53,6 +65,7 @@ export async function login(username, password) { export async function fetchConfig() { const res = await apiFetch('/api/ui/config') + if (!res.ok) throw new Error(`config load failed (${res.status}): ${await res.text()}`) return res.json() } @@ -241,6 +254,15 @@ export async function saveReplacePlaylist(id, name, replace) { if (!res.ok) throw new Error(await res.text()) } +export async function saveLocalOnly(id, name, localOnly) { + const res = await apiFetch('/api/ui/config/local-only', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ id, name, local_only: localOnly }), + }) + if (!res.ok) throw new Error(await res.text()) +} + export async function saveCleanDownloads(enabled) { const res = await apiFetch('/api/ui/config/clean-downloads', { method: 'POST', diff --git a/src/web/frontend/src/lib/utils.js b/src/web/frontend/src/lib/utils.js index 56bbd2f1..6d97d2c0 100644 --- a/src/web/frontend/src/lib/utils.js +++ b/src/web/frontend/src/lib/utils.js @@ -2,7 +2,11 @@ function escHtml(s) { return s.replace(/&/g, '&').replace(//g, '>') } -const SENSITIVE_KEY_RE = /password|token|api_key|secret|accesstoken/i +// Mask by meaning, not exact spelling: API keys arrive as API_KEY, +// APIKEY, ADMIN_API_KEY or X-API-KEY depending on the service, and the +// same goes for tokens and passwords. An unrecognized secret-looking key +// must be masked, never shown, so match the separator-optional forms. +const SENSITIVE_KEY_RE = /pass(word|wd)?|token|secret|credential|api[-_]?key|access[-_]?key|auth[-_]?key|private[-_]?key/i export function highlightEnv(text, masked = false) { return text.split('\n').map(line => { diff --git a/src/web/sample.env b/src/web/sample.env index f6f33587..39fc9d6b 100644 --- a/src/web/sample.env +++ b/src/web/sample.env @@ -1,6 +1,10 @@ # === Web UI === # Web cache limit in MB (cover art, backgrounds) (default: 500) # WEB_CACHE_MB=500 +# Kill a run started from the web UI after this many minutes. Without it a +# wedged run keeps the UI refusing new runs until it is stopped by hand. +# 0 disables the timeout. (default: 240) +# RUN_TIMEOUT=240 # === Discovery Config === @@ -52,6 +56,15 @@ LIBRARY_NAME= # KEEP_PERMISSIONS=true # Comma-separated list (no spaces) of download services, in priority order (supported: lidarr,slskd,youtube) (default: youtube) # DOWNLOAD_SERVICES=youtube +# Time budget (minutes) for a single download service before Explo gives up on it +# and lets the next service in DOWNLOAD_SERVICES try the remaining tracks. +# 0 (default) waits indefinitely. Example chain -- prefer full albums from Lidarr, +# fall back to slskd for anything it hasn't delivered in 15 minutes: +# DOWNLOAD_SERVICES=lidarr,slskd +# LIDARR_FALLBACK_AFTER=15 +# SLSKD_FALLBACK_AFTER=10 +# LIDARR_FALLBACK_AFTER=0 +# SLSKD_FALLBACK_AFTER=0 # Path templating, Options are Artist, Album, TrackName, TrackNumber, File, Ext (eg. "{{Artist}}/{{Album}}/{{File}}") # PATH_TEMPLATE="" @@ -85,6 +98,11 @@ LIBRARY_NAME= # SLSKD_MIGRATE_DOWNLOADS=false # Overwrite track metadata when moving file (default: false) # SLSKD_OVERWRITE_METADATA=false +# Download the whole release the matched file belongs to instead of just the +# recommended track. Only the recommended track is added to the playlist; the +# rest simply land in the library. Off by default -- it multiplies transfer +# volume per recommendation. (default: false) +# SLSKD_ALBUM_MODE=false # Directory where slskd downloads tracks (default: /slskd/) # PS! This is only needed on the binary version, in docker it's set through volume mapping # SLSKD_DIR=/slskd/ @@ -101,6 +119,31 @@ LIBRARY_NAME= # MIN_BIT_DEPTH=8 # Minimal Bitrate (default: 256) # MIN_BITRATE=256 +# Highest bitrate in kbps to accept, so a single track cannot arrive as a 138 MB +# WAV. Worked out from file size when a peer reports no bitrate, so it applies +# to files that describe themselves and files that do not. 0 disables the +# ceiling. (default: 0) +# MAX_BITRATE=0 +# Which end of the accepted range to reach for first. "none" leaves the order of +# EXTENSIONS deciding, as it always has. "smaller" takes the best lossy file and +# falls back to lossless only when there is none -- note it still prefers a 320 +# over a 256, since wanting a small file does not mean wanting a bad one. +# "larger" is the reverse. Ranking only; nothing is rejected. (default: none) +# SIZE_PREFERENCE=none +# Which release album mode takes when several match. "fuller" takes the most +# complete directory, as it always has. "smaller" takes the one closest to the +# album's real track count, so a 21-track deluxe edition does not arrive in +# place of the 11-track album -- note this needs ENRICH_TRACK_METADATA, since +# that is where the real track count comes from, and falls back to "fuller" +# without it. (default: fuller) +# RELEASE_PREFERENCE=fuller +# Trim a release down to the original album before downloading it. Extra discs a +# peer has flattened into one directory are dropped (a real double album is kept +# whole when ENRICH_TRACK_METADATA confirms it), and bonus tracks appended past +# the album's real length are dropped after them. The recommended track is always +# kept, even when it is itself a bonus track. Set to false to take releases +# exactly as the peer shares them. (default: true) +# PREFER_ORIGINAL_RELEASE=true # Comma-separated (without spaces) keywords to avoid, when filtering slskd results (default: live,remix,instrumental,extended,clean,acapella) # FILTER_LIST=live,remix,instrumental,extended,clean,acapella