feat: secondary manifest repository signal (CM-1393) - #4545
Draft
joanagmaia wants to merge 1 commit into
Draft
Conversation
Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>
This was referenced Sep 1, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds secondary manifest repository signals to improve package-to-repository coverage across registry workers.
Changes:
- Adds shared ordered fallback resolution with VCS host gating.
- Integrates primary/secondary signals across package ecosystems.
- Documents the architecture decision in ADR-0021.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
services/libs/data-access-layer/src/packages/packages.ts |
Returns Packagist homepage data. |
services/apps/packages_worker/src/utils/resolveManifestRepo.ts |
Adds shared manifest resolver. |
services/apps/packages_worker/src/utils/__tests__/resolveManifestRepo.test.ts |
Tests resolver behavior. |
services/apps/packages_worker/src/rubygems/types.ts |
Extends RubyGems repository types. |
services/apps/packages_worker/src/rubygems/runRubyGemsCoreLoop.ts |
Persists RubyGems signals. |
services/apps/packages_worker/src/rubygems/normalize.ts |
Adds RubyGems fallbacks. |
services/apps/packages_worker/src/pypi/upsertProject.ts |
Persists PyPI signals. |
services/apps/packages_worker/src/pypi/normalize.ts |
Classifies PyPI fallback fields. |
services/apps/packages_worker/src/pypi/__tests__/normalize.test.ts |
Updates classification expectations. |
services/apps/packages_worker/src/packagist/upsertPackageInfo.ts |
Adds Packagist homepage fallback. |
services/apps/packages_worker/src/packagist/__tests__/persistPackageInfo.test.ts |
Tests Packagist fallback behavior. |
services/apps/packages_worker/src/nuget/types.ts |
Extends NuGet repository types. |
services/apps/packages_worker/src/nuget/runNuGetEnrichmentLoop.ts |
Persists NuGet signals. |
services/apps/packages_worker/src/nuget/normalize.ts |
Adds NuGet project URL fallback. |
services/apps/packages_worker/src/npm/upsertPackage.ts |
Persists npm signals. |
services/apps/packages_worker/src/npm/types.ts |
Models npm bug URLs. |
services/apps/packages_worker/src/npm/normalize.ts |
Adds npm fallback chain. |
services/apps/packages_worker/src/maven/runMavenEnrichmentLoop.ts |
Adds Maven homepage fallback. |
services/apps/packages_worker/src/cargo/types.ts |
Adds Cargo fallback statistics. |
services/apps/packages_worker/src/cargo/normalizeRepos.ts |
Builds Cargo repository choices. |
services/apps/packages_worker/src/cargo/enrich.ts |
Persists Cargo choices and signals. |
docs/adr/README.md |
Indexes ADR-0021. |
docs/adr/0021-secondary-manifest-repository-signal.md |
Records the fallback policy. |
Suppressed comments (2)
services/apps/packages_worker/src/maven/runMavenEnrichmentLoop.ts:344
- Maven's prior
declaredclaim is not replaced here. URL changes/removal leave stale links, and primary A becoming secondary A cannot downgrade because keep-highest retains the old primary score. Transactionally delete Maven-owned declared links before writing the current result, including whenrepositoryUrlis null.
await writeRepoLink(t, packageId, repositoryUrl, changed, fallbackRepo ? 'secondary' : 'primary')
services/apps/packages_worker/src/cargo/enrich.ts:250
rc.signalis not guaranteed to be stored when a crate's primary URL disappears but its homepage resolves to the same repo. The existing primary row is retained by the keep-highest conflict policy, and the preceding prune keeps the matching repo ID. Delete Cargo-owned declared claims before relinking so a current secondary signal can replace stale primary evidence.
SELECT rc.package_id, r.id, $(source), rc.signal, 'no_evidence', NULL,
s.confidence, NOW(), NOW()
FROM ${STAGING_SCHEMA}.repo_choice rc
JOIN repos r ON r.url = rc.repository_url
JOIN packages p ON p.id = rc.package_id
CROSS JOIN LATERAL (SELECT ${CARGO_CONFIDENCE} AS confidence) s
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+69
to
70
| repository_url = CASE WHEN e.declared_repository_url IS NOT NULL OR rn.repository_url IS NOT NULL | ||
| THEN rn.repository_url ELSE p.repository_url END, |
Comment on lines
+92
to
+95
| const linkChanged = await upsertPackageRepo(t, pkgId, repoId, { | ||
| source: 'declared', | ||
| signal: resolvedRepo.signal, | ||
| }) |
Comment on lines
143
to
146
| const linkChanged = await upsertPackageRepo(t, packageDbId.toString(), repoId, { | ||
| source: 'declared', | ||
| signal: normalized.resolvedRepo.signal, | ||
| }) |
Comment on lines
+75
to
81
| if (resolvedRepo) { | ||
| const repo = await getOrCreateRepoByUrl(t, resolvedRepo.repo.url, resolvedRepo.repo.host) | ||
| const linkChanged = await upsertPackageRepo(t, id, repo.id, { | ||
| source: 'declared', | ||
| signal: resolvedRepo.signal, | ||
| }) | ||
| const removedFields = await removeDeclaredPackageRepo(t, id, repo.id) |
Comment on lines
+238
to
+243
| if (!declaredRepositoryUrl) { | ||
| const tracker = entries.find(([k, v]) => /bug|issue|tracker/i.test(k) && REPO_HOST.test(v))?.[1] | ||
| if (tracker) { | ||
| declaredRepositoryUrl = tracker | ||
| declaredRepositoryField = 'bug_tracker' | ||
| } |
Comment on lines
+266
to
+270
| const scmRepositoryUrl = normalizeScmUrl(result.scmUrl) | ||
| const fallbackRepo = scmRepositoryUrl | ||
| ? null | ||
| : resolveManifestRepo([{ field: 'url', url: result.homepageUrl, signal: 'secondary' }]) | ||
| const repositoryUrl = scmRepositoryUrl ?? fallbackRepo?.repo.url ?? null |
Comment on lines
+28
to
33
| * `cargo_sync.repo_choice` then picks one repo per crate: the declared `repository` | ||
| * field (`primary`) or, when that is absent or unparseable, the `homepage` — accepted | ||
| * only on a recognized VCS host, since a homepage is free-form. `documentation` is not | ||
| * staged from the dump and is almost always docs.rs, which that gate rejects anyway. | ||
| * | ||
| * Normalization runs in TypeScript because the bulk set-based cargo pipeline |
Comment on lines
+233
to
234
| // Many projects only declare a Homepage, or only a Bug Tracker, that is itself the repo. | ||
| if (!declaredRepositoryUrl && homepage && REPO_HOST.test(homepage)) { |
Comment on lines
+19
to
+29
| /** | ||
| * Resolves a package's repository from the manifest fields that may carry it, in | ||
| * declaration order: the first candidate is the ecosystem's canonical repository | ||
| * field (`primary`), every later one a fallback (`secondary`). | ||
| * | ||
| * Fallback fields are free-form (homepage, docs, bug tracker), so they are only | ||
| * accepted on a recognized VCS host — an arbitrary `https://example.com/a/b` | ||
| * canonicalizes fine but is not a repo. The primary field keeps its historical | ||
| * behavior and accepts `other` hosts (self-hosted Gitea, cgit, SVN). | ||
| */ | ||
| export function resolveManifestRepo( |
Comment on lines
+39
to
+42
| const signal: PackageRepoSignal = candidate.signal ?? (index === 0 ? 'primary' : 'secondary') | ||
| if (signal === 'secondary' && repo.host === 'other') continue | ||
|
|
||
| return { repo, signal, field: candidate.field } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of 3 for CM-1392.
Stack: #4544 → #4545 (this PR) → #4546. Based on #4544 — review that one first; this diff is only meaningful on top of it.
Problem
Every registry writer only created a link when the ecosystem's canonical repository field parsed — npm
repository, cargorepository, rubygemssource_code_uri, NuGet<repository>, POM<scm><url>. A large share of packages leave that field empty while publishing the same repo URL inhomepage,bugs.url,projectUrl,bug_tracker_uri, or the POM<url>. Those packages got no repo link at all — invisible to criticality, blast radius, and Insights.What this does
One shared helper,
resolveManifestRepo(candidates), resolves a package's repo from an ordered candidate list. The first candidate is the ecosystem's canonical field and resolves asprimary; every later one resolves assecondary. Writers persist the returnedsignal— no writer computes a confidence value, CM-1306's function already pricessecondaryat −0.10.repository→homepage→bugs.urlHomepage→ bug trackerrepository→homepagesource_code_uri→homepage_uri→bug_tracker_urisupport.source→homepage<repository>→projectUrl<scm><url>→ POM<url>Host gate. Candidates go through the shared
canonicalizeRepoUrl. Asecondarycandidate is rejected when canonicalization yieldshost === 'other'— recognized VCS hosts only, since a fallback field is free-form andhttps://example.com/docs/introcanonicalizes into a plausibleowner/reposhape without being a repository. Theprimarycandidate keeps its historical behaviour and still acceptsother, so existing links to self-hosted Gitea, cgit, and SVN are unaffected. Packagist already applied this gate locally; it is now the shared rule.Cargo differs in mechanics only: its pipeline is set-based SQL over a dump, so
normalizeReposstages bothdeclared_repository_urlandhomepageintorepo_norm, and a newrepo_choicetable applies the same first-wins-with-host-gate rule in SQL.documentationis not staged — it is almost always docs.rs, which the host gate rejects anyway.Maven moves onto the shared canonicalizer as part of this PR; its own
parseRepoUrlused substring host matching and would otherwise have accepted hosts the other ecosystems reject.Per-run counters (
primary_field_hit,fallback_hit_by_field,no_signal) log at loop end, following the existing per-run totals pattern.Before merge
Record per-ecosystem
package_reposrow counts so the coverage uplift is measurable against the counters.Decision record
ADR-0021 — includes why fallback candidates are host-gated while primary ones are not.