Skip to content

Add immutable-field sorting to /search - #186

Open
afoote-mitre wants to merge 1 commit into
devfrom
af/178-search-sorting
Open

afoote-mitre wants to merge 1 commit into
devfrom
af/178-search-sorting

Conversation

@afoote-mitre

@afoote-mitre afoote-mitre commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add an optional sort object to POST /search, allowing clients to retrieve matching records by CVE ID or publication date in either direction. This makes recently published records accessible without changing the existing default order or result limits.

Partially addresses #178. Numeric CVE ID ordering and access beyond the existing result window remain deferred; this PR should not close the issue.

Why These Changes Were Needed

Change Why it was needed Primary files
Add a validated, case-insensitive sort object Previously every search used CVE ID ascending. Clients can now select cveId or datePublished, with asc or desc. The property is required when the object is supplied; an omitted order defaults to ascending. Unsupported properties, malformed objects, and extra keys return HTTP 400. utils/searchSort.js, utils/constants.js, utils/errors.js, routes/index.js, controllers/middleware.js
Apply sorting before pagination and semantic candidate selection Sorting only a returned page or bounded candidate subset would not establish the requested order across matching records. The OpenSearch query now applies the selected sort before those limits. Existing semantic filtering and full-record retrieval preserve the selected candidate order. utils/searchQueryBuilder.js
Use immutable fields and deterministic tie handling A mutable field such as dateUpdated can change between page requests, moving a record across an offset boundary and causing clients to miss it or receive it twice. Sorting is therefore limited to CVE ID and publication date. Publication-date sorts place missing dates last and break ties with CVE ID ascending. utils/searchSort.js, utils/searchQueryBuilder.js
Check publication-date sorting capabilities during readiness A searchable date field alone does not establish that the index supports the new sort. Readiness now also requires publication-date aggregation/doc-value capability, consistent with the existing CVE ID sorting check. utils/searchReadiness.js
Document and cover the new endpoint behavior Clients need to understand supported values, defaults, sorting examples, and how the existing pagination and semantic limits interact with the chosen order. Unit, route, OpenAPI, and Postman coverage capture these behaviors and invalid inputs. README.md, routes/swagger.js, api-docs/openapi.json, test/postman/buildCollection.js, test/postman/CVE-Search-API.postman_collection.json, associated test files

Supported Sorting Options

sort.property sort.order: "asc" sort.order: "desc"
cveId CVE ID in ascending textual order CVE ID in descending textual order
datePublished Oldest publication date first Most recent publication date first

Property and order values are case-insensitive. Omitting sort defaults to cveId ascending; supplying sort requires property, while an omitted order defaults to asc. Publication-date sorts place missing dates last in either direction and break ties with CVE ID ascending. Mutable fields such as dateUpdated are not supported.

Defaults and Missing Properties

Request input Behavior
No sort field CVE ID ascending in textual order, equivalent to {"sort":{"property":"cveId","order":"asc"}}.
"sort": {"property":"cveId"} CVE ID ascending; omitted order defaults to asc.
"sort": {"property":"datePublished"} Oldest publication date first; omitted order defaults to asc. Missing dates remain last and ties use CVE ID ascending.
"sort": {} HTTP 400 SEARCH_SORT: property is required when the sort object is supplied.
"sort": {"order":"desc"} HTTP 400 SEARCH_SORT: supplying only a direction does not default the missing property.
"sort": null, or an explicit null/empty property or order HTTP 400 SEARCH_SORT; defaults apply to omitted values, not explicitly invalid values.

CVE ID sorting is currently alphabetical/textual, not numeric: in ascending order, CVE-2024-10000 precedes CVE-2024-9999. Publication-date sorting should be used for publication recency rather than inferring recency from CVE sequence numbers.

Why Mutable Sort Fields Are Excluded

Separate page requests do not share a snapshot. If a client reads page 1 using dateUpdated descending and a record from page 2 is updated before the next request, that record can move onto the already-read page 1 and be missed. Records shifted in the other direction can be returned twice. An immutable CVE ID tie-breaker does not prevent movement when the primary sort field changes.

For that reason, this PR accepts only immutable cveId and datePublished sort fields and rejects mutable fields such as dateUpdated. This reduces sort-key-driven movement but does not eliminate shifts caused by new records, deletions, or changed filter membership. Reconsidering mutable sorting requires a pagination design that preserves a shared snapshot across page requests. The internal, request-scoped PIT traversal in #156 does not provide that cross-request guarantee.

Why Numeric CVE ID Sorting Is Deferred

Numeric CVE ID ordering must be applied before OpenSearch pagination and candidate limits to provide globally correct results. Sorting only the records retrieved by the API would reorder that page or bounded candidate subset, but could not account for matching records that OpenSearch did not return.

We are deferring numeric ordering until we have an indexed normalized sort key that orders IDs by numeric year and sequence. Once the key is available, the API can use it for sorting before pagination and candidate selection. Script-based sorting is not being pursued as part of this change because it would add request-time computation and require a separate performance assessment.

Until then, cveId continues to use textual ordering; datePublished is available for publication-recency ordering.

Example

Retrieve the most recently published critical matches first:

{
  "cvssV3_1Severity": "CRITICAL",
  "sort": {
    "property": "datePublished",
    "order": "desc"
  }
}

@afoote-mitre
afoote-mitre marked this pull request as draft October 15, 2026 13:06
@afoote-mitre
afoote-mitre marked this pull request as ready for review October 15, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant