diff --git a/docs/decisions/0027-resolve-a-reference-across-several-targets.md b/docs/decisions/0027-resolve-a-reference-across-several-targets.md new file mode 100644 index 00000000..24955fce --- /dev/null +++ b/docs/decisions/0027-resolve-a-reference-across-several-targets.md @@ -0,0 +1,201 @@ +# 27. Resolve a reference across several targets + +Date: 2026-09-10 + +## Status + +Accepted + +Extends [ADR 20 (Resolve a reference’s fields from the target’s own collection)](./0020-resolve-a-references-fields-from-the-targets-own-collection.md), +whose `lookup` names one target, and +[ADR 24 (Carry data on a reference edge)](./0024-carry-data-on-a-reference-edge.md), +whose `local` lookup stores a referent through that one target’s declaration. +Amends [ADR 21 (Type a filter by what its field keys on)](./0021-type-a-filter-by-what-its-field-keys-on.md), +which rejected two of the things decided here. Leaves +[ADR 19 (Filter across collections through declared joins)](./0019-filter-across-collections-through-declared-joins.md) +untouched, and states what it cannot reach. + +## Context + +SCHEMA-AP-NDE ranges `CreativeWork.creator` over `Person` **or** +`Organization`. A `lookup` names one `target`, so a deployment declared +`target: 'Person'` and got exactly half of the field: every person labelled and +typed `PersonReference`, every organization unlabelled and _also_ typed +`PersonReference`. Measured on the reporting deployment’s facet, the split was +exact – seven labelled values, all persons; three null, all organizations. A +consumer routing a click on `creator.id` to a person page followed it for an +organization and landed nowhere, with nothing in the response to warn them. + +Two things have moved since the issue was filed. First, the declaration now +sits **inside an edge**: `creator` is an inline `CreatorRole` carrying the role +a publisher wrapped the agent in, and the polymorphic lookup is the edge’s +nested `creator`, reached through `identity`. Second, `local: true` stores +what the work states about its creator, so an organization is no longer +_unlabelled_ – the referring document names it. That narrowed the defect to +two things a label mechanism can never fix: the facet buckets, still labelled +through the single target, and `__typename`, still claiming `PersonReference`. + +Retargeting is not a workaround. Pointing `creator` at the type most referents +are co-typed with (`Term`, in that corpus) took the facet from seven labelled +values to five. No single target covers a range the profile itself declares as +a disjunction. + +ADR 21 had rejected both a list of targets per reference and an output-side +interface, on the grounds that coarse discovery already answered the input-side +question and that a shared fragment was “buildable but unearned” on the output +side. Both grounds were about convenience. A type claim that is false for half +the values is not a convenience problem. + +## Decision + +### A lookup names several targets, in order of precedence + +`target` – and an `idOnly`’s `labelSource` – accepts a `string` or a +`readonly string[]`. One reading serves both: a single name is the list of +one, and no consumer branches on which it was given. Every named type must be a +Root Type with a label field; a name declared twice is rejected; `joinable` is +refused when there is more than one (below). + +**Declaration order is precedence.** A referent two collections hold – two +Root Types whose `class` selections overlap – belongs to the first target +declared. So does a stored referent whose `rdf:type` matches none of them. The +rule is per value: one field serves persons and organizations side by side. +Precedence is declaration order rather than an option because it is the only +lever an author already has, and because overlap is the author’s own choice – +two Root Types selecting overlapping classes is legal for facets and joins +already, and silently choosing is what the rest of the system does. + +### Each referent is read through the target it belongs to + +Three declarations flow through _naming the target_: the label field, the +document key ([ADR 22](./0022-key-a-root-type-on-a-declared-field.md)) and the +facet policy. The reporting deployment’s two targets disagree on the one that +changes what is stored – `Person` is keyed on an authority IRI, `Organization` +on nothing – so requiring the targets to agree would reject the schema the +issue came from. + +Each referent is therefore treated as a document of whichever target it +matches: re-keyed through that target’s key, admitted to a facet by that +target’s policy, labelled from that target’s collection. Nothing changes for +the targets themselves; `Person` keeps its key without knowing it appears in a +polymorphic lookup. The interface an author learns is one sentence longer than +before, which is the point of doing the dispatch per entry rather than +demanding alike targets. + +### A stored referent carries a discriminator + +A `local` lookup stores what the referring document states, and in the +reporting corpus seven creators in eight are never identified. No collection +will ever answer for them, so the collections cannot be the only evidence of +kind. The extraction reads the referent’s `rdf:type` – one `OPTIONAL` triple +per referent, emitted as `rdf:type` itself so framing carries it as `@type` – +and the projection matches it against each target’s `class` in declaration +order. The entry is projected through the matching declaration and records +which under the reserved physical name `_target`, which no type may declare. + +That discriminator is also what re-keys the entry and admits it to a facet, so +the three readings cannot disagree. An engine adapter reads it back when no +collection answers, so an unidentified organization is served as an +organization. A single-target lookup stores none: there is nothing to tell +apart. + +The nested object an engine declares is the union of the targets’ fields, so +two targets declaring one field must declare it alike – same kind, same arity, +same Roles, and for a reference the same strategy and referent – since every +one of those decides the field’s physical shape, and a shape that depended on +which target was listed first would store one target’s referents wrongly. +`searchSchema` rejects the pair otherwise; a field only one target declares is +fine, the other simply never fills it. This holds for a `local` lookup today +and for every lookup once +[#818](https://github.com/ldelements/lde/issues/818) drops the flag, which is +why the multi-target projection lives in the shared nesting body rather than +behind a `local`-specific switch. + +### The surface serves an interface, and a filter named for the set + +A GraphQL surface serves a lookup naming several targets as an **interface** +named for the set in declaration order – `PersonOrOrganizationReference` – +implemented by each target’s own `‹Target›Reference`. Two fields naming the +same targets share one interface, exactly as two lookups on one target share +one reference type ([ADR 20](./0020-resolve-a-references-fields-from-the-targets-own-collection.md)). +The name is derived, not declared: no new option, and the same mechanism that +derives `‹Target›Reference`. + +The interface carries only what is true of every referent: `id`, nullable if +any member’s is, and every `output` field all targets declare alike, with the +weakest nullability any of them keeps – an implementation may promise more than +its interface, never less. Per-type fields stay on the members, reached +through an inline fragment. `__typename` resolves per referent to the target +whose collection answered for it, or, for a stored referent no collection +answers for, to the target its discriminator names. The port marks each nested +document with that target under a symbol key, so the mark can never collide +with a declared field. A bare IRI no collection holds is the one referent with +no kind to report; an interface must still resolve to some object type, so it +is served as the first target declared. + +An interface rather than a union, because union members share no fields, so +even `creator { id name }` would need a fragment and every existing consumer +would break. An interface keeps that selection working unchanged. A `type` +scalar on a flat reference was rejected in the issue for a reason that still +holds: the moment a reference carries more than a label, per-type fields have +nowhere to live. + +On the input side, the field’s filter is named for the set too – +`PersonOrOrganizationFilter { in: [IRI!] }` – rather than for the first target, +which would lie about half the values, or `IRIFilter`, which would claim the +ids belong to no collection here. ADR 21’s coarse discovery still finds it, +since its element type is `IRI`. Refined discovery, which resolves through one +target’s own `id`, does not reach it; ADR 21 already called refined “a +precision tool, not a completeness claim” and named polymorphic ranges as +where it under-reports. This makes the under-report visible in the name. + +### No join + +A reference naming several targets refuses `joinable`. An engine reference +names one collection, and ids that live in several have no single collection +to reference. Its labels, facets and id filters all work from the referring +document. What stays out of reach is a **join predicate** – a condition on the +referent’s own fields, “works whose creator died before 1900”. The issue lists +three ways to recover one (a shared collection of the common fields; the same +collection denormalised to the union of fields with a discriminator; one +reference field per target disjoined at query time), none needed for what this +record delivers and each costing a second copy of every agent or two +unverified engine behaviours. Deferred, to +[#845](https://github.com/ldelements/lde/issues/845) and a record of its own. + +## Consequences + +- A field whose referent may be of several kinds labels every value, types + every value truthfully, and serves each kind’s own fields through a fragment. + Facet buckets are labelled from whichever collection holds the value. +- **Breaking** for a schema-level reader: `labelSourceNameOf` and + `labelTargetNameOf` become `labelSourceNamesOf` and `labelTargetNamesOf`, + returning a list; `localLookupTypeOf` becomes `localLookupTargetsOf`; + `inheritedFacetKeys` becomes `inheritedFacetPolicies`, keyed by target. A + declaration naming one target is unchanged, stores what it stored, and emits + the GraphQL it emitted. +- A multi-target lookup costs one query per target collection per level where + a single-target one costs one – still one round-trip per level, the queries + running concurrently – and asks every collection even for a selection + reduced to `id`, because the answer is what types the referent. Its stored + entries cost one short string each, and its extraction one `OPTIONAL` triple + per referent plus every keyed target’s key hop. +- The framing depth of a type nesting a multi-target lookup is the furthest + any target reaches: the referent is framed once, and the frame has to hold + whichever declaration it turns out to match. +- `_target` joins `id`, `and` and `or` as a name no type may declare. +- Label precedence is settled **per type**, in the order its fields first name + the collections: a page’s labels are one map keyed by IRI, so two fields of + one type ordering the same targets differently cannot each get their own + answer for an IRI both collections hold. Before, the same overlap was + labelled by whichever collection answered last – now it is deterministic + and documented, and the projected lookup (which does resolve per field) + can disagree with a bucket label only in that corner. +- A declared type may not be named as the joined name of a target set + (`PersonOrOrganization` beside a lookup over `Person` and `Organization`): + the interface and the type’s filter would share a name, so the GraphQL + surface refuses the schema, naming both. +- Amends ADR 21: a list of targets per reference and an output-side interface + are both accepted, for a reason ADR 21 did not weigh – a type claim that is + false for half the values. Its input-side reasoning stands: an interface is + output-only, and the `IRI` scalar remains the input-side abstraction. diff --git a/docs/reference/search-api-graphql.md b/docs/reference/search-api-graphql.md index be036f31..85986c4d 100644 --- a/docs/reference/search-api-graphql.md +++ b/docs/reference/search-api-graphql.md @@ -167,12 +167,23 @@ editor. `id`, since a referent needs no identity – so a client selects a nested object’s fields directly and renders one referent at a time. A `local` lookup gets the nullable `id` too, and for the same reason: it carries what the - document states about an endpoint whether or not the endpoint is identified; scalars/booleans + document states about an endpoint whether or not the endpoint is identified. + A lookup naming [several targets](./search#a-referent-of-several-kinds) is + served as an **interface** named for the set (`creator` over `Person` and + `Organization` → `PersonOrOrganizationReference`), implemented by each + target’s own reference type; it carries `id` plus every field the targets + declare alike, with the weakest nullability any of them keeps, and resolves + per referent to the type whose collection answered for it, or that a stored + referent was projected through – so `__typename` is true per referent, and + `... on PersonReference { birthDate }` reaches what only a person has. An + IRI no collection holds has no kind to report and is served as the first + target declared, since an interface must resolve to some type; scalars/booleans per kind; `date` → ISO 8601 string; nullability from `required` / `array` / `kind`. - **`where`** one input per `filterable` field, typed by what the field keys on: a `keyword` holds literals (`KeywordFilter`), a `reference` holds identity - (`‹Target›Filter`, or `IRIFilter` when it names no target), and the numeric + (`‹Target›Filter`, or `IRIFilter` when it names no target, or a filter named + for the set – `PersonOrOrganizationFilter` – when it names several), and the numeric kinds take `IntRange` / `FloatRange` / `DateRange`, a `boolean` a plain `Boolean`. Every type also gets **`id: ‹Type›Filter`** – the document’s IRI, declared by no type and filterable on all of them @@ -328,7 +339,12 @@ when a shorter, higher-precision list is what you want. **Known limit**: the refined strategy resolves only when the target is itself a root collection. A `ref` to a type no collection serves has no `‹Type›Where.id` to match against, so fall back to the coarse strategy – which is also the right -one for a reference declared with no target at all (`IRIFilter`). +one for a reference declared with no target at all (`IRIFilter`), and for one +declared with [several](./search#a-referent-of-several-kinds): its filter is +named for the set (`PersonOrOrganizationFilter`) rather than for any one +member’s `id`, so resolving through `PersonWhere.id` does not reach it, by +design – the name is truthful about what the field admits. Its `in` element is +still `IRI`, so coarse discovery finds it. Two further notes. `IRI` is wire-compatible with `String`, but GraphQL checks variable usage **nominally**, so a variable must be declared `[IRI!]` rather than diff --git a/docs/reference/search-pipeline.md b/docs/reference/search-pipeline.md index 6260b907..9988e6d6 100644 --- a/docs/reference/search-pipeline.md +++ b/docs/reference/search-pipeline.md @@ -424,7 +424,12 @@ guarantees one output triple per genuine value: because a `local` lookup stores its endpoint by id whether or not the document says anything about it – conjoining would drop the id along with the absent fields. It subsumes the key hop below, the key field being one of the target's - own. + own. A lookup naming [several targets](./search#a-referent-of-several-kinds) + expands through every one of them inside the same `OPTIONAL` union – a + referent is of one kind, and the other targets' branches bind nothing for it – + and reads the referent’s `rdf:type` in an `OPTIONAL` of its own, emitted as + `rdf:type` so framing carries it as `@type`: that is what the projection + matches against each target’s `class` to tell which kind the referent is. - **A key hop stays inside its branch.** A reference naming a target that declares a [document key](./search#document-key) gets its branch extended with an `OPTIONAL` hop reading the referent’s key field, emitted under the target’s diff --git a/docs/reference/search-typesense.md b/docs/reference/search-typesense.md index 1dc62891..07baccc2 100644 --- a/docs/reference/search-typesense.md +++ b/docs/reference/search-typesense.md @@ -200,15 +200,25 @@ down. - runs the search; - resolves reference (and reference-facet) labels **per reference field** from the collection of the `SearchType` its `labelSource` names – all - sources bundled into a single lookup. A reference without a `labelSource` - stays id-only. With `labelCacheTtlMs` set, each label-source collection is - instead loaded once into an in-memory cache; + sources bundled into a single lookup. A reference naming + [several targets](./search#a-referent-of-several-kinds) sends its IRIs to + every one of their collections in the same round-trip. An IRI two of them + hold is labelled by the collection the type’s fields name first – label + precedence is settled per type, not per field, since one page’s labels are + one map keyed by IRI. A reference without a + `labelSource` stays id-only. With `labelCacheTtlMs` set, each label-source + collection is instead loaded once into an in-memory cache; - reconstructs the logical `SearchResult` (`parseSearchResponse`) – language maps, labelled references, labelled facet buckets, and one nested Search Document per referent of a surfaced inline reference (each referent’s values grouped, `id` only where the referent had one). Nesting is rebuilt here, below every API surface, so a second surface inherits it rather than - reimplementing it. + reimplementing it. A projected lookup naming several targets fetches from + each target’s collection concurrently, still one round-trip per level, and + marks each nested document with the target whose collection answered for it + (`NESTED_DOCUMENT_TYPE`, a symbol key, so it never surfaces as a field); a + stored referent no collection answers for is read through the target its + stored discriminator names. A label source is just another `SearchType` in the schema (with an `output`, `searchable` text field under its diff --git a/docs/reference/search.md b/docs/reference/search.md index fff09ff6..28ed9c4a 100644 --- a/docs/reference/search.md +++ b/docs/reference/search.md @@ -338,7 +338,9 @@ fields are read from, and the name its emitted type derives from (GraphQL: named per query rather than per declaration, by a [projection](#projecting-what-a-lookup-carries); asked for nothing in particular, it carries the target's label. Only an `inline` reference’s -`ref.typeName` resolves to a declared Reference Type. +`ref.typeName` resolves to a declared Reference Type. Where the referent may be +of several kinds, `target` lists them – see +[A referent of several kinds](#a-referent-of-several-kinds). Note what this makes true of `kind`: **a `reference` holds identity, a `keyword` holds a literal.** A field over an IRI-valued property is a `reference` whatever @@ -603,6 +605,61 @@ See [ADR 26](../decisions/0026-fan-out-a-qualified-edge-into-one-entry-per-tuple Out of scope for now: faceting an edge's own values, which the current engine cannot serve correctly. +#### A referent of several kinds + +A profile often ranges one property over several classes – SCHEMA-AP-NDE’s +`creator` is a `Person` _or_ an `Organization`. A lookup naming one target can +label and type only half of such a field: every organization comes back +unlabelled, and typed as a person. So a `lookup` (and an `idOnly`’s +`labelSource`) may name **several** targets: + +```ts +{ + name: 'creator', + kind: 'reference', + path: `${SCHEMA}creator`, + output: true, + ref: { strategy: 'lookup', target: ['Person', 'Organization'], local: true }, +} +``` + +Each referent then resolves against every named collection and is read through +the declaration of whichever one holds it: its label, its own fields, its +[key](#document-key) and its [facet policy](#facet-policy) are those of _its_ +target, per value. A GraphQL surface serves the field as an interface named for +the set (`PersonOrOrganizationReference`), implemented by each target’s own +reference type, so `__typename` says which kind each referent is and an inline +fragment reaches the fields only one kind has. The interface itself carries +`id` and whatever every target declares alike. + +**Order is precedence.** A referent two collections hold – two Root Types +whose `class` selections overlap – belongs to the first target listed, and so +does a stored referent whose `rdf:type` matches none of them. List the most +specific type first. + +**A stored referent says which kind it is.** A [`local`](#data-on-the-edge) +lookup stores what the referring document states, and most such referents are +never identified, so no collection can ever say what they are. The extraction +therefore reads the referent’s `rdf:type` and the projection matches it against +each target’s `class`, projecting the entry through the matching declaration +and recording which under a reserved physical name (`_target`, which no type +may declare). A stored referent that no collection answers for – unidentified, +or identified but not indexed – is then still served as the kind it was stated +to be. The nested object declares the union of the targets’ fields, so two +targets that both declare a field must declare it alike – same `kind`, same +arity, same Roles, and for a reference the same strategy and referent; +`searchSchema` rejects the pair otherwise. + +**No join.** A reference naming several targets can never be +[`joinable`](#filtering-across-collections): an engine reference names one +collection, and ids that live in several have no single collection to +reference. Its labels, facets and id filters all work from the referring +document; a condition on the referent’s own fields (“works whose creator died +before 1900”) does not, and needs a representation of its own +([#845](https://github.com/ldelements/lde/issues/845) lists three). + +See [ADR 27](../decisions/0027-resolve-a-reference-across-several-targets). + #### Naming the label field That word is `label` by default, but a type may name its own display field with diff --git a/packages/search-api-graphql/coverage/coverage-summary.json b/packages/search-api-graphql/coverage/coverage-summary.json new file mode 100644 index 00000000..b4972113 --- /dev/null +++ b/packages/search-api-graphql/coverage/coverage-summary.json @@ -0,0 +1,45 @@ +{ + "total": { + "lines": { "total": 457, "covered": 457, "skipped": 0, "pct": 100 }, + "statements": { "total": 468, "covered": 468, "skipped": 0, "pct": 100 }, + "functions": { "total": 119, "covered": 119, "skipped": 0, "pct": 100 }, + "branches": { "total": 330, "covered": 321, "skipped": 0, "pct": 97.27 }, + "branchesTrue": { "total": 0, "covered": 0, "skipped": 0, "pct": "Unknown" } + }, + "/Users/david/src/lde-issue-717-polymorphic-lookup/packages/search-api-graphql/src/build-schema.ts": { + "lines": { "total": 328, "covered": 328, "skipped": 0, "pct": 100 }, + "functions": { "total": 83, "covered": 83, "skipped": 0, "pct": 100 }, + "statements": { "total": 331, "covered": 331, "skipped": 0, "pct": 100 }, + "branches": { "total": 235, "covered": 229, "skipped": 0, "pct": 97.44 } + }, + "/Users/david/src/lde-issue-717-polymorphic-lookup/packages/search-api-graphql/src/facet-batch.ts": { + "lines": { "total": 37, "covered": 37, "skipped": 0, "pct": 100 }, + "functions": { "total": 14, "covered": 14, "skipped": 0, "pct": 100 }, + "statements": { "total": 42, "covered": 42, "skipped": 0, "pct": 100 }, + "branches": { "total": 20, "covered": 20, "skipped": 0, "pct": 100 } + }, + "/Users/david/src/lde-issue-717-polymorphic-lookup/packages/search-api-graphql/src/handler.ts": { + "lines": { "total": 16, "covered": 16, "skipped": 0, "pct": 100 }, + "functions": { "total": 5, "covered": 5, "skipped": 0, "pct": 100 }, + "statements": { "total": 17, "covered": 17, "skipped": 0, "pct": 100 }, + "branches": { "total": 19, "covered": 19, "skipped": 0, "pct": 100 } + }, + "/Users/david/src/lde-issue-717-polymorphic-lookup/packages/search-api-graphql/src/language.ts": { + "lines": { "total": 11, "covered": 11, "skipped": 0, "pct": 100 }, + "functions": { "total": 4, "covered": 4, "skipped": 0, "pct": 100 }, + "statements": { "total": 12, "covered": 12, "skipped": 0, "pct": 100 }, + "branches": { "total": 8, "covered": 7, "skipped": 0, "pct": 87.5 } + }, + "/Users/david/src/lde-issue-717-polymorphic-lookup/packages/search-api-graphql/src/print-sdl.ts": { + "lines": { "total": 11, "covered": 11, "skipped": 0, "pct": 100 }, + "functions": { "total": 2, "covered": 2, "skipped": 0, "pct": 100 }, + "statements": { "total": 11, "covered": 11, "skipped": 0, "pct": 100 }, + "branches": { "total": 7, "covered": 7, "skipped": 0, "pct": 100 } + }, + "/Users/david/src/lde-issue-717-polymorphic-lookup/packages/search-api-graphql/src/projection.ts": { + "lines": { "total": 54, "covered": 54, "skipped": 0, "pct": 100 }, + "functions": { "total": 11, "covered": 11, "skipped": 0, "pct": 100 }, + "statements": { "total": 55, "covered": 55, "skipped": 0, "pct": 100 }, + "branches": { "total": 41, "covered": 39, "skipped": 0, "pct": 95.12 } + } +} diff --git a/packages/search-api-graphql/src/build-schema.ts b/packages/search-api-graphql/src/build-schema.ts index 9dae0ff8..9f8738c9 100644 --- a/packages/search-api-graphql/src/build-schema.ts +++ b/packages/search-api-graphql/src/build-schema.ts @@ -5,6 +5,7 @@ import { GraphQLFloat, GraphQLInputObjectType, GraphQLInt, + GraphQLInterfaceType, GraphQLList, GraphQLNonNull, GraphQLObjectType, @@ -19,9 +20,11 @@ import { type GraphQLOutputType, } from 'graphql'; import { + NESTED_DOCUMENT_TYPE, type Criterion, type Filter, type LocalizedValue, + type NestedDocument, type RootType, type SearchEngine, type ReferenceField, @@ -35,8 +38,10 @@ import { AND_KEY, facetableFields, filterableFields, - labelTargetNameOf, - localLookupTypeOf, + fieldNamed, + labelSourceNamesOf, + labelTargetNamesOf, + localLookupTargetsOf, referenceFields, filterOn, filterOperatorFor, @@ -407,6 +412,15 @@ export function buildGraphQLSchema( // collide: searchSchema resolves its typeName to a declared Reference Type // and rejects duplicate names schema-wide. const referenceTypes = new Map(); + /** + * The interface a lookup naming several targets is served as, by the + * targets’ joined name – a namespace of its own, apart from the object + * types above, so an inline Reference Type that happens to be named + * `PersonOrOrganization` can never be mistaken for it. Beside each, the + * targets behind it. + */ + const interfaceTypes = new Map(); + const interfaceMembers = new Map(); /** * The lookup targets whose emitted type must carry a NULLABLE `id`: those * some field reaches through a {@link ReferenceStrategy.local local} lookup, @@ -433,12 +447,14 @@ export function buildGraphQLSchema( walked.add(searchType.name); for (const field of referenceFields(searchType)) { if (field.ref?.strategy === 'lookup' && field.ref.local === true) { - nullableIdTargets.add(field.ref.target); + for (const target of labelSourceNamesOf(field)) { + nullableIdTargets.add(target); + } } - const nested = - nestedReferenceType(schema, field) ?? - localLookupTypeOf(field, schema); - if (nested !== undefined) { + const referenceType = nestedReferenceType(schema, field); + for (const nested of referenceType === undefined + ? localLookupTargetsOf(field, schema) + : [referenceType]) { collect(nested); } } @@ -478,14 +494,26 @@ export function buildGraphQLSchema( const nestedFilters = new Map(); const referenceFilters = new Map(); - /** The name a reference’s emitted type is keyed under: a `lookup`’s target, - * an `inline`’s reference type, an `idOnly`’s declared name (which names a - * filter’s target, never an object type – an `idOnly` surfaces as its bare - * IRI). */ - function referencedTypeName( - ref: NonNullable, - ): string | undefined { - return ref.strategy === 'lookup' ? ref.target : ref.typeName; + /** The name a reference’s emitted type is keyed under: a `lookup`’s target + * – or its targets joined with `Or` where it names several, the one name + * its interface and its filter share – an `inline`’s reference type, an + * `idOnly`’s declared name (which names a filter’s target, never an object + * type – an `idOnly` surfaces as its bare IRI). */ + function referencedTypeName(field: ReferenceField): string | undefined { + const ref = field.ref; + if (ref === undefined) { + return undefined; + } + return ref.strategy === 'lookup' + ? polymorphicName(labelSourceNamesOf(field)) + : ref.typeName; + } + + /** The one name several targets share: `PersonOrOrganization`. Derived, in + * declaration order, so two fields naming the same targets meet one type – + * exactly as two lookups on one target share `‹Target›Reference`. */ + function polymorphicName(targets: readonly string[]): string { + return targets.join('Or'); } /** @@ -505,13 +533,146 @@ export function buildGraphQLSchema( // as the IRI itself rather than as an object – there is no type to // register, and no name needed to register one under. field.ref.strategy === 'idOnly' || - referencedTypeName(field.ref) === undefined + referencedTypeName(field) === undefined ) { return; } + // A lookup naming several targets is served as an INTERFACE over the + // targets’ own reference types, each registered as it would be for a + // lookup naming it alone – so a consumer selecting only what every target + // carries (`id`, the label) needs no fragment, and `__typename` says + // which target a referent came from. + const targets = labelSourceNamesOf(field); + if (field.ref.strategy === 'lookup' && targets.length > 1) { + for (const target of targets) { + registerObjectType(target, field, owner); + } + registerInterfaceType(targets, field, owner); + return; + } // Guaranteed by the guard above: idOnly is out, and the other two strategies // each name their referent. - const typeName = referencedTypeName(field.ref) as string; + registerObjectType(referencedTypeName(field) as string, field, owner); + } + + /** + * The interface a lookup naming several targets is served as: named for the + * targets (`PersonOrOrganizationReference`), carrying what every one of them + * carries – `id`, and every `output` field they all declare alike – and + * resolved per referent to the target whose collection answered for it, + * which the port marks on each nested document. + * + * A field the targets share is offered with the weakest nullability any of + * them keeps: an implementation may promise more than its interface, never + * less. + */ + function registerInterfaceType( + targets: readonly string[], + field: SearchField, + owner: SearchType, + ): void { + const typeName = polymorphicName(targets); + if (interfaceTypes.has(typeName)) { + return; + } + const graphQLName = `${typeName}Reference`; + // The joined name is derived, so a declared type may spell the same + // thing – `PersonOrOrganization` as a Reference Type – and its filter + // would then share a name with this interface’s. Refused, naming both. + if (takenTypeNames.has(graphQLName) || takenTypeNames.has(typeName)) { + throw new Error( + `Reference type “${typeName}” (field “${field.name}” of “${owner.name}”) would be served as “${graphQLName}”, which collides with another type name; rename one.`, + ); + } + takenTypeNames.add(graphQLName); + // Reserve the joined name too, so a Reference Type spelling it that + // registers LATER is refused with the same message. + takenTypeNames.add(typeName); + const members = targets.map( + (target) => rootTypesByName.get(target) as RootType, + ); + const nullableId = members.some((member) => + nullableIdTargets.has(member.name), + ); + interfaceMembers.set(typeName, targets); + interfaceTypes.set( + typeName, + new GraphQLInterfaceType({ + name: graphQLName, + description: `A reference to a ${targets.join(' or a ')}; select \`__typename\` to tell which, and the fields of one through an inline fragment on its type.`, + resolveType: (source: Source) => { + const resolved = (source as NestedDocument)[NESTED_DOCUMENT_TYPE]; + // A document the port did not type – a referent no collection + // holds, or one built by hand – is read as the first target + // declared: an interface must resolve to SOME object type, and the + // first target is the precedence every other reading of several + // targets applies. + return `${ + typeof resolved === 'string' && targets.includes(resolved) + ? resolved + : targets[0] + }Reference`; + }, + fields: (): Record< + string, + GraphQLFieldConfig + > => ({ + id: { + type: nullableId ? iriScalar : new GraphQLNonNull(iriScalar), + }, + ...Object.fromEntries( + outputFields(members[0]) + .map((declared) => sharedField(declared, members, nullableId)) + .filter((shared): shared is SearchField => shared !== undefined) + .map((shared) => [shared.name, outputFieldConfig(shared)]), + ), + }), + }), + ); + } + + /** + * A field every member declares alike – same kind, same arity – with the + * nullability relaxed unless every member requires it. `undefined` where the + * members do not all carry it: the interface promises only what holds of + * every referent. + */ + function sharedField( + declared: SearchField, + members: readonly RootType[], + nullableId: boolean, + ): SearchField | undefined { + const counterparts = members.map((member) => + fieldNamed(member, declared.name), + ); + if ( + counterparts.some( + (counterpart) => + counterpart === undefined || + counterpart.output !== true || + counterpart.kind !== declared.kind || + (counterpart.array === true) !== (declared.array === true), + ) + ) { + return undefined; + } + // A shared reference field is served as one emitted type on every + // member: `searchSchema` holds the targets to the same strategy and the + // same referent for a name they share, so nothing is checked here. + const required = + !nullableId && + counterparts.every((counterpart) => counterpart?.required === true); + return { ...declared, required } as SearchField; + } + + /** Register the object type one referenced shape is served as, under + * `typeName`: a lookup’s target root type, or an inline reference’s + * Reference Type. */ + function registerObjectType( + typeName: string, + field: ReferenceField, + owner: SearchType, + ): void { if (referenceTypes.has(typeName)) { // Fields sharing a referent share one emitted type, and cannot disagree // about its FIELDS: a lookup's come from the target that names the type, @@ -536,14 +697,22 @@ export function buildGraphQLSchema( // searchSchema rejects a lookup whose target it cannot find, and an inline // reference that resolves to no Reference Type. const nested = ( - field.ref.strategy === 'lookup' - ? rootTypesByName.get(field.ref.target) + field.ref?.strategy === 'lookup' + ? rootTypesByName.get(typeName) : nestedReferenceType(schema, field) ) as SearchType; + // Every interface this type will implement: one per lookup naming this + // target among several. Resolved lazily, so registration order between + // the interface and its members does not matter. + const implemented = (): GraphQLInterfaceType[] => + [...interfaceMembers] + .filter(([, members]) => members.includes(typeName)) + .map(([key]) => interfaceTypes.get(key) as GraphQLInterfaceType); referenceTypes.set( typeName, new GraphQLObjectType({ name: graphQLName, + interfaces: implemented, // A thunk, so a Reference Type nesting another one resolves whatever // the registration order is (the graph is acyclic by searchSchema). fields: (): Record< @@ -655,10 +824,13 @@ export function buildGraphQLSchema( resolve: (source) => iriOf(source[field.name]) ?? null, }; } - const referenceType = referenceTypes.get( - (field.ref === undefined - ? undefined - : referencedTypeName(field.ref)) ?? '', + const typeName = referencedTypeName(field) ?? ''; + // A lookup naming several targets is served as their interface. + const referenceType = ( + field.ref?.strategy === 'lookup' && + labelSourceNamesOf(field).length > 1 + ? interfaceTypes.get(typeName) + : referenceTypes.get(typeName) )!; return field.array === true ? { @@ -737,14 +909,14 @@ export function buildGraphQLSchema( if (existing !== undefined) { return existing; } - const identityTarget = labelTargetNameOf(field, schema); + const identityTargets = labelTargetNamesOf(field, schema); const nestedWhere = nestedWhereInputFor(referenceType); const created = new GraphQLInputObjectType({ name: `${referenceType.name}Filter`, description: `A condition on ${referenceType.name}: the ids its entries reference, or a condition on one entry.`, isOneOf: true, fields: () => ({ - ...(identityTarget !== undefined && { + ...(identityTargets.length > 0 && { in: { type: new GraphQLList(new GraphQLNonNull(iriScalar)) }, }), where: { type: nestedWhere }, @@ -792,9 +964,12 @@ export function buildGraphQLSchema( return keywordFilter; } // A lookup keys on its `target`, an idOnly/inline on its `typeName`: - // one reading, so a filter is typed by whatever names the referent. - const target = - field.ref === undefined ? undefined : referencedTypeName(field.ref); + // one reading, so a filter is typed by whatever names the referent. A + // lookup naming several targets keys on all of them, so its filter + // is named for the set (`PersonOrOrganizationFilter`): truthful, and + // still an `IRI` list, so coarse discovery finds it – refined + // discovery, which resolves through one target’s own `id`, does not. + const target = referencedTypeName(field); return target === undefined ? iriFilter : targetFilter(target); } case 'range': @@ -1153,6 +1328,10 @@ export function buildGraphQLSchema( return new GraphQLSchema({ query: new GraphQLObjectType({ name: 'Query', fields: queryFields }), + // An interface's implementations are reachable only through it, so they + // are listed explicitly; every other reference type is reachable already + // and listing it twice is harmless. + types: [...referenceTypes.values(), ...interfaceTypes.values()], }); } diff --git a/packages/search-api-graphql/src/projection.ts b/packages/search-api-graphql/src/projection.ts index 6558f967..858cb80b 100644 --- a/packages/search-api-graphql/src/projection.ts +++ b/packages/search-api-graphql/src/projection.ts @@ -13,7 +13,7 @@ import { import { fieldNamed, nestedReferenceType, - rootTypeNamed, + referencedTargetsOf, } from '@lde/search/adapter'; type Fragments = Readonly>; @@ -164,34 +164,40 @@ function fromSelections( if (field.kind !== 'reference' || field.ref?.strategy !== 'lookup') { continue; } - // The target's own declaration decides what its selections mean, so the - // level below is read against it rather than against this type. + // The targets' own declarations decide what the selections mean, so the + // level below is read against them rather than against this type. // `searchSchema` rejects a lookup whose target it cannot resolve, so a // schema always resolves its own – which is why this reads the schema - // rather than taking a resolver that could answer nothing. - const target = rootTypeNamed(schema, field.ref.target) as SearchType; - // Only what the target actually serves. A selection carries more than + // rather than taking a resolver that could answer nothing. A lookup + // naming several targets is served as an interface, and a client + // reaches one target’s own fields through an inline fragment on its + // type; the fragment’s fields are read like any other, and each target + // takes the ones it serves. + const targets = referencedTargetsOf(field, schema); + // Only what a target actually serves. A selection carries more than // that: `id` is on the referring document already, and every GraphQL // client worth the name injects `__typename` into every selection set – // asking the engine for either would fail the query at the port's guard. const wanted = fieldsOf(selected.selectionSet, fragments) .map((node) => node.name.value) - .filter((name) => servesField(target, name)); + .filter((name) => targets.some((target) => servesField(target, name))); const entry = (projection[selected.name.value] ??= {}); entry.fields = [...new Set([...(entry.fields ?? []), ...wanted])]; if (selected.selectionSet !== undefined) { - const below = fromSelections( - [selected.selectionSet], - target, - fragments, - schema, - ); - if (Object.keys(below).length > 0) { - // Merged level by level, not key by key: one lookup selected twice – - // two fragments each spreading it – must union what each asked for, - // or the second selection silently replaces the first and a field - // the client asked for is never fetched. - entry.resolve = mergeProjections(entry.resolve, below); + for (const target of targets) { + const below = fromSelections( + [selected.selectionSet], + target, + fragments, + schema, + ); + if (Object.keys(below).length > 0) { + // Merged level by level, not key by key: one lookup selected twice + // – two fragments each spreading it – must union what each asked + // for, or the second selection silently replaces the first and a + // field the client asked for is never fetched. + entry.resolve = mergeProjections(entry.resolve, below); + } } } } diff --git a/packages/search-api-graphql/test/polymorphic-lookup.test.ts b/packages/search-api-graphql/test/polymorphic-lookup.test.ts new file mode 100644 index 00000000..4a9e5d10 --- /dev/null +++ b/packages/search-api-graphql/test/polymorphic-lookup.test.ts @@ -0,0 +1,377 @@ +import { describe, expect, it } from 'vitest'; +import { + graphql, + Kind, + parse, + printSchema, + type FragmentDefinitionNode, + type OperationDefinitionNode, +} from 'graphql'; +import { + NESTED_DOCUMENT_TYPE, + searchSchema, + type FacetsOutcome, + type SearchEngine, + type SearchField, + type SearchQuery, + type SearchResult, + type SearchType, +} from '@lde/search'; +import { buildGraphQLSchema } from '../src/build-schema.js'; +import { projectionFor } from '../src/projection.js'; + +const person: SearchType = { + name: 'Person', + class: 'https://schema.org/Person', + labelField: 'name', + fields: [ + { + name: 'name', + kind: 'text', + locales: ['nl'], + output: true, + searchable: { weight: 1 }, + }, + { name: 'birthDate', kind: 'keyword', output: true }, + // Shared with Organization, but required only here: the interface offers + // it with the weaker promise. + { name: 'homepage', kind: 'keyword', output: true, required: true }, + { + name: 'birthPlace', + kind: 'reference', + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + // Shared, and pointing at one type on both: reaches the interface. + { + name: 'address', + kind: 'reference', + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + ], +}; +const organization: SearchType = { + name: 'Organization', + class: 'https://schema.org/Organization', + labelField: 'name', + fields: [ + { + name: 'name', + kind: 'text', + locales: ['nl'], + output: true, + searchable: { weight: 1 }, + }, + { name: 'location', kind: 'keyword', output: true }, + { name: 'homepage', kind: 'keyword', output: true }, + { + name: 'address', + kind: 'reference', + output: true, + ref: { strategy: 'lookup', target: 'Place' }, + }, + ], +}; +const place: SearchType = { + name: 'Place', + class: 'https://schema.org/Place', + fields: [ + { + name: 'label', + kind: 'text', + locales: ['nl'], + output: true, + searchable: { weight: 1 }, + }, + ], +}; +const work: SearchType = { + name: 'CreativeWork', + class: 'https://schema.org/CreativeWork', + fields: [ + { name: 'title', kind: 'text', locales: ['nl'], output: true }, + { + name: 'creator', + kind: 'reference', + array: true, + output: true, + filterable: true, + facetable: true, + ref: { strategy: 'lookup', target: ['Person', 'Organization'] }, + }, + // A second field naming the same targets meets the same interface. + { + name: 'contributor', + kind: 'reference', + array: true, + output: true, + ref: { strategy: 'lookup', target: ['Person', 'Organization'] }, + }, + ], +}; +const schema = searchSchema(work, person, organization, place); +const gqlSchema = buildGraphQLSchema(schema); +const sdl = printSchema(gqlSchema); + +describe('serving a lookup over several targets', () => { + it('emits one interface per target set, implemented by each target’s reference type', () => { + expect(sdl).toMatch( + /interface PersonOrOrganizationReference \{\s+id: IRI!\s+name: \[LanguageString!\]!\s+homepage: String\s+address: PlaceReference\s+\}/, + ); + expect(sdl).toMatch( + /type PersonReference implements PersonOrOrganizationReference \{/, + ); + expect(sdl).toMatch( + /type OrganizationReference implements PersonOrOrganizationReference \{/, + ); + expect(sdl).toMatch(/creator: \[PersonOrOrganizationReference!\]!/); + expect(sdl).toMatch(/contributor: \[PersonOrOrganizationReference!\]!/); + expect( + sdl.match(/^interface PersonOrOrganizationReference /gm), + ).toHaveLength(1); + // Only what every target carries alike reaches the interface. + expect(sdl).not.toMatch( + /interface PersonOrOrganizationReference \{[^}]*birthDate/, + ); + expect(sdl).not.toMatch( + /interface PersonOrOrganizationReference \{[^}]*location/, + ); + // The members keep everything of their own, promises included. + expect(sdl).toMatch( + /type PersonReference implements[^}]*homepage: String!/, + ); + expect(sdl).toMatch( + /type PersonReference implements[^}]*birthDate: String/, + ); + }); + + it('refuses a declaration spelling the derived name itself', () => { + // A Reference Type named `PersonOrOrganization` would put its filter + // under the interface's filter name; refused rather than served wrongly. + const spelled: SearchType = { + name: 'PersonOrOrganization', + fields: [{ name: 'note', kind: 'keyword', output: true }], + }; + const nesting = (fields: readonly SearchField[]): SearchType => ({ + ...work, + fields, + }); + const note: SearchField = { + name: 'note', + kind: 'reference', + output: true, + ref: { strategy: 'inline', typeName: 'PersonOrOrganization' }, + }; + // Whichever registers first, the other is refused. + for (const fields of [ + [...work.fields, note], + [note, ...work.fields], + ]) { + expect(() => + buildGraphQLSchema( + searchSchema(nesting(fields), person, organization, place, spelled), + ), + ).toThrow(/collides with another type name/); + } + }); + + it('refuses a declaration spelling the derived name itself', () => { + // A Reference Type named `PersonOrOrganization` would put its filter + // under the interface's filter name; refused rather than served wrongly. + const spelled: SearchType = { + name: 'PersonOrOrganization', + fields: [{ name: 'note', kind: 'keyword', output: true }], + }; + const nesting = (fields: readonly SearchField[]): SearchType => ({ + ...work, + fields, + }); + const note: SearchField = { + name: 'note', + kind: 'reference', + output: true, + ref: { strategy: 'inline', typeName: 'PersonOrOrganization' }, + }; + // Whichever registers first, the other is refused. + for (const fields of [ + [...work.fields, note], + [note, ...work.fields], + ]) { + expect(() => + buildGraphQLSchema( + searchSchema(nesting(fields), person, organization, place, spelled), + ), + ).toThrow(/collides with another type name/); + } + }); + + it('shares a reference field only where every target points it at one type', () => { + // Person.affiliation looks up an Organization, Organization.affiliation a + // Place: two emitted types, so the interface offers neither. + expect(sdl).not.toMatch( + /interface PersonOrOrganizationReference \{[^}]*affiliation/, + ); + }); + + it('refuses a declaration whose derived interface name is taken', () => { + expect(() => + buildGraphQLSchema( + searchSchema(work, person, organization, place, { + name: 'PersonOrOrganizationReference', + class: 'https://schema.org/Thing', + fields: [{ name: 'note', kind: 'keyword', output: true }], + }), + ), + ).toThrow(/would be served as “PersonOrOrganizationReference”/); + }); + + it('types the filter by the target set, still over IRIs', () => { + expect(sdl).toMatch( + /input PersonOrOrganizationFilter \{\s+in: \[IRI!\]\s+\}/, + ); + expect(sdl).toMatch( + /input CreativeWorkCriterion @oneOf \{[^}]*creator: PersonOrOrganizationFilter/, + ); + }); + + it('reports the target each referent came from, and serves its own fields through a fragment', async () => { + const engine: SearchEngine = { + schema, + async search(): Promise { + return { + total: 1, + facets: {}, + hits: [ + { + id: 'https://w/1', + document: { + title: { nl: ['Werk'] }, + creator: [ + { + id: 'https://p/1', + name: { nl: ['Frank Koel'] }, + birthDate: '1901', + [NESTED_DOCUMENT_TYPE]: 'Person', + }, + { + id: 'https://o/1', + name: { nl: ['Zusters Franciscanessen'] }, + location: 'Roermond', + [NESTED_DOCUMENT_TYPE]: 'Organization', + }, + // Untyped – a hand-built document – reads as the first. + { id: 'https://x/1', name: { nl: ['Onbekend'] } }, + ], + }, + }, + ], + }; + }, + searchFacets: async ( + _searchType: SearchType, + queries: readonly SearchQuery[], + ): Promise => + queries.map(() => ({ facets: {} })), + }; + + const result = await graphql({ + schema: gqlSchema, + source: `{ + creativeWorks { + items { + creator { + __typename + id + name { value } + ... on PersonReference { birthDate } + ... on OrganizationReference { location } + } + } + } + }`, + contextValue: { engine, acceptLanguage: ['nl'] }, + }); + + expect(result.errors).toBeUndefined(); + const [item] = (result.data as { creativeWorks: { items: unknown[] } }) + .creativeWorks.items; + expect(item).toEqual({ + creator: [ + { + __typename: 'PersonReference', + id: 'https://p/1', + name: [{ value: 'Frank Koel' }], + birthDate: '1901', + }, + { + __typename: 'OrganizationReference', + id: 'https://o/1', + name: [{ value: 'Zusters Franciscanessen' }], + location: 'Roermond', + }, + { + __typename: 'PersonReference', + id: 'https://x/1', + name: [{ value: 'Onbekend' }], + birthDate: null, + }, + ], + }); + }); +}); + +describe('projecting a selection over several targets', () => { + function infoFor(query: string) { + const document = parse(query); + const operation = document.definitions.find( + (definition): definition is OperationDefinitionNode => + definition.kind === Kind.OPERATION_DEFINITION, + ); + const fragments = Object.fromEntries( + document.definitions + .filter( + (definition): definition is FragmentDefinitionNode => + definition.kind === Kind.FRAGMENT_DEFINITION, + ) + .map((fragment) => [fragment.name.value, fragment]), + ); + return { + fieldNodes: operation!.selectionSet.selections.filter( + (selection) => selection.kind === Kind.FIELD, + ), + fragments, + }; + } + + it('asks for what any target serves, fragments included, and descends each target’s own lookups', () => { + expect( + projectionFor( + infoFor(`{ + creativeWorks { + items { + creator { + __typename + id + name { value } + ... on PersonReference { birthDate birthPlace { label { value } } } + ... on OrganizationReference { location } + address { label { value } } + } + } + } + }`) as never, + work, + schema, + ), + ).toEqual({ + creator: { + fields: ['name', 'birthDate', 'birthPlace', 'location', 'address'], + resolve: { + birthPlace: { fields: ['label'] }, + address: { fields: ['label'] }, + }, + }, + }); + }); +}); diff --git a/packages/search-api-graphql/vite.config.ts b/packages/search-api-graphql/vite.config.ts index d4ea5d9a..dbea2aaa 100644 --- a/packages/search-api-graphql/vite.config.ts +++ b/packages/search-api-graphql/vite.config.ts @@ -29,7 +29,10 @@ export default mergeConfig( // Re-anchored for the selection-set projection: its defensive reads // (an unresolvable target, an absent field list) are reachable only // from a direct caller, since GraphQL validates the query first. - branches: 97.05, + // Re-anchored for the polymorphic lookup: the shared-field reading + // of its interface no longer re-checks what `searchSchema` already + // guarantees of a field two targets share, so that branch is gone. + branches: 97.27, statements: 100, }, }, diff --git a/packages/search-pipeline/src/extraction.ts b/packages/search-pipeline/src/extraction.ts index abe3109d..d9ef358e 100644 --- a/packages/search-pipeline/src/extraction.ts +++ b/packages/search-pipeline/src/extraction.ts @@ -14,13 +14,11 @@ import { fieldNamed, irAlias, isInlineReference, - labelSourceNameOf, - localLookupTypeOf, + localLookupTargetsOf, + referencedTargetsOf, referenceTypeNamed, - rootTypeNamed, } from '@lde/search/adapter'; import type { - ReferenceField, RootType, SearchField, SearchSchema, @@ -31,6 +29,8 @@ const factory = new AstFactory(); const parser = new Parser(); const generator = new Generator(); +const RDF_TYPE = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type'; + /** Options for {@link extractionQuery}. */ export interface ExtractionOptions { /** @@ -81,6 +81,12 @@ export interface ExtractionOptions { * `OPTIONAL`, so a referent with no key candidate keeps its row. The root side * needs nothing: a key field is a declared field, so its own branch and * template triple are already there. + * - **references naming several targets**: the referent may be of any of them, + * so its branch also reads the referent’s `rdf:type` (`OPTIONAL`, emitted + * as is – framing carries it as `@type`), which is what the projection + * matches against each target’s `class` to tell which one it is. Every + * target then contributes its own expansion or key hop, so whichever the + * referent turns out to be, its declaration is in the frame. * * Wire the result into a `SparqlConstructReader` (see `searchStages`), which * runs it per batch with the roots injected as VALUES. @@ -196,36 +202,43 @@ function buildFor( factory.gen(), ), ]; - const local = localTargetOf(field, schema, onPath); - if (local === undefined) { - // Only where no local expansion follows: a `local` lookup reads the - // target’s every path-bearing field, and a keyed target’s key field is - // one of them, so emitting the hop as well would state it twice. - const keyed = keyedTargetOf(field, schema); - if (keyed !== undefined) { - const built = buildKeyHop( - keyed.target, - keyed.keyField, - value, - counter, - ); - template.push(built.triple); - patterns.push(built.pattern); + const targets = referencedTargetsOf(field, schema); + if (targets.length > 1) { + const built = buildTypeHop(value, counter); + template.push(built.triple); + patterns.push(built.pattern); + } + const local = localTargetsOf(field, schema, onPath); + const expansions: PatternGroup[] = []; + for (const target of targets) { + if (!local.includes(target)) { + // Only where no local expansion follows: a `local` lookup reads the + // target’s every path-bearing field, and a keyed target’s key field + // is one of them, so emitting the hop as well would state it twice. + const keyed = keyedFieldOf(target); + if (keyed !== undefined) { + const built = buildKeyHop(target, keyed, value, counter); + template.push(built.triple); + patterns.push(built.pattern); + } + continue; } - } else { - const nested = buildFor(local, value, schema, counter, onPath); + const nested = buildFor(target, value, schema, counter, onPath); template.push(...nested.template); - // `OPTIONAL`, unlike an inline reference’s conjoined nesting: this - // referent is stored by id whether or not the referring document says - // anything about it, and conjoining would drop both together. - if (nested.branches.length > 0) { - patterns.push( - factory.patternOptional( - [factory.patternUnion(nested.branches, factory.gen())], - factory.gen(), - ), - ); - } + expansions.push(...nested.branches); + } + // `OPTIONAL`, unlike an inline reference’s conjoined nesting: this + // referent is stored by id whether or not the referring document says + // anything about it, and conjoining would drop both together. Several + // targets’ expansions share the one UNION: a referent is of one kind, + // and the branches of the others simply bind nothing for it. + if (expansions.length > 0) { + patterns.push( + factory.patternOptional( + [factory.patternUnion(expansions, factory.gen())], + factory.gen(), + ), + ); } branches.push(factory.patternGroup(patterns, factory.gen())); } @@ -234,45 +247,60 @@ function buildFor( } /** - * The Root Type a {@link ReferenceStrategy.local} lookup expands into here, or - * `undefined` where the field declares none – or where that type is already on - * this path, which is where the recursion stops. + * The Root Types a {@link ReferenceStrategy.local} lookup expands into here: + * none where the field declares none, and never one already on this path, + * which is where the recursion stops. */ -function localTargetOf( +function localTargetsOf( field: SearchField, schema: SearchSchema, visiting: ReadonlySet, -): RootType | undefined { - const local = localLookupTypeOf(field, schema); - return local === undefined || visiting.has(local.name) ? undefined : local; +): readonly RootType[] { + return localLookupTargetsOf(field, schema).filter( + (target) => !visiting.has(target.name), + ); } /** - * The keyed Root Type a reference points at, with the field its key is read - * from: a `lookup`’s `target` or an `idOnly`’s `labelSource` - * ({@link labelSourceNameOf}) that declares a {@link RootType.key}, or - * `undefined` for every other field. Naming the target is exactly the boundary - * the projection re-keys along, so the extraction reads the same declarations - * rather than a rule of its own: a reference that names no target keeps the - * node IRI, and needs no hop. + * The field a Root Type’s key is read from, when it declares a + * {@link RootType.key}. Naming the target is exactly the boundary the + * projection re-keys along ({@link referencedTargetsOf}), so the extraction + * reads the same declarations rather than a rule of its own: a reference that + * names no target keeps the node IRI, and needs no hop. */ -function keyedTargetOf( - field: SearchField, - schema: SearchSchema, -): { readonly target: RootType; readonly keyField: KeyedField } | undefined { - if (field.kind !== 'reference') { - return undefined; - } - const targetName = labelSourceNameOf(field as ReferenceField); - const target = - targetName === undefined ? undefined : rootTypeNamed(schema, targetName); - if (target?.key === undefined) { +function keyedFieldOf(target: RootType): KeyedField | undefined { + if (target.key === undefined) { return undefined; } // `searchSchema` guarantees a declared, path-bearing key field, so the target // – which came out of the schema – always has one. - const keyField = fieldNamed(target, target.key.field) as KeyedField; - return { target, keyField }; + return fieldNamed(target, target.key.field) as KeyedField; +} + +/** + * The one-triple hop that reads a referent’s `rdf:type`, for a reference whose + * referent may be of several kinds: emitted under `rdf:type` itself, which + * framing turns into the node’s `@type`, and bound in an `OPTIONAL` so an + * untyped referent keeps its row and falls back to the first target declared. + */ +function buildTypeHop( + referent: TermVariable, + counter: VariableCounter, +): { readonly triple: TripleNesting; readonly pattern: Pattern } { + const type = factory.termVariable(`t${counter.next++}`, factory.gen()); + const predicate = factory.termNamed(factory.gen(), RDF_TYPE); + return { + triple: factory.triple(referent, predicate, type), + pattern: factory.patternOptional( + [ + factory.patternBgp( + [factory.triple(referent, predicate, type)], + factory.gen(), + ), + ], + factory.gen(), + ), + }; } /** A key field as the schema guarantees it: path-bearing. */ diff --git a/packages/search-pipeline/test/extraction.test.ts b/packages/search-pipeline/test/extraction.test.ts index 0ed63c05..127c02ea 100644 --- a/packages/search-pipeline/test/extraction.test.ts +++ b/packages/search-pipeline/test/extraction.test.ts @@ -11,10 +11,9 @@ import { defineSearchType, searchSchema } from '@lde/search'; import { fieldNamed, irAlias, - labelSourceNameOf, - localLookupTypeOf, + localLookupTargetsOf, + referencedTargetsOf, referenceTypeNamed, - rootTypeNamed, } from '@lde/search/adapter'; import type { SearchSchema, SearchType } from '@lde/search'; import { extractionQuery, extractionQueryString } from '../src/extraction.js'; @@ -720,26 +719,27 @@ describe('extraction ⟷ projection contract', () => { } // A `local` lookup: the projection shapes the referent through the // TARGET’s own declaration, so it reads that type’s aliases off it. - const local = localLookupTypeOf(field, schema); - if (local !== undefined && !onPath.has(local.name)) { - for (const alias of projectionReads(local, schema, onPath)) { + const local = localLookupTargetsOf(field, schema).filter( + (target) => !onPath.has(target.name), + ); + for (const target of local) { + for (const alias of projectionReads(target, schema, onPath)) { aliases.add(alias); } - continue; } // A reference into a keyed type: the projection reads the referent’s key // candidates off the frame, under the TARGET’s alias for its key field. - const targetName = labelSourceNameOf(field); - const target = - targetName === undefined - ? undefined - : rootTypeNamed(schema, targetName); - const keyField = - target?.key === undefined - ? undefined - : fieldNamed(target, target.key.field); - if (target !== undefined && keyField !== undefined) { - aliases.add(irAlias(target, keyField)); + for (const target of referencedTargetsOf(field, schema)) { + if (local.includes(target)) { + continue; + } + const keyField = + target.key === undefined + ? undefined + : fieldNamed(target, target.key.field); + if (keyField !== undefined) { + aliases.add(irAlias(target, keyField)); + } } } return aliases; diff --git a/packages/search-pipeline/test/polymorphic-extraction.test.ts b/packages/search-pipeline/test/polymorphic-extraction.test.ts new file mode 100644 index 00000000..406f7c7d --- /dev/null +++ b/packages/search-pipeline/test/polymorphic-extraction.test.ts @@ -0,0 +1,151 @@ +import { describe, expect, it } from 'vitest'; +import { defineSearchType, searchSchema } from '@lde/search'; +import { irAlias } from '@lde/search/adapter'; +import { extractionQueryString } from '../src/extraction.js'; + +const SCHEMA = 'https://schema.org/'; +const RDF_TYPE = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type'; + +/** Keyed on an authority IRI: what makes the hop load-bearing. */ +const person = defineSearchType({ + name: 'Person', + class: `${SCHEMA}Person`, + labelField: 'name', + key: { field: 'sameAs' }, + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA}name>`, + locales: ['nl'], + output: true, + searchable: { weight: 1 }, + }, + { + name: 'sameAs', + kind: 'reference', + path: `<${SCHEMA}sameAs>`, + array: true, + }, + ], +}); + +const organization = defineSearchType({ + name: 'Organization', + class: `${SCHEMA}Organization`, + labelField: 'name', + fields: [ + { + name: 'name', + kind: 'text', + path: `<${SCHEMA}name>`, + locales: ['nl'], + output: true, + searchable: { weight: 1 }, + }, + { + name: 'location', + kind: 'keyword', + path: `<${SCHEMA}location>`, + output: true, + }, + ], +}); + +describe('extracting a lookup that names several targets', () => { + it('reads the referent’s rdf:type, so the projection can tell which target it is', () => { + const work = defineSearchType({ + name: 'Work', + class: `${SCHEMA}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `<${SCHEMA}creator>`, + array: true, + output: true, + ref: { strategy: 'lookup', target: ['Person', 'Organization'] }, + }, + ], + }); + const query = extractionQueryString( + work, + searchSchema(work, person, organization), + ); + + // Emitted as `rdf:type` itself, which framing carries as `@type`, and + // bound OPTIONAL so an untyped referent keeps its row. + expect(query).toContain(`<${RDF_TYPE}>`); + expect(query).toContain('OPTIONAL'); + // Every keyed target contributes its key hop, under its own alias – a + // plain lookup expands nothing else. + expect(query).toContain(irAlias(person, person.fields[1])); + expect(query).not.toContain(irAlias(organization, organization.fields[1])); + }); + + it('expands a local lookup through every target, in one OPTIONAL union', () => { + const edge = defineSearchType({ + name: 'CreatorEdge', + fields: [ + { + name: 'creator', + kind: 'reference', + path: `<${SCHEMA}creator>`, + output: true, + ref: { + strategy: 'lookup', + target: ['Person', 'Organization'], + local: true, + }, + }, + ], + }); + const work = defineSearchType({ + name: 'Work', + class: `${SCHEMA}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `<${SCHEMA}creator>`, + array: true, + output: true, + ref: { strategy: 'inline', typeName: 'CreatorEdge' }, + }, + ], + }); + const query = extractionQueryString( + work, + searchSchema(work, edge, person, organization), + ); + + expect(query).toContain(`<${RDF_TYPE}>`); + // Each target's fields under that target's own aliases: the projection + // reads a referent through whichever declaration its type matches. + for (const field of person.fields) { + expect(query).toContain(irAlias(person, field)); + } + for (const field of organization.fields) { + expect(query).toContain(irAlias(organization, field)); + } + }); + + it('reads no rdf:type for a lookup naming one target', () => { + const work = defineSearchType({ + name: 'Work', + class: `${SCHEMA}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `<${SCHEMA}creator>`, + output: true, + ref: { strategy: 'lookup', target: 'Person' }, + }, + ], + }); + expect( + extractionQueryString(work, searchSchema(work, person)), + ).not.toContain(RDF_TYPE); + }); +}); diff --git a/packages/search-typesense/src/collection-definition.ts b/packages/search-typesense/src/collection-definition.ts index c794d385..b42d78bc 100644 --- a/packages/search-typesense/src/collection-definition.ts +++ b/packages/search-typesense/src/collection-definition.ts @@ -11,7 +11,8 @@ import { isInlineReference, isInternalField, joinGraph, - localLookupTypeOf, + localLookupTargetsOf, + TARGET_FIELD, nestedFieldName, nestedReferenceType, physicalFields, @@ -148,25 +149,24 @@ export function buildCollectionDefinition( } /** - * The type a field nests, or `undefined` when it nests nothing. Throws when the - * type carries an inline reference the caller gave no schema to resolve: the - * collection would silently store the reference as a string the projection - * never writes, so every document would fail to import. + * The types a field nests – one for an inline reference or a single-target + * `local` lookup, several for a `local` lookup naming several targets – or + * none when it nests nothing. Throws when the type carries an inline reference + * the caller gave no schema to resolve: the collection would silently store + * the reference as a string the projection never writes, so every document + * would fail to import. */ -function nestedTypeOf( +function nestedTypesOf( searchType: SearchType, field: SearchField, schema: SearchSchema | undefined, -): SearchType | undefined { +): readonly SearchType[] { const nested = - schema === undefined - ? undefined - : (nestedReferenceType(schema, field) ?? - localLookupTypeOf(field, schema)); + schema === undefined ? [] : nestedTypesOfNestedField(field, schema); // Reached only for a field carrying a Role, so an inline reference here // stores entries: a Role-less one is an internal reading device, pruned // before the writer. - if (nested === undefined && isInlineReference(field)) { + if (nested.length === 0 && isInlineReference(field)) { throw new Error( `Building the collection for “${searchType.name}” needs the search schema its surfaced inline reference “${field.name}” resolves against; pass it as the collection-definition option “schema”.`, ); @@ -246,13 +246,14 @@ function typesenseFields( if (isInternalField(field)) { return []; } - const nested = nestedTypeOf(searchType, field, schema); - if (nested !== undefined) { + const nested = nestedTypesOf(searchType, field, schema); + if (nested.length > 0) { return [ - ...nestedFields( + ...nestedObjectFields( field.name, field, nested, + nested.length > 1, schema as SearchSchema, defaultLocale, false, @@ -373,6 +374,67 @@ function typesenseFields( * the children rather than declared, and this is the one place that invariant * can be got wrong. */ +function nestedObjectFields( + prefix: string, + reference: SearchField, + nestedTypes: readonly SearchType[], + polymorphic: boolean, + schema: SearchSchema, + defaultLocale: string | undefined, + withinArray: boolean, + onPath: ReadonlySet, +): CollectionFieldSchema[] { + // `polymorphic` is about the DECLARATION, `nestedTypes` about what is + // descended into here: a lookup naming several targets of which only one is + // still off the path stores a discriminator all the same. + if (!polymorphic) { + return nestedFields( + prefix, + reference, + nestedTypes[0], + schema, + defaultLocale, + withinArray, + onPath, + ); + } + // A lookup naming several targets stores a referent of whichever kind it + // turns out to be under ONE nested object, so the object declares the union + // of the targets’ fields – each declared once, and `searchSchema` has + // already refused two targets declaring one name differently – plus the + // discriminator the projection writes to say which kind an entry is. + const byName = new Map(); + for (const nestedType of nestedTypes) { + for (const declared of nestedFields( + prefix, + reference, + nestedType, + schema, + defaultLocale, + withinArray, + onPath, + )) { + if (!byName.has(declared.name)) { + byName.set(declared.name, declared); + } + } + } + const flattensToArray = withinArray || reference.array === true; + const children = [...byName.values()].filter( + (declared) => declared.name !== prefix, + ); + children.push(targetField(prefix, flattensToArray)); + return [ + { + name: prefix, + type: flattensToArray ? 'object[]' : 'object', + index: children.some((child) => child.index !== false), + optional: reference.required !== true, + }, + ...children, + ]; +} + function nestedFields( prefix: string, reference: SearchField, @@ -413,8 +475,11 @@ function nestedFields( if (isInternalField(field)) { continue; } - const deeper = nestedTypeOfNestedField(field, schema); - if (deeper !== undefined && walked.has(deeper.name)) { + const deeper = nestedTypesOfNestedField(field, schema); + // Descend into the types not yet on the path; where every one is, the + // cycle stops here. + const descend = deeper.filter((type) => !walked.has(type.name)); + if (deeper.length > 0 && descend.length === 0) { // The cycle stops here, and so does the frame – but a value is still // stored: the extraction falls back to the target's key hop, and the // projection writes the referent as an `{id}` object beside its identity @@ -432,6 +497,9 @@ function nestedFields( index: false, optional: true, }, + // A cut lookup naming several targets still says which one it + // stored, exactly as an expanded one does. + ...(deeper.length > 1 ? [targetField(boundary, boundaryArray)] : []), { // Always a Root Type, so always identified: `searchSchema` rejects // inline cycles, so the only way to arrive at a type already on the @@ -445,12 +513,13 @@ function nestedFields( ); continue; } - if (deeper !== undefined) { + if (descend.length > 0) { children.push( - ...nestedFields( + ...nestedObjectFields( nestedFieldName(prefix, field.name), field, - deeper, + descend, + deeper.length > 1, schema, defaultLocale, flattensToArray, @@ -563,14 +632,28 @@ function nestedIdentityFields( ]; } -/** The type a *nested* field itself nests: another inline reference’s type, or - * the Root Type a {@link ReferenceStrategy.local local} lookup projects its - * endpoint through. */ -function nestedTypeOfNestedField( +/** The stored discriminator of a lookup naming several targets + * ({@link TARGET_FIELD}): a stored value, never indexed. */ +function targetField(prefix: string, array: boolean): CollectionFieldSchema { + return { + name: nestedFieldName(prefix, TARGET_FIELD), + type: array ? 'string[]' : 'string', + index: false, + optional: true, + }; +} + +/** The types a field nests: another inline reference’s type, or the Root + * Types a {@link ReferenceStrategy.local local} lookup projects its endpoint + * through. */ +function nestedTypesOfNestedField( field: SearchField, schema: SearchSchema, -): SearchType | undefined { - return nestedReferenceType(schema, field) ?? localLookupTypeOf(field, schema); +): readonly SearchType[] { + const referenceType = nestedReferenceType(schema, field); + return referenceType === undefined + ? localLookupTargetsOf(field, schema) + : [referenceType]; } /** diff --git a/packages/search-typesense/src/lookup.ts b/packages/search-typesense/src/lookup.ts index 4521981d..db8fbe7b 100644 --- a/packages/search-typesense/src/lookup.ts +++ b/packages/search-typesense/src/lookup.ts @@ -10,7 +10,7 @@ import { fieldNamed, labelFieldOf, nestedReferenceType, - rootTypeNamed, + referencedTargetsOf, } from '@lde/search/adapter'; import { escapeFilterValue } from './query-compiler.js'; @@ -31,18 +31,26 @@ import { escapeFilterValue } from './query-compiler.js'; export type ResolvedReferents = | { readonly via: 'lookup'; - /** The Root Type these referents are declared by – reconstruction reads - * them through the target’s own declaration, never the referrer’s. */ - readonly target: RootType; - readonly documents: ReadonlyMap>; - /** Keyed by the reference field’s name on the type this level carries. */ - readonly children: ReadonlyMap; + /** By IRI. A lookup naming several targets holds referents of any of + * them here, each saying which. */ + readonly documents: ReadonlyMap; } | { readonly via: 'nested'; readonly children: ReadonlyMap; }; +/** One referent a lookup level fetched. */ +export interface ResolvedReferent { + /** The Root Type this referent is declared by – the target whose collection + * answered for it. Reconstruction reads it through that target’s own + * declaration, never the referrer’s. */ + readonly target: RootType; + readonly document: Record; + /** The levels below, keyed by the reference field’s name on `target`. */ + readonly children: ReadonlyMap; +} + /** Typesense caps a filter list; the same batch size the label lookup uses. */ const BATCH_SIZE = 200; @@ -104,45 +112,67 @@ export async function resolveProjection( // hand-built query from throwing here rather than at the port’s guard. return undefined; } - const target = rootTypeNamed(schema, field.ref.target); - const collection = - target === undefined ? undefined : collections.get(target.class); - if (target === undefined || collection === undefined) { - return undefined; - } - const include = includeFields(target, level.fields, level.resolve); + const targets = referencedTargetsOf(field, schema).filter((target) => + collections.has(target.class), + ); const iris = distinctIris(parents, name); - // Nothing to read: no referent named, or the selection reduced to the - // `id` the referring document already carries. - if (iris.length === 0 || include.length <= 1) { + if (targets.length === 0 || iris.length === 0) { return undefined; } - const documents = await fetchReferents( - client, - collection, - iris, - include, - onError, - ); - return [ - name, - { - via: 'lookup', - target, - documents, + // One fetch per target, concurrently: a referent lives in one of the + // collections, and which one is what a lookup naming several targets + // exists to find out – so with several, every collection is asked even + // for a selection reduced to `id`, since the answer types the referent. + // With one target, a selection reduced to the `id` the referring + // document already carries reads nothing. + const perTarget = await Promise.all( + targets.map(async (target) => { + const include = includeFields(target, level.fields, level.resolve); + if (targets.length === 1 && include.length <= 1) { + return undefined; + } + const fetched = await fetchReferents( + client, + collections.get(target.class) as string, + iris, + include, + onError, + ); // The level below reads the IRIs off the documents this one just // fetched – one more round-trip for the page, whatever its size. - children: await resolveProjection( + const children = await resolveProjection( client, level.resolve, target, schema, collections, - [...documents.values()], + [...fetched.values()], onError, - ), - }, - ] as const; + ); + return { target, fetched, children }; + }), + ); + // Declaration order is precedence: an IRI two collections hold belongs + // to the target declared first. + const documents = new Map(); + for (const resolved of perTarget) { + if (resolved === undefined) { + continue; + } + for (const [iri, document] of resolved.fetched) { + if (!documents.has(iri)) { + documents.set(iri, { + target: resolved.target, + document, + children: resolved.children, + }); + } + } + } + if (perTarget.every((resolved) => resolved === undefined)) { + return undefined; + } + return [name, { via: 'lookup', documents }] as const; }), ); for (const level of levels) { diff --git a/packages/search-typesense/src/query-compiler.ts b/packages/search-typesense/src/query-compiler.ts index cdc0bc2f..e05791db 100644 --- a/packages/search-typesense/src/query-compiler.ts +++ b/packages/search-typesense/src/query-compiler.ts @@ -23,7 +23,7 @@ import { isRangeFacet, isWelded, joinGraph, - localLookupTypeOf, + localLookupTargetsOf, pageForOffset, physicalFields, nestedReferenceType, @@ -268,16 +268,24 @@ function collectSearchable( if (isInternalField(field)) { continue; } + const referenceType = nestedReferenceType(schema, field); const nested = - nestedReferenceType(schema, field) ?? localLookupTypeOf(field, schema); + referenceType === undefined + ? localLookupTargetsOf(field, schema) + : [referenceType]; // Cut on the CHILD, not on entry, and against the set this level was // reached with – the same boundary the collection's own walk cuts at // (`nestedFields`). Returning on entry instead skipped the companions of // the level the collection HAD declared, leaving them indexed and absent - // from `query_by` wherever a type reached itself. - if (nested !== undefined && !onPath.has(nested.name)) { + // from `query_by` wherever a type reached itself. Several targets each + // contribute their searchable fields: the one nested object declares them + // all, and free text has to reach a referent of either kind. + for (const nestedType of nested) { + if (onPath.has(nestedType.name)) { + continue; + } collectSearchable( - nested, + nestedType, locale, schema, qualify(prefix, field.name), diff --git a/packages/search-typesense/src/search.ts b/packages/search-typesense/src/search.ts index b1caa61c..961af012 100644 --- a/packages/search-typesense/src/search.ts +++ b/packages/search-typesense/src/search.ts @@ -1,6 +1,7 @@ import type { Client } from 'typesense'; import type { SearchParams } from 'typesense/lib/Typesense/Documents.js'; import { + NESTED_DOCUMENT_TYPE, type FacetBucket, type FacetsOutcome, type LocalizedValue, @@ -31,13 +32,15 @@ import { isUnsatisfiable, joinGraph, labelFieldOf, - labelSourceNameOf, - labelTargetNameOf, - localLookupTypeOf, + labelSourceNamesOf, + labelTargetNamesOf, + localLookupTargetsOf, nestedReferenceType, outputFields, physicalFields, + referencedTargetsOf, referenceFields, + storedTargetOf, } from '@lde/search/adapter'; import { buildSearchParams, @@ -45,7 +48,11 @@ import { type BuildSearchParamsOptions, } from './query-compiler.js'; import { deriveCollectionName } from './collection-name.js'; -import { resolveProjection, type ResolvedReferents } from './lookup.js'; +import { + resolveProjection, + type ResolvedReferent, + type ResolvedReferents, +} from './lookup.js'; /** Where the engine reads documents – plus every query-compiler knob * ({@link BuildSearchParamsOptions}), declared once there and forwarded @@ -189,7 +196,12 @@ export function createTypesenseSearchEngine< const typesByName = new Map( [...schema.values()].map((searchType) => [searchType.name, searchType]), ); - const labelSources = new Map>( + // Per field, one source per target it names, in order of precedence – a + // reference naming several targets reads labels from every one of them. + const labelSources = new Map< + string, + ReadonlyMap + >( [...schema.values()].map((searchType) => [ searchType.class, new Map( @@ -197,21 +209,19 @@ export function createTypesenseSearchEngine< // An INLINE reference names its label target one level in, through // its identity companion: the companion holds that field's ids, so // that field's target is what can label them. - .filter((field) => labelTargetNameOf(field, schema) !== undefined) - .map((field) => { - const source = typesByName.get( - labelTargetNameOf(field, schema) as string, - ) as RootType; - const labelField = labelFieldOf(source) as TextField; - return [ - field.name, - { + .filter((field) => labelTargetNamesOf(field, schema).length > 0) + .map((field) => [ + field.name, + labelTargetNamesOf(field, schema).map((targetName) => { + const source = typesByName.get(targetName) as RootType; + const labelField = labelFieldOf(source) as TextField; + return { collection: collections.get(source.class) as string, labelField, queryBy: physicalFields(labelField).search.join(','), - }, - ]; - }), + }; + }), + ]), ), ]), ); @@ -242,7 +252,9 @@ export function createTypesenseSearchEngine< type, [ ...new Map( - [...sources.values()].map((source) => [source.collection, source]), + [...sources.values()] + .flat() + .map((source) => [source.collection, source]), ).values(), ], ]), @@ -252,7 +264,7 @@ export function createTypesenseSearchEngine< // not re-derive or re-resolve them on every search. const outputReferenceSources = new Map< string, - readonly { name: string; source: LabelSource }[] + readonly { name: string; sources: readonly LabelSource[] }[] >( [...schema.values()].map((searchType) => { const sources = labelSources.get(searchType.class); @@ -272,13 +284,13 @@ export function createTypesenseSearchEngine< // the entry – and their FACET buckets are labelled by a separate // path that reads the companion. nestedReferenceType(schema, field) === undefined && - localLookupTypeOf(field, schema) === undefined, + localLookupTargetsOf(field, schema).length === 0, ) .map((field) => ({ name: field.name, - source: (sources as Map).get( + sources: (sources as Map).get( field.name, - ) as LabelSource, + ) as readonly LabelSource[], })), ]; }), @@ -441,6 +453,7 @@ export function createTypesenseSearchEngine< [response], labelSources.get(searchType.class), outputReferenceSources.get(searchType.class) ?? [], + distinctLabelSources.get(searchType.class) ?? [], ), ); // What the caller asked to resolve, level by level. Independent of the @@ -520,6 +533,7 @@ export function createTypesenseSearchEngine< responses, labelSources.get(searchType.class), outputReferenceSources.get(searchType.class) ?? [], + distinctLabelSources.get(searchType.class) ?? [], ), ); // multi_search reports a failed entry inline instead of rejecting the @@ -632,37 +646,44 @@ async function loadAllLabels( */ function labelLookupGroups( responses: readonly TypesenseSearchResponse[], - sources: ReadonlyMap | undefined, - outputSources: readonly { name: string; source: LabelSource }[], + sources: ReadonlyMap | undefined, + outputSources: readonly { name: string; sources: readonly LabelSource[] }[], + ordered: readonly LabelSource[], ): LabelLookupGroup[] { if (sources === undefined || sources.size === 0) { return []; } - const irisByCollection = new Map< - string, - { source: LabelSource; iris: Set } - >(); - const add = (source: LabelSource, iri: string): void => { - let group = irisByCollection.get(source.collection); - if (group === undefined) { - group = { source, iris: new Set() }; - irisByCollection.set(source.collection, group); + // One group per collection, in the order the TYPE’s fields first name them + // – seeded up front rather than in the order the hits happen to mention + // them, because `fetchLabels` reads group order as precedence and a label + // is resolved per type, not per field: an IRI two collections hold gets the + // label of the one the type declares first, whichever field carried it. + const irisByCollection = new Map( + ordered.map((source) => [ + source.collection, + { source, iris: new Set() }, + ]), + ); + // An IRI of a field naming several targets travels to EVERY one of their + // collections: which holds it is not known until one answers. + const add = (fieldSources: readonly LabelSource[], iri: string): void => { + for (const source of fieldSources) { + irisByCollection.get(source.collection)?.iris.add(iri); } - group.iris.add(iri); }; for (const response of responses) { // Hits only carry labels for OUTPUT reference fields (reconstructDocument // skips non-output fields); `outputSources` pairs each with its resolved // source, precomputed per type. for (const hit of response.hits ?? []) { - for (const { name, source } of outputSources) { + for (const { name, sources: fieldSources } of outputSources) { const raw = hit.document[name]; if (Array.isArray(raw)) { for (const value of raw) { - add(source, String(value)); + add(fieldSources, String(value)); } } else if (typeof raw === 'string') { - add(source, raw); + add(fieldSources, raw); } } } @@ -670,19 +691,18 @@ function labelLookupGroups( // like `class`); resolve them in the same lookup. Skip a non-source facet // (e.g. a keyword facet) in one check instead of probing every bucket. for (const facet of response.facet_counts ?? []) { - const source = sources.get(facet.field_name); - if (source === undefined) { + const facetSources = sources.get(facet.field_name); + if (facetSources === undefined) { continue; } for (const bucket of facet.counts) { - add(source, bucket.value); + add(facetSources, bucket.value); } } } - return [...irisByCollection.values()].map(({ source, iris }) => ({ - source, - iris: [...iris], - })); + return [...irisByCollection.values()] + .filter(({ iris }) => iris.size > 0) + .map(({ source, iris }) => ({ source, iris: [...iris] })); } /** @@ -797,12 +817,20 @@ export async function fetchLabels( return; } for (const hit of result.hits ?? []) { + const id = String(hit.document.id); + // First group wins: the groups arrive in the order a field declares its + // targets, so an IRI two collections hold is labelled by the target + // declared first – the same precedence the lookup and the projection + // apply. + if (labels.has(id)) { + continue; + } const label = localizedValue( hit.document, groupPerSearch[index].source.labelField, ); if (label !== undefined) { - labels.set(String(hit.document.id), label); + labels.set(id, label); } } }); @@ -823,7 +851,10 @@ function mergeLabels( const merged = new Map(); for (const map of maps) { for (const [iri, label] of map) { - merged.set(iri, label); + // First source wins, as it does for a fetched lookup (`fetchLabels`). + if (!merged.has(iri)) { + merged.set(iri, label); + } } } return merged; @@ -889,7 +920,7 @@ export function parseSearchResponse( // its IRIs. const referenceFacets = new Set( referenceFields(searchType) - .filter((field) => labelTargetNameOf(field, schema) !== undefined) + .filter((field) => labelTargetNamesOf(field, schema).length > 0) .map((field) => field.name), ); const facets: Record = {}; @@ -965,19 +996,19 @@ function logicalValue( nested, labels, schema, - resolved?.children, + resolved?.via === 'nested' ? resolved.children : undefined, ); } // Checked BEFORE the plain lookup below, because a `local` lookup // resolves through the same kind of level while storing a different // shape: its stored value is the endpoint's own document, not an id, so // reading it as one would stringify the object into a bogus `id`. - const localType = localLookupTypeOf(field, schema); - if (localType !== undefined) { + const localTargets = localLookupTargetsOf(field, schema); + if (localTargets.length > 0) { return localLookupValue( flat[field.name], field, - localType, + localTargets, resolved, labels, schema, @@ -1071,18 +1102,9 @@ function nestedValue( (referent): referent is Record => typeof referent === 'object' && referent !== null, ); - const documents: NestedDocument[] = referents.map((referent) => { - const document = reconstructDocument( - referent, - referenceType, - labels, - schema, - children, - ); - return typeof referent.id === 'string' - ? { id: referent.id, ...document } - : document; - }); + const documents: NestedDocument[] = referents.map((referent) => + nestedDocument(referent, referenceType, labels, schema, children), + ); if (documents.length === 0) { return undefined; } @@ -1111,14 +1133,16 @@ function nestedValue( function localLookupValue( raw: unknown, field: ReferenceField, - target: SearchType, + targets: readonly RootType[], resolved: ResolvedReferents | undefined, labels: ReadonlyMap, schema: SearchSchema, ): SearchValue | undefined { const fetched = - resolved?.via === 'lookup' ? resolved.documents : new Map(); - const entries = (Array.isArray(raw) ? raw : [raw]) + resolved?.via === 'lookup' + ? resolved.documents + : new Map(); + const documents = (Array.isArray(raw) ? raw : [raw]) .filter( (entry): entry is Record => typeof entry === 'object' && entry !== null, @@ -1126,16 +1150,67 @@ function localLookupValue( .map((entry) => { const id = typeof entry.id === 'string' ? entry.id : undefined; const authoritative = id === undefined ? undefined : fetched.get(id); - return authoritative ?? entry; + // An entry no collection answered for is read through the declaration + // it was stored under – which it says itself where there were several + // to choose from, and the only one otherwise. + const referent: ResolvedReferent = authoritative ?? { + target: storedTargetOf(entry, targets), + document: entry, + children: new Map(), + }; + return typedDocument( + targets.length > 1, + referent.target, + nestedDocument( + referent.document, + referent.target, + labels, + schema, + referent.children, + ), + ); }); - return nestedValue( - entries, - field, - target, + if (documents.length === 0) { + return undefined; + } + return field.array === true ? documents : documents[0]; +} + +/** + * A nested document read through one of a lookup’s **several** targets, marked + * with that type’s name ({@link NESTED_DOCUMENT_TYPE}) so a surface can tell + * which it is – the marker is a symbol, so it never surfaces as a field. A + * single-target lookup marks nothing: there is nothing to tell apart. + */ +function typedDocument( + polymorphic: boolean, + target: RootType, + document: NestedDocument, +): NestedDocument { + return polymorphic + ? { ...document, [NESTED_DOCUMENT_TYPE]: target.name } + : document; +} + +/** One nested document rebuilt from a flat referent, carrying its `id` where + * it has one – the shared body of every nesting that reads a referent. */ +function nestedDocument( + referent: Record, + searchType: SearchType, + labels: ReadonlyMap, + schema: SearchSchema, + children: ReadonlyMap | undefined, +): NestedDocument { + const document = reconstructDocument( + referent, + searchType, labels, schema, - resolved?.children, + children, ); + return typeof referent.id === 'string' + ? { id: referent.id, ...document } + : document; } /** @@ -1157,20 +1232,21 @@ function lookupValue( return undefined; } const iris = Array.isArray(raw) ? (raw as string[]) : [String(raw)]; + const polymorphic = referencedTargetsOf(field, schema).length > 1; const documents: NestedDocument[] = iris.map((iri) => { const referent = resolved.documents.get(iri); return referent === undefined ? { id: iri } - : { + : typedDocument(polymorphic, referent.target, { id: iri, ...reconstructDocument( - referent, - resolved.target, + referent.document, + referent.target, labels, schema, - resolved.children, + referent.children, ), - }; + }); }); return field.array === true ? documents : documents[0]; } @@ -1191,7 +1267,7 @@ function referenceValue( // a label, even if the (cached, full-collection) map happens to hold this // IRI from another source. const label = - labelSourceNameOf(field) === undefined ? undefined : labels.get(iri); + labelSourceNamesOf(field).length === 0 ? undefined : labels.get(iri); return label === undefined ? { id: iri } : { id: iri, label }; }); return field.array === true ? references : references[0]; diff --git a/packages/search-typesense/test/polymorphic-lookup.test.ts b/packages/search-typesense/test/polymorphic-lookup.test.ts new file mode 100644 index 00000000..6f68dcef --- /dev/null +++ b/packages/search-typesense/test/polymorphic-lookup.test.ts @@ -0,0 +1,518 @@ +import { describe, expect, it } from 'vitest'; +import { + defineSearchType, + NESTED_DOCUMENT_TYPE, + searchSchema, + type NestedDocument, + type SearchQuery, +} from '@lde/search'; +import { TARGET_FIELD } from '@lde/search/adapter'; +import { createTypesenseSearchEngine } from '../src/search.js'; +import { buildCollectionDefinition } from '../src/collection-definition.js'; +import { fakeTypesenseClient, filterByIds } from './fake-typesense-client.js'; + +const person = defineSearchType({ + name: 'Person', + class: 'https://example.org/Person', + labelField: 'name', + fields: [ + { + name: 'name', + kind: 'text', + locales: ['nl', 'und'], + output: true, + searchable: { weight: 1 }, + }, + { name: 'birthDate', kind: 'keyword', output: true }, + ], +}); + +const organization = defineSearchType({ + name: 'Organization', + class: 'https://example.org/Organization', + labelField: 'name', + fields: [ + { + name: 'name', + kind: 'text', + locales: ['nl', 'und'], + output: true, + searchable: { weight: 1 }, + }, + { name: 'location', kind: 'keyword', output: true }, + ], +}); + +/** A plain lookup at the root, faceted: what the issue was filed against. */ +const work = defineSearchType({ + name: 'CreativeWork', + class: 'https://example.org/CreativeWork', + fields: [ + { name: 'title', kind: 'text', locales: ['nl'], output: true }, + { + name: 'creator', + kind: 'reference', + array: true, + output: true, + filterable: true, + facetable: true, + ref: { strategy: 'lookup', target: ['Person', 'Organization'] }, + }, + ], +}); + +/** The same range one level in: an edge nesting a `local` lookup. */ +const creatorEdge = defineSearchType({ + name: 'CreatorEdge', + fields: [ + { name: 'role', kind: 'keyword', output: true }, + { + name: 'creator', + kind: 'reference', + output: true, + ref: { + strategy: 'lookup', + target: ['Person', 'Organization'], + local: true, + }, + }, + ], +}); + +const edgedWork = defineSearchType({ + name: 'Painting', + class: 'https://example.org/Painting', + fields: [ + { + name: 'creator', + kind: 'reference', + array: true, + output: true, + filterable: true, + ref: { strategy: 'inline', typeName: 'CreatorEdge', identity: 'creator' }, + }, + ], +}); + +const schema = searchSchema(work, edgedWork, person, organization, creatorEdge); +const collections = { + Person: 'people', + Organization: 'organizations', + CreativeWork: 'works', + Painting: 'paintings', +}; + +const base: SearchQuery = { + where: [], + orderBy: [], + limit: 10, + offset: 0, + facets: [], + locale: 'nl', +}; + +/** `p/1` is a person, `o/1` an organization, `both` is in both collections. */ +const people: Record> = { + 'https://p/1': { + id: 'https://p/1', + name_nl: 'Frank Koel', + birthDate: '1901', + }, + 'https://both': { id: 'https://both', name_nl: 'Als persoon' }, +}; +const organizations: Record> = { + 'https://o/1': { + id: 'https://o/1', + name_nl: 'Zusters Franciscanessen', + location: 'Roermond', + }, + 'https://both': { id: 'https://both', name_nl: 'Als organisatie' }, +}; + +/** The lookups a search made, told from the label round-trip by their + * `include_fields`. */ +function client(rootResponse: Record) { + const searches: Record[] = []; + const fake = fakeTypesenseClient({ + multiSearch: (search) => { + if (search.query_by_weights !== undefined) { + return rootResponse; + } + if (search.include_fields !== undefined) { + searches.push(search); + } + const documents = search.collection === 'people' ? people : organizations; + const include = + search.include_fields === undefined + ? undefined + : new Set(String(search.include_fields).split(',')); + const hits = filterByIds(String(search.filter_by)) + .filter((id) => documents[id] !== undefined) + .map((id) => ({ + document: + include === undefined + ? documents[id] + : Object.fromEntries( + Object.entries(documents[id]).filter(([key]) => + include.has(key), + ), + ), + })); + return { found: hits.length, hits }; + }, + }); + return { fake, searches }; +} + +const typeOf = (document: unknown) => + (document as NestedDocument)[NESTED_DOCUMENT_TYPE]; + +describe('resolving a lookup over several targets', () => { + const hits = { + found: 1, + hits: [ + { + document: { + id: 'https://w/1', + title_nl: 'Werk', + creator: [ + 'https://p/1', + 'https://o/1', + 'https://both', + 'https://gone', + ], + }, + }, + ], + }; + + it('asks every target’s collection, and types each referent by the one that answered', async () => { + const { fake, searches } = client(hits); + const engine = createTypesenseSearchEngine(fake.client, schema, { + collections, + }); + + const result = await engine.search(work as never, { + ...base, + resolve: { creator: { fields: ['name', 'birthDate', 'location'] } }, + }); + const creators = (result.hits[0].document as Record) + .creator as readonly NestedDocument[]; + + expect(searches.map((search) => search.collection).sort()).toEqual([ + 'organizations', + 'people', + ]); + expect(creators.map(typeOf)).toEqual([ + 'Person', + 'Organization', + // Held by both: the target declared first wins. + 'Person', + // Held by neither: a bare id, of no type at all. + undefined, + ]); + expect(creators[0]).toMatchObject({ + id: 'https://p/1', + name: { nl: ['Frank Koel'] }, + birthDate: '1901', + }); + expect(creators[1]).toMatchObject({ + id: 'https://o/1', + name: { nl: ['Zusters Franciscanessen'] }, + location: 'Roermond', + }); + expect(creators[2]).toMatchObject({ name: { nl: ['Als persoon'] } }); + expect(creators[3]).toEqual({ id: 'https://gone' }); + // Each collection is asked only for what its own declaration serves. + const included = Object.fromEntries( + searches.map((search) => [search.collection, search.include_fields]), + ); + expect(included.people).not.toContain('location'); + expect(included.organizations).not.toContain('birthDate'); + }); + + it('fetches from every collection even for a selection reduced to id, since the answer types the referent', async () => { + const { fake, searches } = client(hits); + const engine = createTypesenseSearchEngine(fake.client, schema, { + collections, + }); + + const result = await engine.search(work as never, { + ...base, + resolve: { creator: { fields: [] } }, + }); + const creators = (result.hits[0].document as Record) + .creator as readonly NestedDocument[]; + + expect(searches).toHaveLength(2); + expect(creators.map(typeOf)).toEqual([ + 'Person', + 'Organization', + 'Person', + undefined, + ]); + }); + + it('labels facet buckets from whichever collection holds the value', async () => { + const { fake } = client({ + ...hits, + facet_counts: [ + { + field_name: 'creator', + counts: [ + { value: 'https://p/1', count: 3 }, + { value: 'https://o/1', count: 2 }, + { value: 'https://both', count: 1 }, + ], + }, + ], + }); + const engine = createTypesenseSearchEngine(fake.client, schema, { + collections, + }); + + const result = await engine.search(work as never, { + ...base, + facets: ['creator'], + }); + + expect((result.facets as Record).creator).toEqual([ + { value: 'https://p/1', count: 3, label: { nl: ['Frank Koel'] } }, + { + value: 'https://o/1', + count: 2, + label: { nl: ['Zusters Franciscanessen'] }, + }, + { value: 'https://both', count: 1, label: { nl: ['Als persoon'] } }, + ]); + }); +}); + +describe('a stored referent of several possible kinds', () => { + const hits = { + found: 1, + hits: [ + { + document: { + id: 'https://w/2', + creator: [ + // Identified and indexed: the collection's record replaces it. + { + role: 'schilder', + creator: { + id: 'https://o/1', + name_und: 'Zusters', + [TARGET_FIELD]: 'Organization', + }, + }, + // Identified, not indexed: what was stated, read through the + // declaration it was stored under. + { + role: 'drukker', + creator: { + id: 'https://o/gone', + name_und: 'Drukkerij', + location: 'Venlo', + [TARGET_FIELD]: 'Organization', + }, + }, + // Never identified: same, minus the id. + { + role: 'auteur', + creator: { name_und: 'Jan Jansen', [TARGET_FIELD]: 'Person' }, + }, + ], + creator_id: ['https://o/1', 'https://o/gone'], + }, + }, + ], + }; + + it('reads an entry no collection answers for through its stored discriminator', async () => { + const { fake } = client(hits); + const engine = createTypesenseSearchEngine(fake.client, schema, { + collections, + }); + + const result = await engine.search(edgedWork as never, { + ...base, + resolve: { + creator: { resolve: { creator: { fields: ['name', 'location'] } } }, + }, + }); + const entries = (result.hits[0].document as Record) + .creator as readonly Record[]; + const endpoints = entries.map((entry) => entry.creator as NestedDocument); + + expect(endpoints.map(typeOf)).toEqual([ + 'Organization', + 'Organization', + 'Person', + ]); + expect(endpoints[0]).toEqual({ + id: 'https://o/1', + name: { nl: ['Zusters Franciscanessen'] }, + location: 'Roermond', + [NESTED_DOCUMENT_TYPE]: 'Organization', + }); + expect(endpoints[1]).toEqual({ + id: 'https://o/gone', + name: { und: ['Drukkerij'] }, + location: 'Venlo', + [NESTED_DOCUMENT_TYPE]: 'Organization', + }); + expect(endpoints[2]).toEqual({ + name: { und: ['Jan Jansen'] }, + [NESTED_DOCUMENT_TYPE]: 'Person', + }); + // The discriminator is a storage detail, never a field. + for (const endpoint of endpoints) { + expect(endpoint).not.toHaveProperty(TARGET_FIELD); + } + }); +}); + +describe('a lookup naming one target', () => { + it('reads nothing for a selection reduced to id', async () => { + const single = defineSearchType({ + ...work, + name: 'SingleWork', + class: 'https://example.org/SingleWork', + fields: [ + { + ...work.fields[1], + ref: { strategy: 'lookup', target: 'Person' }, + }, + ], + }); + const { fake, searches } = client({ + found: 1, + hits: [{ document: { id: 'https://w/3', creator: ['https://p/1'] } }], + }); + const engine = createTypesenseSearchEngine( + fake.client, + searchSchema(single, person), + { collections: { ...collections, SingleWork: 'singles' } }, + ); + + const result = await engine.search(single as never, { + ...base, + resolve: { creator: { fields: [] } }, + }); + + expect(searches).toHaveLength(0); + expect( + (result.hits[0].document as Record).creator, + ).toEqual([{ id: 'https://p/1', label: { nl: ['Frank Koel'] } }]); + }); +}); + +describe('an edge stating no endpoint', () => { + it('reconstructs the entry without one', async () => { + const { fake } = client({ + found: 1, + hits: [ + { document: { id: 'https://w/4', creator: [{ role: 'drukker' }] } }, + ], + }); + const engine = createTypesenseSearchEngine(fake.client, schema, { + collections, + }); + const result = await engine.search(edgedWork as never, base); + expect(result.hits[0].document).toEqual({ creator: [{ role: 'drukker' }] }); + }); +}); + +describe('declaring the collection for a stored referent of several kinds', () => { + it('declares the union of the targets’ fields once, plus the discriminator', () => { + const definition = buildCollectionDefinition(edgedWork, { schema }); + const names = definition.fields?.map((field) => field.name) ?? []; + + expect(names).toContain('creator.creator.id'); + expect(names).toContain('creator.creator.birthDate'); + expect(names).toContain('creator.creator.location'); + expect(names).toContain(`creator.creator.${TARGET_FIELD}`); + // The shared label field is declared once, not once per target. + expect( + names.filter((name) => name === 'creator.creator.name_[^_]+'), + ).toHaveLength(1); + expect( + definition.fields?.find( + (field) => field.name === `creator.creator.${TARGET_FIELD}`, + ), + ).toMatchObject({ type: 'string[]', index: false, optional: true }); + }); + + it('declares a single-valued referent as one object with one discriminator', () => { + const single = defineSearchType({ + ...edgedWork, + name: 'Print', + class: 'https://example.org/Print', + fields: [ + { + name: 'creator', + kind: 'reference', + output: true, + ref: { + strategy: 'lookup', + target: ['Person', 'Organization'], + local: true, + }, + }, + ], + }); + const definition = buildCollectionDefinition(single, { + schema: searchSchema(single, person, organization), + }); + expect( + definition.fields?.find((field) => field.name === 'creator'), + ).toMatchObject({ type: 'object' }); + expect( + definition.fields?.find( + (field) => field.name === `creator.${TARGET_FIELD}`, + ), + ).toMatchObject({ type: 'string' }); + }); + + it('still declares the discriminator where the descent is cut', () => { + // Person reaches back to itself and to the root: both already on the + // path, so the nesting stops – but the projection stores the referent + // as an `{ id, _target }` object all the same. + const knowing = defineSearchType({ + ...person, + fields: [ + ...person.fields, + { + name: 'related', + kind: 'reference', + output: true, + ref: { + strategy: 'lookup', + target: ['Person', 'Painting'], + local: true, + }, + }, + ], + }); + const painting = defineSearchType({ + ...edgedWork, + labelField: 'title', + fields: [ + { + name: 'title', + kind: 'text', + locales: ['nl'], + output: true, + searchable: { weight: 1 }, + }, + ...edgedWork.fields, + ], + }); + const definition = buildCollectionDefinition(painting, { + schema: searchSchema(painting, knowing, organization, creatorEdge), + }); + const names = definition.fields?.map((field) => field.name) ?? []; + expect(names).toContain('creator.creator.related'); + expect(names).toContain(`creator.creator.related.${TARGET_FIELD}`); + }); +}); diff --git a/packages/search-typesense/test/qualified-relation.test.ts b/packages/search-typesense/test/qualified-relation.test.ts index 5cec28ee..b672c0f5 100644 --- a/packages/search-typesense/test/qualified-relation.test.ts +++ b/packages/search-typesense/test/qualified-relation.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest'; import { defineSearchType, searchSchema } from '@lde/search'; -import { labelTargetNameOf } from '@lde/search/adapter'; +import { labelTargetNamesOf } from '@lde/search/adapter'; import { buildCollectionDefinition } from '../src/collection-definition.js'; import { buildSearchParams } from '../src/query-compiler.js'; @@ -765,9 +765,9 @@ describe('a facet policy over the companion', () => { describe('labelling the buckets of an edge’s facet', () => { it('reads the label target through the identity companion', () => { // An inline reference names no label source of its own, so reading only - // `labelSourceNameOf` left its buckets unlabelled – the facet policy was + // `labelSourceNamesOf` left its buckets unlabelled – the facet policy was // inherited one level in, but the labels were not. - expect(labelTargetNameOf(work.fields[0], schema)).toBe('Person'); + expect(labelTargetNamesOf(work.fields[0], schema)).toEqual(['Person']); }); it('names nothing for an inline reference without an identity', () => { @@ -787,11 +787,11 @@ describe('labelling the buckets of an edge’s facet', () => { }); expect( - labelTargetNameOf( + labelTargetNamesOf( displayOnly.fields[0], searchSchema(displayOnly, person, creatorEdge), ), - ).toBeUndefined(); + ).toEqual([]); }); }); diff --git a/packages/search/src/adapter.ts b/packages/search/src/adapter.ts index 3437e99f..bfe312e4 100644 --- a/packages/search/src/adapter.ts +++ b/packages/search/src/adapter.ts @@ -22,7 +22,10 @@ export { isInlineReference, nestedReferenceType, nestedFieldName, - localLookupTypeOf, + localLookupTargetsOf, + referencedTargetsOf, + storedTargetOf, + TARGET_FIELD, identityFieldOf, identityFieldName, referenceFields, @@ -36,8 +39,8 @@ export { OR_KEY, labelFieldOf, labelFieldNameOf, - labelSourceNameOf, - labelTargetNameOf, + labelSourceNamesOf, + labelTargetNamesOf, documentKeyOf, DEFAULT_LABEL_FIELD, isRangeFacet, diff --git a/packages/search/src/engine.ts b/packages/search/src/engine.ts index e9335473..3fca8171 100644 --- a/packages/search/src/engine.ts +++ b/packages/search/src/engine.ts @@ -169,8 +169,26 @@ export type SearchValue = */ export interface NestedDocument { readonly [field: string]: SearchValue | undefined; + /** + * The `name` of the Root Type this document is read through, where the + * field that carries it is a `lookup` naming **several** targets – the + * declaration it was resolved or stored against. Such a lookup serves + * referents of any of its targets under one field, and this is what tells a + * surface which one each referent is, so it can type the value per referent + * instead of claiming one kind for all of them. Absent where there is + * nothing to tell apart: a single-target lookup, or an inline reference’s + * entry. Keyed by a symbol so it can never collide with a declared field + * name. + */ + readonly [NESTED_DOCUMENT_TYPE]?: string; } +/** The key under which a {@link NestedDocument} carries the `name` of the + * Root Type it is read through. */ +export const NESTED_DOCUMENT_TYPE: unique symbol = Symbol( + 'lde.search.nestedDocumentType', +); + /** * A JSON-LD-style language map (`@container: @language`, `@set` arrays); the key * `und` carries untagged (`@none`) values. The surface flattens it to a diff --git a/packages/search/src/index.ts b/packages/search/src/index.ts index 73e5072f..93746734 100644 --- a/packages/search/src/index.ts +++ b/packages/search/src/index.ts @@ -80,6 +80,7 @@ export type { // Engine port + the logical result document returned across it. An engine is // bound to the whole SearchSchema at construction by its adapter factory. +export { NESTED_DOCUMENT_TYPE } from './engine.js'; export type { SearchEngine, SearchResult, diff --git a/packages/search/src/join-graph.ts b/packages/search/src/join-graph.ts index d2def7bd..83e84a0c 100644 --- a/packages/search/src/join-graph.ts +++ b/packages/search/src/join-graph.ts @@ -1,5 +1,5 @@ import { - labelSourceNameOf, + labelSourceNamesOf, referenceFields, type ReferenceField, type RootType, @@ -177,7 +177,9 @@ function declaredEdges( // field, which a Reference Type cannot carry (a nested field is `output` // only). So a joinable edge always has a collection at the far end without // a rule of its own. - const target = byName.get(labelSourceNameOf(field) as string) as RootType; + // One target, never several: `validateSearchType` refuses `joinable` on a + // reference naming more than one. + const target = byName.get(labelSourceNamesOf(field)[0]) as RootType; const claimed = claimedBy.get(target.name); if (claimed !== undefined) { throw new Error( diff --git a/packages/search/src/project.ts b/packages/search/src/project.ts index db242275..3488f283 100644 --- a/packages/search/src/project.ts +++ b/packages/search/src/project.ts @@ -10,21 +10,25 @@ import { displayFieldName, documentKeyOf, fieldNamed, - inheritedFacetKeys, + identityFieldOf, + inheritedFacetPolicies, inlineFramingDepth, irAlias, isAbsoluteIri, isInternalField, isInlineReference, isoToUnixSeconds, - labelSourceNameOf, - localLookupTypeOf, + localLookupTargetsOf, physicalFields, + referencedTargetsOf, referenceTypeNamed, - rootTypeNamed, + storedTargetOf, + TARGET_FIELD, type KeywordField, type ProjectionValue, + type FacetKeys, type ReferenceField, + type ReferenceType, type RootType, type SearchField, type SearchSchema, @@ -143,16 +147,27 @@ function pruneInternalFields( if (schema === undefined || field.kind !== 'reference') { continue; } - const nestedType = + const nestedTypes: readonly SearchType[] = field.ref?.strategy === 'inline' - ? referenceTypeNamed(schema, field.ref.typeName) - : localLookupTypeOf(field, schema); + ? [referenceTypeNamed(schema, field.ref.typeName)].filter( + (type): type is ReferenceType => type !== undefined, + ) + : localLookupTargetsOf(field, schema); const nested = document[field.name]; - if (nestedType === undefined || nested === undefined) { + if (nestedTypes.length === 0 || nested === undefined) { continue; } for (const referent of Array.isArray(nested) ? nested : [nested]) { - pruneInternalFields(referent as ProjectedNode, nestedType, schema); + pruneInternalFields( + referent as ProjectedNode, + nestedTypes.length === 1 + ? nestedTypes[0] + : storedTargetOf( + referent as ProjectedNode, + nestedTypes as readonly RootType[], + ), + schema, + ); } } } @@ -359,13 +374,13 @@ function applyField( // is an entry like any other, minus its `id`. Same reason as above for // needing a schema: without one there is no target to project through. if (field.kind === 'reference' && schema !== undefined) { - const localType = localLookupTypeOf(field, schema); - if (localType !== undefined) { + const localTargets = localLookupTargetsOf(field, schema); + if (localTargets.length > 0) { const endpoints = applyNestedReferents( document, valuesOf(node, alias), field, - localType, + localTargets, schema, context, ); @@ -382,13 +397,12 @@ function applyField( return applyText(document, langValuesOf(node, alias), field); case 'keyword': return applyFacet(document, literalsOf(node, alias), field, schema); - case 'reference': - return applyFacet( - document, - referenceValues(node, alias, field, schema), - field, - schema, - ); + case 'reference': { + const targetsById = new Map(); + const values = referenceValues(node, alias, field, schema, targetsById); + rememberTargets(document, targetsById); + return applyFacet(document, values, field, schema, targetsById); + } case 'integer': return setNumber( document, @@ -451,32 +465,120 @@ function referenceValues( alias: string, field: ReferenceField, schema: SearchSchema | undefined, + targetsById: Map, ): readonly string[] { - const targetName = labelSourceNameOf(field); - if (schema === undefined || targetName === undefined) { - return irisOf(node, alias); - } // Guaranteed declared: `searchSchema` validates that every named target // resolves, and the projection only ever runs against a type of its schema. - const target = rootTypeNamed(schema, targetName) as RootType; - if (target.key === undefined) { + const targets = referencedTargetsOf(field, schema); + if (targets.length === 0) { return irisOf(node, alias); } - const keyFieldName = target.key.field; return valuesOf(node, alias) .map((value) => { const iri = iriString(value); - return iri === undefined - ? undefined - : documentKeyOf( - target, - iri, - isObject(value) ? keyCandidatesOf(value, target, keyFieldName) : [], - ); + if (iri === undefined) { + return undefined; + } + // Each referent is re-keyed through the target IT belongs to – the one + // its `rdf:type` matches – so a person keyed on an authority IRI and an + // organization keyed on nothing store, under one field, the ids their + // own collections file them under. + const target = targetOfReferent(value, targets); + const keyed = + target.key === undefined + ? iri + : documentKeyOf( + target, + iri, + isObject(value) + ? keyCandidatesOf(value, target, target.key.field) + : [], + ); + // Remembered only where there was a choice: a single-target reference + // stays byte-identical to what it was, symbol keys included. Keyed by + // what the field STORES – the value after its own `transform`, which + // `applyFacet` applies before it reads this map. + if (keyed !== undefined && targets.length > 1) { + targetsById.set( + field.transform === undefined ? keyed : field.transform(keyed), + target, + ); + } + return keyed; }) .filter((value): value is string => value !== undefined); } +/** + * The Root Type a framed referent belongs to, among the targets its reference + * names: the first whose `class` is among the referent’s `rdf:type`s, which + * the extraction emits and framing carries as `@type`; else the first target + * declared. One target needs no evidence at all. This is the one reading of + * *which of several targets is this*, so re-keying, the facet policy and the + * stored discriminator ({@link TARGET_FIELD}) cannot disagree. + */ +function targetOfReferent( + value: unknown, + targets: readonly RootType[], +): RootType { + if (targets.length === 1 || !isObject(value)) { + return targets[0]; + } + const raw = value['@type']; + const classes = new Set( + (Array.isArray(raw) ? raw : [raw]).filter( + (type): type is string => typeof type === 'string', + ), + ); + return targets.find((target) => classes.has(target.class)) ?? targets[0]; +} + +/** + * The targets the reference values of a projected node were resolved through, + * by id – kept beside the node under a symbol, so it never reaches the + * writer, for the one reader that needs it after projection: the identity + * companion of an inline reference, which admits an entry’s id to a facet by + * the policy of the target that id belongs to. Only a reference naming + * several targets has anything to remember. + */ +const REFERENT_TARGETS: unique symbol = Symbol('referentTargets'); + +type WithReferentTargets = ProjectedNode & { + [REFERENT_TARGETS]?: Map; +}; + +function rememberTargets( + document: ProjectedNode, + targetsById: ReadonlyMap, +): void { + if (targetsById.size === 0) { + return; + } + const node = document as WithReferentTargets; + const remembered = (node[REFERENT_TARGETS] ??= new Map()); + for (const [id, target] of targetsById) { + remembered.set(id, target); + } +} + +/** + * Whether an id deserves a facet bucket, by the {@link FacetKeys facet policy} + * of the target it belongs to: admitted where that target declares none, and + * by its `only` where it does. `targetName` is the target the id was resolved + * or stored against; an id whose target is unknown falls back to the first + * declared, the same precedence every other reading of several targets uses. + */ +function admitsFacet( + policies: ReadonlyMap, + targets: readonly RootType[], + targetName: string | undefined, + id: string, +): boolean { + // Policies are inherited from targets, so where any exists a target does. + const policy = policies.get(targetName ?? targets[0].name); + return policy === undefined || policy.only(id); +} + /** * Project a text field. **Display** (when `output`) preserves *every* language * present – one label per language, or every label of it for an `array` field @@ -575,7 +677,7 @@ function foldedSearchValue(values: readonly string[]): string { * already-read raw values). * * A reference inheriting a {@link FacetKeys facet policy} from the type it - * names ({@link inheritedFacetKeys}) also writes the `${name}_facet` companion + * names ({@link inheritedFacetPolicies}) also writes the `${name}_facet` companion * the engine facets instead of the field: the subset of **what the field * stores** that the policy admits. So it is written after the field’s own * `transform` and the IRI filter, from the same `values` – for a keyed target @@ -599,6 +701,7 @@ function applyFacet( raw: readonly string[], field: KeywordField | ReferenceField, schema: SearchSchema | undefined, + targetsById?: ReadonlyMap, ): void { // A `reference` stores identity, so what it stores must be an IRI whatever // route the value arrived by. {@link iriString} guards the graph path, but a @@ -613,21 +716,27 @@ function applyFacet( const folded = dedupe(values.map((value) => fold(value))); const names = physicalFields(field, schema); const searchField = names.search[0]; - const policy = inheritedFacetKeys(field, schema); + const policies = inheritedFacetPolicies(field, schema); + const inherited = policies.size > 0; // `names.facet` is the companion exactly when a policy is inherited – - // physicalFields reads the same `inheritedFacetKeys`. + // physicalFields reads the same `inheritedFacetPolicies`. const facetField = names.facet as string; // The companion is a subset of what the field STORES – for a single-valued // field its first value, not the first admitted of all of them, or the facet // would count a value no filter on the field can reproduce. const stored = field.array === true ? values : values.slice(0, 1); - const admitted = policy === undefined ? [] : stored.filter(policy.only); + const targets = referencedTargetsOf(field, schema); + const admitted = !inherited + ? [] + : stored.filter((id) => + admitsFacet(policies, targets, targetsById?.get(id)?.name, id), + ); if (field.array === true) { setArray(document, field.name, stored); if (field.searchable) { setArray(document, searchField, folded); } - if (policy !== undefined) { + if (inherited) { setArray(document, facetField, admitted); } return; @@ -636,7 +745,7 @@ function applyFacet( if (field.searchable) { setString(document, searchField, folded[0]); } - if (policy !== undefined) { + if (inherited) { setString(document, facetField, admitted[0]); } } @@ -707,7 +816,7 @@ function applyInlineReference( document, valuesOf(node, alias), field, - referenceType, + [referenceType], schema, context, ); @@ -746,16 +855,27 @@ function applyIdentityCompanion( // keeps the first referent and drops the rest, and a companion holding an id // from a dropped one would match a filter whose hit shows no such entry. const stored = field.array === true ? referents : referents.slice(0, 1); - const ids = dedupe( - stored.flatMap((referent) => - (Array.isArray(referent[identity]) - ? referent[identity] - : [referent[identity]] - ) - .map(identityValue) - .filter((id): id is string => id !== undefined), - ), - ); + // Each id with the target it was resolved through, for the facet policy + // below: a stored endpoint says so itself ({@link TARGET_FIELD}), a bare id + // was remembered when the entry was projected ({@link rememberTargets}). + const targetById = new Map(); + for (const referent of stored) { + const remembered = (referent as WithReferentTargets)[REFERENT_TARGETS]; + const raw = referent[identity]; + for (const value of Array.isArray(raw) ? raw : [raw]) { + const id = identityValue(value); + if (id === undefined || targetById.has(id)) { + continue; + } + targetById.set( + id, + isObject(value) && typeof value[TARGET_FIELD] === 'string' + ? value[TARGET_FIELD] + : remembered?.get(id)?.name, + ); + } + } + const ids = [...targetById.keys()]; if (ids.length === 0) { return; } @@ -763,12 +883,18 @@ function applyIdentityCompanion( // Same rule as every other facetable reference: where the target declares a // facet policy, the facet reads a narrowed companion of its own, so an // excluded id is never seen by the engine rather than merely unlabelled. - const policy = inheritedFacetKeys(field, schema); - if (policy !== undefined) { + const policies = inheritedFacetPolicies(field, schema); + if (policies.size > 0) { + const targets = referencedTargetsOf( + identityFieldOf(field, schema) as SearchField, + schema, + ); setIdentity( document, names.facet as string, - ids.filter(policy.only), + ids.filter((id) => + admitsFacet(policies, targets, targetById.get(id), id), + ), field, nested, ); @@ -871,7 +997,7 @@ function applyNestedReferents( document: ProjectedNode, values: readonly unknown[], field: ReferenceField, - nestedType: SearchType, + nestedTypes: readonly SearchType[], schema: SearchSchema, context: ProjectionContext, ): readonly ProjectedNode[] { @@ -880,8 +1006,28 @@ function applyNestedReferents( // one entry per combination BEFORE it is projected (ADR 26). const referents = values .filter(isObject) - .flatMap((value) => tuplesOf(value, nestedType, field)) - .map((tuple) => projectFields(tuple, nestedType, schema, context, true)) + .flatMap((value) => { + // A lookup naming several targets projects each referent through the + // one it belongs to, and stores which – so an engine adapter can read + // an entry no collection answers for through the right declaration. + const nestedType = + nestedTypes.length === 1 + ? nestedTypes[0] + : targetOfReferent(value, nestedTypes as readonly RootType[]); + return tuplesOf(value, nestedType, field).map((tuple) => { + const projected = projectFields( + tuple, + nestedType, + schema, + context, + true, + ); + if (nestedTypes.length > 1 && Object.keys(projected).length > 0) { + projected[TARGET_FIELD] = nestedType.name; + } + return projected; + }); + }) // Fields, not identity, are what makes something a referent: a literal // value object under the alias (dirty source data), or a node this // reference type reads nothing from, projects nothing and is no referent. @@ -900,7 +1046,7 @@ function applyNestedReferents( // `id` is its key and stays part of what makes an entry distinct. const distinct = dedupeBy(referents, (referent) => JSON.stringify( - nestedType.class === undefined + nestedTypes[0].class === undefined ? { ...referent, id: undefined } : referent, ), diff --git a/packages/search/src/query.ts b/packages/search/src/query.ts index 43917281..3c6666c2 100644 --- a/packages/search/src/query.ts +++ b/packages/search/src/query.ts @@ -4,6 +4,8 @@ import { ID_FIELD, isoToUnixSeconds, nestedReferenceType, + referencedTargetsOf, + type SearchField, type SearchSchema, type SearchType, } from './schema.js'; @@ -535,7 +537,7 @@ export function validateQuery( issues.push({ part: 'facets', field: name, reason: 'not-facetable' }); } } - collectProjectionIssues(query.resolve, searchType, schema, issues); + collectProjectionIssues(query.resolve, [searchType], schema, issues); for (const sort of query.orderBy) { if ( sort.field !== 'relevance' && @@ -568,40 +570,61 @@ export function validateQuery( */ function collectProjectionIssues( projection: ReferenceProjection | undefined, - searchType: SearchType, + searchTypes: readonly SearchType[], schema: SearchSchema, issues: QueryIssue[], ): void { for (const [name, level] of Object.entries(projection ?? {})) { - const field = fieldNamed(searchType, name); - if (field === undefined) { + // A level is read against every type it may be reached through: a lookup + // naming several targets resolves a referent of any of them, so the level + // below is valid wherever ANY of those targets serves it – a field one + // target declares and another does not is exactly what a polymorphic + // selection asks for. + const fields = searchTypes + .map((searchType) => fieldNamed(searchType, name)) + .filter((field): field is SearchField => field !== undefined); + if (fields.length === 0) { issues.push({ part: 'resolve', field: name, reason: 'unknown-field' }); continue; } // Descending into an inline reference is free, so a nested level is valid // wherever the entries exist – what it is *for* is the lookup below it. - const nested = nestedReferenceType(schema, field); - if (nested !== undefined) { + // `searchSchema` holds the targets of one lookup to one declaration per + // name, so a name is nested or a lookup, never both; each is still read + // on its own here rather than one deciding for the other. + const nested: SearchType[] = []; + for (const field of fields) { + const referenceType = nestedReferenceType(schema, field); + if (referenceType !== undefined) { + nested.push(referenceType); + } + } + if (nested.length > 0) { collectProjectionIssues(level.resolve, nested, schema, issues); - continue; } - if (field.kind !== 'reference' || field.ref?.strategy !== 'lookup') { - issues.push({ part: 'resolve', field: name, reason: 'not-resolvable' }); + const lookups = fields.filter( + (field) => field.kind === 'reference' && field.ref?.strategy === 'lookup', + ); + if (lookups.length === 0) { + if (nested.length === 0) { + issues.push({ part: 'resolve', field: name, reason: 'not-resolvable' }); + } continue; } - const targetName = field.ref.target; - const target = [...schema.values()].find( - (rootType) => rootType.name === targetName, + const targets = lookups.flatMap((field) => + referencedTargetsOf(field, schema), ); - if (target === undefined) { + if (targets.length === 0) { // searchSchema rejects a lookup whose target it cannot resolve, so this // is a query built against a different schema than the engine serves. issues.push({ part: 'resolve', field: name, reason: 'not-resolvable' }); continue; } for (const wanted of level.fields ?? []) { - const targetField = fieldNamed(target, wanted); - if (targetField === undefined || targetField.output !== true) { + const served = targets.some( + (target) => fieldNamed(target, wanted)?.output === true, + ); + if (!served) { issues.push({ part: 'resolve', field: `${name}.${wanted}`, @@ -609,7 +632,7 @@ function collectProjectionIssues( }); } } - collectProjectionIssues(level.resolve, target, schema, issues); + collectProjectionIssues(level.resolve, targets, schema, issues); } } diff --git a/packages/search/src/schema.ts b/packages/search/src/schema.ts index bf184da9..fed4dca1 100644 --- a/packages/search/src/schema.ts +++ b/packages/search/src/schema.ts @@ -236,7 +236,19 @@ export type ReferenceStrategy = } | { readonly strategy: 'lookup'; - readonly target: string; + /** + * The Root Type whose collection the fields are read from – or + * **several**, where the referent may be any of them: a `creator` that + * is a `Person` or an `Organization`. Each value then resolves against + * every named collection and reports which one held it, so a surface can + * type it per value rather than claiming one kind for all of them. + * + * Order is precedence. A value that several collections hold – two Root + * Types whose `class` selections overlap – belongs to the first target + * declared, and so does a stored referent whose `rdf:type` matches none + * of them. List the most specific type first. + */ + readonly target: string | readonly string[]; /** * Also project the target’s **own fields from this document’s frame**, so * the reference stores what the referring document states about the @@ -287,7 +299,7 @@ export type ReferenceStrategy = * identifies the edge. Naming it is also what gives the companion a * **target**: the named field’s own `target` is the Root Type whose keys * these ids are, which is what a facet policy is inherited through - * ({@link inheritedFacetKeys}) and what types the filter at the surface. + * ({@link inheritedFacetPolicies}) and what types the filter at the surface. * * The companion holds ids only, so a facet over it is exact and * identity-keyed: an entry whose endpoint the graph named inline – a @@ -310,20 +322,22 @@ export interface ReferenceField extends SearchFieldBase, Searchable { readonly from?: ProjectionValue; /** * The `name` of the Root Type whose collection labels this reference’s facet - * buckets. Only an `idOnly` reference declares one: a `lookup` reads its - * labels from the `target` it already names, and an `inline` reference - * carries the referent’s own fields. The named type must declare an `output`, + * buckets – or several, in order of precedence, where the referent may be + * any of them (the same range a `lookup`’s `target` states). Only an + * `idOnly` reference declares one: a `lookup` reads its labels from the + * `target` it already names, and an `inline` reference carries the + * referent’s own fields. Each named type must declare an `output`, * `searchable` text field under its {@link SearchTypeBase.labelField} name * (`label` by default; validated by {@link searchSchema}), so an engine can * both reconstruct the label and search it (typeahead). */ - readonly labelSource?: string; + readonly labelSource?: string | readonly string[]; /** * Turn this reference into an **engine-level join**, so a query can filter * this type by a condition on the referent – `“every object published by * institution X”` in one round-trip instead of two. Valid only where the * reference names the type it resolves against – a `lookup`’s `target` or an - * `idOnly`’s {@link ReferenceField.labelSource} ({@link labelSourceNameOf}) – + * `idOnly`’s {@link ReferenceField.labelSource} ({@link labelSourceNamesOf}) – * which already asserts that this field’s values are ids of documents in that * type’s collection, exactly the fact a join needs. * @@ -474,7 +488,7 @@ export interface KeyField { * A {@link RootType}’s **facet policy**: which of its documents get a facet * bucket, as a predicate over the document key. Declared once, on the type, * and inherited by every facetable reference that *names* the type – a - * `lookup`’s `target`, an `idOnly`’s `labelSource` ({@link labelSourceNameOf}) + * `lookup`’s `target`, an `idOnly`’s `labelSource` ({@link labelSourceNamesOf}) * – because *which of a type’s ids deserve a bucket* is a fact about the type, * not about each field that points at it. A per-field policy would be one rule * declared N times, and forgetting one would silently reintroduce the buckets @@ -725,7 +739,7 @@ export function referenceTypeNamed( /** * The {@link RootType} a declaration names – a `lookup`’s `target`, an - * `idOnly`’s {@link ReferenceField.labelSource} ({@link labelSourceNameOf}), a + * `idOnly`’s {@link ReferenceField.labelSource} ({@link labelSourceNamesOf}), a * join edge – or `undefined` when the schema declares no Root Type by that * name. The one reading of *which type does this point at*, so the projection * (which re-keys a reference through its target’s {@link KeyField}), the @@ -880,35 +894,43 @@ function framingReach( // the cut as “reach 0, and nothing else to count” left the innermost // referent’s key one hop outside the frame, so it stored a node IRI that // matches nothing in the target’s collection. - const local = localLookupTypeOf(field, schema); - if (local !== undefined && !onPath.has(local.name)) { - furthest = Math.max(furthest, hops + framingReach(schema, local, onPath)); - continue; + // + // Several targets each contribute their own reach: a referent is framed + // once, and the frame has to hold whichever declaration it turns out to + // match, so the furthest of them decides. + // The field’s own traversal counts whatever it names. + furthest = Math.max(furthest, hops); + const local = localLookupTargetsOf(field, schema).filter( + (target) => !onPath.has(target.name), + ); + for (const target of local) { + furthest = Math.max( + furthest, + hops + framingReach(schema, target, onPath), + ); } // A keyed target’s key field is itself path-bearing, so its own traversal // counts too – the extraction reads it off the referent with that path. - const keyPath = keyedTargetKeyPath(field, schema); - furthest = Math.max( - furthest, - hops + (keyPath === undefined ? 0 : pathHopCount(keyPath)), - ); + // Counted for every target that is not locally expanded above, cut ones + // included: there the extraction falls back to exactly this hop. + for (const target of referencedTargetsOf(field, schema)) { + if (local.includes(target)) { + continue; + } + const keyPath = keyPathOf(target); + furthest = Math.max( + furthest, + hops + (keyPath === undefined ? 0 : pathHopCount(keyPath)), + ); + } } return furthest; } -/** The `path` of the key field of the keyed Root Type a reference names, when - * it names one that declares a {@link RootType.key}. */ -function keyedTargetKeyPath( - field: SearchField, - schema: SearchSchema, -): string | undefined { - if (field.kind !== 'reference') { - return undefined; - } - const targetName = labelSourceNameOf(field); - const target = - targetName === undefined ? undefined : rootTypeNamed(schema, targetName); - if (target?.key === undefined) { +/** The `path` of a Root Type’s key field, when it declares a + * {@link RootType.key}. */ +function keyPathOf(target: RootType): string | undefined { + if (target.key === undefined) { return undefined; } // `searchSchema` guarantees a key field that is declared and path-bearing – @@ -1058,7 +1080,7 @@ function assertIdentityCompanion( } if ( identityField.kind !== 'reference' || - labelSourceNameOf(identityField) === undefined + labelSourceNamesOf(identityField).length === 0 ) { throw new Error( `${where} names identity “${identity}”, which names no target: an identity companion holds ids of documents in a collection, so the field it harvests must be a reference declaring a “lookup” target or a label source.`, @@ -1329,15 +1351,41 @@ export function labelFieldOf(searchType: SearchType): TextField | undefined { } /** - * The Root Type a reference resolves labels from, by name: a `lookup`’s - * `target`, an `idOnly`’s {@link ReferenceField.labelSource}, or `undefined` - * when it resolves none. One reading for the two declarations, so a consumer - * never branches on the strategy to find the collection. + * The Root Types a reference resolves labels from, by name and in order of + * precedence: a `lookup`’s `target`, an `idOnly`’s + * {@link ReferenceField.labelSource}, or none when it resolves none. One + * reading for the two declarations, so a consumer never branches on the + * strategy to find the collection – and one **shape** for the one-target and + * the several-target declaration, so a consumer never branches on that either: + * a single name is the list of one. + */ +export function labelSourceNamesOf(field: ReferenceField): readonly string[] { + const declared = + field.ref?.strategy === 'lookup' ? field.ref.target : field.labelSource; + return declared === undefined + ? [] + : typeof declared === 'string' + ? [declared] + : declared; +} + +/** + * The Root Types a reference resolves against ({@link labelSourceNamesOf}), + * resolved in the given schema and in order of precedence. A name the schema + * does not declare as a Root Type is skipped – `searchSchema` rejects such a + * declaration, so this only happens for a field read against a foreign schema, + * where it resolves what it can. */ -export function labelSourceNameOf(field: ReferenceField): string | undefined { - return field.ref?.strategy === 'lookup' - ? field.ref.target - : field.labelSource; +export function referencedTargetsOf( + field: SearchField, + schema: SearchSchema | undefined, +): readonly RootType[] { + if (schema === undefined || field.kind !== 'reference') { + return []; + } + return labelSourceNamesOf(field) + .map((name) => rootTypeNamed(schema, name)) + .filter((target): target is RootType => target !== undefined); } /** @@ -1380,34 +1428,117 @@ function assertResolvableLabelSources( ); } } - const sourceName = - field.kind === 'reference' && field.ref?.strategy === 'lookup' - ? field.ref.target - : labelSource; - if (sourceName === undefined) { + if (field.kind !== 'reference') { continue; } - const source = rootTypeNamed(schema, sourceName); - if (source === undefined) { - // A name that IS declared, just not as a Root Type, is the confusing - // case: telling the author to declare a type they already declared - // would send them looking in the wrong place. Only a Root Type has a - // collection to resolve against, so name that instead. - throw new Error( - referenceTypeNamed(schema, sourceName) === undefined - ? `Reference “${searchType.name}.${field.name}” names unknown label source “${sourceName}”; declare a SearchType with that name.` - : `Reference “${searchType.name}.${field.name}” names label source “${sourceName}”, which is a Reference Type; a label source must be a Root Type, since a resolved label is read from that type’s own collection.`, - ); + const sources: RootType[] = []; + for (const sourceName of labelSourceNamesOf(field)) { + const source = rootTypeNamed(schema, sourceName); + if (source === undefined) { + // A name that IS declared, just not as a Root Type, is the confusing + // case: telling the author to declare a type they already declared + // would send them looking in the wrong place. Only a Root Type has a + // collection to resolve against, so name that instead. + throw new Error( + referenceTypeNamed(schema, sourceName) === undefined + ? `Reference “${searchType.name}.${field.name}” names unknown label source “${sourceName}”; declare a SearchType with that name.` + : `Reference “${searchType.name}.${field.name}” names label source “${sourceName}”, which is a Reference Type; a label source must be a Root Type, since a resolved label is read from that type’s own collection.`, + ); + } + if (labelFieldOf(source) === undefined) { + throw new Error( + `Reference “${searchType.name}.${field.name}” uses label source “${sourceName}”, which must declare an output, searchable text field “${labelFieldNameOf(source)}”.`, + ); + } + sources.push(source); } - if (labelFieldOf(source) === undefined) { + assertCompatibleTargets(searchType, field, sources); + } + } +} + +/** + * The targets of a reference naming **several** may not disagree about a + * field they both declare: a stored referent of either kind lands in one + * nested object, which an engine declares once, so `birthDate` cannot be a + * `date` on one target and a `keyword` on the other, nor a list on one and a + * single value on the other – nor an id on one and a nested document on the + * other, nor indexed on one and stored only on the other. Everything that + * decides a nested field’s physical shape has to agree: kind, arity, the + * Roles it opts into, and for a reference the strategy and what it points at. + * A field only one target declares is fine – the other simply never fills it + * – and so is a `text` field whose locales differ, since display stores every + * present language regardless. + */ +function assertCompatibleTargets( + searchType: SearchType, + field: ReferenceField, + targets: readonly RootType[], +): void { + const seen = new Map(); + for (const target of targets) { + for (const declared of target.fields) { + const earlier = seen.get(declared.name); + if (earlier === undefined) { + seen.set(declared.name, { target, field: declared }); + continue; + } + if (describeShape(earlier.field) !== describeShape(declared)) { throw new Error( - `Reference “${searchType.name}.${field.name}” uses label source “${sourceName}”, which must declare an output, searchable text field “${labelFieldNameOf(source)}”.`, + `Reference “${searchType.name}.${field.name}” names targets “${earlier.target.name}” and “${target.name}”, which both declare “${declared.name}” but not alike (${describeShape(earlier.field)} vs ${describeShape(declared)}); a referent of either kind is stored in one shape, so a field the targets share must be declared the same on both.`, ); } } } } +/** + * Everything that decides a field’s physical shape, spelled out for a message + * comparing two declarations: kind and arity, the Roles it carries, and for a + * reference its strategy, what it points at, and whether it stores a copy. + */ +function describeShape(field: SearchField): string { + const roles: string[] = ['filterable', 'facetable', 'sortable'].filter( + (role) => field[role as 'filterable' | 'facetable' | 'sortable'] === true, + ); + if (field.searchable !== undefined) { + roles.push('searchable'); + } + const ref = + field.kind === 'reference' && field.ref !== undefined + ? [ + field.ref.strategy, + ...labelSourceNamesOf(field), + ...(field.ref.strategy === 'inline' ? [field.ref.typeName] : []), + ...(field.ref.strategy === 'lookup' && field.ref.local === true + ? ['local'] + : []), + ] + : []; + return [ + field.array === true ? `${field.kind} list` : field.kind, + ...(ref.length > 0 ? [`→ ${ref.join(' ')}`] : []), + ...(roles.length > 0 ? [`(${roles.join(', ')})`] : []), + ].join(' '); +} + +/** + * The Root Type a stored referent was projected through, among the targets + * its lookup names: the one its discriminator names ({@link TARGET_FIELD}), + * or the first declared where it carries none – a single-target lookup stores + * none, and the first target is also what an unmatched referent was projected + * through. The one reading of a stored discriminator, so the projection + * (which prunes through it) and an engine adapter (which reconstructs through + * it) cannot read it differently. + */ +export function storedTargetOf( + entry: Readonly>, + targets: readonly RootType[], +): RootType { + const name = entry[TARGET_FIELD]; + return targets.find((target) => target.name === name) ?? targets[0]; +} + /** * One structural problem {@link validateSearchType} found: a field declares a * capability or property its `kind` cannot honour, or the declaration is @@ -1441,6 +1572,8 @@ export interface SearchTypeIssue { | 'joinable-not-allowed' | 'joinable-without-label-source' | 'joinable-with-inline-ref' + | 'joinable-with-several-targets' + | 'duplicate-target' | 'reserved-field-name' | 'key-field-unknown' | 'key-field-not-reference' @@ -1484,9 +1617,29 @@ export const ID_FIELD = 'id'; export const AND_KEY = 'and'; export const OR_KEY = 'or'; +/** + * The physical name under which a stored referent of a lookup naming + * **several** targets records which of them it belongs to: the `name` of the + * Root Type whose `class` its `rdf:type` matched first, or the first target + * declared where none did. Written by the projection, read by an engine + * adapter to reconstruct the referent through the right declaration when no + * collection answers for it – an unidentified referent, or an identified one + * whose document is not indexed. A single-target lookup stores none: there is + * nothing to tell apart. + * + * Reserved schema-wide ({@link validateSearchType} rejects a field of this + * name), so it can never shadow a declared field of a stored entry. + */ +export const TARGET_FIELD = '_target'; + /** The logical field names no {@link SearchType} may declare, each because a * surface already gives that name a meaning of its own. */ -const RESERVED_FIELD_NAMES: readonly string[] = [ID_FIELD, AND_KEY, OR_KEY]; +const RESERVED_FIELD_NAMES: readonly string[] = [ + ID_FIELD, + AND_KEY, + OR_KEY, + TARGET_FIELD, +]; /** Kinds that can feed full-text search (project a folded search field). */ const SEARCHABLE_KINDS: readonly FieldKind[] = ['text', 'keyword', 'reference']; @@ -1622,19 +1775,31 @@ export function validateSearchType( field.output === true && ((field.ref?.strategy === 'inline' && field.ref.typeName === undefined) || - (field.ref?.strategy === 'lookup' && field.ref.target === undefined)) + (field.ref?.strategy === 'lookup' && + labelSourceNamesOf(field as ReferenceField).length === 0)) ) { issue('missing-ref-type-name'); } // A join addresses the referent's collection – the one a lookup's // `target` or an idOnly's `labelSource` names. With neither, the flag // states an edge to nowhere. - if ( - field.joinable === true && - labelSourceNameOf(field as ReferenceField) === undefined - ) { + const targets = labelSourceNamesOf(field as ReferenceField); + if (field.joinable === true && targets.length === 0) { issue('joinable-without-label-source'); } + // An engine reference names ONE collection, and ids that live in + // several have no single collection to reference – so a reference whose + // referent may be any of several types can never be an engine edge. Its + // labels, facets and id filters all work from this document; a + // condition on the referent’s own fields does not (#717). + if (field.joinable === true && targets.length > 1) { + issue('joinable-with-several-targets'); + } + // Precedence between targets is declaration order, so a name declared + // twice would have to mean two different ranks. + if (new Set(targets).size !== targets.length) { + issue('duplicate-target'); + } // An inline reference is stored as a NESTED OBJECT, not as an id an // engine can point a reference field at, so the two cannot both hold: // the collection definition would emit the nesting and silently drop the @@ -1895,46 +2060,48 @@ export function identityFieldName(name: string): string { } /** - * The Root Type whose labels a reference’s **facet buckets** read – its own - * label source, or, for an inline reference, the one its - * {@link ReferenceStrategy.identity identity companion} points at. + * The Root Types whose labels a reference’s **facet buckets** read, by name and + * in order of precedence – its own label sources, or, for an inline reference, + * those its {@link ReferenceStrategy.identity identity companion} points at. + * Empty for a field that labels no bucket. * - * The same one-level-in reading {@link inheritedFacetKeys} makes, and for the - * same reason: the companion holds that field’s ids, so the type that names - * those ids is the type that can label them. Kept together with it so a facet - * cannot inherit a policy from one type and its labels from another – or, as - * happened first, inherit the policy and no labels at all. + * The same one-level-in reading {@link inheritedFacetPolicies} makes, and for + * the same reason: the companion holds that field’s ids, so the types that + * name those ids are the types that can label them. Kept together with it so a + * facet cannot inherit a policy from one type and its labels from another – + * or, as happened first, inherit the policy and no labels at all. */ -export function labelTargetNameOf( +export function labelTargetNamesOf( field: SearchField, schema: SearchSchema | undefined, -): string | undefined { +): readonly string[] { if (field.kind !== 'reference') { - return undefined; + return []; } - return labelSourceNameOf(identityFieldOf(field, schema) ?? field); + return labelSourceNamesOf(identityFieldOf(field, schema) ?? field); } /** - * The Root Type a {@link ReferenceStrategy.local local} lookup projects its - * referents through, or `undefined` for every other field. Such a reference - * stores nested documents shaped by the **target’s own declaration** – so it - * needs no reference type of its own, and reconstructs through the same path a - * resolved referent does. + * The Root Types a {@link ReferenceStrategy.local local} lookup projects its + * referents through, in order of precedence – or none for every other field. + * Such a reference stores nested documents shaped by the **target’s own + * declaration** – so it needs no reference type of its own, and reconstructs + * through the same path a resolved referent does. With several targets, each + * referent is projected through the one its `rdf:type` matches + * ({@link TARGET_FIELD}). */ -export function localLookupTypeOf( +export function localLookupTargetsOf( field: SearchField, schema: SearchSchema | undefined, -): RootType | undefined { +): readonly RootType[] { if ( - schema === undefined || field.kind !== 'reference' || field.ref?.strategy !== 'lookup' || field.ref.local !== true ) { - return undefined; + return []; } - return rootTypeNamed(schema, field.ref.target); + return referencedTargetsOf(field, schema); } /** @@ -1942,7 +2109,7 @@ export function localLookupTypeOf( * {@link ReferenceStrategy.identity identity companion} harvests, or * `undefined` where the field declares none (or the schema cannot resolve its * reference type). The one reading of *which nested field identifies the edge*, - * so the projection (which harvests it), {@link inheritedFacetKeys} (which + * so the projection (which harvests it), {@link inheritedFacetPolicies} (which * reads its target’s facet policy) and an adapter’s filter compiler cannot * resolve it differently. */ @@ -2318,7 +2485,7 @@ export function unixSecondsToIso(seconds: number): string { * The facet is the one member that depends on more than the declaration: a * facetable reference whose target declares a {@link FacetKeys facet policy} * facets a `${name}_facet` companion rather than itself, and only the `schema` - * can resolve that target ({@link inheritedFacetKeys}). Without one the field + * can resolve that target ({@link inheritedFacetPolicies}). Without one the field * facets on its own name – the same reading the projection makes without a * schema, where it cannot re-key a reference either. */ @@ -2348,7 +2515,7 @@ export function physicalFields( // otherwise a level further in than an engine can weld a condition to. const nestsAnObject = identityFieldOf(field, schema) !== undefined || - (localLookupTypeOf(field, schema) !== undefined && + (localLookupTargetsOf(field, schema).length > 0 && field.filterable === true); const identity = nestsAnObject ? identityFieldName(field.name) : undefined; const filtered = identity ?? field.name; @@ -2358,7 +2525,7 @@ export function physicalFields( facet: field.facetable !== true ? undefined - : inheritedFacetKeys(field, schema) === undefined + : inheritedFacetPolicies(field, schema).size === 0 ? filtered : `${filtered}_facet`, identity, @@ -2366,9 +2533,13 @@ export function physicalFields( } /** - * The {@link FacetKeys facet policy} a field inherits: the `facetKeys` of the - * Root Type it names ({@link labelSourceNameOf}), when the field is a facetable - * reference and the schema resolves that type. The boundary is *naming the + * The {@link FacetKeys facet policies} a field inherits, keyed by the `name` of + * the Root Type declaring each: the `facetKeys` of every type the field names + * ({@link labelSourceNamesOf}) that declares one, when the field is a facetable + * reference and the schema resolves those types. Empty where nothing is + * inherited. With several targets, an id is admitted to a bucket by the policy + * of the target it belongs to, and unconditionally where that target declares + * none. The boundary is *naming the * target* – the same line along which a reference is re-keyed and a join is * drawn – so a reference that names no type inherits nothing, whatever type * its values happen to point at; and a `derive`d reference, which produces its @@ -2384,21 +2555,24 @@ export function physicalFields( * rather than declaring it twice is what keeps *which of a type’s ids deserve * a bucket* a fact about the type. */ -export function inheritedFacetKeys( +export function inheritedFacetPolicies( field: SearchField, schema: SearchSchema | undefined, -): FacetKeys | undefined { +): ReadonlyMap { + const policies = new Map(); if ( schema === undefined || field.kind !== 'reference' || field.facetable !== true || field.derive !== undefined ) { - return undefined; + return policies; } const identity = identityFieldOf(field, schema); - const targetName = labelSourceNameOf(identity ?? field); - return targetName === undefined - ? undefined - : rootTypeNamed(schema, targetName)?.facetKeys; + for (const target of referencedTargetsOf(identity ?? field, schema)) { + if (target.facetKeys !== undefined) { + policies.set(target.name, target.facetKeys); + } + } + return policies; } diff --git a/packages/search/src/testing.ts b/packages/search/src/testing.ts index 723a933f..d6df1bda 100644 --- a/packages/search/src/testing.ts +++ b/packages/search/src/testing.ts @@ -11,7 +11,9 @@ import { facetableFields, filterableFields, ID_FIELD, - inheritedFacetKeys, + identityFieldOf, + inheritedFacetPolicies, + referencedTargetsOf, nestedReferenceType, outputFields, type RootType, @@ -269,8 +271,19 @@ export function describeSearchEngineContract( // second assertion bites. for (const searchType of types()) { for (const field of facetableFields(searchType)) { - const policy = inheritedFacetKeys(field, engine().schema); - if (policy === undefined) { + // A bucket keyed on an id of one target is admitted by that + // target’s policy, and a target declaring none admits every one of + // its ids. Which target a bucket belongs to is not visible here, so + // the contract is checked only where every target constrains: then + // a bucket failing every policy belongs to no admitting target. + const policies = [ + ...inheritedFacetPolicies(field, engine().schema).values(), + ]; + const targets = referencedTargetsOf( + identityFieldOf(field, engine().schema) ?? field, + engine().schema, + ); + if (policies.length === 0 || policies.length < targets.length) { continue; } const result = await engine().search(searchType, { @@ -280,7 +293,9 @@ export function describeSearchEngineContract( }); const buckets = result.facets[field.name] ?? []; for (const bucket of buckets) { - expect(policy.only(bucket.value)).toBe(true); + expect(policies.some((policy) => policy.only(bucket.value))).toBe( + true, + ); expect(bucket.label).toBeDefined(); } if (field.output !== true || field.filterable !== true) { @@ -288,7 +303,7 @@ export function describeSearchEngineContract( } const excluded = result.hits .flatMap((hit) => referenceIds(hit.document[field.name])) - .find((id) => !policy.only(id)); + .find((id) => !policies.some((policy) => policy.only(id))); if (excluded === undefined) { continue; } diff --git a/packages/search/test/polymorphic-lookup.test.ts b/packages/search/test/polymorphic-lookup.test.ts new file mode 100644 index 00000000..88d50ec1 --- /dev/null +++ b/packages/search/test/polymorphic-lookup.test.ts @@ -0,0 +1,641 @@ +import { describe, expect, it } from 'vitest'; +import { projectDocument, type SearchDocument } from '../src/project.js'; +import { + defineSearchType, + inheritedFacetPolicies, + inlineFramingDepth, + labelTargetNamesOf, + localLookupTargetsOf, + physicalFields, + referencedTargetsOf, + searchSchema, + storedTargetOf, + TARGET_FIELD, + validateSearchType, +} from '../src/schema.js'; +import { validateQuery } from '../src/query.js'; + +const SCHEMA_ORG = 'https://schema.org/'; +const alias = (type: string, field: string) => `urn:lde:${type}/${field}`; + +/** + * SCHEMA-AP-NDE ranges `creator` over Person OR Organization. The two differ + * in the one declaration that changes what a reference stores: Person is keyed + * on an authority IRI, Organization on nothing – and Person admits only its + * authority-keyed ids to a facet. + */ +const person = defineSearchType({ + name: 'Person', + class: `${SCHEMA_ORG}Person`, + labelField: 'name', + key: { field: 'sameAs' }, + facetKeys: { only: (id) => id.startsWith('https://rkd/') }, + fields: [ + { + name: 'name', + kind: 'text', + path: `${SCHEMA_ORG}name`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + name: 'sameAs', + kind: 'reference', + path: `${SCHEMA_ORG}sameAs`, + array: true, + }, + { + name: 'birthDate', + kind: 'keyword', + path: `${SCHEMA_ORG}birthDate`, + output: true, + }, + ], +}); + +const organization = defineSearchType({ + name: 'Organization', + class: `${SCHEMA_ORG}Organization`, + labelField: 'name', + fields: [ + { + name: 'name', + kind: 'text', + path: `${SCHEMA_ORG}name`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + { + name: 'location', + kind: 'keyword', + path: `${SCHEMA_ORG}location`, + output: true, + }, + ], +}); + +/** A plain lookup at the root, faceted: what the issue was filed against. */ +const work = defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + array: true, + output: true, + filterable: true, + facetable: true, + ref: { strategy: 'lookup', target: ['Person', 'Organization'] }, + }, + ], +}); + +/** The same range, one level in: an edge nesting a `local` lookup. */ +const creatorEdge = defineSearchType({ + name: 'CreatorEdge', + fields: [ + { + name: 'role', + kind: 'keyword', + path: `${SCHEMA_ORG}roleName`, + output: true, + }, + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + output: true, + ref: { + strategy: 'lookup', + target: ['Person', 'Organization'], + local: true, + }, + }, + ], +}); + +const edgedWork = defineSearchType({ + name: 'EdgedWork', + class: `${SCHEMA_ORG}Painting`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + array: true, + output: true, + filterable: true, + facetable: true, + ref: { strategy: 'inline', typeName: 'CreatorEdge', identity: 'creator' }, + }, + ], +}); + +const schema = searchSchema(work, edgedWork, person, organization, creatorEdge); + +const RKD = 'https://rkd/artists/1'; +const rembrandt = { + '@id': 'https://id.example/rembrandt', + '@type': [`${SCHEMA_ORG}Person`, `${SCHEMA_ORG}Thing`], + [alias('Person', 'name')]: [{ '@value': 'Rembrandt' }], + [alias('Person', 'sameAs')]: [{ '@id': RKD }], +}; +const sisters = { + '@id': 'https://id.example/zusters', + '@type': `${SCHEMA_ORG}Organization`, + [alias('Organization', 'name')]: [{ '@value': 'Zusters Benedictinessen' }], +}; +/** Typed as neither: the fallback case. */ +const untyped = { + '@id': 'https://id.example/unknown', + [alias('Person', 'name')]: [{ '@value': 'Onbekend' }], +}; + +describe('declaring several targets', () => { + it('reads them in order, through one shape for one and for many', () => { + expect(referencedTargetsOf(work.fields[0], schema)).toEqual([ + person, + organization, + ]); + expect(labelTargetNamesOf(edgedWork.fields[0], schema)).toEqual([ + 'Person', + 'Organization', + ]); + expect(localLookupTargetsOf(creatorEdge.fields[1], schema)).toEqual([ + person, + organization, + ]); + expect(localLookupTargetsOf(work.fields[0], schema)).toEqual([]); + }); + + it('inherits each target’s facet policy under its own name', () => { + const policies = inheritedFacetPolicies(work.fields[0], schema); + expect([...policies.keys()]).toEqual(['Person']); + // A policy anywhere among the targets earns the field a facet companion. + expect(physicalFields(work.fields[0], schema).facet).toBe('creator_facet'); + }); + + it('rejects a target named twice: order is precedence, and a rank cannot be held twice', () => { + expect( + validateSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + output: true, + ref: { strategy: 'lookup', target: ['Person', 'Person'] }, + }, + ], + }), + ).toContainEqual({ field: 'creator', reason: 'duplicate-target' }); + }); + + it('rejects an empty target list as naming no type at all', () => { + expect( + validateSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + output: true, + ref: { strategy: 'lookup', target: [] }, + }, + ], + }), + ).toContainEqual({ field: 'creator', reason: 'missing-ref-type-name' }); + }); + + it('refuses joinable: an engine reference names one collection', () => { + expect( + validateSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + joinable: true, + ref: { strategy: 'lookup', target: ['Person', 'Organization'] }, + }, + ], + }), + ).toContainEqual({ + field: 'creator', + reason: 'joinable-with-several-targets', + }); + }); + + it('reserves the discriminator’s name', () => { + expect( + validateSearchType({ + name: 'Person', + class: `${SCHEMA_ORG}Person`, + fields: [{ name: TARGET_FIELD, kind: 'keyword', output: true }], + }), + ).toContainEqual({ field: TARGET_FIELD, reason: 'reserved-field-name' }); + }); + + it('resolves every named target, not just the first', () => { + expect(() => + searchSchema(person, { + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + output: true, + ref: { strategy: 'lookup', target: ['Person', 'Nowhere'] }, + }, + ], + }), + ).toThrow(/names unknown label source “Nowhere”/); + }); + + it('rejects targets that declare one field differently', () => { + // A referent of either kind is stored in one nested object, so a field + // the targets share must be one shape. + const otherOrganization = defineSearchType({ + ...organization, + fields: [ + organization.fields[0], + { + name: 'birthDate', + kind: 'date', + path: `${SCHEMA_ORG}foundingDate`, + output: true, + }, + ], + }); + expect(() => searchSchema(work, person, otherOrganization)).toThrow( + /both declare “birthDate” but not alike \(keyword vs date\)/, + ); + // Arity counts too: one nested object cannot hold a value and a list + // under one name. + const listedOrganization = defineSearchType({ + ...organization, + fields: [ + organization.fields[0], + { + name: 'birthDate', + kind: 'keyword', + array: true, + path: `${SCHEMA_ORG}foundingDate`, + output: true, + }, + ], + }); + expect(() => searchSchema(work, person, listedOrganization)).toThrow( + /not alike \(keyword vs keyword list\)/, + ); + // So does everything else that decides the stored shape: a Role, and for + // a reference what it points at and whether it stores a copy. + const filteringOrganization = defineSearchType({ + ...organization, + fields: [ + organization.fields[0], + { + name: 'birthDate', + kind: 'keyword', + path: `${SCHEMA_ORG}foundingDate`, + output: true, + filterable: true, + }, + ], + }); + expect(() => searchSchema(work, person, filteringOrganization)).toThrow( + /not alike \(keyword vs keyword \(filterable\)\)/, + ); + const place = defineSearchType({ + name: 'Place', + class: `${SCHEMA_ORG}Place`, + fields: [ + { + name: 'label', + kind: 'text', + path: `${SCHEMA_ORG}name`, + locales: ['und'], + output: true, + searchable: { weight: 1 }, + }, + ], + }); + const address = (local: boolean) => ({ + name: 'address', + kind: 'reference' as const, + path: `${SCHEMA_ORG}address`, + output: true, + ref: { strategy: 'lookup' as const, target: 'Place', local }, + }); + expect(() => + searchSchema( + work, + place, + defineSearchType({ + ...person, + fields: [...person.fields, address(false)], + }), + defineSearchType({ + ...organization, + fields: [...organization.fields, address(true)], + }), + ), + ).toThrow( + /not alike \(reference → lookup Place vs reference → lookup Place local\)/, + ); + }); + + it('accepts an idOnly reference labelling its buckets from several sources', () => { + const labelled = defineSearchType({ + name: 'Work', + class: `${SCHEMA_ORG}CreativeWork`, + fields: [ + { + name: 'creator', + kind: 'reference', + path: `${SCHEMA_ORG}creator`, + facetable: true, + labelSource: ['Person', 'Organization'], + ref: { strategy: 'idOnly' }, + }, + ], + }); + const idOnly = searchSchema(labelled, person, organization); + expect(labelTargetNamesOf(labelled.fields[0], idOnly)).toEqual([ + 'Person', + 'Organization', + ]); + }); +}); + +describe('reading a stored discriminator', () => { + it('names the target stored, and falls back to the first where none is', () => { + expect( + storedTargetOf({ [TARGET_FIELD]: 'Organization' }, [ + person, + organization, + ]), + ).toBe(organization); + expect( + storedTargetOf({ name_und: 'Onbekend' }, [person, organization]), + ).toBe(person); + }); +}); + +describe('two targets nesting one name inline', () => { + it('must nest the same Reference Type', () => { + const addressOf = (typeName: string) => ({ + name: 'address', + kind: 'reference' as const, + path: `${SCHEMA_ORG}address`, + output: true, + ref: { strategy: 'inline' as const, typeName }, + }); + const postal = defineSearchType({ + name: 'PostalAddress', + fields: [ + { + name: 'street', + kind: 'keyword', + path: `${SCHEMA_ORG}streetAddress`, + output: true, + }, + ], + }); + const visiting = defineSearchType({ ...postal, name: 'VisitingAddress' }); + expect(() => + searchSchema( + work, + postal, + visiting, + defineSearchType({ + ...person, + fields: [...person.fields, addressOf('PostalAddress')], + }), + defineSearchType({ + ...organization, + fields: [...organization.fields, addressOf('VisitingAddress')], + }), + ), + ).toThrow( + /reference → inline PostalAddress vs reference → inline VisitingAddress/, + ); + }); +}); + +describe('framing depth over several targets', () => { + it('frames as far as the furthest target reaches', () => { + // Person's key field is one more hop than Organization's fields: the + // frame has to hold whichever declaration the referent turns out to match. + expect(inlineFramingDepth(schema, edgedWork)).toBe(2); + const shallow = searchSchema( + defineSearchType({ + ...edgedWork, + name: 'ShallowWork', + }), + defineSearchType({ + ...creatorEdge, + fields: [ + creatorEdge.fields[0], + { + ...creatorEdge.fields[1], + ref: { strategy: 'lookup', target: 'Organization', local: true }, + }, + ], + }), + organization, + ); + expect(inlineFramingDepth(shallow, shallow.get(edgedWork.class)!)).toBe(2); + }); +}); + +describe('projecting a plain lookup over several targets', () => { + const node = { + '@id': 'https://ex/work/1', + [alias('Work', 'creator')]: [rembrandt, sisters, untyped], + }; + const document = projectDocument(node, work, schema); + + it('re-keys each referent through the target its rdf:type matches', () => { + // The person through Person's key, the organization through nothing. + expect(document.creator).toEqual([ + RKD, + 'https://id.example/zusters', + 'https://id.example/unknown', + ]); + }); + + it('admits each id to a facet by the policy of its own target', () => { + // The organization has no policy to fail; the untyped referent fell back + // to the first target, whose policy it does not satisfy. + expect(document.creator_facet).toEqual([RKD, 'https://id.example/zusters']); + }); + + it('judges a transformed id by its target still', () => { + // A `transform` changes what the field stores; the policy is looked up by + // the stored value, so the organization keeps its unconditional admission + // and the person is still held to Person's policy. + const transformed = defineSearchType({ + ...work, + name: 'TransformedWork', + fields: [ + { + ...work.fields[0], + transform: (value: string) => value.replace('https://', 'http://'), + }, + ], + }); + const projected = projectDocument( + { + ...node, + [alias('TransformedWork', 'creator')]: node[alias('Work', 'creator')], + }, + transformed, + searchSchema(transformed, person, organization), + ); + expect(projected.creator_facet).toEqual(['http://id.example/zusters']); + }); +}); + +describe('projecting a local lookup over several targets', () => { + const node = { + '@id': 'https://ex/work/2', + [alias('EdgedWork', 'creator')]: [ + { + [alias('CreatorEdge', 'role')]: [{ '@value': 'etser' }], + [alias('CreatorEdge', 'creator')]: [rembrandt], + }, + { + [alias('CreatorEdge', 'role')]: [{ '@value': 'uitgever' }], + [alias('CreatorEdge', 'creator')]: [sisters], + }, + { + [alias('CreatorEdge', 'role')]: [{ '@value': 'drukker' }], + [alias('CreatorEdge', 'creator')]: [untyped], + }, + { + // Named inline as an organization, with no id: the case a collection + // can never answer for, and the one the discriminator exists for. + [alias('CreatorEdge', 'role')]: [{ '@value': 'opdrachtgever' }], + [alias('CreatorEdge', 'creator')]: [ + { + '@type': `${SCHEMA_ORG}Organization`, + [alias('Organization', 'name')]: [{ '@value': 'Gemeente' }], + [alias('Organization', 'location')]: [{ '@value': 'Maastricht' }], + }, + ], + }, + ], + }; + const document = projectDocument(node, edgedWork, schema); + const entries = document.creator as readonly SearchDocument[]; + const endpointOf = (index: number) => + entries[index].creator as SearchDocument; + + it('projects each endpoint through the target it matches, and says which', () => { + expect(endpointOf(0)).toMatchObject({ + id: RKD, + name_und: 'Rembrandt', + [TARGET_FIELD]: 'Person', + }); + expect(endpointOf(1)).toMatchObject({ + id: 'https://id.example/zusters', + name_und: 'Zusters Benedictinessen', + [TARGET_FIELD]: 'Organization', + }); + expect(endpointOf(3)).toMatchObject({ + name_und: 'Gemeente', + location: 'Maastricht', + [TARGET_FIELD]: 'Organization', + }); + expect(endpointOf(3)).not.toHaveProperty('id'); + }); + + it('falls back to the first target declared for a referent matching none', () => { + expect(endpointOf(2)).toMatchObject({ + id: 'https://id.example/unknown', + [TARGET_FIELD]: 'Person', + }); + }); + + it('admits each harvested id to the facet by the policy of its own target', () => { + expect(document.creator_id).toEqual([ + RKD, + 'https://id.example/zusters', + 'https://id.example/unknown', + ]); + expect(document.creator_id_facet).toEqual([ + RKD, + 'https://id.example/zusters', + ]); + }); + + it('stores no discriminator for a lookup naming one target', () => { + const single = searchSchema( + defineSearchType({ ...edgedWork, name: 'SingleWork' }), + defineSearchType({ + ...creatorEdge, + fields: [ + creatorEdge.fields[0], + { + ...creatorEdge.fields[1], + ref: { strategy: 'lookup', target: 'Person', local: true }, + }, + ], + }), + person, + ); + const singleWork = single.get(edgedWork.class)!; + const projected = projectDocument( + { + ...node, + [alias('SingleWork', 'creator')]: node[alias('EdgedWork', 'creator')], + }, + singleWork, + single, + ); + const [first] = projected.creator as readonly SearchDocument[]; + expect(first.creator).not.toHaveProperty(TARGET_FIELD); + }); +}); + +describe('validating a projection over several targets', () => { + const base = { + where: [], + orderBy: [], + limit: 10, + offset: 0, + facets: [], + locale: 'nl', + }; + + it('accepts a field any target serves, and rejects one none does', () => { + expect( + validateQuery( + { + ...base, + resolve: { creator: { fields: ['birthDate', 'location'] } }, + }, + work, + schema, + ), + ).toEqual([]); + expect( + validateQuery( + { ...base, resolve: { creator: { fields: ['deathDate'] } } }, + work, + schema, + ), + ).toEqual([ + { part: 'resolve', field: 'creator.deathDate', reason: 'unknown-field' }, + ]); + }); +}); diff --git a/packages/search/test/qualified-relation.test.ts b/packages/search/test/qualified-relation.test.ts index 69d3ca01..b6c5734d 100644 --- a/packages/search/test/qualified-relation.test.ts +++ b/packages/search/test/qualified-relation.test.ts @@ -3,7 +3,7 @@ import { projectDocument, type SearchDocument } from '../src/project.js'; import { defineSearchType, inlineFramingDepth, - labelTargetNameOf, + labelTargetNamesOf, physicalFields, searchSchema, } from '../src/schema.js'; @@ -1412,11 +1412,11 @@ describe('the label target a facet reads', () => { it('is the identity companion’s target, one level in', () => { // An inline reference names no label source of its own, so reading only // its own declaration would leave its facet buckets unlabelled. - expect(labelTargetNameOf(work.fields[0], schema)).toBe('Person'); + expect(labelTargetNamesOf(work.fields[0], schema)).toEqual(['Person']); }); it('is nothing for a field that references nothing', () => { - expect(labelTargetNameOf(creatorEdge.fields[0], schema)).toBeUndefined(); + expect(labelTargetNamesOf(creatorEdge.fields[0], schema)).toEqual([]); }); it('is nothing against a schema that does not declare the edge’s type', () => { @@ -1425,7 +1425,7 @@ describe('the label target a facet reads', () => { // reading every schema-less path in the projection makes. const foreign = searchSchema(person); - expect(labelTargetNameOf(work.fields[0], foreign)).toBeUndefined(); + expect(labelTargetNamesOf(work.fields[0], foreign)).toEqual([]); }); }); diff --git a/packages/search/test/schema.test.ts b/packages/search/test/schema.test.ts index 76d528ea..9d9b3f28 100644 --- a/packages/search/test/schema.test.ts +++ b/packages/search/test/schema.test.ts @@ -16,7 +16,7 @@ import { isoToUnixSeconds, isInternalField, isRangeFacet, - labelSourceNameOf, + labelSourceNamesOf, nestedFieldName, nestedReferenceType, outputFields, @@ -1564,26 +1564,43 @@ describe('searchSchema validation', () => { ); }); - it('reads the label source name off a lookup’s target and an idOnly’s labelSource', () => { + it('reads the label source names off a lookup’s target and an idOnly’s labelSource', () => { expect( - labelSourceNameOf({ + labelSourceNamesOf({ name: 'publisher', kind: 'reference', ref: { strategy: 'lookup', target: 'Organization' }, }), - ).toBe('Organization'); + ).toEqual(['Organization']); expect( - labelSourceNameOf({ + labelSourceNamesOf({ name: 'license', kind: 'reference', labelSource: 'Term', ref: { strategy: 'idOnly' }, }), - ).toBe('Term'); + ).toEqual(['Term']); + // Several targets, in the order declared: the one shape for one and + // for many, so no reader branches on which it was given. + expect( + labelSourceNamesOf({ + name: 'creator', + kind: 'reference', + ref: { strategy: 'lookup', target: ['Person', 'Organization'] }, + }), + ).toEqual(['Person', 'Organization']); + expect( + labelSourceNamesOf({ + name: 'creator', + kind: 'reference', + labelSource: ['Person', 'Organization'], + ref: { strategy: 'idOnly' }, + }), + ).toEqual(['Person', 'Organization']); // Resolves nothing: no target, no label source. expect( - labelSourceNameOf({ name: 'license', kind: 'reference' }), - ).toBeUndefined(); + labelSourceNamesOf({ name: 'license', kind: 'reference' }), + ).toEqual([]); }); it('rejects a labelSource on anything but an idOnly reference', () => { diff --git a/packages/search/vite.config.ts b/packages/search/vite.config.ts index 92874588..ec1f86b4 100644 --- a/packages/search/vite.config.ts +++ b/packages/search/vite.config.ts @@ -12,7 +12,7 @@ export default mergeConfig( thresholds: { functions: 100, lines: 100, - branches: 99.72, + branches: 99.74, statements: 100, }, },