Skip to content

Backend formatter unconditionally prepends separator regardless of field value #8175

Description

@grantfitzsimmons

Describe the bug
When the Agent formatter is used in backend query results (e.g., query builder, reports, labels), separators defined in the formatter XML are unconditionally prepended to field values. This causes output like "Smith, " when firstName is NULL or empty, instead of the expected "Smith".

The frontend correctly handles this case by only emitting the separator when the field value is non-empty, but the backend SQL generation does not.

Affected formatters
All formatters that use the sep attribute on <field> elements, including but not limited to:

  • Agent (agentType=1): lastName + , firstName + - text3
  • Address: address + address2 + , city + , state + postalCode
  • CollectingEvent: stationFieldNumber + ; locality.localityName + ; locality.geography.fullName + ; locality.latitude1 + , locality.longitude1

To Reproduce

  1. Create a Query Builder query that includes a formatted Agent field (one-to-one relation)
  2. Ensure some Agent records have a lastName but firstName is NULL or empty
  3. Run the query
  4. Observe the formatted result shows e.g. "Smith, " (with trailing comma and space) instead of "Smith"
  5. The same data on the frontend (forms, subview cells) does NOT show the trailing comma

Expected behavior
Separators should only appear between non-empty field values. "Smith" + empty firstName should produce "Smith", not "Smith, ".

Root Cause
In specifyweb/backend/stored_queries/format.py, the apply_stringish inner function within make_expr unconditionally prepends the separator:

# Line 230-232 (before fix)
sep = fieldNodeAttrib.get('sep')
if sep is not None:
    e = concat(sep, e)

This generates SQL like CONCAT(', ', firstName). When firstName is NULL, CONCAT returns '' (via blank_nulls wrapping → IFNULL). However, under some query conditions, the separator string is still present in the intermediate concatenation before NULL-coalescing, leading to "Smith, " output.

For comparison, the frontend correctly handles this at specifyweb/frontend/js_src/lib/components/Formatters/formatters.ts:204-207:

return {
    formatted: formatted?.toString() ?? '',
    separator: (formatted ?? '') === '' ? '' : separator,
};

Additional context
This issue was introduced when the backend formatter was implemented without the separator-null-guard logic that exists in the frontend. The frontend (for previews in the table format editor and elsewhere) has handled this correctly since the formatter was first implemented, as seen in the formatField function in formatters.ts

Reported by
Marion Depraetere at MfN Berlin in BoldDesk ticket #1016

I have set up formatted fields for the Collectors and Determiners fields. The issue is that there is a display problem, but not for all collections and not for all Agents. So far, I have only seen the problem in the Mammals collection.

I would therefore like the Agents to appear in the format ‘First name, Last Name’. Normally, the comma does not appear if the “Last Name” field is empty. However, in the Mammals collection this does not work, whereas it works fine for the other collections. Even stranger is that it only works for some Agents (see Budinger agent in the second screenshot)

Activity

  1. added
    2 - QueriesIssues that are related to the query builder or queries in general
    2 - Exporting DataIssues that are related to exporting data to DwC, GBIF, IPT, Web Portal, etc.
    on Jun 10, 2026
  2. added a commit that references this issue on Oct 23, 2026
    1921050
  3. added this to the 7.12.3 milestone on Oct 23, 2026
  4. linked a pull request that will close this issuefeat(dwca): add visual editor #8522on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

2 - Exporting DataIssues that are related to exporting data to DwC, GBIF, IPT, Web Portal, etc.2 - QueriesIssues that are related to the query builder or queries in general

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions