FOUR-33159 POC Performance issue when opening the column configuration in a saved search

[devmiguelangel]

Issue & Reproduction Steps

FOUR-33159 POC Performance issue when opening the column configuration in a saved search

Problem

Opening Columns on a Request Saved Search with fulltext PMQL (e.g. (fulltext LIKE "%test%")) took ~76s and could 500 with MySQL error 1038 (Out of sort memory).

Two independent bottlenecks were hit on every load:

1. HasDataColumns::getData() re-executed the full Saved Search PMQL (including LOWER(data) LIKE '%test%') just to sample recent data JSON for column names.
2. For unscoped (global fulltext) searches, SavedSearch::getProcessesAttribute() fell back to Process::get() (all processes) and then ran a recursive ExportManager screen walk per process, adding ~50s of PHP.

Solution

Column discovery no longer depends on the user's PMQL or on sort_buffer_size.

• Bounded, PMQL-free sampling — HasDataColumns::getData() now uses column_discovery_query (type base query, optionally scoped by process_id), orders by indexed id DESC, and limits to 10 rows. The fulltext clause is never applied. MySQL 1038 is caught and returns an empty sample instead of a 500.
• Variable Finder instead of ExportManager — new ProcessVariableDiscoveryService resolves data.* fields from var_finder_variables (indexed) for both scoped and unscoped searches. Screen fallback uses ScreensInProcess (direct BPMN screenRef scan) capped to the 25 most recently updated processes, not the recursive export graph.
• Separate pagination for available columns — GET /api/1.1/processes/variables?onlyAvailable= now paginates Saved Search available + data columns independently. Process variables load in a second phase. Empty processIds returns the unscoped catalog instead of an empty page. Active columns are excluded in SQL. Fixed a duplicate paginate() call in the controller.
• Caching — Cache::flexible + lock + last-known-good for both the discovery service and data_columns, keyed on saved search / process / Variable Finder state.
• Frontend — VariablesList.vue pages available columns until meta.last_page, then switches to process variables. The edit screen no longer background-fetches every page.

Scope

• Saved Search listing results are unchanged; scopeFilter() / valueAliasFullText() are untouched.
• Task and Collection Saved Searches still resolve columns.
• No MySQL configuration change is required.

Related Tickets & Packages

https://processmaker.atlassian.net/browse/FOUR-33159
https://github.com/ProcessMaker/package-savedsearch/pull/696

Code Review Checklist

• I have pulled this code locally and tested it on my instance, along with any associated packages.
• This code adheres to ProcessMaker Coding Guidelines.
• This code includes a unit test or an E2E test that tests its functionality, or is covered by an existing test.
• This solution fixes the bug reported in the original ticket.
• This solution does not alter the expected output of a component in a way that would break existing Processes.
• This solution does not implement any breaking changes that would invalidate documentation or cause existing Processes to fail.
• This solution has been tested with enterprise packages that rely on its functionality and does not introduce bugs in those packages.
• This code does not duplicate functionality that already exists in the framework or in ProcessMaker.
• This ticket conforms to the PRD associated with this part of ProcessMaker.

ci:package-savedsearch:FOUR-33159

.

[cursor]

PR Summary

Medium Risk
Changes a hot API path used when editing saved-search columns, including pagination semantics and new caching; behavior is heavily tested but unscoped discovery may differ slightly from loading all processes via ExportManager.

Overview
Fixes slow/failing Saved Search column configuration by reshaping GET /api/1.1/processes/variables and centralizing how data.* fields are resolved.

onlyAvailable now paginates saved-search available + data columns on their own via getAvailableColumnsPaginator, instead of merging them into the process-variables page (which dropped fields and skewed totals). Empty processIds no longer returns an empty list; it uses unscoped discovery (Variable Finder or a bounded screen scan).

Adds ProcessVariableDiscoveryService: prefers indexed Variable Finder queries, falls back to ScreensInProcess (replacing recursive ExportManager walks), and wraps results in Cache::flexible with locks and last-known-good payloads. The controller excludes active saved-search columns in SQL for the finder path, orders by field, and on MySQL 1038 logs and falls back to screen-based discovery.

Feature tests cover fulltext saved searches with empty process IDs, multi-page onlyAvailable, active-column exclusion, no PMQL/LOWER scans for available columns, and sort-memory fallback.

^{Reviewed by Cursor Bugbot for commit c171a6e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[decisions-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube