Terminate message queries that exceed a configurable time limit

[ramonsmits]

Customer debug packages showed audit message queries against MessagesViewIndexWithFullTextSearch running for up to ~65 minutes. RavenDB's own Databases.QueryTimeoutInSec (default 300) does not stop them because the server renews the operation deadline while a query keeps making progress, so large scans/sorts run for hours and their sorted results spill into the server's Temp folder, filling the disk.

Query time limit

The message view and search queries now run under a linked CancellationTokenSource that cancels after a configurable time limit, default 1 minute, maximum 1 hour. Cancelling the client request aborts the call to the database server, which does terminate the query server-side. Invalid values fall back to the default.

One QueryTimeLimit helper in ServiceControl.Infrastructure serves all persisters. Its rule is that whatever exception surfaces after the deadline fired is the timeout: RavenDB and Npgsql raise OperationCanceledException, Microsoft.Data.SqlClient raises SqlException("Operation cancelled by user"). The SQL Server path is covered by a test against the real container that slows the command down server-side.

The limit applies to exactly these queries:

• Audit RavenAuditDataStore: GetMessages, QueryMessages, QueryMessagesByReceivingEndpoint, QueryMessagesByReceivingEndpointAndKeyword, QueryMessagesByConversationId, read from ServiceControl.Audit/QueryTimeoutInSeconds. QueryAuditCounts (licensing throughput) and QuerySagaHistoryById are not under the limit.
• Primary IMessagesViewDataStore on RavenDB, SQL Server and PostgreSQL: GetAllMessages, GetAllMessagesForEndpoint, GetAllMessagesByConversation, GetAllMessagesForSearch, SearchEndpointMessages, read from ServiceControl/QueryTimeoutInSeconds. On SQL Server and PostgreSQL these queries also use the limit as their per-command timeout, so Database/CommandTimeout cannot undercut it.

Other paged index queries (GetFailedMessages, groups, event log, ...) are unchanged.

What a caller sees

• A query that runs out of its time is answered with 504 Gateway Timeout and a application/problem+json body whose detail names the setting, on both hosts.
• The primary's scatter-gather keeps partial data: a local timeout is absorbed the way a remote failure already was, a remote 504 or error is a missing instance rather than an instance with no data. The response lists every missing instance in the X-Particular-Incomplete-Results header (instanceId:timeout|unavailable|error, exposed through CORS) and carries no ETag. Only when no queried instance answered and one of them timed out does the composite itself answer 504.
• The in-process audit counts used for licensing throughput fail when any instance is missing, instead of recording a partial sum as the day's throughput.
• The remote instance HttpClient timeout is ServiceControl/QueryTimeoutInSeconds plus 30 seconds instead of the 100 second default, so raising the limit on both instances does not make the primary give up on the audit instance first.

ServicePulse follow-up: the audit list already treats a non-2xx answer (the new 504) as a failed query; reading X-Particular-Incomplete-Results on a 200 is still to be added.

Docs: Particular/docs.particular.net#8486