test(solr): verify replication and shard lifecycle operations

.github/workflows/ci.yml

@@ -4,6 +4,13 @@ on:
     branches: [master]
   pull_request:
     branches: [master]
+  workflow_dispatch:
+    inputs:
+      enable_destructive_solrcloud_tests:
+        description: "Run opt-in SPLITSHARD and MIGRATE checks on the disposable two-node cluster"
+        required: false
+        default: false
+        type: boolean
 jobs:
   test:
     strategy:
@@ -45,7 +52,7 @@ jobs:
             --volume "$RUNNER_TEMP/solr-backups:/var/solr/backups" \
             --publish 8983:8983 \
             --env SOLR_JAVA_MEM="-Xms512m -Xmx512m" \
-            --env SOLR_OPTS="-Dsolr.allowPaths=/var/solr/backups" \
+            --env SOLR_OPTS="-Dsolr.allowPaths=/var/solr/backups -Dsolr.disable.shardsWhitelist=true -Dsolr.disable.allowUrls=true -Dsolr.security.allow.urls.enabled=false" \
             "solr:${{ matrix.solr-version }}" \
             solr-precreate integration
       - name: Wait for Solr and integration core
@@ -67,6 +74,7 @@ jobs:
         env:
           SOLR_BASE_URL: http://localhost:8983/solr
           SOLR_CORE: integration
+          SOLR_VERSION: "${{ matrix.solr-version }}"
           SOLR_BACKUP_LOCATION: /var/solr/backups
       - name: Print Solr logs
         if: always()
@@ -124,7 +132,7 @@ jobs:
             --volume "$RUNNER_TEMP/solr-backups:/var/solr/backups" \
             --publish 8983:8983 \
             --env SOLR_JAVA_MEM="-Xms512m -Xmx512m" \
-            --env SOLR_OPTS="-Dsolr.allowPaths=/var/solr/backups" \
+            --env SOLR_OPTS="-Dsolr.allowPaths=/var/solr/backups -Dsolr.disable.shardsWhitelist=true -Dsolr.disable.allowUrls=true -Dsolr.security.allow.urls.enabled=false" \
             --entrypoint /bin/bash \
             "solr:${{ matrix.solr-version }}" \
             -lc 'mkdir -p /var/solr/data && cp /tmp/security.json /var/solr/data/security.json && chmod 666 /var/solr/data/security.json && exec docker-entrypoint.sh solr-precreate integration'
@@ -149,6 +157,7 @@ jobs:
         env:
           SOLR_BASE_URL: http://localhost:8983/solr
           SOLR_CORE: integration
+          SOLR_VERSION: "${{ matrix.solr-version }}"
           SOLR_SECURITY: "true"
           SOLR_USERNAME: integration-admin
           SOLR_PASSWORD: integration-password
@@ -158,6 +167,8 @@ jobs:
         run: docker logs solr-secure || true
 
   cloud-integration:
+    env:
+      SOLR_DESTRUCTIVE_OPT_IN: "${{ github.event_name == 'workflow_dispatch' && inputs.enable_destructive_solrcloud_tests == true && 'true' || 'false' }}"
     strategy:
       fail-fast: false
       matrix:

docs/administration.md

@@ -161,3 +161,18 @@ The alias and cluster-status integration checks exercise alias read-back, collec
 The V1 parameter spellings are intentional: this client calls `/solr/admin/collections`, not the Solr 10 V2 `/api/collections` resources. The V2 API maps `split.key` to `splitKey` and `target.collection` to `target`; those V2 names must not be sent through this V1 client.
 
 Reference guides: [Solr 8.1 Collections API](https://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-8.1.pdf), [Solr 9.2 Collection Management](https://solr.apache.org/guide/solr/9_2/deployment-guide/collection-management.html), [Solr 9.10 Shard Management](https://solr.apache.org/guide/solr/9_10/deployment-guide/shard-management.html), and [Solr 10.0 Collections API](https://solr.apache.org/guide/solr/10_0/configuration-guide/collections-api.html).
+
+## Opt-in SPLITSHARD and MIGRATE verification
+
+These operations are intentionally excluded from normal push/pull-request CI. The opt-in suite uses only collections created by the fixture with unique `integration_*` names. It refuses to run unless all of the following are true:
+
+- `SOLR_CLOUD=true` and `SOLR_MULTINODE=true`.
+- `SOLR_DESTRUCTIVE_OPT_IN=true` is explicitly set.
+- `SOLR_BASE_URL` resolves to localhost / loopback; remote Solr endpoints are skipped.
+- The disposable cluster reports at least two live nodes; the split test additionally confirms its new collection starts with one shard and the migration test confirms the target collection is empty.
+
+Run it from the GitHub Actions **CI** workflow using **Run workflow** and enable `enable_destructive_solrcloud_tests`. This provisions the workflow's ephemeral two-node SolrCloud cluster and runs the opt-in tests for the pinned Solr version matrix. Leaving the input disabled keeps both tests skipped, including on ordinary pull requests.
+
+The tests use asynchronous request IDs and poll `REQUESTSTATUS` for up to 180 seconds, then remove completed status records. They verify active child-shard state and post-split document visibility, and verify that every routed document is searchable in the migration target. They do not claim to test concurrent writes, rollback of a partially completed operation, or recovery after a cluster/node failure.
+
+If a request times out or fails, do not rerun against a reused cluster. Preserve the CI job logs, inspect `CLUSTERSTATUS` and `REQUESTSTATUS`, and discard the ephemeral cluster. Fixture cleanup deletes only the collections it created; if Solr still has an operation in flight and refuses deletion, destroy the ephemeral cluster rather than attempting cleanup against a shared server. The suite is an opt-in verification harness, not permission to point these destructive actions at a user-owned cluster. A workflow that is skipped or has not been manually dispatched is not runtime evidence; record successful Solr versions individually before updating compatibility status. Keep the workflow run URL and tested Solr version alongside the compatibility record.

docs/api-coverage.md

@@ -24,9 +24,10 @@ A generic request method does not mean that every endpoint has a typed wrapper,
 | Core Admin | `coreAdmin(action, params)` with `CoreAdminParams<A>` action-specific models for STATUS, CREATE, RELOAD, UNLOAD, RENAME, SWAP, MERGEINDEXES, and SPLIT; `CoreAdminStatusResponse` for STATUS and `CoreAdminCommandResponse` as the default non-STATUS response envelope | STATUS models per-core paths/config metadata, optional index statistics, initialization failures, and unknown fields. Other actions default to a named extensible response-header envelope; callers may supply a more specific generic. Unit/type fixtures cover minimal and detailed STATUS payloads plus CREATE/RELOAD response inference. Isolated CREATE, STATUS, RELOAD, and fixture-cleanup UNLOAD are integration-tested on Solr 8.11.4/9.11.0/10.1.0; remaining actions retain serialization/type coverage only and are not claimed as runtime-certified. See [Core Admin evidence](administration.md#core-admin-status-response) and Issue #229.
 | Collections | `collections(action, params)` with action-specific `CollectionAdminParams<A>` and `CollectionAdminResponse<A>` conditional response contracts | `LIST` uses `SolrCollectionListResponse`; collection lifecycle/policy, replica, and role actions use named extensible response envelopes; alias, cluster, shard, backup/restore, and async actions retain their named response models. Runtime evidence remains partial, not blanket certification of every action: exact Solr 8.11.4/9.11.0/10.1.0 evidence is listed in the [per-action Collections matrix](collections-api-coverage.md). `MODIFYCOLLECTION`, `SPLITSHARD`, `MIGRATE`, `ADDROLE`, and `REMOVEROLE` remain runtime-unverified. See [backup and restore docs](backup-restore.md), [Collections action matrix](collections-api-coverage.md), and Issues #80–#85, #121, and #230. |
 | Security | `security(area, command)` | Security-enabled standalone matrix covers anonymous 401 responses for both authentication and authorization, authenticated reads, authentication property mutation/read-back, and malformed JSON rejection on Solr 8.11.4/9.11.0/10.1.0. Unit tests verify Basic/Bearer credentials and Authorization headers are not exposed through client HTTP errors. See Issues #86–#88. |
-| System | `systemInfo()`, `systemProperties()`, `systemThreads()`, `systemLogging()` | System info, non-empty JVM/process properties, a non-empty thread dump, and logger metadata are exercised on pinned Solr 8.11.4/9.11.0/10.1.0. Optional fields remain version-dependent; response models preserve unknown fields. See Issues #89 and #91. |\n| Metrics | `metrics()` with typed group/registry/prefix/regex filters; exported `SolrMetricsJsonResponse`, `SolrMetricsRegistry`, and open `SolrMetricValue`; JSON/Prometheus response union | JVM response-format coverage remains distinct (observed JSON on Solr 8/9 and Prometheus exposition on 10.1). A `node` group integration check runs on all pinned versions, asserting a non-empty JSON registry on Solr 8/9 and Prometheus samples on 10.1; registry, prefix, and regex filters are asserted against JSON responses on Solr 8/9 (Solr 10.1 supports Prometheus/OpenMetrics, not the JSON writer). Consumer fixtures verify the extensible registry/value envelope; plugin-specific metrics remain open and unenumerated. See Issues #90, #91, and #236. |
+| System | `systemInfo()`, `systemProperties()`, `systemThreads()`, `systemLogging()` | System info, non-empty JVM/process properties, a non-empty thread dump, and logger metadata are exercised on pinned Solr 8.11.4/9.11.0/10.1.0. Optional fields remain version-dependent; response models preserve unknown fields. See Issues #89 and #91. |
+| Metrics | `metrics()` with typed group/registry/prefix/regex filters; exported `SolrMetricsJsonResponse`, `SolrMetricsRegistry`, and open `SolrMetricValue`; JSON/Prometheus response union | JVM response-format coverage remains distinct (observed JSON on Solr 8/9 and Prometheus exposition on 10.1). A `node` group integration check runs on all pinned versions, asserting a non-empty JSON registry on Solr 8/9 and Prometheus samples on 10.1; registry, prefix, and regex filters are asserted against JSON responses on Solr 8/9 (Solr 10.1 supports Prometheus/OpenMetrics, not the JSON writer). Consumer fixtures verify the extensible registry/value envelope; plugin-specific metrics remain open and unenumerated. See Issues #90, #91, and #236. |
 | Backup and restore | `replication("backup" | "restore" | "restorestatus" | "deletebackup")`, `collections("BACKUP" | "RESTORE" | "LISTBACKUP" | "DELETEBACKUP" | "REQUESTSTATUS" | "DELETESTATUS")` | Typed core and SolrCloud backup/restore parameters and response models, including async status polling. Full core restore and SolrCloud collection backup/restore cycles are integration-tested on Solr 8.11.4/9.11.0/10.1.0 using local filesystem repositories and shared SolrCloud storage. See [backup and restore docs](backup-restore.md). |
-| Replication | `replication(command, params)`, `replicationFile(params)`; exported command-specific parameter and response models | V1 ReplicationHandler commands for index version, details, file list, leader/follower control (Solr 10 fetchindex/abortfetch; Solr 8/9 snappull/abort), core backup/restore/status/delete, and raw file-content streaming. Integration coverage includes core backup/restore cycles on Solr 8.11.4/9.11.0/10.1.0. An end-to-end leader/follower fetch cycle is not yet certified. See Issue #95 and [replication docs](replication.md). |
+| Replication | `replication(command, params)`, `replicationFile(params)`; exported command-specific parameter and response models | V1 ReplicationHandler commands for index version, details, file list, leader/follower control (fetchindex/abortfetch across Solr 8/9/10), core backup/restore/status/delete, and raw file-content streaming. Integration coverage includes core backup/restore cycles on Solr 8.11.4/9.11.0/10.1.0. A disposable two-core leader/follower fetch/read-back check is implemented but remains pending runtime verification until its pinned-version CI jobs pass; it uses `fetchindex`/`abortfetch` across Solr 8/9/10. Abort and polling lifecycle behavior remains unit-only. See Issue #95 and [replication docs](replication.md). |
 | Streaming Expressions | `stream()` | Sends `expr` with `wt=json` and parses the complete JSON response through the shared transport. It is not an incremental tuple-stream parser: no chunk-boundary protocol or memory-bound/backpressure guarantee is claimed. SolrCloud search is integration-tested on pinned Solr 8.11.4/9.11.0/10.1.0; unit tests cover JSON parsing, caller cancellation during body consumption, and cleanup after body-stream errors. See [Streaming Expressions](streaming.md) and Issue #93. |
 | Export | `export()`, `exportStream()`; exported `SolrExportParams`, `SolrExportSort`, and `SolrExportFieldType` | Export requests require typed `fl` and `sort` parameters, support repeated filter queries, structured or raw sort expressions, batch sizing, JSON/JavaBin writers, and an explicit custom-parameter escape hatch. The raw Response preserves backpressure-aware streaming; `exportStream()` exposes its byte body directly. Sort fields must be single-valued docValues fields (int, long, float, double, or string); field lists additionally support date and boolean and may be multi-valued. Solr 10.1 `includeStoredFields` support is exposed but requires version-specific verification. Existing raw Export response tests pass on pinned Solr 8.11.4/9.11.0/10.1.0; typed parameter serialization and streamed chunk consumption have unit coverage. See [Export docs](export.md) and Issues #92–#94. |
 | Advanced / Raw | `requestRaw(path, options)` with typed method/query/header/body/control options and a raw Fetch `Response` | Relative paths and same-origin absolute URLs use the shared transport. Cross-origin URLs, embedded credentials, and URL fragments are rejected before Fetch, preventing configured credentials from being sent to unrelated hosts. Generic request()/requestRaw() transport behavior has unit coverage only; arbitrary custom endpoint behavior remains unverified until endpoint-specific integration tests exist. See [raw requests](raw-requests.md) and Issue #98. |