Keep ensemble order and arguments in cache keys and subsets

[marisbasha]

Summary

Ensemble response caching ignores the order of the ensemble, so after re-ordering, responses come back in the old network_id order. Ensemble subsets drop their constructor arguments, and simulate_from_dataset fails when the last batch is smaller.

Changes

Response cache keys

make_hashable sorted lists, so context_aware_cache gave the same key for the same names in a different order. After sort(), inside rank_by_validation_error() or select_items() with a permuted list, the Ensemble response methods (flash_responses, moving_edge_responses, validation_losses and the others cached on self.names) returned the result cached for the previous order. network_id then no longer lined up with ensemble.names, task_error() or the colours. Arguments such as speeds=[25, 19] and speeds=[19, 25] also shared a key. Lists and tuples now keep their order. Sets and dicts are still sorted, and frozensets are now sorted like sets. The caches are in memory, and the joblib response caches on disk do not use make_hashable, so no stored responses are invalidated.

Subsets

ens[0:2] and ens[[0, 1]] rebuilt the subset from the model names with default arguments. root_dir, best_checkpoint_fn, best_checkpoint_fn_kwargs, checkpoint_mapper, network_class, connectome_getter and recover_fn were dropped, and the names were resolved under flyvis.results_dir. A subset of an ensemble loaded from another root silently pointed at the same-named models in results_dir, and custom best-checkpoint settings were reset. Subsets are now built from the model paths with the original constructor arguments. try_sort is not passed on, so the subset keeps the order given by the key.

simulate_from_dataset

The per-batch responses were combined with np.stack, which raised whenever the number of stimuli was not a multiple of batch_size. They are now concatenated.

Testing

New tests/test_cache_utils.py checks that the cache follows a change of order and that make_hashable keeps list and tuple order but not set or dict order. In tests/test_ensemble.py (require_download), a new test copies three pretrained models to another root and checks that ens[0:2] and ens[[2, 0]] point at the copies, in that order, with the same best_checkpoint_fn_kwargs. Another runs simulate_from_dataset with 3 stimuli and batch_size=2. All four fail on main and pass with the change.

On the pretrained ensemble flow/0000 (50 models), inside rank_by_validation_error(reverse=True), main returns min_validation_losses in the old order and the branch in the new one. select_items with the same three names in a new order also returns the old order on main. Moving edge responses cached on disk with main's cache_utils load from the cache on the branch.

Full suite (-m "not require_download and not require_large_download and not gpu", test_examples.py and test_sintel.py excluded): 196 passed, 12 skipped on main; 198 passed, 12 skipped on the branch. require_download tests against the pretrained models: 21 passed on main; 23 passed on the branch. tests/test_sintel.py, run offline with the mock data: 14 passed on both. ruff check is clean on the changed files; ruff format is clean on the changed lines (it would reformat an untouched block in ensemble.py, left as is).

[Copilot]

Copilot review overview

🟢 Approval recommended

The implementation correctly addresses each reported regression with focused coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes order-sensitive caching, preserves ensemble configuration in subsets, and supports uneven final simulation batches.

Changes:

• Preserve list/tuple order in cache keys.
• Retain ensemble paths and constructor arguments in subsets.
• Concatenate variable-sized response batches and add regression tests.

File	Description
flyvis/​utils/​cache_utils.py	Preserves sequence order while normalizing unordered collections.
flyvis/​network/​ensemble.py	Retains subset configuration and concatenates simulation batches.
tests/​test_cache_utils.py	Tests order-sensitive cache behavior.
tests/​test_ensemble.py	Tests subset configuration and uneven batch handling.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 38.95%. Comparing base (92b3845) to head (cc90adb).

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #30      +/-   ##
==========================================
+ Coverage   38.68%   38.95%   +0.26%     
==========================================
  Files          75       75              
  Lines        9738     9738              
==========================================
+ Hits         3767     3793      +26     
+ Misses       5971     5945      -26     

Flag	Coverage Δ
unittests	38.95% <100.00%> (+0.26%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.