Skip to content

Fix unsorted and repeated row selection on a DynamicTable read from HDF5 - #1607

Open
aaronkanzer wants to merge 1 commit into
hdmf-dev:devfrom
aaronkanzer:fix/h5py-unsorted-row-selection
Open

aaronkanzer wants to merge 1 commit into
hdmf-dev:devfrom
aaronkanzer:fix/h5py-unsorted-row-selection

Conversation

@aaronkanzer

Copy link
Copy Markdown

Selecting rows of a DynamicTable with a list of indices works in memory but fails after the table is read from HDF5 if the indices are not increasing or repeat. This affects calls like nwbfile.units.get([5, 2]), nwbfile.electrodes[[3, 1]] and table["col"][[2, 0]].

from hdmf.common import DynamicTable, SimpleMultiContainer, get_manager
from hdmf.backends.hdf5 import HDF5IO

t = DynamicTable(name="t", description="d")
t.add_column("b", "b")
for i in range(4):
    t.add_row(b=i * 1.5)
print(t.get([3, 1]))  # works in memory
with HDF5IO("t.h5", "w", manager=get_manager()) as io:
    io.write(SimpleMultiContainer(name="root", containers=[t]))
io = HDF5IO("t.h5", "r", manager=get_manager())
t = io.read()["t"]
t.get([3, 1])

Observed (hdmf 6.2.0 and dev): TypeError: Indexing elements must be in increasing order. The same happens for t["b"][[1, 1]], for string columns, and for DynamicTableRegion.get([2, 0]).

Expected: rows 3 and 1, in that order, as for the in-memory table.

Cause: Data.get passes the list straight to h5py.Dataset, and h5py only accepts increasing indices without duplicates. DynamicTableRegion.get with index=False already works around this, and so does EnumData (#1534). Plain columns, id, and DynamicTableRegion.get(..., index=True) do not.

Fix: when an h5py.Dataset is indexed with a 1-D integer list or array that is not increasing, Data.get reads the unique sorted rows and puts them back in the requested order. Negative indices are converted first. DynamicTableRegion.get now goes through Data.get instead of indexing self.data directly. Increasing selections take the same path as before. Adds round-trip tests for unsorted, repeated and negative indices, full rows, and a region column.

@aaronkanzer
aaronkanzer force-pushed the fix/h5py-unsorted-row-selection branch from e4678bd to 5dfb028 Compare October 24, 2026 12:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant