[avgvi]

Makes SafeDisk.with_numpy() reads as fast as the default pickle Disk, and extends what it stores so that numpy data works in keys as well as values. Existing entries are unchanged: the stored bytes of every type 5.8.3 already handled are identical, and 5.8.3 and this branch read each other's plain array entries.

Speed

with_numpy() now keeps the .npy headers it has parsed or written, in two bounded in-memory tables, instead of parsing each header with numpy's literal parser on every read. The size of the data is still checked against the header on every read.

384-value float32 array	upstream 5.6.3	5.8.3	this PR
hit	4.4 µs	12.9 µs	4.5 µs
write	38.8 µs	47.5 µs	35 µs

Key cost on a with_numpy() disk, compared with 5.8.3: str, int, float and bytes keys are unchanged, and so are keys of with_types classes (21.1 → 21.2 µs). A tuple of plain values costs about 0.3 µs more (4.82 → 5.13 µs), for the check that converts numpy scalars in it. Plain SafeDisk is unchanged.

New in keys

• numpy booleans, integers, floats, complex numbers, strings and bytes become the Python values item() gives, also inside tuples, lists, dicts, sets and frozensets, so int64(1) and 1 name the same entry and memoize hits for either.
• Arrays, alone or inside other keys, and as memoize arguments. Every array in a key is written in C order, so an equal array of the same dtype and shape names the same entry. Another dtype, shape or time unit names another entry.
• A key without numpy is pickled exactly as a plain SafeDisk pickles it, at any protocol.

New in values (and keys)

Type	Stored as	Reads back as
datetime64 / timedelta64 scalar	('scalar', dtype name, int64 count)	same unit, NaT included; a unit multiple over 9 digits raises TypeError at write
array of objects	('objects', shape, items), each item checked like any value	object array, C order
numpy.recarray	('records', array)	recarray
numpy.ma.MaskedArray	('masked', data, mask, fill_value, hard_mask)	MaskedArray with its mask, fill value and hard mask
numpy.memmap	its data, as a plain array	plain ndarray

All of these are persistent IDs, as 'npy' and 'scalar' already were: rows name no module or function. A value holding an array of objects is read back once before it is stored, so a self-referencing array raises TypeError instead of being written unreadable. numpy.matrix, other ndarray subclasses, arrays with object fields, longdouble and structured scalars are still refused, with messages that say what to store instead. Entries and keys using the new kinds read as unreadable on 5.8.3 and earlier.

Fix

Values whose pickling handed out short-lived views of one array (for example an object whose __reduce__ yields slices) could be stored with one view's data in place of later ones on 5.8.2 and 5.8.3. The pickler's memo now holds each array object it has seen, so a freed view's id is never reused for another.

Review and tests

An independent design and logic review ran on an earlier version of this branch; this PR includes its changes: persistent IDs in place of module-level rebuild functions, the key fast paths, the write-time time-unit check, and its test gaps (pinned key bytes, the key pickle scan, two time units in one value, recursion, the header size limit, protocols 2, 4 and 5).

• pytest: 985 passed on numpy 2.5 (arrays.py 100% covered, 99.22% overall) and 983 on numpy 1.26, Python 3.12.
• blue, flake8, isort, pylint, mypy, doc8 and the docs build pass.
• Stored bytes for existing value and key types compared byte for byte with 5.8.3; 5.8.3 and this branch read each other's array entries.
• embetter's cached(), unchanged, now caches 2-D rows, numpy numbers and image arrays on this branch.

CHANGELOG.md has the entries under Unreleased; they don't name this PR's number yet.

[read-the-docs-community]

Documentation build overview

📚 diskcache-community | 🛠️ Build #35690291 | 📁 Comparing 731edce against latest (ac57330)

  🔍 Preview build  

3 files changed

± api.html
± index.html
± tutorial.html