Repository navigation
0.10.0: backup and restore, less memory - #54
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Settings -> Backup downloads one ZIP with everything: a consistent copy of the database (SQLite's backup) and the data folders, by the parts of settings.BACKUP_PARTS (sensors and settings always; training images, people and faces, history and downloaded models ticked by default). Camera passwords are masked unless ticked. The download is streamed. Restore replaces everything: the upload is checked and extracted next to the live data and its database migrated there; then the runtime is stopped, its database closed (Database.close, waited for, so its worker threads cannot write into the restored one), its models let go (Runtime.release: two sets at once ran the test VM out of memory), the data swapped and a new runtime started. Any failure puts the old data back; a restore cut off by a stop is undone or finished at the next start. Requests made meanwhile wait for the new runtime. Entities of sensors, classes and people that are not in the backup are removed; the others keep their unique IDs. A newer backup is refused. Nothing added later is missing from a backup: every table and data folder must belong to a part (tests/test_backup.py), and a backup made by each database version is kept in tests/assets/backups and restored by every later version (tests/backup_fixture.py adds the next one). Also: ingress_stream, without which the Supervisor refused uploads over 16 MB (videos and sensor exports too), and Database.init no longer lowers user_version when an older version opens a newer database. Tested between two apps on a Home Assistant VM: 485 MB restored in 9 s, every entity kept its ID, faces recognised right after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ease checks A full backup holds the faces and the numbers computed from them unless People and their faces is unticked, and camera passwords only when ticked: README, SECURITY.md, CLAUDE.md, DOCS.md and SCOPE.md say so. README and CLAUDE.md move everything with Settings -> Backup; the checks before a stable release move a backup between two apps both ways. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bundles A review of 0.10.0b1's backup and restore found: - The order in which a restore moved the database and folders aside came from a set. Cut off after a folder but before the database, the next start's clean-up deleted the live database. The database files now always go first, and clean_leftovers removes the database only when it was moved aside (test reproduces the cut-off). - Model files came back from a backup unchecked, and the app loads a model it finds without checking it again: only a registry's file with its SHA-256 is restored now (backbones.all_specs, also used by the model download script). - A newer version can add a choice to a setting without a new database version, so the schema check let such a backup through. Every sensor of a backup is now validated like a sensor bundle import, before anything is stopped: imports.validated_sensor, shared by both. - The backup's database may hold only tables and indexes (a trigger would run with the app's writes). - A sensor bundle given to restore, or a backup given to the sensor import, says where it goes. - The staging folder is left out of Home Assistant's backups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er callback WebKit reported "ResizeObserver loop completed with undelivered notifications" on the dashboard once in CI (iPhone project, 0.10.0b2's first run). bind:clientWidth sets state in the observer's callback, and Svelte then flushes every update the page has waiting - here the status poll's "Learning faces" chip, which resized a card and with it the observed frame - while the browser is still delivering size changes. Reproduced in isolation in WebKit; setting the size in a timeout (or an animation frame) does not report it. DetectionBoxes now measures its layer with its own observer and a timeout (an animation frame never comes in a hidden tab). The pitfall is in CLAUDE.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The history was extracted next to the database and moved to the media folder during the swap, while the app is stopped. On Home Assistant OS both are on one disk and the move is a rename, but the media folder may be network storage: then gigabytes were copied with the app stopped and requests held, long enough for Home Assistant's watchdog to restart the app mid-swap, and the next start put the old data back. Each folder is now staged on its own disk (<media>/.restore for the history, backup._stage_root), so the swap only renames; both staging folders are cleared afterwards and at start-up. A test checks that no file moves between the two disks during the swap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oard ResizeObserver error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three kinds of sensor are short sections instead of a three-column table, each screenshot sits with its section at full width, the Home Assistant screenshots wrap from side by side to stacked, and the code blocks fit a phone without scrolling sideways. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ONNX Runtime's CPU arena kept the largest working memory any image had needed for the rest of the run (D-FINE S about 150 MB on Home Assistant OS), and glibc kept memory freed after a model run as well. Sessions now run without the arena and each check ends with malloc_trim (about 1 ms). With the default models the app is about 190 MB smaller (a test installation with nine sensors and faces: 943 -> 677 MB on average) for about 6 % more time per check. The detectors' footprints are measured again this way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loading scikit-learn and SciPy cost about 90 MB of memory on Home Assistant OS for a few matrix products; the dev installation with nine sensors went from 677 to 537 MB on average. classifier.fit minimises the same objective as LogisticRegression(class_weight="balanced") with lbfgs (L2 on the weights, not the intercept; one vector for two states, softmax for more) with L-BFGS. The problem is convex: on real DINOv2 embeddings the heads matched a fully converged scikit-learn to within 0.05 percentage points, closer than scikit-learn's own default stop. Heads are stored as <id>.npz (no pickle) with a format number. The old pickles (<id>.joblib) are retrained once at startup from the stored embeddings and left in place for an older version; a state sensor whose first model is training publishes nothing instead of "unknown". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… scikit-learn) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…model 0.10.0b4 saved heads only as <id>.npz. Taken back to 0.10.0b3 or 0.9.2 (or restoring a 0.10 backup there), a state sensor had no <id>.joblib and stayed untrained until its next label, or used an old pickle without the newer labels. Every training (and the startup, for heads trained before) now puts a marker in <id>.joblib that older versions cannot read, so they retrain from the stored images. Coming forward again, a head older than the sensor's model is retrained too. Checked on the test Home Assistant: b4 -> b3 -> 0.9.2 -> b4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on /alive - Database.close counts the connections still in use before the pool is disposed of (afterwards it counted none, so a restore never waited for them). - Entities of sensors and people the backup does not have are removed once the restored data runs and MQTT is connected; a restore that fails keeps them. - A value only outside today's limits (api.imports.OutOfLimits) no longer refuses a restore; settings this version does not know still do. - Free disk space is checked on the media folder's disk too when it is another one, and again once the upload is in (an upload without Content-Length was not checked). - The heads folder keeps only <id>.npz and the marker files; an old pickle becomes the marker. - Home Assistant's watchdog asks /api/v1/alive, which is never held while a restore swaps the data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pip-audit blocked the 0.10.0 tag: Pillow 12.3.0 has 24 known vulnerabilities, fixed in 13.0.0. RELEASING.md: the backup check now gives the test sensors what the release adds, restores into an emptied app and back, and compares everything, not only counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes 0.10.0b7 to stable 0.10.0.
New
images, people and their faces, history and downloaded AI models — to move VisionState to
another Home Assistant, or between the stable and the beta app. Everything is ticked; untick what
you do not need (each shows its size). Camera passwords are left out unless you tick them.
IDs in Home Assistant; sensors and people that are not in the backup are removed only once the
restored data runs. The backup is checked first (its version, every sensor's settings, and the
free disk space, also where the history goes when the media folder is on network storage), and
if anything goes wrong, even if the app is stopped in the middle, it keeps what it had. A backup
from an older version is upgraded; one from a newer version is refused (update the app first).
The sensors whose camera password must be entered again are listed. Only the AI models and
trained sensor models VisionState itself makes come back from a backup.
Better
each check instead of being held for the rest of the run, and the state classifier no longer
loads scikit-learn. On a test installation with nine sensors and face recognition the app went
from about 940 MB to about 580 MB. A check takes a few percent longer.
a moment, and the sensors keep their last state meanwhile). Going back to 0.9.2 afterwards
works too: it retrains them the same way.
Fixed
Home Assistant.
longer makes the next update repeat its database upgrade.
security fixes.
Help wanted: Raspberry Pi and other ARM boards
The lower memory use may cost a little more CPU. On an x86 PC the difference is negligible, but on
ARM it has not been measured yet. If you run VisionState on a Raspberry Pi or another ARM board,
please share in issue #52 how memory and CPU
compare with 0.9.2. Even rough numbers or "no difference I can see" help.
🤖 Generated with Claude Code