Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
d5491e3
Back to the beta channel after 0.9.2
oleost Oct 19, 2026
b757829
Frontend: source-map-js 1.2.2 (GHSA-68fv-2mgg-jv7q)
oleost Oct 19, 2026
31b92bf
Merge remote-tracking branch 'origin/main' into beta
oleost Oct 19, 2026
32065ef
0.10.0b1: backup of everything, and restoring it
oleost Oct 19, 2026
b052456
Docs: backups in the privacy notes, the contributor guide and the rel…
oleost Oct 19, 2026
90bd78b
0.10.0b2: a restore cut off keeps the database; backups trusted like …
oleost Oct 19, 2026
bab1af1
Frontend: measure the detection boxes' layer outside the ResizeObserv…
oleost Oct 19, 2026
fd3a37c
Backup: extract each folder on the disk it goes to
oleost Oct 19, 2026
e640ce3
0.10.0b3: restoring with the media folder on network storage; a dashb…
oleost Oct 19, 2026
9e5e4f5
README: a layout that reads well on phones
oleost Oct 20, 2026
96d02f3
Memory: no ONNX Runtime arena, give freed memory back after every check
oleost Oct 20, 2026
9000901
Classifier: logistic regression in NumPy instead of scikit-learn
oleost Oct 20, 2026
62742e7
0.10.0b4: much less memory (no ONNX Runtime arena, classifier without…
oleost Oct 20, 2026
41967d4
Heads: older versions retrain after going back instead of finding no …
oleost Oct 20, 2026
3edc4d1
0.10.0b5: older versions retrain on data from 0.10.0b4
oleost Oct 20, 2026
ba6e6fc
Restore: wait for the database, keep entities until it runs, watchdog…
oleost Oct 20, 2026
2193e92
0.10.0b6: restoring a backup, watchdog and Home Assistant entities
oleost Oct 20, 2026
5511a6d
0.10.0b7: Pillow 13.0.0; a full backup round trip before every promote
oleost Oct 22, 2026
b83de96
0.10.0: backup and restore, less memory
oleost Oct 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,13 @@ Every fact has **one** home; other files link to it instead of repeating it.
- **No telemetry, nothing "phones home".** Download counts on GHCR are the only usage signal.
- **Only reachable through Home Assistant Ingress** (requests from other addresses are refused);
outside Home Assistant the app has no login — documented, keep it that way.
- **Camera credentials never leak:** `redact.py` masks them in logs, errors and exports.
- **Camera credentials never leak:** `redact.py` masks them in logs, errors and exports (a full
backup holds them only when the user ticks *Camera passwords*, unticked by default).
- **No hard-coded defaults** outside `settings.py` / the registries; the UI reads them from
`GET /api/v1/config`.
- **A backup holds everything:** every table and data folder belongs to a part of
`settings.BACKUP_PARTS` (tests fail otherwise), and a new database version gets a kept backup in
`tests/assets/backups/` (`python -m tests.backup_fixture`) that every later version restores.
- Home Assistant state values stay lower-case keys (`open`, `car_parked`); unique IDs and topics
never change once released.
- Marketing text (README, release notes) promotes VisionState on its own merits — no comparisons
Expand Down Expand Up @@ -118,7 +122,7 @@ itself (`__main__.py`).
and `VISIONSTATE_MEDIA` folder. Home Assistant then builds the Dockerfile itself, the same image
as a release (*Local apps* in the app store; after a change: *Rebuild*). It shares MQTT topics and
unique IDs with an installed VisionState, so never run both at once; it has its own data (move
sensors with export/import). This is how changes are tried on Home Assistant during development;
everything with Settings → Backup, or single sensors with export/import). This is how changes are tried on Home Assistant during development;
a beta is released when the maintainer asks for one.

## UI testing routine
Expand Down Expand Up @@ -151,7 +155,7 @@ Many users run Home Assistant on phones and tablets.
`--app <slug>` (photos from GitHub). Running it twice changes nothing.
- **Docs screenshots** (`docs/promo/`): seed the demo sensors on a local backend with a local MQTT
broker (so the header shows *MQTT connected*), then `node e2e/promo.mjs <app URL>` in
`visionstate/frontend` (1440 px wide, device scale 1.2, one row of the README the same height).
`visionstate/frontend` (1440 px wide, device scale 1.2).
The Home Assistant screenshots (`ha-*.png`) are taken by hand on a real Home Assistant. Show the
maintainer before publishing.

Expand All @@ -173,5 +177,10 @@ Many users run Home Assistant on phones and tablets.
`untrack`, name the real dependencies with `void x`, and give a polling loop a counter so a
restart ends the old loop. The e2e test "no page asks the server more often than it polls"
counts requests per page.
- State set in a ResizeObserver's callback — which is what `bind:clientWidth` does — flushes every
update the page has waiting while the browser is still delivering size changes; when one of
them resizes what is observed, WebKit reports "ResizeObserver loop completed with undelivered
notifications" (seen once in CI on the dashboard, never locally). Measure in a timeout instead,
like `measure` in `DetectionBoxes.svelte` (not the next animation frame: none comes in a hidden tab).
- The Chrome tab Claude drives counts as hidden (`document.visibilityState`); views that only poll
while visible look fine there. Use `setVisibility` (e2e) or override it by hand.
86 changes: 52 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,26 +31,33 @@

## Three kinds of sensor

| | What you get | How |
|---|---|---|
| 🚪 **States** | *Garage door open, closed or halfway*, gate shut, lights on in the shed | Click a few examples of each state; it learns in about a second |
| 🧍 **Objects & people** | *Emma is home*, *our car is in the driveway*, a cat on the lawn, an unknown person at the door | Pick from 80 objects — no training. Name people by their face once; every camera knows them |
| 🔢 **Readings** | The water, gas or power meter — a cheap ESP32-CAM above it is enough —, the minutes left on the washer | Draw a box around the number. Counters only go up and land in the Energy dashboard |
### 🚪 States

*Garage door open, closed or halfway*, the gate shut, the lights on in the shed.

Click a few examples of each state; it learns in about a second.

### 🧍 Objects & people

*Emma is home*, *our car is in the driveway*, a cat on the lawn, an unknown person at the door.

Pick from 80 objects — no training. Name people by their face once; every camera knows them.

<p align="center">
<img src="docs/promo/objects.png" alt="An object sensor recognising Emma by her face and telling our car from any car" width="900">
<img src="docs/promo/objects.png" alt="An object sensor recognising Emma by her face and telling our car from any car" width="900"><br>
<em>Emma recognised by her face, and our own car told apart from any other car.</em>
</p>

<table>
<tr>
<td width="50%" valign="top"><img src="docs/promo/reading.png" alt="A reading sensor reading the wheels of a real water meter" width="100%"></td>
<td width="50%" valign="top"><img src="docs/promo/people.png" alt="The People page with the faces taught for Emma" width="100%"></td>
</tr>
<tr>
<td valign="top"><b>Real meters</b> — the rolling wheels of a water meter, read right even while a wheel is half way between two digits.</td>
<td valign="top"><b>People</b> — a few photos from your phone are enough to start; faces never leave your Home Assistant.</td>
</tr>
</table>
<p align="center">
<img src="docs/promo/people.png" alt="The People page with the faces taught for Emma" width="900"><br>
<em>A few photos from your phone are enough to start; faces never leave your Home Assistant.</em>
</p>

### 🔢 Readings

*The water, gas or power meter*, the minutes left on the washer.

Draw a box around the number. Counters only go up and land in the Energy dashboard.

## Read a water, gas or power meter with an ESP32-CAM

Expand All @@ -61,6 +68,11 @@ before it is published (a counter never goes down or jumps), and the total lands
dashboard — with a rate (m³/h, kW) if you want one. For a dark meter cabinet it switches a light
on for each reading; a battery camera can trigger a check when it has a new picture.

<p align="center">
<img src="docs/promo/reading.png" alt="A reading sensor reading the wheels of a real water meter" width="900"><br>
<em>The rolling wheels of a real water meter, read right even while a wheel is half way between two digits.</em>
</p>

## Highlights

- 🧠 **100 % local on any CPU** — Intel, AMD, Raspberry Pi 4/5. No cloud, no GPU, no subscription, no telemetry.
Expand All @@ -72,28 +84,27 @@ on for each reading; a battery camera can trigger a check when it has a new pict

## In Home Assistant

<table>
<tr>
<td width="50%" valign="top"><img src="docs/promo/ha-states.png" alt="A garage door state sensor as a Home Assistant device" width="100%"></td>
<td width="50%" valign="top"><img src="docs/promo/ha-objects.png" alt="The driveway object sensor as a Home Assistant device, with Emma and our car" width="100%"></td>
</tr>
<tr>
<td valign="top"><b>State sensor</b> — the state with its confidence, the last frame, a button to check now and a switch to pause it.</td>
<td valign="top"><b>Object sensor</b> — an on/off sensor and a count for every object, person and own label.</td>
</tr>
</table>
<p align="center">
<img src="docs/promo/ha-states.png" alt="A garage door state sensor as a Home Assistant device" width="340">
<img src="docs/promo/ha-objects.png" alt="The driveway object sensor as a Home Assistant device, with Emma and our car" width="340">
</p>

- **State sensor** — the state with its confidence, the last frame, a button to check now and a switch to pause it.
- **Object sensor** — an on/off sensor and a count for every object, person and own label.

Use them like any other sensor — for example, tell me when the garage door has been open for
10 minutes:

```yaml
# Example: tell me when the garage door has been open for 10 minutes
triggers:
- trigger: state
entity_id: sensor.garage_door
to: open
for: "00:10:00"
actions:
- action: notify.mobile_app_phone
- action: notify.notify
data:
message: The garage door has been open for 10 minutes.
message: Garage door open!
```

## Install
Expand All @@ -108,16 +119,22 @@ with the **MQTT** integration, and a camera.
fresh frame right away.

New features come to a separate **beta** app first: add `https://github.com/oleost/VisionState#beta`
instead. Run only one of the two at a time (they publish the same sensors); move sensors with
**Export** and **Import**.
instead. Run only one of the two at a time (they publish the same sensors); move everything with
**Settings → Backup**, or single sensors with **Export** and **Import**.

The user guide — every entity, triggers, faces, readings, review — is on the app's
*Documentation* tab ([`visionstate/DOCS.md`](visionstate/DOCS.md)).

## How it works

```
camera ─► your region ─► pretrained model (ONNX, CPU) ─► small per-sensor model ─► debounce ─► MQTT ─► Home Assistant
camera
▼ your region
▼ pretrained model (ONNX, CPU)
▼ small per-sensor model
▼ debounce
▼ MQTT
Home Assistant
```

- **States:** [DINOv2](https://github.com/facebookresearch/dinov2) turns the region into a feature
Expand All @@ -135,9 +152,10 @@ camera ─► your region ─► pretrained model (ONNX, CPU) ─► small per-s

- Nothing leaves your machine; there is no telemetry.
- Only reachable through Home Assistant Ingress (a Home Assistant login).
- Camera passwords are hidden in logs and removed from exports.
- Camera passwords are hidden in logs and removed from exports; a backup holds them only if you
tick it.
- Face recognition is off until you turn it on for a camera; faces are kept in the app's own
storage and exported only if you tick it.
storage and exported only if you tick it (a backup holds them unless you untick it).

## Contributing

Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ Supported: the latest stable release and the current beta.

What VisionState relies on, so you know what counts as a vulnerability: it is only reachable
through Home Assistant Ingress (other requests are refused); camera credentials are never written to
logs, errors or exports; nothing is sent anywhere but to your own Home Assistant and MQTT broker.
logs, errors or exports (a full backup holds them only when the user ticks *Camera passwords*); nothing is sent anywhere but to your own Home Assistant and MQTT broker.
17 changes: 12 additions & 5 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ docs/ SCOPE.md, ARCHITECTURE.md, RELEASING.md, WHEEL_READER_PL
```

**Stack:** Python 3.14, FastAPI, uvicorn, SQLAlchemy (SQLite), asyncio; ONNX Runtime (CPU only),
NumPy, Pillow, PyAV (bundled FFmpeg), scikit-learn; aiomqtt, websockets, httpx. Frontend: Svelte 5 +
NumPy, Pillow, PyAV (bundled FFmpeg); aiomqtt, websockets, httpx. Frontend: Svelte 5 +
Vite + TypeScript, plain CSS with design tokens, a hash router (Ingress-safe), bundled fonts. Docker
image `python:3.14-slim`, built by GitHub Actions and published on GHCR. Quality: pytest (with a fake
camera, Home Assistant and MQTT), ruff, pyright, svelte-check, Playwright, an image smoke test.
Expand Down Expand Up @@ -52,7 +52,7 @@ API as `request.app.state.runtime`.
| `db.py` | SQLAlchemy tables and the schema migrations (`MIGRATIONS`, `SCHEMA_VERSION`) |
| `storage.py` | Image files on disk: training samples, history frames, thumbnails |
| `engine/` | The runtime — see below |
| `api/` | The REST API (`/api/v1/…`), one file per topic: `sensors.py` (CRUD, quality, export), `history.py` (the history: one filter for the list, its count and facets; history frames), `samples.py` (training images), `teach.py` (taught boxes; what is taught from a history frame is written into its row: `remember_taught`), `faces.py` (teaching a face on a sensor; the people and their faces, shared by every sensor), `cameras.py` (camera and entity lists, the light, previews), `review.py` (review queue and answers), `imports.py` (import), `system.py` (UI config, status, AI models and deleting downloaded ones, storage, review rules, review reminder, teaching and face limits), `common.py` (input models and views shared by them) |
| `api/` | The REST API (`/api/v1/…`), one file per topic: `sensors.py` (CRUD, quality, export), `history.py` (the history: one filter for the list, its count and facets; history frames), `samples.py` (training images), `teach.py` (taught boxes; what is taught from a history frame is written into its row: `remember_taught`), `faces.py` (teaching a face on a sensor; the people and their faces, shared by every sensor), `cameras.py` (camera and entity lists, the light, previews), `review.py` (review queue and answers), `imports.py` (import), `backup.py` (backup of everything, restore), `system.py` (UI config, status, `alive` for Home Assistant's watchdog, AI models and deleting downloaded ones, storage, review rules, review reminder, teaching and face limits), `common.py` (input models and views shared by them) |
| `sources.py` | Camera sources: Home Assistant camera, HTTP snapshot, RTSP; address checks and size limits; the Home Assistant REST client (states, services, notify services, the app's page) |
| `mqtt.py` | MQTT topics, Home Assistant discovery messages, the bridge (connect, publish, commands) |
| `ha_events.py` | Home Assistant WebSocket: state changes of trigger entities, entity registry |
Expand All @@ -64,6 +64,7 @@ API as `request.app.state.runtime`.
| `teach.py` | Object sensors: comparing detected boxes with the boxes the user taught |
| `imaging.py` | Decoding, regions (ROI), crops, change signatures, drawing boxes |
| `uploads.py`, `bundle.py` | Uploaded files (images, ZIP, video) → frames; sensor export/import bundles |
| `backup.py` | A backup of everything (database copy + files, by the parts of `settings.BACKUP_PARTS`) and restoring one: staged, swapped in, undone on failure |
| `redact.py` | Masks credentials in URLs, messages and log lines |

### `engine/`
Expand Down Expand Up @@ -127,7 +128,7 @@ error is kept in `LiveState.error` and shown in the UI.
the media folder (see `storage.Storage.face_path`).
- **Files**: `<data>/samples/<sensor>/` (training images; for object sensors the crops of taught
boxes — next to the database that lists them, so a backup holds both), `<data>/thumbs/sample/`,
`<data>/heads/` (trained heads), `<data>/models/` (downloaded models; models and thumbnails are
`<data>/heads/` (trained heads, `<sensor id>.npz`), `<data>/models/` (downloaded models; models and thumbnails are
left out of backups); `<media>/history/<sensor>/` and `<media>/thumbs/history/`. Versions before
the move kept training images in `<media>/samples/`: they are read there until
`engine.HistoryMixin.move_samples` has copied them (`settings.SAMPLES_MOVE`). A file is removed
Expand All @@ -146,7 +147,7 @@ error is kept in `LiveState.error` and shown in the UI.
|---|---|
| `lib/api.ts` | The only place that builds API URLs; one function per endpoint |
| `lib/types.ts` | Types of what the API returns |
| `lib/app.svelte.ts` | Shared state: config from `GET /api/v1/config`, status polling |
| `lib/app.svelte.ts` | Shared state: config from `GET /api/v1/config`, status polling, the last restored backup (Settings is made again with it) |
| `lib/router.svelte.ts` | Hash routes (`paths`), a page's query (`route.query`, `setQuery`) |
| `lib/history.ts` | The history filter: from and to the page URL, and as the API takes it |
| `lib/ui.ts` | UI constants (tabs per sensor kind, wizard texts); `tokens.css` colours, type, spacing |
Expand All @@ -172,7 +173,13 @@ config in the builder of its kind in `mqtt.py` (`_state_entities`, `_object_enti
in Home Assistant hang on them).

**A new column**: the field in `db.py`, a new `SCHEMA_VERSION` with its entry in `MIGRATIONS`, and
a test that an old database is upgraded (see `tests/test_entity_ids.py`).
a test that an old database is upgraded (see `tests/test_entity_ids.py`). Then a backup made by the
new version: `python -m tests.backup_fixture` (in `visionstate/backend`), committed —
`tests/test_backup.py` asks for it.

**A new table or folder of data**: its part in `settings.BACKUP_PARTS` (a folder that is made again
by itself: `BACKUP_LEFT_OUT`), so backups hold it; `tests/test_backup.py` fails until it is there.
A folder is a `Settings` property under the data or media folder.

**A new API endpoint**: in the `api/` file of its topic, input as a pydantic model, its function
in `lib/api.ts`, its type in `lib/types.ts`.
Expand Down
11 changes: 10 additions & 1 deletion docs/RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,13 +77,22 @@ all of it and tell the maintainer what was checked and what was found.
- restart Home Assistant, reboot the host and restart the MQTT broker: the app and its entities
come back by themselves;
- make a backup of the app and restore it: the sensors are back, with their training images;
- **backup round trip** (Settings → Backup): first give the standing test sensors what the
release adds (a new setting, a new kind of data, a person…), so that the backup has to carry
it. Download a full backup from the beta app (camera passwords ticked), restore it into a
freshly installed app (a local app with its data and media folder emptied first) and back into
the beta app. Compare both sides: every sensor with all its settings, its training images and
what it was taught, people and their faces, the app's settings and limits, downloaded models,
history, and the Home Assistant entities with their IDs. Anything that does not come back
blocks the release;
- **soak for 1 hour** with the sensors checking: the app's memory (Supervisor app stats) and its
data on disk level off instead of growing, the log stays quiet;
- open the app in Home Assistant at desktop width and on a phone (Ingress);
- prepare the **update from the last stable release**: the stable app is installed there too. It
shares MQTT topics and unique IDs with the beta app, so never run both: stop the beta app,
start the stable one, and give it one sensor of each kind (exports of the standing test sensors,
or `scripts/seed_demo.py --app <slug>` once the stable release can teach faces from photos).
`scripts/seed_demo.py --app <slug>`, or a backup of the beta app once the stable release can
restore one).
3. **What changed since the last stable release**: new or upgraded dependencies have a licence that
fits Apache-2.0 (`git diff vX.Y.Z -- visionstate/backend/requirements.txt
visionstate/frontend/package.json`); `homeassistant:` in config.yaml still names the oldest Home
Expand Down
Loading
Loading