PhotoGraph

Claim-centric knowledge graphs for personal photo management — the implementation behind the paper of the same name (IJCAI-ECAI 2026 Joint Workshop on GenAIK and NORA).

PhotoGraph turns a photo collection into a knowledge graph you can question and correct. Ask it "Marie and James at the zoo" and it does not score an embedding against your words: it resolves Marie and James to people in the graph, the zoo to a place recorded as one, and returns the photographs where all three hold — with the evidence that says why.

Why a claim graph

A conventional schema records that a photo contains a dog. This one records that according to a vision model, with 0.87 confidence, this region of this image depicts a dog — and that a person later accepted, corrected or rejected that reading. Claims supersede rather than overwrite, so provenance survives and disagreement between models and people is representable rather than lost.

That is what makes the captions explainable: hovering a word in the interface highlights the segmentation mask of the object it refers to, because the token carries the claim id that carries the evidence.

Getting started

Needs PostgreSQL 16 with pgvector, Redis, Node 20 and Python 3.13. Full instructions, including model weights, are in docs/INSTALL.md.

./setup.sh                                            # dependencies, database, schema
./run.sh                                              # backend, worker, frontend, services
backend/.venv/bin/python -m scripts.seed_demo_user    # demo@example.com / demo12345

Then open http://localhost:5173 and upload some photographs. This repository ships none — see Data and privacy.

How it works

frontend (React 19, TanStack Query)
    │
backend (FastAPI) ──── PostgreSQL + pgvector      knowledge graph, embeddings
    │                  Redis                      job queue
    │
    └── worker (arq) ── per-image pipeline, four independently retryable steps
            │
            ├── vlm-service    objects, people, tags, relationships, captions
            ├── face-service   detection, 512-d embeddings, clustering
            ├── sam-service    bbox → segmentation mask
            └── geo-service    GPS → place, landmark confirmation

Each upload creates an ImagePipeline row whose steps are tracked in the database, so progress survives a restart and any step can be retried alone.

Further reading: architecture · knowledge graph · backend · frontend · services · database

Models

Vision-language	Qwen3-VL-30B-A3B-Instruct, Q4_K_M, served by llama.cpp
Face recognition	InsightFace buffalo_l, 512-d embeddings
Segmentation	SAM 3
Geocoding	Google Places (optional; skipped without an API key)

Roughly 20 GB of weights, downloaded by setup.sh. Developed against a single NVIDIA RTX 4090. The vision model is the slow step and dominates throughput; the others run in parallel behind it. Prompts live in captions/service.py and search/llm_parser.py, and are tuned for Qwen3-VL — another model may need them adjusted.

Tests

cd backend  && ./.venv/bin/python -m pytest
cd frontend && npm test

Tests that need photographs skip rather than fail. To run them, drop your own JPEGs into backend/tests/data/, as that directory's README explains.

Limitations

A research prototype, not a product.

• Identities need review. Face matching is deliberately permissive, so automatically assigned names are suggestions. Person centroids are computed only from faces a human has confirmed, which contains the drift without eliminating it.
• Not every face is found. People photographed from behind, at a distance or in poor light often yield no usable face, so they cannot be searched by name. This is the main limit on person queries.
• Extraction sometimes fails, usually a timeout under load. Failed steps are recorded and can be retried from the interface.
• Single node. No horizontal scaling, no rate limiting, JWT auth only.

Data and privacy

This repository contains no photographs, no database and no credentials, by design.

The system processes personal photographs and derives face embeddings from them. Under GDPR those embeddings are biometric data (Art. 9) and the photographs are personal data. If you run this on real images:

• Everything stays on machines you control — except image bytes sent to whichever vision endpoint you configure, and GPS coordinates sent to Google Places if you enable landmark lookup. Both are your choice and both are off unless configured.
• uploads/, logs/, *.db and all image formats are gitignored. Check before you commit.
• Face crops live in backend/uploads/faces/ and embeddings in the kg_nodes table. backend/scripts/delete_all_data.py removes both.
• Obtain consent from the people in your photographs before processing them, and have a lawful basis before processing biometric data.

Citing this work

Omair Shahzad Bhatti, Abdulrahman Mohamed Selim, László Kopácsi, Michael Barz and Daniel Sonntag. PhotoGraph: Claim-Centric Knowledge Graphs for Personal Photo Management. IJCAI-ECAI 2026 Joint Workshop on GenAIK and NORA, 2026.

@inproceedings{bhatti2026photograph,
  title     = {PhotoGraph: Claim-Centric Knowledge Graphs for Personal Photo
               Management},
  author    = {Bhatti, Omair Shahzad and Selim, Abdulrahman Mohamed and
               Kop{\'a}csi, L{\'a}szl{\'o} and Barz, Michael and
               Sonntag, Daniel},
  booktitle = {IJCAI-ECAI 2026 Joint Workshop on Generative AI and Knowledge
               Graphs (GenAIK) and Neuro-symbolic Reasoning and Applications
               (NORA)},
  year      = {2026},
  url       = {https://openreview.net/forum?id=7q0HDlrVfs}
}

Licence

CC BY-NC 4.0 — share and adapt for non-commercial purposes with attribution. The paper is separately licensed CC BY 4.0 on OpenReview.