I spent five years building backend services and REST APIs — Go and Python, PostgreSQL, Docker, Kubernetes — and I now write the coding tasks and verifiers used to train and evaluate AI agents.
Most of that work comes down to one question: did this actually get fixed, or does it just look fixed? I take a real repository, plant a defect, write the gold solution, and build the grader that decides whether a fix is real. The grader is the hard part. It has to pass any reasonable solution and reject everything else — including the ones that make the symptom disappear without touching the cause.
So I write the grader, then I attack it: what is the laziest change that turns this green, and would my checks catch it?
20+ task environments built this way, 4 approved through both automated and human review.
swe-arrow-range
A containerized bug-fix task built from a real commit in the arrow library, shipped
with the cheat solutions already written. Delete the test file, exit zero from conftest,
skip everything, monkeypatch the library in memory — five of them, each proven to score
zero on every push. The grader takes only your changes to library source and replays
them onto a clean checkout, so nothing you leave elsewhere in the tree comes along.
Python · Docker · pytest · GitHub Actions
root-cause-diagnosis · on the Hub 18 backend and infrastructure incidents where the obvious answer is the symptom, not the cause. Every case carries a decoy that is true, visible in the evidence, and still wrong. Naming the symptom scores −0.40 against +1.10 for the real cause. Python · verifiers v1
incident-triage Multi-turn investigation over MCP tools. The evidence is hidden behind correct tool use — the queue backlog only appears if you ask for debug logs. The trajectory is scored, not just the answer: a correct guess made without calling a single tool scores below a half-correct answer reached by investigating. Python · verifiers v1 · MCP
Codalyra-AI — multi-agent code review. Four specialist agents (logic, security, performance, quality) review a diff and a synthesis agent merges their findings. A single-prompt baseline runs alongside every review, so the extra structure is measured rather than assumed. Python · FastAPI · React · PostgreSQL · Redis · Celery · Docker