Autonomy Test Harness

A scenario-based test system for robot navigation. It drives Nav2 with its minimal TurtleBot 4 simulation (ROS 2 Jazzy, Gazebo Harmonic) through randomized, versioned scenarios, injects scheduled lidar faults between the simulated sensor and Nav2, judges every run from simulator ground truth (never from the robot's own estimate), and runs headless in Docker under a Jenkins pipeline.

Why it matters. A navigation stack that reports SUCCEEDED is not necessarily at the goal. Testing it means controlled, repeatable stress (the same scenarios with and without each fault), a verdict that does not trust the system under test, and enough runs to put an uncertainty on every pass rate.

Headline result

A paired experiment: the same generated scenarios with no fault and under each lidar fault type, one run per scenario and condition, with Nav2's pass rate and a 95% Wilson confidence interval. Each fault starts 5.0 s after the goal is sent and lasts 10.0 s (scan_dropout, freeze) or 60.0 s (range_noise, ray_dropout, latency, sector_blackout). Because the scenarios are paired, each fault condition is also compared with the no-fault run of the same scenario: "lost / gained" counts scenarios that passed only without the fault or only with it, and the exact McNemar test asks whether that split could be chance.

Condition	Fault window (s after dispatch)	Passed	95% Wilson interval	Lost / gained vs. none	Exact McNemar p	Errored after retry
none	-	30 of 30 (100.0%)	[0.8865, 1.0]	-	-	0
range_noise	5.0-65.0	29 of 30 (96.7%)	[0.8333, 0.9941]	1 / 0	1.0	0
ray_dropout	5.0-65.0	30 of 30 (100.0%)	[0.8865, 1.0]	0 / 0	1.0	0
scan_dropout	5.0-15.0	0 of 30 (0.0%)	[0.0, 0.1135]	30 / 0	1.86e-09	0
latency	5.0-65.0	29 of 30 (96.7%)	[0.8333, 0.9941]	1 / 0	1.0	0
freeze	5.0-15.0	0 of 30 (0.0%)	[0.0, 0.1135]	30 / 0	1.86e-09	0
sector_blackout	5.0-65.0	24 of 30 (80.0%)	[0.6269, 0.9049]	6 / 0	0.0312	0

• Nav2 tolerates noisy, thinned or delayed scans (range noise, ray dropout, latency), loses some runs under a forward sector blackout, and fails every run when new scans stop arriving (whole-scan dropout, frozen scans). In those runs the controller aborted with Unable to transform goal pose into costmap frame because the map to odom transform had gone stale (scan dropout log, freeze log). AMCL publishes that transform when it processes a new scan, so with no new scan the controller can no longer place the goal and gives up rather than continuing on odometry. The Oracle records these runs as timeout (goal not reached), with Nav2's ABORTED in the detail.
• The ground-truth oracle catches false successes. In 5 runs Nav2 reported SUCCEEDED but the robot's true final position was outside the Oracle's 0.3 m tolerance: exp30-013 under range_noise (0.331 m), exp30-004 under sector_blackout (0.336 m), exp30-009 under sector_blackout (0.317 m), exp30-010 under sector_blackout (0.313 m), exp30-016 under sector_blackout (0.402 m). Nav2's goal checker accepts 0.25 m, measured on its own (AMCL) pose estimate. A check based on the robot's own estimate would have passed those runs. The likely cause, inferred but not measured (the AMCL pose is recorded only in the bags, which are not committed), is that the fault shifted AMCL's estimate away from the true pose, so Nav2 stopped where it believed the goal was.

This README is generated by python scripts/build_docs.py from docs/README.template.md and the files under results/; every number in it comes from a generated results file, stamped with the command, seed and commit that produced it (details below).

What I built

• Scenario generator (harness/generator/): draws obstacles, start and goal from a seed, rejects infeasible draws by reason, and writes versioned YAML scenarios that regenerate exactly from their seed.
• Fault injector (harness/faults/): a ROS 2 node between the simulated lidar and Nav2 with scheduled, seeded faults: range noise, ray dropout, whole-scan dropout, latency, freeze, and sector blackout.
• Oracle (harness/oracle/): judges collision, goal reached, and timeout from Gazebo's true poses only, reading the recorded ROS bag.
• Runner (harness/runner/, scripts/robust.sh): one child process per scenario with a hard wall-clock limit and a teardown check; a long step that crashes or stalls is retried with the next method on a ladder, while a real result (a failing test or verdict) is reported as it is.
• Experiment and reporting (harness/experiment/, harness/reporting/): paired conditions, Wilson intervals, resumable runs, JUnit XML with one test case per scenario, and a Markdown summary.
• CI: a Jenkins controller configured entirely as code (ci/jenkins/, ci/Jenkinsfile) that builds the image per commit, runs the unit and property tests, and runs the smoke suite on change and the full suite nightly. A GitHub Actions workflow runs the tests that need neither ROS 2 nor the simulator, and a Ruff lint.

Architecture

flowchart LR
    GEN["Generator<br/>seed -> feasible Scenarios (YAML)"] --> RUN
    subgraph CTR["ath-sim container (CPU rendering, no display)"]
        RUN["Runner<br/>one child process per Scenario,<br/>hard wall-clock limit, teardown check"]
        RUN --> SIM["Gazebo Harmonic + TurtleBot 4<br/>stock tb4_simulation_launch.py"]
        SIM -- "/scan" --> FI["Fault injector node<br/>scheduled faults"]
        FI -- "/scan_ath" --> NAV["Nav2 (unmodified)<br/>AMCL, costmaps, MPPI, collision monitor"]
        SIM -- "Gazebo pose stream" --> GT["Ground-truth reader<br/>/ath/ground_truth/robot_pose"]
        RUN -- "NavigateToPose goal" --> NAV
        GT --> BAG["ROS bag (MCAP)"]
        FI --> BAG
        NAV --> BAG
        BAG --> ORA["Oracle<br/>collision / goal / timeout<br/>from ground truth only"]
    end
    ORA --> REP["Reports<br/>junit.xml + summary.md"]
    REP --> CI["Jenkins<br/>smoke on change, full nightly"]

Loading

Nav2 reads the lidar through a parameter file identical to its stock nav2_params.yaml except the lidar topic names, which point at the injector's /scan_ath; nothing else about Nav2 or the simulation changes. Collisions are judged geometrically from true poses and shapes (the model has no contact sensor).

Quickstart without ROS or Docker

The generator, schema, fault models, oracle geometry, statistics and reporting are plain Python, so their tests run from a fresh clone with uv (it installs the pinned Python and the locked dependencies):

git clone https://github.com/OctaviusLeo/Autonomy-Test-Harness.git
cd Autonomy-Test-Harness
uv sync --frozen
uv run --frozen pytest -m "not sim and not ros"

Regenerate the committed smoke-suite scenarios from their seed and check they match byte for byte:

uv run --frozen python -m harness.generator.cli generate --config config/generator_smoke.yaml --out /tmp/fixed10 --prefix fixed10
diff -r -x generation.json scenarios/fixed10 /tmp/fixed10 && echo identical

Tests marked ros need rclpy and run in the ath-sim image; tests marked sim need the running simulator.

CONTRIBUTING.md lists the checks to run before a commit and how the README and results are regenerated.

Run the full harness

Everything that touches ROS, Gazebo, or Nav2 runs in the ath-sim image. It was developed on WSL 2 with Docker Engine and needs no GPU and no display. Run shell scripts as bash scripts/<name>.sh.

bash scripts/build_image.sh                     # ath-sim:dev from the working tree
bash scripts/test_unit.sh                       # all unit and property tests in the image, with coverage
bash scripts/build_image.sh --tag ath-sim:$(git rev-parse HEAD)   # reported runs use a clean commit
bash scripts/run_suite.sh smoke                 # the smoke suite (scenarios/fixed10)
bash scripts/run_suite.sh full                  # the full suite (fixed10 + experiment30, no faults)
bash scripts/run_experiment.sh                  # the paired experiment (resumable)
bash scripts/rerun_one.sh --config config/generator.yaml --seed <seed> [--condition <name>]

Scenario format

A Scenario is YAML (scenarios/example_valid.yaml is a complete example) with these fields:

Field	Meaning
schema_version	Format version; an unsupported version is rejected by name
scenario_id	Name; the bag and the run records are stored under it
seed	This Scenario's own seed; regenerates it alone
world	Arena (empty_10x10, a walled square arena with a matching map)
time_limit_s	Simulation seconds after goal dispatch within which the goal must be reached
start, goal	{x, y, yaw} in the map frame (equal to the Gazebo world frame), meters and radians
obstacles	Each {shape: box or cylinder, size, pose}; spawned at run time and not on Nav2's map
faults	Each {type, window: {start_s, end_s}, params}; the window is simulation seconds after dispatch

harness/schema/scenario.py defines the bounds; anything out of bounds, of the wrong type, missing, unknown, or of an unsupported version or world is rejected with an error naming the field.

The Generator (python -m harness.generator.cli generate --config <yaml> --out <dir> --prefix <id>) draws obstacles, start, and goal from a seed, rejects infeasible draws by reason (start inside an inflated obstacle, off the map, or goal unreachable by a 4-connected flood fill on the inflated grid), and writes each Scenario with its own seed so it can be regenerated and re-run alone.

Continuous integration

bash scripts/jenkins_up.sh builds and starts a local controller (Compose project ath-jenkins) at http://127.0.0.1:8091/ with anonymous read-only access; it is configured entirely from ci/jenkins/ (Configuration as Code, pinned plugins). The job autonomy-test-harness runs ci/Jenkinsfile from file:///repo (this repository, mounted read-only): it polls for changes every few minutes and runs nightly. Each build builds ath-sim:<commit>, runs the unit tests in it, then runs the smoke suite (on a change) or the full suite (on the timer), publishes the JUnit reports (one test case per Scenario), and archives the summary and the bags of failed runs. It fails if a unit test fails, a Scenario errors, or a smoke-suite Scenario fails its Verdict.

The controller runs on the author's machine, so its builds do not show as GitHub checks. The evidence is committed in ci/jenkins_run_log/: the console logs and JUnit test reports of a green build, a deliberately red build (a failing unit test), and the green build after its revert, plus verification.json and the job's trigger configuration.

These are the commands the pipeline runs (it adds --image ath-sim:<commit> --baked and output paths):

bash scripts/test_unit.sh
bash scripts/run_suite.sh smoke

On GitHub, .github/workflows/tests.yml runs the tests that need neither ROS 2 nor the simulator on a hosted runner for every push and pull request, and lints the repository with Ruff (rules in ruff.toml).

Repository layout

Path	Contents
harness/	All logic: schema, generator, faults, runner, oracle, experiment, reporting, world
scripts/	Thin entry points (image build, suites, experiment, Jenkins helpers, gate checks)
tests/	Unit and property tests (pytest, Hypothesis) with fakes for processes, Docker, Gazebo and Nav2
scenarios/	The committed scenario sets (fixed10, experiment30) and an example
config/	Generator, fault, experiment and run settings; the Nav2 parameter file
docker/, ci/	The ath-sim image, the Jenkins controller and pipeline, and the Jenkins run evidence
results/	Generated results (never edited by hand)
docs/	This README's template, versions with evidence (docs/ENVIRONMENT.md), and run evidence
.kiro/specs/	The requirements, design and task list the project was built from

Results

Experiment details

Total run time: 7742.1 s (2.15 h) over 1 segment(s), the sum of every run's wall-clock duration plus report generation. Base set scenarios/experiment30, set seed 303.

Fault settings: range_noise (sigma_m 0.1; window 5.0-65.0 s after dispatch); ray_dropout (prob 0.5; window 5.0-65.0 s after dispatch); scan_dropout (no parameters; window 5.0-15.0 s after dispatch); latency (latency_ms 300.0; window 5.0-65.0 s after dispatch); freeze (no parameters; window 5.0-15.0 s after dispatch); sector_blackout (center_deg 270.0, width_deg 90.0; window 5.0-65.0 s after dispatch).

Oracle: footprint radius 0.18 m, collision below 0.02 m clearance, goal within 0.3 m and 0.3 rad of the goal, judged from ground truth only.

Generated by bash scripts/run_experiment.sh into results/experiment/experiment.json and results/summary.md. Command bash scripts/run_experiment.sh, seed 303, commit 4849ce0e6aa64c9da66078c9eaddeb5043d76320.

Paired counts, McNemar p-values, false successes and collisions: results/experiment/paired_analysis.json, generated from the committed JUnit files. Command python scripts/paired_analysis.py, seed 303, commit eb50bf52c80a97f6dc01760f7cdf1ccdc24e59ef.

The commit hashes in the result stamps and in the Jenkins evidence come from the history before local IDE files were removed from it; docs/COMMIT_MAP.md maps each one to the commit in this history with the same code.

Smoke suite stability

The 10 fixed Scenarios, 3 times with no Fault: repeat 1: 10 of 10, repeat 2: 10 of 10, repeat 3: 10 of 10 passed; outcome changed between repeats for 0 Scenario(s). So the smoke suite is every fixed10 Scenario (config/smoke_suite.yaml). results/m2_fixed10/summary.json. Command bash scripts/run_suite.sh smoke --repeat 3 --out $ATH_SHARED/runs/m2-fixed10, seed 0, commit 73a0d036dd438fe3616569ed31eac9b999c0b74a.

Fault injection check

One run of exp30-000 per Fault type, each bag checked scan by scan against the Fault's window: range_noise confined True (Verdict passed True); ray_dropout confined True (Verdict passed True); scan_dropout confined True (Verdict passed False); latency confined True (Verdict passed True, measured delay 0.302 s); freeze confined True (Verdict passed False); sector_blackout confined True (Verdict passed True). results/m3_fault_check.json. Command bash scripts/run_suite.sh runs/m3_conditions --out $ATH_SHARED/runs/m3-conditions, seed 0, commit b06c031f8945fd526723032ba29741793be5db1b.

Platform baseline and headless rendering

One NavigateToPose goal from a script in the headless stock simulation: Nav2 reported SUCCEEDED and the true final pose was 0.1358 m from the goal (tolerance 0.3 m); rendering method egl (llvmpipe (LLVM 20.1.2, 256 bits)). results/m0_baseline.json. Command bash scripts/run_m0.sh, seed 0, commit 814a2c39bed7ff27bb2f3c8a87d1d108db3c3047.

Headless-render check in the container with no display: PASS; a box with its near face 2.0 m ahead of the lidar measured 2.0009 m (tolerance 0.05 m), 360 of 360 ranges finite. results/headless_render_check.txt (command bash scripts/check_headless_render.sh, seed 0, commit eab421795f3efad6cf9e0ef2269c98b714bc47ed).

Known limitations

• The injected faults are models, not field data. They are simple, seeded lidar perturbations applied between the simulated sensor and Nav2; real sensor failures look different.
• Software rendering. Sensors render on the CPU with Mesa (llvmpipe); lidar geometry is exact, but timing differs from a GPU setup.
• Not bit-for-bit repeatable. Gazebo and Nav2 timing vary between runs of the same Scenario; the smoke suite's triple run reports how often outcomes changed.
• No contact sensor. The TurtleBot 4 model has none, so collisions are decided from true poses and shapes with a clearance threshold. The recorded collisions are small overlaps of the disc footprint with an obstacle: exp30-008 under sector_blackout (minimum clearance -0.005 m), exp30-011 under sector_blackout (minimum clearance -0.006 m).
• Goal reached means reached once. The Oracle passes a run at the first ground-truth sample within the position and orientation tolerance before the time limit; a robot that passed through the goal and then left it would still pass.
• Frozen scans keep their original timestamp. The freeze fault republishes the last scan before its window unchanged, header stamp included, so Nav2 may discard the stale scans and see the freeze as a dropout. That may explain why freeze and whole-scan dropout gave identical results; it was not checked.
• The latency runs predate a fix. Until commit 70d933c the injector released scans out of order at the end of a latency window: once the window closed, new scans passed straight through while the scans queued in its last 300 ms were released after them. The committed latency-condition runs were made before that fix; only scans in the 300 ms after the window closed were affected.
• One arena, one robot, one parameter set. Results describe Nav2's stock TurtleBot 4 configuration in one walled arena with obstacles that are not on the map.
• One run per scenario and condition. The intervals reflect the sample of scenarios, not run-to-run variation of the same scenario.
• Image rebuilds age. Pinned ROS package versions disappear from the ROS apt repository after its next sync; the image recorded for each run identifies what was tested.
• Odometry drift is not implemented: intercepting odometry and TF would change the simulation's wiring rather than use Nav2's published interfaces.

Versions with evidence are in docs/ENVIRONMENT.md. Code is released under the MIT License. Third-party packages and images are listed with their licenses in THIRD_PARTY.md; apart from a copy of Nav2's stock parameter file and the parameter file derived from it (both Apache-2.0), none are redistributed here. There is no packaging or release.

How this was built

This project was built with Claude Code (Anthropic's agentic coding tool) under a spec-driven workflow: written requirements, then a design, then a task list organized into milestones (the spec files under .kiro/specs/), each milestone ending in an automated gate script that had to pass before the next one started. The commit history shows this openly; commits carry a Co-Authored-By trailer naming the model that helped write them. Every reported number comes from a script in this repository and records the command, seed and commit that produced it, so nothing was typed, estimated or borrowed. I set the goals and constraints in the specification, I am accountable for the verification and the results, and I can explain and change any part of this code.