test-microvm: pause and resume a microVM through OpenVMM state control by esaurez · Pull Request #405 · microsoft/nvx

Enrique Saurez (esaurez) · 2026-10-06T13:34:29Z

Summary

This picks up nanvix/openvmm#114, which adds --microvm-state-control. It is an authenticated host endpoint that pauses, resumes, and queries a running microVM without a snapshot, and it holds guest time while the VM is paused. This PR:

bumps the openvmm gitlink to that PR's head;
specifies host pause in the time ABI;
adds a CI scenario that exercises it on every backend.

Depends on nanvix/openvmm#114. Its base, #404 (AMD CPUs), has merged, so this PR now targets dev directly with its one commit, 77018ef. The gitlink points at the head of nanvix/openvmm#114, 647e635c3 on branch esaurez/microvm-vm-state-control, two commits on nanvix/openvmm#115 (07850d27a, merged, the commit #404 pins). Once #114 merges, I will repoint the gitlink to the merged commit. Until then, it stays a draft.

Changes

openvmm: 07850d27a (time ABI: boot AMD CPUs on an EPYC Milan profile and AMD host profiles #404) → 647e635c3.
scripts/nvx_tools/state_control.py is a client for state-control protocol version 1, which OpenVMM's Guide documents in reference/openvmm/management/state_control_protocol.md. It shares the control console's endpoint streams through a new public EndpointStream and connect_endpoint() in control_session.py.
test-microvm gets a new pause-resume scenario, in the default suite and under --debug-kernel. It runs a managed VM at the largest requested vCPU count:
- A host that does not authenticate, or that presents the wrong capability, gets no response.
- A busy loop spans a 30 s pause, which is longer than the 21 s RCU stall timeout and the debug kernel's 20 s soft-lockup threshold. Two 2 s pauses follow. Every pause and resume is repeated, which must change nothing, and the VM is queried while paused. The transition count must rise by one per change.
- Guest uptime must count only the time the VM ran, to within 1 s. rcu_stall_count must stay 0, and no time ABI violation may occur.
- On Linux hosts, OpenVMM may use at most 10% of a pause, plus 0.2 s, in CPU time while paused.
- The guest wall clock must return to within 1.5 s of host UTC within 150 s, through a wall-clock discipline step.
- A later host sees the same broker instance and transition count. A pause outlives the host that made it. OpenVMM must still be running when it is terminated while paused, and it must exit with the termination status.
- The scenario prints one NVX-PAUSE-RESUME: evidence line and adds about a minute per backend.
Docs:
- doc/design/time-abi.md gets a new "Host pause" section. It covers held monotonic time; the wall clock, which follows host UTC and is stepped by the discipline; the rejection of periodic and TSC-deadline LAPIC timers; and the interaction with snapshots.
- machine-and-device-abi.md, host-attachment-model.md, and concurrency-and-trust-boundaries.md describe the endpoint.
- ci.md and usage.md cover the scenario.
There is no manifest field and no control-contract revision change, because the control-console protocol is unchanged.

Testing

ruff check, ruff format --check, and strict pyright for Linux and Windows are clean.
The CI unittest list passes on Windows: 751 tests on the branch rebased on time ABI: boot AMD CPUs on an EPYC Milan profile and AMD host profiles #404, 735 before. Before the rebase it also passed on Linux (689 tests). The new tests cover:
- the client's wire protocol;
- offline checks of the scenario against a model VM with a fake clock: frozen and advancing guest time, CPU use while paused, replies to unauthenticated hosts, a busy loop that misses the pause, and wall-clock repair with and without a discipline step.

End to end, test-microvm --scenario managed-lifecycle --scenario pause-resume passes at 8 vCPUs on two backends:

MSHV on an AMD EPYC 9V74 (Zen 4) host. No pinned profile covers Zen 4 yet, so a test-only wrapper added --cpu-profile host, which test-microvm cannot pass.
WHP on an AMD EPYC 7763 (Milan) host, on amd.milan.v1.

NVX-PAUSE-RESUME: backend=mshv processors=8 busy_loop_lead_s=3.0 paused_s=34.0 guest_minus_host_running_s=0.007 rcu_stalls=0 wall_clock_lag_after_resume_s=34.4 wall_clock_repair_s=8.1 discipline_steps=1 last_step_ns=34010475477 busy_cpu_s_per_s=1.01 paused_cpu_s=0.00
NVX-PAUSE-RESUME: backend=whp processors=8 busy_loop_lead_s=3.0 paused_s=34.1 guest_minus_host_running_s=0.003 rcu_stalls=0 wall_clock_lag_after_resume_s=34.9 wall_clock_repair_s=8.2 discipline_steps=1 last_step_ns=34099961222

Windows reports no VMM CPU time, so the WHP line omits the two CPU fields.

Not run: KVM and Intel hosts. CI runs the scenario on its Intel runners once this PR leaves draft.

Copilot

Copilot review overview

🔵 Needs a closer look

Cross-platform runtime validation remains pending, the OpenVMM dependency is still draft, and the busy-workload overlap check has an unresolved false-pass path.

Review effort: Balanced
Findings: 1

Open (1)

Overlap check timestamps pause before acknowledgment · New

What changed in this PR

Integrates OpenVMM’s authenticated pause/resume endpoint and validates held guest time across host pauses.

Changes:

Updates the OpenVMM pin and adds a protocol client.
Adds cross-backend pause/resume testing.
Documents endpoint security, lifecycle, and time semantics.

File	Description
`openvmm`	Pins the state-control implementation.
`.github/actions/validate-nvx/action.yml`	Runs the new client tests.
`scripts/nvx_tools/control_session.py`	Exposes shared endpoint streams.
`scripts/nvx_tools/state_control.py`	Implements protocol v1 client.
`scripts/nvx_tools/microvm_tests.py`	Adds the pause/resume scenario.
`scripts/test_state_control.py`	Tests protocol encoding and validation.
`scripts/test_microvm_tests.py`	Tests scenario behavior and dispatch.
`doc/usage.md`	Documents scenario selection.
`doc/ci.md`	Describes CI acceptance checks.
`doc/design/time-abi.md`	Specifies guest time during pauses.
`doc/design/machine-and-device-abi.md`	Defines the endpoint’s ABI role.
`doc/design/host-attachment-model.md`	Documents restore attachment behavior.
`doc/design/concurrency-and-trust-boundaries.md`	Documents authentication and serialization.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        time.sleep(1)
+        busy_after = _process_cpu_seconds(pid)
+        # The VM has run without a pause so far, so its uptime follows the host.
+        pause_uptime = uptime_start + time.monotonic() - host_start


Pick up nanvix/openvmm's authenticated microVM state-control endpoint (--microvm-state-control), which pauses, resumes, and queries a running microVM without a snapshot and holds its guest time while it is paused. Add a client for its protocol, sharing the control console's endpoint streams, and a pause-resume scenario that runs in the default suite and the debug-kernel jobs. On a managed VM it checks that hosts without the capability get no response; that a 30 s pause and two 2 s pauses are idempotent and stop a busy guest; that guest uptime counts only the time the VM ran; that no RCU stall or time ABI violation occurs; that the wall-clock discipline steps the guest clock back to host UTC; that a pause outlives the host that made it; and that OpenVMM can be terminated while the VM is paused. Specify host pause in the time ABI, and describe the endpoint in the machine ABI, the host attachment model, and the trust boundaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🟡 Changes recommended

The CPU-budget check can fail spuriously, and the documented OpenVMM provenance is stale.

Review effort: Balanced
Findings: 2 · 1

Open (3)

Base CPU budget on measured pause duration · New
Overlap check timestamps pause before acknowledgment
Update OpenVMM pin and documented provenance · New

+    used = cpu_after - cpu_before
+    if (
+        used
+        > PAUSE_RESUME_PAUSED_CPU_SHARE * seconds
+        + PAUSE_RESUME_PAUSED_CPU_ALLOWANCE_SECONDS
+    ):
+        raise RuntimeError(
+            f"OpenVMM used {used:.2f} s of CPU time while paused for {seconds:.0f} s"
+        )


+version 1 is documented in OpenVMM's Guide, in
+``reference/openvmm/management/state_control_protocol.md``.


Copilot AI balanced review requested due to automatic review settings October 6, 2026 13:34

Enrique Saurez (esaurez) mentioned this pull request Oct 6, 2026

openvmm: add an authenticated microVM host pause endpoint nanvix/openvmm#114

Draft

Copilot started reviewing on behalf of Enrique Saurez (esaurez) October 6, 2026 13:35 View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Comment thread scripts/nvx_tools/microvm_tests.py

time.sleep(1)

busy_after = _process_cpu_seconds(pid)

# The VM has run without a pause so far, so its uptime follows the host.

pause_uptime = uptime_start + time.monotonic() - host_start

Enrique Saurez (esaurez) force-pushed the esaurez/microvm-state-control branch from b43c5ce to 77018ef Compare October 6, 2026 15:55

Copilot AI balanced review requested due to automatic review settings October 6, 2026 15:55

Enrique Saurez (esaurez) changed the base branch from dev to fix/issue-396-amd-cpu-profiles October 6, 2026 15:55

Copilot started reviewing on behalf of Enrique Saurez (esaurez) October 6, 2026 16:10 View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Base automatically changed from fix/issue-396-amd-cpu-profiles to dev October 6, 2026 17:25

Navigation Menu

Repository navigation

test-microvm: pause and resume a microVM through OpenVMM state control - #405

test-microvm: pause and resume a microVM through OpenVMM state control#405
Enrique Saurez (esaurez) wants to merge 1 commit into
devfrom
esaurez/microvm-state-control

Enrique Saurez (esaurez) commented Oct 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		version 1 is documented in OpenVMM's Guide, in
		``reference/openvmm/management/state_control_protocol.md``.

Conversation

Enrique Saurez (esaurez) commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Enrique Saurez (esaurez) commented Oct 6, 2026 •

edited

Loading