Skip to content

feat(metrics): report VMM thread CPU usage - #6264

Open
Richard-Durkee wants to merge 1 commit into
firecracker-microvm:mainfrom
Richard-Durkee:feat/vmm-thread-rusage-metrics
Open

Richard-Durkee wants to merge 1 commit into
firecracker-microvm:mainfrom
Richard-Durkee:feat/vmm-thread-rusage-metrics

Conversation

@Richard-Durkee

@Richard-Durkee Richard-Durkee commented Oct 12, 2026 •

Copy link
Copy Markdown

Changes

Add vmm.thread_cpu_us: the CPU time the VMM thread used since the last metrics flush. It is read with clock_gettime(CLOCK_THREAD_CPUTIME_ID) when the metrics are written.

Reason

All device emulation runs on the VMM thread, but no metric shows how busy that thread is. thread_cpu_us / flush interval gives its utilization, so a saturated VMM thread can be spotted without host tools like top -H.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check CONTRIBUTING.md.

PR Checklist

  • I have read and understand CONTRIBUTING.md.
  • I have run tools/devtool checkbuild --all to verify that the PR passes
    build checks on all supported architectures.
  • I have run tools/devtool checkstyle --no-clippy to verify that the PR
    passes the automated style checks.
  • I have described what is done in these changes, why they are needed, and
    how they are solving the problem in a clear and encompassing way.
  • I have updated any relevant documentation (both in code and in the docs)
    in the PR.
  • I have mentioned all user-facing changes in CHANGELOG.md.
  • If a specific issue led to this PR, this PR closes the issue.
  • When making API changes, I have followed the
    Runbook for Firecracker API changes.
  • I have tested all new and changed functionalities in unit tests and/or
    integration tests.
  • I have linked an issue to every new TODO.

  • This functionality cannot be added in rust-vmm.

@Richard-Durkee
Richard-Durkee force-pushed the feat/vmm-thread-rusage-metrics branch from cff330e to 9c64355 Compare October 12, 2026 08:11
@ilstam ilstam self-assigned this Oct 14, 2026
@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.35%. Comparing base (329519d) to head (9c64355).

⚠️ Current head 9c64355 differs from pull request most recent head 731f435

Please upload reports for the commit 731f435 to get more accurate results.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6264      +/-   ##
==========================================
- Coverage   83.59%   83.35%   -0.24%     
==========================================
  Files         282      278       -4     
  Lines       33585    31980    -1605     
==========================================
- Hits        28074    26656    -1418     
+ Misses       5511     5324     -187     
Flag Coverage Δ
5.10-m5n.metal 83.49% <100.00%> (-0.24%) ⬇️
5.10-m6a.metal 82.87% <100.00%> (-0.31%) ⬇️
5.10-m6g.metal 80.46% <100.00%> (-0.42%) ⬇️
5.10-m6i.metal 83.49% <100.00%> (-0.26%) ⬇️
5.10-m7a.metal-48xl 82.86% <100.00%> (-0.30%) ⬇️
5.10-m7g.metal 80.46% <100.00%> (-0.42%) ⬇️
5.10-m7i.metal-24xl 83.47% <100.00%> (-0.24%) ⬇️
5.10-m7i.metal-48xl 83.47% <100.00%> (-0.26%) ⬇️
5.10-m8g.metal-24xl 80.46% <100.00%> (-0.40%) ⬇️
5.10-m8g.metal-48xl 80.46% <100.00%> (-0.40%) ⬇️
5.10-m8i.metal-48xl 83.47% <100.00%> (-0.25%) ⬇️
5.10-m8i.metal-96xl 83.46% <100.00%> (-0.26%) ⬇️
5.10-m9g.metal-48xl 80.46% <100.00%> (-0.42%) ⬇️
6.1-m5n.metal 83.62% <100.00%> (-0.23%) ⬇️
6.1-m6a.metal 83.02% <100.00%> (-0.29%) ⬇️
6.1-m6g.metal 80.46% <100.00%> (-0.41%) ⬇️
6.1-m6i.metal 83.62% <100.00%> (-0.24%) ⬇️
6.1-m7a.metal-48xl 83.01% <100.00%> (-0.28%) ⬇️
6.1-m7g.metal 80.46% <100.00%> (-0.41%) ⬇️
6.1-m7i.metal-24xl 83.64% <100.00%> (-0.22%) ⬇️
6.1-m7i.metal-48xl 83.64% <100.00%> (-0.24%) ⬇️
6.1-m8g.metal-24xl 80.45% <100.00%> (-0.39%) ⬇️
6.1-m8g.metal-48xl 80.46% <100.00%> (-0.41%) ⬇️
6.1-m8i.metal-48xl 83.64% <100.00%> (-0.24%) ⬇️
6.1-m8i.metal-96xl 83.64% <100.00%> (-0.24%) ⬇️
6.1-m9g.metal-48xl 80.46% <100.00%> (-0.41%) ⬇️
6.18-m5n.metal 83.63% <100.00%> (-0.22%) ⬇️
6.18-m6a.metal 83.01% <100.00%> (-0.29%) ⬇️
6.18-m6g.metal 80.56% <100.00%> (-0.39%) ⬇️
6.18-m6i.metal 83.63% <100.00%> (-0.24%) ⬇️
6.18-m7a.metal-48xl 83.01% <100.00%> (-0.28%) ⬇️
6.18-m7g.metal 80.56% <100.00%> (-0.39%) ⬇️
6.18-m7i.metal-24xl 83.64% <100.00%> (-0.23%) ⬇️
6.18-m7i.metal-48xl 83.64% <100.00%> (-0.24%) ⬇️
6.18-m8g.metal-24xl 80.56% <100.00%> (-0.39%) ⬇️
6.18-m8g.metal-48xl 80.56% <100.00%> (-0.41%) ⬇️
6.18-m8i.metal-48xl 83.64% <100.00%> (-0.24%) ⬇️
6.18-m8i.metal-96xl 83.64% <100.00%> (-0.24%) ⬇️
6.18-m9g.metal-48xl 80.56% <100.00%> (-0.41%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Richard-Durkee
Richard-Durkee force-pushed the feat/vmm-thread-rusage-metrics branch from 9c64355 to f998e10 Compare October 15, 2026 18:16
Comment thread src/vmm/src/logger/metrics.rs Outdated
/// Metric for signaling a panic has occurred.
pub panic_count: SharedStoreMetric,
/// CPU time the VMM thread used since the last metrics flush, in microseconds.
pub thread_cpu_us: SharedIncMetric,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we name it "main_thread_cpu_us" for clarity? Similar for the other variables and function names (last_main_thread_cpu_us, update_main_thread_cpu_usage).

/// The metrics are flushed from the event loop, so the calling thread is the VMM thread.
fn update_thread_cpu_usage(&mut self) {
let thread_cpu_us = get_time_us(ClockType::ThreadCpu);
METRICS

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of adding to it every time, can't we just set it to get_time_us(ClockType::ThreadCpu) directly?

The main thread runs the event loop and emulates every device that has
no worker thread of its own, but no metric shows how busy it is, so a
saturated main thread can only be spotted with host tools. Add
vmm.main_thread_cpu_us, the CPU time the main thread has used, read
from CLOCK_THREAD_CPUTIME_ID each time the metrics are written. The
seccomp filters already allow clock_gettime.

Signed-off-by: Richard Durkee <56232628+Richard-Durkee@users.noreply.github.com>
@Richard-Durkee
Richard-Durkee force-pushed the feat/vmm-thread-rusage-metrics branch from f998e10 to 731f435 Compare October 29, 2026 04:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants