From 180df311e2f9975610c02d2a8c1c5303d66f6bf6 Mon Sep 17 00:00:00 2001 From: Thanos Makatos Date: Fri, 23 Oct 2026 08:11:11 +0000 Subject: [PATCH 1/2] Document every controller and threshold engine setting Add a Configuration section that explains where configuration files are read from, how defaults apply, and every controller and threshold engine setting, and move the threshold validation and backoff description into it. AI-generated: everything. --- README.md | 153 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 130 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index e35b01c..d0da2f5 100644 --- a/README.md +++ b/README.md @@ -84,13 +84,32 @@ managed and unmanaged classifications across daemon restarts. +------------------------------------------+ ``` -## Configuration includes +## Configuration + +The daemon reads its controller configuration from +`/etc/io-thread-controller.json`, or from the file given with `--config`. The +controller configuration names the scaling engine and the directories holding +the engine and backend files: the active engine reads +`/.json`, and each backend reads its own file from +`backend_config_dir`. Every file is JSON, and unknown keys are rejected so that +misspelt settings fail loudly. + +If the controller file does not exist, the daemon runs with built-in defaults, +which `--dump-config` prints. A missing engine file likewise selects the +engine's defaults. Settings omitted from a file that does exist take the +defaults listed below. + +At startup the daemon logs the effective controller, engine, and backend +configuration, after defaults, command-line overrides, and duration rounding +have been applied. + +### Includes Controller, engine, and backend JSON files may contain an `include` field. Its path is resolved relative to the including file, loaded recursively, and merged before the including file's keys are applied. -## Durations +### Durations Durations are integer seconds in settings ending in `_secs`. Every duration, including engine ones, must be a whole multiple of `scale_poll_secs`, and zero @@ -100,9 +119,115 @@ therefore invalidate other settings. To try a new poll period without rewriting the other settings, set `round_up_durations` or pass `--round-up-durations`. Each offending duration is -then rounded up to the next multiple with a warning. At startup the daemon logs -the effective controller, engine, and backend configuration, which shows the -rounded values. +then rounded up to the next multiple with a warning, and the effective +configuration logged at startup shows the rounded values. + +Percentages are written as numbers from 0 to 100 in settings ending in +`_percent`. + +### Controller + +The first five settings are required when the controller file exists. + +| Setting | Default | Meaning | +|---|---|---| +| `engine` | required | Scaling engine name, e.g. `threshold`. | +| `engine_config_dir` | required | Directory holding `.json`. | +| `backend_config_dir` | required | Directory holding backend files, e.g. `qemu.json`. | +| `vm_state_path` | required | VM ownership file. | +| `scale_poll_secs` | required | Poll period in seconds; must be at least 1. | +| `round_up_durations` | `false` | Round durations up to poll multiples instead of rejecting them. | +| `min_thread_count` | `1` | Fewest worker threads the controller will set. | +| `max_thread_count` | `8` | Most worker threads the controller will set. | +| `host_cpu_scale_up_ceiling_percent` | `90` | Host CPU use that blocks scale-up; `0` disables. | +| `cooldown_secs` | `30` | Per-VM pause after a successful scale action. | +| `max_instances_adjusted_per_poll` | `0` | Most VMs resized per poll; `0` means no limit. | +| `dry_run` | `false` | Log decisions without resizing pools. | +| `enable_per_vm_status_line` | `true` | Write a status line per VM every poll. | +| `enable_per_vm_main_log` | `false` | Send per-VM status lines to the main log instead of per-VM files. | +| `enable_aggregate_status_line` | `true` | Write one fleet-wide status line every poll. | +| `print_status_header` | `false` | Log a legend of the status-line fields at startup. | +| `refresh_cgroup_on_each_read` | `false` | Re-read the VM's cgroup CPU limits on every refresh. | +| `experimental_psi_monitoring` | `false` | Sample host pressure-stall information every poll. | + +Every `scale_poll_secs` the controller refreshes each VM, asks the engine for +decisions, and applies them. The thread-count bounds are hard limits: an +engine proposal outside `min_thread_count` and `max_thread_count`, or above the +VM's vCPU count, is not applied. `min_thread_count` must be at least 1 and no +greater than `max_thread_count`. When the controller file does not exist, +`min_thread_count` defaults to 2 rather than 1. + +`host_cpu_scale_up_ceiling_percent` protects the host: while overall host CPU +use is at or above it, scale-ups are blocked, but scale-downs and reverts still +apply. `cooldown_secs` starts after every successful scale-up, scale-down, or +revert and blocks further automatic scale-ups and scale-downs of that VM until +it expires; reverts are never blocked by it. + +`max_instances_adjusted_per_poll` limits how disruptive a single poll can be. +When more VMs need resizing, scale-downs are applied before scale-ups, and the +remaining VMs are reconsidered on the next poll. + +`vm_state_path` records which VMs the controller manages and which it leaves +to the operator, as classified by the backend when it first sees each VM. The +default, `/run/io-thread-controller/vm-ownership.json`, survives daemon +restarts but not a host reboot. + +By default each VM's status line goes to +`/var/log/io-thread-controller/vms/.log`, together with that VM's log +events; see [Status line](#status-line) for the format. +`enable_per_vm_main_log` sends the status lines to the main log instead. + +`dry_run` and `print_status_header` can also be enabled with `--dry-run` and +`--print-status-header`. `refresh_cgroup_on_each_read` and +`experimental_psi_monitoring` are described under +[Cgroup throttling](#cgroup-throttling) and +[Experimental PSI monitoring](#experimental-psi-monitoring). + +### Threshold engine + +The threshold engine reads `threshold.json` and sizes each VM's worker pool to +the CPU load it observes. + +| Setting | Default | Meaning | +|---|---|---| +| `scale_up_threshold_percent` | `95` | Average per-thread CPU above which a thread is added. | +| `scale_down_sustain_secs` | `30` | How long scale-down must stay possible before it happens. | +| `max_scale_down_step` | `0` | Most threads removed at once; `0` means no limit. | +| `scale_up_min_gain_percent` | `5` | IOPS gain a scale-up must deliver; `0` disables the check. | +| `scale_down_revert_drop_percent` | `5` | IOPS loss that reverts a scale-down; `0` disables the check. | +| `scale_validation_delay_secs` | `20` | How long to measure after an action before validating it. | +| `scale_validation_delay_per_thread_secs` | `0` | Extra validation time per thread in the new pool. | +| `block_scale_up_when_cgroup_throttled` | `true` | Skip scale-up while the VM's cgroup is being CPU-throttled. | +| `post_revert_up_grace_secs` | `10` | Time after a reverted scale-up during which metrics are ignored. | +| `post_revert_up_recovery_secs` | `20` | Time after the grace period spent re-measuring IOPS. | +| `post_revert_up_backoff_max_secs` | `300` | Cap on the post-revert hold; `0` disables backoff. | +| `scale_down_backoff_max_secs` | `1200` | Cap on the scale-down sustain time; `0` disables backoff. | + +Each poll, the engine compares the pool's average per-thread CPU use with +`scale_up_threshold_percent`. Above it, the engine proposes one more thread. +Otherwise it works out the smallest pool that would keep the average at or +below the threshold. If that is smaller than the current pool for +`scale_down_sustain_secs`, the engine shrinks the pool to it, by at most +`max_scale_down_step` threads at a time. + +After a successful action, the engine measures IOPS for +`scale_validation_delay_secs`, plus `scale_validation_delay_per_thread_secs` +for each thread in the resulting pool, which gives larger pools more time to +settle. A scale-up is reverted unless IOPS gained at least +`scale_up_min_gain_percent`. A scale-down is reverted when IOPS fell by more +than `scale_down_revert_drop_percent` while CPU use is above the scale-up +threshold; if CPU use stayed low, demand simply left, and the smaller pool is +kept. Setting either percentage to zero disables that direction's validation. + +Repeated failures back off per VM. After a reverted scale-up, scale-up is +blocked for `post_revert_up_grace_secs`, while the pool settles, plus +`post_revert_up_recovery_secs`, while the engine measures a fresh IOPS baseline +for the next attempt. That hold doubles with each consecutive failed scale-up, +up to `post_revert_up_backoff_max_secs`, and scale-down remains allowed +throughout. After a reverted scale-down, the next scale-down needs +`scale_down_sustain_secs` of low utilisation doubled once per consecutive +failure, up to `scale_down_backoff_max_secs`. A kept action resets its +direction's backoff, as does an applied action in the opposite direction. ## Thread-name filtering @@ -145,24 +270,6 @@ queries and thread-count changes need an independently stateful handle for each VM. This keeps the controller independent of libvirt, QMP, control sockets, and other transport details. -After a successful action, the engine samples for -`scale_validation_delay_secs`. The optional -`scale_validation_delay_per_thread_secs` setting adds that much time for each -thread in the resulting pool, allowing larger pools more time to settle. -A scale-up is reverted unless IOPS gained at least -`scale_up_min_gain_percent`; a scale-down is reverted when IOPS lost more than -`scale_down_revert_drop_percent`. Setting either percentage to zero disables -that direction's validation. - -Repeated failures back off per VM. After a reverted scale-up, scale-up is -blocked for `post_revert_up_grace_secs` plus `post_revert_up_recovery_secs`; -that hold doubles with each consecutive failed scale-up, up to -`post_revert_up_backoff_max_secs`. After a reverted scale-down, the next -scale-down needs `scale_down_sustain_secs` of low utilisation doubled once per -consecutive failure, up to `scale_down_backoff_max_secs`. A kept action -resets its direction's backoff, as does an applied action in the opposite -direction. Setting a maximum to zero disables that direction's backoff. - ## QEMU backend The QEMU backend discovers active VMs through libvirt and sends QMP commands From b0ec84fdb939f8b8eb94cef61505570d2829f79a Mon Sep 17 00:00:00 2001 From: Thanos Makatos Date: Fri, 23 Oct 2026 14:22:21 +0000 Subject: [PATCH 2/2] Use more sensible config defaults Signed-off-by: Thanos Makatos --- README.md | 2 +- io-thread-controller.d/engines/threshold.json | 2 +- io-thread-controller.json | 9 +++++---- src/config.rs | 9 +++++---- 4 files changed, 12 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index d0da2f5..39cea25 100644 --- a/README.md +++ b/README.md @@ -143,7 +143,7 @@ The first five settings are required when the controller file exists. | `cooldown_secs` | `30` | Per-VM pause after a successful scale action. | | `max_instances_adjusted_per_poll` | `0` | Most VMs resized per poll; `0` means no limit. | | `dry_run` | `false` | Log decisions without resizing pools. | -| `enable_per_vm_status_line` | `true` | Write a status line per VM every poll. | +| `enable_per_vm_status_line` | `false` | Write a status line per VM every poll. | | `enable_per_vm_main_log` | `false` | Send per-VM status lines to the main log instead of per-VM files. | | `enable_aggregate_status_line` | `true` | Write one fleet-wide status line every poll. | | `print_status_header` | `false` | Log a legend of the status-line fields at startup. | diff --git a/io-thread-controller.d/engines/threshold.json b/io-thread-controller.d/engines/threshold.json index 110084e..94792de 100644 --- a/io-thread-controller.d/engines/threshold.json +++ b/io-thread-controller.d/engines/threshold.json @@ -2,7 +2,7 @@ "scale_up_threshold_percent": 95, "scale_down_sustain_secs": 120, "max_scale_down_step": 0, - "scale_up_min_gain_percent": 5, + "scale_up_min_gain_percent": 1, "scale_down_revert_drop_percent": 3, "scale_validation_delay_secs": 30, "scale_validation_delay_per_thread_secs": 0, diff --git a/io-thread-controller.json b/io-thread-controller.json index 9efee46..ed369fb 100644 --- a/io-thread-controller.json +++ b/io-thread-controller.json @@ -1,12 +1,13 @@ { - "scale_poll_secs": 10, + "scale_poll_secs": 5, "round_up_durations": false, - "enable_per_vm_status_line": true, + "enable_per_vm_status_line": false, + "enable_per_vm_main_log": false, "enable_aggregate_status_line": true, "min_thread_count": 1, - "max_thread_count": 8, + "max_thread_count": 16, "host_cpu_scale_up_ceiling_percent": 90, - "cooldown_secs": 10, + "cooldown_secs": 30, "engine": "threshold", "engine_config_dir": "/etc/io-thread-controller.d/engines", "backend_config_dir": "/etc/io-thread-controller.d/backends", diff --git a/src/config.rs b/src/config.rs index 9e4b415..4a24601 100644 --- a/src/config.rs +++ b/src/config.rs @@ -76,10 +76,11 @@ pub struct Config { #[serde(default)] pub round_up_durations: bool, /// Emit a `vm=` status line per tracked instance on - /// every tick (default `true`). - #[serde(default = "default_true")] + /// every tick (default `false`). + #[serde(default)] pub enable_per_vm_status_line: bool, - /// Also emit each per-VM status record on the main controller target. + /// Also emit each per-VM status record on the main controller target (default `false`). + /// Has no effect is `enable_per_vm_status_line` is `false`. #[serde(default)] pub enable_per_vm_main_log: bool, /// Emit a single aggregate status line summarising every @@ -242,7 +243,7 @@ impl Default for Config { host_cpu_scale_up_ceiling: 0.9, cooldown_secs: default_cooldown_secs(), vm_state_path: Path::new("/run/io-thread-controller/vm-ownership.json"), - enable_per_vm_status_line: true, + enable_per_vm_status_line: false, enable_per_vm_main_log: false, enable_aggregate_status_line: true, print_status_header: false,