Skip to content

slurm: put slurm_install_prefix clients on the slurmd PATH (fixes Pyxis with a custom prefix) - #1411

Merged
michael-balint merged 1 commit into
masterfrom
dholt/slurmd-prefix-path
Sep 23, 2026
Merged

michael-balint merged 1 commit into
masterfrom
dholt/slurmd-prefix-path

Conversation

@dholt

@dholt dholt commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Summary

slurmd runs with systemd's minimal PATH, so when slurm_install_prefix is not /usr/local the Slurm client binaries are not resolvable from slurmd's children. Enroot's 50-slurm-pmi.sh hook (installed by the pyxis role) calls scontrol and fails, and every Pyxis container refuses to start:

pyxis: container start failed with error code: 1
    [ERROR] Command not found: scontrol
    [ERROR] /etc/enroot/hooks.d/50-slurm-pmi.sh exited with return code 1

This adds a slurmd.service.d/10-slurm-path.conf drop-in that prepends {{ slurm_install_prefix }}/bin and /sbin to slurmd's PATH. The existing unconditional start slurmd task (restart + daemon-reload) in compute.yml runs after this, so the override is active on every convergence without a new handler.

The same correction was part of the closed consolidation PR #1396; this is the minimal standalone version.

Validation

  • Reproduced on a fresh Ubuntu 24.04 single-GPU deployment with slurm_install_prefix=/opt/deepops/slurm, Slurm 26.05.3, Pyxis 0.11.1: srun --gpus=1 works, srun --container-image=... fails exactly as above.
  • Role lint (ansible-lint, production profile): 0 failures.
  • Syntax check across all release playbooks: pass.
  • git diff --check: clean.
  • Live re-validation with the fix is scheduled as part of the release QA rerun; results will be added here.

Notes

On Ubuntu 23.10+ Pyxis additionally needs the unprivileged user-namespace AppArmor exception for enroot-nsenter, which #1392 addresses. This PR is independent of that change.

slurmd runs with systemd's minimal PATH, so with a custom
slurm_install_prefix the Slurm client binaries are not resolvable from
slurmd's children. Enroot's slurm-pmi hook (installed by the pyxis role)
calls scontrol and fails with "Command not found: scontrol", and every
Pyxis container then refuses to start:

    pyxis: container start failed with error code: 1
    [ERROR] Command not found: scontrol
    [ERROR] /etc/enroot/hooks.d/50-slurm-pmi.sh exited with return code 1

The default /usr/local prefix only works because /usr/local/bin is on the
default PATH. Install a slurmd drop-in that prepends the configured prefix.
The existing unconditional slurmd restart with daemon-reload in compute.yml
applies it on every convergence.

Reproduced on a fresh Ubuntu 24.04 single-node deployment with
slurm_install_prefix=/opt/deepops/slurm and Slurm 26.05.3; the same
correction was part of the closed consolidation PR #1396.

Signed-off-by: Doug Holt <dholt@nvidia.com>
@michael-balint
michael-balint merged commit 266e208 into master Sep 23, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants