Skip to content

arch: Validate device IDs and correct device initialization - #3067

Draft
ggorman wants to merge 2 commits into
fix-default-device-idfrom
fix-device-id-validation
Draft

ggorman wants to merge 2 commits into
fix-default-device-idfrom
fix-device-id-validation

Conversation

@ggorman

@ggorman ggorman commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Stacked on #3066; the base switches to main once that merges.

  1. Raise on a device ID that does not exist. An explicit deviceid at or beyond the number of devices (the visible ones if e.g. CUDA_VISIBLE_DEVICES is set) was ignored by the generated code, which ran on the current device; the memory query raised IndexError (an off-by-one on the visible devices) or wrapped around them. DeviceID._arg_check now raises InvalidArgument: Device 2 does not exist: 2 device(s) available (CUDA_VISIBLE_DEVICES=0,1) before anything is launched. Tests: TestDeviceID::test_nonexistent_deviceid[None|0,1]; test_deviceid_per_rank now picks devices that exist on any node.
  2. Select the device before initializing it, and only if there is one. The OpenACC runtime was initialized before selecting the device, so every MPI rank not on the default device also created a context on it (seen with nvidia-smi on two H200s: odd ranks held contexts on both GPUs; after the fix one each). With no device found, the MPI default selection computed rank % 0, a SIGFPE with an OpenMP runtime reporting zero devices. Tests: test_gpu_openacc.py::TestCodeGeneration::test_init_acc_env; test_gpu_openmp.py::test_init_omp_env_w_mpi expectation updated.

Each test fails on the parent commit.

Coordination with #2930: this PR validates the device ID in arguments(), so with both changes #2930's test_no_visible_devices raises inside op.arguments() (verified for the masks -1, '' and NoDevFiles); that call needs to move inside pytest.raises. Whichever lands second rebases.

Refs #2711, #2930. Out of scope, as follow-ups: visible-device lists naming devices the process cannot see (e.g. a container with --gpus device=1 and CUDA_VISIBLE_DEVICES=1), UUID/MIG masks, and validating deviceid in estimate_memory(). Does not close #2711.

🤖 Generated with Claude Code

An explicit `deviceid` at or beyond the number of devices (visible ones, if
e.g. CUDA_VISIBLE_DEVICES is set) was silently ignored by the generated
code, which then ran on the current device, while the memory query either
raised IndexError (an off-by-one on the visible devices) or wrapped around
them. Check it along with the other runtime arguments and raise an
InvalidArgument naming the device and the number available
…is one

The OpenACC runtime was initialized before selecting the device, so every
rank not using the default device also created a context on it. And with no
device found, the MPI device selection computed `rank % 0`

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arch jitting, archinfo, ... GPU

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant