feat!: apps SDK, decorator-native GPU compute with live dev sessions and a new rp CLI

[KAJdev]

Brings decorator-native GPU compute into the core runpod package, with queue, API, and task resources sharing one app lifecycle and a live development workflow.

Worker execution is versioned independently through runpod-sdk-runtime, allowing runtime images and SDK clients to evolve on separate release cycles. Flash lifecycle commands live under rp flash, while account, secret, registry, pod, and SSH commands stay at the root.

Completes CON-605
Completes CON-601
Completes CON-750

[deanq]

Review from a facade-delegation + correctness pass on the apps SDK (I'm consuming this contract from runpod-flash's compat facade).

Requesting changes on the two verified merge-blockers below; the rest are important-but-negotiable. Three findings were reproduced locally (noted inline).

Critical:

• placement.py datacenter enum pin -> schedulable nowhere (verified 3.14.6)
• dev/task teardown orphans billed endpoints & pods on the failure path (the SLS-344 class)

Facade note: the sentinel dispatch wire contract matches cleanly and delegates today. Three seams still force flash to keep its own code and are worth landing here: a public header builder (facade currently imports the private targets._headers), an id-only stub/client (Queue/Api require app=), and a remote-class execution seam (apps is function-only; flash's RemoteClassWrapper has no landing spot). Happy to file these as separate issues if you'd prefer.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate findings affect mutation safety, runtime packaging, resource placement, cleanup, and live development.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds decorator-native GPU compute, Apps SDK resources, shared lifecycles, live development sessions, and new rp flash CLI workflows.

Changes:

• Adds deployment, placement, task, queue, API, and runtime support.
• Adds live development, custom-image execution, and end-to-end examples.
• Updates CLI workflows, exports, authentication, scaffolding, and tests.

File summaries

File	Reviewed scope
tests/test_shared/test_auth.py	Authentication coverage
tests/test_init.py	Public API exports
tests/test_cli/test_init_command.py	Project scaffolding
tests/test_cli/test_cli_utils/test_sync.py	Sync utilities
tests/test_cli/test_cli_utils/test_info.py	Info utilities
tests/test_cli/test_cli_groups/test_ssh_dispatch.py	SSH dispatch
tests/test_cli/test_cli_groups/test_project_helpers.py	Project helpers
tests/test_cli/test_cli_groups/test_project_commands.py	Project commands
tests/test_cli/test_cli_groups/test_pod_commands.py	Pod commands
tests/test_cli/test_cli_groups/test_exec_helpers.py	Exec helpers
tests/test_cli/test_cli_groups/test_exec_functions.py	Exec functions
tests/test_cli/test_cli_groups/test_exec_commands.py	Exec commands
tests/test_cli/test_cli_groups/test_config_commands.py	Config commands
tests/test_apps/test_watch.py	Watch behavior
tests/test_apps/test_volume.py	Volume behavior
tests/test_apps/test_shim.py	Runtime shim
tests/test_apps/test_secret.py	Secret resources
tests/test_apps/test_retry.py	Retry behavior
tests/test_apps/test_registry.py	Registry resources
tests/test_apps/test_placement.py	Placement behavior
tests/test_apps/test_model.py	Model resources
tests/test_apps/test_manage.py	Resource management
tests/test_apps/test_logs.py	Logs behavior
tests/test_apps/test_job.py	Job behavior
tests/test_apps/test_images.py	Image handling
tests/test_apps/test_compat_lock.py	Compatibility locking
tests/test_apps/test_build.py	Build behavior
tests/test_apps/test_auth.py	Apps authentication
tests/test_apps/__init__.py	Apps test package
tests/e2e/task_pod_smoke.py	Task pod smoke test
tests/e2e/matrix/run.py	E2E matrix runner
tests/e2e/matrix/main.py	E2E matrix entrypoint
tests/e2e/examples/run_all.sh	Example runner
tests/e2e/examples/README.md	Example documentation
tests/e2e/examples/13_streaming.py	Streaming example
tests/e2e/examples/12_custom_image.py	Custom image example
tests/e2e/examples/11_train_eval.py	Training and evaluation example
tests/e2e/examples/10_cached_models.py	Cached model example
tests/e2e/examples/09_secrets.py	Secrets example
tests/e2e/examples/08_volumes.py	Volumes example
tests/e2e/examples/07_pipelines.py	Pipelines example
tests/e2e/examples/06_tasks.py	Tasks example
tests/e2e/examples/05_api_service.py	API service example
tests/e2e/examples/04_dependencies.py	Dependencies example
tests/e2e/examples/03_gpu.py	GPU example
tests/e2e/examples/02_invocation_styles.py	Invocation styles
tests/e2e/examples/01_hello_world.py	Hello-world example
tests/e2e/apps_live_smoke.py	Live Apps smoke test
setup.py	Package entry points
runpod/serverless/utils/rp_upload.py	Upload utility
runpod/rp_cli/update.py	CLI update support
runpod/rp_cli/__init__.py	CLI package
runpod/cli/utils/rp_sync.py	Legacy sync utility
runpod/cli/utils/rp_info.py	Legacy info utility
runpod/cli/groups/ssh/commands.py	SSH commands
runpod/cli/groups/project/starter_templates/llama2/src/handler.py	Llama starter handler
runpod/cli/groups/project/starter_templates/llama2/builder/requirements.txt	Llama build requirements
runpod/cli/groups/project/starter_templates/default/src/handler.py	Default starter handler
runpod/cli/groups/project/starter_templates/default/builder/requirements.txt	Default build requirements
runpod/cli/groups/project/starter_templates/default/.runpodignore	Default ignore rules
runpod/cli/groups/project/helpers.py	Project helpers
runpod/cli/groups/project/commands.py	Project commands
runpod/cli/groups/project/__init__.py	Project CLI package
runpod/cli/groups/pod/commands.py	Pod commands
runpod/cli/groups/exec/helpers.py	Exec helpers
runpod/cli/groups/exec/functions.py	Exec functions
runpod/cli/groups/exec/commands.py	Exec commands
runpod/cli/groups/exec/__init__.py	Exec CLI package
runpod/cli/groups/config/commands.py	Config commands
runpod/cli/entry.py	CLI entrypoint
runpod/cli/__init__.py	CLI package
runpod/apps/watch.py	App watching
runpod/apps/volume.py	Volume resources
runpod/apps/utils/network.py	Network utilities
runpod/apps/utils/names.py	Naming utilities
runpod/apps/utils/lookup.py	Lookup utilities
runpod/apps/utils/events.py	Event utilities
runpod/apps/utils/client.py	Client utilities
runpod/apps/utils/__init__.py	Apps utilities package
runpod/apps/stubs.py	SDK stubs
runpod/apps/shim.py	Runtime shim
runpod/apps/serialization.py	Serialization
runpod/apps/secret.py	Secret resources
runpod/apps/schedule.py	Scheduling
runpod/apps/registry.py	Registry resources
runpod/apps/protocol.py	Worker protocol
runpod/apps/model.py	Model resources
runpod/apps/markers.py	Route markers
runpod/apps/manage.py	Resource management
runpod/apps/logs.py	Logs
runpod/apps/job.py	Jobs
runpod/apps/invoker.py	Invocation
runpod/apps/init.py	Apps initialization
runpod/apps/images.py	Images
runpod/apps/errors.py	Apps errors
runpod/apps/entrypoint.py	Runtime entrypoint
runpod/apps/discovery.py	App discovery
runpod/apps/discovery_state.py	Discovery state
runpod/apps/datacenter.py	Datacenter support
runpod/apps/context.py	Apps context
runpod/apps/auth.py	Apps authentication
runpod/apps/__init__.py	Apps package
runpod/api/queries/apps.py	Apps queries
runpod/api/mutations/apps.py	Apps mutations
runpod/api/graphql.py	GraphQL transport
runpod/__init__.py	Public exports
requirements.txt	Dependencies
pytest.ini	Test configuration
pyproject.toml	Project tooling
examples/apps/web_service.py	Web service example
examples/apps/train_and_eval.py	Training example
examples/apps/streaming.py	Streaming example
examples/apps/README.md	Apps documentation
examples/apps/hello_world.py	Hello-world example
examples/apps/gpu_inference.py	GPU inference example
docs/cli/start_here.md	CLI getting started
docs/cli/references/command_line_interface.md	CLI reference
.dockerignore	Docker ignore rules

Review details

Suppressed comments (16)

runpod/apps/api.py:82

• Task CPU specs accept a list of alternative instance IDs, and placement computes candidates using all of them, but this conversion silently sends only the first ID to deployCpuPod. A volume may be placed for a later alternative that the pod request never uses, causing an avoidable deployment failure; preserve the alternatives or resolve one consistently before placement.

            instance_ids = pod_input.pop("instanceIds", None)
            if instance_ids:
                pod_input["instanceId"] = instance_ids[0]

runpod/apps/api.py:66

• delete_endpoint exposes a boolean success result, but the cleanup and management callers treat any non-exceptional return as success. A false result is therefore counted as an endpoint deletion and can allow the environment/app to be deleted or the local session state to be cleared while the endpoint remains. Propagate false as an error or require every caller to check the result.

    async def delete_endpoint(self, endpoint_id: str) -> bool:
        mutation = app_mutations.MUTATION_DELETE_ENDPOINT
        data = await self._execute(mutation, {"id": endpoint_id})
        return bool(data.get("deleteEndpoint"))

runpod/apps/deploy.py:228

• The deployed worker template never receives the target environment name. App._resolve() falls back to app.env when FLASH_ENVIRONMENT is absent, so rp flash deploy --env prod can make nested worker-to-worker calls send sentinel headers for default/the source app environment instead of prod. Inject the resolved environment name into the worker env or pass it through the endpoint payload construction.

        "FLASH_RESOURCE_NAME": spec.name,
        # version-triggering: a new build recreates all workers
        "RUNPOD_BUILD_ID": build_id,
        **render_env(spec.env),
    }

runpod/apps/dev.py:410

• A failed deletion is reported but the unconditional clear() discards its endpoint ID. The session then cannot retry cleanup, and the CLI still exits normally while the endpoint remains billable; retain failed IDs and surface cleanup failure instead of clearing all tracking state.

            except Exception as exc:
                emit(sink, "delete_failed", resource)
                log.warning("failed to delete dev endpoint %s: %s", endpoint_id, exc)
        self._endpoints.clear()

runpod/apps/dev.py:150

• Live dev endpoint workers are not given FLASH_RESOURCE_NAME or RUNPOD_RESOURCE_NAME. If a worker calls its own handle with .remote(), _is_current_worker() cannot take the local fast path and _resolve_dev_sibling() routes the call back through the same live endpoint, changing deployed behavior and potentially causing an unintended extra request. Include the resource name in the dev template environment, as the deployed endpoint path already does.

                {"key": "RUNPOD_DEV_APP", "value": app.name},
                *(

runpod/apps/discovery.py:117

• When a directory module creates an App and then raises, the tolerant path keeps that newly registered, possibly incomplete app in the global registry. Because found is collected after the loop, discovery can return and deploy this partial app instead of reporting the failed module; roll back registrations made by a failed import before continuing.

            except DiscoveryError as exc:
                if strict:
                    raise
                failures.append(str(exc))
                log.warning("%s", exc)
                continue

runpod/apps/discovery.py:80

• The timeout raises while the daemon import thread is still allowed to execute. That thread can later register an App, invoke user code, or mutate sys.modules after discovery has moved on and cleared DISCOVERY_ENV, making a timed-out module's side effects race with deployment. Run timed imports in an isolatable subprocess, or do not continue after a timed-out thread.

    if thread.is_alive():
        raise DiscoveryError(
            f"importing {path} timed out after {IMPORT_TIMEOUT_SECONDS}s; "
            f"module-level code must not block (guard it with "
            f'`if __name__ == "__main__":` or runpod.is_local())'

runpod/apps/discovery.py:56

• Loading every file under a synthetic top-level module name leaves __package__ unset, so a normal project module such as pkg/main.py cannot import from .helpers import .... Directory discovery suppresses that import error and then reports no app, making package-based projects undeployable. Preserve the package-qualified import context (or set __package__/search locations) when loading discovered files.

    module_name = f"_runpod_discovered_{path.stem}_{abs(hash(str(path)))}"
    spec = importlib.util.spec_from_file_location(module_name, path)

runpod/apps/markers.py:25

• LiveTarget._sync_source sends requests to /_runpod/sync, but this path is not reserved for API routes. A user can register that path and collide with the internal source-sync endpoint during live development. Add it to the reserved paths (along with any other runtime control paths) before accepting route markers.
  runpod/apps/targets.py:235
• This retry loop is also used by QueueClient.run() and runsync(), which are non-idempotent POST submissions. If a 5xx or connection failure occurs after the service accepted the job, retrying can enqueue or execute the same function more than once; restrict retries for submissions or attach a stable idempotency key.
  runpod/apps/targets.py:62
• When the same parameter is supplied positionally and by keyword, body.update(kwargs) silently overwrites the positional value instead of matching Python's TypeError: got multiple values behavior. A call such as fn(1, a=2) therefore executes remotely with a=2, unlike local and task execution; reject duplicate parameter names while building the input.
  runpod/apps/targets.py:601
• _synced_hash is scoped to the target, not to a worker. Because this sync request goes through the load balancer, the first call can upload source to worker A and mark the hash; a later call routed to a newly scaled or restarted worker B then skips the upload and that worker has no live module/routes. Track synchronization per worker or synchronize before each worker can receive a request.
  runpod/apps/tasks.py:225
• A task resolves a shared volume using only its own spec. If another queue/task in the app uses the same volume, the volume can be created in a datacenter that fits this task but not the sibling, contradicting the shared-volume placement contract and causing the later resource to fail placement. Resolve against all app specs sharing this volume.
  runpod/apps/tasks.py:132
• For a CPU5 task without an explicit datacenter, this sends the CPU3 location list even though CPU5 stock is restricted to CPU5_DATACENTERS. The fallback can therefore make unnecessary capacity requests against locations that cannot host the requested flavor; select the CPU5 pool when the instance ID starts with cpu5.
  runpod/apps/tasks.py:340
• When termination fails, this still clears pod_id, so TaskJob.wait()/cancel() lose the only identifier needed to retry cleanup and the pod remains until the one-hour terminateAfter backstop. Retain the ID on failure (or retry deletion) and clear it only after a successful termination.
  runpod/rp_cli/main.py:412
• Cancellation of _run_entrypoint_cancellable abandons the daemon thread, but _session immediately enters its finally block and deletes the live endpoints. The user thread can still be blocked in a remote call and continue issuing requests (including task provisioning) while teardown is running, leaving work or resources behind. Coordinate cancellation with the runner and wait for it, or defer teardown until the in-flight call has stopped.

• Files reviewed: 146/148 changed files
• Comments generated: 12
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[KAJdev]

Should be ready for a re-review. Took a look at all these comments, main things were addressed.

• rebase with the new REST wrapper API client for all supported operations
• resource validation and GPU/datacenter/volume placement, including multi-GPU and multi-volume configurations
• endpoint and pod cleanup across partial startup, failures, timeouts, and cancellation plus retry policies that avoid duplicating mutations
• nested sync/async worker calls, concurrent source synchronization, result/error handling, and dependency packaging
• independently versioned runtime images, custom-image bootstrapping, and runtime publication.
• .remote() calls on queue based endpoints now use /runsync first, then switch to polling during longer requests

[deanq]

Re-review after the REST-wrapper rebase — the earlier blockers all look resolved (placement datacenter enum, dev teardown orphaning, the targets.py pod leaks, etc.). A few new issues in the new/rebased code, inline below.

The tasks.py gpu="any" one is the same enum→device-name class as the placement.py fix from last round and would block task deploys; the rp_info.py get_pod change is a retry regression on the rp pod connect path. The rest are robustness/correctness.

[deanq]

gpu="any" (and GpuGroup.ANY) can't deploy. spec.gpu normalizes to ["any"], then _device_names(["any"]) → GpuGroup("any").device_names() is [], so [] or ["any"] falls back to the literal ["any"] and gpuTypeIdList=['any']. The pod deploy API can't resolve 'any' as a device name, so the task never launches — whereas gpu=None correctly expands to the full device list. Worth normalizing the ANY sentinel to the same full-list expansion as None. (Same enum→wire-value class as the placement.py datacenter fix from the last round, one module over.)

[deanq]

session._retry_connection = False doesn't do anything — aiohttp.ClientSession has no _retry_connection attribute, so this just sets an unused instance attr. The comment says it prevents aiohttp from replaying PUT/DELETE after an ambiguous disconnect, but aiohttp doesn't auto-retry requests, so there's nothing to disable here. Either drop it as dead code, or — if replay of non-idempotent calls (delete_endpoint, terminate_pod) is a genuine concern — handle it explicitly.

[deanq]

This changes failure-path behavior: previously pod = get_pod(pod_id) or {} tolerated a transient None/empty response and the while loop retried until timeout. Now if not pod: raise ValueError(...) aborts on the first falsy return, so a pod that's still provisioning (or a momentary empty GraphQL response) makes get_pod_ssh_ip_port — and thus rp pod connect / SSHConnection — crash immediately instead of waiting for RUNNING. Suggest keeping the tolerate-and-retry within the timeout window, and only raising once the deadline is hit.

[deanq]

The docstring (line 26) lists ADA_80_PRO as covering H100 PCIe + 80GB HBM3 + NVL, but POOLS_TO_TYPES[ADA_80_PRO] only has NVIDIA_H100_80GB_HBM3 (PCIe and NVL aren't in GpuType at all). So device_names() omits the PCIe/NVL variants — a task requesting ADA_80_PRO can't schedule onto those machines, and resolve_gpu_string("H100") only ever matches the one HBM3 device. Either add the missing device types or narrow the docstring to match reality.

[deanq]

run_rest_request_async returns None on a 204 / empty body, but here current = await run_rest_request_async('GET', ...) is dereferenced right after with current.get('type'). If the GET yields no body, current is None and this raises AttributeError instead of a meaningful error. A None guard (treat as "no existing endpoint") would be safer.

[deanq]

ast.walk yields nested defs too, so _bare_function_source picking the first FunctionDef whose name matches can select an inner function (or a same-named sibling in the parsed snippet) rather than the target — and lines[node.lineno-1:] then ships the wrong body to the worker. It's the REPL last-resort path, but it fails silently (wrong remote code) rather than erroring. Consider matching only the top-level / outermost def.

[deanq]

instance_ids = pod_input.pop('instanceIds', None) or [pod_input.get('instanceId')] becomes [None] when the input has neither key, and the loop then deploys a candidate with instanceId=None — producing a confusing server-side error instead of a clear client-side validation failure. Guard the empty case before the loop.