gpu: a shared op layer over the backends, views as the currency, and a fourth backend to prove it - #2
Merged
Merged
Conversation
…o GPU cuda.h dlopens libcuda and looks kernels up by name, so a stand-in libcuda.so.1 on LD_LIBRARY_PATH is a whole installation. The stand-in allocates host memory, copies with memcpy, and turns a launch into one trace line: kernel name, grid, block, shared-memory bytes, and every argument, with pointers printed as (allocation, byte offset) so two runs compare with diff. The argument layout of each kernel comes from preprocessing the .cu itself, which also makes the launch contract a checked fact: 116 of the 117 kernels take their pointers first and then 4-byte scalars (tl_rope_dpos is the one that interleaves). run.sh builds the test suite and the CUDA checkers on Linux in a container and traces them; compare.sh traces a base ref and the working tree and diffs. bench/cuda/check/trace_sweep.cpp reaches what those programs do not, so all 117 kernels appear, and it calls the four zero-then-scatter ops with a non-zero output offset, where the trace shows the pre-zeroing clearing from the buffer's base instead. This is for the backend-layer refactor: no machine here runs a CUDA kernel, and what a host-side change must preserve is exactly what the trace holds.
…n once Every backend declared every op itself, twice (the real branch and the disabled-platform stub), and an op's offset was a loose int64_t beside a void* whose meaning differs per backend. This puts a shared layer between array.h and the backends and moves the first family onto it. gpu_abi.h is what that layer and a backend agree on: the op vocabulary (it lived in metal.h, which cuda.h and webgpu.h included for it), `span` (a device handle and a byte offset, the currency ops take), each view's access, the launch grid, and the kernel ABI: for a kernel id, views in the order the kernel declares its buffers and a params struct of 4-byte fields in the order it takes its scalars. The canonical order is the CUDA kernel's, since the .cu is the one source no machine here can run. A backend's device core gains one entry point, dispatch(kernel, views, params, grid). Metal binds view i at index i and the params after them. CUDA derives residency from each view's access and expands the params into argv four bytes apiece, which is launch_'s own contract read the other way round. WebGPU's kernels predate the ABI (one params layout for every entry point, an operation number per family), so its core carries a marshal from the canonical params; that stays inside webgpu.h. tl::gpu becomes a namespace with a using-directive for the selected backend instead of an alias, so a shared op declared in it is found first and everything else still falls through to the backend. gpu_ops.h holds binary, unary and unary_ext, once, with no backend in sight; their six per-backend definitions and six stubs are gone. MSL's ew_params is reordered to the canonical layout. cuda's kernel_name_ no longer answers tl_sgemm for an id it has no kernel for. The shared launch counts per kernel id, so a test can tell a kernel that ran from an op that fell back: the new test runs each shared op on views at distinct non-zero offsets and checks the census (mutating Metal's dispatch to drop the offset fails it). tools/cuda_trace shows no removed line against the previous commit: the generic expansion hands the driver what the hand-written launches did.
binary_bcast, compare, clamp, scalar_binary, row_op, row_logsumexp, layer_norm, index_select, gather_from_axis, xent_bwd and adam_step are each one kernel with the same buffers on every backend, so each becomes one function in gpu_ops.h over a canonical params struct in gpu_abi.h. Their per-backend definitions and stubs go: 840 lines out of the three backend headers, 190 into the shared two. What differed was only layout and launch shape. MSL's ew_bcast, clamp and scalar params are reordered to the canonical (CUDA argument) order. The rank-2 broadcast kernel reads its cell from a flat index on CUDA and from a 2-D thread position on Metal and WebGPU, which is a property of the kernels, so each backend states it (traits::cells_2d) and the shared policy picks the grid. A row reduction's per-thread scratch rides in the grid (CUDA's shared-memory bytes; the others size theirs in the kernel). cuda.h's kernel table learns the ids these ops dispatch, and the named getters they used (compare_, clamp_, scalar_binary_) go. Metal's dispatch declines an id it has no MSL kernel for instead of throwing. WebGPU's marshal takes the read views' element offsets, so a kernel that reads a third operand at an offset (layer_norm's bias) places it, and an offset nobody consumed fails the dispatch instead of being dropped. array.h passes views as array::device_span(). tools/cuda_trace also syntax-checks the CUDA-only benches it does not run. Against the previous commit the trace differs in one line's position: adam_step uploads a host-born gradient after its three state buffers rather than before, because residency now follows argument order.
…ts own way The rest of the op surface moves behind gpu_ops.h, so an op's signature now exists in one place and no backend carries a stub. The model path's single-kernel ops (rmsnorm, rmsnorm_res, swiglu, gemv_bf16_row, gemv_q4, kv_append, kv_fill, merge_heads) join the shared ones. Their canonical layouts are the CUDA kernels': MSL's kv params split into an append and a fill struct, merge_heads gets its own, and rmsnorm becomes two entry points over one core (add_rmsnorm_ beside rmsnorm_), as CUDA already had. The row-GEMV launch shape, which metal.h had copied from cuda.h by hand, is one policy (row_reduce); Metal's q4 GEMV had a fixed 256 threads and now takes it too, which is 20-31% faster at Qwen2.5's shapes. The ops whose algorithm differs by backend stay the backend's: a scatter into a zeroed buffer against a gather, shape metadata in a device buffer against a params block, one kernel against a split and a combine. Each is a static member of the backend's `own` struct with the shared signature, written over its dispatch where one kernel suffices. gpu_ops.h detects the member and forwards, or answers false: a backend declares what it has and nothing else, and a signature that drifts is a compile error rather than a silent CPU fallback. That replaces tools/check_backend_parity.py and its allowlist, which checked the same thing with regular expressions. Views change what callers can say. array.h's rope no longer sends a view with an offset to the CPU. The q4 scales are `qw.at(N*K/2)` instead of pointer arithmetic that Metal undid by subtracting. kv_cache takes views. The Qwen2.5 driver reads q, k and v as views of the fused projection on every backend, so caps::flat_addressing, copy_out (48 dispatches a token on Metal) and the unfused prefill buffers are gone; check_qwen still matches the numpy oracle token for token. CUDA's pad, fold, index_add and scatter_to_axis cleared their output from the buffer's base whatever offset the kernel then wrote at. The evaluator never passes one, so nothing was wrong yet. The trace against the previous commit differs in exactly those four memsets (#24+0 -> #24+64) and nowhere else, across every test and checker. tools/cuda_trace gains a compile-only mode that builds every test, checker, bench and model driver against the CUDA branch.
gpu.h included all three backend headers, and each compiled to a branch of stubs where its gate did not hold: three copies of "no device here", kept alive so that tl::gpu could alias one of them on a build that fits none. Each backend header is now gated whole, and gpu.h includes only the one it selects, falling to gpu_null.h. That file is the null backend: available() is false, alloc() is null, dispatch() declines, `own` is empty. It is also everything gpu.h asks of a backend with nothing filled in, which makes it the template a new backend starts from, and adding one touches its own header, its kernels, and one branch of gpu.h's selection. Verified on a Linux build with no GPU backend (113 cases in cpu, gpu and auto modes), on Metal, on WebGPU under Deno, and by tools/cuda_trace, whose trace is identical to the previous commit's.
…ckend The layers (shared ops over a device core), gpu::span and access, the kernel ABI and how each backend realizes a launch from it, the two kinds of op (single-kernel, and backend-own through `own`), launch policy and traits, the steps for a new op and a new backend (gpu_null.h is the template), what is not shared yet, and the four ways a change is verified on a machine with no NVIDIA GPU.
The conformance test found the first thing the shared layer had wrong. Run each op on views at non-zero offsets, inside buffers with sentinels on both sides, against a plain host loop: on WebGPU the values were right and the sentinels were gone. `out` marked the device copy live without an upload, which is sound when a kernel writes a whole fresh buffer and loses the rest of a host-filled one when it writes a view of it. cuda.h had the same rule (its comment says a partial write must be device_rmw_), and both backends had copied the HOST/DEVICE/BOTH machine from each other. gpu::residency (gpu_abi.h) is that machine, once, with a fourth state: `none`, a fresh allocation nobody filled, which alloc's host_fill argument (ignored on both mirrored backends until now) tells apart from `host`. An `out` uploads first only when the host had filled the buffer; an output into a fresh buffer, which is nearly every output, still brings nothing. cuda.h and webgpu.h keep their mirror tables and do the copying, and ask residency when. tools/cuda_trace shows no existing line changed. The census also counts backend-own ops (gpu::ops_run), so a second test runs one graph per op family through the evaluator in GPU mode and requires the device to have been reached: the oracle comparisons pass either way, which is how Metal and CUDA could fall back to the CPU entirely and stay green. Mutating Metal's pad to drop its output offset fails the first test.
gpu_host.h is a backend whose device is the CPU and whose kernels are plain loops. -DTENSORLIB_HOST_GPU=ON selects it ahead of any real one. It was written from gpu_null.h and docs/backends.md alone: a device core of about 190 lines (lifecycle, memory, dispatch's switch, traits, caps), about 270 of kernels that follow the kernel ABI, and four backend-own ops. The backend itself is that header and one branch of gpu.h's selection; with only those, 110 of the suite's 115 cases passed, the conformance tests among them. So "a backend is a small core plus kernels" is something the build checks, and the shared layer and its conformance tests run on a machine with no GPU, where the CPU fallback never enters them (CI: host-backend, on ubuntu and macOS). Each kernel in it is also the plainest statement of what its id computes. The other five cases are what this commit changes outside that header. Two were the host kernels' own (softmax of huge logits, a row with no finite max). Three were the suite asking a platform macro what a backend can do, and each `#if !defined(TENSORLIB_WEBGPU)` would have needed the new backend's name added. Those questions are asked in code now: gpu::has_<op> (whether the selected backend runs a backend-own op, a constant the detection already had) and traits::times_launches for the profile test's device times, which every backend states. tl::profile: a backend that does not record its own launches (traits::profiles_launches) gets a row per launch from the shared layer, by kernel name (gpu::kop_name), so a new backend is profiled from its first kernel. The host backend keeps a device's form, a launch leaving work pending until a flush, so the evaluator's flush and barrier paths and their profile rows are exercised on it too. Metal, WebGPU under Deno, the null backend on Linux and check_qwen pass; tools/cuda_trace is identical to the previous commit.
yhirose
added a commit
to yhirose/culebra
that referenced
this pull request
Sep 22, 2026
…nds supply a device core cpp-tensorlib 62289b2 -> 5911d39 (PR yhirose/cpp-tensorlib#2): every op is written once over spans in gpu_ops.h, a backend is a dispatch plus its own ops, residency is decided in one place, and a host reference backend proves a fourth one fits in a single header.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Every GPU backend declared every op itself, twice (the real branch and a disabled-platform stub), and an op's buffer was a bare
void*whose meaning differs per backend, with the offset a looseint64_tbeside it on some ops and absent on others. Adding WebGPU took about 2,000 lines across 12 places, and roughly half the later commits towebgpu.hwere stubs added because an op had landed elsewhere.This puts a shared layer between
array.hand the backends.gpu_abi.h: the op vocabulary (it lived inmetal.h),gpu::span(a device handle and a byte offset, the currency every op takes), each view's access (in/out/inout), the launch grid, the launch policy, the residency state machine, and the kernel ABI. For a kernel id, the views come in the order the kernel declares its buffers and the params struct is a run of 4-byte fields in the order it takes its scalars. The canonical order is the CUDA kernel's, since the.cuis the one source no development machine here can run.gpu_ops.h: every op, written once. An op's signature exists there and nowhere else.dispatch(kernel, views, params, grid),traits,caps. Metal binds view i at buffer index i. CUDA derives residency from each view's access and expands the params intoargvfour bytes apiece. WebGPU's kernels predate the ABI, so its core carries a marshal from the canonical params. That stays insidewebgpu.h.ownstruct.gpu_ops.hdetects the member and forwards, or answers false. There are no stubs, and a signature that drifts is a compile error rather than a silent CPU fallback. This replacestools/check_backend_parity.pyand its allowlist.gpu_null.hreplaces the stub branch each backend header carried.gpu.hincludes only the backend it selects.gpu_null.his also the template a new backend starts from.gpu_host.his a fourth backend, written fromgpu_null.handdocs/backends.mdalone: a device that is the CPU, with kernels that are plain loops. About 190 lines of core, 270 of kernels, four own ops. With only that header and one branch ingpu.h, 110 of the suite's 115 cases passed.-DTENSORLIB_HOST_GPU=ONselects it, so the shared layer and its conformance tests run on a machine with no GPU (new CI jobhost-backend).What callers gain
array.h's rope no longer sends a view with an offset to the CPU.qw.at(N*K/2)instead of pointer arithmetic that Metal undid by subtracting.kv_cachetakes views. The Qwen2.5 driver reads q, k and v as views of the fused projection on every backend, socaps::flat_addressing,copy_out(48 dispatches a token on Metal) and the unfused prefill buffers are gone.metal.hhad copied fromcuda.hby hand, is one shared policy. Metal's q4 GEMV had a fixed 256 threads and now takes it too: 20 to 31% faster at Qwen2.5's shapes.Bugs found on the way
pad,fold,index_addandscatter_to_axiscleared their output from the buffer's base whatever offset the kernel then wrote at. The evaluator never passes one, so nothing was wrong yet.outmarked the device copy live with no upload, which is sound for a whole fresh buffer and loses the rest of a host-filled one when a kernel writes a view of it. The new conformance test found this on WebGPU.gpu::residencygains anonestate (a fresh allocation), told apart fromhostbyalloc'shost_fillargument, which both mirrored backends ignored until now.cuda.h'skernel_name_answeredtl_sgemmfor any id it had no kernel for.#if !defined(TENSORLIB_WEBGPU)). Those aregpu::has_<op>andtraitsqueries now, so a new backend edits no test.How it was verified
No machine here runs CUDA kernels, and CI compiles them without running them.
tools/cuda_trace(first commit) puts a stand-inlibcuda.so.1in front of the backend in a Linux container and records every launch: kernel, grid, block, shared-memory bytes and each argument, with pointers printed as (allocation, byte offset) so two runs diff. All 117 kernels are reached. Across the whole migration the trace changed in four memsets (the pre-zeroing fix above) and one reordered upload inadam_step. Nothing else.ctest, andcheck_qwenagainst the numpy oracletools/cuda_trace/compare.shdispatchorpadfails the view testsNot verified: whether the CUDA kernels still compute the right thing. The trace shows the same kernels get the same arguments, which is what a host-side change that leaves the
.cualone has to preserve. Acteston NVIDIA hardware is still needed before this merges.Not done here
tl::profilehooks and theTL_PROFILEautostart still sit in each backend.caps::model_path = false.cuda.h,webgpu.h).kopstill lists Metal-only kernel ids.docs/backends.mdcovers the layers, the kernel ABI, and how to add an op or a backend.