gpu: launch recording, split policy and the model path, all in one place - #3
Merged
Merged
Conversation
… time Every kernel launch on every backend is one row under tl::profile, made by gpu::launched(kernel name) in gpu_abi.h; a backend's launch primitive calls it once per launch — shared ops and own ops alike — and adds the device time where it has one. CUDA, Metal and WebGPU each had their own copy of the record, the shared layer a fourth for the backends that had none, and traits::profiles_launches said which was which; the flag is gone. TL_PROFILE=1 now starts at the first launch as well as the first evaluation, so a decoder on the model path, which never reaches the evaluator, is profiled from its first kernel (the chat decoder: 0 scopes, 2376 launches). The CUDA trace against master is identical; the host backend records a kernel that ran and its four own ops, so its suite sees the same rows.
A reduction that leaves the device short of busy groups is split over more of them and combined after: CUDA's decode GEMV, its bf16 GEMM tiles and its decode attention each did that arithmetic in their own words, and Metal's GEMV and attention copied it by hand with the numbers of a 16-core GPU. Now policy::split_parts and split_chunk (gpu_abi.h) are the arithmetic, under a split_rule that is the kernel's side of it (the target, the shortest k, the shortest part, the granule), and the target comes from traits::fill_groups (164 on CUDA, 64 on Metal) rather than a constant of one backend. CUDA's numbers are preserved exactly, since attn_dpos_chunk in the kernel source reproduces them for the captured-graph path: the launch trace against master is identical, and a test pins the rules' values. Metal's GEMV now rounds its part count up rather than down and its attention rounds a chunk to four keys, as CUDA's does; decode throughput is unchanged (121 tok/s either way on Qwen2.5-0.5B). CUDA's f32 GEMM wave plan stays its own, written against the trait.
A fused op — rmsnorm, swiglu, the decode GEMV, the cache writes, rope, the attention, split_heads / merge_heads, argmax — was a kernel a backend had or a false the model had to route around, and caps::model_path said which backend was which. Now gpu_ops.h holds one generic composition of each out of the tier-0 ops (gpu::generic), and the op takes it when the shared launch declines or `own` has no member. A backend with only tier 0 runs the whole f32 model path: the KV cache, the decoder's imperative step, the model-path tests, all of it now runs on WebGPU and on the host backend, and caps::model_path is gone. An operand the tier has no reader for (a bf16 cache or weight, int4 weights) still declines, under the caps that name it. A composition names no backend and no array — spans, tier-0 ops, and the device core's alloc / release / cpu_barrier / sync_to_host — and no launch in it writes a buffer it reads, which WebGPU forbids within a dispatch (and invalidates the whole encoder over). A second test runs each composition beside the backend's kernel, where it has one, and requires the same numbers; the eps in rmsnorm is the kind of thing it catches. The CUDA trace against master adds the new test's launches and changes nothing else.
…ees on sgemm_wave_chunk_ divided traits::fill_groups (int64_t) by variables the function keeps as long, and passed the mismatch straight into std::min with no explicit template argument. Every platform this library builds on has long == int64_t except Windows (LLP64, long is 32-bit), where deduction had no common type and the build failed. One cast at the top of the function keeps the rest of it in long, as it was before the trait existed.
yhirose
added a commit
to yhirose/culebra
that referenced
this pull request
Sep 22, 2026
cpp-tensorlib 5911d39 -> 6abecc6 (PR yhirose/cpp-tensorlib#3): every launch is recorded in one place regardless of backend, a split against the device's fill is one shared rule stated beside each kernel, and the fused model-path ops fall back to a generic composition of tier-0 ops where a backend has no kernel for them, so WebGPU and the host reference backend now run the whole f32 model path. Verified on NVIDIA hardware.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2. Three commits, each independently landable:
A launch is recorded in one place, and a backend only stamps its time. Every kernel launch on every backend is one row under
tl::profile, made bygpu::launched(kernel name)ingpu_abi.h; a backend's launch primitive calls it once per launch — shared ops and own ops alike — and adds the device time where it has one.traits::profiles_launchesis gone.TL_PROFILE=1now starts at the first launch as well as the first evaluation, so a decoder on the model path, which never reaches the evaluator, is profiled from its first kernel.A split is one rule of the device's fill, stated beside its kernel.
policy::split_parts/split_chunkunder asplit_rule(gpu_abi.h) replace CUDA's decode-GEMV split-K, bf16-GEMM z-split and decode-attention split-KV arithmetic, and Metal's hand-copied GEMV and attention splits, with the target coming fromtraits::fill_groups(164 on CUDA, 64 on Metal) rather than a constant of one backend. CUDA's numbers are preserved exactly — the launch trace against master is identical — sinceattn_dpos_chunkin the kernel source reproduces them for the captured-graph path.The model path out of tier 0, once, under every fused kernel.
gpu_ops.hnow holds one generic composition of each fused op (rmsnorm,swiglu, the decode GEMV, the cache writes,rope, the attention,split_heads/merge_heads,argmax) out of the tier-0 ops, and the op takes it when the shared launch declines orownhas no member. A backend with only tier 0 now runs the whole f32 model path — WebGPU and the host reference backend both do, andcaps::model_pathis gone. A second test runs each composition beside the backend's kernel and requires the same numbers.Verified locally (native ctest +
tensorlib_check_qwen, host backend--gpu/--auto, WebGPU under Deno, CUDA host-side trace against origin/master) at each commit; mutation checks confirm the profile and generic-composition tests actually bite.