Repository navigation
agx: exact division, log and sin/cos for compute; graphics keeps the base math - #14
joshuaswarren wants to merge 22 commits into
Conversation
|
Ran same-boot graphics A/Bs of
These are performance-localization experiments, not proposed reverts to merge or evidence that the old numerical behavior is correct in every case. M1 MacBook Pro / J293, G13GEach row is a separate matched A/B block. Arrows show installed control → candidate. Quake values are medians of three measured timedemos per arm; each mark score is one fixed suite per arm.
The fresh-base check and the math-removed variant make a broad build-environment difference a poor explanation for the larger losses. Small differences in these short runs should not be treated as gains. On the M1 Air / J313, full #14 gave 511.5 → 446.6 FPS and 7,423 → 6,160 vkmark points. With compiler math restored to the base and the memory fixes retained, the separate A/B gave 486.9 → 507.2 FPS and 7,344 → 7,378 points. The control FPS varied, but removing the math returned the candidate to roughly its original throughput. On M3 Pro / J516S, full #14 was essentially unchanged: 185.9 → 185.7 FPS, glmark2 718 → 718, vkmark 2,209 → 2,220. The diagnostic splits above were run on the M1s, not M3. A separate Quake's full-screen gamma shader uses This points toward checking cheaper API-compliant math paths while retaining the necessary correctness fixes. In particular, a bitwise difference from a correctly rounded CPU reference alone does not establish an ordinary-division conformance failure: the Vulkan precision table permits 2.5 ULP over the specified divisor range. These graphics measurements do not establish the numerical conformance of a replacement or measure MLX/model performance. Reproduction, exact diagnostic changes, and measurement scopeAll variants start from the exact #14 head above. The original base is the exact Memory fixes with base compiler math: restore Omit only log lowering: remove this one line in - NIR_PASS(_, nir, agx_nir_lower_log);Division/reciprocal-only math split: apply the following diagnostic diff to #14. It restores the base sin/cos filter and omits the new log/trig passes; both new division passes remain. diff --git a/src/asahi/compiler/agx_compile.c b/src/asahi/compiler/agx_compile.c
index 003fd738b6e..6ede7262e61 100644
--- a/src/asahi/compiler/agx_compile.c
+++ b/src/asahi/compiler/agx_compile.c
@@ -6808,11 +6808,9 @@ agx_calc_stats(agx_context *ctx, unsigned size, struct agx2_stats *stats)
}
static bool
-agx_lower_sincos_filter(const nir_instr *instr, const void *g17p)
+agx_lower_sincos_filter(const nir_instr *instr, UNUSED const void *_)
{
- /* Only the native G17P sequence is lowered here; every other target goes
- * through agx_nir_lower_sincos, the exact reduction. */
- if (!(uintptr_t)g17p || instr->type != nir_instr_type_alu)
+ if (instr->type != nir_instr_type_alu)
return false;
nir_alu_instr *alu = nir_instr_as_alu(instr);
@@ -9859,7 +9857,6 @@ agx_preprocess_nir_internal(nir_shader *nir, bool g17p)
NIR_PASS(_, nir, nir_lower_load_const_to_scalar);
NIR_PASS(_, nir, nir_lower_flrp, 16 | 32 | 64, false);
NIR_PASS(_, nir, agx_lower_sincos, g17p);
- NIR_PASS(_, nir, agx_nir_lower_sincos);
NIR_PASS(_, nir, nir_shader_intrinsics_pass, agx_lower_front_face,
nir_metadata_control_flow, NULL);
if (g17p)
@@ -9889,7 +9886,6 @@ agx_preprocess_nir_internal(nir_shader *nir, bool g17p)
* reciprocals it builds are not refined twice.
*/
NIR_PASS(_, nir, agx_nir_lower_fdiv);
- NIR_PASS(_, nir, agx_nir_lower_log);
NIR_PASS(_, nir, nir_opt_deref);
NIR_PASS(_, nir, nir_lower_vars_to_ssa);J293/J313 retained kernel Quake: vkQuake 1.36.0, demo1, 969 frames, actual 1920×1080 verified from each run's game PNG. One discarded warm-up per arm, then control/candidate/candidate/control/control/candidate. Vsync/MSAA off, Executable SHA256:
J293 raw measured Quake samples, control / candidate:
Other suites: Omarchy M Test graphics benchmark suite 1 settings, 1920×1080, 3 seconds per scene. glmark2 2023.01, Wayland off-screen: build/VBO, linear texture, Phong shading, high-poly bump, effect2d kernel, desktop blur, jellyfish, terrain. vkmark 2025.01, headless: device-local vertex, anisotropic texture, Phong, cel, blur, desktop, cube. All scenes completed on the Apple GPU. Marks were not repeated, and this does not measure sustained clocks, thermals, stutter or full conformance. Generated-by: LLM |
|
The split and the plan for #14. The memory fixes cost nothing in your A/B, and the math costs a lot. So they should not travel together.
NOT measured yet: any candidate. The numbers above are yours. I have not reproduced the Quake or vkmark costs, and I have not measured the base against the table. |
d8af310 to
b717bba
Compare
joshuaswarren
changed the title
|
The head of this PR is now MEASURED on a native build of this head (CPU only): the five probes pass, 352 of 352 compiler tests pass, and NOT measured: any GPU run of this head, and the vkQuake and vkmark A/B. Could you run the same A/B on this head? The bound test of the base against the precision table is queued. One finding from the CTS runs of the #6 stack: none of the tests that went from Fail to Pass was a precision test. The precision groups pass on the base on the M1 Max. |
|
Measurement of the base against the Vulkan division and reciprocal table, on the GPU. Setup: 16-inch M1 Max, a compute shader, 328,192 pairs (262,144 random plus 66,048 around the 2^-126 normal limit). The oracle is the correctly rounded quotient with denormals flushed. I count only pairs with a divisor in the specified range, 2^-126 to 2^126 (324,976 pairs). Tolerance 2.5 ULP.
All 169 base cases have an exact quotient of exactly 2^-126, the smallest normal number. The base returns a flushed zero there. All 25 special mismatches are the mirror case at the same boundary. So the base is off only where the correct result is within one rounding step of the denormal limit. A result that close to the denormal range may be flushed under the spec, so this is not an ordinary conformance failure. Everywhere else the base division is inside 2.5 ULP, and its reciprocal is within 1 ULP. What this means for the plan:
I will offer two cheap versions and measure both:
Which one should land depends on whether compute users need the rounding without NOT measured: log, log2, pow, sin and cos against the table. The bound test only covers division and reciprocal so far. |
|
Now log, log2, exp, exp2, pow, sin and cos, on the GPU. Setup: 16-inch M1 Max, a compute shader, 262,144 random inputs per operation, a float64 reference. The bounds are the Vulkan precision table.
So the base meets the table for all of these on this chip. The exact log, sin and cos sequences are not needed for the table. With the division result above, the base sequences meet the whole table. The old head is more accurate than the table asks. That accuracy is what costs the graphics speed. Consequence for this PR: the exact sequences are an option for code that wants correctly rounded results. They are not a conformance fix. The head already limits them to compute. You may prefer them only for NOT measured:
|
|
The decision is to keep head Could you rerun your vkQuake, vkmark and glmark2 A/B on I am measuring the compute cost with an MLX decode A/B on the 16-inch M1 Max. I will post it here. |
hk, spirv: image texel visibility and the free-memory reserve (split out of #14)
|
The compute cost after the last change, measured on an M2 Max by another lane, same method as before (five interleaved rounds per model, one boot, release builds, digests equal in every round). Arms: base
For comparison, the head before ( These absolute numbers are the base tree of #14, not the cooperative matrix stack of #6. NOT measured here: graphics on this head (expected to equal the base by construction; the vkmark A/B of |
|
The per-operation table of the head
So the remaining cost sits in the exact log and sin/cos for compute. The decode cost stands at -2.5 and -1.1 percent with equal outputs. The ULP figures near zero are the near-zero metric, not typical error. |
|
Status, no change to head |
…duction
Three fp32 rounding defects on Honeykrisp (Apple M1 G13G, Mesa 26.1.7)
that MLX patches around in its shaders:
a) GLSL "/" was lowered by nir_opt_algebraic (lower_fdiv) to
a * frcp(b), and frcp was only Newton-refined when the instruction
was `precise`. Even with an exact reciprocal a * RN(1/b) is not
RN(a/b): 80/25 and 10/25 come out one ulp low, 6.55859375/0.030016447
one ulp high. Measured on 1008 pairs: 299 mismatches (298 at 1 ulp,
1 at 2 ulp), 1/b: 204 mismatches; 1M random pairs: 305792 mismatches.
fma() itself is IEEE (0/1000 mismatches), the MLX note
blaming fma contraction was a misdiagnosis.
b) log(x) was flog2(x) * ln2 on the hardware log2, which is only
absolutely accurate: log(3) = 1.0986121892929077 instead of
1.0986123085021973, 2792/4107 mismatches on [0.5, 64] with errors up
to 357 ulp near 1 (log(1.0000001) = 1.65e-7 vs 1.0e-7); log2 had
2905/4107 mismatches.
c) sin/cos reduced the argument as fract(x / 2pi) in fp32, so the
reduced angle lost ulp(x) of precision: sin(3.0) 14 ulp off,
sin(1e5) 1.06e-3 absolute error, sin(1e8) = 0.0, sin(-1e-3) 571 ulp
off, sin(pi_f32) = 0, and the hardware sin_pt_2 core itself is 2 to
3 ulp off (cos(1e-3) 3 ulp).
The lowering moves to a new the AGX compiler pass file agx_nir_lower_math.c,
built only from the exact fp32 fma and marked no-fast-math so
nir_opt_algebraic cannot reassociate it:
- lower_fdiv is dropped from agx_nir_options; fdiv32 becomes Markstein
division (y = refined rcp, q = a y, r = fma(-b, q, a), q + r y) with
the NaN/zero guard keeping q for b = 0, overflow and signed zero.
frcp32 is always refined. fdiv16 keeps a * frcp(b).
- fmul(flog2(x), ln2) as emitted by nir_flog, and any remaining flog2,
become a double-float evaluation of log(m) = 2s + 2s^3 P(s^2) with
s = (m-1)/(m+1) kept as a pair, m folded into [sqrt(1/2), sqrt(2)),
e ln2 split hi/lo and Fast2Sum accumulation (about 40 bits before the
final rounding). Zero/subnormal/negative/inf/NaN are handled by bit
tests, not the hardware log2.
- fsin/fcos: Cody-Waite in radians with pi/2 in three pieces, the k C2
product and its subtraction as TwoProduct/TwoSum so the reduced
argument is a pair accurate to an ulp, for |x| < 2^22; an integer
Payne-Hanek (24-bit mantissa times a 128-bit window of 2/pi, 64
fraction bits kept) for |x| >= 2^22 up to FLT_MAX. The quadrant then
selects a Taylor polynomial in quadrant units (sin degree 13, cos
degree 14, leading terms as pairs, low part of the argument fed to
the derivative). The hardware sin_pt_1/sin_pt_2 core (nir_op_fsin_agx)
is no longer used by the lowering. fp16 sin/cos go through the fp32
path.
Verification (standalone Vulkan compute reproducer):
- division: 1008-pair table incl. 80/25, 10/25, 1000 random: 0
mismatches (stock 299); 1/b 0 (stock 204); 1M random pairs 0
mismatches for both a/b and 1/b; b = 0, inf, NaN, overflow, flushed
subnormal and -0/5 = -0 match host IEEE.
- log: 4107 points on [0.5, 64] plus pinned exact cases: 0 mismatches
(stock 2792), log(3) = 1.0986123085021973; log2 0 (stock 2905);
1M random on [1e-37, 1e38]: log 3 at 1 ulp, log2 14 at 1 ulp,
rest exact.
- sin/cos at 1e5, 5e5, 1e6, 5e6, 1e8: all 0 ulp from the double
reference except cos(5e6) at 1 ulp (stock: 2.8e5 to 1.06e9 ulp);
1310 mixed points max 1 ulp; 1M random over [1e-3, 1e9] and [-10, 10]:
sin 42346 and cos 41184 at 1 ulp, rest exact, max 1 ulp (stock
>= 8 ulp for 627689 / 512816 of them).
- MLX suites on the same Release tree, stock vs fixed:
primitive 98/98, eq_math 7/7, fast_ops 30/30, runtime 40/40,
reduce_ops 34/34 both; complex_ops 30/33 both with the same three
pre-existing cases (failed assertions 31 -> 29). The upstream
ops_tests log(3) pin through libmlx: FAIL on stock, PASS on fixed,
as do 80/25, 10/25 and sin/cos up to the 1e5 limit mlx enforces.
- timing: 16M-element elementwise dispatch is memory bound at ~2.45 ms
on both drivers (sin 2.474 -> 2.463 ms median, within spread). A
32-deep in-register chain shows the ALU cost: sin 6.64 -> 39.6 ms,
log 2.57 -> 34.8 ms, div 2.84 -> 7.24 ms, mul unchanged.
The sin zero-return selects the argument when it compares equal to zero. The comparison flushes, so a subnormal argument matches and leaks out unflushed through the select. From flushed sources (in-shader arithmetic) the argument is already zero and nothing changes; from unflushed sources (buffer loads) a shader declaring DenormFlushToZero must see a flushed, sign-preserving zero, which the hardware fmul path it sits behind does not give it. Gate the fix on the DenormFlushToZero fp32 execution mode alone. The CTS float_controls sin cases are all DenormFlushToZero (optionally with SignedZeroInfNanPreserve on other operands); SignedZeroInfNanPreserve alone never exercises sin and preserving the subnormal argument is correct under DenormPreserve and under no mode at all, so those modes keep the existing emission byte-identical. Division is deliberately left unkeyed: the CTS fp32 group exercises OpFDiv only under DenormFlushToZero with in-shader-flushed denorm inputs (expectations V_ZERO/V_UNUSED) and has no DenormPreserve or rounding-mode division cases, and the previously-landed exact sequence flushes the subnormal-result band itself (it is FTZ-by-construction), so no existing sequence satisfies DenormPreserve there and none is required by any current CTS case. Pinned by test-lower-math.cpp: the default-mode fdiv/frcp op histograms are exact pins of the sequences the MLX decode digests were measured against, and the sin case distinguishes the keyed FTZ emission.
…ction step Stress testing on an M2 Pro found a third division failure class the original review missed (2026-09-13): when the exact quotient sits just below the normal boundary, the GPU zeroes the correction step's intermediate value and the Markstein step is silently skipped, giving 1 ulp errors -- 18,382 wrong results over 1M random pairs. The first commit in this series refined a * rcp(b) in place, so the flush landed on an intermediate, not on the result. Divide the normalized mantissas instead and restore the exponent afterwards, so the refinement always runs on operands near one and the flush happens on the exact quotient, before rounding. The rounding decision is integer: the half-ulp point is exact by construction, and the residual of the second Markstein step is exact by the fma contract, so the comparison against it decides the tie without a third refinement. The reciprocal takes the same path, and the log2 series divisor (near two) keeps the unguarded Newton form. This re-lands the division part of the fix first made on 2026-09-13 and pulled with it during the 2026-09-20 A/B isolation windows; it was never re-applied to the omacom/mesa stack. Verified against an exact model over 262,144 pairs covering edge bit patterns and the binade boundaries, and by an independent exact-rational checker; see the receipt in the PR.
nir_builder_opcodes_h.py renders the intrinsic builders from a mako template that references FP_MATH_CTRL, but the index was never imported or passed to render(), so mako resolved the name to UNDEFINED and silently skipped the builder-defaulting block for every FP_MATH_CTRL intrinsic. NoContraction on cooperative-matrix operations was therefore dropped in every driver, not just this one, and the generated header is missing the twelve fp_math_ctrl defaulting blocks. Import the index and hand it to the template. The generated header grows exactly 24 lines, twelve two-line defaulting blocks.
Stress runs in review (2026-09-13) found that a precise log2(x) * ln2 gave different bits depending on whether ln2 was a literal or read from a buffer, on 26.8% of inputs. The stack carries an amended form of the log fix in which a multiply of a hardware flog2 by the fp32 ln2 constant is rewritten to the correctly rounded soft log. That reopens the finding from the other side: the literal form now gets the single-rounding soft log, while the buffer form still computes soft log2 and multiplies, so the two forms of the same source expression differ by 1 ulp on 25.3% of inputs (66,332 of 262,144 in the literal_vs_buffer reproducer), and the precise multiply check fails the same way. Drop the literal-only rewrite. log2 is lowered to the double-float soft log2 everywhere, and log() keeps the one rounded multiply the source asks for, from a literal or from a buffer: the two forms now agree bit for bit on every input, powers of two still come out exact because the series returns the exponent unchanged, and log stays within 1 ulp on the 262,144-input reproducer.
The Vulkan SPIR-V environment requires only 2.5 ulp for OpFDiv (docs.vulkan.org/spec/latest/appendices/spirvenv.html), but the correctly rounded division kept as the AGX default costs ~2.2% 4B prefill on G13G. A cheaper guarded Markstein default (common 2^-3 operand scale for d >= 2^126, float selects for the specials) was tried and measured at 8,376 over-bound results over the 328,192-pair set: the compute pipeline preserves denormals, so any float-path special handling has to re-add the integer flush machinery, and nothing is left to save. The normalized correctly rounded division therefore stays the default for every fdiv and frcp. The test pins the contract the default actually must meet: within 2.5 ulp of the correctly rounded (with FTZ) quotient over the 262,144-pair precise_math regression input plus a 66,048-pair 2^-126-straddle boundary set (the M2 Pro failure class), with bitwise specials. On 1d72152 the measured maximum is 0 ulp on both sets, division and reciprocal. The exact-rounding contract remains in precise_math.py for the nir_fp_exact path; rusticl does not advertise CL_FP_CORRECTLY_ROUNDED_DIVIDE_SQRT, so no OpenCL correctly-rounded selection exists in this driver today.
Since 43b2756, every fp32 frcp and every fdiv(1, x) takes the normalized correctly rounded division. After constant folding that is about 28 ops; "hk: enable cooperative matrices by default on G14X" used about 5. Decode kernels take many reciprocals (sigmoid, softmax normalizers), so this change costs decode speed. A per-commit bisect of "hk: enable cooperative matrices by default on G14X" to "tests: pin the 2.5 ulp default-division bound for AGX" on an M1 Max (G13C, MLX held fixed, 5 alternating rounds) put -6.63 tok/s of 2B decode on 43b2756, negative in 5 of 5 rounds. 443153a recovered 2.1 tok/s; no other commit moved decode beyond noise. For x in [2^-126, 2^126), the normalized reciprocal is the frcp refinement on x, scaled by a power of two. So 1/x is frcp plus two fma there. The other exponents have fixed results, chosen by selects: zero and denormals give +-inf, exactly 2^126 gives +-2^-126, the rest up to inf gives +-0, and NaN gives the canonical NaN. fdiv(1, x) takes the same path. General division does not change. The results are bit-identical to "tests: pin the 2.5 ulp default-division bound for AGX". tests/div_window.py compares every fp32 reciprocal (2^32) and 2^32 hashed quotients with "tests: pin the 2.5 ulp default-division bound for AGX" and with the correctly rounded FTZ quotient: 0 mismatches on G13C. The same sweep on G13G found 0 mismatches for the frcp scaling that this path uses. precise_math.py and ulp_bound.py pass unchanged (max 0 ulp on both sets). d64 decode tok/s on the M1 Max, MLX bf62cfb, medians of two 5-round runs ("hk: enable cooperative matrices by default on G14X" / "tests: pin the 2.5 ulp default-division bound for AGX" / this commit): Qwen3.8-2B 4-bit: 104.95, 103.48 / 101.20, 100.11 / 104.03, 103.53 Qwen3-4B 4-bit: 66.58, 66.34 / 64.12, 63.82 / 64.97, 64.77 9B qwen3_5 4-bit: 14.23, 14.21 / 14.08, 14.13 / 14.17, 14.13 Greedy token digests do not change.
155e743 dropped the rewrite of fmul(flog2(x), ln2) into the single rounding soft log, so that a precise log2(x) * ln2 gives the same bits from a literal ln2 and from a buffer (found in review). The cost is accuracy for every default log(). Over all 2,130,706,432 positive normal fp32 inputs against fp64 on an M1 Max (G13C): log() before 155e743: max 0.5004 ulp, mean 0.250, 3,448 not correctly rounded log() after 155e743: max 1.2388 ulp, mean 0.343, 536,324,864 not correctly rounded log2() is the same on both (max 0.5025 ulp). Restore the rewrite, but skip an exact multiply. A default log() may be evaluated more accurately than written, so it gets the soft log again. A precise multiply keeps the rounding the source asks for, so the literal and buffer forms still agree bit for bit.
The exact reciprocal took 16 ops: one Newton step inside [2^-126, 2^126)
and selects for the other exponents. The hardware rcp already gives
those results. It flushes denormal inputs to +-inf and results below
2^-126 to +-0. Where the Newton step is NaN (x zero, denormal, infinite
or NaN), keep the hardware value, then make NaN the canonical NaN. This
is the "hk: enable cooperative matrices by default on G14X" reciprocal plus the canonical NaN: 7 ops.
The results are bit-identical to "agx: round log() once unless the multiply is exact". tests/div_window.py, all
2^32 fp32 reciprocals and 2^32 hashed quotients against "agx: round log() once unless the multiply is exact" and
against the correctly rounded FTZ quotient on an M1 Max (G13C): 0
mismatches. precise_math.py and ulp_bound.py pass (max 0 ulp).
d64 decode tok/s on the M1 Max, MLX bf62cfb, 5 alternating rounds,
medians ("agx: round log() once unless the multiply is exact" / this commit / "hk: enable cooperative matrices by default on G14X"):
Qwen3.8-2B 4-bit: 103.19 / 104.35 / 104.75, ahead in 5 of 5 rounds,
median paired gain +0.76
Qwen3-4B 4-bit: 64.80 / 64.87 / 66.68
The 4B gap is general division, which stays on the exact path.
DivDefaultFastSequencePinned and FrcpDefaultSequencePinned counted the ALU ops of the "hk: enable cooperative matrices by default on G14X" division and reciprocal. Every exact lowering since "agx: flush the exact quotient in precise fp32 division, not the correction step" emits other ops, so agx_tests has failed these three pins on the PR omacom#6 head (found by w7Q, the driver's meson suite: 61/64). The bits of division and reciprocal are pinned on hardware by tests/precise_math.py, ulp_bound.py and div_window.py. What this file can pin is the float-control contract: DenormFlushToZero emits exactly the default lowering. Compare the two op histograms for fdiv and frcp instead of fixed counts.
…it late Symptom: any shader converting fp32 to a 64-bit integer aborts in the backend with "Unhandled ALU op fdiv" (agx_compile.c agx_emit_alu); the MLX suite the copy-offset tests crashes on "dtype converting copies cover the integer family and bool". Fragment shaders with perspective-correct varyings hit the same path. Cause: "agx: correctly rounded fdiv/frcp, faithful log/log2, exact sin/cos reduction" dropped lower_fdiv from the compiler options so nir_opt_algebraic no longer splits fdiv into fmul/frcp, and only ran agx_nir_lower_fdiv in agx_preprocess_nir. nir_lower_int64 (f2u64/f2i64: fdiv by 2^32, plus the frem that lower_fmod turns into a second fdiv) and agx_nir_lower_interpolation run after preprocessing, so their fdiv reached the backend, which has no fdiv. Fix: agx_nir_lower_fdiv_late runs in agx_optimize_nir after the int64 cleanup loop and lowers only fdiv, leaving frcp alone so the reciprocals built by the first pass and by the log lowering are not refined twice. Verification: hk/precise-math driver alone reproduced the abort on the 13-inch M1; with this commit the copy-offset tests passes 25/25 .
nir_lower_int64 (float to 64-bit integer conversions) and smooth offset interpolation emit a dynamic fdiv after preprocessing. The backend has no fdiv, so agx_nir_lower_fdiv_late lowers it. Three unit tests pin that the late pass lowers fdiv like the first pass, leaves a plain reciprocal alone, and does not refine an existing reciprocal a second time. MEASURED: the AGX compiler unit tests pass on a native arm64 build, 63 of 63, including the three new ones. NOT measured: a shader that goes through nir_lower_int64 or the interpolation pass end to end; the tests build the division directly. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…it needs ulp_bound.py divided the error by np.spacing of the float64 reference. np.spacing is negative for a negative reference, so the distance came out negative and any error on a negative reference passed the > 2.5 check; for a positive reference the float64 spacing made every inexact result fail instead. Use the absolute float32 spacing of the reference. Add a --self-test mode that needs no GPU: it checks the distance for 0, 1 and 3 ULP errors in both directions on positive and negative references, that a 3 ULP error on a negative reference exceeds 2.5, and that the old expression gives a negative distance for it. The script also compiles precise_math.comp, which this tree did not have; add it. MEASURED: the self test passes; the old expression gives -1610612736 for a 3 ULP error at -1.5, which passes the old check. NOT measured: the GPU run of ulp_bound.py with the corrected distance; the numbers from the old script are not to be relied on. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
Measure positive binary32 ULPs, reject non-finite results for finite references, and add seven CPU controls. Restore unchanged public shader/runner fixtures from Joshua Warren source f144c79; provenance is recorded beside the tests. No driver code changes. Generated-by: LLM
Graphics shaders pay for the exact sequences and gain nothing the Vulkan precision table asks for. A same-boot A/B of the full change on an M1 showed about 18 percent lower vkmark and 7 percent lower vkQuake. Lower the exact sequences only for compute shaders, where the rounding contract matters. Graphics keeps the base reciprocal and quotient, the base log and the base sin/cos sequence. A fdiv that is emitted after preprocessing (int64 lowering, perspective divide) is still lowered, as a multiply by the hardware reciprocal, because the backend has no fdiv. Not measured: speed or ULP of this change. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
The AGX math pass lowers graphics-stage fdiv only at 32 bits and leaves the 16-bit division untouched for a backend that has no fdiv, so any fp16 division in a graphics shader cannot compile. Pin the graphics contract in the CPU suite: a fragment-stage fdiv at 16 and 32 bits must lower to fmul + frcp in both the early and the late pass, with no ffma and no surviving fdiv. The 16-bit halves of this pin fail deliberately until the source fix lands; the 32-bit halves pin that the coming fix keeps the existing hardware-reciprocal quotient.
b717bba to
6902830
Compare
Two integration slips from rebasing the math range onto the current integration/aurora, both breaking the build: - The conflict resolution at the preamble-retry wrapper left the superseded libagx_frcp refinement and the old three-argument agx_nir_lower_fdiv in agx_compile.c alongside their replacement in agx_nir_lower_math.c, giving agx_nir_lower_fdiv conflicting types. Drop the two dead helpers; the dispatch in agx_nir_lower_math.c is the sole lowering now. - The compute guard in agx_lower_sincos still referenced the bool g17p after upstream widened the parameter to the three-valued native target code (1=G17P, 2=M5 native fragment depth, 0=older). Key the guard on native != 1: compute keeps the exact agx_nir_lower_sincos reduction on every target except G17P, and the explicit-RTE repair path, which calls the base pass with native=1, still lowers compute.
nir_lower_idiv marks everything it emits exact and opens emit_udiv with frcp(u2f32(denominator)). The refinement that correctly rounds that reciprocal used to apply in every shader stage; the math range now refines exact reciprocals in compute only, so a fragment shader with variable integer division keeps the raw hardware reciprocal. Pin the production pattern with the real lowering: run nir_lower_idiv on a fragment udiv, run the early pass, and require the refinement steps to appear, while an unmarked fast-math frcp stays untouched. The idiv half fails deliberately until the precision source fix lands.
The early pass lowers through nir_def_replace, which removes the lowered fdiv instruction, while the fixture builder cursor stayed parked after that instruction; the late-half fdivs were then inserted behind a removed instruction and crashed the suite. Reset the cursor to the end of the entrypoint implementation before building them. No assertion changes.
The math range left graphics-stage division half-wired: fdiv at 16 bits fell through to a backend that has no fdiv, and the correctly rounded reciprocal that nir_lower_idiv builds (an exact frcp over an integer conversion) was refined in compute only, so integer division in vertex and fragment shaders kept the raw hardware reciprocal. Lower graphics fdiv at 16 bits the same way as at 32, multiplying by the hardware reciprocal. Refine the exact integer-conversion reciprocal in graphics stages with the same correctly rounded helper the compute path uses, restoring the refinement scope of the pass this range replaced: keyed on the exact marking and, for application shaders, on an integer-conversion input, while ordinary graphics reciprocals keep the hardware instruction and all compute sequences are untouched. The refinement's float controls are scoped to its own emissions and restored afterwards, and the late pass stays fdiv-only, so nothing is refined twice. Failing before this change at 7f9dd7c, witnessed by CI run 40967791964 on both architectures: GraphicsFdivLowersBothWidthsInEarlyAndLatePasses and GraphicsIdivReciprocalRefinedPlainFrcpUntouched fail while the other 155 tests in the target pass.
|
Source review at c9cdabb (delta from 7f9dd7c: one commit, Result: no defect found in the delta. Traced against the two failing tests (I did not run them; I read the code and
Checked in the diff:
Observation, not a defect in this diff: Not checked: the CI results for this head, |
|
Follow-up at c9cdabb: the CI run for this head has finished. |
Current status: NOT READY TO MERGE / CPU regressions fixed (29 October)
Current head:
c9cdabbd1a7dbe36290eae553d3c6aea6d3d7f29, on Aurora672657e05e6b382a0203975f7a26c46f9a674973.The clean failing-before run at
7f9dd7cddfc46a27e90709e0b5b54aceae16529afailed exactly the two new graphics tests on both architectures (run 40967791964). The minimal shared correction now lowers fp16 graphics division like fp32 and restores exact integer-division reciprocal refinement in graphics. The compute branch and test assertions were unchanged.Full CI is now green on x86_64 and aarch64. This is CPU/compiler evidence, not on-device graphics or numerical qualification. Exact-head source review, Chris's graphics/math/precision hold, GPU precision/graphics comparison and compute qualification remain. No READY claim.
Historical evidence and original PR description (head bindings remain as recorded)
NOT READY TO MERGE
Head:
b717bba9992d55a95de848389b707a54e72c8d31.GitHub reports MERGEABLE against
integration/aurora: no text conflicts. This is not a test pass. Its merge status is UNSTABLE.This public head is still the candidate. The graphics 16-bit fdiv has a blocking source defect: it survives lowering, but the backend has no fdiv instruction. The recorded 352/352 CPU tests do not cover that case. The new fixture has not run, and production code is not yet fixed. Prove the failure, fix the shared lowering, run the final tests and obtain the focused source review. Chris's graphics/math/precision hold remains open. Local da4deb0 measurements belong to another commit and do not qualify this head. Keep this PR in draft.
Earlier review, source and test notes are kept below with their original bindings. They do not change this status.
Status: DRAFT until a GPU run
This PR is now the compiler math only. The memory fixes moved to #15. The head changed on 25 Oct. It drops them and adds one commit that keeps the cost out of graphics.
What it changes (17 commits)
fdivafter preprocessing. The backend has nofdiv, so a late pass lowers it.agx: keep the exact division, log and sin/cos lowering for compute) limits the exact sequences to compute. Graphics keeps the base reciprocal, quotient, log, sin and cos. A latefdivin graphics becomes a multiply by the hardware reciprocal.Why the last commit: a same-boot A/B on the M1 of the earlier head showed about 18 percent lower vkmark and 7 percent lower vkQuake. The M3 Pro was unchanged. The exact division explains the vkmark loss, and the log pass explains vkQuake. The Vulkan precision table asks for 2.5 ULP for division. It gives limits for log, log2 and pow. It does not ask for exact results.
MEASURED, native arm64 build of exactly this head on a 16-inch M1 Max, CPU only:
meson test: 120 pass and 19 fail. The same 19 fail on the base built the same way.MEASURED earlier, on a whole-suite CTS run of the #6 stack on the 16-inch M1 Max: none of the 34 Fail to Pass tests was a precision test. 30 were memory model tests, fixed by the image visibility change now in #15. The other 4 were timing tests. So the precision groups of the CTS pass on the base on that chip.
NOT measured: