Block FP32 vision attention value accumulation

[Mikyx-1]

Gemma 3 vision attention on main calls MulByConstAndAdd for every query/source pair, repeatedly loading V and updating the output vector. This change packs V per head and computes four query rows together, keeping FP32 accumulators in SIMD registers and sharing V loads.

Fixes #1018

Scope and implementation

• Targets dev. The current patch and measurements were developed against main at 3ed403e; dev has replaced this loop with FlashAttention, so integration requires a port and fresh validation. The measurements below do not establish a speedup over dev.
• Preserves the source-position accumulation order, supports row/vector tails, and uses the existing thread pool.
• Adds 1.125 MiB of scratch for 4,096 patches × 72 FP32 values, reused across heads.
• Includes only the kernel, its integration, and required CMake/Bazel header entries.

Measurement setup

• CPU: Intel Core i5-12400F; six pinned physical CPU workers, AVX2, spin waits enabled.
• Platform/build: Linux x86-64, GCC 13.3.0, Release (-O3 -DNDEBUG).
• Model: Gemma 3 4B, 4b-it-sfp-padded.sbs weights.
• Input: repository paligemma/testdata/image.ppm, resized to 896 × 896 (4,096 patches).
• Three serial A/B pairs, alternating order: before/after, after/before, before/after.
• Each process performed one warmup encoding and one measured encoding, with a common baseline-calibrated MatMul plan to isolate this change from autotuning differences.
• Both local A/B builds included the same pre-existing KV-cache changes; the baseline was not a pristine main build. Those changes are excluded from this PR.

Image-encoder latency

Pair	Before	After
1	38.019 s	19.455 s
2	38.899 s	19.985 s
3	38.887 s	19.703 s
Median	38.887 s	19.703 s

1.97× faster image encoding; 49.3% lower latency (19.18 s saved per image).
Timing excludes model/image loading, resizing, language-model prefill, text generation, and output-file writes. Each pass recomputes the image embeddings; there is no previous-image cache.

Validation

• Complete image embeddings were byte-identical in all three pairs: 256 × 2,560 FP32 values per run. Generated text was not compared.
• Local kernel comparisons passed 182 cases per target on AVX2 and EMU128, covering tails, unaligned/padded data, adjacent output heads, and varied values.
• All 12 related regression tests passed; all four kernel tests also passed AddressSanitizer/UndefinedBehaviorSanitizer with leak detection (kernel/tests instrumented).
• CLI and API server builds passed. The isolated PR source also compiled against clean main headers, and its four kernel tests passed.
• Tests, benchmarks, reports, and artifacts remain local and are not committed.

Measurements cover one image/model on this CPU with unlocked clocks; speed and numerical equivalence beyond the tested cases are not established.

[Mikyx-1]

Closing: I mistakenly benchmarked this optimization against main instead of dev. The reported 1.97× speedup does not apply to dev, which already uses a different FlashAttention implementation.

[jan-wassenberg]

:) Sorry about that. Yes, main is quite a ways behind, but it would indeed have been a nice optimization prior to our Flash attention.