Repository navigation
Expose native FP32 recurrent state storage for ExLlamaV3 - #503
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ExLlamaV3 1.6.3 defaults GatedDeltaNet/KDA recurrent state storage to FP16 and exposes
Cache(recurrent_dtype=torch.float)for FP32. Tabby currently cannot select that native option, so upgrading an existing FP32 setup silently changes its storage precision.Add
model.recurrent_state_fp32and the corresponding model-load field. The default remains false, preserving current upstream behavior. When true, forwardtorch.floatthrough the shared cache constructor for main and draft caches, including quantized K/V modes. This changes recurrent storage only; scans still compute in FP32. FP32 doubles recurrent state and checkpoint memory relative to FP16.Validation: eight isolated cache-forwarding cases cover default/explicit FP16, FP32, quantized K/V, and main/draft paths. A real 48-layer recurrent CUDA model on ExLlamaV3 1.6.3 / Torch 2.13.0 loaded with all 48 initialized recurrent tensors in
torch.float32(301,989,888 bytes). Coding, image, eight tool-use cases, a 40,934-token dispersed lookup and warm continuation passed. JSON grammar, stream cancellation, and the next request also passed. These are brief compatibility observations, not broad quality or speed claims.