Fix 4-bit 70B VRAM estimate: published Q4_K_M is 42.5 GB, not ~35 GB - #839
Open
prosoftkeys wants to merge 1 commit into
Open
Fix 4-bit 70B VRAM estimate: published Q4_K_M is 42.5 GB, not ~35 GB#839prosoftkeys wants to merge 1 commit into
prosoftkeys wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "Estimate VRAM requirements" section advises that a 4-bit quantized 70B model runs on ~35 GB of VRAM. The published files are larger than that, and the gap lands on a pod size boundary.
Q4_K_M for Llama 3.1 70B ships at 42.52 GB from two independent publishers:
bartowski/Meta-Llama-3.1-70B-Instruct-GGUF:Meta-Llama-3.1-70B-Instruct-Q4_K_M.ggufmradermacher/Meta-Llama-3.1-70B-Instruct-GGUF:Meta-Llama-3.1-70B-Instruct.Q4_K_M.ggufThat is 4.82 bits per weight rather than 4.0, because K-quants keep attention layers at higher precision. The ~35 GB figure is 70.6B x 4 bits, which is the theoretical rung rather than the file anyone downloads.
Why it matters for pod selection: 42.52 GB is 39.60 GiB, so on a 40 GB card the weights alone leave 0.40 GiB for the KV cache and everything else. Someone sizing from this line picks a 40 GB pod and hits OOM during load. A 48 GB card is the honest recommendation at 4-bit.
I have also added one line to the Resource calculators list. Disclosure: I maintain vramcalculator.com, so please treat that half as a suggestion and drop it if you would rather not add another link. The correction above stands on its own and is the reason I opened this.