Skip to content

fix: emit real I2_S tensors for offline-quantized LlamaModel checkpoints #623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Pujitha Paladugu (pujitha24) wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: emit real I2_S tensors for offline-quantized LlamaModel checkpoints #623

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions utils/convert-hf-to-gguf-bitnet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -804,7 +804,9 @@ def write_tensors(self):
data_torch = data_torch.unsqueeze(0).expand((4, *origin_shape)) >> shift
data_torch = data_torch & 3
data_torch = (data_torch.float() - 1).reshape((origin_shape[0] * 4, *origin_shape[1:]))
data_torch = data_torch / scale_map[name.replace(".weight", "")].float()
# For I2_S output: keep as ternary {-1,0,1}, scale is passed separately to quantize_to_i2_s
if self.ftype != gguf.GGMLQuantizationType.I2_S:
data_torch = data_torch / scale_map[name.replace(".weight", "")].float()

# use the first number-like part of the tensor name as the block id
bid = None
Expand Down Expand Up @@ -866,7 +868,14 @@ def write_tensors(self):

i2_scale = None
if self.ftype != gguf.GGMLQuantizationType.F32 and extra_f16 and not extra_f32:
if self.ftype == gguf.GGMLQuantizationType.TL1 and suit_i2:
if self.ftype == gguf.GGMLQuantizationType.I2_S and suit_i2 and name.replace(".weight", "") in scale_map:
# Only tensors from offline-quantized checkpoints (uint8 + weight_scale) are
# already ternary; a plain LlamaForCausalLM checkpoint has no such scale and
# must not be blindly ternarized here, so it falls through to the F16 branch below.
data_qtype = gguf.GGMLQuantizationType.I2_S
override_scale = scale_map[name.replace(".weight", "")].item()
data = quantize_to_i2_s(data, override_scale=override_scale)
elif self.ftype == gguf.GGMLQuantizationType.TL1 and suit_i2:
data, i2_scale = transform_to_tl1(data)
assert data.dtype == np.uint8
assert i2_scale.dtype == np.float32
Expand Down