marksverdhei/GLM-4.7-Flash-FP8

GLM-4.7-Flash FP8

FP8 quantized version of zai-org/GLM-4.7-Flash.

Also, see Unsloth's new unsloth/GLM-4.7-Flash-FP8-Dynamic

Quantization Details

• Method: FP8 E4M3 per-tensor quantization with embedded scales
• Original size: ~62GB (BF16)
• Quantized size: ~30GB (FP8)
• Preserved in BF16: lm_head, embed_tokens, layernorms, router weights

Performance

Tested on 2x RTX 3090 (24GB each) with vLLM 0.13.0:

Setting	Value
Tensor Parallel	2
Context Length	8192
VRAM per GPU	14.7 GB
Throughput	19.4 tokens/sec

Note: RTX 3090 lacks native FP8 support, so vLLM uses the Marlin kernel for weight-only FP8 decompression. GPUs with native FP8 (RTX 40xx, Ada Lovelace+) will achieve higher throughput.

Usage with vLLM

Requires vLLM 0.13.0+ and transformers 5.0+ for glm4_moe_lite architecture support.

from vllm import LLM, SamplingParams

llm = LLM(
    model="marksverdhei/GLM-4.7-Flash-fp8",
    tensor_parallel_size=2,
    max_model_len=8192,
    enforce_eager=True,  # Optional: disable CUDA graphs to save VRAM
)

outputs = llm.generate(["Hello, world!"], SamplingParams(max_tokens=100))
print(outputs[0].outputs[0].text)

License

MIT (same as base model)