AI & Machine Learning
GPTQ
Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers. Covers loading pre-quantized GPTQ models, quantizing your own model, choosing group size, selecting ExLlamaV2, Marlin, or Triton kernels, and QLoRA fine-tuning with PEFT. Use when fitting 70B-class models onto limited or consumer GPUs, cutting memory about 4x versus FP16, speeding up inference, finding pre-quantized checkpoints on HuggingFace, or fine-tuning a quantized model with LoRA. For slightly better accuracy on newer GPUs use AWQ instead, and for simple 8-bit or on-the-fly quantization use bitsandbytes.
- Publisher / source
- KalarisLabs/research-agent-skills (found on the CSV list)
- Upstream record
- KalarisLabs/research-agent-skills
- Version
- Version 2
- Published
- Publication review
- Approved for publication
- Content fingerprint
37b1142e0a4d3d19fd3d98228fb7431a241004279331b9a9da8b0409de5b6365