metadata
base_model:
- meta-llama/Meta-Llama-3-8B
base_model_relation: quantized
license: llama3
Model Card
- Base model:
meta-llama/Meta-Llama-3-8B - Quantization method: LNQ with GuidedQuant Hessian
- Target bit-width: 4
- Backend kernel: Any-Precision-LLM kernel (
ap-gemv) - Calibration data: RedPajama (1024 sentences / 4096 tokens)
- Calibration objective: Next-token prediction
- num_groups (for GuidedQuant Hessian): 1
How to run
- Follow the instruction in https://github.com/snu-mllab/GuidedQuant.