# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("samgreen/internlm2-chat-20b-nf4", trust_remote_code=True, device_map="auto")Quick Links
InterLM2-Chat NF4 Quant
Usage
As of 2024/1/17, Transformers must be installed from source and bitsandbytes >=0.42.0 is required in order to load serialized 4-bit quants.
pip install -U git+https://github.com/huggingface/transformers bitsandbytes
Quantization config
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
Not necessary for inference, just load the model without specifying any quantization/load_in_*bit.
Model Details
- Downloads last month
- 17
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="samgreen/internlm2-chat-20b-nf4", trust_remote_code=True)