ISTA-DASLab
/

Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16

Text Generation

text-generation-inference

Model card Files Files and versions

Resources

View closed (1)

DeepSeek-V2.5?

#6 opened about 1 year ago by

Repetitive generation without additional EOS token

#5 opened over 1 year ago by

Biomed Foundation Model

#4 opened over 1 year ago by

Yi-34B AQLM?

#3 opened over 1 year ago by

~8 tok/sec with ~5k context on vLLM with Flash Attention and `kv_cache_dtype="fp8"` on 3090TI 24GB VRAM

#2 opened over 1 year ago by