Instructions to use wenbopan/Faro-Yi-34B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use wenbopan/Faro-Yi-34B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="wenbopan/Faro-Yi-34B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("wenbopan/Faro-Yi-34B")
model = AutoModelForCausalLM.from_pretrained("wenbopan/Faro-Yi-34B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use wenbopan/Faro-Yi-34B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "wenbopan/Faro-Yi-34B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wenbopan/Faro-Yi-34B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/wenbopan/Faro-Yi-34B

SGLang

How to use wenbopan/Faro-Yi-34B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "wenbopan/Faro-Yi-34B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wenbopan/Faro-Yi-34B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "wenbopan/Faro-Yi-34B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wenbopan/Faro-Yi-34B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use wenbopan/Faro-Yi-34B with Docker Model Runner:
```
docker model run hf.co/wenbopan/Faro-Yi-34B
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

The Faro chat model focuses on practicality and long-context modeling. It handles various downstream tasks with higher quality, delivering stable and reliable results even when inputs contain lengthy documents or complex instructions. Faro seamlessly works in both English and Chinese.

Faro-Yi-34B

Faro-Yi-34B is an improved Yi-34B-200K with extensive instruction tuning on Fusang-V1. Compared to Yi-34B-200K, Faro-Yi-34B has gained greater capability in various downstream tasks and long-context modeling thanks to the large-scale synthetic data in Fusang-V1.

Just like Yi-34B-200K, Faro-Yi-34B supports up to 200K context length.

How to Use

Faro-Yi-9B-200K uses chatml template. I recommend using vLLM for long inputs.

import io
import requests
from PyPDF2 import PdfReader
from vllm import LLM, SamplingParams

llm = LLM(model="wenbopan/Faro-Yi-34B")

pdf_data = io.BytesIO(requests.get("https://arxiv.org/pdf/2303.08774.pdf").content)
document = "".join(page.extract_text() for page in PdfReader(pdf_data).pages) # 100 pages

question = f"{document}\n\nAccording to the paper, what is the parameter count of GPT-4?"
messages = [ {"role": "user", "content": question} ] # 83K tokens
prompt = llm.get_tokenizer().apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
output = llm.generate(prompt, SamplingParams(temperature=0.8, max_tokens=500))
print(output[0].outputs[0].text)
# Yi-9B-200K:      175B. GPT-4 has 175B \nparameters. How many models were combined to create GPT-4? Answer: 6. ...
# Faro-Yi-9B-200K: GPT-4 does not have a publicly disclosed parameter count due to the competitive landscape and safety implications of large-scale models like GPT-4. ...

Or With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('wenbopan/Faro-Yi-34B', device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained('wenbopan/Faro-Yi-34B')
messages = [
    {"role": "system", "content": "You are a helpful assistant. Always answer with a short response."},
    {"role": "user", "content": "Tell me what is Pythagorean theorem like you are a pirate."}
]

input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
generated_ids = model.generate(input_ids, max_new_tokens=512, temperature=0.5)
response = tokenizer.decode(generated_ids[0], skip_special_tokens=True) # Aye, matey! The Pythagorean theorem is a nautical rule that helps us find the length of the third side of a triangle. ...