Instructions to use Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1")
model = AutoModelForCausalLM.from_pretrained("Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M

Use Docker

docker model run hf.co/Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M

LM Studio
Jan

vLLM

How to use Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M

SGLang

How to use Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1 with Ollama:
```
ollama run hf.co/Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1 with Docker Model Runner:
```
docker model run hf.co/Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M
```

Lemonade

How to use Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1:Q4_K_M

Run and chat with the model

lemonade run user.Tamil-Mistral-7B-Instruct-v0.1-Q4_K_M

List all available models

lemonade list

Atomic Chat

Model Card for Tamil-Mistral-7B-Instruct-v0.1

The Tamil-Mistral-7B-Instruct-v0.2 Large Language Model (LLM) is an improved instruct fine-tuned version of Tamil-Mistral-7B-Instruct-v0.1. Tamil LLM: A Breakthrough in Tamil Language Understanding In the realm of language models, the fine-tuned Tamil Mistral model represents a significant advancement. Unlike its English counterpart, the Tamil Mistral model is specifically tailored to comprehend and generate text in the Tamil language. This innovation addresses a critical gap, as the English Mistral model fails to effectively engage with Tamil, a language rich in culture and heritage. Through extensive fine-tuning with a base Tamil Mistral model, this iteration has been meticulously enhanced to grasp the nuances and intricacies of the Tamil language. As a result, we are delighted to present a revolutionary model that enables seamless interaction through text. Welcome to the future of conversational Tamil language processing with our instructive model.

Dataset

Tamil open instruct dataset (400k) instruction google translated

Training time

18 hrs to train on NVIDIA RTX A6000 48GB with batch size of 30

Kaggle demo link

https://www.kaggle.com/code/hemanthkumar21/tamil-mistral-instruct-v0-1-demo/

from transformers import  (AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig,TextStreamer,pipeline)
import torch
model_name = "Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1"
nf4_config = BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type="nf4",bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(model_name,device_map='auto',quantization_config=nf4_config,use_cache=False,low_cpu_mem_usage=True )
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
streamer = TextStreamer(tokenizer)
pipe = pipeline("text-generation" ,model=model, tokenizer=tokenizer ,do_sample=True, repetition_penalty=1.15,top_p=0.95,streamer=streamer)
prompt = create_prompt("வாழ்க்கையில் ஆரோக்கியமாக இருப்பது எப்படி?")
result=pipe(prompt,max_length=512,pad_token_id=tokenizer.eos_token_id)

result:
- உடற்பயிற்சி - ஆரோக்கியமான உணவை உண்ணுங்கள் -2 புகைபிடிக்காதே - தவறாமல் உடற்பயிற்சி செய்</s>

Instruction format

To harness the power of instruction fine-tuning, your prompt must be encapsulated within ~~and~~ tokens. This instructional format revolves around three key elements: Instruction, Input, and Response. The Tamil Mistral instruct model is adept at engaging in conversations based on this structured template. E.g.

# without Input
prompt_template =<s>"""சரியான பதிலுடன் வேலையை வெற்றிகரமாக முடிக்க, தேவையான தகவலை உள்ளிடவும்.

### Instruction:
{}

### Response:"""

# with Input
prompt_template =<s>"""சரியான பதிலுடன் வேலையை வெற்றிகரமாக முடிக்க, வழங்கப்பட்ட வழிகாட்டுதல்களைப் பின்பற்றி, தேவையான தகவலை உள்ளிடவும்.

### Instruction:
{}

### Input:
{}

### Response:"""

Python function to format query

def create_prompt(query,prompt_template=prompt_template):
    bos_token = "<s>"
    eos_token = "</s>"
    if query:
      prompt_template = prompt_template.format(query)
    else:
      raise "Please input with query"
    prompt = bos_token+prompt_template #eos_token
    return prompt

Demo Video

Model Architecture

This instruction model is based on Mistral-7B-v0.1, a transformer model with the following architecture choices:

Grouped-Query Attention
Sliding-Window Attention
Byte-fallback BPE tokenizer

Troubleshooting

If you see the following error:

Traceback (most recent call last):
File "", line 1, in
File "/transformers/models/auto/auto_factory.py", line 482, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
File "/transformers/models/auto/configuration_auto.py", line 1022, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
File "/transformers/models/auto/configuration_auto.py", line 723, in getitem
raise KeyError(key)
KeyError: 'mistral'

Installing transformers from source should solve the issue pip install git+https://github.com/huggingface/transformers

This should not be required after transformers-v4.33.4.

Limitations

The Mistral 7B Instruct model is a quick demonstration that the base model can be easily fine-tuned to achieve compelling performance. It does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.

Quantized Versions:

coming s00n

How to Cite

@misc{Tamil-Mistral-7B-Instruct-v0.1, 
      url={[https://huggingface.co/Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1]https://huggingface.co/Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1)}, 
      title={Tamil-Mistral-7B-Instruct-v0.1}, 
      author={"hemanth kumar"}
}

Downloads last month: 397

Safetensors

Model size

7B params

Tensor type

F16

Model tree for Hemanth-thunder/Tamil-Mistral-7B-Instruct-v0.1

Base model

Hemanth-thunder/Tamil-Mistral-7B-v0.1

Quantized

(3)

this model

Quantizations

2 models

Hemanth-thunder
/

Tamil-Mistral-7B-Instruct-v0.1