Update README.md

README.md

@@ -1,3 +1,121 @@
----
-license: gemma
----
+---
+license: gemma
+library_name: vllm
+pipeline_tag: image-text-to-text
+extra_gated_heading: Access Gemma on Hugging Face
+extra_gated_prompt: To access Gemma on Hugging Face, you’re required to review and
+  agree to Google’s usage license. To do this, please ensure you’re logged in to Hugging
+  Face and click below. Requests are processed immediately.
+extra_gated_button_content: Acknowledge license
+base_model: google/gemma-3-27b-it
+---
+
+# FP8 Dynamic Quantized Gemma-3-27b-it
+
+### Features
+- Image text to text
+- Tool chain
+
+ 
+## 1. What FP8‑Dynamic Quantization Is
+* **FP8 format**  
+  * 8‑bit floating‑point (1 sign bit + 5 exponent bits + 2 mantissa bits).  
+  * Drastically shrinks weight/activation size while keeping floating‑point behavior.
+* **Dynamic scheme (`FP8_DYNAMIC`)**  
+  * **Weights:** *static*, **per‑channel** quantization (each out‑feature channel has its own scale).  
+  * **Activations:** *dynamic*, **per‑token** quantization (scales are recomputed on‑the‑fly for every input token).
+* **RTN (Round‑To‑Nearest) PTQ**  
+  * Post‑training; no back‑prop required.  
+  * No calibration dataset needed because:  
+    * Weights use symmetric RTN.  
+    * Activations are quantized dynamically at inference time.
+
+## 2. Serving the FP8 Model with vLLM
+
+```
+vllm serve BCCard/gemma-3-27b-it-FP8-Dynamic \
+  --tensor-parallel-size 4 \
+  --gpu-memory-utilization 0.9  \
+  --max-model-len 8192 \  
+  --enforce-eager \  
+  --api-key bccard \
+  --served-model-name gemma-3-27b-it
+```
+
+## 3. Quantization Code Walk‑Through (Shared Knowledges)
+
+[LLM Compressor](https://github.com/vllm-project/llm-compressor) is an easy-to-use library for optimizing models for deployment with vllm, including:
+
+Comprehensive set of quantization algorithms for weight-only and activation quantization
+Seamless integration with Hugging Face models and repositories
+safetensors-based file format compatible with vllm
+Large model support via accelerate
+
+```
+from transformers import AutoProcessor, Gemma3ForConditionalGeneration
+from llmcompressor.modifiers.quantization import QuantizationModifier
+from llmcompressor.transformers import oneshot
+
+model_name = "google/gemma-3-27b-it"
+
+processor = AutoProcessor.from_pretrained(model_name)
+model = Gemma3ForConditionalGeneration.from_pretrained(model_name, device_map="auto", torch_dtype="auto", trust_remote_code=True)
+
+recipe = QuantizationModifier(
+    targets="Linear",
+    scheme="FP8_DYNAMIC",
+    ignore=['re:.*lm_head', 're:vision_tower.*', 're:multi_modal_projector.*'],
+)
+
+SAVE_DIR = "gemma-3-27b-it-FP8-Dynamic"
+oneshot(model=model, recipe=recipe, output_dir=SAVE_DIR)
+processor.save_pretrained(SAVE_DIR)
+```
+
+## 4. Gemma 3 model card
+
+**Model Page**: [Gemma](https://ai.google.dev/gemma/docs/core)
+
+**Terms of Use**: [Terms][terms]
+
+**Authors**: Google DeepMind, BC Card (Quatization)
+
+### Description
+
+Gemma is a family of lightweight, state-of-the-art open models from Google,
+built from the same research and technology used to create the Gemini models.
+Gemma 3 models are multimodal, handling text and image input and generating text
+output, with open weights for both pre-trained variants and instruction-tuned
+variants. Gemma 3 has a large, 128K context window, multilingual support in over
+140 languages, and is available in more sizes than previous versions. Gemma 3
+models are well-suited for a variety of text generation and image understanding
+tasks, including question answering, summarization, and reasoning. Their
+relatively small size makes it possible to deploy them in environments with
+limited resources such as laptops, desktops or your own cloud infrastructure,
+democratizing access to state of the art AI models and helping foster innovation
+for everyone.
+
+### Inputs and outputs
+
+-   **Input:**
+    -  Text string, such as a question, a prompt, or a document to be summarized
+    -  Images, normalized to 896 x 896 resolution and encoded to 256 tokens
+       each
+    -  Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and
+       32K tokens for the 1B size
+
+-   **Output:**
+    -   Generated text in response to the input, such as an answer to a
+        question, analysis of image content, or a summary of a document
+    -   Total output context of 8192 tokens
+
+### Citation
+
+```none
+@article{gemma_2025,
+    title={Gemma 3 FP8 Dynamic},
+    url={https://bccard.ai},    
+    author={BC Card},
+    year={2025}
+}
+```