Update README.md

README.md

@@ -9,4 +9,186 @@ pipeline_tag: image-text-to-text
 library_name: transformers
 tags:
 - merge
----
+---
+
+# N2-Eye: Multimodal Conversational AI
+
+N2-Eye is a multimodal language model that combines the power of LiquidAI's LFM2-1.2B language model with OpenAI's CLIP vision encoder to enable image understanding and conversation capabilities.
+
+## Model Details
+
+- **Base Language Model**: LiquidAI/LFM2-1.2B (1.26B parameters)
+- **Vision Encoder**: OpenAI CLIP-ViT-Base-Patch32
+- **Model Type**: Image-Text-to-Text (Multimodal Conversational)
+- **Training Dataset**: CRAG-MM Multi-Turn Public Dataset
+- **License**: MIT
+- **Framework**: PyTorch + Transformers
+
+## Architecture
+
+N2-Eye uses a modular architecture that combines:
+
+1. **Language Model**: LFM2-1.2B for text generation and conversation
+2. **Vision Encoder**: CLIP for image understanding (frozen during training)
+3. **Projection Layer**: A trainable MLP that maps CLIP features to the language model's embedding space
+
+The model processes images by:
+- Encoding images with CLIP to extract visual features
+- Projecting these features through a learnable projection layer
+- Integrating projected features into the language model at special `<image>` token positions
+
+## Training Details
+
+### Dataset
+- **Source**: CRAG-MM Multi-Turn Public Dataset (v0.1.1)
+- **Format**: Multi-turn conversations with images
+- **Preprocessing**: Conversations formatted with ChatML-style tokens
+
+### Training Configuration
+- **Batch Size**: 2 per device (with gradient accumulation steps: 4)
+- **Learning Rate**: 2e-5
+- **Training Length**: 1 epoch on validation split
+- **Precision**: bfloat16
+- **Max Sequence Length**: 2048 tokens
+- **Optimization**: Gradient checkpointing enabled
+
+### Special Tokens
+- `<image>`: Placeholder for image embeddings in conversation
+- System prompt: "You are a helpful assistant trained by Liquid AI. You can see and understand images."
+
+## Usage
+
+### Basic Inference
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM, CLIPProcessor
+from PIL import Image
+
+# Load components
+tokenizer = AutoTokenizer.from_pretrained("GoofyLM/N2-Eye")
+clip_processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
+
+# Load the multimodal model (requires custom loading due to architecture)
+# See the training code for complete loading implementation
+
+# Prepare conversation
+conversation = """<|im_start|>system
+You are a helpful assistant trained by Liquid AI. You can see and understand images.<|im_end|>
+<image>
+<|im_start|>user
+What do you see in this image?<|im_end|>
+<|im_start|>assistant
+"""
+
+# Process inputs
+text_inputs = tokenizer(conversation, return_tensors="pt")
+image = Image.open("your_image.jpg")
+image_inputs = clip_processor(images=image, return_tensors="pt")
+
+# Generate response
+with torch.no_grad():
+    outputs = model.generate(
+        input_ids=text_inputs.input_ids,
+        attention_mask=text_inputs.attention_mask,
+        images=image_inputs.pixel_values,
+        max_new_tokens=150,
+        do_sample=True,
+        temperature=0.7
+    )
+
+response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(response)
+```
+
+### Chat Template
+
+N2-Eye uses ChatML format for conversations:
+
+```
+<|im_start|>system
+You are a helpful assistant trained by Liquid AI. You can see and understand images.<|im_end|>
+<image>
+<|im_start|>user
+{user_message}<|im_end|>
+<|im_start|>assistant
+{assistant_response}<|im_end|>
+```
+
+## Capabilities
+
+N2-Eye can:
+- Understand and describe images in detail
+- Answer questions about visual content
+- Engage in multi-turn conversations that reference images
+- Combine visual and textual information for comprehensive responses
+
+## Limitations
+
+- **Image Token Handling**: Requires specific placement of `<image>` tokens in conversation format
+- **Single Image**: Currently optimized for single image per conversation
+- **Training Scale**: Trained on a limited dataset (validation split only)
+- **Frozen Vision**: CLIP encoder is frozen, limiting adaptation to new visual domains
+
+## Technical Implementation
+
+### Model Architecture Classes
+
+The implementation includes several key components:
+
+1. **MultimodalLFM2Model**: Main model class combining language and vision
+2. **CRAGMMDataset**: Dataset handler for CRAG-MM format
+3. **MultimodalTrainer**: Custom trainer for multimodal inputs
+
+### Key Features
+
+- **Gradient Checkpointing**: Memory-efficient training
+- **Custom Collation**: Handles multimodal batch processing
+- **Flexible Image Integration**: Dynamic matching of image features to token positions
+- **Safe Serialization**: Custom saving to handle shared tensors
+
+## Requirements
+
+```
+torch
+transformers
+datasets
+Pillow
+clip-by-openai
+```
+
+## Training Your Own Version
+
+To retrain or fine-tune N2-Eye:
+
+1. Install dependencies
+2. Prepare your dataset in CRAG-MM format
+3. Modify configuration in the training script
+4. Run the training pipeline
+
+See the included training script for complete implementation details.
+
+## Citation
+
+If you use N2-Eye in your research, please cite:
+
+```bibtex
+@misc{n2eye2025,
+  title={N2-Eye: Multimodal Conversational AI},
+  author={GoofyLM Lab},
+  year={2025},
+  publisher={Hugging Face},
+  howpublished={\url{https://huggingface.co/GoofyLM/N2-Eye}}
+}
+```
+
+## Acknowledgments
+
+- **LiquidAI** for the LFM2-1.2B base model
+- **OpenAI** for the CLIP vision encoder
+- **CRAG-MM** dataset contributors for training data
+- **Hugging Face** for the transformers library and model hosting
+
+## License
+
+This model is released under the MIT License. See the LICENSE file for details.