---
license: mit
datasets:
- HuggingFaceFV/finevideo
language:
- en
pipeline_tag: video-text-to-text
library_name: diffusers
tags:
- Torch
- CustomModel
- HRM
---

# Model Card for TTV-HRM

<!-- Provide a quick summary of what the model is/does. -->

TTV-HRM is a lightweight text-to-video generation model that uses a Hierarchical Reasoning Module (HRM) to generate short video clips from textual prompts, optimized for efficiency on T4 GPUs.
<!-- 
## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

<!-- TTV-HRM (Text-to-Video Hierarchical Reasoning Model) is a compact PyTorch-based model for generating low-resolution video clips (8 frames at 32x32) conditioned on text prompts. It features a GPT-2 text encoder, 3D CNN video tokenizer/detokenizer, rotary embeddings, and a dual-level HRM with adaptive reasoning steps (up to 3) for iterative refinement. The model emphasizes interpretability and low compute, serving as a research baseline for multimodal generation. -->

<!-- - **Developed by:** [Ahsan Umar](https://huggingface.co/codewithdark)
- **Funded by [optional]:** Self-funded
- **Shared by [optional]:** XCollab
- **Model type:** Transformer-based generative model
- **Language(s) (NLP):** English (text conditioning)
- **License:** MIT
- **Finetuned from model [optional]:** None (from scratch)
 -->
<!-- ### Model Sources [optional] -->

<!-- Provide the basic links for the model. -->

<!-- - **Repository:** https://huggingface.co/XCollab/TTV-HRM
- **Paper [optional]:** Technical Report (forthcoming)
- **Demo [optional]:** Colab Notebook (see "How to Get Started")

## Uses -->

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

<!-- ### Direct Use -->

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
<!-- 
- Research prototypes for text-to-video synthesis.
- Generating simple animations (e.g., "a red ball moving left to right").
- Educational demos in AI courses.

Foreseeable users: Researchers, students, hobbyists. Affected: End-viewers of generated content (potential for misleading visuals). -->

<!-- ### Out-of-Scope Use -->

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

<!-- - High-fidelity or long-form video production.
- Real-time generation.
- Factual depictions of real events/people (hallucination risk).
- Malicious deepfakes or harmful content.

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

<!-- Technical: Low resolution (32x32) results in pixelated outputs; limited to 8 frames with potential motion artifacts. No explicit safety mechanisms.

Sociotechnical: Biases from FineVideo dataset (Western-centric, underrepresents diversity). Risks include generating stereotypical or misleading content, amplifying societal biases.

Limitations: Moderate quality (FID ~118); early halting may under-refine complex prompts. -->

<!-- ### Recommendations -->

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
<!-- 
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

- Review outputs for biases/artifacts.
- Fine-tune on diverse data.
- Apply post-processing (e.g., upscaling).
- Use content filters for deployment.
 -->
<!-- 
## Training Details

### Training Data

This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering.

Trained on 50 samples from the [FineVideo Dataset](https://huggingface.co/datasets/HuggingFaceFV/finevideo) (train split). Videos: 8-frame clips resized to 32x32, normalized [-1,1]. Text: Descriptive captions tokenized to 77 tokens. No additional filtering; random 90/10 train/eval split.

### Training Procedure -->

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

<!-- Trained end-to-end with MSE loss on video reconstruction. 3 epochs, batch size 2, AdamW optimizer (lr=1e-4, wd=0.01), linear warmup scheduler.

#### Preprocessing [optional]

Videos: Uniform frame sampling, resize, normalize. Text: GPT-2 tokenization with padding/truncation.

#### Training Hyperparameters

- **Training regime:** fp16 mixed precision

#### Speeds, Sizes, Times [optional] -->

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

<!-- - Model size: ~20M parameters.
- Training time: ~2 min on T4 GPU (50 samples, 3 epochs).
- Checkpoint size: ~80 MB (safetensors).

## Evaluation
 -->
<!-- This section describes the evaluation protocols and provides the results. -->

<!-- ### Testing Data, Factors & Metrics

 #### Testing Data -->

<!-- This should link to a Dataset Card if possible. --> 

<!-- 10% held-out split from FineVideo (5 samples).

#### Factors -->

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

<!-- Prompt complexity (simple vs. descriptive).

#### Metrics -->

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

<!-- - FID: Fréchet Inception Distance (frame-wise; measures distribution similarity).
- MSE: Mean Squared Error (reconstruction quality). -->

<!-- ### Results

| Epoch | MSE (Train) | FID (Eval) |
|-------|-------------|------------|
| 1     | 0.4523     | 145.23    |
| 2     | 0.3871     | 132.45    |
| 3     | 0.3412     | 118.67    |
 -->
<!-- #### Summary

Best model at Epoch 3 (FID=118.67). Decreasing trends indicate learning; moderate quality due to small dataset.

## Model Examination [optional]
 -->
<!-- Relevant interpretability work for the model goes here -->

<!-- Halt predictor enables inspection of reasoning steps (e.g., avg 2.5 steps per generation). Attention maps visualizable for text-video alignment. -->

<!-- 
## Technical Specifications 
### Model Architecture and Objective

Transformer blocks (4 total: 2 per HRM level) with cross-attention. Objective: Minimize MSE between generated and target videos.

### Compute Infrastructure -->

<!-- [More Information Needed]

#### Hardware

NVIDIA T4 GPU.

#### Software

PyTorch 2.0+, Transformers, Datasets. -->

<!-- ## Citation [optional] -->

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**
```
@misc{ttv_hrm_2025,
  author = {Ahsan Umar, XCollab Team},
  title = {TTV-HRM: Text-to-Video Hierarchical Reasoning Model},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/XCollab/TTV-HRM}
}
```
<!-- 
**APA:**
XCollab Team. (2025). TTV-HRM: Text-to-Video Hierarchical Reasoning Model. Hugging Face. https://huggingface.co/XCollab/TTV-HRM --> -->