You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

HTDN (Hybrid Transformer Dynamics Network) - 399M

HTDN is a custom-built, 399-parameter language model developed from scratch by AutoCode-Ai. It introduces a novel architecture combining a continuous Dynamics Layer with attractors and standard Transformer components, fully implemented in JAX/Flax.

📌 Model Overview

Developer: AutoCode-Ai
Parameters: ~399M
Framework: JAX / Flax
Tokenizer: bert-base-multilingual-cased
Hardware Used for Training: TPU v5e-8
Languages: Multilingual (with strong capabilities in Arabic text generation)
Format: Orbax Checkpoints

🏗️ Architecture Details

Unlike standard transformer-only models, HTDN utilizes a hybrid block structure to manage continuous dynamics and memory alongside causal attention:

Dynamics Layer: Features u and v states with 22,000 attractors and trainable continuous time-step (dt) dynamics, processing temporal sequences efficiently.
Causal Self-Attention: 8 attention heads with Rotary Position Embeddings (RoPE) for robust context modeling.
SwiGLU FFN: A highly optimized Feed-Forward Network utilizing the Swish-Gated Linear Unit for better activation routing.

Hyperparameters

Parameter	Value
`d_model`	1024
`d_u` & `d_v`	512
`hidden_react`	3072
`n_attractors`	22,000
`num_heads`	8
`num_layers`	4
`max_len`	64

💻 How to Load and Use

Because this model uses a custom JAX/Flax architecture and is saved using Orbax Checkpoints, you cannot load it directly using the standard transformers AutoModelForCausalLM. Below is a guide on how to instantiate the model and restore the weights.

1. Install Dependencies

pip install jax flax orbax-checkpoint transformers

Downloads last month: -; Downloads are not tracked for this model. How to track