1990two/liquid_state_space

Liquid State Space Model - The Classics Revival

Continuous-Time Adaptive Sequence Processing with Learned Dynamics

Experimental Research Code - Functional but unoptimized, expect rough edges

What Is This?

Liquid State Space Model enhances traditional state space models with liquid neural network dynamics and adaptive time constants. The system learns content-dependent time evolution, making it naturally adaptive to different sequence characteristics and potentially more efficient than transformers for long sequences.

Core Innovation: Time constants and state dynamics adapt based on input content, creating a continuous-time sequence processor that adjusts its temporal behavior to match data requirements.

Architecture Highlights

• Adaptive Time Constants: Learn content-dependent evolution speeds
• Continuous-Time Dynamics: Proper differential equation integration
• HiPPO Initialization: Theoretically grounded memory representation
• Liquid Evolution: Neural ODEs for state transitions
• Efficient Long Sequences: O(L) complexity vs O(L²) attention
• Language Model Ready: Drop-in transformer replacement

Quick Start

from liquid_state_space import LiquidSSMLanguageModel

# Create liquid SSM language model
model = LiquidSSMLanguageModel(
    vocab_size=32000,
    d_model=512,
    state_dim=256,
    num_layers=6,
    max_seq_len=2048
)

# Process sequences
input_ids = torch.randint(0, 32000, (batch_size, seq_len))
outputs = model(input_ids, labels=target_ids)

# Generate text
generated = model.generate(
    input_ids[:1],
    max_length=100,
    temperature=1.0
)

Current Status

• Working: Adaptive time constants, continuous dynamics, HiPPO matrices, language modeling, text generation
• Rough Edges: No optimization for very long sequences (>4k), numerical stability could be improved
• Still Missing: Distributed training, advanced initialization schemes, memory compression
• Performance: Competitive with small transformers, needs scaling validation
• Memory Usage: Lower than transformers for long sequences, higher for short ones
• Speed: Good sequential processing, benefits from specialized ODE solvers

Mathematical Foundation

The core state space model follows:

dx/dt = A(t,x)·x + B·u
y = C·x + D·u

With adaptive time constants:

τ(x,u) = base_τ × (1 + η·MLP([x;u]))
effective_dt = min(target_dt, min(τ)/10)

HiPPO matrices initialize A for optimal memory:

A_ij = √(2i+1)√(2j+1)  if i > j
A_ii = -(2i+1)

Liquid evolution uses:

dx/dt = -x/τ + A·x + B·u + noise·exploration_rate

Research Applications

• Long-range sequence modeling
• Time series prediction with adaptive dynamics
• Scientific computing with learned ODEs
• Efficient transformer alternatives
• Continuous-time natural language processing

Installation

pip install torch numpy scipy
# Download liquid_state_space.py from this repo

The Classics Revival Collection

Liquid State Space Model is part of a larger exploration of foundational algorithms enhanced with modern neural techniques:

• Evolutionary Turing Machine
• Hebbian Bloom Filter
• Hopfield Decision Graph
• Liquid Bayes Chain
• Liquid State Space Model ← You are here
• Möbius Markov Chain
• Memory Forest

Citation

@misc{liquidssm2025,
  title={Liquid State Space Model: Continuous-Time Adaptive Sequence Processing},
  author={Jae Parker 𓅸 1990two},
  year={2025},
  note={Part of The Classics Revival Collection}
}