crasmarum/born-attention-lm

Born-Rule Complex-Valued Language Model (rope20k)

A character-level autoregressive language model whose attention replaces the usual softmax(QKᵀ/√d) with the Born rule from quantum mechanics: complex queries/keys and attention weights formed from normalized |⟨Q,K⟩|². Built and trained with CNet, a from-scratch complex-valued C++/CUDA framework (no PyTorch/TensorFlow).

• Paper: https://arxiv.org/abs/2610.08592
• Code: https://github.com/crasmarum/CNet
• Architecture: complex embedding → L × [per-token norm → Born attention (+ RoPE) → complex FFN] → per-token projection → Born-rule sequence cross-entropy.
• This checkpoint (rope20k.mod): width E = 160, context N = 64 tokens, L = 8 blocks, ~2.1M complex parameters; Born attention + RoPE + TokenNorm, trained ~20k steps on tiny-shakespeare (character level, vocab
  65. with the Born-loss probability floor ce_eps = 1e-3.
• Reported result: val 1.52 nats/char on tiny-shakespeare (see the paper for the Born-vs-softmax parity and the RoPE / TokenNorm ablations).

Format

rope20k.mod is a CNet checkpoint, not a PyTorch/safetensors file. It loads only through the CNet binary — transformers / from_pretrained will not read it.

Usage

Download the weights:

from huggingface_hub import hf_hub_download
path = hf_hub_download(repo_id="crasmarum/born-attention-lm",
                       filename="rope20k.mod")

Build CNet and generate (see the repo for build instructions):

# build
cd CNet && make dp_shakespeare        # or `make generate` for the CPU-only sampler

# sample from the trained model
./dp_shakespeare -generate true -model rope20k.mod -prompt "ROMEO:" -gen_len 600 -temp 0.8

# report the exact held-out Born loss on tiny-shakespeare
./dp_shakespeare -eval_loss true -model rope20k.mod -data tiny_shakespear2.txt

Method (one line)

For complex Q, K, attention weight a_ij ∝ |⟨q_i, k_j⟩|² (causal, row- normalized) — the Born probability p = |ψ|² applied to attention scores; the output distribution is likewise p_k = |z_k|² / Σ|z|².

Limitations

Research model: character-level, trained on tiny-shakespeare. Not instruction-tuned and not intended for production text generation.

Citation

@article{crasmaru2026cnet,
  title   = {CNet: A Complex-Valued Deep Learning Framework with Wirtinger
             Autodifferentiation and FFT–Hadamard Convolution},
  author  = {Crasmaru, Marcel},
  journal = {arXiv preprint arXiv:2610.08592},
  year    = {2026}
}