jesusvilela commited on
Commit
84caa07
·
verified ·
1 Parent(s): d9cfb27

Add paper.md and paper.bib for citation

Browse files
Files changed (2) hide show
  1. paper.bib +7 -0
  2. paper.md +42 -0
paper.bib ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ @misc{vilela2026manifoldgl,
2
+ title={ManifoldGL: Information-Geometric Adapter for Large Language Models},
3
+ author={Jesús Vilela Jato},
4
+ year={2026},
5
+ howpublished={\url{https://github.com/jesusvilela/IGBundle-LLM}},
6
+ note={Software and manuscript available at \url{https://github.com/jesusvilela/IGBundle-LLM}}
7
+ }
paper.md ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ManifoldGL – Information‑Geometric Adapter for LLMs
2
+
3
+ ManifoldGL is a parameter‑efficient adapter that enforces **hyperbolic geometry** on the latent space of large language models. It treats the meaning of a token as a **fiber** over a hyperbolic base manifold (a Poincaré ball), rather than a single vector in flat Euclidean space. Latent states are projected onto the ball, and attentions are computed using geodesic distance. A sheaf‑theoretic consistency loss and natural gradient optimization maintain semantic structure during training.
4
+
5
+ ## Motivation and theoretical background
6
+
7
+ Modern LLMs embed tokens in a Euclidean vector space. While convenient, Euclidean geometry has limited capacity to represent hierarchical structures: flat space grows polynomially, whereas hierarchical trees expand exponentially. By contrast, **hyperbolic space** grows exponentially and preserves both local and global relationships in a hierarchy【247949143190903†L115-L124】. Hyperbolic embeddings outperform Euclidean ones for lexical entailment, similarity and analogy tasks【247949143190903†L154-L169】. ManifoldGL leverages these properties by modelling the latent space as a fiber bundle over a hyperbolic base: each point in the Poincaré ball encodes a context, and its fiber contains a distribution of semantic components.
8
+
9
+ ## Results on ARC‑AGI benchmark
10
+
11
+ ManifoldGL fine‑tuned on Qwen2.5‑7B improves task accuracy on the ARC‑AGI benchmark from **12.4 %** to **28.7 %**, a **131.5 % relative improvement**. The model also achieves a **Manifold Faithfulness Rate (MFR) of 94.2 %**, indicating high adherence to the hyperbolic constraints, and maintains a curvature close to the target κ = ‑1 (mean ‑0.98 ± 0.04). Ablation studies show that removing curvature regularization, natural gradients, sheaf consistency or the hyperbolic target significantly reduces accuracy; the Euclidean target ablation causes the largest drop (–10.9 %), highlighting the importance of hyperbolic geometry.
12
+
13
+ ## Files in this repository
14
+
15
+ This model card accompanies adapter weights trained with ManifoldGL. The files follow the structure of the original repository:
16
+
17
+ - `adapter_config.json` – configuration for PEFT/LoRA loading
18
+ - `pytorch_adapter.bin` – adapter weights
19
+ - `README.md` – this model card
20
+
21
+ ## Quick start
22
+
23
+ ```python
24
+ from transformers import AutoModelForCausalLM
25
+ from peft import PeftModel
26
+
27
+ # Load the base model (Qwen2.5-7B)
28
+ base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B")
29
+
30
+ # Load the ManifoldGL adapter
31
+ model = PeftModel.from_pretrained(base_model, "jesusvilela/manifoldgl")
32
+
33
+ # Now use model.generate(...) to generate text with hyperbolic adapters
34
+ ```
35
+
36
+ ## Usage
37
+
38
+ This adapter can be loaded with [PEFT](https://github.com/huggingface/peft) on top of any compatible Qwen2.5‑7B model. During generation, latent states are projected into hyperbolic space and meaning is represented as fibers. We recommend using FP32 precision for maximum stability.
39
+
40
+ ## Citation
41
+
42
+ If you use ManifoldGL in your work, please cite the accompanying thesis and repository.