Spaces:
Running
on
CPU Upgrade
Running
on
CPU Upgrade
Fix RoPE comments: smaller k is faster oscillation
#5
by
6S-bobby
- opened
app/src/content/article.mdx
CHANGED
|
@@ -1146,15 +1146,15 @@ def apply_rope_simplified(x, pos, dim=64, base=10000):
|
|
| 1146 |
Formula:
|
| 1147 |
ΞΈ_{p,k} = p * base^(-k / (dim/2))
|
| 1148 |
|
| 1149 |
-
- Small k (early dimension pairs) β
|
| 1150 |
-
- Large k (later dimension pairs) β
|
| 1151 |
|
| 1152 |
"""
|
| 1153 |
rotated = []
|
| 1154 |
for i in range(0, dim, 2):
|
| 1155 |
k = i // 2 # index of this dimension pair
|
| 1156 |
|
| 1157 |
-
# Frequency term: higher k β
|
| 1158 |
inv_freq = 1.0 / (base ** (k / (dim // 2)))
|
| 1159 |
theta = pos * inv_freq # rotation angle for position p and pair k
|
| 1160 |
|
|
|
|
| 1146 |
Formula:
|
| 1147 |
ΞΈ_{p,k} = p * base^(-k / (dim/2))
|
| 1148 |
|
| 1149 |
+
- Small k (early dimension pairs) β fast oscillations β capture fine detail.
|
| 1150 |
+
- Large k (later dimension pairs) β slow oscillations β capture long-range info.
|
| 1151 |
|
| 1152 |
"""
|
| 1153 |
rotated = []
|
| 1154 |
for i in range(0, dim, 2):
|
| 1155 |
k = i // 2 # index of this dimension pair
|
| 1156 |
|
| 1157 |
+
# Frequency term: higher k β slower oscillation
|
| 1158 |
inv_freq = 1.0 / (base ** (k / (dim // 2)))
|
| 1159 |
theta = pos * inv_freq # rotation angle for position p and pair k
|
| 1160 |
|