Abstract
Position encoding is a foundational design choice in transformer architectures, enabling models to exploit token order without recurrence. Rotary Position Embedding (RoPE), introduced by Su et al. (2021), represents a significant departure from additive absolute and relative position schemes. Rather than augmenting token representations with fixed or learned position vectors, RoPE encodes position information by rotating query and key vectors in a complex-valued embedding space. This formulation achieves relative position sensitivity through an elegant mathematical identity: the inner product of two rotated vectors depends only on their position difference. In this paper, we examine the theoretical foundations of RoPE, derive its key properties from first principles, analyze its relationship to other position encoding families, and survey the empirical evidence for its effectiveness across large-scale language models including LLaMA, PaLM 2, and Gemini. We further discuss extensions to long-context generalization via interpolation and extrapolation schemes, including NTK-aware scaling and YaRN, and close with open problems in position encoding research.
1. Introduction
The transformer architecture (Vaswani et al., 2017) processes tokens in parallel, abandoning the sequential inductive bias of recurrent networks. This parallelism comes at a cost: the model has no innate sense of order. Without explicit position information, a permutation of the input sequence would yield identical attention scores. The original transformer addressed this by adding sinusoidal positional encodings to token embeddings before the first layer—a fixed, non-learned encoding defined by sine and cosine functions of varying frequencies.
While sinusoidal encodings and their learned variants have served well, they carry an important limitation: they inject position information additively, at a single point in the pipeline. The attention mechanism, which operates on queries and keys derived from these embeddings, must then implicitly disentangle content and position information through learning. This creates a form of representational entanglement that can impede generalization, particularly when context lengths at inference time differ from those seen during training.
Relative position encodings (Shaw et al., 2018; Raffel et al., 2020; Press et al., 2022) address part of this problem by modifying attention logits to depend on the relative position between query and key positions rather than their absolute indices. These methods are generally more robust to distributional shift in sequence length. However, many such schemes introduce significant implementation complexity, require modifications to the attention computation graph, or add learnable parameters that must be tuned.
Rotary Position Embedding (RoPE), proposed by Su et al. (2021), offers a compelling synthesis. It achieves relative position sensitivity by rotating query and key vectors by position-dependent angles in a carefully structured embedding subspace. The rotation is applied to pairs of embedding dimensions using rotation matrices parameterized by the token’s absolute position and a set of per-dimension base frequencies. The key insight is that when two rotated vectors are dotted together in the attention score computation, the absolute position components cancel, leaving only the relative position difference. RoPE thus achieves the desiderata of relative encoding with the implementation simplicity of an absolute scheme.
The adoption of RoPE in production-scale models has been rapid. LLaMA (Touvron et al., 2023a), LLaMA 2 (Touvron et al., 2023b), Mistral (Jiang et al., 2023), PaLM 2 (Anil et al., 2023), and Gemini (Team et al., 2023) all employ RoPE or closely related rotary schemes. Understanding its theoretical properties, empirical behavior, and failure modes is therefore practically important for anyone working with modern language model infrastructure.
This paper proceeds as follows. Section 2 reviews the landscape of position encoding methods and situates RoPE within it. Section 3 derives the RoPE formulation, establishes its core properties, and analyzes the geometry of rotary embeddings. Section 4 examines context-length generalization, surveying interpolation and extrapolation techniques. Section 5 discusses open problems and directions for future research. Section 6 concludes.
2. Related Work
Position encoding in transformers has evolved through several generations of increasingly sophisticated approaches. We briefly survey the most relevant prior work.
Absolute sinusoidal encodings. Vaswani et al. (2017) introduced fixed sinusoidal position encodings of the form $PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d})$ and $PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d})$, where $pos$ is the token index and $i$ indexes embedding dimensions. These encodings were added to token embeddings before the first layer. Their geometric interpretation is that each position maps to a unique point on a high-dimensional torus, with the frequencies chosen so that different dimensions capture structure at different scales. While effective for shorter sequences, these encodings do not generalize well beyond training lengths.
Learned absolute encodings. Devlin et al. (2019) replaced sinusoidal encodings with learned embedding vectors in BERT. This approach performs comparably in-distribution but often degrades more severely on out-of-distribution lengths since the model learns nothing about positions not encountered during training.
Relative position encodings (Shaw et al., 2018). Shaw et al. proposed modifying attention logits by adding learned relative position bias terms $a_{ij}^K$ and $a_{ij}^V$ that depend on the clipped relative distance $\text{clip}(i – j, -k, k)$. This approach explicitly models pairwise token distances, improving generalization, but requires storing and indexing a set of relative position matrices.
T5 relative biases (Raffel et al., 2020). The T5 model introduced a simpler relative bias scheme that adds a scalar bias to attention logits based on bucketed relative position. Buckets use a logarithmic spacing for larger distances. This approach has become popular due to its simplicity but still requires additional learnable parameters.
ALiBi (Press et al., 2022). Attention with Linear Biases subtracts a fixed penalty proportional to query-key distance from attention logits. ALiBi requires no trainable parameters and exhibits strong length generalization in practice, though its linearity imposes a particular inductive bias that may not suit all tasks.
Kerple (Chi et al., 2022). Chi et al. formalized a class of relative position encodings based on positive-definite kernel functions applied to position differences, providing a principled framework for analyzing generalization properties. RoPE can be situated within this framework as a kernel with specific frequency-based parameterization.
RoPE (Su et al., 2021). The work we analyze in depth. RoPE applies position-dependent rotations to queries and keys, achieving relative encoding through geometric structure. Its adoption in LLaMA (Touvron et al., 2023a) and subsequent models catalyzed widespread interest.
3. Technical Analysis
3.1 Formulation
Let $\mathbf{q}_m, \mathbf{k}_n \in \mathbb{R}^d$ denote query and key vectors at positions $m$ and $n$ respectively, where $d$ is the head dimension. The attention logit is $a_{mn} = \mathbf{q}_m^\top \mathbf{k}_n / \sqrt{d}$.
RoPE transforms $\mathbf{q}_m$ and $\mathbf{k}_n$ by position-dependent rotation matrices before computing their inner product. Specifically, for a $d$-dimensional vector $\mathbf{x}$ at position $p$, the rotary transformation is defined by partitioning dimensions into $d/2$ pairs $(x_{2i-1}, x_{2i})$ and rotating each pair by angle $p \cdot \theta_i$:
$$f(\mathbf{x}, p)_i = \begin{pmatrix} x_{2i-1} \cos(p\theta_i) – x_{2i} \sin(p\theta_i) \\ x_{2i-1} \sin(p\theta_i) + x_{2i} \cos(p\theta_i) \end{pmatrix}$$
where the base frequencies are $\theta_i = b^{-2(i-1)/d}$ for base $b = 10000$ (following the sinusoidal convention). In matrix form, the full transformation can be written as:
$$f(\mathbf{x}, p) = \mathbf{R}_p \mathbf{x}$$
where $\mathbf{R}_p$ is a block-diagonal orthogonal matrix with $2 \times 2$ rotation blocks $\begin{pmatrix} \cos(p\theta_i) & -\sin(p\theta_i) \\ \sin(p\theta_i) & \cos(p\theta_i) \end{pmatrix}$ along the diagonal.
3.2 The Relative Position Property
The central theoretical property of RoPE is that the inner product of rotated query and key vectors depends only on their position difference. Formally:
$$\langle f(\mathbf{q}, m), f(\mathbf{k}, n) \rangle = \langle \mathbf{R}_m \mathbf{q}, \mathbf{R}_n \mathbf{k} \rangle = \mathbf{q}^\top \mathbf{R}_m^\top \mathbf{R}_n \mathbf{k} = \mathbf{q}^\top \mathbf{R}_{n-m} \mathbf{k}$$
The last equality follows from the multiplicative structure of rotation matrices: $\mathbf{R}_m^\top \mathbf{R}_n = \mathbf{R}_{n-m}$ since rotation matrices form a group under multiplication and $\mathbf{R}_p^\top = \mathbf{R}_{-p}$. Thus, the attention score depends on the content vectors $\mathbf{q}$, $\mathbf{k}$ and the relative offset $n – m$, but not on $m$ or $n$ individually. This is exactly the property we want from a relative position encoding, derived here without any modification to the attention computation beyond the pre-rotation of queries and keys.
3.3 Complex Number Interpretation
RoPE admits an elegant complex number interpretation that makes its geometry transparent. Represent each 2D subspace of the embedding as a complex number: $x_{2i-1} + i \cdot x_{2i} \in \mathbb{C}$. Then the rotary transformation is simply multiplication by the unit complex number $e^{i p \theta_i}$:
$$f(\mathbf{x}, p)_i = (x_{2i-1} + i x_{2i}) \cdot e^{i p \theta_i}$$
The inner product in $\mathbb{R}^2$ corresponds to the real part of the complex inner product, so:
$$\text{Re}\left[ \overline{f(\mathbf{q}, m)_i} \cdot f(\mathbf{k}, n)_i \right] = \text{Re}\left[ \overline{q_i} \cdot k_i \cdot e^{i(n-m)\theta_i} \right]$$
This form makes explicit that the contribution of each frequency pair to the attention score is a cosine of the relative position modulated by the magnitude and phase of the query-key interaction in that frequency band. Higher-indexed dimensions ($i$ near $d/2$) have smaller $\theta_i$ (lower frequency), acting as slow-varying position signals; lower-indexed dimensions have higher frequency, providing fine-grained local resolution. This multi-scale structure is directly analogous to the sinusoidal encoding’s frequency hierarchy.
3.4 Orthogonality and Norm Preservation
Since $\mathbf{R}_p$ is an orthogonal matrix, $\|f(\mathbf{x}, p)\|_2 = \|\mathbf{x}\|_2$. This is a desirable property: position encoding does not distort the magnitude of representations, only their direction in the embedding space. In contrast, additive encodings change both direction and magnitude, potentially interfering with learned feature norms. The norm-preserving property of RoPE is relevant for training stability, particularly in models that rely on careful initialization or normalization schemes.
3.5 Decay of Attention with Distance
An important empirical property of RoPE is that attention scores between distant tokens tend to decay on average, which provides a useful inductive bias for local coherence. This can be understood theoretically by observing that for random query and key vectors, the expected value of $\mathbf{q}^\top \mathbf{R}_{n-m} \mathbf{k}$ decreases with $|n – m|$ as the rotation phases become less aligned across frequency pairs. Su et al. (2021) showed that under reasonable assumptions about query-key statistics, the expected inner product magnitude $\mathbb{E}[|\langle f(\mathbf{q},m), f(\mathbf{k},n) \rangle|]$ is a decreasing function of $|n-m|$. This implicit decay is a key advantage over ALiBi’s explicit linear penalty, as it allows the model to learn both local and global attention patterns while biasing toward locality by default.
3.6 Context Length Generalization
Despite its theoretical elegance, RoPE exhibits a well-known failure mode: performance degrades sharply when the context length at inference exceeds the maximum length seen during training ($L_{\text{train}}$). The intuition is straightforward—positions beyond $L_{\text{train}}$ correspond to rotation angles that the model has never optimized against, and the resulting key-query interactions fall outside the distribution of training angles.
Chen et al. (2023) proposed position interpolation (PI) as a remedy: rather than assigning position $p$ angle $p \theta_i$, rescale positions by a factor $s = L_{\text{train}} / L_{\text{target}}$, so position $p$ receives angle $(p / s) \theta_i$. This maps any inference position into the training range. With a small amount of fine-tuning (as few as 1000 gradient steps on long-context data), LLaMA extended from 2k to 32k tokens with minimal quality degradation. The trade-off is that interpolation compresses relative position signals—tokens at positions 0 and 1 become nearly indistinguishable in some frequency bands—which can impair short-range resolution.
NTK-aware interpolation (bloc97, 2023) addresses this by modifying the base frequency rather than rescaling positions uniformly. The NTK (Neural Tangent Kernel) perspective suggests that the high-frequency dimensions should not be compressed, since they carry most of the fine-grained positional information. Replacing the base $b = 10000$ with a scaled $b’ = b \cdot (L_{\text{target}} / L_{\text{train}})^{d/(d-2)}$ achieves an approximately uniform distribution of position signal across frequency bands at the new context length.
YaRN (Peng et al., 2023) refines this further with a per-frequency interpolation scheme that applies different scaling factors to different frequency bands and introduces a temperature parameter to adjust attention entropy. YaRN achieves state-of-the-art context extension without fine-tuning and can extend LLaMA 2 from 4k to 128k tokens with targeted fine-tuning.
4. Discussion
The rapid adoption of RoPE across diverse production models raises several interesting questions about what properties of position encoding matter most for downstream performance.
Why does relative encoding matter at scale? Empirically, models with relative position encodings tend to be more robust to test-time length shifts than those with absolute encodings. The theoretical argument—that attention scores depending only on relative position generalize better because the function to be learned is invariant to absolute position shifts—is plausible but not fully proven. A confounding factor is that most large models trained with RoPE are also trained on longer contexts than their absolute-encoding predecessors, making it difficult to isolate the encoding choice.
Base frequency and the effective context window. The choice of base $b = 10000$ implies that the lowest-frequency dimension ($i = d/2$) completes one full rotation every $2\pi \cdot 10000^{(d-2)/d} \approx 10000$ positions. For models trained on sequences longer than this, some frequency pairs may aliase—the same effective angle appears at multiple positions—which can confuse the model. This has motivated the use of larger bases ($b = 500000$ in LLaMA 3) for models trained on long contexts. The relationship between base frequency, training length, and effective context remains an area of active investigation.
Interaction with attention sinks. Xiao et al. (2023) observed that large language models with absolute position encodings tend to develop attention sinks—a small number of initial tokens receive disproportionately large attention weights regardless of content. This phenomenon is partly a consequence of softmax’s requirement to sum to one: models learn to route excess attention weight to stable anchor tokens. RoPE models also exhibit attention sinks, but their geometry differs. Understanding the interaction between rotary position encoding and attention sink behavior is relevant for efficient inference techniques like StreamingLLM (Xiao et al., 2023).
Multi-dimensional extensions. RoPE was designed for 1D sequences. Extensions to 2D and 3D spatial data (images, video, point clouds) require new designs. Su et al. (2021) briefly discussed 2D generalizations; more systematic treatments have appeared in the context of vision transformers and multi-modal models. The question of how to assign rotation angles to multi-dimensional positions while preserving the relative position property remains an open design problem.
Learned vs. fixed frequencies. The standard RoPE uses fixed base frequencies $\theta_i = b^{-2(i-1)/d}$. One could imagine learning these frequencies jointly with the model, allowing each attention head to specialize its positional sensitivity. Preliminary experiments (Su et al., 2021; Ding et al., 2023) suggest modest gains from frequency learning, but the practical benefits appear small compared to the added complexity, which may explain why fixed frequencies remain the dominant choice.
RoPE and sparse attention. Long-context models often combine RoPE with sparse attention patterns (local windows, strided global tokens). The interaction is non-trivial: sparse attention patterns that are effective with absolute encodings may not be optimal with RoPE, since the implicit locality bias from rotary attention decay interacts with the explicit sparsity pattern. Optimal sparse pattern design for RoPE models remains underexplored.
5. Conclusion
Rotary Position Embedding represents a significant advance in position encoding design for transformer language models. By encoding position as a rotation in a complex-valued embedding space, RoPE achieves relative position sensitivity through an elegant mathematical identity without requiring modifications to the attention computation beyond a pre-rotation of queries and keys. Its norm-preserving property, implicit distance decay, and implementation simplicity have driven adoption across the most capable open and closed language models available today.
We have derived RoPE’s core properties from first principles, examined its complex-valued geometry, and analyzed methods for extending its effective context window beyond training length. The interpolation and NTK-aware scaling approaches demonstrate that RoPE’s frequency structure can be exploited to achieve controlled context extension, though fundamental questions remain about the relationship between base frequency, training length, and generalization.
Open problems include: the optimal base frequency for long-context training, principled methods for multi-dimensional rotary encoding, the interaction between rotary attention and sparse patterns, and a deeper theoretical account of why relative position encoding benefits generalization. As language models continue to scale both in parameter count and context length, position encoding will remain a foundational design variable whose properties deserve rigorous theoretical and empirical investigation.
References
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
- Shaw, P., Uszkoreit, J., & Vaswani, A. (2018). Self-attention with relative position representations. Proceedings of NAACL-HLT 2018.
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT 2019.
- Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1), 5485–5551.
- Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864.
- Press, O., Smith, N. A., & Lewis, M. (2022). Train short, test long: Attention with linear biases enables input length extrapolation. Proceedings of ICLR 2022.
- Chi, T.-A., Fan, T.-H., Ramadge, P. J., & Rudnicky, A. I. (2022). KERPLE: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35.
- Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi�re, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023a). LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
- Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023b). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
- Chen, S., Wong, S., Chen, L., & Tian, Y. (2023). Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595.
- Peng, B., Quesnelle, J., Fan, H., & Shinn, E. (2023). YaRN: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071.
- Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2023). Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453.
- Anil, R., Chowdhery, A., Roberts, A., Askell, A., Aakanksha, C., et al. (2023). PaLM 2 technical report. arXiv preprint arXiv:2305.10403.
- Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023). Mistral 7B. arXiv preprint arXiv:2310.06825.