Latest Long-Context Modeling Research Papers

The newest Long-Context Modeling papers from across the field — arXiv, NeurIPS, CVPR, Nature, and more — refreshed daily and ranked by relevance. Distill AI tracks Long-Context Modeling so you don’t have to: get the standout work delivered to your inbox every morning, with 2-sentence summaries and the option to chat with any paper.

Recent papers

• Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese
  Pedro Carvalho Brom · arXiv · Oct 29, 2026

  Building a language model of one's own no longer requires a data centre. This tutorial trains one from scratch, tokenizer included, on one consumer GPU, and shows how to tell whether it is good for the tasks of a specialist domain. It propo…

• Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models
  Dmytro Babarytskyi, Oleksandr Babarytskyi · arXiv · Oct 29, 2026

  Large Language Models (LLMs) rely on Multi-Head Attention (MHA) for sequence modeling, but scaling to long contexts is constrained by the quadratic cost of attention, motivating sub-quadratic recurrent alternatives. This paper formalizes a …

• Multi-State Linear Attention with Generalized Second-Order Routing
  Hee Seung Hwang, Sanghyuk Chun, Xindi Wu, Olga Russakovsky et al. · arXiv · Oct 27, 2026

  Self-attention plays a key role in long-sequence modeling, but its quadratic time complexity fundamentally limits the scaling of Large Language Models. Recently, Multi-State Linear Attention (MSLA) mechanisms have been proposed to make sequ…

• Zooming without Zooming v2: From Seeing Details to Connecting Evidence for Generalizable Multimodal Perception
  Lai Wei, Yatai Li, Liangbo He, Jun Lan et al. · arXiv · Oct 27, 2026

  Multimodal perception forms the foundation of multimodal intelligence. In particular, spatial perception and multimodal long-context understanding are two pivotal tasks that underpin embodied AI and multimodal agents. However, synthesizing …

• Why Full-Duplex Speech Models Collapse in Long Conversations
  Yeonju Kim, Bella Godiva, Se Jin Park, Yong Man Ro · arXiv · Oct 23, 2026

  Full-duplex speech models listen and speak at the same time, which makes them promising for natural, open-ended conversation. But can they sustain a coherent conversation for tens of minutes? Because they generate a token every frame, even …

• Head Dimension Matters in Language Models for Long-Context Retrieval
  Seiya Tokui, Shoichiro Yamaguchi, Sosuke Kobayashi · arXiv · Oct 23, 2026

  Multi-head attention retrieves and combines context representations through query--key interactions and value aggregation, with head dimension constraining the per-head representation space. In long-context processing, it remains unclear wh…

• HERD: Harness-Evolving Recursive Distillation for Continual Learning
  Xun Sun, Thanh Thi Nguyen, Thuy-Trang Vu · arXiv · Oct 22, 2026

  Continual learning requires a model to keep acquiring new skills and knowledge without compromising its existing capabilities. Parametric methods suffer from catastrophic forgetting and loss of plasticity. Non-parametric methods, such as pr…

• Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering
  Aman Behera, Namit Solanki, Mehul Anand · arXiv · Oct 22, 2026

  Long-context models often use answer-bearing evidence less effectively when it appears away from a prompt boundary. We compare this position bias across Transformer, state-space, and hybrid Mamba architectures by moving one gold document th…

• RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training
  Dor Tsur, Maor Ashkenazi, Maximilian Golub, Peter Hicks et al. · arXiv · Oct 22, 2026

  Speculative decoding is a popular decode acceleration method. As such, it is particularly relevant to long-output generation, which requires an increasingly large KV cache. KV cache compression offers a way to constrain costs in long-contex…

• CulturalRiddles: A Multicultural Benchmark of Riddles
  My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier et al. · arXiv · Oct 21, 2026

  As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factua…

• Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling Law
  Ziqing Qiao, Xiaoyue Xu, Sikui Zhang, Yinuo Xu et al. · arXiv · Oct 21, 2026

  Long-context continual pre-training (LCCP) is widely used to improve long-context capability, but how its benefits scale with the number of tokens used for LCCP remains unclear. We investigate this question through modeling, monitoring, and…

• Cache Engineering for Agents
  Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren et al. · arXiv · Oct 20, 2026

  Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can a…

• When Copying Is Hard: Copy-Constrained Decoding for Exact Span Reproduction
  Jinghui Zhang, Lang Gao, Zongfang Liu, Ruihong Zeng et al. · arXiv · Oct 20, 2026

  Precisely reproducing a target span verbatim within a long context is a fundamental capability test for LLMs: even when the target string is present in context, models often still drift within the span or fail to stop correctly, revealing l…

• KEEP-Bench: Evaluating Evidence Preservation in Context Compression
  Ziqing Man, Di Wu, Jia-Chen Gu, Junyi Zhang et al. · arXiv · Oct 19, 2026

  Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, s…

• TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
  Liana Patel, Rishabh Ranjan, Ion Stoica, Matei Zaharia et al. · arXiv · Oct 19, 2026

  Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters. On the other hand, language models process all input tokens through each model layer, resulting in signifi…

• When the harness decides success: state that cannot be re-observed
  Raimondas Lencevicius · arXiv · Oct 19, 2026

  Agent harnesses decide what a language model sees across turns: they drop old turns, summarize them, or replay the model's own earlier reasoning. We ask when these choices change whether an agent succeeds, not only what it costs, and report…

• Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference
  Zhenxu Tian, Kebin Liu, Zhengwu Yang, Yi Su et al. · arXiv · Oct 16, 2026

  Long-context decoding in large language models (LLMs) is increasingly bottlenecked by the memory and computation required to attend over an ever-growing Key-Value (KV) cache at each autoregressive step. Although token-level sparse attention…

• Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries
  Anikait Singh, Teresa Zhang, Yoonho Lee, Roshen Sanjay Nair et al. · arXiv · Oct 15, 2026

  Reasoning over long document contexts remains a challenge for large language models (LLMs). Standard retrieval methods are inherently reactive, bottlenecking agents with complex, cross-document reasoning at inference time, while precomputed…

• Hy-Memory-30B-A3B: Attributed Reinforcement Learning for Long-Term Memory Writing
  Tianxiang Fei, Jiazheng Kang, Mao Zheng, Xingyan He et al. · arXiv · Oct 15, 2026

  External-memory systems extend language-model applications beyond a context window, but their quality depends critically on the model executing the write path: deciding what to retain, how precisely to represent it, and how new evidence sho…

• Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning
  Shiwen Qin, Cameron Barker, Henry Gouk, Elliot J. Crowley et al. · arXiv · Oct 15, 2026

  Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that re…

• Latent Compaction with Test-Time Training
  Yingfa Chen, Xingyu Shen, Chaojun Xiao, Xu Han · arXiv · Oct 14, 2026

  Long contexts enlarge key--value (KV) caches, increasing memory use and attention costs during generation. Compressing these caches must preserve information for unseen questions. We introduce **Latent Compaction**, which learns a compact d…

• ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models
  Rusali Saha, Nitya Khurana, Ruihong Huang, Kuan-Hao Huang · arXiv · Oct 14, 2026

  Cultural attitudes and social norms evolve over time, yet most cultural benchmarks for large language models (LLMs) treat culture as a static attribute of a country or region. We introduce ChronoCulture, a benchmark of 10,360 temporal pairs…

• FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding
  Shuxuan Li, Jialiang Kang, Yanghao Zhou, Ziye Zhu et al. · arXiv · Oct 14, 2026

  Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs, where KV-cache access costs can be amortized over multiple accepted tokens. Yet most draft…

• MemDuet: Learning Dual-Process Memory for Long-Context Reasoning
  Jiangcheng Song, Haodong Jing, Yulin Mu, Lesong Tao et al. · arXiv · Oct 14, 2026

  Reasoning over infinitely long documents is becoming a reality with recurrent memory agents. However, increasing context length exacerbates the trade-off between reasoning performance and inference efficiency. Inspired by Kahneman's account…

• Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy
  Naji Alazhar, Wassel Alazhar · arXiv · Oct 9, 2026

  Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not handle draft…

• Taking Tokenization off the Host for Agentic LLM Serving
  Mohammad Siavashi, Pascal Schwinte, Ali Banaei, Gerald Q. Maguire et al. · arXiv · Oct 9, 2026

  Agentic workloads resend long contexts that barely change between calls. Prefix caching now skips most of their prefill, yet every request must still be tokenized in full before the cache can even be consulted. Tokenization, a CPU stage lon…

• VFold: Symmetry-Aware Cross-Layer Value Cache Compression
  Neha Verma, Sungwon Kim, Kenton Murray, Kevin Duh · arXiv · Oct 8, 2026

  While caching key-value (KV) states accelerates Large Language Model (LLM) decoding, this cache can dominate memory usage at long context lengths. One solution is to compress this memory by exploiting inter-layer cache similarities. However…

• Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog
  Hadeel Al-Negheimish, Jasna Ilieva, Yoon Kim · arXiv · Oct 8, 2026

  Current frontier LLMs can theoretically process long contexts with 1M tokens or more. But to what extent can they go beyond simple retrieval and perform deeper reasoning over such long contexts? We empirically investigate long-horizon reaso…

• Decoupling Exploration from Optimization in RLVR
  Saif Punjwani, Micah Goldblum · arXiv · Oct 7, 2026

  Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas abs…

• YANchor-4B: Effective Long-Horizon Reasoning in O(N) Time with O(1) Memory
  Huishan Ji, Hua Xu, Weiming Zhang, Qirui Ye · arXiv · Oct 7, 2026

  Long-horizon reasoning demands access to earlier information at a manageable generation cost. Full-history attention incurs growing storage and computation, while recurrent compression can lose precise details. Therefore, we present YANchor…