Latest Multimodal Learning Research Papers

The newest Multimodal Learning papers from across the field — arXiv, NeurIPS, CVPR, Nature, and more — refreshed daily and ranked by relevance. Distill AI tracks Multimodal Learning so you don’t have to: get the standout work delivered to your inbox every morning, with 2-sentence summaries and the option to chat with any paper.

Recent papers

• Conserved Multimodal Nonlinear Dynamics in Brain Activity and Structure
  Calvin Grant · Zenodo (CERN European Organ... · Dec 5, 2026

  SUMMARY The brain has been generating the same rhythms — delta, theta, alpha, beta, gamma — in every subject ever recorded. The boundaries between them are reproducible to within a hertz across species, ages, anesthesia, wakefulness, epilep…

• Conserved Multimodal Nonlinear Dynamics in Brain Activity and Structure
  Calvin Grant · Zenodo (CERN European Organ... · Dec 5, 2026

  SUMMARY The brain has been generating the same rhythms — delta, theta, alpha, beta, gamma — in every subject ever recorded. The boundaries between them are reproducible to within a hertz across species, ages, anesthesia, wakefulness, epilep…

• Towards Early and Accurate Disease Detection Through Multimodal Predictive Modeling: Fusion of Electronic Health Records, Medical Imaging, And Omics Data Using Interpretable Machine Learning.
  Muhammad Ahsan Hayat, Jahangir Baig, Shayan Ahmed, Ahmed Faraz Ayubi · Policy Research Journal · Nov 3, 2026

  Early detection of disease is a cornerstone for improving patient outcomes, reducing costs, and enabling preventative interventions. Traditional predictive models often rely on a single type of data (e.g., imaging, clinical labs, or genomic…

• VeriCUA: Verification as Agentic Exploration for Computer-Using Agents
  Qiushi Sun, Da Yu, Alexander Knop, Amer Sinha et al. · arXiv · Oct 29, 2026

  Building and scaling computer-using agents (CUAs) across desktop, mobile, and web environments requires reliable task-completion verification for data curation, evaluation, and reinforcement learning. Typical automated approaches prompt vis…

• PhysCount: Do MLLMs Count Visual Appearances or Physical Entities?
  Lidong Lu, Jiaxin Lv, Liangrui Dong, Xinyi Mao et al. · arXiv · Oct 29, 2026

  Images reveal appearances, but counting often asks about physical objects. A single entity may be visible directly, in several reflections, or through glass, so detecting every visible occurrence does not determine how many entities are pre…

• CrossWeave: Cross-Modal Scene and Instance Retrieval from Sparse 2D-3D Alignment
  Aadith Warrier, Gnana Prakash Punnavajhala, Siddharth Tourani, Muhammad Haris Khan et al. · arXiv · Oct 29, 2026

  Cross-modal scene and instance retrieval aims to retrieve a corresponding 3D environment or object given query 2D images, serving as a foundational mechanism for spatial reasoning in robotics and AR/VR. Existing methods achieve this by lear…

• A Unified Benchmark for Aerial-Enhanced Autonomous Driving: The KITScenes Aerial Dataset
  Daniel Lengerer, Richard Schwarzkopf, Jonas Merkert, Fabian Immel et al. · arXiv · Oct 29, 2026

  Aerial imagery provides complementary spatial context for autonomous driving by exposing road structure beyond the immediate perception range of onboard sensors. Yet map construction from aerial imagery, aerial-assisted online HD mapping, a…

• HiFOR: Hierarchical Foresight with Action-Guided World Grounding for Vision-Language-Action Model
  Rongxing Ding, Hongyu Qu, Kehong Wu, Xudong Wang et al. · arXiv · Oct 29, 2026

  Vision-Language-Action (VLA) models have demonstrated promising capabilities in robotic manipulation by integrating visual perception, language understanding, and action generation. Recent advances have further incorporated future predictio…

• ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning
  Chaehyun Kim, Chi-Pin Huang, Jisoo Kim, Min-Hung Chen et al. · arXiv · Oct 27, 2026

  Vision-Language-Action (VLA) models have made substantial progress in robotic manipulation, but their generalization remains constrained by the diversity and coverage of action-labeled robot demonstrations. Human videos provide much broader…

• Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?
  Jialv Zou, Xiaotong Li, Zhengrong Yue, Xu Huang et al. · arXiv · Oct 27, 2026

  Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause una…

• VideoLVR: Latent Visual Reasoning over Grounded Video Evidence
  Shilong Li, Xingyuan Bu, Hao Shi, Yun Xing et al. · arXiv · Oct 27, 2026

  Chain-of-thought reasoning has substantially advanced video understanding in multimodal large language models (MLLMs). However, language-based reasoning traces do not provide a reliable mechanism for models to revisit relevant visual eviden…

• Zero-Shot Multimodal Event Spotting in Soccer Broadcasts
  Sadia Kamal, Mahmudul Hasan, Tony Braskich · arXiv · Oct 27, 2026

  Automatic soccer event spotting supports highlight generation, video indexing, and sports analytics, but existing methods depend on large annotated datasets and soccer-specific training. We present a zero-shot framework that detects, classi…

• PerceptRL: Making Visual Perception Rewardable in MLLMs from Answers to Interactions
  Tongkun Guan, Zhibo Yang, Mingkun Yang, Haocheng Wang et al. · arXiv · Oct 27, 2026

  Fine-grained visual perception remains a bottleneck during post-training of multimodal large language models (MLLMs), as perceptual errors can constrain gains from reasoning-focused optimization. Although image-caption pre-training improves…

• When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models
  Ananth K Suresh, Aveen Dayal, Gouthaman KV, Aditya Pola et al. · arXiv · Oct 27, 2026

  For reliable deployment, multimodal large language models (MLLMs) should recognize when an image provides insufficient evidence to answer a question and withhold unsupported answers. Yet refusal alone is not necessarily evidence of such rec…

• PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations
  Hanqing Wang, Zhaotian Cai, Ruichuan An, Bohan Zeng et al. · arXiv · Oct 27, 2026

  Vision--Language--Action (VLA) models generate robot actions from visual observations and language instructions, but their understanding of physical dynamics remains limited. Video world models can capture scene dynamics, yet generating fut…

• EmbeddingGemma 2: Unified and Efficient Multimodal Representations
  Henrique Schechter Vera, Sahil Dua, Jovan Andonov, Sara Smoot et al. · arXiv · Oct 27, 2026

  We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings …

• TimeLineXR: An Interactive Exploration of Time in Vision-Language Models
  Nidham Tekaya, Markus Passecker, Matthias Zeppelzauer · arXiv · Oct 27, 2026

  Recently, it has been shown that vision-language models (VLMs) have a certain intrinsic understanding of time. They can estimate the temporal origin of objects and encode time along a low-dimensional, non-linear manifold inside their embedd…

• VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation
  Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma et al. · arXiv · Oct 26, 2026

  Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that wri…

• MedVTok: A General-Purpose Medical Visual Tokenizer
  Chenglong Ma, Yuanfeng Ji, Junzhi Ning, Jiyao Liu et al. · arXiv · Oct 26, 2026

  Multimodal medical AI requires a common visual interface for encoding heterogeneous imaging evidence for patient condition modeling. However, existing medical visual tokenizers are often tied to a specific dimension and imaging modality, fo…

• Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent
  Benjamin Pikus, Daniel Saunders, Mircea Mironenco, Salman Mohammadi et al. · arXiv · Oct 26, 2026

  A robot policy needs to understand what a task is asking for in a scene, and how to move the robot to accomplish that task. To achieve these two skills, we introduce Zero-real Agent Policy (ZAP), a recipe that teaches a small off-the-shelf …

• Multimodal Concert Video Editing: System Design and Exploratory Evaluation
  Lennart Schiweck, Adrian Kempf, Michelle Retig, Moritz Feucht et al. · arXiv · Oct 26, 2026

  Editing multi-camera concerts requires coordinated decisions about musical relevance, visual quality and cut timing. We present a modular framework that combines instrument activity from multitrack audio, visible instruments and image sharp…

• VPR-Space: A Benchmark for Visual Place Recognition in Space Analog Domains
  Richard Fechner, Peer Neubert · arXiv · Oct 26, 2026

  Visual Place Recognition (VPR) is an important means for loop closure detection in visual Simultaneous Localization and Mapping (SLAM). While state-of-the-art VPR and SLAM approaches excel in urban and feature-rich natural environments, the…

• EARS: Evidence-Aware Retrieval via Sound and Motion for Budgeted Video Reasoning
  Shih-Yao Lin · arXiv · Oct 26, 2026

  Uniform frame sampling can miss brief evidence in sparse-event videos when the VLM sees only K frames. We propose EARS (Evidence-Aware Retrieval via Sound and motion), a training-free, query-free front-end that ranks frames using optical-fl…

• Mixed-Signal Front-End Design for Concurrent Acquisition of Electrophysiological and Hemodynamic Brain Signals
  M.Karpagam · Progress in Electronics and... · Oct 25, 2026

  Based on experimental evidence, multimodal neuro engineering applications in the case of electrophysiological monitoring and hemodynamic monitoring of a brain region demands simultaneous functionality, but is difficult because of the differ…

• MiD-VLM: Detecting and Grounding Global Semantic Misinformation in Image-Caption Pairs
  Tarunya Prasad, Bharatesh Chakravarthi · arXiv · Oct 23, 2026

  Semantic misinformation can arise when an authentic image is paired with a caption that changes or distorts its narrative. Because the image itself may contain no visible signs of manipulation, methods designed to detect pixel-level artifac…

• Beyond Discrete Tokens: Preserving Numerical Continuity in Medical Vision-Language Models
  Dianye Huang, Jiamei Guo, Yue Zhou, Mingde Li et al. · arXiv · Oct 23, 2026

  Objective: To develop a unified vision-language model for prompt-conditioned quantitative ultrasound measurement that addresses the mismatch between continuous measurement targets and discrete token-based numerical prediction. Methods: This…

• Closing the Modality Gap via Simplex-Constrained Representations
  Shubham Gupta, Siva Reddy, Perouz Taslakian, Valentina Zantedeschi et al. · arXiv · Oct 23, 2026

  While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent \emph{modality gap}. We propose addressing this separation through a purely geometric approach…

• Multimodal Generative AI Assists Comprehensive Echocardiographic Interpretation
  Yi Qin, Marawan Elbatel, Shanshan Song, Taoran Huang et al. · arXiv · Oct 23, 2026

  Echocardiography is the cornerstone of cardiovascular diagnostics, yet its global utility is constrained by a critical shortage of experienced interpreters. Current artificial intelligence (AI) solutions remain limited, typically operating …

• Video-MemComp: Taylor-Guided Sensitivity-Aware Memory Management for Streaming Video Understanding
  Yonghui Niu, Yuhao Ji, Chang Nie, Jiao Zhang et al. · arXiv · Oct 23, 2026

  The rapid growth of video data has created an urgent need for efficient video understanding systems that can handle long and continuous visual inputs, posing significant challenges for Multimodal Large Language Models (MLLMs) due to severe …

• SP-Guide: Enhancing Aerial Vision-Language Navigation with Spatial Priors
  Haoxiang Tang, Dong Zhou, Shen Tan, Minxuan Jiang et al. · arXiv · Oct 23, 2026

  Aerial vision-language navigation (AVLN) requires a UAV to reach a language-described target from partial egocentric observations while reasoning over large outdoor scenes. Existing approaches typically fuse view-centered RGB observations a…