Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning

Summary

Visual chain-of-thought methods allow multimodal large language models to reason through intermediate image generation, but synthesizing explicit visual steps creates major inference overhead during proactive video reasoning. To eliminate runtime synthesis, Internalized Visual Thinking introduces a post-training framework that jointly optimizes textual prediction and next-embedding prediction across unlabeled videos. By predicting latent representations of future frames alongside target text answers during training, the model captures motion, object transitions, interactions, and latent intent. At inference time, the model produces answers directly without generating or re-encoding future frames. IVT outperforms text-only post-training across six evaluation settings while matching or exceeding Visual CoT performance and cutting end-to-end latency by more than 5x.

Context

Multimodal large language models often rely on visual chain-of-thought to reason about spatial, temporal, and embodied environments by generating intermediate images. While this provides visual foresight, generating explicit images introduces substantial inference overhead that poses problems for proactive video reasoning tasks.

Approach / What changed

Internalized Visual Thinking (IVT) is a post-training framework that jointly optimizes textual prediction alongside next-embedding prediction over unlabeled videos. For a partially observed video, IVT predicts latent representations of future frames and textual answers together during training, allowing the model to internalize visual dynamics and answer directly at inference without synthesizing or re-encoding future frames.

Takeaways

• Internalized Visual Thinking jointly optimizes textual prediction and next-embedding prediction during post-training, learning latent representations of future frames from unlabeled video.
• IVT generates final answers directly at inference time without synthesizing or re-encoding intermediate future frames in pixel space.
• Compared to Visual CoT, IVT delivers comparable or superior performance across evaluations while reducing end-to-end latency by more than 5x.

Related reading

• Apple · Aug 25, 2026

  STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation

  Existing multimodal models struggle with structural fragmentation, often sacrificing visual fidelity via discrete tokenization or breaking structural symmetry by pairing causal text generation with diffusion-based denoising. STARFlow2 addresses these limitations by leveraging autoregressive normalizing flows, which share the causal mask, key-value cache mechanism, and left-to-right structure of large language models. Built upon the Pretzel architecture, the system vertically interleaves a frozen pretrained vision-language model stream with a TARFlow stream through residual skip connections under a single causal mask. A deep-shallow flow design combined with a unified FAE latent space allows both text and continuous visual outputs to enter the key-value cache directly without re-encoding. Benchmark experiments validate strong performance across multimodal understanding and continuous image generation tasks.

• Amazon · May 26, 2026

  Diverse reasoning traces teach LLMs to make better decisions

  Large language models typically rely on supervised fine-tuning with a single human-verified reasoning trace per example, which limits reasoning breadth and causes mode collapse when attempting parallel reasoning. Researchers presented set-supervised fine tuning (SSFT) and global forking policy optimization (GFPO) to teach models diverse problem-solving strategies. SSFT represents reasoning as a set of complete solution paths paired with global forking tokens, utilizing bipartite matching to achieve order-invariant training and specialized tokens. GFPO applies lightweight reinforcement learning to select the optimal forking token upfront, detaching gradients from generated rollouts to ensure stable optimization. On standard reasoning and coding benchmarks, the combined framework increased single-shot accuracy by 5% to 7% while improving pass@k diversity without degrading pass@1 accuracy.