arXiv v2 · November 2026

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

David Huang1·Lianlei Shan2

1 University of Toronto · 2 Tsinghua University

DLWM keeps several interpretations of an image in parallel and learns which paths to expand, stop or merge before answering.

Are the benches facing the water or the path?

A bench beside the water, with a footpath curving past the trees on the right.
A single bench viewed from across a stream, with a park path on the far side.
See answers from 10 people per photo
Left photo
10 water
Right photo
6 path · 3 water · 1 other (“no”)

VQAv2 validation set ↗ · Question IDs 35767000 (left) and 131965002 (right)

Figure 1 · Paper illustration

One question, several paths

Which edge of a bench is the backrest? That small detail can change the answer. In Figure 1, a single reasoning path settles on one reading early. DLWM can carry competing readings forward before choosing an answer.

Figure 1 depicts a separate lakeside scene and compares explicit reasoning, a single latent path and several DLWM paths.
Figure 1. The paper compares three reasoning strategies using a bench scene.Full size ↗

Figure 2 · Architecture

How DLWM reasons

DLWM architecture, from the vision-language encoder through world generation and controller actions to the weighted answer.
Figure 2. The generator, shared reasoner, controller, and prediction head.Full size ↗

Form hypotheses

Each world is a candidate interpretation of the image and question, represented by a latent vector. The generator learns to produce a diverse set of worlds.

Allocate compute

A shared Transformer updates the worlds. A learned controller spends its budget on expanding, stopping, or merging paths, and decides when to answer.

Combine predictions

Each surviving world predicts the next answer token. A learned score sets its weight in the final probability distribution.

Experiments

Accuracy and inference cost

DLWM has the highest reported accuracy on all four benchmarks among the methods shown. It also uses less memory than LVR and LaRe. Heima uses the least memory and has the highest throughput.

Benchmark accuracy (%) Table 1(a)
MethodGQAVQAv2ScienceQAMMVP
LVR 66.474.872.160.9
LaRe 66.975.072.661.7
Heima 65.274.371.559.8
DLWM 69.376.075.266.1
Inference cost Table 1(b)
MethodMemory (GB)Samples/s
LVR 19.06.1
LaRe 19.55.9
Heima 13.89.2
DLWM 14.58.9
Complete comparison in Table 1 ↗

Code and citation

The Quick Start covers training on GQA, VQAv2 and ScienceQA. Evaluation also includes MMVP as a zero-shot benchmark. The reproducibility notes describe the released configurations and how they relate to the paper. The original checkpoints, predictions and logs are unavailable in this release.

@misc{huang2026dlwmdiverselatentworld,
      title={DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning}, 
      author={David Huang and Lianlei Shan},
      year={2026},
      eprint={2606.15160},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.15160}, 
}
Photo credits

Left photo · Bare trees and benches. by Ronald Saunders · CC BY-SA 2.0 (cropped preview)

Right photo · Bench Pavilion Gardens Buxton by jillyspoon · CC BY-ND 2.0