DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
1 University of Toronto · 2 Tsinghua University
DLWM keeps several interpretations of an image in parallel and learns which paths to expand, stop or merge before answering.
Are the benches facing the water or the path?
See answers from 10 people per photo
- Left photo
- 10 water
- Right photo
- 6 path · 3 water · 1 other (“no”)
VQAv2 validation set ↗ · Question IDs 35767000 (left) and 131965002 (right)
One question, several paths
Which edge of a bench is the backrest? That small detail can change the answer. In Figure 1, a single reasoning path settles on one reading early. DLWM can carry competing readings forward before choosing an answer.

How DLWM reasons

Form hypotheses
Each world is a candidate interpretation of the image and question, represented by a latent vector. The generator learns to produce a diverse set of worlds.
Allocate compute
A shared Transformer updates the worlds. A learned controller spends its budget on expanding, stopping, or merging paths, and decides when to answer.
Combine predictions
Each surviving world predicts the next answer token. A learned score sets its weight in the final probability distribution.
Accuracy and inference cost
DLWM has the highest reported accuracy on all four benchmarks among the methods shown. It also uses less memory than LVR and LaRe. Heima uses the least memory and has the highest throughput.
| Method | GQA | VQAv2 | ScienceQA | MMVP |
|---|---|---|---|---|
| LVR | 66.4 | 74.8 | 72.1 | 60.9 |
| LaRe | 66.9 | 75.0 | 72.6 | 61.7 |
| Heima | 65.2 | 74.3 | 71.5 | 59.8 |
| DLWM | 69.3 | 76.0 | 75.2 | 66.1 |
| Method | Memory (GB) | Samples/s |
|---|---|---|
| LVR | 19.0 | 6.1 |
| LaRe | 19.5 | 5.9 |
| Heima | 13.8 | 9.2 |
| DLWM | 14.5 | 8.9 |
Code and citation
The Quick Start covers training on GQA, VQAv2 and ScienceQA. Evaluation also includes MMVP as a zero-shot benchmark. The reproducibility notes describe the released configurations and how they relate to the paper. The original checkpoints, predictions and logs are unavailable in this release.
@misc{huang2026dlwmdiverselatentworld,
title={DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning},
author={David Huang and Lianlei Shan},
year={2026},
eprint={2606.15160},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2606.15160},
}
Photo credits
Left photo · Bare trees and benches. by Ronald Saunders · CC BY-SA 2.0 (cropped preview)
Right photo · Bench Pavilion Gardens Buxton by jillyspoon · CC BY-ND 2.0

