OVOW

KO | EN

gitlite — search

#react #ai #machine-learning #rust #docker #macos #nodejs #java #kubernetes #awesome-list #vue #pytorch #security #openai

(★ 21)

[ECCV 2026] One Video, One World: Turning Monocular Video into Physical 4D Scenes

OVOW Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# One Video, One World: Turning Monocular Video into Physical 4D Scenes

**Junhao Chen**&emsp;
**Boran Zhang**&emsp;
**Mingjin Chen**&emsp;
**Henghaofan Zhang**&emsp;
**Saining Zhang**<br>
**Congcong Zhu**&emsp;
**Hao Zhao**&emsp;
**Ruqi Huang**&emsp;
**Zhihao Li**&emsp;
**Yufei Wang**

Tsinghua University&emsp;·&emsp;SparcAI Inc.&emsp;·&emsp;University of Science and Technology of China<br>
The Hong Kong Polytechnic University&emsp;·&emsp;University of Electronic Science and Technology of China&emsp;·&emsp;Nanyang Technological University

<br>

</div>

> **OVOW reconstructs instance-level, simulation-ready 4D mesh scenes from monocular video.** Given a single video, our method decomposes the scene into physically independent mesh instances and recovers rigid-body motions and non-rigid mesh deformations, yielding instance-level meshes ready for downstream physics simulation and editing. We demonstrate results of multi-object collisions, rigid-body motions, and deforming object motions across tabletop, indoor, and in-the-wild scenarios.

---

## 📖 Abstract

We introduce **OVOW**, the first **training-free** system that reconstructs *instance-level, simulation-ready* 4D mesh scenes from a single monocular video. Recent 4D reconstruction achieves impressive rendering quality, but its outputs (e.g., implicit fields, Gaussian primitives, or point clouds) lack the watertight topology, instance separation, and standardized physical interfaces required by physics simulators and embodied AI. OVOW closes this gap with a four-stage pipeline: a vision-language model discovers, labels, and motion-classifies all instances; category-aware reconstruction yields per-instance meshes for rigid objects and topology-consistent mesh sequences for deformable ones; an iterative render-match-optimize procedure recovers metric scale and 6-DoF pose trajectories; and physics-grounded assembly enforces ground contact and inter-object support. Crucially, we model all motion, rigid and non-rigid, through direct vertex deformation without category-specific priors or skeleton rigging, producing watertight mesh scenes ready for downstream physics simulation and editing. We further establish the first benchmark for *structured Video-to-4D* evaluation, with metrics for geometric correctness, instance separation, and physical plausibility beyond visual fidelity; the same pipeline doubles as a scalable engine for *synthesizing* paired video-to-4D simulation data for future 4D world models and embodied AI. Across two synthetic benchmarks (static and 4D), OVOW attains the best overall layout and geometry accuracy and the lowest photometric and semantic error among all baselines, and on monocular video runs one to two orders of magnitude faster than the baselines, while downstream physics simulation confirms its physical stability.

## ✨ Highlights

- **First structured Video-to-4D system.** Turns a monocular video into instance-level, *simulation-ready* 4D mesh scenes — watertight, instance-separated, and physics-engine ready (URDF).
- **Fully training-free.** Composes pretrained foundation models into a tightly integrated pipeline; no task-specific training.
- **Unified motion model.** All motion — static, rigid, and deformable — is modeled via direct vertex deformation, with **no** category priors or skeleton rigging.
- **New benchmark & data engine.** The first benchmark scoring geometric correctness, instance separation, and physical plausibility beyond visual fidelity; the same pipeline synthesizes paired *video ↔ 4D scene* data at scale.
- **Fast & stable.** On video, amortizes to **3.35 s/frame** — one to two orders of magnitude faster than single-image baselines — while reconstructed scenes stay physically stable under gravity in simulation.

## 🔧 Method

OVOW is a fully training-free pipeline with four stages:

| # | Stage | What it does | Foundation models |
|---|-------|--------------|-------------------|
| 1 | **VLM-Guided Scene Decomposition** | Discovers, uniquely names, and motion-classifies every instance (static / rigid / deformable); produces dense per-frame masks. | Qwen3-VL, SAM3 |
| 2 | **Instance-Level Mesh Reconstruction** | Amodal inpainting + feed-forward image-to-3D for static/rigid objects; topology-consistent mesh sequences for deformable ones; metric scale recovery. | FLUX.2, Hi3DGen, Motion324, VGGT |
| 3 | **Spatiotemporal Pose & Deformation Recovery** | Iterative render-match-optimize for metric scale, orientation, and per-frame 6-DoF pose trajectories; decouples global motion from local vertex deformation. | RoMa v2, FoundationPose |
| 4 | **Physics-Grounded Scene Assembly** | Ground-plane estimation and contact projection enforce ground contact and inter-object support; exports a coherent, simulation-ready scene with recovered HDR lighting. | — |

## 🌐 Interactive Demos

Explore browser-native, interactive 3D/4D reconstructions on the **[project page](https://OneVideoOneWorld.github.io)** — instance-level meshes with recovered rigid and non-rigid motion, spanning video-to-4D and image-to-3D examples.

## 🚀 Code

> **The code is currently under internal company review and will be released here soon.**

Please ⭐ **star** and **watch** the repo to be notified as soon as it is available.

## 📌 Citation

If you find our work useful, please consider citing:

```bibtex
@misc{chen2026videoworldturningmonocular,
      title={One Video, One World: Turning Monocular Video into Physical 4D Scenes},
      author={Junhao Chen and Boran Zhang and Mingjin Chen and Henghaofan Zhang and Saining Zhang and Congcong Zhu and Hao Zhao and Ruqi Huang and Zhihao Li and Yufei Wang},
      year={2026},
      eprint={2606.31388},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.31388},
}
```

## 🔗 Links

- **arXiv:** https://arxiv.org/abs/2606.31388
- **Project Page:** https://OneVideoOneWorld.github.io
- **Code:** https://github.com/yisuanwang/OVOW

OVOW

OVOW Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

OVOW

OVOW Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README