awesome-vln

KO | EN

gitlite — search

#javascript #hacktoberfest #react #go #android #machine-learning #deep-learning #linux #cli #pytorch #security #ios #mcp #openai

(★ 236)

A curated list of research papers in Vision-Language Navigation (VLN)

File Explorer

Download Latest Version (.zip)

LICENSE
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Awesome Vision-Language Navigation

A curated list of research papers in Vision-Language Navigation (VLN). Link to the code and website if available is also present. You can also find more embodied vision papers in **[ awesome-embodied-vision](https://github.com/ChanganVR/awesome-embodied-vision)**.

## Contributing

Please feel free to contact me via email (liudq@mail.ustc.edu.cn) or open an issue or submit a pull request.

To add a new paper via pull request:

1. Fork the repo, edit `README.md`.

2. Put the new paper at the correct chronological position as the following format:

```
 - **Paper Title** 
 *Author(s)* 
 Conference, Year. [[Paper]](link) [[Code]](link) [[Website]](link)
 ```

3. Send a pull request. Ideally, I will review the request within a week.

## Papers

### Tasks:

- **Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments** 
 *Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, Anton van den Hengel* 
 CVPR, 2018. [[Paper]](https://arxiv.org/abs/1711.07280) [[Code]](https://github.com/peteanderson80/Matterport3DSimulator) [[Website]](https://bringmeaspoon.org)

- **HoME: a Household Multimodal Environment** 
 *Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, Aaron Courville* 
 NIPS Workshop, 2017. [[Paper]](https://arxiv.org/abs/1711.11017) [[Code]](https://github.com/ml-lab/home-platform)

- **Talk the Walk: Navigating New York City through Grounded Dialogue** 
 *Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, Douwe Kiela* 
 arXiv, 2019. [[Paper]](https://arxiv.org/abs/1807.03367) [[Code]](https://github.com/facebookresearch/talkthewalk)

- **Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments** 
 *Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, Yoav Artzi* 
 CVPR, 2019. [[Paper]](https://arxiv.org/abs/1811.12354) [[Code]](https://github.com/lil-lab/touchdown) [[Website]](https://sites.google.com/view/streetlearn/touchdown)

- **Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention** 
 *Khanh Nguyen, Debadeepta Dey, Chris Brockett, Bill Dolan* 
 CVPR, 2019. [[Paper]](https://arxiv.org/abs/1812.04155) [[Code]](https://github.com/debadeepta/vnla) [[Video]](https://youtu.be/Vp6C29qTKQ0)

- **Learning To Follow Directions in Street View** 
 *Karl Moritz Hermann, Mateusz Malinowski, Piotr Mirowski, Andras Banki-Horvath, Keith Anderson, Raia Hadsell* 
 AAAI, 2020. [[Paper]](https://arxiv.org/abs/1903.00401) [[Website]](https://sites.google.com/view/streetlearn/aaai-2020)

- **REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments** 
 *Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, Anton van den Hengel* 
 CVPR, 2020. [[Paper]](https://arxiv.org/abs/1904.10151)

- **Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation** 
 *Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, Jason Baldridge* 
 ACL, 2019. [[Paper]](https://arxiv.org/abs/1905.12255) [[Code]](https://github.com/google-research/google-research/tree/master/r4r)

- **Vision-and-Dialog Navigation** 
 *Jesse Thomason, Michael Murray, Maya Cakmak, Luke Zettlemoyer* 
 CoRL, 2019. [[Paper]](https://arxiv.org/abs/1907.04957) [[Website]](https://cvdn.dev/)

- **Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning** 
 *Khanh Nguyen, Hal Daumé III* 
 EMNLP, 2019. [[Paper]](https://arxiv.org/abs/1909.01871) [[Code]](https://github.com/khanhptnk/hanna) [[Video]](https://youtu.be/18P94aaaLKg)

- **Talk2Nav: Long-Range Vision-and-Language Navigation with Dual Attention and Spatial Memory** 
 *Arun Balajee Vasudevan, Dengxin Dai, Luc Van Gool* 
 arXiv, 2019. [[Paper]](https://arxiv.org/abs/1910.02029) [[Website]](https://people.ee.ethz.ch/~arunv/talk2nav.html)

- **Cross-Lingual Vision-Language Navigation** 
 *An Yan, Xin Wang, Jiangtao Feng, Lei Li, William Yang Wang* 
 arXiv, 2019. [[Paper]](https://arxiv.org/abs/1910.11301) [[Code]](https://github.com/zzxslp/Crosslingual-VLN)

- **Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments** 
 *Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, Stefan Lee* 
 arXiv, 2020. [[Paper]](https://arxiv.org/abs/2004.02857) [[Code]](https://github.com/jacobkrantz/VLN-CE)

- **Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation** 
 *Muhammad Zubair Irshad, Chih-Yao Ma, Zsolt Kira* 
 ICRA, 2021. [[Paper]](https://arxiv.org/abs/2104.10674) [[Code]](https://github.com/GT-RIPL/robo-vln)

### Roadmap (Chronological Order):

- **Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments** 
 *Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, Anton van den Hengel* 
 CVPR, 2018. [[Paper]](https://arxiv.org/abs/1711.07280) [[Code]](https://github.com/peteanderson80/Matterport3DSimulator) [[Website]](https://bringmeaspoon.org)
 
- **Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation** 
 *Xin Wang, Wenhan Xiong, Hongmin Wang, William Yang Wang* 
 ECCV, 2018. [[Paper]](https://arxiv.org/abs/1803.07729)
 
- **Speaker-Follower Models for Vision-and-Language Navigation** 
 *Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, Trevor Darrell* 
 NeurIPS, 2018. [[Paper]](https://arxiv.org/abs/1806.02724) [[Code]](https://github.com/ronghanghu/speaker_follower) [[Website]](http://ronghanghu.com/speaker_follower/)
 
- **Shifting the Baseline: Single Modality Performance on Visual Navigation & QA** 
 *Jesse Thomason, Daniel Gordon, Yonatan Bisk* 
 NAACL, 2019. [[Paper]](https://arxiv.org/abs/1811.00613) [[Poster]](https://jessethomason.com/personal_site/www/publication_supplements/NAACL19_poster.pdf)
 
- **Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation** 
 *Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang* 
 CVPR, 2019. [[Paper]](https://arxiv.org/abs/1811.10092)
 
- **Self-Monitoring Navigation Agent via Auxiliary Progress Estimation** 
 *Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, Caiming Xiong* 
 ICLR, 2019. [[Paper]](https://arxiv.org/abs/1901.03035) [[Code]](https://github.com/chihyaoma/selfmonitoring-agent) [[Website]](https://chihyaoma.github.io/project/2018/09/27/selfmonitoring.html)
 
- **The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation** 
 *Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, Zsolt Kira* 
 CVPR, 2019. [[Paper]](https://arxiv.org/abs/1903.01602) [[Code]](https://github.com/chihyaoma/regretful-agent) [[Website]](https://chihyaoma.github.io/project/2019/02/25/regretful.html)
 
- **Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation** 
 *Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, Siddhartha Srinivasa* 
 CVPR, 2019. [[Paper]](http://arxiv.org/abs/1903.02547) [[Code]](https://github.com/Kelym/FAST) [[Video]](https://www.youtube.com/watch?v=AD9TNohXoPA)
 
- **Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout** 
 *Hao Tan, Licheng Yu, Mohit Bansal* 
 NAACL, 2019. [[Paper]](https://arxiv.org/abs/1904.04195) [[Code]](https://github.com/airsplay/R2R-EnvDrop)
 
- **Multi-modal Discriminative Model for Vision-and-Language Navigation** 
 *Haoshuo Huang, Vihan Jain, Harsh Mehta, Jason Baldridge, Eugene Ie* 
 NAACL Workshop, 2019. [[Paper]](https://arxiv.org/abs/1905.13358)
 
- **Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation** 
 *Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, Kate Saenko* 
 ACL, 2019. [[Paper]](https://arxiv.org/abs/1906.00347)
 
- **Chasing Ghosts: Instruction Following as Bayesian State Tracking** 
 *Peter Anderson, Ayush Shrivastava, Devi Parikh, Dhruv Batra, Stefan Lee* 
 NeurIPS, 2019. [[Paper]](https://arxiv.org/abs/1907.02022) [[Code]](https://github.com/batra-mlp-lab/vln-chasing-ghosts) [[Video]](https://www.youtube.com/watch?v=eoGbescCNP0)
 
- **Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters** 
 *Federico Landi, Lorenzo Baraldi, Massimiliano Corsini, Rita Cucchiara* 
 BMVC, 2019. [[Paper]](https://arxiv.org/abs/1907.02985) [[Code]](https://github.com/aimagelab/DynamicConv-agent)
 
- **Transferable Representation Learning in Vision-and-Language Navigation** 
 *Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhaes, Jason Baldridge, Eugene Ie* 
 ICCV, 2019. [[Paper]](https://arxiv.org/abs/1908.03409)
 
- **Robust Navigation with Language Pretraining and Stochastic Sampling** 
 *Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah Smith, Yejin Choi* 
 EMNLP, 2019. [[Paper]](https://arxiv.org/abs/1909.02244) [[Code]](https://github.com/xjli/r2r_vln)
 
- **Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling** 
 *Tsu-Jui Fu, Xin Wang, Matthew Peterson, Scott Grafton, Miguel Eckstein, William Yang Wang* 
 arXiv, 2019. [[Paper]](https://arxiv.org/abs/1911.07308)
 
- **Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation** 
 *Juncheng Li, Xin Wang, Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang, William Yang Wang* 
 CVPR, 2020. [[Paper]](https://arxiv.org/abs/1911.07450)

- **Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks** 
 *Fengda Zhu, Yi Zhu, Xiaojun Chang, Xiaodan Liang* 
 CVPR, 2020. [[Paper]](https://arxiv.org/abs/1911.07883)
 
- **Perceive, Transform, and Act: Multi-Modal Attention Networks for Vision-and-Language Navigation** 
 *Federico Landi, Lorenzo Baraldi, Marcella Cornia, Massimiliano Corsini, Rita Cucchiara* 
 arXiv, 2019. [[Paper]](https://arxiv.org/abs/1911.12377) [[Code]](https://github.com/aimagelab/perceive-transform-and-act)
 
- **Just Ask: An Interactive Learning Framework for Vision and Language Navigation** 
 *Ta-Chung Chi, Mihail Eric, Seokhwan Kim, Minmin Shen, Dilek Hakkani-tur* 
 AAAI, 2020. [[Paper]](https://arxiv.org/abs/1912.00915)
 
- **Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training** 
 *Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, Jianfeng Gao* 
 CVPR, 2020. [[Paper]](https://arxiv.org/abs/2002.10638) [[Code]](https://github.com/weituo12321/PREVALENT)
 
- **Multi-View Learning for Vision-and-Language Navigation** 
 *Qiaolin Xia, Xiujun Li, Chunyuan Li, Yonatan Bisk, Zhifang Sui, Jianfeng Gao, Yejin Choi, Noah A. Smith* 
 arXiv, 2020. [[Paper]](https://arxiv.org/abs/2003.00857)
 
- **Vision-Dialog Navigation by Exploring Cross-modal Memory** 
 *Yi Zhu, Fengda Zhu, Zhaohuan Zhan, Bingqian Lin, Jianbin Jiao, Xiaojun Chang, Xiaodan Liang* 
 CVPR, 2020. [[Paper]](https://arxiv.org/abs/2003.06745) [[Code]](https://github.com/yeezhu/CMN.pytorch)
 
- **Take the Scenic Route: Improving Generalization in Vision-and-Language Navigation** 
 *Felix Yu, Zhiwei Deng, Karthik Narasimhan, Olga Russakovsky* 
 arXiv, 2020. [[Paper]](https://arxiv.org/abs/2003.14269)
 
- **Sub-Instruction Aware Vision-and-Language Navigation** 
 *Yicong Hong, Cristian Rodriguez-Opazo, Qi Wu, Stephen Gould* 
 arXiv, 2020. [[Paper]](https://arxiv.org/abs/2004.02707)
 
- **Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments** 
 *Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, Stefan Lee* 
 ECCV, 2020. [[Paper]](https://arxiv.org/abs/2004.02857) [[Code]](https://github.com/jacobkrantz/VLN-CE) [[Website]](https://jacobkrantz.github.io/vlnce)
 
- **Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling** 
 *Tsu-Jui Fu, Xin Eric Wang, Matthew Peterson, Scott Grafton, Miguel Eckstein, William Yang Wang* 
 ECCV, 2020. [[Paper]](https://arxiv.org/abs/1911.07308)
 
- **Improving Vision-and-Language Navigation with Image-Text Pairs from the Web** 
 *Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, Dhruv Batra* 
 ECCV, 2020. [[Paper]](https://arxiv.org/abs/2004.14973)
 
- **Soft Expert Reward Learning for Vision-and-Language Navigation** 
 *Hu Wang, Qi Wu, Chunhua Shen* 
 ECCV, 2020. [[Paper]](https://arxiv.org/abs/2007.10835)
 
- **Active Visual Information Gathering for Vision-Language Navigation** 
 *Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, Jianbing Shen* 
 ECCV, 2020. [[Paper]](https://arxiv.org/abs/2007.08037) [[Code]](https://github.com/HanqingWangAI/Active_VLN)
 
- **Environment-agnostic Multitask Learning for Natural Language Grounded Navigation** 
 *Xin Eric Wang, Vihan Jain, Eugene Ie, William Yang Wang, Zornitsa Kozareva, Sujith Ravi* 
 ECCV, 2020. [[Paper]](https://arxiv.org/abs/2003.00443)

- **Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation** 
 *Muhammad Zubair Irshad, Chih-Yao Ma, Zsolt Kira* 
 ICRA, 2021. [[Paper]](https://arxiv.org/abs/2104.10674) [[Code]](https://github.com/GT-RIPL/robo-vln) [[Website]](https://zubair-irshad.github.io/projects/robo-vln.html) [[Video]](https://www.youtube.com/watch?v=y16x9n_zP_4)

awesome-vln

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

awesome-vln

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README