AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers (ICRA'2024)

Yongchao Chen, Jacob Arkin, Charles Dawson, Yang Zhang, Nicholas Roy, Chuchu Fan
Massachusetts Institute of Technology
arXiv Appendix Website Code and Dataset


We combine LLMs with the traditional task and motion planner to transform language instruction into control signals. The LLM's role is as the translator and semantic checkers.

Video

Abstract

Recent advances in large language models (LLMs) have shown promise for translating natural language into robot action sequences for complex tasks. However, existing approaches either translate the natural language directly into robot trajectories or factor the inference process by decomposing language into task sub-goals and relying on a motion planner to execute each sub-goal. When complex environmental and temporal constraints are involved, inference over planning tasks must be performed jointly with motion plans using traditional task-and-motion planning (TAMP) algorithms, making factorization into subgoals untenable.

Rather than using LLMs to directly plan task sub-goals, we instead perform few-shot translation from natural language task descriptions to an intermediate task representation that can then be consumed by a TAMP algorithm to jointly solve the task and motion plan. To improve translation, we automatically detect and correct both syntactic and semantic errors via autoregressive re-prompting, resulting in significant improvements in task completion.


Part1 of full prompt used by our AutoTAMP framework. The green part is the task explanation and the blue part is the few-shot examples.


Part2 of full prompt used by our AutoTAMP framework. This shows the left prompt after task explanation and few-shot examples, and also the structures of syntactic checking and semantic checking.

Related Links

This work is part of a broader research thread around language-instructed task and motion planning, which allows us to transform from natural language instruction into robot trajectory.

Other work on natural language to STL translation and LLM-based agents from our lab include:

BibTeX

@article{chen2023autotamp,
  title={AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers},
  author={Chen, Yongchao and Arkin, Jacob and Zhang, Yang and Roy, Nicholas and Fan, Chuchu},
  journal={arXiv preprint arXiv:2306.06531},
  year={2023}
}
}