In context
In the mid-2020s, large language models (LLMs) were increasingly applied to robot task planning, but their lack of real-world scene perception often led to unexecutable plans—a critical bottleneck for deploying service robots in unstructured home environments. This research, published in the Chinese journal Robot, addressed that gap by grounding LLM reasoning in physical scene information.
What was reported
Researchers from Beijing University of Posts and Telecommunications and Nanyang Technological University proposed TaPA, an embodied task planning framework that aligns scene information with LLMs to generate executable action sequences. The framework uses a multimodal instruction-tuning dataset synthesized from triples of scene information, human instructions, and action plans, which is used to fine-tune a pre-trained LLaMA-7B model. During inference, an open-vocabulary object detector (Detic) provides the LLM with a list of objects present in the scene, gathered from multi-view RGB images collected by the robot.
Experiments were conducted in the AI2-THOR simulator, with training data expanded from 80 to 6,400 scenes via object replacement strategies, yielding 15,000 training samples. The evaluation involved 30 volunteers voting on plan executability. TaPA achieved a task planning success rate 6.38% higher than GPT-3.5, and also outperformed LLaMA and LLaVA baselines. The framework was designed to handle more complex and diverse tasks than prior benchmarks like ALFRED, supporting varied instructions and deployment environments.
Why it mattered
By coupling LLM reasoning with real-time visual perception, TaPA demonstrated a practical path to making LLM-based planners physically grounded, a key step toward reliable home service robots. Its data synthesis and perception strategies offered a scalable approach for adapting general-purpose models to specific deployment scenarios, potentially accelerating commercial adoption in domestic automation.
“The proposed TaPA framework outperforms the existing GPT-3.5 model by 6.38% in success rate of task planning, effectively facilitating the process of home service robot deployment.”
Source: 《机器人》期刊 (robot.sia.cn) · Published 2025-08-19 · “面向居家服务机器人任务规划的具身大模型”