Planning with Shifting Object Positions in LLM-Generated Robot Workflows

Background

In our research on embodied AI, a Large Language Model (LLM) translates natural-language intentions into executable Python programs that control a quadruped robot. A Vision-Language Model (VLM) builds a scene graph of the environment, which the LLM uses to generate the program. When the environment changes, the LLM patches the running program at runtime.

Problem

Object positions in the scene graph are not stable. Due to imprecise perception, an object first estimated at (1.0, 1.0) may later appear at (1.2, 1.2) as the robot sees it from new angles, even though it never moved. During execution, this raises several questions:

• Patch or not? Is the shift a real change that requires a patch, or just estimation noise? Patching on every update causes oscillating plans and wasted LLM calls.
• Stale values: Coordinates may already be bound in the running program, e.g. passed to an ongoing move_to(1.0, 1.0). Should the action be interrupted and redirected, completed and corrected afterwards, or left as is?
• Object identity: Is the object at (1.2, 1.2) the same one, or a new object? Wrong decisions lead to duplicate entries in the scene graph and plans that visit the same object twice or skip a real one.
• Literal vs. reference: Should the generated code contain literal coordinates (resolved at generation time) or object references (resolved at execution time)?

Goal and Tasks

The thesis investigates how shifting position estimates affect planning and runtime patching, and develops strategies to handle them:

1. Analysis: Inject controlled position drift in simulation (Gazebo/ROS 2) and categorize its effects on the generated programs and patches.
2. Concept: Design strategies, e.g. thresholds or hysteresis for patch triggers, late binding via object references instead of literal coordinates, rules for redirecting running actions, and identity tracking for drifting objects.
3. Implementation: Prototype the strategies in the existing pipeline.
4. Evaluation: Compare against the current system (success rate, number of patches, unnecessary path length, duplicate or missed objects).

Requirements

Good Python skills and interest in robotics and LLMs. Experience with ROS 2 is a plus.

Contact

Justus Flerlage j.flerlage@tu-berlin.de