01 · Contact-rich surface interaction and coverage
Grounded Task Axes · Version 2
From Language to Manipulation: Composing Object-Centric Skills from Scratch.
GTA-2 turns high-level language instructions into executable, interpretable, and correctable robot manipulation skills through task-axis controller compositions—zero-shot, without task-specific demonstrations or training.
01 · Task breadth
Fourteen different real-world tasks—
no demonstrations provided.
The benchmark spans contact-rich surface interaction and coverage, precision alignment and fastening, articulated-object and device operation, semantic ordering and multi-object organization, deformable-object manipulation, obstacle-aware pick-and-place, and pouring and material transfer.
14 generated-plan comparisons
02 · Pouring, dispensing, and material transfer
Grasp the Cup by the Handle and Gently Pour into the Box
03 · Deformable object manipulation
Cut the Dough/Sponge with the Knife in Hand
04 · Precision alignment, insertion, fastening, and hanging
Unscrew the Black Nut
05 · Device operation
Start the Microwave
06 · Device operation
Pick Up the Phone Call
07 · Clutter cleanup and multi-object organization
Sweep the Trash into the Dustpan
08 · Basic pick and place for VLA
Put the Ball into the Basket (There Is a Wall Obstacle in Between)
09 · Clutter cleanup and multi-object organization
Put the Fruits into the White Bowl / Put the Others into the Pink Container
10 · Semantic reasoning and ordering
Fix the Typo
11 · Basic pick and place for VLA
Put the Ball into the Basket
12 · Basic pick and place for VLA
Put the Cube into the Drawer
13 · Contact-rich surface interaction and coverage
Iron the Tie
14 · Articulated objects and appliances
Close the Drawer
02 · Overview
Manipulation stays explicit
from intent to control.
The system separates reasoning about what the task means from the geometric and control decisions required to execute it.Most reusable robot skills are still large units: pick, place, open. GTA-2 composes below that level.
Sub-atomic composition means constructing each subtask from task-relevant keypoints, axes, constraints, and controllers—the reusable structure beneath a conventional primitive.
- 01
Decompose task intent
Expose semantic subtasks in an ordered, inspectable plan.
- 02
Compose at the sub-atomic level
Build each subtask from object-centric task-axis controller compositions.
- 03
Correct locally
Route feedback to the responsible stage while preserving decisions that are already correct.
Read the full abstract+
Robotic manipulation tasks are often decomposed into reusable behaviors or skills, yet these units can remain too coarse to expose the geometric, control, and scene-dependent decisions required for execution. We introduce Grounded Task Axes version 2 (GTA-2), a modular multi-agent vision-language model (VLM) framework that constructs executable manipulation skills from reusable object-centric task-axis components. Rather than predicting actions end-to-end or relying on predefined task-level skill primitives, GTA-2 explicitly represents manipulation structure through semantic subtasks, task-relevant keypoints and axes, controller compositions, and scene-dependent parameters. Four specialized VLM agents progressively decompose the task, generate an abstract task-axis skill, assign controller parameters, and ground the required visual features in an RGB-D observation. This separation between abstract skill construction and scene-specific instantiation enables zero-shot generation of new manipulation behaviors without task-specific robot demonstrations, policy training, or fine-tuning. Because the intermediate decisions remain explicit, users can also provide targeted feedback to the responsible agent while preserving components of the generated skill that are already correct. We evaluate GTA-2 on a real robot across 14 diverse manipulation tasks and compare it against the generalist VLA policy π0.5 and two Code-as-Policies baselines using either task-axis controllers or conventional robot primitives. GTA-2 achieves stronger zero-shot execution across the benchmark, while targeted refinement further improves and stabilizes skill generalization across scene configurations.
03 · Method
Four VLMs—each specialized
in a different aspect of manipulation.
Four specialized VLM agents transform task intent into a grounded controller composition. Scroll to follow the representation as it becomes executable.
Task promptput the bottle into the boxRGB-D scene · controller library
Generated skill04Move over the box−
01Approach bottle for pre-grasp+
02Grasp the bottle+
03Lift the bottle up+
1PosAlign(bottle_cap, box_inside, o_z = 0.15 m)2PosAlign(bottle_cap, box_inside, o_y = 0.00 m)3PosAlign(bottle_cap, box_inside, o_x = 0.00 m)4AxisAlign(eef_z, box_inside_norm, o_θ = 180°)Ptarget+Raxis+Fcontact+Gstate
05Place down the bottle+
06Release and retract+
00
Task prompt + RGB-D scene
Begin with a task instruction and an RGB-D scene.
The instruction specifies the desired outcome while the observation and reusable controller library provide the scene and action vocabulary used by the four agents.- Input
- “put the bottle into the box”
- Output
- Inputs for the four-agent pipeline
Inputs
- Task prompt: put the bottle into the box
- RGB-D scene observation
- Reusable task-axis controller library
01
Task Decomposer
Convert the instruction into ordered subtasks.
The Task Decomposer exposes the temporal and semantic structure of the manipulation behavior before any controller is selected.- Input
- Task prompt + scene context
- Output
- Ordered Subtasks
Ordered Subtasks
- Approach bottle for pre-grasp
- Grasp the bottle
- Lift the bottle up
- Move over the box
- Place down the bottle
- Release and retract
02
Skill Generator
Map each subtask to an abstract skill recipe.
For each subtask, the Skill Generator selects symbolic object features, task axes, controller compositions, and priorities from the reusable controller library.- Input
- Ordered subtasks + controller library
- Output
- Abstract Skill Recipes
Highlighted subtask
- 04 · Move over the box
Keypoints
- bottle_cap
- box_inside
Controllers
- PosAlign(bottle_cap, box_inside, o_z)
- PosAlign(bottle_cap, box_inside, o_y)
- PosAlign(bottle_cap, box_inside, o_x)
- AxisAlign(eef_z, box_inside_norm, o_θ)
03
Parameter Setter
Assign the controller parameter groundings.
The Parameter Setter replaces symbolic placeholders with numerical values inferred for the observed task and scene.- Input
- Abstract skill recipes
- Output
- Parameter Groundings
Parameter Groundings
- Hover height (o_z): 0.15 m
- Vertical offsets (o_x, o_y): 0.00 m
- EEF angle offset (o_θ): 180°
- Grasp depth (o_d): 0.02 m
04
Vision Grounder
Ground only the requested scene features.
The Vision Grounder instantiates the keypoints and axes requested by the lifted skill as metric features in the current RGB-D scene.- Input
- Requested symbolic keypoints + axes
- Output
- 3D Groundings
3D Groundings
- bottle_cap
- box_inside
- eef_tcp
- eef_z_axis
- bottle_cap_normal
- bottle_inside_normal
05
Robot-agnostic execution interface
Compile the grounded skill into robot execution.
The grounded task-axis skill is compiled into an executable robot script whose controllers are evaluated at runtime.- Input
- Grounded task-axis skill
- Output
- Executable robot script
Runtime Commands
- Position
- Orientation
- Force
- Gripper
Inspect the source figure.
The web-native diagram now follows the paper’s bottle-to-box example and its intermediate outputs.04 · Targeted refinement
Correct the responsible stage,
not the whole behavior.
Because intermediate decisions remain explicit, feedback can update decomposition, skill structure, parameters, or visual grounding independently.Execution failures are attributed to the agent that made the responsible decision. The rest of the generated behavior is retained.
Task→Skill→Parameters→Vision
15 scene configurations per task. Colored segments identify the responsible feedback stage; white segments are successful executions.
05 · Full results
Task generalization across fourteen
real-world manipulation tasks.
Success rates across 14 tasks and five methods, evaluated over 20 real-robot executions for each task–method pair.
Open accessible results table
| Task | One-Success GTA-2 | Zero-Shot GTA-2 | Pi-0.5 | CaP-TAC | CaP-Primitive |
|---|---|---|---|---|---|
| Put the Ball into the Basket | 90% | 85% | 55% | 40% | 75% |
| Put the Cube into the Drawer | 100% | 85% | 40% | 45% | 60% |
| Pick Up the Phone Call | 100% | 75% | 5% | 40% | 45% |
| Iron the Tie | 85% | 75% | 0% | 30% | 45% |
| Put the Fruits into the White Bowl / Put the Others into the Pink Container | 85% | 75% | 5% | 20% | 40% |
| Close the Drawer | 90% | 65% | 10% | 40% | 40% |
| Grasp the Cup by the Handle and Gently Pour into the Box | 80% | 55% | 0% | 15% | 0% |
| Put the Ball into the Basket (Obstacle) | 95% | 80% | 35% | 45% | 55% |
| Erase the Red Scribble | 100% | 80% | 40% | 60% | 80% |
| Start the Microwave | 100% | 75% | 10% | 30% | 50% |
| Fix the Typo | 85% | 75% | 0% | 35% | 50% |
| Unscrew the Black Nut | 90% | 75% | 0% | 40% | 50% |
| Cut the Dough/Sponge with the Knife in Hand | 90% | 75% | 10% | 20% | 5% |
| Sweep the Trash into the Dustpan | 80% | 60% | 0% | 20% | 0% |
Comparison with Pi-0.5 and the Code-as-Policies baselines.
06 · More behaviors
Additional robot behaviors
beyond the benchmark.
Additional trials probe open-world placement, geometric tool paths, underspecified language, small objects, and semantic destinations.
01Semantic placement
Marker into a cup
A compact open-world placement behavior from a direct language instruction.
02Tool trajectory
Draw a square around the ball
A geometric drawing request becomes a constrained waypoint trajectory.
03Instruction ambiguity
A different interpretation
Given “Fix the typo,” GTA-2 arranges the loose letters into “TEACH” instead of correcting the word we had in mind. Well, that’s technically not incorrect…
04Precision placement
Ping-pong ball into a bowl
A small-object grasp and placement behavior with a narrow target region.
05Semantic grounding