World Action Agent: Leveraging VLMs for Robotic Manipulation through World Action Rehearsal

Meta description
World Action Agent uses visual action workspaces, rehearsal and skill transfer to reach 75.6% average success on LIBERO-Pro robot manipulation tasks.

Tags
Vision-Language Models, Robot Manipulation, World Action Agent, LIBERO-Pro, Qwen3.5-9B

Featured image
Abstract technology style, no real people, no company logos, no text.

World Action Agent, or WAA, is a multi-agent harness through which general-purpose vision-language models, VLMs, pilot robots with basic tools, making every decision within a visual action workspace. The design matters because it reframes how VLMs are used: instead of being asked only to produce constraints or code from a camera view, they operate inside a workspace that shows contact views, rehearses actions and corrects what is seen before execution. The latest reported result is a 75.6% average success rate on LIBERO-Pro when WAA uses skills evolved only from LIBERO-90.

Background: VLMs Move From Viewing Scenes to Acting in Them

General-purpose vision-language models bring broad knowledge and spatial reasoning to robot manipulation. Yet existing systems have used them in two limiting ways. Some use them indirectly, to predict constraints or write programs. Others give them a view of the scene rather than a world in which to act. The gap is practical: a good description of a scene does not automatically give a robot a place to decide, test, correct and execute. That distinction is what the new work targets. The practical consequence is that strong visual reasoning can be wasted if the robot cannot test an action, see its effect and correct it before committing.

Development Process: Building the Visual Action Workspace

WAA creates that place through three workspace properties. Contact views are selected automatically from scene geometry and present the scene around the current interaction. Action rehearsal turns each action into an editable proposal. The agent, alone or through an Imagination Agent, previews and revises that proposal against planning feedback before execution. In-view correction closes the loop between observation, rehearsal and low-level execution, letting the agent remove residual offsets in the view where it observes them. Together, the three parts keep the VLM inside a loop of seeing, trying, fixing and acting.

Latest Progress: Skills, Transfer and Smaller Pilots

Through the same workspace, WAA acquires embodied procedural knowledge in two ways. It evolves multimodal skills from expert videos and human teaching under evidence-based review, then consults them through a Skill Agent. It also uses its interaction traces to train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLA agents, code-as-policy agents and a visual-harness baseline with the same backbone. The same skills remain effective on RoboSuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.

Current Stage: A Harness, Not a Single Model

These numbers place WAA at a distinct stage of VLM robot manipulation. It is not presented as one model that replaces low-level control, but as a harness that organizes perception, rehearsal, correction and skill use around VLMs. The LIBERO-Pro score shows what the workspace can do when skills come only from LIBERO-90. The RoboSuite transfer suggests the skill library is not narrowly tied to one evaluation suite, and the Qwen3.5-9B jump from 1.7% to 43.3% shows harness traces can help smaller VLMs pilot the system. For readers following robot learning, the signal is that harness design can compete with model size alone. The next question is how far this visual action workspace can carry when tasks demand more varied contact geometry, longer sequences and broader physical experience.