ACE-CaP-0: Real-World Code-as-Policies in the Era of Experience

Abstract

Recent advances in frontier models have pushed the boundaries of agentic robot control, enabling agents to perform complex robotic tasks through reasoning and interaction. However, repeated calls to these models during execution can substantially increase task completion time. Code as Policies (CaP) offers a route for coding agents to acquire physical skills through experience and retain them in executable programs, allowing the reasoning invested in development to be reused across runs. Developing reliable code policies while limiting the cost of physical experimentation remains a central challenge. We investigate human guidance as a source of priors for feasible actions and simulation as an auxiliary environment for exploring and testing candidate policies. Our findings favor robot demonstrations available on request and simulation used selectively during real-world policy refinement. These findings inform ACE-CaP-0, an agentic recipe that couples policy revision with experience acquisition through Demonstrations on Request and Simulation on Demand. Across eight real-world manipulation tasks spanning alignment, multi-object handling, and bimanual coordination, the resulting frozen programs achieve 92.1% success (221/240 trials). Mean execution times across tasks range from 0.97 to 2.27 minutes. The programs execute with runtime perception and control, without a human or coding agent in the loop. These results highlight CaP’s potential to turn experience into reliable, reusable physical skills.

What experience to acquire, and when

A study on charger insertion motivates the recipe. Six development conditions, one development run each, address three questions: what form of guidance to provide, when to provide it, and how to bring simulation into exploration. All use the same foundation model, task instruction, and interfaces, with at most ten human-assisted scene resets. Each resulting program is frozen and run for 30 real-world trials.

What form of guidance?

Three forms of assistance, each available when the agent asks for it, in development without simulation. Robot demonstrations give the highest final success, and both demonstration conditions take less development time and fewer output tokens than text alone.

Guidance form for charger insertion. Textual guidance: 60% final real-world success, 5.9 hours of development, 0.86 million output tokens. Robot-egocentric human video: 73.3%, 3.8 hours, 0.75 million. Robot demonstration with video, trajectory and key joint pose: 86.7%, 3.7 hours, 0.73 million. — Guidance form (paper, Fig. 2). Final success rate over 30 real-world trials of each frozen program, development time, and output token usage.

Textual guidance
60% 5.9 h, 0.86 M tokens
Advice about manipulation strategies, action constraints, and corrections. No demonstration.
Robot-egocentric human video
73.3% 3.8 h, 0.75 M tokens
Text, plus videos of a person performing the task with their hands, recorded from the robot's execution viewpoint.
Robot demonstrations
86.7% 3.7 h, 0.73 M tokens
Text, plus kinesthetic demonstrations that provide video, arm trajectories, and key joint poses.

When to provide it?

The same textual guidance and robot demonstrations, handed over at the outset or requested by the agent. On request, final success is 86.7% against 53.3%, development time falls from 7.2 to 3.7 hours, and output tokens from 1.16 to 0.73 million.

Guidance timing for charger insertion. Upfront guidance: 53.3% final real-world success, 7.2 hours of development, 1.16 million output tokens. On-request guidance: 86.7%, 3.7 hours, 0.73 million. — Guidance timing (paper, Fig. 3). On-request guidance is the same development run as robot demonstrations above.

Upfront guidance
53.3% 7.2 h, 1.16 M tokens
The agent receives textual guidance and robot demonstrations at the start of development.
On-request guidance
86.7% 3.7 h, 0.73 M tokens
The agent decides whether and when to ask, before its first physical attempt or during later revision.

How should simulation come in?

Simulation as a separate stage before transfer, or as a tool the agent calls on during real-world development. Simulation on demand reaches 100% final success against 86.7% for real-only development, in 2.6 rather than 3.7 hours, with more output tokens.

Simulation protocols for charger insertion. Real-only: 86.7% final real-world success, 3.7 hours of development, 0.73 million output tokens. Sequential real-sim-real: 0%, 3.5 hours, 1.86 million. Simulation on demand: 100%, 2.6 hours, 1.09 million. — Simulation protocols (paper, Fig. 4). Real-only is the same development run as robot demonstrations and on-request guidance above.

Real-only
86.7% 3.7 h, 0.73 M tokens
Development through real-world execution and feedback, without simulation. Text and robot demonstrations on request.
Sequential real-sim-real
0% 3.5 h, 1.86 M tokens
Demonstrations first, then a simulation built from the real setup and ten rounds of simulated exploration in four parallel environments, then transfer. The transferred program fails at grasping the plug: it was refined against cleaner simulated images and depth without the real sensor's close-range blind regions.
Simulation on demand
100% 2.6 h, 1.09 M tokens
Development stays in the real-world loop. The agent decides whether and when to simulate, which task segment and initial state to examine, and which revisions to test. Text and robot demonstrations on request.

Percentages are final real-world success over 30 trials of the frozen program. Development time is elapsed hours, including model calls, simulated experiments, and real-world interaction; tokens are output tokens across all model calls during development. Each condition is a single development run on one task, so the 30 trials measure the reliability of the resulting program, while the repeatability of the development process remains to be established.

One development loop, two sources of experience

The working artifact is a program. The agent starts from a task instruction, observations of the workspace, and the robot's perception and control interfaces, and writes code that specifies action sequences, geometric computations, motion parameters, and gripper commands.

At each step it weighs the current program against the evidence it has and the question that is still open. It can revise the code, request a demonstration, organize a simulated experiment, or run a candidate on the robot. Neither demonstrations nor simulation is required in every iteration.

Diagram of ACE-CaP-0. Policy development: the coding agent develops an evolving code policy through a shared robot toolbox and real-world interaction, requesting demonstrations and simulation as needed. Policy execution: the frozen code policy drives the robot with no human or coding agent in the loop. — From policy development to standalone execution (paper, Fig. 5). During development the agent writes and revises the policy using real-world feedback, through a robot toolbox it also uses for direct exploration; it can request kinesthetic demonstrations or test candidate code in simulation for a chosen task segment and initial state. The resulting policy is frozen and executes without a human or coding agent in the loop.

Demonstrations on request

The agent asks for a demonstration when it judges that seeing how a person performs the task with the robot would help it build or revise the policy. A person physically guides the arms through the task; the agent receives video, recorded arm trajectories, and key joint poses that capture the arm configurations at critical stages, and reads them together. The demonstrated action order and arm roles inform the program's structure; approach directions, motion paths, and key joint poses inform individual actions.

From a charger-insertion development log

Messages are translated from the Chinese originals and abridged. In this session the protocol had the agent begin with one whole-task demonstration; the later question was the agent's own decision.

Agent asks
Please guide the robot arm by hand through the whole task once: insert the white charger on the table into the power strip. Do it the way you think is best. Please note how you grasp the charger so that the pins end up pointing down, and push down until its bottom sits flat on the strip. Pause for a second or two at the key positions (before the grasp, after the grasp, when aligned with the socket, after insertion) so that I can find the key poses in the recording, and open and close the gripper as you go.
Person demonstrates

Guiding the left arm to the strip

Pushing in, hand squeezing the gripper fingers

Fingers closed, pressing the charger's top

End state

What the agent receives besides video: the recorded left-arm fingertip height and gripper opening of this demonstration. The closing press, with the gripper shut, is the step the agent first overlooked.
Agent reads the recording
during the held push the operator's hands PINCHED the gripper fingers (much more squeeze than the robot's 2 N); after release the charger stood deep; the closed fingertips then visibly pressed its top

From the agent's own notes, in English in the original.
Agent asks again, after three failed insertions
I have now failed this step three times. Holding the charger, I align it with the slots and push down, but the pins go in only 5–8 mm and then stop: the gripper squeezes with about 2 N at most, and pushing further makes the charger slide up between the fingers. As soon as I open the gripper the charger springs out and falls over. How did you manage it in your demonstration? A. Release as soon as the pins enter, then press the top with closed fingertips. B. Push the pins in deep before releasing (how deep?). C. Support the charger with the other hand. D. Something else.
Person answers
B. Push in deep first: until the gripper itself bottoms out, rather than the pins, which is hard and may not succeed. Only then release.

What ends up in the program

def release_and_press(ctx, arm, xy, z_table, stroke_mm):
    """Let go of the charger standing in the socket at XY (the table at Z_TABLE) without touching it, then with the
    closed fingertips find its top and press it home (the 17:02 demo's last step), and touch it again to check.

    Pressed up to PRESS_TRIES times until a light touch finds the top at the seated height (operator 11-11 00:03:
    用合拢的夹爪从充电器顶上竖直往下用力压到底). Returns {'ok', 'seated', 'touch' (first touch over the table),
    'after' (last touch), 'presses'}."""
    gripper(ctx, arm, max(stroke_mm - RELAX_BELOW_MM, 0))
    gripper(ctx, arm, RELEASE_MM)
    straight(ctx, arm, [xy[0], xy[1], z_table + CLEAR_ABOVE_TABLE], speed=LIFT_SPEED)
    gripper(ctx, arm, 0)
    t = touch_top(ctx, arm, xy, z_table)
    ...

Agent-written code from the same session, unedited apart from truncation. The docstring cites the demonstration and the operator's advice (the Chinese reads: "with the gripper closed, press straight down hard on the charger's top until it is fully in").

Simulation on demand

A simulated experiment starts from a development question: does an approach path avoid an obstruction, how does a motion parameter change the interaction, in what order should coordinated actions run. The agent chooses the task segment, the initial state, and the candidate changes to test, then interprets the outcome against the question that motivated it. The conclusion applies to the behavior and conditions that experiment examined; physical execution then tests the revision on the robot.

From a parcel-reorientation development log

One evening, two simulated experiments, each answered in about ten minutes. Messages are translated from the Chinese originals and abridged.

On the robot
The parcel lies very close to the robot, rotated 45°. The program finds no start pose that both hands can reach and stops without moving.
Agent revises the code and asks for an experiment
Scene: the box lies close to the robot, between the arms, rotated 45°, label not facing up. The program should push the box outwards before turning it, but before the push it finds no grasp both hands can reach (28 candidates rejected), so it stops. Change: when sliding or pushing, also try contact points offset by 3 and 4.5 cm with the hand tilted 15° or 20°; if one pair of faces fails, try the other pair. Please use this box position and angle (x 0.24–0.27 m, 42°–48°, both directions), run the whole flow five times, and check whether the push, turn, flip and pull-back all succeed.
In simulation

One of the five initial states in the twin

The same state after the final revision: label up

Five initial states run in four parallel digital twins, with the candidate code unchanged and no agent in the loop. The previous code reproduces the real stop. The revision succeeds in four of five; in the fifth the backend refuses a two-arm straight-line move, because the right arm cannot reach the pose 77% of the way along the line.
On the robot
The next real run meets the same refusal, 65% of the way along the line.
Agent finds the cause
The right wrist's last joint sits within a few hundredths of a radian of its limit at the pre-pose, while the start pose was solved with the opposite half-turn, so the straight line between them runs into the limit. The fix rejects candidates whose waypoints mix half-turn solutions. Replaying the planner on 674 box measurements recorded on the robot leaves 672 plans unchanged; the other two are this scene.
In simulation
The same five states again. The previous code reproduces the refusal; the fix succeeds in five of five.
On the robot
The next real run finds a start pose, pushes the parcel 9.3 cm and turns it, with no refusal. A flip check later in that run failed for an unrelated reason and was revised separately.

Real top-down camera view: a clear bottle with a white cap on a white table. — Real camera

Digital-twin rendering of the same scene: the bottle at the same place and tilt. — Digital twin

The same initial state of bottle uncapping, seen by the real camera and rendered in the twin. The twins mirror the station's kinematics, camera poses, fixed obstacles and safety checks. That makes them good at questions of geometry and control flow like the one above; the reliability of contact, such as a grasp or an insertion, is left to the robot.

The policy is a program

At the end of development the agent produces a condensed code policy that retains the task logic and execution details worked out through interaction, expressed in its action structure, geometric computations, and motion parameters. Once frozen, its decision rules stay fixed, while runtime observations still change the values those rules compute and the actions that result. It executes without a human or coding agent in the loop.

g = M.grasp_standing(ctx, arm, t['xy'], t['z_top'], t['close_yaw'], CARRY_Z)
run.log('grasp', **g)
if not g['ok']:
    to_safe(ctx, arm)
    return run.report(False, 'grasp: ' + g['why'])

c = M.carry_level(ctx, arm, target['xy'], (CARRY_Z, CARRY_LOW_Z), slot_yaw)
run.log('carry', **c)
if not c['ok']:
    M.straight(ctx, arm, [t['xy'][0], t['xy'][1], g['z_grasp'] + 0.002], speed=0.3)
    return run.report(False, 'carry: ' + c['why'] + ' (charger held back at its start)')

z_table = table_z(plane, target['xy'][0], target['xy'][1])
ins = M.find_slots(ctx, arm, target['xy'], s['axis'], z_table, g['top_above_tip'], log=run.log)
run.log('insert', **{k: v for k, v in ins.items() if k != 'touches'})
if not ins['ok']:
    M.straight(ctx, arm, c['tip'], speed=0.4)
    return run.report(False, 'insert: ' + ins['why'] + ' (charger still held above the strip)')
xy = ins['xy']
# operator 23:35: straighten it, then push hard to the bottom (holding)
st = M.straighten(ctx, arm, g['stroke_mm'])
push = M.guarded(ctx, arm, [xy[0], xy[1], z_table + M.FLOOR_ABOVE_TABLE], M.PUSH_RISE_A, speed=M.TOUCH_SPEED)
run.log('push', stroke_mm=st['stroke_mm'], z=round(push['tip'][2] - z_table, 4), fired=push['fired'],
        why=push['why'], largest=push['largest'])

p = M.release_and_press(ctx, arm, xy, z_table, st['stroke_mm'])
run.log('press', **p)
M.straight(ctx, arm, [xy[0], xy[1], z_table + M.CLEAR_ABOVE_TABLE], speed=0.4)
fid = V.observe(ctx, ['front', 'wrist_right', 'wrist_' + arm])
run.log('final_view', frame=fid)
if not p['ok']:
    ctx.value('piper_park')
    return run.report(False, 'press: ' + p['why'], frame=fid, press=p)
M.gripper(ctx, arm, 0)
ctx.value('piper_park')
return run.report(True, socket=xy, kind=target['kind'], press=p, frame=fid)

The main sequence of an agent-written charger-insertion program from a development session: grasp, carry, feel for the slots, straighten, push, release and press, look, report. Two debugging early-exits are omitted.

Eight tasks

The tasks span alignment, object reorientation, multi-object handling, and bimanual coordination. They include inserting a charger into a socket, removing a bottle cap while stabilizing the bottle, and placing a pair of shoes into a box.

The dual-arm platform: two Agilex PIPER X arms on a tabletop, a camera on a mast between them and a camera on each wrist, beside representative scenes of the eight tasks. — Platform and tasks (paper, Fig. 6): two Agilex PIPER X arms and three Orbbec DaBai DC1 cameras, giving one top-down view and two wrist-mounted views.

221 of 240 real-world trials succeed

Each program is developed with at most ten human-assisted scene resets, then frozen and run for 30 trials that are separate from development. Seven of the eight tasks reach at least 90%.

Success rate over 30 evaluation trials, mean physical execution time of the frozen program, and output tokens generated across all model calls during policy development (paper, Table 1 and Fig. 7).

Charger Insertion succeeds in all 30 trials and Chicken Plating in 29. Bread Packing, at 25 of 30, is the one task below 90%. Development uses between 0.75 and 2.14 million output tokens per task.

Mean physical execution times range from 0.97 to 2.27 minutes, and six of the eight tasks average less than two minutes. Parcel Reorientation is the shortest at 0.97 minutes; Bottle Uncapping and Shoe Packing are the longest at 2.23 and 2.27 minutes, both with 90% success.

Contributions and author list

Core contributors: Changxu Wei^1,2, Yunpeng Qing^1,3, Guozheng Ma^1,4
Contributors: Bowen Liu¹, Zexin He¹, Yilun Kong¹, Haoming Gu¹, Jingyuan Liu¹, Tianyu Zhu¹, Jianfei Wang¹, Junqiang Kou¹, Huinan Mo¹, Hao Lv¹, Pengbin Li¹, Yuesong Wang¹, Bo Li¹, Huan Zhou¹, Shunyu Liu⁴, Chunxiao Liu¹, Bixuan Jin¹, Shaopeng Yin¹, Fei Wang¹
Advisors: Zhi Hou^1,†, Jiaxing Huang², Changqing Zou³, Dacheng Tao⁴, Xiaogang Wang¹
Affiliations: ¹ACE Robotics, ²The Hong Kong Polytechnic University, ³Zhejiang University, ⁴Nanyang Technological University. ^†Project leader.

Cite

@misc{acecap0,
  title  = {ACE-CaP-0: Real-World Code-as-Policies in the Era of Experience},
  author = {{ACE-CaP Team} and Wei, Changxu and Qing, Yunpeng and
            Ma, Guozheng and Liu, Bowen and He, Zexin and Kong, Yilun and
            Gu, Haoming and Liu, Jingyuan and Zhu, Tianyu and
            Wang, Jianfei and Kou, Junqiang and Mo, Huinan and Lv, Hao and
            Li, Pengbin and Wang, Yuesong and Li, Bo and Zhou, Huan and
            Liu, Shunyu and Liu, Chunxiao and Jin, Bixuan and
            Yin, Shaopeng and Wang, Fei and Hou, Zhi and Huang, Jiaxing and
            Zou, Changqing and Tao, Dacheng and Wang, Xiaogang},
  year   = {2026},
  note   = {Project page: https://acerobotics2025.github.io/ACE-CaP/}
}