pepijn223/act_openarm_screw_picking_v33a_real

ACT: OpenArm screw picking (v33a-real)

LeRobot ACTPolicy for bimanual OpenArm screw picking: "Pick up the bolts and screws and place them into the red container." Trained on 61 real teleoperated episodes plus 91 Genesis sim episodes in the eval-scene layout. The run used 152 episodes and 143,275 frames, for 20 epochs (22,380 steps at batch size 128) on one H100.

Status: experimental. In the Genesis sim eval (60 s, fixed long-edge-table scene), this checkpoint placed 0 of 4 screws with 113 contacts. It has not been tested on the real robot.

Inputs

key	shape	notes
observation.images.head	3×270×480	head camera, RGB float in [0,1], then ImageNet mean/std normalization
observation.images.gripper_a	3×270×480	left wrist camera, same normalization
observation.images.gripper_b	3×270×480	right wrist camera, same normalization
observation.state	128	motion history (see below), normalized with hist_mean / hist_std from action_stats.npz

Motion history. These are the 8 executed actions of the last 8-step segment, flattened to 8×16. Each row is the measured camera pose after that action, expressed relative to the measured pose at the segment start. The two jaw slots (7 and 15) hold the commanded openings. Before anything has executed, use 8 identity rows: zero translation, quaternion (1,0,0,0), and openings = init_opening. serve_act.py (ACTSession.acknowledge) shows exactly how it is built.

Output

The policy predicts a chunk of 32×16 normalized actions. Denormalize them with mean and std from action_stats.npz. Each row holds, per arm: camera-pose xyz, quaternion wxyz, and gripper opening (0 = closed). Arm a uses columns 0–7, arm b columns 8–15.

Actions are chained-relative per 8-step segment. For segment s (actions 8s…8s+7), measure the wrist-camera pose at the start of the segment, then compose each relative action onto it: target = measured_segment_start ∘ action. Use wam_session.chained_targets for this, and replay_core.relative_actions for the inverse. Run the actions at 25 Hz. Replanning every 16 to 32 actions worked best in sim.

Files

• config.json, model.safetensors: LeRobot ACT weights; load with ACTPolicy.from_pretrained.
• action_stats.npz: action mean/std, hist_mean/hist_std, init_opening.
• serve_act.py, act_common.py: the inference session used in eval (normalization, history tracking, denormalization).
• train_meta.json: training configuration.

Caveats

• The sim episodes in this run use next-measured-pose targets. A later fix switched to commanded-pose targets, because under per-segment re-anchoring the measured targets under-shoot by about 1 cm. Real-episode targets come from the recorded robot data.
• Camera intrinsics, crops, and the camera↔TCP extrinsics must match the training rig.