RoboDojo - Real
note
SE3 Labs real-world instantiation of 10 RoboDojo simulation tasks. For each task, the official language instruction and scoring criteria are preserved, then reconciled with the photographed SE3 materials. Real-world adaptations are explicitly labeled; they are not claims about the original benchmark.
Official RoboDojo simulation-task catalog · Official repository · SE3 scoring-sheet reference · Source check: October 13, 2026
Tasks
Generalization
Memory
Precision
Long-Horizon
Instruction-Conditioned
Scoring-sheet model
note
Each task is scored on a milestone ladder extended from RoboDojo's, judged on the final state of the episode. Extra early rungs credit grasps, transports, and per-object placements, so partial progress is visible. The failure mode and execution metrics are recorded for diagnosis and do not change the score.
- Progress score: the highest milestone whose condition holds at the end of the episode, or 0. Because the ladders add rungs, mean progress is not directly comparable to RoboDojo's.
- Success: a separate Yes/No answer confirming whether all requirements for the 100-point milestone were satisfied.
- Failure mode: one of none · missed grasp · drop · wrong object or target · placement or insertion miss · knocked over or disturbed · stalled or timed out · no return to origin · safety stop.
- Execution metrics: per-task counts of observable events, such as failed grasps, drops, and wrong placements, counted over the whole episode.
- Protocol: every task starts at zero joint angles with open grippers and has a 180-second time limit. Run at least 20 episodes per task per policy from the same recorded set of start layouts, with the operator blind to the policy.
- Reporting: success rate with a 95% Wilson interval, mean progress, the share of episodes reaching each milestone, the failure-mode histogram, and the mean of each execution metric.
Material-difference register
| Task | RoboDojo simulation | SE3 Labs real setup | Required adaptation |
|---|---|---|---|
| Pack Objects Into Box | 4 generic objects + aligned box | Car, shoe, toothbrush, dual-face hammer + cardboard box | Define each object's “front” operationally |
| Hang Mugs | 3 mugs | 2 ceramic mugs + wooden tree | Normalize milestones to 2 mugs |
| Cover Blocks | Red, green, blue blocks | Green, light-blue, dark-blue blocks + identical yellow cups | Change recall order; randomize block positions |
| Press By Number | 2 red count buttons + blue confirm | Green/cyan count buttons + dark-blue confirm; cards show 9 and 1 | Bind roles by position, not simulated colors |
| Plug In Charger | Rigid simulated charger and strip | USB wall adapter + multi-outlet strip with flexible cord | Mark target socket; keep strip de-energized |
| Insert Tubes | 3 tubes | 2 capped specimen tubes + multi-slot rack | Use 2 marked target wells; photo shows terminal state |
| Classify Objects | 3 abstract categories | Writing tools, toy vehicle, camera equipment | Freeze a ground-truth taxonomy |
| Play Stacking Toy | 4 pegs; groups of 4, 3, 2, 1 | 5 pegs, each seeded with 1 starter piece; 4 table pieces | Place 1 piece per group onto its seeded peg |
| Align Blocks | Set square + 3 blocks | Clear acrylic ruler + 3 purple cubes | Use ruler as pushing tool; define an alignment tolerance |
| Classify Objects By Language | 3 unseen categories + 3 baskets | Peppers, figurines, watches + blue/orange/yellow bins | Parameterize category-to-bin mapping in the prompt |