Classify Objects By Language

RoboDojo simulation reference

SE3 Labs real setup

Task ID
SE3 language instruction	“Put the into the left blue bin, the into the middle orange bin, and the into the right yellow bin.”
Official language instruction	Put category₁ objects into the left basket, category₂ objects into the middle basket, and category₃ objects into the right basket, then reset the robot arm.
RoboDojo category	Instruction-Conditioned
Materials / ground truth	toy bell peppers = red, green, yellow; figurines = two small figures; watches = red-band analog, yellow-band analog, black-band wearable. Bins are blue (left), orange (middle), and yellow (right).
Difference	class membership and bin assignment are expressed only by language; bin color is not a semantic shortcut.
RoboDojo task page	Classify Objects By Language

Progress score

Score the highest milestone whose condition holds at the end of the episode; the episode succeeds at 100. Also record the failure mode and the execution metrics below.

An object is sorted when it is in its instructed bin. A category counts when all of its objects are in the instructed bin and that bin holds nothing else.

Progress	Condition
0	No milestone below is reached.
5	One object is in its instructed bin and the gripper is open.
15	Two objects are in their instructed bins.
25	Three objects are in their instructed bins.
40	One category counts.
70	Two categories count.
100	All three categories count and all eight objects are assigned. Both arms are back at origin.

Execution metrics

Record these alongside progress; they do not change it. Count over the whole episode, including events later corrected.

Metric	What to count	Answer
Failed grasps	Grasps that close on or around a task object without lifting it clear of the table.	0 · 1 · 2 · 3+
Drops	Held objects that fall before they are released at a target.	0 · 1 · 2 · 3+
Wrong-bin placements	Objects released into a bin other than the instructed one.	0 · 1 · 2 · 3+
Objects moved out	Objects taken out of a bin after being released there.	0 · 1 · 2 · 3+

Randomization

Each episode samples every variable below and records its values.

Variable	Domain
, ,	Any order of toy peppers, figurines, and watches; balance the six orders across episodes.
Start layout	Bins stay blue, orange, yellow from left to right, each with a random position in its third and yaw within ±10°; object positions and yaw anywhere, clear of the arms.

Schematic start layout, not to scale. Hover a shape for details.

Python SDK

from se3labs.sdk.eval.tasks import TaskRegistry

definition = TaskRegistry.get_task("ROBODOJO-REAL-39")
# Sample one episode's parameters and initial layout.
task = definition.sample(seed=42)
# Sample 20 episodes with separate seeds; seed=42 reproduces the same set.
tasks = definition.sample_set(seed=42, episodes=20)

Use definition.sample_set(seed=42, episodes=24) for balanced coverage: each of the six category assignments appears four times.

Create and run a job:

from se3labs.sdk import eval_client

job = TaskRegistry.get_job(
    "ROBODOJO-REAL-39", policy=policy, seed=42, episodes=20,
)
report = eval_client.serve_policy("your-station", "your-relay:7443", job)

task.layout.visualize("layout.png") saves the sampled layout.