Skip to main content

Agentic Control

Before you start, complete the station setup.

ACT Harness exposes arm motion, grippers, and cameras as shell commands and Python tools. Use it to let an agent observe the scene, choose a motion, and inspect the result on a YAM station.

Define the episode sequence in my_eval.py. For example, sample twenty Plug In Charger setups from the task registry:

my_eval.py
from se3labs.sdk.eval.tasks import TaskRegistry

TASKS = TaskRegistry.get_task("ROBODOJO-REAL-20").sample_set(seed=42, episodes=20)

Each entry defines one episode, with its own instruction, layout, scoring, and time limit. You can also supply a list or tuple of custom task specs. Start the harness from that file's directory and leave it running:

python -m se3labs.sdk.examples.act_harness serve \
  --station example-station-id --address example-station-addr \
  --tasks my_eval:TASKS --no-keyboard

Run the agent on the same machine so it can access the harness's local socket and camera files. Give it the bundled instructions by printing them into its context, or installing them into its skills directory:

python -m se3labs.sdk.examples.act_harness skill

For example, to install the instructions for Claude Code:

python -m se3labs.sdk.examples.act_harness skill --install .claude/skills

The agent can then use these commands from another terminal:

python -m se3labs.sdk.examples.act_harness wait
python -m se3labs.sdk.examples.act_harness send status
python -m se3labs.sdk.examples.act_harness cameras --out ./frames
python -m se3labs.sdk.examples.act_harness send move left 0 0.05 0
python -m se3labs.sdk.examples.act_harness end-episode

wait waits for a live rollout before sending motion. Each action returns JSON; the agent should check ok and inspect the camera paths returned by cameras. In this example, move left 0 0.05 0 moves the left arm 5 cm along its base-frame Y axis. Translations use metres; rotations use degrees.

For an agent with a Python tool loop, the same interface is available through Harness, tool_definitions(), and call_tool(). Pass the definitions to your agent and dispatch its chosen tool name and argument dictionary with call_tool(harness, name, arguments). See the ACT Harness reference for the available tools and motion limits.

end-episode requests scoring, reset, and homing. The harness stays running between station jobs and reuses the supplied sequence for each job; use Ctrl+C in its terminal to stop serving. Scores and progress are available in the station dashboard.

Continue with reading the evaluation results.