se3labs.interface.eval.taskTasks and Scoring
Task definition: what the robot is asked to do, how the human operator sets the scene, and how an episode is scored.
The station shows description and reset_instruction to the operator, hands instruction to
the policy, and presents scoring to the operator after every episode.
Task_ParameterValue
Task_ParameterValue: tp.TypeAlias = str | int | float | tuple[str | int | float, ...]
Task_ScoreKind
class Task_ScoreKind(Enum)
Attributes
Binary = 'binary'#-
Answered yes / no. Contributes
weightwhen yes,0when no. Choice = 'choice'#-
Answered by picking exactly one of the item's
choices. Contributes that choice'sscore. Count = 'count'#-
An exact nonnegative integer observation. Recorded without contributing points.
Task_ScoreChoice
Task_ScoreChoice(score: float, description: str)
Attributes
score: float#-
Points this choice earns. Not necessarily unique across the item's choices: several judgements may be worth the same, e.g.
[2, 1, 1, 1, 1, 0]. description: str#-
The exact judgement this choice stands for, shown to the operator. Unique within the item.
Task_ScoreItem
Task_ScoreItem(
key: str,
prompt: str,
weight: float | None = None,
)
Attributes
key: str#-
Stable identifier used as the answer key, e.g.
"towel_folded". prompt: str#-
Question shown to the operator, e.g.
"Is the towel folded in half with edges aligned?". kind: Task_ScoreKind#choices: tuple[Task_ScoreChoice, ...] | None = None#-
The options for
Choice, in presentation order;Nonefor other kinds. weight: float | None = None#-
Binaryonly: points when yes (default1).0records the item without scoring it. Other kinds must leave thisNone. max_points: float#-
Read-only property. The most this item can contribute.
Methods
contribution(answer: bool | int) -> float#-
Points for a binary answer or choice index; counts contribute zero.
Task_ScoreAnswers
Task_ScoreAnswers: tp.TypeAlias = dict[str, bool | int]
Operator answers keyed by Task_ScoreItem.key: bool for binary, a 0-based option index
for choice, or a nonnegative integer for count. Every item of the sheet must be answered.
Task_ScoreSheet
Attributes
success: Task_ScoreItem#-
The binary item deciding success, asked first. Yes means success.
items: tuple[Task_ScoreItem, ...] = ()#-
Further criteria presented to the operator after
success, in this order. all_items: tuple[Task_ScoreItem, ...]#-
Read-only property.
max_total: float#-
Read-only property.
Methods
total(answers: Task_ScoreAnswers) -> float#-
Weighted points earned, in
[0, max_total]. normalized(answers: Task_ScoreAnswers) -> float#-
total / max_totalin[0, 1];0if no item carries weight. is_success(answers: Task_ScoreAnswers) -> bool#-
Whether the operator answered yes to the binary success item.
Task_Spec
Task_Spec(
name: str,
description: str,
reset_instruction: str,
max_rollout_time: float,
instruction: str | None = None,
seed: int | None = None,
)
Attributes
name: str#-
Stable task identifier, e.g.
"fold_towel". description: str#-
Short human-readable summary shown to the operator.
reset_instruction: str#-
Step-by-step instructions for the operator to set up the scene before an episode.
scoring: Task_ScoreSheet#max_rollout_time: float#-
Wall-clock budget for one episode from the first policy inference, unit: [s]. The station stops the episode and asks the operator to score it as it stands when this elapses.
instruction: str | None = None#-
Natural-language command handed to the policy.
Nonefor tasks that are not language-conditioned. layout: LayoutSample | None = None#-
Concrete initial object poses for the operator to reproduce.
seed: int | None = None#-
Seed used to realize this task, when sampled from a definition.
parameters: dict[str, Task_ParameterValue] = field(default_factory=dict)#-
Realized semantic variables and calibration values, recorded with the episode.