Tasks and Scoring

Task definition: what the robot is asked to do, how the human operator sets the scene, and how an episode is scored.

The station shows description and reset_instruction to the operator, hands instruction to the policy, and presents scoring to the operator after every episode.

Task_ScoreKind

class Task_ScoreKind(Enum)

Attributes

Binary = 'binary'

Answered yes / no. Contributes weight when yes, 0 when no.

Choice = 'choice'

Answered by picking exactly one of the item's choices. Contributes that choice's score.

Task_ScoreChoice

Task_ScoreChoice(score: float, description: str)

Attributes

score: float

Points this choice earns. Not necessarily unique across the item's choices: several judgements may be worth the same, e.g. [2, 1, 1, 1, 1, 0].

description: str

The exact judgement this choice stands for, shown to the operator. Unique within the item.

Task_ScoreItem

Task_ScoreItem(
    key: str,
    prompt: str,
    kind: Task_ScoreKind,
    choices: tuple[Task_ScoreChoice, ...] | None = None,
    weight: float | None = None,
)

Attributes

key: str

Stable identifier used as the answer key, e.g. "towel_folded".

prompt: str

Question shown to the operator, e.g. "Is the towel folded in half with edges aligned?".

kind: Task_ScoreKind

choices: tuple[Task_ScoreChoice, ...] | None = None

The options for Choice, in the order they are presented; required then, None for Binary.

weight: float | None = None

Binary only: points when yes (default 1). 0 records the item without scoring it. A Choice item carries its points in its choices and must leave this None.

max_points: float

Read-only property. The most this item can contribute.

Methods

contribution(answer: bool | int) -> float

Points earned for answer: a bool for a binary item, a choice index for a choice item.

Task_ScoreAnswers

Task_ScoreAnswers: tp.TypeAlias = dict[str, bool | int]

Operator answers keyed by Task_ScoreItem.key: bool for a binary item, the 0-based index of the chosen option for a choice item. Every item of the sheet must be answered.

Task_ScoreSheet

Task_ScoreSheet(success: Task_ScoreItem, items: tuple[Task_ScoreItem, ...] = ())

Attributes

success: Task_ScoreItem

The binary item that decides whether the episode passed. Always present, always asked first.

items: tuple[Task_ScoreItem, ...] = ()

Further criteria presented to the operator after success, in this order.

all_items: tuple[Task_ScoreItem, ...]

Read-only property.

max_total: float

Read-only property.

Methods

total(answers: Task_ScoreAnswers) -> float

Weighted points earned, in [0, max_total].

normalized(answers: Task_ScoreAnswers) -> float

total / max_total in [0, 1]; 0 if no item carries weight.

is_success(answers: Task_ScoreAnswers) -> bool

The answer to the success item.

Task_Spec

Task_Spec(
    name: str,
    description: str,
    reset_instruction: str,
    scoring: Task_ScoreSheet,
    max_rollout_time: float,
    instruction: str | None = None,
)

Attributes

name: str

Stable task identifier, e.g. "fold_towel".

description: str

Short human-readable summary shown to the operator.

reset_instruction: str

Step-by-step instructions for the operator to set up the scene before an episode.

scoring: Task_ScoreSheet

max_rollout_time: float

Wall-clock budget for one episode from the first policy inference, unit: [s]. The station stops the episode and asks the operator to score it as it stands when this elapses.

instruction: str | None = None

Natural-language command handed to the policy. None for tasks that are not language-conditioned.