Skip to main content

Tasks and scoring

Module: se3labs.interface.eval.task

Task definition: what the robot is asked to do, how the human operator sets the scene, and how an episode is scored.

The station shows description and reset_instruction to the operator, hands instruction to the policy, and presents scoring to the operator after every episode.

Task_ScoreKind​

class Task_ScoreKind(Enum)

Task_ScoreKind.Binary​

Binary = 'binary'

Answered yes / no. Contributes weight when yes, 0 when no.

Task_ScoreKind.Choice​

Choice = 'choice'

Answered by picking exactly one of the item's choices. Contributes that choice's score.

Task_ScoreChoice​

Task_ScoreChoice(score: float, description: str)

Task_ScoreChoice.score​

score: float

Points this choice earns. Not necessarily unique across the item's choices: several judgements may be worth the same, e.g. [2, 1, 1, 1, 1, 0].

Task_ScoreChoice.description​

description: str

The exact judgement this choice stands for, shown to the operator. Unique within the item.

Task_ScoreItem​

Task_ScoreItem(
    key: str,
    prompt: str,
    kind: Task_ScoreKind,
    choices: tuple[Task_ScoreChoice, ...] | None = None,
    weight: float | None = None,
)

Task_ScoreItem.key​

key: str

Stable identifier used as the answer key, e.g. "towel_folded".

Task_ScoreItem.prompt​

prompt: str

Question shown to the operator, e.g. "Is the towel folded in half with edges aligned?".

Task_ScoreItem.kind​

kind: Task_ScoreKind

Task_ScoreItem.choices​

choices: tuple[Task_ScoreChoice, ...] | None = None

The options for Choice, in the order they are presented; required then, None for Binary.

Task_ScoreItem.weight​

weight: float | None = None

Binary only: points when yes (default 1). 0 records the item without scoring it. A Choice item carries its points in its choices and must leave this None.

Task_ScoreItem.max_points​

max_points: float

Read-only property.

The most this item can contribute.

Task_ScoreItem.contribution​

contribution(answer: bool | int) -> float

Points earned for answer; raises ValueError if the answer does not fit the item.

Task_ScoreAnswers​

Task_ScoreAnswers: tp.TypeAlias = dict[str, bool | int]

Operator answers keyed by Task_ScoreItem.key: bool for a binary item, the 0-based index of the chosen option for a choice item. Every item of the sheet must be answered.

Task_ScoreSheet​

Task_ScoreSheet(success: Task_ScoreItem, items: tuple[Task_ScoreItem, ...] = ())

Task_ScoreSheet.success​

success: Task_ScoreItem

The binary item that decides whether the episode passed. Always present, always asked first.

Task_ScoreSheet.items​

items: tuple[Task_ScoreItem, ...] = ()

Further criteria presented to the operator after success, in this order.

Task_ScoreSheet.all_items​

all_items: tuple[Task_ScoreItem, ...]

Read-only property.

Task_ScoreSheet.max_total​

max_total: float

Read-only property.

Task_ScoreSheet.validate_answers​

validate_answers(answers: Task_ScoreAnswers) -> None

Every item answered, nothing extra, every answer of the right kind and range.

Task_ScoreSheet.total​

total(answers: Task_ScoreAnswers) -> float

Weighted points earned, in [0, max_total].

Task_ScoreSheet.normalized​

normalized(answers: Task_ScoreAnswers) -> float

total / max_total in [0, 1]; 0 if no item carries weight.

Task_ScoreSheet.is_success​

is_success(answers: Task_ScoreAnswers) -> bool

The answer to the success item.

Task_Spec​

Task_Spec(
    name: str,
    description: str,
    reset_instruction: str,
    scoring: Task_ScoreSheet,
    max_rollout_time: float,
    instruction: str | None = None,
)

Task_Spec.name​

name: str

Stable task identifier, e.g. "fold_towel".

Task_Spec.description​

description: str

Short human-readable summary shown to the operator.

Task_Spec.reset_instruction​

reset_instruction: str

Step-by-step instructions for the operator to set up the scene before an episode.

Task_Spec.scoring​

scoring: Task_ScoreSheet

Task_Spec.max_rollout_time​

max_rollout_time: float

Wall-clock budget for one episode from the first policy inference, unit: [s]. The station stops the episode and asks the operator to score it as it stands when this elapses.

Task_Spec.instruction​

instruction: str | None = None

Natural-language command handed to the policy. None for tasks that are not language-conditioned.