Evaluation Messages

Status, episode results, and rejection details returned by the Evaluation Client.

These protobuf messages support attribute access, such as status.state. Repeated fields are sequences: iterate over status.episodes or use len(status.episodes). Enum fields contain integer values; compare them with the named constants in pb.

from se3labs.interface.wire import pb, status_pb

Optional fields support message.HasField("field_name"). For mutually exclusive fields, message.WhichOneof("group_name") identifies the active field. A false boolean or index zero can be a present answer; testing the field's truthiness does not check presence.

status_pb.Status

Latest job snapshot reported by a station and cached by the relay. Poll get_status() again for updates. A cached snapshot does not establish that the station is still connected.

Field	Value
station_id	str. ID of the station that produced this update.
state	str. Job state: CREATED (created), WAITING_POLICY (awaiting a policy connection), RUNNING (operator prompts and episode execution), FINALIZING (finishing), SUCCEEDED (completed normally), FAILED (execution failed), or CANCELLED (operator stopped or aborted). SUCCEEDED does not mean every episode passed the task; inspect each episode's scored and success fields.
episodes_total	int. Number of episodes requested for this job.
episodes	Sequence of pb.EpisodeResult. Finished episodes in execution order. The running episode is not included; len(status.episodes) counts finished episodes, including unscored ones.
updated_unix_ms	int. When the station produced this update, in milliseconds since the Unix epoch. This is not the time of the poll.
message	str. Progress detail or terminal reason. May be empty.

pb.EpisodeResult

One finished episode, with the score sheet as the operator answered it.

Field	Value
episode_id	str. Station-assigned episode ID, such as ep_0000.
end_condition	str. Why execution ended: timeout (rollout budget elapsed), policy_completed (the policy raised EndEpisode), lease_lost (worker lost control permission), authority_lost (a higher-priority controller or operator held the arms), or start_pose_unreached (arms did not settle; no rollout ran).
scored	bool. Whether the operator submitted a score sheet. When false, items is empty and success and score are not meaningful.
success	bool. The operator's answer to the task's success question. Check scored first.
score	float. Points earned divided by the sheet's maximum, between 0.0 and 1.0; zero if no item carries weight. Check scored first.
items	Sequence of pb.ScoredItem. The whole sheet, success item first, each item with its answer.

pb.Reject

Details available as SessionRejected.reject when opening a session fails.

Field	Value
code	pb.RejectCode (int). Reason for refusing the session.
message	str. Human-readable explanation supplied by the relay or station.

pb.ScoredItem

One score question with the operator's answer and the points it earned.

Field	Value
item	pb.ScoreItem. The original question and its available choices.
yes	bool. A binary item's answer. Part of WhichOneof("answer").
choice	int. Zero-based index into item.choices. The selected label is answer.item.choices[answer.choice].description. Part of WhichOneof("answer").
points	float. Points earned for this answer, before normalizing the episode score.

Use WhichOneof("answer") to distinguish yes from choice; false and choice index zero are valid answers, not missing values.

pb.RejectCode

Why the relay or station refused a policy session.

Value	Meaning
REJECT_CODE_UNSPECIFIED = 0	No specific code was supplied; read the rejection message.
REJECT_CODE_UNKNOWN_STATION = 1	No matching station is attached and waiting. Check the station ID and ask the operator to start evaluation.
REJECT_CODE_STATION_BUSY = 2	Another policy session holds the station. Wait for it to finish.
REJECT_CODE_PROTOCOL_VERSION = 3	SDK and station protocol versions differ. Use matching releases.
REJECT_CODE_BAD_REQUEST = 4	The task or episode count was rejected. Read the message for the invalid value.

pb.ScoreItem

A question from the submitted Task_ScoreItem, included with its answer.

Field	Value
key	str. Stable identifier used as the answer key.
prompt	str. Question shown to the operator.
kind	pb.ScoreKind (int). Whether the question takes a binary answer or a choice index.
choices	Sequence of pb.ScoreChoice. Available options in display order; empty for a binary question.
weight	float. Points for a yes answer to a binary question. Absent for a choice question. Check presence with HasField("weight").

pb.ScoreKind

How the operator answers a score question.

Value	Meaning
SCORE_KIND_UNSPECIFIED = 0	No question kind was supplied; not valid for a submitted task.
SCORE_KIND_BINARY = 1	A yes/no question, earning its weight for yes and zero for no.
SCORE_KIND_CHOICE = 2	A choice question, earning the selected choice's score.

pb.ScoreChoice

One option in a choice question, corresponding to Task_ScoreChoice.

Field	Value
score	float. Points earned by selecting this option; nonnegative.
description	str. Label shown to the operator.