Evaluation Messages

Status, episode results, and rejection details returned by the Evaluation Client.

These protobuf messages support attribute access, such as status.state. Repeated fields are sequences: iterate over status.episodes or use len(status.episodes). Enum fields contain integer values; compare them with the named constants in pb.

from se3labs.interface.wire import pb, status_pb

Optional fields support message.HasField("field_name"). For mutually exclusive fields, message.WhichOneof("group_name") identifies the active field. A false boolean or index zero can be a present answer; testing the field's truthiness does not check presence.

status_pb.Status

Latest job snapshot reported by a station and cached by the relay. Poll get_status() again for updates. A cached snapshot does not establish that the station is still connected.

Field	Value
station_id	str. ID of the station that produced this update.
state	str. Job state: CREATED (created), WAITING_POLICY (awaiting a policy connection), RUNNING (operator prompts and episode execution), FINALIZING (finishing), SUCCEEDED (completed normally), FAILED (execution failed), or CANCELLED (operator stopped or aborted). SUCCEEDED does not mean every episode passed the task; inspect each episode's scored and success fields.
episodes_total	int. Number of episodes requested for this job.
episodes	Sequence of pb.EpisodeResult. Finished episodes in execution order. The running episode is not included; len(status.episodes) counts finished episodes, including unscored ones.
updated_unix_ms	int. When the station produced this update, in milliseconds since the Unix epoch. This is not the time of the poll.
message	str. Progress detail or terminal reason. May be empty.

pb.EpisodeResult

One finished episode, with the score sheet as the operator answered it.

Field	Value
episode_id	str. Station-assigned episode ID, such as ep_0000.
end_condition	str. Why execution ended: timeout (rollout budget elapsed), policy_completed (the policy raised EndEpisode), lease_lost (worker lost control permission), authority_lost (a higher-priority controller or operator held the arms), or start_pose_unreached (arms did not settle; no rollout ran).
scored	bool. Whether the operator submitted a score sheet. When false, items is empty and success and score are not meaningful.
success	bool. Whether the success question's answer satisfies the task. Check scored first.
score	float. Points earned divided by the sheet's maximum, between 0.0 and 1.0; zero if no item carries weight. Check scored first.
items	Sequence of pb.ScoredItem. The whole sheet, success item first, each item with its answer.
task	pb.Task. The exact instruction, layout, parameters and scoring used for this episode.

pb.Reject

Details available as SessionRejected.reject when opening a session fails.

Field	Value
code	pb.RejectCode (int). Reason for refusing the session.
message	str. Human-readable explanation supplied by the relay or station.

pb.ScoredItem

One score question with the operator's answer and the points it earned.

Field	Value
item	pb.ScoreItem. The original question and its available choices.
yes	bool. A binary item's answer. Part of WhichOneof("answer").
choice	int. Zero-based index into item.choices. The selected label is answer.item.choices[answer.choice].description. Part of WhichOneof("answer").
count	int. An exact nonnegative count; contributes no points. Part of WhichOneof("answer").
points	float. Points earned for this answer, before normalizing the episode score.

Use WhichOneof("answer") to distinguish yes, choice, and count. False and zero are valid answers, not missing values.

pb.Task

Field	Value
name	str.
description	str.
reset_instruction	str.
scoring	pb.ScoreSheet.
max_rollout_time_s	float.
instruction	str. Check presence with HasField("instruction").
layout	pb.LayoutSample.
seed	int. Check presence with HasField("seed").
parameters	Mapping of str to pb.TaskParameter.

pb.RejectCode

Why the relay or station refused a policy session.

Value	Meaning
REJECT_CODE_UNSPECIFIED = 0	No specific code was supplied; read the rejection message.
REJECT_CODE_UNKNOWN_STATION = 1	No matching station is attached and waiting. Check the station ID and ask the operator to start evaluation.
REJECT_CODE_STATION_BUSY = 2	Another policy session holds the station. Wait for it to finish.
REJECT_CODE_PROTOCOL_VERSION = 3	SDK and station protocol versions differ. Use matching releases.
REJECT_CODE_BAD_REQUEST = 4	The submitted episode tasks were rejected. Read the message for the invalid value.

pb.ScoreItem

A question from the submitted Task_ScoreItem, included with its answer.

Field	Value
key	str. Stable identifier used as the answer key.
prompt	str. Question shown to the operator.
kind	pb.ScoreKind (int). Whether the question takes a binary answer, a choice index, or a count.
choices	Sequence of pb.ScoreChoice. Available options in display order; empty for binary and count questions.
weight	float. Points for a yes answer to a binary question. Absent for other kinds. Check presence with HasField("weight").

pb.ScoreSheet

Field	Value
success	pb.ScoreItem.
items	Sequence of pb.ScoreItem.

pb.LayoutSample

Field	Value
seed	int.
objects	Sequence of pb.LayoutObject.
table	pb.LayoutFootprint.
excluded	Sequence of pb.LayoutFootprint.

pb.ScoreKind

How the operator answers a score question.

Value	Meaning
SCORE_KIND_UNSPECIFIED = 0	No question kind was supplied; not valid for a submitted task.
SCORE_KIND_BINARY = 1	A yes/no question, earning its weight for yes and zero for no.
SCORE_KIND_CHOICE = 2	A choice question, earning the selected choice's score.
SCORE_KIND_COUNT = 3	An unscored exact nonnegative integer observation.

pb.ScoreChoice

One option in a choice question, corresponding to Task_ScoreChoice.

Field	Value
score	float. Points earned by selecting this option; nonnegative.
description	str. Label shown to the operator.

pb.LayoutObject

Field	Value
id	str.
label	str.
instruction	str.
footprint	pb.LayoutFootprint.

pb.LayoutFootprint

Field	Value
shape	pb.LayoutShape.
pose	pb.Pose2.

pb.TaskParameter

Field	Value
values	Sequence of pb.ParameterValue.
sequence	bool.

pb.LayoutShape

Field	Value
circle_radius	float. Part of WhichOneof("shape").
rectangle	pb.Rectangle2. Part of WhichOneof("shape").
polygon	pb.Polygon2. Part of WhichOneof("shape").

pb.Pose2

Field	Value
x	float.
y	float.
yaw	float.

pb.ParameterValue

Field	Value
text	str. Part of WhichOneof("value").
integer	int. Part of WhichOneof("value").
real	float. Part of WhichOneof("value").

pb.Rectangle2

Field	Value
width	float.
height	float.

pb.Polygon2

Field	Value
shell	pb.PolygonRing.
holes	Sequence of pb.PolygonRing.

pb.PolygonRing

Field	Value
vertices	Sequence of pb.Point2.

pb.Point2

Field	Value
x	float.
y	float.