Engineering

Trace events and the fixture contract

ยท Open

This page is supposed to hold two documents: the runlog trace event schema, and the evalset fixture JSONL contract, both written as docs with sequence diagrams.

Not written yet, waiting on P-5, P-6

The runlog trace event schema and the evalset fixture contract. No spec or shipped code for either tool was found in the calabrodesign repo or in the linked calabro-mcp pointer as of 2026-09-21. Inventing a plausible JSON shape and drawing a sequence diagram of it would produce a document that looks exactly like a real one and describes nothing.

Checked 2026-09-21.

Why this is not stubbed

A schema document is the most copyable artifact an engineering site produces. Someone reads it, believes it, and writes a producer against it. A made-up schema is worse than an empty page because it fails silently and at someone else's expense.

What the finished pages need

For the trace event schema:

  1. Every field, its type, and whether it is required.
  2. What identifies a run, and what identifies a step within it.
  3. Ordering guarantees. Whether events can arrive out of order, and what a consumer is allowed to assume.
  4. The versioning rule. What a producer does when the shape changes, and what a consumer does when it sees a version it does not know.
  5. A worked example: one real run, start to finish, as it appears on the wire.

For the fixture contract:

  1. The JSONL line shape, field by field.
  2. What a grader receives and what it must return.
  3. How a fixture declares which graders apply to it.
  4. What happens to a fixture whose grader throws.

Item three on the first list is usually where these documents are thinnest and where the bugs actually live.

What is documented in the meantime

The constraints that will shape both of these are written up: the constraints. The local-first and worker rules in particular will decide what a trace event is allowed to contain.