Playground

A harness for watching cortex work and finding out why it works the way it does.

Not a benchmark — benchmarks/polyglot scores cortex. This one shows its reasoning so a bad habit can be traced to the prompt or tool description that caused it.

The commands

./run.sh <task> [label]             # run cortex on one task, keep everything
./batch.sh [-n runs] [task ...]      # repeat a suite and keep going on failure
./report.sh [runs/<task-label> ...]  # summarize pass/fail, cost and behavior
./trace.sh <view> <trace>            # read what happened inside one run

What a run keeps

Each run gets its own directory under runs/ holding a pristine copy of the task's seed project, the trace cortex wrote, cortex's own stdout, and verifier output. The seed is never touched, so a run is repeatable and two runs are directly comparable:

trace.jsonl — the wire-level record
diff.patch — exactly what the agent changed
stdout.txt / stderr.txt — the agent's own output
verify_stdout.txt / verify_stderr.txt — final project verification
status.txt — git status of the work tree

The trace

Newline-delimited JSON with one discriminator, kind. Most kinds are axon event names. Three are cortex's own:

kind	what it holds
`run`	which build and which models produced this trace
`wire_request`	the exact messages sent to a model, and to which one
`wire_reply`	what came back, and how long it took

The wire kinds are the reason this harness exists. Events are emitted downstream of a request — they describe the reply and never the prompt — so the composed system message, the windowed conversation and the pruner's own curator calls are all invisible without them. Prompt engineering against the event stream alone is guesswork.

The views

trace.sh offers six views, each one jq filter over the trace:

./trace.sh timeline runs/report-final/trace.jsonl
./trace.sh system   runs/report-final/trace.jsonl   # the prompt under test
./trace.sh prompt   runs/report-final/trace.jsonl pruner
./trace.sh tools    runs/report-final/trace.jsonl   # shows a wrong approach fastest

report.sh is the first thing to read after a suite. It flags failed verification, tool errors, runtime errors, no-op prunes, large call counts, and no-diff runs before you dive into individual traces.

Continue to Running tasks and Reading traces for the details.