Running tasks

run.sh drives cortex through one playground task and keeps everything it did.

Usage

./run.sh <task> [label]

What happens, step by step

1. run.sh ledger iter1 copies tasks/ledger/seed intoruns/ledger-iter1/work.
2. It commits the seed in a fresh per-run git repo. git diff in the work directory is then exactly what the agent changed, with no other setup.
3. A .gitignore is written so build artefacts (__pycache__/,*.pyc, .pytest_cache/) don't fill the diff with binary noise and bury the one source change that matters.
4. It builds the cortex in the working tree — not whatever was on$PATH. The whole point is to test the code you're working on.
5. It detects the seed project's verifier and passes it to cortex with --verifyand one repair attempt by default.
6. It runs cortex non-interactively on tasks/<task>/TASK.txt, writing stdout and stderr to the run directory.
7. cortex announces its trace file on stderr; run.sh copies it into the run directory so the directory is self-contained once the cache is cleaned.
8. It runs the seed project's verifier (go test ./..., python -m pytest -q, or npm test) and records the result.
9. It stages all changes and writes diff.patch — staged so files the agent created show up as additions rather than untracked paths the patch never mentions.

The run directory

runs/ledger-iter1/
├── work/           # pristine seed copy + the agent's changes
├── cortex          # the freshly-built binary
├── trace.jsonl     # the wire-level record
├── diff.patch      # exactly what the agent changed
├── stdout.txt      # the agent's answer (stdout)
├── stderr.txt      # notices, tool activity, errors
├── verify.status   # verifier exit status
├── verify_stdout.txt
├── verify_stderr.txt
└── status.txt      # git status of the work tree

Batch runs

./batch.sh -n 3 report pipeline

Batch runs keep going after failures and write a markdown summary under runs/. Use this for prompt or tool changes so one unlucky run does not hide the rest of the suite.

Set PLAYGROUND_REPAIR_ATTEMPTS=0 when you want to measure one unaided turn.