The programs

Six real programs, each written by a fresh Opus session from a spec the lead wrote, each finding two to four toolchain bugs and several gaps, each followed by a step that closed them. They are the corpus the rounds and the measurements run on.

programspecplanlineswhat it is, and what it found
1, jobq01-job-queueprogram-13,057a durable lease-based job queue over HTTP on the store recipe; found that hand-written deadlines lie, and that invariant kept none of eight candidates; the round 7 to 10 program
2, logstat02-log-analyzerprogram-2769a log analyzer; the first program, written in 25 minutes; the round 1 to 6 program and the stronger form of measurement 1
3, kv03-kv-storeprogram-31,619a TCP key-value store with a line protocol; found six runtime bugs (step 12) and became durable
4, notes04-web-backendprogram-42,160a notes service over HTTP, the first built from two recipes; found that a started process was never freed and that nothing checked an implementation against its recipe (step 19)
5, agent05-agent-harnessprogram-54,551an agent harness with permissions, budgets, retries, and a scripted mock model; found an authority hole and asked for the delayed send (step 24)
6, ledger06-ledgerprogram-64,450a double-entry payments ledger with holds, captures, settlement, idempotency, seven nevers and four invariants; found restart: :never restarting anyway and replay memory unbounded (steps 29 and 29b)

The changes to program 1

  • 01b-job-queue-change — change 1: scheduled jobs, backoff, retry, the tries rename; the maintenance round (control-run-8), the small-model round (control-run-9), the Elixir round (control-run-10)
  • 01c-job-queue-change-2 — change 2: the folder checked at open, 503 and the service still answering, /queues, verify; generation two of the erosion-round
  • 01d-job-queue-change-3 — change 3: the store restarts itself from its log, a budget on the child line, a chaos switch; generation three, the BEAM’s row answered (back in 106 ms)
  • 01e-job-queue-change-4 — change 4: idempotent creates, the archive beside the log; generation four, nothing eroded, the Mo queue found nine times slower on the lease path
  • 01f-job-queue-change-5 — change 5: a lease handed to another worker, a queue renamed with jobs in flight; generation five, the first Mo-only defect, the laws silent

The corpus and the menu

  • corpus — the 50 single-construct files and the rejects/
  • program-menu — the programs that were considered, and program 7 (a Redis subset against Redis’s own tests), the one chapter 1 says answers the BEAM
  • q14-first-real-program
  • bodies-as-cache — every program regenerated from its stripped spec, twice, all at 1.0
  • model-bakeoff — the same corpus brief by three models