jobq regenerated — report
Every function body in examples/programs/jobq/ was written again from the modules’ intents,
types, signatures, contracts and tests, and from mo-wiki/spec/programs/01-job-queue.md. The
tests were not touched. Ten modules, 2,135 lines stripped to signatures, now 3,114 lines.
Done-when
- Wall-clock: 26 minutes 19 seconds (01:15:13 to 01:41:32 UTC, 16 Sep 2026), from the first
read of
examples/README.mdto the last green run. mo checkclean on all ten modules;mo check --recipe Recipes.Store.Store store.momatches 11 signatures and passes the recipe’s 8 tests.mo testgreen on all ten: 55 tests, 4 properties over 200 seeds each, 0 failed, 0 skipped.mo test --sim 100green;queue.moholds 3 of 3 tests under faults,server.mo2 of 2 under its# sim: --faults 20 --until 0.5line (re-run at 400 seeds to check it is not flaky).- All five
# run:lines byte-identical to their.expectedfiles, with the expected exit codes (0, 0, 2, 1, 1).data/demo/jobq.loganddata/compact/jobq.logare unchanged by the runs. mo fmt --checkclean on all ten.mo test --write --sim 100has rewritten everyverified:line; the.mo.idssidecar’s changes are confined to the tenjobq/entries.- Every
moprocess ran undertimeout 120with a 4 GiB watchdog; the watchdog never fired (the slowest single command wasmo run -- check, 1.3 s).
Loops to green, by cause
Five failures after an edit of mine. Every one was fixed by the next edit.
| # | Module | Cause | Diagnostic / test | Fixed first try |
|---|---|---|---|---|
| 1 | board.mo | return board written as a case arm’s expression in unindexed | MO0102 | yes |
| 2 | books.mo | a boolean expression wrapped onto a second line as a case arm’s value in held_twice? | MO0102 | yes |
| 3 | moves.mo | with_jobs folded with reduce, so the anonymous function captured fs | MO0409 | yes |
| 4 | server.mo | Time.fixture() in unknown(), a function outside a test | MO0403 | yes |
| 5 | server.mo | the fault test failed under --sim 100 --faults 20 --until 0.5, seed 6243096878674939956: assert turn(...) == "", left "a 503 changed the jobs" | test, not a diagnostic | yes |
Loop 5 is the only one that was a thinking mistake rather than a syntax one. turn read the jobs
before a request as snapshot(service) or [], so a snapshot the faults had already knocked out
read as no jobs at all, and the next 503 looked like a change. A turn that cannot see the jobs
beforehand has no evidence, so it now says nothing (turn matches on the Option and a new
helper took does the lease).
Not counted as loops, since they were the state the strip left behind rather than my edits: three
signatures whose continuation line had been eaten with the body (Board.with_number’s
Map(K, Set(UInt64))), Journal.put_all’s StoreError), Main.ran’s Problem), all MO0103),
and the two empty fn update bodies (MO0101, “expected case”). Journal.put_all’s requires
went with its signature line; it was put back as
requires pairs.all?(fn(pair) key?(pair.0) and value?(pair.1) end), which is what its
test rejects "a put of many with a key holding a space" trips.
Two more diagnostics came from a throwaway probe file, not from the program: MO0101 (an
assignment as a case arm’s expression) and MO0403 (a capability is passed down as a parameter, never returned). The second decided how the crowd of 1,200 connections is held — see decision 12.
No run ever printed the wrong bytes: all five # run: lines matched their .expected file the
first time they were run.
Decisions the spec did not cover
-
How a replay knows which ids were handed out. The spec says a job’s id is
j_and “a counter that never repeats, per service”, but not how a restart learns the counter. The log carries a record of its own under the keyids, and ids are reserved a thousand at a time: a create whosenext_idhas reached the reserved ceiling writesSET ids <next_id + 1000>in the same append as the job’s record, so a replay resumes above every id ever handed out — including one whose job was deleted and one whose write failed.data/demo’sSET ids 1000is what makes the first job the check createsj_1000. -
An id is spent before its record is durable.
Moves.creatingraisesnext_idon the books it hands toBooks.committed, not on the bookscommittedgives back, so a create answered503still burns its id. Without this a torn write could hand the same id to two jobs, which is what “a job is never held by two workers at once” turns into once the log is replayed. -
A lease’s place in the index is a whole second.
Board.duebuckets each queue’s leased jobs by the second its lease runs out (whole seconds since 2000, anInt64). A look gathers every bucket at or below the current second and then filters on the exact millisecond deadline, so the index is coarse and the answer is not. The spec only asked that a lease be “a deadline, not a timer”. -
Which looks expire a lease, and how far each one looks. The spec names “a lease request, an ack, a fail, a read, or the listener’s
Idle”. Read here means bothGET /jobs/{id}, which looks at its own job, andGET /jobs, which looks at its queue’s leases whenqueue=is given and at every queue’s when it is not.DELETElooks too, so it does not answer409on a lease that has already run out, and so does a refused ack or fail. Every expiry a look finds goes into the same append as the change the call makes, so a look costs at most one write. -
Every response’s error sentence. The spec gives statuses, not words. All of them are the program’s own and
jobq.expectedpins them:a request needs authorization: Bearer <token>,no such job,no route <path>,this route takes GET, DELETE,the job is leased,the caller does not hold a live lease on the job,queue must be 1 to 64 letters, digits, - and _,max_attempts must be a whole number from 1 to 100,lease_ms must be a whole number from 100 to 3600000,state must be queued, leased, done, or dead,the body is not JSON. -
The listener’s
idle:is 5 seconds, against an acceptor mailbox of 4,096. The runtime counts a connection that has sent no whole request against that bound, so 1,200 silent clients leave 2,896 of headroom;idle:is what frees them, and 5 seconds is short enough that a crowd cannot accumulate and long enough that a real client on a local socket is never cut off. The acceptor’sIdleis turned into aSweepask that waits 10 seconds, which bounds every file call the sweep makes;Idleonly arrives when no client came for 5 seconds, so nothing is waiting on that ask. -
jobq checkand its own log.jobq.check.logis removed before the service opens and again after the script, andStore.writing_toresets the replayed line count and the cut-short flag along with the name and size. Sodata/demo’sjobq.log, whose last line is cut short, is replayed but never written to, and the check’s own log does not start life needing a rewrite.Books.openedreplays the folder’sjobq.logfirst and then continues overplace.logwhen the two differ. -
A hand-edited log cannot crash the service.
Job.job_ofrefuses a record whose state isqueuedand whoseattemptshave already reachedmax_attempts— a state no move of the program can reach, since a fail or an expiry on the last attempt isdead. Without the refusal such a record would reachJob.leased, whoserequires job.attempts < job.max_attemptswould crash the queue process rather than refuse the line. -
What a sweep answers.
Moves.swepthands back the jobs whose leases it ended as aListedoutcome, andJobq.Queueturns that into the count itsSweepmessage replies with. The spec asks for the sweep, not for its answer. -
A job is written by hand, the health counts by
Json.encode.Job.shownis written out field by field, as the module’s comment says it must be;Api.healthisJson.encodeof theHealthstruct, whose field order already is the spec’s (queued, leased, done, dead, uptime_ms). -
compactcannot say why a folder would not open.Jobq.Main’suse Jobq.Store{...}line does not importStoreError, sojobq compactreports any failure to open asis not a folder jobq can read. AddingStoreErrorto thatuseline would have meant editing a module header the strip left intact, which this run did not do. -
The crowd holds its connections by recursion. A capability cannot be returned from a function (MO0403), so a
List(Conn)of 1,200 silent clients was not available.Main.crowdedopens one connection, plays the rest of the script inside that frame, and closes it on the way out, so all 1,200 are open while the producer’s request is answered. 1,200 frames plus the script’s 43 is well under the 10,000-call depth limit. The count that could not be opened goes to stderr, since the system’s descriptor limit decides it; on this VM all 1,200 opened. -
Six private helpers beyond the stripped signatures, each named in its module:
Queue.counted(aSweep’s count out of an outcome),Server.ended_by(theIdleask),Server.tried(a job’s number and attempts, what a503must not have changed),Server.took(the lease half of a turn),Main.trip_query(what a path holds after its?) andMain.crowded. -
What a
503is allowed to have changed. The fault test compares each job’s(number, attempts)before and after, not the jobs themselves: a lease that runs out between the two looks is a legitimate change that leavesattemptsalone, while a lease that landed without its answer arriving does not. So the test asserts exactly the thing the never is about, and not the clock.
Contracts the reader expects to see
The requires the spec asks for are on Job.job (queue name, payload, max_attempts) and
Job.leased (lease_ms), each with its test rejects; Store.put and Journal.put_all carry the
store’s. Moves.leasing has ensures result.outcome is Handed(job) implies handed_to?(books, job, worker) — leased to the caller with attempts one higher — and Moves.acking has ensures result.outcome is Found(job) implies job.state == Done.
Invariants left out of process Service, and why. The process as the spec left it declares
none, and none could be added that a message can break:
- the id counter never goes backwards —
state.books.next_idis only ever set fromBooks.next_id, whichcreatingraises by one andopenedraises to the log’s ceiling. No message lowers it, so no test could trip it. It is documentation. - a job’s attempts never exceed its max_attempts — already a
neverinJobq.Jobover everyJobvalue the run holds, which is stronger than an invariant over one process’s state, andleased’srequiresrefuses the only move that could break it. - no job is held by two workers at once — already a
neverinJobq.Books, over theSteps of every write, which covers the writes a restart replays as well as the ones this process made. - the store is open before a job is served —
readiedopens it at the first message after a start or a restart andserved_byanswers503when it cannot, so no message can leave a call served against an unopened store.
The four nevers in Jobq.Books are checked over the Steps a write makes. A call that looks at
a job and changes nothing makes no Step, so a lease that ran out blocks its job past a look
holds vacuously; it is the expiry written into every look’s append that actually keeps it, not the
never.
Deadlines: the within: count
31 within: sites; 4 of them are in test helpers (Moves.with_jobs, Queue.ask, Queue.got,
Server.sent). Of the 27 in the program:
- 7 derive from an enclosing deadline. All are in
Jobq.Journal: five pass a caller’sDeadlinestraight through (list,size,fold_lines, a compaction’srename, an append’s size check), and two tighten it withat_mostby a number chosen for that call — 5 s for an append, so a failed one can still be looked at, and 20 s for a compaction’s write, so the rename has time left. The chain above them is derived too: the worker asks the service within 60 s, andreply_byinside the update is what remains of that one ask, so nothing the service does for a request outlives the request. - 20 are numbers chosen for the call, because there is no enclosing deadline to derive from.
13 are in
Jobq.Store, whose recipe functions keep their own deadlines formainand for the recipe’s tests (5 s append and size, 20 s + 5 s compaction, 600 s replay, 10 s list and stat). 7 are inJobq.MainandJobq.Server, wheremainhas no asker and the acceptor’sIdlecarries no reply: 21 minutes for the opening replay, 5 s to bind a listener, 10 s for the one-shot client and for the script and check-log housekeeping, 30 s forexchange.reply, 60 s for the worker’s ask, 10 s for theIdlesweep.
So one number — the worker’s 60 seconds — is the root of every deadline a request’s work runs on, and the store’s own numbers are the root of the rest.
Questions for a running service the runtime could have answered
Kept for directions 37 to 40, from what was actually wanted while getting this green:
- Which process holds the books right now, and whether
state.openedis true — the “folder cannot be read” path was reasoned about rather than looked at. - What a service’s
next_idandreservedare between two updates, to see theSET idsline land without reading the log. - The last ten messages the queue process took, to tell a
503from a timed-out ask apart from a503from a refused write — the distinction loop 5 turned on. - How many connections the acceptor’s mailbox is holding, and how many of them have sent nothing,
during
crowd 1200. - The slowest update of a check run, to see whether the crowd or the replay dominates.