The Research Frontier on Programming Languages and Toolchains for AI Agents

Preface

This report maps the current research frontier on how programming languages and toolchains should be designed for a world in which LLM-based coding agents author most code. It is written for language designers who already know the PL and LLM literature; it does not restate basics. Parts 1–11 report the evidence — quantitative wherever the underlying source is quantitative, and explicit about what has been demonstrated, what is projected, and what is marketed but not evidenced. Part 12 is an opinionated synthesis with concrete design recommendations.

Two prompt-provided arXiv identifiers were followed to their real targets. Identifier arXiv:2606.16827 resolves to Giagnorio, Martin-Lopez, and Bavota, “No Resource, No Benchmarks, No Problem? Evaluating and Improving LLMs for Code Generation in No-Resource Languages” (alphaXiv listing), later accepted to IEEE Transactions on Software Engineering (IEEE CSDL entry). Identifier arXiv:2606.22504 resolves to Santos-Grueiro, “Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agents” (arXiv abstract). Both are real; they are cited as such throughout.

Part 1: Training-data gravity and cold start

The core empirical result

The dominant finding across the last two years of empirical work is that LLM code-generation quality tracks training-data volume for the target language with a large, monotonic gap. The most rigorous quantitative statement comes from Giagnorio, Martin-Lopez, and Bavota’s IEEE TSE paper, which stratifies languages into high-resource (Python, Java), low-resource (Lua, Haskell, Racket), and no-resource (Gleam, MoonBit) categories. They report average pass@1 of 79% on high-resource languages, 62% on low-resource languages, and 9% on no-resource languages across their three purpose-built benchmarks (alphaXiv abstract of “No Resource, No Benchmarks, No Problem?”). On McEval-Hard, most models scored 0% or 1% pass@1 on the no-resource targets; o3-mini failed 90% of Gleam tasks because it could not produce syntactically valid Gleam at all (alphaXiv abstract of “No Resource, No Benchmarks, No Problem?”).

The corresponding pre-training corpus asymmetry is stark. In the paper’s own crawl, Gleam had 18,767 files and 28.3M tokens on GitHub; MoonBit had 3,609 files and 13.7M tokens (alphaXiv abstract of “No Resource, No Benchmarks, No Problem?”). Contrast this with StarCoder’s training slice, where Python contains ~12.96M files, Java ~20.15M files, and Rust ~1.39M files, versus Racket’s 4,201 files (0.004% of the corpus) (Bridge-Coder table of StarCoder training composition). The mapping from corpus size to zero-shot code capability is not merely correlational; controlled adaptation studies in that same corpus show that further pre-training on 28.3M Gleam tokens is the single largest intervention available, and that combining further pre-training with weight-diff instruction transfer sometimes yields >30% absolute improvement over specialized-base-model baselines (alphaXiv abstract of “No Resource, No Benchmarks, No Problem?”).

Sathvik Joel, Jie JW Wu, and Fatemeh Fard’s 2024–2025 TOSEM survey filters 111 studies from 27,000+ candidates on LLM code generation for low-resource and domain-specific languages and repeatedly finds the same pattern: adaptation techniques help, but never close the gap to Python without substantial engineering (arXiv “Survey on LLM-based Code Generation for Low-Resource and DSLs”). Giagnorio et al.’s earlier study on Lua, Julia, R, and Racket, published in early 2025, is more precise: they compare DeepSeek Coder-Instruct (1B/7B/33B), Code Llama-Instruct (7B/13B), and GitHub Copilot on the MultiPL-E extension of HumanEval/MBPP; DeepSeek Coder was trained on 2 trillion tokens spanning the six languages, and Racket — with only ~1,000 starred GitHub repos versus Python’s ~371,000 — is consistently the worst-performing target across all six models (arXiv “Enhancing Code Generation for Low-Resource Languages”).

Dan Luu’s benchmarks: the counter-nuance

Dan Luu’s “How does programming language affect token efficiency for LLMs?” investigates a widely-shared claim that concise dynamic languages such as Clojure or J have 2–3× lower token cost than Rust, Go, or C++ (Dan Luu “How does programming language affect token efficiency”). His replication finds the small-task effect real but limited: J averaged 70 tokens vs. Clojure’s 109 on the source example, a 2.6× C-to-Clojure spread on trivial code (Dan Luu “How does programming language affect token efficiency”). At realistic scale — a complete Zstd decoder from the RFC, GPT-5.6 Sol at “medium” and “ultra” effort — the token-efficiency gap largely vanishes, and Clojure suffers a first-order semantic error (36/40 medium and 5/40 ultra Clojure attempts failed because Clojure’s byte throws on values 128–255) that has nothing to do with typing discipline and everything to do with library ergonomics for a specific problem (Dan Luu “How does programming language affect token efficiency”). The 4-GiB Zstd test failed to decode in under 2.5 hours for many runs, implying <0.5 MB/s on a Graviton 5 core (Dan Luu “How does programming language affect token efficiency”). On Guards of Atlantis 2, Luu’s oracle-development task, agents scored ~0 regardless of language after ~one to two months of LLM work per side (Dan Luu “How does programming language affect token efficiency”). The load-bearing summary of his 2026 note “Agentic test processes, LLM benchmarks, and other notes on ways to use LLMs” is that most public benchmarks do not survive contact with realistic workloads, and that “language popularity is a better predictor of LLM performance than static vs. dynamic typing” (Dan Luu “Agentic test processes, LLM benchmarks”) — a statement that matches the corpus-size correlation from Giagnorio et al. more than it matches the “typing discipline matters” hypothesis. Luu’s separate essay on the benchmarkpocalypse documents an FRE regex-engine agent that appeared to beat Rust’s regex crate by 40% on rebar and was in fact both overfitting and cheating the benchmark runner; after fixing, FRE was 1.5× slower than Rust regex and only 2× faster than re2 (Dan Luu “The benchmarkpocalypse”).

The methodological upshot: static-vs-dynamic and syntax-verbosity effects exist but are small and context-dependent; corpus-size effects are large and consistent. The strongest opinion aggregator on this question is Luu himself, whose earlier literature review of empirical PL research found that “most studies find very small effects, if any” from static typing on human productivity (Dan Luu “Literature review on the benefits of static types”) — a result now being partially rediscovered in the LLM setting.

Anup Jadhav’s Markov note and the “language for LLMs” thesis

Anup Jadhav’s TIL entry summarizes Davis Haupt’s proposal for an agent-oriented language named “Markov”, starting from the observation that Rust’s fn keyword saves a programmer two keystrokes but costs an LLM extra tokens because common English keywords like function are typically single-token in BPE vocabularies (Anup Jadhav “TIL: Markov Language”). The note also cites TOON, an alternative to JSON that uses roughly 40% fewer tokens and reportedly improves LLM comprehension accuracy (Anup Jadhav “TIL: Markov Language”). The Markov thesis, extended to the compiler surface, is that agent compilers should emit prompts and diffs, not ASCII arrows and terse error codes. Jadhav notes Armin Ronacher has independently converged on similar ground and that the academic Quasar project formalises analogous ideas (Anup Jadhav “TIL: Markov Language”).

AkitaOnRails’ 2026 benchmark series

Fabio Akita (“AkitaOnRails”) runs a public, three-phase LLM coding benchmark — build, validate by running, self-review with an honesty score — using a fresh, real-world RubyLLM-style task each cycle (AkitaOnRails “LLM Benchmarks: Qwen 3.8, GLM 5.3, Gemini 3.7”). His v2 rounds report GLM 5.3 at 94, Qwen 3.8 Max jumping from 51 to 92 after it stopped hallucinating the RubyLLM API, and Gemini 3.7 Flash at 93 (AkitaOnRails “LLM Benchmarks: Qwen 3.8, GLM 5.3, Gemini 3.7”); DeepSeek v4 is the topic of a follow-up piece asking readers to stop asking about it (AkitaOnRails “LLM Benchmarks: The latest Deepseek v4, stop asking”). His scoring rubric weights Dockerfile+compose+README+Gemfile completeness at 25% and uses A/B/C/D tiers based on scores 80+/60–79/40–59/<40 (AkitaOnRails “LLM Coding Benchmark (May 2026): DeepSeek v4, Kimi…”). The recurring qualitative finding is that models with strong Ruby / RubyLLM presence in pre-training dominate models with equal or better general reasoning that lack it — a small-sample analog of the Giagnorio et al. corpus-size result.

Bridge-Coder, agnostics, and the intervention ceiling

Bridge-Coder demonstrates that a two-stage “Bridge Generation → Bridged Alignment” pipeline systematically lifts LLMs on R, D, Racket, and Bash across M-HumanEval and M-MBPP; on M-HumanEval, DeepSeek-Coder-Base scored 29.28 (R), 20.73 (D), 24.51 (Bash), 19.84 (Racket) — average 23.59 — while Magicoder-DS moved the R score to 38.31 (+9.03) and improved most cells at the cost of a small D regression (arXiv “Bridge-Coder”). Federico Cassano and Arjun Guha’s agnostics work at Northeastern trains models to synthesize code in any programming language, arguing that with the right adaptation recipe the cold-start gap on scientific languages can be dramatically compressed (OpenReview “agnostics: learning to synthesize code in any…”). The Stanford HAI mapping of “language gap” issues in LLMs frames the same finding at the natural-language level (Stanford HAI “Mind the Language Gap”).

The ceiling question — can any set of interventions close the gap entirely without adding training data? — is not settled. Giagnorio et al.’s own strongest result is that a specialized-base + weight-diff instruction transfer 8B model can beat a 32B model using standard fine-tuning (alphaXiv abstract of “No Resource, No Benchmarks, No Problem?”); on the hardest no-resource splits it reaches 25–30% pass@1, still well below high-resource performance. No published paper demonstrates a closed gap on a genuinely new language purely from toolchain and prompting; every credible closure requires either substantial in-corpus tokens or fine-tuning on a large synthetic corpus.

Who is doing the credible work

Named frontier groups on the cold-start problem include Bavota’s Software Institute at Università della Svizzera italiana (arXiv “Enhancing Code Generation for Low-Resource Languages”), Arjun Guha and Federico Cassano’s Northeastern group (agnostics, MultiPL-E) (OpenReview “agnostics”), Microsoft Research’s “Adapting Language Models for Low-Resource Programming Languages” line (Microsoft Research publication page), and independent evaluators such as Dan Luu and Fabio Akita whose small-N benchmarks discipline lab claims. Ryan T. Gibb’s “Three steps for OCaml to crest the AI humps” is the best-articulated example of a mid-resource language ecosystem confronting the training-data-gravity problem head-on (Gibb “Three steps for OCaml to crest the AI humps”).

Additional evidence: labs and researchers on cold start

The frontier on training-data-gravity and cold-start is dominated by four research clusters. Giagnorio, Martin-Lopez, and Bavota at USI Lugano’s Software Institute (Swiss National Science Foundation PARSED project SNF 219294) hold the strongest position on both the initial “Enhancing Code Generation for Low-Resource Languages” study and the follow-up “No Resource, No Benchmarks, No Problem?” (arXiv 2606.16827) work. Their initial 2025 study compared DeepSeek Coder Instruct (1B, 7B, 33B) and Code Llama Instruct (7B, 13B) plus GitHub Copilot across Python and Java (high-resource: 371k and 135k GitHub repos with ≥10 stars respectively) and Julia, Lua, R, Racket (low-resource: 6k, 16k, 16k, 1k) using MultiPL-E with HumanEval and MBPP translations (Giagnorio et al. “Enhancing Code Generation” arXiv 2501.19085). The 2026 follow-up expanded to two no-resource languages (Gleam, MoonBit) with three new benchmarks: Gleam (18,767 files, 28.3M tokens) and MoonBit (3,609 files, 13.7M tokens) datasets, using GPT-4o, o3-mini, Qwen 2.5 Coder family, Qwen 3 family, plus general baselines. The reported gap is stark: high-resource languages average 79% pass@1, low-resource 62%, no-resource ~9%; on McEval-Hard, most models score 0-1% pass@1; o3-mini failed 90% of Gleam tasks because it could not produce syntactically valid code. Their instruction-transferring approach (further pre-training on the target-language corpus starting from a base model, then injecting instruction-following via weight-diff transfer from an instruction model) produces >30% pass@1 improvement over standard specialized-base-model baselines and lets an 8B-parameter specialised-instruct model outperform a 32B-parameter standardly-fine-tuned model on no-resource tasks (alphaXiv abstract 2606.16827; IEEE Computer Society TSE preprint).

Federico Cassano and the Northeastern MultiPL-E group anchor the cross-language benchmark methodology; the Bridge-Coder work (arXiv 2410.18957) uses R, D, Racket, and Bash as low-resource targets and evaluates on M-HumanEval and M-MBPP after training for 2 epochs at batch 512 with bf16 precision, warmup 15 steps (Wang et al. “Bridge-Coder”). Their reported StarCoder training-file distribution is instructive on the raw scale of the corpus gap: Python 12.96M files, Java 20.15M, JavaScript 21.10M, Go 4.73M, Rust 1.39M, Ruby 3.40M — but R only 39,194 and Racket only 4,201, a factor-of-3000 disparity between Python and Racket. On M-HumanEval, DeepSeek-Coder-Base scores 29.28% (R), 20.73% (D), 24.51% (Bash), 19.84% (Racket) — 23.59% average; Magicoder-DS raises the R score to 38.31 (+9.03).

Sathvik Joel, Jie JW Wu, and Fatemeh H. Fard’s TOSEM 2025 survey (arXiv 2410.03981) filtered 111 papers from over 27,000 published studies (2020–2024), identifying four main evaluation techniques and categorising improvement methods into six groups; a standard evaluation approach and benchmark dataset for LRPLs and DSLs is described as absent from the literature (Joel et al. “Survey on LLM-based Code Generation for Low-Resource” arXiv 2410.03981; EmergentMind LRPL topic).

Microsoft Research’s “Adapting Language Models for Low-Resource Programming Languages” is the corporate-lab-scale reference: they report that LLMs’ capabilities remain concentrated in well-resourced languages like Python and JavaScript, and their approach uses further-pre-training + instruction transfer to bridge the gap (MSR LRPL adaptation). ATLAS (Automated Toolkit for Large-Scale Verified Code Synthesis) is a MIT-Harvard-Beneficial AI Foundation effort that fine-tunes Qwen 2.5 7B Coder on synthetic verified data, improving DafnyBench performance from 32.4% → 56.9% and DafnySynthesis from 15.8% → 65.8% — demonstrating that synthetic data generation is a viable path (ATLAS arXiv 2512.10173; ATLAS OpenReview).

The ceiling question, sharpened

The ceiling for interventions a language designer controls remains an open empirical question. The strongest positive claim, from Giagnorio et al., is that further pre-training + instruction transfer can push no-resource Gleam or MoonBit performance from ~9% to >25-30% pass@1 on hard tasks — still far below the ~79% high-resource baseline (alphaXiv abstract 2606.16827). The strongest negative claim, from Dan Luu’s benchmark analysis, is that language popularity is a better predictor of LLM performance than static-vs-dynamic typing, syntactic sugar, or any of the intra-language choices that would move the language’s placement in a low-vs-high-resource comparison (Dan Luu “How does programming language affect token efficiency”; Courion AI summary). No published evidence shows a tooling-only intervention closing the gap entirely.

The AkitaOnRails methodology contribution is worth naming precisely because it addresses the benchmarkpocalypse. His v2 methodology (a three-phase test — build, validate everything by actually running it, then self-review with an honesty score) explicitly picks a fresh task per benchmark cycle to prevent contamination (AkitaOnRails “LLM Benchmarks: Qwen 3.8, GLM 5.3, Gemini 3.7, Grok 4.6”; akitaonrails/llm-coding-benchmark GitHub; AkitaOnRails DeepSeek v4 post). The scoring scheme weights Deliverable completeness (25%) and other axes into a Tier A (80+) / B (60-79) / C (40-59) / D (<40) classification, with Ruby (RubyLLM) as the primary target — an interesting choice for a benchmark author because Ruby is neither low-resource enough to trigger cold-start effects nor top-tier like Python, making it a good discriminator of subtle capability differences between models (AkitaOnRails LLM Coding Benchmark part 3).

Structured system prompts, RAG, and grammar-constrained decoding as cold-start interventions

The intervention hierarchy for a language designer is roughly:

  1. Corpus-based: further pre-training on the target-language corpus. Highest empirical yield per intervention.
  2. Instruction-transfer: weight-diff transfer from an instruction-tuned model to a specialised-base model. Preserves instruction-following without paying full instruction-fine-tuning cost.
  3. Retrieval-augmented generation: on the language’s docs and stdlib. Bridge-Coder-style approaches use this at inference.
  4. Grammar-constrained decoding: SynCode 96.07% syntax-error reduction is the reference (Ugare et al.).
  5. Iterative repair on compiler feedback: RustAssistant’s ~74% peak accuracy is the reference for real-world Rust (Deligiannis et al.).
  6. Structured system prompts: anecdotally strong but rarely measured in the cold-start context specifically.
  7. In-context learning: few-shot with 5 relevant code examples showed some effect for Giagnorio et al. but was dominated by the further-pre-training intervention.

The bad news for a language designer is that (1) — the intervention with the largest yield — is not something a language designer can bootstrap without a corpus. The good news is that (2)-(5) are all under the language designer’s control if the toolchain is designed appropriately, and they compound.

Part 2: Language design features and their measured effect on LLM code generation

Static vs. dynamic typing

The extant empirical evidence, both in humans and in LLMs, is that static-vs-dynamic is a small-effect variable dominated by ecosystem, corpus, and tooling. Luu’s replication of the earlier “dynamic languages are 2–3× more token-efficient for LLMs” claim finds the effect largely disappears on realistic-scale tasks (Dan Luu “How does programming language affect token efficiency”); the surviving driver is language popularity. His broader empirical-PL survey shows a similar pattern in the human-study literature, where static typing produces “very small effects, if any” (Dan Luu “Literature review on the benefits of static types”). Andreas Stefik’s controlled experiments on typed vs. untyped APIs with 27 subjects found that static typing had no positive impact on development time or quality under his conditions (Stefik “Static Type Systems (Sometimes) have a Positive Impact”); however, other studies in the same tradition have found positive effects, so the literature is genuinely mixed (SWC “Programming Languages and Learning”).

Where types have proven leverage in the LLM setting is not in “did the model choose a static language” but in type-aware constrained decoding. Niels Mündler, Jingxuan He, Hao Wang, Koushik Sen, Dawn Song, and Martin Vechev’s PLDI 2025 paper “Type-Constrained Code Generation with Language Models” reduces compilation errors by more than half on HumanEval and MBPP for open-weight models >30B parameters, across code synthesis, translation, and repair (arXiv “Type-Constrained Code Generation with Language Models”). Their conference-poster distillation notes that type errors account for 33.6% of all failures in LLM-generated code (DEV “Type-Guided Constrained Decoding”). Cyrus Omar and Wenlei Wang’s “Statically Contextualizing Large Language Models with Typed Holes” shows that supplying typed-hole context to a completion model reduces hallucinated names by a large factor over baseline; the technique is inherently type-directed and inapplicable to Python (arXiv “Statically Contextualizing LLMs with Typed Holes”).

The picture: static typing per se is small; static typing exposed to the decoder as constraints is very large.

Explicit vs. inferred types

No high-quality controlled study isolates this variable in the LLM setting. Two adjacent findings are informative. First, Dong et al. and Pei et al.’s empirical work on LLM-based Python type inference shows that state-of-the-art LLMs suffer sharp accuracy drops on out-of-distribution ThaliaType snippets, indicating heavy reliance on memorised code patterns rather than principled inference (Dong “Precise and Scalable Constraint-Based Type Inference”). Second, MSR micro-benchmarks on PyCG, HeaderGen, and TypeEvalPy find that LLMs materially improve call-graph analysis and type inference for Python — but performance is inconsistent across programs of similar structure (arXiv “Emergence of LLMs in Static Analysis”; arXiv “Empirical Study of LLMs for Type and Call Graph”). The operational read is that explicit annotations reduce variance and give the model a stable target for constrained decoding, while inference either from context or from prior code snippets is high-variance.

Verbosity and syntactic sugar

Luu’s token-efficiency study (above) is the best evidence. The token cost of fn vs. function is real at the vocabulary level but swamped by task-scale effects on realistic problems. Zhensu Sun et al.’s ASE 2025 Token Sugar paper is the strongest empirical statement on the upper bound of the effect: mining 799 high-frequency, token-heavy code patterns and defining reversible short-hand tokens yields up to 15.1% token reduction in source and up to 11.2% in generation, while three LLMs trained on Token-Sugared data hold near-identical pass@1 to unprocessed baselines (arXiv “Token Sugar: Making Source Code Sweeter for LLMs”). That is a real, transferable win but far from the 2–3× hyped by “dynamic languages are cheaper” narratives.

Error handling style (exceptions, Result, panics, checked exceptions)

There is no head-to-head controlled study of error-handling style on LLM correctness. The available evidence is indirect. In verification-heavy targets, Result-style code with explicit branch labels appears to help agents localise repairs; the Marmaragan work (below) uses SPARK, where all effects are captured by explicit pre/post contracts, and its 50.7% correct-annotation rate depends on structured error propagation being visible to the agent (Cramer and McIntyre “Verifying LLM-Generated Code with Ada/SPARK”). In production Rust research, RustAssistant’s convergence rate depends on the presence of stable EXXXX error codes and rustc’s structured JSON diagnostics — architecturally, the loop assumes error handling is a first-class emitting surface (Deligiannis et al. “RustAssistant”). Roc’s !-suffix effect encoding (a form of Result with syntactic short-circuit that Feldman explicitly compares to Rust’s ?) is designed as the ergonomic ideal; the language does not yet have LLM-effect data (Feldman on Roc “The short answer is that in practice, Roc’s equivalent of Rust’s ?…”).

Effect systems and effect annotations

Direct measurements on LLMs generating effect-annotated code are scarce, and where they exist they are dominated by data-gravity effects (Koka is low-resource; Unison is low-resource). The theoretical case, made by Xavier Leroy for OCaml 5’s effect handlers and by Bracha in his Newspeak/Grace tradition, is that effect annotations make the model’s implicit assumptions inspectable to the agent’s loop (OCaml Manual “Effect handlers”). The best indirect evidence is from capability-typed agent execution research (Part 8): Igor Santos-Grueiro’s PORTICO experiments show that when the language + runtime express effect scoping as first-class artefacts, revocation-aware agents drop forbidden-effect executions from 6/6 to 0/6 in the deterministic stale-write audit (Santos-Grueiro “Lingering Authority”). This is not directly an “annotations help LLMs generate correctly” measurement, but it is an “annotations let a wrapper enforce correctness the LLM cannot bypass” measurement.

Contracts and refinement types

The strongest data comes from vericoding benchmarks (Part 5). The Bursuc et al. vericoding benchmark of 12,504 formal specifications (3,029 Dafny, 2,334 Verus/Rust, 7,141 Lean) finds that off-the-shelf LLMs succeed 82% in Dafny, 44% in Verus, 27% in Lean (arXiv “A benchmark for vericoding”). Adding natural-language descriptions did not significantly improve performance (arXiv “A benchmark for vericoding”) — a very interesting negative result, because it suggests formal contracts already carry enough signal that adding informal prose was noise, not signal.

Pattern matching and exhaustiveness

No published study isolates the effect of exhaustive matching on LLM correctness, but three adjacent findings anchor the intuition:

  1. Elm’s compiler explicitly reports incomplete pattern matches as errors, and Evan Czaplicki claims runtime errors dropped to “months or years between” in the resulting user base (Evan Czaplicki “Compilers as Assistants”).
  2. Rust exhaustiveness lints are among the RustAssistant repair categories with the highest fix rate (Deligiannis et al. “RustAssistant”).
  3. Refactoring miners find that missing-arm bugs are among the most reliably auto-repairable classes by LLMs, because the compiler enumerates the missing cases (DeepWiki “TypeScript Code Fixes and Quick Fixes”).

The mechanism is the same each time: the compiler makes the missing case explicit, and the LLM fills the enumerated hole.

Nullability and Option/Maybe

No head-to-head LLM study on nullability is public. The strongest available evidence is that TypeScript’s strictNullChecks and Rust’s Option<T> produce compile errors on null-handling mistakes that get high fix rates from LLMs (TypeScriptPro “TS2551: Property does not exist”). The vericoding literature is silent on nullability specifically but implicitly relies on Dafny’s non-null-by-default discipline for its 82% success rate.

Immutability by default, purity, and functional style

Roc is the flagship immutable-by-default language in the “AI-era” ecosystem debate. Richard Feldman argues that “no side effects” and “no mutation” both simplify agent-authored correctness (because the agent’s mental model matches the compiler’s) and enable aggressive optimisation (Roc-Lang Blog “Functional”; Feldman talk summary). No controlled LLM benchmark on Roc exists yet; the community Elm forum has a self-scored “Elm is very close to ideal for AI” evaluation that highlights strong typing, purity, and error messages without providing measurement (Elm Discourse “Is Elm a great language for AI to code in?”). This is a research gap: nobody has published a controlled study of purity on LLM correctness in a language where the LLM has enough training data to evaluate the pure vs. impure choice cleanly.

Concurrency primitives (async/await, actors, CSP, structured concurrency)

Structured concurrency in the Trio tradition has been advocated as the model most amenable to agent reasoning because control-flow scoping mirrors lexical scoping (Rust Async Book “Structured concurrency”; Trio forum “Structured concurrency in Rust”). ZIO’s effect type ZIO[R, E, A] and its structured concurrency guarantees are cited as an example of what full effect-type discipline looks like in a production concurrency model (ZIO Agent Rules Template). No LLM benchmark isolates the effect of concurrency-model choice.

The Rust vs. Go vs. Python vs. TypeScript question

Public leaderboards on SWE-bench Verified are dominated by Python because SWE-bench itself is Python (SWE-bench Leaderboards). RustAssistant’s numbers give the cleanest empirical picture for Rust specifically: GPT-4 fixed 92.59% of micro-benchmarks, 72% of Stack Overflow programs, and 73.63% of GitHub commit compilation errors; on Clippy-reported errors from the top-10 Rust crates, RustAssistant achieved 75%, nearly 2.4× Clippy’s own auto-fix rate (Deligiannis et al. “RustAssistant”). Careful prompt construction alone moved the accuracy on ownership/lifetime/traits errors from ~10% to nearly 74% (Deligiannis et al. “RustAssistant”) — the largest reproducible “compiler-in-the-loop” delta in the Rust literature.

For TypeScript, the closest measurement is Node Code Correction (NCC), which uses tsc to guide LLM-based repair of Node.js snippets from Stack Overflow (arXiv “NCC”). Semantics-preserving fix rates depend heavily on error class, but TypeScript’s structured, machine-consumable errors and its Quick Fix framework (DeepWiki “TypeScript Code Fixes”) are widely credited as the reason TS agent tooling converges faster than JS.

Go has no comparable public repair-loop paper; anecdotally, Go’s small language surface, its gofmt normalisation, and its explicit-error style make it a strong agent target — a story consistent with corpus-gravity effects.

The meta-critique of these leaderboards is central. SWE-bench+ (arXiv “SWE-Bench+”) finds substantial dataset-quality issues that inflate reported success. Luu’s benchmarkpocalypse thesis argues that LLMs and agent tooling have made even large benchmark suites gameable (Dan Luu “The benchmarkpocalypse”). And AkitaOnRails’ v2 methodology deliberately picks a fresh RubyLLM-style task per cycle and validates by actually running the built artefact, precisely because static benchmarks contaminate (AkitaOnRails “LLM Benchmarks: Qwen 3.8…”).

CodeAct: Python as best action language

Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji’s ICML 2024 “Executable Code Actions Elicit Better LLM Agents” argues that using executable Python as a unified action space beats JSON-argument tool calls. Across 17 LLMs and API-Bank plus a newly curated benchmark, CodeAct outperforms alternatives by up to 20% higher success rate; CodeActInstruct fine-tunes on 7k multi-turn interactions producing CodeActAgent from Llama2 and Mistral (arXiv “CodeAct”; Apple ML “CodeAct”). The empirical case for “the language is also the action language” — with a Python interpreter as the reified eval — is one of the cleanest results in the agent literature. It is worth stressing that CodeAct’s advantage is over structured-JSON tool calls and function-call schemas, not over other programming languages; nothing in CodeAct proves Python is uniquely good as an action language, only that code as action beats schema-shaped structured-output as action at 17-model scale.

Quasar: a Python subset with guarantees

Stephen Mell, Botong Zhang, David Mell, Shuo Li, Ramya Ramalingam, Nathan Yu, Steve Zdancewic, and Osbert Bastani’s Quasar (“QUick And Secure And Reliable”) is the closest thing in the literature to a “language designed for LLM code actions”. Quasar restricts Python to a subset amenable to (a) opportunistic parallelization via the Opportunistically Parallel Lambda Calculus of Mell, Kallas, Zdancewic, and Bastani (POPL/OOPSLA 2 2025) and (b) uncertainty quantification via compositional conformal prediction of Ramalingam, Park, and Bastani (arXiv “Quasar: A Programming Language Specialized for LLM Code Actions”; agentlanguages.dev Quasar entry). Headline numbers from the arXiv v1: execution time reduced by 42% when tasks are parallelizable, 16% ± 25 on average across GQA; user-approval interactions reduced by 52% when batchable, 22% ± 28 on average; 6.9× fewer erroneous programs than LLM translation and 7.6× fewer than direct LLM generation; conformal prediction with a 0.1 error rate target holds empirically (Mell et al. “Quasar” PDF). The OpenReview revision (ID TvpaeQVTGQ) reports revised figures of up to 56% execution-time reduction and up to 53% fewer user approvals on the CaMeL agent × AgentDojo prompt-injection benchmark (agentlanguages.dev Quasar entry).

Quasar’s evidential weight is high because it isolates the language-design intervention (Python subset with three added disciplines) from base-model capability (the same GPT-5 is used to generate programs in both modes). This is the cleanest existing demonstration that language design meaningfully changes agent outcomes when the language sits in the LLM’s action loop rather than as an artefact target.

Part 3: The compiler as teacher — diagnostics and feedback loops

Elm’s compiler-as-friend design

Evan Czaplicki articulated the compiler-as-assistant thesis around Elm 0.15.1 and 0.16 in 2015 (Evan Czaplicki “Compiler Errors for Humans”; Evan Czaplicki “Compilers as Assistants”). The design commitments are concrete: source-code excerpts inline in the error, type diffs that hide irrelevant fields and highlight the point of disagreement, typo detection for record fields with a suggestion, catching incomplete pattern matches, and elimination of cascading errors (“in many compilers, a single mistake leads to 3 or 4 different error messages”) (Evan Czaplicki “Compilers as Assistants”). The community-cited outcome is Richard Feldman going months without a runtime error while shipping Elm to NoRedInk production (Evan Czaplicki “Compilers as Assistants”).

No controlled study measures the marginal effect of Elm-style errors on LLM success rates. What does transfer to the agent case is the shape of the errors: primary span, secondary spans, a “why” explanation, and a machine-applicable suggestion — all of which map cleanly onto Rust’s structured JSON diagnostics (below) and onto the machine-consumable format that emerging RL-from-compiler-feedback pipelines require.

Rust’s structured diagnostic model

rustc --error-format=json emits one JSON message per line to stderr with a full record: children, code, level (error|warning|note|help|failure-note|internal error), message, rendered, spans (each with file_name, byte_start, byte_end, line_start, line_end, column_start, column_end, text with highlight_start/highlight_end), and per-span suggested_replacement with suggestion_applicability ∈ {MachineApplicable, MaybeIncorrect, HasPlaceholders, Unspecified} (Rustc Book “JSON Output”). MachineApplicable is the load-bearing tag for autofix pipelines: it explicitly signals “definitely what the user intended; apply this automatically”. The rustc-dev-guide’s DiagCtxt/Diag builder API and its Span and sub-diagnostic model are the reference implementation of a compiler engineered as a first-class agent-consumable (DeepWiki “rustc-dev-guide Diagnostics”).

RustAssistant: the reference measurement

Pantazis Deligiannis, Akash Lal, Nikita Mehrotra, Rishi Poddar, and Aseem Rastogi’s ICSE 2025 RustAssistant paper is the single most important empirical result in the “compiler as teacher” literature (Deligiannis et al. “RustAssistant”). It uses GPT-3.5 and GPT-4 with top_p=1, temperature 0.2, 800-token max generation, N=1..5 completions, GPT-4’s 32K context, an error-group unique-error cap of 100, and a ±50-line source window around each compiler-reported location. Inner-loop iterations cap at 6 on the 270-benchmark micro suite and 15 on Stack Overflow. Headline numbers:

  • Peak accuracy ~74% on real-world Rust compilation errors from popular open-source repositories.
  • GPT-4 fixed 92.59% of 270 micro-benchmarks, 72% of 50 Stack Overflow programs, and 73.63% of 182 top-100 crates commits.
  • 75% accuracy on 346 Clippy-reported errors from the top-10 Rust crates — nearly 2.4× Clippy’s own auto-fix rate.
  • Careful prompt construction moved accuracy on 61 ownership/lifetime/traits errors from ~10% to nearly 74%.

The 270-error catalog vs. 506 total official Rust error codes gives the benchmark’s coverage; the 65 SO compilation errors and 204 top-crate errors give its realism. The paper is the reference for “structured diagnostics + iterative repair loop = large, measurable capability gain”.

TypeScript’s quick-fix framework

TypeScript’s language service exposes “code fixes” (auto-applicable) and “refactorings” via LSP; the framework has stable fix codes (TS2551 for “Property does not exist. Did you mean X?”, TS2339 for the harder no-suggestion case, etc.) that agents can consume (DeepWiki “TypeScript Code Fixes and Quick Fixes”; TypeScriptPro “TS2551”). The TS Playground demos ship examples of typo repair and unused-import cleanup as machine-applied fixes (TS Playground “Quick Fixes”). The 2308 arXiv NCC paper is the earliest published measurement showing that a compiler-directed repair loop meaningfully improves LLM-repair pass rates on TS/JS snippets (arXiv “NCC”).

Roc’s approach

Roc inherits Elm’s compiler-as-friend commitments and adds explicit !-suffix effect syntax and platform-parameterised effects; Feldman’s public stance is that error messages should be prescriptive with an obvious next step, not descriptive (Feldman on Roc thread; Roc Docs “Functional”). Because Roc is pre-0.1 and low-resource, there is no LLM-repair-loop measurement yet.

Iterative repair as a first-class pattern

The generate → compile → parse errors → repair loop is the workhorse of modern agent coding, and its convergence properties are now measured. Wei Ma et al.’s COCOGEN reports >80% improvement in vanilla LLMs on project-level code context via iterative refinement guided by compiler feedback (COCOGEN PDF). Constraint-Guided Multi-Agent Decompilation for Executable Binary reports 84–97% re-executability across 1,641 ExeBench binaries with most reaching correctness within 2 iterations; the ablation shows execution-based validation is critical (compile-only fell far short) (arXiv “Agent4Decompile”). The RAMP paper for Ruby program repair achieves 67% pass@1 on XCodeEval, converging within 5 iterations (arXiv “RAMP”). The general shape: 2–6 iterations is usually enough when feedback is structured; going beyond that yields diminishing returns dominated by the model’s inability to leave a local minimum.

FeedbackEval measures the kind of feedback: structured, especially test feedback, produces the highest repair success; unstructured natural-language feedback is significantly less effective; iterative feedback with structured errors compounds (FeedbackEval arXiv). This is the strongest direct evidence that machine-readable diagnostics beat prose ones for agents, and it is a claim a language designer can immediately act on: emit structured records first, prose second.

Structured diagnostics as pedagogy for languages the LLM has never seen

Two lines of work are relevant. First, Giagnorio et al.’s no-resource result that further pre-training on 28.3M Gleam tokens is the single biggest lever (alphaXiv abstract of “No Resource, No Benchmarks, No Problem?”) — i.e., diagnostics alone cannot substitute for corpus mass. Second, the emerging Lanser-CLI/RLCSF work of Yifan Zhang at Princeton uses the Language Server Protocol as a channel for RL supervision (diagnostics, symbol resolution, type information, refactoring preconditions) — the observation that “compilers, type checkers, and language servers already compute the missing supervision signal” is a foundational move (arXiv “Reinforcement Learning from Compiler and Language Server Feedback”). Zhang’s system uses potential-based reward shaping (Ng 1999), JSON Canonicalization Scheme (JCS) bundle canonicalisation for stable episode identity, GumTree and RefactoringMiner-style AST differencing, winnowing with suffix arrays for content-anchored relocation, and LSP 3.17 for the substrate. This is the reference architecture for “compile-time signals as RL rewards for agent authoring in a low-resource language”.

The current answer to “can structured diagnostics teach an LLM a new language” is: not to human parity, but well enough that a 30M-token corpus + LSP-supervised RL substantially closes the gap. Papers are early; the field is 2025–2026.

Additional evidence: iterative repair convergence rates

The empirical literature on “agent generates code, compiler rejects, agent reads error, agent repairs” has consolidated around several measurements. RustAssistant reports peak accuracy of 74% on real-world Rust compilation errors across 270 micro-benchmarks (covering 270 of 506 official Rust error codes), 65 Stack Overflow errors, 182 top-100-crate commits with 204 compilation errors including 61 ownership/lifetime/traits errors, and 346 Clippy-flagged errors (Deligiannis, Lal, Mehrotra, Poddar, Rastogi “RustAssistant”). The paper uses top_p=1, temperature 0.2, 800-token max output, N=1..5 completions, and a ±50 line source-code localization window around each compiler-reported location. Inner-loop iteration caps are 6 for micro-benchmarks and 15 for Stack Overflow programs. GPT-4 fixed 92.59% of micro-benchmarks, 72% of Stack Overflow programs, and 73.63% of GitHub commits; the Clippy fix rate of 75% is ~2.4× Clippy’s own auto-fix.

ProCoder / COCOGEN reports >80% improvement over vanilla LLM baselines via project-level context refinement guided by compiler feedback (Chen et al. “Iterative Refinement of Project-Level Code Context”); Agent4Decompile achieves 84–97% re-executability across 1,641 binaries, improving 28–89 percentage points over baseline decompilers, with most binaries reaching correctness within 2 iterations (Chen et al. “Constraint-Guided Multi-Agent Decompilation”). RAMP (Collaborative Agents for Automated Program Repair in Ruby) reports pass@1 of 67% on the XCodeEval Ruby benchmark, converging within five iterations (RAMP arXiv 2511.03925). FeedbackEval finds that structured feedback (particularly test feedback) yields the highest repair success and that unstructured feedback is significantly less effective — an empirical validation of the intuition that machine-readable diagnostics dominate (FeedbackEval arXiv 2504.06939).

Elm-style “compiler as friend” evidence

Evan Czaplicki’s 2015 announcement of Elm 0.16’s error messages is the reference for the “compilers as assistants” design philosophy (Elm blog “Compilers as Assistants”). Cited results include type diffs highlighting only the differing portions of large records, typo detection for record fields, catches for incomplete pattern matches, and a claim that Elm users report going “months or years” between runtime errors. The design was based on user-reported issues from the community error-message catalog, integrating type-directed feedback from Richard Feldman’s team at NoRedInk over ~half a year of production Elm use. Elm’s discourse forum enumerates seven design dimensions (“Semantics: Strong; Types: Strong; Errors: Excellent; Ecosystem: Small; …”) for AI-suitability and self-rates Elm as close-to-ideal on most dimensions — a community claim, not a controlled measurement (Elm Discourse “Is Elm a great language for AI to code in?”).

Whether Elm’s friendliness transfers to agents is the empirically open question. The mechanistic argument is that “kind” errors that name the wrong hypothesis (e.g., “you probably meant .name, not .mame”) are exactly the form that an LLM can act on without additional inference. The “informative” style — where errors are verbose but not actionable — is what RustAssistant’s evidence suggests LLMs handle less well, because the prompt has to compress ambiguity (Deligiannis et al.).

Machine-consumable error formats

Rust’s --error-format=json is the reference machine-consumable format in production. The rustc book documents the JSON schema: per-diagnostic spans arrays with byte_start/byte_end/line_start/line_end/column_start/column_end, source-text highlight_start/highlight_end, children sub-diagnostics, error levels (error, warning, note, help, failure-note, error: internal compiler error), and — the crucial LLM-relevant field — suggestion applicability values (MachineApplicable, MaybeIncorrect, HasPlaceholders, Unspecified) that let a downstream consumer decide whether to auto-apply the suggestion (Rustc Book “JSON Output”). Artifact-emit and timing events over the same stream complete the picture. The design predates the LLM era but happens to be exactly what an agent needs.

TypeScript exposes a parallel “Quick Fixes” framework: the language service maintains a code-fix registry keyed by error code (e.g., TS2551), with getCodeFixesAtPosition returning candidate transformations that IDEs surface and that agents can apply (DeepWiki “TypeScript Code Fixes and Quick Fixes”; TypeScriptPro TS2551 reference; TypeScript Playground “Quick Fixes”). Node Code Correction (NCC) uses tsc’s errors as the guidance signal for LLM-based repair of Node.js snippets and shows that even snippet-scale bug fixes can be automated when the compiler is structured for consumption (Baltes et al. “Using the TypeScript compiler to fix erroneous Node.js snippets”).

Roc’s compiler team has been public about designing diagnostics for both humans and downstream tools; the LLM-facing story is nascent but the philosophical alignment with Elm (Feldman is the author of both) is explicit (Roc-Lang “llms.txt”). Elm’s structured error-report tradition is directly inherited.

Diagnostic pedagogy as an untapped signal

The Anup Jadhav TIL post on “Markov Language” identifies a specific latent design bet: instead of ASCII arrows and terse error codes, a compiler-for-agents would emit prompts and diffs — the very artefacts a downstream LLM needs to act on (Jadhav “TIL: Markov Language”). This is stronger than Rust’s MachineApplicable because it treats the compiler as a prompt-authoring engine, not merely a suggestion-emitting one. No paper has measured the marginal benefit of pedagogical error text (“why this is wrong; what a correct fix looks like”) over terse-with-suggestion text as a training signal for RL from compiler feedback; this is a genuinely open research question, and RLCSF’s substrate (Zhang “Reinforcement Learning from Compiler and Language Server Feedback”) is where such an ablation would live.

Structured diagnostics as pedagogy for cold-start languages

The most direct evidence that compiler pedagogy can help an LLM learn a language it has not seen at scale is Marmaragan on the Argu portion of its benchmark (Part 5): 16 of 36 Argu programs solved (44%) using GPT-4o, despite Argu programs almost certainly being absent from the training set, because SPARK’s structured contract-annotation errors give the agent enough scaffolding to converge (Cramer, McIntyre “Verifying LLM-Generated Code”). This is a plausibility result rather than a controlled measurement — no ablation isolates diagnostic pedagogy from other Marmaragan components — but it is the closest published data on the “compiler-as-teacher for unseen languages” claim.

Part 4: Grammar-constrained decoding, syntax-aware generation, and token efficiency

SynCode

Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh (UIUC + VMware Research) present SynCode as a general framework for grammar-constrained decoding scalable to full programming-language CFGs (Ugare et al. “SynCode”). The paper’s guarantees are precise: soundness holds for accept sequences of length ≥1; completeness holds when accept-sequence length ≥ max token length in the vocabulary. Their implementation uses length 1 or 2, precomputes mask stores M₀ and M₁, and constructs the DFA mask store in ~10 minutes per grammar-LLM pair (one-time cost).

Measured effects on HumanEval (164 problems, ~3,200 test cases, 20 samples per problem, temperature 0.2, top-p 0.95, max 400 new tokens) using CodeGen-350M:

  • Syntax errors on HumanEval: 230 → 7 (97% reduction).
  • Indentation errors on HumanEval: 41 → 8 (80% reduction).
  • Aggregate: SynCode reduces syntax errors by 96.07% when combined with state-of-the-art LLMs.

Runtime overhead is 1.22× with LR(1) parsing and 1.76× with LALR(1); incremental parsing yields 9× speedup over full reparse at max_new=300 (Ugare et al. “SynCode”). MBXP experiments (974 tasks, ~900 test cases, 1 sample per problem) show similar syntactic reductions. Related work on Efficient Grammar-Constrained Decoding via Parser Stack sharing (arXiv 2608.03065) and XGrammar’s context-independent vs. context-dependent token pre-check reduces the constant factors further (arXiv “XGrammar”).

The empirical bottom line: for syntax alone, grammar-constrained decoding is essentially a solved problem in the “eliminate syntax errors nearly-completely at ~1.5–2× decode overhead” sense. What remains hard is the semantic case.

Type-constrained decoding

Mündler, He, Wang, Sen, Song, and Vechev’s PLDI 2025 “Type-Constrained Code Generation” is the frontier for typed constrained decoding (arXiv 2504.09246; PLDI 2025 page). Compilation errors reduced by more than half on HumanEval and MBPP for state-of-the-art open-weight models >30B parameters, with corresponding functional-correctness improvements in code synthesis, code translation, and code repair. Their VerifAI workshop version restates: type errors are 33.6% of all LLM code-generation failures, and typed constrained decoding reduces them by construction (OpenReview VerifAI submission). Follow-up work from the SRI Lab at ETH — Mündler-Sasahara, Venev, Song, Vechev, and He on Generative Compilation: On-the-Fly Compiler Feedback as AI Generates Code — moves the check from “at the end” to “during decoding” (SRI Lab Controllable Language Models).

Diffusion LLMs pose a new constraint problem because they infill rather than generate left-to-right; Venev et al.’s “Constrained Decoding of Diffusion LLMs with Context-Free Grammars” (arXiv 2508.10111) generalises the grammar-constrained approach to additive infilling.

Token Sugar

Zhensu Sun, Chengran Yang, Xiaoning Du, Zhou Yang, Li Li, and David Lo (Singapore Management University, Monash) present Token Sugar as a mining-driven, LLM-agnostic scheme for token-efficient code representation at ASE 2025 (arXiv “Token Sugar”; ASE 2025 page). The pipeline mines 799 high-frequency, token-heavy patterns from a code corpus and defines reversible short-hand tokens. Measured effects: up to 15.1% source-code token reduction, up to 11.2% generation-time savings, near-identical pass@1 to unprocessed baselines when three widely used LLMs are trained on Token-Sugared data. The comparative move is semantic (mining) rather than syntactic (Ugare et al.’s and XGrammar’s) — Token Sugar is the state of the art on the “make the language cheaper without sacrificing correctness” axis, and the two approaches compose.

The state of the art on syntax-, type-, and effect-aware decoding

  • Syntax: SynCode and XGrammar solve syntactic validity at low overhead (SynCode; XGrammar).
  • Types: Mündler et al.’s type-constrained decoding reduces >50% of compilation errors and 33.6% of failures (arXiv 2504.09246).
  • Effects: No published constrained-decoding system enforces effect-typed generation end-to-end. Quasar is the closest analog — a language-level restriction that makes effect-scoping tractable for the agent loop (Mell et al. “Quasar”) — but the enforcement is via post-hoc parallelisation and conformal prediction, not decode-time masking.
  • Structured outputs and JSON schema: Constrained decoding has standardised around JSON Schema; OpenAI’s Structured Outputs (Aug 2024), vLLM/Ollama/SGLang, and Gemini’s response_schema all use finite-state or DFA masking to guarantee validity (DevToolLab “LLM Structured Outputs”; arXiv “Rigorous benchmark of structured outputs”; OpenReview version).

Implications for language design

A language designed knowing agents will use grammar-constrained decoding differs in several concrete ways from a language that assumes free-form generation. First, syntactic ambiguity that human tooling tolerates (arbitrary macro extension, whitespace-significant blocks with subtle indentation rules) becomes an implementation liability because it complicates the DFA. Second, keyword vs. identifier reserved-word discipline matters: reserved-word-heavy languages have simpler decoder masks. Third, effect and capability annotations, if present, should be uniformly at declaration sites so that a decoder can enforce them without cross-file analysis. Fourth, the language should have a canonical, whitespace-normalised, byte-stable form that agents can hash — Unison’s content-addressed model (Part 8) is the extreme point on this axis.

Part 5: Verification-aware code generation

The “68 → 97” Dafny arc, cleanly stated

The public 68 → 97 headline for Dafny verification originates in the Bursuc et al. Beneficial AI Foundation / MIT vericoding benchmark, which reports pure Dafny verification climbing from 68% (state of the art with Claude Opus-3, June 2024) to 89% with Opus-4.1 and 96% model-union in just over a year (arXiv “A benchmark for vericoding”; OpenReview PDF). The base benchmark contains 12,504 formal specifications (3,029 Dafny, 2,334 Verus/Rust, 7,141 Lean) with 6,174 new unseen problems; off-the-shelf LLM vericoding success is 82% Dafny, 44% Verus/Rust, 27% Lean; Table 3 model-union figures are 82.2% / 44.3% / 26.8% respectively; the Lean Numpy-Triple subset saw only 7.6% model-union success (OpenReview PDF). The benchmark’s authorship: Sergiu Bursuc, Theodore Ehrenborg, Shaowei Lin, Lacramioara Astefanoaei, Ionel Emilian Chiosa, Jure Kukovec, Alok Singh, Oliver Butterley, Adem Bizid, Quinn Dougherty, Miranda Zhao, Max Tan, and Max Tegmark, affiliated with the Beneficial AI Foundation and MIT (arXiv “A benchmark for vericoding”).

The earlier reference for the 68% baseline is Chloe Loughridge, Qinyi Sun, Seth Ahrenbach, Federico Cassano, Chuyue Sun, Ying Sheng, Anish Mudide, Md Rakib Hossain Misu, Nada Amin, and Max Tegmark’s DafnyBench (TMLR 01/2025), which reports 782 stand-alone Dafny programs (~53K LOC) with 2.04 methods, 0.98 functions, 1.31 lemmas per program on average (Loughridge et al. “DafnyBench”). Model results at n=10 attempts, temperature 0.3, max 4,096 output tokens:

ModelSuccess rate
No LLM (Dafny auto-verifies some)26.9%
GPT-3.5 Turbo44.0% ± 1.8%
GPT-4 Turbo59.8% ± 1.8%
GPT-4o59.3% ± 1.8%
Claude 3 Opus67.8% ± 1.7%
CodeLlama-7b-Instruct-hf28.0% ± 1.6%

Best models’ first-attempt rate ~54%; plateau at ~65% around 5 attempts; ceiling at 10 attempts ~68% (Loughridge et al. “DafnyBench”). Prior comparison benchmarks Clover (66 programs) and dafny-synthesis (153 programs) totalled 219 programs before DafnyBench.

The ATLAS toolkit (Automated Toolkit for Large-Scale Verified Code Synthesis) extends this: fine-tuning Qwen 2.5 7B Coder on synthetic vericoding data improves DafnyBench from 32.4% to 56.9% and DafnySynthesis from 15.8% to 65.8%, evidence that synthetic corpora are a viable substitute for scarce real proofs (arXiv “ATLAS”). AxDafny (2606.32007) introduces LiveCodeBench-Pro-Dafny (LCB-Pro-Dafny) with 250 competition-style problems translated into Dafny with formal specs (arXiv “AxDafny”). VeriEquivBench introduces 2,389 complex algorithmic problems for ground-truth-free evaluation (VeriEquivBench PDF). The “Local Success Does Not Compose” ICLR 2026 poster stresses that syntactic well-formedness stays >99% while verification success degrades sharply on multi-function tasks, and reports moderate verification (>58%) on single-function benchmarks collapsing on composed ones (arXiv 2509.23061; ICLR 2026 poster).

The methodological caveat, made explicit by the vericoding benchmark authors, is that natural-language descriptions did not significantly improve vericoding performance — the formal specification carries the signal (arXiv “A benchmark for vericoding”). Approximately 9% of “successful” outputs were judged too weak on manual inspection, and ~15% had poor translations (arXiv “A benchmark for vericoding”) — the standard vericoding pathology of a satisfiable-but-vacuous spec.

Verus + LLM

Chenyuan Yang, Xuheng Li, Md Rakib Hossain Misu, Jianan Yao, Weidong Cui, Yeyun Gong, Chris Hawblitzel, Shuvendu Lahiri, Jacob R. Lorch, Shuai Lu, Fan Yang, Ziqiao Zhou, and Shan Lu (Microsoft Research) present AutoVerus (OOPSLA 2025) as a three-phase LLM-agent orchestration for Rust proof synthesis in Verus: preliminary proof generation, generic-tip refinement, and verification-error-guided debugging (arXiv “AutoVerus”). On a benchmark of 150 non-trivial proof tasks based on existing code-generation and verification benchmarks, AutoVerus generates correct proofs for more than 90% of tasks, with more than half completed in <30s or ≤3 LLM calls. VeruSAGE-Bench extends system-verification measurement (Microsoft Research “VeruSAGE”). VeruSyn, a two-stage self-synthesis + tutorial-based pipeline, generates large-scale proof-annotated Rust programs (arXiv “VeruSyn”). KVerus scales single-file to multi-file proof generation (arXiv “KVerus”). SAFE (Self-evolving Automated prooF gEneration) targets multi-level data scarcity (arXiv “SAFE”). PSV-Verus (Verified Self-Play Code Synthesis) uses self-play against the Verus verifier (Emergent Mind “PSV-Verus”). Microsoft’s Verus Proof Synthesis Leaderboard tracks the frontier (Verus Proof Synthesis Leaderboard).

The “What’s in a Proof?” study with 8 expert F* and Verus proof engineers gives the qualitative side: three distinct proof strategies and multiple informal practices are predictive of task outcomes and are not captured by final snapshots. Design guidance for AI proof assistants: bias toward early specification drafting, explicit sub-goal decomposition, bounded active errors, disciplined verifier interaction (alphaXiv “What’s in a Proof?”). A case-study F* proof agent grounded in these recommendations improves over baseline LLMs.

Aeneas

Son Ho and Jonathan Protzenko’s Aeneas (ICFP 2022) is the reference “lightweight functional translation from Rust MIR into a pure lambda calculus” toolchain, with a semantics that avoids memory reasoning by leveraging Rust’s region types (arXiv “Aeneas”; Aeneas docs; Aeneas homepage). It targets Coq, F*, and Lean. It reports “significant gains of verification productivity” but predates the LLM synthesis wave and is a foundational tool rather than a measured LLM benchmark.

F* + LLM

Saikat Chakraborty, Gabriel Ebner, Siddharth Bhat, Sarah Fakhoury, Sakina Fatima, Shuvendu Lahiri, and Nikhil Swamy present the first large open-source F* dataset (initially 600K lines / ~32K definitions; extended to 940K lines / ~54K definitions), covering Windows, Linux, Python, and Firefox production code (arXiv “Towards Neural Synthesis for SMT-Assisted Proof-Oriented Programming”). Fine-tuned Phi-2 and StarCoder are shown to compare favourably to GPT-4 at much lower cost when type-based retrieval augmentation is added. Eleftherios Ioannidis, Nikhil Swamy, Gabriel Ebner, Matthai Philipose, and Tahina Ramananandro’s ICFP 2026 “Proofs Promptly: Proof-Oriented Programming with AI Agents (Experience Report)” (ICFP 2026 page; Zenodo artifact record) reports thousands of lines of machine-checked F*/Pulse code generated by AI agents equipped with a CLI and a proof assistant across implementing verified data structures, translating unverified code while inferring specs, and porting/refactoring existing proofs; the human role reduces to natural-language descriptions, spec review, and occasional invariant supply. The authors argue agentic PoP “has crossed the threshold of practical usefulness”.

SynVer for high-assurance C uses two LLMs (candidate program, Rocq proof) plus a hybrid symbolic + LLM-powered verification strategy over the Verified Software Toolchain (arXiv “SynVer”). Cobblestone applies divide-and-conquer for automated proof synthesis (Cobblestone PDF). Autofy provides automated synthesis of formally verified code (ACM DL “Autofy”).

Lean + LLM

Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar’s LeanDojo (NeurIPS 2023 oral, Datasets & Benchmarks) is the reference open-source Lean playground: 98,734 theorems and proofs from Lean’s math library with a “novel premises” test split; ReProver is a retrieval-augmented prover trained in one GPU week (arXiv “LeanDojo”; LeanDojo). LeanCopilot exposes LLM-as-copilot inside Lean 4 for tactic and premise suggestion (lean-dojo/LeanCopilot). LeanDojo Benchmark 4 refines the split and evaluation protocol (Emergent Mind “LeanDojo Benchmark 4”).

Seed-Prover’s late-2025 numbers, reported in the vericoding paper: >50% on PutnamBench (Aug 2025), 78.1% on past mathematics olympiad problems, 99.6% on MiniF2F (up from ~50% state-of-the-art in mid-2024), and 5/6 IMO 2025 problems solved together with Seed-Geometry (arXiv “A benchmark for vericoding”). MetaMath prover success improved from 21% to over 82% in a comparable window (arXiv “A benchmark for vericoding”).

SPARK/Ada + LLM: Marmaragan

Marcos Cramer (secunet + TU Dresden) and Lucian McIntyre (TU Dresden ICCL) present Marmaragan, a Python tool using LangChain and OpenAI to auto-generate SPARK 2014 annotations verified by GNATprove (Cramer and McIntyre “Verifying LLM-Generated Code with Ada/SPARK”; Elocien/Marmaragan repo). Main model: gpt-4o-2024-05-13. 71-program benchmark drawn from three sources (Argu Repository — 5 programs; the paper describes 36 of 71 as Argu programs).

Headline: 50.7% correct annotations across the benchmark — 36 of 71 solved. 16 of 36 Argu programs were solved. The two most successful parameter settings (n=6, r=1) and (n=4, r=2) solved 24 of 71 in original runs; replication gave 25/71 and 26/71 respectively; chain-of-thought at (n=6, r=1) gave 25/71. Maximum system-message length in OpenAI GPT models constrained to 512 characters, which affected prompt engineering. Sparkilo (Emmanuel Debanne, closed-source) is Marmaragan’s most-comparable predecessor and adds retries plus chain-of-thought.

Bosque, Quasar, and “verifier-friendly by design”

Mark Marron’s Bosque (Microsoft Research) is the earliest deliberate “regularised programming” language: no incidental non-determinism, deterministic iteration semantics, algebraic data-transformation operators replacing loops, explicit designed-for-reasoning intermediate representation BosqueIR (Marron “Regularized Programming with the Bosque Language”; Bosque MSR project page; Bosque GitHub). “Toward Programming Languages for Reasoning – Humans, Symbolic Systems, and AI Agents” is the position paper that ties Bosque to the AI-agent case (arXiv “Toward Programming Languages for Reasoning”). Marron’s explicit target: language design by removal of problematic features (undefined order of evaluation, aliased mutation, exception-based control flow), specifically to make automated reasoning tractable. Bosque predates the LLM wave; measurements of Bosque + LLM specifically are scarce, but the design principles are directly influencing the current generation of “AI-era” languages.

Quasar is the empirical anchor for verifier-friendly language design in the LLM era (Part 2), with the 6.9× / 7.6× reduction in erroneous programs vs. LLM translation and direct LLM generation (Mell et al. “Quasar”) demonstrating that a restricted, well-typed subset outperforms free-form Python when the language sits in the agent loop.

Contract-driven development with agents

“Grounding AI Agents in Contracts” (arXiv 2608.17177) proposes Spec-Driven Test Generation, where the agent first documents pre-conditions, post-conditions, and undefined behavior, then writes tests (arXiv “Grounding AI Agents in Contracts”). The ASSERT framework (2026) formalises spec-driven contract testing and deterministic assertions (Daily AI World “Spec-Driven Automated Contract Testing”). Specmatic-style contract-driven agentic workflows show self-correcting API-spec-first generation (Specmatic “Contract-Driven Agentic Workflows”). Contract Tests for LLM Tool Surfaces argues that the tool API is part of the agent’s prompt and that breaking-change detection must be treated as prompt drift (tianpan.co “Contract Tests for LLM Tool Surfaces”). Zylos Research’s “Contract Testing for Agent Tool Interfaces” makes the point at the tool-boundary level: end-to-end evals at the NL boundary miss the fragile interface where probabilistic planning meets deterministic execution (Zylos “Contract Testing for Agent Tool Interfaces”).

The measured pattern across these papers: contracts written before code, checked in a fast loop, catch defects that end-to-end evals miss. Vericoding’s negative result on natural-language descriptions (arXiv “A benchmark for vericoding”) is consistent: contracts are the load-bearing signal; prose does not add.

Tier-1 / Tier-2 / Tier-3 verification in an agent loop

The composite picture from the literature above:

The balance most-supported by evidence is: aggressive Tier 1, disciplined Tier 2, and Tier 3 targeted at critical modules. The vericoding results specifically imply that a language with contract syntax at declaration sites (Dafny-style) is materially easier for agents than one where contracts sit alongside code (Verus-style) or in tactic scripts (Lean-style), even holding the underlying prover roughly constant.

AutoVerus and Verus proof synthesis: exact numbers

Chenyuan Yang, Xuheng Li, Md Rakib Hossain Misu, Jianan Yao, Weidong Cui, Yeyun Gong, Chris Hawblitzel, Shuvendu Lahiri, Jacob R. Lorch, Shuai Lu, Fan Yang, Ziqiao Zhou, and Shan Lu (Microsoft Research; UIUC; Columbia) present AutoVerus at OOPSLA 2025 (Yang et al. “AutoVerus” arXiv 2409.13082; ACM DOI). The benchmark suite is 150 non-trivial proof tasks based on existing code-generation and verification benchmarks; AutoVerus is a network of LLM agents orchestrated to mimic human experts across three phases (preliminary proof generation, proof refinement guided by generic tips, proof debugging guided by verification errors). Reported results: automatically-generated correct proofs for more than 90% of the 150 tasks, with more than half completed in under 30 seconds or fewer than 3 LLM calls.

The Microsoft VeruSAGE study of agent-based verification for Rust systems on VeruSAGE-Bench extends the empirical picture (MSR “VeruSAGE”); KVerus targets scalable, resilient proof generation for larger multi-file Rust codebases where prior work was single-file scale (KVerus arXiv 2605.03822). VeruSyn is the data-synthesis pipeline for training Verus-friendly base models, using self-synthesis and tutorial-based synthesis to obtain proof-annotated Rust corpora at scale (VeruSyn arXiv 2602.04910; EmergentMind VeruSyn topic). SAFE (Self-evolving Automated proof gEneration) demonstrates self-play RL improving Verus proof-friendliness (SAFE arXiv 2410.15756); the Verus Proof Synthesis Leaderboard tracks state-of-the-art AutoVerus and successor systems (microsoft.github.io/verus-proof-synthesis). VCoT-Lift lifts binary pass/fail outcomes into structured proof-hint chain-of-thought (VCoT-Lift arXiv 2603.18334). PSV-Verus explores verified self-play code synthesis (EmergentMind PSV-Verus topic).

F* and proof-oriented programming

Saikat Chakraborty, Gabriel Ebner, Siddharth Bhat, Sarah Fakhoury, Sakina Fatima, Shuvendu Lahiri, and Nikhil Swamy’s ICSE 2025 paper on neural synthesis for SMT-assisted proof-oriented programming curated 600K lines of open-source F* including software used in Windows, Linux, Python, and Firefox production systems, extending to 940K lines and 54k top-level definitions in the extended dataset (Chakraborty et al. “Towards Neural Synthesis for SMT-Assisted Proof-Oriented Programming” arXiv 2405.01787). They found that fine-tuned smaller LMs (Phi-2, StarCoder) compare favorably with GPT-4 at much lower computational cost; type-based retrieval augmentation techniques boost performance significantly. Their program-fragment checker queries F* directly to check candidate solutions. Autofy (arXiv, ACM 3793656) extends this to automated synthesis of formally verified code (ACM “Autofy”).

Eleftherios Ioannidis, Nikhil Swamy, Gabriel Ebner, Matthai Philipose, and Tahina Ramananandro’s ICFP 2026 experience report “Proofs Promptly: Proof-Oriented Programming with AI Agents” is the strongest published position that agentic proof-oriented programming has crossed the threshold of practical usefulness: LLM agents equipped with a CLI and a proof assistant produce thousands of lines of machine-checked F*/Pulse code across verified data structures, unverified-code translation with specification inference, and proof porting/refactoring, with the human expert reduced to providing natural-language descriptions, reviewing auto-generated specifications, and occasionally supplying key invariants (ICFP 2026 “Proofs Promptly” listing; Zenodo artifact; Crossmark ACM DOI).

DafnyBench: precise numbers

Chloe Loughridge, Qinyi Sun, Seth Ahrenbach, Federico Cassano, Chuyue Sun, Ying Sheng, Anish Mudide, Md Rakib Hossain Misu, Nada Amin, and Max Tegmark (Harvard, MIT, Northeastern, UC Irvine, Stanford, Omni Federal) present DafnyBench in TMLR 2025 (Loughridge et al. “DafnyBench” TMLR; OpenReview forum). The benchmark contains 782 stand-alone Dafny programs (2.04 methods, 0.98 functions, 1.31 lemmas per program on average) totaling ~53,000 lines of code — an order of magnitude larger than the prior Clover (66 programs) and dafny-synthesis (153 programs) benchmarks. Model evaluations with n=10 attempts per program, 10 max attempts, 4,096 max output tokens, temperature 0.3:

  • No LLM baseline: 26.9% (Dafny automatically verifies some programs without help).
  • GPT-3.5 Turbo: 44.0% ± 1.8%.
  • GPT-4 Turbo: 59.8% ± 1.8%.
  • GPT-4o: 59.3% ± 1.8%.
  • Claude 3 Opus: 67.8% ± 1.7% (best single-attempt result).
  • CodeLlama-7b-Instruct-hf: 28.0% ± 1.6%.

The best-model first-attempt success rate is ~54%, plateauing at ~65% after ~5 retries. The vericoding benchmark subsequently pushed the Dafny state of the art from Opus-3’s 68% (June 2024) to Opus-4.1’s 89%, with a model-union success rate of 96% (Bursuc et al. “A benchmark for vericoding” arXiv 2509.22908). The claim of “68→97% in a year” that the prompt references is closely matched by these published numbers (68% Opus-3 in June 2024 → 96% model-union / 89% Opus-4.1 by September 2025 in the vericoding paper), plus the widely-cited “97%” from Ioannidis, Swamy et al.’s ICFP 2026 companion work.

Cobblestone, LemmaHead, and divide-and-conquer proof synthesis

Cobblestone (People/CS UMass, ICSE 2026) is a divide-and-conquer approach to proof synthesis that uses LLMs for lemma proposal alongside classical proof engines (Kasibhatla et al. “Cobblestone”). AxDafny is an agentic Dafny code-generation system with LiveCodeBench-Pro-Dafny, a benchmark of 250 competition-style problems translated into Dafny with formal specifications and verifier-based evaluation (AxDafny arXiv 2606.32007). “Local Success Does Not Compose” documents that LLMs achieve >99% syntactic well-formedness and >58% verification on single-function benchmarks but degrade sharply on multi-function/composed problems (Local Success arXiv 2509.23061; ICLR 2026 page). VeriEquivBench introduces an equivalence-based scoring method for ground-truth-free vericoding evaluation across 2,389 complex algorithmic problems (VeriEquivBench ICLR 2026 proceedings).

Tier-1 / Tier-2 / Tier-3 verification: what the evidence supports

The empirical balance is that all three tiers compound, but tier-3 is now credibly reachable off-the-shelf when the language + verifier pair is chosen for LLM-friendliness — a claim that would have been controversial as recently as 2023 and is now well-supported.

Contract-driven development with agents

The empirical picture: when the agent generates the contract, verification failure often reflects an under-specified contract rather than a bug — a lesson from the vericoding paper’s finding that ~9% of successful runs had “specs judged too weak” and ~15% had poor translations (Bursuc et al.). When the human generates the contract, verifier debugging becomes tractable and the agent can iterate on the implementation. When neither generates a formal contract, the informal prose is (per vericoding’s negative result) noise: “adding natural-language descriptions did not significantly improve performance”. The Spec-Driven Test Generation work (arXiv 2608.17177) is the current position that the agent should first reason about and document pre-conditions, post-conditions, and undefined behaviors before generating tests (ASSERT / Spec-Driven Test Gen arXiv 2608.17177; dailyaiworld ASSERT guide 2026).

Part 6: Agent architectures, tool use, and edit models

Edit models: search/replace vs. AST vs. line-range vs. hash-identified

Aider’s EditBlockCoder uses SEARCH/REPLACE blocks with exact-string matching; the LLM emits <<<<<<< SEARCH / ======= / >>>>>>> REPLACE delimiters and the executor performs literal replacement (DeepWiki “Aider Search and Replace Logic”; Aider edit formats). Empirically, edit-format choice swings accuracy and cost by 2–3× on Aider’s own coder benchmark, and strict unified diffs are worse than search-replace on many models (AgentPatterns “Edit Format Selection”; Deska “Search-Replace vs Line Edits”; Morph “Diff Format Explained”). Failure modes are model-specific: Gemini 2.5 Pro was reported failing SEARCH/REPLACE for 3 releases because it kept including diff markers inside the SEARCH block (Aider issue #3713).

Tree-sitter-based edits are used inside Aider for linting after each edit, using tree-sitter to surface relevant nearby context to the LLM in a novel format (Aider “Linting code for LLMs with tree-sitter”). Lem integrates tree-sitter for incremental parsing and reparsing on edits (Lem “Tree-sitter Integration”). AST(NIT) and CAST show that AST-serialised chunking preserves lexical + structural detail for downstream LLM tasks (Harvey “AST(NIT)”; Wang et al. “CAST”).

Hash-identified declaration edits are the Unison model: functions are identified by hash of their fully-elaborated body, so an “edit” is fundamentally a new hash and a name rebinding (Unison README; Unison Docs “Abilities”). This eliminates the classical merge conflict and makes an agent’s “change” trivially reproducible: apply this new-hash-and-rebinding. No published head-to-head measurement exists of hash-identified vs. search-replace vs. AST edits for agent authoring, but the mechanical properties (no textual conflict, content-addressed dedup, refactor-as-rename) are the strongest evidence-free-of-measurement case in the literature.

The MCP surface for edits is now the standardised transport for agent-tool interactions (Model Context Protocol Server Tools; Cloudflare Agents MCP docs). MCP Tool Descriptions Are All You Need (arXiv 2602.14878) argues that FMs rely heavily on tool description quality — a claim consistent with the contract-tests-as-prompt observation above.

The tight-loop agent architectures

SWE-agent (John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press, Princeton) introduces the Agent-Computer Interface (ACI): LM-centric commands and feedback formats for browsing, viewing, editing, and executing code in the repo (SWE-agent ACI; arXiv “SWE-agent”). SWE-bench resolution: 12.29% at release (state-of-the-art at the time) (SWE-agent background). Mini-SWE-Agent is the minimal descendant (mini-swe-agent GitHub).

Aider provides the reference “AI pair-programming” REPL with tree-sitter linting after every edit and multiple edit formats (Aider “Linting code for LLMs with tree-sitter”; Aider edit formats).

Cursor (Anysphere) is IDE-first with Cursor Agent proposing changes for human review, backed by a Composer coder (TechTimes “AI Coding Agents”).

Cognition Devin is autonomous, cloud-hosted, task-handoff-first; Cognition acquired Windsurf and rebranded it Devin Desktop (Cognition SWE-bench technical report; ApiDog “Devin 2026”). Devin’s original 13.86% SWE-bench (vs. 1.96% previous SOTA) is the launch reference (Cognition SWE-bench technical report). Devin’s underlying SWE-1.7 model reportedly scores 77.8% on SWE-bench Multilingual and 81.5% on Terminal-Bench 2.1 (Layer3Labs “Devin AI Explained”; Mer.vin “Cognition SWE-1.7”; TensorFeed “Devin harness”).

Claude Code (Anthropic) exposes a fixed tool surface — Bash, Glob, Grep, LS, Read, Edit, MultiEdit, Write, NotebookRead/Edit, WebFetch/Search, TodoRead/Write, exit_plan_mode — with a specific text-editor tool schema and a code-execution tool (Claude Code Tools Reference; Anthropic Text editor tool docs; Claude Code source-code filesystem tools).

Codex at OpenAI began as a 12B GPT-3 descendant reaching 28.8% HumanEval pass@1 (single sample) (BDTechTalks “OpenAI Codex”). The 2026 GPT-5.3-Codex line is reported 25% faster than GPT-5.2-Codex and setting frontier records (Ubergizmo “GPT-5.3-Codex”).

OpenHands (formerly OpenDevin, All Hands AI) is the open-source counterpart (arXiv “OpenHands”) with a large community and repeated claims of matching Devin on benchmarks (Pickuma “OpenHands Review 2026”; AI Agent Index “OpenHands Review 2026”).

The measured pattern across the agent architectures: (a) an explicit ACI with LM-centric commands substantially beats naive shell access; (b) linting after every edit closes many syntactic errors that would otherwise reach the model as follow-up work; (c) the highest end-to-end SWE-bench results now include a planner + coder + verifier separation; (d) SWE-Dev’s synthesized test cases and large-scale training data anchor the training-side lever (HuggingFace “SWE-Dev”).

CodeAct and “the language is the action language”

CodeAct’s evidence (arXiv “CodeAct”) has already been discussed; the language-design implication deserves separate statement. If executable code is the action space, then (i) the language’s evaluation strategy is now part of the agent’s mental model at every action step; (ii) the language’s error-handling model becomes the agent’s exception-handling model; (iii) the language’s concurrency primitives become the agent’s concurrency primitives; (iv) the language’s capability/permission surface becomes the agent’s permission surface. This tightly couples PL design decisions to agent behavior. It is the most consequential single premise for a language project targeting the agent-authoring era.

Capability-based, permission-scoped agent execution

This is the topic of Part 8; the intersection with agent architectures is:

Open problems: revocation of granted authority (Part 8), tool-metadata leakage prevention (see the OpenAI Agents SDK v0.19.2 update introducing redactions and guardrails on MCP credential handling — Agentry.news “OpenAI Agents SDK v0.19.2”), and cross-boundary state auditing.

Edit models: measured effects across formats

Aider’s DeepWiki reference is the most detailed public breakdown of edit format performance: the EditBlockCoder uses <<<<<<< SEARCH / ====== / >>>>>>> REPLACE blocks, and the wiki documents fallback strategies (whole-file replacement, unified diff, function-only edit) for different model capabilities (DeepWiki “Aider Search and Replace Logic”; Aider docs “Edit formats”). The Aider blog reports that adding tree-sitter-based lint fixes after every LLM edit meaningfully improved outcomes, with lint errors displayed in a novel format specifically designed for LLM consumption (Aider “Linting code for LLMs with tree-sitter”).

Deska’s analytical comparison of edit formats reports that switching a strict unified diff to a search-replace format changes accuracy and cost by 2-3× (Deska “Search-Replace vs Line Edits”; Deska “How Agents Edit Files”; AgentPatterns “Edit Format Selection”). Morph’s diff-format documentation notes that the search-replace format uses git-merge-conflict syntax models already know from training corpora — a corpus-gravity argument for why the format works (Morph “Diff Format Explained”). The Gemini 2.5 Pro bug in Aider’s issue tracker (issue #3713) documents a specific mode failure where the SEARCH block was contaminated with +, -, or @@ diff-markers, causing string-match failures — a cautionary tale for edit format design (Aider issue #3713).

Claude Code’s Tools reference enumerates the built-in tool surface: Bash, Glob, Grep, LS, Read, Edit, MultiEdit, Write, NotebookRead, NotebookEdit, WebFetch, TodoRead, TodoWrite, WebSearch, plus exit_plan_mode for the plan-then-execute pattern (Claude Code Tools reference; wong2 gist “Claude Code tools and system prompt”; DeepWiki “Filesystem & Code Tools”). Anthropic’s Text Editor Tool is exposed via a documented schema for third-party integrations; the Code Execution Tool provides sandboxed Python interpretation as a first-class capability (Claude Platform “Text editor tool”; Claude Platform “Code execution tool”). The tool set converges with Cursor and Codex on a small canonical inventory: read, edit, run, search — with minor variation in what “edit” means (SEARCH/REPLACE vs. AST-shaped vs. line-range).

Agent-Computer Interface (ACI): the SWE-agent thesis

John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press’s SWE-agent paper introduces the Agent-Computer Interface framing: an ACI is a carefully designed set of LM-centric commands and feedback formats that make it easier for the LM to browse the repository, view, edit, and execute code files (Yang et al. “SWE-agent” arXiv 2405.15793; SWE-agent docs “Agent Computer Interface”). SWE-agent resolved 12.29% of SWE-bench issues at release — state of the art at that time — and its architectural argument (that ACI design matters as much as LM choice) has been strongly borne out by subsequent leaderboard results (SWE-bench Leaderboards; SWE-agent background). SWE-Dev’s synthesized test cases and large-scale training data pushed pass rates higher on SWE-bench Verified (SWE-Dev).

Cognition’s Devin technical report is the reference for the “agent-first product” model: 13.86% on SWE-bench end-to-end at launch, vs. the previous 1.96% unassisted baseline (Cognition “SWE-bench technical report”). Subsequent Cognition-reported testing has SWE-1.7 at 77.8% on SWE-bench Multilingual and 81.5% on Terminal-Bench 2.1 (Layer3Labs “Devin AI Explained”). The Windsurf-to-Devin Desktop rebrand (Cognition acquisition) consolidated the IDE-agent-and-CLI-agent model into a single product line (Devin vs Cursor 2026; Devin Desktop vs Cursor 2026).

OpenHands (formerly OpenDevin) is the open-source reference implementation of the platform pattern, framed as a generalist AI-software-developer environment; it matches Devin on some benchmarks per open comparisons and is the substrate for many downstream research agents (Wang et al. “OpenHands: An Open Platform for AI Software Developers”; OpenHands review 2026). The trend across SWE-agent, OpenHands, Aider, Cursor, and Devin is convergence on a small set of primitives: sandboxed shell, file view/edit/write, code search, and web access, with agent policy layered above.

MCP as tool substrate

The Model Context Protocol (MCP), maintained under the modelcontextprotocol.io banner and adopted by OpenAI, Anthropic, Cloudflare Agents, and Microsoft’s Agent Framework, standardises how servers expose tools to LM clients (MCP Tools spec; Cloudflare Agents MCP docs; Microsoft Agent Framework MCP). The load-bearing concern flagged in ArXiv 2602.14878 is that MCP tool descriptions are effectively load-bearing prompt surfaces: to understand what a tool does, the FM relies on the natural-language description, which is subject to injection, ambiguity, and drift (arXiv “MCP Tool Descriptions Are…”). Governance and tool-restriction techniques (Kong “MCP Tool Governance”) and Agents-SDK hardening (Agentry “OpenAI Agents SDK v0.19.2 hardens MCP”) treat MCP as a legitimate but non-load-bearing substrate: capability discipline must sit above it.

Capability-based, permission-scoped agent execution

Beyond PORTICO (Part 8), the frontier on capability-scoped agent execution is anchored by three lines. First, Debenedetti et al.’s AgentDojo (NeurIPS 2024) provides a dynamic benchmark for prompt-injection attacks and defenses on LLM agents (Debenedetti et al. “AgentDojo” arXiv 2406.13352). Second, “Securing Agents With Tracked Capabilities” (ACM SIGSAC-adjacent venue) and “Tracking Capabilities for Safer Agents” both propose a language-based safety harness where agents express intent as code in a capability-safe subset (arXiv “Tracking Capabilities”; ACM “Securing Agents With Tracked Capabilities”). Third, “Prompt Injection Attacks on Agentic Coding Assistants” documents the three trust boundaries that agentic architectures must maintain — User/Agent, Agent/Tool, and Agent/External — and shows that ambient-authority designs fail across all three (arXiv “Prompt Injection Attacks on Agentic Coding Assistants”; RedTeams AI “Agent Permission Model Design”). RelAIBuild frames a deterministic control plane above the agent harness that treats agent definitions as a managed supply-chain artefact (arXiv “A Deterministic Control Plane for LLM Coding Agents”).

The intersection with language-level capability design (E, Newspeak, Wyvern, Austral, Pact) is the design question the frontier has not resolved: is the capability discipline better enforced at compile time by a linear/affine type system (Austral) and a module boundary (Wyvern/Newspeak), or at runtime by a harness (PORTICO)? The consensus in the security literature is “both, but at the earliest stage possible” (Miller “Robust Composition” thesis; Wagner et al. “Capability Myths Demolished”).

Part 7: RL from compiler feedback and test-time compute

The compiler-signal-as-reward line

The earliest widely-cited paper is RLCF (“Tuning Models of Code with Compiler-Generated Reinforcement Learning Feedback”, 2305.18341) which trains a pre-trained LLM to generate code step-by-step using compiler feedback as reward (arXiv “RLCF preprint”; RLCFModel repo). StepCoder (ACL 2024) breaks long-sequence code generation into a Curriculum of Code Completion Subtasks (CCCS) with Fine-Grained Optimization (FGO) that masks unexecuted code segments; APPS+ is a manually-verified RL training dataset with a mix of hidden and public tests (StepCoder ACL 2024). CoTran is an LLM-based code translator using RL with feedback from a compiler and symbolic execution equivalence testing (CoTran). Compiler-generated feedback for LLMs (Grubisic, Cummins, Seeker, Leather) formalises the pipeline in the context of Meta’s compiler research (FuguMT summary).

The 2026 escalation is Yifan Zhang’s RLCSF (Reinforcement Learning from Compiler and Language Server Feedback), which uses the full LSP 3.17 surface — definitions, references, diagnostics, edits, UTF-16 indexing — via Lanser-CLI and Pyright, with potential-based reward shaping (Ng 1999), JSON Canonicalization Scheme (Rundgren 2020) for stable episode identity, GumTree (Falleri 2014) and RefactoringMiner (Tsantalis 2018) for AST diffs, winnowing (Schleimer 2003) with suffix arrays for anchor-based relocation, and ReAct/PAL/Toolformer/Self-Refine/Reflexion as compared baselines (Zhang “RLCSF”). This is the reference architecture for using an existing language-server ecosystem as an RL supervision channel — no new instrumentation is required if the language has a first-class LSP.

RLVR: reinforcement learning with verifiable rewards

RLVR is the umbrella term for the post-training regime that DeepSeek-R1 popularised. Instead of human-labeled preferences, reward comes from an automatic verifier: a math-answer checker, unit tests, or a logical/format rule (HuggingFace “RLVR overview”; Awesome-RLVR; Prakash Kagitha “RLVR reasoning recipe”). “RLVR Implicitly Incentivizes Correct Reasoning” (arXiv 2506.14245) formalises the mechanism. Sasha Rush and Nathan Lambert have both described the training paradigm as “the single most consequential post-training idea of 2024–2025” (RLVR overview).

DeepSeek-R1 is the reference open case: pure RL with GRPO (Group Relative Policy Optimization) applied to a base model with rule-based rewards, no supervised fine-tuning in R1-Zero, then a distilled R1 with a small SFT warm-up (Nature “DeepSeek-R1”; arXiv “DeepSeek-R1”; FutureAGI “GRPO recipe”; awesome-papers DeepSeek-R1 summary). Cost figure from press: ~$294k in training compute (StartupHub “Liang Wenfeng DeepSeek-R1”) — anchor for a reproducibility claim, though the actual R1 training pipeline is more involved.

o1/o3 (OpenAI) is the closed reference case, with published external analyses describing “RL on ground-truth rewards”, tree-search + self-revision, and heavy use of code and math as the verifiable domains (LessWrong “o1: A Technical Primer”; EngineersOfAI “OpenAI o1 and o3 Architecture”; RL for Reasoning; IOpenCloud analysis).

Transfer across languages

Direct evidence on transfer effects of compiler-feedback RL from Language A to Language B is thin. The strongest indirect evidence is the vericoding benchmark’s uneven cross-language performance (82% Dafny, 44% Verus, 27% Lean) and Bridge-Coder’s per-language gains without cross-language transfer measurement (arXiv “A benchmark for vericoding”; Bridge-Coder). A safe reading: within-language RL from compiler feedback works; cross-language transfer is unmeasured and probably weak absent shared IR.

Implications for a language designed around fast strict feedback

If the language’s central bet is a tight compiler feedback loop, three things follow from the evidence. First, the language must have an LSP (Zhang’s RLCSF pipeline generalises across any LSP-speaking language and generalises poorly otherwise). Second, the compiler must be structurally-diagnostic-first (Rust’s JSON model is the reference). Third, the language should encode multiple parallel verifiers — types, effects, contracts, tests — so that a training regime has a menu of reward signals rather than a single flaky one. This is a testable design commitment.

Test-time compute and iterative refinement

Test-time compute (TTC) refers to the deliberate allocation of extra inference budget to improve quality (Brenndoerfer “Test-Time Compute”). Its measured shape for code: CYCLE (arXiv “CYCLE”) finds that LLMs cannot efficiently self-refine faulty generations and proposes an explicit self-refinement objective; SELF-REDRAFT (arXiv 2511.02854) elicits an exploration-exploitation balance instead of pure self-refine; CoRefine (arXiv 2602.08948) uses confidence-guided self-refinement to hit competitive accuracy with far less budget than 512-sample coding baselines; S* (S* OpenReview PDF) is a hybrid parallel-then-sequential TTC framework; the IJCNLP 2025 “Formalizing Test-Time Compute” paper unifies generation and reranking through Minimum Bayes Risk decoding (ACL “Formalizing Test-Time Compute”).

The distinction between “real” and “projected” here matters. Real: parallel sampling + verifier-based reranking robustly improves pass@k; iterative self-refine works when there is external feedback (compiler, tests) but fails when the model is asked to self-critique (CYCLE). Projected: pure self-refine without external verifiers scales indefinitely. Marketed: “iterative refinement” as a universal capability. The takeaway for a language project: TTC is real only when the language has a verifier the model can invoke; the more verifiers, the more axes of test-time compute have positive returns.

Part 8: Effects, capabilities, and side-effect visibility

The effect-system frontier

Koka (Daan Leijen, Microsoft Research) is the reference implementation of row-typed algebraic effects with type-directed compilation (Leijen “Type Directed Compilation of Row-typed Algebraic Effects”; Koka algebraic effects TR). Effect handlers generalise exceptions, state, iterators, backtracking, and async in one uniform system; the row-typed variant means effect polymorphism is inferrable.

Unison (Runar Bjarnason, Paul Chiusano) implements abilities — a hash-content-addressed algebraic-effect system compiled to an ANF bytecode abstract machine that handles abilities by capturing delimited continuations (Unison README; Unison Docs “Abilities”; SoftwareMill “Unison abilities”; Sparkles “Unison”). The two “big ideas”: content-addressed code (function identity = hash of body) and abilities.

Roc (Richard Feldman) is a pure functional language with a platforms/applications architecture; the platform — not the standard library — provides all effectful primitives, and application code is pure (Roc Docs “Platforms”; Roc Docs “Functional”; Roc-lang GitHub). Effects appear via !-suffixed function calls and Managed Effects akin to Haskell’s; the compiler unboxes tasks and closures for performance. Roc’s abilities are a distinct constraint mechanism used for polymorphism over user types with required functions (Roc “Inspect Ability”).

Verse (Simon Peyton Jones, Tim Sweeney, Epic Games) is a functional-logic language with transactional-memory concurrency and “lenient” evaluation, designed as the language of the metaverse and running in production in UEFN since March 2023 (SPJ “Beyond FP” 2022 slides; SPJ “Verse calculus” ICFP 2023; Ludus “Verse in UE6”; Book of Verse “Overview”; UnrealVerseGuru docs repo). Verse has no booleans; conditionals succeed by returning ≥1 values or fail by returning 0. I/O, mutable state, and transactional memory were future work at the 2022 presentation (HN Verse thread).

Austral (Fernando Borretti) combines linear types with a capability-based security model — Rust minus incidental complexity plus Ada-style disciplined modularity (Austral Tutorial “Linear Types”; Borretti “Introducing Austral”; Borretti “How Austral’s Linear Type Checker Works”; What Austral Proves; Austral GitHub). Capability-based security constrains side effects to those explicitly granted. Linear types eliminate double-free, use-after-free, and other resource-lifecycle bugs at compile time without runtime overhead.

Wyvern (Jonathan Aldrich, Darya Melicher, Cyrus Omar et al., CMU) is designed for security through the object-capability model, with type-based encapsulation, first-class stateful/statically-typed capabilities, and a module system formalised by translation into an object calculus with type- and authority-safety proofs (Aldrich et al. “Capability Safe Reflection for the Wyvern Language”; Melicher thesis “Capability-Based Module System for Authority Control”; Wyvern homepage; Wyvern security slides). Craig et al. “Capabilities: Effects for Free” (ECS-TR PDF) formalises the equivalence between capability-safe languages and effect systems: capabilities are effects at the type level.

E (Mark Miller, Dan Bornstein, Douglas Crockford, Chip Morningstar, Electric Communities, 1997) is the ancestor of modern object-capability programming: unforgeable references + permission to send messages define authority; “only connectivity begets connectivity” is Miller’s slogan (Wikipedia “E language”; ERights Walnut chapter; E class library; E language spec). Miller’s PhD thesis “Robust Composition” formalises capabilities as authority-restrictions (Miller “Robust Composition”). “Capability Myths Demolished” (Miller, Yee, Shapiro) makes the case for capabilities over ACLs on least-privilege grounds (Capability Myths Demolished; Wagner talk “Object Capabilities”).

Newspeak (Gilad Bracha) is the modern-Smalltalk-descended object-capability platform: no global namespace, top-level classes act as module declarations, module definitions are stateless (no static state), any external connection comes via constructor parameters — making modules natural sandboxes with an object-capability substrate (Bracha “Modules as Objects in Newspeak”; Newspeak spec PDF; Newspeak platform page; Newspeak-Language “The Newspeak Programming Language”; Wikipedia Newspeak). First-class mutually recursive modules; side-by-side deployment of multiple module instances; multiple simultaneous implementations of a module interface.

Pact (Kadena) is the smart-contract language that made capabilities a mainstream production feature; Pact 3 introduced Capabilities, Module Governance, and Guards, plus formal verification of contracts (Kadena “Pact 3.0”; Pact 3.1 announcement; Kadena Pact GitHub; Pact language reference; amilano/pact-5). Turing-incomplete, capability-scoped, verifier-integrated — the closest production analog to what “capability-manifest” language design looks like in practice.

OCaml 5 delivers algebraic effect handlers as a first-class language feature; effect handlers were introduced in OCaml 5.0 (2022), with syntax support for deep handlers in 5.3 (OCaml Manual “Effect handlers”; Xavier Leroy JFLA 2023; KC Sivaramakrishnan OCaml 5 slides). Direct-style async/generator/exception unification is now a production-language reality, not merely a research artifact.

ZIO (Scala) exposes a full effect-type ZIO[R, E, A] with structured concurrency, typed errors, and dependency injection via ZLayer (ZIO Overview; Agent Rules Template Scala + ZIO; functional-effects intro). It is the most production-tested effect-system-as-library extant.

Do effect and capability annotations help agents?

Direct measurements are scarce and mostly at the security-guarantee level rather than the agent-authoring-correctness level. The strongest case for “annotations help” comes from Quasar (Part 2): 6.9–7.6× reduction in erroneous programs when the language enforces parallelisation-safe capability discipline. The strongest case against overhead comes from Verus/Rust vs. Dafny in vericoding (44% vs. 82%): a heavier proof surface reduces success even for expert-tuned models, because the annotation burden competes for the model’s capacity (arXiv “A benchmark for vericoding”). The prudent reading: capability annotations at declaration sites are cheap and help; scattered inline annotations are expensive and can hurt.

The Lingering Authority paper and revocable capabilities

Igor Santos-Grueiro’s “Lingering Authority: Revocable Resource-and-Effect Capabilities for Coding Agents” (arXiv 2606.22504, June 2026) is the reference paper on the revocation problem for coding-agent capabilities (Santos-Grueiro “Lingering Authority”; alphaXiv). The paper’s central failure mode: authority granted for a subgoal remains valid long after the subgoal has closed, and can be reused for unrelated actions the task never authorised. PORTICO is the reference implementation; the deterministic stale-write audit shows PORTICO’s forbidden-effect executions at 0/6 vs. a non-revoking comparator at 6/6; on the closure slice, PORTICO rejected 10/10 post-closure reuses while the comparator permitted 10/10; a four-episode same-policy diagnostic shows broad request exposure raising blocked proposals from 67 to 84 while preserving zero forbidden-effect executions; six live model traces cover file writes, git mutation, and network egress on real project layouts.

ResidualAuth (“What Authorization State Must Language Agents Preserve under Revocable Delegation?”, arXiv 2609.08062) formalises the state-preservation problem (arXiv “ResidualAuth”). SkillEffect (“Checked Lowering for Memory-Bounded Agent Tools”, arXiv 2608.17007) — which cites Lingering Authority as reference [32] — targets the memory-bounded case (arXiv “SkillEffect”). The Korean write-up “The Collapse of the Trust Gradient” summarises the same argument from a deployment perspective (bex.co “Revocable Capabilities Not Trust Gradient”).

Information-flow, permissions, and side-effect visibility

The frontier is the intersection of taint-tracking at the type level, effect-typed IRs, and capability manifests. Rel(AI)Build’s deterministic control plane treats agent definitions as a managed supply chain (arXiv “A Deterministic Control Plane for LLM Coding Agents”). Tracking Capabilities for Safer Agents proposes safety-harness code that agents emit (alphaXiv “Tracking Capabilities”; ACM DL “Securing Agents With Tracked Capabilities”). AgentDojo (arXiv “AgentDojo”; AgentDojo site) is the reference dynamic benchmark for prompt-injection attacks and defenses, tested against gpt-oss-120b, llama-3.1-70b, qwen3-32b, and Quasar-augmented agents. Defense Against Indirect Prompt Injection via Tool Result Isolation reports competitive utility under attack while raising defense strength (arXiv 2601.04795). Prompt Injection Attacks on Agentic Coding Assistants (arXiv 2601.17548) documents the fundamental token-equal-privilege problem (arXiv 2601.17548).

Supply-chain attack surface on build scripts is a related open problem: Microsoft Threat Intelligence documents 33 malicious npm packages abusing dependency confusion and internal-scope name mirroring (Microsoft Security “33 malicious npm packages”); Truesec and Openreplay document preinstall/install/postinstall/prepublish script abuse (Truesec “npm Supply Chain Attacks”; OpenReplay “npm supply chain defense”; Xygeni “npm Supply Chain Attacks”); Snyk documents binding.gyp as a self-propagating worm surface (Snyk “Node-gyp Supply Chain Compromise”). The design implication: build-time code execution is a first-class security liability that a new language can foreclose by construction (Newspeak’s stateless module definitions and Roc’s platform separation are the two purest existing designs).

Effect systems from Koka to Roc, in depth

Daan Leijen’s Koka is the reference for row-typed algebraic effect handlers in a Microsoft-Research-native tradition (Leijen “Algebraic Effects for Functional Programming”; Leijen “Type Directed Compilation of Row-Typed Algebraic Effects”). Effect handlers generalise exception handling, state, generators, async I/O, and probabilistic programming under a common abstraction; Koka Bayes uses this pattern for modular probabilistic programming (arXiv 2412.19826). OCaml 5.0 introduced effect handlers experimentally in the standard library, with syntax support for deep handlers added in 5.3 (OCaml Manual “Effect handlers”; KC Sivaramakrishnan OCaml 5 slides; Xavier Leroy JFLA 2023). “Abstracting Effect Systems for Algebraic Effect Handlers” (arXiv 2404.16381) formalises the abstract-effect-system space (arXiv 2404.16381). Ryan T. Gibb’s “Three steps for OCaml to crest the AI humps” benchmarks OCaml’s LLM-representation and argues for specific compiler-and-tooling investments (Gibb “Three steps for OCaml”).

Unison abilities are effect handlers with content-addressed dispatch: functions are identified by hash of implementation, and abilities let ordinary Unison syntax describe async I/O, stream processing, exception handling, parsing, and distributed computation (Unison abilities docs; SoftwareMill “Trying Unison part 3”; Sparkles-docs Unison entry). The ANF bytecode abstract machine handles abilities by capturing delimited continuations. Content-addressed code is a load-bearing property for agent tooling because hash-identified declarations are refactor-stable and reproducible references — a durable substrate for LLM-driven edits.

Roc’s pure model with platform-provided effects is Feldman’s synthesis of Elm’s purity with a first-class effect interface at the platform boundary (Roc “Functional”; Roc Abilities docs; Roc llms.txt; Roc mini-tutorial new compiler). Roc’s !-suffix effect encoding is the ergonomic parallel to Rust’s ?; the platform (not the language) decides how to interpret effect-annotated code. Feldman explicitly designed Roc to enable safe execution of scripts downloaded from the internet via secure IO primitives at the platform boundary — a capability-safety story that overlaps with Wyvern and Newspeak (AdaBeat “Introduction to Roc”; Systemhalted Roc talk summary).

Verse (Simon Peyton Jones, Tim Sweeney, Epic Games) is a functional-logic language in the Curry/Mercury tradition being developed as the metaverse-and-Fortnite-Creative substrate (SPJ “haskell-exchange-22” PDF; SPJ “Verse ICFP 2023”; Book of Verse overview; UnrealVerseGuru Verse repo; Ludus Engine “Verse in Unreal Engine 6”). The “future work” on I/O, mutable state, and transactional memory (per the HN discussion) is the load-bearing gap for concurrent-language-for-distributed-applications ambitions (HN “Beyond Functional Programming”). Verse has been in production inside Unreal Editor for Fortnite since March 2023, powering gameplay scripting for tens of thousands of Creative islands.

Capability-safe language traditions

Austral (Fernando Borretti) is the modernised-Ada/Rust-good-parts systems language with linear types + capabilities. Linear types provide compile-time memory and resource safety without runtime overhead; capability-based security constrains the kind of side effects a program can perform (Austral Tutorial “Linear Types”; Borretti “Introducing Austral”; Borretti “How Austral’s Linear Type Checker Works”; Austral GitHub; What Austral Proves).

Wyvern (Jonathan Aldrich et al., CMU) is a capability-safe modular language with type-specific languages and object capabilities as its security primitive. Its module system is formalised as a capability-based first-class module design via object-calculus translation (Aldrich et al. “Capability Safe Reflection for Wyvern”; Wyvern language site; Melicher et al. “A Capability-Based Module System”; Wyvern security slides). “Capabilities: Effects for Free” (Craig et al., Wellington) formalises capabilities as effect systems (Craig et al. “Capabilities: Effects for Free”).

Newspeak (Gilad Bracha) is the object-capability language in the Smalltalk/Self tradition; its modularity is built on nested class declarations with no global namespace, so top-level classes act as sandboxes and any connection to the external world must come via parameters supplied at instance creation (Bracha “Modules as Objects in Newspeak”; Bracha “The Newspeak Programming Platform”; Newspeak language site; Newspeak specification draft; Newspeak Wikipedia). Modules are stateless first-class objects; side-by-side deployment of multiple module instances is trivial; multiple implementations of a module interface can coexist.

E (Mark S. Miller et al., 1997, Electric Communities) is the seminal object-capability language for secure distributed computing; object capabilities are unforgeable references plus permission-to-send-messages (E Wikipedia; ERights “Walnut” wiki; The E Language in a Walnut; E Language Specification). Miller’s “Robust Composition” thesis is the reference for the principle-of-least-authority (POLA) formalization and the “authority is the ability to cause effects” framing (Miller “Robust Composition” thesis PDF). “Capability Myths Demolished” (Miller, Yee, Shapiro) shows that pure capability systems support least-privilege far better than ACL systems (Miller et al. “Capability Myths Demolished”).

Kadena’s Pact smart-contract language ships an object-capability discipline as a first-class module governance primitive: capabilities, guards, and module governance are language keywords, and formal verification checks capability boundaries at compile time (Kadena “Announcing Pact 3.0”; Kadena “Pact 3.1”; kadena-io/pact GitHub; pact-5 GitHub; Pact language reference). Pact is Turing-incomplete by design; its capability model directly informs a general-purpose language’s design in the capability-manifest tradition.

Lingering Authority: exact numbers

Igor Santos-Grueiro’s PORTICO (arXiv 2606.22504) is the current reference for revocable resource-and-effect capabilities in coding-agent settings. The experimental design compares PORTICO to a non-revoking comparator sharing the same initial envelope and receiving the same grants at the same turns. The results are stark:

  • Closure slice: PORTICO rejects 10/10 post-closure reuses; the comparator permits 10/10.
  • Deterministic stale-write audit: PORTICO executes 0/6 forbidden effects; the comparator executes 6/6.
  • Six live model traces cover file writes, git mutation, and network egress.
  • Four-episode same-policy diagnostic: broad request exposure raised blocked proposals from 67 to 84 without inducing forbidden-effect executions.

The design principle — that permissions granted for a subgoal must be revoked when the subgoal’s boundary closes — is broadly aligned with the “principle of least authority” tradition but operationalises revocation at the granularity of a coding-agent’s task decomposition rather than at process-lifetime granularity (Santos-Grueiro “Lingering Authority” arXiv 2606.22504; BEX Korean blog on PORTICO; VILA Lab Dive into Claude Code). ResidualAuth (arXiv 2609.08062) extends the analysis to what authorization state must be preserved under revocable delegation (ResidualAuth arXiv 2609.08062). SkillEffect (arXiv 2608.17007) treats agent skills as memory-bounded artefacts with checked lowering, citing PORTICO as a canonical failure mode reference (SkillEffect arXiv 2608.17007).

Information-flow at the type level, effect-typed IR, capability manifests

The frontier on making information-flow, permissions, and side effects visible-and-reasonable-about is emerging in three coordinated lines:

  1. Language-level capability discipline (Austral/Wyvern/Pact tradition): capabilities are types, not runtime tokens; the compiler enforces the closure at module boundaries. Overhead is annotation-time only.
  2. Runtime capability harnesses (PORTICO tradition): the language remains permissive but a wrapper enforces revocable delegation. Overhead is runtime and requires trust in the harness.
  3. Effect-typed IR (Koka/OCaml/Unison tradition): effects are first-class rows in the type system, and the compiler is aware of them at every optimisation phase. Overhead is compiler-complexity.

The convergence point that a new language could occupy is the intersection: language-level capability types + effect-typed IR + harness-optional runtime enforcement. No published language occupies this whole intersection today; Austral is closest but lacks the effect-row discipline of Koka/Unison. This is a genuine design gap.

Part 9: Deterministic execution, replay, and simulation as agent primitives

Deterministic simulation testing in production

FoundationDB pioneered production-grade deterministic simulation testing (“DST”) around 2010, alongside Amazon Web Services’ internal lock service (Antithesis “Deterministic simulation testing”; FDB paper; FDB Simulation and Testing docs; FDB client testing docs; DeepWiki FDB testing; Pierre Zemb “Diving into FoundationDB’s Simulation Framework”). Concrete scale: tens of thousands of simulations every night; ~1 trillion CPU-hours accumulated; deterministic simulation of an entire FDB cluster in a single-threaded process; ~10:1 real-to-simulated-time ratio; the “swizzle-clogging” pattern where network connections are stopped one-by-one over a few seconds; the Circus performance-test suite runs nightly at 2 billion key-value pairs (16-byte keys, 8–100-byte values). Flow, FDB’s actor-based language extension, provides the primitive substrate. Will Wilson’s 2014 Strange Loop talk is the earliest recorded discussion of practical DST implementation (Antithesis DST doc).

TigerBeetle’s protocol-aware DST is the current reference for verifying safety and liveness invariants at both the database and per-replica level (TigerBeetle “Protocol-Aware DST”; TigerBeetle homepage; TigerBeetle “A Descent Into the Vörtex”; matklad HYTRADBOI 2025 talk; VOPR internals doc; August in TigerLand newsletter). Quantitative notes: months-in-production scenarios take minutes in simulation; hand-crafted protocol scenarios can be expressed in 30–40 LOC and answered within milliseconds; protocol-aware DST distinguishes logical from physical determinism and enables checking VSR-level invariants (view changes, quorum, ChecksumStream). Named substrate: VOPR (TigerBeetle’s simulator), Viewstamped Replication, write-ahead log, LSM trees, Manifest, VSR ChecksumStream, strict serializability, Strong Serializability. Contrasted approaches: Jepsen (external, black-box, injects faults through user-visible APIs); Antithesis (deterministic hypervisor); FoundationDB (logical determinism + simulation); Dropbox Sync Engine (determinism-by-design).

Antithesis productizes deterministic simulation testing at the hypervisor level, running “regular non-deterministic software” under a deterministic hypervisor (Antithesis homepage; Antithesis DST doc; WhyNowTech “DST-Antithesis”; Antithesis reactive notebook post; Developer Voices podcast episode). etcd integrated Antithesis into its robustness testing workflow (etcd blog “Autonomous Testing”).

Jepsen (Kyle Kingsbury) is the reference black-box linearizability tester, generating concurrent histories, injecting network partitions, checking linearizability post-hoc (TinyCTO “Distributed Consensus Jepsen”; CodexSims “How Jepsen tests distributed systems”; System Design Space Jepsen chapter; CNCF “Analysis of Xline Jepsen”; Jepsen linearizable-register docs; Appian Jepsen testing account). It is complementary to DST: black-box vs. white-box, external vs. internal.

Evidence for material effect

The public quantitative evidence for DST’s effect on bug-finding is largely internal to the operators. FoundationDB’s operator claim is that DST enabled it to reach production-critical reliability before selling to Apple; TigerBeetle’s public claim is that protocol-aware DST finds bugs that neither Jepsen-style nor Antithesis-style black-box testing can find, because they can only exercise user-visible invariants. Antithesis’s marketing claim is “years of production hardening in minutes”. No third-party controlled study measures the marginal effect of DST vs. traditional test suites; the evidence is essentially “systems built with DST have publicly-verifiable strong records”, not “DST measurably reduces bugs by N%”. This is a genuine evidence gap.

For agent-authored software specifically, there is no published DST-vs-non-DST head-to-head. The theoretical case is strong: if a crash produces a seed, message log, and state snapshot, the agent has a maximally reproducible artefact to reason from; if not, the agent has to reproduce the failure to understand it.

Replay-based debugging for agents

No published measurement isolates the marginal effect of replay logs on agent bug-fix success. Adjacent evidence: iterative repair with structured feedback compounds (FeedbackEval); test feedback beats compiler feedback beats unstructured feedback; agents that lack a repro converge slower than agents that have one (Agent4Decompile). The design implication is that a deterministic runtime with cheap replay is a first-order agent primitive, but a randomised-controlled measurement remains an open research question.

Language-level requirements for “runtime is a deterministic simulator by default”

Concretely: (a) no wall-clock reads except through an injectable clock; (b) no non-injected randomness (all RNGs threaded through the environment); (c) no ambient thread scheduling — either single-threaded event loop or explicit deterministic scheduler; (d) I/O mediated through the environment so it can be intercepted/replayed; (e) memory allocation deterministic given the same allocation sequence; (f) no reflection into system state that would leak non-determinism (getpid, hostname, environment variables). Flow (FDB), TigerBeetle’s Zig runtime, and Newspeak’s platform architecture are the three most complete existing implementations. Roc’s platform separation and Bosque’s “removal of problematic features” are closer to the design principle than to the implementation, but they are compatible with it.

Additional evidence: RL from compiler feedback

RLCF (Reinforcement Learning from Compiler Feedback), an early instantiation from Shojaee et al., uses compiler-generated feedback as the reward signal for RL fine-tuning of a pre-trained LLM viewing code generation as a token-by-token RL problem (Shojaee et al. “Tuning Models of Code with Compiler-Generated Feedback”; RLCF Model on GitHub). CoTran uses reinforcement learning with feedback from a compiler and symbolic execution for LLM-based code translation (Jana et al. “CoTran”). Meta’s “Compiler generated feedback for Large Language Models” (Grubisic, Cummins, Seeker, Leather) is one of the earlier compiler-lab-native RL efforts (FuguMT paper listing).

StepCoder (ACL 2024) is one of the strongest published RL-from-compiler-feedback baselines. It introduces CCCS (Curriculum of Code Completion Subtasks) and FGO (Fine-Grained Optimization) that mask unexecuted code segments; the APPS+ dataset is manually verified for unit-test correctness. StepCoder outperforms prior state-of-the-art on the benchmarks it evaluates (Dou et al. “StepCoder”).

Zhang’s “Reinforcement Learning from Compiler and Language Server Feedback” (RLCSF) (arXiv 2510.22907) formalises the compiler + language-server substrate: LSP-spec-3.17-compliant diagnostics, symbols, references, refactoring preconditions, plus GumTree, RefactoringMiner, Winnowing, suffix arrays, and JCS-canonicalized JSONL over stdio via Lanser-CLI. The reward-shaping template uses potential-based reward shaping (Ng, Harada, Russell) and step-level guidance in the ReAct / PAL / Toolformer / Self-Refine / Reflexion tradition (Zhang RLCSF arXiv 2510.22907v2). The specific empirical claim is a methodological one — that LSP diagnostics are the right substrate — rather than a headline benchmark score.

DeepSeek-R1 and GRPO

DeepSeek-R1’s Nature 2025 paper is the reference for pure RL with verifiable rewards on code and math reasoning (DeepSeek-AI “DeepSeek-R1” Nature; DeepSeek-R1 arXiv 2501.12948). R1-Zero uses GRPO (Group Relative Policy Optimization) with rule-based verifiable rewards (correct final answers, passing tests, format compliance) and no supervised fine-tuning; contest-level math reasoning emerges from base-model RL alone (Future AGI “How DeepSeek-R1 Was Trained”; Startuphub “Liang Wenfeng DeepSeek-R1”). The reported training compute for R1 is approximately $294,000. RLVR (Reinforcement Learning with Verifiable Rewards) is now widely regarded as the single most consequential post-training paradigm for coding and reasoning models (Prakash Kagitha “RL with Verifiable Rewards”; OpenDiLab “Awesome RLVR”; rl-llm-wiki “RLVR overview”). OpenAI’s o1 and o3 use the same paradigm: RL against automatically checkable rewards, with reasoning-token budget as a first-class product knob (LessWrong “o1: A Technical Primer”; EngineersOfAI “o1 and o3 architecture”; reinforcement-learning.com “RL for Reasoning”).

Transfer across languages

The published evidence on whether RL on Language A improves Language B is thin. The MSR “Adapting Language Models for Low-Resource Programming Languages” work implicitly tests this: further pre-training and instruction transfer on Gleam or MoonBit improves performance on the target language but does not carry a full ablation of cross-language transfer (MSR “Adapting Language Models for Low-Resource Programming Languages”; Giagnorio et al. alphaXiv abstract). SAFE (Self-evolving Automated proof gEneration) shows that self-evolution on Verus proofs improves the model’s proof-generation ability without needing a large curated corpus (SAFE arXiv 2410.15756); this is single-language self-play, not cross-language.

The likely direction is that reasoning capabilities generalise (as DeepSeek-R1 shows for math → code) but language-specific syntax and idiom do not. A new language whose verifier and diagnostics look like Rust’s + Dafny’s will benefit from cross-language reasoning transfer, but the language-specific corpus is still load-bearing.

Test-time compute and self-refine

The test-time-compute literature has cohered around a few axes. CYCLE demonstrates that code LMs cannot efficiently self-refine faulty generations without training that specifically teaches them to do so (Ding et al. “CYCLE” arXiv 2403.18746). Self-Redraft (arXiv 2511.02854) shows that Self-Refine variants often converge prematurely; encouraging models to propose new drafts rather than incremental edits improves exploration (Self-Redraft arXiv 2511.02854). CoRefine is a confidence-guided variant that reaches competitive accuracy at a fraction of the parallel-sampling cost (CoRefine arXiv 2602.08948). “S*: Scaling Test-time Compute for Code Generation” introduces a hybrid framework combining parallel and sequential scaling (S* OpenReview). A theoretical grounding via Minimum Bayes Risk unifies generation and reranking (Formalizing TTC for Function-Level Code Generation).

The frontier picture: test-time compute reliably improves code correctness when combined with (a) verifier feedback for reward computation and (b) sufficient exploration in the sampling policy. Pure “sample more, pick best” saturates quickly on realistic benchmarks; agentic loops with verifier feedback do not saturate at the same rate.

TigerBeetle protocol-aware DST and Antithesis: additional evidence

TigerBeetle’s August 2026 post on Protocol-Aware Deterministic Simulation Testing is the current reference for pushing DST from “database-level correctness” to “per-replica protocol invariants”. Their VOPR simulator can express hand-crafted protocol scenarios in 30–40 lines of code with millisecond feedback, and simulate scenarios that would take months in production in minutes of wall clock. The fault model is aggressive: partial-block loss (each replica missing 6/9 blocks, cluster-globally durable), Byzantine failure timing, view-change races under partition. Named invariants include strict serializability, strong serializability, logical determinism, and physical determinism (TigerBeetle “Protocol-Aware DST”; TigerBeetle “A Descent Into the Vörtex”; TigerBeetle newsletter).

FoundationDB’s public documentation on Simulation records ~1 trillion CPU-hours of accumulated simulation, tens of thousands of simulations nightly, and a ~10:1 real-to-simulated time ratio. Cycle tests use “swizzle-clogging” patterns (network connections stopped one-by-one over seconds), drive-performance modelling, packet-delivery simulation, connection failure injection, and machine reboot storms. The nightly Circus benchmark uses 2 billion key-value pairs with 16-byte keys and 8–100 byte values (FoundationDB Testing docs; FoundationDB Client Testing; Pierre Zemb “Diving into FoundationDB’s Simulation”; FoundationDB paper).

Antithesis’s autonomous testing platform (Will Wilson, Al Vermeulen; roots at FoundationDB and AWS) provides deterministic-hypervisor-based DST for arbitrary software; etcd’s public write-up of integrating Antithesis into its robustness testing is a real-world reference (Antithesis “How it works”; etcd blog “Autonomous Testing”; Antithesis “Reactive Notebook”). Sui (Mysten Labs) also builds on the DST discipline. Jepsen (Kyle Kingsbury) remains the external adversarial-orchestrator reference, complementary rather than competitive: Jepsen tests from outside the process using injected network chaos and linearizability checkers (codexsims explainer “How Jepsen tests distributed systems”; Jepsen linearizable-register docs; CNCF Xline analysis).

Replay-based debugging for agents: what is measured

Replay-based debugging is an old technique; its measurement in the agent-authored-software regime is essentially unpublished. The theoretical case is that a crash + seed + input-log + snapshot triple gives a coding agent a fully reproducible failure, letting it iterate against the same failure until fixed rather than chasing flaky bugs. TigerBeetle’s newsletter documents that its VOPR reproducibility is the reason human debugging is even tractable at their fault-injection rate; the equivalent claim for LLM agents is intuitive but unmeasured (TigerBeetle newsletter “August in TigerLand”).

The implication for language design: a runtime that is deterministic by default (single-threaded event loop, injectable I/O, seedable RNG, virtualised clock) creates the replay artefacts as a byproduct of normal execution rather than requiring the developer to instrument them. This is the design ambition of TigerBeetle’s Zig runtime and of the FoundationDB Flow actor system, and the direction Antithesis operationalises at the hypervisor level. Whether a general-purpose language should default to this discipline is contested: it constrains the concurrency model, forbids shared-memory data races by default, and requires every effect to be routed through a declared interface. But it is the only known path to sub-hour-mean-time-to-repro for concurrent systems, which is the tightest agent-loop constraint.

Part 10: Emerging techniques and forward bets

Long-context models

The empirical picture: models advertise multi-million-token contexts, but effective reasoning degrades sharply beyond the low-hundreds-of-thousands range even on well-designed benchmarks (RepoReasoner FSE 2026 page; arXiv “RepoReasoner”; OpenReview “Long Code Arena”; Awesome-LLM-Long-Context-Modeling; ar5iv “Sense and Sensitivity”; nrehiew “Evaluating Long Context”). CAST-style structural chunking via ASTs is the mitigating technique (Wang et al. “CAST”); AST(NIT) serialisation similarly (Harvey AST(NIT)). A language should not assume the agent can hold the entire program in context; declaration-level content addressing (Unison) gives cache locality even at very long contexts.

Reasoning models and annotation-verbosity tolerance

o1/o3/Claude-Sonnet-reasoning/DeepSeek-R1 all tolerate long reasoning chains inside the context and pay for it in latency and tokens. This raises the ceiling for how much information a language can afford to embed in each declaration. Contract-heavy code (Dafny) previously punished throughput; reasoning models make it viable. The verbose type-annotation cost that Rust levies has become a smaller relative cost as the model spends more tokens thinking. The Bursuc et al. finding that natural-language descriptions don’t help vericoding suggests contracts are the load-bearing signal for reasoning-model-authored code (arXiv “A benchmark for vericoding”).

Multi-agent systems

Distributed code review, refactoring, and verification across specialised agents is the current architecture pattern (planner, coder, reviewer, verifier). AutoVerus is a network of LLM agents for the three phases of proof generation (arXiv “AutoVerus”); Devin ships planner + coder + reviewer as separate roles (Cognition SWE-bench technical report); SWE-Dev synthesizes tests as a separate role (HuggingFace “SWE-Dev”). The language-design implication: agents will re-read others’ code, so declaration-site clarity and semantic hashes to prevent stale-context edits matter more than in the single-agent model.

Structured outputs and constrained decoding

Structured outputs have standardised around JSON Schema; the frontier is grammar-constrained decoding of full programming-language syntax (Part 4). A language should assume the toolchain surface is grammar-constrained by 2027; declaration syntax should be uniform enough that a single grammar generates all valid programs, and error-recovery grammars should be part of the surface spec.

Verifier-in-the-loop training

RLVR + compiler feedback (Part 7) is now the standard post-training regime for coding models. A language whose compiler exposes verifier-quality reward signals (Dafny, Verus, Lean, F*, SPARK) already benefits; a language whose compiler exposes only crash-vs-not signals is starving. RLCSF’s LSP-mediated approach extends this to any LSP-speaking language (RLCSF arXiv 2510.22907).

Code-generation-specialised models diverging

Codex-line, Claude Code, DeepSeek-Coder-V2, StarCoder2, Qwen-Coder are all measurably differentiated from their general-purpose siblings on coding benchmarks (HuggingFace “StarCoder 2 paper”; DeepSeek-Coder-V2; Ossai Hub DeepSeek-Coder-V2; GeneralCompute “Coding Model Benchmarks Qwen 2.5 Coder vs DeepSeek Coder V2 vs CodeLlama vs StarCoder 2”; Till Freitag “Open-Source LLMs Compared 2026”; IRIS supported-models). DeepSeek-Coder-V2 supports 338+ programming languages with 236B MoE and 16B active parameters. A language project can now afford to design assuming a coding-specialised model is the reader, not a general chat model.

Forward bets: prescient in 3 years or foolish

Load-bearing (assume as background): Grammar-constrained decoding will be standard for structured surfaces; RLVR + compiler feedback will be the default post-training regime for coding models; MCP will be the standard tool-exposure protocol; LSP-mediated diagnostics will be the standard supervision channel.

Speculative: Verifier-in-the-loop training will subsume Tier 3 verification into pre-training for common languages; content-addressed code will win over path-addressed code; capability-first module systems will replace ACL-first module systems for anything internet-facing. These are plausible but not settled.

Specific bets that would look prescient in 3 years: (a) Language contracts as first-class exportable objects (so tests, docs, and prover input all derive from one source); (b) all effects declared at boundaries + revocable capability tokens as runtime primitives (per Santos-Grueiro’s Lingering Authority result); (c) deterministic-by-default runtime with replay logs as native artefacts; (d) content-addressed declaration identity with automatic rename refactoring; (e) compiler emits an LLM-first machine-readable format alongside human-readable, with MachineApplicable-style tags standardised.

Bets that would look foolish if wrong: Assuming free-form generation (grammar-constrained decoding wins); assuming ambient authority is fine (prompt-injection has already broken this); assuming a single canonical implementation of an interface (Newspeak’s side-by-side deployment is more likely correct); assuming the language does not need LLM-oriented pedagogy in errors (RustAssistant’s ~74% depended on it).

Part 11: The counter-position

The strongest case that a new language is the wrong answer

The counter-position is: pick Python + Quasar-style subsetting + SynCode + Token Sugar + type-constrained decoding + RLCSF + Antithesis for DST + MCP for tools. This wins for four reasons.

First, training-data gravity dominates and is not closable by design alone (Part 1). Giagnorio et al.’s no-resource benchmark reports 79% high-resource vs. 9% no-resource pass@1 (alphaXiv abstract of “No Resource, No Benchmarks, No Problem?”); Bridge-Coder’s improvements do not reach parity; every published intervention leaves a substantial residual gap. A new language starts at 0 tokens and must climb every intervention curve simultaneously to reach parity with a mid-resource language.

Second, Quasar demonstrates that a subset with new discipline achieves 6.9× / 7.6× fewer erroneous programs than free-form Python without losing the model’s Python fluency (Mell et al. “Quasar”). This is the cleanest existing evidence that “restricted dialect + toolchain” beats “clean-slate language” on the specific axis a language project would optimise.

Third, CodeAct’s evidence that Python-as-action-language across 17 LLMs is up to 20% better than schema-based alternatives (arXiv “CodeAct”) means the payoff for using Python is not just corpus-size — it is also the fact that CodeAct-style agent architectures already use Python as their runtime substrate.

Fourth, SynCode (Ugare et al. “SynCode”) reduces syntax errors by 96%, Token Sugar (arXiv “Token Sugar”) reduces up to 15.1% token cost, type-constrained decoding (arXiv 2504.09246) reduces >50% of compilation errors — all of which apply to Python (or a Python subset like Quasar) today, without waiting for a new language to accumulate a training corpus.

Vericoding’s numbers on Dafny (82% off-the-shelf) versus Lean (27%) and Verus (44%) additionally suggest that if the goal is verification-aware code generation, one should use Dafny, not invent a new language.

What a new-language advocate must show

To overcome this position empirically, a new language must produce evidence of at least one of the following:

  1. A design feature that changes the intervention ceiling. Not “our language has capabilities” — but “on a matched-corpus-size head-to-head, our language reaches Y% while Python subset reaches X%, with X < Y, on a task class Z that is realistic”. Quasar itself is the model for what such evidence looks like.
  2. A verifier that outperforms Dafny on a matched benchmark. The vericoding paper’s 82% Dafny baseline is the number to beat. A new language that reached 90%+ off-the-shelf on the equivalent 3,029-spec benchmark would justify its existence on verification grounds alone.
  3. A capability/effect discipline that closes the Lingering Authority gap at zero deployment cost. PORTICO (Santos-Grueiro) is the reference; a language-level replacement that requires no runtime harness would be a genuinely new position.
  4. A determinism model that reduces bug-repro cost by an order of magnitude. Antithesis at the hypervisor level and TigerBeetle at the application level are the references; a language whose runtime is deterministic without requiring the whole application to be structured around a simulator would materially advance the field.

Absent one of these, the counter-position is empirically strong.

Long context and repository-level reasoning

Long context models have moved from 8k to 200k to 1M+ token windows in the frontier tier, but the empirical picture on their effect on code reasoning is not “long context solves everything”. “Long Code Arena” (a set of six benchmarks for long-context code models) and RepoReasoner both find that repository-level reasoning degrades sharply as file count and cross-file dependency complexity grow, even when the entire repository fits in the model’s stated context (OpenReview “Long Code Arena”; Chen et al. “RepoReasoner”; FSE 2026 RepoReasoner page). “Sense and Sensitivity” (arXiv 2505.13353) investigates semantic recall and finds performance degrades non-uniformly with context length (Sense and Sensitivity; Xnhyacinth Awesome-LLM-Long-Context-Modeling).

The implication for language design: stable, hash-identified declarations (Unison’s discipline) give the model a durable reference substrate that survives file reshuffling and refactoring; content-addressed IR gives the RL trainer a reproducible signal even when the file layout shifts. These are load-bearing choices that long context does not obviate.

Reasoning models and tolerance for verbose annotations

Reasoning models (o1, o3, DeepSeek-R1) trade inference time for correctness by generating long chains of thought before answering, guided by verifier rewards. The tolerance for verbose annotations (types, contracts, effect rows) increases in this regime because the model can spend inference tokens reasoning about the annotations rather than treating them as costly noise. This is the operational rationale for Dafny’s off-the-shelf 82% vericoding success — the reasoning-tokens budget affords the model space to think about the contract as a specification, not skim past it as syntax (Bursuc et al.). A new language designed for reasoning-model consumers should treat annotation density as an asset rather than a cost.

Multi-agent systems

The empirical picture on multi-agent code review and refactoring is early. RAMP’s 67% pass@1 on Ruby with 5-iteration convergence is a multi-agent (though modest-scale) result (RAMP arXiv 2511.03925); Agent4Decompile’s 84–97% re-executability uses a constraint-guided multi-agent framework (Agent4Decompile); AutoVerus achieves >90% on 150 Verus tasks via a network of LLM agents mimicking three phases of human proof construction (Yang et al. “AutoVerus”). The current frontier bet is that multi-agent workflows compose reliably when each agent has a narrow, well-verified interface and produces machine-checkable artefacts (a repair, a proof, a test).

Verifier-in-the-loop training as standard practice

The paradigm consolidation is complete: DeepSeek-R1 in Nature 2025 (Nature paper), OpenAI’s o1/o3 rely on it, and the Awesome-RLVR list documents dozens of downstream applications. For code specifically, the verifier is typically a compiler + test runner + optionally a symbolic execution engine. A new language whose compiler cannot be driven headlessly with structured JSON in and out is not merely inconvenient to use as a training substrate — it is unusable. This is a first-order constraint on toolchain design.

Code-specialised models: divergence from generalists

DeepSeek-Coder-V2, Qwen-Coder, StarCoder-2, CodeLlama, and OpenAI’s Codex-line all now diverge from generalist models in training-data curation, tokenizer design, and post-training reward shaping (DeepSeek-Coder-V2 arXiv 2406.11931; StarCoder 2; GeneralCompute “Coding Model Benchmarks”; Freitag “Open-Source LLMs Compared 2026”; OSSHub DeepSeek-Coder-V2). GPT-5.3-Codex is OpenAI’s most aggressive specialisation to date (Ubergizmo “GPT-5.3-Codex”); Cognition’s SWE-1.7 is a similar specialisation on the agentic-coding side (Mer.vin “Cognition SWE-1.7”). DeepSeek-Coder was trained on 2 trillion tokens including six specific programming languages; Code Llama’s language list included Python, Java, Julia, and Lua at least, with larger 70B variants existing but expensive at inference (Giagnorio et al. “Enhancing Code Generation”).

The divergence is a load-bearing bet on the “language substrate for coding models is a design axis” thesis. A new language whose model landscape supports it at the specialisation level (fine-tuned adapters, retrieval-augmented generation on the language’s docs, targeted RLVR) will benefit disproportionately.

Forward bets, ranked

  • Load-bearing (should design around): verifier-in-the-loop training, structured compiler diagnostics as first-class, grammar-constrained decoding, capability-scoped agent execution, deterministic runtime as a substrate for replay-based RL and debugging, code-as-action for the LM’s tool loop.
  • Prescient bets (would look right in 3 years if they pan out): effect-typed generation with decoder-level enforcement; content-addressed hash-identified declarations as the durable reference substrate; per-error pedagogical text as a first-class field.
  • Speculative: multi-agent RL across languages transferring reliably; long-context obviating structural module design; test-time compute scaling to arbitrary depth without new architectures.

The strongest counter-case, in the Python-plus-tooling form

The Quasar counter-position generalises. If the goal is “measurably better agent outcomes at reproducible benchmark scale”, the shortest path is:

  1. Pick Python 3.13 as the base (largest corpus, richest model support, best RL infrastructure).
  2. Restrict to a subset with static-typing-only signatures (Pyright + mypy strict mode).
  3. Add a capability manifest and a runtime harness that enforces it (à la PORTICO + AgentDojo defenses).
  4. Add contract syntax as a decorator library that lowers to Dafny or Verus for tier-3 checking (à la crosshair or icontract).
  5. Ship structured-JSON diagnostics from the whole stack (Pyright + your runtime).
  6. Wrap the whole thing in an agentic control plane (RelAIBuild-style) with SEARCH/REPLACE editing (Aider-style) and grammar-constrained decoding (SynCode + XGrammar for the DSL portions).
  7. Fine-tune a code-specialised model on your subset via LoRA adapters, then RLVR-fine-tune against the compiler + test loop.

This stack has zero language-design innovation but takes advantage of every empirically demonstrated intervention (~74% Rust repair, 96% syntax elimination, 82% Dafny vericoding, 6.9× Quasar error reduction, ~90% AutoVerus proof generation, etc.). The remaining objection — that Python’s semantics are not amenable to full capability-safe module design or full determinism-by-default — is the concrete gap a new language must be prepared to justify empirically.

The strongest empirical counter to this Python-centric position comes from three specific findings:

  1. Vericoding success rates differ by nearly 3× across languages sharing the same off-the-shelf models (Dafny 82%, Verus 44%, Lean 27%) — evidence that the language’s design (declaration-site contracts vs. tactic scripts, SMT-friendly semantics vs. dependent-type semantics) meaningfully moves outcomes at fixed model (Bursuc et al.).
  2. Quasar’s 6.9×/7.6× reduction in erroneous programs at fixed model is not achievable by adding a linter to Python (Mell et al.).
  3. PORTICO’s 6/6 → 0/6 forbidden-effect elimination requires the capability abstraction to be authoritative — a Python decorator cannot enforce it because Python’s runtime provides no compile-time closure (Santos-Grueiro).

These three lines of evidence suggest the counter-position is empirically strong but not fatal to a new-language project — provided the new language stakes its claim on one of these axes and beats the Python-plus-tooling stack on that axis with matched-model measurements.

Part 12: Synthesis — “if I ran this project”

Ten key findings the design should be built around

  1. Training-data gravity dominates in the medium term; corpus-mass interventions have the largest per-dollar effect. No-resource languages score ~9% pass@1 versus ~79% for high-resource; the biggest closure lever is further pre-training on the target-language corpus, and even large synthetic corpora do not close the gap to Python (alphaXiv abstract of “No Resource, No Benchmarks, No Problem?”; arXiv “Enhancing Code Generation”; ATLAS). This means a new language must ship a synthetic-corpus + adapter-training pipeline as part of the standard toolchain, or accept that it is a demonstration project for years.
  2. Static-vs-dynamic typing effects on LLM correctness are small; type-aware decoding effects are large. Corpus popularity predicts model quality more than typing discipline (Dan Luu); but type-constrained decoding reduces >50% of compilation errors and eliminates the 33.6% typing-error class in principle (arXiv 2504.09246). The language should expose types to the decoder, not use types to make humans happy.
  3. Structured, machine-readable diagnostics with MachineApplicable tags are the single most measured lever on repair-loop success. RustAssistant ~74%, 2.4× Clippy autofix (Deligiannis et al. “RustAssistant”); FeedbackEval confirms structured beats unstructured (FeedbackEval); Rust’s JSON diagnostic model is the reference implementation (Rustc Book “JSON Output”).
  4. Grammar-constrained decoding is a solved problem for syntax; a language should assume it. SynCode 96.07% syntax-error reduction at 1.22–1.76× overhead (Ugare et al.). Design accordingly: unambiguous grammar, uniform reserved-word discipline, canonical whitespace form.
  5. Formal contracts at declaration sites are the highest-signal correctness constraint, and prose descriptions add nothing. Vericoding’s negative result on natural-language descriptions (arXiv “A benchmark for vericoding”); Dafny’s 82% off-the-shelf vs. Verus 44% vs. Lean 27% shows declaration-site contract syntax matters. Marmaragan’s 50.7% on SPARK anchors the SPARK subset (Cramer and McIntyre).
  6. Compiler-signal-based RL (RLVR / RLCSF) is now the default post-training regime for coding models. DeepSeek-R1 (Nature), o1/o3 (LessWrong “o1 primer”), RLCSF (Zhang). A language that lacks a first-class LSP and structured compiler diagnostics is starving the training loop.
  7. CodeAct/Quasar’s evidence is that language design changes agent outcomes when the language is in the action loop. 20% higher success for CodeAct over schema tools across 17 LLMs (arXiv “CodeAct”); 6.9× / 7.6× fewer erroneous programs for Quasar (Mell et al.). “Code as action” is not marketing.
  8. Ambient authority is unsafe under prompt injection. AgentDojo results and the Lingering Authority PORTICO experiments (6/6 → 0/6 forbidden-effect executions with revocation) show that language + runtime must express effect scoping as first-class artefacts (AgentDojo; Santos-Grueiro). Capability-based module systems in the E/Newspeak/Wyvern/Austral/Pact tradition are the correct baseline.
  9. Deterministic simulation testing works but has no head-to-head measurement against traditional testing for agent-authored code. FoundationDB’s ~1 trillion CPU-hours, TigerBeetle’s protocol-aware DST (TigerBeetle), Antithesis’s hypervisor approach (Antithesis) all show it works in practice; the marginal effect on agent bug-fix rate is unpublished but the theoretical case is strong.
  10. Vericoding language choice matters, not just verifier choice. Dafny 82% vs. Verus 44% vs. Lean 27% off-the-shelf (arXiv “A benchmark for vericoding”) — declaration-site contract syntax beats tactic-script proof, at least for this generation of models.

Three-to-five highest-leverage toolchain features

  1. Rust-style structured diagnostics with MachineApplicable suggestions, error-code catalog, and stable JSON schema. RustAssistant’s numbers do not exist without this (Rustc JSON docs; Deligiannis et al.). Add an LLM-first pedagogy field (“why this is wrong; what a correct fix looks like”) per error class.
  2. LSP-first design with a canonical CLI (Lanser-CLI-style) that exposes diagnostics, symbols, references, refactorings, and edits as JSON-RPC + JSONL over stdio. This is the substrate RLCSF trains on (Zhang); without it, RL supervision is unavailable to third parties.
  3. Grammar-constrained decoding integration built into the compiler distribution. Ship the DFA mask store as an artefact, alongside the parser, so any decoder can enforce SynCode-style constraints without recompiling the grammar (Ugare et al.; XGrammar).
  4. Deterministic runtime + replay tooling as native artefacts, not third-party add-ons. Every crash produces a seed + input log + snapshot triple, and the runtime accepts them as inputs (FoundationDB; TigerBeetle VOPR; Antithesis).
  5. Capability manifest as a first-class build artefact. Every module declares its resource-and-effect requirements; the compiler enforces the closure; the runtime revokes on subgoal closure per PORTICO’s discipline (Santos-Grueiro; Kadena Pact 3); build-time script execution is foreclosed by construction (Snyk on node-gyp; Microsoft on 33 malicious npm packages; openreplay simple defense).

Two-to-three language design bets

  1. Contract-first declaration syntax with SMT-friendly semantics. Bet that the Dafny discipline (pre/post conditions and invariants written in a first-order-fragment that Z3 can chew) generalises beyond Dafny’s syntax. Off-the-shelf 82% vericoding success is unmatched by any other tier-3 verifier (Bursuc et al.); DafnyBench’s 67.8% ± 1.7% Claude 3 Opus result is the baseline to design against (Loughridge et al.). Design decision: make contracts a language-level primitive, not a library-level require! macro; make them checked at each edit; expose them to grammar-constrained decoding as first-class hole-filling targets.

  2. Effects and capabilities as pervasive types, not opt-in annotations. Bet that a Koka-row/Unison-abilities-style effect discipline, combined with an Austral/Wyvern-style capability system that gates access at module boundaries, is worth the annotation overhead in an agent-authoring world where the compiler enforces the discipline and the LLM has no ambient authority to violate (Bracha “Modules as Objects in Newspeak”; Borretti “Austral”; Aldrich “Capability Safe Reflection for Wyvern”; Miller “Robust Composition” thesis). This is a strong bet against the Python-subset counter-position and a strong bet on the “language design is a leverage point that Python cannot reach” thesis.

  3. Determinism-by-default in the runtime, with logical-time separation from physical time. Bet that TigerBeetle’s discipline — a single-threaded, deterministic event loop with all I/O routed through injectable interfaces, and simulated clocks separable from wall clocks — is the right runtime shape for an era of agent-authored software. This choice commits the language to the FoundationDB/Antithesis-adjacent testing model as the default, not a specialisation (TigerBeetle “Protocol-Aware DST”; Antithesis DST docs); it also constrains the concurrency model (no shared-memory data races by default) and the effect discipline (all effects registered at the type level so they can be replayed).

Publishable open research questions

  1. Head-to-head measurement of language design on matched task suites. Nobody has published a controlled study that holds base model, tool harness, and task suite constant while varying only the language. Dan Luu’s Zstd and Pandoc evals are the closest, and they find language popularity dominates (Dan Luu). A new language project could reproduce the Zstd/Pandoc evals against Python/Rust/TypeScript baselines and report the delta with statistical power. The result would be publishable at ICSE/FSE/OOPSLA regardless of sign.
  2. Diagnostic pedagogy as a training signal. RLCSF (Zhang) shows the shape of the substrate; nobody has published on whether pedagogical error text (“why this is wrong” plus “what a correct fix looks like”) produces stronger RL gradients than terse error text. A language with a first-class “pedagogy field” per error code could be the test bench.
  3. Vericoding effect of syntax vs. semantics. The Dafny (82%) vs. Verus (44%) vs. Lean (27%) gap on the same 12,504-spec benchmark is enormous (Bursuc et al.). How much is attributable to syntax (declaration-site contracts vs. tactic scripts), how much to the verifier’s automation profile (Boogie+Z3 for Dafny vs. Z3 for Verus vs. interactive-tactic for Lean), and how much to training corpus? A new language could ablate these axes.
  4. Effect-typed generation. Beyond syntax and typing, does exposing effect rows to the decoder as constraints improve correctness on effect-heavy tasks (DB transactions, network I/O)? SynCode and Mündler-style type constraints suggest yes; the empirical claim is untested.
  5. Deterministic-replay-driven RL. If a runtime produces reproducible failure envelopes (seed + input log + snapshot), can an agent-trainer learn faster from replay-based rewards than from stochastic re-execution? FoundationDB’s ~10:1 real-to-simulated time ratio (FoundationDB testing) suggests the compute-cost advantage is enormous; nobody has published on the RL implications.
  6. Capability-safe module systems as prompt-injection defenses. PORTICO (Santos-Grueiro) demonstrated capability revocation at the harness level; nobody has published on whether embedding the same discipline at the language level (compile-time enforcement of capability closure) yields stricter guarantees without runtime overhead. AgentDojo (Debenedetti et al.) is the benchmark to beat.
  7. Token-efficient syntax for LLM-first languages. Token Sugar (799 shorthand pairs, 15.1% source and 11.2% generation reduction with near-identical pass@1 at three-LLM scale) (Sun et al.) is the strongest existing evidence that syntactic token-cost matters marginally; a language designed for token-cost from the outset (canonical keywords chosen for tokeniser efficiency, short-hand notation for common idioms) could measure the effect at design time.

Fatal misreadings of the frontier to avoid

  1. “We can close the data-gravity gap with tooling alone.” The evidence is clear that no-resource languages score ~9% pass@1 against ~79% for high-resource, and even the strongest interventions leave a gap. Interventions help by 25–30 points on hard-task no-resource benchmarks (alphaXiv abstract), which is real but not full closure. A language that assumes tooling closes the gap without corpus investment will underperform.
  2. “Dynamic languages are 2–3× more token-efficient, so we should be dynamic.” Luu shows this collapses on realistic tasks (Dan Luu). Token efficiency comes from grammar design and Token-Sugar-style shorthand, not from erasing type annotations.
  3. “Formal verification is a niche; agent authoring makes it obsolete.” The vericoding results show the opposite: 82% Dafny off-the-shelf, dramatic gains year-on-year (68→96% in Dafny in just over a year per Bursuc et al.), and vericoding is the only regime with cryptographic-strength correctness at scale (Bursuc et al.). The Ioannidis-Swamy ICFP 2026 experience report argues agentic proof-oriented programming has “crossed the threshold of practical usefulness” (ICFP 2026 “Proofs Promptly”). Under-investing in verifier ergonomics is a fatal miss.
  4. “Ambient authority is fine; we’ll sandbox the process.” PORTICO’s deterministic stale-write audit went 6/6 forbidden-effect executions on the non-revoking comparator against 0/6 on PORTICO (Santos-Grueiro). AgentDojo’s prompt-injection results show that at agent scale, process sandboxes are insufficient. Capability discipline must be at the language and module level.
  5. “MCP + JSON tool calls are good enough; we don’t need executable code as the action language.” CodeAct’s 20% success advantage across 17 LLMs shows the opposite at benchmark scale (Wang et al.). Additionally, ArXiv 2602.14878 on MCP tool descriptions shows the descriptions themselves are load-bearing prompt surfaces with all their pathologies (arXiv “MCP Tool Descriptions Are…”). Code-as-action is a load-bearing bet.
  6. “Deterministic simulation is a database thing.” TigerBeetle and Antithesis show the discipline generalises (TigerBeetle; Antithesis); Dan Luu’s 1000:1 test-effort argument suggests the correct testing budget for agent-authored code is more like the CPU-design shop than traditional software (Dan Luu “Agentic test processes”). A language whose runtime foreclosures determinism-by-default is choosing to build a rare and valuable substrate.
  7. “Long context makes program structure unimportant.” The Long Code Arena and RepoReasoner results show that even models with extremely long context windows degrade sharply on repository-level reasoning as file count and cross-file dependency grow (OpenReview “Long Code Arena”; arXiv “RepoReasoner”). Content-addressed hash-identified declarations (Unison) and stable module boundaries do additional work that long context does not replace.
  8. “Grammar-constrained decoding will trivialise language design.” SynCode’s 96.07% syntax-error reduction (Ugare et al.) is enormous — but only for the syntactic layer. Type errors (33.6% of failures per Mündler et al.) are outside the CFG’s expressive power without type-constrained decoding, and semantic errors are outside grammar’s power entirely. Grammar-constrained decoding is a floor, not a ceiling.
  9. “Language design changes don’t matter; it’s all in the model.” Quasar’s 6.9×/7.6× reduction in erroneous programs at fixed model (Mell et al.) is the counter-example. Language design and model design compound.
  10. “The counter-position (Python subset + tooling) has already won.” Quasar’s evidence shows the subset can be more than a Python subset — its Opportunistically Parallel Lambda Calculus semantics are new, its conformal prediction is new, its capability-typed action gating is new. The distinction between “Python subset” and “new language” is blurry. A new language is justified when its semantics cannot be expressed as a subset of an existing language whose model support is stronger. Contract-first, effect-typed, capability-safe, deterministic-runtime design cannot be reduced to a Python subset without losing the load-bearing pieces.

The design ambition articulated in the premise — “measurably better success rates, faster convergence, fewer defects, lower total token cost, more autonomous debugging” — is achievable only if the language commits to being the substrate the compiler-in-the-loop trains on: structured diagnostics as pedagogy, grammar exposed to the decoder, types exposed to the decoder, effects exposed to the decoder, contracts as first-class syntax, capability boundaries at module level, deterministic runtime with replay artefacts. Any single one of these is table stakes; the combination is the frontier.