No. 4 Saturday 17 October Game dev, software and AI

Machines read the code; people pick the risks

The Claude Code team explains how it reviews a flood of agent-written code without slowing down. Also: Postgres 19 slips under AI-found bugs, a test harness that rescues a cheap model, and one Unity game shipping to ten platforms from one branch.

The short version

  • Software and AI The Claude Code team now lets agents do most code reading and spends human review by risk tier: low-risk changes need no person, high-risk ones need two.
  • Software and AI PostgreSQL 19 slips to about November 19 after AI-assisted review found so many defects that 8 to 10 features were pulled.
  • Software and AI A paired experiment found a test harness did nothing for a frontier model at 7 times the cost, but took a cheaper model from 3 of 10 to 10 of 10 accepted runs.
  • Game dev A four-person Unity team ships to about ten platforms from one branch. Its rule: ask what input a device can produce, not what device it is.
  • Game dev GameDiscoverCo finds about 17% of engaged PC players also play on a Steam Deck, and 64.4% play no console at all.
Software and AI Lesson

Anthropic rebuilt code review

The team behind Claude Code, which an AI mostly writes, explains how it checks a flood of changes: machines do most of the reading, and people step in where the risk is.

The path a change takes before users see it, as described by Addy Osmani, Morgan Westlee Lunt and Qing Wang of the Claude Code team.

Claude writes about 90% of Claude Code's own code. Merged pull requests more than doubled from June to September, while the share later reverted fell by over 40%. Across Anthropic's monorepo, daily changes went from about 2,500 in May to over 30,000 in September.

Every pull request now gets an AI review: several agents hunt for bugs in parallel, separate verifier agents vote on each finding, and only agreed findings are posted, sorted as important, nit or pre-existing. A separate security pass and required CI follow. Then a classifier sets the risk tier: low risk needs no human approval, medium needs one person and high needs two. Sensitive paths such as billing, the auto-updater and CI workflows always need a person.

The reviewer is itself tested. Past incidents and reverts form an evaluation set that runs daily, and every change to the reviewer is scored against it. Noise is measured by how authors respond to comments: fixed, dismissed or intentional.

54%of internal pull requests now get substantive review comments, up from 16%

Technical details

Convergence numbers after the team tuned the reviewer: pull requests passing after one review-and-fix round rose from 65% to 78%, those needing five or more rounds fell from 8.9% to 1.5%, and mean reviews to pass dropped from 2.08 to 1.32. Safeguards include a three-round cap, diff-only re-review after early passes, flagging bug classes instead of single lines, and never repeating a finding.

About 59% of what the reviewer kept flagging was old code that predated its first pass. Agent approvals never count as human approvals, and approvals apply to one specific commit.

Two files steer it. CLAUDE.md describes the project's stack, conventions and goals and is loaded on every review, so it should stay short. REVIEW.md is written by code owners and says what matters in their area, what failure looks like, what to skip, and to cite file and line as evidence.

Why it mattersAny team using coding agents hits the same wall: more code than people can read. This is a worked answer with numbers, and most of it (a verifier step, risk tiers, a REVIEW.md per area) can be copied at small scale.

For youYour own repos already run an AI review pass; the parts to borrow are the risk tier (which plugin modules always need your eyes) and a REVIEW.md per area. Also the eval set built from your past reverts.

pull requests
Proposed changes to a codebase, reviewed before they are merged into the main version.
reverted
Undone after merging, usually because the change caused a problem.
monorepo
One shared code repository holding many projects.
CI
Continuous integration: automatic builds and tests that run on every proposed change.
classifier
A model that sorts each item into one of a few categories, here low, medium or high risk.
evaluation set
A fixed collection of test cases used to score a system the same way every time.

Sources Claude Code blog: solving code reviewSimon Willison

More like this Less like this

Review is the new bottleneck

Finding problems got cheap. Fixing them, and deciding which need a person, did not.

Software and AI News

Postgres 19 slips after AI review finds too many bugs

AI-assisted review found defects in PostgreSQL 19 faster than its developers could fix them, so several features were pulled and the release moved back about six weeks.

8-10features reverted after beta; about one is normal

Nov 19rough new release date

Committer Peter Eisentraut says the cause is not AI-written code, which the reverted work predates. Cheap bug finding raised the bar on code already merged, and even three-line fixes eat the schedule.

Technical details

Among the pulled features are the SQL/PGQ graph queries, now aimed at version 20. Still in 19: REPACK CONCURRENTLY and pg_plan_advice. Eisentraut also blames an April feature freeze that overlapped a security-fix crunch, which pulled senior reviewers away until August, and suggests LLM-assisted fuzzing of feature interactions during development.

Why it mattersPointing review agents at an old codebase finds more than a team can fix. Plan the fixing capacity before turning the reviewer on.

For youSupabase will carry Postgres 19 later than planned; no action, but do not plan around SQL graph queries this year.

reverted
Taken back out of the code after being merged.
beta
A test release that is feature complete but not yet stable.

Sources Peter EisentrautThe Register

More like this Less like this

Software and AI Lesson

Make checking cheap enough that the agent does it

Addy Osmani argues the slow step in AI coding is now proving the code works, and lists the checks that let an agent prove it without a person reading every line.

His starting point: a human reading diffs is the slowest feedback loop there is. People should define what success means and review the evidence; the agent should run the checks.

  1. Keep a few end-to-end tests as ground truth

    A small set of tests that drive the real app, owned by people, that agents may not weaken.

  2. Let the agent run and watch the app

    A feature map plus a command or skill that launches the app and reports what it sees.

  3. Write properties, not examples

    Property-based tests state a rule in plain English first, then check it against many generated inputs.

  4. Diff old against new

    Run both versions on the same inputs and keep a list of differences you approved.

  5. Make the loop fast and repeatable

    Seed randomness, freeze clocks, quarantine flaky tests. Aim for under a second; a 15-minute run makes agents batch changes and lose the failing step.

Technical details

Libraries he names: fast-check for TypeScript, proptest for Rust, Hypothesis for Python. Save every minimized failure as a regression test. He reports no benchmark results; the post is a practice guide.

Why it mattersIt is the practical half of today's lead story: review scales only when most checking happens before a human looks.

For youProperty tests fit your SaaS pricing and parsing code first; fast-check drops into the existing TypeScript test runner.

end-to-end tests
Tests that use the whole app the way a user would, from the screen to the database.
Property-based tests
Tests that state a rule (sorting never loses items) and check it against hundreds of random inputs.
flaky tests
Tests that sometimes pass and sometimes fail without any code change.

Sources Addy Osmani: Agentic Verification

More like this Less like this

Software and AI Lesson

A test harness rescued the cheap model and only slowed the expensive one

One engineer gave two AI models the same coding task ten times each, with and without a set of automated checks to run while working. The checks made the cheaper model reliable and barely helped the stronger one.

Naresh Jain ran 10 paired runs per model on one Java service (validate a request, call an order API, publish to Kafka), judged by 178 outside acceptance tests. The harness gave the agent executable contracts, examples and test feedback while it worked.

With harness Without harness
Sol, runs accepted 10 of 10 10 of 10
Sol, average cost $2.68 $0.41
Sol, average time 19 min 53 s 6 min 51 s
Luna, runs accepted 10 of 10 3 of 10
Luna, average cost $0.27 $0.09
Luna, average time 35 min 15 min 56 s

For a frontier model, run the tests once as a final gate. For a cheaper model, keep the harness on the whole time: harnessed Luna was still about a third cheaper than unharnessed Sol.

Technical details

Token use: Sol about 11.3M with harness and 1.08M without; Luna about 22.9M and 5.86M. All runs used low reasoning effort. Unharnessed Luna passed 94.8% of functional checks but failed at protocol edges: status codes at the timeout boundary, an extra Kafka __TypeId__ header, 415 where the brief required 400, and a mismatched environment variable name.

Caveats from the write-up: one greenfield service, one reasoning setting, the cost of writing the harness not counted, and no independent replication.

Why it mattersIt turns "should the agent run tests while it works?" into a cost question with an answer: it depends on the model, and the cheap model plus checks can beat the expensive one alone.

For youWorth testing on your subagent setup: a cheaper model with your test hooks on, against your default model with tests only at the end.

harness
The checks and tools wrapped around an AI agent, such as tests it can run and contracts it must satisfy.
Kafka
A widely used system for passing messages between services.
acceptance tests
Tests written from the requirements that decide whether the work is done.
frontier model
One of the most capable and most expensive AI models available.

Sources martinfowler.com: Are Harnesses Worth the Cost?

More like this Less like this

Ship to every screen

Porting lessons, player data, and the tools under the engine.

Game dev Lesson

One branch, ten platforms: how a four-person team ports a Unity game

A tiny studio with one programmer is shipping its cooking game to PC, consoles and phones from a single copy of the code. Its write-up lists what made that possible and what it would do earlier.

Foxy Dumplings screenshot: two furry paws stir a bowl of flour and egg on a kitchen counter, with a recipe progress bar and three gold stars rating the step perfect.
Image: Soft Crunch Games via Unity

Soft Crunch Games targets Windows, macOS, Linux, Switch, Switch 2, PS4, PS5, Xbox Series, iOS and Android with Foxy Dumplings. One programmer handles all code, ports, builds and certification.

  • Ask what input a device can produce

    Never check which device is plugged in. A Steam Deck touchscreen can report itself as a mouse with one button, which once made a two-button minigame impossible to finish.

  • Push platform code to the edges

    Saves, accounts, achievements, cloud sync and DLC checks each sit behind an interface; a small factory picks the platform version at startup, and store SDKs are stripped from builds that do not need them.

  • One button prompt, everywhere

    A single prompt prefab, used in almost 200 places, swaps its glyphs the moment the player picks up a pad or moves the mouse.

  • Profile on the weakest machine early

    A kitchen render texture went from about 32 MB to about 4 MB, and Japanese font atlases from about 48 MB to 4 to 8 MB per font.

  • Treat certification as design

    Leaving out text entry and in-game purchases removed whole classes of console work. Getting dev kits running earlier is their main regret.

Technical details

About 40 platform and store configurations are compile-checked, and the build menu has more than 70 entries. Compile-time cases use scripting define symbols set by build scripts, such as leaving Steamworks out of non-Steam PC builds; Unity's own platform defines are kept for code that cannot compile elsewhere. Localization keys can have platform variants, so "Achievement" becomes "Trophy" on PlayStation.

The render texture change fills about 8 times fewer pixels per frame on Steam Deck. The original Japanese fallback atlases were 4096 by 4096 and re-uploaded 16 MB per new character. The public demo drew more than 18,000 players and over a thousand survey responses. The game supports 14 languages.

Why it mattersThese are engine-agnostic habits. A small team that adopts them before its second platform avoids rewriting input and save code for every port.

For youThe capability rule maps straight onto Enhanced Input in your plugins: query the active mapping context, never the platform name. Today's Try this starts there.

branch
A separate line of development in version control; one branch means one shared copy of the code.
certification
The checks a console maker runs before a game may be sold on its platform.
SDKs
Software development kits: libraries a store or platform provides for features like achievements.
prefab
A reusable, pre-built object in Unity that can be placed in many scenes and updated in one place.
font atlases
Images that pack all the letters of a font so the GPU can draw text quickly.
render texture
An image the game draws into off screen, then shows or reuses elsewhere.

Sources Unity: How Soft Crunch Games built Foxy Dumplings

More like this Less like this

Game dev Trend

Most engaged PC players skip consoles; one in six has a Steam Deck

A survey of over 16,000 Steam players found most of them play no console at all, and far more use a Steam Deck than its share of Steam playtime suggests.

Venn chart of PC, Switch and Steam Deck players: 67.2% PC only, 10.9% PC and Switch, 7.9% PC and Steam Deck, 5.5% all three.
Image: GameDiscoverCo

17%of PC players also play on a Steam Deck

64.4%of PC players play no console

Earlier GameDiscoverCo estimates put the Deck at only 2 to 4% of Steam playtime, so many owners use it alongside a desktop. Switch was the most common console among these players at 7.5% alone; 22% pair PC with a phone and no console.

Technical details

The sample is engaged Steam users who answered GameDiscoverCo's Steam Fan Snapshot, so it skews toward heavy players. Among players of PC for at least an hour a month: PC only 42.4%, PC and mobile without console 22%, PC and console without mobile 11.5%, all three 15.6%.

Why it mattersSteam Deck support (controller input, readable text at 800p) reaches a bigger slice of a PC audience than playtime figures imply.

For youSupports putting Steam Deck Verified on the action RPG's launch checklist rather than treating it as a later port.

Steam Deck
Valve's handheld gaming PC that runs the Steam library.

Sources GameDiscoverCo

More like this Less like this

Game dev News

A security engineer says Epic's new Linux anti-cheat is easy to fool

ContextYesterday's issue covered Easy Anti-Cheat Linux v2, which checks for Secure Boot and a signed kernel using the PC's TPM security chip.

Epic's planned Linux anti-cheat trusts the PC to report how it started up. A well-known security engineer shows two cheap ways to make a tampered PC report a clean start.

Matthew Garrett describes two routes. On many desktop boards, shorting the TPM's reset pin lets software replay a clean boot record. Or an external TPM on a USB-to-SPI bridge, which he puts at about $20, stands in for the real one.

  • Outside managed company fleets, a server cannot tie a TPM to one specific PC.
  • Banning known-bad TPMs server side could raise the cost of cheating.
  • He grants it may still deter casual cheaters.
Technical details

The design Garrett analyzes combines eBPF, Secure Boot, measured boot and TPM quotes. In the reset-pin route, a modified kernel replays the expected measurement log so the quote looks legitimate. The external route uses a Microchip MCP2210 bridge with /dev/tpmrm0 pointed at it.

Why it mattersIf you plan to gate online play on hardware attestation, treat it as one signal among several, not proof.

For youNo action for a single-player RPG. If you add PvP over EOS later, keep server-side validation of gameplay as the real defense.

TPM
A security chip in most modern PCs that stores keys and records how the machine started up.
Remote attestation
A signed report from a machine's security chip that a distant server checks before trusting it.
Secure Boot
A PC feature that only starts operating system code signed by a trusted key.
kernel
The core of the operating system, which controls hardware and memory.

Sources Matthew Garrett

More like this Less like this

Game dev Release

BRUTAL

A free C# and Vulkan framework for building your own engine, not an engine itself.

Dean Hall's studio RocketWerkz released the code foundation it spent about four years building, so other teams can build engines on it instead of depending on Unity or Unreal.

Its core is an interop layer that gives C# direct access to native APIs, plus libraries for math, memory, strings, job scheduling, logging and localization. It runs on Windows and Linux.

RocketWerkz built Kitten Space Agency on it, and 80 Level reports Ludeon used it as the base of the engine behind RimWorld's move away from Unity. Hall frames it as a hedge against depending on a few engine vendors.

Language
current C#
Graphics
Vulkan
Platforms
Windows, Linux
Cost
free, including commercial use with credit
In use
Kitten Space Agency

Try itgithub.com/ahwootech/BRUTAL

Technical details

80 Level names no specific license; check the repository before shipping on it. The team says it is not meant to replace Unity or Unreal.

Why it mattersTwo shipped or shipping projects now run on it, which makes "own your engine in C#" a real option for teams leaving Unity rather than a rewrite from zero.

For youNot for your Unreal work. Worth a look only as a reference for job scheduling and native interop patterns.

Vulkan
A low-level graphics programming interface used by many modern games.
interop layer
Code that lets one language call functions written in another, here C# calling native C and C++ libraries.
job scheduling
Splitting work into small tasks that run across all CPU cores.

Sources 80 LevelGitHub: BRUTAL

More like this Less like this

Game dev Release

Raylib 6.2 redesigns image drawing and passes a security audit

The small, beginner-friendly C game library had its biggest update in six months, with a cleaner drawing API and fixes from a professional security review.

500+commits since the last release

20new core functions

File functions now return values in one standard way, so code that checks them may need small updates.

Technical details

The CPU image drawing API now mirrors the GPU drawing API. 300+ changes, 619 core functions in total, 10 new examples. The audit was by Radically Open Security.

Why it mattersIf you use raylib for prototypes or tools, read the migration notes before upgrading.

API
The set of functions a library offers to the code that uses it.

Sources GameFromScratch

More like this Less like this

A safe room for each agent

Separate databases, sealed machines, and release notes to act on.

Software and AI Release

Supabase CLI 2.119

One local Supabase stack per git worktree, with or without Docker.

Supabase's command-line tool can now run several separate local copies of a project on one machine, so parallel AI agents each get a database without stepping on each other.

An experimental native runtime starts the stack without Docker, and services other than Postgres start only when first used.

Start, cached
37 s default, 5.2 s Docker, 3.9 s native
Download
1.9 GB, 550 MB, 360 MB
Idle memory
2.6 GiB default, 290 MiB new
Remote branch ready
median 102 s to 21 s

Try itsupabase start --runtime native

Technical details

Enable with SUPABASE_EXPERIMENTAL_STACK=1 supabase init or stack = true under [experimental] in supabase/config.toml. --eager starts everything at once; --output-format json prints URLs and keys. Native runs on Linux amd64 and arm64 (glibc 2.35+) and macOS 14+ on Apple silicon. Remote branch p95 time to ready fell from 7.6 minutes to 41 seconds; branches carry no secrets or production data by default.

Why it mattersRunning two or three agents at once is common now; one shared local database is where they collide. Numbers are Supabase's own.

For youDirectly useful for your SaaS repos: one worktree per Claude Code agent, each with its own stack and migrations.

git worktree
A second working folder linked to the same repository, so two branches can be open at once.
Docker
A tool that runs software in isolated containers.
runtime
The machinery that actually runs the services, here either Docker or native processes.

Sources Supabase blog

More like this Less like this

Software and AI Release

Chainguard puts coding agents in throwaway machines with signed results

A security company now runs AI coding agents inside small sealed virtual machines that never hold real passwords, and signs a proof that the tests passed.

Both products are in beta and support only Claude for now.

No long-lived secrets sit inside the agent's machine; outgoing traffic rules are optional and logged.
Technical details

Workspaces syncs through chainctl develop. Checks can be triggered from the CLI, a pre-commit hook, a GitHub App or an MCP server, and a pass produces an attestation tied to environment, verdict and commit. Chainguard Actions (over 1,000 rebuilt GitHub Actions) is generally available. No published performance comparison; the attestation means trusting Chainguard.

Why it mattersAgent security is moving from "trust the prompt" to "contain the machine". Even without the product, the pattern (sandbox, injected short-lived keys, attested test runs) is worth copying.

For youYour cloud routines already run in containers; the idea to borrow is the credential proxy, so no long-lived tokens sit where the agent can read them.

MicroVM
A very small virtual machine that starts in moments and isolates code like a full computer.
CI
Continuous integration: automatic builds and tests that run on every proposed change.
Credential proxy
A middleman that adds passwords or keys to requests so the agent never sees them.

Sources ChainguardInfoQ

More like this Less like this

Software and AI Release For practitioners

Claude Code 2.1.302: forked subagents by default, saner scheduled runs

The latest Claude Code update turns on a cheaper kind of helper agent everywhere and fixes scheduled cloud runs that could silently repeat their work.

Show the details

Released October 16. The changes that alter day-to-day behavior:

  • Fork subagents on by default

    In claude -p, the Agent SDK, the desktop app and VS Code. A fork inherits the parent's context. Turn off with CLAUDE_CODE_FORK_SUBAGENT=0.

  • Scheduled runs fail honestly

    Failed cloud routine runs no longer rerun from scratch an hour later (which could repeat work such as posting twice) and show as failed, not Completed.

  • Hooks queue past 12

    At most 12 prompt or agent hooks of one event run at once; the rest wait.

  • Safer default mode

    Headless and SDK sessions with no permission mode start in default on third-party providers, with telemetry off, or on a fresh install. Pass --permission-mode auto to keep auto.

  • Pick the auto mode classifier model

    New CLAUDE_CODE_AUTO_MODE_MODEL sets the model for the client-side safety check.

Technical details

Also: desktop and VS Code always run the main conversation's subagents in the background; a subagent's own Agent calls run in the foreground by default. Server-side auto mode now compacts an over-long conversation for the safety check instead of denying every checked tool call. Fixes for a prompt cache miss after ToolSearch reloads, MCP servers on protocol 2026-07-28 that omit some fields, worktree isolation under git 2.56+, and a new Idle compaction toggle in /config. The standalone Claude Design app at claude.ai/design closes December 14, 2026.

Why it mattersThe fork default changes cost and context behavior for anyone running Claude Code headless; check scripts that relied on the old foreground behavior.

For youCheck the Radar routine and your other cloud routines: failed runs will now stay failed instead of silently rerunning, and headless runs may start in default mode.

subagents
Helper agents that the main agent starts to handle part of a task.
headless
Running without an interactive screen, for scripts and automations.
hooks
Scripts Claude Code runs automatically at set moments, such as before a tool call.

Sources Claude Code v2.1.302

More like this Less like this

Learn one thing

What is a test harness, and why do coding agents need one?

A harness is the set of automated checks an agent can run on its own work while it builds: tests, contracts and fake versions of outside services. It turns "looks right" into "passes".

Think of it like thisLike a climbing harness clipped to a rope: it does not climb for you, but every slip is caught at once instead of at the bottom.

An agent writes code, runs the checks, reads what failed and tries again, many times a minute. Without checks it can only guess, and a person has to find the mistakes later by reading.

Today's stories show both sides. The Claude Code team moved most checking before a human looks, and Osmani lists what a fast loop needs. The paired experiment shows the cost: the loop saved a cheaper model but only slowed a stronger one.

A good harness is fast (seconds, not minutes), deterministic (same result every run) and owned by people, so the agent cannot quietly weaken the tests to pass.

The inner loop a harness makes possible; a person reviews the evidence at the end.
contracts
Machine-checkable descriptions of what a service must accept and return.
deterministic
Giving the same result every time it runs with the same input.

Try this today

Game dev 45 minutes For practitioners

Replace one platform check with a capability check

Find the places your game asks which device it is on, and change one to ask what input is actually available, the rule a four-person team uses to ship one game to ten platforms.

You needA game project with keyboard, gamepad or touch input, in any engine.

  1. List the device checks

    Search the project for platform names and device checks (Steam Deck, Switch, mobile, "is gamepad connected"). Write each one down with what it decides.

  2. Name the real question

    For each, write the capability it stands for: can the player press two mouse buttons, is touch the last input, is text entry allowed.

  3. Change one to a capability query

    Rewrite the riskiest one to read the active input scheme or last input device, the way Foxy Dumplings switches schemes on the player's last input.

  4. Feed prompts from the same place

    Point one button prompt at that active scheme so its glyph changes when the player switches device.

  5. Switch devices mid-play

    Play, then move from pad to mouse to touch (or a Steam Deck touchscreen) and watch the prompt and controls follow.

You know it worked whenSwitching input mid-play updates the prompt at once, and the code you changed no longer mentions a platform name.

For youIn your UE5 plugins, look for platform branches in the interaction and combat input code; replace one with a query on the active Enhanced Input mapping context.

capability check
Asking whether the needed input or feature is available, instead of asking which device the game runs on.
input scheme
A named set of controls, such as gamepad or keyboard and mouse, that the game switches between.

Sparks

What is catching on right now, and the idea worth borrowing from each.

The lab

Research from the front line, in plain words: what it solves, how, and what you could do with it.

Game dev PreprintarXiv preprint

Agents that build 3D models by writing code get far better when a loop runs the code, measures the result and lets a vision model judge it.

3DCodeVerse: Exploring 3D Worlds as CodeYipeng Gao, Ziyao Zeng, Jinfa Huang et al.

3DCodeVerse overview: a text prompt and reference photo of a Kyoto temple garden go to a Blender Python script, which renders the garden; around it are example scenes, graphics, 3D objects and articulated objects.
Image: Gao et al., arXiv
The problem
Code written in one shot to build a 3D asset (a Blender script, a three.js scene) often fails to run or looks wrong, and there was no large open set of runnable 3D programs to learn from.
The idea
A dataset of about 2 million runnable 3D programs across Blender Python, CadQuery, OpenSCAD, GLSL, URDF and three.js, plus a multi-agent harness: a planner splits the asset, sub-agents write code, and feedback from running it, spatial measurements and a vision-language judge drives up to 4 refinement rounds.
The result
Fine-tuning a 9B model lifted its run rate on their benchmark from 0% to 84.9%; a 27B model went from 51.9% to 92.9%. With the harness, human raters preferred the output over the same model's native harness 69 to 89% of the time for 3D objects. All comparisons are the authors' own.
Use it
Copy the loop for procedural assets in a web or engine pipeline: generate the script, run it, check bounding boxes and proportions, render a view for a vision model to critique, and stop after a few rounds. The dataset is licensed for non-commercial use only.

For youA cheap test: have Claude Code write a three.js prop generator with a render-and-critique step, before trying the same with editor Python in Unreal.

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadarXiv 2610.22341Project page

Software and AI PreprintarXiv preprint

On real customer delivery work the best agent resolves 14 of 95 tasks; the misses come from unwritten rules, not code.

FDE-Gym: The Last Mile of AI DeliveryDongxu Yang, Hongda Yan, Renxin Wang et al. (DeepLethe, Tsinghua, Stanford and others)

FDE-Gym overview: real deliveries in 18 sectors feed 20 scenarios with 130+ scripted people, then 95 expert tasks built with 10,000+ expert hours, reviewed against 270+ shortcut strategies, forming the benchmark.
Image: Yang et al., arXiv
The problem
Agent benchmarks hand over clear tasks. Real delivery means vague requests, live systems and constraints that only surface when you ask the right person.
The idea
95 tasks from 20 customer scenarios in 18 sectors, adapted by experts from real projects, with over 130 scripted stakeholders to question. Scores run from 0 (system unchanged) to 1 (the reference solution), with penalties for needless messages, trial runs and waiting.
The result
With constraints hidden, Claude Opus 5.5 scored 0.56 and resolved 14 tasks; Claude Sonnet 5.5 scored 0.49 with 15; GPT-6 Astra 0.38. Hiding the rules cost the top three models 0.28 points on average; one small model scored below doing nothing. The benchmark is the authors' own and not yet reproduced.
Use it
Give agents that work in someone else's system an explicit interview step: list the unknowns, ask, write down the constraints, then act. Budget the questions, since the scoring penalizes needless ones.

For youMatches your client-facing SaaS work: add a "list unknowns and ask" step to the skills that touch a customer's data or deployment.

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadarXiv 2610.22350CodeProject page

Software and AI PreprintarXiv preprint

A small model that runs real bug fixes backwards plants realistic new bugs: cheap test tasks for coding agents.

SWE-at-Scale: Scalable Issue Synthesis from Historical Bug FixesSilin Chen, Zihan Zhang, Xiaodong Gu et al.

SWE-at-Scale pipeline: stage one reverses real developer fixes into training pairs; stage two uses a bug-inducing model to write a patch or a bug pattern that is applied to a clean repository to make a buggy one.
Image: Chen et al., arXiv
The problem
Training and testing coding agents needs many verifiable bug-fix tasks. Bugs invented by a language model tend to be unrealistic or repetitive.
The idea
Invert 13,407 real fixes from 688 repositories into bug-insertion examples, fine-tune a 9B model on them, and keep a planted bug only if it breaks a test that passed before.
The result
Human raters judged 94.0% of the bugs from the stronger variant usable, against 72.2% for the best rewrite baseline. 88% of usable patches were unique, at 17,450 tokens per task against about 2.29 million for a feature-adding baseline. Four agents that score 69.6 to 97.0 on SWE-bench Verified scored 16.8 to 37.0 on these tasks.
Use it
Build a private evaluation for your agent setup from your own git history: take fix commits that touch a test, reverse them, and keep only the versions that turn a green test red.

For youA way to score changes to your Claude Code skills and hooks on your own repos instead of public benchmarks.

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadarXiv 2610.21359Code

Also moving

This week on Toa Radar

Read the weekly summary: The week proof became the bottleneck
Tune mode is on in this browser. Turn it off