No. 17 Friday 30 October Game dev, software and AI

Who checks the agents' work

Meta catches bugs with tests written for one change and then thrown away. A team that let agents write everything got an app that runs but a build that stays red. And a game engine lets its agents play-test their own work.

The short version

  • Software and AI Meta now runs AI-written tests overnight on yesterday's risky code changes; they caught 74 serious privacy bugs before release in the first half of 2026.
  • Software and AI Thoughtworks let agents write all the code for a week: about 80% done, a build often red, and two thirds of the time spent at the whiteboard.
  • Game dev A research game engine lets agents pause, step and play the game they are building; dropping it cut quality more than dropping the reviewers.
  • Software and AI The top coding agent setups now sit within three points of each other; cost per task is where they differ.
  • Game dev South Korea fined Google and Apple $47.6m and ordered them to let apps link to outside payment options.
Software and AI Lesson

Meta writes a test for each risky change, then throws it away

Every night, Meta has AI models write fresh tests aimed at the previous day's riskiest code changes. A test that fails on the change is evidence of a bug, and only the strongest cases reach an engineer.

ContextMost AI code review leaves comments for a person to judge. Meta's approach produces a failing test instead, which shows the behaviour change directly.

Meta pipeline diagram. Top: code under test, mutants, an LLM writes extra tests, tests that pass on the parent and fail on mutants and the diff become weak catches. Below: the dodgy diff and intent-aware workflows that make mutants.
Image: Meta Engineering

Meta calls them Catching JiTTests, for just-in-time tests. They run overnight on spare machines, against changes submitted the day before that a risk model marks as risky. The tests are written for that one change and are not kept in the test suite.

There are two ways in. One treats the change itself as a possible bug and looks for any behaviour that differs from the code before it. The other has a model guess what the change is meant to do, list how it could go wrong, plant those mistakes as small variants of the code, and write tests that tell them apart. Tests that fail on the real change become candidates.

Most candidates are false alarms, so a panel of AI models judges each one, and a rule-based checker gives a second, deterministic verdict. Only the highest ranked go to the engineer who wrote the change, who mostly answers one question: did you mean the behaviour to change?

Meta reports 589 engineers engaged with the alerts (64% of those contacted), a 60% true-positive rate by their judgment, and 74 serious privacy bugs caught before release in the first half of 2026, when privacy incidents fell 20%. The paper was presented at FSE 2026.

74serious privacy bugs caught before release, first half of 2026 (Meta-reported)

Technical details

The two workflows are called dodgy diff and intent-aware. The intent-aware one reads the diff, its title and summary, describes the risks in text, and turns each risk into a mutant that still passes the existing tests. Tests that kill those mutants and fail on the real diff are called weak catches.

Risky diffs are picked with a model similar to Meta's Diff Risk Score, trained on past changes. The false-positive filter pairs an ensemble of LLM judges, which is not deterministic, with RubFake, a rule-based environment that is.

Meta frames this as human on the loop rather than in the loop: engineers set the criteria and review results in aggregate, and rarely read the generated test code.

Why it mattersIt turns AI review from opinions into evidence a person can check in seconds. The parts (pick risky changes, generate tests, filter hard, show only the best) work at any team size.

For youCopy the shape for agent-written PRs in the SaaS repos: a nightly job where Claude Code writes throwaway tests against yesterday's riskiest diffs and reports only tests that fail on the change. Start with the intent-aware prompt.

mutants
Copies of the code with a small deliberate mistake planted in them, used to check whether tests notice.
true-positive rate
The share of alerts that turned out to be real problems.
FSE
The ACM conference on the Foundations of Software Engineering, a major research venue.

Sources Meta Engineering: Catching bugs before they land

More like this Less like this

Checking the work

When agents write most of the code, the checks are the job.

Software and AI Lesson

No human code for a week: the app ran, the build stayed red

Thoughtworks had agents write every line of an app for a week, twice, with different teams. They got most of the way, but the checks kept failing.

80%of the app done after a week

2/3of the team's time spent at the whiteboard, not building

Rachel Laycock, Thoughtworks' CTO, says the build was often red. Possible causes: too many agents in parallel, code not modular enough, and changes arriving faster than checks could finish. "Built is not the same as working in production."

Why it mattersThe design work did not shrink; it became most of the job. And agents can outrun your checks, which is exactly where bugs get through.

For youWhen running several agents on one repo, cap parallelism to what CI can keep green; a red main branch should stop the routine.

Sources Rachel Laycock: Why are good developers so wary of agentic development?

More like this Less like this

Software and AI Lesson

How a dozen people changed 250 million lines, before AI

ContextTitus Winters led C++ libraries at Google and co-wrote Software Engineering at Google.

In 2010 a volunteer team of about twelve changed code across Google's codebase of close to 250 million lines, used by about 10,000 engineers. Titus Winters says the recipe still applies when agents make the edits.

The sequence, as Winters describes it on The Pragmatic Engineer. No language models were involved; the edits were generated by tools.

  1. Parse everything

    A batch job read every build target into a syntax tree, so the code could be searched by structure, not text.

  2. Find the exact patterns

    Match the precise calls that need to change, and generate the edits by machine.

  3. Run old and new side by side

    Both versions of the API work at once, so nothing breaks while the change is under way.

  4. Ship in shards

    Split the edits by directory and send them out in batches that owners can review.

  5. Block the old way

    Stop new code from bringing the old pattern back in.

  6. Delete

    Remove the old API once nothing uses it.

Technical details

The batch job was a MapReduce whose mappers parsed each build target; the clang-format toolkit came out of this work. Winters also says, from conversations rather than measured data, that at other companies code output is up hugely, outages are up about 30%, and features shipped are roughly flat.

Why it mattersAgents make large edits cheap, but a big change still lands safely only in this order. Steps 3 and 5 are the ones agent migrations tend to skip.

For youUse this order for the next cross-plugin API change: keep the old function alive, let agents migrate one plugin at a time, then add a lint or hook that blocks the old call before deleting it.

syntax tree
A structured map of source code that tools can search and edit safely, unlike plain text.
API
The set of functions other code calls to use a library.

Sources The Pragmatic Engineer: Software engineering at Google with Titus Winters

More like this Less like this

Game tech to borrow

An engine agents can play, faces on a standard rig, and store and engine changes.

Game dev Lesson

A game engine built for agents to play-test their own work

Researchers built a 2D web engine that agents can read, pause and play, plus a team of agents where builders are checked by a separate auditor, player and reviewer. The engine mattered more than the reviewers.

VibeGame's loop, from the paper. The game is stored as checked text files, and agents can step the running game frame by frame and press buttons in it.

Without the agent-readable engine, the score fell from 53.45 to 43.12; without the reviewing roles, to 48.82.
Technical details

Eight agents in all: orchestrator, designer, artist, architect, programmer, auditor, player and reviewer. The engine is built on Phaser, a web 2D framework; scenes, nodes, assets and scripts are schema-checked text with explicit references. Agents can pause, advance a set number of frames, read and change properties such as positions and velocities, send actions like "press jump", and take screenshots.

On a 30-task subset of GameCraft-Bench, scored by a model judge, it beat the best baseline by 11.70 points with a smaller model and Codex by 4.19 with a stronger one. In a blind study, 30 people preferred its games 72.7% of the time on average. Limits: 2D only, and saved lessons from past games sometimes hurt when the next game was a different type.

Why it mattersAgents find many game bugs only by playing. Giving them a way to pause, step and press buttons is a smaller job than building an agent team, and it helped more.

For youAdd a debug hook to the three.js games (pause, advance N frames, inject input, dump state as JSON) and a console command set in UE, so Claude Code can play-test a change instead of only compiling it.

play-test
Running the game and playing it to find problems, rather than only reading the code.

Sources arXiv: VibeGameProject page

More like this Less like this

Game dev News

Korea fines Google and Apple over in-app payments

South Korea's regulator fined both stores and ordered them to let apps link to outside payment options.

$32mGoogle (KRW 42.9 billion)

$15.6mApple (KRW 20.81 billion)

The regulator objected to forced store payments, a 26% fee on third-party payments and unexplained review delays. Apple disagrees and is reviewing its options.

Why it mattersIf the orders stick, mobile games sold in Korea could send players to cheaper payment pages.

For youNot urgent for the Flutter apps; watch how each store implements external links before planning a Korea pricing change.

Sources PocketGamer.biz: South Korea fines Google and Apple

More like this Less like this

Game dev Release

MetaWrap

Fits a sculpt, scan or generated model to MetaHuman topology and moves its skin textures across.

ContextMetaHuman is Epic's system for realistic digital humans with a ready-made facial rig.

A new Blender add-on wraps your own character onto Epic's MetaHuman face layout and sends it to Unreal with the real MetaHuman rig.

Elvis Morelli's suite works on a whole head, skin only, or a full character, automatically or guided by matching landmark points. MetaSculpt adds scan detail and wrinkle maps; MetaGroom moves hair between Unreal and Blender.

Price
$158
Needs
Blender 5.0+, Unreal 5.6+
Runs on
Windows only

Why it mattersCustom faces on the standard rig mean animation, lip sync and tools built for MetaHumans work on your own characters.

For youA cheap route to distinct NPC faces for the action RPG without hand-rigging; test one head before buying it for a crowd.

topology
The layout of a 3D model's polygons; a rig only works on the layout it was built for.

Sources CG Channel: MetaWrap

More like this Less like this

Game dev Do this

Godot 4.8 is feature-complete: what to check before it ships

ContextGodot is a free, open-source game engine.

Godot 4.8's second beta adds no new features, only fixes, and asks people to test their projects now. Two changes can break existing games.

From the beta 2 notes: 144 fixes from 59 contributors in two weeks.

  1. Install .NET 10 if you use C#

    The C# build of 4.8 now needs .NET 10 or newer, up from .NET 8.

  2. Open a copy of your project

    Never the original: beta builds can change saved files.

  3. Check your particles

    The default range of the particle angle curve changed to -1 to 1, so some effects may look different.

  4. Report what broke

    Regressions found now get fixed before the final release.

Technical details

Also in beta 2: building the engine from source needs Python 3.10, the Compatibility renderer no longer draws multiply blending too dark, buttons check the visible viewport area for touch input, and Unicode data moves to 18.0.0. Beta 1 brought the game view embedded in the editor by default, a Trail3D node, mip-level texture streaming and screen-space contact shadows.

Why it mattersTesting a beta costs an hour now; finding a break after the release costs a patch.

For youOnly relevant to Godot side projects; skip unless one is active.

Sources Godot: Dev snapshot 4.8 beta 2GameFromScratch: Godot 4.8 beta 1

More like this Less like this

Budgets and fences

Limits for agents that run on their own, and what the top ones cost.

Software and AI Release

GitHub's automations get spending caps and learn from results

GitHub's Copilot Automations, agents that run on triggers, can now stop at a credit limit, start each other, and suggest changes to their own prompts based on what people did with their work.

A run stops when it hits its per-run limit; daily, weekly and monthly limits make new runs wait for the reset. Automations can be started from outside systems through an API, by 24 more GitHub events, or by another automation finishing or failing.

  • Custom MCP servers, with only the tools you pick switched on
  • Proposes prompt, tool or trigger changes from signals such as merged PRs and changed labels; you accept or reject each
  • On Pro, Pro+, Max, Business and Enterprise plans

Why it mattersA per-run cap is the cheapest guard against an unattended agent burning money in a loop, whichever tool runs it.

For youCopy two ideas into the morning routines: a hard spend cap per run, and a weekly job that proposes prompt edits from what got merged or reverted.

MCP
Model Context Protocol: a standard way to plug tools and data sources into an AI agent.

Sources GitHub: tools, triggers and cost controlsGitHub: automations that self-improve

More like this Less like this

Software and AI Trend

The top coding agents sit within three points

ContextArtificial Analysis is an independent company that benchmarks AI models; its Coding Agent Index scores a model and its agent tool together.

On one independent score for coding agents, the leading setups are close together. Cost per task now separates them more than skill.

Coding Agent Index, model plus agent tool. Gemini 4 Carbon also costs about 23% less per task than Claude Opus 5.5 on the firm's main index.

Claude and Gemini scores are from Artificial Analysis. Fusion's is reported by GitHub and Mistral's by Mistral. Nathan Lambert argues the firm's v5 index is flawed but useful, and that a coding-only index would help.

Technical details

Artificial Analysis puts cost per Intelligence Index task at $7.76 for Gemini 4 Carbon, $10.04 for Claude Opus 5.5 at max effort and $6.25 for GPT-6 Astra, with Carbon using 123k output tokens per task against Opus's 173k. Carbon is listed at $4 in and $20 out per million tokens, with a 50% launch discount and no end date. Mistral Large 4 is an open-weight model at $0.70 in and $2.10 out on Mistral's API.

Why it mattersWhen the top tools score alike, pick on price, speed and how well they fit your workflow, and rerun your own tasks rather than trusting one number.

For youNo reason to switch from Claude Code; worth a monthly check of cost per task for the routine's heavy desks.

Sources Artificial Analysis: Gemini 4 Carbon is the new #1Interconnects: the benchmarker's dilemmaMistral: Large 4 open weightsGitHub: Copilot Fusion

More like this Less like this

Software and AI Release For practitioners

Claude Code 2.1.314 fences unattended agents in

The latest Claude Code release adds a setting that stops the agent writing files outside its project, and fixes several ways long unattended runs failed quietly.

Show the details

The four changes that matter most for agents left running on their own, from the release notes.

  • No writes outside the project

    The new permissions.blockWritesOutsideWorkingDirectories setting makes Write, Edit and NotebookEdit refuse paths outside the working directories, in every permission mode.

  • Headless failures now fail loudly

    claude -p used to exit with code 0 and no output when startup failed. It now prints the error and exits with 1, so scripts and schedulers notice.

  • Hooks see Monitor commands

    Hooks that match Bash now also run on shell commands started by the Monitor tool. They can block or ask, but not approve or rewrite them.

  • Leaner long runs

    In long headless and SDK turns with many subagents, the parent no longer keeps each subagent's tool results until the turn ends. PreCompact hook output is cut at 10,000 characters.

Why it mattersUnattended agents fail in quiet ways: a wrong exit code, a write in the wrong folder, a hook that never fired. This release closes four of them.

For youTurn on the write fence in the routine's settings and make the scheduler treat a non-zero exit as a failed run worth a notification.

headless
Running a tool from a script with no person watching its screen.
hooks
Your own scripts that Claude Code runs before or after its actions, which can allow or block them.
subagents
Helper agents a main agent starts for parts of a task.

Sources Claude Code v2.1.314 release notes

More like this Less like this

Learn one thing

What is mutation testing?

Mutation testing checks your tests by planting small deliberate bugs in the code and seeing whether the tests notice. A test suite that misses a planted bug would probably miss a real one too.

Think of it like thisA fire drill with a fake alarm: you learn whether people respond before a real fire does.

A tool makes many copies of the code, each with one small change: a plus turned into a minus, a less-than turned into less-or-equal, a line removed. Each copy is a mutant. Run the tests on every mutant; if a test fails, the mutant is killed. Mutants that survive show behaviour your tests never pin down.

It used to be too slow and noisy for most teams, because tools made thousands of trivial mutants. Language models change that: they can plant a few realistic mistakes based on what a change is meant to do, then write the tests that catch them. That is how Meta's catching tests work.

Killed mutants mean your tests would notice that kind of mistake; survivors are the gaps.
test suite
All the automated tests for a project, run together.

Try this today

Software and AI 20 minutes For practitioners

Write the migration rule down instead of hoping examples win

In twenty minutes you find out whether your coding agent follows the new pattern you want or copies the old code around it, and fix it with one line.

You needA repo mid-migration (old and new patterns both present) and a coding agent that reads CLAUDE.md or AGENTS.md.

  1. Pick a pattern in flux

    Something the old code does one way and new code should do another: ID types, field casing, an old helper you are replacing.

  2. Run a task without a rule

    Ask the agent for a small addition next to old-style files, five times in fresh sessions. Count how often it copies the old pattern.

  3. Add an explicit priority rule

    Put one line in CLAUDE.md or AGENTS.md naming the new pattern and saying it overrides what existing files do.

  4. Run the same task again

    Five more fresh sessions, same prompt. Count again.

## Patterns in migration
- New code uses string IDs, not integer IDs.
  This overrides the older pattern you will see in existing files.

You know it worked whenWithout the rule, most runs copy the old pattern; with it, nearly all follow the new one. In the paper, existing code beat the agent's own default in 359 of 360 trials, and a rule in the instruction file worked as well as one in the prompt.

For youTry it on the plugin repos where newer modules use a different naming or interface style than the first ones.

CLAUDE.md
A file of standing instructions that Claude Code reads at the start of every session; AGENTS.md plays the same role for other agents.

Sparks

What is catching on right now, and the idea worth borrowing from each.

The lab

Research from the front line, in plain words: what it solves, how, and what you could do with it.

Software and AI PreprintarXiv preprint

Agents copy what your repo already does; only a written rule reliably changes that

Precedent or Intent? What Coding Agents Follow When Evolving Existing CodeHaichao Zhu, Geyang Xu, Zhaorui Yang, et al.

The problem
When a task wants a different pattern from the existing code, nobody knew which a coding agent would follow: the code around it, its own habits, or the instruction.
The idea
5,625 controlled trials with four coding agents and six pairs of interchangeable choices, such as integer or string IDs and field casing, with hidden tests holding the task fixed. They varied the agents' defaults, how many existing examples there were and where they sat, and explicit rules. Then they checked on four real open-source repos.
The result
Existing code beat the agent's own default in 359 of 360 trials. Three supporting files nearby gave 96% adoption; examples in sibling folders, 4%. A written priority rule got 59 of 60 joint successes against 38 of 60 for a code example alone, and worked as well in CLAUDE.md or AGENTS.md as in the prompt.
Use it
When moving a codebase to a new pattern, write the rule into the agent's instruction file. A few new-style examples lose to the old files nearby.

For youAdd explicit pattern rules to each plugin repo's CLAUDE.md where newer modules differ from the first ones.

More like this Less like this

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadPaper

Game dev PreprintarXiv preprint

One picture becomes game-ready hair cards with spring bones, in about a second

HairDeck: Animation-Ready Stylized Hair Cards from a Single ImageYuze He, Yixiang Wei, Wang Zhao, et al.

HairDeck pipeline: a hairstyle as a set of cards where one card is one token; a set diffusion transformer conditioned on a point cloud or image denoises card tokens; a decoder outputs cards, and a chain model adds spring-bone chains.
Image: He et al., arXiv
The problem
Stylized game characters use hair made of flat cards moved by spring bones, placed by hand. Image-to-3D tools make one fused mesh, and the one earlier card generator is slow.
The idea
Treat a hairstyle as an unordered set of cards, each one a single token, and generate them all at once with a diffusion model; a flag per slot decides how many cards exist. The same recipe generates the spring-bone chains.
The result
On a 100-hairstyle test set it beats the earlier card method: shape error 0.0117 to 0.0066, card orientation error 40.0 to 16.8 degrees, and 33.7 seconds to 1.01 seconds per hairstyle. Limits: trained on stylized anime hair, and it makes card geometry and UVs, not textures.
Use it
Use it for a first pass on stylized character hair, then retarget the chains to your engine's spring-bone system, such as Kawaii Physics in Unreal. Watch for texture generation.

For youRelevant only if the action RPG goes stylized; otherwise note it.

More like this Less like this

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadPaperProject pageCode

Game dev Lab reportNVIDIA Research technical report

Record gameplay once in Unreal, then replay it to render any camera or pass

Isekai3D SDG: A Platform for Diverse Synthetic Gameplay Visual DataRafal Karp, Gene Liu, Shih-Hsin Li, et al. (NVIDIA)

The problem
World models and game agents need gameplay video paired with camera, character and input data. Re-simulating for every camera or render pass, and copying huge datasets per experiment, is slow.
The idea
Unreal plugins record a session once (the engine's replay system plus a file of positions, camera poses and inputs, which can run without rendering), then replay it to capture any camera or projection and passes such as depth, normals and object IDs. A packaged build runs without the editor.
The result
No accuracy benchmark; it is a systems report. Recording several characters in one shared world gave about 1.3 to 2 times the throughput. Building a 10,000-hour dataset as references to stored video took 3 minutes, against 16 hours to copy the footage.
Use it
Borrow the pattern for your own Unreal tools: record deterministic gameplay once, then re-render it for trailers, regression screenshots or training data. No code release is mentioned.

For youRecord and replay is a fit for a regression-screenshot tool across the UE plugins' demo maps.

More like this Less like this

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadNVIDIA Research

Also moving

This week on Toa Radar

Read the weekly summary: The week agents got fences and rulebooks
Tune mode is on in this browser. Turn it off