No. 30 Thursday 12 November Game dev, software and AI

Tests the agent cannot rewrite

When an agent writes the code, keep the checks somewhere it cannot quietly edit: a table of cases, or tests proven against the real service. Plus a flood of AI-found bugs, what reliability layers really buy, and a shooter recompiled for Apple silicon.

The short version

  • Software and AI A Thoughtworks engineer argues that when AI writes the code, the tests are your contract with it, so keep them as data in tables the agent cannot quietly change.
  • Software and AI AWS built a free local AWS emulator with agents, first proving each test against real AWS and then letting agents code until the same tests passed.
  • Software and AI VulnCheck counts 442 CVEs credited to Anthropic in seven months. Only 217 of nearly 38,000 findings are marked fixed, so expect a long tail of patches.
  • Software and AI Two field tests of agent add-ons: on a coding task, cheap checks beat voting; on code search, plain file tools held up against most indexes.
  • Game dev A fan project recompiles Black Ops II into native Apple silicon code ahead of time, with agents doing the grind; it runs at 51 to 60 fps on an M1.
Software and AI Lesson

Keep the contract in a table

If a coding agent can edit both the code and its tests, a passing test proves little. One engineer's fix: write the expected behaviour as plain tables of cases, and let the agent write only the code that reads them.

ContextMartin Fowler's site publishes long-form articles by Thoughtworks engineers; this one is by Matteo Vaccari.

The tests-as-data loop as the article describes it: you own the cases, the agent owns the code.

Vaccari interviewed 26 developers at a company that leans heavily on coding agents. Many had stopped trusting their tests, because the agent rewrote the tests along with the code whenever it changed something. His answer: when AI writes most of the code, you are its customer, and customer tests, not developer tests, become the contract between you.

So move that contract out of test code and into test data: a table with one row per case (input, expected output, expected error), plus a small runner that reads the table and calls the real system. You draft a sample of the format, the agent writes the runner, you check the runner against production code, and both of you add rows. Tables are quick to read, which makes reviewing them realistic.

The payoff is a simple rule about what may change. Refactoring touches only production code; adding a feature touches code and data; improving coverage touches only data. A diff that breaks the rule is a signal to stop and look.

Technical details

The article shows a CSV greeting table (columns test_name, resource, name, greeting, error) and an access-control table for an admin dashboard (role, session state, resource, expected result, logged). It ties the idea to older practice: XP acceptance tests, Fit, Go's table-driven tests and approval testing, which AI now makes cheap to set up because the custom runner costs minutes.

The author cites outside reports that agent-written code has raised defects (Faros AI, DX, a CloudBees survey of 213 tech leaders); those figures are cited, not measured by him. He lists limits: it suits production systems rather than prototypes, an external format only pays if people review the tables, and it does not replace exploratory testing.

Why it mattersAn agent that can rewrite its own tests can make any change look green. Separating the expected behaviour from the code that checks it gives you something stable to review.

For youPlugin damage, inventory stacking and interaction rules are natural tables; keep them in a data folder the agent may read but not edit, and review only the rows.

customer tests
Tests written from the user's point of view: given this input, the system must answer this, regardless of how the code inside is arranged.
runner
A small program that reads each row of the table, feeds it to the software, and compares the answer with the expected one.
refactor
Changing how code is organised without changing what it does.

Sources martinfowler.com: Tests as data

More like this Less like this

Checks the agent cannot bend

Tables you own, and tests proven on the real thing.

Software and AI Lesson

Which files an agent should touch, by task

With tests kept as data, every kind of task has an expected footprint. Check a diff against this table before you read the code.

From Vaccari's article: what should change for each kind of work.

Should change Should stay put
New feature Production code and test data The test runner
Refactor Production code only Test data and the runner
More coverage Test data only Production code and the runner
Better test format Test data and the runner Production code

If an agent's refactor edits the table of cases, treat it as a bug, not a cleanup.

Why it mattersA one-line rule per task turns review from reading every change into checking that the right folders moved.

For youAdd this as a short section in each plugin repo's agent instructions so the agent states the task type before it starts.

Sources martinfowler.com: Tests as data

More like this Less like this

Software and AI Release

AWS let agents build its emulator, but only against tests proven on real AWS

Amazon released a free tool that imitates its cloud on your own machine, so code can be tested without an account or a bill. Agents wrote all of it, held to tests that were first checked against the real service.

How the emulator was built, per Swami Sivasubramanian's post on November 11:

The key step is the second one: a test only counts after it passes against the real service, so the agents chase recorded behaviour, not their own guesses.
Technical details

The tests come from AWS's public API models, written in its Smithy format. Agents adjust each scenario until it passes against real AWS, which records actual behaviour; then agents build the emulator until it passes too. A failing test marks a gap. Several adversarial security-review agents and a general reviewer sit behind the merge gate, and deterministic checks stop agents from touching code they should not.

Sivasubramanian says an internal version came together over a weekend. In his own anecdote, rebuilding a replication engine that once took about four months stalled for four nights until the agent had a spec and a test environment, then took about two hours. Simon Willison notes it is written in Rust, counts 29,684 commits in its first three weeks, and needs Docker on macOS and Windows.

Why it mattersIt is the tests-as-contract idea at scale: agents can write a whole service when the bar they must clear was set by reality, not by them.

For youFor the SaaS backends, the same trick works small: record a few real Supabase responses once, freeze them as fixtures, and let the agent build against those.

emulator
A program that imitates another system closely enough that software can run against it as if it were the real thing.
fidelity tests
Tests that check the copy behaves exactly like the original, using scenarios a real customer would run.
merge gate
Automatic checks every change must pass before it is allowed into the main code.

Sources All Things Distributed: AWS Emulator and the engineer's jobaws/aws-emulator on GitHubSimon Willison: aws-emulator

More like this Less like this

Bugs arrive faster than fixes

What AI bug hunting is producing, and one way to keep up.

Software and AI Trend

AI bug hunting is finding far more than anyone fixes

A security firm counted the public vulnerabilities credited to Anthropic's bug-hunting program: hundreds in seven months, but a tiny share of its findings are fixed so far, and maintainers rate most as less severe.

Flow chart: 37,966 findings split into 30,617 not in the ledger and 7,349 in it; of those, 217 fixed, 419 withdrawn, 1,654 duplicates, 4,866 disclosed and 193 not yet disclosed.
Image: VulnCheck

442CVEs credited to Anthropic in seven months

217of 37,966 findings marked fixed

2known to be exploited

VulnCheck's November update says Anthropic's model rated 91.5% of its September findings critical or high, against 51.3% by the maintainers. Between September 8 and November 2 the ledger gained 11,813 findings and 15 fixes.

Technical details

In curl, 45 reports led to 34 published so far: 19 judged not vulnerabilities, 6 confirmed low severity and 9 pending. VulnCheck's author suspects a ledger bug inflates the disclosed count, and the chart strikes through an earlier figure. Anthropic claims a 94.2% true-positive rate; that is its own number.

Why it mattersDependency updates will keep coming in larger batches, and many will be labelled more urgent than they are. Triage by exposure, not by the label.

For youPin and batch dependency updates on the Node services, and read the maintainer's severity, not the reporter's.

CVEs
Common Vulnerabilities and Exposures: public ID numbers given to known security bugs so everyone refers to the same one.
ledger
Anthropic's public list of what its bug-hunting program found and what happened to each finding.

Sources VulnCheck: Anthropic Glasswing November update

More like this Less like this

Software and AI Lesson

Patch every copy of a library at once

Geoffrey Huntley argues that if bugs now arrive faster than upstream fixes, teams should keep their own patches and apply them everywhere from one definition: laptops, CI, agent sandboxes and servers.

In a write-up of his talk, Huntley uses Nix: a patch file sits next to the package definition, an overlay routes every consumer to the patched build, and patching a core library rebuilds whatever depends on it. A note in the agent instructions tells the next agent to remove temporary patches after seven days.

  • Back pressure: hooks that refuse commits that do not build.
  • Static checks that enforce module boundaries.
  • Property-based tests for behaviour.
  • Agents run loose only on throwaway machines, never near production.
Technical details

Huntley's example patches git to disable force-push and shows that an OpenSSL patch recompiles dependents such as Nginx and Postgres; CI builds once and publishes to a cache. He names Bombadil and Hegel for property testing; both come from Antithesis, a company he is affiliated with. The post has no code; his demo repo is ghuntley/nix-demo.

Why it mattersWhen fixes lag reports, waiting for upstream leaves every machine exposed; a single patched definition closes the gap everywhere at once.

Nix
A package manager that builds software from exact written recipes, so the same recipe gives the same result on every machine.
overlay
In Nix, a layer that swaps in your version of a package for everything that uses it.
property-based tests
Tests that check a rule holds for many generated inputs, instead of a few hand-picked examples.

Sources Geoffrey Huntley: the answer to AI is engineering

More like this Less like this

Measure before you add a layer

Two field tests of add-ons people bolt onto agents.

Software and AI Lesson

Reliability layers mostly sort answers, they rarely fix them

Two engineers tested common add-ons that are meant to make an AI pipeline more accurate. The free check was the most useful; the expensive ones changed little or broke right answers.

Tokens spent per run on a medical-code extraction task, lower end of each reported range:

The check fixed nothing but split answers well (about 5 in 6 right among those it accepted). Self-correction fixed nothing; voting netted minus one to plus one codes and destroyed 2 to 5 right ones; the judge lifted F1 from 0.397 to 0.420. The baseline was right about 1 time in 3.

Technical details

Recalling codes from memory invented codes 13 to 18% of the time; picking from a menu of valid codes was about ten times better. A refusal rule passed 21 to 23% of records at 0.74 to 0.82 accuracy. The authors' advice: report yield (how many records you keep) next to accuracy.

Why it mattersBefore paying for voting or a judge model, turn free-text answers into choices from a list and add deterministic checks; that is where the gains were.

For youFor any classifier step in the SaaS products, make the model pick from an enum and validate it in code before adding a second model.

F1
A single score that balances how many answers were right against how many right answers were missed.
self-correction
Asking the model to review and fix its own answer.

Sources InfoQ: LLM reliability layers under test

More like this Less like this

Software and AI Lesson

Plain file tools held their own against code indexes

A streaming company tested six ways for a coding agent to find code, from plain file search to search indexes. Plain tools were competitive on concrete tasks; indexes helped mainly when the task was vaguely described.

Stacked bar charts of tool calls per task for five setups, split into knowledge base, search, read, edit, shell and other; the file-system baseline uses mostly search and read calls.
Image: JioHotstar via InfoQ

JioHotstar ran Claude Sonnet as the agent on SWE-bench subsets, CodeScaleBench and an internal question set, comparing a file-tools baseline with Sourcebot, Graphify, CocoIndex Code, Repowise and an in-house RAG index. Sourcebot beat the baseline on harder tasks and cost less; on easier ones, retrieval added overhead and did not reliably cut tokens.

Why it mattersAgents already reach for text search and file reads. An index is worth its upkeep only if your traces show it helps on your own work.

For youKeep the Unreal plugin repos on plain tools; if search gets slow in the engine source, try a structural search before any embedding index.

RAG
Retrieval-augmented generation: fetching relevant documents and adding them to the model's prompt.
SWE-bench
A benchmark of real GitHub issues that a coding agent must fix in the actual repository.

Sources InfoQ: JioHotstar benchmarks retrieval for coding agents

More like this Less like this

Game dev: ports, tools, a closure

A native Mac port, two free tools, and a warning.

Games and software Lesson

Black Ops II, translated to native Apple silicon code

ContextRadar covered a similar recompile of a racing game to the browser on November 8; this one targets native Mac code through LLVM and leans on agents.

A fan project converts your own Steam copy of a 2012 Windows shooter into a native Mac program before it runs, with no emulator, and reports 51 to 60 frames per second on a base M1.

A first-person view of the Zombies mode running on a Mac, a burning zombie in a dark street, with a frame counter reading 55.
Image: bo2-mac

What the converter does on your Mac, in about 1 hour 45 minutes on an M1:

  1. Check

    Accepts only the exact Steam build, matched by checksum.

  2. Lift and fold

    Ghidra lists the functions, Remill turns each into LLVM's portable code, and clang compiles it for ARM64.

  3. Run

    A hand-written Windows layer sends Direct3D 11 to Metal and audio to CoreAudio.

Technical details

The README says AI agents supply the throughput (writing shims, bisecting millions of calls, reading code) while the human sets goals and corrects from what is on screen; the Show HN post names the model. Measured only on one M1 (16 GB): campaign 51 to 60 fps, multiplayer against bots 53 to 60. Known issues: no online play, a fixed 1280 by 720 window, and very high temperatures in long sessions. Code is GPL-3.0; users must decrypt their own executables with a separate tool.

Why it mattersAgents make the slow, mechanical parts of a port affordable for one person, while a human still judges the result. That shifts what legacy code is worth rescuing.

For youThe lift, fold and runtime split is a template for moving old C++ tools between platforms with agents doing the shims.

LLVM
A widely used compiler toolkit with a portable in-between code that can be compiled for many processors.
ARM64
The processor family in Apple silicon Macs and most phones.
Metal
Apple's graphics programming interface.

Sources adrbn/bo2-mac on GitHub

More like this Less like this

Game dev Release

Editable Motion Paths

Keyframes become points and their easing becomes handles on a 3D curve.

A free Blender add-on draws the path a bone or object travels as a curve you can grab and reshape in the viewport, and the animation updates as you do.

Two Blender viewports: a hand's swing drawn as a coloured oval path with a selected point and handle, and a character with several coloured motion arcs around the arm.
Image: Editable Motion Paths, Blender Extensions

Mesh-editing moves such as snapping, extrude, knife and bevel work on the path and rewrite the animation curves live, so cleanup happens where you see the motion instead of in the Graph Editor. It works on existing rigs with no setup or baking, and adds ghost frames, easing presets and breakdown tools. Its author says the add-on was built heavily with AI help.

Price
Free
Needs
Blender 4.4 or newer
Setup
None, works on existing rigs

Why it mattersAdjusting an arc where you can see it is faster than nudging abstract curves, especially for combat swings and jumps.

For youTry it on one attack animation for the action RPG before FBX export and compare cleanup time with the Graph Editor.

keyframes
Poses saved at specific moments; the software fills in the motion between them.
Graph Editor
Blender's panel of value-over-time curves, where animators normally fine-tune motion.

Sources Blender Extensions: Editable Motion Paths80 Level: edit animations like a mesh

More like this Less like this

Game dev Release

Sign in with Google for Unity

Returns a signed Google ID token to C# that your backend can verify.

Google released a free plugin that adds Google sign-in to Unity games on Android and iOS, configured in the editor instead of by hand-editing platform files.

On Android it uses Credential Manager and on iOS Google's sign-in library; you paste a web client ID and an iOS client ID into the editor. The token works with Firebase Auth, Supabase, Auth0 or your own server. It replaces an older plugin that broke with current Android tooling, and it only signs players in: account management is up to you.

Platforms
Android and iOS
Needs
Unity 6.5
Output
Google ID token
Source
Open, on GitHub

Why it mattersMobile sign-in has been one of the fiddliest parts of a Unity build; an official, maintained plugin removes a common source of store-review failures.

For youSupabase accepts a Google ID token directly, so a Unity mobile prototype could share accounts with the SaaS backends.

ID token
A signed piece of data that proves who the user is, which a server can check without asking Google again.
Credential Manager
Android's built-in system for passwords, passkeys and sign-in providers.

Sources googlesamples/siwg-mobile-unity-pluginPocketGamer.biz: Google releases the plugin

More like this Less like this

Game dev News

Fortis Games shuts down

A mobile studio that grew through acquisitions closed after its only backer stopped funding it, before any of its games fully launched.

440jobs at risk

0games fully launched

Las Vegas Sands, its sole financier, ended funding. Twilight Towers, in soft launch since September 2025 with weak retention, closes on November 16.

Why it mattersScaling a team before one game proves it keeps players leaves a studio exposed to a single funding decision.

soft launch
Releasing a game in a few countries first to test retention and revenue before a worldwide launch.

Sources PocketGamer.biz: Fortis Games shuts down

More like this Less like this

Learn one thing

What is a test oracle?

A test oracle is whatever tells you the right answer for a test: a table of expected results, a trusted older system, or the real service. A test is only as good as its oracle.

Think of it like thisIt is the answer key for an exam. If the student can edit the answer key, every exam gets full marks.

Running a test is easy; knowing what the result should be is the hard part. That knowledge can come from a person writing expected values, from a specification, from a second implementation, or from recording what the real system does.

Coding agents make the question urgent. If the same agent writes the code and the expected values, the oracle is no longer independent, and passing proves little. Today's stories show two ways to keep it independent: tables of cases a person owns, and tests that must first pass against real AWS before agents build against them.

A test compares what the software says with what the oracle says. Keep the oracle out of the agent's reach.
specification
A written description of what software must do, independent of how it does it.

Try this today

Games and software 45 minutes For practitioners

Move one module's tests into a table the agent cannot edit

Find out whether your agent can refactor a module while the expected behaviour stays fixed, and catch it when it tries to change the rules.

You needA small module with clear inputs and outputs (a pricing rule, a damage formula), and Claude Code or another agent with file permission rules.

  1. Write the table

    Put five to ten cases in a CSV under tests/cases/: one row per case with the input, the expected output and any expected error, like the article's greeting table.

  2. Get a runner

    Ask the agent to write a small test runner that reads every row, calls the module and compares results. Check it yourself by breaking one row on purpose: the run must fail.

  3. Lock the cases

    Add the deny rule below to .claude/settings.json so the agent's file tools cannot edit the table. Shell commands are not covered, so still check the diff.

  4. Refactor

    Ask the agent to refactor the module. Afterwards, run git diff: only production code should have changed.

{
  "permissions": {
    "deny": ["Edit(/tests/cases/**)"]
  }
}

You know it worked whenThe refactor passes against the unchanged table, git diff shows no changes under tests/cases/, and a deliberately broken version of the module fails the run.

For youStart with one plugin's damage or stacking rules; the same CSV can later feed a Modular Test Framework test in Unreal.

deny rule
A Claude Code permission setting that blocks a tool from acting on matching files, whatever the agent decides.

Sparks

What is catching on right now, and the idea worth borrowing from each.

The lab

Research from the front line, in plain words: what it solves, how, and what you could do with it.

Software and AI PreprintarXiv preprint

Agent notes help more when each lesson links to the run it came from

Don't Just Trust Your Memory: Evidence-Linked Memory Reduces Memory-Induced Degradation in LLM AgentsYukyeong Lee, Yumin Choi, Jinheon Baek, et al.

Two-part figure: an agent fails a task with memory that it solves without it; then the agent checks a doubtful insight against its compilation and source run before revising it.
Image: Lee et al., CC BY 4.0
The problem
Lessons an agent saved from earlier runs can make it fail tasks it would solve without them, usually because it follows a wrong or stale lesson.
The idea
Store each lesson as a chain: a short insight, a summary of why that run went well or badly, and the raw run. The agent reads the insights, and follows the links to the evidence when a lesson is vague, conflicts with another or came from a failure.
The result
Across 8 suites in 5 benchmarks, average success rose from 79.5 with no memory and 87.0 with insights alone to 90.2. Tasks lost because of memory fell from 6.5% to 4.5%, for about 1.17 times the cost of insights alone.
Use it
Next to each lesson in an agent's instruction file, link the transcript or pull request it came from, and add one line: if a lesson looks wrong or conflicts, open its source before acting.

For youLink each lesson in the routine notebook and CLAUDE.md files to the session or commit it came from.

More like this Less like this

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadPaper (arXiv)Code

Software and AI PaperACM Transactions on Software Engineering and Methodology

About a quarter of active GitHub projects already show coding-agent traces

Agentic Much? Adoption of Coding Agents on GitHubRomain Robbes, Théo Matricon, Thomas Degueule, et al. (Univ. Bordeaux, CNRS, Univ. Rennes, Inria)

The problem
Surveys say many developers use coding agents, but nobody had measured adoption from what agents leave behind in real repositories.
The idea
Scan 128,018 GitHub projects for agent traces: instruction files such as CLAUDE.md, ignore-file entries, and agents named as commit co-authors, then compare agent-assisted commits with human-only ones.
The result
Estimated adoption was 22.20% to 28.66% on February 21, 2026, spread across project ages, languages and topics, including established organisations. Agent-assisted commits are larger than human-only ones.
Use it
Expect bigger diffs from agent-assisted work and size code review to match. The released pipeline can measure the same signals in your own organisation's repositories.

For youRun the pipeline on the plugin repos to see how commit size changed since agents took over.

More like this Less like this

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadPaper (ACM TOSEM)Preprint (arXiv)Code

Game dev PreprintarXiv preprint

A game test bed that checks agents against the engine's state, not their own claims

GTAGym: Real AAA Game Tasks Environment for Evaluation and Reinforcement LearningJiajun Chen, Shicheng Fang, Jiahao Lu, et al.

Grid of game screenshots labelled by task family, such as road navigation, interception, aquatic and aerial navigation, with an instruction under each, plus shooting and sniping tasks.
Image: Chen et al., arXiv
The problem
Training or grading game-playing agents in a big commercial game needs people to watch and judge whether each task was done.
The idea
Turn 75 GTA V story missions into 150 tasks, rebuild the 3D ones in a browser engine where success is read from game state, and generate new task variants from templates to make training environments with automatic rewards.
The result
The best model scored 70.9 against 89.8 for people. Training an open 27B model on 50 generated navigation tasks lifted on-foot accuracy from 54.2% to 72.9%, and success on 100 tasks in the real game from 15% to 23%; shooting accuracy fell.
Use it
Copy the two parts that transfer: a verifier that reads success from your engine's state, and a template generator for task variants. Together they give automated playtests of your own game.

For youAn engine-state verifier is what an agent-driven Unreal playtest needs to know a quest step really completed.

More like this Less like this

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadPaper (arXiv)Code

Also moving

This week on Toa Radar

Read the weekly summary: The week the checks moved outside the agent
Tune mode is on in this browser. Turn it off