No. 14 Tuesday 27 October Game dev, software and AI

Steer the fleet, trust the hardware

One person and a fleet of Claude agents put F-Zero X on a 3DS at 60 fps, and published how. Plus what trimming agent history really costs, a survey on AI time savings, scoped Supabase tokens and a Godot translation exploit.

The short version

  • Games and software One developer directed Claude agents to port F-Zero X to the Nintendo 3DS in 11 weeks: 777 prompts, 588 subagents, and a steady 60 fps on the New 3DS.
  • Games and software The rules behind it: a killswitch and a log receipt for every change, and a progress file written before any agent finishes.
  • Software and AI A study of 42 ways to trim an agent's history found that none saved money once prompt caching was priced in.
  • Software and AI In Atlassian's survey, developers saving 10+ hours a week with AI fell from 68% to 26%, and 65% spend time fixing AI work that missed the point.
  • Game dev A community translation file can run code in a Godot 4 game, and ActionForge offers free retargeting in the browser.
Games and software Lesson
A New 3DS mid-race: F-Zero X on the top screen, and a dashboard on the bottom screen with energy, race position, lap times and a course map.
Image: G-Diffuser postmortem

One person, 588 agents, F-Zero X on a 3DS

One developer brought the racing game F-Zero X to the Nintendo 3DS in 11 weeks without opening an editor. Claude agents wrote all the code; the human set goals, played the builds and approved every merge.

ContextF-Zero X is a 1998 Nintendo 64 racer. Fans rebuilt its source code from the game (a decompilation), and others ported that to PC; this project took the PC port to Nintendo's 3DS handheld.

The project, called G-Diffuser, ran from August 11 to October 26. One long-lived Claude Code session acted as orchestrator: it held the project memory, wrote briefs for background agents, merged their branches and was the only voice the human heard. The postmortem sums up the human's role as "a captain, not a programmer".

The postmortem counts 777 human prompts, 588 subagents, 1,288 workflow agents and about 22.6 billion tokens, which the author estimates at $19,559 at API list prices. On a New 3DS, every one-second window of a 3D race measured between 59 and 61 fps; the older, slower 3DS reaches a 60 fps median.

The speed came from a renderer that writes the 3DS graphics chip's own commands from data prepared ahead of time, instead of translating Nintendo 64 draw calls every frame. Hardware counters showed the old 3DS was starved by instruction-cache misses, so race loops were resized to fit its 16 KB cache. A renderer that won in the emulator lost on hardware and was dropped.

588subagents, steered through 777 human prompts

Technical details

Releases: v0.9.0 on September 4 (New 3DS only); v1.0.0 on October 19 (old 3DS support, the native renderer on by default, the 64DD Expansion Kit on New 3DS, and a bottom-screen dashboard called DASH); v1.1.0 on October 26 (2 to 4 player multiplayer, and both 3D eyes drawn in one GPU submission). No game data ships; players extract their own.

Old 3DS: the first boot on hardware ran at about 10 fps. Now 85% of one-second windows in a 3D race sit at 60, with a 10th percentile of 56, though the heaviest courses can still fall to 30. Drawing both eyes in one submission lifted the 3D menu from 40.2 to 44.6 fps. The port layer is about 181,000 lines under port/3ds, tests included.

Shelved along the way: a GPU vertex-transform attempt, a working pose-replay fallback (the human chose to aim for a real 60 instead), and a retained renderer that only won in the emulator. The Expansion Kit stayed New 3DS only because it was too slow on the old model.

Cost: the $19,559 is the author's estimate at list prices; the page also logs $15,567 in billing cache writes. The orchestrator's memory file reached 591 lines.

Why it mattersThis is one of the most detailed public records of a long, agent-built project: what the human did, what the agents did, what it cost and what broke. What transfers is the discipline around the agents, not the model.

For youUE plugin work has the same shape: an orchestrator session with a memory file, a worktree subagent per feature, and a killswitch plus a log receipt for each change. Copy the rules in the next block into your subagent brief template.

decompilation
Source code rebuilt by hand from a finished game's machine code, so it can be changed and recompiled.
orchestrator
The one agent session that hands out work to other agents, keeps the plan and merges what they return.
subagents
Extra AI agents started for one task each, which report back to the agent that started them.
instruction-cache
A tiny, fast memory on the processor that holds the code it is about to run; code that does not fit has to be fetched slowly.
emulator
A program that imitates a console on a PC, handy for testing but not always faithful to real hardware speed.

Sources G-Diffuser postmortemPostmortem: the agentic workflowRepository

More like this Less like this

Agents on a long leash

The rules that kept a fleet of agents honest, where AI time savings went, and fixes for runs nobody watches.

Games and software Lesson

The rules every agent brief carried

The F-Zero port's agents all worked from the same short list of rules, and most were written after something went wrong, like agents dying with no commits or killing each other's test runs.

Every background agent got its own git worktree and branch, a committed brief, and a progress file it had to write before finishing. These rules were copied into every brief.

  1. Keep upstream untouched

    Changes live as pure patches on top of pristine upstream code, and a clean round trip must pass.

  2. Give every change a killswitch

    Each change gets a settings key, so it can be switched off on the console without a rebuild.

  3. Demand a receipt

    Each change prints a log line that proves it actually engaged.

  4. Commit early, progress file first

    An agent that dies on a token limit or an API overload with no commits loses everything, so it writes its notes and commits before anything else.

  5. One emulator, one lock

    A lock file makes sure no agent kills another agent's test run.

  6. Review after merging

    After each batch of merges, an adversarial pass looks for damage from combining branches and marks each finding confirmed or suspected.

Technical details

What went wrong anyway: agents killed each other's emulator runs and deleted SD-card files; one run was credited to the wrong binary; a round shipped with stereo silently off; two changes together destroyed the HUD; patch hunks were carried twice; and the decompiled source tree was corrupted in the final week.

Verification: tick-exact input replays with per-frame state hashes compared new render paths with old ones, an emulator pool of three slots ran tests in parallel, and from October 21 each console got one combined test build instead of one card per feature. Twice, the human slept while a fleet worked against a frozen main branch and left a candidate branch and a status note for the morning.

Two more rules from the briefs: take a screenshot of a race frame with every change switched on before staging, and merge settings keys rather than overwrite the player's own choices.

Why it mattersEach rule makes an agent's work either recoverable or provable. That is what lets one person trust hundreds of agents they never watch.

For youAdd the killswitch and receipt rules to the UE plugin subagent brief: a console variable per change and a UE_LOG line that proves it ran.

git worktree
A second working folder attached to the same repository, so two agents can work on different branches at once.
killswitch
A setting that turns one change off without rebuilding, so you can compare with and without it.
adversarial pass
A review whose only job is to look for what is broken, not to confirm that things work.

Sources Postmortem: the agentic workflow

More like this Less like this

Software and AI Trend

Developers saving 10+ hours a week with AI fell from 68% to 26%

Atlassian's yearly survey says almost every team now uses AI, but far fewer developers report big time savings, and many spend hours steering and fixing what it produces.

Share of developers in Atlassian's 2026 survey of 1,155 developers and 405 engineering leaders.

A vendor survey, self-reported. The 2025 edition asked 3,500 people in six countries, so the drop is not a like-for-like comparison.

Technical details

Other figures: 95% say their organization uses AI and 87% say it helps them ship faster. Top friction sources are finding information (51%), meetings (50%), context switching (45%) and technical debt (44%).

60% say AI use is treated as a productivity signal, and 41% sometimes use AI more than needed because they feel expected to. Only 42% of leaders say they have metrics that separate real gains from activity. Developers with a clear AI strategy report less friction (41%) than those without one (22%).

Why it mattersThe savings now go into steering and checking AI output. Teams that measure outcomes rather than AI usage are the ones that can tell whether it pays.

For youTrack your own number: time spent steering and fixing agent output per shipped feature, not prompts sent.

friction
Anything that slows developers down that is not the work itself, such as hunting for information, meetings or switching tasks.

Sources Atlassian: State of Developer Experience 2026InfoQ

More like this Less like this

Software and AI Release

Claude Code 2.1.311 steadies long, unattended runs

The latest Claude Code update fixes several things that misbehave when agents run for hours without anyone watching, and makes the ultrathink keyword raise effort for a whole turn.

Released October 26. The changes that matter most for long runs:

  • Big requests keep going

    A request over 200,000 tokens used to stop and ask for long-context credits; it now compacts and continues.

  • Stop hooks fire once

    The SubagentStop hook ran about twice a minute per subagent; it now runs once, when the subagent finishes.

  • Late MCP servers reach subagents

    Subagents can use tools from MCP servers that connect after they start.

  • Worktrees stay put

    A subagent's worktree is no longer removed while a background agent is still using it.

  • ultrathink means xhigh

    In the terminal, the keyword runs that turn at xhigh effort on current models, even when typed mid-turn.

Technical details

Also fixed: long freezes from a CLAUDE.md with a very long list item; Write and Edit stalling up to 5 seconds on thousands of changed lines; interrupts that did nothing while an MCP server hung; endless retries of errors that cannot succeed. Shell timeouts now show the output so far.

DISABLE_COMPACT can now only turn compaction off, not back on, from settings files. Cloud routines no longer stay locked after their owner leaves and rejoins the organization.

Why it mattersHooks that counted subagent stops were overcounting, and runs near the context limit could halt in the night. Both now behave.

For youIf any routine hook counts SubagentStop events, recheck its numbers after updating.

compacts
Summarizes the conversation so far so it fits in the model's working memory, then carries on.
hook
A script Claude Code runs automatically at a set moment, such as before a tool call or when an agent stops.
MCP
Model Context Protocol, a standard way to plug tools and data sources into an AI agent.

Sources Claude Code v2.1.311 release

More like this Less like this

Plumbing for agents

Tools agents use well, tool calls that parse, history that is cheaper to keep, and keys scoped to the job.

Software and AI Lesson

What MCPCon learned about tools agents use well

At this year's agent and MCP conference, the talks turned to security and scale. The practical advice: label what each tool can do, keep its description small, and keep instructions short.

At AGNTCon + MCPCon in San Jose on October 22 and 23, OpenAI said it now requires every tool in its plugin directory to declare whether it only reads or can destroy. Only 56% of plugins use server instructions, and about 1,000 characters is a safe length. Three in four tool schemas are under 1 kB, but one tool reached 260 kB.

The Agentic AI Foundation grew from 3 to 10 projects, adding Anthropic's Agent Skills, OpenAI's Open Responses, Alibaba's OpenSandbox and Hugging Face's OpenEnv.

  • Label each tool read-only or destructive.
  • Keep schemas small: oversized ones get trimmed, argument descriptions first.
  • Keep server instructions near 1,000 characters.
  • Support stateless HTTP; some platforms say about 70% of traffic uses it.
Technical details

About a third of the protocol's 116 conformance scenarios now test security. The July 28 spec added stateless servers, and Claude Code gained MCP tasks for tool calls that run from minutes to days. The largest single server schema seen was 1.4 MB.

Chrome's WebMCP origin trial was presented as 2.5 to 7.5 times faster than scraping the page (vendor-reported). Cumulative MCP SDK downloads stand at 4.17 billion. The next event is in San Francisco on April 28 and 29, 2027.

Why it mattersMany agent tool failures start in the tool's own text. These are cheap fixes any MCP server author can make this week.

For youAudit your own MCP servers: add read-only or destructive annotations, trim the schemas and check instruction length. Pair it with the AT-AT paper in The lab.

MCP
Model Context Protocol, a standard way to plug tools and data sources into an AI agent.
schema
A formal description of a tool's inputs: names, types and what each one means.
stateless HTTP
A connection style where each request stands alone, so servers are easier to scale and restart.

Sources InfoQ

More like this Less like this

Software and AI Lesson

Strict decoding fixed every broken tool call

Ubicloud found that an open model served with vLLM's defaults formatted about 2 in every 25 tool calls wrong. Forcing its output to follow the tool's schema fixed them all.

92%valid tool calls, default settings

100%with constrained decoding

Tested on Kimi K2.6 with the K2VV tool-call benchmark; the cost was negligible. Vendor-reported.

Technical details

vLLM 0.28 by default scored 92%. Its own constrained decoding scored about 91% because it became opt-in. Ubicloud used XGrammar structural tags (tool name, call index, a JSON schema per tool) and applied the fix to Kimi and DeepSeek models. vLLM 0.31 added a server-side flag; the article does not give its exact name.

Why it mattersIf you send agent work to an open model you host, check that strict tool-call decoding is on before blaming the model.

vLLM
Popular open-source software for serving AI models on your own GPUs.
constrained decoding
Limiting which words the model may produce next, so its output always fits a required format.

Sources Ubicloud

More like this Less like this

Software and AI Lesson

Trimming an agent's history saved money in 0 of 42 setups

Deleting old turns from a coding agent's history looks like a saving. A study found it never cut cost once caching was priced in, because every edit makes the provider process the whole history again.

26/42cheaper at flat token prices

0/42cheaper once caching is priced in

Researchers at Simula and CISPA ran 9,400 SWE-bench tasks with two agents and 42 ways of deleting history. Agents also went back to re-read what had been deleted.

Technical details

Setup: Mini-SWE-agent and Trae agent on DeepSeek-V4-Flash served with vLLM, 100 SWE-bench Verified tasks. Deletion varied by scope (reasoning, tool output or both), granularity, amount and timing.

Deleting right away lost solved tasks in 19 of 42 setups; deleting two turns later kept them in 37 of 42, though 21 runs grew longer. Deleting tool output raised re-reads up to 7.5 times with the delay and 10.6 times without. With caching priced in, 26 of 42 setups were significantly more expensive.

Why it mattersIf your provider caches the start of each request, cutting old context mid-session breaks the cache and costs more. Measure cost per solved task with cache prices before adding a pruning step.

For youDo not add mid-session history pruning to long routines to save money; hand big jobs to subagents with fresh context and keep the main session's opening stable.

caching
The provider keeps the processed start of a request and charges much less when the next request begins the same way.
SWE-bench
A benchmark of real bug fixes from open-source projects, used to score coding agents.

Sources Vallecillos-Ruiz et al., arXiv

More like this Less like this

Software and AI Do this

Supabase tokens can now be limited to one project

A Supabase access token used to open every project its owner could reach. Scoped tokens can be limited to one project and to just the permissions an agent or script needs.

Scoped personal access tokens became generally available on September 28, and Supabase's MCP server now checks them tool by tool.

Legacy token Scoped token
Reach Every organization and project you can access Chosen organizations, or a single project
Permissions Full access No access, read-only, full, or one toggle per permission
Expiry Works until deleted Up to one year
An agent tries a write it should not It runs It fails with a Postgres permission error

Start from No access, not Read-only: the read-only preset still exposes API keys and secrets.

Technical details

Scoped tokens start with sbp_fc. A refused call returns 403 with a list of missing_permissions. In the MCP server, read-only SQL needs Database read and apply_migration needs Migrations read-write. Scope and permissions cannot be edited after creation.

Gotchas: commands that use SUPABASE_DB_PASSWORD bypass token permissions; the browser login flow of the CLI still creates full-access tokens; supabase link needs read access on Project Settings, API Keys and API Key Secrets. Old tokens keep working, marked Legacy.

Why it mattersAn agent holding a full-access token can change any project you own. A scoped token turns a mistake or a prompt injection into a permission error.

For youSwap the token in the Claude Code Supabase MCP config for one project-scoped token per SaaS. See Try this today.

personal access token
A password-like key that lets scripts and tools act on your account without logging in.
MCP server
A plug-in that gives an AI agent a set of tools, here for working with Supabase projects.

Sources Supabase

More like this Less like this

Game dev toolbox

A Godot exploit to know about, free retargeting, and the week's art tools.

Game dev News

A translation file can run code in a Godot game

A researcher showed that a community translation file loaded by a Godot 4 game can run code of the attacker's choosing. It only matters for games that load translations they did not ship themselves.

The chain, from the researcher's reading of the Godot 4.7 source:

Confirmed on a Godot 4.7 Windows export in 10 of 10 runs; the path exists in the source from 4.1.
Technical details

The author says they reported it privately on September 23 with no reply, then published early instead of on the planned December 22. Pull request 123857 restricts plural rules to pure math functions; an older one, 123130, would still allow seed(). Neither had merged by October 26. The author's own severity score is 7.8 out of 10; no equivalent path was found in Godot 3.5.

Why it mattersMod and community translation support is an attack surface. Until a fix merges, check the plural-forms header against the gettext grammar, or do not load outside .po files at all.

.po translation file
A plain-text file of translated strings in the common gettext format, often contributed by fans.
static initializer
Code in a script that runs once, automatically, as soon as the script is loaded.

Sources axeghost write-upProof of concept

More like this Less like this

Game dev Release

ActionForge

A free, in-browser alternative to Mixamo for rigging, retargeting and generating animations.

ActionForge is a free animation tool that runs in the browser: drop in a character, attach animations to it, and export for your engine, without an account or an install.

GameFromScratch covered it on October 26. It takes .glb and .fbx characters, retargets animations, layers them by body part, poses with live IK, and generates 8-direction locomotion sets in a Motion Lab. It can attach props to hands and ships a CC0 animation library.

Price
Free
Install
None, runs in the browser
Inputs
.glb, .fbx
Export presets
Unreal, Unity, Godot, Blender
Tool licence
Not stated

Why it mattersPlaceholder and prototype animation without an account or Mixamo's terms. The claims come from coverage, not a test, so check the output on your own rig first.

For youTry it for Fab plugin demo maps: an 8-direction set on the UE5 mannequin for the combat plugin showcase.

retargets
Moves an animation made for one skeleton onto a character with a different skeleton.
IK
Inverse kinematics: you place a hand or foot and the joints in between bend to follow.
locomotion sets
The walk, run and strafe animations a character needs to move in every direction.
CC0
A licence that puts work in the public domain, free for any use without credit.

Sources GameFromScratchActionForge

More like this Less like this

Game dev Release

Also new for game artists: cage painting, an MCP server in a lighting tool, live generated video

This week's art tool releases include a fix for a classic texture baking headache, a lighting tool that AI assistants can drive, and open models that turn rough blockouts into video in real time.

From CG Channel's coverage of October 25:

  • Substance 3D Painter 12.2

    Paint the bake cage directly on the mesh instead of setting one global distance, plus reworked polygon fill with loop fill.

  • Gaffer 1.7.5

    The open-source lookdev and lighting tool adds an MCP server, so AI assistants can run Python and drive headless renders.

  • EdenLive

    Open models (5B and 14B, Apache 2.0) that turn blockouts or reference video into photoreal video; over 24 fps at 720p on an RTX 5090 with the 5B model, vendor-reported.

  • ArcBrush 1.7

    Adds a PBR texture node and a trim sheet node, with exports for Unreal, Unity and Godot.

Why it mattersMCP servers are reaching art and lighting tools, so agents can start driving parts of the art pipeline the way they already drive code.

bake cage
An inflated copy of a low-detail model used to capture detail from a high-detail one into textures.
lookdev
Look development: setting up materials and lighting so assets look right before final rendering.
trim sheet
One texture holding many reusable strips of detail, shared across many models to save memory.
blockouts
Rough, untextured shapes used to plan a level or scene before real art exists.

Sources CG Channel: Substance 3D Painter 12.2CG Channel: software roundup

More like this Less like this

Learn one thing

What is a git worktree, and why does each agent get its own?

A worktree is an extra working folder attached to the same git repository, checked out on its own branch. Several agents can each edit their own copy at the same time without overwriting one another.

Think of it like thisOne recipe book, several kitchens. Each cook works from a photocopy at their own bench, and the head chef decides which changes go back into the book.

Normally a repository has one folder, so only one branch can be checked out at a time. The git worktree command adds more folders that share the same history but each hold a different branch. Creating one takes seconds and copies no history.

That is why agent setups lean on them. In the F-Zero port, every background agent got a worktree and a branch named for its change, never touched the main branch, and the orchestrator merged the results. Claude Code's latest update fixed a bug where a worktree could be removed while a background agent was still using it.

Separate folders, one history: parallel agents stay out of each other's way until a merge.
branch
A named line of changes in git that can grow separately and be merged back later.
orchestrator
The agent that hands out work to other agents and merges what they return.

Try this today

Software and AI 15 minutes For practitioners

Give your AI agent a Supabase token that can only read one project

In a quarter of an hour, an agent connected to your database can look but cannot change anything, and cannot touch your other projects.

You needA Supabase project and an agent that reaches it through the Supabase MCP server or CLI.

  1. Create a scoped token

    In the Supabase Dashboard, open your access tokens and create a new one limited to a single project.

  2. Start from No access

    Pick the No access preset, then switch on only Database read. Avoid the Read-only preset: it still includes API keys and secrets.

  3. Set an expiry and copy it

    Choose an expiry (up to one year), read the risk rating and the list of MCP tools it covers, and copy the token; it is shown once.

  4. Swap it in

    Put it in the MCP server's config or in SUPABASE_ACCESS_TOKEN, replacing the old full-access token.

  5. Retire the old one

    Delete the token marked Legacy once everything works, and make sure nothing passes SUPABASE_DB_PASSWORD, which bypasses token permissions.

You know it worked whenAsk the agent to run a read query: it works. Ask it to insert a row: it fails with a Postgres permission error. A call it is not allowed to make returns 403 with missing_permissions.

For youDo it per SaaS project, so each Claude Code session only sees the database it is working on.

MCP server
A plug-in that gives an AI agent a set of tools, here for working with Supabase projects.

Sparks

What is catching on right now, and the idea worth borrowing from each.

The lab

Research from the front line, in plain words: what it solves, how, and what you could do with it.

Software and AI PreprintarXiv preprint

Let a coding agent rewrite your tools' descriptions and error messages from logged runs.

AT-AT: Automatic Tool Optimization from Agent TracesMaya Ravichandran, Pragya Tripathi, Dharamendra Kumar, et al. (Netflix)

The problem
Tool descriptions, errors and outputs are written once by hand and never improved, even though agents' failures with them are logged.
The idea
Offline, with the agent frozen, a lightweight coding agent reads past runs and their scores, then edits tool descriptions and, where allowed, what a tool returns on error. Each edit is checked on held-out tasks, with no new live runs needed during optimization.
The result
Telecom pass rate on tau2-bench rose from 29.5% to 36.75% (significant); Airline gained 7 points (not significant), Retail none. On a production Netflix MCP server the pass rate went from 54.5% to 72.7%; on another, tool errors fell from 73 to 37 and cost dropped. One optimizer run, small task sets.
Use it
Export the failed runs on one of your MCP servers, have an agent propose rewrites of tool descriptions and error messages, and compare pass rate, tool errors and cost on tasks it did not see. Error text that says what to do next is the main lever.

For youRun it by hand on your own MCP servers and skills: feed a subagent a week of failed tool calls and let it rewrite the error messages first.

More like this Less like this

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadPaperCode

Software and AI PreprintarXiv preprint

A 1.5 ms check stops coding agents from acting on files and names that do not exist.

StateMirror: Pre-Execution Detection of Hallucinated Tool Calls in Coding AgentsShiwen Shan, Yintong Huo, Zhiying Wu, et al. (Sun Yat-sen University, Singapore Management University)

Two examples: an agent's proposed import, the predicate extracted from it, and the repository evidence showing the symbol or path is absent, flagged as an undefined symbol and an invalid import path.
Image: Shan et al., arXiv
The problem
Coding agents often propose actions that cannot work, such as importing a symbol that is not exported. In 4,398 public SWE-agent runs, recovering from one cost a median of 3 extra steps and 22,400 tokens.
The idea
Before an action runs, extract what it assumes about the repository (this path exists, this module exports that name) by parsing the code and command, then check those facts against the live repository. No language model involved.
The result
On 2,508 actions it reached 0.940 precision and 35.3% higher F1 than the best LLM judge, in 1.5 ms instead of a median 17.7 s, with zero tokens. With its feedback, agents fixed over 95% of the bad calls; on 79 live tasks run time fell 15.8% and tokens 11.8%, with no loss in tasks solved.
Use it
A pre-tool hook that checks referenced paths and imports exist, and blocks the call with a short message naming the nearest real path, captures much of the idea cheaply.

For youA PreToolUse hook doing an existence check on Bash, Read and Edit targets is an afternoon's work and costs no tokens.

More like this Less like this

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadPaper

Game dev PreprintarXiv preprint

Unity's physics becomes the answer key for playable AI video worlds, and they mostly fail it.

WMEval: Evaluating Interactive World Models Grounded on Physical SimulationsLanxiang Hu, Yixin Huang, Abhilash Shankarampeta, et al. (UC San Diego, Unity)

Overview diagram: actions go both to Unity scenes and to world models such as Project Genie and Matrix-Game; states like velocity, lighting and momentum are extracted from each and scored.
Image: Hu et al., arXiv, CC BY 4.0
The problem
Benchmarks for world models (video generators you can steer like a game) score how frames look, not whether the physics is right.
The idea
Play the same inputs in nine Unity scenes and in each world model, then compare the generated video with the simulator's true state: control, consistency, motion, fluids, cause and effect, and lighting.
The result
In a domino chain, Project Genie kept the cause and effect in 3 of 15 runs against Unity's 13. Its billiard motion was off by 26.8% against Unity's 4.9%. Matrix-Game 3.0's character consistency fell from 91.2% to 16.0% under a full camera turn.
Use it
Use a deterministic engine scene as the reference for anything generated: check AI-made gameplay or physics against a scripted ground truth before trusting it. And do not plan physics-heavy gameplay on today's world models.

More like this Less like this

  1. Idea
  2. Prototype
  3. Code released
  4. Shipping

ReadPaperCode

Also moving

This week on Toa Radar

Read the weekly summary: The week agents got fences and rulebooks
Tune mode is on in this browser. Turn it off