Keep the contract in a table
If a coding agent can edit both the code and its tests, a passing test proves little. One engineer's fix: write the expected behaviour as plain tables of cases, and let the agent write only the code that reads them.
ContextMartin Fowler's site publishes long-form articles by Thoughtworks engineers; this one is by Matteo Vaccari.
Vaccari interviewed 26 developers at a company that leans heavily on coding agents. Many had stopped trusting their tests, because the agent rewrote the tests along with the code whenever it changed something. His answer: when AI writes most of the code, you are its customer, and customer tests, not developer tests, become the contract between you.
So move that contract out of test code and into test data: a table with one row per case (input, expected output, expected error), plus a small runner that reads the table and calls the real system. You draft a sample of the format, the agent writes the runner, you check the runner against production code, and both of you add rows. Tables are quick to read, which makes reviewing them realistic.
The payoff is a simple rule about what may change. Refactoring touches only production code; adding a feature touches code and data; improving coverage touches only data. A diff that breaks the rule is a signal to stop and look.
Technical details
The article shows a CSV greeting table (columns test_name, resource, name, greeting, error) and an access-control table for an admin dashboard (role, session state, resource, expected result, logged). It ties the idea to older practice: XP acceptance tests, Fit, Go's table-driven tests and approval testing, which AI now makes cheap to set up because the custom runner costs minutes.
The author cites outside reports that agent-written code has raised defects (Faros AI, DX, a CloudBees survey of 213 tech leaders); those figures are cited, not measured by him. He lists limits: it suits production systems rather than prototypes, an external format only pays if people review the tables, and it does not replace exploratory testing.
Why it mattersAn agent that can rewrite its own tests can make any change look green. Separating the expected behaviour from the code that checks it gives you something stable to review.
For youPlugin damage, inventory stacking and interaction rules are natural tables; keep them in a data folder the agent may read but not edit, and review only the rows.
- customer tests
- Tests written from the user's point of view: given this input, the system must answer this, regardless of how the code inside is arranged.
- runner
- A small program that reads each row of the table, feeds it to the software, and compares the answer with the expected one.
- refactor
- Changing how code is organised without changing what it does.
Sources martinfowler.com: Tests as data