The agent sets up, the solver hunts, a test proves it
Researchers paired a coding agent with a tool that mathematically searches for crashing inputs. The agent did the setup work people usually skip, and every bug it reported had to come with a test that reproduces it.
ContextCrossHair is an open-source Python tool that explores a function's possible inputs to find ones that break its stated rules; it has rarely been used on real web apps because it needs hand-written rules and cannot talk to a real database.
The team from Chalmers University of Technology and the University of Edinburgh built a fixed pipeline around Claude Code. First the agent writes stand-ins for the database calls, so the solver can run without a real database. Then it writes contracts, statements of what each function expects and promises, for the riskiest functions. CrossHair then searches each function for inputs that break them, with no AI involved.
A second agent sorts every finding into real bug, bad contract or setup artifact, and each real bug must be reproduced by a unit test, and where possible by an end-to-end test on the unmodified app. Across 11 open-source Python web apps, including Zulip, NetBox and Apache Superset, with over 1.1 million lines, the setup let CrossHair run 81% of 5,406 contracted functions, against 4.5% without it.
The pipeline reported 1,050 reproduced bugs: 221 found by CrossHair and the rest flagged while the agent wrote contracts. A baseline where the same model simply read the code found more in total but missed 312 of these. CrossHair's finds were mostly crashes on odd but allowed inputs, such as empty strings, missing fields and unusual characters; the reading baseline mostly found wrong outputs.
1,050bugs, each reproduced by a test, in 11 Python web apps
Technical details
The paper is arXiv 2611.15596 by George Granberry, Wolfgang Ahrendt, Andrew D. Gordon and Moa Johansson. The harness ran Claude Opus 5.5 through Claude Code. Stubs are repaired up to two times against a checking script; up to 800 risk-ranked functions per repository get contracts; CrossHair gets 30 seconds per condition.
Of the 1,050 bugs, 920 were reproduced end to end. Framework set-up alone (without the synthesized database stand-ins) let CrossHair run 29.5% of functions. One author blind-reviewed 50 bugs from each method and judged 72% and 70% real, not a significant difference.
Limits the authors list: a single run, no maintainer review yet, one reviewer, and only Django and SQLAlchemy apps. LLM cost was estimated at no less than $2,821 for the pipeline across 11 repositories against about $712 for the reading baseline, priced rather than billed.
Why it mattersAgents that report bugs tend to report some that are not real. Requiring a reproducing test for each one, and pairing the agent with a tool that searches exhaustively, is a pattern any team can copy without this paper's code.
For youThe witness-test rule fits a review subagent on the Node and Supabase SaaS work today: no bug claim lands without a failing test. CrossHair itself only covers Python, so the transferable part is the stub-plus-contract setup and the evidence rule.
- contracts
- Statements attached to a function about what inputs it accepts and what it promises to return, which a tool can check.
- end-to-end test
- A test that drives the whole running application the way a user would, instead of calling one function directly.
- deterministic
- Gives the same answer every time for the same input, with no randomness or guessing.
Sources arXiv: Bug Finding With Agentic Symbolic ExecutionCrossHair on GitHubArtifact on figshare