If you have three minutes: open MR !5, then the test MR the agents opened by themselves, MR !6. Together they are the whole product: a green suite that misses 4 bugs, and verified tests that catch them, 1 minute 42 seconds later.
When a merge request opens, a GitLab trigger starts three agents that run the full loop with no human gates: analyse, write tests, verify, retry, publish. A person approves the outcome by merging the test MR. Agents on this platform have no merge tool, and we wouldn’t give them one.
| Criterion | Our claim | Evidence |
|---|---|---|
| Technological implementation | A custom Duo flow with three agents, started by a “merge request created” trigger, running TestProof’s own AST mutation engine inside the flow’s CI job. A multi-stage pipeline with a mutation gate, an agent guard, SAST, Pages, and a real-world benchmark job. | Flow in the AI Catalog Flow source (YAML) Pipeline config Mutation engine |
| Design | One complete path from a new MR to a verified, test-only MR, with the evidence in the MR description, a summary note on the original MR, and an independent CI re-check. | MR !5 thread MR !6 description Agent session 10034500 Flow job log |
| Potential impact | Every team that trusts a green pipeline has this gap. On real open-source libraries, TestProof found 7 real test gaps, including an input that would crash humanize.metric() with no test failing. |
Benchmark dashboard Pinned projects Survivor triage |
| Innovation | Mutation testing becomes the agent’s objective feedback loop: the agent checks its own work against a judge it cannot influence, and CI checks it again. | The verify command Safety model |
| Security | A live prompt-injection test: hostile text in an MR description, a code comment and a docstring. The flow flagged and ignored all of it and changed only tests/; nothing was approved or merged. A branch that did change app code and lowered min_score was blocked by guard_test_only. |
Write-up Hostile MR !9 Agent MR !10 (tests only) Guard blocks !8 |
| Reuse | Any GitLab project can adopt TestProof with a 3-line include: of our CI/CD component, which runs on a runner image we publish to the project’s container registry. We consume our own component on MR pipelines. |
Component
Container registry
package_image job
Self-test on !11 |
| Planning | A second custom flow, TestProof Test Debt, with a mention trigger. Mentioned on !9, it ran the mutation analysis and opened one issue listing the 4 surviving mutants (the same ids and 50% score as !9’s CI job), each with its diff and a suggested test, and listed the three injection attempts as suspicious instead of following them. | Issue #2 Flow pipeline Flow in the AI Catalog Flow definition |
| Monitoring | A nightly pipeline schedule runs the full mutation analysis on main and appends one point (date, commit, score, killed, survived) to trend.json, carried forward from the previous run. First point: commit 3ba43a1, 15/15 killed. |
Scheduled pipeline
mutation_trend job
trend.json |
| Platform breadth | Besides the flow, a custom read-only agent, Ask TestProof, for Duo Chat: it explains survivors from the job logs and code and has no write tools. In a live session on !5 it explained why comparison_boundary.L23 survived (citing app/orders.py:23 and the boundary value 100), proposed the killing test without claiming it works, and declined when asked to open an MR. |
Chat session (screenshots + transcript) Agent in the AI Catalog Agent definition |
| Presentation | The video shows one real run end to end. This site replays run 3 with its real timestamps and links every artifact. | Run replay README |
Times are from the trigger (comment or MR creation) to the flow’s first note and to the test MR’s creation, read from the GitLab API. *All steps finished; the platform reported CheckpointAccessDeniedError while closing the session.
gitlab-ai-hackathon/…/showcase with LICENSE at the root.
All pipelines, including the flow’s own duo_workflow jobs and the benchmark.
A custom flow (three agents) in the AI Catalog, enabled with mention, assign, reviewer and “merge request created” triggers, a second flow that files test-debt issues (TestProof Test Debt), a read-only custom agent (Ask TestProof), plus agent-config.yml and AGENTS.md.
This site runs on Cloud Run, built by Cloud Build from site/ with deploy_cloud_run.sh.
The order-pricing app shows the loop clearly. The benchmark runs the engine on real projects in CI; the agent flow doesn’t run there, because those repositories aren’t ours to open MRs on.
Equivalent mutants are counted in the score exactly as the runner reports them, and labelled as equivalent instead of being dropped. The triage lives in bench/triage.json.
100% means every planted bug was caught, not that the code is correct. Inconclusive mutants are excluded from the score and always shown.
The engine mutates Python ASTs and reads unittest and pytest output. Other languages need their own operators.