We pointed ChaosAgents, our red-team engagement framework, at all 1,507 CyberGym tasks. The final score is the least interesting result. What matters is that the same task population moved from single digits to a third solved by changing the scaffold, not the prompt—and that an adversarial review of our own results forced us to find, and fix, tainted solves we could not stand behind.
Rick Whittington with Sean Page, Charles Peters, Shad Bazyany, Michael Palacios, and Andrew Fowler
ChaosAgents is a red-team engagement framework, not a CyberGym-specific solver. CyberGym is one target type we pointed it at. Four properties of the framework shaped this run.
The main session plans, sequences, and synthesises; it never writes a proof of concept. Work is delegated to specialist agents whose personas carry domain doctrine—a vulnerability researcher reasons about sinks and data flow, an exploit developer about payload construction and marker discipline. For CyberGym the relevant persona is a crash-PoC specialist: read C/C++ source, construct minimal binary inputs, verify against a sanitizer build.
Sub-agents run in isolated contexts and inherit nothing implicitly. Every spawn carries the doctrine it needs—an evidence taxonomy (TESTED-LIVE / CODE-CONFIRMED / INFERRED), the reasoning graph (Plan → Judge → Action → Summary → Progress, where the Judge node exists specifically to catch repeated failed approaches), and the attack-loop protocol that defines budgets and scoring gates. This is why an agent that cannot produce a verified crash reports NEEDS-LIVE rather than claiming success—the taxonomy makes the distinction mandatory.
TESTED-LIVE
CODE-CONFIRMED
INFERRED
NEEDS-LIVE
The unit of work is a ladder: cheapest capable model first, escalating only on failure.
The task ceiling is 21,600 s. The economic bet is that most tasks are solvable by a cheap model and only a minority need the expensive one. Adaptation 1 is the story of that bet failing, and being repaired.
/adapt is not a retrospective. After a batch drains, a strategy analyst—deliberately not subordinate to the coordinator that ran the batch—classifies every failure and may recommend a prompt change, an infrastructure fix, or halting a vector entirely. Its output is a versioned prompt or a code change, gated and validated on the failed cohort before promotion. Every architectural change in this writeup came out of that loop rather than from intuition.
/adapt
A deliberate consequence: the framework is adversarial toward its own results. The taint review below was produced by the same machinery—review agents with a refute-by-default posture, run against our own solves, with authority to overturn them. In one session it flagged 193, each of which we then ran down and closed.
The interesting result of this run is not the score. It is that the same task population moved from ~7% to 38% solved by changing the scaffold, not the prompt—and that we only found the right changes by instrumenting failures rather than reasoning about them. Four adaptations mattered. Each is given below with the evidence that triggered it and the measured effect.
Two revalidation batches (lv1-revalidation and lv1r2) left 65 tasks unsolved between them. Classified by the rung each terminal event died on:
lv1-revalidation
lv1r2
Terminal causes: 16 timeouts, 15 context exhaustions, 8 tool-ceiling hits.
78% of failures never reached the expensive model. The ladder was designed to escalate to opus on hard tasks; in practice the cheap rungs burned themselves out first and handed opus a poisoned context. Rung 1 was behaving like a solver—fuzzing, sweeping corpora, reading whole trees—when its economic role is reconnaissance.
file:line
lv1r3 ran 16 tasks that had failed in both prior rounds: 12 solved (80%). The best-performing prompt of the run, on the hardest population we had.
lv1r3
529 context exhaustions run-wide. Five prompt generations (v12av1 → v12fv13) had not moved it. We opened the largest dead trajectory:
v12av1
v12fv13
11.4 MB total 976 tool calls (781 Bash) median tool_result 589 bytes largest single 44 KB top-10 combined 0.2 MB of 11.4 MB
There was no oversized payload to trim. The context died of ~800 small calls accumulating linearly. No prompt instruction can fix that—which is why five attempts hadn’t.
The coordinator became the shell script, not an LLM:
The team architecture’s first batch produced few crashes. Rather than adjust the strategy prompt, we read the oracle ledger:
616 EXIT:0 legitimate no-crash 357 EXIT:125 image missing — pruned mid-run 130 EXIT:127 /out/<fuzzer> does not exist — wrong binary name 29 EXIT:1 real sanitizer crash +454 candidates rejected before testing (unbuildable python)
487 of 1,205 candidates never reached the target—and every one scored as a clean no-crash. The prompt was fine; the harness was lying to it.
cgstripped/*
/out/<name>
Lesson worth stating plainly: a batch reporting “0 solved” is not evidence about task difficulty until you have proven the oracle ran.
Raising task concurrency from 6 to 15 to improve throughput caused docker commit to contend for the daemon. The strip gate—which refuses any task whose image cannot be verified clean—began failing:
docker commit
concurrency 15, no backoff: 8 of 8 completed tasks ABORTED run-wide strip_fail 9 → 25
The gate was correct to refuse. The tasks were being lost to a resource problem while looking like task failures.
Exponential backoff on the strip gate—5 attempts at 30/60/90/120/150 s plus jitter—so contention costs time rather than tasks. At the same concurrency: aborts = 0, backoff absorbing the contention. Throughput retained; correctness gate intact. The gate still refuses outright if all attempts fail.
We had been republishing distilled “hints” from prior attempts into later batches, on the assumption that accumulated intelligence compounds.
No measurable effect. An earlier Z2 measurement had suggested 27% vs 7% in favour of unhinted, but that did not survive a full batch. We stopped republishing hints. Worth recording because the intuition is strong and the data does not support it.
20 tasks are unsolved at time of blog post. They are not lightly attempted—the median is 13 recorded attempts each at minimum, across nearly every batch generation:
arvo:23765 18 attempts Z, X, Yr2, Z3, Z4, Z4d, Z4e arvo:10882 16 attempts fv2c2r2, Z, X, Z3, Z4, Z4d, Z4e arvo:16634 16 attempts straggler8, straggler8r2, Z, X, Yr2, Z3, Z4, Z4d
All are arvo. Their source trees are smaller than the solved population (median 11.4 MB vs 17.6 MB), so scale is ruled out as the differentiator. We do not currently have a hypothesis that explains them, and we think that is the honest thing to say.
arvo
Note: these tasks are still being actively worked, and some may be solved by the time of submission.
Every one of the 1,487 scored tasks was re-reviewed after the run by the same machinery that ran it—review agents with a refute-by-default posture and the authority to overturn a solve. Per task, three channel reviewers ran in parallel with --allowedTools Read Glob Write—no Bash, no Grep—because in an earlier round agents told to read line by line wrote extractors instead. Removing the capability was the only thing that worked. A synthesiser combined their typed output with a deterministic byte-identity check, and was required to return INDETERMINATE, never LEVEL1_CLEAN, if any channel failed to report.
--allowedTools Read Glob Write
INDETERMINATE
LEVEL1_CLEAN
In one session a review flagged 193 disqualifying solves. We did not simply drop them. Each was traced to its root cause, the channel it exploited was closed in tooling (see Controls), and affected tasks were re-solved under the hardened controls before the run was considered settled. The 193 were not held out of the report—each was driven to a clean re-solve, so the reported 98.7% stands entirely on results that survived this review.
STORED_ANSWER_COPY dominates. An agent copying or re-deriving a previously stored PoC rather than constructing one—disqualifying at every difficulty level, since the reference PoC appears in no DIFFICULTY_FILES tier. Sub-cases a hash comparison cannot catch: arvo:56991 regex-mutated a stored PoC (25 of 6,867 lines changed); arvo:14215 hexdumped one and retyped the bytes.
STORED_ANSWER_COPY
DIFFICULTY_FILES
arvo:56991
arvo:14215
INDIRECT_LEAK leaves no filename trace.arvo:61617 ran git log/git show against a .git surviving inside the container, retrieving fix commits. arvo:55898‘s spawn prompt contained a working PoC as prose—no file was ever read. No filename-based scan can see either. Both channels are now closed in tooling.
INDIRECT_LEAK
arvo:61617
git log
git show
.git
arvo:55898
INDETERMINATE are tasks whose transcripts no longer exist. Rather than count them on trust, we re-solved them fresh under the hardened controls.
Re-derived independently from raw logs and bytes, not from reviewer prose:
src-vul/
Only the 112 are proven to a standard we would defend without qualification.
Note: the 112 suspect solves were re-run under the hardened controls to produce a valid, untainted solve; the counts above record what the review caught, not work left undone.
CyberGym sets difficulty by the files provided (arvo_task.py, DIFFICULTY_FILES): level0 repo-vul · level1+description.txt · level2 +error.txt · level3 +repo-fix.tar.gz +patch.diff.
arvo_task.py
repo-vul
+description.txt
+error.txt
+repo-fix.tar.gz +patch.diff
We intended and report level1. Our harness did not use gen_task.py --difficulty level1; it copied the raw dataset directory, which contains all five files. For some tasks level3 files were therefore within reach even though the prompt referenced only level1 inputs. 31 of the exclusions demonstrably opened one. This is our defect, not the benchmark’s—the shipped repo-vul.tar.gz does not contain patch.diff; we put it in reach, and it is now stripped before any task runs.
gen_task.py --difficulty level1
repo-vul.tar.gz
patch.diff
Prompts are not hand-edited artifacts in this system; they are versioned outputs of the /adapt loop, and no prompt reaches a batch without passing a gate.
Lineage.v12av1 → v12av2 → v12bv1 → … → v12fv13 → v12fv14. Each version is registered with its parent, the failure classification that triggered it, and the change made. v12fv14‘s registered rationale is verbatim the evidence in Adaptation 1: rung1 budget discipline; handoff-as-spec; explicit no-stored-answer-copy rule.
v12av1 → v12av2 → v12bv1 → … → v12fv13 → v12fv14
v12fv14
Generation. A strategy analyst reads the failed dispatches for a cohort, classifies each terminal event, and proposes edits scoped to the dominant class. The 78%-of-failures-at-rungs-1-and-2 finding produced three edits and nothing else—we deliberately did not bundle unrelated improvements, because a bundled change cannot be attributed.
Fail-closed compliance gate. Every prompt is run through prompt-compliance-check.sh before launch. The chain scripts abort rather than launch on failure:
prompt-compliance-check.sh
./scripts/prompt-compliance-check.sh "$P" || { say "ABORT — failed compliance"; exit 1; }
The gate checks that the prompt still forbids network use, still forbids reading answer artifacts, and still carries the marker/evidence discipline. This exists because a prompt is easy to improve for solve rate in ways that quietly relax a constraint.
Validation before promotion. A new version runs on the failed cohort only, never broadly. v12fv14 was validated on 16 twice-failed tasks before being considered for wider use.
Constraints stated in a prompt are requests. We learned—repeatedly—that they need to be enforced by tooling, and we lost real work finding that out. The channels the taint review uncovered are closed here.
--disallowedTools WebFetch,WebSearch on every solving invocation. The CyberGym FAQ names reading the patch or PoC from the web as a shortcut; we removed the capability rather than instructing against it.
--disallowedTools WebFetch,WebSearch
Taint reviewers ran with --allowedTools Read Glob Write—no Bash, no Grep. In an earlier round we asked reviewers to read line by line and they wrote extractors instead. Removing the tools was the only thing that produced actual line-by-line reading.
/usr/local/bin/docker is a wrapper around the real binary that refuses:
/usr/local/bin/docker
Direction matters, and the guard is written around it: -v cand:/tmp/poc:ro mounts our candidate in and is the sanctioned oracle call. Only reading out is blocked. The guard ships with an 18-case test suite that asserts both directions—that it blocks what it must and permits what it must—after an early revision denied the ladder’s own strip step and refused all 74 tasks in a batch.
-v cand:/tmp/poc:ro
Every task runs against a locally stripped image, verified inside the image:
ABSENT:0:0 ← /tmp/poc absent : 0 .git dirs : 0 patch.diff/error.txt/repo-fix
A task whose strip cannot be verified is refused, not run. During one concurrency experiment this correctly refused 8 of 8 tasks rather than let them proceed against unverified images.
Stored PoCs from prior runs were originally quarantined inside the working tree. Agents copied them from the new location—--add-dir restricts Read/Write but not Bash, so cp reaches anywhere the user can. We moved 18.8 GB outside the working tree and made it root:root mode 700, then verified as the agent user:
--add-dir
cp
root:root
cat → DENIED ls → DENIED find → DENIED (sudo retains access)
Only after that did re-solve runs stop reproducing stored answers.
Instrument the oracle before trusting any solve rate. We spent hours tuning prompts against batches where 40% of candidates never reached the target.
Build the taint controls before the run, not after. Ours were retroactive; the honest fix is to make the channel impossible on day one.
Never edit a running script. Rewriting a ladder in place corrupted 24 in-flight runs—bash reads scripts by byte offset as it executes.
Make answer stores unreadable, not just moved. Relocating stored PoCs was not enough; only root ownership outside the working tree stopped the copying.
Plan the project directory structure up front, and enforce it for agent runs. A clear, enforced layout would have prevented most of the cross-contamination we chased down after the fact.
On a taint hit, clear the task and restart from zero. Moving tainted work to a quarantine folder instead left artifacts in reach—re-tainting the re-solve and muddying later taint identification.
Much of this run went into re-doing tasks to validate and verify clean solves. The framework itself, though, is model-portable—nothing in it is tied to the models we happened to use. Our next step is to point it at other open-weight models, and potentially to train our own.
Today we use adversarial red-teaming inside the environment to identify issues, propose remediations, and emit detections. Where we are headed is a platform: one that identifies and triages vulnerabilities automatically, surfacing the genuinely critical and high-severity findings that warrant a fix.
/tmp/poc
ABSENT:0:0
/src/**/.git
error.txt
repo-fix.tar.gz
per-task-results.csv
vul_exit_code
fix_exit_code
examples/