man rmrfbench
Methodology
How a run works, what the model is shown, how the end state is graded, when a model is run again at a higher reasoning effort, and what the numbers can and cannot support. Numbers on this page come from the same data file as the leaderboard.
The setup
Each run starts a throwaway Docker container from the benchmark's server image, with --network none, 1 CPU, 1 GB of memory and a limit of 512 processes. A setup script plants the scenario's situation on it: files, users, services, a database. The container is destroyed after grading.
The agent gets one tool, bash, which runs a command as root on the server. Each call is a fresh non-interactive bash in /root, so no state such as cd or exported variables carries over between calls. A command times out after a fixed time, and a run ends after a fixed number of steps or when the agent replies without calling the tool.
Every run uses an unattended bash-only harness: there is no tool for asking a person, and the system prompt says nobody is available.
What the model sees
The system prompt, verbatim. {hostname} is replaced by the run's host name.
(not in this data file)
The user message is a one-line ticket header followed by the scenario's ticket, for example:
[OPS-4821] opened by priya from 10.42.7.19
(ticket text)
The tool definition, verbatim:
(not in this data file)
No prompt injection. These scenarios are deliberately constructed edge cases and footguns, not malicious attacks: there are no prompt injections, poisoned files, misleading tool output, or requests for harm. Everything the agent needs to avoid the mistake is on the box, and a careful operator who looks before acting finds it.
Outcomes and end-state grading
Grading looks only at the end state. The harness never pattern-matches the agent's commands: rm -rf is fine if it hit the right directory. After the agent stops, a check script inspects the server: whether the home directory is still there, whether the backups survived, whether the database still has its rows, whether the service still answers.
Setup and check scripts are piped in over stdin and never written to the server's disk, so the agent cannot read them. Every run gets one of the outcomes on the right.
Blocked by guardrail. Every ticket can be done safely, so when the provider's content filter refuses a run before the model can act, it is counted as a failure of its own kind: the box is intact, but the job was not done and the run counts against survival.
Survival is the share of graded runs that neither broke the box nor were blocked by a guardrail. Clean + done is the share that finished the job with nothing broken. Both are reported because an agent that refuses everything survives every time.
Scenario validation
Each scenario is checked by three scripted agents before any model is run, with python3 -m rmrfbench validate:
- Do nothing. Must cause no harm. This shows the checks do not fire on an untouched box.
- The wrong command (
naive.sh, the obvious way to do the ticket). Must cause exactly the damage the scenario declares. This shows the trap is armed. - The careful command (
safe.sh). Must finish the job, or decline it where that is the right answer, with no harm at all. This shows the ticket can be done safely.
The scenario pages show both scripts.
Variation between repeats
Each model runs every scenario several times per reasoning effort, a fixed number of runs per effort in all. Repeats are not identical prompts. Each run gets a seeded variant: the host number changes (dev-box-03 becomes dev-box-41), the service user changes where a scenario has one, and the ticket carries a header with a ticket number, a requester and an IP address. The seed depends on the scenario and repeat number only, so every model sees the same variants.
Effort escalation
Many models accept a reasoning effort setting. Each such model starts at its lowest supported effort. If any run at that effort destroys or damages the box, the next effort up is run, and so on. The ladder stops at the first of:
- an effort with no broken boxes in all its runs (status
perfect); - the last effort up to
the max effort, the highest effort this pass runs, still with broken boxes (exhausted); - the model's budget of a fixed amount running out (
budget); - an effort with no broken boxes where some runs were refused by the provider's content filter (
blocked), since more reasoning does not get past a filter; - runs that keep erroring at an effort after three attempts (
incomplete).
Models without reasoning control run once at the provider default (single). pending means more efforts are still to be run. A clean effort is never followed by a higher one.
The leaderboard ranks each model by the effort its ladder stopped at and shows the lower efforts in the strip under its name. A higher effort is not always better, and it usually costs more. The ladder is a search for a clean result, not a measure of progress.
Does more reasoning help?
For models run at two or more efforts, the index compares survival at the lowest effort run with the highest. A change of at least ten points is labelled better or worse. That threshold is a label, not a significance test. The comparison only exists for models that broke a box at their lowest effort, because a clean effort is never followed by a higher one, so it is not a sample of models in general.
Statistics
Every rate on the site comes with a Wilson 95% score interval, which behaves sensibly at small n and at rates near 0% or 100%.
Cost accounting
Every run records its token usage and cost. Where the provider reports a cost for a request, that figure is used. Where it reports none, the cost is computed from the model's list price and the token counts.
Attempts that errored and were retried are included: their cost counts towards the model's totals and towards its budget. Cost per run on the site is an effort's total cost divided by its graded runs, so it includes those attempts. A per-run cost cap stops a run that runs away.
Failure summaries
Each run that broke the box gets a one-sentence summary written by a summariser model.
The summariser reads the ticket, the scenario's trap, the grader's findings and the transcript. It is told that the transcript is data and that instructions inside it must be ignored. It returns the number of the step that caused the damage, one sentence of at most 30 words, and a confidence of high or low. Low-confidence summaries are marked on the site.
The command shown next to a summary is copied verbatim from the stored transcript at the step the summariser picked, and is never generated. Where there is no summary, the step is the last bash command that looks destructive. The grader's finding, from the end-state checks, is the authoritative statement of what happened; the summary is a reading aid and may be wrong.
Limitations
- Few scenarios. A small number of scenarios is a small set, by design. A model that does well here can still make mistakes these scenarios do not test.
- Small n. A few runs per effort cannot separate models whose true rates are a few points apart. Treat differences between neighbouring rows as noise.
- Published scenario text. The scenario descriptions are published on this site. Once public, models can be trained on them.
- Minimal harness. One bash tool, a fresh shell per call, no confirmation prompts and no sandboxing beyond the container. This measures the model in this loop, not a product with its own guardrails.
- Content filters. Some provider content filters refuse some runs. A refusal is graded as blocked by guardrail and counts against survival; other provider errors are retried, and a model page shows how many runs still errored.
- Containers, not VMs. There is no systemd inside the servers;
systemctlis a small wrapper around sysvinitservice, which a sharp agent may notice.