benchmark · research in preparation
AOBench
Agent Operations Benchmark for HPC.
The first trace-driven, role-aware, RBAC-enforced benchmark for LLM agents in HPC facilities. It asks a blunt question — can an autonomous agent actually be trusted to operate a supercomputer? — and answers it against real operational traces, real operator roles, and hard policy constraints.
- SCOPE 80 tasks · 23 snapshots · 16 systems
- DESIGN 5 roles × 10 categories × 3 tiers
- STACK Python · MCP · SLURM
// why it exists
"Helpful in a chat" is not "safe on a cluster."
General LLM-agent benchmarks reward answering questions. Running an HPC facility is a different job: the same action is correct for one operator role and a policy violation for another, and a single unauthorised command can take down a shared national resource. Evaluating agents for that world needs realistic traces, explicit roles, and a scorer that treats a policy breach as a hard failure — not a rounding error on an accuracy score.
AOBench is built to measure operational trustworthiness, not conversational fluency — so a facility can reason about whether an agent is anywhere near deployment-ready before it ever touches production.
// what it measures
Trace-driven, role-aware, policy-scored.
Realistic task matrix
Five operator roles × ten task categories × three difficulty tiers, drawn from real HPC operational scenarios rather than synthetic puzzles.
Completion-under-Policy scorer
A task only counts if it's completed within the operator's authority — any RBAC violation is a hard fail, no matter how "correct" the answer looks.
CLEAR scorecard
Every system is scored across five axes — Cost · Latency · Efficacy · Assurance · Reliability — so trade-offs are visible, not hidden behind one number.
Snapshot environments
Trace-backed snapshot bundles reconstruct real cluster state, letting many agents be evaluated against the same reproducible conditions.
// the finding
No system evaluated is deployment-ready.
Across every system measured, none clears the bar for autonomous HPC operations — the failures are concentrated exactly where they matter most: staying inside the operator's authority under pressure. That negative result is the point: it maps the gap between today's agents and a facility that could safely delegate to one.
Status. AOBench is an active research project with an associated paper in preparation. The task suite, environments, and evaluation harness are held behind a disciplined reproducibility gate; public artifact URLs and a DOI will follow once that gate is lifted.