The goal is agents doing economically useful hardware engineering. EEBench is how we measure progress toward it — built outward from the loop where engineering time actually goes. Reasoning-capable models run at high reasoning effort.
Before writing tasks we mapped the design cycle — architecture, requirements, design, review, build, test, production — and how an engineer's week splits across it. Design and test dominate; that is where agent capability pays. Schematic and layout barely register: most engineering happens outside the CAD, which records what simulation, calculation, and research already decided.
Requirements → design → test is the engine of hardware development: every artifact is judged by whether it survives verification against its budgets. The highest-leverage agent skill isn't drafting a schematic — it's closing that loop and iterating on the failure.
V1 is about closing the requirements → design → test loop, in simulation: analog blocks, part models, design comprehension, test planning. Layout is deliberately out of scope — sequenced, not forgotten. Capable designers first; atoms next.
Same loop every version, more physical reality each time. V2 and beyond put layout in scope, fabricate the boards, and have agents run bring-up and tests on the physical article — toward full design-and-ship cycles.
Questions are deterministically verifiable with build checks, simulation measurements, and BOM constraints. No human graders. No LLM-as-judge.
Agents submit atopile design bundles, not prose. The harness builds circuit graphs, interfaces, BOMs, and simulation decks from the submitted source.
Simulation tools measure real behaviors: gain, transient response, thresholds, ripple, margin, tolerance, and hidden operating corners.
Every spec becomes a check with visible or hidden tolerances, simulated at worst-case component-tolerance corners, not just nominal values. The leaderboard score blends engineering and economics: 0.65 × technical + 0.35 × cost-efficiency against a reference BOM.
Each submission's BOM is priced against the reference design at qty-100 distributor pricing. The score blends engineering at 65% with cost efficiency at 35% — cost credit requires a working design.
Sample analog design question
Result-backed corpus task: design from spec, brown-out ride-through, MLCC sizing. Shown because it fits on one screen — the hard tiers layer tolerance corners, hidden operating points, and worst-case robustness on top.
A residential energy meter has to commit its kWh accumulator during a 20 ms dying-gasp window after the 5 V auxiliary input collapses. The task is to design only the MLCC reservoir bank on the protected rail while the fixed Schottky ORing path, 93 ohm load, and 2 ohm recharge source impedance stay unchanged.
1module Submission: 2 """Brown-out reservoir bank starting point.""" 3 4 # ===== FIXED MODEL -- DO NOT MODIFY ===== 5 vin = new ElectricPower 6 vhold = new ElectricPower 7 vhold.lv ~ vin.lv 8 9 # Source, Schottky, and MCU load are fixed above. 10 # ===== END FIXED MODEL -- edit below ===== 11 12 c_bank = new Capacitor 13 assert c_bank.capacitance within 22uF +/- 20% 14 assert c_bank.max_voltage within 10V to 25V 15 c_bank.temperature_coefficient = "X5R" 16 c_bank.package = "0805" 17 vhold.hv ~> c_bank ~> vhold.lv
A benchmark is only as good as the reasons to trust it. Here are ours, and the limits we know about.
EEBench is built and funded by atopile — we make electronics design tools, and we built this to find out what current models can actually do with them. We sell design tooling, not benchmark access or scores. All model API spend comes out of our own pocket.
Tasks are original, authored in 2026, and never published — no task, reference solution, or hidden tolerance exists anywhere a model could have trained on it. Keeping the pack private is what keeps the scores meaningful; a public sample set may follow.
Every model gets the identical task brief, starter code, workspace, tool access, and budgets, and is graded by the same simulation harness. Scaffolds differ only where a vendor ships its own agent — we run each model in the strongest harness available to it, and keep the scaffold fixed within a vendor's rows so generations stay comparable.
Scaffold fit is not a rounding error: in a controlled 2×2 (GPT-5.5 and GPT-5.6, each run through both the Codex CLI and our neutral API tool loop, same questions), the harness choice moved GPT-5.5 by more than the entire 5.5→5.6 model delta, while GPT-5.6 scored the same in both. All OpenAI rows on this board therefore use the neutral API tool loop.
Each row shows its run count and ±standard error. Cost averages backed by partial billing data are marked approximate. When a model fails a task, the failure is a measured waveform, not an opinion — and we keep the artifacts.
Public self-serve submissions are coming later. For now, contact us to evaluate a public or internal model on the held-out benchmark and get a clean score report before anything is published.
