ArchEval: How Well Can AI Agents Actually Design Computer Hardware?

A research team from Harvard and Georgia Tech has built a benchmark that puts AI agents through the same kind of challenges a computer architect faces: picking CPU mechanisms, tuning memory systems, configuring accelerators, and satisfying hard design constraints. The results are a useful reality check for anyone wondering how much of their hardware design work an AI tool might someday handle. The short answer is that today's best agents are solid optimization assistants when given enough feedback, but they fall apart quickly when that feedback disappears.

What Is ArchEval and What Does It Actually Test?

ArchEval is a benchmark platform containing 20 design challenges drawn from five hardware domains. It hooks up to eight real simulators and scores each agent on whether it produces a design that beats a known baseline. That makes results concrete and comparable, not just a matter of opinion.

The five domains covered are CPU core mechanisms, system architecture, memory systems, hardware accelerators, and compute-in-memory. Each challenge requires more than generating code. An agent must read a workload, pick an appropriate mechanism, drive a simulator, interpret results, and decide whether a candidate design is worth pursuing further. That last step, exercising judgment before committing to an evaluation, is where current agents struggle most.

How Does the Benchmark Structure Work?

Each challenge runs under three difficulty levels that strip away scaffolding one layer at a time. The differences between levels reveal exactly which AI capabilities are mature and which are not yet reliable.

L1 (full harness) gives the agent a working simulator loop with repeated feedback after each run. L2 (simulator-code container) provides the simulator source code but requires the agent to assemble its own workflow. L3 (agent-only) offers no runnable feedback at all before the agent submits its final design. Every run captures the full decision trajectory, so researchers can trace failures back to specific steps like workload analysis, constraint handling, or performance prediction.

What Did the Results Show?

At L1, all four tested agents met or exceeded the baseline, meaning structured simulator feedback is enough for current AI to produce genuinely useful hardware designs. That is an encouraging result for any team that wants to use an LLM as a co-pilot in a simulation-heavy workflow. The picture changes sharply at L2 and L3.

When simulator source was available but unstructured (L2), many agents failed to turn that code into productive experiments. At L3, only GPT-5.5 combined with Codex stayed above baseline, reaching 1.21x geometric mean performance and a 65% win rate against the baseline designs. The other three agents tested fell below baseline. Even the best agent had only a 15% pass rate on performance modeling tasks, meaning it could predict how a design would perform before running a simulator. That gap between simulation-assisted results and unaided prediction is the clearest signal that something fundamental is still missing.

What Does This Mean for Embedded and FPGA Engineers?

If you are building custom accelerators on an FPGA or tuning a microcontroller peripheral configuration, the practical takeaway is straightforward. Pairing an LLM agent with a real simulation or synthesis feedback loop is likely to give you useful suggestions today. Using an agent to reason about design tradeoffs without any tool-in-the-loop is a much riskier proposition.

The benchmark also highlights a capability gap that matters for anyone automating parts of a hardware design flow. Current agents are weak at discovering novel mechanisms independently and at knowing in advance whether a design idea is worth the simulation time. Until those two capabilities improve, the most productive role for AI in a hardware project is running structured experiments faster, not replacing the engineering judgment that decides which experiments to run.

What Are the Current Limits of This Research?

ArchEval covers 20 challenges across a focused set of simulators, which is a strong starting point but not exhaustive coverage of the full hardware design space. The benchmark also evaluates agents against a fixed baseline, so it measures improvement over a known reference rather than absolute design quality. The paper does not evaluate agents on physical implementation constraints like timing closure or power delivery, which are critical for real tape-outs and FPGA bitstream generation.

As AI agents gain better simulator-tool integration and more calibrated performance prediction, benchmarks like ArchEval will become the standard way to track whether those improvements translate into designs engineers can actually trust.