What open-weight models get wrong on analog and mixed-signal work, measured.

A design team considering an on-premises model wants one question answered: on our work, how often is it wrong, and how would we know. Published coding benchmarks do not answer it. They measure code that runs and is checked by tests, where a wrong answer throws an exception and announces itself. The published analog benchmarks score circuit questions and designs, and their results are in the second table below, with where we stand on repeating them. The failure that costs an analog team time is a different case, where nothing errors at all and a plausible wrong number goes downstream, and we found no benchmark that counts it separately.

So we built a harness that measures that case specifically, on analog and mixed-signal material, and we published its mistakes along with its results. The models we measured are not named on this page, and the section near the end says why. The published results keep the names their papers gave them.

Models change every few months and these numbers change with them. Keeping a team current on them is part of full support, the $2,500 monthly retainer, described after the second table.

The bottom line

On site, the cheapest setup we can recommend costs $42,000 up front and about $19,400 a year for a team of up to 25 engineers. Hosted behind the gate, the same team pays nothing up front and $700 to $5,750 a year.

1. Overall accuracy on our four tests, from the first table. These are the three open-weight models with the best accuracy that we can price. Row A also scored 100% on every test, but it publishes no parameter count, so its hardware cannot be costed yet. Row F's estimated median wait for a correct answer is 11 seconds, and its weak spot is arithmetic done in the reply (33%), which the script mode removes. Row B's wait is 14 seconds, on twice F's cards. Row D is right on all four tests, arithmetic included, and fits one card, but its wait is 275 seconds, so a team that wants faster answers buys more cards. AnalogCoder and the buck tasks run on these three next and can change the order.
2. ESTIMATE from the break-even model under the third table: RTX PRO 6000 cards at $16,000 or H200 NVL at $31,500, plus hosts, sized for 20 answers per engineer per working day with the slowest tenth inside five minutes and one spare set, on whichever card comes out cheaper. A year means the capital spread over three years, plus power and upkeep. Up to about 25 engineers the fleet is one set of cards and the spare, so it costs the same for 5 engineers as for 25.
3. Tiers 2 and 3 on The Offering. The gate, which I build with your team, holds back anything foundry-covered or design-identifying before a prompt leaves. Anthropic's commercial terms say it may not train models on customer content, and they treat that content as the customer's confidential information. Its API deletes inputs and outputs within 30 days by default, keeps content its safety systems flag or the law requires for longer, and offers zero data retention by agreement. Whether your foundry NDA allows any of it is your legal team's decision.
4. The three closed references in the first table scored 92%, 100% and 100% overall, and the newest frontier model we ran solved all twelve of our buck tasks with no simulator.
5. ESTIMATE from measured spend. Our hosted design tasks on 8 October 2026 cost $0.006 to $0.037 each, across three frontier models, alone and with the simulator. A datasheet-sized prompt of 20,000 tokens with a 1,000-token answer costs about $0.05 at the newest model's list price of $2 per million tokens in and $10 per million out. At the same 20 answers per engineer per working day over 230 working days, that is $28 to $230 per engineer a year.

Our measurements, in one table

Thirteen rows. Ten open-weight models that could in principle be installed on site, and three closed hosted models kept as reference and never treated as candidates, because a model whose weights are not released cannot be installed anywhere. Open-weight rows ran three replicates of every task, so 78 trials each. Reference rows ran one, so 26.

Every percentage below is correct answers over graded trials, and the column after the four tests is the share of trials that produced nothing to grade. Those two numbers have to be read together. A model that is right whenever it answers, and that fails to answer one trial in five, is a different purchase from a model that answers every time and is wrong more often. The last column is how long an engineer would wait for a correct answer on one workstation card, and it is the column that separates this field.

MEASURED from the trial records, graded under the current harness rules. The first run, on 23 September 2026, gave every column except computing in the reply. That run's reply limit of 16,384 tokens cut the open-weight rows off while they were still reasoning, so the test was run again on 27 September at 131,072 tokens, eight times the budget, and that second run is what the column shows for the open-weight rows. Each of those figures rests on three trials or fewer, G's on one, and every wrong answer in the column was silently wrong. Rows H and J had every reply cut by their hosting before an answer appeared. J's three trials were run a third time on 7 October on an endpoint that allows longer replies, and each ended in a provider error after 6,344 to 27,656 tokens of reasoning, so J stays unmeasured in this test. The totals line counts every trial of the ten open-weight rows, graded or not: computing in the reply, 17 of the 30 came back right, five silently wrong and eight cut off; writing a script, at the first run's limit, 81 of the 90 came back right. The closed references answered it inside the first run's limit. Overall accuracy is the first run. "Reading a datasheet" is the ten questions the material can settle and "knowing when to abstain" is their ten twins, so a reply that could not be read counts against whichever of the two it answered. "Unmeasured" means no trial in that test produced an answer to grade; it is not a score of zero. The nothing-to-grade column counts trials that were cut at the limit, empty, filtered by the provider, or lost to a failed request after its retries.

ESTIMATE for the last column, derived as set out under accuracy and time below. Rows B, C and E need two, seventeen and three of the cards to hold their weights. Rows H and J had their reply budget capped by their hosting, so their figures rest on one and three correct answers and read low. A and K publish no parameter count, so no rate can be derived for them.

The finding that should drive the decision

The data reduction family asks the same three questions two ways. One mode says compute the quantities and return the numbers. The other says write a Python script, and the harness runs it in a sandbox and grades what it prints. Same models, same campaign, same definitions in the prompt, same grader, same tolerances. The only difference is whether the model does the arithmetic or writes the method and something else executes it.

MEASURED from the campaign result files.

With the same models, questions and grader, writing a script beat computing the answer, even when computing had eight times the reply budget.

The compute-it mode needs a long reply. These models work through a 63-step array in their reasoning channel, and at the first run's limit of 16,384 tokens one row spent 16,382 of them on reasoning alone, so we raised the budget eightfold and ran the family again. Eight rows then produced at least one gradeable answer, and five got all three right. Three rows still had replies cut by their hosting, two of them every reply, and those trials are recorded as unmeasured rather than failed.

With eight times the budget, the compute-it mode got 17 of 30 right across the ten rows. Five of those 30 were silently wrong and eight were cut off. The script mode produced no silently wrong answers at all. Its failures announce themselves, one of them by calling a numpy function with a keyword argument it does not accept and stopping before printing anything. The compute-it mode's failures do not: a well-formed object, a plausible peak integral non-linearity outside tolerance, and nothing in the reply to say so.

The published results, and our remeasurement

Each published figure sits beside our own run of the same model. The numbered notes under the table say what each figure measures and where it comes from.

1. AnalogCoder, Lai et al., 2024, and AAAI 2025. The model writes Python that builds one of fifteen transistor-level blocks (amplifiers, inverters, current mirrors, op-amps) and runs a SPICE simulator; a checker simulates the design and feeds any failure back for further rounds. The figure is the mean first-try pass rate over tasks 1 to 15, derived from the paper's per-task table, which we read twice and checked against the totals in the benchmark's repository. In the paper every model except GPT-4o also had the benchmark's circuit library, which plays no part in these fifteen tasks. On the four op-amp tasks the two open models scored 0 to 7%.
2. Our rerun of GPT-4o, the 2024-05-13 snapshot, ran on 8 October 2026 with the paper's prompt, feedback loop and temperature of 0.5: 59 of 75 trials passed, 5 per task where the paper ran 15, a mean first-try pass rate of 78.7%. The gap to the paper is in the op-amp tasks 11, 13 and 14 and in task 4; tasks 12 and 15 failed every trial, close to the paper's 13.3% on each. The benchmark had to pass its own control first, and it did on 7 October 2026: 15 of 15 of its correct answers accepted and 4 of 4 deliberately broken ones rejected. That took pinning the simulator to ngspice 34, the version the benchmark's Python library supports; the paper does not say which version it ran. On ngspice 42 the op-amp checks reject the benchmark's own correct answers. o3 and GPT-6.1 Sol run next.
3. Buck converter design workflows, Nau, Krummenauer and Zimmermann, 2025: 269 tasks that tune, adapt or optimise a buck converter, from a textbook buck to a two-phase synchronous step-down controller, with LTspice. A task is solved when the simulated result is within 5% of its target. The figures are the model alone and with LTspice in the loop without retrieval, from the paper's Table 2. With retrieval and LTspice together, o3 reached 90.7%. The paper does not release its tasks.
4. The paper's tasks are not released, so we wrote twelve of our own: set the duty cycle, inductor, output capacitor or switching frequency of three bucks, one of them two-phase, solved when SIMPLIS puts the result within 5% of its target. With the simulator in the loop a model may run up to four simulations before it answers. Before any model ran, the 36-deck control held: every exact answer passed and every answer 12% off failed. Run on 8 October 2026, each model on OpenAI's own endpoint. Counts: GPT-4o 5 of 12 alone and 11 with the simulator, o3 8 and 12, GPT-6.1 Sol 12 and 12. Twelve tasks give wide intervals: 8 of 12 is anywhere from 39% to 86% at 95% confidence. Our tasks differ from the paper's, so compare the direction of change between the two columns; the levels do not carry over.
5. CIRCUIT, Skelic et al., 2025: 510 analog circuit questions with numerical answers, right when the final number is within 0.001 of the key. The figures are zero-shot with netlists, the paper's best setting for GPT-4o, from its Table 1. The question set is not public.
6. Ours is a question set of the same shape, written here, each answer computed two independent ways. A 120-question pilot on the paper's two OpenAI models tunes its difficulty to these figures; the full set is then frozen and run on this year's models. Gemini 1.5 Pro has no endpoint on the aggregator we use, so it is not run.

We found no published benchmark that scores a model driving SIMPLIS; ours is described under What is being measured next.

DOCUMENTED for the published figures, from each paper as read on 7 and 8 October 2026. DERIVED for the AnalogCoder means. MEASURED for ours. AMSbench (Shi et al., 2025, about 8,000 questions across eight models) is the largest published set, and it is left out of the table because its per-model figures have not been read for this page.

Where this harness differs from the published benchmarks