Open source vs frontier models

Created on July 19, 2026

2026   ·   llm   open-source   benchmarks   ai   ·   research

Privacy regulation and AI sovereignty create an incentive towards open-weights-only stacks. I wanted to explore how close is the frontier of open-source only models in comparison to closed sourced frontier models (Opus and ChatGPT). I ran a small benchmark study to measure three axes at once: accuracy, cost, and speed.

Evaluation setup

In my open open source setup models, I use the following set DeepSeek-V4-Pro, Qwen3-235B-Thinking, GLM-5.2. I used four ensembling strategies:

  • Synthesizer: Several OSS models answer concurrently, then an OSS synthesizer model reads every member’s answer and its reasoning and writes the final answer by deciding which line of reasoning to trust
  • Consensus adjudicator: This is shown the vote tally of the responses of multiple OSS models, it defaults to the majority answer and overturns it only when it can name a concrete, specific error in the consensus reasoning thereby fixing the synthesizer’s habit of talking itself into a lone wrong answer
  • Self-consistency: Each OSS model answers k=3 times and the final answer is the plurality vote, with no aggregator model reading anything (OpenRouter Fusion also used this)
  • OpenRouter Fusion: OpenRouter’s own hosted mixture-of-models router that blends multiple providers behind a single endpoint (turns out to be the cheapest option on all five benchmarks).

No closed model was used anywhere in the ensembling stack, not even as judge. For frontier models, I used Claude Opus 4.8 and ChatGPT 5.5. For the benchmarks, I used GPQA-Diamond (198 questions), AIME 2025 (30), SuperGPQA (500), HLE-math (128), SimpleQA (1000). Finally, I used OpenRouter via Inspect to run the evaluation pipelines, with live per-token pricing for cost.

Comparison with Claude Opus 4.8

I used Claude Opus 4.8 as the reference bar. ChatGPT was generally strictly better on these benchmarks but the ensembles had mixed performance comparison. Scroll through this visual to see the results in a nutshell.

The accuracy finding is more nuanced than a single number lets on: open source wins where benchmarks are dense enough for ensembling to help, and loses where the questions are genuinely at the frontier of difficulty. GPQA-Diamond and SuperGPQA fall in the first category; HLE-math and AIME fall in the second. Ensembling can also have big impact on costs with simple tasks being delegated to local models.

Cost and Speed tradeoff

Accuracy parity is only half the story. The chart below shows the full trade-off space for the models with costs in $ also included.

Open-source ensembles vs. the frontier Up & left is better — higher accuracy, lower cost.
Frontier (closed) Open-source ensemble bubble size ∝ tokens used

On cost, open source can undercut the frontier substantially. This to me was the largest insight. Fusion (by OpenRouter) was the cheapest contestant on all five benchmarks: $1.33 vs ChatGPT’s $33.15 on SimpleQA,† $2.79 vs $23.92 on HLE. The catch is that naive ensembling goes the other direction: self-consistency, which samples every model several times, was the most expensive option of all (up to $66 on SimpleQA). How you ensemble matters more than whether you ensemble.

On speed, open source loses. Flip to vs Speed. A single frontier call returns in seconds to minutes; open ensembles ran ten times longer or more (HLE: ~20 minutes per question for some strategies). Fanning out to three reasoning models and synthesizing is inherently slow. No amount of accuracy parity changes this, and it is the honest weak spot of the approach.

† SimpleQA Fusion uses OpenRouter's built-in web search, not a tool-free setup — which explains the accuracy spike to 97%.

Closing thoughts

Open source models have truly come a long way. However, frontier models certainly are worth their premium because as the envelope of capabilities increases, the bar of problems users want to tackle also raises higher.