Claude Opus 5.5 vs a self-hosted Qwen3.8 Flash on a real production task
H. Kamkar
A full run on Claude Opus 5.5 cost at least $78.56. The same run on a self-hosted Qwen3.8 Flash cost $1.20, and Opus scored only 16 points higher. Whether that gap is worth 65 times the price depends on the task, so here are the numbers: where the open model held up, where it fell short, and what the hardware took.
What we tested
We had a production data pipeline running on Claude Opus 5.5. A domain expert reviewed its answers by hand, kept them by default and corrected the ones that were wrong. Those reviewed answers are our reference. We wanted to know whether an open-weight model that we host ourselves could match them.
We picked Qwen3.8-Flash-Next-FP8 (Qwen below) and served it with vLLM on two deeplit® Cloud setups: one server with two NVIDIA RTX PRO 6000 cards, and one with a single NVIDIA B200. Both models got the same prompt for the same 125 records. Opus ran at Extra high effort, one step below its Max setting. Qwen ran with thinking on at its highest level.
The short version:
Qwen matched the reference 77% of the time. The expert kept 93% of Opus's answers unchanged.
The widest gap is on the one question that asks for a list: Qwen matched 64% of the reference lists exactly. It left items out far more often than it made them up.
A full run cost at least $78.56 on Opus at API list prices, and about $1.20 on either GPU setup. That is about 1/65 of the Opus cost.
Opus finished in about 10 minutes with 20 agents in parallel. Qwen took about 20 minutes on the two-card server and about 11 minutes on the B200.
Because the reference starts from Opus, and the prompt was tuned on Opus, this test measures how close Qwen gets to Opus. It does not measure true accuracy for either model.
The task
The work is under NDA, so we don't say what the questions ask or what the data contains. Every score, cost and hardware figure is from the real runs.
For each record, the model answers three yes/no questions and produces one list. The reference covers between 59 and 98 records per question, 288 answers in total. Each question is graded only on the records where it applies, and a record the model did not answer counts as a miss.
We also scored a simple baseline: always giving the most common answer for each question. A model that does not clearly beat that is not adding much.
Both Qwen setups used the same container image and the same flags, with temperature 0 and up to 64,000 tokens per request. The hourly prices are deeplit® Cloud's flat rates for the two setups.
Why Flash, and why FP8
This was production work on a project budget, not a research benchmark. The open-weight model had to run on one server at an hourly rate the project could carry, so we took the Flash model in its FP8 release. FP8 stores each weight in one byte instead of the two of a 16-bit release, which halves the memory the weights need. A larger open-weight model, or the same model at 16 bits, needs more or bigger cards and a higher hourly rate.
Fitting the model on the GPUs
Qwen3.8-Flash-Next has 125 billion core parameters, a 51-billion-parameter n-gram embedding table and 4 billion for multi-token prediction, about 180 billion in total. Only about 6 billion are active for each token: it is a mixture-of-experts model, and for each token a router picks 10 of its 512 experts, plus one shared expert. That is what keeps it fast and cheap to run. The FP8 checkpoint is 172.8 GiB, and each RTX PRO 6000 has about 95 GiB of memory. On paper, the weights alone would leave only about 17 GiB across both cards for the KV cache and everything else serving needs.
It fits because the n-gram table, about 46.6 GiB, is a large lookup table that does not need to sit on the GPU, and vLLM v0.31.0 keeps it in the server's ordinary RAM by default. Our configuration had no offload setting, and the startup log showed EngramConfig(cpu_offload=True, ...). The official vLLM recipe says the same, and documents --engram-config '{"cpu_offload":true}' to set it explicitly.
Model weights
Size
Full FP8 checkpoint
172.8 GiB
On the two GPUs
126.2 GiB (63.1 GiB per card)
In system RAM (n-gram table)
about 46.6 GiB
At runtime each card used about 85.5 of its 95 GiB: 63.1 GiB of weights, 19.5 GiB of KV cache and a few GiB of working memory.
The single B200 holds the model the same way, with about 131 GiB on the card and the n-gram table in host RAM. At 64 requests at a time its KV cache was 98 to 99% full, and vLLM had to pause and recompute requests many times. Running about 40 requests at a time might be cheaper. We have not tested that.
More cards did not help. On four RTX PRO 6000 cards, each request ran at 9 tokens per second, 2.5 times slower than on two cards, at twice the hourly price. These cards have no NVLink, so the traffic between them crosses PCIe, and four cards exchange more of it per token than two. We did not profile it, but that is the likely cause.
A 180-billion-parameter model ran on two cards plus system RAM, or on a single B200, at about a cent per record.
Results
Each cell is the share of answers that match the reference. For Opus, that is the share the expert kept unchanged.
Question
Records checked
Opus 5.5 (kept after review)
Qwen, 2× RTX PRO 6000
Qwen, 1× B200 (2 runs)
Baseline (most common answer)
Yes/no 1
98
90%
74%
76 to 77%
67%
Yes/no 2
65
100%
82%
83 to 85%
65%
Yes/no 3
66
95%
86%
85 to 88%
53%
List, exact match
59
90%
64%
64 to 66%
42%
All answers
288
93%
77%
77 to 79%
58%
With one run on the two-card server and two on the B200, the GPU cannot be separated from run-to-run noise: on every question, the three runs are within 3 points of each other.
Treating the 288 answers as independent, Qwen's match rate on the two-card server is between 72% and 81% at 95% confidence. They come from 125 records, so the real range is somewhat wider.
On yes/no 3 Qwen comes closest, 86% (57 of 66 records), and with that few records the gap to Opus could be noise. On yes/no 1 Qwen scored 74%, only 7 points above always giving the most common answer.
How far each model sits above the baseline shows what it adds over guessing the most common answer (from the rounded shares above):
Question
Opus 5.5 above the baseline
Qwen above the baseline
Yes/no 1
23 points
7 points
Yes/no 2
35 points
17 points
Yes/no 3
42 points
33 points
List, exact match
48 points
22 points
All answers
35 points
19 points
Over all 288 answers, Qwen delivers a little more than half of what Opus adds over the baseline.
Qwen leaves list items out
Counting individual list items across the 59 checked records shows what happened on the two-card run:
Opus 5.5
Qwen
Reference items named
237
46
Items named that are not in the reference
45
13
Reference items missed
2
193
Share of reference items found
99%
19%
Share of named items that are in the reference
84%
78%
For Opus, the 45 extra items are the ones the expert removed, and the 2 missed items are the ones the expert added.
Of the items Qwen named, 78% are in the reference, close to the 84% of Opus's items the expert kept. Qwen just named very few. Opus went the other way: it found almost everything and named 45 items the expert removed.
Because the reference lists started from Opus's lists, an item Opus missed only shows up if the expert added it. A correct item that only Qwen found counts against Qwen unless the expert had added it too.
Missing items and invented items need different fixes. Misses call for smaller questions or a second pass. Invented items call for a check against the source. A single match rate hides which one you have.
Repeat runs give different answers
Running the same prompt on the same GPU twice did not give the same answers, even at temperature 0. vLLM batches requests together, which changes the order of floating-point sums, and answers that were close to a tie go the other way.
Across Qwen's three runs with the same prompt, the match rate per question moved 2 to 3 points:
Question
Range over 3 runs
Yes/no 1
74 to 77%
Yes/no 2
82 to 85%
Yes/no 3
85 to 88%
List, exact match
64 to 66%
With 59 to 98 records per question, differences of a few points between any two runs are noise.
Time and cost: self-hosted vs the API
Opus 5.5
Qwen, 2× RTX PRO 6000
Qwen, 1× B200
Time for 125 records
about 10 minutes
about 20 minutes
about 11 minutes
Speed per request
not measured
23 tokens per second
38 to 39 tokens per second
Whole server
not measured
about 1,000 tokens per second
about 1,750 tokens per second
Cost for 125 records
at least $78.56
about $1.20
about $1.20
Cost per record
$0.63
about $0.01
about $0.01
Each Qwen run read about 1.3 million tokens and generated about 1.2 million, and a run cost about the same on both setups. The B200 costs 1.8 times more per hour and finishes in a little over half the time, so the choice between them is about how fast you need results. Part of the B200's speed per request comes from its memory bandwidth, about 5 times that of the RTX PRO 6000 Server Edition in this server, so this is not a clean test of one card against two.
Where the Opus bill went
Opus ran inside Claude Code rather than on a metered API account, so we converted the logged tokens to Anthropic's API list prices: $4 per million input tokens, $20 per million output tokens, $5 or $8 per million tokens written to the cache (five-minute or one-hour) and $0.20 per million read from it.
Part of the bill
Tokens
Cost
Writing context to the cache
9.5 million
$48.33
Reading context from the cache
143.2 million
$28.64
Output
78,875
$1.58
Fresh input
1,872
$0.01
Total
$78.56
The run made 767 agent calls, about 6 per record, and each one wrote or reread the context the agents work from. The record agents came to $57.77 and the main session that ran them to $20.79. The output, the part that holds the answers, was $1.58. The log undercounts output for 518 of the 767 agent calls, which is why we say "at least".
So most of the cost was the agent layer, and an integration without it would likely cost less. If a direct integration cut the bill to a fifth, which we did not measure, a Qwen run would still be more than ten times cheaper. Agents multiply the tokens behind every answer, which is how a per-token bill outruns a budget; AI token cost: why the bill outruns the budget covers that pattern and the flat-rate alternative.
Running it every month
At 125 records the difference is small money. At 100,000 records a month, this run's per-record costs come to roughly $63,000 for Opus and roughly $1,000 of server time for Qwen. The Opus figure is an upper one: it carries the agent layer, and Anthropic's Batch API halves list prices for work that can wait. On the two-card server that load needs about 270 hours, so one server can carry it. Kept for this job all month at its flat hourly rate, that server costs about $2,600, about 24 times less than the Opus estimate. These are straight-line estimates, not measurements.
Our result is one task. Two outside reference points put it in context.
Artificial Analysis's Intelligence Index, which combines ten evaluations, scores Qwen3.8 Flash Next at 40 and Claude Opus 5.5 at Extra high effort at 56. That points the same way as our result.
A published comparison on pulling red flags out of SEC 8-K filings found GLM-5.2, an open-weight model, about 5 points behind Claude Opus 4.8, with Kimi K2.6 far behind. GLM-5.2 has about 750 billion parameters, far more than Flash, and the study graded against a reference another AI model wrote, so its result does not carry over directly to a Flash-size model on our task.
Limits of this test
This test favors Opus in two ways.
The reference is Opus's own output, and a reviewer looking at an existing answer is more likely to accept it than one labeling from scratch. Opus's 93% is the share the expert kept, not its accuracy.
The prompt was written and tuned on Opus. A prompt shaped around one model's habits tends to suit that model best.
Opus also ran only once, so we don't know how much its answers move between runs. Qwen's three runs show that a few points either way is normal.
We think the direction of the result would hold, especially on the lists. The size of the gap could be smaller under a blind review.
How to run this comparison on your own workload
Our numbers belong to one task, but the method transfers:
A reference from your own production output. Real records, checked by someone who knows the domain. Say whether the review was blind; ours was not.
A baseline. Score the most common answer for each question, and report every model against it.
Misses and inventions counted apart. For any list output, count what the model found and what it made up.
Three runs, not one. On a batching server the same prompt moves a few points between runs. Treat a gap of that size as noise.
Cost per record at the price you would pay. Include the orchestration: in our run, the agent layer cost far more than the answers.
What we take from it
For this task, Qwen can't replace Opus as it is. The gap is widest on complete lists, and on one yes/no question Qwen barely beats a simple baseline.
The hardware side was simpler than the model size suggests. A 180-billion-parameter model ran on two cards plus system RAM, or on a single B200, at about a cent per record. vLLM keeps the n-gram table in RAM by default.
That cost gap is large enough to justify more work on the self-hosted side. The next things we would try are splitting the list question into smaller checks, sending only the list question to a larger model, and running fewer requests at a time on the B200. We have not tested these yet.
If you are thinking about moving a workload off Opus 5.5 (Extra high effort), run the same comparison on your own data before you decide. The Qwen runs above ran on deeplit® Cloud at a flat hourly rate, so a comparison on your own records costs GPU hours, not tokens. Book a deployment call to talk it through.
Frequently asked questions
Is a self-hosted open-weight model cheaper than the Claude API?+−
For this job, yes. A run of 125 records cost about $1.20 of GPU time, against at least $78.56 at Claude API list prices. At 100,000 records a month, a server kept for the job costs about $2,600, against roughly $63,000 for Opus. Two things narrow the gap: most of the Opus bill was the agent layer, so a direct integration would cost less, and GPU time is not the whole cost, since the engineering behind a self-hosted model is usually the larger part. The cheaper model also matched the reference 77% of the time, where the expert kept 93% of Opus's answers.
Can Qwen3.8-Flash-Next-FP8 run on two RTX PRO 6000 cards?+−
In our setup, yes. About 126 GiB of the model sat on the two cards and the 46.6 GiB n-gram table sat in system RAM. We used vLLM v0.31.0, which keeps the table out of GPU memory by default. The server had 256 GB of RAM.
Is one B200 better than two RTX PRO 6000 cards?+−
Per run, they cost the same, about $1.20 for 125 records. The B200 finished in about 11 minutes instead of 20, and each request ran faster. Its KV cache was nearly full at 64 requests at a time, so fewer parallel requests may suit it better. Four RTX PRO 6000 cards were slower than two, most likely because the cards talk over PCIe without NVLink.
Would a larger open-weight model close the gap?+−
We have not tested one. One published comparison put GLM-5.2, a much larger open-weight model, about 5 points behind Claude Opus 4.8 on SEC filings, but a model that size needs far more GPU memory than Flash in FP8. We chose Flash for the project's budget; with a larger budget, a larger open-weight model is the next thing to test.
Why is Opus the reference?+−
The pipeline already ran on Opus, and a domain expert reviewed its answers, keeping them by default and correcting the wrong ones. We used that reviewed output as the reference, so the test shows how close Qwen gets to Opus. A blind review could change the size of the gap.
Why does Qwen give different answers at temperature 0?+−
vLLM processes many requests together, and the batch changes the order in which numbers are added up. Small rounding differences then tip answers that were close to a tie. In our three runs, the match rate per question moved 2 to 3 points.
Written by
H. Kamkar
Founder, deeplit®
Building private AI infrastructure for startups selling into regulated industries.
H. Kamkar
