Stealth Model Watch
Independent benchmark · August 21–22, 2026
Ox Alpha’s first real benchmark: 58.4% — not the 80% you heard
An independent developer just completed the first full-scale third-party measurement of OpenRouter’s anonymous stealth model: every one of the 113 DeepSWE software-engineering tasks, end to end, in a single 20-hour run. The result replaces a viral rumor with a documented, reproducible number — and it lands Ox Alpha level with Claude Opus 4.8.
(Model identity confirmed Aug 26: Z.AI announced it as GLM-5.3-Flash, with an official AA Intelligence Index of 57 — level with Claude Opus 4.8. See the confirmation page.)
- Resolve rate
- 58.4%
- 66 / 113 tasks solved
- 95% CI (Wilson)
- [49.2%, 67.1%]
- all 113 tasks ran — no sampling
- Comparison
- ≈ Opus 4.8
- Claude Opus 4.8 scores 59%
- Wall clock
- 20h 39m
- Aug 21 – Aug 22, end to end
- Hidden signal
- 80%
- of tasks land ≥90% of their tests
- Run cost
- $0
- stealth preview pricing
At a glance
The rumor, the number, the reference
Bar length = share of the 113 DeepSWE tasks fully solved.
The rumor traces to a 10-task comparison on X ↗ ; the measured score is from the full 113-task run ↗ .
The correction
Rumor vs. measurement
Where the 80% came from — and why it doesn't survive contact with a full run.
The “~80% pass rate” that circulated after launch traces back to a 10-task comparison posted on X ↗ — Ox Alpha ahead of Fable 5, GLM-5.3, Grok-4.6 and GPT-5.6 Sol. Ten questions is a party trick, not a benchmark. It was useful early signal, and we linked it as exactly that.
The new run is a different species of evidence. All 113 tasks of the DeepSWE suite ran
under a standard agentic harness (pier +
mini-swe-agent, Docker backend, stock agent
configuration), with the harness itself probed and validated before the sweep started,
and every decision — backend choice, concurrency, retries, even a mid-run disk crisis —
documented in the public report. The tester’s own verdict:
“The rumored ~80% pass rate is completely incorrect.”
Reading the number
What the binary score hides
DeepSWE scoring is all-or-nothing: reward 1 requires every fail-to-pass test to pass and no regression. One assertion away from a solve still scores zero.
| Outcome | Tasks | Share | What it means |
|---|---|---|---|
| Solved (reward 1) | 66 | 58.4% | full fail-to-pass sweep, no regressions |
| Regression-only | 2 | 1.8% | feature worked — broke an unrelated passing test |
| Near-miss (≥90% f2p) | 22 | 19.5% | ten of these failed on a single test |
| Genuine failure (<90% f2p) | 23 | 20.4% | the real misses |
The mean partial score across all runs is 0.9366, and 80% of tasks landed at least 90% of their fail-to-pass tests. Sharpest illustration: on one task the model passed 48/49 new tests and 51,468/51,469 existing ones — and scored exactly the same zero as a model that did nothing.
So both numbers are true at once: 58.4% is the correct leaderboard-comparable score, and it simultaneously understates how often the model essentially solves the task.
The actionable finding
9.7% was lost to tool calls, not reasoning
The most practically useful discovery in the report — and deliberately left unmitigated, because suppressing it would have produced non-comparable numbers.
| Failure mode | Tasks | What happened |
|---|---|---|
| Reasoning / incomplete | 31 | got some or most tests, missed the rest |
| RepeatedFormatError | 11 | aborted after 3 consecutive responses with no tool call |
| AgentTimeoutError | 5 | exceeded each task’s own 90-minute budget |
Ox Alpha intermittently returns a response containing no tool call at all. Across the first 28 trials measured, 46% of episodes hit at least one such turn; when it happens three times in a row, the episode aborts. That accounts for 11 of the 47 non-solves — 9.7% of the entire benchmark — tasks abandoned before reasoning even got a chance to fail.
In the tester’s words, the headline “understates the model’s reasoning ability while correctly reporting its agentic ability.” For anyone driving the model through long agent loops, this is the spec sheet line that matters most.
Under pressure
What 20 hours of load confirmed
A benchmark sweep is also a stress test. These listing claims held up under sustained real use.
- Million-token context, used for real. Context grows every step in agentic loops; 96.1% prompt-cache hit kept ~1.1B input tokens tractable across the run.
- Reasoning cannot be turned off. Confirmed against the API: mandatory reasoning with default effort “max” — matching the [1210] error-code fingerprint already in our tokenizer evidence.
- Genuinely multi-language. Solved tasks span Python, TypeScript and Go — this is not a Python-only measurement wearing a general crown.
- $0 pricing, logged prompts. The free stealth preview held for the whole 20-hour sweep; OpenRouter applies daily request caps to zero-priced models instead.
Interpretation
What it means for the identity hunt
Our read — clearly labeled as interpretation, not evidence.
A capability profile is a fingerprint too. Ox Alpha now has a measured one: top-tier agentic coding at roughly Claude Opus 4.8’s level, million-token context actually exercised, native video input on the listing, and an unusual tool-call reliability gap that a frontier lab’s flagship agent model would rarely ship with. Whoever built this can train top-class coding models — but may not have polished the serving layer around them.
Does it pick a winner among our four theories? Not directly. No public GLM, MiMo or Hunyuan variant has published a same-harness DeepSWE score to compare against, and Ox Alpha has not yet been added to the official leaderboard. The hard identity evidence remains where it was: tokenizer batteries, error codes and video-token math pointing at the GLM family, with MiMo counter-signals keeping the race open.
What the run does change: “a frontier model built for efficient coding and sustained agentic work” is no longer marketing copy — it is now a measured claim. And on Aug 26 the identity landed too: Z.AI announced Ox Alpha as GLM-5.3-Flash.
Fine print
Method & caveats
- Harness: pier 0.3.1 + mini-swe-agent with its stock configuration, Docker backend honoring each task’s declared resources, concurrency 4, seed-pinned task order.
- Single attempt per task — with intermittent format failures, per-task outcomes carry real run-to-run variance. The 95% CI covers generalizing beyond this suite; the 58.4% itself is exact.
- Not confidential: stealth models on OpenRouter log prompts for evaluation. Reasoning effort stayed at the model default (“max”) — a low-effort run would likely trade some accuracy for fewer timeouts.
- This page links and summarizes the tester’s public report; we did not rerun the benchmark ourselves. Numbers above are quoted from it as of Aug 23, 2026.