u/nurge86

Five days ago I posted the first Routerly benchmark campaign (MMLU / HumanEval / BIRD, 10 seeds, paired t-tests, semantic-intent routing vs direct Claude Sonnet 4.6). Today I published the full results write-up. Short recap for anyone who missed the first thread:

• MMLU: 83.5% vs 86.5% Sonnet, $0.00344 vs $0.01118 per run, 69% cheaper, delta not significant (p = 0.19)
• HumanEval: 95.0% vs 97.0% Sonnet Pass@1, $0.03191 vs $0.04889 per run, 35% cheaper, delta not significant (p = 0.40)
• BIRD (SQL): 44.5% vs 55.5% Sonnet, accuracy gap was significant (p = 0.02). Flagged as a backend pool failure, not a routing failure.

Full write-up with the PDF audit is here: https://blog.routerly.ai/we-ran-200-questions-per-model

0.2.0 is the first release that directly reflects what that campaign told me. Releasing in the next few days. I wanted to share what is actually changing and why, because I think the reasoning is more interesting than the changelog.

What I changed

1. SQL pool rebuild. The BIRD result was not acceptable and I did not want to hide it. The cheap tier on SQL tasks is replaced. Re-run on BIRD is running this week and will be published regardless of outcome.
2. Routing decomposition is now observable per request. In the first campaign I found that the LLM-routing policy on MMLU was spending 80% of its total cost on the routing call itself. 0.2.0 exposes this breakdown in the response metadata, so you can see routing cost vs inference cost per call instead of guessing.
3. Semantic-intent policy is the new default. The embedding-based router (text-embedding-3-small, ~$0.000002 per query) matched or beat the LLM-routing policy on every benchmark while being roughly 3 orders of magnitude cheaper to run. Routing distribution on MMLU went from 96% DeepSeek under the LLM policy to a 76/24 DeepSeek/Sonnet split under semantic-intent, which is what closed the accuracy gap. Keeping LLM routing as an option for users who want fully dynamic decisions, but the default moves.
4. Statistical rigor baked into the benchmark harness. The follow-up at 55 seeds (vs 10 in the original run) is now the standard campaign shape. 10 seeds of n=20 gave roughly 80% power to detect a ~7.7 pp gap, which is too coarse for honest claims on small deltas.

What I did not fix and why

Opus 4.6 as an always-on ceiling is still more accurate than any routed configuration on a handful of MMLU subjects (graduate-level physics, professional law). I am not pretending routing beats Opus on the hardest slice of the distribution. The pitch is that most production traffic is not that slice, and the savings on the rest pay for the few calls where you still want to hit Opus directly.

Release

0.2.0 drops in the next few days. I will post a second update with the 55-seed numbers and the rebuilt SQL pool results as soon as the campaign is complete. Expect the data to either confirm the first round or embarrass me publicly, which is the point of running it.

Full write-up of the first campaign (metrics, routing distributions, link to the PDF audit) is here: https://blog.routerly.ai/we-ran-200-questions-per-model

If you want to try Routerly on your own workload before 0.2.0 ships, everything else is at routerly.ai. Happy to answer anything in the comments, especially methodology critiques.

Five days ago I posted the first Routerly benchmark campaign (MMLU / HumanEval / BIRD, 10 seeds, paired t-tests, semantic-intent routing vs direct Claude Sonnet 4.6). Today I published the full results write-up. Short recap for anyone who missed the first thread:

• MMLU: 83.5% vs 86.5% Sonnet, $0.00344 vs $0.01118 per run, 69% cheaper, delta not significant (p = 0.19)
• HumanEval: 95.0% vs 97.0% Sonnet Pass@1, $0.03191 vs $0.04889 per run, 35% cheaper, delta not significant (p = 0.40)
• BIRD (SQL): 44.5% vs 55.5% Sonnet, accuracy gap was significant (p = 0.02). Flagged as a backend pool failure, not a routing failure.

Full write-up with the PDF audit is here: https://blog.routerly.ai/we-ran-200-questions-per-model

0.2.0 is the first release that directly reflects what that campaign told me. Releasing in the next few days. I wanted to share what is actually changing and why, because I think the reasoning is more interesting than the changelog.

What I changed

1. SQL pool rebuild. The BIRD result was not acceptable and I did not want to hide it. The cheap tier on SQL tasks is replaced. Re-run on BIRD is running this week and will be published regardless of outcome.
2. Routing decomposition is now observable per request. In the first campaign I found that the LLM-routing policy on MMLU was spending 80% of its total cost on the routing call itself. 0.2.0 exposes this breakdown in the response metadata, so you can see routing cost vs inference cost per call instead of guessing.
3. Semantic-intent policy is the new default. The embedding-based router (text-embedding-3-small, ~$0.000002 per query) matched or beat the LLM-routing policy on every benchmark while being roughly 3 orders of magnitude cheaper to run. Routing distribution on MMLU went from 96% DeepSeek under the LLM policy to a 76/24 DeepSeek/Sonnet split under semantic-intent, which is what closed the accuracy gap. Keeping LLM routing as an option for users who want fully dynamic decisions, but the default moves.
4. Statistical rigor baked into the benchmark harness. The follow-up at 55 seeds (vs 10 in the original run) is now the standard campaign shape. 10 seeds of n=20 gave roughly 80% power to detect a ~7.7 pp gap, which is too coarse for honest claims on small deltas.

What I did not fix and why

Opus 4.6 as an always-on ceiling is still more accurate than any routed configuration on a handful of MMLU subjects (graduate-level physics, professional law). I am not pretending routing beats Opus on the hardest slice of the distribution. The pitch is that most production traffic is not that slice, and the savings on the rest pay for the few calls where you still want to hit Opus directly.

Release

0.2.0 drops in the next few days. I will post a second update with the 55-seed numbers and the rebuilt SQL pool results as soon as the campaign is complete. Expect the data to either confirm the first round or embarrass me publicly, which is the point of running it.

Full write-up of the first campaign (metrics, routing distributions, link to the PDF audit) is here: https://blog.routerly.ai/we-ran-200-questions-per-model

If you want to try Routerly on your own workload before 0.2.0 ships, everything else is at routerly.ai. Happy to answer anything in the comments, especially methodology critiques.

Five days ago I posted the first Routerly benchmark campaign (MMLU / HumanEval / BIRD, 10 seeds, paired t-tests, semantic-intent routing vs direct Claude Sonnet 4.6). Today I published the full results write-up. Short recap for anyone who missed the first thread:

• MMLU: 83.5% vs 86.5% Sonnet, $0.00344 vs $0.01118 per run, 69% cheaper, delta not significant (p = 0.19)
• HumanEval: 95.0% vs 97.0% Sonnet Pass@1, $0.03191 vs $0.04889 per run, 35% cheaper, delta not significant (p = 0.40)
• BIRD (SQL): 44.5% vs 55.5% Sonnet, accuracy gap was significant (p = 0.02). Flagged as a backend pool failure, not a routing failure.

Full write-up with the PDF audit is here: https://blog.routerly.ai/we-ran-200-questions-per-model

0.2.0 is the first release that directly reflects what that campaign told me. Releasing in the next few days. I wanted to share what is actually changing and why, because I think the reasoning is more interesting than the changelog.

What I changed

1. SQL pool rebuild. The BIRD result was not acceptable and I did not want to hide it. The cheap tier on SQL tasks is replaced. Re-run on BIRD is running this week and will be published regardless of outcome.
2. Routing decomposition is now observable per request. In the first campaign I found that the LLM-routing policy on MMLU was spending 80% of its total cost on the routing call itself. 0.2.0 exposes this breakdown in the response metadata, so you can see routing cost vs inference cost per call instead of guessing.
3. Semantic-intent policy is the new default. The embedding-based router (text-embedding-3-small, ~$0.000002 per query) matched or beat the LLM-routing policy on every benchmark while being roughly 3 orders of magnitude cheaper to run. Routing distribution on MMLU went from 96% DeepSeek under the LLM policy to a 76/24 DeepSeek/Sonnet split under semantic-intent, which is what closed the accuracy gap. Keeping LLM routing as an option for users who want fully dynamic decisions, but the default moves.
4. Statistical rigor baked into the benchmark harness. The follow-up at 55 seeds (vs 10 in the original run) is now the standard campaign shape. 10 seeds of n=20 gave roughly 80% power to detect a ~7.7 pp gap, which is too coarse for honest claims on small deltas.

What I did not fix and why

Opus 4.6 as an always-on ceiling is still more accurate than any routed configuration on a handful of MMLU subjects (graduate-level physics, professional law). I am not pretending routing beats Opus on the hardest slice of the distribution. The pitch is that most production traffic is not that slice, and the savings on the rest pay for the few calls where you still want to hit Opus directly.

Release

0.2.0 drops in the next few days. I will post a second update with the 55-seed numbers and the rebuilt SQL pool results as soon as the campaign is complete. Expect the data to either confirm the first round or embarrass me publicly, which is the point of running it.

Full write-up of the first campaign (metrics, routing distributions, link to the PDF audit) is here: https://blog.routerly.ai/we-ran-200-questions-per-model

If you want to try Routerly on your own workload before 0.2.0 ships, everything else is at routerly.ai. Happy to answer anything in the comments, especially methodology critiques.

Quick update on Routerly, the LLM routing layer I am building.

Five days ago I did something I probably should not have done from a marketing standpoint: I ran a formal benchmark campaign against my own product. Today I published the full results, including the ones that were bad.

The TL;DR:

• Knowledge tasks (MMLU): 69% cheaper than Claude Sonnet 4.6 with no detectable accuracy drop
• Coding tasks (HumanEval): 35% cheaper, again no detectable accuracy drop
• SQL tasks (BIRD): honest miss, the cheap model in my SQL pool was too weak and the accuracy gap was real

Two things happened after I posted.

First, nobody was annoyed that I showed a losing result. The opposite. The BIRD section is the thing that made people trust the rest of the data. Being upfront about what did not work turned out to be the best signal of all.

Second, the feedback loop became the roadmap. 0.2.0 is not a feature list I invented at a whiteboard, it is literally the list of problems the benchmark exposed:

• Rebuild the SQL backend pool so the routing has something good to route to
• Expose routing cost vs inference cost per request (turns out LLM-based routing was eating 80% of total cost on MMLU, and I only caught it because I instrumented it)
• Make the embedding-based router the default since it matched or beat LLM routing at a fraction of the cost
• Scale the benchmark harness from 10 seeds to 55 so I can actually make statistical claims on small deltas

Lesson I am taking away from this: if you are building on top of LLM APIs, running a real benchmark on your own product is the fastest product research you can do. It tells you exactly where the ceiling is, exactly where the floor is, and exactly which piece is stopping you from closing the gap.

0.2.0 ships in the next few days. I will publish the 55-seed re-run and the rebuilt SQL numbers alongside it, win or lose.

Full write-up of the first campaign with all the numbers and the PDF audit: https://blog.routerly.ai/we-ran-200-questions-per-model

0.2.0 release notes and the 55-seed re-run will land on routerly.ai in the next few days.

Happy to talk shop with anyone building on the same stack.

Five days ago I posted the first Routerly benchmark campaign (MMLU / HumanEval / BIRD, 10 seeds, paired t-tests, semantic-intent routing vs direct Claude Sonnet 4.6). Today I published the full results write-up. Short recap for anyone who missed the first thread:

• MMLU: 83.5% vs 86.5% Sonnet, $0.00344 vs $0.01118 per run, 69% cheaper, delta not significant (p = 0.19)
• HumanEval: 95.0% vs 97.0% Sonnet Pass@1, $0.03191 vs $0.04889 per run, 35% cheaper, delta not significant (p = 0.40)
• BIRD (SQL): 44.5% vs 55.5% Sonnet, accuracy gap was significant (p = 0.02). Flagged as a backend pool failure, not a routing failure.

Full write-up with the PDF audit is here: https://blog.routerly.ai/we-ran-200-questions-per-model

0.2.0 is the first release that directly reflects what that campaign told me. Releasing in the next few days. I wanted to share what is actually changing and why, because I think the reasoning is more interesting than the changelog.

What I changed

1. SQL pool rebuild. The BIRD result was not acceptable and I did not want to hide it. The cheap tier on SQL tasks is replaced. Re-run on BIRD is running this week and will be published regardless of outcome.
2. Routing decomposition is now observable per request. In the first campaign I found that the LLM-routing policy on MMLU was spending 80% of its total cost on the routing call itself. 0.2.0 exposes this breakdown in the response metadata, so you can see routing cost vs inference cost per call instead of guessing.
3. Semantic-intent policy is the new default. The embedding-based router (text-embedding-3-small, ~$0.000002 per query) matched or beat the LLM-routing policy on every benchmark while being roughly 3 orders of magnitude cheaper to run. Routing distribution on MMLU went from 96% DeepSeek under the LLM policy to a 76/24 DeepSeek/Sonnet split under semantic-intent, which is what closed the accuracy gap. Keeping LLM routing as an option for users who want fully dynamic decisions, but the default moves.
4. Statistical rigor baked into the benchmark harness. The follow-up at 55 seeds (vs 10 in the original run) is now the standard campaign shape. 10 seeds of n=20 gave roughly 80% power to detect a ~7.7 pp gap, which is too coarse for honest claims on small deltas.

What I did not fix and why

Opus 4.6 as an always-on ceiling is still more accurate than any routed configuration on a handful of MMLU subjects (graduate-level physics, professional law). I am not pretending routing beats Opus on the hardest slice of the distribution. The pitch is that most production traffic is not that slice, and the savings on the rest pay for the few calls where you still want to hit Opus directly.

Release

0.2.0 drops in the next few days. I will post a second update with the 55-seed numbers and the rebuilt SQL pool results as soon as the campaign is complete. Expect the data to either confirm the first round or embarrass me publicly, which is the point of running it.

Full write-up of the first campaign (metrics, routing distributions, link to the PDF audit) is here: https://blog.routerly.ai/we-ran-200-questions-per-model

If you want to try Routerly on your own workload before 0.2.0 ships, everything else is at routerly.ai. Happy to answer anything in the comments, especially methodology critiques.