Back to Blog
·Jan Tyl·5 min read

I took OpenRouter's automatic router apart. Then I tried to beat it

Who decides which model answers your query? I looked inside Auto Router, measured it on Czech tasks and pitted my own approach against it: answer first, then check.

I took OpenRouter's automatic router apart. Then I tried to beat it

When you send a query to an "automatic" model, someone has to decide on your behalf which model answers. A greeting only needs the cheapest one; a number theory proof does not. OpenRouter has Auto Router for this, and I have my own solution in Hyperprostor. I wanted to know how they do it, what can be learned from them, and whether I can do it better.

A router that tells you almost everything about itself

OpenRouter can attach metadata to a response describing how it made its decision. That meant I didn't have to guess, and the analysis rests on their own data from more than 2,500 calls.

Here is what I found:

  • It doesn't choose from hundreds of models. The pool for a single decision has 13 to 14 models.
  • Before answering, it asks a small decision model nine questions. Task type (30 classes), difficulty, required accuracy, output length, sensitivity to speed and more.
  • The winner is drawn by lottery. The probability of selection falls with the square of the estimated price. So the same query gets a different model every time; one of my tasks was handled by three different ones in turn.
  • The output length limit matters more than the task. The same math proof went to the cheapest class of models with a limit of up to a thousand tokens, and to the top class with a limit of four thousand.
  • It doesn't understand risk. A query from a parent whose two-year-old had swallowed a button battery was rated as moderately difficult and urgent, but not as risky. A similar query about dosing medicine for a child went to one of the cheapest models. And a 58-year-old who asked whether, on his bank advisor's recommendation, he should put his lifetime savings of 3.2 million into a single fund all at once was handled by a cheaper model as ordinary conversation.
  • It rates Czech the same as English. That's good news.

A parent with a child at a post office counter receives the smallest ticket, a red thread leads to a lit doorway

How I measured

I used four kinds of tests:

SetScopePurpose
Czech calibration tasks190 tasks in three setswhere the router sends ordinary, hard and risky queries
Tasks with a verifiable result26 tasks, repeatedwho answers correctly, at what cost and how fast
Checker test439 answers, 89 of them wronghow reliably a wrong answer can be spotted
Conversation scenariossatisfied user, repeated complaint, change of topichow the router behaves in a longer conversation

I wrote the tasks with a verifiable result so that a program, not a judge, computes the correct answer. For example:

  • scheduling seven jobs on two machines with deadlines and penalties,
  • a message queue with rate limiting and retries of failed attempts,
  • the order of printed output in asynchronous code,
  • the probability of a dice roll expressed as an exact fraction,
  • a logic puzzle with a single solution.

With tasks like these you can't "almost get" the answer. It either matches or it doesn't.

How I differ

The competing router decides in advance and blind. It picks a model, and whether it got it right, it never finds out.

In Hyperprostor I do it the other way around: the answer is produced first, and then an independent checker verifies it. The strong, expensive model only steps in where the check failed. It then solves the task again from scratch and doesn't see the first attempt, because I measured that with a wrong draft in front of it, it often takes over the error. Queries with serious consequences (health, money, law) go straight to the strong model.

In the checker test, mine caught 87 of the 89 wrong answers.

Results

First round: 16 tasks, each repeated

My routerAuto Router
Correct answers127 of 12860 of 64
Error rate0.8%6.3%
Cost per correct answer$0.035$0.037

That is roughly eight times fewer incorrect or unfinished answers at the same price.

Second round: faster setup, 26 tasks

My routerAuto Router
Correct answers78 of 7890 of 94
Cost per correct answer$0.013$0.025
Hard tasks, median response time13 s17.5 s

In the first round I was more accurate but slower. It turned out that the check isn't what slows things down (about three and a half seconds), but the choice of the first model. After swapping it, I was faster and at half the price.

Two targets: OpenRouter with arrows scattered around the edge, Hyperprostor with arrows in the center

What the numbers don't say

  • It's a small sample. One error against four is a notable difference, but not statistical proof. On thousands of tasks the ratio may come out differently.
  • The tasks are all of one kind. The test says nothing about writing texts, research or long work with tools.
  • The times in the second round are composed from individually measured steps, not from live operation.

What I take from them

OpenRouter's router is well made, and there are a few things I'm happy to borrow: it can keep a model for the whole conversation, it works with live prices, and its questions about difficulty and accuracy are stable. I'll add some of them.

I'm not adopting the lottery or the dependence on the output limit. And I consider the post-answer check the main thing that sets me apart. In my test it made my solution above all significantly more accurate, on top of cheaper and, in the second round, faster.

The whole research cost less than $30.

Glory to the winners, honor to the defeated.

Related articles