Accuracy is the wrong metric for an agent router

Any multi-agent system needs something that decides which agent gets a task. Most demos write a keyword match, report accuracy, and ship.

I built three routers and measured them properly. The keyword baseline scored a respectable 0.625 accuracy and is completely unusable in production, because it claimed 8 out of 8 tasks that no agent should have touched. Ask it to translate a sentence into Japanese and it confidently hands the job to the agent that reviews pull requests.

Its abstention rate is not low. It is zero. And accuracy cannot see that.

The numbers

5 agents, 8 skills, 48 held-out tasks — 40 in scope, 8 deliberately not.

Four measurements, because one is not enough:

• accuracy — correct routes over in-scope tasks; declining counts as a miss
• precision — of the in-scope tasks it claimed, how many were right
• abstention — of the out-of-scope tasks, how many it correctly refused
• balanced — the mean of accuracy and abstention

That last one exists to kill both degenerate strategies. A router that always answers scores 0 on abstention. A router that never answers scores 0 on accuracy. Averaging them means neither cheat looks good.

The error profiles are completely different

keyword      wrong agent 15 · declined a real task  0 · claimed an out-of-scope task 8
learned      wrong agent  2 · declined a real task 18 · claimed an out-of-scope task 2
embedding    wrong agent  3 · declined a real task  3 · claimed an out-of-scope task 1

Three routers, three entirely different failure modes, and a single accuracy column collapses all of it.

The keyword router never says no. Not "rarely" — it has no mechanism that fires. Its 0.625 accuracy is arithmetically true and operationally meaningless.

The learned router says no far too often. Precision 0.909 — when it commits, it is almost always right. But it declined 18 of 40 real tasks, so nearly half the work falls on the floor. Safe and useless.

The threshold mattered more than the model

Both learned routers take a min_confidence below which they decline. Sweeping that one constant moves the result further than switching model architectures does.

Embedding router, sweeping the threshold:

Read the last two columns together. Every point of precision past 0.30 is bought by dropping real work. At 0.60 the router is flawless and answers almost nothing.

My first guess for this constant was 0.35. That guess cost 17 points of accuracy and 9 of balanced score — a bigger swing than I would have got by switching from scikit-learn to PyTorch at the wrong threshold.

The honest caveat: with 48 tasks there is no separate development set, so 0.30 was chosen on the same data it is reported on. That is overfitting. The number to trust is the shape of the curve — steeply falling accuracy past the knee — not the specific optimum. In a real deployment this belongs in config, tuned on held-out traffic, and revisited whenever an agent is added.

Routing learned from capability cards

The A2A protocol idea worth stealing even if you never adopt the spec: an agent's capability declaration is data it publishes about itself, not configuration you write about it.

{
  "name": "billing-agent",
  "skills": [{
    "skill_id": "explain_invoice",
    "description": "Break down what a customer was charged for...",
    "examples": [
      "why is my invoice higher this month",
      "I was charged twice for March",
      "my card was declined, what now"
    ]
  }]
}

Those examples are not documentation. They are the training set — all three routers fit on exactly (example, agent-name) pairs flattened out of the cards. Adding a sixth agent means publishing a card, not editing a router.

Which also means the quality of your routing is downstream of how well your agents describe themselves. That is the same property that makes an MCP tool description the most consequential line in the server: a capability nobody can recognise is a capability nobody invokes.

The evaluation tasks are written deliberately unlike the card examples, because a router scored on the strings it trained on tells you nothing:

The baseline is not a straw man

Worth being explicit, because a comparison against deliberately bad code proves nothing. KeywordRouter weights each agent's terms by inverse document frequency across agents, so a word every agent uses counts for almost nothing. Without that, the agent with the longest card wins every ambiguous task, and keyword routing looks worse than it deserves.

It is a properly built baseline. It still has no way to decline, and that is the point — the flaw is structural, not a matter of effort.

What transfers

• Measure refusal as its own number. A router, a classifier or a gate that cannot decline will confidently mishandle everything outside its world, and your headline metric will not show it.
• Report a balanced score so neither "always answer" nor "never answer" can look good.
• Sweep the abstention threshold before comparing models. It is one constant and it moved more than the architecture did.
• Say where the threshold came from. Tuned on the eval set is overfitting; saying so is the difference between a result and a claim.

Everything here is in a2a-task-router — python scripts/run_eval.py --sweep reproduces the table above, and pytest -q runs 30 tests with no downloads.