AI Experiments

One question at a time, measured on a public benchmark, with the exact request and response kept for every call. A result that comes out negative stays negative here.

Experiments

Can a decision model re-rank retrieval better than hybrid search?
A hybrid first stage returns 20 candidates per question. Each one is handed to a model as a single typed question, question and passage in, one number out, and the candidates are re-sorted by that number. Nothing is trained. The hosted decision model gains; both open-weights checkpoints leave the ranking worse than doing nothing.
Jev vs the floor
+0.0478
nDCG@10, paired
Laya vs the floor
−0.0241
nDCG@10, paired
Questions
323
BEIR NFCorpus (test)
Scoring calls
25,840
$0.16 spent

Decision Arena is the sibling experiment: the same two models playing highway-env, Snake and Blackjack with zero training, every request and answer on the page. This site asks the same models a retrieval question instead of a game one.

AI Experiments

Experiments

Can a decision model re-rank retrieval better than hybrid search?

Related