An experiment in re-ranking search results
Someone searches 3,633 medical abstracts for
shelf life
The search found the right abstract. It put it seventh.
Search is good at finding and bad at ordering. A second, slower reader could put the best ones first. We tested four of them on 323 questions.
- 01Half-lives of serum PCB congener concentrations in environmentally exposed early adolescents.
- 02Comparative and meta-analytic insights into life extension via dietary restriction.
- 03Microbiology of fresh and restructured lamb meat: a review.
- 04Ten years of life: Is it a matter of choice?
- 05Modifying the processing and handling of frozen broccoli for increased sulforaphane formation.
- 06Longevity and diet. Myth or pragmatism?
- 07Oxidative stability and shelf-life evaluation of selected culinary oils.the right answer
- 08Extending healthy ageing: nutrient sensitive pathway and centenarian population
- 09A statistical regression model for the estimation of acrylamide concentrations in French fries for excess lifetime cancer risk assessment.
- 10Rye Bread Consumption in Early Life and Reduced Risk of Advanced Prostate Cancer
and 10 more
So add a second reader: a re-ranker.
The search is fast because it only compares words and meanings, across thousands of abstracts. A re-ranker is slower and more careful. It takes the 20 the search kept, reads the question beside each one, scores how well that abstract answers it, and puts them back in order of score.
All four re-rankers we tested work exactly like this. They differ only in how well they score. Here is one of them, Jev, on this question.
In the search's order, the right answer is at 07.
The experiment: four re-rankers, one test.
We gave four re-rankers the same job on the same questions, then graded every order they made.
The four
A closed model reached over the internet. Asked of each abstract: how relevant is it, 0 to 4?
An open model, 421M parameters, run on our own machine. Asked the same question, word for word.
The same Laya after its makers fine-tuned it for questions of this kind. Same question again.
A small model, 22M parameters, trained on real Bing searches. Here as the yardstick for the other three.
The same test for all four
- 01323 health questions
From a public test set. For each one, the abstracts that answer it were marked in advance.
- 02The search keeps 20
The same 20 abstracts per question, for every re-ranker.
- 03Each re-ranker reorders them
Only the reader changes. Nothing is tuned to these questions.
- 04We grade the order
Did the right answers move up?
How an order is graded
Only the top ten places count, and higher places are worth more. Here is “shelf life” graded three ways.
Place 1 is worth 1.00, place 7 0.33, place 10 0.29. Below the line, nothing.
With one right answer, the grade is simply the worth of its place. With several, each counts by its place, and the best possible order still scores 1.000. The measure is called .
Grade all 323 questions and average. The search's own order, with no re-ranker at all, averages 0.331. That is the number to beat.
What we found: one clear winner, and two that made it worse.
The search's own order scores 0.331 on the whole top ten. Here is where each re-ranker landed on the same 323 questions, and how that holds up graded seven other ways.
Grade it any way you like
How good is the order of the whole top ten? The headline measure. The search's own order scores 0.331.
Better than the search alone. 14% above it. Better on 145 questions, worse on 60; 118 unchanged.
Better than the search alone. 5% above it. Better on 102 questions, worse on 84; 137 unchanged.
Worse than the search alone. 7% below it. Better on 90 questions, worse on 125; 108 unchanged.
Worse than the search alone. 8% below it. Better on 86 questions, worse on 136; 101 unchanged.
Each re-ranker is compared with the search question by question, with a . With eight ways to grade, a borderline result can clear the bar by chance: the cross-encoder's gain on the whole top ten is one of those.
Jev against the yardstick
The cross-encoder is the tool this job usually gets. Graded on the whole top ten and compared question by question, Jev was better on 124 questions and the cross-encoder on 67; 132 were tied. On this measure, Jev is really ahead.
Across all eight ways of grading, Jev is ahead on six. On top result right and first right answer, they tie: Jev's lead is in the rest of the top ten, not the very first place.
The catch
Jev is the only one that runs on someone else's servers, and it is by far the slowest. Re-ranking one question means 20 calls, and made one after another they took about 6.4 seconds, against 0.15 for the cross-encoder. Whether the calls can run at once was not tested.
| Re-ranker | Seconds per question | Cost for all 323 |
|---|---|---|
| Jev | 6.40 | $0.16 |
| Cross-encoder | 0.15 | free |
| Laya | 0.80 | free |
| Laya typed | 0.79 | free |
Jev's time is a trip across the internet; the other three ran on one local graphics card. The cost is what the provider charged.
See it for yourself.
Everything above came from averages and one example. Here are all 323 questions. It opens on “shelf life”, the question from the top. Pick any other and watch each re-ranker reorder its 20.
1 answer. Tap it to highlight it.
Loading the rankings…
What this does not answer.
The numbers above are easy to read too much into. Each of these is a question the run cannot settle.
- Whether Jev is fast enough for someone waiting on a search.One question is 20 calls, about 6.4 seconds made one after another. Whether they can run at once, and how fast that would be, was not tested.
- Whether Jev beats a bigger cross-encoder.The one here is the smallest common model, 22M parameters. A team choosing for real would try a larger one.
- How much of Jev's order is really Jev's.Its scores come back with two decimals, so 1,450 of the 6,459 abstracts it scored tie with another, and a tie keeps the search's order.
- Whether training would rescue Laya.Neither Laya was trained on this task here. Their numbers are the starting point a later fine-tune would be measured against.
- Anything the search never found.On 71 of the 323 questions, none of the 20 abstracts was a right answer, so no re-ranker could help.
- Whether it holds beyond health questions.One collection of medical abstracts, one kind of question.