Adaptive reranking with Jev and a logistic regression

Ask a cross-encoder reranker to find "B2B software companies with a subscription product" in a corpus of 650,000 Swiss company records and the top of the list comes back full of hosting providers and managed-IT shops. Each one shares words with the query, and none of them sells subscription software.

We replaced that reranker with Jev, the model we use as a judge, and the accuracy of our results went up sharply. The cost went up too: reading a document with Jev costs several times what Voyage AI's rerank-2.5-lite charges to score one. So we replaced the serial retrieve-then-rerank workflow with a loop. Jev reads documents one at a time and scores how well each fits the query. A logistic regression trains on those scores, using each document's embedding vector as features. After every round of judging, the regression scores every unread document in the corpus and picks the next batch for Jev to read.

Jev cannot afford to read 650,000 documents. The regression can score all of them in four seconds, so it acts as a cheap proxy that decides where Jev should spend its budget. Because the regression's weight vector lives in embedding space, it can be L2-normalised and sent to the vector index as a kNN query, which pulls in documents the original search never returned.

Setup

The corpus is roughly 650,000 Swiss company records from the commercial register, enriched with our own pipelines. Each record is represented by an embedding vector. Queries are natural-language descriptions of a market: "German-speaking plumbing contractors", "companies selling SaaS to other businesses", "trade associations". Given a query, the task is to find every document in the corpus that matches it, which is the same task behind building a market in Prospex.

Why the reranker failed

We had been using rerank-2.5-lite to order a pool of 6,000 candidates against the query. Two failure modes stood out.

The reranker scored word overlap. For the subscription-software query, the reranker's first page was full of documents sharing the word "software" or "subscription" while describing unrelated businesses. The companies an engineer would recognise as subscription-software companies (SonarSource, InvestGlass, SOPHiA GENETICS) ranked lower.

Rerankers struggle with conditional instructions. Voyage's reranker accepts instructions, and it still stumbled on exclusion clauses. One of our queries ended with an exclusion clause that read "keep the companies that market their own property". The reranker scored each document against the full query text, so documents matching the "keep" clause scored highest, and the exclusion filter then removed exactly those. Of 29 excluded documents we labelled, 28 were strong matches. In a second query the same failure removed 31 of 36 labelled strong matches.

We benchmarked the reranker against Jev on ten queries with 970 blind labels (strong, partial, none) from a labelling model that never saw which system had ranked a row.

Top 20 of 2,500, ten queries	Reranker	Jev
Strong matches	137	180
Unrelated documents	27	4
AUC, strong vs. none (mean)	0.94	1.00
Cost per 25,000 documents	$0.25	$1.40

Deep retrieval helps, up to a point

The original candidate pool was the query's nearest neighbours. A plumbing contractor that did not rank near the query embedding was never seen by anything downstream. We replaced that with a deeper search: 2,000 results per seed, using the query text, example documents and generated sub-queries as seeds, fused by reciprocal rank. At the same budget of 2,000 judge requests, the number of strong matches found went up several times over.

Query	Strong matches, original pool	Strong matches, deep retrieval
Plumbing contractors	41	1,397
Industrial SMEs	151	746
SaaS companies	154	251
Trade associations	106	134

Embedding similarity finds documents that sound like the query. A bus operator that qualifies as a regulated corporate entity reads like a transport company, and the search leaves it out. In half of the queries we tested, deep retrieval missed 8 to 33% of the strong matches that Jev finds when given a broader reading budget.

The adaptive loop

Each round of judging produces labels, and those labels can train a classifier that decides what to read next. The idea is old: Rocchio's relevance feedback did this in 1971 by shifting a query vector towards the documents a user marked relevant. What is new here is pool-based active learning with Jev supplying the feedback, on dense embeddings, and with the classifier choosing the next batch out of a corpus of 650,000.

1. Read the first 300 documents in the search's order.
2. Label each document read so far: a score of 0.5 or above is a positive.
3. Fit a logistic regression (C=4, class_weight="balanced") on the embedding vectors against those labels.
4. Score every unread document in the corpus with the regression, and read the 300 it ranks highest.
5. Stop once a round comes back under 5% positive, after at least two rounds and only if widening the search finds nobody new. Stop in any case at 6,000 total reads.

The model is refit from scratch every round on all the labels accumulated so far, which takes about four seconds on a single core.

Query	Requests to 95%, search order	Requests to 95%, adaptive	Stopped after	Recall
Prospecting agencies	714	397	600	95.0%
Trade associations	1,343	622	1,200	99.6%
SaaS companies	1,965	1,046	1,500	98.7%
Regional software	3,839	1,293	1,800	98.3%
Industrial SMEs	2,057	1,321	1,800	99.4%
Wholesale & distribution	3,816	1,700	2,100	98.3%
Plumbing contractors	1,881	1,478	2,100	99.9%
Regulated entities	3,827	1,806	2,400	98.7%
Go-to-market tools	3,936	1,825	2,400	98.9%
Real estate	3,657	2,119	3,000	99.9%

The regression also generalises past what the search returned. Trained on the first 600 documents the search produced, it separates positives from negatives among documents the search never reached, with an AUC between 0.82 and 1.0.

The weight vector is a query

A logistic regression on embedding features produces a weight vector that lives in the same space as the data. Normalise it to unit length and you have a direction. Send that direction to the vector index as a kNN query and it returns documents the regression would score highly, including documents the original text-based search missed entirely.

On the four queries where the text search was weakest, Jev scored 300 documents returned exclusively by the direction query, documents no previous run had seen.

Query	Positives among direction-only results
Regulated entities	41%
Wholesale & distribution	43%
Go-to-market tools	61%
Regional software	38%

The direction query for the regulated-entities query returned Swiss Re, Johnson & Johnson, power plants and a bus operator: documents that match the query without sounding like it. This is the same mechanism behind finding companies similar to a list you already have, run against a description instead of a seed company.

Results and limitations

Query	Jev requests	Wall-clock (s)	Matches found	Recall
Prospecting agencies	600	75	37	87.5%
Trade associations	1,500	138	277	97.5%
Regional software	3,304	275	1,285	96.1%
Plumbing contractors	5,403	400	3,295	99.9%
Regulated entities	6,000 (cap)	508	1,955	85.9%
Go-to-market tools	6,000 (cap)	511	2,791	89.0%

Two queries hit the budget cap while they were still finding matches at 6,000 reads, so their recall figures are a floor on what the loop would reach with a larger budget. The plumbing query found 3,295 matches after 5,403 judge calls, against 2,009 matches after 6,000 calls under the previous system, where a reranker sorted the same candidate pool. Recall is what a target-account list is judged on, and the companies a query misses are the ones nobody ever learns were missing.

Takeaway

Jev plus a logistic regression in the embedding space finds more of the relevant set, for less money, than either a better embedding or a better reranker did. The part we did not expect is the last one: the regression's weight vector is itself a query, and it reaches documents the original search would never have returned.