demo-bm25-qwen-judge

run live · 2026-11-04T23:04:29+00:00 · system demo-bm25 (demo-bm25:k=3:k1=1.5:b=0.75:corpus=992deb87cf07)

18 questions · 1 errors · 0 from cache · 1071.68 s · results hash a155eac9a986f217

Metrics

metricmeanmedian p10minmax nundefined
hit_rate@1 1.000 1.000 1.000 1.000 1.000 161
recall@1 0.906 1.000 0.500 0.500 1.000 161
precision@1 1.000 1.000 1.000 1.000 1.000 161
ndcg@1 1.000 1.000 1.000 1.000 1.000 161
hit_rate@3 1.000 1.000 1.000 1.000 1.000 161
recall@3 1.000 1.000 1.000 1.000 1.000 161
precision@3 0.396 0.333 0.333 0.333 0.667 161
ndcg@3 1.000 1.000 1.000 1.000 1.000 161
hit_rate@5 1.000 1.000 1.000 1.000 1.000 161
recall@5 1.000 1.000 1.000 1.000 1.000 161
precision@5 0.238 0.200 0.200 0.200 0.400 161
ndcg@5 1.000 1.000 1.000 1.000 1.000 161
mrr 1.000 1.000 1.000 1.000 1.000 161
exact_match 0.059 0.000 0.000 0.000 1.000 170
token_f1 0.418 0.412 0.043 0.000 1.000 170
rouge_l 0.387 0.389 0.043 0.000 1.000 170
faithfulness 0.909 1.000 0.500 0.500 1.000 116
answer_relevancy 0.294 0.250 0.250 0.000 0.500 170
context_precision 0.521 0.333 0.000 0.000 1.000 161

Distributions

hit_rate@1

mean 1.000 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 16 00.51

recall@1

mean 0.906 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 3 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 13 00.51

precision@1

mean 1.000 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 16 00.51

ndcg@1

mean 1.000 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 16 00.51

hit_rate@3

mean 1.000 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 16 00.51

recall@3

mean 1.000 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 16 00.51

precision@3

mean 0.396 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 13 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 3 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 0 00.51

ndcg@3

mean 1.000 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 16 00.51

hit_rate@5

mean 1.000 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 16 00.51

recall@5

mean 1.000 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 16 00.51

precision@5

mean 0.238 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 13 0.3-0.4: 0 0.4-0.5: 3 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 0 00.51

ndcg@5

mean 1.000 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 16 00.51

mrr

mean 1.000 over 16 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 16 00.51

exact_match

mean 0.059 over 17 question(s)

0.0-0.1: 16 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 1 00.51

token_f1

mean 0.418 over 17 question(s)

0.0-0.1: 3 0.1-0.2: 1 0.2-0.3: 1 0.3-0.4: 3 0.4-0.5: 3 0.5-0.6: 3 0.6-0.7: 0 0.7-0.8: 1 0.8-0.9: 1 0.9-1.0: 1 00.51

rouge_l

mean 0.387 over 17 question(s)

0.0-0.1: 3 0.1-0.2: 3 0.2-0.3: 1 0.3-0.4: 2 0.4-0.5: 2 0.5-0.6: 3 0.6-0.7: 0 0.7-0.8: 1 0.8-0.9: 1 0.9-1.0: 1 00.51

faithfulness

mean 0.909 over 11 question(s)

0.0-0.1: 0 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 2 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 9 00.51

answer_relevancy

mean 0.294 over 17 question(s)

0.0-0.1: 1 0.1-0.2: 0 0.2-0.3: 12 0.3-0.4: 0 0.4-0.5: 0 0.5-0.6: 4 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 0 00.51

context_precision

mean 0.521 over 16 question(s)

0.0-0.1: 5 0.1-0.2: 0 0.2-0.3: 0 0.3-0.4: 4 0.4-0.5: 0 0.5-0.6: 0 0.6-0.7: 0 0.7-0.8: 0 0.8-0.9: 0 0.9-1.0: 7 00.51

Worst-performing questions by mean score

idquestion / answerscoreretrieved
dense-weakness
Where are dense retrievers weaker than BM25, and what is the common remedy?
metrics: APITimeoutError: Request timed out.
reference, metrics and judge verdicts

reference: On rare identifiers, product codes and exact phrases; combining both kinds of retrieval in a hybrid search is the common remedy.

 - dense-retrieval, hybrid-search, tf-idf
relevant: dense-retrieval, hybrid-search
out-of-scope
Who won the 1998 FIFA World Cup?
I could not find an answer in the documents.
reference, metrics and judge verdicts

reference: The documents do not cover this.

hit_rate@1=-, recall@1=-, precision@1=-, ndcg@1=-, hit_rate@3=-, recall@3=-, precision@3=-, ndcg@3=-, hit_rate@5=-, recall@5=-, precision@5=-, ndcg@5=-, mrr=-, exact_match=0.00, token_f1=0.33, rouge_l=0.17, faithfulness=-, answer_relevancy=0.00, context_precision=-

faithfulness: answer makes no factual claims
answer_relevancy: rated 1/5 - The answer does not address the question at all.
context_precision: no retrieved contexts
 0.125
postings-compression
How are postings lists compressed?
Postings lists are often compressed.
reference, metrics and judge verdicts

reference: By storing the gaps between consecutive document identifiers and encoding them with variable-byte or similar encodings.

hit_rate@1=1.00, recall@1=1.00, precision@1=1.00, ndcg@1=1.00, hit_rate@3=1.00, recall@3=1.00, precision@3=0.33, ndcg@3=1.00, hit_rate@5=1.00, recall@5=1.00, precision@5=0.20, ndcg@5=1.00, mrr=1.00, exact_match=0.00, token_f1=0.00, rouge_l=0.00, faithfulness=-, answer_relevancy=0.25, context_precision=0.33

faithfulness: answer makes no factual claims
answer_relevancy: rated 2/5 - The answer is too vague and does not provide specific details on how postings lists are compressed.
context_precision:
  • no The passage is about compressing postings lists but does not provide information on how gaps between document identifiers are stored and encoded.
  • no The passage is about a different ranking technique and does not contain information on how postings lists are compressed.
  • yes The passage explains how gaps between document identifiers are stored and encoded using variable-byte or similar encodings.
 0.673 inverted-index, hybrid-search, reranking
relevant: inverted-index
bi-encoder
Why can document embeddings be computed ahead of time in a bi-encoder?
Document embeddings can therefore be computed once, ahead of time, and only the query has to be encoded at search time.
reference, metrics and judge verdicts

reference: Because the query and the document are encoded independently.

hit_rate@1=1.00, recall@1=1.00, precision@1=1.00, ndcg@1=1.00, hit_rate@3=1.00, recall@3=1.00, precision@3=0.33, ndcg@3=1.00, hit_rate@5=1.00, recall@5=1.00, precision@5=0.20, ndcg@5=1.00, mrr=1.00, exact_match=0.00, token_f1=0.30, rouge_l=0.15, faithfulness=1.00, answer_relevancy=0.25, context_precision=0.00

faithfulness:
  • yes Document embeddings can therefore be computed once, ahead of time. - The context directly states this in point [1].
  • yes Only the query needs to be encoded at search time. - The context directly states this in point [1].
answer_relevancy: rated 2/5 - The answer mostly off-topic, it focuses on the search time encoding instead of explaining why document embeddings can be computed ahead of time.
context_precision:
  • no This passage does not contain information that the reference answer relies on.
  • no This passage does not contain information that the reference answer relies on.
  • no This passage does not contain information that the reference answer relies on.
 0.696 dense-retrieval, reranking, tf-idf
relevant: dense-retrieval
ann-indexes
Which approximate nearest neighbour indexes are used for dense retrieval?
Searching millions of vectors exactly is slow, so systems use approximate nearest neighbour indexes such as HNSW graphs or inverted file indexes with product quantisation.
reference, metrics and judge verdicts

reference: HNSW graphs or inverted file indexes with product quantisation.

hit_rate@1=1.00, recall@1=1.00, precision@1=1.00, ndcg@1=1.00, hit_rate@3=1.00, recall@3=1.00, precision@3=0.33, ndcg@3=1.00, hit_rate@5=1.00, recall@5=1.00, precision@5=0.20, ndcg@5=1.00, mrr=1.00, exact_match=0.00, token_f1=0.53, rouge_l=0.53, faithfulness=0.50, answer_relevancy=0.25, context_precision=0.00

faithfulness:
  • no HNSW graphs are used for dense retrieval. - The context does not mention HNSW graphs being used for dense retrieval.
  • yes inverted file indexes with product quantisation are used for dense retrieval. - The context states that inverted file indexes with product quantisation are used for approximate nearest neighbour indexing, which is a form of dense retrieval.
answer_relevancy: rated 2/5 - The answer mostly off-topic as it discusses approximate nearest neighbour indexes for dense retrieval, but does not specify which indexes are used.
context_precision:
  • no The passage does not mention any approximate nearest neighbour indexes or their usage.
  • no The passage does not mention any approximate nearest neighbour indexes or their usage.
  • no The passage does not mention any approximate nearest neighbour indexes or their usage.
 0.702 dense-retrieval, hallucination, hybrid-search
relevant: dense-retrieval
bm25-name
What does the name BM25 stand for?
It belongs to the probabilistic relevance framework developed by Stephen Robertson and Karen Sparck Jones, and the name stands for "Best Matching 25", the 25th variant tried in the Okapi information retrieval system at City University London.
reference, metrics and judge verdicts

reference: Best Matching 25, the 25th variant tried in the Okapi system.

hit_rate@1=1.00, recall@1=1.00, precision@1=1.00, ndcg@1=1.00, hit_rate@3=1.00, recall@3=1.00, precision@3=0.33, ndcg@3=1.00, hit_rate@5=1.00, recall@5=1.00, precision@5=0.20, ndcg@5=1.00, mrr=1.00, exact_match=0.00, token_f1=0.43, rouge_l=0.43, faithfulness=-, answer_relevancy=0.25, context_precision=0.00

faithfulness: answer makes no factual claims
answer_relevancy: rated 2/5 - The answer provides irrelevant information about the name's origin instead of directly addressing what BM25 stands for.
context_precision:
  • no The passage does not contain information about the meaning of BM25.
  • no The passage does not contain information about the meaning of BM25.
  • no The passage does not contain information about the meaning of BM25.
 0.702 bm25, dense-retrieval, hybrid-search
relevant: bm25
rrf-constant
What value of k is usually used in reciprocal rank fusion?
Reciprocal rank fusion is a simple way to merge ranked lists.
reference, metrics and judge verdicts

reference: A value of 60.

hit_rate@1=1.00, recall@1=1.00, precision@1=1.00, ndcg@1=1.00, hit_rate@3=1.00, recall@3=1.00, precision@3=0.33, ndcg@3=1.00, hit_rate@5=1.00, recall@5=1.00, precision@5=0.20, ndcg@5=1.00, mrr=1.00, exact_match=0.00, token_f1=0.00, rouge_l=0.00, faithfulness=-, answer_relevancy=0.25, context_precision=1.00

faithfulness: answer makes no factual claims
answer_relevancy: rated 2/5 - The answer does not specify the value of k used in reciprocal rank fusion, only explaining what it is.
context_precision:
  • yes The passage directly states that a value of 60 for k is the usual default for reciprocal rank fusion.
  • no The passage is about term frequency saturation and does not mention reciprocal rank fusion or the value of k.
  • no The passage discusses parameter k1 for term frequency saturation, not reciprocal rank fusion and the value of k.
 0.710 hybrid-search, hallucination, bm25
relevant: hybrid-search
faithfulness
How can the faithfulness of an answer be measured?
Faithfulness can be measured by splitting an answer into individual claims and checking each claim against the retrieved passages.
reference, metrics and judge verdicts

reference: Split the answer into claims and check each claim against the retrieved passages; the score is the fraction of supported claims.

hit_rate@1=1.00, recall@1=1.00, precision@1=1.00, ndcg@1=1.00, hit_rate@3=1.00, recall@3=1.00, precision@3=0.33, ndcg@3=1.00, hit_rate@5=1.00, recall@5=1.00, precision@5=0.20, ndcg@5=1.00, mrr=1.00, exact_match=0.00, token_f1=0.53, rouge_l=0.53, faithfulness=-, answer_relevancy=0.25, context_precision=0.00

faithfulness: answer makes no factual claims
answer_relevancy: rated 2/5 - The answer focuses on a method for measuring faithfulness rather than directly addressing how faithfulness of an answer can be measured.
context_precision:
  • no The passage is about measuring faithfulness by splitting an answer into claims and checking each claim against the retrieved passages, which is not the reference answer's topic.
  • no The passage discusses chunk size and search engines, which are not relevant to measuring faithfulness of an answer.
  • no The passage is about measuring faithfulness by splitting an answer into claims and checking each claim against the retrieved passages, which is not the reference answer's topic.
 0.713 hallucination, chunking, inverted-index
relevant: hallucination
vocabulary-mismatch
Which retriever suffers from vocabulary mismatch, and which kind of retrieval can match documents that use different words?
Its main weakness is vocabulary mismatch: a relevant document that uses different words from the query receives no credit for those words.
reference, metrics and judge verdicts

reference: BM25 suffers from vocabulary mismatch; dense retrieval can match documents that use different words from the query.

hit_rate@1=1.00, recall@1=0.50, precision@1=1.00, ndcg@1=1.00, hit_rate@3=1.00, recall@3=1.00, precision@3=0.67, ndcg@3=1.00, hit_rate@5=1.00, recall@5=1.00, precision@5=0.40, ndcg@5=1.00, mrr=1.00, exact_match=0.00, token_f1=0.39, rouge_l=0.39, faithfulness=1.00, answer_relevancy=0.25, context_precision=0.00

faithfulness:
  • yes Its main weakness is vocabulary mismatch: a relevant document that uses different words from the query receives no credit for those words. - The context directly states this as BM25's main weakness.
answer_relevancy: rated 2/5 - The answer mostly off-topic, only briefly mentioning vocabulary mismatch without addressing the specific retriever and kind of retrieval needed.
context_precision:
  • no This passage does not contain information relevant to the vocabulary mismatch issue or how dense retrieval can match documents with different words.
  • no This passage does not contain information relevant to the vocabulary mismatch issue or how dense retrieval can match documents with different words.
  • no This passage does not contain information relevant to the vocabulary mismatch issue or how dense retrieval can match documents with different words.
 0.715 bm25, dense-retrieval, tf-idf
relevant: bm25, dense-retrieval
bm25-k1
What does the k1 parameter control in BM25?
Term frequency saturation is controlled by the parameter k1.
reference, metrics and judge verdicts

reference: k1 controls term frequency saturation: how quickly repeated occurrences of a term stop adding to the score.

hit_rate@1=1.00, recall@1=1.00, precision@1=1.00, ndcg@1=1.00, hit_rate@3=1.00, recall@3=1.00, precision@3=0.33, ndcg@3=1.00, hit_rate@5=1.00, recall@5=1.00, precision@5=0.20, ndcg@5=1.00, mrr=1.00, exact_match=0.00, token_f1=0.35, rouge_l=0.26, faithfulness=1.00, answer_relevancy=0.25, context_precision=0.33

faithfulness:
  • yes Term frequency saturation is controlled by the parameter k1. - The context directly states that term frequency saturation is controlled by the parameter k1.
answer_relevancy: rated 2/5 - The answer does not directly address the k1 parameter in BM25 but instead talks about term frequency saturation, which is not the focus of the question.
context_precision:
  • no The passage only mentions k1 in the context of term frequency saturation and does not explain its role in BM25.
  • no The passage discusses hybrid search and dense retrievers but does not explain k1 in BM25.
  • yes The passage explains how k1 controls term frequency saturation, which is relevant to the k1 parameter in BM25.
 0.722 bm25, dense-retrieval, hybrid-search
relevant: bm25

Errors (1)