Is a high score a promise the answer is right?

No, and treating it as one is the most common misreading. A high score says the passage is about the same thing as the question. Whether it contains the answer, whether the answer is current, and whether the model used it correctly are three further questions the score has no opinion on.

Why did an obviously relevant document not match?

Usually vocabulary. Your material says the phrase your team uses and the visitor typed the phrase customers use. The fix is to add the customer phrasing to the document, which costs a sentence and closes a class of failures rather than one instance of it.

Can I see the scores for answers that succeeded?

Not in the interface. The only scores displayed are the best match figures on refused questions. If you want a sense of the margin on answers, the practical route is to raise the threshold one step and read what starts being refused.

Metrics and terms

The number that decides whether the model is called

Every question produces a score before it produces anything else. It is a measure of how close the nearest piece of your material sits to what was asked, and it is the gate: above the threshold there is an answer, below it there is a refusal. Understanding what it does and does not represent explains most of the behaviour people find surprising.

shown in the product

Each row of "Unanswered questions" shows a "best match" percentage next to when the question was last asked. That figure is computed and displayed. No score is shown for questions that were answered, and there is no tile anywhere giving an average score, so treat the displayed figure as evidence about refusals only.

What it means

A retrieval score is a similarity figure between the meaning of a question and the meaning of a stored passage. It is not a probability that the answer will be correct, not a confidence rating from a model, and not a quality judgement about your writing. It is closeness in meaning, nothing more. A question that shares vocabulary and subject with a passage scores high whether or not that passage answers it, and a question that means exactly what a passage means in entirely different words can score lower than you would expect. The system uses the best score among the retrieved passages as its gate, so the whole behaviour of the assistant on any given question turns on a single comparison between one question and one passage.

How it is actually calculated

What is compared, and how many

Retrieval returns six passages by default, capped at roughly four thousand tokens of context, with the last three conversation turns kept for continuity. The highest scoring of those passages is what the threshold is checked against.

So the gate is decided by one passage and the answer is written from up to six. That gap explains a common complaint: an answer can be built partly from material that scored poorly, as long as one passage cleared the bar.

Where the threshold sits

The owner chooses from three settings: Cautious at 0.5, Balanced at 0.35 which is the default, and Willing at 0.22. Those are absolute cut offs on the score, not percentiles, so they behave differently on different material.

This is why the same threshold can be strict on one assistant and loose on another. Dense, specific documents produce higher scores for genuine matches; sprawling general documents produce middling scores for everything, and a middling score for everything means the bar catches nothing.

What the displayed percentage covers

The only place a score is shown is against each unanswered question, as a best match percentage. That is the score of the nearest passage for a question that was refused, which by definition sits below the threshold.

Scores for questions that were answered are not displayed anywhere. You cannot look at yesterday's good answers and see which ones squeaked over the line, which is a real limit when you are deciding whether to tighten the setting.

How the number gets moved without anything improving

How scores rise without retrieval getting better

Pad your documents with the vocabulary of every question you expect. Scores rise because similarity rises, and the passages that now win are the ones stuffed with keywords rather than the ones containing the answer. This is the oldest trick in retrieval and it still works, which is the problem.

Consolidate everything into fewer, longer documents. Long passages touch more subjects and therefore match more questions, so average scores go up and precision goes down. It also degrades every answer built on them, because a passage about six things says little about any of them.

The honest version of the same move is the opposite: split material into focused passages that each answer one thing. Scores for the right questions go up and scores for the wrong questions go down, which is what you actually wanted.

What to look at instead, or alongside

The best match column on each unanswered row, read as a decision between writing new material and rewording old material.
Whether the answer cited the passage you expected, which is a better check on retrieval than any score.
Your own test questions, asked in the widget in a visitor's words rather than your internal ones.
The threshold setting itself, changed one step and observed, since a score means little without knowing what bar it faced.

Questions

Is a high score a promise the answer is right?: No, and treating it as one is the most common misreading. A high score says the passage is about the same thing as the question. Whether it contains the answer, whether the answer is current, and whether the model used it correctly are three further questions the score has no opinion on.
Why did an obviously relevant document not match?: Usually vocabulary. Your material says the phrase your team uses and the visitor typed the phrase customers use. The fix is to add the customer phrasing to the document, which costs a sentence and closes a class of failures rather than one instance of it.
Can I see the scores for answers that succeeded?: Not in the interface. The only scores displayed are the best match figures on refused questions. If you want a sense of the margin on answers, the practical route is to raise the threshold one step and read what starts being refused.

Keep reading

Try it on your own material

Upload a document or point it at your site, paste one line of HTML, then ask it something only your business could answer.

Start free See what it does