Metrics and terms
How long the assistant takes, and what that does not tell you
This is the tile most likely to be read as an achievement. It is not one. It is a property of the system, it will always be small, and it moves for reasons that have almost nothing to do with whether your assistant is any good. It is worth watching for exactly one thing, which is a change.
shown in the product
The tile is labelled "Median latency" and hinted "to full answer". It is one number over the window with no distribution behind it on that page. It is not a first response time in the support sense, because there is no queue and no person waiting to pick anything up.
What it means
Answer latency is the interval between a question arriving and a complete answer existing. It is a machine measurement, made entirely on our side, and it should not be confused with the experience of waiting, because an answer that arrives progressively feels quicker than one that appears all at once even when the interval is identical. The figure is also a mixture of populations rather than one thing: refusals return immediately without a model, cached answers return almost immediately, and genuine model calls take as long as they take. A single figure covering all three describes none of them well, which is the main thing to hold in mind when reading it.
How it is actually calculated
What is timed
The interval from the question being received to the full answer being available, in milliseconds, recorded per assistant message. The tile presents one figure over the last thirty days.
It is server side timing. Nothing in it accounts for the visitor's network, their device, or how long the panel took to appear on the page, so it is a floor on what somebody experienced rather than a measurement of it.
The mixture inside the number
Refusals skip the model entirely and return at once. Cached answers return without a model call. Both sit in the same population as answers that took a full model call, and both drag the figure down.
So this tile falls when your cache rate rises and falls when your refusal rate rises, neither of which is a speed improvement. An assistant that got worse in a specific way will show a better latency figure, which is a good reason not to put a target on it.
One figure is not a distribution
The page shows a single value, hinted as being to the full answer. There are no percentiles and no worst case on that page, and the slow tail is where a latency problem actually lives.
A stable figure with a small number of very slow answers looks identical to a stable figure with none. If you suspect a tail, the way to see it is to read the conversations rather than to look at the tile again.
How the number gets moved without anything improving
How the figure improves while the product does not
Raise the refusal threshold. More questions get the instant fallback, the fast population grows, and the average time to an answer falls. The assistant is answering less and looking quicker.
Shorten replies. A shorter answer finishes sooner, so the figure improves and the number of answers that needed the extra room and did not get it is invisible on this page.
The accidental version is a repetitive traffic week filling the cache, which does the same thing for a few days and then stops. Nobody decided anything and the tile moved anyway.
What to look at instead, or alongside
- Whether the answer was right, which is the only property of an answer that survives being slow.
- Your own timing of the widget on a real page, since the visitor's wait includes things this figure cannot see.
- The cache share and the refusal share read next to it, because they explain most of the movement.
- A read of the slowest conversations if anybody has complained, since the tail is invisible in a single figure.
Questions
- Is this the same as first response time?
- No, and the confusion is worth heading off. First response time in support measures how long a person took to pick something up. There is no queue here and nobody picks anything up, so the concept does not transfer. This is machine time to a finished answer.
- Our figure went down. Did something improve?
- Possibly, and there are three duller explanations to rule out first: the cache share went up, the refusal share went up, or the reply length setting was shortened. Check those before writing it up as a performance win.
- Should we put a target on it?
- No. A latency target is an instruction to refuse faster and answer more briefly, both of which the number rewards. If speed genuinely matters to you, watch it for changes and act on the ones you cannot explain.
Keep reading
- Deflection rate: our figure counts refusals, not resolutionsThe Deflection tile is the share of assistant messages that were not refusals. Support teams usually mean something wider by the word.
- Fallback rate: how often the assistant declines to answerThe refusal share is the other half of the Deflection tile. What triggers a fallback, what it costs, and why a low one is not automatically good.
- Coverage gaps: questions your material does not answerEvery refused question is a documented hole in your knowledge base, written by a real visitor. How the list is built and how to work through it.
- Every metric and termWhat each one means, and which of them this product actually computes.
Try it on your own material
Upload a document or point it at your site, paste one line of HTML, then ask it something only your business could answer.