Can a cached answer be out of date?

For at most a day, and not across a content update. Entries expire after twenty four hours and the cache is cleared when a source finishes syncing, so the window where a stale answer can be served is bounded by whichever of those comes first.

Why 0.97 rather than something looser?

Because a loose bar starts serving the answer to a similar question rather than the same one, and that failure is invisible to everybody except the visitor who got it. A high bar means the cache catches rewordings and misses everything else, which is the conservative side to err on.

Does the cache ever cross between assistants?

No. Entries are scoped per assistant, so a question asked on one site cannot be served an answer generated for another. That is a boundary in the design rather than a setting anybody can adjust.

Metrics and terms

The share of answers that cost nothing to produce

Support traffic is unusually repetitive: a small set of questions produces most of the volume, phrased slightly differently every time. Matching those by meaning rather than by text catches almost all of it, and every catch is an answer produced without spending anything. This is the tile that quietly decides your bill.

shown in the product

The tile is labelled "Served from cache" and hinted "cost nothing". The hint is literal rather than promotional: those answers did not call a model and did not add to the spend figure next to them. Refusals are excluded from the share, so an assistant cannot improve this number by declining to answer.

What it means

A cache rate is the share of answers served from something already computed rather than produced fresh. What makes this one work is that the matching is by meaning rather than by exact text, so a question worded differently from the one that filled the cache still hits it. An exact string cache would miss nearly all real support traffic, because people do not type the same sentence twice. The rate is a cost figure first and a latency figure second, and it is not a quality figure at all: a cached answer is exactly as good or as bad as the answer that was cached, which is why the rules governing what goes in and how long it stays there matter more than the percentage.

How it is actually calculated

The formula and its exclusion

The share of assistant messages answered without calling a model, with refusals excluded. Refusals also skip the model, so counting them would inflate this figure by rewarding an assistant for declining to answer, which is the opposite of what a cost saving means.

The distinction is possible because a billed answer records which model produced it and a cached or refused answer records none. Subtracting the refusals from the messages with no model leaves the cache hits.

What matches, and how closely

A new question is compared by meaning against previous questions for the same assistant and hits at 0.97 similarity. That is a deliberately high bar: it catches rewordings of the same question and does not catch a related but different one.

The comparison is scoped per assistant, so one owner's cache can never serve another owner's answer. The query has already been prepared for retrieval by that point, so a lookup costs one comparison and no extra call.

How long an entry lives

Entries live one day, and at most two hundred are kept per assistant. Both limits exist to stop a cached answer outliving the material it was based on.

The cache is also cleared when a source finishes syncing. That is the important rule: the moment you update a document, answers built on the old version stop being served. It also means the rate dips after every content change, which is correct behaviour appearing as a worse number.

What moves the rate that has nothing to do with you

Traffic concentration. An assistant whose visitors ask five questions will show a high rate; one whose visitors ask five hundred different things will show a low one, with identical material and identical settings.

Volume and timing too. With entries living a day, an assistant with steady daily traffic caches usefully and one with a weekly spike mostly does not. None of that is a judgement on the assistant.

How the number gets moved without anything improving

How the share rises without anything getting cheaper per visitor

Stop updating your material. Nothing clears the cache, entries live their full day, and the rate settles higher. Your answers get staler at exactly the same speed, which is the trade nobody makes deliberately and several people make by neglect.

The accidental inflation is subtler: a burst of traffic from one campaign asks the same question hundreds of times, the rate climbs, and it tells you about the campaign rather than about the assistant. Read this tile alongside the conversation count or it will move without meaning anything.

Note the direction the honest version runs. Publishing more material usually lowers this figure for a while, because the cache clears and because visitors start asking things they previously could not. A falling cache rate after a content push is a sign of work, not of regression.

What to look at instead, or alongside

The spend figure itself, which is the thing you actually care about and does not need a ratio to be read.
Spend divided by answered messages, tracked month to month, since that captures cache savings and model choice together.
The conversation count next to it, to tell a genuine repetition pattern from a traffic spike.
The date of your last content change, because a high rate immediately after one means the clear did not happen when you thought.

Questions

Can a cached answer be out of date?: For at most a day, and not across a content update. Entries expire after twenty four hours and the cache is cleared when a source finishes syncing, so the window where a stale answer can be served is bounded by whichever of those comes first.
Why 0.97 rather than something looser?: Because a loose bar starts serving the answer to a similar question rather than the same one, and that failure is invisible to everybody except the visitor who got it. A high bar means the cache catches rewordings and misses everything else, which is the conservative side to err on.
Does the cache ever cross between assistants?: No. Entries are scoped per assistant, so a question asked on one site cannot be served an answer generated for another. That is a boundary in the design rather than a setting anybody can adjust.

Keep reading

Try it on your own material

Upload a document or point it at your site, paste one line of HTML, then ask it something only your business could answer.

Start free See what it does