Seven things a model should never be trusted to notice

An instruction is text the model reads alongside everything else and weighs against the rest of the context. It usually wins. It can lose to a long conversation, an unusual phrasing, a retrieved passage that pulls hard in another direction, or a visitor who frames the question in a way the instruction did not anticipate.

A rule runs before the model does. It matches on the message, and if it matches, the outcome is decided: this conversation goes to a person, or produces a fixed reply, and the knowledge base is never consulted. The behaviour does not vary with phrasing, temperature, conversation length or the day.

The question to ask about any escalation case is simple. If this behaves correctly nine times and incorrectly once, is the once acceptable? For a question about opening hours, yes. For the seven cases below, no. That is the entire criterion, and it is worth applying explicitly rather than by feel.

When somebody types a request to speak to a human being, they have told you the automated path has ended. There is nothing left for the model to interpret, and any interpretation it performs can only make things worse: a retrieved passage about your contact page, an offer to try rephrasing, a cheerful suggestion that it might be able to help after all.

This one is trivially matchable, which is why it is the strongest candidate for a rule. The phrasings are short and there are not many of them. Human, person, agent, real person, speak to someone, talk to a human, get me a person. A short list catches nearly all of them, and the cost of a false positive is a handover offered to somebody who did not need it, which is not a harm.

Askably handles this case before the knowledge base is touched at all: a recognised request for a person produces the handover form rather than an answer. The design point is not the specific implementation, it is that the branch happens ahead of retrieval, so nothing in the indexed material can talk the system out of it.

On a health site, a symptom is not a topic, it is a liability. The problem is that symptoms arrive attached to administrative questions rather than on their own. Somebody asks how to book an appointment and adds, in the last clause, what is happening to them. An instruction to avoid clinical advice is competing with a message that is genuinely about booking.

A rule that matches symptom vocabulary and hands over regardless of what the rest of the message says is cruder and better. It will hand over some conversations that could have been answered. That is a cost worth paying, because the alternative failure is a business publishing clinical advice under its own name.

Allergy questions deserve the same treatment on any site that sells food or anything applied to skin, not only on health sites. An allergy question looks like a product question and reads like one to a retrieval system: it matches your ingredients page, and the ingredients page is usually true. Usually is not the standard. Ingredient lists change, suppliers change, and a page can be correct on the day it was written and dangerous a year later. Route it to somebody who can check the current specification.

A visitor reporting a charge they did not make is on a clock. In many payment systems the ability to act depends on how quickly the report reaches somebody who can act, and a conversation that spends four exchanges being helpful is a conversation that has cost them something real.

There is also a second party involved whose interests the assistant cannot represent. A fraud report is not only a support request, it is the start of a process that may involve your payment provider, your bank and possibly a regulator. Nothing in that process is served by an automated reply summarising your refund policy.

The rule should be broad and the response should be short: acknowledge, hand to a person, and if you have a phone route for this, name it in the same message. Match on the obvious vocabulary, and accept the false positives. Somebody asking a general question about card security who gets handed to a person has lost nothing.

A safeguarding disclosure is somebody telling you, often obliquely and often in the wrong channel, that a person is at risk. Businesses that work with children, vulnerable adults, or in education and care already have a defined procedure and a named person responsible for it. That procedure exists because the correct handling is specific and the consequences of improvisation are severe.

An assistant has exactly one job here, which is to get out of the way and put the conversation in front of the person who owns that procedure, quickly and without asking follow up questions. Follow up questions are not neutral in a disclosure, they can shape what somebody says next, and a support widget has no business shaping that.

A legal threat belongs in a rule for a different reason. Once somebody has mentioned a solicitor, a lawyer, a claim or a regulator, anything said in your name is potentially evidence. An improvised automated reply is a written statement by your business about a matter in dispute, composed by a system with no knowledge of the dispute. Acknowledge, route, and say nothing about the substance.

Somebody writing because a customer has died usually opens with an administrative question. How do I cancel this account. How do I stop these emails. What do I need to send you. The administrative question is answerable, and that is exactly the trap: an assistant that answers it correctly has behaved appallingly.

This is the clearest case in the list for a rule rather than a judgement, because the signal is a short and stable set of phrasings and the cost of missing it is high. Match on the vocabulary of death directly, hand over immediately, and do not attempt the administrative question even though the material contains the answer.

It is also the case most often left out entirely, because nobody enjoys writing the rule and it never comes up in a launch checklist. Write it before launch anyway. The alternative is discovering the gap through a conversation you will not want to read.

A rule that lives only in a configuration screen is a rule that disappears the first time somebody rebuilds the setup, changes vendor, or tidies up settings they do not recognise. Keep the list in a document owned by a person, alongside the reason each entry is there.

The reason matters as much as the entry. Six months on, somebody will look at the allergy rule, see it firing on questions that could have been answered, and remove it as an over-correction. A single line explaining why it exists prevents that, and it costs nothing to write at the time.

Review the list when the business changes rather than on a calendar. New product lines, a new regulated activity, a new country, a new audience: each of those can add an entry. Nothing else needs to touch it.