The number moved, and it was probably not the assistant

It requires no setup. The data already exists on both sides of the change, the two numbers are easy to put next to each other, and the story writes itself in one sentence. It also arrives at exactly the moment somebody asks whether the thing was worth doing, which is when a clean answer is most wanted and least available.

The comparison assumes one thing: that the only relevant difference between the two months is the change you made. On a live website that assumption is almost never true, and the failure is not subtle. In a typical quarter a site will have had at least a redesign or a template change, a price or a promotion, a campaign, and a change of season, and each of those moves support volume on its own.

So the honest starting position is that you have two months that differ in several ways and one number that differs. Attribution from that is a story, and a story told confidently enough becomes a fact somebody repeats in a board pack.

Every business has a shape to its year, and most people underestimate their own because they are inside it. School terms move traffic for anything to do with families, travel, tutoring, childcare and uniforms. Tax and filing deadlines move it for anything to do with accounts. Weather moves it for trades, gardens, roofing, heating and anything outdoors. Public holidays move it for everything, including by removing working days from the denominator.

There is a shape to the week too, and it matters when the two months you are comparing contain different numbers of working days, or when one contains a holiday week. A month with a bank holiday in it has fewer working days, fewer replies going out, more messages accumulating and a longer apparent response time, and none of that reflects any change in how anybody worked.

The correction is unglamorous. Compare like for like where you have it: this month against the same month last year, as well as against last month. Where you do not have last year, because the site is new or the setup changed, say the comparison is weak rather than making it and hoping. A weak comparison honestly labelled is usable. A weak comparison presented as a result is not.

A redesign is treated as a visual change and it is rarely only that. It moves where the widget sits and how visible it is. It changes what is above the fold, which changes what people read before they ask. It renames navigation, which changes what people can find. It frequently changes which pages exist and how they are structured, which changes what is indexed and therefore what can be answered at all.

The consequence is that a redesign moves the composition of questions, not only the count. A per topic breakdown will shift even if nothing about the assistant changed, because the population of people reaching it changed and so did what they had already seen.

There is a sharper version of this that catches people out. A redesign that improved the help pages reduces the number of questions asked, because more people find the answer before asking. That shows up as lower volume, which the person presenting the assistant's numbers will happily read as successful deflection, and which is actually somebody else's win entirely. Both effects are good. Attributing one to the other is how a team ends up investing in the wrong thing next quarter.

Change a price and the question mix changes immediately and dramatically. People ask about the new price. They ask whether their existing arrangement is affected. They ask what the alternatives are. They ask whether they can lock in the old rate. None of those questions existed the month before and none of them are covered by material written before the change.

That produces a spike in unanswered questions that has nothing to do with the system's quality, and it produces it at precisely the moment somebody is looking at the numbers, because a price change is the kind of event that prompts a review.

It also changes who is on the site. A price rise brings existing customers who would not otherwise have visited. A promotion brings people who are not customers at all. And a price change generates complaints, which are a different kind of conversation, longer and more likely to escalate, and they will skew any average length or handover rate you happen to be tracking.

This is the confounder that is most often read backwards. A campaign brings colder traffic: people less familiar with what you do, more likely to ask basics, less likely to have read anything, less likely to buy. Their questions are more varied and more of them fall outside your material.

So the unanswered rate goes up during a campaign, and the naive reading is that quality dropped. It did not. The population changed and the material was never written for that population. That is a useful finding and it is an entirely different finding from the one somebody is about to write down.

The general form of this is worth keeping in mind whenever a rate moves. A rate can change because handling changed, or because the mix of what is being handled changed, and those look identical in the total. If every individual topic improved and the overall figure got worse, the mix moved. Checking the per topic breakdown before reading the total takes a minute and prevents the most embarrassing category of conclusion.

The single highest value thing here is boring and cheap: a dated list of everything that changed, kept by whoever changes things. One line each. Prices, page structure, campaigns, staffing, opening hours, product availability, anything switched on or off, anything that went wrong.

It costs a line per change and it converts an unanswerable question into a readable one. Without it, every before and after comparison is reconstructed from memory months later, and memory is worse than useless here because it retains the change you were involved in and drops the four you were not.

It works best when it is not owned by the person doing the analysis. The people who change prices, run campaigns and edit templates are not the people reading support numbers, and they will not think to mention it. One shared list, appended to as a habit, read before any chart is opened.

When you genuinely need attribution, the way out is to stop comparing across time. Time is the axis that carries every confounder at once. Split by something else and most of them stop mattering.

The practical version on a small site is by section. Put it on the documentation and not on the product pages, or on one brand and not the other, and compare the two in the same weeks, in the same season, under the same campaign. That is not a controlled experiment, because the sections differ in other ways, but it removes the season, the redesign and the campaign in one move, which is most of your problem.

Two honest caveats. This needs enough volume in both halves to see anything, which many small sites do not have, and a split that shows the assistant to some visitors and not others changes what your own team sees day to day, which they should be told about. A staged rollout by section is usually more acceptable than a hidden split, and it produces nearly the same evidence.

The professional move is to report the number, list the confounders, and say which direction each of them probably pushed. The figure fell. In the same period we redesigned the help section, ran a campaign for two weeks, and moved from our busiest month to our quietest. Two of those push the number down for reasons unrelated to the assistant.

That reads as weaker than a clean claim and it is more credible, and more useful, because it tells the reader what to do next rather than what to believe. It also protects you. A claim of credit that somebody later unpicks costs more than the credit was worth, and support numbers get unpicked whenever they are used to justify a budget.

The version of this to avoid is the one that sounds humble and is not: presenting the number with a footnote nobody reads. Put the confounders in the same sentence as the figure. If they belong in a footnote, they are not confounders, they are decoration.