Why a subdomain of a site you already crawled was never read

A crawl of the main site finished and none of the documentation, help centre or shop pages are in it, even though they are linked from the navigation.

In rough order of how often each one turns out to be the answer. Work down rather than across: each carries a way to tell whether it is yours before you change anything.

• 1

  The subdomain is a different host, and only www is forgiven

  Why

  Every link found on a page is compared against the host the crawl started from, with a leading www removed from each side before comparing. docs.example.com does not survive that comparison against example.com, so the link is dropped before it is ever fetched.

  How to confirm it is this one

  Ask the assistant something on the Test tab that only the subdomain answers. A grounded reply shows a numbered source under it, linking to the page it used. If you get the fallback message instead, and the question then turns up on the Insights tab under Unanswered questions, nothing from that host is indexed.

  Fix

  Add a second crawl source with the subdomain as its start URL. Each source keeps its own start host, so the two crawls do not interfere, and both feed the same assistant.

• 2

  The subdomain is in your sitemap and was skipped anyway

  Why

  Sitemap addresses are convenient but not privileged. Every address read out of a sitemap goes through the same site test before it is queued, so a sitemap that spans several hosts contributes only the entries on the host you started from.

  How to confirm it is this one

  Open /sitemap.xml on the start host and search it for the subdomain. Finding those addresses there, next to a document count that clearly does not include them, is this cause and no other.

  Fix

  Same answer: a second source pointed at the subdomain. There is no version of this where one crawl covers both hosts from the form.

• 3

  The subdomain was added to the allowed sites instead of as a source

  Why

  Two different lists get confused here. Allowed sites controls where the widget is permitted to load and talk, which is an installation question. Sources control what the assistant knows. Adding a host to the first does nothing whatsoever to the second.

  How to confirm it is this one

  Look at the Install tab and the Knowledge tab side by side. If the subdomain appears under allowed sites and there is no source whose name points at it, this is what happened.

  Fix

  Add the source. Keep the allowlist entry too if the widget is genuinely installed there, and note that the number of distinct sites one assistant may be installed on is capped by the plan: one on Free, three on Starter, five on Growth, ten on Agency. Sources are not capped that way, only the pages they index are.

• 4

  The second source ran and stopped at the page allowance

  Why

  The budget for any crawl is the plan's page number minus everything the assistant already has. A main site that used most of the allowance leaves very little for the subdomain, and none at all if it used all of it.

  How to confirm it is this one

  The new source shows Failed and carries the message about the page limit for your plan with the number in it, or it shows Ready with a document count far below the size of the subdomain.

  Fix

  Lower Max pages on the main site source and re-crawl it, which frees the allowance for the documentation people actually ask about, or move up a plan.

• 5

  The subdomain has its own robots.txt and it disallows the crawl

  Why

  robots.txt is read from the host a crawl starts on, so a second source pointed at the subdomain reads that subdomain's file rather than the main site's. Documentation and help hosts are frequently the ones with the restrictive rules, because they are frequently the ones somebody put behind a tool with its own defaults.

  How to confirm it is this one

  Load /robots.txt on the subdomain itself, not on the apex. The two are separate files and are often nothing like each other.

  Fix

  Change the rules on that host so the pages you want indexed are allowed. A missing robots.txt is read as allowing everything, so deleting a file that was serving no purpose is also a valid answer. Do not look for a way around the rule: it is your own host, so the honest fix is to say what you actually mean in the file rather than to route past it.

Turning the same-host restriction off entirely is possible through the API and is almost always a mistake. Without it the crawl follows every external link it meets, which on a site with a blog roll or a partner page means spending your whole page allowance indexing other people's writing and then answering visitors out of it. Two sources with two start hosts costs nothing extra and cannot wander.

Do the two sources answer as one assistant?

Yes. Sources are how material is grouped for indexing and re-syncing, not how it is searched. Retrieval runs across everything that assistant has, so a question is answered from whichever source holds the best passage.

Is a different port or scheme also a different site?

For the crawler the comparison is on the hostname alone, so a port does not split it. For the widget allowlist an origin includes the scheme and the port and they very much do split it, which is a separate list with separate rules.

Can I crawl a subdomain I do not control?

You can point a crawl at anything publicly reachable and permitted by its own robots.txt, but you should not. The assistant answers as your business, so indexing somebody else's pages means quoting them to your visitors under your name.