One document, a Ready badge, and nothing saying why

The source on the Knowledge tab says Ready and the line under its name says 1 document, on a site with far more pages than that.

In rough order of how often each one turns out to be the answer. Work down rather than across: each carries a way to tell whether it is yours before you change anything.

• 1

  There is no sitemap at the address the crawler looks at

  Why

  The sitemap is tried first because it is cheaper and more complete than following links. It is looked for at /sitemap.xml on the scheme and host of your start URL, and nowhere else. A sitemap announced in robots.txt at some other path, or split into files under a different name, is not found, and the crawl quietly falls back to following links.

  How to confirm it is this one

  Open your start URL, then put /sitemap.xml on the end of the host and load that. If you get a 404, a redirect to the home page, or an HTML page rather than XML, the crawl had no sitemap to work from and everything depended on the links.

  Fix

  Publish a sitemap at that exact address, or make it the thing your existing sitemap path redirects to. A sitemap index living there works too: it is followed, and up to 25 sitemap documents are read out of it.

• 2

  The navigation is built in the browser, so there are no anchor tags to follow

  Why

  Link discovery parses the HTML that was delivered and collects anchor tags carrying an href. A menu assembled by script after delivery contributes no anchors, and neither do buttons wired up with click handlers. Fragment, mailto, tel and javascript links are skipped as well, which on some sites removes most of what is left.

  How to confirm it is this one

  View source on the start page, the real source rather than the developer tools inspector, and search for the text a href. Count what comes back. If the delivered document contains a handful of anchors and your menu has forty items, the menu is not in the document.

  Fix

  A sitemap is the cheapest answer here, because it sidesteps discovery entirely. Failing that, make sure the navigation is present in the delivered HTML, or add the important sections as separate sources one at a time.

• 3

  The start URL redirects to a host the same-site test rejects

  Why

  Redirects are followed, so the page that gets read is the one you land on. The same-site test, though, compares the host you typed against the host of every link found, and it only strips a leading www before comparing. Land on a different host and every link on the page you fetched belongs to that other host, fails the comparison, and is dropped.

  How to confirm it is this one

  Paste the start URL into a browser and read the address bar once it has settled. If the host you end up on differs from the host you typed by anything more than a leading www, a country or language subdomain for example, that is the whole cause.

  Fix

  Start the crawl from the address it actually lands on. Note that example.com and www.example.com genuinely are one site here, so a www redirect is never the problem.

• 4

  Depth is set to zero, or too low for how the site is laid out

  Why

  Links are only collected from a page whose depth is below the cap. The start page is depth zero, so a cap of zero means links are never collected from anything and exactly one page is indexed. Above zero the cap is a real limit on how far a chain of links is walked: a page reachable only through four clicks is never discovered under the default of three.

  How to confirm it is this one

  Look at the Depth box on the crawl form. Zero explains a single document on its own. For any other value, start a second crawl from the same URL with Depth at 6 and compare the document counts on the two source cards.

  Fix

  Raise Depth. Pages found through the sitemap are queued at depth 1 regardless, so a site with a working sitemap barely feels this setting, which is another reason to fix the sitemap first.

• 5

  robots.txt disallows the paths your links point at

  Why

  robots.txt is read from the start URL's host and every discovered link is checked against it before being queued. A disallowed link is skipped and counted as skipped, with nothing said. The start URL itself is not checked, and neither are addresses that came from the sitemap, which is why this cause produces exactly one page rather than none.

  How to confirm it is this one

  Load /robots.txt on the same host and read the Disallow lines against the paths your navigation points at. A 404 there rules this out completely, because a missing robots.txt is read as allowing everything.

  Fix

  Change the rules so the paths you want indexed are allowed. The crawler identifies itself in its user agent and links to a page about itself, so you can write a rule aimed at it specifically and leave every other crawler where it is. Do not go looking for a route around the file: it is your own host, and a rule that says something you do not mean is a problem well beyond this one crawl.

• 6

  The page allowance for this assistant was already spent

  Why

  The budget for a crawl is your plan's page number minus everything already indexed for that assistant, across all of its sources. A budget of one leaves room for the start page and nothing else, and a budget of zero stops the crawl before it starts.

  How to confirm it is this one

  A budget of zero is explicit: the source shows Failed and carries the message about the page limit for your plan, naming the number. Short of that, add up the document counts on every source card for that assistant and compare with your plan: 50 pages on Free, 500 on Starter, 5,000 on Growth, 50,000 on Agency.

  Fix

  Delete a source you no longer need, which removes its documents with it, or move up a plan. Trimming the crawl with a lower Max pages does not help, because the ceiling is the plan number and not the field.

Include and exclude patterns are the cause nobody guesses, because the crawl form on the Knowledge tab does not send them: they only exist on a source created through the API. They are globs, matched against the whole URL and against the path, and a pattern such as docs/* matches neither, since the path begins with a slash and the whole URL begins with the scheme. A pattern that matches nothing filters out everything. If the source was made through the API, start a second crawl from the same URL with no patterns at all and compare the two document counts.

Why does a crawl that read one page still say Ready?

Because nothing failed. The status describes whether the run completed, not whether the result is any good, and inventing an error out of a small number would mean guessing at how many pages your site is supposed to have. The document count on the card is the number to read, and it is deliberately shown next to the name rather than buried.

Does it index PDFs and images it finds linked from a page?

No. Anything whose response is not HTML is skipped, and addresses ending in the usual image, stylesheet, script, font, archive, audio, video and installer extensions are never fetched at all. A linked PDF has to be uploaded as a file if you want it answered from.

How hard does the crawler hit the site?

Gently, on purpose. There is a short delay between requests and a small limit on how many run at once, because the crawl runs against somebody's live site and a burst that looks like an attack is a worse outcome than a slow crawl.