Pages whose text only arrives in the browser

A page you can read perfectly well in a browser produces no document, or produces one with none of the content you expected in it.

In rough order of how often each one turns out to be the answer. Work down rather than across: each carries a way to tell whether it is yours before you change anything.

• 1

  The document sent by the server is a shell

  Why

  Extraction prefers real structure: headings, paragraphs, list items, tables, quotes and code blocks, in that order of usefulness. When that yields very little it falls back to the whole container's flat text, which loses the heading structure but beats extracting nothing. When even that is tiny and the HTML around it was large, the page is refused rather than indexed.

  How to confirm it is this one

  View source on the page, the delivered source rather than the inspector, and search for a full sentence you can see on screen. If it is not in there, this is it. To see the refusal in as many words, save that page and upload it on the Knowledge tab: the source card carries the message saying the page builds its content with JavaScript, which cannot be read by fetching the page.

  Fix

  Three things work. Serve a version of the page whose content is in the delivered HTML. Build that route ahead of time, so the delivered document arrives complete instead of being assembled on arrival. Or take the content out of the page altogether and upload it as a file, which is often the fastest of the three and is the right answer for pricing tables and policy text that changes rarely.

• 2

  The text is there but marked hidden

  Why

  Hidden content is stripped before extraction, and this is a security control rather than tidiness: white-on-white text, zero-size text and elements marked hidden are how a page carries instructions a human reviewer never sees. Elements with a hidden attribute, with aria-hidden set to true, or with a style that hides them are removed, along with comments.

  How to confirm it is this one

  View source, find the missing text, and look at the element wrapping it. Collapsed accordion panels, inactive tab panels and off-canvas menus are usually marked aria-hidden or hidden until opened, and that is exactly the markup that gets dropped.

  Fix

  Do not fight the stripping, because you do not want it turned off. Move the answers out of collapsed panels into the body of the page, or upload the same content as a file. An FAQ hidden behind twelve accordion toggles is worth restructuring anyway, since it is the material visitors ask about most.

• 3

  A protection layer served the crawler something other than the page

  Why

  The crawler sends a bot user agent that identifies itself and links to a page about the bot, which is the honest thing to do and also makes it trivially easy for a protection layer to challenge or refuse. What comes back is then a challenge page or a refusal, not your content.

  How to confirm it is this one

  The source card carries an error line naming the address and the status it returned, which is the clean case. The messier case is a challenge page returning 200, and that looks identical to a shell in view source except that the words on it are about verifying a browser rather than about your business.

  Fix

  Allow the crawler through by name in your protection rules. Its user agent identifies it and points at a page describing it, so a rule can be written narrowly rather than by opening the site up.

• 4

  The content is in a linked file rather than in the page

  Why

  Any response that is not HTML is skipped, so a page whose real content is a linked PDF or a document viewer contributes only whatever wrapper text surrounds it. Addresses ending in the common image, stylesheet, script, font, archive, audio, video and installer extensions are not even fetched.

  How to confirm it is this one

  Look at what the page actually contains. If the useful part sits inside an embedded viewer, or behind a download link, the crawl read the wrapper and there was nothing else there to read.

  Fix

  Upload the file itself on the Knowledge tab. Files are parsed properly rather than skipped, and a manual or a price list is usually better material than the page that links to it.

It is worth being clear about why the refusal exists at all, because at first glance indexing the shell looks harmless. It is not. A document made of navigation labels and a cookie notice matches questions about navigation labels and cookie notices, so the assistant confidently cites a page that never contained the answer, and the person reading the reply has no way to tell. A page that is refused shows up as a missing document, which is a problem you can see. A page indexed empty shows up as an assistant that is subtly wrong, which is a problem you hear about from a customer.

Could you not just drive a browser and wait for the page?

It would work and it would change what this costs to run, for a failure with three good workarounds and one root cause most sites can fix in their own build. Uploading the content as a file takes a few minutes and produces better material than a scraped page would.

The page works when I save it and upload it. Why?

Because your browser had already assembled it before you saved. An uploaded HTML file goes through exactly the same extraction, so it is a fair test of what the extractor can see, and it is also a legitimate way to get one stubborn page indexed.

Does search indexing prove my page is fine?

No. Search engines drive a browser and this does not, so a page can be perfectly indexed by a search engine and unreadable here. View source is the only test that matches what this actually does.