Source material

The crawler will not open your manual, and the manual resists being split

A PDF manual is often the most authoritative document a company owns and the one that behaves worst when it is indexed. It was laid out for print, which means its structure lives in typography and position rather than in the text, and almost everything that makes it a good manual on paper is something a text extractor either loses or turns into noise.

Why this one is harder than it looks

First, it will not arrive by itself. A crawl follows links and a sitemap and skips any response that is not a web page, so a PDF linked from your documentation is passed over rather than read. This is the single most common surprise: the manual is on the site, the site was crawled, and the manual is simply not in the material. It has to be uploaded as a file, deliberately, and nothing about the crawl will tell you it was missed.

Second, a scanned manual contains no text at all. Pages are read as text, page by page, and text that exists only as pixels inside an image is not read. A manual produced by scanning paper, or one whose diagrams and tables were pasted in as pictures, can upload successfully and contribute almost nothing. The upload looks like it worked, which is the worst way for this to fail.

Third, page furniture becomes noise in every passage. The product name in the header, the confidentiality line in the footer, the page number, the chapter title repeated on every recto. Extracted linearly, these strings land in the middle of the text, appearing every few hundred words, and they are the same on every page, so they contribute nothing distinguishing while taking up room in the material handed to the model.

Fourth, layout betrays the reading order. A two-column page can be read straight across, interleaving two unrelated sentences into nonsense. A table that continues onto the next page loses its header, so the continuation is a grid of values with no idea what they are values of. A sidebar or a callout box lands wherever its text happens to sit in the file, which is frequently in the middle of a sentence it has nothing to do with.

What it has to contain

Structure rather than wording. A passage pulled out of this document has to stand on its own, because that is the only form in which it will ever be read.

Structural requirements
Real text, not pictures of textThe document has to have a text layer. Select a paragraph in a viewer and copy it. If nothing comes across, the file is images and needs converting to text before it is worth uploading.
A single column reading orderOne column per page, so the extracted order matches the order a person reads in. Multi-column layouts are for print and they do not survive being read as a stream.
No repeated headers or footersThe version prepared for upload has the running heads, the footers and the page numbers removed. They exist for a printed object and they are pure repetition in every passage of the extracted text.
Tables that fit on one page with their headerAny table that spans a page break either fits on one page, repeats its header on the continuation, or is written out as prose rows. A continuation without a header is a set of values without labels.
One file per topic, each with its subject in its opening linesA per-topic upload whose first lines name the topic, so every passage from it sits in a document that is about one thing rather than in a four-hundred page document that is about everything.

The rewrite, in order

Check whether there is any text in there at all

Open the manual and try to select a paragraph and copy it. If you get the words, there is a text layer and the file can be read. If you get nothing, or you get a picture, the manual is scanned and uploading it will produce a source that contributes essentially nothing while appearing to have been added successfully.

Do this per section rather than once, because mixed documents are common. A manual can be perfectly readable text for two hundred pages and then contain a chapter that was pasted in from a scan, or an appendix of screenshots with all the specifications inside the pictures. That chapter is invisible and nothing will say so.

Where a section is images, it has to be converted to text before upload. That is real work with real cost, and it is worth doing only for the sections people ask about. A scanned appendix nobody references is not worth converting; a scanned specifications table that generates half your questions is.

Strip the page furniture and unpick the columns

Prepare a version of the manual for upload that is not the version you print. Take out the running headers, the footers, the page numbers, the confidentiality notice and the document reference that repeats on every page. None of them mean anything in a passage and all of them appear in every passage.

Reflow anything in two or more columns into one. This is the change that fixes the most severe corruption, because a two-column page read straight across produces sentences that are grammatically plausible and factually invented, alternating halves of two different paragraphs. That is much harder to notice than an obviously broken passage.

Deal with the floating material at the same time. Callout boxes, margin notes and sidebars land at unpredictable points in the extracted stream. Either move each one inline, immediately after the paragraph it belongs to, or pull it out into its own short section with a heading that says what it is about.

Rescue the tables

Find every table that crosses a page boundary. On the continuation page the header is gone, and what remains is rows of values with no columns to name them. Either resize the table so it fits on one page, repeat the header row at the top of each continuation, or convert the whole table into one sentence per row with the column names written into the sentence.

The prose version is the most robust, and for tables people ask specific questions about it is worth the space. A row that reads the compact model has a capacity of that many litres and weighs that much is answerable on its own; the same row inside a grid, one page after its header, is not.

Apply the same test to any table whose meaning lives outside its cells: a merged heading covering three columns, a shaded row indicating something, an asterisk pointing to a footnote at the bottom of the page. Each of these carries meaning that does not come across as text, and each has to be written into the cells or into the surrounding sentences instead.

Split by topic, and be honest about what it costs

A four-hundred page manual uploaded as one file is one source whose passages all come from an enormous document about everything. Splitting it into per-topic files, one for installation, one for maintenance, one for specifications, one for safety, gives each set of passages a document that is about a single subject and makes it far easier to see later which topics are actually being asked about.

The cost is real and worth stating plainly. It is manual work each time the manual is revised, because a new edition means splitting again rather than replacing one file. It uses more of your page allowance, which is set by plan: fifty on the free tier, five hundred on Starter, five thousand on Growth and fifty thousand on Agency. And you have to keep track of which split files correspond to which edition, or you will end up with two editions of the same chapter uploaded at once, which produces exactly the contradiction you would expect.

One thing you do not have to worry about is accidental duplication. Two files whose extracted text comes out identical are stored once, so a printer-friendly variant of a chapter you have already uploaded adds nothing rather than doubling it. That protects you from the most common accident, but not from two genuinely different editions, which differ in their text and so both count.

What happens if you skip it

The source that uploaded successfully and answers nothing

The distinctive failure of a PDF manual is silence where there should be an answer. The manual is definitive, it covers the question in detail, and the assistant declines because nothing in the extracted text resembles the question closely enough. The material contains page headers, page numbers, and paragraphs of interleaved column text that read as gibberish to a matcher, and the actual specification was a picture.

The louder version is a specification quoted from the wrong row. A table split across a page break yields a passage of bare values, the model is handed the numbers without the column names, and the customer is told a capacity or a torque figure that belongs to a different model. In a manual that is not an embarrassment, it is a safety problem, and it is the one case in this whole family where a wrong answer can hurt somebody.

Both failures share a cause and a cost: the upload appeared to work. Nothing is missing from the list of sources, nothing is flagged as unreadable, and the owner reasonably assumes the manual is covered. The gap is only discovered by asking the assistant a question you already know the answer to, which is the single most useful thing to do after uploading any manual.

Check it against this

Before you index it

  • Text can be selected and copied from every section of the file
  • The upload version has no running headers, footers or page numbers
  • Multi-column pages have been reflowed into a single column
  • No table continues onto another page without its header
  • Callouts and margin notes sit next to the text they belong to
  • Large manuals are split per topic, one edition at a time
  • Someone has asked the assistant a question they know the answer to

Questions

Can I just link to the PDF and let the crawler find it?
No. A crawl follows links and a sitemap but skips anything that is not a web page, so a linked PDF is passed over. It has to be uploaded as a file. This catches people out constantly, because the crawl reports success and the manual simply is not in the material.
How do I know whether my manual is scanned?
Try to select and copy a paragraph in any viewer. Text comes across if there is a text layer and nothing comes across if the page is an image. Check several sections, not one, because a manual that is text throughout can still have an appendix or a diagram chapter that was pasted in as pictures.
Is it better to upload the manual or to publish it as web pages?
Web pages, where you have the option. A page per topic gives you headings that survive, tables converted properly, text that can be corrected without regenerating a document, and a crawl that picks up changes. Upload the file where the manual is the authoritative artefact and republishing it is not realistic.

Keep reading

Try it on your own material

Upload a document or point it at your site, paste one line of HTML, then ask it something only your business could answer.