Website Crawl Errors

Open Training → Knowledge. A website source marked Action needed shows a short error with its address. Open the source to see the full error. For a problem with one page, find that page and open its details.

Check the reported problem

• The page cannot be opened: verify the address and whether it redirects or requires a login.
• A firewall blocks access: ask your website administrator to inspect the blocked request. Do not disable protection for the whole website just to train a page.
• The website could not connect to its database: ask your website administrator to restore the database connection. During background training, Boei tries twice more. If the problem continues, training stops for the remaining pages of that website. After the website works again, retry the affected pages.
• No readable content: check that the page actually contains useful text, or add that content as a file or text source.
• Search engine results cannot be used as a knowledge source: add the original page address instead. Search links in a bulk import stay visible with this reason. If a previously trained search page is retried, Boei removes its old knowledge and frees its training space.
• The address is a sitemap: add it as a website source for discovery rather than treating the XML itself as knowledge.
• Not enough training space: free space or upgrade, then resume. This is a capacity problem, not a website error.
• Training stopped or only partly succeeded: inspect the details, then retry the affected content.

When part of a sitemap cannot be read

A sitemap is a list of pages on your website. Some websites split that list across several addresses. If Boei cannot read one of them, the source shows Action needed with the address and reason.

Boei still processes pages from the lists it can read, including when you first create a chatbot. During later updates, Boei adds new pages and refreshes pages that need an update. It keeps existing pages because an unread list does not mean those pages were removed from your website.

If you receive an update email, it includes the address that could not be read alongside the results for pages that did update.

Check the address in the error. Once it works again, choose Sync now if your account includes Auto Sync. On Starter without Auto Sync, you can retry individual content pages; checking the whole sitemap requires Auto Sync. Boei can check for removed pages again after it has read the full list. The warning clears after a successful update. If content pages are still blocked by a firewall, that warning stays until those pages can be read too.

Retry carefully

After correcting the cause, retry one affected page first. Check its extracted content and try a question with Test chatbot. If that succeeds, retry the remaining affected selection.

A discovered address is not automatically usable knowledge. Wait for the ready status before expecting the chatbot to answer from it. Existing usable content remains available during replacement processing.

Large or slow sites

Training runs in the background and is paced per website. A large site can take longer even when Boei has spare workers, because repeatedly requesting the same host too quickly can cause blocks or overload. Leave the run processing and return to its progress, or use the completion-email option.

If a problem persists, contact support with the page address, the displayed error and when it happened. Do not include passwords or private access tokens.