Website Training

Train your chatbot on your website content. Boei discovers pages on your site, lets you select which ones to train, and extracts the useful content automatically.

How to get there: Go to Setup → Chatbot in the top menu → click your chatbot → Training → Website tab.

Adding Pages

Click Add New Pages to see the available discovery methods:

Site Crawl

Boei's crawler visits your website and follows links to discover pages, similar to how a search engine works.

1. Click Add New Pages → Site Crawl
2. Enter your domain URL
3. Optionally set path filters to include or exclude certain sections of your site
4. Click Crawl domain
5. Boei discovers pages and lists them with status "Fetched"
6. Select pages and click Train (or use Select All to train everything)

The crawler discovers pages by following links from your starting URL. Pages behind login walls or not linked from anywhere won't be found.

If your crawl fails, it's usually because a firewall is blocking the crawler. See Crawl errors for troubleshooting steps.

Sitemap Import

If your website has a sitemap (most do), you can import pages directly from it.

1. Click Add New Pages → Sitemap Import
2. Enter your sitemap URL (usually yoursite.com/sitemap.xml)
3. Boei parses the sitemap and lists all discovered pages
4. Select pages and click Train

When you add a URL, Boei automatically looks for your sitemap, so you rarely need to paste the sitemap URL by hand.

Page Links

Extract and import all links found on a specific page. Useful for sites that have an HTML sitemap or a links/resources page.

1. Click Add New Pages → Page Links
2. Enter the URL of the page containing links
3. Boei extracts all links from that page
4. Select which linked pages to train

Bulk Upload

Paste a list of URLs directly.

1. Click Add New Pages → Bulk Upload
2. Paste your URLs (one per line)
3. Click submit to add them for training

Which Discovery Method Should I Use?

Recommendation: Use Sitemap Import if your site has a sitemap, especially for sites that change regularly. Pair it with auto-sync (Growth plan and above) to keep your chatbot current without any manual work. Use Site Crawl for a quick start if you're not sure about your sitemap.

Content Filtering

Boei automatically strips common boilerplate elements when crawling pages: navigation menus, footers, sidebars, cookie banners, comment sections, ads, and similar non-content elements. This happens by default so your training data stays clean without any setup.

For advanced control, you can set your own CSS include/exclude selectors per website source to further refine what content gets extracted. See CSS filters for details.

Page Status

Each page shows a status indicator:

• Green - Successfully trained
• Yellow - Processing (Boei is reading and learning from the page)
• Red - Failed (click for error details)
• Gray - Discovered but not yet trained

Page Credits

Every 5,000 characters of cleaned page content counts as 1 page credit (minimum 1 per page). Each page shows its credit cost in the list. See Training credits for plan limits and details.

Keeping Content Up to Date

• Auto-sync periodically re-checks your sitemap sources and individual pages, picks up new and changed pages, and cleans up removed ones. Available on Growth plan and above.
• Manual retrain lets you update specific pages on demand. Select pages and click Retrain.
• Bulk operations let you retrain, untrain, or delete many pages at once.

Long crawls resume from where they left off if interrupted, so a large site doesn't have to restart from zero after a timeout.

Tips

• Train your whole site. Don't overthink which pages to include. Train everything and let the AI find the right answers. You can always untrain irrelevant pages later.
• Use sitemap import for ongoing accuracy. Combined with auto-sync, your chatbot stays current as you publish and remove pages.
• Check the quality dashboard to spot topics where your chatbot needs better content.