Skip to main content

​
Overview

We support Automatic Recrawl to ensure your site’s content stays fresh. This involves periodically re-crawling your site and re-parsing its contents. This ensures your AI assistant always provides up-to-date answers.

​
Enabling Automatic Recrawl

The New Domain wizard will ask you if you wish to configure Automatic Recrawl, and ask you to specify a crawl interval in hours. The minimum crawl interval is 1 hour. Once you have added a domain, you can browse to the Domain Settings page to toggle Automatic Recrawl on or off, or change the recrawl interval.

​
Smart Recrawl

Smart recrawl uses your sitemap lastmod time to only crawl pages which have changed since the previous crawl. This speeds up recrawls by only crawling updated content, including new pages. It also lowers the load on your server. If your sitemap does not implement lastmod, this setting will not work.

​
Manual crawl

If you wish to trigger an immediate crawl, you can do this from the Domains page, by clicking ‘Crawl again’.
Note that triggering a manual crawl does not reset the interval for automatic crawling.

​
Viewing crawls

The Crawls page allows you to view the status of a given crawl, including the number of URLs detected and scraped, as well as the crawl duration.