Knowledge sources and crawling

Three knowledge sources

Every answer the assistant gives comes from one of three sources you control:

• Crawled pages — Pages we fetch automatically from your website. Good for product pages, FAQs, blog posts — anything that's already public.
• Uploaded documents — PDF, DOCX, and DOC files you upload. Good for price lists, terms, internal handbooks, anything that isn't on the public site.
• Articles you write — Knowledge articles you author directly in the dashboard. Good for filling gaps the crawl missed or for answers you want phrased a specific way.

One site, multiple sources

A Clarifier site can pull content from more than one place. The site has a primary domain (set when you create it) and you can add additional crawl sources for sibling subdomains or specific URLs. All sources feed the same knowledge base — visitors get answers from any of your content regardless of which source it came from.

• Primary domain — The URL you used to create the site. Cannot be removed; the widget can always be embedded here.
• Additional subdomain — A sibling domain like blog.acme.com or shop.acme.com — must share the same registrable domain as the primary. Each subdomain source is scanned independently and gets its own include/exclude patterns. The widget can also be embedded on any added subdomain.
• Single page — One specific URL — useful for a page the scan missed, a freshly published page you don't want to wait for the next full re-crawl to pick up, or a high-importance page outside your normal patterns.

All sources share the same per-plan crawl-page limit. Pro's 5,000 pages are split however you like across primary, subdomains, and single-page sources combined.

Scan vs. crawl

When you add a site, Clarifier first runs a scan: it reads your sitemap and a sample of pages to suggest URL patterns to include or exclude. The scan is fast and doesn't fetch every page. The crawl is the actual fetch — it follows the patterns you've approved and downloads each page's content. Both can be re-run from the dashboard whenever you change your site.

Include and exclude patterns

After the scan you'll see URL patterns grouped by structure — for example, /produkter/*, /blog/*, /admin/*. Tick the ones to include in the crawl and untick the ones to skip. Common things to exclude: admin pages, search-result pages, paginated archives, customer-account pages. The crawl only fetches pages that match an included pattern and don't match any excluded one.

robots.txt and blocking

Clarifier respects robots.txt. If your robots.txt blocks our crawler, the dashboard will tell you and link to a fix — usually removing a Disallow rule that targets all crawlers. While the crawl is blocked you can still build a knowledge base from uploaded documents and hand-written articles; the assistant just won't know about your live pages.

What happens after a crawl

Once content is fetched (whether from the crawl, a document upload, or an article), it goes through the same three-step processing pipeline:

• The text is split into overlapping chunks — roughly 1,500 characters each, with 200-character overlap so context isn't cut at chunk boundaries.
• Each chunk is converted into a 1,536-dimensional embedding vector using OpenAI's text-embedding-3-small model.
• Vectors are stored in our search index. When a visitor asks a question, we embed their question the same way and retrieve the closest chunks, which the language model uses to write the answer.

Per-plan limits

The crawl page limit caps how many pages we'll fetch from your site. The knowledge sources limit caps how many uploaded documents and hand-written articles you can have per site combined.

If your site has more pages than your plan allows, use exclude patterns to skip low-value sections (search results, archives, account pages) so the crawl focuses on the pages that matter for visitor questions.

Keeping the assistant up to date

When you publish new content, re-run the crawl from the dashboard. Re-uploading a document replaces the old one; editing an article updates it in place. There's no automatic re-crawl yet — refresh manually whenever your site changes meaningfully.