CertantBot, the Certant crawler

When CertantBot visits

Certant customers can add a website as a source for their own knowledgebase. CertantBot then reads the pages of that site and stores them as text the customer can search and ask questions of. Before a crawl starts, the customer has to confirm that they own the site or are authorised to index it.

CertantBot stays on the site the customer added. Unless the customer proves they own the site, one Certant account can keep at most 25 pages from it on a free trial, or 100 pages on a paid plan.

Rules it follows

• robots.txt: It obeys the Disallow and Allow rules for CertantBot, or for * if your robots.txt has no section for CertantBot.
• Crawl-delay: It waits the number of seconds you set between requests, up to 10 seconds.
• noindex: A page marked noindex is skipped and is not added to the customer's knowledgebase.
• 429 and 503 responses: It waits for the time given in Retry-After. If your site keeps returning these, it slows down and then pauses the crawl.
• Pace: By default it fetches about 2 pages a second, with no more than 4 requests in flight to one site at a time.

It does not log in, fill in or submit forms, or try to get past bot challenges such as CAPTCHAs. Pages behind a login or a challenge are reported to the customer as blocked.

Block CertantBot

To stop CertantBot reading your site, add these lines to the robots.txt file at the root of your site:

User-agent: CertantBot
Disallow: /

To keep it out of one part of the site only, put that path after Disallow: instead, for example Disallow: /members/.

Prove you own your site

If you are a Certant customer and the site is yours, proving it lets your account keep more pages from the site: up to 1,000 on a free trial and up to 5,000 on a paid plan. Your Certant account shows a verification token when you add the site. Use any one of these three methods:

1. DNS record: Add a TXT record named _certant-verify.<host> with the value certant-verify=<token>. For www.example.com the record name is _certant-verify.www.example.com.
2. Response header: Send X-Certant-Verification: <token> with the page you start the crawl from. If you cannot set headers, put <meta name="certant-verification" content="<token>"> in that page's head instead.
3. File: Publish a file at /.well-known/certant-verification.txt that contains the token.

Certant checks again at the start of every crawl, so removing the record, header or file takes the site back to the standard limit.