KapllanBot

KapllanBot is the web crawler of Kapllan AI. It keeps the index of Kapllan's search engine up to date: it revisits public pages when they are likely to have changed, and reads new pages that sites list in their sitemaps and feeds.

It identifies itself in every request with this user agent:

KapllanBot/1.0 (+https://kapllan.ai/bot)

What KapllanBot respects

• robots.txt, by the standard (RFC 9309). KapllanBot reads the rules for KapllanBot, and it also obeys the rules you set for all crawlers (*) and for Googlebot: if either of those groups closes a page, KapllanBot does not fetch it either.
• Crawl-delay in robots.txt, in seconds.
• One request at a time to your site, and never more than one per second. If your server answers slowly, KapllanBot waits longer.
• 429 Too Many Requests and 503: KapllanBot slows down, and waits as long as your Retry-After header asks (up to a day).
• noindex (<meta name="robots" content="noindex"> or the X-Robots-Tag header): the page is not indexed. nofollow: its links are not followed.
• Conditional requests: KapllanBot sends If-None-Match and If-Modified-Since, so an unchanged page costs you a 304 Not Modified and no transfer.

KapllanBot reads robots.txt again at least once a day, so a change there takes effect within 24 hours.

How to stop or limit KapllanBot

To keep KapllanBot out of your whole site, add this to your robots.txt:

User-agent: KapllanBot
Disallow: /

To keep it out of part of your site:

User-agent: KapllanBot
Disallow: /private/

To ask for more time between requests (here, 10 seconds):

User-agent: KapllanBot
Crawl-delay: 10

If you have already closed a part of your site to Googlebot or to all crawlers (*), you do not need to do anything: KapllanBot follows those rules too.

What we do with your pages

Kapllan's search engine indexes public pages so that people can find them. A search result shows your page's title, a short passage and a link to your page. We do not republish your pages; the text keeps its owners' rights.

How to verify that a visitor is really KapllanBot

Anyone can copy a user agent. The real KapllanBot always comes from the addresses in our published list, and those addresses resolve to crawler.kapllan.io:

1. Look up the address from your logs in reverse DNS: host 135.181.61.188 gives crawler.kapllan.io.
2. Look up the name forward and check that it gives the same address: host crawler.kapllan.io gives 135.181.61.188.
3. Or compare the address with our list: kapllan.ai/bot-ips.json.

A request that says "KapllanBot" and comes from any other address is not from us.

Contact

Questions, problems, or a request to remove pages from our index: hello@kapllan.ai. Please include your site's address and, if you can, the times and addresses from your logs. We answer within one working day.