AutoPlanX runs two small crawlers. Both read public web pages only, at a low rate, and both obey robots.txt.
User-Agent: AutoPlanXAuthorityBot/1.0 (+https://autoplanx.online/bot; crawler@autoplanx.online)
Purpose: link research. It reads public pages on a small number of sites in the topics our websites cover, to see which pages link to which.
What it does not do: it does not log in, submit forms, collect personal data, solve CAPTCHAs, or try to get around a block.
Rate: one request at a time per site, with a pause of at least two seconds between requests, and a fixed small daily page limit. It backs off when a site answers 429 or 5xx, and stops for the day when a site refuses it.
User-Agent: AutoPlanX-Cortex-Research/1.0
Purpose: topic research. It reads a limited number of public articles that rank for a search topic, to understand what readers are already given.
Both crawlers read /robots.txt before any other page. If your robots.txt cannot be read, the Authority crawler does not crawl your site at all. To block a crawler, add for example:
User-agent: AutoPlanXAuthorityBot Disallow: / User-agent: AutoPlanX-Cortex-Research Disallow: /
The crawlers currently run from 152.228.132.15. If more addresses are used in the future, they will be listed here first.
Email crawler@autoplanx.online to ask for a slower rate, a removal, or with any question.