Crawler documentation
Meet WebAtlasBot.
The official crawler of the MevaSearch search engine, building a safe and filter-free map of the internet.
Crawler documentation
The official crawler of the MevaSearch search engine, building a safe and filter-free map of the internet.
WebAtlasBot identifies itself with the following User-Agent. We recommend verifying requests via reverse DNS.
WebAtlasBot/2.0 (+https://webatlasindex.pl/bot.php)
The crawler minimizes server load. It respects robots.txt directives, honors Crawl-delay, and checks noindex/canonical tags before saving a page.
Collected content feeds the WebAtlas Index — our own database used for search and semantic analysis by MevaSearch.
Block or throttle WebAtlasBot with standard robots.txt rules:
User-agent: WebAtlasBot
Disallow: /private/
Crawl-delay: 5The bot also honors the meta tag <meta name="robots" content="noindex">.
WebAtlasBot collects data for the MevaSearch engine, focusing on high-quality, educational and technical content, aiming to build a knowledge base resistant to algorithmic manipulation.
AdultBot and WebOrbitonBot are sibling crawlers of the same project, run by the same team and held to the same standards described on this page.