Crawler documentation

Meet WebAtlasBot.

The official crawler of the MevaSearch search engine, building a safe and filter-free map of the internet.

How it works

Identification

WebAtlasBot identifies itself with the following User-Agent.

WebAtlasBot/2.0 (+https://webatlasindex.pl/bot.php)

Verification

All WebAtlas crawlers connect only from the IP addresses listed in ip-ranges.json. A request with our User-Agent from any other address does not come from WebAtlas.

Behavior

The crawler minimizes server load. It respects robots.txt directives, honors Crawl-delay, and checks noindex/canonical tags before saving a page.

WebAtlas Index

Collected content feeds the WebAtlas Index — our own database used for search and semantic analysis by MevaSearch.

Crawl control

Block or throttle WebAtlasBot with standard robots.txt rules:

User-agent: WebAtlasBot
Disallow: /private/
Crawl-delay: 5

The bot also honors the meta tag <meta name="robots" content="noindex">.

Why are we visiting?

WebAtlasBot collects data for the MevaSearch engine, focusing on high-quality, educational and technical content, aiming to build a knowledge base resistant to algorithmic manipulation.

Related crawlers

AdultBot and WebOrbitonBot are sibling crawlers of the same project, run by the same team and held to the same standards described on this page.

About Crawlers Webmaster tools API Pricing Support
Add your page