MeyaBot is a name for Meya's web crawler used as part of our GPT/LLM AI training (learn more about MeyaGPT). It is used to crawl and extract content to train an AI customer service chatbot.

Access Control

It is generally used by customers to crawl their own websites to augment their existing knowledge base (KB), help center or other documentation. If you feel that someone is crawling your content without your permission, you can restrict MeyaBot using one or more of the following mechanisms.

User-Agent

You can identify MeyaBot by looking at it's user agent string in the request:

MeyaBot/1.0 (+https://docs.meya.ai/docs/meya-bot)

robots.txt

MeyaBot will observe your robots.txt file. You can also selectively provide instructions to MeyaBot:

User-agent: MeyaBot
Disallow: /not-for-meya

IP Address

All MeyaBot requests will originate from an IP address found this list:

https://cdn.meya.ai/meya-bot/ips.txt

This list of IPs can be used to further restrict access to your content.