Last Updated: 07/19/2025
MediaThriveBot is the web crawler operated by MediaThrive ("we," "us," or "our"). This page provides detailed information about our bot, its behavior, and how website owners can control its access to their content.
MediaThriveBot is an automated web crawler that systematically browses the World Wide Web to collect data for our services. Our bot helps us:
MediaThriveBot identifies itself with the following user agent string:
Mozilla/5.0 (compatible; MediaThriveBot/2.0; +https://mediathrive.com/bot)
Our crawler operates from the following IP address ranges:
MediaThriveBot adjusts its crawling frequency based on several factors:
MediaThriveBot strictly adheres to the robots.txt protocol. We respect the crawl directives specified in your robots.txt file, including:
To control MediaThriveBot's behavior on your site, you can use these examples:
To allow MediaThriveBot but limit its crawl rate:
User-agent: MediaThriveBot
Crawl-delay: 10
Allow: /
To block MediaThriveBot entirely:
User-agent: MediaThriveBot
Disallow: /
To block MediaThriveBot from specific sections:
User-agent: MediaThriveBot
Disallow: /private/
Disallow: /members/
Allow: /
MediaThriveBot is designed to be polite and avoid overloading servers:
MediaThriveBot collects:
Data collected by MediaThriveBot is used for:
If you encounter any issues with MediaThriveBot, please contact us:
Please include the following information in your report:
If you need to adjust how MediaThriveBot crawls your site beyond robots.txt controls, please reach out to us with your specific requirements.
No, MediaThriveBot is independently operated by MediaThrive and is not affiliated with any major search engine.
Yes, MediaThriveBot can render and execute JavaScript to access dynamic content on modern web applications.
No, MediaThriveBot only accesses publicly available content and does not attempt to bypass authentication systems.
Check that the User-Agent matches our official string and that the IP address is within our published ranges. For additional verification, you can perform a reverse DNS lookup which should resolve to our domains.
Yes, MediaThriveBot respects the nofollow attribute on links.
MediaThriveBot's operations comply with relevant laws and regulations governing web crawling and data collection. For more information about our legal policies, please refer to our:
If you have specific legal inquiries regarding MediaThriveBot, please contact our legal team at [email protected].