Allow‑list MediaThriveBot behind Cloudflare/WAF

Allow‑listing MediaThriveBot when you use Cloudflare “Under Attack” mode or a WAF

When Cloudflare (or another WAF) challenges every request, legitimate crawlers can be blocked.
Follow this guide to let MediaThriveBot reach your site safely while keeping full protection for everyone else.

1  Why you need this

• MediaThriveBot pulls your public articles, podcasts, or other assets so they appear in MediaThrive.

• If Cloudflare’s Under Attack Mode, Bot Fight Mode, or a strict WAF rule is enabled, the bot will receive JavaScript or CAPTCHA challenges that it cannot solve.

• By adding a lightweight allow‑list rule you bypass only traffic that (a) presents our unique User‑Agent string and(b) includes your private verification token in an HTTP header. Everything else is still fully protected.

2  Grab your Verification Token

1. Sign in to MediaThrive.

2. Go to Team ▸ Settings.

3. In the Development panel copy the Verification Token (click the clipboard icon).

4. Keep the token secret; treat it like a password.
   You can regenerate it at any time — just remember to update your Cloudflare rule afterward.

Image

3  How MediaThriveBot identifies itself

MediaThrive always sends both of these on every crawl request.

4  Create a Cloudflare Custom Rule

Cloudflare’s WAF Custom Rules let you skip specific protections when a condition matches (recommended over simple User‑Agent allow‑lists). (Cloudflare Docs, Cloudflare Docs)

1. Log in to the Cloudflare dashboard and select your domain.

2. Go to Security ▸ WAF ▸ Custom Rules and click Create rule.

3. Give the rule a clear name, e.g. “Allow MediaThriveBot”.

4. In Rule expression, paste:

   text
   Copy
   (http.user_agent contains "MediaThriveBot") and
   (http.request.headers["X-MediaThrive-Token"] eq "YOUR_TOKEN_HERE")
   
   

5. Choose action → Skip and tick:

   • WAF Managed Rules

   • Bot Fight Mode / Super Bot Fight Mode

   • Rate Limiting / DDoS (as needed)

   (If “Skip” is unavailable on your plan, select Allow instead.)

6. Save and deploy the rule. Place it high in the rule‑order list to ensure it runs before generic blocking rules.

Result: Only requests that match both the special User‑Agent and your unique token bypass Cloudflare challenges; every other crawler is still filtered.

5  Testing

1. From a terminal or script, run:

   bash
   Copy
   curl -I https://yourdomain.com \
     -A "Mozilla/5.0 (compatible; MediaThriveBot/2.0; +https://mediathrive.com/bot)" \
     -H "X-MediaThrive-Token: YOUR_TOKEN_HERE"
   
   

2. You should receive an HTTP 200 (or 304/301) without a Cloudflare challenge page.

3. Remove the header or change the token to confirm Cloudflare blocks the request.

6  Using other WAFs / Firewalls

7  Regenerating or revoking a token

1. In Team ▸ Settings click the ↻ icon next to Verification Token.

2. Update your Cloudflare (or WAF) rule with the new value.

3. Old tokens stop working immediately.

8  FAQ

What if I only add a User‑Agent allow‑list?
That works, but anyone can fake a User‑Agent. The extra token header proves the request really comes from MediaThrive.

Can I limit by IP instead?
Our crawler is distributed across multiple regions for resilience. Header‑based allow‑listing is simpler and stays up‑to‑date automatically.

Does this weaken my security?
No. The rule skips protection only for authenticated MediaThrive traffic. All other visitors (including malicious bots) still go through your usual Cloudflare defenses.

Add the Settings screenshot right after Section 2 so users can quickly locate the Verification Token field.