Select one or more tags, then press “Search Plugins”

Find Plugin with any / all of the selected criteria
Search Plugin

All-IN Crawler & AI Agent Manager Wordpress Plugin - Rating, Reviews, Demo & Download

All-IN Crawler & AI Agent Manager Preview Wordpress Plugin - Rating, Reviews, Demo & Download
No ratings yet
Free
Follow for free plugins, new theme releases and theme news

Plugin Description

Much of your traffic is bots. Some bring you customers: Google, Bing, ChatGPT search, social previews. Others only cost you server resources: aggressive AI training crawlers, SEO tools, scrapers and vulnerability scanners. All-IN Crawler & AI Agent Manager lets you decide, per category or per bot, who gets in and how fast, without hiding your site from search engines or AI assistants.

  • About 120 known bots in 12 categories: search engines, AI search & assistants, AI training crawlers, SEO tools, social previews, monitoring, feed readers, archives, scripts & HTTP libraries, vulnerability scanners, unidentified bots and empty User-Agents.
  • Allow, rate-limit or block per category, with per-bot overrides and custom User-Agent rules (text or regex).
  • One-click presets: Balanced, Maximum AI visibility, Protect content from AI training, Lockdown.
  • IP / CIDR allow and block lists (IPv4 and IPv6), and excluded paths for webhooks and payment callbacks.
  • Protect content from AI training, stay in AI answers: training crawlers (GPTBot, ClaudeBot, CCBot…) are handled separately from AI search and assistant agents (OAI-SearchBot, ChatGPT-User, Claude-User, Perplexity-User…).
  • Real search engines only: forward-confirmed reverse DNS plus optional official crawler IP lists (Google, Bing, Apple, OpenAI, Perplexity; off until you turn them on) catch fake Googlebots and GPTBots.
  • Overload protection: rate limits per IP or per bot with 429 + Retry-After, load shedding with 503 + Retry-After when server load per CPU core crosses your threshold, and an optional per-IP limit for “human” visitors (logged-in users are exempt). SEO-safe status codes only, never cached.
  • robots.txt and llms.txt: manage robots.txt rules (including Google-Extended, Applebot-Extended and an optional Crawl-delay) and generate an llms.txt that skips noindex content.
  • Block before PHP: export rules to .htaccess (Apache/LiteSpeed) or nginx. Inside WordPress, requests are decided while plugins load, before the theme runs.
  • Dashboard and reports: 7/14-day statistics, top denied IPs with one-click blocking, a denied-request log, a User-Agent tester and weekly or monthly email reports.
  • Optional bot-list feed that adds new AI crawlers between releases (add-only and safety-checked). Off by default, and it only downloads from a URL you enter.
  • Fast storage: APCu, then Redis/Memcached, then the database.
  • Fails open: if anything goes wrong inside the plugin, visitors are let through and the error is logged.
  • Full Hebrew translation, RTL ready.

Full documentation: see docs/USER-MANUAL.md (site owners) and docs/DEVELOPER.md (hooks & architecture).

More from A-in

This plugin is complete on its own: no licence key, no locked features. Premium plugins, add-ons and support from the same developer are available at a-in.co.il.

External services

All-IN Crawler & AI Agent Manager works without any external service. These features contact third parties:

  • Official crawler IP lists (Bot Lists tab “Download official crawler IP lists”, off by default, opt-in). Nothing is downloaded until you tick the box and save; the “Download official IP lists” button is also disabled until then. Once enabled, the site downloads the public IP range files published by the crawler operators once a day, to verify that a request claiming to be their bot really comes from them. No data about your site or visitors is sent: it is a plain HTTPS GET with a generic all-in-crawler-ai-agent-manager/<version> User-Agent, so the publisher only sees your server’s IP address. Untick the box to stop the downloads at any time. Sources:
    • Google: https://developers.google.com/static/search/apis/ipranges/ (Terms, Privacy)
    • Bing: https://www.bing.com/toolbox/bingbot.json (Terms, Privacy)
    • Apple: https://search.developer.apple.com/applebot.json (Terms, Privacy)
    • OpenAI: https://openai.com/gptbot.json, searchbot.json, chatgpt-user.json (Terms, Privacy)
    • Perplexity: https://www.perplexity.com/perplexitybot.json, perplexity-user.json (Terms, Privacy)
  • Bot-list feed (Lists Bot list updates, off by default). The plugin ships its extra bot definitions in data/bots.json and needs no download. If you enter a feed URL of your own (for example your agency’s feed), the site requests that URL once a day, sending your site URL in the User-Agent and, if you set one, your access key as a Bearer token. There is no default remote feed.
  • DNS verification of search engine bots uses reverse and forward DNS lookups through your server’s resolver. No third-party API is called.

Screenshots

  1. Dashboard with statistics per category and bot, server load and top denied IPs.

    Dashboard with statistics per category and bot, server load and top denied IPs.

  2. Bot rules with one-click presets, category defaults and per-bot overrides.

    Bot rules with one-click presets, category defaults and per-bot overrides.

  3. Protection settings: monitor or enforce mode, fake search-engine protection and server-load shedding.

    Protection settings: monitor or enforce mode, fake search-engine protection and server-load shedding.

  4. User-Agent tester and settings export / import.

    User-Agent tester and settings export / import.

  5. robots.txt management with a live preview of the added lines.

    robots.txt management with a live preview of the added lines.

Similar Plugins:

Rating Widget: Post Rating, 5 Star Rating, Reviews, Thumbs Up & Down, Reaction Dreams Agent Ready – AI Agent Readiness & LLM Optimization WordPress Plugin Default ThumbnailReviews And Rating – Google Reviews

Reviews & Comments