AI Crawlers Visibility, Analytics & Control

Description

AI Crawlers Visibility, Analytics & Control helps WordPress site owners understand and manage how AI systems access their content. Monitor visits from GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and 78+ known AI crawlers, then choose which bots to monitor, discourage through robots.txt, or block with a PHP 403 response.

The plugin combines AI crawler analytics, AI bot control, and AI discoverability tools in one WordPress dashboard. It includes an AI Visibility Score, llms.txt and llms-full.txt, Markdown for Agents, JSON-LD, downloadable reports, email alerts, and an authenticated REST API.

It works alongside SEO and security plugins. It does not replace traditional SEO, a firewall, or server-level bot protection.

AI SEO, GEO, and AEO for WordPress

Traditional SEO helps search engines index a site. AI SEO, Generative Engine Optimization (GEO), and Answer Engine Optimization (AEO) focus on making content easier for AI crawlers and answer engines to discover and understand.

This plugin supports the technical foundations of AI visibility by helping you:

• Measure which AI crawlers access your content and which pages they request.
• Check discovery, structured-data, and content-readiness signals.
• Publish AI-readable discovery files and Markdown versions of public content.
• Provide machine-readable site identity and article information with JSON-LD.
• Control crawler access without changing the content human visitors see.

No plugin can guarantee inclusion, citations, rankings, or traffic from ChatGPT, Claude, Gemini, Perplexity, or other AI services.

Why monitor AI crawlers on WordPress?

AI companies use automated AI crawlers to index, train on, or retrieve website content. Common examples include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Bytespider (ByteDance), CCBot (Common Crawl), PerplexityBot, and many others.

With this AI crawlers plugin you can:

• See which AI crawlers visit your site and which pages they request.
• Compare AI bot traffic across date ranges and spot trends over time.
• Block AI crawlers with robots.txt rules or PHP-level 403 responses.
• Keep logs for compliance, debugging, and content licensing decisions.
• See a 7-day crawler summary on the WordPress admin dashboard.

How it works

1. Enable crawler tracking and choose how long activity logs should be retained.
2. Review crawler trends, top bots, most-crawled pages, and recent requests.
3. Use robots.txt or optional PHP blocking for crawlers you do not want accessing your content.
4. Run an AI Visibility Score scan and enable the discoverability features that fit your site.

AI crawler analytics dashboard

Understand which AI systems are interested in your site instead of relying on guesswork.

• Dashboard with total requests, unique AI crawlers, pages crawled, and blocked attempts.
• Charts for AI crawler requests over time and traffic by bot category.
• Top AI crawlers and most-crawled pages tables with period comparison.
• Recent AI crawler activity log with filters for bot, HTTP status, and content type.
• Download CSV analytics reports for preset or custom date ranges.
• Blocked-attempt logging when PHP blocking is enabled.

AI Visibility Score

Find technical signals that can limit AI discovery and get prioritized recommendations for improvement.

• Analyze any page or your entire site for AI readiness.
• 28 automated checks across discovery, structured data, content structure, and technical signals.
• 0–100 score with letter grade and prioritized recommendations.
• Site-wide batch scan, scan history, comparison diff, and PDF export.

AI discoverability: llms.txt, Markdown, and JSON-LD

• Auto-generate /llms.txt and optional /llms-full.txt for AI systems.
• Choose public post types, set URL limits, and include or exclude individual content.
• Exclude content by URL pattern and automatically exclude noindex content.
• Respect noindex metadata from Yoast SEO, Rank Math, All in One SEO, SEOPress, and Genesis.
• Regenerate discovery files when content is published or on a schedule.
• Optionally add LLMs: and LLMs-full: references to robots.txt.
• Serve .md URL endpoints for public content and taxonomy archives.
• Respond to Accept: text/markdown requests from compatible AI agents.
• Output Organization, WebSite, and optional Article JSON-LD with AI discovery pointers.

WooCommerce AI Search Optimization

When WooCommerce is active, optimize products for AI search engines:

• Include products in llms.txt with price, stock status, ratings, and attributes.
• Variable product price ranges and sale price display.
• Product Markdown endpoints with YAML commerce frontmatter (SKU, price, availability).
• Optional Product JSON-LD schema (skipped when major SEO plugins already output it).
• Product data panel tab: exclude from AI and set a short AI description.
• HTTP Link header pointing to /llms.txt (rel="ai-content-index").
• Product-focused Visibility Score checks (schema, SKU, price).

78+ built-in AI crawlers

• Pre-configured detection for major AI, search, and scraper bots.
• Category labels (training, search, assistant, agentic, scraper, and more).
• Per-crawler info with user-agent pattern details.
• Enable or disable monitoring for individual AI crawlers.
• Includes crawlers associated with OpenAI, Anthropic, Google, Perplexity, Meta, Amazon, Apple, ByteDance, Common Crawl, and others.

Custom AI crawler detection

• Add your own user-agent patterns for bots not in the built-in list.
• Assign company name and category for consistent reporting.

robots.txt control for AI crawlers

Publish clear crawler preferences and identify bots that continue visiting after being disallowed.

• Add Disallow rules for selected AI crawlers.
• Bulk templates to disallow or allow entire categories (training, scraper, and more).
• Compliance monitor shows top non-compliant crawlers that still visit.
• Rules are appended to your WordPress robots.txt output.
• Live preview before you publish changes.

Block AI crawlers with PHP (403)

robots.txt communicates a preference; PHP blocking enforces access control at the WordPress application level.

• Hard-block selected AI crawlers before WordPress serves a response.
• Block specific IP addresses with the same 403 PHP response.
• Customize the plain-text message returned to blocked bots (since 1.1.0).
• Blocked requests are logged in the AI crawler analytics dashboard.
• Works alongside robots.txt for stronger enforcement when needed.

Data management

Crawler activity is stored in your WordPress database, giving you control over retention and deletion.

• Configurable log retention (7 days to 1 year, or keep all data).
• Optional delete-all-data action from the admin.
• Delete plugin data on uninstall (optional setting).
• Multi-select crawler, category, HTTP status, and content-type activity filters.
• Download reports for preset or custom date ranges.

Reports and integrations

• Daily, weekly, or monthly HTML email reports with configurable recipients and alert types.
• Email summaries cover crawler activity, traffic changes, blocked attempts, new crawlers, and robots.txt non-compliance.
• Test email delivery and view the next scheduled report from the Reports page.
• Authenticated REST API for analytics summaries, recent activity, CSV exports, visibility scans, and blocked attempts.
• Professional report emails include key metrics, the top crawler, and most-crawled pages.

Clean admin experience

• WordPress admin menu: AI Crawlers
• Redesigned UI with grouped sidebar navigation and dark mode.
• Dashboard, Crawlers, Control, Visibility Score, Reports, and Settings in one place.
• Built for site administrators — no code required.

Who is this plugin for?

• Bloggers and publishers who want to know when AI crawlers access their articles.
• Business and marketing sites tracking AI bot traffic to key landing pages.
• Developers and agencies managing AI crawler policy across client WordPress sites.
• Site owners exploring robots.txt and PHP options for AI crawler control.

Privacy and data ownership

• AI crawler activity logs are stored locally in your WordPress database.
• The analytics dashboard tracks detected crawler requests, not general human visitor analytics.
• Log retention and deletion are controlled from the plugin settings.
• Scheduled reports are sent through the standard WordPress wp_mail() function.
• Core crawler monitoring does not require a separate analytics account.