crawl-readiness-mcp

MCP server that checks whether a website is set up for AI the way its owner intends: can GPTBot, ClaudeBot, PerplexityBot and 50+ other AI crawlers read it, and what does its access policy allow? Generates llms.txt, an AI-crawler robots.txt and JSON-LD, reads your Answer Analysis, and checks whether a page permits an AI use.

Installation

MCP client config (paste into mcp.json)

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "crawl-readiness-mcp"
      ]
    }
  }
}

Agent install prompt

Copy and paste into Cursor / Claude / other agents to install this MCP for you

Please read this MCP repository and install it into my current environment:
https://github.com/Grubbomatic/crawl-readiness-mcp

Known working client config (can write into mcp.json):
{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": [
        "crawl-readiness-mcp"
      ]
    }
  }
}

Please: 1) check README for deps and env vars; 2) write config for my client; 3) if secrets are missing, tell me what to fill — do not invent values.

GitHub

Related servers