polite-crawler
LocalA polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
Installation
{
"mcpServers": {
"polite-crawler": {
"command": "npx",
"args": [
"polite-crawler"
]
}
}
}