scrapling-fetch-mcp

An MCP server that helps AI assistants access text content from websites that implement bot detection, bridging the gap between what you can see in your browser and what the AI can access.

Intended Use

This tool is optimized for low-volume retrieval of documentation and reference materials (text/HTML only) from websites that implement bot detection. It has not been designed or tested for general-purpose site scraping or data harvesting.

Note: This project was developed in collaboration with Claude Sonnets 3.7 and 4.5, using LLM Context.

Installation

Requirements

Python 3.10+
uv package manager

Install

# Install scrapling-fetch-mcp
uv tool install scrapling-fetch-mcp

# Install browser binaries (REQUIRED - large downloads)
uvx --from scrapling-fetch-mcp scrapling install

Important: The browser installation downloads hundreds of MB of data and must complete before first use. If the MCP server times out on first use, the browsers may still be installing in the background. Wait a few minutes and try again.

Setup with Claude Desktop

Add this configuration to your Claude Desktop MCP settings:

MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "scrapling-fetch": {
      "command": "uvx",
      "args": ["scrapling-fetch-mcp"]
    }
  }
}

After updating the config, restart Claude Desktop.

What It Does

This MCP server provides two tools that Claude can use automatically when you ask it to fetch web content:

Page fetching: Retrieves complete web pages with support for pagination
Pattern extraction: Finds and extracts specific content using regex patterns

The AI decides which tool to use based on your request. You just ask naturally:

"Can you fetch the docs at https://example.com/api"
"Find all mentions of 'authentication' on that page"
"Get me the installation instructions from their homepage"

Protection Modes

The tools support three levels of bot detection bypass:

basic: Fast (1-2s), works for most sites
stealth: Moderate (3-8s), handles more protection
max-stealth: Maximum (10+s), for heavily protected sites

Claude automatically starts with basic mode and escalates if needed.

Tips for Best Results

Just ask naturally - Claude handles the technical details
For large pages, Claude can page through content automatically
For specific searches, mention what you're looking for and Claude will use pattern matching
The metadata returned helps Claude decide whether to page or search

Limitations

Designed for text content only (documentation, articles, references)
Not for high-volume scraping or data harvesting
May not work with sites requiring authentication
Performance varies by site complexity and protection level

Built with Scrapling for web scraping with bot detection bypass.

License

Apache 2.0

Features

Rate this server

Features

Rate this server