Skip to content
OpenSourceSoftware.ai
Scrapling logo

Scrapling

D4Vinci · Data engineering

GitHub stars

90,186
—Awaiting first sync

Awaiting first sync

No measured star history yet

What is Scrapling?

Scrapling puts web scraping within reach from three directions: a terminal command for one-off pages, an MCP server that hands the same tools to an AI agent, and a Python framework for full crawls. Each uses the same fetchers and parser, so a selector worked out in the shell carries over to the agent or the spider.

Scrapling is an open-source web scraping toolkit with a command line, an interactive scraping shell and an MCP server for AI agents. It fetches pages over fast HTTP or a real browser, extracts data with CSS, XPath or text search, and saves pages as Markdown, text or HTML. A Python library and concurrent spiders cover larger crawls.

Core capabilities

  • Save any page as Markdown, text or HTML from the terminal without writing code.
  • Let Claude, Cursor and other agents browse and extract data through its MCP server.
  • Grow into Python spiders with sessions, proxy rotation, pause and resume, and exports.

Scrape from the terminal

The scrapling extract commands save a page straight to a Markdown, text or HTML file, with options to pick content by CSS selector, wait for an element or render the page in a browser. An optional interactive shell built on IPython helps develop selectors, converts copied curl requests into Scrapling requests and opens results in your browser.

  • CLI
  • Interactive shell
  • Markdown

A scraping tool for AI agents

The MCP server lets Claude, Cursor and other MCP clients scrape through Scrapling with HTTP requests, browser actions, screenshots, saved login state and captured API responses. It trims and cleans page content before sending it to the model, blocks ads and includes protections against prompt injection. An Agent Skill teaches coding agents the current library API.

  • MCP
  • AI agents
  • Agent Skill

From one page to a full crawl

In Python, fetchers cover fast HTTP requests and full browser automation with persistent sessions and proxy rotation. Elements can be found by CSS, XPath, text or regex and tracked again after a site changes. Spiders add concurrency, per-domain throttling, optional robots.txt compliance, pause and resume, and JSON, JSONL, CSV or XML export. A Docker image ships with the browsers installed.

  • Python
  • Web crawling
  • Data extraction
  • Docker

Where it fits

Use cases

  1. 01

    Grab a page without writing a scraper

    Run scrapling extract with a URL and an output file, pick the content with a CSS selector, and get Markdown, text or HTML. Switch from get to fetch when the page needs a browser to render.

  2. 02

    Give an AI assistant a scraping tool

    Install the MCP extra and add the scrapling-mcp command to Claude Desktop, Claude Code or Cursor. The agent can then fetch pages, take screenshots and pull out the data you ask for.

  3. 03

    Build a dataset from a whole site

    Start in the interactive shell to test selectors, then move them into a spider that crawls concurrently, can obey robots.txt, resumes after a stop and exports JSON, JSONL or CSV.