Skip to content
OpenSourceSoftware.ai
Crawl4AI logo

Crawl4AI

unclecode · Data engineering

GitHub stars

86,461
—Awaiting first sync

Awaiting first sync

No measured star history yet

What is Crawl4AI?

Crawl4AI is built for the step before an AI feature can use the web: getting pages into a form a model can read. It drives real browsers, cleans what they return into Markdown or structured data, and runs on your own machine, so teams keep control of what is crawled, how fast and through which proxies.

Crawl4AI is an open-source web crawler and scraper that turns websites into clean, LLM-ready Markdown for RAG, AI agents and data pipelines. Use it as a Python library, from the crwl command line, or as a self-hosted Docker server with a REST API, MCP, a monitoring dashboard and a playground. A hosted Crawl4AI Cloud is also available.

Core capabilities

  • Convert pages into clean Markdown with headings, tables, code and cited links kept.
  • Extract structured JSON with CSS or XPath schemas, or with an LLM of your choice.
  • Run your own server with a REST API, MCP, a monitoring dashboard and a playground.

Markdown and structured data from any page

Pages come back as Markdown that keeps headings, lists, tables and code, with links turned into numbered citations. Filters can strip menus and footers or keep only what matches a query. For structured data, CSS, XPath and regex schemas work without an LLM, and LLM extraction fills a typed JSON schema with the provider you choose.

  • Markdown
  • Data extraction
  • RAG

Crawl dynamic sites with a real browser

Deep crawls use breadth-first, depth-first or best-first strategies and can resume after a crash. Browser sessions keep logins and cookies across steps, run JavaScript, scroll infinite pages and take screenshots or PDFs. Proxies, stealth mode and Chromium, Firefox or WebKit are configurable for sites that are hard to reach.

  • Web crawling
  • Browser automation
  • Playwright

Library, command line, server or cloud

Install it with pip and call it from Python, or use the crwl command to fetch a page, deep crawl a site or ask a question about a page. The Docker image adds a token-protected REST API, an MCP endpoint for agents, a monitoring dashboard and a playground. Crawl4AI Cloud is a paid hosted option with a free monthly allowance.

  • Self-hosted
  • Docker
  • CLI
  • MCP

Where it fits

Use cases

  1. 01

    Feed a documentation site into a RAG system

    Deep crawl a docs site breadth first, keep the main content as Markdown, and index the pages in a vector database so an assistant can answer from them.

  2. 02

    Give an AI agent web access

    Run the Docker server with your own API token and connect Claude Code or another agent through MCP, so it can fetch pages, screenshots and PDFs from your machine.

  3. 03

    Collect product or listing data

    Describe the fields once, generate a CSS or XPath schema, and extract the same JSON from many pages without calling an LLM for every page.