Skip to content
OpenSourceSoftware.ai
Firecrawl logo

Firecrawl

firecrawl · Data engineering

GitHub stars

197,051
—Awaiting first sync

Awaiting first sync

No measured star history yet

What is Firecrawl?

Most of the web is built for browsers, not for software that needs to read it. Firecrawl takes a URL and returns the readable content as Markdown or structured data, which suits search, chatbots and data pipelines. The open-source version runs on your own servers, while some features exist only in Firecrawl's hosted cloud.

Firecrawl is an API that scrapes a web page, crawls a whole site or maps its URLs, and returns clean Markdown, HTML or structured JSON. You can run the open-source version on your own server with Docker Compose and call it from SDKs, the command line or an MCP server.

Core capabilities

  • Scrape any URL to Markdown, HTML or schema-based JSON with one API call.
  • Crawl a site with path filters and depth limits, or map every URL on it.
  • Self-host with Docker Compose and call it from SDKs, the CLI or an MCP server.

Scrape, crawl and map

Scrape turns one URL into Markdown, HTML or JSON that follows a schema you define. Crawl follows links from a start page as a background job you poll, with include and exclude path patterns, a discovery depth limit, and robots.txt respected. Map lists the URLs on a site without scraping them. Each is a REST endpoint, with SDKs listed for Python, Node, Go, Java, Ruby, Rust, .NET, PHP and Elixir.

  • Scrape
  • Crawl
  • Map
  • Markdown
  • SDKs

Self-hosting and what stays in the cloud

Clone a tagged release, create a small .env file and run docker compose up --build -d; the API listens on port 3002. The stack has the API, workers, Playwright, Redis, RabbitMQ and PostgreSQL. It starts without authentication and without persistent volumes, so add both before real use. The main code is AGPL-3.0 and the SDKs are MIT. Cloud-only: Fire-engine anti-bot, screenshots and page actions, Interact, Agent, Browser, and enterprise controls. JSON extraction needs your own OpenAI-compatible or Ollama model.

  • Docker Compose
  • AGPL-3.0
  • Self-hosting
  • Cloud-only limits

Connect it to AI tools

Firecrawl's docs describe a local MCP server, firecrawl-mcp, that can be pointed at a self-hosted instance with FIRECRAWL_API_URL. It runs over stdio by default or as a local streamable HTTP server, and needs an API key only if your instance requires one. A command-line tool, firecrawl-cli, is also available for scripts.

  • MCP
  • AI agents
  • CLI

Where it fits

Use cases

  1. 01

    Feed company pages to an AI assistant

    Send each URL to the scrape endpoint with the markdown format and use the result as clean text in your retrieval pipeline. Add the json format with a schema when you want fields such as price or title instead of a page.

  2. 02

    Copy a documentation site into a knowledge base

    Start a crawl at the docs URL, use includePaths and excludePaths to stay inside the sections you want, and set a depth limit. Poll the job until it completes, then store each page's Markdown.

  3. 03

    Give a coding agent web access

    Run the firecrawl-mcp server from Firecrawl's docs with FIRECRAWL_API_URL pointing at your own instance. Claude, Cursor and other MCP clients can then scrape pages through your server.