What is Firecrawl, and why does it matter for AI?

The problem: the web is built for people

Language models such as ChatGPT, Claude or Gemini only know what was in their training data. Anything that happened afterwards is unknown to them. If you need current prices, a new law or today's manual for a piece of software, you have to give the AI a web page to read.

That is harder than it sounds. A typical web page is not just the text you read. It also contains menus, ads, cookie banners, footers and a lot of program code. Many pages only load their actual content afterwards using JavaScript, a programming language that runs in the browser. A simple program that only fetches the source code then often sees an almost empty page.

For an AI, this clutter is costly twice over. Models process text in small units called tokens, and every unnecessary menu costs computing time and money. Clutter can also make answers worse, because the model has to separate what matters from what does not.

Tools like Firecrawl address exactly this. They fetch a page, run the JavaScript like a real browser, strip out navigation and ads and return the core content as plain text. This is usually done in Markdown, a simple text format with headings and lists that AI models handle well.

What Firecrawl actually does

Firecrawl is primarily an application programming interface, or API. Developers send a request and get data back. According to the company, as of October 2026 it offers several core functions.

Search queries the web and can return the page content for each result straight away. Scrape reads a single address and returns text, HTML, a screenshot or structured data. Crawl follows a website's links and reads many subpages at once. Map only lists a site's addresses without reading them. With the Agent function you describe in plain language what data you need, for example the prices of all of a provider's plans, and Firecrawl searches and collects it on its own. Interact controls a browser that clicks, scrolls and fills in forms.

For AI assistants there is a so-called MCP server. MCP, the Model Context Protocol, is an open standard that lets AI programs use external tools. This allows Claude, ChatGPT, Codex or Cursor, for example, to search the web and read pages through Firecrawl without anyone writing their own code.

Since September 2026, Firecrawl has also bundled sources that do not come directly from the open web under the name Alexandria: official data interfaces, licensed publishers and its own indexes, for example of research papers and software documentation. According to the company, it pays some of these providers, such as Wikimedia Enterprise for Wikipedia data. According to the company, millions of requests for Wikipedia data alone run through Firecrawl every month.

Typical uses include chatbots that are meant to know a company's website, research agents, price monitoring and loading documentation into coding assistants.

Who is behind it: founders, stories and funding

Firecrawl was created by Caleb Peffer (CEO), Nicolas Silberstein Camara (CTO) and Eric Ciarla. According to TechCrunch, they founded their company in 2022. At the time they were building Mendable, an AI chat for software documentation, and took part in start-up accelerator Y Combinator's summer 2022 batch. By their own account they sold Mendable to customers such as Snapchat, MongoDB and DoorDash and realised along the way that cleanly reading web pages was the hardest part. Firecrawl grew out of that in spring 2024.

The small company first made headlines in early 2025 with an unusual job ad: on Y Combinator's job board it was looking to hire an AI agent as an employee, for 10,000 to 15,000 US dollars a year. When no suitable agent turned up, it raised the budget, according to TechCrunch, to one million US dollars for several agents and the developers who built them. Applications poured in, yet by August 2025 none had been hired. The founders found that vetting and managing such agents is a job in itself.

There are stories behind the money too. According to TechCrunch, Shopify CEO Tobias Lütke signed up as a customer through the website like anyone else. The team emailed him but got no reply. Two months later Shopify asked about an enterprise contract, and after a second email Lütke came on board as an investor. In August 2025 Firecrawl raised 14.5 million US dollars in a Series A round led by Nexus Venture Partners. At the time it cited 350,000 developers, was already profitable according to Peffer, and counted some of the world's largest hedge funds among its customers alongside Shopify, Replit and Zapier. In September 2026 a Series B of 75 million US dollars followed, led by Smash Capital. Firecrawl now speaks of more than 1.5 million users.

For context: all user and customer figures come from the company itself and have not been independently verified. Firecrawl has not published a valuation.

Costs, open source and free alternatives

Firecrawl bills in credits. Under the price list in effect since 4 September 2026, reading a page usually costs one credit and a search with up to ten results costs two. Without a credit card you get 1,000 free credits every month, enough for about 1,000 pages. Paid plans start at 16 US dollars a month when billed annually and go up to 599 US dollars for one million credits, plus VAT where applicable.

You do not have to use the cloud, though. The core is published on GitHub under the AGPL-3.0 licence and can be run with Docker on your own machines, including servers in Germany. There are no licence fees. In essence, the AGPL requires that if you offer a modified version to others over a network, you publish your changes as well. According to Firecrawl's own guide, however, the self-hosted version lacks quite a bit: reading pages, crawling websites, listing addresses and search work. The particularly stubborn anti-bot technology called Fire-engine is not included, screenshots and clicks do not work by default, and Agent and Interact are cloud-only. For AI features you need your own language model, for example via Ollama. Security, updates and backups are your job.

There are also free projects. Crawl4AI is a Python library under the Apache 2.0 licence with around 85,000 stars on GitHub that turns web pages into AI-readable text without keys or fees; proxies and bot blocking are up to you. Jina Reader, also Apache 2.0, returns a page's text when you put r.jina.ai in front of its address, but does not follow links. ScrapeGraphAI (MIT licence) pulls structured data out of pages based on a plain-language description and can use local models. Browser Use (MIT licence) lets an AI operate a browser by itself. Commercial competitors include Exa, Tavily, Apify, Browserbase and Bright Data.

Our assessment: almost every provider publishes comparisons in which it comes out on top. Such rankings are marketing. With Firecrawl, users mainly criticise the cost at very large volumes and the fact that heavily protected sites do not always work. The company's claim that Firecrawl covers 96 percent of the web has not been independently verified. Test any tool on the pages you actually need.

The sensitive side: robots.txt, server load and copyright

Website operators can use a file called robots.txt to specify which automated visitors may read what. It is a request, not a technical barrier. Firecrawl says it respects these rules by default, including rules aimed specifically at the name FirecrawlAgent. Critics on GitHub complained that operators cannot reliably recognise Firecrawl requests in their server logs because no fixed browser identifier is documented. In 2025 a user also pointed to a setting in the code that allows robots.txt to be bypassed for individual customer teams. How it is used today is not publicly documented. Firecrawl itself states that it is solely the users' responsibility to respect websites' policies.

Then there is server load. Many operators, especially small projects, complain about AI crawlers putting heavy strain on their servers. For this reason the network provider Cloudflare has blocked AI crawlers by default for new customers since July 2025 and is trialling pay-per-crawl models. Since 15 September 2026, operators there can also allow search engines while disallowing AI training, and new ad-supported domains get stricter default settings. AI crawlers therefore have a dubious reputation, partly because some simply ignore robots.txt. Firecrawl says it wants to pay website owners and publishers when AI uses their content, and in September 2026 announced a self-service system for this.

In the EU, copyright law permits the automated analysis of text, known as text and data mining, under certain conditions. Under Section 44b of the German Copyright Act, which implements Article 4 of the EU's DSM Directive, it is allowed for lawfully accessible content as long as the rights holder has not objected. For online content, such an objection only counts if it is machine-readable. In December 2025 the Hamburg Higher Regional Court ruled in Kneschke v. LAION that a notice in ordinary terms of use is not enough. robots.txt is seen as a typical example of a machine-readable reservation. Germany's Federal Court of Justice heard the case on 3 September 2026 (case I ZR 281/25) and, according to reports, plans to rule on 17 December 2026; a referral to the EU Court of Justice is also possible.

It is unclear whether an agent merely reading a page to answer a question counts as text and data mining at all. Anyone who redistributes content, for example by copying and publishing whole articles, needs a separate legal basis anyway. A website's terms of use can add further contractual limits.

Data protection, and what it means for you

Web pages often contain personal data: names, photos, comments, contact details. The EU General Data Protection Regulation (GDPR) applies to them even when they are publicly available online. Whoever collects such data with Firecrawl or a similar tool is usually responsible for it themselves, not the tool provider.

On 7 July 2026 the European Data Protection Board, the body of EU data protection authorities, adopted draft guidelines on web scraping for generative AI. Comments can be submitted until 30 October 2026. According to the draft, scraping is not banned in principle but usually relies on legitimate interest, which must be carefully weighed. Only necessary data may be collected, and particularly sensitive data such as health information enjoys special protection. Signals such as robots.txt or CAPTCHAs are meant to feed into the balancing test: ignoring them makes your case weaker.

Firecrawl is a US company, so your requests and the retrieved content pass through its servers. According to its pricing page, a data processing agreement is available from the Standard plan upwards, a feature for removing personal information is included in all plans, and large customers can opt for zero data retention. Whether that is sufficient for your situation is something you need to check yourself. If you run the open core or a free alternative yourself on servers in the EU, you avoid routing data through the US provider, but you remain just as responsible for the data you collect.

Our assessment: Firecrawl solves a real technical problem; without tools like it, AI agents could hardly use the current web. Legally and ethically, however, it is a tool like a browser or a photocopier: what matters is what you do with it. Reading individual pages you would be allowed to open anyway carries little risk. Anyone systematically collecting entire websites, processing personal data or reusing content should respect robots.txt and terms of use and seek legal advice if in doubt.

Disclosure: Simply Understood itself uses web research tools, including Firecrawl.