The first time I used Firecrawl I understood immediately why it had become the default web-to-LLM pipeline for developers building RAG applications. You give it a URL. You get back clean markdown. No navigation, no cookie banners, no ad scripts, no boilerplate. Just the content. Ready for embedding, chunking, indexing, or feeding directly into a language model.
The experience was genuinely impressive. Firecrawl had clearly invested in solving the hardest parts of web extraction for AI use cases — JavaScript rendering, anti-bot handling, content isolation, output normalisation. The product worked.
Then I looked at the pricing. The Hobby plan was £19 per month for 500 credits. The Standard plan was £49 for 3,000. The Growth plan was £99 for 10,000. For a developer building a RAG pipeline that needed to scrape several hundred URLs from documentation sites, news sources, and research papers, the numbers added up quickly. For a solo founder or indie hacker building something on a bootstrap budget, £99 per month for web-to-text conversion is a significant line item.
I also noticed that Firecrawl's own benchmarks — published in their documentation and referenced in independent comparisons — showed the tool achieving successful extraction on about 34% of heavily protected sites. Better than alternatives, but a meaningful failure rate on the commercial and media sites where the most valuable content often lives.
The LLM-Ready Web Scraper was built with a different set of assumptions.
First, no subscription. Pay per URL. The base rate is £0.005 per URL crawled — a hundred URLs costs fifty pence. An AI developer building a knowledge base from 1,000 documentation pages pays £5.00 for the extraction layer. Not £99 per month whether they use it or not.
Second, vertical processing. Generic web extraction treats all content identically — a legal case, a product listing, a research paper, and a news article get the same extraction logic. But these content types have completely different structures. A legal document's value is in the structured hierarchy of clauses and precedents. A medical article's value is in the clinical claims and the supporting evidence. A property listing's value is in the specific details — bedrooms, price, location — that the standard extraction would bury in surrounding text.
The actor includes seven vertical processing modes: General, Legal, Medical, Property, E-commerce, Research, and News. Each mode applies domain-specific CSS selector priorities and content isolation rules that improve extraction quality for that content type. A legal document processed in Legal mode prioritises judgment body selectors and preserves structural hierarchy. A property listing processed in Property mode hunts for price, bedroom count, and location fields specifically.
Third, RAG-native output. The actor returns not just extracted text but a pre-computed token estimate — how many tokens this content will consume in an LLM context window — and optional text chunking, where the content is split into configurable token-sized segments ready for vector embedding. You do not have to run a second pass to chunk your content before indexing it. The extraction and chunking happen in the same actor run.
The use case that prompted this build was a RAG pipeline I was building for a client who needed their AI assistant to answer questions from a knowledge base assembled from about 400 URLs across four different websites. They wanted clean markdown out of each URL, chunked at 512 tokens, with token counts included so they could predict their embedding API costs before committing.
With Firecrawl's Standard plan, the extraction alone would have cost £49 per month. The chunking would have been a separate step in the pipeline. The token estimation would have been another step. With the LLM-Ready Web Scraper, the entire operation — extraction, chunking, token estimation — runs in a single actor call for approximately £2.00 in PPE.
Independent benchmarks have tested similar tools on scrapy extraction accuracy across protected sites and found significant variation. The actor's use of Playwright with stealth options handles JavaScript-heavy pages that fail on simpler HTTP fetch approaches. The fallback to Cheerio for simpler pages reduces execution time and cost on static content that does not require a full browser.
The developer market for affordable, scriptable, non-subscription web-to-LLM pipelines is large and underserved. Firecrawl is excellent at what it does. But not every developer building a RAG application needs a £99/month subscription. Some need a tool that charges by the URL and gets out of the way.
That is what this actor is.