How to Scrape Any Website with Claude Code Using Firecrawl CLI

TL;DR: Claude Code’s built-in web fetch fails on sites with bot detection or heavy JavaScript. Install Firecrawl CLI (one command) and Claude can scrape anything - Amazon, LinkedIn, directories, job boards. Faster and more token-efficient. 6 copy-paste scraping prompts below.

If you’ve tried using Claude Code for web scraping, you’ve hit the wall. Bot restrictions, JavaScript rendering failures, pages that return garbage. Something as simple as scraping an Amazon product page becomes a multi-hour headache.

The built-in web fetch works fine for simple, static pages. But the modern web runs on JavaScript, and most valuable data lives behind bot detection. That’s where Firecrawl comes in.

What Firecrawl Fixes

Firecrawl is a web scraping tool that just launched a CLI that plugs directly into Claude Code. It handles the two things that break normal web fetch:

Bot detection bypass. Sites like Amazon, LinkedIn, and most directories block automated requests. Firecrawl gets around this so Claude can access the data.

JavaScript rendering. Dynamic pages that load content via JavaScript return empty or broken results with normal web fetch. Firecrawl renders the JavaScript first, then returns the clean content.

It’s also faster and more token-efficient than the built-in web fetch, which means you’re not burning through your usage on failed requests and retries.

Setup (2 Minutes)

1. Get a Firecrawl API key. Go to firecrawl.dev and create a free account. The free tier gives you 500 credits per month.

2. Install the CLI. Run: npm install -g firecrawl-cli

3. Add your API key. Set FIRECRAWL_API_KEY as an environment variable in Claude Code.

4. Test it. Ask Claude to scrape any website using Firecrawl. If it returns clean data, you’re set.

6 Scraping Prompts

Competitor Pricing: “Use Firecrawl to scrape [competitor URL]. Find their pricing page and extract all plan names, prices, features included, and any limits. Organize into a comparison table and save to a file on my desktop.”

Lead List from a Directory: “Use Firecrawl to scrape [directory URL]. Pull the business name, phone number, email, website URL, and address for each listing. Save to a CSV file called leads.csv. Get the first 50 results.”

Amazon Product Research: “Use Firecrawl to scrape this Amazon search results page: [URL]. For each product, pull the product name, price, rating, number of reviews, and ASIN. Save to a spreadsheet. Top 20 results.”

Job Posting Data: “Use Firecrawl to scrape [job board URL]. Pull the job title, company name, location, salary range, and posting date for each listing. Save to a CSV. Filter for [specific role].”

Content Research: “Use Firecrawl to scrape [blog or news site URL]. Pull the article titles, publish dates, authors, and first 200 words of each post from the last 30 days. Save to a file for content research.”

Real Estate Listings: “Use Firecrawl to scrape [real estate site URL] for listings in [city]. Pull the address, price, bedrooms, bathrooms, square footage, and listing URL. Save to a spreadsheet sorted by price.”

Tips

Always say “use Firecrawl” in your prompt so Claude uses it instead of built-in web fetch
Specify exact data fields you want extracted
Use CSV format for spreadsheet-ready output
Scrape in batches (“first 50 results”) rather than trying to get everything at once
Add delays between requests if a site still blocks you
Firecrawl is open source - you can self-host for unlimited free scraping

FAQ

Is Firecrawl free? The free tier gives you 500 credits per month. Each page scrape costs roughly 1 credit. For individual use and testing, free is enough. Paid plans remove limits.

Is this legal? Web scraping publicly available data is generally legal. Don’t scrape personal data, copyrighted content, or data behind logins you don’t own. Check the site’s terms of service.

Can it scrape sites that require login? The basic CLI works on public pages. For authenticated scraping, you’d need to pass session cookies or use Firecrawl’s more advanced features.

How does this compare to browser automation (Playwright, Puppeteer)? Firecrawl is simpler - no browser setup, no selectors, no scripts. You just tell Claude what data you want and it handles extraction. Browser automation gives more control but requires more technical setup.