I do a fair amount of scraping in Laravel, and it always splits into two worlds. Guzzle for the pages that are just HTML, and a mess of Puppeteer scripts shelled out from PHP for the ones that render in JavaScript. Two toolchains, two mental models, glue in between.
Larascraper collapses that into one. You write a class, you describe what you want, and it runs through a real headless browser or plain HTTP with the same code. This post walks through the whole thing.
How to Install
composer require edulazaro/larascraper
The browser driver drives Puppeteer, so it pulls a headless browser the first time you scrape a JavaScript site. The HTTP driver needs nothing extra.
A scraper is a class
You extend Scraper and implement handle(). Inside it, $this->scrape($url) starts a fetch, and a terminal decides what comes back.
use EduLazaro\Larascraper\Scraper;
use EduLazaro\Larascraper\ScraperResponse;
class BikeScraper extends Scraper
{
protected function handle(string $url): ScraperResponse
{
return $this->scrape($url)->run();
}
}
Call it from anywhere:
$response = BikeScraper::run('https://example.com/bikes/4');
$response->data; // the raw HTML
$response->success; // true or false
$response->error; // the error message if it failed
BikeScraper::run() always hands you a ScraperResponse; with no crawler, $response->data is the raw HTML. That is rarely what you want. You want structured data, and for that you write a Crawler.
Crawlers turn a page into data
A Crawler gets a small query builder over the DOM via $this->filter():
use EduLazaro\Larascraper\Crawler;
class BikeCrawler extends Crawler
{
protected function handle(): array
{
return [
'name' => $this->filter('h1.title')->text(),
'price' => $this->filter('.price')->text(),
'specs' => $this->filter('.spec')->each(fn ($node) => $node->text()),
];
}
}
Chain it onto the fetch and you get the parsed array back:
protected function handle(string $url): ScraperResponse
{
return $this->scrape($url)->crawl(BikeCrawler::class)->run();
}
BikeScraper::run('https://example.com/bikes/4')->data;
// ['name' => 'Black Eagle', 'price' => '2,499', 'specs' => [...]]
The input is generic. Usually it is an HTML string, but $this->filter($selector, 'xml') treats it as XML, and $this->raw() gives you the untouched input for when you would rather reach for regex, simplexml or json_decode.
Driving a JavaScript page
A lot of the web is empty without JavaScript, so Larascraper runs a real browser by default and lets you act on the page before reading it: click, type, wait, scroll.
protected function handle(string $url): ScraperResponse
{
return $this->scrape($url)
->type('#search', 'black eagle')
->press('Enter', waitForNavigation: true)
->waitForSelector('.results')
->crawl(BikeCrawler::class)
->run();
}
The actions run in the browser, in order, and the crawler sees the final DOM. There is a full set: click(), select() (one value or an array for multi-selects), check() and uncheck() for checkboxes hidden inside widgets, hover(), scroll(), waitForSelector(), waitForNavigation().
When a page is plain HTML, skip the browser entirely:
$this->scrape($url)->driver('http')->run();
Same shape, no browser. Actions need the browser, so mixing them with the HTTP driver throws.
Waits that do not fail the run
By default a wait that times out fails the scrape, which is usually correct. But an empty result set is a valid outcome, not an error. Make the wait optional:
->waitForSelector('.results', ['optional' => true, 'timeout' => 8000])
Now an empty search continues instead of blowing up, and the crawler just returns nothing. You can also race several selectors and take whichever lands first:
->waitForSelector(['.results', '.no-results'])
Retry until it works
Captchas, sessions that need warming up, flaky pages: some flows only work on the second or third try. repeatUntil() loops a block until a condition holds, always bounded so it can never hammer a server:
use EduLazaro\Larascraper\Support\Condition;
$this->scrape($url)
->repeatUntil(
Condition::selectorMissing('#captcha'),
fn ($b) => $b
->solveCaptcha('#captcha-img', '#captcha-input')
->clickAndWait('#verify'),
max: 5,
delay: 1500,
)
->crawl(ResultCrawler::class)
->run();
A throw inside one attempt counts as a failed attempt, not a dead run: the loop re-checks the condition and tries again, up to max. A genuine misconfiguration, like an unknown captcha solver or a 4xx from a bad API key, aborts immediately instead of burning retries.
Solving Captchas
Simple image captchas go through OCR, no API cost:
->solveCaptcha('#captcha-img', '#captcha-input')
The OCR packages are optional; install them once with php artisan larascraper:install --captcha. For the distorted ones that tesseract cannot read, there is an OpenAI vision solver you opt into per call:
->solveCaptcha('#captcha-img', '#captcha-input', ['solver' => 'vision'])
It reads OPENAI_API_KEY from the environment. OCR stays the default, so you only pay for the hard ones.
Downloading files
For a PDF or any binary produced by a click or a form submit, use capture():
$response = $this->scrape($url)
->clickAndWait('a.download')
->capture(['expect' => 'application/pdf'])
->file();
$response->file; // the bytes
$response->contentType; // application/pdf
Using Spiders and concurrency
One scraper fetches one page. For thousands, you write a Spider. It has a handle() where you call your scrapers, and pool() runs them concurrently:
use EduLazaro\Larascraper\Spider;
class CatalogSpider extends Spider
{
protected int $concurrency = 20;
public function handle(): array
{
$ids = range(1, 5000);
return $this->pool(
$ids,
BikeScraper::class,
fn ($response, $id) => $response->success ? $response->data : null,
$this->concurrency,
);
}
}
$bikes = array_filter(CatalogSpider::run());
PHP is single-threaded, so this leans on Fibers. Each scraper suspends when it fetches, the Spider batches all the in-flight requests into one wave, sends them together, then resumes each Fiber with its response. Real concurrency lives on the HTTP driver, and a shared cookie Session threads through every request so a login survives the whole crawl.
Wrapping up
That is the entire surface: a Scraper to fetch, a Crawler to parse, browser actions when the page needs them, a Spider when one page is not enough. The same class shape whether you go through a browser or plain HTTP.
👉 Package on Packagist.
👉 Source on GitHub.
Written against version 3.3. If you try it and something is missing, open an issue.
Top comments (0)