Scraping a Website Using a Symfony Console Command (Clean & Production-Friendly)

Marvelous Akpotu — Wed, 25 Feb 2026 17:48:35 +0000

Web scraping doesn’t belong in controllers.

It’s long-running.
It may fail.
It’s often scheduled.
It’s automation.

That’s exactly why Symfony Console Commands are perfect for it.

Symfony Console Commands allow you to create custom CLI tasks that run inside your Symfony application with full access to its services and dependency injection container. They are ideal for the following:

Background jobs
Automation
Data processing
Any long-running operations that shouldn’t live inside controllers.

In this article, we’ll:

Scrape country data
Parse HTML with DomCrawler
Sort results
Display a clean CLI table

This is the GitHub repo to follow along: https://github.com/Marvelxy/symfony-web-scraper

We’ll use a free scraping sandbox from www.scrapethissite.com :

https://www.scrapethissite.com/pages/simple/

Required Packages

If you don't already have them:

composer require symfony/http-client
composer require symfony/dom-crawler
composer require symfony/css-selector

Create the Command

php bin/console make:command app:create-countries

Now, let’s focus only on the parts that matter.

Inject the HttpClient

Instead of manually creating a client, inject it:

use Symfony\Contracts\HttpClient\HttpClientInterface;

public function __construct(
    private HttpClientInterface $client
) {}

Clean, testable, and native Symfony DI.

Fetch the Page

$response = $this->client->request('GET', self::URL);
$html = $response->getContent();

This gives us the raw HTML.

Parse with DomCrawler

use Symfony\Component\DomCrawler\Crawler;

$crawler = new Crawler($html);

Now, we can extract the block of each country:

$countryInfo = [];

$crawler->filter('.country')->each(function (Crawler $row) use (&$countryInfo) {
    $countryInfo[] = [
        $row->filter('.country-name')->text(),
        $row->filter('.country-capital')->text(),
        $row->filter('.country-population')->text(),
        $row->filter('.country-area')->text(),
    ];
});

If you’ve used JavaScript’s querySelectorAll(), this will feel familiar.

Sort the countries alphabetically

usort($countryInfo, function ($a, $b) {
    return strcasecmp($a[0], $b[0]);
});

Format Output Like a Pro

Instead of dumping raw arrays, format the output properly in a table

printf(
    "%-45s | %-20s | %15s | %15s\n",
    "Country name",
    "Capital",
    "Population",
    "Area (km2)"
);

Then loop through the results and print them.

You can also add a small multibyte-safe padding helper to ensure alignment works even with Unicode characters.

The result looks like a professional terminal table instead of debug output.

Run It

php bin/console app:create-countries

You should see an output similar to this:

Why Use a Symfony Command for Scraping?

Because it gives you:

Separation of concerns
Cron scheduling capability
Clean architecture
Reusability
Easy refactoring into async jobs

This is how scraping should be structured in real applications, not inside controllers.

Production Tips

Before scraping real websites:

Check Terms of Service
Respect robots.txt
Avoid aggressive request rates
Add delays if scraping multiple pages. Example:

sleep(1);

Full Source Code

I’ve published the complete working project on GitHub, including:

The full command implementation
Proper formatting helpers
Setup instructions

GitHub repo: https://github.com/Marvelxy/symfony-web-scraper

Final Thoughts

Symfony + Console Commands + DomCrawler is an underrated combination.

If you’re building:

Data aggregation tools
Monitoring systems
Intelligence platforms
Background automation jobs

This pattern scales cleanly and keeps your application architecture solid.

In part 2, I will store the result in the database.

Install PHP AMQP Extension on Mac

Marvelous Akpotu — Sun, 13 Feb 2022 14:37:20 +0000

I spent several hours trying to set up PHP amqp extension on mac. When I finally got it to work, I created this post as a reference for myself in the future, and also to assist anyone trying to install it.

First, follow the usual steps to install rabbitmq from the URL:
https://rabbitmq.com/install-homebrew.html

After installing rabbitmq and adding it to your path, you can then follow these steps found in this URL:
https://blog.karatos.in/a?ID=00850-2e534416-7a8d-4bce-b77c-ea946f63e10c

The steps below are similar to the ones in the above URL, however, I had to modify it to make it work in my case.

Download the latest version of AMQP or any version of your choice from this URL: https://pecl.php.net/package/amqp

Extract the zipped file and visit it on your terminal.

$ cd/Users/***/Downloads/amqp-<VERSION>/amqp-<VERSION>

$ phpize

$ ./configure -with-php-config=/opt/homebrew/bin/php-config --with-amqp --with-librabbitmq-dir=/opt/homebrew/Cellar/rabbitmq-c/<VERSION>

In my case, I ran the previous command with version 0.11.0 as seen below:
./configure -with-php-config=/opt/homebrew/bin/php-config --with-amqp --with-librabbitmq-dir=/opt/homebrew/Cellar/rabbitmq-c/0.11.0

Run the following commands:

$ make

$ make test

$ make install

That's it. amqp should be installed by now. Next, you have to add the extension to your PHP configuration.

Open the php.ini file at /opt/homebrew/etc/php/8.1/php.ini, if the extension have been added (extension="amqp.so"), remove it and create ext-amqp.ini in the folder /opt/homebrew/etc/php/8.1/conf.d and add the following content:

[ampq]
extension="amqp.so"

Save the file and restart php.

$ brew services restart php

Next, check if it amqp is installed successfully:

$ php -i|grep amqp

Happy hacking 🚀

References

DEV Community: Marvelous Akpotu