Installing Scrapy in Fedora34 with pypy( pypy3 )

Amit Kumar Malaker — Mon, 19 Jul 2021 17:23:56 +0000

Pre-context:

The thing is I love fast things. And I am sure the reader also pursue same nature like me. Hence, pypy3 is the choice.
Fedora is nice but for me it is new and new thing is not always hassle free so to make your life better this instructions will be life saving.

First install pypy3 and respective pip pkgmanager:

sudo dnf install pypy3
wget https://bootstrap.pypa.io/get-pip.py -o get-pip.py
pypy3 get-pip.py

Then install the below dependencies which are the root cause of scrapy installation failures. So do it carefully.

sudo dnf install pypy3-devel
sudo dnf install libxml2-devel
sudo dnf install libxslt-devel
sudo dnf install @development-tools
sudo dnf install libffi-devel
install them if you find them: [zlib1g-dev, libssl-dev] (optional)

Now create virtual environment:

pypy3 -m venv venv

Activate the virtual environment:

source venv/bin/activate

Now install scrapy:

pypy3 -m pip install setuptools
pypy3 -m pip install scrapy

Hopefully scrapy will be installed successfully.
Now create scrapy project.

scrapy startproject demo

Create a spider:

cd demo
scrapy genspider demo demo

In fine, this is the way to have our beloved scrapy working in fedora.
If you have any problem you can comment below. I will try my best to resolve it.

DEV Community: Amit Kumar Malaker

Installing Scrapy in Fedora34 with pypy( pypy3 )

Pre-context: