Fantastic. We use a similar approach at work. For our case, each Scrapy project is run as a docker image for specific URL domains. Just curious, does this approach still allow middleware to be declared into the custom settings?
There is an option to declare middleware in the custom settings yes, but I think as far as creating the middleware inside one script this is something I haven't come across. My guess would be that if you're thinking of middleware then the whole scrapy framework is the best, indeed that is what it's there for.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Fantastic. We use a similar approach at work. For our case, each Scrapy project is run as a docker image for specific URL domains. Just curious, does this approach still allow middleware to be declared into the custom settings?
Hi Musale! Glad you liked the article.
There is an option to declare middleware in the custom settings yes, but I think as far as creating the middleware inside one script this is something I haven't come across. My guess would be that if you're thinking of middleware then the whole scrapy framework is the best, indeed that is what it's there for.