Hi I'm Ian! I'm building ScrapeOps, a job monitoring & scheduling tool for web scrapers that you can integrate with 3 lines of code. Follow me for in-depth web scraping content.
You need to create a Item Pipeline like this in your pipelines.py file.
# pipelines.py
importmysql.connectorclassSaveMySQLPipeline:def__init__(self):self.conn=mysql.connector.connect(host='localhost',user='root',password='*******',database='dbname')## Create cursor, used to execute commands
self.cur=self.conn.cursor()## Create quotes table if none exists
self.cur.execute("""
CREATE TABLE IF NOT EXISTS products (
id int NOT NULL auto_increment,
asin text,
title text,
image VARCHAR(255),
PRIMARY KEY (id)
)
""")defprocess_item(self,item,spider):## Define insert statement
self.cur.execute(""" insert into quotes (asin, title, image) values (%s,%s,%s)""",(item["asin"],item["Title"],item["MainImage"]))## Execute insert of data into database
self.conn.commit()defclose_spider(self,spider):## Close cursor & connection to database
self.cur.close()self.conn.close()
Hi I'm Ian! I'm building ScrapeOps, a job monitoring & scheduling tool for web scrapers that you can integrate with 3 lines of code. Follow me for in-depth web scraping content.
To scrape those extra pricing details you will need to find the selectors for them and add those field to the item.
When I open that page, I don't see the fields as Amazon is probably only showing them based on the geography you are in.
So if you create new selectors for those prices you want and add them to the item, then you can update the mysql storage pipeline to store that data as well.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
You need to create a Item Pipeline like this in your
pipelines.py
file.And then enable it in your
settings.py
file.firstly thank you for answer.
'tutorial.pipelines.TutorialPipeline': 300 <<< I guess a comma is needed here.
secondly, the price information cannot be scraped in the above codes. what could be the reason for this?
For example, can we register mysql by scraping 3 different prices of the following product?
amazon.com/dp/B07KSJLQCD
List Price: $25.00
Price: $17.45
Lightning deal " if any"
MS -> My Telegram t.me/smesut
To scrape those extra pricing details you will need to find the selectors for them and add those field to the item.
When I open that page, I don't see the fields as Amazon is probably only showing them based on the geography you are in.
So if you create new selectors for those prices you want and add them to the item, then you can update the mysql storage pipeline to store that data as well.