DEV Community

Discussion on: Scraping Millions of Google SERPs The Easy Way (Python Scrapy Spider)

Collapse
 
kai_gray_b8562555da86b018 profile image
Kai Gray

This is really great, thank you! How could I get the original query to be written out alongside the result? For instance, if the search query is "soap" I'd be interested in the CSV having a column that listed that search query. I see ['search_information']['query_displayed'] in the response.text but cannot print it out as part of "item"

Many thanks

Collapse
 
ana123kots profile image
ana123kots • Edited

Hi, had the same issue. I solved it by added 2 lines after creating the item dic

query_displayed = di['search_information']
item['query_displayed'] = query_displayed['query_displayed']