Contents: intro, imports, what will be scraped, process, code, links, outro.
Intro
This blog post is a continuation of DuckDuckGo web scraping series. Here you'll see how to scrape Ad Results using Python with selenium library. An alternative API solution will be shown.
Prerequisites: familiarity with selenium library and regular expressions.
Imports
from selenium import webdriver
What will be scraped
Process
Selecting title, link, snippet, source, sitelinks.
What I was doing is basically applying .results--ads selector before additional selector for title, link, snippet, source, and sitelinks to get most accurate selector.
A CSS selectors reference.
Code
from selenium import webdriver
driver = webdriver.Chrome(executable_path='path/to/chromedriver.exe')
driver.get('https://duckduckgo.com/?q=minecraft&kl=us-en&ia=web')
for result in driver.find_elements_by_css_selector('.results--ads .result__body.links_main.links_deep'):
title = result.find_element_by_css_selector('.js-result-title-link').text
link = result.find_element_by_css_selector('.js-result-title-link').get_attribute('href')
source = result.find_element_by_css_selector('.js-result-extras-url').text
snippet = result.find_element_by_css_selector('.js-result-snippet > a').text
print(f'{title}\n{source}\n{snippet}\n{link}\n')
for sitelink in driver.find_elements_by_css_selector('.js-sitelinks-title'):
sitelink_title = sitelink.text
sitelink_url = sitelink.get_attribute('href')
print(f'{sitelink_title}\n{sitelink_url}\n')
driver.quit()
---------------------
'''
Minecraft Video Games - Save On Minecraft Video Games
https://duckduckgo.com/y.js?ad_provider=bingv7aa&eddgt=g4RZhjUMFYuy1q3_sLWq3g%3D%3D&rut=8a6411c3b42b508666bdc647000f144258f958962f56d893c5620653f3ec2ebd&u3=https%3A%2F%2Fwww.bing.com%2Faclick%3Fld%3De8tlRvYW3t4dG5g%2DpvABGA4DVUCUyowbncAXntvrmjGeojwWrZFDK5oXEtgdBhIetjLMVkFSjF0OcmSEtTDNDjOilIW6B52S2hqo8v3CNZwdowb3t5LSt8wF5kfdECLDNLzZ7Js%2DlU0dmHtbUHnpUDy6SbSukR8g%2DVGUPe9w69ESsNB3%2DQnM9LOi5pAGiyHJ5pnYlulA%26u%3DaHR0cCUzYSUyZiUyZmNsaWNrc2VydmUuZGFydHNlYXJjaC5uZXQlMmZsaW5rJTJmY2xpY2slM2ZsaWQlM2Q0MzcwMDA2MTI4NjIwMTM3OSUyNmRzX3Nfa3dnaWQlM2Q1ODcwMDAwNjc2NjE3NDYwNSUyNmRzX2FfY2lkJTNkNjYyNTg3MTQ4JTI2ZHNfYV9jYWlkJTNkMTI0MTgyNjIzMjQlMjZkc19hX2FnaWQlM2QxMjM1MjMxMDcyNTIlMjZkc19hX2xpZCUzZGt3ZC0zMDQ0Mzg5ODczNzglMjYlMjZkc19lX2FkaWQlM2Q3NTE3OTIwMzk4NDc3OSUyNmRzX2VfdGFyZ2V0X2lkJTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTIzNSUyNiUyNmRzX3VybF92JTNkMiUyNmRzX2Rlc3RfdXJsJTNkaHR0cHMlM2ElMmYlMmZ3d3cud2FsbWFydC5jb20lMmZzZWFyY2glMmYlM2ZxdWVyeSUzZG1pbmVjcmFmdCUyYnRoZSUyYmdhbWUlMjZhZGlkJTNkMjIyMjIyMjIyMjQ0MjU1OTM4MDIlMjZ3bWxzcGFydG5lciUzZHdtdGxhYnMlMjZ3bDAlM2RlJTI2d2wxJTNkcyUyNndsMiUzZGMlMjZ3bDMlM2Q3NTE3OTIwMzk4NDc3OSUyNndsNCUzZGt3ZC03NTE3OTI5MjkyMTM5OCUzYWxvYy0yMzUlMjZ3bDUlM2QxNDExNzElMjZ3bDYlM2QlMjZ3bDclM2QlMjZ3bDE0JTNkbWluZWNyYWZ0JTI2dmVoJTNkc2VtJTI2Z2NsaWQlM2Q1ODFiZmJjMGM2NGIxZDc1ZTk4YjliMzBlNWU3MmYxNiUyNmdjbHNyYyUzZDNwLmRzJTI2JTI2bXNjbGtpZCUzZDU4MWJmYmMwYzY0YjFkNzVlOThiOWIzMGU1ZTcyZjE2%26rlid%3D581bfbc0c64b1d75e98b9b30e5e72f16&vqd=3-61454750645793260001748454990905959497-291233585751825993113979507856558647155&iurl=%7B1%7DIG%3D67B7A1B78A264F3A851C7D08BD88545F%26CID%3D3BBC77207A5960C0343D67557B8661B7%26ID%3DDevEx%2C5642.1
Free 2-day Shipping On Millions of Items. No Membership Fee. Shop Now!
This collection includes: The Minecraft base game, 1000 Minecoins and the Starter ...
Curbside Pickup · Shop Everyday Low Prices
PC Gaming‧Virtual Reality‧Video Game Accessories‧Video Game Consoles
walmart.com
Save 5% w/ RedCard at Target™ - Mindcraft Game at Target™
https://duckduckgo.com/y.js?ad_provider=bingv7aa&eddgt=g4RZhjUMFYuy1q3_sLWq3g%3D%3D&rut=882b95c16f4fc14eddef3dd23b4753f8eb11a9051a45d274d9a412b416bcdfc8&u3=https%3A%2F%2Fwww.bing.com%2Faclick%3Fld%3De8G2FCJ0SWLPN18IxZVTwm5zVUCUz2jK7DkoJHozC22Ot_lZykLOLwJ_t0YFXATV0BgHJTj2gyEal6nI1B61u7ILh%2DjB8Rs9qklWqoOhqZcqIbL0qEFCL2itcztPQNH5PBoNrsojS_oRwHEPHy%2D77oS7JBFxJrssZHaQL3KwBdy_uIl3odAlaOJGj521JDMr6dhnO0aQ%26u%3DaHR0cHMlM2ElMmYlMmZjbGlja3NlcnZlLmRhcnRzZWFyY2gubmV0JTJmbGluayUyZmNsaWNrJTNmbGlkJTNkNDM3MDAwNTgyOTA5MjE0MTMlMjZkc19zX2t3Z2lkJTNkNTg3MDAwMDA4MDAyMzE1MjAlMjZkc19hX2NpZCUzZDQwNTUyNDU5NCUyNmRzX2FfY2FpZCUzZDkyMjQzMzE1OTIlMjZkc19hX2FnaWQlM2QxMDAzMDAyMTM2NjQlMjZkc19hX2xpZCUzZGt3ZC0yMDgzMTQ3MDEwMiUyNiUyNmRzX2VfYWRpZCUzZDgxMDIwMzM0NTUyMTgwJTI2ZHNfZV90YXJnZXRfaWQlM2Rrd2QtODEwMjA0Mjk3NjU2NDAlM2Fsb2MtMjM1JTI2JTI2ZHNfdXJsX3YlM2QyJTI2ZHNfZGVzdF91cmwlM2RodHRwcyUzYSUyZiUyZnd3dy50YXJnZXQuY29tJTJmcyUyZm1pbmRjcmFmdCUyYmdhbWUlM2ZyZWYlM2R0Z3RfYWR2X1hTMDAwMDAwJTI2QUZJRCUzZG1zbiUyNmZuZHNyYyUzZHRndGFvJTI2REZBJTNkNzE3MDAwMDAwMTI1MTk1MjIlMjZDUE5HJTNkVmlkZW8lMjUyQkdhbWVzX0NvbnNvbGUlMmJHYW1lcyUyNTdDVmlkZW8lMjUyQkdhbWVzX0Vjb21tX0hhcmRsaW5lcyUyNmFkZ3JvdXAlM2RDb25zb2xlJTJiVmlkZW8lMmJHYW1lXzIlMjZMSUQlM2Q3MDAwMDAwMDExNzYzMTAlMjZMTk0lM2RtaW5kY3JhZnQlMmJnYW1lJTI2TVQlM2RlJTI2bmV0d29yayUzZHMlMjZkZXZpY2UlM2RjJTI2bG9jYXRpb24lM2QxNDExNzElMjZ0YXJnZXRpZCUzZGt3ZC04MTAyMDQyOTc2NTY0MCUzYWxvYy0yMzUlMjZnY2xpZCUzZDliNzQ1YWE2MWYwZjEzYjEyOWQzM2E1NjNjYTNlZWM3JTI2Z2Nsc3JjJTNkM3AuZHMlMjZkc19ybCUzZDEyNDY5NzglMjZkc19ybCUzZDEyNDgwOTklMjZtc2Nsa2lkJTNkOWI3NDVhYTYxZjBmMTNiMTI5ZDMzYTU2M2NhM2VlYzc%26rlid%3D9b745aa61f0f13b129d33a563ca3eec7&vqd=3-205393906844187692533645762826856793138-291233585751825993113979507856558647155&iurl=%7B1%7DIG%3D67B7A1B78A264F3A851C7D08BD88545F%26CID%3D3BBC77207A5960C0343D67557B8661B7%26ID%3DDevEx%2C5661.1
Get the Latest Mindcraft Game. Lots of Mindcraft Game to Choose From. Fast & Free 2-Day Shipping on Orders $35+.
target.com
PC Gaming
https://duckduckgo.com/y.js?ad_provider=bingv7aa&eddgt=g4RZhjUMFYuy1q3_sLWq3g%3D%3D&rut=f9cd2547f922699dc0a2cfca5575b64a2d4a833cd2cac10c903d7bd44293c1d9&u3=https%3A%2F%2Fwww.bing.com%2Faclick%3Fld%3De8zWLFQwqJ1QTiTZTbYLfjXjVUCUy4NTNgKV3RmTlAZdQ%2DUZQnw4wf2ggXuBOMiKnU1TNZ4K5b1tgu_07X6M11e09DY71s2AMuS%2DOs94Pvgkxf19vVVStcYiNdkEdGDvSTMT6LX7zU4mzv9duuDrD6_QElBkpycUIcO_iWQH2FprtmtHTbAK793EnLQG5KaYQY0EzIfQ%26u%3DaHR0cCUzYSUyZiUyZmNsaWNrc2VydmUuZGFydHNlYXJjaC5uZXQlMmZsaW5rJTJmY2xpY2slM2ZsaWQlM2Q0MzcwMDA2MTI4NjIwMTM3OSUyNmRzX3Nfa3dnaWQlM2Q1ODcwMDAwNjc2NjE3NDYwNSUyNmRzX2FfY2lkJTNkNjYyNTg3MTQ4JTI2ZHNfYV9jYWlkJTNkMTI0MTgyNjIzMjQlMjZkc19hX2FnaWQlM2QxMjM1MjMxMDcyNTIlMjZkc19hX2xpZCUzZGt3ZC0zMDQ0Mzg5ODczNzglMjZkc194X2FkeGlkJTNkODM3MDAwMDk1MjM4NjA0OTUlMjZkc194X2FkeHR5cGUlM2QxJTI2ZHNfYV9jaWQlM2Q2NjI1ODcxNDglMjZkc19hX2ZpaWQlM2QxNjQxMDQ3Njk0ODElMjZkc19lX2FkaWQlM2Q3NTE3OTIwMzk4NDc3OSUyNmRzX2VfdGFyZ2V0X2lkJTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTIzNSUyNiUyNmRzX3VybF92JTNkMiUyNmRzX2Rlc3RfdXJsJTNkaHR0cHMlM2ElMmYlMmZ3d3cud2FsbWFydC5jb20lMmZicm93c2UlMmZ2aWRlby1nYW1lcyUyZnBjLWdhbWluZyUyZjI2MzZfMTA3NzUwNCUzZmFkaWQlM2QyMjIyMjIyMjIyNDQyNTU5MzgwMiUyNndtbHNwYXJ0bmVyJTNkd210bGFicyUyNndsMCUzZGUlMjZ3bDElM2RzJTI2d2wyJTNkYyUyNndsMyUzZDc1MTc5MjAzOTg0Nzc5JTI2d2w0JTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTIzNSUyNndsNSUzZDE0MTE3MSUyNndsNiUzZDczNTMwMTA1ODk2NDQlMjZ3bDclM2QlMjZ3bDE0JTNkbWluZWNyYWZ0JTI2dmVoJTNkc2VtJTI2Z2NsaWQlM2RkMjg1N2M5NThlMjgxOTNlNGQ3NTFjNjVhZDg0NTY0YyUyNmdjbHNyYyUzZDNwLmRzJTI2JTI2bXNjbGtpZCUzZGQyODU3Yzk1OGUyODE5M2U0ZDc1MWM2NWFkODQ1NjRj%26rlid%3Dd2857c958e28193e4d751c65ad84564c&vqd=3-1966876518034573175047393153744481480-291233585751825993113979507856558647155&iurl=%7B1%7DIG%3D67B7A1B78A264F3A851C7D08BD88545F%26CID%3D3BBC77207A5960C0343D67557B8661B7%26ID%3DDevEx%2C5647.1
Virtual Reality
https://duckduckgo.com/y.js?ad_provider=bingv7aa&eddgt=g4RZhjUMFYuy1q3_sLWq3g%3D%3D&rut=16e5db074dc2fc09f50561d0c1bc52559a473774d1a0c0e583cd4a27350dbdab&u3=https%3A%2F%2Fwww.bing.com%2Faclick%3Fld%3De8LyhkfwFWgmN0YU2ii3oCXjVUCUwHHiKzeQKJO8IZ2bAozw2nr3rdwptE%2D6O6e9OSSn7AyzP5XLHmKfOCV9Q1smXJOvnBQWvGnqa96sYZZcUgON4k6Qz4u8n0ZpWd2hvhUXPeqWC_Dfm1r4UFN3FIZN715EUuKTCVy1zV1fJanIeJN40x_M3aZP41N2CWtybi1z3nKA%26u%3DaHR0cCUzYSUyZiUyZmNsaWNrc2VydmUuZGFydHNlYXJjaC5uZXQlMmZsaW5rJTJmY2xpY2slM2ZsaWQlM2Q0MzcwMDA2MTI4NjIwMTM3OSUyNmRzX3Nfa3dnaWQlM2Q1ODcwMDAwNjc2NjE3NDYwNSUyNmRzX2FfY2lkJTNkNjYyNTg3MTQ4JTI2ZHNfYV9jYWlkJTNkMTI0MTgyNjIzMjQlMjZkc19hX2FnaWQlM2QxMjM1MjMxMDcyNTIlMjZkc19hX2xpZCUzZGt3ZC0zMDQ0Mzg5ODczNzglMjZkc194X2FkeGlkJTNkODM3MDAwMDk1MjM4NjA0OTglMjZkc194X2FkeHR5cGUlM2QxJTI2ZHNfYV9jaWQlM2Q2NjI1ODcxNDglMjZkc19hX2ZpaWQlM2QxNjQxMDQ3Njk0ODQlMjZkc19lX2FkaWQlM2Q3NTE3OTIwMzk4NDc3OSUyNmRzX2VfdGFyZ2V0X2lkJTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTIzNSUyNiUyNmRzX3VybF92JTNkMiUyNmRzX2Rlc3RfdXJsJTNkaHR0cHMlM2ElMmYlMmZ3d3cud2FsbWFydC5jb20lMmZjcCUyZnZpcnR1YWwtcmVhbGl0eSUyZjU0NTQzMDElM2ZhZGlkJTNkMjIyMjIyMjIyMjQ0MjU1OTM4MDIlMjZ3bWxzcGFydG5lciUzZHdtdGxhYnMlMjZ3bDAlM2RlJTI2d2wxJTNkcyUyNndsMiUzZGMlMjZ3bDMlM2Q3NTE3OTIwMzk4NDc3OSUyNndsNCUzZGt3ZC03NTE3OTI5MjkyMTM5OCUzYWxvYy0yMzUlMjZ3bDUlM2QxNDExNzElMjZ3bDYlM2Q3MzUzMDEwNTg5NjQ1JTI2d2w3JTNkJTI2d2wxNCUzZG1pbmVjcmFmdCUyNnZlaCUzZHNlbSUyNmdjbGlkJTNkZmMyZGM3MGMwOGE4MTgzZTZkZTJhN2ZjMmIwNGE5MGMlMjZnY2xzcmMlM2QzcC5kcyUyNiUyNm1zY2xraWQlM2RmYzJkYzcwYzA4YTgxODNlNmRlMmE3ZmMyYjA0YTkwYw%26rlid%3Dfc2dc70c08a8183e6de2a7fc2b04a90c&vqd=3-137089535419097050706382597529438450029-291233585751825993113979507856558647155&iurl=%7B1%7DIG%3D67B7A1B78A264F3A851C7D08BD88545F%26CID%3D3BBC77207A5960C0343D67557B8661B7%26ID%3DDevEx%2C5649.1
Video Game Accessories
https://duckduckgo.com/y.js?ad_provider=bingv7aa&eddgt=g4RZhjUMFYuy1q3_sLWq3g%3D%3D&rut=ca63a9771de0e54103a90a811dd0ce07bd70f1c159fdb5cd137454cf58e2ebda&u3=https%3A%2F%2Fwww.bing.com%2Faclick%3Fld%3De8WDkz53ENl9Vn9gNtzmnmTjVUCUyqohFt10XkMWzIvjF7WnpaV07v7WA9aeyNFwOIRFn0AJgTgtv7sTAnurbHPtG0mcK5CGOXffNh5EACy3_UBRk0FC7JMCbz1NQZAJzCJidRLfKLfjbO2U8WN1kz8bcrAKm6epDBlUobUPgKP_hZoSpkv5XV4mkXPXLGrQ6Px1_%2DyA%26u%3DaHR0cCUzYSUyZiUyZmNsaWNrc2VydmUuZGFydHNlYXJjaC5uZXQlMmZsaW5rJTJmY2xpY2slM2ZsaWQlM2Q0MzcwMDA2MTI4NjIwMTM3OSUyNmRzX3Nfa3dnaWQlM2Q1ODcwMDAwNjc2NjE3NDYwNSUyNmRzX2FfY2lkJTNkNjYyNTg3MTQ4JTI2ZHNfYV9jYWlkJTNkMTI0MTgyNjIzMjQlMjZkc19hX2FnaWQlM2QxMjM1MjMxMDcyNTIlMjZkc19hX2xpZCUzZGt3ZC0zMDQ0Mzg5ODczNzglMjZkc194X2FkeGlkJTNkODM3MDAwMDk1MjM4NjA0OTIlMjZkc194X2FkeHR5cGUlM2QxJTI2ZHNfYV9jaWQlM2Q2NjI1ODcxNDglMjZkc19hX2ZpaWQlM2QxNjQxMDQ3Njk0NzglMjZkc19lX2FkaWQlM2Q3NTE3OTIwMzk4NDc3OSUyNmRzX2VfdGFyZ2V0X2lkJTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTIzNSUyNiUyNmRzX3VybF92JTNkMiUyNmRzX2Rlc3RfdXJsJTNkaHR0cHMlM2ElMmYlMmZ3d3cud2FsbWFydC5jb20lMmZjcCUyZnZpZGVvLWdhbWUtYWNjZXNzb3JpZXMlMmYxMjI5MDE5JTNmYWRpZCUzZDIyMjIyMjIyMjI0NDI1NTkzODAyJTI2d21sc3BhcnRuZXIlM2R3bXRsYWJzJTI2d2wwJTNkZSUyNndsMSUzZHMlMjZ3bDIlM2RjJTI2d2wzJTNkNzUxNzkyMDM5ODQ3NzklMjZ3bDQlM2Rrd2QtNzUxNzkyOTI5MjEzOTglM2Fsb2MtMjM1JTI2d2w1JTNkMTQxMTcxJTI2d2w2JTNkNzM1MzAxMDU4OTY0MyUyNndsNyUzZCUyNndsMTQlM2RtaW5lY3JhZnQlMjZ2ZWglM2RzZW0lMjZnY2xpZCUzZDMzOTMyZjNmZmJlNTE2ZTRiMTg4NGZjZmQ4NDlhOTZkJTI2Z2Nsc3JjJTNkM3AuZHMlMjYlMjZtc2Nsa2lkJTNkMzM5MzJmM2ZmYmU1MTZlNGIxODg0ZmNmZDg0OWE5NmQ%26rlid%3D33932f3ffbe516e4b1884fcfd849a96d&vqd=3-129360418447431572000570497592988537963-291233585751825993113979507856558647155&iurl=%7B1%7DIG%3D67B7A1B78A264F3A851C7D08BD88545F%26CID%3D3BBC77207A5960C0343D67557B8661B7%26ID%3DDevEx%2C5651.1
Video Game Consoles
https://duckduckgo.com/y.js?ad_provider=bingv7aa&eddgt=g4RZhjUMFYuy1q3_sLWq3g%3D%3D&rut=ea9f551cf1965b8afd7f6ae3ce1ce7cc41ae5adac906667d499c042fb5c61c31&u3=https%3A%2F%2Fwww.bing.com%2Faclick%3Fld%3De82MvYsV6CrsVtx4guYk_cSjVUCUxe_GSgLMhNbftx6vplN7hSUg%2DWEt3bwmwQXmRqKGccgyd4S6deyAa4qVcibAy%2DJ6hANa1TQJhruJEuEQdWs0dyggjvY29KW0yHLL6uzEk81JDc3iX%2Dixoeq_yQkUUe9ThBiDA9zCk%2DSpa%2D7_dhP7gTWzZtOxQo_KSuw_zVnndZ1w%26u%3DaHR0cCUzYSUyZiUyZmNsaWNrc2VydmUuZGFydHNlYXJjaC5uZXQlMmZsaW5rJTJmY2xpY2slM2ZsaWQlM2Q0MzcwMDA2MTI4NjIwMTM3OSUyNmRzX3Nfa3dnaWQlM2Q1ODcwMDAwNjc2NjE3NDYwNSUyNmRzX2FfY2lkJTNkNjYyNTg3MTQ4JTI2ZHNfYV9jYWlkJTNkMTI0MTgyNjIzMjQlMjZkc19hX2FnaWQlM2QxMjM1MjMxMDcyNTIlMjZkc19hX2xpZCUzZGt3ZC0zMDQ0Mzg5ODczNzglMjZkc194X2FkeGlkJTNkODM3MDAwMDk1MjM4NjA0ODklMjZkc194X2FkeHR5cGUlM2QxJTI2ZHNfYV9jaWQlM2Q2NjI1ODcxNDglMjZkc19hX2ZpaWQlM2QxNjQxMDQ3Njk0NzUlMjZkc19lX2FkaWQlM2Q3NTE3OTIwMzk4NDc3OSUyNmRzX2VfdGFyZ2V0X2lkJTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTIzNSUyNiUyNmRzX3VybF92JTNkMiUyNmRzX2Rlc3RfdXJsJTNkaHR0cHMlM2ElMmYlMmZ3d3cud2FsbWFydC5jb20lMmZjcCUyZnZpZGVvLWdhbWVzJTJmMjYzNiUzZmFkaWQlM2QyMjIyMjIyMjIyNDQyNTU5MzgwMiUyNndtbHNwYXJ0bmVyJTNkd210bGFicyUyNndsMCUzZGUlMjZ3bDElM2RzJTI2d2wyJTNkYyUyNndsMyUzZDc1MTc5MjAzOTg0Nzc5JTI2d2w0JTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTIzNSUyNndsNSUzZDE0MTE3MSUyNndsNiUzZDczNTMwMTA1ODk2NDIlMjZ3bDclM2QlMjZ3bDE0JTNkbWluZWNyYWZ0JTI2dmVoJTNkc2VtJTI2Z2NsaWQlM2Q5NTE5MmJhOGVlMzAxYjVkMjFjMTRiNThjZDljMjZhMiUyNmdjbHNyYyUzZDNwLmRzJTI2JTI2bXNjbGtpZCUzZDk1MTkyYmE4ZWUzMDFiNWQyMWMxNGI1OGNkOWMyNmEy%26rlid%3D95192ba8ee301b5d21c14b58cd9c26a2&vqd=3-317251499328598894584215891959984246336-291233585751825993113979507856558647155&iurl=%7B1%7DIG%3D67B7A1B78A264F3A851C7D08BD88545F%26CID%3D3BBC77207A5960C0343D67557B8661B7%26ID%3DDevEx%2C5653.1
'''
Using DuckDuckGo Ad Results API
SerpApi is a paid API with a free plan.
The difference is that is much faster, you don't need to render the page with selenium and you get an immediate structured JSON string.
import json
from serpapi import GoogleSearch
params = {
"api_key": "YOUR_API_KEY",
"engine": "duckduckgo",
"q": "minecraft",
"kl": "us-en"
}
search = GoogleSearch(params)
results = search.get_dict()
print(json.dumps(results['ads'], indent=2, ensure_ascii=False))
-------------------------
'''
[
{
"position": 1,
"title": "Minecraft Video Games - Save On Minecraft Video Games",
"link": "https://duckduckgo.com/y.js?ad_provider=bingv7aa&eddgt=0QdhC1y9z11BeQmahIracw%3D%3D&rut=a216fdd3b3188c92c2685df6a9583e121c134f2e105aeb38b383790e88600a0c&u3=https%3A%2F%2Fwww.bing.com%2Faclick%3Fld%3De8fgTI6SxasnT8IzGsTX1JCzVUCUxMOeN8XH_JyCXYHwFn9yeuq0eYsP2PfLKnYp3LD3CS5kmbcD1H8u51k8PPBW9DT_DUpkBQ5cUHNA9L232R4PfCEH26_I9uM_0tQpgEgrRdyXP9QUO_GQ9gGi570b_S06VtYpB6imjvtR77kU_r7jJankDSwMkwV195m8N_6IkGSw%26u%3DaHR0cCUzYSUyZiUyZmNsaWNrc2VydmUuZGFydHNlYXJjaC5uZXQlMmZsaW5rJTJmY2xpY2slM2ZsaWQlM2Q0MzcwMDA2MTI4NjIwMTM3OSUyNmRzX3Nfa3dnaWQlM2Q1ODcwMDAwNjc2NjE3NDYwNSUyNmRzX2FfY2lkJTNkNjYyNTg3MTQ4JTI2ZHNfYV9jYWlkJTNkMTI0MTgyNjIzMjQlMjZkc19hX2FnaWQlM2QxMjM1MjMxMDcyNTIlMjZkc19hX2xpZCUzZGt3ZC0zMDQ0Mzg5ODczNzglMjYlMjZkc19lX2FkaWQlM2Q3NTE3OTIwMzk4NDc3OSUyNmRzX2VfdGFyZ2V0X2lkJTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTE5MCUyNiUyNmRzX3VybF92JTNkMiUyNmRzX2Rlc3RfdXJsJTNkaHR0cHMlM2ElMmYlMmZ3d3cud2FsbWFydC5jb20lMmZzZWFyY2glMmYlM2ZxdWVyeSUzZG1pbmVjcmFmdCUyYnRoZSUyYmdhbWUlMjZhZGlkJTNkMjIyMjIyMjIyMjQ0MjU1OTM4MDIlMjZ3bWxzcGFydG5lciUzZHdtdGxhYnMlMjZ3bDAlM2RlJTI2d2wxJTNkcyUyNndsMiUzZGMlMjZ3bDMlM2Q3NTE3OTIwMzk4NDc3OSUyNndsNCUzZGt3ZC03NTE3OTI5MjkyMTM5OCUzYWxvYy0xOTAlMjZ3bDUlM2QxOTAlMjZ3bDYlM2QlMjZ3bDclM2QlMjZ3bDE0JTNkbWluZWNyYWZ0JTI2dmVoJTNkc2VtJTI2Z2NsaWQlM2RlNTY1NGI2MGMxMmYxZmZmNjQ1NzJiOTBlMmY4NWFiYSUyNmdjbHNyYyUzZDNwLmRzJTI2JTI2bXNjbGtpZCUzZGU1NjU0YjYwYzEyZjFmZmY2NDU3MmI5MGUyZjg1YWJh%26rlid%3De5654b60c12f1fff64572b90e2f85aba&vqd=3-87030390511408672693321996432917478138-182266568508120047137554084072326458307&iurl=%7B1%7DIG%3DB0E51F11F26E4D7F834E827F1130C03E%26CID%3D3AE9D303FD2763300696C376FCDA62DF%26ID%3DDevEx%2C5744.1",
"source": "walmart.com",
"snippet": "Free 2-day Shipping On Millions of Items. No Membership Fee. Shop Now!",
"sitelinks": [
{
"title": "PC Gaming",
"link": "https://duckduckgo.com/y.js?ad_provider=bingv7aa&eddgt=0QdhC1y9z11BeQmahIracw%3D%3D&rut=04e2b38a60eeb50ba10dccecfa7ad8cd86bc5b14745cc956182b9046e50cc0f3&u3=https%3A%2F%2Fwww.bing.com%2Faclick%3Fld%3De8jvj5mDwm7A8EWjuXtddsyjVUCUwxgrLM2RsbdFtuow%2DGguQDaQGvPeWxXitMufZG_oupGdkH%2D0JqG5tu_8LpfupXuIJnjgUWfg0EZOAsTSyFJtxOz5YJ5%2DsUoKDxr8ArveNQGoEI7BdU0IgoGXZjCiy_WVf0L0osJFCNQIx8BH1SkgDVm_kaP4x0nhEigFiZC38LEA%26u%3DaHR0cCUzYSUyZiUyZmNsaWNrc2VydmUuZGFydHNlYXJjaC5uZXQlMmZsaW5rJTJmY2xpY2slM2ZsaWQlM2Q0MzcwMDA2MTI4NjIwMTM3OSUyNmRzX3Nfa3dnaWQlM2Q1ODcwMDAwNjc2NjE3NDYwNSUyNmRzX2FfY2lkJTNkNjYyNTg3MTQ4JTI2ZHNfYV9jYWlkJTNkMTI0MTgyNjIzMjQlMjZkc19hX2FnaWQlM2QxMjM1MjMxMDcyNTIlMjZkc19hX2xpZCUzZGt3ZC0zMDQ0Mzg5ODczNzglMjZkc194X2FkeGlkJTNkODM3MDAwMDk1MjM4NjA0OTUlMjZkc194X2FkeHR5cGUlM2QxJTI2ZHNfYV9jaWQlM2Q2NjI1ODcxNDglMjZkc19hX2ZpaWQlM2QxNjQxMDQ3Njk0ODElMjZkc19lX2FkaWQlM2Q3NTE3OTIwMzk4NDc3OSUyNmRzX2VfdGFyZ2V0X2lkJTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTE5MCUyNiUyNmRzX3VybF92JTNkMiUyNmRzX2Rlc3RfdXJsJTNkaHR0cHMlM2ElMmYlMmZ3d3cud2FsbWFydC5jb20lMmZicm93c2UlMmZ2aWRlby1nYW1lcyUyZnBjLWdhbWluZyUyZjI2MzZfMTA3NzUwNCUzZmFkaWQlM2QyMjIyMjIyMjIyNDQyNTU5MzgwMiUyNndtbHNwYXJ0bmVyJTNkd210bGFicyUyNndsMCUzZGUlMjZ3bDElM2RzJTI2d2wyJTNkYyUyNndsMyUzZDc1MTc5MjAzOTg0Nzc5JTI2d2w0JTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTE5MCUyNndsNSUzZDE5MCUyNndsNiUzZDczNTMwMTA1ODk2NDQlMjZ3bDclM2QlMjZ3bDE0JTNkbWluZWNyYWZ0JTI2dmVoJTNkc2VtJTI2Z2NsaWQlM2RkMTU0ZTc5MjE2NDgxYWE2NGZkMDg3NjIzNTlmNWFjZSUyNmdjbHNyYyUzZDNwLmRzJTI2JTI2bXNjbGtpZCUzZGQxNTRlNzkyMTY0ODFhYTY0ZmQwODc2MjM1OWY1YWNl%26rlid%3Dd154e79216481aa64fd08762359f5ace&vqd=3-220109179843301254675732288972445008198-182266568508120047137554084072326458307&iurl=%7B1%7DIG%3DB0E51F11F26E4D7F834E827F1130C03E%26CID%3D3AE9D303FD2763300696C376FCDA62DF%26ID%3DDevEx%2C5749.1"
},
{
"title": "Virtual Reality",
"link": "https://duckduckgo.com/y.js?ad_provider=bingv7aa&eddgt=0QdhC1y9z11BeQmahIracw%3D%3D&rut=233ab76f169eac87c1482731a761f5b40bdf142a70d7804182cb7936070ede82&u3=https%3A%2F%2Fwww.bing.com%2Faclick%3Fld%3De8HEspBp8JBHVa1ItwUzszTDVUCUy39V5wJqekOq3UHWELZ6v8PE7Nw0GmZUt8B_nUo_mt3lcfo8dqm8tE_nHmCtI_YUS5NSkp4im%2DqgRXe964XNGRIZ8yhuY3Jl5XcyjJtfSHS1crmNv4H6yxHYRd0pS7yV5PMaaeRQ7jI_v8icUTDuEjixnIyOaf3MmtY8AQyw30jw%26u%3DaHR0cCUzYSUyZiUyZmNsaWNrc2VydmUuZGFydHNlYXJjaC5uZXQlMmZsaW5rJTJmY2xpY2slM2ZsaWQlM2Q0MzcwMDA2MTI4NjIwMTM3OSUyNmRzX3Nfa3dnaWQlM2Q1ODcwMDAwNjc2NjE3NDYwNSUyNmRzX2FfY2lkJTNkNjYyNTg3MTQ4JTI2ZHNfYV9jYWlkJTNkMTI0MTgyNjIzMjQlMjZkc19hX2FnaWQlM2QxMjM1MjMxMDcyNTIlMjZkc19hX2xpZCUzZGt3ZC0zMDQ0Mzg5ODczNzglMjZkc194X2FkeGlkJTNkODM3MDAwMDk1MjM4NjA0OTglMjZkc194X2FkeHR5cGUlM2QxJTI2ZHNfYV9jaWQlM2Q2NjI1ODcxNDglMjZkc19hX2ZpaWQlM2QxNjQxMDQ3Njk0ODQlMjZkc19lX2FkaWQlM2Q3NTE3OTIwMzk4NDc3OSUyNmRzX2VfdGFyZ2V0X2lkJTNka3dkLTc1MTc5MjkyOTIxMzk4JTNhbG9jLTE5MCUyNiUyNmRzX3VybF92JTNkMiUyNmRzX2Rlc3RfdXJsJTNkaHR0cHMlM2ElMmYlMmZ3d3cud2FsbWFydC5jb20lMmZjcCUyZnZpcnR1YWwtcmVhbGl0eSUyZjU0NTQzMDElM2ZhZGlkJTNkMjIyMjIyMjIyMjQ0MjU1OTM4MDIlMjZ3bWxzcGFydG5lciUzZHdtdGxhYnMlMjZ3bDAlM2RlJTI2d2wxJTNkcyUyNndsMiUzZGMlMjZ3bDMlM2Q3NTE3OTIwMzk4NDc3OSUyNndsNCUzZGt3ZC03NTE3OTI5MjkyMTM5OCUzYWxvYy0xOTAlMjZ3bDUlM2QxOTAlMjZ3bDYlM2Q3MzUzMDEwNTg5NjQ1JTI2d2w3JTNkJTI2d2wxNCUzZG1pbmVjcmFmdCUyNnZlaCUzZHNlbSUyNmdjbGlkJTNkYWM2MjU2MDFkNGU0MTY5ZTE1ZDMzNmYxZjczMzIyMjclMjZnY2xzcmMlM2QzcC5kcyUyNiUyNm1zY2xraWQlM2RhYzYyNTYwMWQ0ZTQxNjllMTVkMzM2ZjFmNzMzMjIyNw%26rlid%3Dac625601d4e4169e15d336f1f7332227&vqd=3-134614315498574247865815717038458011354-182266568508120047137554084072326458307&iurl=%7B1%7DIG%3DB0E51F11F26E4D7F834E827F1130C03E%26CID%3D3AE9D303FD2763300696C376FCDA62DF%26ID%3DDevEx%2C5751.1"
}
...
{
"position": 2,
"title": "Discover Savings Today - Get Free Shipping w/ RedCard",
"link": "https://duckduckgo.com/y.js?ad_provider=bingv7aa&eddgt=0QdhC1y9z11BeQmahIracw%3D%3D&rut=04ff8dd1af518abff4214d1e44198e92cf70219ab9cc762bcbfc2f047af1c083&u3=https%3A%2F%2Fwww.bing.com%2Faclick%3Fld%3De84Ph8X4w260Gl92Z%2DIMdNlDVUCUxi6IcTvbnqI%2DRZRBILc0WlQAVwo6NvwHp54gu9smxxtlnCx_pc%2DHIb12pJzkYMFXQaTFi1wetFpfKNmUv68wi61B%2DOBrJ6Ja%2DqR86VrPNq6WCDA28OomiMpwZJy%2DI4oUuNsqwbD_OFazgGMvNjERIw4YRFJu%2De7nxmtFFQk0IIKg%26u%3DaHR0cHMlM2ElMmYlMmZjbGlja3NlcnZlLmRhcnRzZWFyY2gubmV0JTJmbGluayUyZmNsaWNrJTNmbGlkJTNkNDM3MDAwNTgyOTkyNzkyNjUlMjZkc19zX2t3Z2lkJTNkNTg3MDAwMDY0NTIwMTEwNzElMjZkc19hX2NpZCUzZDQwNTUwNzc5OSUyNmRzX2FfY2FpZCUzZDkyMjQ0MjI0NjQlMjZkc19hX2FnaWQlM2QxMTMxMTExNzMxNDAlMjZkc19hX2xpZCUzZGt3ZC0zMDAzOTg2MzAzNzIlMjYlMjZkc19lX2FkaWQlM2Q4MTg0NDk3MDA2MjIxMCUyNmRzX2VfdGFyZ2V0X2lkJTNka3dkLTgxODQ1MDYyNjUyMDI2JTNhbG9jLTE5MCUyNiUyNmRzX3VybF92JTNkMiUyNmRzX2Rlc3RfdXJsJTNkaHR0cHMlM2ElMmYlMmZ3d3cudGFyZ2V0LmNvbSUyZnMlMmZtaW5lY3JhZnQlMmJtaW5lY3JhZnQlMmJtaW5lY3JhZnQlM2ZyZWYlM2R0Z3RfYWR2X1hTMDAwMDAwJTI2QUZJRCUzZG1zbiUyNmZuZHNyYyUzZHRndGFvJTI2REZBJTNkNzE3MDAwMDAwMTI1MzkwMzYlMjZDUE5HJTNkRWxlY3Ryb25pY3NfVmlkZW8lMmJHYW1lJTJiQ29uc29sZXMlMjU3Q0VsZWN0cm9uaWNzX0Vjb21tX0hhcmRsaW5lcyUyNmFkZ3JvdXAlM2RWaWRlbyUyYkdhbWUlMmJDb25zb2xlc18yJTI2TElEJTNkNzAwMDAwMDAxMTc2MjQ2JTI2TE5NJTNkbWluZWNyYWZ0JTJibWluZWNyYWZ0JTJibWluZWNyYWZ0JTI2TVQlM2RlJTI2bmV0d29yayUzZHMlMjZkZXZpY2UlM2RjJTI2bG9jYXRpb24lM2QxOTAlMjZ0YXJnZXRpZCUzZGt3ZC04MTg0NTA2MjY1MjAyNiUzYWxvYy0xOTAlMjZnY2xpZCUzZDM1MjExNTNiYmQxNDEyNDNhYWZhNzVjYjllYzNlY2NkJTI2Z2Nsc3JjJTNkM3AuZHMlMjZkc19ybCUzZDEyNDY5NzglMjZkc19ybCUzZDEyNDgwOTklMjZtc2Nsa2lkJTNkMzUyMTE1M2JiZDE0MTI0M2FhZmE3NWNiOWVjM2VjY2Q%26rlid%3D3521153bbd141243aafa75cb9ec3eccd&vqd=3-252781822407565903154811433641671895283-182266568508120047137554084072326458307&iurl=%7B1%7DIG%3DB0E51F11F26E4D7F834E827F1130C03E%26CID%3D3AE9D303FD2763300696C376FCDA62DF%26ID%3DDevEx%2C5763.1",
"source": "target.com",
"snippet": "Shop Minecraft Minecraft Minecraft & More. Get Minecraft Minecraft Minecraft at Target™ Today. Save on Minecraft Minecraft Minecraft. Quick & Easy Returns In-Store."
}
]
'''
Links
GitHub Gist • DuckDuckGo Ad Results API
Outro
If you have any questions or something isn't working correctly or you want to write something else, feel free to drop a comment in the comment section or via Twitter at @serp_api.
Yours,
Dimitry, and the rest of SerpApi Team.
Top comments (0)