DEV Community: Illia Zub

How we reverse-engineered Google Maps pagination

Illia Zub — Tue, 05 Dec 2023 11:01:52 +0000

In this story, you'll see the process of decoding URL parameters for pagination on Google Maps. It involved deobfuscation of Closure-compiled JavaScript, reverse-engineering of Protobuf data structures, and a bit of math. We tried to decode URL parameters by ourselves, by using pbtk, and attempted to outsource this work. In the end, we succeeded after several pair programming sessions.

How Google Maps pagination works

We can get the link for the next page only by clicking on the “next page” button.

Links look like this.

Page 1

https://www.google.com/search?tbm=map&authuser=0&hl=en&gl=us&pb=!4m8!1m3!1d24182.00605141337!2d-74.0083012!3d40.7455096!3m2!1i1024!2i768!4f13.1!7i20!8i20!10b1!12m18!2m3!5m1!6e2!20e3!6m11!4b1!23b1!26i1!27i1!41i2!45b1!63m0!67b1!73m0!74i150000!89b1!10b1!16b1!19m4!2m3!1i360!2i120!4i8!20m57!2m2!1i203!2i100!3m2!2i4!5b1!6m6!1m2!1i86!2i86!1m2!1i408!2i240!7m42!1m3!1e1!2b0!3e3!1m3!1e2!2b1!3e2!1m3!1e2!2b0!3e3!1m3!1e3!2b0!3e3!1m3!1e8!2b0!3e3!1m3!1e3!2b1!3e2!1m3!1e9!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e10!2b0!3e4!2b1!4b1!9b0!22m3!1sjWLeXbmnHIXt-gTm2ouwDg%3A23!2zMWk6Mix0OjEyNjk2LGU6MSxwOmpXTGVYYm1uSElYdC1nVG0yb3V3RGc6MjM!7e81!24m40!1m12!13m6!2b1!3b1!4b1!6i1!8b1!9b1!18m4!3b1!4b1!5b1!6b1!2b1!5m5!2b1!3b1!5b1!6b1!7b1!10m1!8e3!14m1!3b1!17b1!20m2!1e3!1e6!24b1!25b1!26b1!30m1!2b1!36b1!43b1!52b1!55b1!56m2!1b1!3b1!26m4!2m3!1i80!2i92!4i8!30m28!1m6!1m2!1i0!2i0!2m2!1i458!2i768!1m6!1m2!1i974!2i0!2m2!1i1024!2i768!1m6!1m2!1i0!2i0!2m2!1i1024!2i20!1m6!1m2!1i0!2i748!2m2!1i1024!2i768!34m9!3b1!4b1!6b1!8m2!1b1!3b1!9b1!12b1!14b1!37m1!1e81!42b1!46m1!1e9!47m0!49m1!3b1!50m40!1m39!2m7!1u3!4sOpen+now!5e1!9s0ahUKEwj35K6aqYrmAhWFtp4KHWbtAuYQ_KkBCIMHKBc!10m2!3m1!1e1!2m7!1u2!4sTop+rated!5e1!9s0ahUKEwj35K6aqYrmAhWFtp4KHWbtAuYQ_KkBCIQHKBg!10m2!2m1!1e1!2m7!1u1!4sCheap!5e1!9s0ahUKEwj35K6aqYrmAhWFtp4KHWbtAuYQ_KkBCIUHKBk!10m2!1m1!1e1!2m7!1u1!4sUpscale!5e1!9s0ahUKEwj35K6aqYrmAhWFtp4KHWbtAuYQ_KkBCIYHKBo!10m2!1m1!1e2!3m1!1u2!3m1!1u1!3m1!1u3!4BIAE!59BQ2dBd0Fn&q=Coffee&tch=1&ech=1&psi=jWLeXbmnHIXt-gTm2ouwDg.1574855312836.1

Page 2

https://www.google.com/search?tbm=map&authuser=0&hl=en&gl=us&pb=!4m8!1m3!1d24182.00605141337!2d-74.0083012!3d40.7455096!3m2!1i1024!2i768!4f13.1!7i20!8i20!10b1!12m18!2m3!5m1!6e2!20e3!6m11!4b1!23b1!26i1!27i1!41i2!45b1!63m0!67b1!73m0!74i150000!89b1!10b1!16b1!19m4!2m3!1i360!2i120!4i8!20m57!2m2!1i203!2i100!3m2!2i4!5b1!6m6!1m2!1i86!2i86!1m2!1i408!2i240!7m42!1m3!1e1!2b0!3e3!1m3!1e2!2b1!3e2!1m3!1e2!2b0!3e3!1m3!1e3!2b0!3e3!1m3!1e8!2b0!3e3!1m3!1e3!2b1!3e2!1m3!1e9!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e10!2b0!3e4!2b1!4b1!9b0!22m3!1sjWLeXbmnHIXt-gTm2ouwDg%3A78!2zMWk6Mix0OjEyNjk2LGU6MSxwOmpXTGVYYm1uSElYdC1nVG0yb3V3RGc6Nzg!7e81!24m40!1m12!13m6!2b1!3b1!4b1!6i1!8b1!9b1!18m4!3b1!4b1!5b1!6b1!2b1!5m5!2b1!3b1!5b1!6b1!7b1!10m1!8e3!14m1!3b1!17b1!20m2!1e3!1e6!24b1!25b1!26b1!30m1!2b1!36b1!43b1!52b1!55b1!56m2!1b1!3b1!26m4!2m3!1i80!2i92!4i8!30m28!1m6!1m2!1i0!2i0!2m2!1i458!2i768!1m6!1m2!1i974!2i0!2m2!1i1024!2i768!1m6!1m2!1i0!2i0!2m2!1i1024!2i20!1m6!1m2!1i0!2i748!2m2!1i1024!2i768!34m9!3b1!4b1!6b1!8m2!1b1!3b1!9b1!12b1!14b1!37m1!1e81!42b1!46m1!1e9!47m0!49m1!3b1!50m40!1m39!2m7!1u3!4sOpen+now!5e1!9s0ahUKEwj35K6aqYrmAhWFtp4KHWbtAuYQ_KkBCIMHKBc!10m2!3m1!1e1!2m7!1u2!4sTop+rated!5e1!9s0ahUKEwj35K6aqYrmAhWFtp4KHWbtAuYQ_KkBCIQHKBg!10m2!2m1!1e1!2m7!1u1!4sCheap!5e1!9s0ahUKEwj35K6aqYrmAhWFtp4KHWbtAuYQ_KkBCIUHKBk!10m2!1m1!1e1!2m7!1u1!4sUpscale!5e1!9s0ahUKEwj35K6aqYrmAhWFtp4KHWbtAuYQ_KkBCIYHKBo!10m2!1m1!1e2!3m1!1u2!3m1!1u1!3m1!1u3!4BIAE!59BQ2dBd0Fn&q=Coffee&tch=1&ech=2&psi=jWLeXbmnHIXt-gTm2ouwDg.1574855312836.1

They lead to a f.txt file that contains the next page results.

The Plan

Find out how the pb (protobuf) string is constructed.
Generate the next page link by setting the required parameters.
Catch and parse the data.

Decoding the URL parameters for Google Maps pagination

Google Maps URLs contain the pb parameter contains string-encoded Protobuf. The format is the same as for the data parameter in the browser URL on Google Maps. It contains !-separated values. There are several answers on StackOverflow, gists on GitHub, some blog posts about decoding, and even a kinda official guide on reverse engineering protobuf, but none of this touches pagination.

We tried to use pbtk but it wasn't able to extract structures and crashed. Several attempts of reading pretty-printed obfuscated JavaScript didn't work.

After pairing with Milos, we found out most of the variables in Google Maps pagination URI: latitude, longitude, altitude_in_feets, pagination_offset, some parameter that is equal to psi but I don't know what it's meaning. psi changes after each page reload and it's in window.APP_OPTIONS[11].

psi parameter in the window.APP_OPTIONS on Google Maps

Another moving part is a list of filters, but we don't know how to parse them.

List of filters in Google Maps URLs for pagination

We understood that we can make the first request for Google Maps, extract variables and construct pagination URI. Like for our API to scrape YouTube.

if offset
  long ||= -73.91476977539236
  lat ||= 40.68525694561602
  alt ||= 120027.44487325678

  offset ||= 40

  psi ||= "b24JYPPGOoaJrwTXlbHACw"

  google_query = "#{query_scheme_and_domain}/search?tbm=map&authuser=0&hl=en&gl=ua&pb=!4m12!1m3!1d#{alt}!2d#{lat}!3d#{long}!2m3!1f0!2f0!3f0!3m2!1i1920!2i549!4f13.1!7i20!8i#{offset}!10b1!12m8!1m1!18b1!2m3!5m1!6e2!20e3!10b1!16b1!19m4!2m3!1i360!2i120!4i8!20m65!2m2!1i203!2i100!3m2!2i4!5b1!6m6!1m2!1i86!2i86!1m2!1i408!2i240!7m50!1m3!1e1!2b0!3e3!1m3!1e2!2b1!3e2!1m3!1e2!2b0!3e3!1m3!1e3!2b0!3e3!1m3!1e8!2b0!3e3!1m3!1e3!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e9!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e10!2b0!3e4!2b1!4b1!9b0!22m3!1s#{psi}!2s1i%3A2%2Ct%3A12696%2Ce%3A1%2Cp%3A#{psi}%3A1273!7e81!24m56!1m16!13m7!2b1!3b1!4b1!6i1!8b1!9b1!20b0!18m7!3b1!4b1!5b1!6b1!9b1!13b0!14b0!2b1!5m5!2b1!3b1!5b1!6b1!7b1!10m1!8e3!14m1!3b1!17b1!20m4!1e3!1e6!1e14!1e15!24b1!25b1!26b1!29b1!30m1!2b1!36b1!43b1!52b1!54m1!1b1!55b1!56m2!1b1!3b1!65m5!3m4!1m3!1m2!1i224!2i298!89b1!26m4!2m3!1i80!2i92!4i8!30m28!1m6!1m2!1i0!2i0!2m2!1i458!2i549!1m6!1m2!1i1870!2i0!2m2!1i1920!2i549!1m6!1m2!1i0!2i0!2m2!1i1920!2i20!1m6!1m2!1i0!2i529!2m2!1i1920!2i549!31b1!34m16!2b1!3b1!4b1!6b1!8m4!1b1!3b1!4b1!6b1!9b1!12b1!14b1!20b1!23b1!25b1!26b1!37m1!1e81!42b1!46m1!1e9!47m0!49m1!3b1!50m73!1m68!2m7!1u3!4sOpen+now!5e1!9s0ahUKEwiPrMCgia3uAhUjlYsKHanUBmYQ_KkBCNUJKBg!10m2!3m1!1e1!2m7!1u2!4sTop+rated!5e1!9s0ahUKEwiPrMCgia3uAhUjlYsKHanUBmYQ_KkBCNYJKBk!10m2!2m1!1e1!2m7!1u1!4sCheap!5e1!9s0ahUKEwiPrMCgia3uAhUjlYsKHanUBmYQ_KkBCNcJKBo!10m2!1m1!1e1!2m7!1u1!4sUpscale!5e1!9s0ahUKEwiPrMCgia3uAhUjlYsKHanUBmYQ_KkBCNgJKBs!10m2!1m1!1e2!2m7!1u16!4sVisited!5e1!9s0ahUKEwiPrMCgia3uAhUjlYsKHanUBmYQ_KkBCNkJKBw!10m2!16m1!1e1!2m7!1u16!4sHaven%27t+visited!5e1!9s0ahUKEwiPrMCgia3uAhUjlYsKHanUBmYQ_KkBCNoJKB0!10m2!16m1!1e2!3m1!1u3!3m1!1u2!3m2!1u1!3e1!3m11!1u16!2m4!1m2!16m1!1e1!2sVisited!2m4!1m2!16m1!1e2!2sHaven%27t+visited!4BIAE!2e2!3m2!1b1!3b1!59BQ2dBd0Fn!65m0!69i540&q=#{safe_query}&tch=1&ech=1&psi=#{psi}.1611230833287.1"
end

The quick check confirmed that the algorithm works.

$ bundle exec rails runner 'puts Search.new(engine: :google_maps, q: "coffee", lat: 36.3996184, long: -113.9511419, alt: 2124931.1267513777, offset: 20, psi: "rZkJYJuoINHmrgTP1IVI").query_randomized' | xargs curl -s -k -x $HTTP_PROXY -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.46' - > f.txt

# "https://www.google.com/search?tbm=map&authuser=0&hl=en&gl=ua&pb=!4m8!1m3!1d2124931.1267513777!2d36.3996184!3d-113.9511419!3m2!1i1024!2i768!4f13.1!7i20!8i20!10b1!12m25!1m1!18b1!2m3!5m1!6e2!20e3!6m16!4b1!23b1!26i1!27i1!41i2!45b1!49b1!63m0!67b1!73m0!74i150000!75b1!89b1!105b1!109b1!110m0!10b1!16b1!19m4!2m3!1i360!2i120!4i8!20m65!2m2!1i203!2i100!3m2!2i4!5b1!6m6!1m2!1i86!2i86!1m2!1i408!2i240!7m50!1m3!1e1!2b0!3e3!1m3!1e2!2b1!3e2!1m3!1e2!2b0!3e3!1m3!1e3!2b0!3e3!1m3!1e8!2b0!3e3!1m3!1e3!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e9!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e10!2b0!3e4!2b1!4b1!9b0!22m3!1sgpYJYKSwMsH9rgT0_7nAAg!2zMWk6Mix0OjEyNjk2LGU6MSxwOmdwWUpZS1N3TXNIOXJnVDBfN25BQWc6MjM!7e81!24m55!1m15!13m7!2b1!3b1!4b1!6i1!8b1!9b1!20b0!18m6!3b1!4b1!5b1!6b1!13b0!14b0!2b1!5m5!2b1!3b1!5b1!6b1!7b1!10m1!8e3!14m1!3b1!17b1!20m4!1e3!1e6!1e14!1e15!24b1!25b1!26b1!29b1!30m1!2b1!36b1!43b1!52b1!54m1!1b1!55b1!56m2!1b1!3b1!65m5!3m4!1m3!1m2!1i224!2i298!89b1!26m4!2m3!1i80!2i92!4i8!30m28!1m6!1m2!1i0!2i0!2m2!1i458!2i768!1m6!1m2!1i974!2i0!2m2!1i1024!2i768!1m6!1m2!1i0!2i0!2m2!1i1024!2i20!1m6!1m2!1i0!2i748!2m2!1i1024!2i768!34m17!2b1!3b1!4b1!6b1!7b1!8m4!1b1!3b1!4b1!6b1!9b1!12b1!14b1!20b1!23b1!25b1!26b1!37m1!1e81!42b1!46m1!1e9!47m0!49m1!3b1!50m72!1m68!2m7!1u3!4sOpen+now!5e1!9s0ahUKEwizhPrmpK3uAhXBvosKHfR_DigQ_KkBCMgJKBY!10m2!3m1!1e1!2m7!1u2!4sTop+rated!5e1!9s0ahUKEwizhPrmpK3uAhXBvosKHfR_DigQ_KkBCMkJKBc!10m2!2m1!1e1!2m7!1u1!4sCheap!5e1!9s0ahUKEwizhPrmpK3uAhXBvosKHfR_DigQ_KkBCMoJKBg!10m2!1m1!1e1!2m7!1u1!4sUpscale!5e1!9s0ahUKEwizhPrmpK3uAhXBvosKHfR_DigQ_KkBCMsJKBk!10m2!1m1!1e2!2m7!1u16!4sVisited!5e1!9s0ahUKEwizhPrmpK3uAhXBvosKHfR_DigQ_KkBCMwJKBo!10m2!16m1!1e1!2m7!1u16!4sHaven%27t+visited!5e1!9s0ahUKEwizhPrmpK3uAhXBvosKHfR_DigQ_KkBCM0JKBs!10m2!16m1!1e2!3m11!1u16!2m4!1m2!16m1!1e1!2sVisited!2m4!1m2!16m1!1e2!2sHaven%27t+visited!3m1!1u2!3m2!1u1!3e0!3m1!1u3!4BIAE!2e2!3m1!3b1!59BQ2dBd0Fn!65m0!69i540&q=coffee&tch=1&ech=1&psi=rZkJYJuoINHmrgTP1IVI.1611241093531.1"

# Hit next page in browser, copy URL and curl it.

$ curl -s -k -x $HTTP_PROXY -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.46' 'https://www.google.com/search?tbm=map&authuser=0&hl=en&gl=ua&pb=!4m8!1m3!1d2124931.1267513777!2d-113.9511419!3d36.3996184!3m2!1i1024!2i768!4f13.1!7i20!8i20!10b1!12m25!1m1!18b1!2m3!5m1!6e2!20e3!6m16!4b1!23b1!26i1!27i1!41i2!45b1!49b1!63m0!67b1!73m0!74i150000!75b1!89b1!105b1!109b1!110m0!10b1!16b1!19m4!2m3!1i360!2i120!4i8!20m65!2m2!1i203!2i100!3m2!2i4!5b1!6m6!1m2!1i86!2i86!1m2!1i408!2i240!7m50!1m3!1e1!2b0!3e3!1m3!1e2!2b1!3e2!1m3!1e2!2b0!3e3!1m3!1e3!2b0!3e3!1m3!1e8!2b0!3e3!1m3!1e3!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e9!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e10!2b0!3e4!2b1!4b1!9b0!22m3!1srZkJYJuoINHmrgTP1IVI!2zMWk6Mix0OjEyNjk2LGU6MSxwOnJaa0pZSnVvSU5IbXJnVFAxSVZJOjIz!7e81!24m55!1m15!13m7!2b1!3b1!4b1!6i1!8b1!9b1!20b0!18m6!3b1!4b1!5b1!6b1!13b0!14b0!2b1!5m5!2b1!3b1!5b1!6b1!7b1!10m1!8e3!14m1!3b1!17b1!20m4!1e3!1e6!1e14!1e15!24b1!25b1!26b1!29b1!30m1!2b1!36b1!43b1!52b1!54m1!1b1!55b1!56m2!1b1!3b1!65m5!3m4!1m3!1m2!1i224!2i298!89b1!26m4!2m3!1i80!2i92!4i8!30m28!1m6!1m2!1i0!2i0!2m2!1i458!2i768!1m6!1m2!1i974!2i0!2m2!1i1024!2i768!1m6!1m2!1i0!2i0!2m2!1i1024!2i20!1m6!1m2!1i0!2i748!2m2!1i1024!2i768!34m17!2b1!3b1!4b1!6b1!7b1!8m4!1b1!3b1!4b1!6b1!9b1!12b1!14b1!20b1!23b1!25b1!26b1!37m1!1e81!42b1!46m1!1e9!47m0!49m1!3b1!50m72!1m68!2m7!1u3!4sOpen+now!5e1!9s0ahUKEwig-8Lpp63uAhVRs4sKHU9qAQkQ_KkBCN8KKBY!10m2!3m1!1e1!2m7!1u2!4sTop+rated!5e1!9s0ahUKEwig-8Lpp63uAhVRs4sKHU9qAQkQ_KkBCOAKKBc!10m2!2m1!1e1!2m7!1u1!4sCheap!5e1!9s0ahUKEwig-8Lpp63uAhVRs4sKHU9qAQkQ_KkBCOEKKBg!10m2!1m1!1e1!2m7!1u1!4sUpscale!5e1!9s0ahUKEwig-8Lpp63uAhVRs4sKHU9qAQkQ_KkBCOIKKBk!10m2!1m1!1e2!2m7!1u16!4sVisited!5e1!9s0ahUKEwig-8Lpp63uAhVRs4sKHU9qAQkQ_KkBCOMKKBo!10m2!16m1!1e1!2m7!1u16!4sHaven%27t+visited!5e1!9s0ahUKEwig-8Lpp63uAhVRs4sKHU9qAQkQ_KkBCOQKKBs!10m2!16m1!1e2!3m11!1u16!2m4!1m2!16m1!1e1!2sVisited!2m4!1m2!16m1!1e2!2sHaven%27t+visited!3m1!1u2!3m2!1u1!3e0!3m1!1u3!4BIAE!2e2!3m1!3b1!59BQ2dBd0Fn!65m0!69i540&q=coffee&tch=1&ech=1&psi=rZkJYJuoINHmrgTP1IVI.1611241905488.1' > correct.txt

# Compare f.txt and correct.txt in a text editor - they almost the same.

We didn't want to add new parameters (long, lat, alt) to our API for pagination specifically, so we tried to found ways to convert alt from zoom. But those formulas don't equal the altitude in pagination URLs that Google Maps use.

Also, altitude depends on the number of pixels per inch which is different on different devices, and Google re-scales map to fit all places on the map. (This was irrelevant actually). Milos combined multiple formulas from the JS code in Google Maps to the formula to convert altitude to zoom for the given latitude.

const EARTH_RADIUS_IN_METERS = 6371010
const TILE_SIZE = 256
const SCREEN_PIXEL_HEIGHT = 768

var zoom = (altitude, latitude) => Math.log(1 / Math.tan(Math.PI / 180 * 13.1 / 2) * (SCREEN_PIXEL_HEIGHT / 2) * 2 * Math.PI / (TILE_SIZE * altitude / (EARTH_RADIUS_IN_METERS * Math.cos(Math.PI / 180 * latitude)))) / Math.LN2;

zoom(24182.00605141337, 40.7455096);

// => 14

The last missing part is the reverse formula.

Convert zoom to an altitude

We've simplified the zoom formula in Wolfram Alpha.

The initial formula for zoom = f(alt, lat)

The simplified formula in Wolfram Alpha. Pretty nice, huh?

After several hours of reading middle-school math books on logarithmic equations, we reversed the formula.

The Ruby code of alt = f(zoom, lat):

EARTH_RADIUS_IN_METERS = 6371010
TILE_SIZE = 256
SCREEN_PIXEL_HEIGHT = 768
RADIUS_X_PIXEL_HEIGHT = 27.3611 * EARTH_RADIUS_IN_METERS * SCREEN_PIXEL_HEIGHT

def altitude(zoom, latitude)
  (RADIUS_X_PIXEL_HEIGHT * Math.cos((latitude * Math::PI) / 180)) / ((2 ** zoom) * TILE_SIZE)
end

Code to generate `pb` parameter

The pb parameter for the Google Maps pagination is a function of ll from the URL and theoffset.

ll can contain negative and positive latitude, longitude, and zoom. We decided to extract those parameters with regular expression.

PAGINATION_PARAMETERS_REGEX = %r{
  \A                                      # Start of string
  (?:\s*)                                 # initial possible whitespace
  @(?<latitude>[-+]?\d{1,2}(?:[.,]\d+)?)  # latitude: @10.78472
  (?:\s*,\s*)                             # separator between latitude and longitude
  (?<longitude>[-+]?\d{1,3}(?:[.,]\d+)?)  # longitude: @-110
  (?:\s*,\s*)                             # separator between longitude and zoom
  (?<zoom>\d{1,2}(?:[.,]\d+)?)z           # zoom: 9.22
  \z                                      # End of string
}x

Conversion of the ll URL parameter to pb for the specific results offset (start) on Google Maps looks like this.

def pagination(ll, start)
  extracted_parameters = ll.match(PAGINATION_PARAMETERS_REGEX)

  return "" unless extracted_parameters

  "!4m8!1m3!1d" +
    altitude(extracted_parameters[:zoom].to_f, extracted_parameters[:latitude].to_f) +
    "!2d" +
    extracted_parameters[:longitude] +
    "!3d" +
    extracted_parameters[:latitude] +
    "!3m2!1i1024!2i768!4f13.1!7i20!8i" +
    (start || "0") +
    "!10b1!12m25!1m1!18b1!2m3!5m1!6e2!20e3!6m16!4b1!23b1!26i1!27i1!41i2!45b1!49b1!63m0!67b1!73m0!74i150000!75b1!89b1!105b1!109b1!110m0!10b1!16b1!19m4!2m3!1i360!2i120!4i8!20m65!2m2!1i203!2i100!3m2!2i4!5b1!6m6!1m2!1i86!2i86!1m2!1i408!2i240!7m50!1m3!1e1!2b0!3e3!1m3!1e2!2b1!3e2!1m3!1e2!2b0!3e3!1m3!1e3!2b0!3e3!1m3!1e8!2b0!3e3!1m3!1e3!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e9!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e10!2b0!3e4!2b1!4b1!9b0!22m3!1s!2z!7e81!24m55!1m15!13m7!2b1!3b1!4b1!6i1!8b1!9b1!20b0!18m6!3b1!4b1!5b1!6b1!13b0!14b0!2b1!5m5!2b1!3b1!5b1!6b1!7b1!10m1!8e3!14m1!3b1!17b1!20m4!1e3!1e6!1e14!1e15!24b1!25b1!26b1!29b1!30m1!2b1!36b1!43b1!52b1!54m1!1b1!55b1!56m2!1b1!3b1!65m5!3m4!1m3!1m2!1i224!2i298!89b1!26m4!2m3!1i80!2i92!4i8!30m28!1m6!1m2!1i0!2i0!2m2!1i458!2i768!1m6!1m2!1i974!2i0!2m2!1i1024!2i768!1m6!1m2!1i0!2i0!2m2!1i1024!2i20!1m6!1m2!1i0!2i748!2m2!1i1024!2i768!34m16!2b1!3b1!4b1!6b1!8m4!1b1!3b1!4b1!6b1!9b1!12b1!14b1!20b1!23b1!25b1!26b1!37m1!1e81!42b1!46m1!1e9!47m0!49m1!3b1!50m53!1m49!2m7!1u3!4s!5e1!9s!10m2!3m1!1e1!2m7!1u2!4s!5e1!9s!10m2!2m1!1e1!2m7!1u16!4s!5e1!9s!10m2!16m1!1e1!2m7!1u16!4s!5e1!9s!10m2!16m1!1e2!3m11!1u16!2m4!1m2!16m1!1e1!2s!2m4!1m2!16m1!1e2!2s!3m1!1u2!3m1!1u3!4BIAE!2e2!3m1!3b1!59B!65m0!69i540"
end

Putting all together

# https://regex101.com/r/nOoiJ6/2
PAGINATION_PARAMETERS_REGEX = %r{
  \A                                      # Start of string
  (?:\s*)                                 # initial possible whitespace
  @(?<latitude>[-+]?\d{1,2}(?:[.,]\d+)?)  # latitude: @10.78472
  (?:\s*,\s*)                             # separator between latitude and longitude
  (?<longitude>[-+]?\d{1,3}(?:[.,]\d+)?)  # longitude: @-110
  (?:\s*,\s*)                             # separator between longitude and zoom
  (?<zoom>\d{1,2}(?:[.,]\d+)?)z           # zoom: 9.22
  \z                                      # End of string
}x

EARTH_RADIUS_IN_METERS = 6371010
TILE_SIZE = 256
SCREEN_PIXEL_HEIGHT = 768
RADIUS_X_PIXEL_HEIGHT = 27.3611 * EARTH_RADIUS_IN_METERS * SCREEN_PIXEL_HEIGHT

def pagination(ll, start)
  extracted_parameters = ll.match(PAGINATION_PARAMETERS_REGEX)

  return "" unless extracted_parameters

  "!4m8!1m3!1d" +
    altitude(extracted_parameters[:zoom].to_f, extracted_parameters[:latitude].to_f) +
    "!2d" +
    extracted_parameters[:longitude] +
    "!3d" +
    extracted_parameters[:latitude] +
    "!3m2!1i1024!2i768!4f13.1!7i20!8i" +
    (start || "0") +
    "!10b1!12m25!1m1!18b1!2m3!5m1!6e2!20e3!6m16!4b1!23b1!26i1!27i1!41i2!45b1!49b1!63m0!67b1!73m0!74i150000!75b1!89b1!105b1!109b1!110m0!10b1!16b1!19m4!2m3!1i360!2i120!4i8!20m65!2m2!1i203!2i100!3m2!2i4!5b1!6m6!1m2!1i86!2i86!1m2!1i408!2i240!7m50!1m3!1e1!2b0!3e3!1m3!1e2!2b1!3e2!1m3!1e2!2b0!3e3!1m3!1e3!2b0!3e3!1m3!1e8!2b0!3e3!1m3!1e3!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e9!2b1!3e2!1m3!1e10!2b0!3e3!1m3!1e10!2b1!3e2!1m3!1e10!2b0!3e4!2b1!4b1!9b0!22m3!1s!2z!7e81!24m55!1m15!13m7!2b1!3b1!4b1!6i1!8b1!9b1!20b0!18m6!3b1!4b1!5b1!6b1!13b0!14b0!2b1!5m5!2b1!3b1!5b1!6b1!7b1!10m1!8e3!14m1!3b1!17b1!20m4!1e3!1e6!1e14!1e15!24b1!25b1!26b1!29b1!30m1!2b1!36b1!43b1!52b1!54m1!1b1!55b1!56m2!1b1!3b1!65m5!3m4!1m3!1m2!1i224!2i298!89b1!26m4!2m3!1i80!2i92!4i8!30m28!1m6!1m2!1i0!2i0!2m2!1i458!2i768!1m6!1m2!1i974!2i0!2m2!1i1024!2i768!1m6!1m2!1i0!2i0!2m2!1i1024!2i20!1m6!1m2!1i0!2i748!2m2!1i1024!2i768!34m16!2b1!3b1!4b1!6b1!8m4!1b1!3b1!4b1!6b1!9b1!12b1!14b1!20b1!23b1!25b1!26b1!37m1!1e81!42b1!46m1!1e9!47m0!49m1!3b1!50m53!1m49!2m7!1u3!4s!5e1!9s!10m2!3m1!1e1!2m7!1u2!4s!5e1!9s!10m2!2m1!1e1!2m7!1u16!4s!5e1!9s!10m2!16m1!1e1!2m7!1u16!4s!5e1!9s!10m2!16m1!1e2!3m11!1u16!2m4!1m2!16m1!1e1!2s!2m4!1m2!16m1!1e2!2s!3m1!1u2!3m1!1u3!4BIAE!2e2!3m1!3b1!59B!65m0!69i540"
end

def altitude(zoom, latitude)
  ((RADIUS_X_PIXEL_HEIGHT * Math.cos((latitude * Math::PI) / 180)) / ((2 ** zoom) * TILE_SIZE)).to_s
end

Parse the paginated data

We already have an API to scrape Google Maps that extracts data from the inline JavaScript in the HTML. Milos refactored it to support extraction from inline JS in HTML and from pagination responses. What we can say here is our parser gets the data from the deeply nested arrays and objects.

It's possible to extract data from complex single-page applications without browser automation. For us, it's more fun to understand how the scraped website works instead of tuning timeouts in waitFor function calls. It also runs faster and is simpler to maintain. If this is something that excites you, we'd love for you to join us.

Playwright’s getByRole is 1.5x slower than CSS selectors

Illia Zub — Tue, 14 Nov 2023 19:05:15 +0000

Playwright’s getByRole uses of querySelectorAll('*') and matches elements by the accessible name.

Introduction

In the realm of web automation and testing with Playwright, understanding the performance of various locator strategies is key. This article delves into the getByRole locator's efficiency compared to CSS selectors, offering insights into the technical workings and practical implications of these choices.

I wanted to use Page#getByRole's underlying CSS selectors in SerpApi code base. But the getByRole locator was 1.5 times slower compared to the standard CSS selectors, prompting an investigation into the root cause. This performance discrepancy, likely stemming from Playwright’s use of querySelectorAll('*') and matching elements by the accessible name, raises essential considerations for developers prioritizing speed in their automation scripts.

Deep Dive: How `getByRole` Works

The getByRole function in Playwright is more than just a method to locate web elements; it's a complex mechanism with multiple layers of interaction within the Playwright architecture. Let's demystify this process with an example. Consider this code:

await page.getByRole('button', { name: 'Enter address manually' }).click();

This command sets off a cascade of actions within Playwright:

Page#getByRole creates a Locator
Locator#click delegates call to Frame#click passing the Locator#_selector
Frame#click delegates to Channel#click. Frame inherits _channel from ChannelOwner. ChannelOwner#_channel is a JS Proxy object based on the EventEmitter
Client Frame dispatches an event to the server Frame#click
FrameSelector#resolveInjectedForSelector injects the FrameExecutionContext#injectedScript script to the page, controlled by Playwright. The InjectedScript#constructor adds the engine for getByRole locator.
createRoleEngine calls parseAttributeSelector and queryRole
queryRole calls querySelectorAll('*') and matches the element

Compared to the getByRole, the locator with the regular CSS selector just traverses DOM and is 1.5 times faster.

Performance

Comparative tests reveal that a regular locator using CSS selectors outperforms getByRole by 1.5 times. Interestingly, the $.then method trailed, being 2x slower in our tests.

console.time("getByRole");
for (let i = 0; i < 100; i++) {
  const textContent = await page.getByRole('button', { name: 'Enter address manually' }).textContent()
}
console.timeEnd("getByRole");

console.time("locator");
for (let i = 0; i < 100; i++) {
  const textContent = await page.locator('.ektjNL').textContent()
}
console.timeEnd("locator");

console.time("$.then");
for (let i = 0; i < 100; i++) {
  const textContent = await page.$('.ektjNL').then(e => e.textContent())
}
console.timeEnd("$.then");

// Output:
// getByRole: 677.5ms
// locator: 497.306ms
// $.then: 1.135s

Conclusion

In web automation, understanding Playwright's locators — getByRole and CSS selectors — is key. getByRole excels in clarity, while CSS selectors win in speed. This matters in large test suites where every second counts. Choose wisely: getByRole for readability, CSS selectors for efficiency.

When counting lines in Ruby randomly failed SerpApi deployments

Illia Zub — Thu, 07 Sep 2023 09:49:04 +0000

Recently, we observed the occasional failing deployments only on two of our servers. The failed servers even were closing my regular SSH connection. In this story, you'll learn how we reduced memory usage and made one piece of SerpApi code 1.5x faster.

TL;DR

str.count($/) was 1.5x faster compared to str.lines.count and didn't allocate additional memory.

Investigation

Only two servers faced the random failed deployments.

#<Thread:0x000055a170560e70 digital_ocean.rb:80 run> terminated with exception (report_on_exception is true):
SerpApi/vendor/bundle/ruby/2.7.0/gems/net-ssh-5.2.0/lib/net/ssh/transport/server_version.rb:54:in `readpartial': Connection reset by peer (Errno::ECONNRESET)

These servers also randomly closed my SSH connections.

$ ssh server-2
Last login: Fri Feb 24 14:23:29 2023 from {remote_ip}
client_loop: send disconnect: Broken pipe

DigitalOcean server's graphs shown that memory usage was near to 95% percent on these two servers. Load average was occasionally peaking at 12 compared to 2 on other servers.

We checked the Puma server flamegraph. Most of the wall time on production were the SearchSplitter#do_one_request and Puma thread pool.

We used rbspy to generate the flamegraph:

$ rbspy record -p $PID_OF_PUMA_PROCESS

The flamegraph didn't reveal anything actionable and we moved to memory profiling.

Memory profiling

Here's the script we used:

require 'memory_profiler'

user = User.find_by(email: "me")

report = MemoryProfiler.report do
  threads = []

  (1..5).map do |i|
    (1..5).map do |j|
      threads << Thread.new {
        search = Search.new(engine: "google", q: "#{i} * #{j}", user: user)
        SearchProcessor.process(search)
      }
    end
  end

  threads.each(&:join)
end

report.pretty_print

It turned out that the top allocator was line counting in the response validator.

elsif response[:html].lines.count < 50 && response[:html].match?(/<div id=\"main\"><div id=\"cnt\"><div class=\"dodTBe\" id=\"sfcnt\"><\/div><H1>.+<\/H1>.+<p>.+<\/p>$/)

Why the String#lines and Array#count were the top allocators of the entire app?

`String#lines`

The HTML file size varies from 180 KB (regular organic results) to 1.3 MB (Google Shopping with num=100). The String#lines allocated an array multiple time per each search because we send multiple requests concurrently per each search.

Thanks to @guilhermesimoes's gist, we found that str.each_line.count should be faster. But it was not optimal and we found a way to improve the solution.

Solution

The solution was super simple — str.count($/). Here's the diff:

- elsif response[:html].lines.count < 50 && response[:html].match?(/<div id=\"main\"><div id=\"cnt\"><div class=\"dodTBe\" id=\"sfcnt\"><\/div><H1>.+<\/H1>.+<p>.+<\/p>$/)
+ elsif response[:html].count($/) < 50 && response[:html].match?(/<div id=\"main\"><div id=\"cnt\"><div class=\"dodTBe\" id=\"sfcnt\"><\/div><H1>.+<\/H1>.+<p>.+<\/p>$/)

To make sure the problem was solved, we benchmarked multiple ways of counting string lines in Ruby. We reused and adopted the gist above to exclude File#read from the benchmark and added String#count to the benchmark.

Benchmark

String#count was 1.5 times faster that other options:

Warming up --------------------------------------
                size    31.000  i/100ms
              length    75.000  i/100ms
               count    77.000  i/100ms
   each_line + count    81.000  i/100ms
           count($/)   196.000  i/100ms
Calculating -------------------------------------
                size      1.529k (±33.9%) i/s -      4.774k in   5.015361s
              length      1.434k (±38.8%) i/s -      5.025k in   5.139834s
               count      1.335k (±40.7%) i/s -      4.697k in   5.079353s
   each_line + count      1.411k (±39.5%) i/s -      5.022k in   5.110146s
           count($/)      2.231k (± 2.6%) i/s -     11.172k in   5.012323s

Comparison:
           count($/):     2230.5 i/s
                size:     1529.0 i/s - 1.46x  (± 0.00) slower
              length:     1434.2 i/s - 1.56x  (± 0.00) slower
   each_line + count:     1411.0 i/s - 1.58x  (± 0.00) slower
               count:     1334.9 i/s - 1.67x  (± 0.00) slower

Here's the script:

require "benchmark/ips"

html = File.read(Rails.root.join("spec/data/google/superhero-movies-mobile-63f582a0defa1345501c6b50-2023-02-23.html"))

Benchmark.ips do |x|
  x.report("size") { html.lines.size < 50 }
  x.report("length") { html.lines.length < 50 }
  x.report("count") { html.lines.count < 50 }

  x.report("each_line + count") { html.each_line.count < 50 }

  x.report("count($/)") { html.count($/) < 50 }

  x.compare!
end

Memory usage

count($/) doesn't allocate a new array compared to lines/each_line/etc.

We used the awesome heap-profiler and heapy Ruby gems to profile memory usage.

`lines`/`readlines`/`each_line`/etc.

html.lines.count allocated the new array and referenced the original string for each iteration in the benchmark.

$ bundle exec heapy read ./tmp/lines_count/allocated.heap 49 --lines=1

Analyzing Heap (Generation: 49)
-------------------------------

allocated by memory (204879705) (in bytes)
==============================
  204872652  tmp/html_length_vs_count_vs_size_bench.rb:6

object count (5406)
==============================
  5301  tmp/html_length_vs_count_vs_size_bench.rb:6

High Ref Counts
==============================
  5300  tmp/html_length_vs_count_vs_size_bench.rb:6

We also used the predefined $/ line separator to allocate even less memory.

`count($/)`

Most of these memory allocations and all of the object reference counts were gone when we used the String#count($/).

$ bundle exec heapy read ./tmp/count_nl/allocated.heap 48 --lines=1

Analyzing Heap (Generation: 48)
-------------------------------

allocated by memory (2547465) (in bytes)
==============================
  2540804  tmp/html_length_vs_count_vs_size_bench.rb:4

object count (105)
==============================
  27  /usr/local/lib/ruby/gems/2.7.0/gems/activesupport/lib/active_support/deprecation/proxy_wrappers.rb:172

High Ref Counts
==============================
  73  /usr/local/lib/ruby/gems/2.7.0/gems/activesupport/lib/active_support/deprecation/proxy_wrappers.rb:172

Code

require "heap-profiler"

HeapProfiler.report(Rails.root.join('tmp/lines_count')) do
  html = File.read(Rails.root.join("spec/data/google/superhero-movies-mobile-63f582a0defa1345501c6b50-2023-02-23.html"))

  100.times { html.lines.count < 50 }
  # 100.times { html.count($/) < 50 }
end

Comparison process

The heap diff comparison was a bit manual because the heapy diff did't provide a diff. We commented / uncommented 100.times { html.lines.count < 50 } and replaced paths in the command above.

## Profile heap
$ bundle exec rails r tmp/html_length_vs_count_vs_size_bench.rb

## Read summary of heap allocations
$ bundle exec heapy read ./tmp/count_nl/allocated.heap

## Read a specific generation (48) limiting number of lines to output (1)
$ bundle exec heapy read ./tmp/count_nl/allocated.heap 48 --lines=1

Results

Immediately after the fix was deployed, memory usage on the affected servers decreased and stabilized. Then memory usage fluctuated again, but deployments and SSH connections stabilized.

Observations and final thoughts

The initial assumption was the Puma graceful restart. During the phased-restart, Puma spawned additional workers to switch to the new code version (which was expected). It wasn't clear why SSH connections were dropping on two DigitalOcean droplets only.

Doubling the amount of RAM would also solve the problem, but it wouldn't be as efficient at this point. The fix was deployed half a year ago and the issue is definitely solved.

Update Sep 20th, 2023

Thanks to @Freaky from Reddit for a wonderful feedback and cooperation:

@Freaky tracked down a performance regression in String#count in 3.1+
@Freaky brought his old SIMD bytecount C port out of mothballs
MRI maintainer nobu is evaluating it for inclusion in MRI
Freaky noted the bytecount Rust crate uses an SSE4.1 intrinsic in SSE2 code and submitted a fix
A similar fix for the aho-corasick Rust crate was made in response

I love it when you get this ripple effect from something that initially seems pretty innocuous.

If you enjoy working on such challenges, come work here with us: https://serpapi.com/careers#open-roles

Reverse Engineering Poocoin API (Part 1)

Illia Zub — Thu, 24 Aug 2023 08:25:40 +0000

What will be scraped
Explanation
Links
Outro

What will be scraped

Explanation

We want to reverse engineer URL data parameter:

# full url 
https://poocoin.app/api2/candles-bsc?data=[YmLNHK6TU[Kblm4UXqKeF2FTYSSW2KsZ32XfnO6TU[KblJ1UVeONWKGSYebblF{U2S[fWqIVYeObmVzXYqCN1:VTUCQSGmOWHiWUWSCOl6VWU[OSFG2UVSCe2eqTYOKcYixZmetNFmrc4qOblG{TX25e2GYVnukcW[7Z4mKOlmrRkSTS2FyXXuHb1:IXUS[bl1zUVeSN16uVYiOb2qsVWSKfl2GXUSSb1[IUlSLcWqVRYeOblqFVnmKd1mucIWlS2[6[H2Hd1mrc3mOWG[1TXm4bWmuSoqbWYi4TXqwbV2J[{GQSWl1Uoq[OF6V[HiOSFqGUnqkNl2sWYeOWFG5XX2KNWG7VUKSWHirUWWXSV6FVlW[flVzTX5xQR%3E%3E

# data parameter
[YmLNHK6TU[Kblm4UXqKeF2FTYSSW2KsZ32XfnO6TU[KblJ1UVeONWKGSYebblF{U2S[fWqIVYeObmVzXYqCN1:VTUCQSGmOWHiWUWSCOl6VWU[OSFG2UVSCe2eqTYOKcYixZmetNFmrc4qOblG{TX25e2GYVnukcW[7Z4mKOlmrRkSTS2FyXXuHb1:IXUS[bl1zUVeSN16uVYiOb2qsVWSKfl2GXUSSb1[IUlSLcWqVRYeOblqFVnmKd1mucIWlS2[6[H2Hd1mrc3mOWG[1TXm4bWmuSoqbWYi4TXqwbV2J[{GQSWl1Uoq[OF6V[HiOSFqGUnqkNl2sWYeOWFG5XX2KNWG7VUKSWHirUWWXSV6FVlW[flVzTX5xQR%3E%3E

data parameter contains token id, time interval, date, and limit argument which I believe is for candles amount to display.

Open dev tools, find relevant XHR request and go to JS source:

Format JavaScript code:

Go up in the stack trace to see the formation of data URL argument:

Evaluate F, W, B I variables to understand what's happening:

B = w(476):

w() returns function name:

var n = ["map", "70209pSjRdP", "13mXPpTk", "32970804XTFOSy", "33EDoYDa", "16914prTfjD", "charCodeAt", "680Pigxyq", "5194335PIAMSW", "stringify", "430dfxvAD", "21ASPctV", "&l=1", "4868dULKHq", "955100otFPGh", "substring", "QWRkcmVzcyI6IjB4MGM1REEwZjA3OTYyZGQwMjU2YzA3OTI0ODY", "4483450KFkYTE", "2303RvgaFj", "lpr", "host", "indexOf"];

function w(e, t) { return n[e - 467] }

I contains base64 encoded characters shifted by one charater code to the right:

let obfuscatedFunctionName = w(476) // B = w(476),
obfuscatedFunctionName === n[9] && obfuscatedFunctionName === "QWRkcmVzcyI6IjB4MGM1REEwZjA3OTYyZGQwMjU2YzA3OTI0ODY" // => true

let base64CharactersOfParamsBase64 = btoa("" + prefixOfEncodedParams + obfuscatedFunctionName + suffixOfEncodedParams).split("") // I = (I = (I = (I = (I = btoa("" + W + B + S)).split(""))

let base64CharactersPlusOneOfParamsBase64 = base64CharactersOfParamsBase64.map(function(e) {
  let charCodePlusOne = e.charCodeAt() + 1 // e[w(488)](0) + 1; w(488) === "charCodeAt"
  return charCodePlusOne
}).map(function(e) { // w(482) === "map"
  return String.fromCharCode(e)
}).join("")

Which is effectively the same as:

let base64CharactersPlusOneOfParamsBase64 = base64CharactersOfParamsBase64.map(e => String.fromCharCode(e.charCodeAt() + 1)).join("")

Example value and test

base64CharactersPlusOneOfParamsBase64 === '[YmLNHK6TU[Kblm4UXqKeF2FTYSSW2KsZ32XfnO6TU[KblJ1UVeONWKGSYebblF{U2S[fWqIVYeObmVzXYqCN1:VTUCQSGmOWHiWUWSGOl6VRU[OSFG2UVSCe2eqTYOKcYixZmetNFmrc4qOblG{TX25e2GYVnukcW[7Z4mKOlmrRkSOWGKFUmSsNF2rTYmOWFFzUXqofmqIVUKTSFVyUmeKOWqFRYe[WHtxUUKKOV6FSUGOflFzUnmKd1mucIWlS2[6[H2Hd1mrc3mOWG[1TXm4bWmuSoqbWYi4TXqwbV2J[4mPSHyrXlSCNV6FXUWPNmlxUWeSN11xXYiSWHe5Xn2Ge2KrWYmPfnyuXUKPcF1zUlePflKrTX5xQR>>' // => true
base64CharactersPlusOneOfParamsBase64 === I // => true

Now compare it with string in request:

https://poocoin.app/api2/candles-bsc?data=[YmLNHK6TU[Kblm4UXqKeF2FTYSSW2KsZ32XfnO6TU[KblJ1UVeONWKGSYebblF{U2S[fWqIVYeObmVzXYqCN1:VTUCQSGmOWHiWUWSGOl6VRU[OSFG2UVSCe2eqTYOKcYixZmetNFmrc4qOblG{TX25e2GYVnukcW[7Z4mKOlmrRkSOWGKFUmSsNF2rTYmOWFFzUXqofmqIVUKTSFVyUmeKOWqFRYe[WHtxUUKKOV6FSUGOflFzUnmKd1mucIWlS2[6[H2Hd1mrc3mOWG[1TXm4bWmuSoqbWYi4TXqwbV2J[4mPSHyrXlSCNV6FXUWPNmlxUWeSN11xXYiSWHe5Xn2Ge2KrWYmPfnyuXUKPcF1zUlePflKrTX5xQR%3E%3E

In the next part, we'll write the actual code that will construct the URL and retrieve the data.

Outro

If you have anything to share, any questions, suggestions, or something that isn't working correctly, feel free to reach out via Twitter at @ilyazub.

P.S. Thanks to @dmitryzub for help on this project and for co-authoring this article.

Mastering SerpApi: An In-depth Features, Best Practices, and Competitive Edge

Illia Zub — Thu, 24 Aug 2023 07:58:30 +0000

In this episode, Ryan and Illia share answers to frequently asked questions about SerpApi usage, features, and best practices.

YouTube | Spotify

Episode Breakdown

00:00:00 - Video Intro
00:01:30 - An introduction to SerpApi, including its key features and benefits
00:02:55 - Comparison of SerpApi to competitors discussing why SerpApi is a superior choice
00:04:00 - Why use SerpApi compared to an official Google API
00:04:55 - Clarification about data warehouse
00:05:33 - SLA and success rate
00:07:03 - Addressing the legality of web scraping, terms of service, and the US Legal Shield
00:08:30 - the US Legal Shield offering
00:09:05 - Understanding searches and credits, including how they are counted, and discussing duplicates
00:11:00 - Search Archive API, no_cache parameter
00:12:00 - async parameter and multi-threading
00:14:45 - Ludicrous Speed plans for fast latency-sensitive use-cases
00:15:45 - API Status page showcase
00:16:40 - Hourly throughput limit
00:18:15 - Traffic spike management
00:19:30 - Incident response and downtime handling
00:21:40 - unit, integration, and e2e tests
00:22:30 - Experimental AI-based data extraction
00:23:40 - Sponsorship and discounts
00:25:30 - Langchain, BagyAGI and SerpApi cooperation started during Replit's hackathon about LLMs
00:28:00 - Google SERPs and Google Ads inconsistencies
00:30:00 - Improving search engine ranking
00:32:00 - SerpApi's followers on Twitter SEO
00:33:48 - Using the dashboard effectively, including how to find the API key and how to use the account API for credit tracking
00:42:00 - Wrapping up and explaining what will be discussed in the next webinar

At its core, the SerpApi Podcast revolves around search engines results scraping, covering areas like parsing, evasion of blocking, web automation, proxies, the legality of scraping, performance enhancement, data extraction, and validation.

Practical applications for SERP data scraping span a wide array, including programmatic search engine optimization (SEO) and local SEO, machine learning (ML), artificial intelligence (AI), large language models (LLMs), news monitoring, open-source intelligence (OSINT), voice assistants,
and e-commerce competitor analysis.

Software Engineer's Life Balance with Dmytro Vasyliev | #SerpApiPodcast, Episode 10

Illia Zub — Sat, 24 Jun 2023 19:51:00 +0000

In the 10th episode of SerpApi Podcast, we discuss work-life balance with Dmytro Vasyliev: "How can you find a balance between your professional activity and personal life?", "How can you maintain your health and avoid burnout?", etc.

Watch this episode · Listen this episode

Chapters

[00:00:00] Introduction
[00:02:03] What is life balance for Dmytro?
[00:03:00] How Dmytro organizes his day?
[00:05:10] Was you always organized in life or you had to do something in order to feel better?
[00:06:35] About life spheres
[00:12:10] Burnout and how to handle it
[00:14:51] Work-life balance: what helps to be more focused?
[00:29:55] Work from home or from office?
[00:33:08] Work with a coach: when, why and what a result is
[00:38:35] The first time with a psychologist
[00:42:20] Training with a personal trainer - how does it works?
[00:46:26] A few exercises to keep an eye on life balance
[00:50:32] Organization of a to-do list
[00:58:08] Epilogue

SerpApi Podcast is about SERP (search engine results pages) and e-commerce data scraping: parsing, circumvention of blocking, web automation, proxies, legal part of scraping, performance, data extraction and validation.

Use cases for SERP data scraping: programmatic search engine optimization (SEO) and local SEO, machine learning (ML), artificial intelligence (AI), large language models (LLMs), news monitoring, open-source intelligence (OSINT), voice assistants, e-commerce competitor research.

The podcast is brought to you by SerpApi team: https://serpapi.com

From Software Engineer to Manager with Oleksii Gyturo | #SerpApiPodcast, Episode 9

Illia Zub — Fri, 23 Jun 2023 16:46:21 +0000

In episode we discuss the career growth in management. Oleksii, as an aspiring manager, faced a multitude of questions and challenges. Career growth depends not only on how well a manager copes with the tasks, but also on how open they are to new knowledge and experience.

Watch this episode · Listen this episode

Show notes

[00:00:00] Introduction
[00:02:11] Did you grow from another position or have been hired as the head of mobile?
[00:03:29] You have started as an individual contributor but then you started working on more and more why?
[00:06:55] What was a result of involving you in a leadership strategy?
[00:10:32] Experience exchange (from respondent to interviewer)
[00:20:14] How do you deal with multiple sources of information about the business?
[00:36:04] What is Illia’s mission at SerpApi?
[00:56:02] Long term planning or being spontaneous? How did you manage that?
[01:06:38] Who is an engineering manager?
[01:10:09] Why have you decided to consult businesses?
[01:12:29] Do you treat consulting as a business?
[01:14:34] Epilogue (few words from Oleksii)

The podcast is brought to you by SerpApi team: https://serpapi.com

Insights from an Engineering Director (#SerpApiPodcast, Ep. 8)

Illia Zub — Wed, 17 May 2023 14:14:59 +0000

The success of the company primarily depends on the effective actions and efforts of talented people. Today we would like to introduce you to Milos, one of the Engineering Directors at SerpApi.

Watch this episode · Listen this episode

Show Notes

[00:00] - Intro
[01:26] - What are Milos' responsibilities at SerpApi?
[02:19] - How did you become a manager?
[03:25] - How do you feel about switching from individual contributor to manager?
[05:16] - Where did you gain that knowledge, internally or externally?
[07:41] - Do you prioritize individual tasks?
[09:46] - How do you feel about contributing via others instead of contributing personally?
[11:18] - What is the upper limit for you (number of subordinates)?
[13:22] - What's a goal of a weekly meeting?
[15:13] - How does your team interact with you?
[16:36] - How do you understand and know what is valuable for the business?
[18:19] - Hobbies
[20:29] - How do you organize your time?
[22:23] - Do you work without interruption?
[23:12] - What do you put in the calendar besides meeting?
[24:26] - Milos' actions when he can’t follow the plan ?
[25:00] - What do you do when you finish your work early?
[26:12] - Long term plans or small steps?
[30:40] - How have you changed since you started working in SerpApi?
[32:41] - What are your plans for the future?
[34:20] - What is the feedback about you from the people you manage?
[36:36] - How do you manage disagreements with the people you manage?
[38:15] - How do you decide when to say "no"?
[40:12] - How do you manage to deal with a large amount of work?
[45:48] - What would you like to be changed in your work?
[50:59] - What are your suggestions for yourself in the past?
[53:26] - How do you deal with anxiety?
[56:16] - Do you have a role model?
[57:13] - Epilogue (few words from Milos)

Links and resources

#SerpApiPodcast is about SERP (search engine results pages) and ecommerce data scraping: parsing, circumvention of blocking, web automation, proxies, legal part of scraping, performance, data extraction and validation.

Use cases for SERP data scraping: SEO, local SEO, ML models, news monitoring, OSINT, voice assistant, ecommerce competitor research.

The podcast is brought to you by SerpApi team: https://serpapi.com

How to reverse engineer a JSON API on a single page application

Illia Zub — Mon, 26 Dec 2022 13:13:19 +0000

Websites like Bing Image Search and Walmart render pages with JavaScript and deliver page content via JSON APIs. While it's possible to scrape dynamic web pages using the browser automation, I prefer fetching data from the API endpoints directly. It usually (not always) works faster and more reliable.

I was debugging the Bing Image Search to help implementing our new Bing Reverse Image Search API. Initially, I've used mitmproxy because Ctrl+Shift+F in the browser dev tools haven't found the request. Then I've figured out how to filter network requests in the browser dev tools, examined the response, and made a draft data adapter.

Algorithms to reverse engineer a JSON API on the SPA

Two ways I've used to reverse engineer a JSON API used on the Bing Image Search: mitmproxy and browser developer tools. I explain the devtools process because it's used more often.

Browser devtools

Ctrl+F in the Network tab of browser dev tools.

Go to the Preview tab of the JSON response.
Expand JS object recursively (my Brave Browser doesn't search in the collapsed JSON 😕)

Ctrl+F the target string

Copy property path

Navigate up and down in JS object (with arrow keys) to learn its structure and create an adapter.
Copy as cURL and transform response with jq to check my assumption.

`mitmproxy`

Ctrl+Shift+F in the browser dev tools no longer searches across all responses.

I've proxied the browser network connections via mitmproxy. Then filtered response bodies with ~bs "TEXT_FROM_THE_HTML_ELEMENT_I_"LOOKING_FOR".

Start mitmproxy with view filter

$ mitmproxy --view-filter '~bs "Freshsales"'

Start chromium-based browser with the target URL and the following flags and parameters

Proxy requests via mitmproxy: --proxy-server='http://127.0.0.1:8080'.
Use incognito mode (1) with a temporary user profile (2) ignoring insecure connections (3) and certificate errors (4): --temp-profile -incognito --user-data-dir="mktemp -d" --no-first-run --ignore-certificate-errors --allow-insecure-localhost. (I ignore certificate errors in a temporary browser profile to not install mitmproxy's certificates system-wide.)

$ brave-browser 'https://www.bing.com/images/search?view=detailV2&insightstoken=bcid_RLKVsIV2BwkFXg*ccid_spWwhXYH&form=SBIHMP&iss=SBIUPLOADGET&sbisrc=ImgPicker&idpbck=1&sbifsz=927+x+524+%c2%b7+25.15+kB+%c2%b7+png&sbifnm=serpapi-serpbear.png&thw=927&thh=524&ptime=223&dlen=34344&expw=798&exph=451&selectedindex=0&id=-1051855017&ccid=spWwhXYH&vt=2&sim=11' --proxy-server='http://127.0.0.1:8080'  --temp-profile -incognito --user-data-dir="`mktemp -d`" --no-first-run --ignore-certificate-errors --allow-insecure-localhost

mitmproxy will display the matched requests

Conclusion

mitmproxy can be used to find the HTTP request with the needed data in addition browser dev tools. At some point, I'll explore tcpdump and wireshark to reverse engineer websites for web scraping and share the learnings with you.

If you have anything to share, any questions, suggestions, or something that isn't working correctly, feel free to reach out via Twitter at @ilyazub_, or @serp_api, or Mastodon at @iz.

Avoiding Puppeteer Antipatterns

Illia Zub — Wed, 25 May 2022 14:05:42 +0000

Puppeteer is a popular browser automation library for NodeJS commonly used for web scraping and end-to-end testing. Since Puppeteer offers a rich API that performs complex interactions with the browser in real-time, there's plenty of room for misunderstandings and antipatterns to creep into your scripts.

In this post, we'll share 9 Puppeteer antipatterns I've used or seen in Puppeteer code over the past few years. While the list isn't exhaustive, we hope it'll elevate your understanding and appreciation of the tool.

I've assumed that readers arrive with a working familiarity with Puppeteer and have written at least a few scripts with it, enough to have encountered some of the quirks and pitfalls of automation.

Overusing `waitForTimeout`

Puppeteer's readme says it best:

Puppeteer has event-driven architecture, which removes a lot of potential flakiness. There's no need for evil sleep(1000) calls in puppeteer scripts.

The trouble is, Puppeteer has page.waitForTimeout(milliseconds), identical semantics to the evil sleep(1000) call! It can be handy to have this escape hatch in the API, but it's also easy to abuse.

Sleeping is evil because it introduces a race condition that involves two possible outcomes.

Sleeping causes a race condition between the browser state and the Puppeteer driver

One outcome is when the sleep duration is too optimistic and the driver wakes up before the browser is in the desired state. In this case, the issue is correctness: the driver will likely wind up missing data from responses that haven't arrived, or throw errors when attempting to manipulate elements that don't exist.

The other outcome is when the sleep duration is too pessimistic and the browser reaches the desired state before the driver wakes up. In this case, the issue is efficiency: elapsed time between the desired state appearing and the driver waking up is wasted. Even a small extra delay becomes painful when incurred repeatedly, such as during automated testing.

Furthermore, a sleep that seems to work can create an illusion of robustness. A duration that's long enough today might be too short tomorrow. Sleeping introduces an overfitted, environment-specific value that may work on one machine, but can easily fail on another. That other machine is often a production deploy on the cloud, where debugging discrepancies against a working local version can be difficult.

The alternatives to waitForTimeout include waitForSelector, which blocks until a selector appears, or the more general waitForFunction, which blocks until a predicate condition becomes true. Puppeteer tests the condition in a tight requestAnimationFrame loop or upon DOM changes with a MutationObserver. Both waitForSelector and waitForFunction adhere to the event-driven model and eliminate race conditions.

There are plenty of occasions when page.waitForTimeout is appropriate, however. It can help with debugging, throttling interaction to simulate human behavior, rate limiting polling of a resource and as a last resort for handling pesky situations that don't respond well to other approaches. But sleeping as the default option when it's entirely possible to block on an explicit predicate is an antipattern.

Assuming Puppeteer's API works like the native browser API

Puppeteer's API has different semantics from the native browser API. For example, Puppeteer's page.click() seems like a straightforward wrapper on the browser's native HTMLElement.click(), but it actually operates quite differently and hides a fair amount of complexity under the hood.

Instead of invoking the click event handler directly on the element as the native .click() does, clicking in Puppeteer scrolls the element into view, moves the mouse onto the element, presses one of a few mouse buttons, optionally triggers a delay, then releases the mouse button. You can also trigger multiple clicks. In other words, Puppeteer performs a click like a human would.

Neither approach is better, but it's a mistake to assume they're the same and indiscriminately use one or the other across the board.

There are times when using the native browser click makes it possible to access an element the mouse is unable to reach via Puppeteer's click, such as when another element is on top of it. In other cases, such as testing, it's desirable to click with the mouse as a human would, using a trusted event. Puppeteer's docs state:

For automation purposes it's important to generate trusted events. All input events generated with Puppeteer are trusted and fire proper accompanying events. If, for some reason, one needs an untrusted event, it's always possible to hop into a page context with page.evaluate and generate a fake event:

await page.evaluate(() => {
  document.querySelector('button[type=submit]').click();
});

The difference in behavior might be most noticable for page.click(), but it's worth keeping the distinction in mind when working with the rest of Puppeteer's API.

Never using `"domcontentloaded"`

The default event for Puppeteer's navigiation API, such as page.goto(url, {waitUntil: "some event"}), is {waitUntil: "load"}. I'm guilty of blindly relying on this default without questioning its implications, but it's worth investigating a bit relative to other options: "domcontentloaded", "networkidle2" and "networkidle0".

MDN says of the load event:

The load event is fired when the whole page has loaded, including all dependent resources such as stylesheets and images. This is in contrast to DOMContentLoaded, which is fired as soon as the page DOM has been loaded, without waiting for resources to finish loading.

Waiting for load is a useful choice in scenarios where it's important to see the page as the user would. Example use cases include snapping screenshots or generating PDFs. "networkidle0" and "networkidle2" resolve the navigation promise when no more than 0 or 2 network requests are active in the past 500 milliseconds. These settings have similar use cases as "load" and offer more control for handling pages that might keep a polling connection or two active after load, or when the driver needs to flush requests before proceeding.

On the other hand, if the goal is to scrape a table of text statistics that arrives from a network request, there's no sense in waiting around for anything more than "domcontentloaded" and the lone data request, which might look like:

await page.goto(url, {waitUntil: "domcontentloaded"});
const el = await page.waitForSelector(yourSelector, {visible: true});
const data = await el.evaluate(el => el.textContent);

...where yourSelector won't be injected into the page until the desired data arrives.

Not blocking images and resources

If the goal is to scrape a simple piece of data, there's no sense in wasting time and bandwidth requesting all images, stylesheets and/or scripts on a page. When possible, consider disabling JS with page.setJavaScriptEnabled(false), stopping the browser with page.evaluate(() => window.stop()) or enabling request interception to block images as follows:

await page.setRequestInterception(true);
page.on("request", req => {
  if (req.resourceType() === "image") {
    req.abort();
  }
  else {
    req.continue();
  }
});

This helps keep your scripts snappy and reduces waste.

As an aside, there's no need to await page.on, which is strictly callback-driven and does not return a promise. Nonetheless, it can be handy to promisify page.on so results can be awaited later. While most events are available in a promise-based function, others such as "dialog" aren't.

You can use .once and .off methods to ensure the handler is removed when no longer needed.

Avoiding `page.evaluate` when trusted events aren't necessary

One can easily wind up frustrated when refactoring working browser console code using jQuery or vanilla JS to Puppeteer's trusted interface: page.$, page.$eval, page.type, page.click, and so on. This is a common situation since experimenting with the DOM in the browser console is a typical first step in building a Puppeteer script. These refactors often go sour when attempting to pass complex structures such as ElementHandles between the browser and Node contexts, or encountering behavioral differences in methods like page.click(), discussed above.

Puppeteer provides page.evaluate() as a highly-generalized function to run code in the browser. It supports passing serializable data and returning deserialized results. For times when trusted events aren't needed, plopping a chunk of working browser code inside a page.evaluate and letting it handle the heavy lifting alleviates the time spent on a rewrite and helps mitigate the potential for subtle regressions.

Misusing developer tools-generated selectors

The developer tools in modern browsers let you copy CSS selectors and XPaths to the clipboard easily:

Copying an element's CSS selector in Chrome

This is a boon for developers who might not be familiar with CSS selectors or XPath, and can save time for those who are.

The problem is, these selectors and paths can be overly-rigid and lead to brittle scripts that break if a wrapper is added as a parent further up the tree, or a sibling element shows up unexpectedly after an interaction on the page.

For example, Chrome gives the very strict CSS selector #answer-60796572 > div > div.answercell.post-layout--right > div.s-prose.js-post-body > pre when #answer-60796572 pre might be more resilient to handling dynamic behavior.

Choosing how to select elements is an art, not a science. It takes time to build a feel for which specificity is appropriate for a particular use case. The browser-generated selectors and paths are handy, but are ripe for misuse. When using these selectors, it's wise to take a step back and consider the broader context such as site-specific behavior and script goals.

SerpAPI's post Web Scraping with CSS Selectors using Python provides a nice introduction to CSS selectors in a web scraping context. (Python isn't essential to the selector information.)

Not using the return value of `.waitForSelector` and `.waitForXPath`

A common, albeit minor, antipattern is to make an extra call to retrieve an element after waiting for it:

await page.waitForSelector(selector);
const elem = await page.$(selector);

// or:
await page.waitForXPath(xpath);
const [elem] = await page.$x(xpath);

Cleaner and more precise is to use the return value of the waitFor calls:

const elem = await page.waitForSelector(selector);

// or:
const elem = await page.waitForXPath(xpath);

Once you've obtained the ElementHandle, you can use elem.evaluate(el => el.textContent) rather than page.evaluate(el => el.textContent, elem). See a Stack Overflow answer of mine for details on passing nested ElementHandles back to the browser.

Using a separate HTML parser with Puppeteer

Puppeteer already has access to the full power of browser JS and operates on the page in real-time, so it's an antipattern to bring in a secondary HTML parser like Cheerio without good reason. Using Cheerio with Puppeteer involves taking serialized snapshots of the entire DOM (i.e. using page.content()), then asking Cheerio to re-parse the HTML before being able to make a selection. This can be slow, adds a potentially confusing layer of indirection between the live page and the separate HTML parser and creates an additional opportunity for misunderstanding the state of the application.

This approach can be beneficial for debugging and when HTML snapshots or particular features of Cheerio are needed (like sizzle selectors), but the common case is to use Puppeteer on its own.

Using Puppeteer when other tools are more appropriate

Web browser automation is a heavy and slow proposition. It involves launching a browser process, then making network calls to navigate the browser to a page, then manipulating the page to accomplish a goal. Sometimes that goal can be achieved directly using a simple HTTP request to a public API or retrieving data baked into a page's static HTML. Using a request library like fetch and an HTML parser like Cheerio, when possible, can be a faster and easier method than Puppeteer.

When collecting requirements for a scraping task, take the time to make sure the data isn't hidden in plain sight. This usually involves viewing the page source to see the static HTML and analyzing network traffic using the browser's developer tools to determine where the data you're after is coming from. Reading the website's FAQ and documentation often points to a public API. Occasionally, other sites offer the same information in a more accessible format.

For one-time scrapes, I'll often select the data we need using browser JS and copy it to the clipboard from the console by hand instead of reaching for Puppeteer. Userscripts, extensions and bookmarklets are other lightweight ways to manipulate a page without Puppeteer or Node.

While these alternatives might not cover many of your Puppeteer needs, it's good to keep more than one tool in your belt.

Wrapping up

In this article, I've presented a handful of Puppeteer antipatterns to be on the lookout for. We hope that a critical examination of them will help keep your Puppeteer code clean, fast, reliable and easy to maintain. As the tool grows, I'll be curious to see how these patterns evolve along with it.

Keep in mind that these guidelines are rules-of-thumb and might need to be broken from time to time. The key is to be aware of the tradeoffs when using them.

Happy automating!

This article is current as of Puppeteer version 13.5.1.

SerpApi Integration for Zapier (wip)

Illia Zub — Thu, 17 Mar 2022 14:33:55 +0000

SerpApi Integration for Zapier is work-in-progress.

Idea

Two years ago we wrote a note about a SerpApi Integration for Zapier. Currently, we're working on supporting all search parameters that SerpApi has.

Work-in-progress SerpApi Integration for Zapier works similarly to the SERPAPI_RANK function from SerpApi Add-On for Google Sheets (tutorial). It is a Zapier Action that makes a search to SerpApi with the specific parameters.

Returned data

Then returns a search result that matches the domain.

Use cases

A matched search result, for example, can be passed to the next Zapier Action like "Create Record in Airtable" or "Create Record in Google Sheets".

SerpApi + Zapier

SerpApi integration for Zapier is in development. We're working on accepting more search parameters that SerpApi supports.

Outro

If you have anything to share, any questions, suggestions, or something that isn't working correctly, reach out via Twitter at @ilyazub_, or @serp_api.

Yours,
Illia, and the rest of the SerpApi Team.

Join us on Reddit | Twitter | YouTube

Request a Feature 💫 or Report a Bug 🐞

Scrape Walmart Search for a specific store

Illia Zub — Thu, 02 Dec 2021 11:50:33 +0000

Walmart responds with results for Sacramento for requests outside of the US.

But how to search for products that are available in a specific store? Any store on Walmart can be chosen without browser automation — only by setting relevant cookies in the plain HTTP request.

To figure out on your own, JS and browser dev tools knowledge will be enough. Some Ruby knowledge is required to understand this post.

Location cookies

I've updated location several times and checked the browser Dev Tools -> Application -> Cookies.

There are several cookies being updated after choosing a different location: locGuestData, locDataV3, assortmentStoreId; ACID, hasACID, hasLocData.

location-data also looks relevant but it contains postal code and address for a store I haven't chosen. Maybe it was used before Walmart migrated to GrapgQL API.

locDataV3 and locGuestData are Base64 and URI-encoded JSON objects. locDataV3 contains more data than locGuestData. But data of locGuestData can be used for both.

ACID is a UUID. It can be generated on the client.

hasACID and hasLocData are flags.

Understanding `locGuestData`

Let's check what's inside this cookie value to understand how to set the store ID.

Example of encoded `locGuestData`

When sending requests to Walmart, locGuestData is a Base64-encoded string.

eyJpbnRlbnQiOiJTSElQUElORyIsInN0b3JlSW50ZW50IjoiUElDS1VQIiwibWVyZ2VGbGFnIjp0cnVlLCJwaWNrdXAiOnsibm9kZUlkIjoiNDExNSIsInRpbWVzdGFtcCI6MTYzNzMyODUwMDUyM30sInBvc3RhbENvZGUiOnsiYmFzZSI6Ijc4MTU0IiwidGltZXN0YW1wIjoxNjM3MzI4NTAwNTIzfSwidmFsaWRhdGVLZXkiOiJwcm9kOnYyOjUyNzNlMDFjLTA4NzAtNGUwOS05ODU4LTAzYTI2ZDQ5N2ZhOSJ9

Example of decoded `locGuestData`

This Base64 string is a encoded JSON object.

JSON.parse(decodeURIComponent(atob("eyJpbnRlbnQiOiJTSElQUElORyIsInN0b3JlSW50ZW50IjoiUElDS1VQIiwibWVyZ2VGbGFnIjp0cnVlLCJwaWNrdXAiOnsibm9kZUlkIjoiNDExNSIsInRpbWVzdGFtcCI6MTYzNzMyODUwMDUyM30sInBvc3RhbENvZGUiOnsiYmFzZSI6Ijc4MTU0IiwidGltZXN0YW1wIjoxNjM3MzI4NTAwNTIzfSwidmFsaWRhdGVLZXkiOiJwcm9kOnYyOjUyNzNlMDFjLTA4NzAtNGUwOS05ODU4LTAzYTI2ZDQ5N2ZhOSJ9")))

{
    "intent": "SHIPPING",
    "storeIntent": "PICKUP",
    "mergeFlag": true,
    "pickup": {
        "nodeId": "4115",
        "timestamp": 1637328500523
    },
    "postalCode": {
        "base": "78154",
        "timestamp": 1637328500523
    },
    "validateKey": "prod:v2:5273e01c-0870-4e09-9858-03a26d497fa9"
}

After changing Walmart store several times, I've seen that nodeId and postalCode.base are changing.

Generate `timestamp` and `acid` for `locGuestData`

timestamp and acid can be generated on every request.

timestamp = Time.now.to_i

acid = SecureRandom.uuid

Base64-encode location data

Next, let's Base64-encode that JSON string as Walmart expects.

timestamp = Time.now.to_i

acid = SecureRandom.uuid

location_guest_data = {
  intent: "SHIPPING",
  storeIntent: "PICKUP",
  mergeFlag: true,
  pickup: {
    nodeId: store_id,
    timestamp: timestamp
  },
  postalCode: {
    base: postal_code,
    timestamp: timestamp
  },
  validateKey: "prod:v2:#{acid}"
}

encoded_location_data = Base64.urlsafe_encode64(JSON.dump(location_guest_data))

Create cookie string

Finally, a location cookie string contains all the required fields.

%(ACID=#{acid}; hasACID=true; hasLocData=1; locDataV3=#{location_guest_data}; assortmentStoreId=#{store_id}; locGuestData=#{encoded_location_data})

Complete function to create Walmart location cookie

Putting all together.

def location_cookie(store_id, postal_code)
  return if store_id.blank?

  timestamp = Time.now.to_i

  acid = SecureRandom.uuid

  location_guest_data = {
    intent: "SHIPPING",
    storeIntent: "PICKUP",
    mergeFlag: true,
    pickup: {
      nodeId: store_id,
      timestamp: timestamp
    },
    postalCode: {
      base: postal_code,
      timestamp: timestamp
    },
    validateKey: "prod:v2:#{acid}"
  }

  encoded_location_data = Base64.urlsafe_encode64(JSON.dump(location_guest_data))

  %(ACID=#{acid}; hasACID=true; hasLocData=1; locDataV3=#{location_guest_data}; assortmentStoreId=#{store_id}; locGuestData=#{encoded_location_data})
end

Then make an HTTP request using the language and libraries you've chosen.

import got from 'got';

const STORE_ID = "4115";
const POSTAL_CODE = "78154";

const locationCookie = getLocationCookie(STORE_ID, POSTAL_CODE);

const htmlResponse = await got('https://www.walmart.com/search?q=cookie', {
  headers: {
    cookie: locationCookie
  }
});

Where to get store ID and postal code

Well, but we wouldn't hard-code store ID and postal code into the web scraping program. A CSV of 4.6k stores can be used to find and store ID dynamically.

Programmatic usage of CSV is out of the scope of this post. All that is needed is to read find store ID and postal code for a specific location in a table.

Updating a list of Walmart stores IDs and locations

Walmart provides several sources to find stores. Data can be populated from one of those sources:

Store Directory

Store Directory contains links on four levels: country, states, cities, and stores. To get the data, iterate over all elements on the specific level and make subsequent requests.

States

Assuming the country is the US, 51 states can be hard-coded. Walmart front-end requests data from the JSON endpoint https://www.walmart.com/store/electrode/api/store-directory. It accepts the st search parameter.

Example: https://www.walmart.com/store/electrode/api/store-directory?st=AL.

It returns a list of cities. Each city object contains city, and storeId or storeCount. The city with storeId contains a single store. The city with storeCount contains multiple stores.

Single store in a city

Request to a specific store returns an HTML page. Example: https://www.walmart.com/store/5744.

Store address and postal code should be extracted from the HTML. Store ID is already in URI.

let postalCode = document.querySelector(".store-address-postal[itemprop=postalCode]").textContent;
let address = document.querySelector(".store-address[itemprop=address]").textContent;

Multiple stores in a city

Request for multiple stores returns a JSON response. Cities with a single store respond with an empty array ([]) so we have to parse HTML.

Example request for multiple stores

https://www.walmart.com/store/electrode/api/store-directory?st=AL&city=Decatur

Sample city from the response

{
  "displayName": "Neighborhood Market",
  "storeName": "Neighborhood Market",
  "address": "1203 6th Ave Se",
  "phone": "256-822-6366",
  "postalCode": "35601",
  "storeId": 2488
}

Putting all together

Pseudo-code to collect store IDs and locations for all US states.

const STATES = ["AL", "TX", "CA", /* ... */];

let walmartStores = [];

for (let state of STATES) {
  let cities = get(`https://www.walmart.com/store/electrode/api/store-directory?st=${state}`);

  for (let { storeId, storeCount, city } of cities) {
    if (storeId && !storeCount) {
      let store = get(`https://www.walmart.com/store/${storeId}`);

      let document = parseHTML(store);

      let postalCode = document.querySelector(".store-address-postal[itemprop=postalCode]").textContent;
      let address = document.querySelector(".store-address[itemprop=address]").textContent;

      walmartStores.push({ postalCode, address, storeId: storeId });
    } else if (!storeId && storeCount > 0) {
      let stores = get(`https://www.walmart.com/store/electrode/api/store-directory?st=${state}&city=${city}`);

      walmartStores.concat(stores);
    }
  }
}

csv.write("walmart_stores.csv", walmartStores);

Existing programs to scrape Walmart Stores

Search on GitHub via grep.app shows four relevant repositories

Akamai edge workers example. But it contains only 471 Walmart stores.

$ curl -s https://raw.githubusercontent.com/akamai/edgeworkers-examples/master/edgecompute/examples/personalization/storelocator/data/locations.json | jq '.elements[].tags | select(."ref:walmart" != null) | .ref' | wc -l
471

scrapehero/walmart_store_locator which scrapes stores by postal codes. But finding a list of actual postal codes turned out to be harder than finding a list of actual US states.
theriley106/WaltonAnalytics which is great to extract data from Walmart but not Walmart stores.
GUI/covid-vaccine-spotter which scrapes stores by postal codes. But finding a list of actual postal codes turned out to be harder than finding a list of actual US states.

So, I've played with Rust and came up with this (rough) program.

After going through compilation errors, it worked well. Thanks to this helpful blog post about async streams in Rust. Every time my program compiled, it actually worked. Fixing compilation errors is hard (for non-rustacean) but there's was no need to debug the program in runtime which is great.

Conclusion

Scraping Walmart is fairly easy — it contains inline JSON data for all products on the search results page.

Update location cookies to specify the location for plain HTTP requests to Walmart.

If you have anything to share, any questions, suggestions, or something that isn't working correctly, feel free to drop a comment in the comment section or reach out via Twitter at @ilyazub_, or @serp_api.

Yours,
Ilya, and the rest of the SerpApi Team.

Join us on Reddit | Twitter | YouTube

DEV Community: Illia Zub

How we reverse-engineered Google Maps pagination

How Google Maps pagination works

The Plan

Decoding the URL parameters for Google Maps pagination

Convert zoom to an altitude

Code to generate pb parameter

Putting all together

Parse the paginated data

Playwright’s getByRole is 1.5x slower than CSS selectors

Introduction

Deep Dive: How getByRole Works

Performance

Conclusion

When counting lines in Ruby randomly failed SerpApi deployments

TL;DR

Investigation

Memory profiling

String#lines

Solution

Benchmark

Memory usage

lines/readlines/each_line/etc.

count($/)

Code

Comparison process

Results

Observations and final thoughts

Update Sep 20th, 2023

Reverse Engineering Poocoin API (Part 1)

What will be scraped

Explanation

Outro

Mastering SerpApi: An In-depth Features, Best Practices, and Competitive Edge

Episode Breakdown

Software Engineer's Life Balance with Dmytro Vasyliev | #SerpApiPodcast, Episode 10

Chapters

From Software Engineer to Manager with Oleksii Gyturo | #SerpApiPodcast, Episode 9

Show notes

Insights from an Engineering Director (#SerpApiPodcast, Ep. 8)

Show Notes

Links and resources

How to reverse engineer a JSON API on a single page application

Algorithms to reverse engineer a JSON API on the SPA

Browser devtools

mitmproxy

Conclusion

Avoiding Puppeteer Antipatterns

Overusing waitForTimeout

Assuming Puppeteer's API works like the native browser API

Never using "domcontentloaded"

Not blocking images and resources

Avoiding page.evaluate when trusted events aren't necessary

Misusing developer tools-generated selectors

Not using the return value of .waitForSelector and .waitForXPath

Using a separate HTML parser with Puppeteer

Using Puppeteer when other tools are more appropriate

Wrapping up

SerpApi Integration for Zapier (wip)

Idea

Returned data

Use cases

SerpApi + Zapier

Outro

Scrape Walmart Search for a specific store

Location cookies

Understanding locGuestData

Example of encoded locGuestData

Example of decoded locGuestData

Generate timestamp and acid for locGuestData

Base64-encode location data

Create cookie string

Complete function to create Walmart location cookie

Where to get store ID and postal code

Updating a list of Walmart stores IDs and locations

Store Directory

States

Single store in a city

Multiple stores in a city

Putting all together

Code to generate `pb` parameter

Deep Dive: How `getByRole` Works

`String#lines`

`lines`/`readlines`/`each_line`/etc.

`count($/)`

`mitmproxy`

Overusing `waitForTimeout`

Never using `"domcontentloaded"`

Avoiding `page.evaluate` when trusted events aren't necessary

Not using the return value of `.waitForSelector` and `.waitForXPath`

Understanding `locGuestData`

Example of encoded `locGuestData`

Example of decoded `locGuestData`

Generate `timestamp` and `acid` for `locGuestData`