Hey All! I’ve been recently exploring Puppeteer in order to automate simple tasks for myself (e.g. capturing screenshots and PDFs). I'm extremely amazed by how easy it is to drive the browser UI, as well as inspect network requests/performance data/etc., and I can only imagine how powerful headless browser automation can be, not only for E2E tests, but also business processes (e.g. automate data collection/entry) 🚀
I'm curious to hear from others, in order to learn whether you've used headless browsers for automation, and if so, what for? (e.g. end-to-end tests, automating a business process, etc.) In particular, it would be awesome to know what tasks you're automating, which specific tools/frameworks you've used (e.g. Selenium), and what your experience was like building/maintaining the automation scripts. Thanks!
Top comments (15)
Like Johannes, I automated e2e for my SPA, using an object to configure my tests.
I also used puppeteer to create a static version of my SPA to use prerendering on my customer website.
If you guys have tools to do this, apart from Jest, I would also be glad to hear from you!
Boringly enough I only used them for e2e testing. But now that you mention it, a browser extension to automate some basic chores would be nice, even though I can't think of a practical use case atm for some reason.
Curious what other people have to offer!
Hi Jonathan,
I'm currently using Headless browser sessions for all of my automated GUI testing. We run our automated GUI tests in the cloud so a dependency on a monitor is out of the question. Our automation is a combination of deployment validation, business logic verifications and multiuser role based E2E testing [with screen captures].
I am using WebdriverIO as the base framework to driver the interactions with headless Chrome.
The troubles with headless automation:
1) You loose the ability of the GPU acceleration you get from running a browser on a full desktop OS. That means that you have to code your automated tests for potential delays in returned results or page rendering.
2) Another issue I run into around resource constrains when running a headless browsers and multiple simultaneous instances. I need to spend some time profiling my automation environment to understand my breaking point.
Hey Ken, I work with Jonathan at Microsoft, and we are curious to know more about your learnings in testing/automation. Would it be possible to take this forward over email? I'm at arjun.attam@microsoft.com
Hi Arjun,
Thanks for reaching out. I'll send you an email shortly and I look forward to chatting with you.
Cheers!
Ken
I scraped Javascript rendered data from my college's website. The website just has the data and it is not even responsive, still everyone is forced to use it on desktop view on mobile devices. I scraped the attendance, LMS and everything using Casper.js and Phantom.js and created RESTful APIs and developed a mobile app using React Native for Android and iOS. 🙄👍 By the way in my liesure, I find new ways to scrape contents. 🖖
Very cool! Have you had trouble maintaining the scraping scripts (e.g. HTML layout changes)? Or does the school website change infrequently enough that scraping it is pretty reliable?
They don't bring any updates. So they don't change any HTML content. Hence till now I didn't face anything of that sort...
Just had a practical idea. I have to download all my business bills to send them to the tax office every month. Quite annoying as I have to collect them from different emails and websites. That would be something worth some automation script, though honestly I think that in the end (due to changing environments) this probably will cost me more time than it will save. I might do it anyway though! :D I also spend like 2000 hours on an app to save me some time on time tracking every day, which doing manually would have approximately just cost me 312 hours in the same time so far. So you could say I have some experience in over the top automation.
That seems quite relatable, and something I wish I could fix as well. I came across Selenium IDE, which seems to be a way to record UI interactions like the ones you mention. Does that sound helpful to you?
Thanks. I worked with Selenium many years before and back then it was not much fun, but I might give it another go. I also found this one: imacros.net/
I used CasperJS to audit pages on a university 'website' (actually many sites on the same domain) to facilitate planning for migration to responsive templates (this was a while back). I generated a site map by inspecting every page and following any links on the domain and logging what was found; including details of custom scripts etc. that would require additional manual intervention.
Some faculties/departments had their own web servers so this was the only practical way to gather all the data in one place.
Nowadays I'm really liking Cypress for end to end testing; though no idea if it could achieve the same results in terms of web-scraping.
Hey Ben, I work with Jonathan at Microsoft, and we are curious to learn more about your experience in testing/automation. Would it be possible to take this forward over email? I'm at arjun.attam@microsoft.com
I use Selenium WebDriver. It has some advantages over Puppeteer:
It can be used with any browser. For some very simple tasks, this isn't so important (but I still use it), but it can be quite important for complete E2E tests.
It can be used with any language. Unlike Puppeteer, which can only be used in JS, Selenium has libraries for all popular languages. This is very useful if you develop in multiple programming languages.
I used Python/Selenion web driver to mass update Google Custom Search engines. Google doesn't expose their API for updating your CSE, so I wrote a scraper to gather all the tokens then used the python requests library to delete/update the lists (I have 11 CSE's I update 2x per week)