I am a fullstack developer specialized in frontend technologies like reactjs. I also love programmimg in go. Currently im working on https://stormkit.io to make developers lives easier.
I like the tool (pricingbot) I will actually propose this to my wife who works in the e-commerce field.
Regarding the API, I used once puppeteer to scrape some content and it was pretty straightforward. I would definitely use the data however. For instance if I am scraping several sites to find the median price of a car, i would rather use the api hoping it already has some data i could rely on. Companies would even pay a lot of money for this IMO.
I am a fullstack developer specialized in frontend technologies like reactjs. I also love programmimg in go. Currently im working on https://stormkit.io to make developers lives easier.
Sure! So what I mean is that the api could be written in a way that it learns the scraped data, and after a while people can use it only to query data. Imagine I scrape the average car price in the market for a model, and I specify several websites to look for. Next time another user might use just the data and they won't have to rewrite the same scraper.
I am saying this because I was using puppeteer and it was is very easy to use, I haven't seen the need for another tool when I used it. The hard part (in terms of effort) was to write the scraper and save the data into the database. It just takes time.
However, ScrapingNinja only returns you raw HTML, no formatted data, maybe it is not that clear on the landing page.
What you are talking about is some kind of product that gets a URL in input and that output formatted data so you don't have to configure XPath / CSS selector whatever.
Thank you very much for your time and feedback.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
I like the tool (pricingbot) I will actually propose this to my wife who works in the e-commerce field.
Regarding the API, I used once puppeteer to scrape some content and it was pretty straightforward. I would definitely use the data however. For instance if I am scraping several sites to find the median price of a car, i would rather use the api hoping it already has some data i could rely on. Companies would even pay a lot of money for this IMO.
Keep up the good work!
Thank you very much!
May I ask what do you mean by “ if they already have some data” ?
Sure! So what I mean is that the api could be written in a way that it learns the scraped data, and after a while people can use it only to query data. Imagine I scrape the average car price in the market for a model, and I specify several websites to look for. Next time another user might use just the data and they won't have to rewrite the same scraper.
I am saying this because I was using puppeteer and it was is very easy to use, I haven't seen the need for another tool when I used it. The hard part (in terms of effort) was to write the scraper and save the data into the database. It just takes time.
Oh, I see, it makes sense now.
However, ScrapingNinja only returns you raw HTML, no formatted data, maybe it is not that clear on the landing page.
What you are talking about is some kind of product that gets a URL in input and that output formatted data so you don't have to configure XPath / CSS selector whatever.
Thank you very much for your time and feedback.