DEV Community: Bartude

Building URL's to crawl based on other websites

Bartude — Tue, 16 Apr 2019 22:17:47 +0000

The idea behind this is that a user, upon registration, selects a number of preferences of their choosing (min/max price, region, etc..). Then based on those preferences, I'm gonna crawl/scrape other websites to find matches for those preferences (For all of these websites, these matches are found using query parameters).

Just a sidenote, this is not to steal other peoples content. The idea is to merely gather up the matches from the different websites and list them all on each users authenticated page.

I'm not sure what's the best course of action to do this. This is what I've thought of so far:

Manually make a map for each website of the existing values and the corresponding query parameters and build the url's using the corresponding website map
Use a browser automation tool like Selenium to automatically browse and build the url each time the preferences are saved (although I'm not sure these are safe to use in production)

Any tips on the best way to move forward with this? (I am open to using API's that would help me do this)

When to install packages and their weight?

Bartude — Tue, 11 Dec 2018 18:30:08 +0000

In web development, either working or not with a framework, the need for installing packages is almost inevitable. There are a few which are really indispensable for each and every person.

Part one of my question is, what should be the deciding factors on when to install an X or Y package or not (and just develop the necessary logic yourself).

Second part of my question is, do installed packages influence a lot of the web apps performance? For example, I once saw an image of a company called Spatie (which develops Laravel packages) create a web app for a client, and the list of packages they were installing was gigantic (my standards, like around 30 or 40).

Explain like I'm five: Is Test Driven Development really that used in most of the programming world?

Bartude — Mon, 13 Nov 2017 11:38:54 +0000

When I'm navigating on the big wide web that is the internet I often see job postings that require experience in TDD. I've been programming for about two years and I've never really used PHPunit(shame) because each time I finish a functionality I test it with various inputs to correct bugs. Can someone explain (like I'm five) what's the importance of programming using TDD?

P.S.: I hope I've posted this in the right place cause this is my first time :p