DEV Community: Md Abu Taher

QA: Npm scripts vs gulp or similar test runners?

Md Abu Taher — Fri, 06 Jan 2023 09:56:58 +0000

This is a question asked on stackoverflow which will most likely be closed because it's not a question asking for a straightforward solution. So I'll share it here with some context and answer.

Question:

What are the advantages and disadvantages of both? I currently have npm scripts for test execution like this in package.json:

  "scripts": {
    "test": "npx playwright test",
    "lint": "eslint . --ext js,ts",
  }

Having never used gulp or grunt, I would like to know if they do have any massive advantages over just using npm scripts? Thank you.

Answer:

You've shown that running tests with npm scripts is simple. Simply run npm run test from the command line to run tests.

The number of npm scripts and their complexity can make them difficult to maintain. In such cases, a task runner like Gulp or Grunt can be helpful.

Gulp and Grunt are JavaScript task runners that automate minification, compilation, unit testing, and linting. Both use configuration files to define tasks that can be run from the command line.

Gulp or Grunt's benefits include:

With a task runner, you can define all your tasks in a single configuration file, making them easier to manage.
Task runners offer more configuration options and can run multiple commands or chain tasks together.
Sharing tasks is easier with a task runner configuration file than with npm scripts.

If npm scripts meet your needs, you don't need a task runner. If your npm scripts are difficult to manage, Gulp or Grunt may be a better option.

State of AI and AI Content Detection

Md Abu Taher — Fri, 23 Dec 2022 01:59:57 +0000

There is a post about Guidelines for AI-assisted Articles on DEV which lists down a few rules for posting on dev.to, and then there was my post about Why you shouldn't rely on ChatGPT so much!.

The drawback of using AI to write things are getting worse.

People are using and abusing AI to write tons of contents, flooding fiverr and upwork with writing related jobs. There are now tons of videos explaining how to earn easy money using AI to write content.
Flooding stackoverflow with wrong answers to the questions, which led to a ban on ChatGPT based solutions.
Google also says AI generated contents are bad for SEO.

The issue with AI writing is almost similar as the issue we had with Co-Pilot. Some people abusing the machine and some are fearing it would replace human.

Let's check this following section written by ChatGPT.

Artificial intelligence (AI) is revolutionizing the transportation industry by optimizing routes, schedules, and other aspects of transportation in order to improve efficiency and customer satisfaction. With the ability to analyze real-time traffic data and provide personalized recommendations and updates, AI-powered systems are making it easier for people to get from point A to point B and enhancing the overall travel experience.

In addition to optimizing transportation for individuals, AI is also being used to streamline logistics and supply chain management, helping businesses and organizations to optimize their operations and reduce costs. By analyzing data on demand, inventory, and transportation, AI-powered systems can make significant improvements to the efficiency of supply chains and logistics operations.

Overall, the use of AI in transportation is on the rise and is expected to continue to grow as more businesses and organizations adopt these technologies to improve efficiency and customer satisfaction.

Now this is easily detected by the AI content detection tools.

Then I asked AI systems to rewrite it and avoid useless words. It generated a more simpler version.

Have you noticed how AI is making it easier and more fun to travel? It helps us get from one place to another faster and more comfortably by analyzing traffic data and giving us personalized recommendations.

And it's not just for people like you and me - businesses use it to make their operations run smoother and save money, too. They do this by looking at data on things like demand, inventory, and transportation to simplify their logistics and supply chains.

AI in transportation is becoming more popular and is expected to keep growing in the future.

And then the AI content detection failed.

In fact there are tools like Content at Scale who charges a hefty amount to provide ai content that sounds less ai-ish, as they mix up a few AI tools, google search and so on, to produce a better content. I am sure there will be more doing the same.

Okay so now what?

There are AI tools that can produce contents, there are AI detection tools that can detect AI contents, but both of them can be easily fooled.

What is the issue here?

The issue is the quality of content we are going to read on a daily basis. Since AI content is basically trained on dataset of real world data, a ton of AI content will sound real, while a ton of real human written content will end up sounding like AI.

If I consume more AI related contents, I am bound to pick up a few of their habits when doing research for my articles, and end up being called an AI.

So basically it's another cat and mouse game once again. People will abuse it as much as possible and there seems to be no solution at sight yet until human creativity catches up, it will for sure. If AI was trained by human, AI can be caught by human as well.

Why you shouldn't rely on ChatGPT so much!

Md Abu Taher — Wed, 21 Dec 2022 03:52:56 +0000

As a developer, I'm always on the lookout for new tools and techniques that can streamline my workflow and save me time. Recently, I decided to give ChatGPT a try for web automation.

My goal was to extract data from HackerNews using ChatGPT, so I fed it a series of prompts and hoped it would generate the appropriate regex for me.

Oh! I was so wrong!

Unfortunately, despite its impressive capabilities and hype, ChatGPT ultimately failed to deliver the goods.

While it did generate some text that sounded like it might be on the right track, it didn't actually solve my problem. This was both good news and bad news.

On the one hand, it showed that ChatGPT was learning and adapting to meet my needs. On the other hand, it didn't actually get the job done.

In the end, I decided to take matters into my own hands and started adjusting the generated regex myself. It took some trial and error, but eventually, I was able to get it working.

Steps I have taken

I told ChatGPT that it should become a regex generator and the output should be in JavaScript.

I simply copied and pasted the text from HackerNews.

Provide ChatGPT with the desired input and output for the regex.

I received a sample output from ChatGPT, and it sounded like it was absolutely accurate.

To validate it, I took it to regex101, which would highlight everything for me.

It looks like it worked, except for one result. So I went back to ChatGPT and provided more information.

But instead of fixing it, it just broke the whole regex. Everything, including time, is now broken.

I continued to iterate on this process, using ChatGPT’s output as a starting point and making manual adjustments until the desired regex was obtained.

At this point, this process needed a lot of tries, which used up my precious credits that I could spend on other things.

¯_(ツ)_/¯

So I decided to take matters into my own hands and tweak the regex myself.

The final regex was this, which worked for most stories.

^(?<rank>\d+)\.\s+(?<title>.+)\s+\((?<site>.+)\)\s+(?<points>\d+)?\s+points?\s+by\s+(?<author>\S+)?\s+(?<time>.+?)\s+\|\s+hide(\s+\|\s+(?<comments>\d+)\s+comments)?

Conclusion of the experiment

Even though there are hundreds of other ways to grab the data from HackerNews, using AI to extract data from a blob of text was an interesting challenge to try with ChatGPT. But this experiment just told me one thing.

Do not blindly trust GPT and variants. It can give you incorrect data with very high confidence and your precious app will be broken in no time.

Okay, I could have given it better input, but keep in mind that I tried this process many times with different input. Meanwhile, it kept adding big blobs of text to the response, even when I didn’t need them. I've noticed that it frequently adds large blobs of text despite my requests.

For example, this prompt specifically mentions avoiding providing explanation, but who cares?

What does ChatGPT have to say about this?

I told ChatGPT about the above situation, and it sympathized with me.

I’m glad you found ChatGPT to be a helpful tool in your workflow, even if it didn’t quite solve the problem you were working on. It’s important to remember that large language models like ChatGPT, while extremely powerful and versatile, still have limitations when it comes to certain tasks.

In your case, using a large language model for web automation was a novel approach, and it’s not surprising that it didn’t work perfectly on the first try. Web automation can be a complex task that requires a deep understanding of web technologies, as well as specific domain knowledge about the sites and data you are trying to access. In these cases, it’s often best to use specialized tools or rely on your own expertise and understanding of the problem.

That being said, it’s always worth considering whether a large language model could be helpful in solving a specific problem. They can be a powerful tool in the right context, and with the right amount of training and guidance, they can learn and adapt to new tasks and environments. The key is to use them in a way that complements your own skills and knowledge, rather than relying on them as a replacement.

Conclusion

This taught me about ChatGPT's limitations. Even though they can write text that sounds human, do a wide range of tasks, and speak many languages, they may not have the technical expertise to do a specific task as well as a specialized tool or a person with expertise in that area.

My own knowledge and skills helped me solve the problem, not ChatGPT. This shows why language models need human understanding and expertise. Large language models can be helpful, but they shouldn't replace human understanding.

You should not depend too much on ChatGPT.

Reflections on My 15 Year Journey from Novice to Intermediate Developer

Md Abu Taher — Tue, 13 Dec 2022 06:02:50 +0000

My self-taught development journey started around 15 years ago when I was a little kid. It went from passion to a job to irritation with tons of ups and downs.

I became a normal developer, freelancer, co-founder, office employee and so on. Sometimes I was spending 20 hours per day in front of computer like crazy due to excitement, sometimes I got burned out from overworking and stress.

I probably did not see everything, but I want to share a few points.

How do you spot a beginner programmer?
Things beginners should know
Things beginners should avoid

Let's get started!

How do you spot a beginner programmer?

Top ⤴️

There are several signs that can indicate that a programmer is a beginner. Some of these signs include:

Lack of familiarity with core programming concepts and best practices.
Struggling to write clean, well-structured, and efficient code.
Reliance on copying and pasting code from online sources without fully understanding how it works.
Difficulty troubleshooting and debugging errors and problems in their code.
Limited knowledge of programming languages and tools.
Lack of experience working on real-world projects or in a team environment.

These are just some of the potential signs that a programmer may be a beginner. It's important to note that being a beginner is not necessarily a bad thing - everyone has to start somewhere, and everyone has different levels of experience and expertise.

However, if you notice some of these signs in a programmer, it may indicate that they are still in the early stages of learning and may need more guidance and support.

Things beginners should know

Top ⤴️

Here are a few tips I'd like to give to someone who is just starting to learn how to code:

Start by making sure that each project or task has clear goals that can be reached.

This can help you focus your work and keep you from getting distracted by code that isn't important.

Use tools like Notion, ClickUp, Github Issue.
Figure out which tasks and features are most important and work on them first.

This can help you make progress on your projects and meet your deadlines while still giving you time to try out new ideas.
Ask programmers with more experience for feedback and help.

This can help you figure out where you might be spending too much time on code that isn't necessary or useful and teach you how to work more efficiently.

Check Github Issues, Stackoverflow, Dev.to, Facebook Groups.
Use tools and methods to better keep track of your time and tasks.

This can be done by using project management software, using agile development methods, or following other best practices for efficient and effective programming.

Use tools like DeskTime, TimeDoctor.
Take breaks often and don't work for long stretches without stopping.

This can help you avoid getting burned out at work and keep your mind on the task at hand.

Pomodoro is a great method to do it.
Don't refactor unless needed.

Rewriting or refactoring code can be a good way to improve how easy it is to read, how easy it is to fix, or how fast it runs. But it can take a lot of time and might not always be necessary.

Before starting a big refactoring project, you should carefully consider whether it will bring real benefits. You shouldn't make changes just to make changes.
Work with other programmers and talk to them.

As a beginner, you can learn a lot from other programmers who know more than you do or have more experience.

By working on projects together, sharing ideas and code, and giving each other feedback and help, you can learn and grow as a programmer more quickly.
Use version control to manage your code.

Version control software, like Git, lets you keep track of changes to your code, work with other people on it, and revert to older versions if you need to.

This can help you organize, speed up, and keep your work safe. It can also help you avoid losing important code or making mistakes.
Put together a portfolio of your work to show possible employers.

When you're just starting out as a programmer, it can be hard to get your first job or project.

Building a portfolio of your work is one way to stand out and show off your skills to potential employers or clients.

This can include finished projects, samples of your code, or other examples of your work that show your skills and potential.
Keep learning new things about programming and keeping up with the latest changes (that is relevant to your goal).

Programming is a field that is always changing, so there are always new languages, frameworks, tools, and methods to learn. You can keep getting better as a programmer and stay ahead of the curve in a field that is getting more and more competitive by staying curious and up-to-date.
Write clean and readable code.

As a beginner programmer, you might be tempted to just focus on getting your code to work and not think about how easy it is to read or fix. But writing code that is clean and easy to read is important for more than one reason.

It can make your code easier to understand and change, which can save you time and effort in the long run. It can also make your code easier to find and use, which can be helpful when working with other programmers or putting your code into larger systems.
Write a lot of tests.

Testing is an important part of the development process, and it can help you find bugs and fix them.

By writing and running tests, you can make sure that your code works the way you want it to, and you can find and fix any problems before they get worse.

This can save you time, effort, and frustration and help you give your users or clients better-quality code.
Take care of both your body and your mind.

Programming can be hard on both your mind and body. It can take long hours of focusing, solving problems, and working with other people, which can be hard on your body and mind.

Taking care of yourself is important if you want to avoid burnout and keep working well. Getting enough sleep, working out regularly, eating well, and taking breaks from your work are all ways to do this.

Taking care of your health will allow you to remain focused, energized, and motivated, allowing you to be a better programmer.
Learn how to debug your code.

Debugging is an important part of development that every programmer needs to know how to do.

By learning to find and fix errors and bugs in your code, you can make it more reliable and faster, and you won't waste time on code that doesn't work the way you want it to.

There are many tools and methods you can use to fix bugs in your code. If you want to become a better programmer, you should learn and practice these skills.
Stick to the rules and best practices for coding.

If you're just starting out as a programmer, you might not know the rules and best practices for writing code in your language or field.

It's important to learn and follow these standards and best practices because they can help you write code that is easier to read, maintain, and use.

They can also help you avoid common mistakes and pitfalls, and they can make it easier for other programmers to understand and use your code.

By following coding standards and best practices, you can improve the quality of your code and make it more useful to you and others.
Take on projects and problems that are hard.

If you're just starting out as a programmer, you might be tempted to take on only small or easy projects so you don't get too busy or frustrated.

But it's important to push yourself and take on bigger, harder projects, because that's how you'll learn the most and grow as a programmer.

You can improve your skills, gain confidence, and build a body of work you can be proud of by taking on hard problems and overcoming obstacles.
Be proactive and take initiative.

Don't just wait for someone to tell you what to do. Look for opportunities to learn and grow, and take on new challenges.
Google everything.

Learn how to google, watch videos and tutorials, visit stackoverflow and reddit. But never copy a code block without understanding how it works.

Also, set yourself a time limit on googling stuff before asking other senior developers.

Things beginners should avoid

Top ⤴️

Here are some things that a new programmer should not do:

Ignore the latest trends or fads in the field of programming.

The world of programming is constantly changing, and there are always new languages, frameworks, tools, and techniques to learn.

However, not all of these developments are equally valuable or relevant, and it can be a waste of time and effort to chase after every new trend or fad.

Instead, focus on the fundamental concepts and principles of programming, and learn the tools and techniques that are most relevant and useful for your goals and projects.
Ignore the pressure to be a perfect programmer.

As a beginner, it is natural to feel pressure to be perfect, to avoid making mistakes, or to measure up to the standards of more experienced programmers.

However, perfection is unattainable, and it is more important to focus on learning and improving.

Be willing to make mistakes, to learn from them, and to continue to grow and develop as a programmer.
Ignore the temptation to compare yourself to others.

As a beginner, it can be easy to compare yourself to other programmers who may have more experience, more skills, or more success than you do.

However, comparisons are rarely useful, and they can be damaging to your confidence and motivation.

Instead, focus on your own goals and progress, and celebrate your own achievements, no matter how small they may seem.
Ignore the voices that tell you that you are not good enough or that you can't do it.

As a beginner programmer, you may encounter many voices that tell you that you are not good enough, that you don't have the right skills or knowledge, or that you should give up.

These voices can come from many sources, including your own doubts, the expectations of others, or the challenges of the field. It is important to ignore these voices and to believe in yourself and your abilities.

You have the potential to become a successful and skilled programmer, and you can achieve your goals if you are willing to put in the effort and dedication.
Ignore the temptation to overcomplicate your solutions.

As a beginner programmer, you may be tempted to overcomplicate your solutions, to add unnecessary features or enhancements, or to try to impress others with your code.

However, simple and straightforward solutions are often the best, and they can be more efficient, maintainable, and effective than complex and sophisticated solutions.

By focusing on the core requirements and avoiding unnecessary complications, you can write code that is more valuable and useful to yourself and others.
Ignore the fear of failure.

As a beginner programmer, you may be afraid of failing, of making mistakes, or of not meeting the expectations of others.

This fear can hold you back and prevent you from taking risks, from exploring new ideas, or from challenging yourself. It is important to ignore this fear and to embrace failure as an opportunity to learn and grow.

By learning from your mistakes and failures, you can become a better programmer, and you can develop the resilience and persistence that are essential for success in the field.

As a beginner, it can be hard to take advice, even if the person giving it means well. But if you can keep going and do what experienced people tell you to do, you'll get better faster than you thought possible. So don't be afraid to take that step and try something new.

Front-end Challenge: Prevent Clicks

Md Abu Taher — Wed, 29 May 2019 18:45:36 +0000

This is a challenge for frontend developers. Test your DOM manipulation and CSS skills.

Create two functions,

`addBlocker()`

Should block all clicks on the page. Nothing should be clickable anymore.
Should print the current mouse position on viewport, and the current element under the mouse on click.

`removeBlocker()`

Should remove the blocker created by addBlocker(), everything should be clickable as before adding the blocker.

The two functions should work on any website including, dev.to and producthunt.com

Here is a video preview of what should be done.

Share your code in the comment section or in a github gist.

Good luck!

Scraping infinite loading (ie: Product Hunt) pages! Without using ajax!

Md Abu Taher — Wed, 29 May 2019 00:36:20 +0000

If you are into building web scrapers, you know how hard it's to scrape infinite loading pages. Most search results you see on google focuses on two methods.

Find the ajax on network tab, try to scrape from it.
Use a combination of document.body.scrollHeight, window.scrollTo and some for loop.

Unfortunately most of them does not work well with lazy loading images, or pages where infinite loading is triggered by smooth scrolling. Fiddling with network tab also seems counter productive in many cases and easily triggered as a bot on some websites.

I was thinking of a solution for this for few years. Tried all different ways and got disappointed because it varied a lot between websites.

Finally it kicked me when I was updating this and this on stackoverflow. Feel free to explore them.

Here is the small preview of what we will be building today. It’s a one minute video (sorry, no audio).

Case:

You need to scrape 100 results from product hunt. The result should contain post title and image url in following structure. It has to stop scraping once the limit has been reached or there are no element left.

[
  {
    "title": "Some product title",
    "img": "https://ph-files.imgix.net/123456-abcdefghijkl"
  }
]

We will be using the surefire method called window.scrollTo, but not with document.body.scrollHeight.

Solution:

PS: If you want to read the code, jump to the final code.

Here what we will do,

We will extract the selector (obviously 🤷).
Then we will find the first element on the page for that selector. We won't continue if there are no element.
Scroll the element into view.
Optional! Wait for a few milliseconds to let it load images and vice versa.
Extract information from that element.
Important! Remove the element from dom.
Important! Scroll to top of the page.
Do the next element or stop if limit has been reached.

The steps marked as IMPORTANT are the key. They will trigger the scroll event on the page without even scrolling manually like others do with document.body.scrollHeight and so on.

Alright, now that you know the solution, you can code the rest of it from the video above.

I'm kidding 😅! Here are the steps!

Extracting the selectors

You probably did this lots of time. But here is a short recap anyway. I will not be writing the selectors directly here because they might change by the time you are reading this post.
Open chrome and load producthunt page. Then right click on a title.

Now pick any of these classes. We will find the right one on next step.

Write down the class name on the console. It will do a instant evaluation, so you will know if the selector is correct or not right away.

Since we have 27 results, we are probably on the right path. Since there are more or less 20-30 results when you load the page first time.

Alright, next we can extract the selector for image.

Fortunately for us, the selector for image is even more straightforward, because we have a nice data attribute there.

However, if you tweak the selector a bit, you will see there are 25 results out of 27 products. Which means it did not load the last two images.

If you scrapped this page right now, you would have 25 proper results.
Additionally I extracted the parent element for each product listing.

Now I see something bit weird, it says 34 results. Which means it did not even load the last 7 results. Not even the title. It's there but not loaded at the moment.

Finally we have three selectors.

Product Entry (Optional): div.white_09016 ul li
Title: .title_9ddaf
Image: [data-test="post-thumbnail"] img

These selectors can change any time since it's a react based website.

Scrape the data

You can execute these code on the browsers console or using some script/library ie: puppeteer has a page.evaluate method for executing functions. I will be using Scratch JS to run the code on the page.

Grab Single Product
Let's create an async function called scrollAndExtract which accepts two parameters called selector and leaf. Leaf is the innerText and src etc.

We need async because we will be using a delay inside the function for showcase purpose.

const scrollAndExtract = async (selector, leaf) => {
  const element = document.querySelector(selector);
  if (element) {
    element.scrollIntoView();
    return element[leaf];
  }
};

Let’s run it,

scrollAndExtract(".title_9ddaf", "innerText").then(console.log);
scrollAndExtract('[data-test="post-thumbnail"] img', "src").then(console.log);

Cool! We got the first title and image url.

Scroll and Remove the element

Next we will remove the element from the view. We can do this in a simpler manner by adding another parameter and tweaking our function for a bit.

Let’s add a remove parameter. If it’s provided, we will remove the element instead of extracting the data.

const scrollAndExtract = async (selector, leaf, remove) => {
  const element = document.querySelector(selector);
  if (element) {
    element.scrollIntoView();
    if (remove) return element.remove(); // <-- Remove and exit
    return element[leaf];
  }
};

Let’s test it out,

scrollAndExtract(".title_9ddaf", "innerText").then(() => {
  scrollAndExtract(".title_9ddaf", null, true);
});

The product title vanished,

Scrape the image

Now we can scrape the image as well in similar fashion.

scrollAndExtract('[data-test="post-thumbnail"] img', "src").then(() => {
  scrollAndExtract('[data-test="post-thumbnail"] img', "src", true);
});

This will extract the src attribute from the image.

Both of them can be merged into a single function which returns an object. We can push it to an array later.

async function extractor() {
  const title = await scrollAndExtract(".title_9ddaf", "innerText");
  await scrollAndExtract(".title_9ddaf", null, true);

  const img = await scrollAndExtract('[data-test="post-thumbnail"] img', "src");
  await scrollAndExtract('[data-test="post-thumbnail"] img', null, true);

  return { title, img };
}

Let’s test it out,

extractor().then(console.log);

Optional: Remove parent container for the title and image
Let’s remove the parent element after scraping the title.

This is optional because the logic will work even without this. But it will save us some space on the viewport and memory as well since we are removing the dom element.

We can remove the parent container and won’t have to worry about removing image or title element since it will be removed as well.

async function extractor() {
  const title = await scrollAndExtract(".title_9ddaf", "innerText");
  const img = await scrollAndExtract('[data-test="post-thumbnail"] img', "src");

  // remove the parent here
  await scrollAndExtract("div.white_09016 ul li", null, true);
  return { title, img };
}

It should work flawlessly,

Loop through 100 elements
We won’t be using a traditional for loop. We will use recursion instead.

Let’s create another function to go through the elements one by one. We will store the results in a result array.

const products = [];
async function hundredProducts() {
  if (products.length < 100) {
    const data = await extractor();
    if (!data.title || !data.img) return null;

    products.push(data);
    return hundredProducts();
  }
}

This will grab the first hundred elements for us. Not only that, it will stop the loop if there are no results from extractor.

We can peek into products array to grab our results.

hundredProducts().then(() => console.log(products));

And bam!

We got 7 results!

Wait! Wut?

Adding small delay to lazily loaded product data with images
As you can see, we got only 7 results. That’s because we told it to stop the loop if there are no image/title. It scrolled too fast to trigger any scroll event and loading new data.

Let’s use a simple delay function, which will wait for a bit before running the loop.

const delay = d => new Promise(r => setTimeout(r, d));

Also, optionally we will scroll to top of the page.

const products = [];
async function hundredProducts() {
  if (products.length < 100) {
    // Let's wait 0.5 seconds before moving to next one
    await delay(500);
    // also trigger a scroll event just in case
    window.scrollTo(0, 0);

    const data = await extractor();
    if (!data.title || !data.img) return null;

    products.push(data);
    return hundredProducts();
  }
}

Final Result
Alright! It’s been a long post and now we have a script and logic to scrape infinity scrolling pages like producthunt.

Here is the complete code which you can run on your browsers console. Make sure un-comment the line to run hundredProducts() and then log products array.

const delay = d => new Promise(r => setTimeout(r, d));

const scrollAndExtract = async (selector, leaf, remove) => {
  const element = document.querySelector(selector);
  if (element) {
    element.scrollIntoView();
    if (remove) return element.remove(); // <-- Remove and exit
    return element[leaf];
  }
};

async function extractor() {
  const title = await scrollAndExtract(".title_9ddaf", "innerText");
  const img = await scrollAndExtract('[data-test="post-thumbnail"] img', "src");

  // remove the parent here
  await scrollAndExtract("div.white_09016 ul li", null, true);
  return { title, img };
}

const products = [];
async function hundredProducts() {
  if (products.length < 100) {
    await delay(500);
    window.scrollTo(0, 0);

    const data = await extractor();
    if (!data.title || !data.img) return null;

    products.push(data);
    return hundredProducts();
  }
}

// hundredProducts().then(() => console.log(products))

Optional: Puppeteer script

If you want to automate this with puppeteer you can put the code inside a page.evaluate function. Here is a snippet and here is the git repo with complete code.

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto("https://www.producthunt.com");
  const productList = await page.evaluate(async () => {
    // paste the final code here
    // ...
    // run the function to grab data
    await hundredProducts();
    // and return the product from inside the page
    return products;
  });

  await browser.close();
})();

Closing Thoughts

This looks ten times bigger than all other posts on the internet. But you know the original version above is a lot smaller as well. No crazy scroll to height or else.

But hopefully I was able to show you a different way than how you normally scrape. Feel free to fiddle and experiment with the data.

Let me know what do you think of this method and what you think is the best method out there for scraping infinity scrolling pages in general.

How NOT to ask for help

Md Abu Taher — Fri, 05 Apr 2019 19:41:23 +0000

Stack Overflow Isn’t Very Welcoming. However an environment does not become toxic on it's own. The people who uses the platform along with unwritten laws keeps making things worse for developers and newbies.

Everyone knows about it, them too!

One thing before I continue,

IF YOU’RE A BEGINNER, JUST IGNORE THE TOXIC WORDS.

Every now and then I would review and answer questions on stackoverflow regarding web scraping and puppeteer. And I know how much toxic stackoverflow can be.

Me? No I am not Jon Skeet, I am just a small human with mere 5k reputation on SO. Many doesn't even know my name.

The toxic environment is on both side. Newbies feels overwhelmed. However I can certainly feel the rules and regulations are there to stop wasting everyone's time. Everything you write on stackoverflow will help someone in future just like someone else's answer helps you.

If you want to ask for help, be prepared to get rejected. You have to learn through rejection. Maybe your question will be bad, maybe it's already answered many times before.

This applies to any and every platform, no matter how newbie-friendly they are, it's chaotic whenever there are thousands of people involved. This also does not mean the platform is toxic. There are happy and grateful people who asks valid questions and get proper answer.

Here are some important points to avoid when asking for help.

"Can you please write the whole damn code for me?"

QUESTION: I have Javascript puppeteer code, and PuppeteerSharp for C#. I know that this libraries are similar, and i know their sites. But my problem that i barely can manage this libraries, there are alot of methods for each lib, and it hard to find needed methods, even i have working example written on JS. Please help me rewrite JS code to C#, so it would do similar things. Or at least function names, for example JS (puppeteer) method = C# (puppeteerSharp) method.

This question asks for help. It's good that they asked for help, asking takes courage. However, no, the people on the internet does not have time or will to work for free. You need to find a very specific problem and ask about it.

"I cannot share my code nor my research done till now. Heck I cannot even share my problem."

Kindly no! Please do share what you did till now! Please tell us where you stuck. We do not know mind reading. We need to understand what's wrong and how we can help.

It shows us your research and effort. We are more willing to help those who works hard. Your research will help you and others who comes to same problem.

"Can you solve all problems I mentioned?"

All questions on same page,

Why it shows infinite while loop?
What is promise.all?
Where do I get $$?
How to get outerHTML?
...

All these shows us you did not do a single research, took a big project/task you were not supposed to take and feeling overwhelmed.

Actually it's better if you can break your problem into multiple chunks. Do not bombard the whole page with questions, you will get downvoted right away. If you split the problems into pieces, it will help others know exactly where to help. Maybe multiple people will help about multiple sub-topics.

Summary

Yes, that's all. Internet is full of toxic people. And if you want to learn within that, you need to know how to ask for help,

Do your research.
Ask very specific questions.
Value time, yours and others.
Respect people.
Share your knowledge.

Have fun programming!

The Reading List Hoarding Problem

Md Abu Taher — Wed, 20 Mar 2019 13:49:39 +0000

Do you like to click that shiny [SAVE] button all the time? This post might be for you.

Here is the current situation of reading list on dev.to,

Note: When I first started writing this article, it was 328, now it's 356 😱, how can I write a post that I myself cannot get rid of? Well friend, I have a reading list hoarding issue at this moment of writing that I want to get rid of. This article is dedicated to me. I just want to share my experience.

Same goes to my bookmark tab, there are at least 2k+ bookmarks there.

Here is current situation of pocket,

Hashnode also has overused bookmark feature:

I have had feeds, pinterest, medium, facebook saved posts, and hundreds of other solutions just to save something for later.

If I remember, I bought books, courses, tutorials, collected them in hundreds of folders, just like the reading list hoarding, and the time I finish my reading list? It doesn't come ever. I simply forget to read them later because I have a busy life.

Anywhere from 2% to 6% of adults have hoarding disorder. Yes, this is a disorder and it exists even on the internet in my opinion.

Thus, we need to talk about it.

Types of hoarders

There are few types of hoarders and you might be one of them.

"I might read it later" Hoarder

Some reader has a good collection, all sorted out properly consisting of titles they have been dying to read. However, some of these readers completely forget the existence of their previous list as soon as they see another awesome/popular post/media.

Once they save it, they begin thinking how they don't have time to read it right now, and vow to read it someday soon, and forget their vow pretty soon.

The cycle continues as they find more awesome posts that they might read later.

"I love this writer, I will read it later" Hoarder

They bookmark all/any video posted by linustechtips (or insert popular tech name), any post written by Molly Struve or Emma Wedekind or (insert your favorite/popular writer).

It shows love and appreciation to the writer. It's alright and encourage them to produce more valuable content.

But as a hoarder, the reading list grows and most simply forgets to read it. They end up in a mental breakdown once someone says something opposing their favorite writer or if they cannot find the content on their saved list.

"This post changed my life" Hoarder

They read the post, start relating their life with the content, they read it with so much passion that it end up having a huge impact on their life.

After that, no other post seems as important as that one. They might bookmark some other post, but end up reading same post over and over.

They save a post just for sake of saving, they read a post only to change their life.

"I will read this again" Hoarder

Sometimes they are not fanatic as "Impacting Hoarder" mentioned above, however they might read the post again just to refresh their brain once in a while.

They will use the content in good use, sometimes in research, sometimes in their content.

However they do not get rid of the bookmark once it's done processing one/few times. They do not love it, but they do not delete it either.

"I disagree and I must prove it properly" Hoarder

Usually the above others might save and read the content in a while. These are few others, who spend time to analyse the posts, line by line, and share their reviews with others.

Some shares short reviews, tweets or videos, but the kind of hoarder on this section are not them.

They form a very decided opinion of the post, read it few times, compile a list of reasons to justify their feelings. With a clever use of bullet points and respectful words, they present their case with other readers in a very persuasive manner.

They read the content and always end up finding a fault somewhere. No matter how much you reason with them, they will not budge.

It's not always negative though, sometimes these kind of hoarder brings very useful information and conversation. You will learn a lot from their opinion and have chance to rethink something from a completely different angle.

What kind of hoarder are you? Does it matter?

There are other types of hoarder outside this list, but it does not matter. At least they are hoarding links, and doing something that might bring value at some point. They are just trying to escape reality every now and then, having some peace in mind.

Once they acknowledge their problem, they will end up bringing more than others who actually does not take any action at all.

Bad sides of hoarding

You are increasing junk usage. The more you do useless stuff, the more server space is gonna be used, more energy usage, more global warming. Just because you saved something useless.
It's hard to find important stuff. If you want to find something you saved at beginning, most of time you will have a hard time scrolling down and pick the important one.
It makes you lose focus. You do not know what to learn anymore. You want to learn web development, mobile development, project management, server management and everything else all at once.
It gives you anxiety. When will you read that huge list? What about the new things you want to learn? Who is going to filter out things you do not want to learn anymore?
It causes you impostor syndrome. You feel there is so much left to learn. Everyone knows a lot more than you and sharing a lot of valuable information. Do you even belong here?

Getting rid of hoarding

This is going to be a long journey just like having an inbox zero. You will need solid proof that you are a hoarder, you will have to convince yourself. (Just like I accepted myself that I hoard links).

Get motivation

Overcoming hoarding will take some impressive commitment. So before you start, you first need to get really motivated about doing so. Getting motivated will help you stay consistent in your plans.

Make a list of strong reasons why you want to stop hoarding. For example, "I want to be able to understand this writer better", "I want to learn JavaScript as soon as possible". I don't know, your list, your reasoning.
Review the list whenever you feel your decision is changing. It will remind you of yourself.

Understand the difference between hoarding and collecting

A hoarder saves links/items in excess and has difficulty discarding or reading them later. It's hard to find and they mostly do not know what they are bookmarking and where it goes. Sometimes they will remember that they have this bookmarked somewhere near 1000th bookmark, but not the actual name anywhere. It just goes there and does nothing.

Collectors have a passion for gathering things as a side interest. Many things can be collected. It is totally up to them what they want to collect. They organize and display the links. They help others with their content, "Top 10 Books I have read last week", "Top 10 public speaking tips" etc.

Start Slow

Do not go and remove all links from every single place. Do not throw out all books at once, do not throw out all junks in dumpster, do not remove all piece of code and start from scratch just randomly.

Open one or collection each week/month. First week go through one writer/category, second week another, and so on.

Handle one post at a time

Stop keeping things "For Now", it won't help much. Handle this one thing just only once so you do not end up handling same link over and over.

If it's a post, read it properly. If it's a video, watch it properly. Decide whether you should watch/read fully at that moment.

One post at a time.

Scale down and organize it

One big glob of thousand line of messy code vs thousand small glob of messy code. Yeah, both are terrible. Messy is terrible.

However, consider one small piece of code at a time, organize it. It might take time, but by end of the day/week, you will end up with some satisfaction.

Same goes for links, if you try to organize all of the links at once, you will get frustrated. Just handle it smoothly and slowly. Just focus on processing and making it shorter.

Scale down when needed, but don't force it.

Make use of the organizing features

On dev.to, you can see the saved posts by category/tags, pocket and other tools also lets you do that.

It's best to make the best use of this. If you are learning javascript, go through that, if you are a beginner or trying to help a beginner, go through that, if you are having problem with testing, go to #testing tag. And the list will clean up, you will end up with a positive and healthy mind.

Toss uninteresting saves

You wanted to learn devOps (just saying for example) and saved a bunch of links. But now you are not interested in it.

Toss it away. Remove it from your list.
Send it to people who will actually benefit from the post. It will increase communication, make you much more helpful to community.

Don't procrastinate

If you do not put a deadline, whatever you left on the reading list will be there forever. Set a note on your calendar, whiteboard or sticky. If you do not read the post by that time, remove it unconditionally.

Keep it simple

You do not need to learn everything at once. Just because you might want to learn kotlin, rust, c, java, php, python, machine learning in future does not mean you should just collect links for them. It will stay there, do not collect it if you do not need it.

Do not leave it there, if you clean this up, it will make you a better you. Cleaning out reading list will help you understand the bad points of hoarding and you can apply it in real life, getting rid of book, course and many other hoarding habit.

Kindly do not save this post for later. Love it and share it with your friends instead to make sure they also benefit from this.

Prevent hoarding

Do not jump into reading just random posts. Do not click save button just yet. Try to understand the point and go through the post quickly for a bit.

Stay focused and organize. Your list will have much more quality. This habit will help you in other areas of your life.

If you ever feel you have this problem out of control. Seek professional help. This is especially true if it is affecting your life in a significant manner. Extreme hoarding is a sign that you have other underlying issues.

Read, Engage and Share instead of Saving. That's one way to contribute and taking action.

Gathering courses, links and books only to never read it again? That's some serious problem. You should take action now!

Maybe I am wrong, maybe this was never an issue. Let me know in the comments what you think about it.

EDIT: It's funny the first reaction this post gets is also a [SAVE] reaction.

VSCode Trick: Reset the Sidebar and Terminal Pane Size

Md Abu Taher — Tue, 12 Mar 2019 09:21:42 +0000

Have you ever resized sidebar and terminal like this?

After that, you either did not care or tried to manually resize it back, right? But did you ever feel you forgot actual size or did not have the satisfaction?

Just double click the borders of the pane you want to resize.

Same applies to extension or any other activity, where you want to see the full contents.

That's it! It is now resized back to appropriate size based on the contents of the sidebar and viewport size.

Peace!

Automated testing with mocha and puppeteer on circleCI

Md Abu Taher — Sun, 16 Sep 2018 18:55:02 +0000

Prefetch

I always had trouble understanding why the code quality is important, I would refuse to spend time on this. But recently I found myself in sticky situation over and over because of the quality. If the quality was good, then I would not have to spend million seconds thinking about the problem and structure. Thus I learned mocha, chai, browser automation libraries like puppeteer. But soon I had problems with the CI and code quality checking tools.

I had a very hard time figuring out how to integrate all these newly acquired knowledge into the projects at once.

It's like trying to build an whole app without writing the first line.

I would feel empty when I saw nice delicious badges on other repos and some repos had multiple green ones and my repos were almost empty, the code was undocumented and such newbie stuff.

Then it hit me, I could just create a sample repo from beginning with everything I want step by step and apply my knowledge there at very basic level so I can repeatedly use same process for other projects and relieve my stress over time. That's how this repo was born.

The result is awesome and I got to add some cool badges too just like others,

Ah! the joy when someone tell you the quality is good (be it just for some peace of mind).

What this is

A sample repo to show how mocha, puppeteer and a ci can be used to automate various tests. Maybe an automation script was created, but depending on the size of project even that script should be tested to make sure everything works as expected when there is a commit pushed.

We can build this using BDD, TDD or any other methodology, but whatever method is being used, the steps will always include,

Creating actual/automation script
Creating test/mocha script
Creating/Preparing CI environment

Right now most CI and servers always prepares the Xvfb and other required libraries, so we don't have to deal with them too much unless explicitely required.

NOTE: This readme/post will avoid going into depth and only discuss matters on top, assuming you already know about nodejs, unit testing, mocha/chai, puppeteer and CI. You don't have to know everything, just having the basics will be enough.

PS: THE REPO IS HERE.

Why tests?

Yeah, why we are just making simple automation scripts?
Right now we are testing only a sample function. It doesn't seem much. However, imagine 100 scripts or thousand scripts. Is it possible for you to run and test every single script to make sure it works? It won't be possible, and it will become a very stupid headache once that happens. So, even if you don't need these kind of tests, they will help you on long run once you have to do same thing over and over again.

If you are building a scraper, and have to browse 100 sites or fill 100 forms, it will help when you got to know the script works and can reach the target website/forms when possible. It will also alert you when things goes south.

Basic example script

We will create a sample script that will browse example website, extract title and outerHTML and close itself. We will treat the script as an independent one, so we won't be dealing with the closing/opening the browser on the testing environment, rather it will just be there to show how it works.

Creating the actual script

We will use node 8 with async-await support, and puppeteer for this.

Basically what we will do is,

Launch the browser.
Navigate to website
Collect title and outerHTML for testing purpose
Close browser and cleanup the processes
Return the data

// app/grab-data.js

const puppeteer = require('puppeteer');

const grabData = async () => {
  const browser = await puppeteer.launch({
    // will greatly affect the results
    headless: true,
    // important for running on various server where root user is present
    args: ['--no-sandbox', '--disable-setuid-sandbox'],
  });

  // actual navigation happens here
  const page = await browser.newPage();
  await page.goto('https://example.com');

  // sample data collection
  const title = await page.title();
  const outerHTML = await page.evaluate(() => document.querySelector('body').outerHTML);
  // cleanup once done
  await browser.close();
  return { title, outerHTML };
};

module.exports = grabData;

Creating test script

When using unit tests, it's important to use timeouts. Make sure to exit the tests, and use proper timeout. Recent mocha will not exit the tests, in that case or in a case where you have infinite timeout, you might end up with a huge cost for the test getting stuck. Mocha by default will run tests one by one. If you want to try asynchronous tests, you can use Ava. Both are fantastic in their own way, popular and has huge support.

You do not need to test every single thing on automation scripts. Just make sure the scripts works at very basic level for each possible scenerio. If you push 100 commits and the automation script has login and such, it will end up doing 100 logins in very short amount of time, and trigger various alerts, unless it's your website you are testing against.

We will create the test script with the actual user flow in mind. Check the test/basic.js for test code. The code is readable even without comments, (and I additionally added too much to the code just to get rid of all eslint errors.)

// test/basic.js

const { expect } = require('chai');
const grabData = require('../app/grab-data');

describe('tests example website', function testExample() {
  this.timeout(30000);

  it('browses the website', async function testBrowser() {
    const { title, outerHTML } = await grabData();
    this.title = title;
    this.outerHTML = outerHTML;
  });

  it('title has text: example', async function testTitle() {
    expect(this.title.toLowerCase()).to.have.string('example');
  });

  it('outerHTML has text: example', async function testOuterHTML() {
    expect(this.outerHTML.toLowerCase()).to.have.string('example');
  });
});

The result:

Code quality

Linting

Sometimes you want to make sure the quality of the work is well and you are not pushing garbage inconsistent code to your repo. If you have multiple contributors to the repo you can even make sure their code passes all linting tests (excluding all eslint-disable hacks), and it returns safely.

Thus we are simply gonna use, eslint --init, and follow the wizard to install airbnb rules, then run using eslint .,

...

opps! and then eslint --fix . --ext .js for auto fixing basic problems.

Also we can include these in the scripts inside package.json and simply call it later on.

Since we have mocha and node as target, we have to manually edit things a bit and add node and mocha as the target environment. Otherwise it will throw errors like, 'describe' is not defined.

And since we are using browser code inside puppeteer, it will throw errors like how document is not defined and such. We need to include browser as targets too, or apply workarounds.

We are all good once we put and lint it. It might throw on some small warnings about best practices, we can fix them if we want, or move on.

The result:

Other quality checks

If not enough, we can use other code quality checking tools such as codacy. All we have to do is add the project and it will do everything else itself. It will clone project, detect programming languages, run various code patterns and calculate project metrics. Finally it will give us a project certification with a grade A to F, A being best, F being worst. The sample repo might get A at beginning, or an F. But if we are not careful enough, it will scream at us later on just like the other tools we are using.

Creating/Preparing CI environment

Headless is a big factor on browsing and testing. Sometimes headless will leave a big footprint and get caught as bot, otherwise it's perfect for testing own websites where we can explicitly exclude it for testing.

Enabling headless will take 6+ seconds to launch on CI environment sometimes, while disabling headless will reduce the time to 1 second. It can be optimized and hacked thru. While this doesn't seem like a big deal, it is indeed a big deal if you have a tight budget and CI costs are really high. Most CI gives you free time to run, but it will quickly add up where you have almost limited resource.

Make sure to enable headless unless explicitly required.

CircleCI has node:8.12.0-browsers and other similar tags that includes the required libraries for headless and xvfb display. Just using their wizard is enough to get you going.

We will need to include the linting, and then we will run the actual test, make sure it's good quality code and works well, without running it ourselves on our own machine. We can even apply the fixing if we are brave enough.

Check out .circleci/config.yml for the code. The result looks beautiful, except it won't if all tests were ended in red.

Final steps

So far we almost copy pasted everything or followed wizard and everything was done. We just need to setup the actual project on circleCI and link the repo. Then push the repo will trigger the build.

That wasn't hard, was it?

It triggers on every commit

It will be very bad to trigger hundreds or thousands of tests on every commit/push. Sometimes you will just update the readme files and won't want to trigger things. And sometimes you will want to trigger only after making a major change. To prevent such worse things from happening, make sure to push to a separate branch and do pull request only if you are satisfied with current code changes. Merging to master branch will trigger the build, which also can be setup as various webhooks and breaking measurements.

So far this repo did not need such a big readme, but I ended up writing it because I was very lazy at writing documentations and ended up confused whenever I needed some script from some old repo to use.

Later I will be able to extend this repo with awesome things like,

process management,
queue management,
docker builds,
code coverage
server deployments.

So much potential!

Why are you programming?

Md Abu Taher — Mon, 30 Jul 2018 15:33:36 +0000

People started to learn web development thinking about future, right now web development is no longer a thing for future, it's present. However, this arises another concern, what's now? AI? Machine Learning? Self driving? Mars? Aliens? Prevent global warming? More programming?

Predictions can go wrong any time, and they are dangerous. However you, those who don't live in the past or present, have some ideas/plans that you are following, right?

So,

What is it? What do you think will be here 5-10-50 years later?
Are you planning for that? What's your plan?
Why are you programming/developing/hacking...or doing whatever you are doing?

Hacking a bit of dev.to with CSS

Md Abu Taher — Thu, 07 Jun 2018 00:39:13 +0000

The problem: No sticky save for lazy developers

When you are browsing this website, you get the titles like this, sometimes there are description, images and tags.

So you have two options here, either open to read it, or save it for later. Sometimes the titles are tricky as this post you are reading right now. Maybe you thought it was something very interesting, but it was actually a trap to get views/likes etc. Let's open the post to see what's inside.

Great! It's a great article to read with lots of valuable information. But wait, you got to hurry up with your train so you have decided to save the post.

The save button is either on the previous page or at the bottom of current post, no save button on the side of the post where you could easily just save it for later. Maybe you can use bookmarks with ctrl + d, but that'd be so boring.

We can either wait for the team to build one for us or we can build one ourselves. Won't it be great to fiddle with some little stuff sometimes?

The Solution: Fiddling with CSS

Okay, let's extract the selector first. I'll right click, select "Inspect Element" and it will be highlighted like this.

Now we know what to break. Let's make some css code for the selector,

#article-reaction-actions {
    position: fixed; /* This will make it floating around */
    z-index: 1000000; /* Just to make sure it's on top of everything */
    bottom: 0; /* So it stays sticky to bottom of my screen */
    background: white; /* Well it's transparent otherwise */
    width: 880px; /* The size of the parent container */
}

I am not going to discuss what each letter of the above snippet does. I'm too lazy to do that right now.

To apply the style, I'll use Stylish, which is a chrome extension. There are firefox and safari version too.

Let's create a new style. when I am reading this post, I will install the chrome extension, come back to this post, and then click the icon.

Then I'll use the above snippets code, enter a nice name and save it.

Great, now when I refresh the page, I'll see this,

Nice, isn't it? I can use this to read first few lines and decide to save or heart it without scrolling to end.

PS:

This was just to relieve my boredom, but I use Stylish for many different websites. I also use Tampermonkey to fiddle with the JS if needed.
Please don't make a change on website or create a chrome extension just for this post. This is no big deal.
Sorry for using posts on images without permission :D, it's dev.to, so I guess it won't hurt.

DEV Community: Md Abu Taher

QA: Npm scripts vs gulp or similar test runners?

Question:

Answer:

State of AI and AI Content Detection

Why you shouldn't rely on ChatGPT so much!

Steps I have taken

Conclusion of the experiment

What does ChatGPT have to say about this?

Conclusion

Reflections on My 15 Year Journey from Novice to Intermediate Developer

Table of contents

How do you spot a beginner programmer?

Things beginners should know

Things beginners should avoid

Front-end Challenge: Prevent Clicks

addBlocker()

removeBlocker()

Scraping infinite loading (ie: Product Hunt) pages! Without using ajax!

Case:

Solution:

Extracting the selectors

Scrape the data

Optional: Puppeteer script

Closing Thoughts

How NOT to ask for help

"Can you please write the whole damn code for me?"

"I cannot share my code nor my research done till now. Heck I cannot even share my problem."

"Can you solve all problems I mentioned?"

Summary

The Reading List Hoarding Problem

Types of hoarders

"I might read it later" Hoarder

"I love this writer, I will read it later" Hoarder

"This post changed my life" Hoarder

"I will read this again" Hoarder

"I disagree and I must prove it properly" Hoarder

What kind of hoarder are you? Does it matter?

Bad sides of hoarding

Getting rid of hoarding

Get motivation

Understand the difference between hoarding and collecting

Start Slow

Handle one post at a time

Scale down and organize it

Make use of the organizing features

Toss uninteresting saves

Don't procrastinate

Keep it simple

Prevent hoarding

VSCode Trick: Reset the Sidebar and Terminal Pane Size

Automated testing with mocha and puppeteer on circleCI

Prefetch

What this is

Why tests?

Basic example script

Creating the actual script

Creating test script

Code quality

Linting

Other quality checks

Creating/Preparing CI environment

Final steps

It triggers on every commit

Why are you programming?

Hacking a bit of dev.to with CSS

The problem: No sticky save for lazy developers

The Solution: Fiddling with CSS

`addBlocker()`

`removeBlocker()`