<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Daniel Cerverizzo</title>
    <description>The latest articles on DEV Community by Daniel Cerverizzo (@dcerverizzo).</description>
    <link>https://dev.to/dcerverizzo</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F131285%2F0ca58fa8-7cc8-4840-b39b-d2a1883d4afa.jpeg</url>
      <title>DEV Community: Daniel Cerverizzo</title>
      <link>https://dev.to/dcerverizzo</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/dcerverizzo"/>
    <language>en</language>
    <item>
      <title>Comprehension Debt: o custo invisível de programar sem entender🇧🇷</title>
      <dc:creator>Daniel Cerverizzo</dc:creator>
      <pubDate>Sat, 31 Jan 2026 04:39:49 +0000</pubDate>
      <link>https://dev.to/dcerverizzo/comprehension-debt-o-custo-invisivel-de-programar-sem-entender-4loi</link>
      <guid>https://dev.to/dcerverizzo/comprehension-debt-o-custo-invisivel-de-programar-sem-entender-4loi</guid>
      <description>&lt;p&gt;&lt;a href="https://dev.to/dcerverizzo/comprehension-debt-the-hidden-cost-of-coding-without-understanding-47ji"&gt;🇺🇸 Read in English&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Afinal, tecnologias que geram código são nossas inimigas?&lt;/p&gt;

&lt;p&gt;A resposta é não. A tecnologia nasce para ajudar.&lt;br&gt;
Mas nem sempre o efeito é exatamente o que esperamos.&lt;/p&gt;

&lt;p&gt;No final de 2022, tivemos o lançamento do ChatGPT, uma das ferramentas mais impactantes desde a invenção da internet. Desde então, vimos mudanças rápidas na sociedade e no mundo do desenvolvimento não foi diferente. Entramos na onda: usamos IA para tirar dúvidas, gerar soluções e, principalmente, escrever código.&lt;/p&gt;

&lt;p&gt;Com esse avanço, algo curioso começou a acontecer. Fóruns técnicos mudaram de comportamento. Plataformas como o Stack Overflow passaram a receber menos perguntas. O padrão mudou: menos dúvidas públicas, mais prompts privados.&lt;br&gt;
Isso indica uma transição importante: as pessoas deixaram de perguntar por quê e passaram a pedir faça por mim.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F6y73x4d3b5x70skgga4n.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F6y73x4d3b5x70skgga4n.jpeg" alt="Imagem de gráfico demonstrando o numero de perguntas geradas no stack over flow" width="800" height="705"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Para um desenvolvedor sênior, usar um prompt para gerar uma classe pode ser apenas um atalho de produtividade.&lt;br&gt;
Mas e para um desenvolvedor iniciante?&lt;/p&gt;

&lt;p&gt;Se até Linus Torvalds já afirmou usar ferramentas de IA, por que nós não poderíamos?&lt;br&gt;
A diferença está em um detalhe que quase não entra na equação:&lt;br&gt;
o iniciante entende o que a máquina está escrevendo?&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fynsvnycgd9vve80ggoa0.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fynsvnycgd9vve80ggoa0.jpeg" alt="Mensagem de commit de Linus afirmando que gerou o codigo usando IA" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Esse ponto nos leva ao conceito de comprehension debt.&lt;/p&gt;

&lt;p&gt;Assim como technical debt surge quando escolhemos atalhos técnicos, comprehension debt surge quando escolhemos atalhos mentais. É o acúmulo de código que funciona, mas que ninguém realmente entende nem mesmo quem escreveu.&lt;/p&gt;

&lt;p&gt;Você copia, cola, ajusta… mas não compreende o motivo.&lt;br&gt;
Isso cria uma falsa sensação de progresso.&lt;br&gt;
A tarefa foi concluída, mas o aprendizado não aconteceu.&lt;/p&gt;

&lt;p&gt;Imagine um navio navegando sem bússola e sem mapa. Ele até anda… mas não sabe para onde está indo.&lt;br&gt;
Programar sem entender é exatamente isso: produzir código sem compreender o “porquê” das decisões.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Não é código ruim.&lt;/em&gt;&lt;br&gt;
&lt;em&gt;É código sem dono.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A IA não &lt;strong&gt;cria&lt;/strong&gt; esse problema.&lt;br&gt;
Ela &lt;strong&gt;amplifica&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;A partir de um prompt, o código aparece rápido e a tarefa parece resolvida. Isso gera a impressão de eficiência, mas também pode gerar aprendizado superficial.&lt;br&gt;
Se a IA resolve rápido, quem aprende?&lt;/p&gt;

&lt;p&gt;Como saber se você está acumulando comprehension debt?&lt;/p&gt;

&lt;p&gt;Algumas perguntas simples ajudam:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Você conseguiria implementar essa funcionalidade sem usar prompts?&lt;/li&gt;
&lt;li&gt;Você sabe explicar o que o código gerado está fazendo?&lt;/li&gt;
&lt;li&gt;Você tem medo de mexer no código sem pedir ajuda da IA?&lt;/li&gt;
&lt;li&gt;Você apenas testa no localhost e torce para funcionar?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Se você não consegue explicar a solução, provavelmente não entendeu o problema.&lt;br&gt;
E se você só testa e torce… algo está errado.&lt;/p&gt;

&lt;p&gt;O custo disso aparece com o tempo.&lt;br&gt;
Você evolui mais devagar, cria dependência das ferramentas e fragiliza sua base como desenvolvedor.&lt;br&gt;
Em um processo seletivo, sem IA, como você se sairia?&lt;/p&gt;

&lt;p&gt;Você troca velocidade hoje por estagnação amanhã.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;O objetivo não é parar de usar IA.&lt;/em&gt;&lt;br&gt;
&lt;em&gt;É mudar como você usa.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Como conselho prático:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Peça para a IA explicar o código que ela gerou.&lt;/li&gt;
&lt;li&gt;Reescreva a solução com suas próprias palavras.&lt;/li&gt;
&lt;li&gt;Simule mentalmente o fluxo do programa.&lt;/li&gt;
&lt;li&gt;Faça pequenos testes e alterações para ver o efeito.&lt;/li&gt;
&lt;li&gt;Tente escrever primeiro e só depois peça revisão da
IA.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Use a IA como copiloto, não como motorista.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Como ferramenta auxiliar, ela é incrível.&lt;br&gt;
Mas só se você estiver navegando com mapa e bússola e não apenas sendo levado pela corrente.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>beginners</category>
      <category>careerdevelopment</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Comprehension Debt: The Hidden Cost of Coding Without Understanding 🇺🇸</title>
      <dc:creator>Daniel Cerverizzo</dc:creator>
      <pubDate>Sat, 31 Jan 2026 04:33:07 +0000</pubDate>
      <link>https://dev.to/dcerverizzo/comprehension-debt-the-hidden-cost-of-coding-without-understanding-47ji</link>
      <guid>https://dev.to/dcerverizzo/comprehension-debt-the-hidden-cost-of-coding-without-understanding-47ji</guid>
      <description>&lt;p&gt;&lt;a href="https://dev.to/dcerverizzo/comprehension-debt-o-custo-invisivel-de-programar-sem-entender-4loi"&gt;🇧🇷 Leia em português&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Are technologies that generate code our enemies?&lt;/p&gt;

&lt;p&gt;The answer is no. Technology exists to help.&lt;br&gt;
But the result is not always what we expect.&lt;/p&gt;

&lt;p&gt;At the end of 2022, ChatGPT was released, one of the most impactful tools since the invention of the internet. Since then, we have seen fast changes in society and software development was no exception. We joined the wave: using AI to solve doubts, generate solutions, and mainly write code.&lt;/p&gt;

&lt;p&gt;With this growth, something interesting started to happen. Technical forums began to change. Platforms like Stack Overflow started receiving fewer questions. The pattern shifted: fewer public doubts, more private prompts.&lt;br&gt;
This shows an important transition: people stopped asking why and started asking do it for me.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F72in5uv01gbylrrq8xa3.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F72in5uv01gbylrrq8xa3.jpeg" alt="Stack Over Flow graphic show after GPT launch decrease number of questions" width="800" height="705"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For a senior developer, using a prompt to generate a class can be just a productivity shortcut.&lt;br&gt;
But what about a beginner?&lt;/p&gt;

&lt;p&gt;If even Linus Torvalds uses AI tools, why couldn’t we?&lt;br&gt;
The difference is a detail that is often ignored:&lt;br&gt;
does the beginner understand what the machine is writing?&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fww3hmb8h76ux164t6dyv.jpeg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fww3hmb8h76ux164t6dyv.jpeg" alt="Linus Pull request message affirm using AI to generate code" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This brings us to the idea of &lt;em&gt;comprehension debt&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Just like technical debt comes from technical shortcuts, comprehension debt comes from mental shortcuts. It is the accumulation of code that works, but no one really understands not even the person who wrote it.&lt;/p&gt;

&lt;p&gt;You copy, paste, and adjust… but you don’t understand why.&lt;br&gt;
This creates a false feeling of progress.&lt;br&gt;
The task is done, but learning did not happen.&lt;/p&gt;

&lt;p&gt;Imagine a ship sailing without a compass or a map. It moves… but it does not know where it is going.&lt;br&gt;
Coding without understanding is exactly that: producing code without knowing the reasons behind the decisions.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;It is not bad code.&lt;/em&gt;&lt;br&gt;
&lt;em&gt;It is ownerless code.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;AI does not &lt;strong&gt;create&lt;/strong&gt; this problem.&lt;br&gt;
It &lt;strong&gt;amplifies&lt;/strong&gt; it.&lt;/p&gt;

&lt;p&gt;With a single prompt, code appears and the task seems solved. This creates a sense of efficiency, but also shallow learning.&lt;br&gt;
If AI solves it fast, who is learning?&lt;/p&gt;

&lt;p&gt;How can you know if you are building comprehension debt?&lt;/p&gt;

&lt;p&gt;Some simple questions help:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Could you implement this feature without using prompts?&lt;/li&gt;
&lt;li&gt;Can you explain what the generated code is doing?&lt;/li&gt;
&lt;li&gt;Are you afraid to change the code without asking AI?&lt;/li&gt;
&lt;li&gt;Do you only test on localhost and hope it works?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you cannot explain the solution, you probably did not understand the problem.&lt;br&gt;
And if you only test and hope… something is wrong.&lt;/p&gt;

&lt;p&gt;The cost appears over time.&lt;br&gt;
You grow slower, become dependent on tools, and weaken your foundation as a developer.&lt;br&gt;
In a job interview, without AI, how would you perform?&lt;/p&gt;

&lt;p&gt;You trade speed today for stagnation tomorrow.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The goal is not to stop using AI.&lt;/em&gt;&lt;br&gt;
&lt;em&gt;It is to change how you use it.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Practical advice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Ask the AI to explain the code it generated.&lt;/li&gt;
&lt;li&gt;Rewrite the solution in your own words.&lt;/li&gt;
&lt;li&gt;Run the program flow in your head.&lt;/li&gt;
&lt;li&gt;Make small changes and see what happens.&lt;/li&gt;
&lt;li&gt;Try to write first and ask for AI review later.&lt;/li&gt;
&lt;li&gt;Use AI as a copilot, not as the driver.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;As a helper tool, it is amazing.&lt;br&gt;
But only if you are navigating with a map and a compass not just following the current.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>beginners</category>
      <category>learning</category>
      <category>career</category>
    </item>
    <item>
      <title>Microfrontends: A Solution or Just Distributed Complexity?</title>
      <dc:creator>Daniel Cerverizzo</dc:creator>
      <pubDate>Fri, 30 Jan 2026 21:58:56 +0000</pubDate>
      <link>https://dev.to/dcerverizzo/strategies-and-best-practices-successfully-implementing-microfrontends-in-your-projects-5anc</link>
      <guid>https://dev.to/dcerverizzo/strategies-and-best-practices-successfully-implementing-microfrontends-in-your-projects-5anc</guid>
      <description>&lt;p&gt;We live in a time where applications keep growing, teams keep getting smaller, and deployment is still slow.&lt;br&gt;
In this context, microfrontends seem like an obvious answer.&lt;br&gt;
But if an AI can generate an architecture in seconds, who takes responsibility for the consequences of that choice?&lt;/p&gt;

&lt;p&gt;The problem is not generating code.&lt;br&gt;
The problem is sustaining decisions.&lt;/p&gt;

&lt;p&gt;Microfrontends are a good solution in many contexts. The real question is: do they make sense in yours?&lt;br&gt;
It is still common to treat this architecture as a silver bullet, when in some scenarios it is just complexity disguised as overengineering. That is why analyzing your ecosystem and its trade-offs is more important than following the latest trend.&lt;/p&gt;

&lt;p&gt;Microfrontends promise autonomy. But when implemented without care, they often multiply problems instead of solving them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What are microfrontends?
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F7y44m40lw5j3knsb8zoe.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F7y44m40lw5j3knsb8zoe.png" alt="Design in excalidraw what are microfrontends one big large square inside there's a square as shell container and every part of the system like dashboard is separete in square and one pipeline" width="800" height="518"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In simple terms, microfrontends apply the same idea of microservices to the frontend: splitting the system into independent parts, each with a clear responsibility.&lt;/p&gt;

&lt;p&gt;Instead of one large frontend, the product is composed of domains. One project handles the dashboard, another handles forms, another handles checkout. Each team becomes the owner of a part of the product not just a screen.&lt;/p&gt;

&lt;p&gt;This separation does not necessarily require different technologies. In most cases, teams use the same stack. What really changes is the boundary of responsibility and the level of isolation between parts.&lt;/p&gt;

&lt;h2&gt;
  
  
  When does it make sense to use microfrontends?
&lt;/h2&gt;

&lt;p&gt;Make sense when the problem is no longer only technical, but also organizational.&lt;/p&gt;

&lt;p&gt;They usually work better when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;the team is large&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;product domains are well defined&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;there is a real deployment bottleneck&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;team dependencies are slowing down delivery&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In this scenario, splitting the frontend by domain allows each part to evolve more independently. Teams gain autonomy and delivery becomes more predictable.&lt;/p&gt;

&lt;h2&gt;
  
  
  When it does NOT make sense
&lt;/h2&gt;

&lt;p&gt;Microfrontends are not a good choice when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;the application is small&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;there is only one team&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;the main problem is messy code&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In these cases, a simpler architecture often solves the problem with less cost.&lt;br&gt;
Before breaking the frontend into pieces, it is worth asking if the problem is really scale or just internal organization.&lt;/p&gt;

&lt;h2&gt;
  
  
  Technical strategies and risks
&lt;/h2&gt;

&lt;p&gt;There are several approaches, and everything depends on scope and time. On the frontend, microfrontends affect both build time and runtime integration, especially in systems that handle a lot of data.&lt;/p&gt;

&lt;p&gt;Module Federation, for example, allows sharing code as if it were an external library. This makes integration easier, but creates runtime dependency. The problem moves from “build is broken” to “version mismatch in production”.&lt;/p&gt;

&lt;p&gt;Web Components help standardize behavior and reuse components across projects. Without this, the system becomes a group of different applications trying to look like a single product.&lt;/p&gt;

&lt;p&gt;The main risks appear quickly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;duplicated dependencies&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;incompatible versions&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;higher observability cost&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;harder debugging&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;The more applications you have, the more discipline you need.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where microfrontends work (and where they break)
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;Without contracts, there is no independence. There is chaos.&lt;/em&gt;&lt;br&gt;
Microfrontends only work when there is a clear agreement between parts: what each domain does, what it exposes, and what it consumes.&lt;/p&gt;

&lt;p&gt;The problem is rarely &lt;strong&gt;technical&lt;/strong&gt; first. It is &lt;strong&gt;organizational&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Microfrontends help when the main issue is coordination between teams. If the problem is only technical, a good refactor may be more effective.&lt;/p&gt;

&lt;p&gt;Standards are no longer optional. Design systems, versioning, and integration rules are not bureaucracy they prevent autonomy from becoming fragmentation.&lt;/p&gt;

&lt;p&gt;A warning sign is version divergence between projects.&lt;br&gt;
For example:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ftxew9dhovr5d2h5nwdb6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ftxew9dhovr5d2h5nwdb6.png" alt="3 squares one represents checkout and other represents dashboard with version react 18 and 15 and one represents returns in error of bundle and version" width="800" height="749"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Checkout: React 18&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Dashboard: React 15&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Does it work? Maybe.&lt;br&gt;
Is it healthy? Probably not.&lt;/p&gt;

&lt;h2&gt;
  
  
  A real scenario
&lt;/h2&gt;

&lt;p&gt;An e-commerce company started with a single frontend responsible for everything: registration, user profile, checkout, and admin panel.&lt;/p&gt;

&lt;p&gt;As the product grew, problems appeared:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;more frequent deployments&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;code conflicts&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;slow reviews&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;delayed releases&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The problem was not &lt;strong&gt;technical&lt;/strong&gt;. It was &lt;strong&gt;organizational&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The solution was to split the frontend by domain:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;checkout&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;user profile&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;admin panel&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;reports&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each project got its own repository and deployment pipeline. For the user, nothing changed. For the teams, everything changed.&lt;/p&gt;

&lt;p&gt;The cost came later: more standards, stronger dependency control, and more concern about observability. Complexity did not disappear it was redistributed.&lt;/p&gt;

&lt;p&gt;In the end, microfrontends are a valid solution in many contexts. But the &lt;em&gt;trade-offs&lt;/em&gt; must be on the table.&lt;/p&gt;

&lt;p&gt;Before adopting them, it is always worth asking:&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Are you using microfrontends to scale your product…&lt;/em&gt;&lt;br&gt;
&lt;em&gt;or to hide technical debt?&lt;/em&gt;&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>frontend</category>
      <category>softwareengineering</category>
      <category>beginners</category>
    </item>
    <item>
      <title>Unlocking the World of International Job Listings: A Node.js and Puppeteer Web Scraping Project🚀</title>
      <dc:creator>Daniel Cerverizzo</dc:creator>
      <pubDate>Mon, 06 Nov 2023 13:32:17 +0000</pubDate>
      <link>https://dev.to/dcerverizzo/unlocking-the-world-of-international-job-listings-a-nodejs-and-puppeteer-web-scraping-project-297m</link>
      <guid>https://dev.to/dcerverizzo/unlocking-the-world-of-international-job-listings-a-nodejs-and-puppeteer-web-scraping-project-297m</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;Job hunting can be a daunting task, especially with countless job platforms offering excellent opportunities. Faced with this challenge, I decided to streamline my job search by consolidating the most frequently visited websites into a single, accessible resource.&lt;/p&gt;

&lt;p&gt;But how did I go about it? My solution was to create a web scraper using cutting-edge technologies such as Puppeteer, Node.js, and MongoDB. This blog post takes you on a journey through the structure and development of this simple yet powerful project.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Quest Begins
&lt;/h2&gt;

&lt;p&gt;The first step in my mission to simplify the job search process was to leverage web scraping. Web scraping allowed me to extract data from multiple job websites, collate it, and present it in a user-friendly format.&lt;/p&gt;

&lt;p&gt;For this, I chose Puppeteer, a headless Chrome browser, and Node.js, a powerful JavaScript runtime. These technologies worked in tandem to retrieve job listings and relevant details. With the data collected, I stored it efficiently using MongoDB, a document-based NoSQL database.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Building Blocks
&lt;/h2&gt;

&lt;p&gt;To commence my project, I initiated the process of creating a web scraper using Puppeteer. This technology granted me access to web pages, from where I could extract crucial job listing data.&lt;/p&gt;

&lt;p&gt;Node.js played a vital role in orchestrating this process. By utilizing JavaScript, I could craft functions to navigate web pages, retrieve job descriptions, and compile the data into structured information.&lt;/p&gt;

&lt;h2&gt;
  
  
  Storing Data for Easy Access
&lt;/h2&gt;

&lt;p&gt;MongoDB, known for its flexibility in handling unstructured data, proved invaluable. It served as the perfect repository for the job listings gathered from web scraping.&lt;/p&gt;

&lt;p&gt;The NoSQL database stored each job listing as a document, making it easier to organize, retrieve, and display the data in a user-friendly manner.&lt;/p&gt;

&lt;h2&gt;
  
  
  Project structure
&lt;/h2&gt;

&lt;p&gt;The project's root directory is where all your project files and subdirectories reside.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;src/&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;src/&lt;/code&gt; directory contains the source code of your project.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;scripts/&lt;/code&gt;: This directory houses the core logic of your web scraping and database operations.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;scraper.js&lt;/code&gt;: The main script for web scraping using Puppeteer and Node.js.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;database.js&lt;/code&gt;: Script for handling MongoDB database operations.&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;code&gt;server.js&lt;/code&gt;: Your main Node.js application file to serve the scraped data to a frontend.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;models/&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;models/&lt;/code&gt; directory contains the data models, schemas, or structures for your project.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;jobSchema.js&lt;/code&gt;: Defines the schema for job listings to be stored in your MongoDB database.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;utils/&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;utils/&lt;/code&gt; directory contains utility files, configurations, and other miscellaneous scripts.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;code&gt;sites.js&lt;/code&gt;: A configuration file listing the websites to scrape, including selectors for job details.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;code&gt;config.js&lt;/code&gt;: Configuration settings for your database connection.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;node_modules/&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;This directory contains the Node.js modules and packages that your project depends on. You don't need to manage this directory manually.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;.gitignore&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;.gitignore&lt;/code&gt; file specifies which files or directories should be ignored when you push your project to a version control system like Git. Commonly, it includes the &lt;code&gt;node_modules/&lt;/code&gt; directory.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;package.json&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;package.json&lt;/code&gt; file lists project metadata and dependencies. It's also where you specify your project's main entry point and various scripts.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;README.md&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;The README file provides essential information about your project, including how to set it up, run it, and any other necessary documentation.&lt;/p&gt;

&lt;p&gt;This structured approach keeps your project organized, making it easier to manage and collaborate with others. The main logic for web scraping and database operations is separated, ensuring a clean and maintainable codebase. You can customize this structure based on your specific project needs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The Scraper Class
&lt;/h3&gt;

&lt;p&gt;Our scraper will be encapsulated in a class for modularity and maintainability. Here's what it looks like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Import necessary modules and configuration&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;puppeteer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;puppeteer&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;sites&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;../utils/sites&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;database&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;../utils/config&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nx"&gt;Scraper&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="nx"&gt;scrapeData&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;site&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// Create a headless Chromium browser using Puppeteer&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;puppeteer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;launch&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;newPage&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

    &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="c1"&gt;// Navigate to the specified website&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;goto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;site&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;600000&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;

      &lt;span class="c1"&gt;// Select all job listings on the page using a provided selector&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;jobList&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;$$&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;site&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;selectors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;list&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;jobData&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;

      &lt;span class="c1"&gt;// Loop through the job listings and extract relevant information&lt;/span&gt;
      &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;job&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;jobList&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;title&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;job&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;$eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;site&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;selectors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;element&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;element&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;textContent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;company&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;job&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;$eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;site&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;selectors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;company&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;element&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;element&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;textContent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;location&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;job&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;$eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;site&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;selectors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;location&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;element&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;element&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;textContent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;link&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;job&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;$eval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;site&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;selectors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;link&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;element&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;element&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;href&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nx"&gt;jobData&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;push&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;company&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;location&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;link&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;

      &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;jobData&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;finally&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="c1"&gt;// Close the browser after scraping is complete&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;close&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="nx"&gt;init&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="c1"&gt;// Connect to the MongoDB database&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;database&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
      &lt;span class="c1"&gt;// Clear existing data in the database&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;database&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;clearData&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
      &lt;span class="c1"&gt;// Scrape data from multiple sites and store it in MongoDB&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;scrapedData&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sites&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;site&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;scrapeData&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;site&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
      &lt;span class="c1"&gt;// Save scraped data to MongoDB&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;database&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;saveDataToMongoDB&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;scrapedData&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Error in app:&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;finally&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Finish!&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;module&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;exports&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;Scraper&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Configuration for websites to scrape&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;sites&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Remotive&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://remotive.com/remote-jobs/software-dev&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;selectors&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;list&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;li.tw-cursor-pointer&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;a.tw-block &amp;gt; span&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;company&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;span.tw-block&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;location&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;span.job-tile-location&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;link&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;a.tw-block&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="nx"&gt;module&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;exports&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;sites&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;The fusion of Puppeteer, Node.js, and MongoDB created a comprehensive solution to simplify job searches. With this project, I centralized data from various websites, making it easier for jobseekers to access the most relevant listings. By sharing this experience, I hope to inspire others to embark on similar projects, harnessing the power of web scraping and innovative technologies. The journey to streamline your job search begins here!&lt;/p&gt;

&lt;p&gt;You can access this project online here: &lt;/p&gt;

&lt;p&gt;&lt;a href="https://jobs-one-drab.vercel.app/"&gt;https://jobs-one-drab.vercel.app/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/Dcerverizzo/web-scraping-jobs"&gt;https://github.com/Dcerverizzo/web-scraping-jobs&lt;/a&gt;&lt;/p&gt;

</description>
      <category>node</category>
      <category>puppeteer</category>
      <category>scraper</category>
      <category>beginners</category>
    </item>
  </channel>
</rss>
