DEV Community: Mattia Zanella

Batch with Node.js

Mattia Zanella — Tue, 27 Apr 2021 13:08:56 +0000

Premise

Let's say that your aunt comes to you asking a favor.

She has this pen drive with A LOT of folders containing mp3s and by what she believes her car cannot read recursively all the folders so she needs one root folder containing all the mp3s.

The USB drive is structured as below:

songs |
      folder_1 |
               foo.mp3
               another-foo.mp3
      folder_2 |
               bar.mp3
               another-bar.mp3
...

And we need to reach this result:

output |
       foo.mp3
       another-foo.mp3
       bar.mp3
       another-bar.mp3
...

Given this overview, you have two option to do this:

Do it manually.
Leave a like ❤️ to this post.

To start we're gonna create a folder, let's call it batch-node.
Inside that folder I've copied the songs folder (above) and I've created an index.js file.
So this is now my batch-node structure:

batch-node |
           index.js
           songs/
...

For this purpose we'll use fs-extra since copy/paste seems like not supported by fs API.

Just add it:

yarn add fs-extra

Let's starting coding.

// index.js

/**
 * fs-extra adds file system methods that aren't included in the native
 * fs module and adds promise support to the fs methods.
 */
const fs = require("fs-extra");

const MP3_DIRECTORY = './songs'
const OUTPUT = './output'

Be carefull, do not create output/ folder since fs-extra will handle this for us otherwise it won't work.

The script is composed by 2 main functions:

a readDirectory to, of course, read a folder and its content.
and a copyToDestination function to copy our song to the folder destination.

Let's add these functions to our index.js:

const copyToDestination = ({ path, file }) => {
    try {
        const fileFormat = path.split('.').pop()
        const isMP3 = fileFormat === 'mp3'
        if (!isMP3) return

        // @see https://stackoverflow.com/a/40019824
        fs.copySync(path, `${OUTPUT}/${file}`)
        console.log(`${ file } has been copied to ${ OUTPUT }`)
    } catch(error) {
        console.error({ error })
    }
}

const readDirectory = dir_name => {
    fs.readdir(dir_name,
        { withFileTypes: false },
        (error, files) => {
            try {
                files.forEach(file => {
                    const path = `${dir_name}/${file}`
                    // @see https://nodejs.org/docs/latest/api/fs.html#fs_stats_isdirectory
                    const isDir = fs.lstatSync(path).isDirectory()
                    const config = { file, path }

                    return isDir ? readDirectory(path) : copyToDestination(config)
                })
            } catch (error) {
                console.error({ error });
            }
        }
    )
}

Pay attention to these lines:

     const fileFormat = path.split('.').pop()
     const isMP3 = fileFormat === 'mp3'
     if (!isMP3) return

Since we have different types of files inside folders (like covers album, ini files and many more) we're returning the function to totally ignore these kinds of file.

Finally we just have to lunch our function at the end of our index.js:

readDirectory(MP3_DIRECTORY)

This should avoid you some boring manual stuff 🙃

Automazione Iliad con Node.js e puppeteer.

Mattia Zanella — Tue, 10 Nov 2020 10:10:26 +0000

TL;DR

Quello che andremo a creare in questo articolo è più o meno questo:

Ho scritto più o meno perché il video è puramente esemplificativo ed esula le seguenti ragioni:

Sarà un cron-job, quindi non una cosa da lanciare manualmente ogni volta.
Solo se l'utilizzo corrente è maggiore a una X% (ora impostata a 70) allora si riceve la mail.

Intro

Una delle cose che non ho mai digerito della mia giovinezza è stata quella di aver speso delle cospique decine di euro per acquistare un Raspberry Pi 3 e averlo usato principalmente per raccogliere la polvere.

Non tutti i mali vengono per nuocere, dicono, e in questo periodo di pandemia spesso mi sono ritrovato a lavorare da remoto facendo affidamento solo ed esclusivamente al tethering del mio telefono.
Ed è qui che nasce l'esigenza di tenere monitorati i consumi effettivi del mio piano Ilaid.

Piccolo disclaimer: la mia è la 1° offerta uscita di Iliad e beneficio di un tetto massimo di 30GB al mese. Se sei anche tu un utente Iliad probabilmente avrai quella più recente da 50.

Come puoi vedere dall'immagine in copertina, Iliad offre un servizio analogo ma solo quando arrivi al 100% di quei 30 o 50GB. Servizio utile come...lascio a te il sillogismo migliore.

Requisiti per lo script

Node.js (versioni LTS consigliate) ma forse dovrei dire NVM
Yarn
Password e nome utente del portale Iliad
Un account email richiesto per poter inviare le email. Account richiesto da nodeMailer

Percisazione prima di iniziare:

Il mio target device sarà un raspberry ma può essere tranquillamente il tuo pc.

Let's rock 🤘

Da terminale:

mkdir iliad-script && cd $_
yarn init -y

// Dependencies required:
yarn add -D dotenv node-cron nodemailer puppeteer

Una volta installate le dipendenze, il 1° step da fare è quello di creare un file .env all'interno del repo poc'anzi creato.
E' un file essenziale e deve avere le seguenti variabili:

# .env
PUPPETEER_PRODUCT=chrome

NODE_ENV=development

ILIAD_LOGIN_PAGE=https://www.iliad.it/account/
ILIAD_USER=YOUR_AMAZING_USERNAME
ILIAD_PASS=YOUR_AMAZING_PASS

# ATTENZIONE: Mail DA dove si inviano le notifiche 👇 (vedi requisiti)
# Nel mio caso è NETSONS
NODEMAILER_HOST=srv-hp4.netsons.net
NODEMAILER_MAIL=YOUR_AMAZING_EMAIL
NODEMAILER_PASS=YOUR_AMAZING_EMAIL_PASS

# ATTENZIONE: Mail dove si riceve le notifiche 👇
EMAIL_RECIPIENT=my_amazing_email@icloud.com

PERCENTACE_GUARD_DEVELOPMENT=5
PERCENTACE_GUARD_PRODUCTION=70

Il 2° step è quello di aggiungere gli script di yarn al package.json

  "scripts" :{
    "start": "node index.js",
    "dev": "node index.js"
  },

A questo punto possiamo iniziare a scrivere il pezzo forte dello script e creiamo un file index.js

require("dotenv").config()

const puppeteer = require("puppeteer")
const mailManager = require('./mailManager.js')

const iliadScraper = async () => {
  const browser = await puppeteer.launch()
  const page = await browser.newPage()

  try {
    await page.goto(process.env.ILIAD_LOGIN_PAGE)

    await page.type("input[type=text]", process.env.ILIAD_USER)
    await page.type("input[type=password]", process.env.ILIAD_PASS)
    await page.click("button[type=submit]")

    await page.waitForNavigation({ waitUntil: "networkidle0" })

    const fatherSelector = ".conso-infos.conso-local"
    await page.waitForSelector(fatherSelector)

    const gridChildValues = await page.$$(
      `${fatherSelector} .conso__content .conso__text`
    )

    const chain = gridChildValues.map((child) => {
      return page.evaluate((element) => element.textContent, child)
    })
    const res = await Promise.all(chain)
    console.log(res)
  } catch (error) {
    console.error(`ERROR::${JSON.stringify(error, null, 2)}`)
    browser.close()
  }
}

iliadScraper()

Trovo verboso dover spiegare ogni metodo di puppeteer quando è già presente una documentazione abbastanza esaustiva.
In sonstanza le operazioni sono:

Avvio di chrome in headless-mode (default è true).
Indirizzo chrome alla pagina di login di Iliad (definito in .env file).
Inserimento di user e pass del profilo Iliad (definito in .env file).
Click del bottone di login.
Attesa della dashboard.

A questo punto mi soffermerei sul log della costante res:

[
  '\n' +
    '                     Chiamate: 1m 34s\n' +
    '                     Consumi voce: 0.00€\n' +
    '                  ',
  '1 SMS\n                     SMS extra: 0.00€\n                  ',
  '\n' +
    '                     64,81mb / 30GB\n' +
    '                     Consumi Dati: 0.00€\n' +
    '                  ',
  '\n' +
    '                     0 MMS\n' +
    '                     Consumi MMS: 0.00€\n' +
    '                  '
]

L'array è la seguente rappresentazione dei box evidenziati:

A questo punto mi sono creato una funzione di cleaning in un file apposito utils/clean.js:

/**
 * @description Since ILIAD's DOM has no ID or specific class for the consume
 * We have to parse all the grid 2x2.
 * @param {Array} arrayResponses
 * @returns {Array}
 */
const clean = (arrayResponses) =>
  arrayResponses.map((response) =>
    response
      .split("\n")
      .map((s) => s.trim())
      .filter(Boolean)
  );

module.exports = clean;

Che successivamente importo in index.js e lo uso per avere una risposta più esaustiva:


...
// Custom utils.
const clean = require('./utils/clean.js')
...
    const res = await Promise.all(chain)
    const cleanedRes = clean(res)
...

A questo punto cleanedRes diventa nel seguente modo:

[
  [ 'Chiamate: 1m 34s', 'Consumi voce: 0.00€' ],
  [ '1 SMS', 'SMS extra: 0.00€' ],
  [ '64,81mb / 30GB', 'Consumi Dati: 0.00€' ],
  [ '0 MMS', 'Consumi MMS: 0.00€' ]
]

Sappiamo che il figlio desiderto per calcolare il consumo corrente è il 3°, quindi:

const extract = require('./utils/extract.js')
...
    // We know that consume is the third child.
    const splittedConsumeValues = cleanedRes[2][0].split("/").map(val => val.trim())
    const consume = {
      current: extract(splittedConsumeValues[0]),
      max: extract(splittedConsumeValues[1])
    }
    console.log(consume)
...

È l'occasione per creare una 2° utility utils/extract.js

  /**
   * @description Given a plain text from DOM, extracting the number.
   * @param {String} str Like 2,64GB or plain 20GB.
   * Attention! can be also something like...200mb, 402.34mb
   * @returns {Number} Pay attention to comma vs dot.
   */
  const extractValue = str => {
    const unit = str.match(/[a-z]+/ig).join()
    const digits = str.match(/\d+/ig).join('.')

    return unit === 'mb'
      ? (parseFloat(digits) / 1000)
      : parseFloat(digits)
  }

  module.exports = extractValue

Che permette di estrarre consumo corrente e il tetto massimo dell'offerta (inseriti poi nell'oggetto consume).
L'oggetto consume ora è:

{ current: 0.06481, max: 30 }

Calcoliamo ora il consumo effettivo in %

...
    const currentProgress = Math.round(consume.current / consume.max * 100)
...

A questo punto abbiamo 0 come consumo in %.
Da qui in poi aggiungo una 3° utility per prevenire l'invio della mail se siamo sotto a una determinata % (definito in .env file).

// utils/mustKillFlow.js

  /**
   * @description This is a utility guard to kill the MAIL flow in specific cases.
   * Pay attentio to PERCENTACE_GUARD, it's a default value fixed to 70.
   * By doing so we're sending email-alert ONLY if the user is over 70% of his GIGA.
   * @param {Number | any} percentage SHOULD be a number of current consume.
   * @returns {Boolean}
   */
  const mustKillFlow = percentage => {
    const alertLimit = process.env.NODE_ENV === 'development'
      ? process.env.PERCENTACE_GUARD_DEVELOPMENT
      : process.env.PERCENTACE_GUARD_PRODUCTION

    return !percentage || isNaN(percentage) || (percentage <= alertLimit)
  }

  module.exports = mustKillFlow

e in index.js avrò dunque:

const mustKillFlow = require('./utils/mustKillFlow.js')
...
    const currentProgress = Math.round(consume.current / consume.max * 100)
    if ( mustKillFlow(currentProgress) ) return browser.close()

    const mailOptions = {
      from: process.env.NODEMAILER_MAIL,
      to: process.env.EMAIL_RECIPIENT,
      subject: 'Report from iliad cron-job ⏰.',
      html: `<p>📵 Your're at more or less ${currentProgress}% of your GIGA. 📵</p>`
    }

    mailManager.sendEmail(mailOptions).catch(console.error)
    browser.close()
} catch(error) {
...

Rimane solo da definire il file:

// mailManager.js, è in root.
const nodemailer = require("nodemailer");

module.exports = {
    async sendEmail(mailOptions) {
        // @see https://ethereal.email/
        const transporter = nodemailer.createTransport({
            host: process.env.NODEMAILER_HOST,
            port: 465,
            secure: true, // true for 465, false for other ports
            auth: {
                user: process.env.NODEMAILER_MAIL,
                pass: process.env.NODEMAILER_PASS
            }
        });

        const sent = await transporter.sendMail(mailOptions);

        console.log("Message sent: %s", sent.messageId);
        // Message sent: <b658f8ca-6296-ccf4-8306-87d57a0b4321@example.com>
    }    
}

Bene, per finalità di utilizzo cambiamo l'ultima parte di index.js da

...
iliadScraper()
...

...
// It will run at 1 A.M and 1 P.M
const timing = process.env.NODE_ENV === 'development' ? '* * * * *' : '00 01,13 * * *'
cron.schedule(timing, iliadScraper)

E nel file .env cambiamo la variabile

NODE_ENV=production

Perfetto, il tutorial è concluso.
Fintanto che terrete in vita il processo, il cron continuerà a girare 2 volte al giorno e verificherà che la vostra soglia non superi il limite impostato.

Create NuxtJS component for Newsletter subscription with Wordpress.

Mattia Zanella — Tue, 28 Jul 2020 09:45:40 +0000

I was wondering if it's possible to create a component for NuxtJS that allows subscribing to the costumers' newsletter
A component like this one:

So I started to surf the internet and surprisely I found a great WP plugin which can deal with REST API.

Requirements

Create an account.
Getting Started by installing base plugin.
Paste your GLOBAL LICENSE CODE on your Wordpress » Newsletter » Settings » License Key field.
By default the 2^ step is not enough if you want to add subscribers via REST API. So you have also to install the free Newsletter API addon (find it on Newsletter addons panel in your blog or get it on your account page - 1^ step). This last step is going to add an "API" item under Newsletter inside WP sidebar.

Now we're ready to create our component.
I'm gonna assume that you already have a project with NuxtJS installed.

Let's code

Template:

<div>
  <h3 class="has-text-centered">Subscribe to our newsletter</h3>
  <div class="control has-icons-left has-icons-right has-text-centered">
    <VZSpinnerPlaceholders
      class="justify-center"
      v-if="isLoading"
      :contrast="true" />
    <div v-else class="input-container is-relative">
      <input
        v-model.trim="email"
        class="input is-large"
        type="email"
        placeholder="Email">
      <span class="add pointer" @click="addSubscription" v-if="isValidEmail"></span>

      <div v-if="result.show" class="result">
        <p> {{ result.message }} </p>
      </div>
    </div>
  </div>
</div>

<VZSpinnerPlaceholders /> it's just a simple component that I created to show a funny spinner.
Since I'm using Bulma's module with NuxtJS you can see the component's output here (Loading).

Logic:

import newsletterManager from "@/mixins/newsletterManager";
...
mixins: [newsletterManager],
data: () => ({
  email: null,
  isLoading: false,
  result: { show: false, message: null }
}),
computed: {
  isValidEmail() {
    const rule = /^(([^<>()\[\]\.,;:\s@\"]+(\.[^<>()\[\]\.,;:\s@\"]+)*)|(\".+\"))@(([^<>()[\]\.,;:\s@\"]+\.)+[^<>()[\]\.,;:\s@\"]{2,})$/i;
    return Boolean(this.email && rule.test(this.email))
  },
  payload() {
    return {
      email: this.email,
      name: 'Client from nuxt',
      gender:'--'
    }
  }
},
methods: {
  async addSubscription() {
    if(!this.isValidEmail) return

    this.isLoading = true
    const subscriptionResponse = await this.MX_NL_addSubscriber(this.payload)
    this.responseHandler(subscriptionResponse)
  },
  responseHandler({ response, isAxiosError }) {
    const isString = s => (typeof s === 'string' || s instanceof String)
    let message

    if(isAxiosError) {
      const text = isString(response.data) ? response.data : response.data.message
      message = `${ text } ❌`
    } else {
      message = 'Subscription completed! ✅'
    }

    this.isLoading = false
    this.result = { show: true, message }
    this.email = null // Flush email
  }
}
...

As you can see, I like using mixins to separate the logic like making the POST to add a subscriber.
This is where MX_NL_addSubscriber comes in:

// mixins/newsletterManager.js
import ENVs from "@/constants/environments";

export default {
  methods: {
    async MX_NL_addSubscriber(payload) {
      try {
        const response = await this.$axios.$post(
          `${ ENVs.getDomainPath() }/wp-json/newsletter/v1/subscribe`,
          payload
        )

        return response
      } catch(error) { // Let's handle it.
        return error
      }
    }
  }
}

I'm not using any secret key for the API since the documentation is quite clear:

Subscribe
When called as REST API it can be invoked without the secret key: in this case more strict contraints are applied to the parameters (like for subscriptions coming from a public subscription form).

So, as you can see in the documentation, it's just a simple CURL:

$ curl -X POST -H "Content-Type: application/json" http://<your-site>/wp-json/newsletter/v1/subscribe
-d '{"email":"test@example.com", "name":"My name", "gender":"f"}'

So this.payload is just an object with those keys.

Result

And this is my wordpress-back office side:

Credits

Once again thank you guys ♥️
// Detect dark theme var iframe = document.getElementById('tweet-1050813729951092739-546'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1050813729951092739&theme=dark" }

My first lambda function with dev.to API

Mattia Zanella — Fri, 08 May 2020 12:42:17 +0000

I'm gonna start this article by mentioning this great article by

Dani VijayFollow

So this example will be more or less the same but, instead return a simple Yey! string, we will fetch DEV's API to get my personal data.

DEV's endpoints are listed here.
I'm gonna call my file as DEV_profile.js and inside I'll put this:

const https = require("https");

const options = {
  hostname: 'dev.to',
  path: '/api/users/me',
  headers: { 'api-key': 'YOUR_AMAZING_TOKEN' }
}

const getProfile = (event, context, callback) => {
  // Just a guard to prevent other verbs
  if (event.httpMethod !== 'GET') return

  const req = https.request(options, res => {
    let content = ''
    res.setEncoding("utf8");

    res.on('data', data => (content += data));

    res.on('end',
      body => callback(null, { statusCode: 200, body: content })
    );
  })

  req.on('error', error => console.error(error))
  req.end();
};

module.exports = { handler: getProfile }

Disclaimer: I'm not a backend developer so I'm not sure what I'm doing 🤪 but to handle CORS issues I'm gonna adding some lines to end of the file:

const middy = require('@middy/core')
const cors = require('@middy/http-cors')

/**
 * Let's "middyfy" our handler, then we will be able to attach middlewares to it
 * and Adds CORS headers to responses
 */
const handler = middy(getProfile).use(cors())

module.exports = { handler }

Following this article helped me a lot during development.

Finally you can watch the result here.

That's all 👋 👋

Nuxt.js - Dynamic layouts by current route.

Mattia Zanella — Tue, 14 Apr 2020 15:52:36 +0000

UPDATE 07/2020

I've updated the snippets below since in specific edge-cases when refreshing the page the middleware didn't work.

Given Nuxt documentation it seems like that it is not so easy for us to have different layouts by dynamic routes.

So I surfed the web and I've found a similar issue and, by the end of the day, this is my solution.

This is my structure:

pages/
   _page.vue
   index.vue

The solution, already mentioned in the issue, is to create a middleware under middleware folder like this one:

/** 
 * @description myMiddleware.js
 * @see https://glitch.com/edit/#!/nuxt-dynamic-layouts?path=middleware/mobile.js:5:1
 */
export default context => {
  const { route: { params } } = context

  switch (params.page) {
    case 'tmp1':
    case 'tmp2':
        return 'tmpLayout'

      default:
        return 'default'
  }
}

And inside _page.vue simply use this line:

  import myMiddleware from '@/middleware/myMiddleware'
  ...
  layout: myMiddleware,

So whenever you navigate through mysite.com/tmp1 or mysite.com/tmp2 our tmpLayout will always used.

UPDATE 07/2020

I've updated the snippets above since in specific edge-cases when refreshing the page the middleware didn't work.