DEV Community: Ulises Gascón

How does the Official Node.js News Feeder work?

Ulises Gascón — Wed, 05 Jul 2023 22:00:28 +0000

Node.js has a new RSS feed that consolidate all the releases and news from the different teams, working groups and projects inside the org.

The Challenge

Node.js as an organization has too many things ongoing all the time. There are many projects, teams, and working groups working on different things. It is hard to keep track of all the things that are happening, so there is a recurrent need for the community to find a better way to be aware of what is going on. This discussion has been going on for a while, and there are many ideas on how to solve this problem, but we decided that RSS is a good way to start as this will also help to promote our activities and achievements outside the Node.js org itself.

Requirements

The teams and working groups should be able to add their own news without having to change their way of working (no PRs, forms...)
The information should be available in a valid RSS feed.
The feed should be updated automatically, but allow for manual news additions and easy content curation.

Decisions made

Use GitHub as the source of truth for the news, so we will use the GitHub API to fetch the relevant information from issues, discussions, releases...
Use a GitHub Action to generate the RSS feed and publish it to GitHub Pages.
Use a GitHub Action to update the feed automatically every week or manually when needed, generating a new commit with the changes in a PR that will be reviewed and curated by the team.
Avoid external dependencies as much as possible, so the solution should be self-contained and easy to maintain.

The Solution

The full source code can be found in this repository. I will explain the most relevant parts of the solution here.

The Architecture

In general terms, the solution is composed of the following parts:

Community

The community is the source of the news. They are the ones who reply to specific issues or discussions related to the news feed, as well as manage the new releases.

Curators

The curators are the ones who review the changes and merge the PRs that update the feed. The feed is automatically updated every week, but it can also be updated manually when needed. There are several scripts in order to collect, process, validate, and publish the feed.

Readers

The readers are the ones who consume the feed. They can be humans or bots. The readers can subscribe to the feed using the following URL: https://nodejs.github.io/nodejs-news-feeder/feed.xml. We provide a Slack channel where the feed is automatically published, so the community can be aware of the latest news.

The Structure

Configuration

There is a config.json file that stores all the references to the external resources (discussions, issues, releases...), API rate limits, and the configuration of the last execution time (lastCheckTimestamp).

The last execution time (lastCheckTimestamp) will prevent us from including already processed information in the feed. This prevents us from using third-party software or reconciling the feed to avoid duplications.

{
  "lastCheckTimestamp": 1688584036809,
  "reposPaginationLimit": 250,
  "releasePaginationLimit": 10,
  "commentsPaginationLimit": 100,
  "breakDelimiter": "</image>",
  "discussionsInScope": [],
  "issuesInScope": []
}

Modularity

The solution is divided into different scripts that do different things, which allows us to reuse the code and make it easier to maintain.

This structure is clearer by checking the package.json.

{
    "scripts": {
        "collect:releases": "node scripts/collect-releases.js",
        "collect:issues": "node scripts/collect-issues.js",
        "collect:discussions": "node scripts/collect-discussions.js",
        "rss:validate": "node scripts/validate.js",
        "rss:build": "node scripts/build.js",
        "rss:format": "node scripts/format.js",
        "rss:format-check": "node scripts/format-check.js"
    }
}

Fetching content from Github

Releases

Node.js uses Github Releases to publish new versions of different projects. There are many projects in the organization, and we keep adding more on a regular basis.

So, this script will do the following:

Fetch all the repositories in the organization.
Fetch the latest releases for each repository.
Filter the releases by the ones that are newer than the last execution time (lastCheckTimestamp).
Format the releases to be included in the feed.
Add the releases to the feed.

Issues

Each project is publishing its news in GitHub Issues as responses.

So, this script:

Fetches all the comments in the issues that are in scope.
Filters the comments by the ones that are newer than the last execution time (lastCheckTimestamp).
Formats the comments to be included in the feed.
Adds the comments to the feed.

Discussions

Discussions are very similar to issues, but they are not supported in the GitHub API REST, so we used the GitHub GraphQL API to fetch the comments.

const comments = await Promise.all(discussionsInScope.map(async ({ discussionId, team }) => {
  const { repository } = await graphql(
    `
    {
      repository(name: "node", owner: "nodejs") {
        discussion(number: ${discussionId}) {
          comments(last: 100) {
            edges {
              node {
                body
                publishedAt
                updatedAt
                databaseId
              }
            }
          }
        }
      }
    }
    `,
    {
      headers: {
        authorization: `token ${process.env.GITHUB_TOKEN}`
      }
    }
  )

  return repository.discussion.comments.edges
    .filter(comment => new Date(comment.node.publishedAt).getTime() > lastCheckTimestamp)
    .map(comment => ({ ...comment.node, team, discussionId }))
}))

See the full file for more details

Updating the feed

In order to update the feed we need to split the current feed by a breakDelimiter that is defined in the config.json file.

//...OMITED...
const feedContent = getFeedContent()
const [before, after] = feedContent.split(breakDelimiter)
const updatedFeedContent = `${before}${breakDelimiter}${relevantReleases}${after}`
overwriteFeedContent(updatedFeedContent)

See the full file for more details

Formatting the feed

We use the library xml-formatter to normalize the feed content. This will help us curate the content later on when reviewing the PR.

import xmlFormat from 'xml-formatter'
import { getFeedContent, overwriteFeedContent } from '../utils/index.js'

const xml = getFeedContent()
const formattedXml = xmlFormat(xml, { indentation: '  ', collapseContent: true })
overwriteFeedContent(formattedXml)

See the full file for more details

Validate the feed

In order to validate the feed, we directly use the W3C Feed Validation Service with an HTTP Request, simulating the form (using the got library) and parsing the response.

  const data = await got.post('https://validator.w3.org/feed/check.cgi', {
    form: {
      rawdata: xml,
      manual: 1
    }
  }).text()

  // Avoid importing CSS in the document
  const dom = new JSDOM(data.replace(/@import.*/gm, ''))

  const title = dom.window.document.querySelector('h2').textContent
  const recommendations = dom.window.document.querySelector('ul').textContent

  console.log(recommendations)

  if (title === 'Sorry') {
    console.log('🚨 Feed is invalid!')
    process.exit(1)
  } else {
    console.log('✅ Feed is valid!')
  }

Note: In order to use the library jsdom to scrape the HTML response we need to avoid the @import statements in the CSS.

The Github Action

Cron Job and Manual Trigger

The GitHub Action is configured to run every week, but it can be triggered manually by using the workflow_dispatch event. This is useful when we want to update the feed manually, for example when we want to add a new news that is not available on GitHub or just want to promote some news quickly.

on:
    workflow_dispatch:
    schedule:
        - cron: '0 0 * * 0'
# ...OMITED...

See the full file for more details

API Limits

The GitHub API has a limit on requests. This process makes many requests to the API, so the best way to overcome this limitation is by using a GitHub Token.

This token can be created by a user and then added to the repository secrets. The GitHub Action will use this token to authenticate the requests to the API, and it will have a higher limit than the anonymous requests.

But the best solution is to use the already available tokens in the GitHub Actions as follows:

# ...OMITED...  
permissions:
  contents: write
  pull-requests: write
  issues: read
  packages: none

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    # ...OMITED...  

    - name: Collect Releases
      run: npm run collect:releases
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

    # ...OMITED...

See the full file for more details

We are passing the secrets.GITHUB_TOKEN as an environment variable GITHUB_TOKEN to the scripts.

Slack Notifications

The feed is published on Slack using the RSS App. This app is listening to the feed and pushing the new items to specific channel(s). In our case, we are using the channel #nodejs-news-feed.

Acknowledgment

Thanks a lot to the Node.js Next 10 team for the support and feedback on this project, especially to Michael Dawson for the guidelines, reviews, and suggestions.

Dockerize Javascript IOT Applications

Ulises Gascón — Mon, 03 Jul 2023 21:52:23 +0000

I am a Docker Captain and I love build Robots and IoT devices.

In those years I have been doing a lot of IoT prototypes for companies and for myself. I have been using Arduino, Raspberry Pi, and other microcontrollers.

There are few things that make IoT development very different from other types of software development. One of them is that you need to have a physical device to test your code or build a good simulator. This means that you need to have a device that you can connect to your computer and run your code on it.

It was very common for me to have a lot of devices, sensors and wires in my backpack while commuting to work. I end up having many funny situations when I was doing IoT development in Coffee Shops around the world, just like the picture above and many others from 2015.

Introduction

When Arduino started to become popular, I was very excited about the idea of building my own IoT devices. I started to learn how to program in C/C++ and I bought a lot of sensors and other components to build my own projects.

But I soon realized that I was spending too much time on the Arduino source code, so I decided to use Javascript instead. I started to use Johnny-Five and I was able to build complex IoT device in a few hours.

Prototype is the new MVP

While the JSDayES 2016 was taking place, I was in one of the stands. During the breaks I was able to hack the welcome candy jar and build a simple IoT device that was controlled by a Node.js server. The device was able to change the color of the light depending on the logic that you define in a simple Node.js App.

Why Johnny Five is so cool?

The library Johnny-Five is a Javascript library that allows you to control Arduino and other microcontrollers using Javascript. It is very easy to use and it has a lot of examples that you can use to learn how to use it.

As a Javascript developer, I was very happy to be able to use the same language to build IoT devices:

The typical blink led example for an Arduino board looks like this:

var five = require("johnny-five");
var board = new five.Board();

board.on("ready", function() {
  var led = new five.Led(13);
  led.blink(1000);
});

In compare to the classic approach:

// the setup function runs once when you press reset or power the board
void setup() {
  // initialize digital pin LED_BUILTIN as an output.
  pinMode(LED_BUILTIN, OUTPUT);
}

// the loop function runs over and over again forever
void loop() {
  digitalWrite(LED_BUILTIN, HIGH);  // turn the LED on (HIGH is the voltage level)
  delay(1000);                      // wait for a second
  digitalWrite(LED_BUILTIN, LOW);   // turn the LED off by making the voltage LOW
  delay(1000);                      // wait for a second
}

It might not look like a big difference, but when you are building complex IoT devices, it makes a big difference. You can use all the Javascript features like Promises, Async/Await, etc. to build your IoT devices and all the compute power that the host device provides, as well all the modern communication that we expect today (HTTP APIS, SDKs..).

This will force you to connect (USB, Bluetooth...) the IoT device to a computer all the time as the device won't have the instructions recorded.

Obviously not all the IOT projects fits this requirements, but for a simple prototype is a fantastic solution.

Why should I Dockerize my project?

Johnny Five is a fantastic library, but some of their dependencies have a complex installation process (under the hood), this sometimes makes difficult to install the library in some environments. Specially if you need to change between devices, it was very frustrating for me at the beginning until I decided to use Docker to make it easier to execute.

Dockerize your JS IoT application

I created this repository to dockerize the blink example, so it is easier to follow and explore the details.

Get the necessary software

Before you begin, you need to have Docker installed on your machine.

Write a Dockerfile

Here is a hyper simplified Dockerfile, if you want to build a production ready example Start from this

If you speak Spanish I also recommend you my ebook "Docker Seguro"

FROM node:18.14.1
WORKDIR /usr/src/app
RUN apt-get update
RUN apt-get -y install udev
COPY package*.json ./
RUN npm install --unsafe-perm --build-from-source=serialport
COPY . .
CMD [ "npm", "run", "start" ]

Build the Docker image

Using the docker build command, create a Docker image.

docker build -t blink .

Run your application in a Docker container

Launch your application using the docker run command, specifying the necessary parameters to configure your container.

Here is the tricky part, you will need to pass the USB device to the container, so it can access the Arduino board.

docker run -it --rm --privileged --device=/dev/ttyUSB0 blink

In my case the port was /dev/ttyUSB0, but this might be different for you. In this Guide you can find more information.

Sometimes access to USB devices from Docker can be tricky, especially in macOS.

Can I pass through a USB device to a container?
Unfortunately, it is not possible to pass through a USB device (or a serial port) to a container as it requires support at the hypervisor level.
Docker: General FAQs for Desktop

But there are many workarounds from the community:

Publish your Docker image

Optionally, you can publish your Docker image to a container registry, such as Docker Hub, for easy access and distribution.

Safely store secrets in Git using Blackbox

Ulises Gascón — Mon, 06 Feb 2023 17:46:09 +0000

Storing secrets in a Git repository can be a dangerous proposition, as the information is often stored in plaintext and can be seen by anyone with access to the repository. It's important to remember that secrets should never be stored in plaintext and that measures should be taken to ensure that the secrets are encrypted and secure.

In this post, I will explain how to use Blackbox to safely store secrets within a Git repository as well as provide examples of how to use it in GitHub Actions as a secret repository to quickly load secrets into your pipelines.

Blackbox in a nutshell

Most developers understand the importance of keeping secrets safe and secure. Unfortunately, it can be difficult to achieve this when dealing with secrets that are stored in a Git repository. Fortunately, Blackbox is a great tool for securely storing secrets within Git. It helps streamline the process of working with secrets while ensuring the highest level of security possible. With its easy-to-use PGP encryption and automated processes, Blackbox ensures developers have a safe and efficient way of managing their secrets.

Instalation

In mac the best way is to use Brew, so you should have Brew installed in your machine, in the first place.

Then, with this command, install Blackbox:

brew install blackbox

In case that you are using a different platform please check the official documentation.

Usage / How to

Initialize the project

blackbox_initialize

Add users

♦️ Note: you need to have the user's pgp public key in your pgp keychain and you MUST be an ADMIN to run this command (you can't add yourself unless you are the first user in the project)

# Add the new key to the project
blackbox_addadmin {email}
# Re-encrypt the files including the new user
blackbox_update_all_files

Remove users

# Remove the user from the project
blackbox_removeadmin {email}
# Re-encrypt the files without the previous key
blackbox_update_all_files

Add a new file

By default, Backbox won't encrypt the new files, so you need to add them manually by using blackbox_register_new_file once your changes are done.

# Create the file
touch secrets/demo_file.txt
# ... do your things in the file...
# Encrypt the file
blackbox_register_new_file secrets/demo_file.txt

♦️ Note: when you encrypt the file, the file is renamed with a .gpg like secrets/demo_file.txt.gpg extension. Now the content is not legible (encrypted). When the file is encrypted, you are ready to commit the changes over that file :-)

Edit a file

You need to decrypt the file first and then re-encrypt the file at the end.

# Decrypt the file
blackbox_decrypt_file secrets/demo_file.txt
# ... make your changes in the file...
# Re-encrypt the file
blackbox_edit_end secrets/demo_file.txt

Once the file is decrypted, it will appear as a new file with the same name without the .pgp extension. The file now looks like this secrets/demo_file.txt and the content is in clear text (decrypted).

You can perform all the changes you want and then encrypt the file again when you are done. Then the plain file will be automatically removed and the encrypted file updated.

Delete a file

Use the command blackbox_deregister_file <file>. Afterwards, is very important to run the command blackbox_shred_all_files in order to remove decrypted files that could still exist in your machine. (Be aware: there is no typo in the command, the 'a' letter is NOT missed from the shred word).

Decrypt all files

If you need to perform changes in several files, the easiest way is to decrypt all the files at once:

blackbox_decrypt_all_files

Afterwards, when you're done, please manually encrypt all the files, one by one, with the command blackbox_edit_end secrets/{file_name}.
There is no option to batch that process yet in Blackbox, but there is a discussion ongoing about it.

Check the differences between a clear file and a encrypted file:

blackbox_diff secrets/demo_file.txt

Using Blackbox in GitHub Actions

One of the coolest features is that you can use pgp to decrypt secrets in any CI process, this is a simple example that

Load the PGP keys in the machine
Clone the secret management repository using a github token
Decrypt one file and store it as .env
Source the .env file to the npm run build process.

name: Pull Request Check

on: [pull_request]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Ensure Node Version
        uses: actions/setup-node@v3

      - name: import GPG key
        env:
          GPG_KEY: ${{ secrets.GPG_KEY }}
        run: |
          echo $GPG_KEY | base64 --decode > signature.asc
          gpg --batch --import signature.asc
      - name: Clone secrets-management
        env:
          GH_EXTENDED_TOKEN: ${{ secrets.GH_EXTENDED_TOKEN }}
        run: git clone https://${GH_EXTENDED_TOKEN}:x-oauth-basic@github.com/UlisesGascon/super-secrets-management.git

      - name: decrypt secrets (local environment)
        env:
          BRANCH: ${{ steps.vars.outputs.short_ref }}
          GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}
        run: gpg --no-tty --batch --passphrase "$GPG_PASSPHRASE" --pinentry-mode loopback --output .env --decrypt super-secrets-management/secrets/app/.env.gpg

      - name: Install dependencies
        run: npm ci

      - name: Build the project
        run: |
          source .env
          npm run build

      #... MORE STEPS

Conclusion

Blackbox is a great tool for securely storing secrets within a Git repository. It provides an easy-to-use PGP encryption system, which ensures that secrets are kept safe and secure. Furthermore, Blackbox can also be used in GitHub Actions as a secret repository, allowing developers to quickly load secrets into their pipelines. With its robust security and automated processes, Blackbox is an excellent tool for securely storing secrets within Git.

You should use the OpenSSF Scorecard

Ulises Gascón — Mon, 23 Jan 2023 20:46:00 +0000

As you may know, the Node.js Ecosystem Security Working Group has defined its priorities for 2023. A key initiative for us will be to assess the organization against the best practices available, such as the OpenSSF Scorecard.

OpenSSF in Node.js

We had a great discussion about the OpenSSF scorecard with the Google Open Source Security Team (GOSST) in the Ecosystem Security Working Group meeting this week.

We began the discussion in this issue, and here you can find the meeting notes:

Assessment against best practices (OpenSSF Scorecards ...) #859

Add OSSF Scorecard #851

Discussion with GOSST about implementing it on Node.js

The Nodejs currently report is located here, also json version available

Agreement to update action version tag by hash in GHA, following this example, lead by GOSST

Agreement to add/document the next steps in this issue in order to provide a good context for the following PRs and TSC Meetings, lead by GOSST

The current score for Node.js is 6.8 out of 10. We will be working to improve this score in the coming months. If you would like to be notified, please subscribe to the Security Working Group repository.

OpenSSF Scorecard in a nutshell

The Scorecard will evaluate the security of your project based on automated checks related to four scenarios.

Malicious maintainers
Build System Compromises
Source Code Compromises
Malicious packages

In order to accomplish this, the scripts are focused in 5 areas (Code Vulnerabilities, Maintenance, Continuous Testing, Source Risk Assestment, Build Risk Assestment and Holistic Security Practices).

Each area has its own associated risk, so the overall score is the average of the five areas. Here, you can check the details of each by consulting the documentation in detail.

The following are the types of questions this score card provides answers to:

Does the project contain a security policy?
Does the project use fuzzing tools?
Does the project use static code analysis tools?
Does the project use Branch Protection?
Does the project have contributors from at least two different organizations?
Does the project cryptographically sign releases?
And many more...

If you are wondering if this is a good idea for your project, I think it is a good idea to at least review your packages in the directory.

OpenSSF Scorecard Implementation

It took less than 5 minutes to install. It quickly analysed the repo and identified easy ways to make the project more secure. Priya Wadhwa, Kaniko

You have two options:

👉 This blog post was originally posted in my blog the 22/01/2023