DEV Community: Ruan Martinelli

A Complete Guide to Buffers in Node.js

Ruan Martinelli — Thu, 05 Aug 2021 13:22:20 +0000

In Node.js, buffers are a special type of object that can store raw binary data. A buffer represents a chunk of memory - typically RAM - allocated in your computer. Once set, the size of a buffer cannot be changed.

A buffer stores bytes. A byte is a sequence of eight bits. Bits are the most basic unit of storage on your computer, they can hold the value of either 0 or 1.

Node.js exposes the Buffer class in the global scope (you don't need to import or require it like other modules). With this API, you get a series of functions and abstractions to manipulate raw binaries.

A buffer in Node.js looks like this:

<Buffer 61 2e 71 3b 65 2e 31 2f 61 2e>

In this example, you can see 10 pairs of letters and numbers. Each pair represents a byte stored in the buffer. The total size of this particular buffer is 10.

You might be asking yourself: "if these are bits and bytes, where are the 0s and 1s?"

That's because Node.js displays bytes using the hexadecimal system. This way, every byte can be represented using just two digits - a pair of numbers and letters from 0-9 and "a" to "f".

Why buffers? Before buffers were introduced, there was no easy way of handling binary data in JavaScript. You would have to resort to primitives such as strings, which are slower and have no specialized tools to handle binaries. Buffers were created to provide a proper set of APIs to manipulate bits and bytes in an easy and performant way.

Working with buffers

Let's see some of the things we can do with buffers.

You will notice that handling buffers is a bit similar to the way we handle arrays in JavaScript. For example, you can .slice(), .concat() and get the .length of a buffer. Buffers are also iterable and can be used within constructs such as for-of.

If you're following the examples on your computer, keep in mind that the Buffer class is exposed globally. You don't need to import or require it as a separate module.

Creating buffers

Buffers are created using these three methods:

Buffer.from()
Buffer.alloc()
Buffer.allocUnsafe()

In the past, buffers were created using the Buffer class constructor (e.g., new Buffer()). This syntax is now deprecated.

Buffer.from()

This method is the most straightforward way to create a buffer. It accepts a string, an array, an ArrayBuffer, or another buffer instance. Depending on which params you pass, Buffer.from() will create a buffer in a slightly different way.

When passing a string, a new buffer object will be created containing that string. By default, it will parse your input using utf-8 as the enconding (see here all enconding types supported):

// Creates a new buffer with the string 'heya!'
// If no enconding is passed in the second parameter, defaults to 'utf-8'.
Buffer.from("heya!");

// Creates the same buffer as the above, but passes 'heya!' as a hex encoded string
Buffer.from("6865796121", "hex");

You can also pass an array of bytes to Buffer.from(). Here I am passing the same string as before ("heya!"), but represented as an array of hexadecimal characters:

// Also writes 'heya!' to the buffer, but passes a array of bytes
Buffer.from([0x68, 0x65, 0x79, 0x61, 0x21]);

If you're not familar with the 0xNN syntax, it means that the characters after 0x should be interpreted as hexadecimal values.

When passing a buffer to Buffer.from(), Node.js will copy that buffer into the current one. The new buffer is allocated in a different area of memory, so you can modify it independently:

const buffer1 = Buffer.from("cars");

// Creates a buffer from `buffer1`
const buffer2 = Buffer.from(buffer2);

// Modify `buffer2`
buffer2[0] = 0x6d; // 0x6d is the letter "m"

console.log(buffer1.toString()); // --> "cars"
console.log(buffer2.toString()); // --> "mars"

These should cover most of the cases where you use Buffer.from(). Refer to the docs for other ways to use it.

Buffer.alloc()

The .alloc() method is useful when you want to create empty buffers, without necessarily filling them with data. By default, it accepts a number and returns a buffer of that given size filled with 0s:

Buffer.alloc(6);
// --> <Buffer 00 00 00 00 00 00>

You can later on fill the buffer with any data you want:

// Creates a buffer of size 1 filled with 0s (<Buffer 00>)
const buff = Buffer.alloc(1);

// Fill the first (and only) position with content
buff[0] = 0x78 // 0x78 is the letter "x"

console.log(buff.toString('utf-8');
// --> 'x'

You can also fill the buffer with other content than 0 and a given enconding:

Buffer.alloc(6, "x", "utf-8");
// --> <Buffer 78 78 78 78 78 78>

Buffer.allocUnsafe()

With .allocUnsafe() , the process of sanitizing and filling the buffer with 0s is skipped. The buffer will be allocated in a area of memory that may contain old data (that's where the "unsafe" part comes from). For example, the following code will most likely always print some random pieces of data every time you run it:

// Allocates a random area of memory with size 10000
// Does not sanitizes it (fill with 0) so it may contain old data
const buff = Buffer.allocUnsafe(10000);

// Prints loads of random data
console.log(buff.toString("utf-8"));

A good use case for .allocUnsafe() is when you are copying a buffer that was safely allocated. Since you will completely overwrite the copied buffer, all the old bytes will be replaced by predictable data:

// Creates a buffer from a string
const buff = Buffer.from("hi, I am a safely allocated buffer");

// Creates a new empty buffer with `allocUnsafe` of the same
// length as the previous buffer. It will be initally filled with old data.
const buffCopy = Buffer.allocUnsafe(buff.length);

// Copies the original buffer into the new, unsafe buffer.
// Old data will be overwritten with the bytes
// from 'hi, I am a safely allocated buffer' string.
buff.copy(buffCopy);

console.log(buffCopy.toString());
// --> 'hi, I am a safely allocated buffer'

In general, .allocUnsafe() should only be used if you have a good reason (e.g., performance optimizations). Whenever using it, make sure you never return the allocated buffer without completely filling it with new data, otherwise you could be potentially leaking sensitive information.

Writing to buffers

The way to write data into buffers is using Buffer.write(). By default, it will write a string encoded in utf-8 with no offset (starts writing from the first position of the buffer). It returns a number, which is the number of bytes that were written in the buffer:

const buff = Buffer.alloc(9);

buff.write("hey there"); // returns 9 (number of bytes written)

// If you write more bytes than the buffer supports,
// your data will truncated to fit the buffer.
buff.write("hey christopher"); // retuns 9 (number of bytes written)

console.log(buff.toString());
// -> 'hey chris'

Keep in mind that not all characters ocuppy a single byte in the buffer (!):

const buff = Buffer.alloc(2);

// The copyright symbol ('©') occupies two bytes,
// so the following operation will completely fill the buffer.
buff.write("©"); // returns 2

// If the buffer is too small to store the character, it will not write it.
const tinyBuff = Buffer.alloc(1);

tinyBuff.write("©"); // returns 0 (nothing was written)

console.log(tinyBuff);
// --> <Buffer 00> (empty buffer)

Also notice that 2 is not the highest number of bytes a character can have. For example, the utf-8 enconding type supports characters with up to 4 bytes. Since you cannot modify the size of the buffer, you always need to be mindful of what you are writing and how much space it will take (size of the buffer vs. size of your content).

Another way to write into buffers is throguh an array-like syntax, where you add bytes to a specific position of the buffer. It's important to notice that any data with more than 1 byte needs to be broken down and set on each position of the buffer:

const buff = Buffer.alloc(5);

buff[0] = 0x68; // 0x68 is the letter "h"
buff[1] = 0x65; // 0x65 is the letter "e"
buff[2] = 0x6c; // 0x6c is the letter "l"
buff[3] = 0x6c; // 0x6c is the letter "l"
buff[4] = 0x6f; // 0x6f is the letter "o"

console.log(buff.toString());
// --> 'hello'

// ⚠️ Warning: if you try setting a character with more than 2 bytes
// to a single position, it will fail:
buff[0] = 0xc2a9; // 0xc2a9 is the symbol '©'

console.log(buff.toString());
// --> '�ello'

// But if you write each byte separately...
buff[0] = 0xc2;
buff[1] = 0xa9;

console.log(buff.toString());
// --> '©llo'

While it's appreciated that you can write to buffers using an array-like syntax, I suggest sticking to Buffer.from() when you can. Managing the length of inputs can be a hard task and will bring complexity to your code. With .from(), you can write things in a buffer worry-free and handle the cases where the input is too large by checking if nothing was written (when it returns 0).

Iterating over buffers

You can use modern JavaScript constructs to iterate over a buffer the same way you would with an array. For example, with for-of:

const buff = Buffer.from("hello!");

for (const b of buff) {
  // `.toString(16)` returns the content in hexadecimal format.
  console.log(b.toString(16));
}

// Prints:
// --> 68
// --> 65
// --> 6c
// --> 6c
// --> 6f
// --> 21

Other iterator helpers such as .entries(), .values() and .keys() are also available for buffers. For example, using .entries():

const buff = Buffer.from("hello!");
const copyBuff = Buffer.alloc(buff.length);

for (const [index, b] of buff.entries()) {
  copyBuff[index] = b;
}

console.log(copyBuff.toString());
// -> 'hello!'

Going further: Buffers and TypedArrays

In JavaScript (I mean JavaScript in general, not restricted to Node.js), memory can be allocated using the special ArrayBuffer class. We rarely manipulate ArrayBuffer objects directly. Instead, we use a set of "view" objects which reference the underlying array buffer. Some of the view objects are:

Int8Array, Uint8Array, Uint8ClampedArray, Int16Array, Uint16Array, Int32Array, etc. See the full list here.

And then there is TypedArray, which is an umbrella term to refer to all of these view objects listed above. All view objects inherit methods from TypedArray via prototypes. The TypedArray constructor is not exposed globally, you always have to use one of the view methods. If you see some tutorial or documentation using new TypedArray(), it means it's using any of the view objects (Uint8Array, Float64Array, etc).

In Node.js, objects created from the Buffer class are also instance of Uint8Array. There are a few small differences between them, which you can read here.

Conclusion

As a beginner, buffers were a topic in Node.js that got me very confused (another one was streams, but that deserves its own post). Hopefully I was able to demystify some of the concepts around buffers and give an overview of the Buffer API.

Authentication with Supabase and React

Ruan Martinelli — Wed, 07 Apr 2021 17:30:11 +0000

Supabase is an open source managed back-end platform. It's a direct alternative to Firebase, which is owned by Google and closed source.

Supabase comes with features such as authentication, object storage, and managed databases. Everything is built on top of open source tools, such as PostgREST and GoTrue. If you want, you can also self-host your own instance of Supabase. As of today, Supabase is in public Beta.

In this tutorial you will learn how to build an a simple React application with authentication using Create React App (CRA). Supabase will serve as the back-end for the authentication part. The application will include sign in, sign up, and a private route than can only be accessed with valid credentials.

If you wanna jump straight to the code, you can check the GitHub repository.

Setting up Supabase

VIsit Supabase's website to create a new account. Click the "Start your project" button and sign in with your GitHub account.

After signing in to the dashboard, hit the green "New Project" button. A modal like this should appear:

1. Create a new project on Supabase

Choose a name for your project and a region close to you. It's required that you set a database password too, but we won't be using any on this tutorial.

It will take a few minutes for the project to be fully created. After it's done, go to Settings > API and copy the URL & Public Anonymous API key. Save the values somewhere, you will need them later on.

2. Copy public credentials

💡 Tip: To make it easier to test the sign up flow, you can also disable email confirmations on Authentication > Settings > Disable Email Confirmations.

Setting up the project

Create a new project using Create React App:

npx create-react-app supabase-auth-react

3. Create a new CRA project

I usually do some cleanup on new CRA projects before start developing. Here's how the project structure looks like after moving files around and deleting a few imports:

.
├── package.json
├── .env.local
└── src
    ├── components
    │   ├── App.js # Moved from src/
    │   ├── Dashboard.js
    │   ├── Login.js
    │   ├── PrivateRoute.js
    │   └── Signup.js
    ├── contexts
    │   └── Auth.js
    ├── index.js # Already created by CRA
    └── supabase.js

4. Overview of the folder structure after cleaning up CRA files

Feel free to recreate the same file structure. Don't worry about adding any code or trying to make sense of all the components just yet, we will go through everything later.

The src/index.js and src/components/App.js were already created by CRA. Here's how they look after cleaning up:

// src/components/App.js

export function App() {
  return (
    <div>
      <h1>supabase-auth-react</h1>
    </div>
  )
}

5. Updated App.js component

// src/index.js
import React from 'react'
import ReactDOM from 'react-dom'

import { App } from './components/App'

ReactDOM.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>,
  document.getElementById('root')
)

6. Updated root component

Setting up the Supabase Client Library

First, install the Supabase JavaScript client library on your project:

npm install @supabase/supabase-js

7. Install the Supabase client

Now add the code to initialize Supabase on src/supabase.js:

// src/supabase.js

import { createClient } from '@supabase/supabase-js'

const supabase = createClient(
  process.env.REACT_APP_SUPABASE_URL,
  process.env.REACT_APP_SUPABASE_PUBLIC_KEY
)

export { supabase }

8. Supabase client initalization

In your .env.local file, add the URL and Public Anonymous API Key saved from the first step:

# .env.local

REACT_APP_SUPABASE_URL="https://YOUR_SUPABASE_URL.supabase.co"
REACT_APP_SUPABASE_PUBLIC_KEY="eyJKhbGciOisJIUzI1Nd2iIsInR5cCsI6..."

9. Set Supabase configuration variables

Create authentication pages

Let's write the code for the Signup, Login and Dashboard components. These will be the three main pages of the application.

For now, let's just focus on writing a boilerplate for those components, without any authentication logic. Start by writing the Signup component:

// src/components/Signup.js

import { useRef, useState } from 'react'

export function Signup() {
  const emailRef = useRef()
  const passwordRef = useRef()

  async function handleSubmit(e) {
    e.preventDefault()

    // @TODO: add sign up logic
  }

  return (
    <>
      <form onSubmit={handleSubmit}>
        <label htmlFor="input-email">Email</label>
        <input id="input-email" type="email" ref={emailRef} />

        <label htmlFor="input-password">Password</label>
        <input id="input-password" type="password" ref={passwordRef} />

        <br />

        <button type="submit">Sign up</button>
      </form>
    </>
  )
}

10. Initial code for the Signup component

The Login component looks very similar to Signup, with a few differences:

// src/components/Login.js

import { useRef, useState } from 'react'

export function Login() {
  const emailRef = useRef()
  const passwordRef = useRef()

  async function handleSubmit(e) {
    e.preventDefault()

    // @TODO: add login logic
  }

  return (
    <>
      <form onSubmit={handleSubmit}>
        <label htmlFor="input-email">Email</label>
        <input id="input-email" type="email" ref={emailRef} />

        <label htmlFor="input-password">Password</label>
        <input id="input-password" type="password" ref={passwordRef} />

        <br />

        <button type="submit">Login</button>
      </form>
    </>
  )
}

11. Initial code for the Login component

The Dashboard is a simple component that displays a greeting message and offers the user to sign out:

// src/components/Dashboard.js

export function Dashboard() {
  async function handleSignOut() {
    // @TODO: add sign out logic
  }

  return (
    <div>
      <p>Welcome!</p>
      <button onClick={handleSignOut}>Sign out</button>
    </div>
  )
}

12. Initial code for the Dashboard component

Routing components with React Router

So far the components are isolated. There is no routing between the Signup, Login and Dashboard pages.

Let's work on that by adding React Router to the project:

npm install react-router-dom

13. Install React Router

In src/components/App.js, declare a route for each of the components created before:

// src/components/App.js

import { BrowserRouter as Router, Switch, Route } from 'react-router-dom'

import { Signup } from './Signup'
import { Login } from './Login'
import { Dashboard } from './Dashboard'

export function App() {
  return (
    <div>
      <h1>supabase-auth-react</h1>

      {/* Add routes here👇 */}
      <Router>
        <Switch>
          <Route exact path="/" component={Dashboard} />
          <Route path="/signup" component={Signup} />
          <Route path="/login" component={Login} />
        </Switch>
      </Router>
    </div>
  )
}

14. Declare routes in the App component

Let's also add links to navigate between the Signup and Login components:

// src/components/Signup.js

import { Link } from 'react-router-dom'

export function Signup() {
  // ...

  return (
    <>
      <form onSubmit={handleSubmit}>{/* ... */}</form>

      <br />

      {/* Add this 👇 */}
      <p>
        Already have an account? <Link to="/login">Log In</Link>
      </p>
    </>
  )
}

15. Add link between Signup and Login pages

// src/components/Login.js

import { Link } from 'react-router-dom'

export function Login() {
  // ...

  return (
    <>
      <form onSubmit={handleSubmit}>{/* ... */}</form>

      <br />

      {/* Add this 👇 */}
      <p>
        Don't have an account? <Link to="/signup">Sign Up</Link>
      </p>
    </>
  )
}

16. Add link between Signup and Login pages

You can test the navigation between components by running the project and clicking on the links or changing the URL in the navigation bar:

17. Demo of the navigation working

Shameless plug: you may notice that the HTML looks a bit different, that's because I am using axist, a tiny drop-in CSS library that I built.

Adding the authentication logic

To set up the authentication logic for the app, we're going to use React's Context API.

The Context API allows sharing data to a tree of components without explicitly passing props through every level of the tree. It's used to share data that is considered "global" (within that component tree).

You can read more about React Context in the official documentation.

In this tutorial, we will use Context to share data associated with the user and the authentication operations. All this information will come from Supabase and will be needed on multiple parts of the app.

Let's start by adding code on src/contexts/Auth.js. First, create a Context object:

// src/contexts/Auth.js

import React, { useContext, useState, useEffect } from 'react'
import { supabase } from '../supabase'

const AuthContext = React.createContext()

// ...

18. Create the Context object

Now, in the same file, create a Provider component called AuthProvider:

// src/contexts/Auth.js

// ...

export function AuthProvider({ children }) {
  const [user, setUser] = useState()
  const [loading, setLoading] = useState(true)

  useEffect(() => {
    // Check active sessions and sets the user
    const session = supabase.auth.session()

    setUser(session?.user ?? null)
    setLoading(false)

    // Listen for changes on auth state (logged in, signed out, etc.)
    const { data: listener } = supabase.auth.onAuthStateChange(
      async (event, session) => {
        setUser(session?.user ?? null)
        setLoading(false)
      }
    )

    return () => {
      listener?.unsubscribe()
    }
  }, [])

  // Will be passed down to Signup, Login and Dashboard components
  const value = {
    signUp: (data) => supabase.auth.signUp(data),
    signIn: (data) => supabase.auth.signIn(data),
    signOut: () => supabase.auth.signOut(),
    user,
  }

  return (
    <AuthContext.Provider value={value}>
      {!loading && children}
    </AuthContext.Provider>
  )
}

// ...

19. Create the AuthProvider

The AuthProvider does three things:

Calls supabase.auth.session to find out the current state of the user and update the user object.
Listens for changes in the authentication state (user signed in, logged out, created a new account, etc.) by subscribing to supabase.auth.onAuthStateChange function.
Prepares the object that will be shared by its children components (value prop). In this case, any components down the tree will have access to the signUp, signIn, signOut functions and the user object. They will be used by the Signup, Login and Dashboard components later on.

The loading state property will make sure the child components are not rendered before we know anything about the current authentication state of the user.

Now, create a useAuth function to help with accessing the context inside the children components:

// src/contexts/Auth.js

// ...

export function useAuth() {
  return useContext(AuthContext)
}

20. Create the useAuth function

You can check how the src/contexts/Auth.js looks after all the changes on the GitHub repository.

Lastly, we need to wrap the Signup, Login and Dashboard components with the AuthProvider:

// src/components/App.js

// ...

import { AuthProvider } from '../contexts/Auth'

export function App() {
  return (
    <div>
      <h1>supabase-auth-react</h1>
      <Router>
        {/* Wrap routes in the AuthProvider 👇 */}
        <AuthProvider>
          <Switch>
            <PrivateRoute exact path="/" component={Dashboard} />
            <Route path="/signup" component={Signup} />
            <Route path="/login" component={Login} />
          </Switch>
        </AuthProvider>
      </Router>
    </div>
  )
}

21. Add the AuthProvider to App.js

Adding authentication to the components

Remember the @TODOs you left earlier in the components? Now it's time to, well, do them.

The functions needed by the components - signUp, signIn and signOut - as well as the user object are available through the Context. We can now get those values using the useAuth function.

Let's start by adding the sign up logic to the Signup component:

// src/components/Signup.js

import { useRef, useState } from 'react'
import { useHistory, Link } from 'react-router-dom'

import { useAuth } from '../contexts/Auth'

export function Signup() {
  const emailRef = useRef()
  const passwordRef = useRef()

  // Get signUp function from the auth context
  const { signUp } = useAuth()

  const history = useHistory()

  async function handleSubmit(e) {
    e.preventDefault()

    // Get email and password input values
    const email = emailRef.current.value
    const password = passwordRef.current.value

    // Calls `signUp` function from the context
    const { error } = await signUp({ email, password })

    if (error) {
      alert('error signing in')
    } else {
      // Redirect user to Dashboard
      history.push('/')
    }
  }

  return (
    <>
      <form onSubmit={handleSubmit}>{/* ... */}</form>

      <br />

      <p>
        Already have an account? <Link to="/login">Log In</Link>
      </p>
    </>
  )
}

22. Add authentication logic to the Signup component

The Login component will look very similar. The main difference is that you will call signIn instead of signUp:

// src/components/Login.js

import { useRef, useState } from 'react'
import { useHistory, Link } from 'react-router-dom'

import { useAuth } from '../contexts/Auth'

export function Login() {
  const emailRef = useRef()
  const passwordRef = useRef()

  // Get signUp function from the auth context
  const { signIn } = useAuth()

  const history = useHistory()

  async function handleSubmit(e) {
    e.preventDefault()

    // Get email and password input values
    const email = emailRef.current.value
    const password = passwordRef.current.value

    // Calls `signIn` function from the context
    const { error } = await signIn({ email, password })

    if (error) {
      alert('error signing in')
    } else {
      // Redirect user to Dashboard
      history.push('/')
    }
  }

  return (
    <>
      <form onSubmit={handleSubmit}>{/* ... */}</form>

      <br />

      <p>
        Don't have an account? <Link to="/signup">Sign Up</Link>
      </p>
    </>
  )
}

23. Add authentication logic to the Login component

Lastly, change the Dashboard so the user can sign out of the application. You can also display some basic information together with the greeting message, such as the user ID:

// src/components/Dashboard.js

import { useHistory } from 'react-router'
import { useAuth } from '../contexts/Auth'

export function Dashboard() {
  // Get current user and signOut function from context
  const { user, signOut } = useAuth()

  const history = useHistory()

  async function handleSignOut() {
    // Ends user session
    await signOut()

    // Redirects the user to Login page
    history.push('/login')
  }

  return (
    <div>
      {/* Change it to display the user ID too 👇*/}
      <p>Welcome, {user?.id}!</p>
      <button onClick={handleSignOut}>Sign out</button>
    </div>
  )
}

24. Add authentication logic to the Dashboard component

Protecting routes

Currently, all the authentication logic is in place, but the Dashboard component remains publicly accessible. Anyone who happens to fall on locahost:3000 would see a broken version of the dashboard.

Let's fix that by protecting the route. If a user that is not authenticated tries to access it, they will be redirected to the login page.

Start by creating a PrivateRoute component:

// src/components/PrivateRoute.js

import React from 'react'
import { Route, Redirect } from 'react-router-dom'

import { useAuth } from '../contexts/Auth'

export function PrivateRoute({ component: Component, ...rest }) {
  const { user } = useAuth()

  return (
    <Route
      {...rest}
      render={(props) => {
        // Renders the page only if `user` is present (user is authenticated)
        // Otherwise, redirect to the login page
        return user ? <Component {...props} /> : <Redirect to="/login" />
      }}
    ></Route>
  )
}

25. Create the PrivateRoute component

The PrivateRoute wraps the Route component from React Router and passes down the props to it. It only renders the page if the user object is not null (the user is authenticated).

If the userobject is empty, a redirect to the login page will be made by Redirect component from React Router.

Finally, update the dashboard route in the App component to use a PrivateRoute instead:

// src/components/App.js

- <Route exact path="/" component={Dashboard} />
+ <PrivateRoute exact path="/" component={Dashboard} />

26. Update the Dashboard component to use PrivateRoute

Done! The dashboard is only available for authenticated users.

Final result

This is how the final version of the application should look like:

26. Final demo of the application

You can see the sign up, login, and sign out working. The dashboard page is also protected, attempting to access it by changing the URL redirects the user to the login page. Notice the user ID showing there too.

Going further

There are a few things that we could add for a more complete authentication flow:

Password reset. I intentionally left it out for simplicity, but Supabase supports password reset through their API. It does all the heavy-lifting for you, including sending the email to the user with the reset instructions.

Authentication providers. When logging in, you can also specify different authentication providers like Google, Facebook and GitHub. Check out the docs.

User operations. You can also add metatada to the authenticated user using the update function. With this, you could build for example a basic user profile page.

Thanks for reading this far!

Deploying to Google Cloud Run with Terraform

Ruan Martinelli — Mon, 15 Mar 2021 07:47:38 +0000

Cloud Run is a serverless platform from Google Cloud to deploy and run containers. It's fully managed, autoscallable, and has a generous free tier. Cloud Run can be used to serve Restful web APIs, WebSocket applications, or microservices connected by gRPC. It also integrates well with other Google Cloud solutions such as Cloud Tasks, Cloud Scheduler, and Pub/Sub.

Terraform is a popular open-source tool for running infrastructure as code. Terraform lets you manage and deploy infrastructure from multiple providers, one of them being Google Cloud. If this is your first time reading about Terraform, you might wanna check this introduction first.

In this article, you will see how to deploy a Cloud Run service to Google Cloud using Terraform. All the infrastructure will be written using HCL, the native syntax for Terraform's language. By the end of the tutorial, you should have a service up and running on Cloud Run and a URL to access it.

Prerequisites

To follow this tutorial you will need:

Terraform CLI. I recommend using the latest version, currently v0.14. Instructions to download and install Terraform can be found here.
Google Cloud SDK. The most recent version should also work well for this tutorial. Installation instructions here.
A Google Cloud account. If you don't have one, create it here.

Initial setup

Start by authenticating the SDK to Google Cloud:

# Authenticate to Google Cloud
gcloud auth application-default login

Follow the web flow to obtain the access credentials. By the end of this step, you will be able to execute commands on the SDK similar to a service account.

Create a new project where your Cloud Run service will be deployed. Replace PROJECT_ID and PROJECT_NAME with the desired values:

gcloud projects create "PROJECT_ID" --name="PROJECT_NAME"

Create a folder with a HCL file inside:

mkdir my-service && cd my-service

touch main.tf

For simplicity, all the Terraform code in the next steps will be added to main.tf.

Creating your first service

Begin by adding the requirements for Terraform and the Google provider on main.tf:

# main.tf

terraform {
  required_version = ">= 0.14"

  required_providers {
    # Cloud Run support was added on 3.3.0
    google = ">= 3.3"
  }
}

This will require the Terraform version to be the latest and the Google provider to be at least on version 3.3 - when Cloud Run support was added.

Now add the Google provider configuration. Replace PROJECT_ID with the value from the previous step:

# main.tf

provider "google" {
  # Replace `PROJECT_ID` with your project
  project = "PROJECT_ID"
}

The Cloud Run API doesn't come enabled on projects by default. Add the following resource to enable it:

# main.tf

# Enables the Cloud Run API
resource "google_project_service" "run_api" {
  service = "run.googleapis.com"

  disable_on_destroy = true
}

Now create the Cloud Run service in the us-central1 region:

# main.tf

# Create the Cloud Run service
resource "google_cloud_run_service" "run_service" {
  name = "app"
  location = "us-central1"

  template {
    spec {
      containers {
        image = "gcr.io/google-samples/hello-app:1.0"
      }
    }
  }

  traffic {
    percent         = 100
    latest_revision = true
  }

  # Waits for the Cloud Run API to be enabled
  depends_on = [google_project_service.run_api]
}

Let's stop for a while and check what the code above is doing:

name: the name of your service. It will be displayed in the public URL.
location: the region where your service will run. See all the options here.
image: The Docker image that will be used to create the container. Cloud Run has direct support for images from the Container Registry and Artifact Registry.
traffic: controls the traffic for this revision. The percent property indicates how much traffic will be redirected to this revision. latest_revision specifies that this traffic configuration needs to be used for the latest revision.
depends_on: waits for a resource to be ready, in this case, the Cloud Run API.

Invoking the service

By default, Cloud Run services are private and secured by IAM. To access them, you would need valid credentials with at least the Cloud Run Invoker permission set.

Let's change that and make the service publicly available through an HTTP endpoint.

There are other ways than HTTP requests to trigger a service. For example, they can be accessed by gRPC requests, WebSockets, and other Google Cloud products like Cloud Scheduler.

Add the following code on main.tf to expose your service:

# main.tf

# Allow unauthenticated users to invoke the service
resource "google_cloud_run_service_iam_member" "run_all_users" {
  service  = google_cloud_run_service.run_service.name
  location = google_cloud_run_service.run_service.location
  role     = "roles/run.invoker"
  member   = "allUsers"
}

The resource above is adding the permission to invoke the service to anyone on the internet. The allUsers identifier is a special value that represents authenticated and unauthenticated users.

To display the service URL in the Terraform command output, add this output to the configuration:

# main.tf

# Display the service URL
output "service_url" {
  value = google_cloud_run_service.run_service.status[0].url
}

Deploying the infrastructure

At this point, you have all it takes to deploy the infrastructure to Google Cloud using Terraform.

Start by initializing the configuration. Run the following command in your terminal:

# Initialize Terraform configuration
terraform init

Run terraform plan to verify the changes that will be applied:

# Plan the changes
terraform plan

If everything is correct, you will see that 3 resources will be created and the service URL will be displayed.

Plan: 3 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + service_url = (known after apply)

Run terraform apply to apply all the changes:

# Apply all the changes
terraform apply

If everything goes well, you will see this at the end of the output:

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

service_url = https://app-cvmew7554a-uc.a.run.app

You can check if the service is running using curl:

curl https://app-cvmew7554a-uc.a.run.app

Hello, world!
Version: 1.0.0
Hostname: localhost

Updating the service

Terraform can be used not just to push your initial infrastructure to Cloud Run, but also to update it.

Cloud Run works with revisions. When a configuration is changed or a new image is added, a new revision is created as a result. You can then redirect all the traffic to the new revision and start serving your updated application.

To update your service, simply change the value in the image property and pass it a new image:

# main.tf

resource "google_cloud_run_service" "run_service" {
  name = "app"

  # ...

  template {
    spec {
      containers {
        # Change `hello-app:1.0` to `hello-app:2.0` 👇
        image = "gcr.io/google-samples/hello-app:2.0"
      }
    }
  }

  # ...
}

With this, Terraform will create a new revision on Cloud Run. The previous revision is preserved, but because of the traffic options defined previously, it won't recieve any traffic.

Run terraform apply to deploy the changes:

# Apply all the changes
terraform apply

You can check that the new image is live using curl:

curl https://app-cvmew7554a-uc.a.run.app

Hello, world!
Version: 2.0.0 # Now serving 2.0.0
Hostname: localhost

Cleaning up

To delete all resources created with Terraform, run the following command and confirm the prompt:

# Destroy all the infrastructure created by Terraform
terraform destroy

This will disable the Cloud Run API, delete the Cloud Run service and its permissions.

The project was created using the gcloud CLI tool, so you will need to delete it manually. For that, you can run:

# Remove the Google Cloud project
gcloud projects delete PROJECT_ID

Thanks for reading this far!

Checkout hyperfoo.io for more Terraform articles!

How to Use Schemas on Fastify for Fun and Profit

Ruan Martinelli — Fri, 11 Dec 2020 15:27:54 +0000

This year, Fastify became my go-to framework for building Node.js APIs.

If the word sounds new to you, Fastify is a web framework for Node.js. It is used to build APIs and services the same way Express does.

Fastify comes with great features that really speed up the process of making applications. Among those features, my favorite one is the fact that the framework is schema-based (I'll explain).

In this post, I will share a few tricks on how you can leverage Fastify's schema capabilities to build APIs at a fast clip.

Schemas

Fastify adopts the JSON Schema format on its core. Many of its features and libraries are built around the popular standard. Ajv, a library to compile and validate JSON Schemas, is a direct dependency of the framework.

By adopting JSON Schema, Fastify opens doors to an entire ecosystem of tools built around it. Below, let's see how to combine all these tools and libraries together with the framework.

1. Validation

One of the ways Fastify uses JSON Schema is to validate data coming from clients. It lets you add input schemas to your routes. For example:

// Schema for `POST /movie` body
const PostMovieBody = {
  type: 'object',
  properties: {
    title: { type: 'string' },
    releaseYear: { type: 'integer', minimum: 1878 },
  },
}

app.post('/movie', {
  schema: {
    // Refence the schema here
    body: PostMovieBody,
  },
  handler: createMovie,
})

1. Using input schemas to validate data in the router.

In this example, any incoming data to POST /movie that doesn't conform to the PostMovieBody schema will throw a validation error.

This way, we are making sure that the handler function doesn't process any invalid or unexpected payloads.

Invalid objects will result in a validation error that looks like this:

POST /movie
{ releaseYear: 2020 } # The `title` parameter was not sent

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "body should have required property 'title'"
}

2. Example of Fastify's default error response.

Note: the content on message comes from Ajv.

2. Serialization

Serialization is the process of converting an object to a format that can be transferred over a network.

With Fastify, you can also define output schemas for JSON payloads. When you do so, any data returned to clients will be serialized and validated according to that definition.

More specifically, defining output schemas helps you in two ways:

Fastify serializes the data with fast-json-stringify. In many cases, it is faster than JSON.stringify.
Ajv validates the response. This will prevent sensitive fields from being exposed.

When declaring output schemas in your routes, each possible status code accepts a definition. For example, you can have schemas defined for 200 and 204 responses.

Tip: to use the same schema for a family of status codes, you can use the '2xx' notation.

Here's how to define an output schema to responses with a 200 status code:

// Generic `Movie` schema
const Movie = {
  type: 'object',
  properties: {
    id: { type: 'integer' },
    title: { type: 'string' },
    releaseYear: { type: 'integer', minimum: 1878 },
  },
}

app.post('/movie', {
  schema: {
    response: {
      // Payloads will be serialized according to the `Movie` schema
      200: Movie,
    },
  },
  // ...
})

3. Using output schemas to serialize and validate data in the router.

In this example, any object returned by the handler that doesn't match the Movie schema will result in an error. By default, the client receives a 400 response - similar to the example #2.

3. Documentation

Documentation is an essential piece in any REST API.

There are many ways to document your application. One of them is manually, where you write routes and definitions by hand in a common format like YAML or JSON.

You can already guess this approach has many problems: outdated schemas, inconsistent validations, type discrepancies, etc.

Another approach is automating your documentation. A tool will automatically generate all the routes and definitions based on an existing schema.

One popular specification for writing documentation is Swagger. Thanks to the official fastify-swagger plugin, you can transform your existing JSON Schema definitions into Swagger ones and expose a beautiful documentation page in a flick.

Adding fastify-swagger to a Fastify application should be straightforward:

const fastify = require('fastify')()

// Register the plugin before your routes
fastify.register(require('fastify-swagger'), {
  exposeRoute: true,
  routePrefix: '/documentation',
  swagger: {
    info: { title: 'movie-api' },
    // Add more options to get a nicer page ✨
  },
})

// Declare your routes here...

4. Registering the `fastify-swagger` plugin.

Now, when you start your Fastify application and navigate to /documentation in a browser this page will pop up:

5. The documentation page generated by `fastify-swagger`.

Note: The more metadata you add to your routes, the more complete your page will look. Check out the route options documentation.

4. Mocking

When testing services or endpoints, many times you will need to provide a fake or simulated input. These inputs are called mock objects. They replicate the structure and behavior of real objects.

You can create mock objects dynamically with the schemas you already have by using json-schema-faker. The library converts existing JSON Schemas into dummy objects that you can use in your tests. Let's see an example.

First, create a helper function (just a wrapper for json-schema-faker):

const jsf = require('json-schema-faker')

/**
 * Creates an object from a JSON Schema. Example:
 * schemaToObject(Movie)
 * => { id: 823, title: 'unicorn', releaseYear: 1942 }
 */
function schemaToObject(schema) {
  return jsf.resolve(schema)
}

6. Creating the `schemaToObject` helper function.

The schemaToObject function does exactly what the name says: given a JSON Schema definition, it returns a matching mock object.

Now let's put it to use. You can call this function whenever you need to create fake objects for your tests. For example, when sending requests to routes:

it('should create a movie', async () =
    // Create a mock object for the request
    const payload = await schemaToObject(PostMovieBody)

    // Calls the POST /movie
    const response = await request.post('/movie', payload)

    expect(response.status).toBe(200)
})

7. Using `schemaToObject` to generate fake payloads.

In this example, we are creating a mock object, POST-ing it to the POST /movie route, and checking the status code.

The schemaToObject function gives you a nice and clean way to test the "happy path" in your tests (when everything meets the expectations).

5. Jest

Jest is a testing framework for JavaScript. One of its features is the possibility to create or import custom matchers.

One of these matchers is jest-json-schema. This package adds a new assertion to Jest: toMatchSchema. It lets you validate an object against an existing JSON Schema definition - it's like Ajv was integrated to Jest.

Note: If you are using another testing framework, there is likely an equivalent to jest-json-schema for it.

Instead of manually asserting the values of each property in an object like this:

it('should create a movie', async () => {
  // ...
  expect(response.title).toBeString()
  expect(response.releaseYear).toBePositive()
})

8. Manually asserting each property in an object.

You can simplify things using toMatchSchema:

import { matchers } from 'jest-json-schema'
import { Movie } from './schemas'

expect.extend(matchers)

it('should create a movie', async () => {
  // ...
  expect(response).toMatchSchema(Movie)
})

9. Asserting all properties using a schema.

Notice I am using the Movie schema defined in example #3.

Of course, this is just simplifying type-checking in your tests. There are still other aspects of your code that need to be tested. Still, based on how easy it is to implement, I believe it's a good addition.

Putting it all together

Let's do a quick recap.

In examples #1 and #3, we have declared two schemas using the JSON Schema format - PostMovieBody and Movie. These schemas are used for:

Validating objects sent to the route.
Serializing and validating objects returned to the clients.
Generating documentation.
Creating mock objects.
Asserting objects on tests.

Now here's the fun part!

Suppose you need to start tracking a new property in your movie objects. For example, you need to save and display the movie poster URL. Let's name the new field posterUrl.

If you were not using a schema-based framework, you would need to go through all your code and update the existing objects to include the new property. This is far from ideal. The chances of missing an assertion in your tests or forgetting to update the documentation are high.

But thanks to the magic of schemas, this process is a breeze. Your definitions are your source of truth. Anything based on the schemas will change once the schema changes.

So, now let's see how we can add the posterUrl property.

The first step is to change the input schema (PostMovieBody) to include the new property:

const PostMovieBody = {
  type: 'object',
  properties: {
    title: { type: 'string' },
    releaseYear: { type: 'integer', minimum: 1878 },
+   posterUrl: { type: 'string' }
  }
}

10. Adding `posterUrl` to the input schema.

Now, since posterUrl must also be serialized and returned to the client, we also add it to the output schema (Movie):

const Movie = {
  type: 'object',
  properties: {
    id: { type: 'integer' },
    title: { type: 'string' },
    releaseYear: { type: 'integer', minimum: 1878 }
+   posterUrl: { type: 'string' }
  }
}

11. Adding `posterUrl` to the output schema.

And that's pretty much it!

Here is what will happen once you restart your server:

Fastify will start checking for posterUrl in the POST /movie route.
The Swagger file will be updated. The posterUrl property will start showing on the documentation page.
Mock objects in your tests will start being generated with a string value for posterUrl.
Tests using the toMatchSchema matcher will start checking for the posterUrl property.

...and you got all that just by changing two lines in your code. How cool is that?

Honorable mention: fluent-schema

If you are used to libraries like Joi or Yup, writing schemas using raw JavaScript objects might feel like a step back.

To overcome that feeling, you can use fluent-schema. It gives you the same compact and programmable interface present in other tools.

For example, we could rewrite the Movie schema in example #3 using fluent-schema:

const S = require('fluent-schema')

const Movie = const schema = S.object()
  .prop('title', S.string())
  .prop('releaseYear', S.number().minimum(1878))

12. Example of the `Movie` schema defined with fluent-schema.

Looks neat, huh?

And that's a wrap! I hope you have enjoyed it. Stay tuned for more Fastify articles. ✌️

Deploying a Node.js API to Cloud Functions with Terraform

Ruan Martinelli — Tue, 01 Dec 2020 21:15:10 +0000

In this tutorial you are going to deploy a simple Node.js API to Google Cloud Functions using Terraform.

Cloud Functions is a compute solution from Google Cloud Platform (GCP) . It provides functions as a service (FaaS), which is a way to run your code "on-demand", without managing any servers.

For deployment we choose Terraform, a command-line tool to build and deploy infrastructure using code. Terraform will help create a predictable and reproducible environment to run your code.

Since it’s not the main focus of this tutorial, we will go with a super simple Node.js API using Fastify. Feel free to use any other languages supported by Cloud Functions on this part.

After finished, you will have an up and running API with an URL that you can make requests to.

Prerequisites

To follow this guide you will need:

Terraform 0.13 or later. You can find the installation instructions here;
Google Cloud SDK. Any recent version should be fine. Installation instructions here;
Node.js 12 or later. If you don’t have Node.js installed, I recommend using nvm for that.

1. Setting up the GCP account

If you are using the Google Cloud SDK for the first time, you will need to authenticate with your Google Account. You can run the following command:

# Authenticate with GCP
gcloud auth application-default login

Now create the project on GCP:

# Create a GCP project
gcloud projects create PROJECT_ID --name="My App"

Note: replace PROJECT_ID with an unique project identifier (e.g. "my-app-2847162").

Set the project you just created as the default one. This will make it easier to run the subsequent commands.

# Set the project as the default one
gcloud config set project PROJECT_ID

Many features on GCP require a billing account linked to the project, Cloud functions is one of them. For this step, you will need to visit the dashboard:

Create a billing account on GCP.

Note: even though Google asks for your credit card, this tutorial should not cost any money to run. The first 2 million invocations of a Cloud Function are free. You will also learn how to shutdown and destroy all the resources created here.

After setting up billing, the account will be listed when you run the following command:

# List billing accounts available
gcloud beta billing accounts list

The output will look something like this:

ACCOUNT_ID            NAME         OPEN  MASTER_ACCOUNT_ID
87PPT3-QECCDL-9OLSSQ  my-account   True

Copy the account ID and run the following command to link the billing account to your project:

# Link a billing account to project
gcloud beta billing projects link PROJECT_ID --billing-account=BILLING_ACCOUNT_ID

Now you are going to structure the project.

2. Structuring the project

Create the files listed below so your repository looks like this:

.
├── terraform
│   ├── modules
│   │   └── function
│   │       ├── main.tf
│   │       ├── outputs.tf
│   │       └── variables.tf
│   ├── main.tf
│   ├── backend.tf
│   ├── outputs.tf
│   └── variables.tf
└── src
    └── index.js

Don’t worry about adding any content now. We will do that on the next step.

Note: This is an opinionated, but common structure for Terraform projects. Terraform doesn’t require your files to be in a certain disposition or have a specific name. Although not recommended, you can build an entire system within a single main.tf file.

The terraform/ folder contain files related to Terraform.

The src folder hosts the code for the Node.js API. Remember, the API code will be very simple, only a single index.js file is enough.

3. Writing the API

Let’s write the API using Fastify.

If you are following this tutorial with a different language, you can add your custom code at this point.

Note: If you never heard about Fastify before, it has a very similar API to other Node.js frameworks like Express.

First, initialize the project with npm and install fastify as a dependency:

# Initialize project
npm init

# Install fastify
npm install fastify

Add this content to the src/index.js file:

// src/index.js

const fastify = require('fastify')
const app = fastify({ logger: true })

app.get('/', async (req, res) => {
  return { works: true }
})

exports.app = async (req, res) => {
  await app.ready()
  app.server.emit('request', req, res)
}

Update the entry point for your code in the package.json file:

// package.json

{
-   "main": "index.js",
+   "main": "src/index.js"
  // ...
}

This will tell Cloud Functions where your API is located. Now let’s jump to the terraform/ folder and start writing the infrastructure code.

4. Writing the infrastructure code

At this point, you already have all the files and folders created inside your terraform/ folder.

Before start adding code to them, let’s take a look at each file’s responsibility:

backend.tf. Declares which Terraform backend you will use.
main.tf. Where you will write the logic for creating resources or invoking modules.
variables.tf. Lists the variables and their values that will be used on main.tf.
outputs.tf. Lists the values your Terraform code will return.
modules/. A place for your Terraform modules. In this case, there will be just one named function.

Note: The name function was chosen for the module because it creates a Cloud Function. Feel free to use a different name.

Note: If you are not familiar with modules in Terraform, think of them as a function or component. It’s something that you write once and then can reuse on other parts of your code.

Begin by declaring which Terraform backend you want to use - where you want to store your Terraform state files.

Let's choose the "local" backend for now, meaning the state files will be stored on your local repository.

# terraform/backend.tf

terraform {
  backend "local" {}
}

Now add the following variables to your terraform/variables.tf file:

# terraform/variables.tf

variable "project" {
    default = "PROJECT_ID"
}
variable "region" {
    default = "us-central1" # Choose a region
}

In terraform/main.tf, declare the provider Terraform will connect to. In your case, the Google Cloud Platform provider (named "google") .

The Google provider has two required parameters, project and region. We can reference the values declared on the step above by accessing the properties in the var object.

# terraform/main.tf

provider "google" {
  project = var.project
  region  = var.region
}

# ⚠️ More code here soon

You will go back to this file soon to add more configuration.

Creating the `function` module

In order to create a Cloud Function on GCP you will need to combine a few resources together:

A storage bucket, to store the code that will be executed by the function
The function itself, to run the code you wrote
An IAM policy, to allow users to invoke the function

These resources will be grouped in the Terraform module you are about to create.

This will make it easier if you want to deploy a second environment (e.g. development & staging) or create multiple functions - you can just invoke the module again with different parameters.

On terraform/modules/function/variables.tf, add the arguments needed by the module. All arguments are required, so don’t add default values.

# terraform/modules/function/variables

variable "project" {}
variable "function_name" {}
variable "function_entry_point" {}

Proceeding to terraform/modules/function/main.tf, add the logic to create the function and all resources needed.

⚠️ This file is a bit dense. Follow through the comments in it to get a better idea of what’s happening.

# terraform/modules/function/main.tf

locals {
  timestamp = formatdate("YYMMDDhhmmss", timestamp())
    root_dir = abspath("../")
}

# Compress source code
data "archive_file" "source" {
  type        = "zip"
  source_dir  = local.root_dir
  output_path = "/tmp/function-${local.timestamp}.zip"
}

# Create bucket that will host the source code
resource "google_storage_bucket" "bucket" {
  name = "${var.project}-function"
}

# Add source code zip to bucket
resource "google_storage_bucket_object" "zip" {
  # Append file MD5 to force bucket to be recreated
  name   = "source.zip#${data.archive_file.source.output_md5}"
  bucket = google_storage_bucket.bucket.name
  source = data.archive_file.source.output_path
}

# Enable Cloud Functions API
resource "google_project_service" "cf" {
  project = var.project
  service = "cloudfunctions.googleapis.com"

  disable_dependent_services = true
  disable_on_destroy         = false
}

# Enable Cloud Build API
resource "google_project_service" "cb" {
  project = var.project
  service = "cloudbuild.googleapis.com"

  disable_dependent_services = true
  disable_on_destroy         = false
}

# Create Cloud Function
resource "google_cloudfunctions_function" "function" {
  name    = var.function_name
  runtime = "nodejs12" # Switch to a different runtime if needed

  available_memory_mb   = 128
  source_archive_bucket = google_storage_bucket.bucket.name
  source_archive_object = google_storage_bucket_object.zip.name
  trigger_http          = true
  entry_point           = var.function_entry_point
}

# Create IAM entry so all users can invoke the function
resource "google_cloudfunctions_function_iam_member" "invoker" {
  project        = google_cloudfunctions_function.function.project
  region         = google_cloudfunctions_function.function.region
  cloud_function = google_cloudfunctions_function.function.name

  role   = "roles/cloudfunctions.invoker"
  member = "allUsers"
}

This file is dealing with all the logic of compressing the source code, storing it in a bucket, creating the Cloud Function and setting the necessary permissions to it.

Using your module

Now that you have your function module ready, you can invoke it in other parts of your Terraform code.

Go back to the entry point file on terraform/main.tf and add the following:

# terraform/main.tf

provider "google" {
  project = var.project
  region  = var.region
}

+ module "my_function" {
+   source               = "./modules/function"
+   project              = var.project
+   function_name        = "my-function"
+   function_entry_point = "app"
+ }

When running the file above, Terraform will look for a main.tf file on the path declared in the source parameter and run the code there along with the other variables.

Note: The function_entry_point must match the name of the exported variable in your Node.js code. You will find exports.app = … on the bottom of src/index.js.

In the terraform/outputs.tf file, add the return values from the module you want to use. Since the module only returns one output value, your file should look like this:

# terraform/outputs.tf

output "function_url" {
    # Access the module output with module.<module_name>.<output_name>
    value = module.my_function.function_url
}

Now let’s see how to deploy all the resources with the Terraform CLI.

5. Deploying

The hard work is already done! Creating the infrastructure should be an easier step.

Run the following commands on the root for your repository to create all the resources and deploy your code:

# Make sure you are on the terraform folder
cd terraform

# Initialize your configuration
terraform init

# Plan the configuration
terraform plan

# Create all the resources
terraform apply

If everything works well, you will see a similar output in your terminal:

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Outputs:

function_url = https://us-central1-my-project-1234567.cloudfunctions.net/my-function

You can verify that it works with a simple curl command. Remember to replace the URL with your own URL.

curl https://us-central1-my-project-1234567.cloudfunctions.net/my-function
{"works":true}

Updating the function

Your first deploy is never final. Eventually, you will want to deploy new versions of the code that runs in the Cloud Function.

After changing and testing your code, you can simply run terraform apply in your terminal. Terraform will compress your source files, store them in the Cloud Storage bucket and update the function with the new code.

Destroying the function

You can clean up all the resources created by running terraform destroy.

The project won’t be deleted this way (it wasn’t created by Terraform). For that, you can run:

# Delete the project
gcloud projects delete PROJECT_ID

6. Going further

This tutorial provides a quick way to get started. Many other good practices can be incorporated to build a more robust application:

Remote Terraform backend. If you check your repository you will notice that a state file was created by Terraform. It’s a good practice to store this file in a remote storage. You can change the backend from "local" to a Cloud Storage bucket, for example. See the list of available backends here.

Multiple environments. You might want to deploy the same infrastructure here under a different environment (e.g. development & production). There are many ways to do it with Terraform and you will find lots of tutorials around.

Continuous deployment. Ideally, you shouldn’t be running terraform plan and terraform apply from your local machine. This should be done as part of the automation process of a CI/CD solution such as Cloud Build or GitHub Actions.

This whole tutorial, plus some of the things are implemented on this repository on GitHub. Check it out!

What You Need To Know About npm Workspaces

Ruan Martinelli — Mon, 23 Nov 2020 08:43:19 +0000

The newest major release of npm came out in October this year. Together with it came one very anticipated feature: npm Workspaces.

Workspaces are a way to work with repositories that have multiple packages - more than one package.json file. These projects are also known as monorepos.

But npm is a bit late to the party. Managing monorepos is already possible with other package managers such as Yarn and pnpm. Libraries like Lerna also bring swift tools to work with multi-package repositories.

That being said, npm Workspaces are still a step in the right direction.

What changes with Workspaces?

If you have a single package.json on your repository, you will not need Workspaces and nothing is going to change for you.

In multi-package repository, npm will now scan your folders looking for other packages and dependencies to install. This was made possible after changes made to Arborist, npm's dependency tree manager.

Duplicated dependencies across packages will be hoisted. This means they are going to be stored on the top-level of the packages. This is done mainly for performance reasons - we all know how big node_modules can get.

Using Workspaces

You can try Workspaces today by updating your npm to version 7. To update, run this command on your terminal:

npm install -g npm@7

Version 7 is still not marked as latest, so running npm install -g npm will still install 6.x.

If you install Node.js 15 today, it should already come with npm 7.

You can create a minimal monorepo setup to play around. Here’s an example of how a simple repository structure would look like:

.
├── package.json
└── packages
    ├── package-a
    │   └── package.json
    └── package-b
        └── package.json

A package.json file on the root of your repository is still needed, even if you don't have any dependencies there. On that file you will tell npm where your subpackages live by adding the workspaces entry:

// ./package.json
{
  // ...
  "workspaces": [
     "./packages/*"
  ]
}

With this, when you run npm install on the root, npm will be smart enough to install package-a and package-b dependencies.

I made a GitHub repository with this example if you wanna check it in more details. Here’s the link: npm-workspaces-demo.

Are Workspaces ready?

“Should I change all my code to use npm Workspaces?”

Workspaces are definitely a welcome addition, but a bit more work is needed to make it a complete solution for managing monorepos.

As of the writing of this post, it still hasn't reached feature parity with Lerna, pnpm or Yarn Workspaces. Technical details about how Workspaces will work are still being voted and discussed on GitHub issues.

The toolset is still limited. There is no npm workspaces subcommand or anything equivalent. If you want to be an early adopter, you’ll need to combine it with tools like nx or Lerna for a complete monorepo workflow.

npm Workspaces vs. Yarn Workspaces

Yarn is the second biggest package manager for JavaScript, so it might be fair to make a comparison.

Yarn Workspaces is around for much longer (it was launched somewhere around 2017). It is a complete feature. The yarn workspaces interface already gives you the tooling that npm is still missing. npm Workspaces is still an MVP.

My personal opinion is that npm Workspaces shouldn't differ that much from existing solutions like Yarn Workspaces - it even borrowed the "Workspaces" name! I believe we can expect a similar API and an easy way to switch between other implementations (but again, that's an opinion).

What’s next for npm Workspaces?

We can expect new things to come for Workspaces. It will not be just an improvement to the npm install command (source).

On GitHub discussions you can see that Workspaces will evolve in a way to bring tools for a more complete workflow.

If you want to keep up-to-date with Workspaces development and new features, there are a few things you can do:

Watch the ongoing Workspaces RFC discussions on this repo;
Keep an eye on the npm Blog for new releases;
Subscribe to hyperfoo.io's newsletter, where I will publish a fresh new article about the next Workspaces iteration. :-)

Thanks for reading!

Cover background by Paweł Czerwiński on Unsplash

Introduction to Terraform

Ruan Martinelli — Fri, 20 Nov 2020 13:59:32 +0000

First, let’s talk about Infrastructure as Code (IaC).

Like the name suggests, Infrastructure as Code is the process of provisioning infrastructure using code. It is a way to automate the creation of the topology needed to run your application.

Think of Infrastructure as a broad term. It can be anything from virtual machines to Kubernetes clusters to storage buckets. Code in this context refers to the language used for writing the automation. The most common ones are JSON, YAML and HashiCorp Language Format (HCL).

IaC: the good parts

Automating your infrastructure has great advantages. Some of my favorite are:

Consistency. Consistency is the name of the game on IaC. Ad-hoc changes to infrastructure can cause a mismatch between development, staging and production environments.

Dynamic provisioning. Infrastructure is transient. When it’s automated, it becomes easier to provision and deprovision resources according to load.

Reproducible environments. Spinning up a sandbox for tests with a certain configuration usually takes a few minutes and has very little human intervention.

Documented infrastructure. The IaC code works as an always up-to-date documentation of your infrastructure. Tools like Git will also give you an audit trail and bring collaboration aspects to it.

Meet Terraform

Terraform is an open source command line tool by HashiCorp.

It gives you the power to manage - build, update and destroy - infrastructure using IaC principles.

Terraform uses the HCL language. With HCL, you declare a blueprint of how you want your infrastructure to look like. Terraform will read that blueprint and provision all the resources you declared.

This is how an HCL file looks like:

provider "aws" {
  version = "~> 3.0"
  region  = "us-east-1"
}

resource "aws_s3_bucket" "bucket" {
  bucket = "my-private-bucket"
  acl    = "private"

  versioning {
    enabled = true
  }
}

1. Example HCL file.

In the example above we are informing Terraform that Amazon Web Services (AWS) is our provider of choice. We are also declaring a resource: a private S3 bucket.

Providers

One very important concept from Terraform are providers.

Providers are the glue between the Terraform CLI and other systems. They work like plugins and live separately from Terraform.

With providers, Terraform is able to connect to cloud vendors (e.g. Google Cloud Platform, AWS), PaaS solutions (e.g. Heroku, Kubernetes) and SaaS applications (e.g. Fastly, GitHub).

State

Terraform keeps track of your infrastructure though a state file.

The state file is the source of truth for your architectural elements. It contains all the resources created by Terraform, plus any metadata needed to make Terraform work.

When you update your configuration - your HCL files - you are declaring a new desired state.

Terraform’s job is to turn that desired state into real-world. It is going to match your current state - state file - and your desired state - your configuration - and come up with a plan of execution.

The plan contains information about what needs to be created, updated or deleted on the infrastructure. Dependencies between resources are solved using a dependency graph. With everything in place, Terraform will then call all the APIs necessary to build and manage all the resources.

Example

Let’s go back to our first example.

Imagine we change the bucket ACL from private to public-read. Here’s what Terraform would do:

Compares the current state with the desired state;
Sees that the ACL of the bucket has changed;
Creates a plan - 1 resource needs to be updated;
Calls the AWS S3 endpoints to update the bucket ACL.

Will it simply update the ACL to public-read? Will it delete the bucket and create a new one?

That depends on which type of resource is modified. In general, Terraform is smart enough to only destroy and recreate resources when absolutely needed.

Is Terraform a good choice for my company?

..or startup, or side project?

Terraform does bring an additional complexity to your workflow. HCL is a new language to learn. The command line tool is still fairly new compared to other IaC solutions and it still shows a few rough edges.

That being said, once you master Terraform, it becomes an indispensable tool. Managing infrastructure with code can actually feel easier and safer than poking around on buttons in a cloud provider dashboard.

The variety of providers on the Terraform Registry also shows how versatile it is. Once you learn it, you can apply the same knowledge on any cloud provider, service, or pizza delivery chain.

That’s it for an introduction, thanks for reading!

This post was originally published on HYPERFOO, go there for more!

Cover background by Paweł Czerwiński on Unsplash

DEV Community: Ruan Martinelli

A Complete Guide to Buffers in Node.js

Working with buffers

Creating buffers

Writing to buffers

Iterating over buffers

Going further: Buffers and TypedArrays

Conclusion

Authentication with Supabase and React

Setting up Supabase

Setting up the project

Setting up the Supabase Client Library

Create authentication pages

Routing components with React Router

Adding the authentication logic

Adding authentication to the components

Protecting routes

Final result

Going further

Deploying to Google Cloud Run with Terraform

Prerequisites

Initial setup

Creating your first service

Invoking the service

Deploying the infrastructure

Updating the service

Cleaning up

How to Use Schemas on Fastify for Fun and Profit

Schemas

1. Validation

2. Serialization

3. Documentation

4. Mocking

5. Jest

Putting it all together

Honorable mention: fluent-schema

Deploying a Node.js API to Cloud Functions with Terraform

Prerequisites

1. Setting up the GCP account

2. Structuring the project

3. Writing the API

4. Writing the infrastructure code

Creating the function module

Using your module

5. Deploying

Updating the function

Destroying the function

6. Going further

What You Need To Know About npm Workspaces

What changes with Workspaces?

Using Workspaces

Are Workspaces ready?

npm Workspaces vs. Yarn Workspaces

What’s next for npm Workspaces?

Introduction to Terraform

IaC: the good parts

Meet Terraform

Providers

State

Example

Is Terraform a good choice for my company?

Creating the `function` module