DEV Community: Chris Castle

Render, Remix, and Strapi: Let's Build a Productivity Tips App

Chris Castle — Mon, 04 Apr 2022 19:55:43 +0000

What are Strapi and Remix?

Strapi is "the leading open-source headless CMS. It's 100% JavaScript, fully customizable, and developer-first." Think of it as an admin GUI to manage your content. That content could be blog posts or product images or videos. And Strapi automatically generates both REST and GraphQL APIs for the content. Render has been an especially popular place to deploy Strapi because it provides one of the quickest paths to get Strapi running in a production environment.

Remix is "a full stack web framework that lets you focus on the user interface...to deliver a fast, slick, and resilient user experience." It comes from some well-known developers from the React ecosystem and has been getting more and more attention since its open source debut at the end of 2021.

Why Strapi and Remix Together?

Strapi and Remix are both popular and growing open source JavaScript projects. In fact, they topped the list of Node.js Framework Rising Stars in 2021.

But more importantly, these two tools complement each other well. Strapi serves as an ideal back end for a Remix front-end. Strapi gives us many things we, as developers, would otherwise have to build ourselves: a clean abstraction on top of a database, a data admin GUI, and an API for the front-end to use. Remix builds upon the vast React ecosystem, letting you quickly create a front-end and scale it up as needed. It addresses a lot of the time-consuming complexity that has become a regular part of a React developer’s work.

A Productivity Tips App

The inspiration for this app came from Render Engineer Shantanu Joshi: "Over time, everyone develops a Swiss army knife of tips, tricks, and hacks to boost productivity. At Render, we created a #productivity-tips Slack channel for anyone to share their best productivity boosters with everyone on the team."

But Slack doesn't provide a great way to discover tips that others have shared in the channel. You can scroll back through the conversation history, but wouldn't it be nice if there was a web page with a simple list? Using Strapi and Remix, we're going to make a web app to catalog all of these tips and share them with others.

Here's a visual overview of what we'll build.

And here are the steps we’ll take to get there:

Set up your development machine
Configure and get familiar with Strapi
Build the front-end with Remix
Deploy everything to Render

Set Up Your Development Machine

You’ll need Node.js, Yarn, git, a Render account, and a Cloudinary account. If you don't have them already, install the tools and create the accounts linked above (everything's free).

Configure and Get Familiar with Strapi

We'll start with an example repository that contains only a few changes from the default Strapi project scaffolding. Following the instructions in the Set up Strapi section of the example repository's README, run Strapi locally to get comfortable with Strapi and understand the minor code changes below.

Here are the changes I made to the Strapi scaffolding. The links will take you to the code for each change.

Use PostgreSQL in production and SQLite for local development.
Use Cloudinary for media storage, resizing, and serving in production, and use the filesystem during local development. No code changes were necessary to use the local filesystem because that is the default built into Strapi.
Tell Strapi what its public URL is when deployed in production.
Create the Tip content type.

Note: Strapi requires you to define Content Types–effectively, the data schema–in code. When you deploy your Strapi service to production, those Content Types are deployed along with the Strapi application code. However, content for each Content Type only lives within the environment in which it's created. You can read more about this in the Strapi documentation.

In our example, we've created a Tip Content Type in development, and we will deploy that to production. However, any content we create in development won't be deployed to production, and any content we create in production won't be available in development.

Once you've completed the instructions to set up Strapi, including adding some sample Tip content in the Strapi Admin GUI, we'll move on to building the Remix service.

Build the Front-End with Remix

Strapi is mostly plug-and-play, requiring you to write code only when you want to customize its default functionality. On the other hand, building a Remix app requires us to write more code. We can start with a Remix scaffolding, but we still need to create the web page views and API requests to to show our productivity tip data. I've broken the process into five steps to guide you through this code. Each step is a separate git branch in the example repository, and the README for each branch explains the changes in that branch.

You can either start with the step-1 branch and work your way through the changes in each branch or skip to step-5, which contains the final example code that we'll deploy to Render. To start at step-1 and work through the branches, follow the README instructions in each branch.

To skip to step-5, fork the example repository, git clone it, switch to the step-5 branch with git checkout step-5, and then install the necessary dependencies with npm install. Now we can connect the Remix app to Strapi's REST API. To do that,

Copy the .env.example file to .env. This file provides our Remix app with two variables it needs to access the Strapi API: STRAPI_URL_BASE and STRAPI_API_TOKEN.
We need to ask Strapi to generate an API token for use in development. In the Strapi Admin GUI, go to Settings --> API Tokens --> Create new API Token. Name it something like Remix Dev and leave it as Read-only.
Copy the token and paste it into .env as the value for STRAPI_API_TOKEN.
Start or restart the Remix dev server by running npm run dev in the Remix app directory, and also make sure the Strapi server is running by running yarn develop in the Strapi app directory.
If you open http://localhost:3000 in a browser, you should see your Productivity Tips app containing any Tip data you've created in the Strapi Admin GUI!

Nice job! Local development is done! Time to deploy our code to Render.

Deploy Everything to Render

We will use Render Blueprints to make deploying and connecting these two services faster and less error-prone. A Render Blueprint is defined using a single render.yaml file. That file lives in the Strapi repository for these two applications.

Follow the instructions in the Deploy to Production on Render section of the Strapi repository's README. You'll make minor changes to the render.yaml file to make sure it deploys the code from your copies of the example repositories. Then you’ll perform the initial deploy, generate a production Strapi API token, and update the deployed production Remix service with that API token.

Now you have your very own Productivity Tips app with which you and your coworkers can collect and share all your best productivity tips! Start adding tips using the Strapi Admin GUI, and then share the Remix service's URL with your team.

Customize

I encourage you to extend the Remix and Strapi applications to customize them for your needs! Here are some ideas for inspiration.

Add a search bar so your coworkers can search for tips about a specific topic or tool. Algolia’s React InstantSearch library might be useful for this.
Add a login page to restrict use of the Remix app to only approved users. Use OAuth with Google or GitHub login so you don’t have to write code to manage user accounts and password resets. Okta has guides for adding Google, GitHub, and many other login providers to a web application. If you don’t want to use a 3rd party login provider, you could store and manage users directly in Strapi.
Make the screenshot thumbnails for each productivity tip clickable and zoomable. Strapi’s Cloudinary plugin that we're using automatically creates small, medium, and large versions of images you attach to a Tip, but I didn’t have time to add more than the thumbnail view to the Remix app.
Replace Pico.css with a different minimal CSS theme, or even add your own styling to match your company or team’s brand colors.

Let me know if you implement any of these! If they would be useful to others, I’ll consider merging them into the example repositories!

Remote Debugging with SSH and VS Code

Chris Castle — Mon, 28 Feb 2022 19:48:15 +0000

I was pretty excited when I discovered Telnet back in the 90s. I could access a command line on servers all over the college campus from my tiny dorm room! I no longer had to trudge through feet of snow and frigid temperatures (I went to college in the U.S. state of Maine) only to find all the computers in the lab occupied. I could connect to them from my room -- even if someone else was already using them!

A more advanced version of me is now shocked at how pervasive the unencrypted Telnet protocol was in the 90s. Back then, I didn't even consider encrypting terminal traffic. Fortunately, others did. After discovering a password sniffer had been used on his university’s network, Finnish student Tatu Ylönen created the first version of the Secure Shell Protocol (or SSH) in 1995, and its use has grown continuously since.

SSH is powerful because it can be used not only to get a command line on a remote computer but also to securely encapsulate almost any kind of data flowing between two machines. It can tunnel pixels from a remote machine's X11 server, sync files with rsync, and even mount a filesystem from a remote machine using SSHFS. SSH has understandably become an essential tool in the software developer's toolkit.

We recently announced the ability to SSH into your Render services, so I was excited to see how I could use it to improve my development workflow.

Why SSH?

We primarily built SSH on Render to allow you to get a command line in containers running a service you’ve deployed. We don’t recommend using it in production regularly, but it can be helpful in a pinch, to run a database migration after a deployment or to check the contents of a file being written to disk.

But there’s a lot more you can do with SSH. For example, you can use scp to copy files to or from your service’s persistent disk. For my own needs, I was curious if I could use Render SSH for port forwarding and eventually remote debugging.

Some of the most frustrating bugs I’ve encountered are those I can’t reproduce locally. I get a bug report from a user along with instructions to reproduce it. I can reproduce the bug in production (running Linux) but surprisingly can’t reproduce it on my Mac. How do I investigate and fix a bug if I can’t reproduce it on my development machine?

Debug Node.js Remotely

Remote debugging to the rescue! Here’s the code for a simplified Node.js HTTP server I will debug remotely.

const http = require('http')

const port = process.env.PORT || '10000'
let host = '127.0.0.1'

if (process.env.NODE_ENV === 'production') {
  console.log('NODE_ENV is set to "production"')
  host = '0.0.0.0'
}

http.createServer((req, res) => {
  console.log('request', req.url)

  if (req.url === '/') {
    res.writeHead(200, { 'Content-Type': 'text/html' })
    res.end('Hello, World!', 'utf-8')
  } else if (req.url === '/exception') {
    try{
      // Some code that causes an exception
      null.toString()
    } catch(e) {
      console.log(e)
      res.writeHead(500, { 'Content-Type': 'text/html' })
      res.end(`An error occurred: ${e.message}`)
    }
  }
}).listen(port, host)
console.log(`Server running at http://${host}:${port}`)

As a refresher, to attach a debugger to the node process if it were running locally, I’d start node with the --inspect flag and then attach VS Code (or one of several other debugger clients) to the process.

$ node --inspect server.js
Debugger listening on ws://127.0.0.1:9229/a9ca58f7-e847-4c31-9e49-82f10ee3a5f5
For help, see: https://nodejs.org/en/docs/inspector
Server running at http://127.0.0.1:10000

Now VS Code would show me variable values and the call stack at any point during the program’s execution.

But getting this working when the node process is running on Render is not as straightforward. Render accepts traffic from the public internet for only one port for a Web Service. My Web Service is already listening on port 10000 for HTTP traffic, so I can’t expose the debug port publicly. Moreover, even when Render starts supporting traffic on multiple ports, exposing the debug port publicly is unsafe! Fortunately, the Node.js documentation provides guidance on enabling remote debugging safely.

Configure Render

The first step is to edit the Start Command for the service from the Render Dashboard. Instead of running node server.js, I have to run node --inspect=9229 server.js. 9229 is the default port the Node.js debugger will listen on, but I want to be explicit to avoid confusion. Then I add an SSH key to my Render account. This only needs to be done once for each Render account.

While the service restarts with the new start command, I’m going to configure VS Code.

Configure VS Code

VS Code requires a launch.json configuration file to connect to the node process as a debug client. To create this, open .vscode/launch.json (create this folder and file if needed) and click Add Configuration… in the bottom right.

Configuring VS Code to remotely debug a Node.js process

Select Node.js: Attach to Remote Program, and VS Code will generate a template with some values that need to be filled in. The address and port are where VS Code will try to connect to the node process, and the remoteRoot is the directory to which code is deployed on Render. This path won’t change if you’re using the native Render Node environment instead of Docker. Here is the launch.json I used.

{
  "version": "0.2.0",
  "configurations": [
    {
      "address": "localhost",
      "localRoot": "${workspaceFolder}",
      "name": "Attach to Remote",
      "port": 9229,
      "remoteRoot": "/opt/render/project/src",
      "request": "attach",
      "skipFiles": [
        "<node_internals>/**"
      ],
      "type": "pwa-node"
    },
  ]
}

Create the SSH Tunnel

Now that the Web Service is using the new Start Command, I can create an SSH tunnel between the node process running on Render and my local development machine. First I grab the ssh command for my Web Service from the Render Dahsboard.

The ssh command for a Web Service

I need to modify it so that ssh creates a tunnel between two ports instead of giving me a command line.

ssh -N -L 9229:localhost:9229 srv-c804oa46fj3d2u40bpq0@ssh.oregon.render.com

The -N flag tells ssh to not execute a remote command after establishing the connection. Normally when you ssh into a machine, a command like bash¹ is run. I don’t need that because I only want ssh to forward TCP traffic back and forth.
The -L flag instructs ssh to create a tunnel for TCP traffic such that connections to port 9229 on my local machine are forwarded to localhost:9229 on the remote machine.
Finally, srv-c804oa46fj3d2u40bpq0@ssh.oregon.render.com is the username and domain name ssh will use to create the connection.

Start Debugging

Now I can start the debugging session! In VS Code I’ve opened the project deployed to Render, switched the left side panel to Run and Debug, and then clicked the play button at the top to start debugging.

VS Code connected as a debug client to a remote node process

Let’s walk through everything I can see and do now. On the bottom, a Debug Console appeared, which now shows me the logs from my service (stdout and stderr). On the left within the Loaded Scripts panel, I select the server.js file so that I can add breakpoints and see which line program execution is on.

To confirm that remote debugging is working the same way local debugging does in VS Code, I add a breakpoint on line 15.

Add a breakpoint to line 15

This is part of the code that is executed when a request to / is made. I make a request to that path in a new browser tab, and it looks like everything is working as expected! I can see that execution paused on line 15, and VS Code shows me the state of the req and res local variables and the call stack.

Program execution paused at the breakpoint

Earlier in this post, I said that remote debugging is helpful to investigate an issue observed in production but not on my local development machine. Let’s say the bug causes an exception, but I don’t know which line of code is generating the exception. I can tell VS Code to automatically pause program execution when it hits both caught and uncaught exceptions.

Break on caught and uncaught exceptions

To test this out, I make a request to /exception. VS Code automatically pauses execution on the line that generates the exception. This example is contrived because I purposely wrote this line to generate an exception. Still, it’s a simple demonstration of how you can remotely debug exceptions before you have to handle a real-life (and more complex) scenario.

VS Code pausing program execution on a caught exception

More to Explore

VS Code’s remote debugging feature can also help if you need to investigate CPU or memory use. The Flame Chart Visualizer for JavaScript extension provides a real-time chart of CPU and memory use of the node process and a flame chart to help you identify CPU hotspots in your code.

Example flame chart

And you can use VS Code to debug more than just Node.js remotely: use it to connect to an Elixir cluster, a Go process, a Ruby process, and much more. There are VS Code extensions available for all popular languages that enable remote debugging for that language.

What are you going to use Render SSH for? Let me know in a comment below!

The shell that’s run is configured per user on the remote machine. You can also override the command by appending a command to the end of the ssh command. ↩

Add Password Protection to Any Site with OAuth2 Proxy - Plus Social Logins

Chris Castle — Mon, 14 Feb 2022 19:51:52 +0000

Continuing on the topic of adding password protection to your site, let’s look at using a reverse proxy server to password protect a service you have deployed to Render. The previous post on this topic covered adding password protection to a static site using PageCrypt. Now we’re expanding beyond static sites.

What’s the problem we’re trying to solve? Say you have an existing Ruby, Node.js, or Python web service (it could be one of our native environments or a Dockerized app), and you want to protect it with password authentication. Maybe you don’t want to touch the existing service’s code because it works well, and you’re concerned about inadvertently introducing a bug. Or perhaps you have multiple services to protect, and you don’t have time to implement password protection for each of them.

An interesting way to solve this problem is to put a reverse proxy in front of your existing web service(s).

What’s a reverse proxy server? It’s a server that acts as a gate and traffic cop guiding incoming traffic to the service that can respond to it. Typically that service is not exposed to the public internet. The scenario we’ll look at below shuttles web traffic between the public internet and a Render Private Service, which is protected from the public internet and only accessible to applications you own. NGINX and Apache are examples of general-purpose web servers that can also be used as reverse proxies, but there are many purpose-built ones like Træfɪk and Caddy.

Reverse proxies are commonly used for load balancing or TLS termination but Render already provides these features to you out of the box. In our scenario, we will deploy a reverse proxy specifically for authentication and authorization.

The reverse proxy we’re going to focus on in this post is called oauth2-proxy.

What is it good for?

Adding social login to your site – i.e., login managed by another service like GitHub, Google, or one of the other fourteen supported OAuth providers
Cases in which you can’t or don’t want to change the code of the web service you want to protect
Maintaining clear separation between auth code and your web service code
Language flexibility: your web service doesn’t need to be written in the same language as oauth2-proxy (which is written in Go).

What are the drawbacks?

Deployed as an extra service, which has a cost (but you can use Render’s free plan)
Configuration of OAuth can be confusing, but the oauth2-proxy documentation explains it well

What is it, and how does it work?

A coworker shared oauth2-proxy as part of a previous Render blog post, and I think it’s pretty cool. It does two things. First, it authenticates a user with an OAuth 2.0 flow. Then, after successful authentication, it acts as a reverse proxy, forwarding web requests to a Private Service behind it. The Private Service shouldn’t be exposed to the public internet, otherwise, the user will be able to access it without authenticating.

Are you wondering what OAuth 2.0 is, or do you need a refresher on it? (If not, you can skip to the diagram.) OAuth 2.0 is a standard defined by the IETF OAuth Working Group. It’s a ubiquitous authorization method that lets you delegate authentication and often authorization to another trusted service. For example, you can use Google to log in to the Render Dashboard.

OAuth 2.0 specifies a multi-step, production-hardened flow we can follow to allow Google to authenticate a user’s credentials and securely tell us that the user is, in fact, the owner of a specific email address, say, user@example.com.¹ Armed with the fact that we confidently know this is user@example.com (and not sneakyblackhat@example.com pretending to be user@example.com), we can then show the user the resources they have access to — e.g., a web page, some data, or their daily New York Times crossword puzzle.

Quick detour: You may have noticed I’ve used two different words that begin with auth: authentication and authorization. People often confuse them or mean both when they say one. So what’s the difference? Authentication verifies that a user is who they say they are. Authorization determines the user’s permissions within your app. In the example above, if a user logs in to Render with Google credentials, Google handles the authentication, assuring us the user is who they say they are. After the user is authenticated, Render needs to decide what services and teams to show them. We look up the services and teams they are authorized to manage — those created by them or shared with them by another Render user.

Traffic Flow

So how does oauth2-proxy work? Let’s look at the flow from the perspective of a user trying to access your app. Note that this flow is specific to oauth2-proxy, not a generic OAuth 2.0 flow.

The browser makes an HTTP request to your site, which is directed to the oauth2-proxy service.
oauth2-proxy responds, prompting the user to authenticate themselves with an OAuth 2.0 provider you’ve configured. In this example, we’ll assume it’s Google.²
The browser is redirected to a Google login page.
The user enters their credentials for Google to verify.
Upon successful authentication, Google redirects the user back to your oauth2-proxy service along with a unique token — a query string parameter that Google refers to as an ID token.
oauth2-proxy makes an HTTP request to Google containing the ID token along with a client ID and client secret that Google has uniquely assigned to your instance of oauth2-proxy. This step is a necessary part of the OAuth flow. A malicious user could spoof the request with a made-up ID token. This out-of-band request between oauth2-proxy and Google verifies the ID token was generated by Google from a recent login for this user.
Google responds, verifying the validity of those three values and finally confirming to oauth2-proxy the user is who they say they are!
By default, oauth2-proxy now authorizes all traffic from this user passing it to your Private Service. However, you can configure oauth2-proxy’s authorization rules in several ways. You can restrict access based on membership in a Google Group, GitHub org, or GitLab project. oauth2-proxy can also pass the username to the Private Service which can implement its own authorization logic.

Implementation

I forked the oauth2-proxy repository and made a few changes so that you can deploy a working example to Render for free.

Before the deployment starts, you’ll be asked for three values: an OAuth provider, client ID, and secret. The oauth2-proxy documentation explains how to generate a client ID and client secret using any one of fourteen supported OAuth providers.

I’ve chosen Google as the OAuth provider for an example deployment: try it out. I'm using oauth2-proxy's default login screen, but you can customize its design. You’ll be prompted to Sign in with Google. After Google authenticates your credentials (which aren’t shared with me or Render), your requests will be proxied to a Node.js service deployed to Render as a Private Service. Private Services on Render are protected from the public internet and only accessible to applications you own. The oauth2-proxy service receives a request from your browser, passes it to the Node.js service, and then passes the Node.js service’s response back to your browser.

More to Explore

If oauth2-proxy doesn't suit your needs, there are some projects that have spun-off from oauth2-proxy like pomerium and BuzzFeed's sso. In addition to the open source library, Pomerium offers a paid service with a GUI to help IT staff more easily manage user permissions. BuzzFeed's sso builds upon oauth2-proxy by separating the domain used for auth from the domain used for the proxy (among several other changes).

If you're interestd in learning more about OAuth, there is ongoing work to evolve the OAuth 2.0 standard. OAuth 2.1 exists in a draft state, and GNAP is a new IETF Working Group developing "a next-generation protocol that encompasses many use cases that are challenging to implement with OAuth 2.0."

Are you using oauth2-proxy or another reverse proxy with password protection? Let me know what worked and didn't work for you! I'm curious to learn more, and I'm sure others will find it helpful to hear about your experience.

It doesn't necessarily have to be an email. It could be a username or a user ID or any other identifier. ↩
You could even deploy your own! ↩

Password Protect Static Sites with PageCrypt

Chris Castle — Wed, 02 Feb 2022 20:25:53 +0000

Password protecting static sites is tricky. You could use window.prompt() to ask a site visitor to enter a password before the site content is revealed, but a resourceful visitor will quickly get around this with right-click, view source (or curl or numerous other ways). All of the site's HTML, CSS, and JavaScript are served to the web browser, and they constitute the entirety of the site.

If you are using a backend API that the static site pulls data from (like many single-page apps do these days), you could require authentication for API requests, but this doesn't protect the HTML, CSS, and JavaScript.

In trying to find a good solution for this for Render users, we discovered PageCrypt. It's a free, open source library that allows you to password protect these static assets securely. Let's investigate how PageCrypt works and test it out!

What's it good for?

Protecting a static site
Cases where you don't need (or can't run) backend code

What are the drawbacks?

Only encrypts a single HTML file by default
Only supports a single shared password (no per-user passwords)

What is it, and how does it work?

PageCrypt is a novel solution to password protecting HTML without a backend. It’s a library you can use as part of your site’s build step or as a command line tool. It uses the Web Crypto API -- currently supported by all major browsers -- and a password to encrypt an HTML page, which you can then host on any static hosting platform, including Render! An HTML page encrypted with PageCrypt prompts the viewer for a password. Upon entering the correct password, the page is decrypted and its content replaces the password prompt.

Example static site password protected with PageCrypt

One potential concern with PageCrypt is that it only encrypts an HTML file by default. If you want to encrypt your CSS and JavaScript files, you’ll have to inline them in the HTML file. The same applies to images and any other binary assets; you’ll have to inline them as Data URIs. As with any authentication and authorization solution, you’ll want to determine what’s acceptable for your security requirements. Maybe you’re comfortable with the risk of images leaking but have higher security requirements for your JavaScript. In that case, the HTML page can link to the image files but should contain all your JavaScript. You can use many static site build tools to automate inlining assets in HTML. Webpack, Gulp, or Grunt are just a few that might be useful.

PageCrypt also doesn't allow users to have individual usernames and passwords. It only works with a single, shared password. If you need the more fine-grained control provided by user accounts, check out a service like Auth0.

Try it out

Adding PageCrypt to the build step of a Hello World static site was straightforward. The instructions in the project’s README provide clear guidance on how to use PageCrypt as a library with browser-executed JavaScript, Node.js, or Deno, and how to use it as a CLI tool. I used the CLI in the build step of my example static site.

Add PageCrypt to your project as a dependency:

$ yarn add pagecrypt

Then update the build step in package.json to use the CLI:

{
...<snip>...
  "scripts": {
    "build": "pagecrypt src/index.html dist/index.html $PASSWORD && cp -R src/css dist/"
  },
...<snip>...
}

The password is set using the $PASSWORD environment variable since we don’t want to store that in the code. Using an environment variable also allows you to change the password and rebuild the site quickly.

Here’s an example deployment of the site. The password is s3cr3t.

To get a deeper understanding of how PageCrypt works, try entering an incorrect password. Then right-click and view the source of the page. PageCrypt generated the contents of this page during the build step. Your original page content is encrypted inside a hidden <pre> element at the bottom of the HTML document.

Encrypted HTML page inside the <pre> element.

After the correct password is entered, your page content is decrypted and shown.

To make it easier for users to access password protected pages, PageCrypt also allows you to create a "magic link" that decrypts the page without prompting the user for a password. The format for the magic link is https://<link-to-your-page>#<password>, placing the password in a URI fragment. Check out the project's README section about magic links to better understand the security implications. When your browser goes to a URL containing a URI fragment, the fragment isn't sent across the internet, but it does remain in the browser's history.

Extend

It would be interesting to extend PageCrypt to do a number of things:

Automate the inlining of CSS, JavaScript, and image files
Encrypt multiple HTML files
Encrypt multiple files, including CSS, JavaScript, and image files

If you do end up extending it in your build process, let me know!

Explore

This version of PageCrypt is a rewrite of an older version of PageCrypt. That older version also inspired a few spin-offs that you might find useful:

Python CLI for PageCrypt
PowerShell CLI for PageCrypt
R-wrapper for PageCrypt
StatiCrypt - A separate but similar project

Now try it out yourself! You can deploy the code to Render for free. The README provides step-by-step instructions.

What's New in PostgreSQL 14?

Chris Castle — Tue, 05 Oct 2021 17:19:07 +0000

PostgreSQL 14, released on September 30, 2021, introduces many new features and improvements. Two I find particularly interesting are JSON syntax conveniences and the multirange data type. But there are also several noticeable performance improvements that many applications will benefit from without any code changes.

JSON Conveniences

PostgreSQL can store and manipulate JSON data. There are JSON and JSONB data types. They are similar, but most of the time you probably want to use JSONB. The JSON data type is stored as a string whereas JSONB is stored in a decomposed binary format that allows PostgreSQL to query and manipulate nested strucutres inside the JSON more efficiently. JSONB also allows you to create indexes based on nested values. I'll be using JSONB exclusively here.

Let's create a table and put some JSON data in it.

CREATE TABLE jsonb_example (
  id bigserial,
  data JSONB
);

INSERT INTO
  jsonb_example (data)
VALUES
  ('{"a": {"b": ["foo", "hello"]}}'),
  ('{"a": {"b": ["chris", "render"]}}');

Here's the table we just created.

SELECT * from jsonb_example;

 id |               data
----+-----------------------------------
  1 | {"a": {"b": ["foo", "hello"]}}
  2 | {"a": {"b": ["chris", "render"]}}
(2 rows)

Cool. But what if we want to query nested values within the JSON? Here's how you used to have to do it with PostgreSQL 13.

SELECT data #> '{a,b,1}'
FROM jsonb_example;

 ?column?
----------
 "hello"
 "render"
(2 rows)

The #> is an operator that tells PostgreSQL to return a JSON sub-object. PostgreSQL has a lot of other operators for working with JSON like #>>, ->, and ->>.

The {a,b,1} is a path into the JSON object that says

Grab the value of property a: {"b": ["foo", "hello"]}
Within that, give me the value of property b: ["foo", "hello"]
Within that array, give me the 2nd element: "hello"

With PostgreSQL 14 you can now access JSON sub-objects in a way that more closely matches how other programming languages reference JSON values. This new functionality is called jsonb subscripting. Here's an example returning the same data as above.

SELECT data['a']['b'][1]
FROM jsonb_example;

   data
----------
 "hello"
 "render"
(2 rows)

We can use the same syntax to filter data with a WHERE clause.

SELECT *
FROM jsonb_example
WHERE data['a']['b'][1] = '"render"';

 id |               data
----+-----------------------------------
  2 | {"a": {"b": ["chris", "render"]}}
(1 row)

And we can update values within the JSON object using the same syntax.

UPDATE jsonb_example SET data['a']['b'][0] = '"bar"';

SELECT *
FROM jsonb_example;

 id |              data
----+---------------------------------
  1 | {"a": {"b": ["bar", "hello"]}}
  2 | {"a": {"b": ["bar", "render"]}}
(2 rows)

This new syntax closely matches how you might access nested values in a JSON object with JavaScript.

const obj = {"a": {"b": ["foo", "hello"]}};
obj['a']['b'][1]; // evaluates to 'hello'

Multirange Data Type

PostgreSQL has had range data types since version 9.2. Now there are multirange data types.

A range data type is useful when you want a single value to represent a range of dates, time, or numbers. For example, you might use a range data type to represent meeting schedules or radio spectrum allocation. With a range data type, you only need a single column to store what would otherwise require two columns--start of range and end of range.

Prior to PostgreSQL 14, that range could only be a single contiguous range. Now with multiranges, multiple discontiguous (but not overlapping) ranges can be stored as a single value.

Let's look at a meeting scheduling example derived from the PostgreSQL documentation.

CREATE TABLE reservation (room int, during tsrange);

INSERT INTO reservation VALUES
    (1108, tsrange('2021-01-01 14:30', '2021-01-01 15:30'));

SELECT * FROM reservation;

 room |                    during
------+-----------------------------------------------
 1108 | ["2021-01-01 14:30:00","2021-01-01 15:30:00")

This row represents a room reservation for a meeting from 14:30 to 15:30 on January 1, 2021. But what if instead of a one hour meeting, we were planning an all-day meeting with a lunch break in the middle? What would a row for that reservation look like?

Prior to PostgreSQL 14's multirange support, it would have to be stored as two rows.

INSERT INTO reservation VALUES
    (1108, tsrange('2021-01-01 09:00', '2021-01-01 12:00')),
    (1108, tsrange('2021-01-01 13:00', '2021-01-01 17:00'));

SELECT * FROM reservation;

 room |                    during
------+-----------------------------------------------
 1108 | ["2021-01-01 09:00:00","2021-01-01 12:00:00")
 1108 | ["2021-01-01 13:00:00","2021-01-01 17:00:00")
(2 rows)

But then we'd need to add a meeting_id column so that we know these two rows represent the same meeting.

With multiranges we only need one row to represent this meeting.

DROP TABLE reservation;
CREATE TABLE reservation (room int, during tsmultirange);

INSERT INTO reservation VALUES
    (1108, tsmultirange(
      tsrange('2021-01-01 09:00', '2021-01-01 12:00'),
      tsrange('2021-01-01 13:00', '2021-01-01 17:00')
    ));

SELECT * FROM reservation;

 room |                                            during
------+-----------------------------------------------------------------------------------------------
 1108 | {["2021-01-01 09:00:00","2021-01-01 12:00:00"),["2021-01-01 13:00:00","2021-01-01 17:00:00")}
(1 row)

Improved Handling of Concurrent Connections

A lot has happened behind the scenes to improve PostgreSQL performance. However, PostgreSQL 14's handling of multiple client connections will likely be the performance improvement noticeable to the most people. The folks at pganalyze did some great performance testing using pgbench to verify this. At 5000 active connections, they saw about a 20% improvement in throughput over PostgreSQL 13. At 10,000 connections, that improvement went up to 50%. Of course, these results are dependent on your hardware, schema, data, and queries, so you may not see the same results, but the consensus seems to be that PostgreSQL's handling of concurrent connections has improved significantly.

And Much More

There are many more new features and improvements, which you can read about in the PostgreSQL 14 release announcement or compare across versions in the feature matrix (deselect all versions except 14 and the one you want to compare to and select "Hide unchanged features").

Render Supports PostgreSQL 14

I'd be remiss not to mention that Render was the first cloud provider I know to support PostgreSQL 14, making it available on the same day PostgreSQL 14 was released and giving our customers access to all these exciting improvements in just a few clicks. Try it out and see how it works for you.

Evolving Alongside your Tech Stack

Chris Castle — Thu, 30 Apr 2020 15:40:59 +0000

This blog post is adapted from a discussion during an episode of our podcast, Code[ish].

Over the last twenty years, software development has advanced so rapidly that it's possible to create amazing user experiences, powerful machine learning algorithms, and memory efficient applications with incredible ease. But as the capabilities tech provides has changed, so too have the requirements of individual developers morphed to encompass a variety of skills. Not only should you be writing efficient code; you need to understand how that code communicates with all the other systems involved and make it all work together.

In this post, we'll explore how you can stay on top of the changing software development landscape, without sacrificing your desires to learn or the success of your product.

User experience depends on technical expertise

When the iPhone first came out in 2007, it was rather limited in technical capabilities. There was no support for multitasking and gestures, no ability to copy and paste text, and there wasn't any support for third-party software. It's not that these ideas were not useful, it’s just that the first generation of the phone's hardware and operating system could not support such features. This serves as a good example to underscore how UX has sometimes been constrained by technology.

Now, the situation has changed somewhat. Tools have advanced to the point where it's really easy to create a desktop or mobile app which accepts a variety of gestures and inputs. The consequences of this are twofold. First, users have come to expect a certain level of quality in software. Gone are the days of simply "throwing something together"; software, websites, and mobile apps all need to look polished. This requires developers to have a high level of design sensibility (or work with someone else who does). Second, it means that the role of the engineer has expanded beyond just writing code. They need to understand why they're building whatever it is they're building, why it's important to their users, and how it functionally integrates with the rest of the app. If you design an API, for example, you’ll need to secure it against abuse; if you design a custom search index, you need to make sure users can actually find what they’re looking for.

On the one hand, because you're running on the same devices and platforms as your users (whether that a smartphone or an operating system), you're intricately familiar with the best UI patterns—how a button should operate, which transitions to make between screens—because every other app has made similar considerations. But on the other hand, you also need to deal with details such as memory management and CPU load to ensure the app is running optimally.

It’s not enough for an app to work well, as it must also look good. It's important to find a balance of both design sensibilities and technical limitations—or at least, a baseline knowledge of how everything works—in order to ship quality software.

Follow everything but only learn some things

When it comes to personal growth, learning to prioritize solutions to the problems you encounter can be critical in your development. For example, suppose you notice one day that your Postgres queries are executing slower than you would like. You should have a general awareness of how higher rates of traffic affects your database querying strategies, or how frequent writes affect the physical tables on disk. But that doesn't necessarily mean that you should sink a massive amount of time and effort to fine-tune these issues towards the most optimal strategy. When developing software, you will always have one of several choices to make, and rarely does one become the only true path forward. Sometimes, having the insight to know the trade-offs and accepting one sub-optimal approach above another makes it easier to cut losses and focus on the parts of your software which matter.

Sometimes, having the insight to know the trade-offs and accepting one sub-optimal approach above another makes it easier to cut losses and focus on the parts of your software which matter.

It seems like every year, a new web framework or programming language is released. This makes it difficult, if not impossible, to follow every single new item when they are announced. The inverse is also true. We might feel that adopting new technologies is one way to stay "relevant," but this attitude can be quite dangerous. If you are an early adopter, you run the risk of being on the hook for finding bugs, distracting you from your actual goal of shipping features for your own application. You should take a calculated approach to the pros and cons of any new tech. For example, switching your database entirely to MemSQL because you heard it's "faster" is less reasonable than making a switch after reading someone's careful evaluation of the technology, and realizing that it matched your own needs as well.

Keeping calm and steady

At the end of the day, you should be very invested in your own stack and the ecosystem you work in. That work can be something as simple as reading Medium posts or following Twitter accounts. Broaden your knowledge of other services outside your own realm of expertise only if you come across someone confronting problems similar to yours. You should own tools which you know how to operate, rather than keep a shed full of all sorts of shiny objects.

CLI Flags in Practice + How to Make Your Own CLI Command with oclif

Chris Castle — Thu, 09 May 2019 16:09:15 +0000

Editor's note: If you like CLIs, you should check out oclifconf taking place on Friday, May 31st, 2019 in San Francisco. It’s the first community get-together for oclif! Space is limited so let us know soon if you are interested in joining.

What is it that makes working from the command line so empowering? It can feel archaic at times, sure, but when you remember the right sequence of words, characters, and symbols for what you’re trying to do, it hits you with a sense of accomplishment and mastery over your tools that no graphical interface can compete with.

So what better way to continue your adventures as a developer than by developing your own CLI tool?

In this post, we’ll go over what type of parameters CLI commands take—also known as "flags", "arguments", and sometimes "options.” Then, we’ll start you off with oclif, the CLI framework that makes it easy to create new CLI commands!

The Syntax of a CLI Command

Any command line interface command has a few standard "parts of speech.” As a user of CLI tools, knowing these parts of speech can help you make fewer typos. It can also help you understand complex commands other people share with you more quickly (like these). If you are designing a CLI tool it is even more important to understand these parts of speech, so you can come up with the most ergonomic interface for your users. Yes, a CLI is a user interface!

Some of you may recognize diagrams like the one below from elementary or primary school. Fortunately, understanding how CLI commands are structured isn’t going to feel like this.

CLI commands are pretty straightforward compared to the typical English sentence.

For starters, let’s look at the parameters that appear to the right of CLI commands. Sure, there are many ways you can pass data to a CLI command, but these three types of parameters to the "right" of the command might be the most common: argument, long flag, and short flag. These two formats for flags are the standard for GNU-style flags. Not all CLIs follow this convention, but it has become the most popular style on Unix-like and POSIX compliant operating systems.

What better way for us to start than with the ls command? It’s one of the most common and simplest commands on Unix-like operating systems. It simply lists the contents of a directory.

Command

$ ls

This command, ls, works on its own, as a standalone command. Without any parameters, this command will list the contents of the current directory.

Argument

$ ls .

But you can do the same thing with an argument! Turns out that ls . and ls are the same thing, with ls simply using an implied . directory. For those that don’t remember or don’t know, . always refers to the current directory.

But now, the argument syntax makes it possible for you to pass any directory path to ls, and to get a look at what’s in there.

$ ls /home/casey/code/some-repo-name

An argument is anything to the right of a command that is not a flag (we’ll get to flags next). And fortunately, an argument can come before or after flags–it can coexist with them happily.

Long Flag

To list files that are normally hidden (like ~/.bashrc), you can use a flag on the ls command. ls --all is the long flag form. A long flag always uses a double dash, and it is always represented by multiple characters.

$ ls --all
$ ls . --all

Short Flag

There is also a short flag form of this flag: ls -a. The a is short for all in this case. A short flag always uses a single dash, and it is always represented by a single letter.

$ ls -a
$ ls . -a

Short flags can stack too, so you don't need a separate dash for each one. Order does not matter for these, unless passing a flag argument.

$ ls -la

Flag Argument

Many flags accept an option called a "flag argument" (not to be confused with a "command argument"). In general a command's parameters can be in any order, but flags that accept options must have the option directly after the flag. That way, the command doesn’t get confused by non-flag arguments.

For an example, here the -x flag does not accept an option but the -f flag does. archive.tar is the option being passed to -f. Both of these are valid.

$ tar -x -f archive.tar
$ tar -xf archive.tar

A flag and its option can be separated by a space or an equals sign =. Interestingly, short flags (but not long flags) can even skip the space, although many people find it much easier to read with the space or equals sign.These three are all valid and equivalent.

$ tar -f archive.tar
$ tar -f=archive.tar
$ tar -farchive.tar

Long flags must have a space or equals sign to separate the flag from its option.

$ git log --pretty=oneline
$ git log --pretty oneline

Other Ways of Passing Data

We've covered parameters, which are arguments, long flags, and short flags. There are two other ways to pass data to a command: environment variables ("env vars"), or standard input ("stdin"). These won't be covered in this blog post, but check out the links to learn more about them.

Building a New Command With `oclif`

Scenario: we want to design an oclif command that takes an input like "Casey", and returns "hi, Casey!". There are many ways the user could pass this in. Here we show an example of each type of input using an argument, a long flag, and a short flag.

First, let’s get started with oclif. Getting a CLI app going is very, very straightforward with it. Open up your terminal and type the following, which will use npx to run oclif and then create a new CLI. npx is a pretty useful command to simplify running CLI tools and other executables hosted on the npm registry.

$ npx oclif single greet-me

We won’t go into the details of the single (vs multi) argument above. Check out the oclif documentation for more about this.

Now, you’ll get the chance to specify some details of your new CLI, including the command name. When it asks you, just press enter, choosing the deafult. It’ll take the greet-me argument you passed into the above command. You can choose the default for most of the questions it asks you. The answers won't make much of a difference for this simple tutorial. However, they are very important to answer accurately if you will be sharing your CLI command with others.

? npm package name: greet-me
? command bin name the CLI will export: greet-me

...

Created greet-me in /home/casey/code/greet-me

Now that we have things set up, let’s check out what’s happening in /greet-me/src/index.ts, where all the important argument and flag-handling code for your CLI will live.

const {Command, flags} = require('@oclif/command')

class GreetMeCommand extends Command {
  async run() {
    const {flags} = this.parse(GreetMeCommand)
    const name = flags.name || 'world'
    this.log(`hello ${name} from ./src/index.js`)
  }
}

GreetMeCommand.description = `Describe the command here
...
Extra documentation goes here
`

GreetMeCommand.flags = {
  // add --version flag to show CLI version
  version: flags.version({char: 'v'}),
  // add --help flag to show CLI version
  help: flags.help({char: 'h'}),
  name: flags.string({char: 'n', description: 'name to print'}),
  // flag with no value (-f, --force)
  force: flags.boolean({char: 'f'}),
}

module.exports = GreetMeCommand

What we can see here is that it accepts a few different flag names out the gate (version, name, help, and force) by registering them in the flags object.

{
    /* … */
    version: flags.version({char: 'v'}),
    /* … */
}

Here, with the version flag, the key acts as the ‘version’ long flag name, and on the right side of the expression, we use the method's in oclif’s flags module to register a flag, a type it’ll return, and the short flag name.

Now, we’re ready to rock: let’s see just how many things oclif handles out of the box by running the CLI. Right now, it’s only available with a slightly awkward command.

$ ./bin/run

But npm allows us to symlink this to the name of our CLI.

$ npm link

...

$ greet-me
> hello world from ./src/index.ts

Excellent! Try passing your name in using -n or --name next–and see if there are any other ways oclif will let you pass in arguments.

`SIGTERM`

While that’s all we’re going to cover in this blog post, oclif has a growing community and its code is open source so there are lots of other ways to learn more. Here are some links to continue exploring oclif.

An episode of the Code[ish] podcast about oclif with Jeff Dickey, one of the creators of oclif, and Nahid Samsami, Heroku’s PM for oclif

13. oclif: An Open Source CLI Framework

Code[ish]

Your browser does not support the audio element.
1x initializing... ×
💻🎙️ oclifconf details if you’ll be in the San Francisco Bay Area on Friday, May 31, 2019
👀🗣️ oclif’s Spectrum community
And finally, oclif’s GitHub repository
oclif
/
oclif

CLI for generating, building, and releasing oclif CLIs. Built by Salesforce.
oclif CLI
🗒 Description

This is the oclif CLI for the Open CLI Framework, that supports the development of oclif plugins and CLIs.

See the docs for more information.

🚀 Getting Started Tutorial

The Getting Started tutorial is a step-by-step guide to introduce you to oclif. If you have not developed anything in a command line before, this tutorial is a great place to get started.

See Usage below for an overview of the oclif CLI.

📌 Requirements

Currently, Node 18+ is supported. We support the LTS versions of Node. You can add the node package to your CLI to ensure users are running a specific version of Node.

📌 Migrating from V1

If you have been using…
View on GitHub

Beyond Web and Worker: Evolution of the Modern Web App on Heroku

Chris Castle — Tue, 14 Aug 2018 18:59:00 +0000

This is the first in a series of blog posts examining the evolution of web app architecture over the past 10 years. This post examines the forces that have driven the architectural changes and a high-level view of a new architecture. In future posts, we’ll zoom in to details of specific parts of the system.

The standard web application architecture suitable for many organizations has changed drastically in the past 10 years. Back in Heroku’s early days in 2008, a standard web application architecture consisted of a web process type to respond to HTTP requests, a database to persist data, and a worker process type plus Redis to manage a job queue.

Modern web app on Heroku in 2008

Now, in 2018, a website (or mobile app) is the primary way many companies interact with customers. Technology is inextricably woven into the fabric of almost every business. As technology is being tasked to do more for a business, we as developers need new breeds of architectures to start from for the modern web app.

Sneak peek: Evolution of the modern web app architecture

The Change Drivers

How have users’ expectations of computers, phones, and software changed in the past decade? What are the drivers of this change? And what do those changes imply for a new web app architecture?

Snappier Web Experiences

The constraints of the web’s request / response pattern led to poor user experiences. Imagine if every installed application on your computer had to re-render the entire screen whenever you clicked something! And now—to simulate the web’s latency—add the restriction of using a computer from twenty years ago but running today’s software. This is what much of the web user experience is like. It’s shocking that we put up with it as much as we do.

Our desire for richer, more interactive, and lower latency web experiences has driven the popularity of single page apps and the JavaScript tools required to build them. Use of convenience libraries like jQuery gave way to libraries like Backbone.js and Knockout.js that finally supported the creation of a cohesive, maintainable app that runs in the user’s browser.

The next generation of libraries—where we are now in 2018—including React, Angular, Ember, and Vue, were born out of our experiences with the first generation plus the exploding popularity of Node.js (here, as a build tool) and rapid improvements to JavaScript (or maybe more accurately, ECMAScript with versions ES2015, 16, 17, etc).

So what architectural changes do we need to make to allow our web application to support single page apps? First, we need a way to more efficiently deliver JavaScript to the user’s browser. Single page apps include much more JavaScript (and possibly more images, videos, CSS, and HTML) than we had previously sent to the browser. Second, it means we need a publicly exposed API from which the single page app can get its dynamic data.

So here’s the list of new concepts we’re going to add to our architecture:

Content Delivery Network -- effectively a global network cache that can deliver JavaScript, HTML, CSS, and images to the user’s browser faster than our web server can.
API Gateway -- a publicly exposed API from which the single page app can securely request the data it needs.

Smartphones

Today most of us expect to find a mobile app for any product or service we want to interact with—or at least a web site that works well on our mobile devices. We now use mobile devices for so many things we could do only in the desktop browser just ten years ago—or even new things we could never do in the desktop browser, like receive a notification based on location, scan a digital coupon, or track a workout. And this trend doesn’t look like it’s going away: in 2011, 10% of the world’s population were smartphone users, and now in 2018, a third are smartphone users. In many countries like the United States, Sweden, and South Korea, 70% of the population is using smartphones.

The iOS App Store and Android Market (now Google Play) appeared in 2008 and ushered in the era of ubiquitous native mobile apps as mobile web browsers hadn’t and still don’t (although they are getting better and better) allow for the same user experience that a native app does.

So what do we need to support a native mobile app in our architecture? We need a way for the app to communicate with our servers—to receive a push notification, receive a digital coupon, or save the GPS data for a user’s workout. Fortunately, we can use one of the concepts we added to support single page apps: an API Gateway.

API as a new UI

It seems that every company wants its own API now. Obviously companies like Heroku, GitHub, and Amazon Web Services whose customers are developers, need an API, but now many companies whose customers aren’t developers—like Bank of America, Macy’s, and Eli Lilly—have APIs. ProgrammableWeb’s API Directory lists over 19,000 public APIs that companies or other organizations have built, most of them from companies whose primary customers are not developers. All these APIs are a kind of new user interface for developers.

An API opens your product up to endless possibilities of creative extension by millions of software developers. Google Maps, Twilio, and Braintree probably didn’t think about ride-sharing as a use case for their APIs, but Uber and Lyft wouldn’t have been possible without a mapping, SMS, and payment API.

An API also provides a standard point of integration with your product. Often the electrical socket analogy is used to explain this. Without a standardized electrical socket, we would have to manually wire each new device we get into our home’s electrical system. The electrical socket gives us a standard interface between a device and the electrical grid. Similarly, because there are common standards for APIs, like REST over HTTP using JSON content type, we as developers need less time and proprietary expertise to integrate two disparate systems.

As you might have guessed, this API can use the same architectural concept we added in the last section: an API Gateway.

Streaming Data (and lots of it)

The software we create is producing data at an exponentially growing rate. And we are demanding much more from that data. We want data not only to build the user’s web or mobile experience, but we want it for reporting, for informing A/B tests or blue/green deploys, for user experience studies, for billing, and for compliance. And often we need a real-time view of that data with the flexibility to create different views to support evolving business requirements.

The “data firehose” has emerged as a metaphor to describe harnessing all of this moving data. We need a data firehose to help move this data around in an organized way. It could be to move data from the point of production to storage, to allow internal services to communicate with each other, or to send real-time data to analysis and visualization tools.

So what do we need to add to our architecture to manage, observe, analyze, and visualize streams of data?

Apache Kafka -- a tool that can manage large streams of data reliably and to which data producers and data consumers can easily attach themselves
AWS S3 / Google Cloud Storage -- effectively unlimited and redundant storage for all the data
AWS RedShift / Google BigQuery -- a data warehouse tool to organize the data and make it queryable quickly
Heroku Dataclips / Metabase / Looker -- data query and visualization tools that are simple to connect to the data warehouse and easy to start using

Adaptability

Ever-increasing complexity (i.e. business complexity, requirement complexity, software becoming more mission-critical to businesses, increased real-time expectations of users, growing data volumes, etc.) means developers need to build and operate more adaptable systems.

An increasingly popular way to build a more adaptable system is to compose it from many small, discrete services. Call this technique what you want—microservices, distributed systems, service-oriented architecture, or even functions-as-a-service—the important concept is the composition of separately-maintained and deployed software components to form a single system.

Building new functionality as separate services instead of within a monolith allows for easier separation of concern. As a developer, I can keep fewer concepts in my head while building or maintaining smaller services. As an engineering manager, my team can work on more in parallel. To be clear, this technique doesn’t come without pitfalls. Check out Ryan Townsend's blog post on his company’s journey of deconstructing a monolith into services.

And Don’t Forget Scale

Clearly, the need for scale is not new in 2018, but the addition of all these new architectural components makes scaling more difficult than it was a decade ago. Previously we could manage most of our scaling needs with three levers: horizontal scaling of web dynos (Heroku’s lightweight container technology), horizontal scaling of worker dynos, and vertical scaling of PostgreSQL. Now we have dozens of levers to worry about. And further compounding the scale problem is the relentless growth of internet users. Half the people on our beautiful blue marble have access to the internet!

This is one area where PaaS and managed services show their value. The good ones reduce the number of levers you have to worry about to scale your system.

Bring It All Together

So let’s look at all the components we need to add to our architecture:

Content Delivery Network -- effectively a global network cache that can deliver JavaScript, HTML, CSS, and images to the user’s browser faster than our web server can.
API Gateway -- a publicly exposed API from which the single page app can securely request the data it needs.
Apache Kafka -- a tool that can manage large streams of data reliably and to which data producers and data consumers can easily attach themselves
AWS S3 / Google Cloud Storage -- effectively unlimited and redundant storage for all the data
AWS RedShift / Google BigQuery -- a data warehouse tool to organize the data and make it queryable quickly
Heroku Dataclips / Metabase / Looker -- data query and visualization tools that are simple to connect to the data warehouse and easy to start using

And the two new concepts we need to think about:

Service-oriented architecture for adaptability
Scalable-first design so that scaling is a normal operations activity, not a herculean one at 2am

Let’s look at our new architecture from a high-level. All of the new components we discussed have been incorporated into a single system.

A modern web app architecture in 2018

Compared to the version from 2008, it looks like a lot to build and manage, right? It could be, if you try to build and manage it all yourself. The truth is that using a PaaS like Heroku makes deploying and managing something like this much easier than building it yourself on an IaaS (or your own data center [shudder]).

Looking to the Future

For sure, we still have many open questions. How do the dynos discover each other and communicate between themselves? What gets served from the CDN and how does it get there? How does Kafka write data to S3 or RedShift? How do code deploys work? How does CI/CD work? How do we monitor the health of all the components? What security concerns do we have to think about?

In follow-up blog posts, we’ll zoom in to specific parts of this system and show you how to deploy them to Heroku. Until then, I hope this high-level architecture serves as a guide to the primary concepts many organizations need to think about as they’re building a web application in 2018.

Many thanks to those who helped me write this post: Vik Rana, Charlie Gleason, Jennifer Hooper, Scott Truitt, Jon Mountjoy, Jon Byrum, Michael Friis, Hunter Loftis, Stephanie Chung, and Jonathan Fulton’s Web Architecture 101 post. Your reviews, discussions, ideas, inspiration, grammar help, drawings, and typo catches are greatly appreciated.

DEV Community: Chris Castle

Render, Remix, and Strapi: Let's Build a Productivity Tips App

What are Strapi and Remix?

Why Strapi and Remix Together?

A Productivity Tips App

Set Up Your Development Machine

Configure and Get Familiar with Strapi

Build the Front-End with Remix

Deploy Everything to Render

Customize

Remote Debugging with SSH and VS Code

Why SSH?

Debug Node.js Remotely

Configure Render

Configure VS Code

Create the SSH Tunnel

Start Debugging

More to Explore

Add Password Protection to Any Site with OAuth2 Proxy - Plus Social Logins

What is it good for?

What are the drawbacks?

What is it, and how does it work?

Traffic Flow

Implementation

More to Explore

Password Protect Static Sites with PageCrypt

What's it good for?

What are the drawbacks?

What is it, and how does it work?

Try it out

Extend

Explore

What's New in PostgreSQL 14?

JSON Conveniences

Multirange Data Type

Improved Handling of Concurrent Connections

And Much More

Render Supports PostgreSQL 14

Evolving Alongside your Tech Stack

User experience depends on technical expertise

Follow everything but only learn some things

Sometimes, having the insight to know the trade-offs and accepting one sub-optimal approach above another makes it easier to cut losses and focus on the parts of your software which matter.

Keeping calm and steady

CLI Flags in Practice + How to Make Your Own CLI Command with oclif

The Syntax of a CLI Command

Command

Argument

Long Flag

Short Flag

Flag Argument

Other Ways of Passing Data

Building a New Command With oclif

SIGTERM

13. oclif: An Open Source CLI Framework

Code[ish]

oclif / oclif

CLI for generating, building, and releasing oclif CLIs. Built by Salesforce.

oclif CLI

🗒 Description

🚀 Getting Started Tutorial

📌 Requirements

📌 Migrating from V1

Beyond Web and Worker: Evolution of the Modern Web App on Heroku

The Change Drivers

Snappier Web Experiences

Smartphones

API as a new UI

Streaming Data (and lots of it)

Adaptability

And Don’t Forget Scale

Bring It All Together

Looking to the Future

Building a New Command With `oclif`

`SIGTERM`

oclif
/
oclif

`oclif` CLI