DEV Community: Nhost

Individual PostgreSQL instances to everyone

Johan Eliasson — Mon, 26 Sep 2022 18:02:22 +0000

Welcome to Nhost’s first-ever launch week!

Today we’re excited to announce that all new projects get their own dedicated Postgres instance with root access. It's finally possible to connect directly to the database with your favorite Postgres client.

Background

When we launched Nhost v2, all databases were hosted and managed on Amazon RDS. The reason why we started with RDS was twofold:

having the most crucial component of all infrastructure managed and scaled by an experienced team on a mature product seemed to be an excellent idea. We wouldn’t have to manage and operate it ourselves.
with v2, all services (e.g., GraphQL, Authentication, and Storage) were moved to Kubernetes because of its flexibility and extensibility. Running a stateful component like Postgres on Kubernetes comes with a complete set of challenges of their own and we wanted to focus on running the stateless components well.

Kubernetes is a complex piece of technology to master but once you do it, it gives infrastructure teams superpowers. All projects running on Nhost have the option to scale vertically (adding resources to existing instances) and horizontally (adding new instances/replicas) on each service individually (GraphQL, Auth, Storage, and now Postgres, but here only vertically). This means your projects can cope with the load of your application, whether sustained or due to spikes in demand while also providing high availability of your products if the underlying infrastructure is misbehaving or faulty. If a node goes down, your services are almost instantly moved to a healthy one. This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched - it just worked without any manual work from us.

The RDS setup comprised a big, database-optimized instance in every region we operate. One instance would hold multiple databases for multiple projects.

We quickly realized that running a multi-tenant database offering on RDS would be problematic because of resource contention and the noisy neighbor effect. The noisy neighbor issue occurs when an application uses the majority of available resources and causes network performance issues for others on the shared infrastructure. A complex query and the absence of an index could decrease the performance on the entire instance and affect not only the offending application but others on the same instance as well.

Although we were able to mitigate this issue by scaling the instances vertically (CPU, memory) and horizontally (scale out / more instances per region), it became painfully clear it wasn’t a definitive solution and that we were not fixing the fundamental problem.

Other, smaller but relevant issues that made us switch were:

RDS for PostgreSQL is not really raw PostgreSQL and it misses some of its flexibility (e.g., postgres is not a superuser)
The set of extensions available is very limited and one cannot change it
No easy way to give users direct access to their databases with the postgres user (this was a highly requested feature)

PostgreSQL running on Kubernetes

After discussing the topic of running stateful workloads on Kubernetes with a couple of industry experts and hearing about some awesome database companies (PlanetScale and Crunchy Data) already doing so, we finally dove in and took the time to research and experiment.

This was a considerable amount of work that required involving the entire team; researching existing solutions to deploy Postgres in Kubernetes, ensuring we could scale the database according to our user's needs and, of course, adapting our internal systems to provision, operate, and scale our users' databases. In addition, we built a one-click process that will be added to the dashboard soon so you can migrate your existing projects from RDS to a dedicated Postgres at your own convenience.

After testing the new setup internally for a few months we launched a private beta with 20 users a couple of months ago. During that period we gathered useful feedback, fixed a couple of issues, and, most notably, heard from most of the users that they were seeing performance improvements.

All in all, we are extremely happy with the result. It is a top priority for us to provide a stable, performant, scalable, and resilient platform so you can build your projects with us and forget about the infrastructure and its operational needs.

It is important to mention that we have the ability to use external PostgreSQL providers if required. If your application has special requirements due to compliance, multi-region needs, or you just happen to like any of those cool database companies out there we can accommodate and connect your application to the database of your choosing.

What does this mean to you?

As mentioned, the overall stability and performance gains are the most important reasons why we are now giving individual instances to everyone, but there are a few other points I would like to mention:

You now own the full PostgreSQL instance. When creating a project, you will be asked for a password for the postgres superuser - you can use any Postgres client to connect directly to your database using the connection string. Be careful, with great power comes great responsibility.
You are now able to install the extensions you need as long as we support them. We will be continuously adding new extensions and will make sure to listen to you on which ones we should prioritize.
You will soon be able to scale up your database and give it as many resources as needed (CPU and memory).
The Hasura GraphQL engine runs alongside your Postgres database, meaning there is little latency to your requests.

What's next?

We are really excited not only about the stability we are able to provide but also about the world of possibilities brought by moving our PostgreSQL offering to Kubernetes. We now have the right foundation in place to look into other features like read replicas or multi-region deployments. Building robust and highly scalable applications should be fun, fast, and easy for everyone. Let us take care of the hard and boring stuff!

P.S: If you like what we are doing, please support our work by giving us a star on GitHub.

Launching Nhost CDN: Nhost Storage Is Now Blazing Fast™

Johan Eliasson — Wed, 29 Jun 2022 06:33:45 +0000

Today we're launching Nhost CDN to make Nhost Storage blazing fast™.

Nhost CDN can serve files up to 104x faster than before so you can deliver an amazing experience for your users.

To achieve this incredible speed, we're using a global network of edge computers on a tier 1 transit network with only solid-state drive (SSD) powered servers. Nhost CDN is live for all projects on Nhost, starting today.

In this blog post we'll go through:

How to start using Nhost CDN?
What is a CDN?
How we built Nhost CDN and what challenges we faced.
Benefits you will notice with Nhost CDN.

Before we kick off, Nhost Storage is built on Hasura Storage which is impressively fast already. With today's launch of Nhost CDN, it's even faster!

How to start using Nhost CDN?

Upgrade to the latest Nhost JavaScript SDK version:

npm install @nhost/nhost-js@latest
# or yarn
yarn add @nhost/nhost-js@latest

If you're using any of the React, Next.js or Vue SDKs, make sure you update them to the latest version instead.

Then, initialize the Nhost Client using subdomain and region instead of backendUrl.

import { NhostClient } from '@nhost/nhost-js';

const nhost = new NhostClient({
  subdomain: '<your-subdomain>',
  region: '<your-region>',
});

You find the subdomain and region of your Nhost project in the Nhost dashboard.

Locally, you use subdomain: 'localhost'. Like this:

import { NhostClient } from '@nhost/nhost-js';

const nhost = new NhostClient({
  subdomain: 'localhost',
});

That's it. Everything else works as before. You can now enjoy extreme speed with the Nhost CDN serving your files.

Keep reading to learn what a CDN is, what technical challenges we faced, and the incredible performance improvements Nhost CDN brings to your users.

What is a CDN?

Before we start diving into technical details and fancy numbers let's briefly talk about what CDNs are and why they are important.

CDN stands for "Content Delivery Network", roughly speaking they are highly distributed caches with lots of bandwidth and located very close to where users live. They can help online services and applications serve content to users by storing copies of it where they are most needed so users don't need to reach the origin. For instance, if your origin is in Frankfurt but users are coming from India or Singapore, the CDN can store copies of your content in caches in those locations and save users the trouble of having to reach Frankfurt for that content. If done properly this has many benefits both for users and for the people responsible for the online services:

From a user perspective: Users will experience less latency because they don't need to reach all the way to Frankfurt to get the content. Instead, they can fetch the content from the local cache in their region. This is even more important in regions where connectivity may not be as good and where packet losses or bottlenecks between service providers are common.
From an application developer perspective: Each request served from a cache is a request that didn't need to reach your origin. This will lower your infrastructure costs as you have to serve fewer requests and, thus, lower your CPU, RAM, and network usage.

Before dropping this topic let's see a quick example, imagine the following scenario:

In the example above, Pratim and Nestor are clients while Nuno is the CDN. In a faraway land, we have our origin, Johan.

When Pratim first asks Nuno about the meaning of life he doesn't know it so he asks Johan about it. When Johan responds Nuno stores a copy of the response and sends it to Pratim.

Later, when Nestor asks Nuno about the meaning of life he already has a copy of the response so Nuno can send it to Nestor right away, reducing latency and saving Johan the trouble of having to respond to the same query again.

This is great but it comes with some challenges. As a continuation, we will talk about some of those, how we are taking care of them for you in our integration with Nhost Storage, and some performance metrics you may see thanks to this integration.

Cache invalidation

As we mentioned previously, CDNs will store copies of your origin responses and serve them directly to users when available. However, things change, so you may need to tell the CDN that the copy of a response is no longer up to date and they need to remove it from their caches. This process is called “cache invalidation” or “purging”.

In the case of Nhost Storage cache-invalidation is handled automatically for you. Every time a file is deleted or changed we instruct the CDN to invalidate the cache for that particular object.

However, this isn't as easy as it sounds as Nhost Storage not only serves static files, it can also manipulate images (i.e. generate thumbnails from an image) and/or generate presigned-urls. This means that for a given file in Nhost Storage there may be multiple versions of the same object that are cached in the CDN. If you don't invalidate them all you may still serve files that were deleted or, worse, the wrong version of an object.

To solve this issue we attach to each response a special header Surrogate-Key with the fileID of the object being served. This means that it doesn't matter if you are serving the original image, a thumbnail, or a presigned url of it, they all will share the same Surrogate-Key. When Nhost Storage needs to invalidate a file what it needs to do is instruct the CDN to invalidate all copies of responses with a given Surrogate-Key.

Security

At this point, you may be considering the security implications of this. What happens if a file is private? Does this mean the CDN will serve the stored copy of it to anyone that requests it or does it mean this is only useful for public files? Well, I am glad you asked. The short answer is that you don't have to worry, you can still benefit from the CDN while keeping your files private.

The longer answer is as follows:

In the CDN we flag cached content that required some form of the authorization header
When a user requests content that was flagged as private we perform a conditional request from the CDN to the origin. The conditional request will authenticate the request and return a 304 if it succeeds.
The CDN will only serve the cached object to the user if the conditional request succeeded.

Even though you still need a round trip to the origin to perform the authentication of the user, you can benefit from the CDN as your request to the origin is very lightweight (just a few bytes with headers going back and forth), and the file will still be served from the CDN cache. You can see below an example of two users requesting the same file:

The cache is empty, CDN requests the file and stores it, total request time from the origin perspective is 5.15s:

time="2022-06-16T12:16:28Z" level=info client_ip=10.128.78.244 errors="[]" latency_time=5.157454279s method=GET status_code=206 url=/v1/files/1ff8ef8d-3240-4cf3-805f-fc3d61d190b2                                           │

Cache has the object already cached but flagged as private so it makes a conditional request to authenticate the user. Total request time from the origin perspective is 218.28ms (after the 304 the actual file is served directly from the CDN without origin interaction):

time="2022-06-16T12:16:41Z" level=info client_ip=10.128.78.244 errors="[]" latency_time=218.283899ms method=GET status_code=304 url=/v1/files/1ff8ef8d-3240-4cf3-805f-fc3d61d190b2

Serving Large Files

Serving large files pose two interesting challenges:

How do you cache large files efficiently?
How do you cache partial content if a connection drops?

These two challenges are related and have a common solution. For instance, imagine you have a 1GB file in your storage and a user starts downloading it, however, the connection drops when the user has downloaded 750MB. What happens when the next user arrives? Do you have to start over? If the file is downloaded fully, do you keep the entire file in the cache?

To support these use cases Nhost Storage supports the Range header. This header allows you to tell the origin you want to retrieve only a chunk of the file. For instance, by setting the header Range: bytes=0-1024 you'd be instructing Nhost Storage to send you only the first 1024 bytes of a file.

In the CDN we leverage this feature to download large files in chunks of 10MB. This way if a connection drops we can store these chunks and serve them later on when a user requests the same file.

Tweaking TCP Parameters

Another optimization we can do in the CDN platform is to tweak the TCP parameters. For instance, we can increase the congestion window, which is particularly useful when the latency is high. Thanks to this we can improve download times even when the file isn't cached already.

Shielding

We mentioned that caches are located close to users, which means that the cache that a user in Cape Town would utilize isn't the same as a user in Paris would. A direct implication of this is that a user in Paris can't benefit from content cached in another location.

This is true up to a certain extent. We utilize a technique called “shielding” which allows us to use a location close to the origin as a sort of “global” cache. With shielding, a cache that doesn't have a copy of the file that is needed will query the shield location instead of the origin. This way you can still reduce the load of your origin and improve your users' experience.

Performance Metrics

To showcase our CDN integration we are going to perform three simple tests:

We are going to download a public image (~150kB)
We are going to download a private image (~150kB)
We are going to download a private large file (45MB)

To make things more interesting we are going to deploy a Nhost app in Singapore while the client is going to be located in Stockholm, Sweden, adding to a latency of ~200ms.

As you can see in the graph below even when the content isn't cached (miss), we experience a significant improvement in download times; downloading the images is done in less than half the time, and downloading the large file takes 30% less time. This is thanks to the TCP tweaks we can apply to the CDN platform

Improvements are more dramatic when the object is already cached, then we see we can get the public image in just 21ms compared to the 2.19s that took to get the file directly from Nhost Storage. Downloading the private image goes down from 2.07s to 403ms, which makes sense as the latency is ~200ms and we need to go back and forth to the origin to ask it to authenticate the user and get back the response before we can serve the object.

Did you build a CDN network?

No, we didn't. We are leveraging Fastly's expertise for that so you get to benefit from their large infrastructure while we get to enjoy their high degree of flexibility to tailor the service to your needs.

Conclusion

Integrating a CDN with a service like Nhost Storage isn't an easy task but by doing so we have increased all metrics allowing you to serve content faster and giving your users a better experience when using your services no matter where your users are.

Hasura Storage in Go: 5x performance increase and 40% less RAM

Johan Eliasson — Sun, 29 May 2022 06:53:04 +0000

Hasura Storage is an open source service that bridges any S3-compatible cloud storage service with Hasura and it is the service we, at Nhost, use to provide storage capabilities to our users.

Its objective is to allow users to combine the features they love about Hasura (permissions, events, actions, presets, etc.) with the convenience of being able to show files online.

The service, written in Node.js, has served us well for quite some time but as the company grew and the number of users increased performance at scale started being a concern, while Node.js may be great for many reasons, performance and scalability aren't one of them.

For those short on time, the goal of this blog post is to showcase the gains we incurred across all metrics by rewriting a Node.js microservice in Golang. Gains that include a 5x increase in the number of requests served while halving memory consumption.

Deciding to rewrite the service

As the need to scale became more important we decided to rewrite the service in go. The reasons behind Golang were many:

Its dependency management system and build system make it a perfect fit for the cloud
Nhost team had plenty of experience with Golang
Even though it is a very verbose language, especially compared to Node.js, it is very easy to learn and fast to write
It is known to be very performant

If you are interested in learning more about the language and its promises ACM has a good article about it.

Rewriting the service

The actual rewrite was quite uneventful. Writing microservices like this is a well-known problem and, while the service is very useful and convenient, it doesn't perform anything too complex. Hasura-storage's innovation and usefulness come from bridging two great services that our users love; s3 and Hasura, not from doing anything whimsical.

Benchmarking the service

When the rewrite was completed we decided to run some benchmarks against both the Node.js and Golang versions of the service. To do so we used k6 and designed the following test:

When a test starts it ramps up its number of workers from 1 to TARGET during the first 10 seconds
Then it runs for 60 seconds more before winding down.
Workers query the service as fast as possible
We run the following tests:
1. download_small_file (100 workers) - Download a 100KB file
2. download_medium_file (100 workers) - Download a 5MB file
3. download_large_file (50 workers) - Download a 45 MB file
4. download_image (100 workers) - Download a 5.3 MB image
5. download_image_manipulated (10 workers) - Download the same image but resize the image and apply some blur on the fly
CPU was limited to 10% of the overall system
RAM was unlimited

Before seeing the conclusions I want to clarify the numbers we are going to see shouldn't be taken at face value, the system used for the benchmark had its CPU allowance quite limited as we wanted to stress both services and see how they behaved under pressure so, what we are interested in isn't the raw numbers, but the difference between the two versions.

Number of requests

We are going to start by looking at the number of requests as this is the main metric that will dictate if the other metrics make sense or not (i.e. decreasing RAM while serving fewer requests might not be something desirable).

As you can see the number of requests we were able to serve under each scenario improved substantially, especially for smaller files (5x)

RAM consumption

RAM is a limited resource and it is not easy to throttle it if a system is reaching its limits. Traditional systems have relied on swapping to disk but this has a dramatic impact on overall performance so it is not an option in modern systems. Instead, modern systems rely on restarting the service when a threshold is reached. It is for this reason that peak memory usage under different scenarios is important, if you reach a certain value your service is restarted, if the service is restarted, it can't serve requests. Below you can see peak usage under the different scenarios described above:

As you can see we managed to improve considerably this metric under all scenarios, especially when downloading large files. If you keep in mind that we were also serving up to 5x more requests this is a very good result.

Response times

Another important metric is response time, here we are looking at two units; minimum response time, which will tell us what is the response when the system is not under pressure, and the P95 which will tell us what was at most the response time for most users (including when the system was under pressure).

Let's start by looking at the minimum response time:

It is hard to see in the test case download_small_file but we improved the response time in that scenario from 29ms in the Node.js case to 7ms in the Golang case. This is a 4x improvement we see across the rest of the scenarios except download_image_manipulated, where we see around a 2x improvement. (we will talk about this scenario later).

And now let's look at the P95

Here we also see a 4x improvement for most cases with the exception of download_image_manipulated or download_large_file where we see substantial improvements but not as dramatic as the rest. This makes sense as downloading large files is going to be I/O NET bound while manipulating images is going to be CPU bound but even then we are happy to see this substantial improvement.

Manipulating images

I wanted to single out the case download_image_manipulated as it is an interesting case. For performance reasons, both versions of Hasura Storage rely on a C library called libvips, this is the reason why Node.js is performing quite nicely here despite the CPU limitations we introduced. However, it is nice to see that even realizing both services are using the same underlying C library we managed to improve all metrics significantly

Deploying the service to production

After the service was rewritten and tested we deployed the service to production. As soon as it was deployed we could see the benefits almost immediately. Below you can see RAM usage in one of the nodes of our cluster:

As you can see we reduced our memory footprint by almost 40%, a significant improvement that will let us serve more users and traffic without increasing our overall infrastructure bill.

Conclusion

We decided to rewrite the service to improve performance metrics and after benchmarking both services side by side we can unequivocally claim that we managed to improve all metrics significantly. We expect to be able to serve more requests while utilizing fewer resources and while also improving the response times for our users which I am sure they will appreciate.

How to Add Authentication to Hasura

Johan Eliasson — Mon, 21 Feb 2022 09:45:18 +0000

Hasura and GraphQL are amazing, but setting up authentication to work with Hasura can be difficult.

In this article you’ll learn how to add authentication to Hasura, so you can sign in users and start using Hasura permissions in your GraphQL API.

Before we continue, let’s just set the context.

Hasura has Authorization (which is different from authentication) built in to handle permissions and access control for your GraphQL API.

But Authentication, which handles users, sign-in flow, tokens, etc, ****is not handled by Hasura. Instead, you need to have your own authentication service that handles all that.

Building such a service is not easy.

But we have a solution for you: Hasura Auth.

Hasura Auth is an open-source service to handle authentication with Hasura. With Hasura Auth you can sign in users and manage roles. Hasura Auth works specifically for Hasura and its permission system.

To get started quickly with authentication for Hasura we recommend creating a Nhost account and creating a Nhost app. At Nhost, we automatically provision a backend for you with Postgres, Hasura, and Hasura Auth. No need worrying about configuration or infrastructure. Instead you can focus on building your app and provide value for your users.

Hasura Auth is Open Source

Hasura Auth is fully open source and is specifically created to handle authentication for Hasura. It’s built with TypeScript.

The code is available at https://github.com/nhost/hasura-auth.

Users in Your Database

Hasura Auth stores your users in your database. Hasura Auth uses its own auth schema with tables related to authentication, such as auth.users.

Users are stored in the auth.users table. If the user signs in with an email and password the password is hashed using bcrypt. However, Hasura Auth has support for even more sign-in methods, such as:

Email and Password
Magic Link
SMS
Social Providers such as Facebook, Google, GitHub, and many more.

Features of Hasura Auth

🧑‍🤝‍🧑 Users are stored in Postgres and accessed via GraphQL
🔑 Multiple sign-in methods.
✨ Integrates with GraphQL and Hasura Permissions
🔐 JWT tokens and Refresh Tokens.
✉️ Emails sent on various operations
✅ Optional checking for Pwned Passwords.
👨‍💻 Written 100% in TypeScript.

Easy to use JavaScript SDK

Hasura Auth is part of the Nhost backend stack, and Nhost comes with an easy-to-use JavaScript SDK written in TypeScript.

This is how you use the Nhost JavaScript client.

First initialize the Nhost client:

import { NhostClient } from '@nhost/nhost-js'

const nhost = new NhostClient({
  backendUrl: '<nhost-backend-url>'
})

Then sign up a new user:

const { session, error } = await nhost.auth.signUp({
  email: 'elon@musk.com',
  password: 'spacex-to-mars',
  options: {
    displayName: 'Elon Musk'
  }
})

That’s it. It’s that simple!

Here’s a demo with authentication with Nhost and our JavaScript client:

Read the full JavaScript SDK documentation if you want to learn more.

You can also use the JavaScript client together with @nhost/react-auth and @nhost/react-apollo if you plan to use React and the Apollo GraphQL client. Support for more GraphQL clients are coming soon.

Get started

You have two options to get started with Hasura Auth to add authentication to Hasura:

Use Nhost (recommended)
Self-hosting

Get started with Nhost (recommended)

Create an account and create a new Nhost app. That’s it 🤯 (Yes, we try to make it easy to build apps).

When you create a new Nhost app we manage all infrastructure and configuration so you get:

Postgres
Hasura
Hasura Auth
Hasura Storage
Serverless Functions

For a more detailed guide, read our Quick start guide.

Get started with self hosting

Hasura Auth is open-source and available as a Docker image (nhost/hasura-auth).

Here’s an example of how you can combine Postgres, Hasura, and Hasura Auth using Docker Compose:

version: '3.6'
services:
  postgres:
    image: postgres
    restart: always
    volumes:
      - ./docker/data/db:/var/lib/postgresql/data
      - ./docker/initdb.d:/docker-entrypoint-initdb.d:ro
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-secretpgpassword}
    ports:
      - '5432:5432'
  graphql-engine:
    image: hasura/graphql-engine:v2.1.1
    depends_on:
      - postgres
    restart: always
    environment:
      HASURA_GRAPHQL_DATABASE_URL: postgres://postgres:${POSTGRES_PASSWORD:-secretpgpassword}@postgres:5432/postgres
      HASURA_GRAPHQL_JWT_SECRET: ${HASURA_GRAPHQL_JWT_SECRET}
      HASURA_GRAPHQL_ADMIN_SECRET: ${HASURA_GRAPHQL_ADMIN_SECRET}
      HASURA_GRAPHQL_UNAUTHORIZED_ROLE: public
      HASURA_GRAPHQL_LOG_LEVEL: debug
      HASURA_GRAPHQL_ENABLE_CONSOLE: 'true'
    ports:
      - '8080:8080'
  hasura-auth:
    image: nhost/hasura-auth:latest
    depends_on:
      - postgres
      - graphql-engine
    env_file:
      - .env
    environment:
      HASURA_GRAPHQL_DATABASE_URL: postgres://postgres:${POSTGRES_PASSWORD:-secretpgpassword}@postgres:5432/postgres
      HASURA_GRAPHQL_GRAPHQL_URL: http://graphql-engine:8080/v1/graphql
    ports:
      - '4000:4000'
    volumes:
      - ./docker/data/mailhog:/maildir

For this example, please use the .env file here.

And start everything with: docker-compose up -d.

Conclusion

We love open source at Nhost. That’s why we’ve open sourced Hasura Auth to make authentication easy with Hasura and make it available for everyone.

Hasura Auth can be used with Nhost if you don’t want to manage infrastructure and configuration your self, or as a self hosted alternative with Docker.

If you want to support our open source, please give us a start on GitHub: https://github.com/nhost/nhost. It would mean a lot. Thanks!

Nhost CLI: From Zero to Production

Vadim Smirnov — Fri, 18 Feb 2022 13:20:58 +0000

In the previous tutorials, that are covered in the documentation, we tested various parts of Nhost, such as:

Database
GraphQL API
Permission
JavaScript SDK
Authentication

All changes we did to our database and API happened directly in the production of our Nhost app.

It’s not ideal for making changes in production because you might break things, which will affect all users of your app.

Instead, it’s recommended to make changes and test your app locally before deploying those changes to production.

To do changes locally, we need to have a complete Nhost app running locally, which the Nhost CLI does.

The Nhost CLI matches your production application in a local environment, this way you can make changes and test your code before deploying your changes to production.

Recommended workflow with Nhost

Develop locally using the Nhost CLI.
Push changes to GitHub.
Nhost automatically applies changes to production.

What you’ll learn in this guide:

Use the Nhost CLI to create a local environment
Connect a GitHub repository with a Nhost app
Deploy local changes to production

Setup the recommended workflow with Nhost

What follows is a detailed tutorial on how you setup Nhost for this workflow

Create Nhost App

Create a new Nhost app for this tutorial.

It’s important that you create a new Nhost app for this guide instead of reusing an old Nhost app because we want to start with a clean Nhost app.

Create new GitHub Repository

Create a new GitHub repository for your new Nhost app. The repo can be both private or public.

Connect GitHub Repository to Nhost App

In the Nhost Console, go to the dashboard of your Nhost app and click Connect to GitHub.

Install the Nhost CLI

Install the Nhost CLI using the following command:



sudo curl -L https://raw.githubusercontent.com/nhost/cli/main/get.sh | bash

Initialize a new Nhost App locally:



nhost init -n "nhost-example-app" && cd nhost-example-app

And initialize the GitHub repository in the same folder:



echo "# nhost-example-app" >> README.md
git init
git add README.md
git commit -m "first commit"
git branch -M main
git remote add origin https://github.com/[github-username]/nhost-example-app.git
git push -u origin main

Now go back to the Nhost Console and click Deployments. You just made a new deployment to your Nhost app!

If you click on the deployment you can see that nothing was really deployed. That’s because we just made a change to the README file.

Let’s do some backend changes!

Local changes

Start Nhost locally:



nhost dev

💡 Make sure you have Docker installed on your computer. It’s required for Nhost to work.

The nhost dev command will automatically start a complete Nhost environment locally on your computer using:

Postgres
Hasura
Authentication
Storage
Serverless Functions
Mailhog

You use this local environment to do changes and testing before you deploy your changes to production.

Running nhost dev also starts the Hasura Console.

💡 It’s important that you use the Hasura Console that is started automatically when you do changes. This way, changes are automatically tracked for you.

In the Hasura Console, create a new table customers with two columns:

id
name

When we created the customers table there was also a migration created automatically. The migration was created at under nhost/migrations/default.



$ ls -la nhost/migrations/default
total 0
drwxr-xr-x  3 eli  staff   96 Feb  7 16:19 .
drwxr-xr-x  3 eli  staff   96 Feb  7 16:19 ..
drwxr-xr-x  4 eli  staff  128 Feb  7 16:19 1644247179684_create_table_public_customers

This database migration has only been applied locally, meaning, you created the customers table locally but it does not (yet) exists in production.

To apply the local change to production we need to commit the changes and push it to GitHub. Nhost will then automatically pick up the change in the repository and apply the changes.

💡 You can commit and push files in another terminal while still having nhost dev running.



git add -A
git commit -m "Initialized Nhost and added a customers table"
git push

Head over to the Deployments tab in the Nhost console to see the deployment.

Once the deployment finishes the customers table is created in production.

We’ve now completed the recommended workflow with Nhost:

Develop locally using the Nhost CLI.
Push changes to GitHub.
Nhost deploys changes to production.

Apply metadata and Serverless Functions

In the previous section, we only created a new table; customers. Using the CLI you can also do changes to other parts of your backend.

There are three things the CLI and the GitHub integration track and applies to production:

Database migrations
Hasura Metadata
Serverless Functions

For this section, let’s do one change to the Hasura metadata and create one serverless function

Hasura Metadata

We’ll add permissions to the users table, making sure users can only see their own data. For this, go to the auth schema and click on the users table. then click on Permissions and enter a new role user and create a new select permission for that role*.*

Create the permission with custom check:



{
  "id": {
    "_eq" : "X-Hasura-User-Id"
  }
}

Select the following columns:

id
created_at
display_name
avatar_url
email

Then click Save permissions.

Now, let’s do a git status again to confirm the permission changes we did were tracked locally in your git repository.

We can now commit this change:



git add -A
git commit -m "added permission for uses"

Now let’s create a serverless function before we push all changes to GitHub so Nhost can deploy our changes.

Serverless Function

A serverless function is a piece of code written in JavaScript or TypeScript that takes an HTTP request and returns a response.

Here’s an example:



import { Request, Response } from 'express'

export default (req: Request, res: Response) => {
  res.status(200).send(`Hello ${req.query.name}!`)
}

Serverless functions are placed in the functions/ folder of your repository. Every file will become its own endpoint.

Before we create our serverless function we’ll install express, which is a requirement for serverless functions to work.



npm install express
# or with yarn
yarn add express

We’ll use TypeScript so we’ll install two type definitions too:



npm install -d @types/node @types/express
# or with yarn
yarn add -D @types/node @types/express

Then we’ll create a file functions/time.ts

In the file time.ts we’ll add the following code to create our serverless function:



import { Request, Response } from 'express';

export default (req: Request, res: Response) => {
  return res
    .status(200)
    .send(`Hello ${req.query.name}! It's now: ${new Date().toUTCString()}`);
};

We can now test the function locally. Locally, the backend URL is http://localhost:1337. Functions are under /v1/functions. And every function’s path and filename becomes an API endpoint.

This means our function functions/time.ts is at http://localhost:1337/v1/functions/time.

Let’s use curl to test our new function:



curl http://localhost:1337/v1/functions/time
Hello undefined! It's now: Sun, 06 Feb 2022 17:44:45 GMT

And with a query parameter with our name:



curl http://localhost:1337/v1/functions/time\?name\=Johan
Hello Johan! It's now: Sun, 06 Feb 2022 17:44:48 GMT

Again, let’s use git status to see the changes we did to create our serverless function.

Now let’s commit the changes and push them to GitHub.



git add -A
git commit -m "added serverless function"
git push

In the Nhost Console, click on the new deployment to see details

After Nhost has finished deploying your changes we can test them in production. First let’s confirm that the user permissions are applied.

Then, let’s confirm that the serverless function was deployed. Again, we’ll use curl:



curl https://your-backend-url.nhost.run/v1/functions/time\?name\=Johan

Conclusion

In this tutorial, we have installed the Nhost CLI and created a local Nhost environment to do local development and testing.

In the local environment, we’ve made changes to our database, to Hasura’s metadata, and created a serverless function.

We’ve connected a GitHub repository and pushed our changes to GitHub.

We’ve seen Nhost automatically deploying our changes and we’ve verified that the changes were applied.

In summary, we’ve set up a productive environment using the recommended Nhost workflow:

Develop locally using the Nhost CLI.
Push changes to GitHub.
Nhost deploys changes to production.

In addition to all that, the Nhost team is always happy to support you with any questions you might have on Discord and Github!

Nhost v2 - The beginning of something big

Johan Eliasson — Tue, 25 Jan 2022 17:16:07 +0000

Today is a big day in the history of Nhost. After months of hard work, we're now launching Nhost v2 into public beta.

For the last two years, we've been on a mission to build the backend for developers. It has taken us a little longer than we originally planned. This is not totally unheard of in IT projects apparently. We started out thinking we would only update a few things but as we dug deeper, the more we realized that if we wanted to be competitive we would eventually have to restructure both our offered services and our infrastructure.

So, last summer, we decided to go all-in on a new version of Nhost with architecture and infrastructure that would enable us to scale our customer's apps. Both in terms of actual performance and in terms of developer productivity.

Yes, we had to do some groundwork, and yes it would take some time. But it's done!

We're now releasing Nhost v2 and we’ve never been in a better position to help developers build apps their users love.

What is Nhost?

If this is the first time you’re reading about Nhost, here are three descriptions of what Nhost is:

Nhost is for the backend, what Netlify and Vercel is for the frontend.
Nhost is an open-source backend to build apps users love.
Nhost is a serverless backend for web and mobile apps. Let's get back to what's new with Nhost v2. But first...

What's the same with Nhost v2?

All Nhost apps (previously called projects) still consist of Postgres and Hasura. Two amazing open-source projects. We remain committed to offering an open data model which has no vendor lock-in for developers.

Generally, we’re still pro,viding a pre-configured backend to build web and mobile apps:

🐘 SQL Database
🌐 GraphQL API
🔒 Authentication
🗄️ File Storage
⚡ Serverless Functions

But we’ve made some updates on pricing, design, services, and infrastructure.

Let’s start a new thing that many have waited for. A free tier!

What's new with Nhost v2?

Rather watch me showing you what's new? Watch this video:

Free tier

Nhost now comes with a free tier which is perfect for testing, building a side project, or for your next hackathon (something we'll do more of in the future).

Once you're ready to go to production we have a new Pro plan. And once your app and business scale up we got you covered with our new infrastructure (more about that further down in the post).

Design

Everything at Nhost has been redesigned. Even our logo had some color adjustments. This means new designs for:

Landing page (https://nhost.io)
Documentation (https://docs.nhost.io)
Dashboard (https://app.nhost.io)

With the new designs, we’re are better able to communicate our service and we’re set up to deliver some awesome updates in the dashboard for 2022.

Hasura Backend Plus is replaced by Hasura Auth and Hasura Storage

Hasura Backend Plus is replaced by Hasura Auth and Hasura Storage. The new authentication service is very similar to Hasura Backend Plus with one big difference; the users table has been moved to the auth schema and merged with the accounts table. This change was necessary to more easily release new features to Hasura Auth and for customers to track migrations separately from Hasura Auth.

Storage has been fundamentally redesigned and now uses Hasura Permissions instead of the previous rules engine using rules.yaml. This makes for a unified permission system across all services of the Nhost stack. We've also added two tables, files and buckets, in a new storage schema where file metadata is stored automatically. This means files and buckets can be treated just as any other datatype and all file metadata is accessible via your GraphQL API.

It’s not only our backend services that were updated. Our client-side SDKs are also fresh!

SDKs

nhost-js-sdk is replaced by @nhost/nhost-js. The previous nhost-js-sdk only had support for authentication and storage, whereas the new @nhost/nhost-js has support for authentication, storage, GraphQL, and functions.

We're also working on our Flutter/Dart SDKs which are to be updated soon.

The SDKs use the app’s domain from Nhost. Previously, we had multiple subdomains per app. Not anymore!

Domains

Previously, we had 3 different subdomains per Nhost app. https://xxx-[hasura | backend | api].nhost.app. Now, there is only a single subdomain per app: https://xxx.nhost.run. Each service is then scoped under /v1/graphql, /v1/auth, /v1/storage, /v1/functions.

Much simpler!

Another thing that should be simple is to have a proper workflow. From local development to production. We’re big fans of Netlify and Vercel and their way of deploying websites from GitHub. To do that for the backend, which we do, we needed an improved version of the CLI.

CLI

We have rebuilt the Nhost CLI in Go for increased stability and performance. The CLI also mimics the Nhost experience locally. We also included some productivity features in the CLI such as:

Separated environment per branch - when you switch git branch, the CLI automatically creates a separate local environment for that branch only.
Serverless functions are automatically rebuilt on every incoming request using ESbuild. Rebuilding usually takes less than 200 ms.
System environment variables are automatically populated with the correct values based on your environment. The CLI is perfect for serious apps with a proper workflow:

Use CLI and develop locally
The CLI will automatically track database migrations and Hasura metadata
Push your code to GitHub
Nhost will automatically deploy database migrations, Hasura metadata, and serverless functions.

Now you understand why Nhost is for the backend, what Netlify and Vercel are for the frontend.

Speaking of serverless functions...

Serverless functions

Previously, we had a “custom API” which was a type of serverless function hosted on Google cloud run, using a mix of Docker and express. This worked OK but had some drawbacks.

Our new serverless functions are built for and deployed on, AWS Lambda (just like Netlify and Vercel) and deployed physically close to your backend in the same AWS region. The new serverless functions also enable simple code, types, and package sharing between the front and backend. Perfect if you want to use JavaScript or TypeScript for both your frontend and backend.

Infrastructure

Previously, we used Digital Ocean to host Nhost projects. We did this with a combination of virtual machines (Droplets), RabbitMQ, and custom scripts to manage docker-compose.yaml files. This was error-prone and we were not able to easily scale a customer's backend.

It was time to level up!

From DigitalOcean to AWS

From now on we’re using AWS, and we rebuilt our infrastructure from the ground up with a single focus in mind: Scale!

We’re now able to scale any Nhost app horizontally in terms of GraphQL, Authentication, Storage, and serverless functions.

The database is still a bottleneck when it comes to scaling, however, we're using AWS RDS for all our client's databases which enables us to deliver:

Automatic software patching
Automatic fail-over
High-availability
Vertically scaling up to 32 vCPUs and 244 GiB of RAM
Read replicas
... and much more!

AWS also has 26 regions around the world, which will help us deliver backends close to our customers. Something that is particularly important with new national data protection laws.

We’ll continue building out our infrastructure in 2022 to support all types of customers, from students, indie hackers, agencies, startups, SMB to enterprise customers.

In summary, Nhost is ready for scale and our new infrastructure will help us ship new features faster.

This is the beginning of something big!

FAQ for customers using Nhost v1

How should I get started with Nhost v2?

If you are brand new to Nhost, we recommend reading our Get Started guide in our documentation.

If you are already hosting your apps on Nhost please read our Migration Guide on how to migrate your old app to Nhost v2.

Do I need to sign up for a new account for Nhost v2?

If you are new to Nhost just sign up for a new account.

If you are an existing customer, just log in with your existing account details. We have ported your login details over.

How do I access my old apps that I created in the previous version of Nhost?

You can still access your old apps at https://console.nhost.io. However, we urge you to migrate your apps to Nhost v2 as we're planning on shuting down the old platform later this year. More information will follow.

How do I get support with Nhost?

For in-depth technical questions, we recommend raising a Github Discussion this keeps it public and searchable for the community.

For quick questions, please join us on Discord. We have an #ask-anything channel where you can get help from us and the community.