DEV Community: arnu515

Using DigitalOcean Droplets as Ephemeral Sandboxes for AI Agents

arnu515 — Mon, 25 May 2026 13:19:29 +0000

Introduction

Agentic AI is a type of artificial intelligence system powered by large language models that operates autonomously using tools to interact with the outside system under minimal human supervision. These AI Agents use tool calling to interact with the outside world and perform tasks on the user's behalf. These tasks can include programming (including performing long-running programming tasks, such as Ralph loops), sending emails, responding to some event, executing code, searching the web, using a computer, and generally anything else that can be represented as a tool call. While this provides LLMs with great utility, it also gives them destructive power that can result in the deletion of files, execution of harmful code, heavy usage of resources, etc.

This is where sandboxing is useful: if AI agents could run in an ephemeral machine where making irreversible changes to the system doesn't have many consequences, it can alleviate some of the pitfalls of providing LLMs with this much power.

Sandboxes are ephemeral machines that are easy and fast to create and destroy, and can also run for long amounts of time. They’re usually deployed on some cloud provider, since cloud machines are easy to spin up and down, can run for long amounts of time, and can be created with various hardware configurations; but there are tools that allow you to create sandboxes on your local machine (you can use docker for this, for example). Thus, this is a great use case for Virtual Private Servers (VPS) such as DigitalOcean Droplets. They can be created and removed with ease, while being billed by the second, so you only pay for what you use. Sandboxes are meant to be ephemeral, i.e. when you delete the sandbox, it loses all its data. Sandboxes are created based on an image, so they start off with a set configuration (such as installed and configured programs, users, keys, etc.).

In this article, you will learn how you can use DigitalOcean Droplets to create ephemeral sandboxes (which can also have persistent storage attached). You'll be introduced to various DigitalOcean services that make using Droplets as sandboxes easier. Using DO over other tools lets you select the location, compute power, and software installed on the sandbox, while being able to take advantage of DigitalOcean's competitive pricing and connecting other services like Volumes, VPCs, the Container Registry, and more to your sandbox right in one place.

Key Takeaways

Isolated Environments for Agentic AI: Running autonomous AI tools in isolated cloud environments mitigates security risks like accidental file deletion, resource draining, or malicious script execution.
Cost Efficiency via Per-Second Billing: Services like DigitalOcean Droplets support per-second billing, enabling rapid programmatic provisioning and teardown without incurring flat hourly or monthly minimums.
Declarative Infrastructure Setup: Utilizing cloud-init configurations provides a robust, code-defined method to inject security rules, users, and environment packages during instance launch without pre-building custom disk images.
Data State Management: Combining ephemeral compute with network-attached storage volumes allows you to persist structural project histories or git trees across individual sandbox replacements.

Prerequisites

To follow along with this article, you will need the following things:

A DigitalOcean Account (you can sign up for $200 free credits using this link)
Some basic Linux command-line knowledge (check these articles for an introduction)
Basic programming knowledge and familiarity with a coding agent for the Ralph loop use case.

Programmatically Creating DigitalOcean Droplets

A Droplet is a Virtual Private Server (a Linux virtual machine in the cloud) which can be configured with a desired amount of (virtual) CPUs, memory, and storage. Droplets can be provisioned and deprovisioned as we need, making them perfect for creating ephemeral sandbox machines. Additionally, you save costs with per-second droplet pricing that was rolled out recently, since you pay down to the exact second for our droplet usage, rather than an hourly or monthly rate.

The DigitalOcean API allows you to create and destroy Droplets programmatically. You can do this by calling API endpoints from an HTTP client, use one of their client libraries, or the CLI. You will need an API Key to authenticate to the API, which you can create in the console. Create an API Key with at least the read, create, and update permissions for Droplets (you may need to add other permissions if you plan on using other features of DigitalOcean like VPCs and volumes).

Save this token to your environment so you can use it in the deployment commands later:

export TOKEN="your_api_token_here"

You can use the doctl CLI or the DigitalOcean API to create a droplet with a provided image and size. You can select one of the base images or a one-click image from the Market, but what if you need something specific that isn't available in these images, such as wanting the sandbox to have specific packages or expose some specific services (like an A2A server).

DigitalOcean lets you specify a shell script to run when the droplet is created, or write a more sophisticated configuration using cloud-init yaml files. They provide a nice alternative to custom images in that they don't cost money to store, don't have to be built in advance, but result in longer ready-times for the sandbox.

Moving Beyond Standard Images

While you can easily deploy base images (like Debian or Ubuntu) or Marketplace apps, out-of-the-box templates often fall short if your sandbox needs specific pre-installed packages or custom background services right from boot.

You could pass a standard bash script to run on startup, but shell scripts can be brittle. A single missing dependency or typo can leave the system in a half-broken state. A more robust alternative is cloud-init. This industry-standard tool uses structured YAML configurations (called cloud-configs) to handle system setup predictably. They provide a lightweight alternative to building custom machine images in advance, avoiding storage costs while keeping your infrastructure defined as code.

Defining the Sandbox Configuration

Let's define a sandbox that installs the latest version of Bun, downloads a web app, exposes an HTTP service, and locks down the firewall.

Create a file named cloud-config and add the following YAML. Notice how the configuration is broken down into declarative blocks: managing user access (users), installing system packages (packages), writing a systemd daemon directly to the filesystem (write_files), and finally executing the deployment commands (runcmd).

#cloud-config

# create a non-root user
users:
  - name: user
    # the user's real name
    gecos: User
    # set their shell to bash
    shell: /bin/bash
    # give them sudo privs
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    # remove password login, we can only log in via ssh keys
    lock_passwd: true
    # set one or more SSH keys for this user
    ssh_authorized_keys:
      # put any ssh keys you want on the user here
      - ssh-ed25519 AAAASOME_FINGERPRINT

# run apt-get update (or equivalent on non-Debian systems)
package_update: true

# install curl and unzip (required to install bun)
# ufw is the firewall
packages:
  - curl
  - unzip
  - ufw

# write arbitrary files
write_files:
  # create systemd service
  - path: /etc/systemd/system/app.service
    permissions: '0644'
    content: |
      [Unit]
      Description=A simple server
      After=network.target

      [Service]
      Type=simple
      User=user
      WorkingDirectory=/home/user
      ExecStart=/opt/bun/bun run /home/user/index.ts
      Restart=always
      Environment=PORT=5000

      [Install]
      WantedBy=multi-user.target

# running arbitrary commands
runcmd:
  # download bun
  - mkdir -p /opt/bun
  - curl -fsSLo /opt/bun/bun.zip [https://github.com/oven-sh/bun/releases/download/bun-v1.3.10/bun-linux-x64.zip](https://github.com/oven-sh/bun/releases/download/bun-v1.3.10/bun-linux-x64.zip)
  - unzip -oqd /opt/bun /opt/bun/bun.zip
  - mv /opt/bun/bun-linux-x64/bun /opt/bun
  - chmod +x /opt/bun/bun
  - rm -r /opt/bun/bun-linux-x64 /opt/bun/bun.zip
  # download the source code
  - curl -fsSLo /home/user/index.ts [https://gist.github.com/arnu515/8c639949ee1a5d226312873151ca40f9/raw/49eeeba4344110d297ed8f30d32ba8f307af4db2/index.ts](https://gist.github.com/arnu515/8c639949ee1a5d226312873151ca40f9/raw/49eeeba4344110d297ed8f30d32ba8f307af4db2/index.ts)
  - chown user:user /home/user/index.ts
  # configure the firewall
  - ufw allow ssh
  - ufw allow 5000/tcp
  - ufw enable
  # enable and start the systemd service
  - systemctl enable --now app

Do not remove the #cloud-config comment at the top of the file, as DigitalOcean requires it to parse the payload. Remember to replace ssh-ed25519 AAAASOME_FINGERPRINT with your actual public key.

Provisioning the Droplet

With the configuration defined, you can deploy the machine. You can do this via the official doctl CLI or by sending the payload directly to the API using curl.

If you have doctl installed and authenticated, run:

doctl compute droplet create \
  --enable-ipv6 \
  --image debian-13-x64 \
  --size s-1vcpu-512mb-10gb \
  --user-data-file cloud-config \
  --region YOUR_REGION_CHOICE \
  sandbox

Alternatively, if you are relying strictly on the API, you can use jq to properly escape the YAML file's quotes and newlines before piping it into curl:

jq -n \
  --arg region "YOUR_REGION_HERE" \
  --arg ud "$(cat cloud-config)" \
  '{
    name: "sandbox",
    region: $region,
    size: "s-1vcpu-512mb-10gb",
    image: "debian-13-x64",
    ipv6: true,
    user_data: $ud
  }' | curl -X POST "[https://api.digitalocean.com/v2/droplets](https://api.digitalocean.com/v2/droplets)" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d @-

To find the exact slug for your preferred data center, run doctl compute regions list.

Monitoring and Testing

The underlying droplet infrastructure takes a few seconds to spin up. You can check the provisioning status by listing your droplets:

doctl compute droplet list

The status will initially show as new. After a brief moment, DigitalOcean will allocate the IP addresses and the status will change to active:

ID           Name        Public IPv4  ...  Status  ...
SOME_NUMBER  sandbox     SOME_IP      ...  active  ...

Once the droplet is active, cloud-init begins executing your configuration in the background. After the system finishes pulling the packages and starting the systemd service, your HTTP server will be live.

Navigate to http://SOME_IP:5000 in your browser to view your live view counter app. You now have a fully reproducible sandbox environment.

Cleaning Up

Because Droplets are billed by the second, you should destroy the sandbox as soon as you are done experimenting. You can delete it using either of the following methods:

doctl compute droplet delete sandbox

# or via the API:

curl -X DELETE "[https://api.digitalocean.com/v2/droplets/DROPLET_ID](https://api.digitalocean.com/v2/droplets/DROPLET_ID)" \
  -H "Authorization: Bearer $TOKEN"

Why Not Containers?

While running OCI containers locally provide a reasonably sandboxed and reproducible environment, they do still run on your local machine and come with downsides because of that, such as not being able to spin up long running tasks, requiring compute/storage resources, requiring additional software (or possibly hypervisors) installed, etc.

Running these sandboxes on a VPS provider like DigitalOcean gives you server-grade hardware and networking while being able to have long-running tasks without keeping your local machine up or using its resources.

Comparison with Other Sandbox Services

There are services like Daytona or sprites.dev which specifically provide preconfigured sandbox instances complete with client libraries and instant configuration. These services usually work by running a container or micro-VM on a server with a preconfigured image, and charging you per-second of compute you use. These services are more convenient, and containers/micro-VMs are faster to start up, but these services tend to cost much more and have higher vendor lock-in compared to using a VPS provider like DigitalOcean. For example, as of the time of writing, running a 2vCPU + 4GiB RAM sandbox with 10GiB of storage for two hours on Daytona costs $0.33, while the same sandbox running for the same time costs $0.036 on DigitalOcean — almost a 10x difference!

If you prioritize spin-up speed over cost, these services may be better than using a cloud provider. There also are plenty of sandbox orchestrating projects on GitHub that you can host on a VPS to create sandboxes within that VPS or a fleet of VPSes, but this article will not be covering that. Let me know if you're interested in such a thing and I'll be sure to cover it!

Unfortunately, there isn't a user-friendly library/command line tool for using DigitalOcean VPSes as sandboxes yet, so you'll have to write scripts that wrap over doctl or use the HTTP API (optionally through a client library) instead. This article will henceforth cover using the doctl command since that's easiest to wrap using something like bash scripts, but these commands are easily translatable to API calls or client library methods.

Using Prebuilt Images

Writing a cloud-config and waiting for it to run every time a sandbox is created is cumbersome. It would be better if a droplet could be created from an image that is already configured according to your needs. DigitalOcean supports uploading custom images (billed at $0.06 per GB per month to store) from which Droplets can be created.

Let's create an image that has a Python interpreter, some JavaScript runtimes, and sane security defaults. This image will be used later in this article as a code execution sandbox. These images must be Unix-like images with a proper filesystem (ext3/ext4), with cloud-init and sshd installed. These images can be raw (.img) or in a virtual machine image format like qcow2 or vdi.

To create such an image, the easiest way is to create a droplet, make your preferred changes to it, and then create a snapshot of it from the console. You can also create images from QEMU/VirtualBox VMs, or use tools like Packer. You could also create a NixOS configuration and make it output a qcow2 image — for maximal reproducibility.

This article will cover the first method. Create a droplet and SSH into it. Then, make any desired changes to the droplet. I chose to update the system, create a new user, give it sudo perms, add an SSH key to it, disable root ssh login, set up ufw, set up fail2ban, and finally, install python3, nodejs, and bun. Feel free to follow these steps, or change them as you wish.

Once your droplet is ready, create a snapshot of it to save it as an image: (these steps can also be done from the DigitalOcean console)

# power off the droplet  
doctl compute droplet-action power-off DROPLET_ID

# create a snapshot — note the region  
doctl compute droplet-action snapshot --snapshot-name 'sandbox-image' --wait DROPLET_ID

# delete the droplet  
doctl compute droplet delete DROPLET_ID

Now, you can use this image to create a droplet. You can only create the droplet in the region you created the snapshot (i.e. the region of the earlier droplet). You can add snapshots (and user-created images) to other regions at no additional cost from the console.

# note the image id of the desired image  
doctl compute image list-user

doctl compute droplet create \
    --image IMAGE_ID \
    --size s-1vcpu-1gb \
    --region REGION \
    --sandbox

Feel free to scour the DigitalOcean one-click app marketplace or for community created images on the web for readymade images you can use.

Persist Sandbox Contents

There are some situations in which you don't want an ephemeral sandbox, but you need compute that can be spun up and down on demand while keeping data changes. A common example of this is something like GitHub codespaces, where you want to persist local code changes the user has made, but spin down the compute when they're done editing. They should then be able to get back to editing from the point they left off later.

DigitalOcean allows you to attach a (network-attached) storage pool that persists over droplet destruction. They're called block storage volumes. They can be created and attached to a droplet to provide it with storage that doesn't get destroyed with the droplet. Do keep in mind that volumes can only be attached to Droplets in the same region.

If you're looking for storage to upload, say build artifacts/output, or other such output files that aren't read from once they're written to, need to be accessed from tooling outside the sandbox, or do not need to be available across sandboxes, but need to be available somewhere, then I recommend using object storage (like DigitalOcean Spaces) instead. This would mean having to write some additional software that your sandbox will have to execute to upload these files to the object storage (it could be as simple as a cURL script). Block storage volumes are recommended in use cases requiring constant access and modification of created files, rather than "upload and forget".

The below command creates a 5GiB volume: (be sure to note the volume's ID)

doctl compute volume create --region blr1 --size 5GiB --fs-type ext4 --fs-label storage sandbox-volume

The volume can be attached to a running droplet like so:

doctl compute volume-action attach VOLUME_ID DROPLET_ID

After the action completes, the volume is available on the droplet at /mnt/sandbox_volume. Writing to any files in this volume will ensure that those files are persisted even after the droplet is destroyed. You can remove a volume from a droplet by using the detach subcommand of volume-action in doctl. If you delete this droplet now, the volume will still remain (but it'll be detached). You are still charged for the volume, even if it isn't attached to anything.

When creating a new droplet, you can directly attach the volume to it using the --volumes flag with the droplet create subcommand. The volume will be attached to the same place and the files that were created earlier will be available. Unlike object storage, the files on the volume cannot be accessed from outside Droplets.

Volumes can only be attached to one droplet at a time (to prevent from data corruption due to race conditions). If you want multiple Droplets to access data from the same volume at the same time, you'll have to use a network filesystem hosted on one of the Droplets with the volume attached.

Attaching a Fixed IP to the Sandbox

Up until now, all the Droplets you've created have had different IPs. This isn't really useful since you'd want to reach your sandboxes over the internet to use them. DigitalOcean allows you to reserve IPv4 and IPv6 addresses that you can attach to Droplets to give them a deterministic address that they can be reached at.

IPv4 reserved addresses cost money — even if they're not attached to a droplet, but IPv6 addresses are free to reserve. If your network supports IPv6, you can get away with not reserving IPv4 addresses and only using IPv6 to communicate with your Droplets.

Create a fixed IP through the following commands:

doctl compute reserved-ip create --region REGION  
doctl compute reserved-ipv6 create --region REGION

You can then attach a reserved IP to an existing droplet using the following command: (the IP and droplet must be in the same region)

doctl compute reserved-ip-action attach IPV4_ADDR DROPLET_ID  
doctl compute reserved-ipv6 attach IPV6_ADDR DROPLET_ID

Unfortunately, there's no way to create a droplet and assign it a reserved IP immediately. You'll have to run the above command to assign it the IP after creation manually. Alternatively, you could fetch the droplet's IP address by using the following commands:

doctl compute droplet get --template '{{.PublicIPv4}}' sandbox  
doctl compute droplet get --template '{{.PublicIPv6}}' sandbox

You may want to take a look at the Reserved IP documentation if you want to do some further configuration, like making the droplet send outbound traffic through the reserved IP. Some of these changes may require the ip command, which you can persist across sandboxes either by updating the image, or by writing a specific cloud-config (I recommend the latter since if your reserved IP changes in the future, that would mean updating the image with the former method).

Use-case: A Ralph loop Agent

If you're familiar with LLM-assisted programming (using large language models to automate and enhance software development tasks), you must have heard of a Ralph loop, which, in its simplest state, is the following bash loop:

while:; do cat prompt.md | CODING_AGENT --allow-all-perms; done

What is a Ralph loop? It's an agent that is fed a prompt from a file over and over again until it performs some big task. At every iteration of the agent, it can edit the prompt file to give instructions to the next iteration of the agent that runs. Ralph loops can either produce bug-free complicated software due to its many iterations and precise instructions, or end up going in a death spiral after one of its iterations makes a mistake.

Why sandbox a Ralph loop? This is the perfect use-case for a sandbox. When a Ralph loop goes rogue and starts breaking apart, you should be able to contain the fallout. There are many cases of agents going rogue and deleting files, writing bad code, running arbitrary commands, etc., and stuff like that should not be happening on a machine you care about. A sandbox is ideal for running Ralph loops, especially in unsupervised cases.

How to set up the sandbox? 1. Base Image: You'd start by using a base image with all of your necessary tools included — such as programming languages, build tools, and an LLM coding tool.

cloud-config: Then, you'd add a cloud-config that creates the initial prompt.md file, clones the repo, sets up credentials & API keys, etc.

Things to consider

The repo should be in a block volume since if the agent goes rogue, you'll still have the repo that you can restore using git after deleting the sandbox. The original Ralph loop article linked above recommends providing detailed specifications and making the agent only do one task per loop.

Make sure your Ralph loop knows how to exit (a common way is to exit if the previous iteration's prompt is the same as the next — but make this clear in the system prompt of the agent).

You should also tell the agent to periodically commit its work using git. You'll need to upload the results (if not using a volume) and delete the sandbox after the agent is done running. You can check if the agent is still active by periodically using the ps command to check for your coding tool's process. It is also a good practice to have a cronjob on your main machine to delete these Ralph sandboxes in case the harness crashes or the agent takes too long to save on compute and tokens.

You'll need to write a harness that spins up these sandboxes with a cloud-config that creates the initial prompt.md and any other supporting files, waits for the agent to finish executing, and then spins down the sandbox. It could also perform some additional steps, like creating a PR from the resulting code, or running a code review, or something else. This harness should be extremely specific to your work, due to the non-deterministic nature of LLMs. A generic harness may make the LLM go rogue more often, and thus, I leave the harness creation to you. Do tell about your experience in the comments!

FAQs

What happens to data stored on a sandbox Droplet when it is destroyed?

Because Droplets act as completely isolated virtual machine instances, destroying a Droplet permanently deletes its root filesystem and any local ephemeral storage. To preserve logs, source trees, or runtime artifacts, you must connect a persistent network attached component like a DigitalOcean Block Storage Volume or upload output artifacts directly to DigitalOcean Spaces object storage prior to executing the destruction command.

How does DigitalOcean’s per-second billing benefit automated AI workflows?

Traditional cloud VPS platforms bill on a strict hourly minimum window even if an instance only runs for a few minutes. DigitalOcean calculates compute usage down to the exact second. For autonomous AI testing workflows—where a harness might spin up an agent, run a suite of tests in two minutes, and instantly destroy the environment—you only pay for those exact 120 seconds of resource usage.

Can I attach a single DigitalOcean Volume to multiple sandbox Droplets simultaneously?

No. DigitalOcean Block Storage Volumes use standard raw block mappings (such as ext4) that are not designed to safely negotiate concurrent write operations across multiple independent operating systems. Attaching a single volume to multiple systems simultaneously can cause catastrophic filesystem corruption. For multi-agent configurations requiring shared access, configure a Network File System (NFS) server wrapper on a host Droplet instead.

Why should I choose cloud-init over standard custom snapshots for my sandbox environments?

While custom machine images provide faster initial boot sequences, they require maintenance overhead, incur minor storage costs ($0.06/GB per month), and must be regenerated whenever global dependency targets shift. cloud-init allows you to manage sandbox setups entirely via infrastructure-as-code configurations, modifying system behaviors dynamically on the fly without changing baseline snapshots.

Conclusion

To combat the unpredictability of agentic-AI systems, they must be run on disposable machines, while still being able to preserve certain changes and outputs of those agents. Sandboxes fill this criterion quite well, not only for agentic applications but also in use cases where quick ephemeral compute is required. DigitalOcean provides raw compute for cheap, and using the techniques prescribed in this article, you can use these resources as a sandboxing tool. Their products are cost-effective and provide more control over using specific sandboxing services, but they may not be DX-optimized or be the fastest to start up. You can reach a middle ground by using a self-hostable sandbox orchestrator over one or multiple Droplets that uses either containers or micro-VMs (Firecracker). Let me know if you want to see more articles about this topic!

I can't wait to see all the things you create with the ideas mentioned in this article! Please tell me about your experiences or queries, if you have any, in the comments below!

This article is part of the DigitalOcean Ripple Writers program. I received compensation and platform credits for writing this content, but all technical assessments, code, and opinions are my own based on hands-on testing.

SupportAddress — An email-first support platform

arnu515 — Mon, 09 Jun 2025 00:04:09 +0000

This is a submission for the Postmark Challenge: Inbox Innovators.

What I Built

SupportAddress is an email-first support platform that lets your customers reach out for support directly through email. There's no learning difficult eccentric software, creating accounts, or checking back at a webpage every now-and-then to see if the support ticket has been updated; all of that is done via email!

Create subgroups within organisations to automatically sort incoming tickets using AI.

Demo

The application is hosted live at https://support.aarnavpai.in

Testing instructions:

Sign in/up at https://support.aarnavpai.in
If you're signing up, you'll receive an OTP in your email. Enter that, and then choose a display name and password.
Once logged in, you'll be taken to the dashboard, where you'll be asked to join or create an organisation. Create an organisation.
Your SupportAddress will be displayed at your organisation's home page. Give this SupportAddress to your customers for them to email any support queries they may have to it.
You may create subgroups (from the organisation actions section at the bottom of the organistation home page). Giving a description to your subgroup lets AI attempt to automatically send any matching support tickets to it.
You can invite other people to your organisation, and once they've joined, you can add them to subgroups. People who are not part of a subgroup will not be allowed to access its tickets.
To create a ticket, email the organisation's SupportAddress, or the SupportAddress of any individual subgroup. The tickets list should update automatically.
Support Agents can assign themselves to a ticket. If they do that, nobody else will be able to reply to or close that ticket, unless that agent unassigns themselves. Replying to a ticket will send an email to the customer with the reply.
Once a ticket is closed, it can only be re-opened if the customer who opened the ticket replies to the ticket mail thread.

Screenshots:

Join/Create organisation page:

Subgroup creation page:

Org Home page:

Sending an email to the SupportAddress:

A new ticket was automatically created and categorised:

Send a reply from the dashboard:

With attachment support:

Code Repository

arnu515 / supportaddress

Email-first customer support

SupportAddress

The email-first support platform that lets your customers reach out for support directly through email.

This is a submission for the Postmark Challenge: Inbox Innovators.

It is hosted live at https://support.aarnavpai.in.

How I Built It

I used QwikJS for this project. Qwik is a relatively new JavaScript framework that uses Resumability for client-side interactivity rather than Hydration. Resumability means that the client-side bundle can essentially "pick-up" where the server-side rendering stopped, i.e. adding event listeners, running on-mount hooks, etc., instead of having to re-render the entire app from scratch like how traditional SSR frameworks do (Next, SvelteKit, etc.). I used QwikJS before for my website, but that website is mostly static content, so to give Qwik a real try, I decided to use it for this project.

Other components of the tech stack include Supabase for the database, auth, storage, and functions; TailwindCSS for the styling; Cloudflare Workers AI for the AI models; and Vercel for deployment. A full list of used libraries can be found in the package.json file in the repository.

Postmark was used for sending and receiving emails. When a customer emails the SupportAddress, Postmark calls a Supabase edge function with the email data. That edge function sanitizes the email, determines whether it belongs to an existing ticket, or creates a new ticket, and uses AI to categorize it into a subcategory. Any attachments received are uploaded to Supabase storage by the function. Finally, if the ticket is closed, the function opens it.

When a Support Agent replies to, or closes a ticket, an email is sent to the customer using Postmark's API. The API makes it very easy to send emails, including with attachments and custom SMTP headers, without having to set up an entire SMTP connection.

I'd like to thank Postmark for giving me an opportunity to explore email parsing, QwikJS, and AI models! I wanted to use more AI in this project, but unfortunately due to me starting this challenge much later than I was supposed to, I wasn't able to add more AI features. Thanks for stopping by!

I wanna Do ____! — An AI powered habit building application

arnu515 — Mon, 20 Jan 2025 00:07:46 +0000

This is a submission for the GitHub Copilot Challenge : New Beginnings

What I Built

A Habit tracking application that uses AI to create tasks and goals for you which help you build good habits!

Demo

The live demo is hosted at: https://do.aarnavpai.in
The source code is at: https://github.com/arnu515/i-wanna-do

Screenshots:

Repo

arnu515 / i-wanna-do

AI powered habit building app!

I wanna Do!

An app that lets you build habits with AI!

This application was built for the GitHub Copilot 1-Day Build Challenge (and it was built in ~19h!)

It is hosted live at https://do.aarnavpai.in.

For more information, check this post on DEV

View on GitHub

Copilot Experience

Copilot was very useful in this task. I'm sure it would have taken me twice as long if it weren't for Copilot. Copilot does repetitive tasks very well, for example, converting an SQL schema into a typescript interface, or converting JSON to typescript interfaces. It can also do all the boilerplate code, so you focus on what matters. Copilot chat is useful for performing tedious refactors that would require command line scripts to do! Copilot chat was also useful in planning this application.

GitHub Models

This application uses GitHub Models to plan out a habit building journey. The LLM converts whatever habit the user wishes to build into a proper roadmap with goals and tasks, breaking down big tasks into smaller ones and making them more approachable.

Conclusion

This was a unique challenge because of the time limit. There were so many things I wanted to add but couldn't because of the time constraints. Nevertheless, it was a lot of fun! I hope GitHub comes back with more challenges in the future!

Easy development environments with Nix and Nix flakes!

arnu515 — Wed, 15 Jan 2025 15:04:37 +0000

In this article, we shall cover declarative development shells with Nix flakes! If you're new to Nix, I recommend checking out the previous two articles in this series to get a better understanding, since this article assumes that you've read the previous two already.

Let's revise: Creating a Shell

The previous article introduced you to the nix shell command, which downloads/builds packages and puts you in a shell environment with them in the $PATH.

You need not run the examples given below, they're just provided for illustration purposes.

By default, nix shell drops you into a shell (which is your login shell) with the packages in $PATH. You can also specify an arbitrary command to run, for example:

# Postgres client
nix shell nixpkgs#postgres_17 --command psql

Similarly, you can also use another shell (that should exist in $PATH)

nix shell nixpkgs#nushell --command nushell  # https://www.nushell.sh/

You can also run multiple commands like so:

nix shell nixpkgs#gnumake --command sh -c "make && make install"

If you want to run the package itself, you need not use nix shell, since the nix run command also exists.

nix run nixpkgs#vim -- --log some.log

There are some more things that nix shell can do, which I will not be covering here, but they are covered in the reference manual.

Using `nix shell` to create a development environment

This can be done, but it is tedious. Imagine having to type dozens of dependencies manually by hand every time you want to enter your development environment, or even if you put it in a script, or use the --stdin argument, you'll still have the risk of installing a different version of the package than required. You may also like to customize the shell environment a bit more, like populating environment variables, running commands before the shell starts, etc.

Instead, there's a better way..

Creating a Development Shell with Flakes

Development shells (or shell environments as they're called in non-flake land) are a feature of Nix that allow you to all the things mentioned above — install specific versions of packages, set environment variables, run commands before the shell starts, etc.

Let's start with the same flake.nix as last time. You could copy it from here, but I'd like to digress a bit by introducing Nix templates!

Templates are a way for you to create, well template folders which are declared by a flake. Nix can then copy those files over from a flake into your current directory with the nix flake init command. Nix makes templates declarative and versioned too!

Contrary to the command's reference manual, there's no need for the template itself to be a flake!

A template is pretty easy to create, you just need to add an attribute set called templates in your flake's outputs. The keys of the attrset are the template names, and the values are an attribute set with three fields: description, a description of the template; path, the path to the template folder; and welcomeText, some text that is output when someone uses the template.

I've made the simple flake from last article into a template hosted at this article series' GitLab page, it looks like this:

// https://gitlab.com/arnu515-tutorials/nix/-/blob/master/flake.nix?ref_type=heads

{
  outputs = {...}: {
    // ...
    templates.simple = {
      description = "A simple flake";
      path = ./a-simple-flake;
      welcomeText = ''
        Welcome to Nix!

        This flake exports a single package, `hello`, which can be
        executed by running:

        $ nix run #.hello

        Check out my article series on Nix for more information!
        https://dev.to/arnu515/my-new-nix-series-2cc3
      '';
    };
  };
}

You can use a template by using the nix flake init command. Beware that nix flake init will copy the template's files to the current directory, but it will not overwrite existing files:

nix flake init --template gitlab:arnu515-tutorials/nix#simple

Note that Nix checks the fragment simple (called a flake output name) starting from the templates attrset, then the rest of the flake, unlike in nix shell/run/build, which started from the packages attrset, then moved to the legacyPackages attrset, then the rest of the flake.

Creating a development shell

Now let's create the development shell. This will be created in the devShells attrset, which looks just like the packages attrset, i.e. it has another attrset inside it for each system (like x86_64-linux), and those attrsets declare development shells inside them.

Just like packages, a development shell named default is the default development shell.

Here's a simple shell that has the hello package:

{
  description = "Nix devshells!";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-24.11";
  };

  outputs = {nixpkgs, ...}: let
    system = "x86_64-linux";
    pkgs = import nixpkgs { inherit system; };
  in {
    devShells.${system}.simple = pkgs.mkShell {
      packages = [ pkgs.hello ];
      shellHook = ''
        ${pkgs.hello}/bin/hello
      '';
    };
  };
}

There's quite a few changes from earlier! First, the hello package was removed, since we don't need it for this demonstration. Then, two variables called system and pkgs were created using the let ... in ... construct. Finally, we create a shell using the pkgs.mkShell function, specifying packages to be put in path, and a shellHook to run before the shell starts. Let's dissect this one-by-one.

The ${...} syntax does string interpolation, replacing itself with the evaluated string inside the braces. pkgs.hello evaluates to the nix store path of the hello package. Thus, we need to append /bin/hello to actually point to the hello executable. Note that just hello could also have been specified, since it will be available in $PATH, but it is better to specify an absolute path like this so you know where your dependencies are coming from.

Next, let's talk about what pkgs = import nixpkgs { inherit system; }; does. The import function takes in a path, and evaluates it if it is a path to a Nix file, or evaluates the default.nix file within the directory if it is a path to a directory (which is the case when a flake reference, like nixpkgs is passed to import), and returns the evaluation. Since nixpkgs points to the nixpkgs flake, which is a directory, it evaluates the default.nix file present within the flake, which in the case of nixpkgs returns a function accepting, among others, an argument for the current system, which is what we have passed to it. It then returns the set of packages for that system, which is what is bound to pkgs.

The pkgs.mkShell function creates a development shell / shell environment for us. A shell environment is actually just another derivation (that's why it is possible to run nix develop on packages itself). In fact, pkgs.mkShell is just a wrapper around stdenv.mkDerivation, with some conveniences like specifying packages instead of nativeBuildInputs, and concatenating shellHooks from all inputs, as visible in its source code. There also exists a pkgs.mkShellNoCC, which does the same as pkgs.mkShell, but does not introduce a C compiler in the shell. This is useful if you're developing projects that do not need a C compiler installed.

Let's enter this shell by running:

$ nix develop .#simple
Hello, world!
(nix:nix-shell-env) bash-5.2$ which cc  # a C compiler was added to thes shell by Nix
/nix/store/888bkaqdpfpx72dd8bdc69qsqlgbhcvf-gcc-wrapper-13.3.0/bin/cc
(nix:nix-shell-env) bash-5.2$ exit

$ which cc  # different C compiler!
/usr/bin/cc

With nix develop, the flake output name is searched starting from devShells, then packages, then legacyPackages, and finally the whole flake.

An actual development environment

The earlier examples were really simple! Let's create an actual development environment with a better shell, a NodeJS install, pnpm, and even a redis database!

{
  description = "Nix devshells!";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-24.11";
  };

  outputs = {nixpkgs, ...}: let
    system = "x86_64-linux";
    pkgs = import nixpkgs { inherit system; };

    # A custom config for valkey
    # Declative configs!
    valkeyConfStr = ''
requirepass super-strong-password
port 12345
    '';

    # This special package writes the valkey configuration to
    # a text file. For more such packages, see:
    # https://nixos.org/manual/nixpkgs/stable/#trivial-builder-writeTextFile
    #
    # The package outputs the path to the file
    valkeyConf = pkgs.writeTextFile {
      name = "valkey.conf";
      text = valkeyConfStr;
    };
  in {
    devShells.${system} = {
      simple = pkgs.mkShell {
        packages = [ pkgs.hello ];
        shellHook = ''
          ${pkgs.hello}/bin/hello
        '';
      };

      default = pkgs.mkShell {
        packages = [
          # newer versions of redis are not packaged in
          # nixpkgs, so we're using valkey, an open-source
          # fork of redis maintained by the Linux Foundation
          pkgs.valkey

          pkgs.nodejs_22
          pkgs.nodePackages.pnpm

          # the friendly interactive shell!
          pkgs.fish
        ];

        shellHook = ''
          ${pkgs.valkey}/bin/valkey-server ${valkeyConf} &
          exec fish
        '';
      };
    };
  };
}

The default shell in the above flake adds Valkey, NodeJS 22, the PNPM package manager, and the fish shell to the environment. It also starts Valkey in the background through a shell hook, passing it a custom config (declared via Nix!) and runs fish so we're dropped in the fish shell instead of our login shell.

The exec command replaces the currently running process with the command specified. If fish was run without exec, it would drop you into your login shell with the shell environment (or run any additional commands in the shellHook if there were any) after you exit fish instead of exiting the devshell as expected. With exec, when you exit fish, it'll exit the environment too.

Let's enter the shell:

$ nix develop
155483:C 15 Jan 2025 15:56:27.317 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
155483:C 15 Jan 2025 15:56:27.317 * Valkey version=8.0.1, bits=64, commit=00000000, modified=0, pid=155483, just started
155483:C 15 Jan 2025 15:56:27.317 * Configuration loaded
155483:M 15 Jan 2025 15:56:27.318 * monotonic clock: POSIX clock_gettime
                .+^+.
            .+#########+.
        .+########+########+.           Valkey 8.0.1 (00000000/0) 64 bit
    .+########+'     '+########+.
 .########+'     .+.     '+########.    Running in standalone mode
 |####+'     .+#######+.     '+####|    Port: 12345
 |###|   .+###############+.   |###|    PID: 155483
 |###|   |#####*'' ''*#####|   |###|
 |###|   |####'  .-.  '####|   |###|
 |###|   |###(  (@@@)  )###|   |###|          https://valkey.io
 |###|   |####.  '-'  .####|   |###|
 |###|   |#####*.   .*#####|   |###|
 |###|   '+#####|   |#####+'   |###|
 |####+.     +##|   |#+'     .+####|
 '#######+   |##|        .+########'
    '+###|   |##|    .+########+'
        '|   |####+########+'
             +#########+'
                '+v+'

155483:M 15 Jan 2025 15:56:27.320 * Server initialized
155483:M 15 Jan 2025 15:56:27.320 * Loading RDB produced by Valkey version 8.0.1
155483:M 15 Jan 2025 15:56:27.320 * RDB age 2934 seconds
155483:M 15 Jan 2025 15:56:27.320 * RDB memory usage when created 0.91 Mb
155483:M 15 Jan 2025 15:56:27.320 * Done loading RDB, keys loaded: 0, keys expired: 0.
155483:M 15 Jan 2025 15:56:27.320 * DB loaded from disk: 0.001 seconds
155483:M 15 Jan 2025 15:56:27.320 * Ready to accept connections tcp

> ps 
   PID TTY          TIME CMD
 153101 pts/3    00:00:00 fish  # this is login shell
 153205 pts/3    00:00:04 fish  # this is the fish process started by `nix develop`
 155483 pts/3    00:00:00 valkey-server  # valkey is running in the background
 155654 pts/3    00:00:00 ps

And you see that valkey has started in the background! To exit valkey, you need to kill it yourself, like so:

> kill 155483

If valkey isn't killed, it'll continue running in the background, even after you exit the devshell!

If you want to kill valkey when you exit the shell, you can change your shellHook to kill the valkey process on exit:

shellHook = ''
  ${pkgs.valkey}/bin/valkey-server ${valkeyConf} &
  fish
  echo Killing valkey server
  # `ps` lists all processes, `grep` searches for a line with `valkey-server`
  # in it. `awk` grabs the first column (the PID) from the output, and 
  # `xargs` sends stdin as arguments to `kill`.
  ps | grep valkey-server | awk "{printf \$1}" | xargs kill
  exit
'';

Improvement needed: If you have another way to achieve this, then please let me know in the comments!

We can't use exec anymore, since we need to execute commands after fish exits.

Now running nix develop starts valkey, drops you into a shell, and when you exit the shell, valkey will automatically be killed!

Pinning package versions:

This usually isn't required, since flakes are already pinned to exact commits via. the flake.lock file, which should be checked into version control. So even if you enter a development shell ten years from now, you'll have the same version of all packages as the flake.lock (provided that GitHub still exists :P).

To update package versions, you can run nix flake update, which will fetch the latest commit of the branch of all your inputs.

But if you still want a specific version of a package, down to the patch version, you can use a nixpkgs input that has exactly that package. Do note that you may have to build that package yourself if the Nix build cache doesn't have it, as is common with quite old versions of packages.

Before doing that, do search nixpkgs for alternate versions of packages that may exist, for example, nixpkgs has node 18, 20, and 22 in it. Many popular packages are split into different nixpkgs for major versions.

With the disclaimers out of the way, let's learn how you can pin a specific version of a package. For this example, we'll add Node 16 (an end-of-life version that you totally shouldn't be using) to the environment.

First, we need to find a commit of the nixpkgs repo that has Node 16. The latest commit should be enough. There are tools like NixHub that help with exactly that. If you check the page for nodejs on NixHub, you can see all the versions of node that were ever published on nixpkgs. Copy the nixpkgs commit for nodejs 16.20.2, i.e. the part in "Nixpkgs Reference", except the flake output name, but do keep it in mind (#nodejs_16), which happens to be a71323f68d4377d12c04a5410e214495ec598d4c.

Now, add that to our environment like so:

{
  description = "Nix devshells!";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-24.11";

    # NEW
    nodejs_16_nixpkgs.url = "github:nixos/nixpkgs/a71323f68d4377d12c04a5410e214495ec598d4c";
  };

  outputs = {nixpkgs, nodejs_16_nixpkgs, ...}: let
    system = "x86_64-linux";
    pkgs = import nixpkgs { inherit system; };

    # NEW
    nodejs_16 = (import nodejs_16_nixpkgs { inherit system; }).nodejs_16;

    # ...
  in {
    devShells.${system} = {
      # ...
      default = pkgs.mkShell {
        packages = [
          pkgs.valkey

          # NEW: replaced node 22 and pnpm with node 16
          nodejs_16

          pkgs.fish
        ];

        # ...
      };
    };
  };
}

Running nix develop will actually give us an error claiming that this version of Node is end-of-life, which it is.

$ nix develop
error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:34:12:
           33|
           34|   strict = derivationStrict drvAttrs;
             |            ^
           35|

       … while evaluating derivation 'nix-shell'
         whose name attribute is located at /nix/store/2csx2kkb2hxyxhhmg2xs9jfyypikwwk6-source/pkgs/stdenv/generic/make-derivation.nix:336:7

       … while evaluating attribute 'nativeBuildInputs' of derivation 'nix-shell'
         at /nix/store/2csx2kkb2hxyxhhmg2xs9jfyypikwwk6-source/pkgs/stdenv/generic/make-derivation.nix:380:7:
          379|       depsBuildBuild              = elemAt (elemAt dependencies 0) 0;
          380|       nativeBuildInputs           = elemAt (elemAt dependencies 0) 1;
             |       ^
          381|       depsBuildTarget             = elemAt (elemAt dependencies 0) 2;

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: Package ‘nodejs-16.20.2’ in /nix/store/bxygxxbgcc7s82wn8a8wdp4gd34hiw4w-source/pkgs/development/web/nodejs/v16.nix:28 is marked as insecure, refusing to evaluate.


       Known issues:
        - This NodeJS release has reached its end of life. See https://nodejs.org/en/about/releases/.

       You can install it anyway by allowing this package, using the
       following methods:

       a) To temporarily allow all insecure packages, you can use an environment
          variable for a single invocation of the nix tools:

            $ export NIXPKGS_ALLOW_INSECURE=1

          Note: When using `nix shell`, `nix build`, `nix develop`, etc with a flake,
                then pass `--impure` in order to allow use of environment variables.

       b) for `nixos-rebuild` you can add ‘nodejs-16.20.2’ to
          `nixpkgs.config.permittedInsecurePackages` in the configuration.nix,
          like so:

            {
              nixpkgs.config.permittedInsecurePackages = [
                "nodejs-16.20.2"
              ];
            }

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
          ‘nodejs-16.20.2’ to `permittedInsecurePackages` in
          ~/.config/nixpkgs/config.nix, like so:

            {
              permittedInsecurePackages = [
                "nodejs-16.20.2"
              ];
            }

Fortunately, the error also provides an easy fix:

$ NIXPKGS_ALLOW_INSECURE=1 nix develop --impure
# valkey output truncated ...
# you can append > /dev/null to the valkey command in shellHook
# to supress this output

$ node -v
v16.20.2

Success! You're now using an old, unsupported version of node!

Be warned! If you're reading this article in the future, and decide to use this version of NodeJS, then don't be surprised if you have to build NodeJS from scratch! (It's not a fast build, by the way).

Automatic shell environments with `direnv`

A devshell discussion with Nix is not complete without introducing direnv. direnv is a tool that very simply runs commands based on the current directory. It is tedious to run nix develop to get into a development shell every time. It's also tedious to remember to exit the dev shell when you're done. direnv automatically does this for you, so it's a valuable addition! It also allows you to use these shells within non-terminal editors like VSCode and JetBrains.

Just a small fix:

There's a small thing we need to change in flake.nix. It's quite a bad idea to start Valkey every time the devshell is opened, since more than one instance of a dev shell can exist at the same time. Instead, wrap the valkey command

{
  # ...
  outputs = {nixpkgs, nodejs_16_nixpkgs, ...}: let
    # ...

    # NEW
    valkeyScript = pkgs.writeShellScriptBin
      "start-valkey"
      ''
        ${pkgs.valkey}/bin/valkey-server ${valkeyConf}
      '';
  in {
    devShells.${system} = {
      # ...
      default = pkgs.mkShell {
        packages = [
          pkgs.valkey

          nodejs_16

          pkgs.fish
        ];

        shellHook = ''
          # NEW: Replace valkey startup and shutdown code with this
          export PATH="${valkeyScript}/bin:$PATH"
          # NEW: Add exec back
          exec fish
        '';
      };
    };
  };
}

This change adds a shell script called start-valkey to the environment, which runs valkey with the specified configuration. Now, to start valkey, you just run start-valkey, and press Ctrl-C to stop it. This allows us to have multiple shells in that environment now.

Install direnv:

direnv is packaged in most linux distros, but since this is a Nix guide after all, let's install it via. nix by running this command:

nix profile install nixpkgs#direnv

You also need to hook direnv into your shell, since direnv needs to know when you've changed directories. Follow the instructions on this page for your shell. For bash, you need to add eval "$(direnv hook bash)" to the end of your ~/.bashrc, and restart your terminal after that.

To tell direnv we want it to start the development shell when we cd into this folder, add an .envrc in the folder, with the following contents:

# Need to set `stty sane` for fish
# https://github.com/direnv/direnv/issues/967#issuecomment-1987134113
if [[ $SHELL =~ "fish" ]]; then
  stty sane
fi

use flake

This will make direnv automatically enter your development environment. Just run direnv allow, to allow executing this .envrc, and you're automatically dropped into the development environment, with glorious Node 16 available!

If you're using fish as your login shell, remove exec fish from the shellHook in the flake, otherwise direnv will go on an infinite loop, since the fish spawned by the devshell spawns direnv which sees that it should enter the devshell, which spawns a fish, which spawns a direnv, ... and so on.

Other alternatives to development shells built with Nix

If writing a devshell on your own seems more complicated than necessary, you can use tools like Devenv or Devbox (by the same team that built NixHub), which are both built on Nix. Devenv provides nice wrappers to automatically add languages, services (like postgres or redis), etc. on top of your flake, without having to do the shenanigans we had to do with Valkey. Devbox on the other hand, lets you skip writing Nix entirely, since they have their own CLI and lock file that pull packages from nixpkgs.

Conclusion

You've now learned one of the most powerful features of nix! In the next few articles, we'll cover packaging for Nix!

If you really liked this article and would like to support me, here are some ways:

Thank you so much!

Getting started with Nix and Nix Flakes

arnu515 — Fri, 03 Jan 2025 12:26:17 +0000

Let's get started with Nix! This article guides you into setting up the Nix package manager, along with flakes, and demonstrates some cool things it can do.

For a quick intro to what Nix is, check out this article I posted, which introduces Nix, and this series of articles to you. This article demonstrates an overview of the Nix package manager, more Nix concepts will be covered in future articles of this series.

Installing Nix

The Nix package manager can be installed on both Linux and Mac, and is also available as a Docker image for you to try out without installing it on bare metal. Windows users will have to use WSL2 to install Nix on their systems (if they're not using Docker).

The Nix website's download page guides you into using their installer to install Nix on your system. However, this article will use [Determinate Systems' Nix installer] instead, since it lets you easily undo all changes their Nix installer makes (i.e. uninstall) with one command, and it also enables Nix flakes by default, which you'd have to enable on your own if you were using the official Nix installer instead.

Their guide, Zero-to-Nix has detailed installation instructions using the Determinate installer, but in essence, it just boils down to running:

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

in a terminal and following the prompts the installer asks. You can check Zero-to-Nix's page for more instructions, or the Determinate installer's README for detailed options. This method will work both on Linux and Mac.

Obligatory disclaimer: Nix flakes, and nix experimental commands (like nix shell and nix profile, which will be covered later) are experimental and may have breaking changes anytime in the future, hence, they are gated by a config option when using the default installer. But these two features have stayed experimental for many years with little-to-no breaking changes, and are thus considered de facto stable by much of the Nix community, and not using them is just giving yourself a handicap for no reason.

Using the official installer?

If you've opted to use the official Nix installer instead, you'll have to manually enable flakes for your user by editing ~/.config/nix/nix.conf, or for all users by editing /etc/nix/nix.conf and adding the following line to the end of the file:

experimental-features = nix-command flakes

You'll have to create the file if it doesn't already exist.

No Systemd?

If you're using a systemd-less distro, like Artix or Void, you can install Nix from their repositories, since it will come preconfigured with whatever init system you'd be using on those distros. The Determinate installer only works on distros with systemd (for a multi-user installation, which is what you want).

If you wish, you can install Nix from your distro's repositories too, instead of using the Determinate installer. I've done this on Alpine, Artix, and Arch. Just make sure the Nix package is not extremely out of date, like in Void's case. You can check if the nix package is out of date by comparing the version in your repos to the version on NixOS's repos

Void linux's Nix package is very out-of-date (by 2-ish years!) as of the time of writing. You'll have to use the Single-user installation mode of the official Nix installer to install Nix on void.

Using Nix packages

Now that you have the Nix package manager installed, you can use it to install any package from the vast library of 120k+ Nix packages on your system, while ensuring that none of your other packages, even those installed by Nix itself, will break due to dependency conflicts.

Let's install the lolcat package. Well, Nix actually allows us to try out the package without installing it first! It downloads the package (or compiles it if the binary isn't availabe in the build cache) and drops you into a shell session containing the requested package in your $PATH.

The below demonstrations will only have their intended effect if you don't already have lolcat installed!

Let's try it out! Run nix shell nixpkgs#lolcat and Nix will download the latest commit of nixpkgs, a collection of lots of Nix packages, find the lolcat package, get its binary from NixOS's binary cache, download it and put it in your $PATH.

Nix will create you a new shell session where lolcat will be available.

✨🪄 Magic! Open a new terminal window, try running lolcat, and see that the command doesn't exist! Let's see what Nix added to our $PATH to make lolcat available:

$ nix shell nixpkgs#lolcat

$ echo $PATH
/nix/store/3mbkj2nlzf87aapwp1ckqrid21p9lb3j-lolcat-100.0.1/bin:... # (truncated)

Nix downloaded lolcat and placed it in a folder in the Nix store (/nix/store by default). The Nix store contains all packages, even multiple versions of the same package that were ever fetched by Nix. My system, about a month old, has close to 34k items in the store! Some of these are packages, some are built derivations (more on those later!). This can be cleaned using the nix store gc command.

We'll learn more about the nix store in further articles, but for a quick rundown, the Nix store is a read-only filesystem which stores things like downloaded and built packages, any packages you create yourself, and anything else included in a nix package like downloaded or locally available source code. All packages are treated as a pure function, and their built output is stored in the store.

Notice the name of the directory where lolcat was downloaded. It contains the hash of the derivation, ensuring integrity of the package, then the name of the package itself, and finally, the version, which in this case is 100.0.1. When you're trying out these commands for yourself, you may have a different hash and/or version of the package.

lolcat is specified to the nix shell command as nixpkgs#lolcat. This is termed as an installable, and in this case, is a flake reference (more about that later). nixpkgs#lolcat is actually a URL with path nixpkgs, and fragment, i.e. the part after the #, lolcat. nixpkgs is an alias which resolves to the nixpkgs-unstable branch of the Nixpkgs GitHub repository, a vast collection of Nix packages. There are other stable branches of nixpkgs, like nixpkgs-24.11, the latest one as of writing. There are quite a few other aliases too, Nix downloads the list from this JSON file. If you omit the URL host, it defaults to the current directory (.).

From the JSON file, the alias for nixpkgs appears to be:

{
  "from": {
    "id": "nixpkgs",
    "type": "indirect"
  },
  "to": {
    "owner": "NixOS",
    "ref": "nixpkgs-unstable",
    "repo": "nixpkgs",
    "type": "github"
  }
}

The object at "to" is one type of flake reference. It can also be written in a URL form like so: github:NixOS/nixpkgs/nixpkgs-unstable. We'll take a look at various types of flake references later.

The fragment of the URL, lolcat, refers to one of the packages exported by the nixpkgs flake. Omitting it defaults to the default package, i.e., literally a package called default. We'll talk more about flakes later.

Actually installing `lolcat`

Having lolcat in a local shell isn't really useful. Let's install lolcat using Nix so that it can be accessed from any shell (that has the Nix profile in $PATH!).

A nix profile is a set of packages that are installed independently from each other. Nix profiles are versioned, so you can roll back to a previous state of your profile at any time!

To install lolcat into your profile, you can run:

nix profile install nixpkgs#lolcat

Now lolcat will be available in any shell! To remove it, run:

nix profile remove lolcat

lolcat will no longer be available in path, but it will still remain in the Nix store. If you wish to use lolcat again, maybe in a temporary shell, it will not have to be downloaded/built again, since it'll already be available in the Nix store.

Notice that this time we pass the package name only, and not a flake reference. To view a list of installed packages in your current profile, run nix profile list.

Finally, let's demonstrate profile versioning. Run nix profile history to see the versions of your profile:

$ nix profile history
Version 1 (2025-01-03):
  flake:nixpkgs#legacyPackages.x86_64-linux.lolcat: ∅ -> 100.0.1

Version 2 (2025-01-03) <- 1:
  flake:nixpkgs#legacyPackages.x86_64-linux.lolcat: 100.0.1 -> ∅

2 will be green, since it is the current version of the profile.

Don't worry about flake:nixpkgs#legacyPackages.x86_64-linux.lolcat for now, that'll be covered later. Just notice that the first history version installed lolcat 100.0.1, and the second history version removed it. Let's roll back to the first version with:

nix profile rollback --to 1

And lolcat is back! If we make a change while we're in this version, Nix'll create a new version for us, without deleting version 2, so you can roll back to any point! For now, let's stick to the same slate, and go back to version 2. I'd leave it as an exercise for you to do the same.

If you wish to learn more about these commands, you can append --help to see a nicely formatted colored manual.

Nix language basics

This article will not teach you the Nix language, since there are much better places to learn that from, such as:

https://learnxinyminutes.com/nix/ (concise)
https://nix.dev/tutorials/nix-language (official)
https://nixcloud.io/tour/?id=introduction/nix (interactive)
https://nix.dev/manual/nix/2.24/language/ (the full reference)

It is highly recommended to learn this language before proceeding to the next section. If you know the language, you'll not be troubled with the syntax when you make your own flakes.

An introduction to flakes

Nix Flakes are an opinionated way to structure a nix expression made up of packages, OS configurations, development shells, modules, images, overlays, etc. If you've read the series introduction, you'll know that every .nix file is just a Nix expression.

Before flakes were a thing, you'd have to create separate nix files for development shells (shell.nix), packages (default.nix), OS configurations (configuration.nix), etc., which may get annoying, but is just a small hindrance. The real advantage of flakes is flake inputs, which let you easily fetch nix packages from anywhere using many methods ("fetchers"), and the nix experimental commands, which are built to work with flakes.

When you create a flake.nix file in a directory, that directory becomes a flake, so it can be used as a path in the URL passed to nix commands. Open an empty directory on your computer, and create a blank file called flake.nix in it. That directory has now become a flake!

We can run the simplest nix command to verify that, nix eval evaluates a Nix expression and prints it to stdout. Without any arguments, it evaluates the default package exported by the flake. Notice the emphasis on evaluates, not runs or installs, i.e. the command just prints the package derivation out to the screen. Some things are lazily evaluated, meaning they aren't evaluated until they're required. A package in a flake is not evaluated when you're just using the flake's development shell. It is only evaluated when you use the package, i.e. when you install it, or use it in another flake.

Running nix eval on an empty flake.nix however, will give you a syntax error, well obviously, since an empty file is not a valid nix expression. A flake is an attrset which has a few fields as described here, notably inputs, a set of other flakes your flake uses, and outputs, a function returning a set of things your flake exposes (which can be used by other flakes!). The declared flakes in inputs will be fetched by Nix, evaluated (lazily), and passed to the outputs function, along with a special parameter self, which is just a reference to the set returned by outputs, so you can reference your own flake in itself. The power of lazy evaluation!

Let's start with a simple "Hello, world!" flake. Create flake.nix in an empty directory on the machine you installed Nix in, and write the following code:

{
  description = "My first flake!";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-24.11";
  };

  outputs = {nixpkgs, ...}: {
    packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hello;
  };
}

This is how a nix flake looks like. It looks quite a lot like JSON, but with semicolons instead of commas, but don't be fooled, Nix is a proper functional language with programming constructs and everything! The inputs attribute set (attrset) declares a set of flakes your flake depends on. In this case, this flake depends on the nixpkgs flake, a collection of 120k+ nix packages. The nixpkgs flake's source code is hosted on GitHub at nixos/nixpkgs, which is what is specified in the flake input. The nixpkgs-24.11 part after the last / is the branch of the repository. Nixpkgs releases a stable branch every six months, in May (05), and November (11, which is what we're using) every year. You can use nixpkgs-unstable if you want the latest packages instead.

The outputs function shows one such construct. Nix attrsets can be condensed with dots, so nixpkgs.url = "foo"; is actually nixpkgs = { url = "foo"; };, similarly with packages.aarch64_linux.hello.

The function syntax of Nix is: param: returned_expr. Nix functions can take in only one parameter, so multiple parameter functions are actually multiple functions with one parameter each, like so: param1: param2: ...: returned_expr. Nix has support for attrset destructuring, so {nixpkgs, ...} means that the parameter to outputs is actually an attrset, and we extract the property nixpkgs, which matches the input property from it. The ... means to ignore any extra properties passed. If it isn't specified, nix will error if properties other than nixpkgs are passed! We know that self is another property passed to output, hence ... is needed.

Finally, in the outputs, we define a single package named hello for 64-bit linux systems (which is what I have), which is just set to the hello package declared by nixpkgs for 64-bit linux systems. Note that the legacy in legacyPackages doesn't actually mean that these packages are legacy (see this for an explanation [TLDR; legacyPackages makes the nix flake show command not evaluate the package, apart from that, they're functionally identical]). You should change x86_64 and linux to your own system's architecture and OS, if they differ from these.

For example, an M-series Mac would be aarch64-darwin.

Now running nix eval tells us that there's no default package, which is true, since we created a package named hello, not one named default. We could rename hello to default, or we could also ask nix eval to evaluate the hello program, since it takes an installable as an argument:

$ nix eval .#hello
«derivation /nix/store/83p9zy4d8lh5fnipz7d1hl7g3rryw6mx-hello-2.12.1.drv»

The . is technically optional, but if it is omitted, bash treats #hello as a comment, so you'll have to quote it. I prefer putting a leading . instead of quoting the whole string.

We get a derivation! Nix saw that our flake exports the hello package or legacyPackage, saw that it is supposed to fetch the hello package from the nixpkgs flakes, fetches that flake if reqiured, and returns the derivation to us.

We can also build and run this package:

$ nix build .#hello
# no output

$ ./result/bin/hello
Hello, world!

The hello executable is actually GNU hello, that in true GNU fashion is an overly complicated program with a seventeen page manual that prints something to the screen, defaulting to Hello, world!. Nix would build this program using its derivation from scratch, if it wasn't available in the build cache, which most packages usually are, so it downloads the program from there instead. The derivation and built (or downloaded) output is stored in the Nix store, as we saw earlier. The result symlink also links to a folder in the store.

The result folder is actually a symlink, which links to the built folder in the nix store:

$ readlink result
/nix/store/ccs8597k5ji5h7ad94wfr329xcxydbla-hello-2.12.1
$ tree result/
result/
|-- bin/
    |-- hello
|-- share/
    ...
|-- man/
    ...

Conclusion

This was a super quick introduction to the Nix package manager and nix flakes. Stay tuned for more articles in the series, the next one lined up is about development shells with flakes. You wouldn't want to miss this one!

If you really liked this article and would like to support me, here are some ways:

Thank you so much!

My new Nix series!

arnu515 — Fri, 27 Dec 2024 09:11:34 +0000

This is a (work-in-progress) series of articles on the Nix ecosystem. Some things planned for the near future are development environments with Nix and direnv, many NixOS related things, and much more. Stay tuned by giving me a follow for when those come out!

A quick introduction to Nix

From the NixOS website, Nix is a tool that takes a unique approach to package management and system configuration. This "unique approach" refers to building packages in isolation from each other, ensuring that if packages work on one machine, they will work on another. This is done in a declarative fashion, essentially meaning that you tell Nix what to do, and it does it for you, without you having to tell Nix how to do it.

This is done using the nix language, which is a purely functional programming language purpose-built to configure the Nix package manager. The language is simple, dynamically typed, and purely functional. You'll pick it up quite easily, but if you want some formal learning, you can check out the Nix Language Basics chapter of the Nix documentation.

Here's a very simple "Hello, world!" in Nix:

"Hello, world!"

Save it as hello.nix and run it with the command (if you have Nix installed) nix eval --file hello.nix to see Hello, world! output to your terminal!

This works because every .nix file contains a nix expression, which are functions, lists, attribute sets (dictionaries), numbers, paths, strings, etc. The nix eval command evaluates the expression and prints it to the screen.

The Nix language is used to create packages for the nix ecosystem. Nix packages, as described earlier, are built in isolation from each other, ensuring that there is no dependency hell and allows multiple versions of a package to seamlessly coexist at the same time. Nix packages are derivations: "a specification for running an executable on precisely defined input files to repeatably produce output files at uniquely determined file system paths". Nix also provides useful builtin functions and a standard environment to make packaging easier. These packages can then be shared, like any other file, and built by other people without any hassle, since Nix guarantees reproducibility. A collection of 120,000+ packages is available in the nixpkgs repository.

Atop all this is the Nix Operating System, or NixOS. NixOS brings all of the declarative goodness of Nix into a Linux distribution. This means that (almost) everything about your system is declarative, including the packages (obviously), the users, the desktop, the login manager, systemd units, containers, and among other things, even the bootloader! Through some extensions, you can also partition disks, build images, and even configure your home folder. A single NixOS configuration is built into multiple pre-configured operating systems using just a single command!

Planned Articles

You can see a very crude list of planned articles for this series on GitLab. The GitLab repository will also contain any source code and large code snippets (not inlined) used in the series.

Support me

Here are some ways you can support me on this journey:

Thank you so much!

This upcoming series will explore a lot of what Nix can do, and what its amazing community has enabled it to do, so stay tuned for more articles in the near future!

The one thing I do not like about the Nix package manager (and a fix for it)

arnu515 — Tue, 16 Jan 2024 16:51:41 +0000

The nix package manager is an awesome package manager for linux and macos, which focuses on declarative packages. This means that you can dump out all the packages you want into a file, and nix will go out and fetch them for you.

This package manager builds itself on the concept of reproducibility, and it boasts a collection of over 80,000 packages, second only to the AUR! But this blog post is not about why nix-os is great, you'll find many other blog posts and videos if you want to learn why.

Many nix package definitions include compiling from source. This can be tedious, since compiling takes a long time and wastes more bandwidth downloading build dependencies. Hence, the nix team has a binary cache, in which they build binaries for all the systems that nix supports for most of the common packages out there, like web browsers, desktop environments, and many more.

But what if your desired package is not in the binary cache? And what if it takes a long time to compile? This is what happened to me, and this is the one thing I don't like about nix.

The AUR provides binary packages, alongside source packages, for example, yay and yay-bin, but nix only provides source packages for most of its packages. Granted you'll see some exceptions, like firefox and firefox-bin, but those are pretty rare.

The fix

In this article, I'll show you how you can create a binary package for your desired program. I wanted to download the SurrealDB package, but the package on nix was a source package, meaning that I had to spend over 50 minutes waiting for a stupid package to compile.

Surreal provides binaries over at its GitHub releases, which I could've downloaded and ran, but I'd have to manually update the package, and as a 10x developer (I'm not one), I'd never manually do anything, but spend 10x the time trying to automate it. Doing this also defeats the reproducibility of nix, hence it's better to create a nix package so that anyone would be able to download your dependencies without having to do anything extra.

Getting set up

First, make sure you have in hand a name for your package, and its version. Generally, binary packages end with -bin, so I'll call my package surrealdb-bin, and I'll be downloading the binary for the latest version as of the time of writing, which is v1.1.0.

Since I use NixOS, and since I want to install surrealdb globally, I'll create my surrealdb package in /etc/nixos, which is the folder which holds my configuration.nix. You may choose to co-locate your package in the directory with your flake.nix, for example, or even push it to GitHub/Lab so you can make your nix config truly reproducible.

This approach will contain two nix files, one for the package itself, and another for an overlay. An overlay gives you the ability to modify nixpkgs without having to publish your package to the nixpkgs repository, and without having to mess around with inputs. Overlays make it very easy to use your package.

Creating the package

Generally, I put my custom packages in the packages/ subfolder, and custom overlays in the overlays/ subfolder, and I name both these files with the name of the package, in this case surrealdb-bin.nix. I'd recommend you follow the same strategy to avoid clutter, but if you're only going to create one package, you can just keep everything in one directory.

I shall refer to packages/surrealdb-bin.nix as the package declaration. Now, put this code in your package declaration:

{ stdenv, fetchzip, autoPatchelfHook, glibc, gcc-unwrapped }: stdenv.mkDerivation rec {
  pname = "PACKAGE_NAME";
  version = "PACKAGE_VERSION";

  src = fetchzip {
    url = "PATH_TO_YOUR_PACKAGE'S_TARBALL_HERE";
    hash = "A_HASH_OF_YOUR_PACKAGE'S_CONTENTS";
  };

  nativeBuildInputs = [ autoPatchelfHook ];
  buildInputs = [
    glibc gcc-unwrapped
    # Any other system binaries your app may need
  ];

  installPhase = ''
    runHook preInstall
    mkdir -p $out/bin
    install -m755 PACKAGE_BINARY_FILE_NAME $out/bin
    runHook postInstall
  '';
}

Let's go over this code step-by-step. If you're familiar with nix syntax, you'll recognise that we're creating a function which destructures its first argument to accept some fields, and returns the output of the function stdenv.mkDerivation. mkDerivation is the helper function that you use to define nix packages. We give it an argument which defines the basic metadata of our package. Here are the fields it defines:

pname: This is the name of the package. You could also use name, but you'll have to also specify the package's version in the name. Using pname makes nix automatically generate the name for us.
version: This is the version of your package. Make sure you're fetching the correct binary!
src: Now here's the meat of the package. We use nix's fetchers to fetch our package's binary from the internet. There are two mainly used fetchers for binary packages: fetchurl and fetchzip. Let's take a closer look at them.

fetchurl: This fetcher directly downloads the file provided to it by the url field.

fetchzip: This file downloads the archive provided to it by the url field, and unarchives it. The fetcher may be named fetchzip, but it also works for other archives like .tar.gz.

I'll be using the fetchzip fetcher to download the tarball of the correct SurrealDB version using a direct link pointing to its GitHub release. You can use nix's string interpolation to generate a proper link. This is what mine would look like:

https://github.com/surrealdb/surrealdb/releases/download/v${version}/surreal-v${version}.linux-amd64.tgz

I'll replace PATH_TO_YOUR_PACKAGE'S_TARBALL_HERE with the above link.

Now for the hash field. You must specify a checksum hash for the binary that you're downloading so that nix can verify that the correct file is being downloaded. The easiest way to set this value is to set hash to something random, install the package, and set the hash to whatever it says in the error that gets thrown.

If you'd like to write hash yourself, the syntax for it will be:

hashingAlgorithm-base64encodedhash

Many hashing algorithms are supported, but the most commonly used ones are md5 and sha256. Make sure to base64 encode your hash value, since nix only accepts that! Don't give it a hex string.

Now let's get back to our package derivation. The rest of the code is instructing patchelf to automatically patch the binary to make it run with nix. Since nix doesn't work like other package managers, binaries which expect shared libraries to be at one particular location will not work, hence we need to use patchelf to update these locations. Thankfully we won't have to manually run patchelf, since nix provides us with the autoPatchelf package. This package is defined in the nativeBuildInputs.

The additional libraries which the package depends on must be specified in the buildInputs array. The best way to find this out, would again be to install the package and add the dependencies it lists out in the error, but generally the two packages I've included should suffice.

Finally, in the installPhase, we define a shell script that runs. The hooks that are called are all related to patchelf, so just make sure they're present in your code. In between the two hook calls, we write code to create a bin/ directory in our package's nix-store path, and we transfer any binaries the package provides to the bin/ folder. Do update the PACKAGE_BINARY_FILE_NAME variable with the name of the binary that gets downloaded by the fetcher. In my case, that'd be surreal.

The install command is actually just some syntactic sugar for the cp command, they both have the same function. The only extra thing is the -m flag, which sets chmod permissions on the binary to make it readable, writable and executable by the user (the 7), and only readable and executable by the user's group and everyone else (the two 5s).

Finally, this is how your package derivation should look like:

# /etc/nixos/packages/surrealdb-bin.nix 
{ stdenv, fetchzip, autoPatchelfHook, glibc, gcc-unwrapped }: stdenv.mkDerivation rec {
  pname = "surrealdb-bin";
  version = "1.1.0";

  src = fetchzip {
    url = "https://github.com/surrealdb/surrealdb/releases/download/v${version}/surreal-v${version}.linux-amd64.tgz";
    sha256 = "2611de5eb7779dfe3b32bb47833fee2e3e168e39e43d76b47ea649b2f8c407fa";
  };

  nativeBuildInputs = [ autoPatchelfHook ];
  buildInputs = [ glibc gcc-unwrapped ];

  installPhase = ''
    runHook preInstall
    mkdir -p $out/bin
    install -m755 surreal $out/bin
    runHook postInstall
  '';
}

Creating the overlay

Now we need to create an overlay to be able to add our package to the existing list of nixpkgs. I'll call my overlay derivation surrealdb-bin.nix, and place it in the overlays/ folder in the /ext/nixos directory.

If you're using a different path than me, be sure to update the relevant imports.

Add these lines to your overlay definition:

self: super: {
  surrealdb-bin = super.callPackage ../packages/surrealdb-bin.nix {};
}

Now this may seem a little scarier, if you're new to the nix language. Since nix functions can only accept one argument, we use nested functions to declare multiple arguments. self and super, as they're commonly called, or more recently final and prev, are both instances of nixpkgs. It's just that super/prev is the version of nixpkgs before the overlay is applied, and self/final is the version of nixpkgs after all overlays are applied.

You should generally only use the super argument. Read the wiki if you want to learn more about overlays.

The overlay functions return type is a set of packages which will be merged into nixpkgs. Here you can see that I'm defining one package called surrealdb-bin, and I'm calling the super.callPackage function and giving it the location of my surrealdb-bin package derivation as the argument. The callPackage function ensures that the package derivation is called with the proper arguments supplied. The blank argument set at the end is just to specify that I don't want to extend the list of arguments passed to the package derivation any further.

Using the overlay

The only thing left to do now, is to actually use the overlay and extend the nixpkgs definition. Now this depends on where you plan to use the overlay, i.e. in a shell.nix, flake.nix, the nixos configuration.nix, or in a home-manager configuration. The steps are different for all of those.

Check the wiki for instructions pertaining to your specific use case. Since I want to install surreal globally, I'll add the overlay to my configuration.nix. All I have to do is add the below line somewhere, and then add surrealdb-bin to the list of environment/user packages.

nixpkgs.overlays = [ (import ./overlays/surrealdb-bin.nix) ];

Don't forget to update the path to the overlay if you're using a different path!

And with a small nixos-rebuild switch, I have access to the surreal command in my environment! Great success! Now if there's a new surreal version published, all I have to do is change the version in my package derivation and rebuild!

You should push your package derivation and overlay to a centralised place like GitHub, GitLab, or any other place that provides direct download links (GitHub Gist / GitLab snippets would be a great place), so that you can download this overlay in any nix configuration and maintain reproducibility, instead of possibly having to duplicate your overlay and maintain two sources of truths.

What if I don't have access to binaries

If your package only has the source available, with no binaries, and still takes a long time to compile, then you should probably leave the compilation to a GitHub action.

Just create a GitHub repository, create an action to build the package, and then upload it as an artifact/release once done.

Then use Nix to fetch that artifact in your package.

Do note that GitHub-provided hosted action runners have a maximum runtime of 6 hours, so if you have a package that takes longer than that (e.g. a web browser), then you need to use self-hosted runners, which have a max runtime of 35 days!

It is risky to host a self-hosted runner on your own machine, since your machine may go down, and you'll have to restart the build process again

Instead it'd be best to use a cloud VPS service like Digital Ocean. If you use that link, or this one, you can get $200 in credit for upto 60 days, that's literally giving you free access to a 16GiB ram + 8 vCPU instance to build all your hopes and dreams on! (You will have to request access to these machines first though).

If you use my link, I also get some credits, so it helps me out too! Thanks for using my link.

Anyway, custom GitHub actions are out of scope of this article, but do let me know if you want me to create another article on that topic in the comments below!

Create a Reddit clone with RedwoodJS

arnu515 — Sun, 20 Feb 2022 14:41:46 +0000

Redwood is an opinionated full-stack javascript web application framework. It is also serverless-ready, meaning it can be deployed on the edge with services like AWS Lambda and Cloudflare Workers. Redwood is super opinionated. It decides your project/directory structure, it decides the frameworks and libraries you use, and it configures everything for you. Some may see a downside to this, but if you're experienced with Redwood's choosing of frameworks, you will have a pleasant time using Redwood.

Redwood was created by Tom Preston-Werner. You may have heard of him before, because he is the guy behind

Github, which is the most popular code host
Jekyll, a ruby-based static-site generator
Gravatar, a very popular avatar service
Semver, the semantic versioning system
TOML, a configuration language, like JSON or YAML, and much more.

Prerequisites

Redwood uses React for the frontend framework, so you'll need to know React.
Redwood uses GraphQL instead of REST APIs, so knowledge of that is required. You can learn it on the official website.
Redwood uses Prisma for interacting with databases, but it's very easy to use, and you can pick it up from this tutorial itself. Prisma works with SQL databases.
A Postgres database running. You can either have the Postgres server installed, or use Docker. I'll be doing the latter in this tutorial.
There are various other libraries used like Jest and Storybook, but these are not needed to follow this tutorial.
I'll be using TypeScript in this tutorial, but feel free to use plain JavaScript. Just be sure to remove any code that is TypeScript-specific.

There are also a few things you'll need installed on your computer:

NodeJS v14 or higher. I'll be using v16. (Psst: For an easy way to manage versions of NodeJS and many others, try https://asdf-vm.com).
The Yarn Package Manager installed. Redwood leverages yarn workspaces, so yarn is needed. You can install it using npm i -g yarn.
A powerful code editor like VSCode or (Neo)Vim. If you're using VSCode, be sure to install the Redwood extension.

Step 0 — Creating your Redwood app

Open an empty folder in your favorite IDE and run the below command in the terminal.

yarn create redwood-app .

If you're like me however, and you've fallen in love with TypeScript, you can create a Redwood typescript app by adding the --typescript flag to the above command like so:

yarn create --typescript redwood-app .

If you want to convert an existing Redwood project to TypeScript, you can run yarn rw setup tsconfig and change your .js files to .ts.

Now open the project in your favorite IDE. I'll use VSCode in this tutorial, since Redwood has first-class support for it. Launch the editor and open the folder, or just run code . in the terminal.

You may be prompted to install recommended extensions, so feel free to install them all, or just some if you don't need certain extensions (like Gitlens, in my case).

Step 1 — Getting to know your project

Let's take a look at the project structure.

There are a few files in the root project. Most of them are configuration files, like jest.config.js. Let's take a look at a specific file called redwood.toml.

[web]
  title = "Redwood App"
  port = 8910
  apiUrl = "/.redwood/functions" # you can customise graphql and dbauth urls individually too: see https://redwoodjs.com/docs/app-configuration-redwood-toml#api-paths
  includeEnvironmentVariables = [] # any ENV vars that should be available to the web side, see https://redwoodjs.com/docs/environment-variables#web
[api]
  port = 8911
[browser]
  open = true

Redwood recommends so many extensions, but not a TOML one! Install this extension for VSCode for TOML highlighting.

If you're unfamiliar with TOML syntax, don't worry, I'll guide you through the config. For now, let's change the port of both the [web] and [api] projects to 3000 and 5000 respectively.

Where your code lives

There are many directories, but the main two directories are web, and api. This is where your Redwood code lives. As the name suggests, web is the frontend project, and api is the backend project.

These two directories are actually their own projects, and Redwood uses Yarn Workspaces to link these two folders together.

Inside the web folder:

The web folder is a regular ol' React application. If you know react, you should be able to read through the structure of this directory. There are just a few things that are different. In the src folder, you can see three more subfolders, components, layouts and pages. The components folder holds any re-usable React components. The layouts folder holds page layouts, which are also React components, and the pages folder, which contains React components mapped to routes.

What is a .keep file?
.keep files are just files that are placed in empty directories so they get committed to a git project. Git doesn't commit empty folders, so .keep files are created to make the folder not empty, and hence, get it committed. These .keep files don't even have to be called .keep, they can be called anything else, but by convention, they're called .keep and .gitkeep.

Inside the api folder:

The api folder is the backend server. This is running fastify under the hood, which is just a faster backend server than express. There are a few config files, and there are three subdirectories.

The db folder contains the schema.prisma file, which is the schema for your database models that is used by Prisma ORM.

The src folder contains all of your source code for the backend.

From the redwood documentation:

directives will contain GraphQL schema directives for controlling access to queries and transforming values.
functions will contain any lambda functions your app needs in addition to the graphql.js file auto-generated by Redwood. This file is required to use the GraphQL API.
graphql contains your GraphQL schema written in a Schema Definition Language (the files will end in .sdl.js).
lib contains a few files:auth.js starts as a placeholder for adding auth functionality and has a couple of bare-bones functions in it to start, db.js instantiates the Prisma database client so we can talk to a database and logger.js which configures, well, logging. You can use this directory for other code related to the API side that doesn't really belong anywhere else.
services contains business logic related to your data. When you're querying or mutating data for GraphQL (known as resolvers), that code ends up here, but in a format that's reusable in other places in your application.

Start the server:

Let's start the server by running the below command:

yarn rw dev

You should now see your application running on http://localhost:3000, or http://localhost:8911, if you didn't change the port in the config. The backend will run on port 5000, or 8910 if you didn't change the port in the config.

If this is what you see, you've successfully created your redwood project!

Replacing SQLLite with Postgres:

SQLLite is the default database used by Redwood, since it doesn't expect everyone to have a fully-fledged database installed and running on their computer. But SQLLite is a file-system based database, and it also lacks in features when compared to Postgres. A file-system based database isn't the best for production, so let's switch over to Postgres.

Postgres needs to be installed on your computer. You can download it and install it, and have a system-wide install of postgres, or you can use Docker to containerize it, which is easier to do. You'll need docker installed, however, and you can get it from here.

Once docker is running, you can create a postgres container using the command below:

docker run -d -p 5432:5432 --name postgres -e POSTGRES_PASSWORD=YOUR_STRONG_PASSWORD postgres

Be sure to replace YOUR_STRONG_PASSWORD to a strong password, since that will be the password of your root account in postgres.

The above command will run the postgres image as a container, with the name postgres (with the --name flag), adds the environment variable POSTGRES_PASSWORD to it (with the -e flag), exposes port 5432 (postgres' default port) back to the host (with the -p flag) and finally, it runs it in the background with the -d flag.

Now, create a new database in the fresh postgres container you just created. Run the below command to get shell access to the container:

docker exec -it postgres bash

If your shell prompt changed, you now have the ability to run commands directly in the postgres container! Now run the below command to create a new database.

createdb -U postgres NAME_OF_YOUR_DATABASE

The -U postgres flag makes it run as the postgres user, which is the default root user. Change NAME_OF_YOUR_DATABASE to anything you want. In my case, I changed it to reddit, which means that a new database with the name reddit has been created for me. Once that's done, exit out of the shell by typing exit and hitting Enter.

Now that you have a postgres database, you just need to tell Prisma to use it. Open the .env file in the project root and add the below code to it.

DATABASE_URL=postgres://postgres:YOUR_STRONG_PASSWORD@localhost:5432/YOUR_DATABASE_NAME

Be sure to replace YOUR_STRONG_PASSWORD and YOUR_DATABASE_NAME with the relevant values. And finally, change the line that says provider = "sqlite" to provider = "postgresql" in the api/db/schema.prisma.

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

Step 2 — Creating Prisma models

Prisma models are definitions for how your database tables will look like. They are written in prisma's own model language in the schema.prisma file. If you're not familiar with this syntax, don't fear, since it looks similar to GraphQL syntax, and I'll guide you with it.

Creating the User model

Open the src/db/schema.prisma file in the api project. Let's delete the example UserExample project, and replace it with our own User model.

model User {
  id        String  @id @default(cuid()) @db.VarChar(255)
  username  String  @unique
  email     String  @unique
  password  String  @db.VarChar(255)
  avatarUrl String  @map("avatar_url")
  roles     String  @default("member")
  bio       String?
  isBanned  Boolean @default(false)  @map("is_banned")

  @@map("users")
}

If you can't see any syntax highlighting, be sure to install the Prisma extension for VSCode.

What the above Prisma code does, is it creates a model named User. A Prisma model is mapped to a table in the database, which in this case will be users, because of the @@map("users"). These are the fields that will be created:

An id filed, which will be the primary key (denoted by @id). It will be a String with the VarChar datatype in Postgres. Since VarChar isn't supported by all databases Prisma supports (like MongoDB), we have to use @db.VarChar instead of directly declaring it as a VarChar type. The id will also be a generated CUID by default. A CUID is a randomly-generated string, like a UUID.
A username and an email field, both of which are Strings and are unique, meaning no two users can have the same email or username. By default, a String will be mapped to Postgres' Text datatype.
A password, which is a String in Prisma, but a VarChar in Postgres
An avatarUrl, which is a String. This will be accessed in JavaScript with avatarUrl, but will be stored in the database as avatar_url, because of @map. I did this because Postgres follows snake_casing, while JavaScript follows camelCasing.
A roles, which is a String, which will contain a comma-separated string of roles. You could use an array here, but I feel like that would be overkill for a field that would usually only have one role. Also member is the default.
A bio, which is an optional string (nullable, in database lingo). This is indicated by the ? after String.
And finally, isBanned, which is a Boolean that defaults to false, and is stored as is_banned in the database.

Now you need to sync the models to your database. Currently, they're only present in the schema.prisma file. To create the tables in the database, run the below command:

yarn redwood prisma migrate dev

You can use the alias rw instead of redwood, and that's what I'll be doing from now on.

Be sure to give it a meaningful name. Treat it like a git commit – the name should reflect the changes you've made. In this case, we've created a User model, so I named it add-user-model.

Now your database will have a table called users with all these fields that you just defined.

Creating the Post model

Now it's time to create a model for holding our posts.

model Post {
  id       String  @id @default(cuid()) @db.VarChar(255)
  title    String
  content  String
  hasMedia Boolean @default(false) @map("has_media")
  mediaUrl String?
  authorId String  @map("user_id") @db.VarChar(255)
  author   User    @relation(fields: [authorId], references: [id])

  @@map("posts")
}

You may see a squiggly line under the author field. Don't worry, we'll solve that soon.

The fields in this model are pretty similar to the ones in the User model, except they have different names. There is one odd one out however, and that is authorId. This authorId field will point to the id of the User that created this post, and this is denoted by the author User line. It has an @relation directive that relates the id field of User to the authorId field of Post. Prisma also requires that we include a backref – a field on the other table that points back to this one indicating the relation. Since this will be a one-to-many (O2M) relation, i.e. one user can have many posts, the post backref in the User model should be an array. You can denote that by putting square brackets ([]) after the type, just like in regular TypeScript.

model User {
  id        String  @id @default(cuid()) @db.VarChar(255)
  username  String  @unique
  email     String  @unique
  password  String  @db.VarChar(255)
  avatarUrl String  @map("avatar_url")
  roles     String  @default("member")
  bio       String?
  isBanned  Boolean @default(false) @map("is_banned")
  post      Post[]  // Add this line to your User model. This is the backref!

  @@map("users")
}

model Post {
  id       String  @id @default(cuid()) @db.VarChar(255)
  title    String
  content  String
  hasMedia Boolean @default(false) @map("has_media")
  mediaUrl String?
  authorId String  @map("user_id") @db.VarChar(255)
  author   User    @relation(fields: [authorId], references: [id])

  @@map("posts")
}

While we're at it, let's also add a Comment model, which will store comments on a post. This model will have two relations — both O2M — one with the User model, and the other with the Post model.

model User {
  id        String  @id @default(cuid()) @db.VarChar(255)
  username  String  @unique
  email     String  @unique
  password  String  @db.VarChar(255)
  avatarUrl String  @map("avatar_url")
  roles     String  @default("member")
  bio       String?
  isBanned  Boolean @default(false) @map("is_banned")

  post     Post[]  // backref to Post
  comments Comment[]  // backref to Comment

  @@map("users")
}

model Post {
  id       String  @id @default(cuid()) @db.VarChar(255)
  title    String
  content  String
  hasMedia Boolean @default(false) @map("has_media")
  mediaUrl String?
  authorId String  @map("user_id") @db.VarChar(255)
  author   User    @relation(fields: [authorId], references: [id])

  comments Comment[]  // backref to Comment

  @@map("posts")
}

// Our comment model
model Comment {
  id       String @id @default(cuid()) @db.VarChar(255)
  content  String
  authorId String @map("user_id") @db.VarChar(255)
  author   User   @relation(fields: [authorId], references: [id])
  postId   String @map("post_id") @db.VarChar(255)
  post     Post   @relation(fields: [postId], references: [id])

  @@map("comments")
}

You should notice that the fields that are in a relation have the same type as the field they're in a relationship with. This is important, since they'll be storing the same type of data.

Let's migrate our database! Run the same command as before, and this time, you can give it a name directly in the command line with the --name argument.

yarn rw prisma migrate dev --name add-post-and-comment-models

With that, our three basic models have been created. Let's now use them in the Redwood project.

Step 2 — Adding authentication to your app

Redwood makes it really easy to add authentication to your application. It handles almost everything that is boring, like sessions and stuff like that.

Let's use the Redwood CLI and sets up authentication for you. Run the below command to do that:

yarn rw setup auth dbAuth

This will setup a local authentication provider that saves your users' credentials on the database. Redwood also supports some authentication-as-a-service providers out-of-the-box, like Auth0 and Magic. Read more about that here.

A few new files have been created. You need to edit these files to make it work with your schema. First, let's edit src/lib/auth.ts. This file contains methods that are used by Redwood under-the-hood to determine if a user is authenticated and authorized to access a resource.

You only need to do one small edit – make Redwood read the roles stored in the users table in the hasRole function. But first. let's make the getCurrentUser function return the whole user, instead of just the user's id:

export const getCurrentUser = async (session) => {
  return await db.user.findUnique({
    where: { id: session.id },
  })
}

In the above snippet, I just removed the select {...} from the query so it returns all fields of the user. We can now use this in the hasRole function. Change out the hasRole function to the one below:

export const hasRole = ({ roles }: { roles: AllowedRoles }): boolean => {
  if (!isAuthenticated()) {
    return false
  }

  // If your User model includes roles, uncomment the role checks on currentUser
  if (roles) {
    if (Array.isArray(roles)) {
      // the line below has changed
      if (context.currentUser.roles)
        return context.currentUser.roles
          .split(',')
          .some((role) => roles.includes(role))
    }

    if (typeof roles === 'string') {
      // the line below has changed
      if (context.currentUser.roles)
        return context.currentUser.roles.split(',').includes(roles)
    }

    // roles not found
    return false
  }

  return true
}

This code should now check the roles in the database instead of returning false by default.

Adding some fields to the User model:

Redwood gives you a hashedPassword, a salt, a resetToken and a resetTokenExpiresAt to store in your database, but the current User model can only store the password. Let's change that by adding three new fields to the User model by changing the User model in schema.prisma to this:

model User {
  id            String    @id @default(cuid()) @db.VarChar(255)
  username      String    @unique
  email         String    @unique
  password      String    @db.VarChar(255)
  // added the below three lines
  salt          String    @db.VarChar(255)
  resetToken    String?   @db.VarChar(255)
  resetTokenExp DateTime? @db.Timestamptz()
  avatarUrl     String    @map("avatar_url")
  roles         String    @default("member")
  bio           String?
  isBanned      Boolean   @default(false) @map("is_banned")

  post     Post[]
  comments Comment[]

  @@map("users")
}

If you were messing around and created a few users, your migration will fail because salt is empty, and salt is not allowed to be empty. So, just add @default("") to the salt field in the schema to ensure that already-existing users won't have null values for required fields.

Next, migrate with the below command:

yarn rw prisma migrate dev --name add-fields-to-user

Of course, you can use your own migration name.

Next, you'll need to generate types so Redwood knows about the new User.

yarn rw generate types

And finally, you need to restart the dev server. Press Ctrl+C (maybe twice) to stop the current running dev server and run yarn rw dev to start it again.

Configuring authentication:

There are just a few final changes you need to make to the src/functions/auth.ts file, such as setting an avatarUrl. For the avatarUrl, we'll use Gravatar, which is a popular avatar service. For that, you just need to use the below URL as the avatarUrl

https://gravatar.com/avatar/EMAIL_HASH?d=mp&s=64

The EMAIL_HASH should be an md5 hash of the user's email. For generating an md5 hash, let's install the md5 package (along with its typescript definitions) with the below commands:

yarn workspace api add md5 && yarn workspace api add -D @types/md5

We're using workspace api add instead of just add because there are two workspaces here, and we just want to add md5 to the api folder, not the web folder.

Now, let's change the src/functions/auth.ts file to make sure it works with our requirements:

import { db } from 'src/lib/db'
import { DbAuthHandler } from '@redwoodjs/api'
import md5 from 'md5'

export const handler = async (event, context) => {
  const forgotPasswordOptions = {
    // handler() is invoked after verifying that a user was found with the given
    // username. This is where you can send the user an email with a link to
    // reset their password. With the default dbAuth routes and field names, the
    // URL to reset the password will be:
    //
    // https://example.com/reset-password?resetToken=${user.resetToken}
    //
    // Whatever is returned from this function will be returned from
    // the `forgotPassword()` function that is destructured from `useAuth()`
    // You could use this return value to, for example, show the email
    // address in a toast message so the user will know it worked and where
    // to look for the email.
    handler: (user) => {
      return user
    },

    // How long the resetToken is valid for, in seconds (default is 24 hours)
    expires: 60 * 60 * 24,

    errors: {
      // for security reasons you may want to be vague here rather than expose
      // the fact that the email address wasn't found (prevents fishing for
      // valid email addresses)
      usernameNotFound: 'Email not found',
      // if the user somehow gets around client validation
      usernameRequired: 'Email is required',
    },
  }

  const loginOptions = {
    // handler() is called after finding the user that matches the
    // username/password provided at login, but before actually considering them
    // logged in. The `user` argument will be the user in the database that
    // matched the username/password.
    //
    // If you want to allow this user to log in simply return the user.
    //
    // If you want to prevent someone logging in for another reason (maybe they
    // didn't validate their email yet), throw an error and it will be returned
    // by the `logIn()` function from `useAuth()` in the form of:
    // `{ message: 'Error message' }`
    handler: (user) => {
      return user
    },

    errors: {
      usernameOrPasswordMissing: 'Both email and password are required',
      usernameNotFound: 'Email ${username} not found',
      // For security reasons you may want to make this the same as the
      // usernameNotFound error so that a malicious user can't use the error
      // to narrow down if it's the username or password that's incorrect
      incorrectPassword: 'Incorrect password for ${username}',
    },

    // How long a user will remain logged in, in seconds
    expires: 60 * 60 * 24 * 365 * 10,
  }

  const resetPasswordOptions = {
    // handler() is invoked after the password has been successfully updated in
    // the database. Returning anything truthy will automatically logs the user
    // in. Return `false` otherwise, and in the Reset Password page redirect the
    // user to the login page.
    handler: (user) => {
      return user
    },

    // If `false` then the new password MUST be different than the current one
    allowReusedPassword: true,

    errors: {
      // the resetToken is valid, but expired
      resetTokenExpired: 'resetToken is expired',
      // no user was found with the given resetToken
      resetTokenInvalid: 'resetToken is invalid',
      // the resetToken was not present in the URL
      resetTokenRequired: 'resetToken is required',
      // new password is the same as the old password (apparently they did not forget it)
      reusedPassword: 'Must choose a new password',
    },
  }

  const signupOptions = {
    // Whatever you want to happen to your data on new user signup. Redwood will
    // check for duplicate usernames before calling this handler. At a minimum
    // you need to save the `username`, `hashedPassword` and `salt` to your
    // user table. `userAttributes` contains any additional object members that
    // were included in the object given to the `signUp()` function you got
    // from `useAuth()`.
    //
    // If you want the user to be immediately logged in, return the user that
    // was created.
    //
    // If this handler throws an error, it will be returned by the `signUp()`
    // function in the form of: `{ error: 'Error message' }`.
    //
    // If this returns anything else, it will be returned by the
    // `signUp()` function in the form of: `{ message: 'String here' }`.
    handler: (payload) => {
      const { username: email, hashedPassword, salt, userAttributes } = payload
      return db.user.create({
        data: {
          email,
          password: hashedPassword,
          avatarUrl: `https://gravatar.com/avatar/${md5(email)}?d=mp&s=64`,
          salt: salt,
          username: userAttributes.name,
        },
      })
    },

    errors: {
      // `field` will be either "username" or "password"
      fieldMissing: '${field} is required',
      usernameTaken: 'Email `${username}` already in use',
    },
  }

  const authHandler = new DbAuthHandler(event, context, {
    // Provide prisma db client
    db: db,

    // The name of the property you'd call on `db` to access your user table.
    // ie. if your Prisma model is named `User` this value would be `user`, as in `db.user`
    authModelAccessor: 'user',

    // A map of what dbAuth calls a field to what your database calls it.
    // `id` is whatever column you use to uniquely identify a user (probably
    // something like `id` or `userId` or even `email`)
    authFields: {
      id: 'id',
      username: 'email',
      hashedPassword: 'password',
      salt: 'salt',
      resetToken: 'resetToken',
      resetTokenExpiresAt: 'resetTokenExp',
    },

    forgotPassword: forgotPasswordOptions,
    login: loginOptions,
    resetPassword: resetPasswordOptions,
    signup: signupOptions,
  })

  return await authHandler.invoke()
}

All I did above was change the hashedPassword field to password, and the username field to email. I also replaced instances of Username in messages to Email, and I added the avatarUrl field.

Adding login and signup pages

Let's add login and signup pages to the frontend. Redwood makes this really easy by providing a generator for us. Run the below command to create a login page, a signup page, and a forgot and reset password page.

yarn rw g dbAuth

I will delete the ForgotPassword and ResetPassword pages, since I won't be adding that functionality to this project.

Next, you need to replace the username field in both Login and SignUp to email, and in SignUp, add a new field called username. I've done it below and here's how your code should look like:

// Routes.tsx

import { Router, Route } from '@redwoodjs/router'

const Routes = () => {
  return (
    <Router>
      <Route path="/login" page={LoginPage} name="login" />
      <Route path="/signup" page={SignupPage} name="signup" />
      <Route notfound page={NotFoundPage} />
    </Router>
  )
}

export default Routes

// LoginPage.tsx

import { Link, navigate, routes } from '@redwoodjs/router'
import { useRef } from 'react'
import {
  Form,
  Label,
  TextField,
  PasswordField,
  Submit,
  FieldError,
} from '@redwoodjs/forms'
import { useAuth } from '@redwoodjs/auth'
import { MetaTags } from '@redwoodjs/web'
import { toast, Toaster } from '@redwoodjs/web/toast'
import { useEffect } from 'react'

const LoginPage = () => {
  const { isAuthenticated, logIn } = useAuth()

  useEffect(() => {
    if (isAuthenticated) {
      navigate(routes.home())
    }
  }, [isAuthenticated])

  const emailRef = useRef<HTMLInputElement>()
  useEffect(() => {
    emailRef.current.focus()
  }, [])

  const onSubmit = async (data) => {
    const response = await logIn({ ...data })

    if (response.message) {
      toast(response.message)
    } else if (response.error) {
      toast.error(response.error)
    } else {
      toast.success('Welcome back!')
    }
  }

  return (
    <>
      <MetaTags title="Login" />

      <main className="rw-main">
        <Toaster toastOptions={{ className: 'rw-toast', duration: 6000 }} />
        <div className="rw-scaffold rw-login-container">
          <div className="rw-segment">
            <header className="rw-segment-header">
              <h2 className="rw-heading rw-heading-secondary">Login</h2>
            </header>

            <div className="rw-segment-main">
              <div className="rw-form-wrapper">
                <Form onSubmit={onSubmit} className="rw-form-wrapper">
                  <Label
                    name="username"
                    className="rw-label"
                    errorClassName="rw-label rw-label-error"
                  >
                    Email
                  </Label>
                  <TextField
                    name="username"
                    className="rw-input"
                    errorClassName="rw-input rw-input-error"
                    ref={emailRef}
                    validation={{
                      required: {
                        value: true,
                        message: 'Email is required',
                      },
                    }}
                  />

                  <FieldError name="username" className="rw-field-error" />

                  <Label
                    name="password"
                    className="rw-label"
                    errorClassName="rw-label rw-label-error"
                  >
                    Password
                  </Label>
                  <PasswordField
                    name="password"
                    className="rw-input"
                    errorClassName="rw-input rw-input-error"
                    autoComplete="current-password"
                    validation={{
                      required: {
                        value: true,
                        message: 'Password is required',
                      },
                    }}
                  />

                  <FieldError name="password" className="rw-field-error" />

                  <div className="rw-button-group">
                    <Submit className="rw-button rw-button-blue">Login</Submit>
                  </div>
                </Form>
              </div>
            </div>
          </div>
          <div className="rw-login-link">
            <span>Don&apos;t have an account?</span>{' '}
            <Link to={routes.signup()} className="rw-link">
              Sign up!
            </Link>
          </div>
        </div>
      </main>
    </>
  )
}

export default LoginPage

import { Link, navigate, routes } from '@redwoodjs/router'
import { useRef } from 'react'
import {
  Form,
  Label,
  TextField,
  PasswordField,
  FieldError,
  Submit,
} from '@redwoodjs/forms'
import { useAuth } from '@redwoodjs/auth'
import { MetaTags } from '@redwoodjs/web'
import { toast, Toaster } from '@redwoodjs/web/toast'
import { useEffect } from 'react'

const SignupPage = () => {
  const { isAuthenticated, signUp } = useAuth()

  useEffect(() => {
    if (isAuthenticated) {
      navigate(routes.home())
    }
  }, [isAuthenticated])

  // focus on email box on page load
  const emailRef = useRef<HTMLInputElement>()
  useEffect(() => {
    emailRef.current.focus()
  }, [])

  const onSubmit = async (data) => {
    const response = await signUp({ ...data })

    if (response.message) {
      toast(response.message)
    } else if (response.error) {
      toast.error(response.error)
    } else {
      // user is signed in automatically
      toast.success('Welcome!')
    }
  }

  return (
    <>
      <MetaTags title="Signup" />

      <main className="rw-main">
        <Toaster toastOptions={{ className: 'rw-toast', duration: 6000 }} />
        <div className="rw-scaffold rw-login-container">
          <div className="rw-segment">
            <header className="rw-segment-header">
              <h2 className="rw-heading rw-heading-secondary">Signup</h2>
            </header>

            <div className="rw-segment-main">
              <div className="rw-form-wrapper">
                <Form onSubmit={onSubmit} className="rw-form-wrapper">
                  <Label
                    name="username"
                    className="rw-label"
                    errorClassName="rw-label rw-label-error"
                  >
                    Email
                  </Label>
                  <TextField
                    name="username"
                    className="rw-input"
                    errorClassName="rw-input rw-input-error"
                    ref={emailRef}
                    validation={{
                      required: {
                        value: true,
                        message: 'Email is required',
                      },
                    }}
                  />
                  <FieldError name="username" className="rw-field-error" />

                  <Label
                    name="name"
                    className="rw-label"
                    errorClassName="rw-label rw-label-error"
                  >
                    Username
                  </Label>
                  <TextField
                    name="name"
                    className="rw-input"
                    errorClassName="rw-input rw-input-error"
                    ref={emailRef}
                    validation={{
                      required: {
                        value: true,
                        message: 'Username is required',
                      },
                    }}
                  />
                  <FieldError name="name" className="rw-field-error" />

                  <Label
                    name="password"
                    className="rw-label"
                    errorClassName="rw-label rw-label-error"
                  >
                    Password
                  </Label>
                  <PasswordField
                    name="password"
                    className="rw-input"
                    errorClassName="rw-input rw-input-error"
                    autoComplete="current-password"
                    validation={{
                      required: {
                        value: true,
                        message: 'Password is required',
                      },
                    }}
                  />
                  <FieldError name="password" className="rw-field-error" />

                  <div className="rw-button-group">
                    <Submit className="rw-button rw-button-blue">
                      Sign Up
                    </Submit>
                  </div>
                </Form>
              </div>
            </div>
          </div>
          <div className="rw-login-link">
            <span>Already have an account?</span>{' '}
            <Link to={routes.login()} className="rw-link">
              Log in!
            </Link>
          </div>
        </div>
      </main>
    </>
  )
}

export default SignupPage

Notice how I've only changed the text but not the name of the inputs? This is because under-the-hood, Redwood still expects your email to be called username.

For one final step, let's add a home page so we don't have to see the Redwood logo anymore. Use the below command to generate an index page at /.

yarn rw g page home /

This command will generate a page called home, but map it to /, instead of /home. Change the code of the newly created HomePage.tsx to this:

import { MetaTags } from '@redwoodjs/web'

const HomePage = () => {
  return (
    <>
      <MetaTags
        title="Redwoodit"
        description="A clone of Reddit using RedwoodJS"
      />

      <h1>Redwoodit</h1>
    </>
  )
}

export default HomePage

And with that, you've added authentication to your Redwood application.

If you visit http://localhost:3000/signup, you can create an account and if you visit http://localhost:3000/login, you can log in to an account.

Conclusion

You've successfully created a Redwood application and added authentication to it. In the next part of this tutorial, you will add support for fetching, creating, updating and deleting posts and comments. If you've gotten stuck anywhere, be sure to check out the source code, the Redwood documentation, or ask in the Redwood Discord/Discourse Forums for help.

Stay tuned for Part 2!

Simple Authenticator - The Simplest TOTP Application

arnu515 — Fri, 14 Jan 2022 07:30:51 +0000

Overview of My Submission

Simple Authenticator is an application that generates TOTP codes for two-factor authentication - like Google Authenticator. It can optionally store your configured applications encrypted in the cloud with MongoDB Atlas.

Github Repository: https://github.com/arnu515/simple-authenticator

Link to download (Android only): https://github.com/arnu515/simpleauthenticator/releases

Submission Category:

[Note]: Choose your own Adventure

Link to Code

arnu515 / simpleauthenticator

Simple TOTP Authenticator App built with Flutter

simpleauthenticator

Simple TOTP Authenticator

Download

I recommend running the application with flutter run, because the APK is not signed. View instructions below.

Get the Android APK file from Releases.

You may get a play-protect warning, and that's because I haven't yet signed the application. This warning can be ignored.

I will publish it soon on F-Droid and Play store.

Run locally

You can run the application on Windows, Mac or Linux by using the below command. Make sure to have the flutter SDK installed.

flutter run --dart-define "API_URL=URL_TO_BACKEND"

If you haven't hosted the backend yourself, you can use https://d13c320db282.up.railway.app.

You can also run the app on Android if you have the Android SDK installed by connecting your phone to your computer and turning on USB debugging, and running the above command.

You can also build the app by replacing flutter run with flutter build…

View on GitHub

Connect VSCode to a Docker container

arnu515 — Sat, 18 Sep 2021 02:40:24 +0000

In the video below, you'll learn how you can connect VSCode to a Docker container using the Remote Containers extension.

Automate your workflow with Microsoft Power Automate

arnu515 — Mon, 12 Jul 2021 16:50:41 +0000

The most tedious part of development is doing the same tasks over and over again. Don't you feel bored having to mention someone on Slack when they get assigned a Github Issue, or having to send an email to your newsletter when you post a tweet.

Sure, you can setup integrations, webhooks and CI jobs to do these tasks, but they're often tedious, require signups to hundreds of services, and all of them might not have the same steps.

Enter Microsoft Power Automate, also called Microsoft Flow, which allows you to automate almost everything in your tech stack. Want to send a message on Slack or Teams, sure, want to SMS a number with Twilio, you can do that too. It also has many listeners, like when a new Tweet is posted, or when a Github Issue is assigned to you.

In this tutorial, we'll take a look at how we can automate a few common tasks with Microsoft Flow.

Prerequisites

An Office365 Microsoft Account
Accounts for the services we'll be automating (Github, Slack, etc).

Flow 1 - Mail me when a Github Issue is assigned to me

First, create a new Automated Cloud flow and give it a name.

Add the When an issue is assigned to me trigger from Github. You will now be asked to sign in to Github, which you should do.

Next, add the Mail action, which uses Sendgrid under the hood.

You can notice that I've added some dynamic elements that were provided to me by the Github Trigger.

You can add more things to your flow, for example, add an item on Microsoft Todo, create a Trello card, or message yourself on Slack.

Click Save when done, and you will now be emailed whenever an issue's been assigned to you!

Flow 2 - Post your website's status to your Twitter everyday at 6 AM

Create a Scheduled Cloud Flow, since this event is based on time.

Make sure the date is set to this date, and the time is set to 6:00 AM.

Also set the task to repeat every 1 day. See the below image if you get confused.

Add a HTTP action that'll query your website's status page. My website uses Cachet to display its status, so I'll tailor my flow to Cachet's API response.

Next, you need to parse the data down to access the status_name field in the JSON response.

Finally, you can tweet out the status of the website. You'll need to sign in to Twitter for this flow to work.

Click Save when done, and now, at 6 AM daily, your followers will be notified of your website's status.

Conclusion

See how easy it is to automate the small things? The best part is that Microsoft Flow is included with the Office365 subscription, so along with Excel, Word and PowerPoint, you also get this monster of an automation tool that really, nobody cares about! So go ahead and share this article with your friends, so they can also use this wonderful tool.

Everything new in Flask 2.0

arnu515 — Tue, 22 Jun 2021 15:49:25 +0000

Photo by Markus Spiske on Unsplash

Flask is one of the most used Python Frameworks in Web Development. It recently came out with a new release - 2.0.0.

In this post, we'll go through the major changes. If you want the full change log, click here.

Changes

Let's go over the major changes introduced in Flask 2.0.0

Dropped support for Python 2 and 3.5

The most breaking change is the dropped support for Python 2 and 3.5. Well, it's about time we move from those versions anyway, since Python 2 is really old!

Also, dropping 3.5 and 2 support is required for the next change:

Type hints

Python 3.7 added support for type annotations. This is where you can add type hints to variables to tell others what type it will be. This is a God send for people with linters and autocompletors, since you no longer have to view docs to find the return type of a certain function.

Keep in mind however, that type hints aren't enforced, meaning that this is completely valid code, as far as the python interpreter goes



a: str = 5

To check your types, use an external linter like mypy

Flask 2.0.0 is now fully typed, so no more awkward moments when you import request from flask and your IDE won't autocomplete its methods.

You can see the difference between the type hints in Flask 2.0.0 and its earlier versions with the help() function:

The above image shows Flask 1.1.4's docstring and the below image shows Flask 2.0.1's docstring. Notice the type hints

New `Config.from_file()` method

You may remember the Config.from_json(), or the app.config.from_json() method that you use to configure Flask with a JSON File. In Flask 2.0.0, it has been deprecated in favour of Config.from_file.

The syntax of this method is as follows:



app.config.from_file(filename: str, loads_function)

Now, if we wanted to implement the old from_json behaviour, look at the code below:



import json

app.config.from_file(filename: str, json.load)

This opens us to do the same to parse TOML files as well:



import toml

app.config.from_file(filename: str, toml.load)

You need to install the toml package for this!

New route decorators

Flask has new decorators for defining routes now. Before we used @app.route(path: str) to define a route in our app, and for methods other than GET we add the methods parameter to our decorator.

Now, Flask has followed ExpressJS's routes and added decorators for defining routes specific to HTTP Methods.



# You can now use these instead of app.route
@app.get(path)
@app.post(path)
@app.put(path)
@app.delete(path)
@app.patch(path)

Smaller changes

Here's a list of smaller changes that may affect your projects:

Some send_file parameters have been renamed, the old names are deprecated. attachment_filename is renamed to download_name. cache_timeout is renamed to max_age. add_etags is renamed to etag.
When loading a .env or .flaskenv file, the current working directory is no longer changed to the location of the file.
helpers.total_seconds() is deprecated. Use timedelta.total_seconds() instead.

And my favorite feature of this list:

flask shell sets up tab and history completion like the default python shell if readline is installed

How to upgrade

Change your Flask dependency in requirements.txt, Pipfile or pyproject.toml to this:



Flask==2.0.0

And run:



pip3 install -r requirements.txt

You can no longer use python2, so beware of that!

Conclusion

This is of course not the full change log. If you want the full change log, click here.

DEV Community: arnu515

Using DigitalOcean Droplets as Ephemeral Sandboxes for AI Agents

Introduction

Key Takeaways

Prerequisites

Programmatically Creating DigitalOcean Droplets

Moving Beyond Standard Images

Defining the Sandbox Configuration

Provisioning the Droplet

Monitoring and Testing

Cleaning Up

Why Not Containers?

Comparison with Other Sandbox Services

Using Prebuilt Images

Persist Sandbox Contents

Attaching a Fixed IP to the Sandbox

Use-case: A Ralph loop Agent

FAQs

What happens to data stored on a sandbox Droplet when it is destroyed?

How does DigitalOcean’s per-second billing benefit automated AI workflows?

Can I attach a single DigitalOcean Volume to multiple sandbox Droplets simultaneously?

Why should I choose cloud-init over standard custom snapshots for my sandbox environments?

Conclusion

SupportAddress — An email-first support platform

What I Built

Demo

Code Repository

arnu515 / supportaddress

Email-first customer support

SupportAddress

How I Built It

I wanna Do ____! — An AI powered habit building application

What I Built

Demo

Repo

arnu515 / i-wanna-do

AI powered habit building app!

I wanna Do!

Copilot Experience

GitHub Models

Conclusion

Easy development environments with Nix and Nix flakes!

Let's revise: Creating a Shell

Using nix shell to create a development environment

Creating a Development Shell with Flakes

Automatic shell environments with direnv

Other alternatives to development shells built with Nix

Conclusion

Getting started with Nix and Nix Flakes

Installing Nix

Using the official installer?

No Systemd?

Using Nix packages

Actually installing lolcat

Nix language basics

An introduction to flakes

Conclusion

My new Nix series!

A quick introduction to Nix

Planned Articles

Support me

The one thing I do not like about the Nix package manager (and a fix for it)

The fix

Getting set up

Creating the package

Creating the overlay

Using the overlay

What if I don't have access to binaries

Create a Reddit clone with RedwoodJS

Prerequisites

Step 0 — Creating your Redwood app

Step 1 — Getting to know your project

Where your code lives

Step 2 — Creating Prisma models

Creating the User model

Creating the Post model

Step 2 — Adding authentication to your app

Adding login and signup pages

Conclusion

Simple Authenticator - The Simplest TOTP Application

Using `nix shell` to create a development environment

Automatic shell environments with `direnv`

Actually installing `lolcat`

New `Config.from_file()` method