DEV Community: Ante Barić

How single message broke all our monitoring and dashboards

Ante Barić — Thu, 02 May 2024 16:22:16 +0000

So.. this just happened (or depending on when you are reading, it happened some time ago) and while I look at my beautiful dashboards working again I am contemplating, despite all the modern tech, how fragile software can be these days…

For the last 1.5 years, I have been using Axiom for all of my logs ingestion, querying, and monitoring needs. It is a great product and I never had one issue with it in my time using it. Spoiler alert, even today, when it failed it was actually my fault, but let's see what happened.

Axiom has a feature called virtual fields that allows you to compute log properties on the fly from other properties. One use case is parsing JSON payloads in logs and then expanding them with fields from that parsed JSON. For example, if I have the log entry:

{
  "path": "/example",
  "message": "{\"event\":\"impression\",\"id\":\"jhfdsf334j24j\"}"
}

I can use the virtual field to extract event property from that log entry and then filter over it as with any other "real" log field. Virtual fields are (like other queries) created in Axiom Processing Language (APL). It looks something like this:

['dataset']
| extend event = iff(isnotnull(parse_json(message)), parse_json(message)["event"], "")

To explain it quickly:

iff returns first non null value (value of event or empty string)
isnotnull checks if the value is null, to avoid parsing fields that are not JSON
parse_json parses value as JSON

With the above I could query all of my logs in Axiom by event field which was great, up until today. I got an email from Axiom saying that one of my monitors was auto-disabled. This was due to query, that monitor was using, failing multiple times. I opened it to see what was going on and quickly closed it because I was doing something else. A few minutes passed and I noticed one of the dashboards I was using was broken, I refreshed the page thinking it was a temporary error, but it did not help. The error said, "cannot index value of type string".

Since I did not change the dashboard or any setup on my side for weeks and after checking Axiom status page I reported an issue on their Discord server. I quickly got a reply that all systems were green from their end. One thing that Axiom representative said is to try and change the query time to something else eg. query logs from yesterday instead of the last X hours how I had it set up. To my surprise changing it to yesterday made my dashboard work again. At this point, I figured that it must be some specific log entry in some period that is causing an error. I played with FROM and TO dates and narrowed it down to the period of 1s containing 40 log entries. And yep, you read it right, in millions of logs it was down to 40 entries in the span of less than a second. Now I had to go in and look up which one of them was faulty.

Before continuing try and guess which one of the above was a culprit?

Knowing one of them was a culprit I quickly opened JavaScript console in Chrome, pipped the entries, and executed JSON.parse on them. All of them returned SyntaxError except for one entry. It was the "email" entry. If you parse that string (with double quotes) as JSON you get back a string email (without double quotes). Which perfectly matched the error from my dashboard saying "cannot index value of type string". My virtual field was trying to extract event property from string email , something like that is a no-op. Since I am using those virtual fields in almost all the places it broke literally everything. What a mess right?

To fix this I had to replace my isnotnull check with something else since it did not cover the above case. Browsing Axiom docs I found gettype utility function that returns the type of data passed to it. I also found exact example in their docs that parse_json can return a string.

By using the above function my fixed virtual field now extracts the value only if the parsed JSON value is a dictionary (or object in JavaScript terms). It looks like this:

['dataset']
| extend event = iff(gettype(parse_json(message)) == "dictionary", parse_json(message)["event"], "")

And just like that, everything was working again, and he lived happily ever after…

Thanks for reading! 🙈

If you enjoyed this post I would appreciate the like or share and if you are interested here are some of my other posts/projects:

How I did 100,000 push-ups in a year 💪
Integrating Rust into Next.js - How-To Developer Guide
Get Komfy App - Stream your favorite video content from your phone directly to your PC or Mac 📺
How I hacked Wordle - the most popular game in the world ⌨️
Jawa - Visual scraper interface - exports to puppeteer script which you can run anywhere and scrape anything 🤖

Integrating Rust into Next.js: How-To Developer Guide

Ante Barić — Tue, 12 Mar 2024 14:55:25 +0000

Rust is one of the latest languages that got a lot of spotlight in the developer community. When talking performance it rivals C/C++ but it is also accessible enough to be used for a wide range of tasks and can run on many different platforms. One of those platforms is, of course, the web. In this post we will dive into how you can start using Rust in your web projects and platforms you are already familiar with.

Web development was caught in the Rust storm and so far it looks like a great fit. Rust entered on a high note with standard JavaScript tooling like bundlers, compilers, test runners, and even runtimes being (re)written in Rust with massive performance gains. As a long-time programmer who worked in production projects with a bunch of different languages like PHP, Python, Objective C, Swift, and JavaScript my goal with learning a new language is always to start writing production code as soon as possible. Don't get me wrong, fundamentals are important but where you see and learn the nuances of the language is solving real problems.

Recently my focus has been on the backend and APIs so my goal with Rust was to start writing new API endpoints for my projects in Rust instead of the usual JavaScript/Node.js. If you read any of my articles you might already know that I use Vercel to host most of my projects. I like the versatility it provides, allows me to deploy a bunch of different frameworks and mostly forget about the hurdle of managing infrastructure for the apps I am working on and just focus on delivering features. Naturally, I was curious if it is possible to build, deploy, and run Rust code on the Vercel platform and if I can integrate it into my current (mostly) JavaScript-based projects.

To my delight, there are multiple ways to deploy Rust code on Vercel. We will go through different options and then go into details with the option I chose to go with.

Using WebAssembly (Wasm) at the Edge. This allows us to write Rust code which we can then compile into .wasm binary which we can import as any other package/file into our JavaScript code.
Using custom Rust runtime. Aside from deploying Node.js and frameworks such as Next.js, Vercel supports a range of custom runtimes that allow the deployment of native code compiled from other languages like Go, Python, and also Rust. Some of those runtimes are Vercel-maintained and some, like Rust runtime, are community-maintained. Both work pretty well (I used PHP and Python runtimes before).
Using Rust runtime in the Next.js project. While using Rust runtime with Vercel is great, a lot of my projects are actually on frameworks like Next.js. To boost the number of projects in which I can use Rust I decided to integrate a template using Rust runtime with Next.js. That way I could gradually implement some features for my projects in Rust. It will also allow me to learn Rust more by using it each day.

Setup

We are going to start with a fresh create-next-app project to show the modifications needed to make this work. This will also provide you with steps you can take to do it in your project as well. Of course, I also have my template linked on GitHub below.

When you have your Next.js project up and running we can start to put Rust in the mix. If you were wondering this should all work on Next.js version 12.x, 13.x and 14.x and also with pages or app directory. To make use of custom runtimes we need to add vercel.json configuration file to our project.

We are also going to run npm install vercel -D to add vercel CLI to our dev dependencies. We will use it to run a local server for our Rust API endpoints. What this will do is tell Vercel to deploy any of our .rs files as serverless API functions with Rust runtime. The next thing is to create top-level api/ directory inside of our project. This differs from pages/api/ directory that you might already have in your Next.js project. If you have any code inside pages/api/ just keep it there as is and proceed to create top-level /api directory for our Rust runtime functions.

You will also need to have Rust installed on your local machine. The quickest way is with rustup, so you can go ahead and install it if you don't have it already. Rust uses cargo as a package manager and build system. We get everything we need to build and run our Rust code locally with a single command. To enable cargo we need to create Cargo.toml configuration file in the root of our project.

If you are familiar with package.json it has some similarities. We add some general metadata like name and version, and we then add dependencies we need to build our API endpoints. In Rust dependencies are called "crates" and are hosted on crates.io We use tokio as async runtime, serde_json for JSON parsing/transform and vercel_runtime which is Rust equivalent of Vercel or Next.js APIs to manipulate requests and responses. Also, if you are using VSCode you can install rust-analyzer extension which will enable all the Rust language features and give you some more help and context when writing code.

Next, we are going to create our first Rust API endpoint. We will name it api/crab.rs as we wrote in our cargo config file.

Now, if you never wrote a line of code in Rust this might not be 100% clear but if you used Vercel or some HTTP server in any language it should be mostly readable. We create a handler function that will be called to respond when API request is made to GET /api/crab. It responds with 200 OK status code and returns a message in JSON format.

We are going to quickly test it by running npx vercel dev. If you created a new project this will prompt you to set up your Vercel project, just follow the prompts to set it up. When it goes through it will say something like "Ready! Available at localhost:3000". If you open http://localhost:3000/api/crab in your browser after a few seconds you should see the response.

Response from our Rust API endpointIf you refresh again you will see the response comes back almost instantly. That is because vercel dev under the hood used cargo to build and run our Rust file on the fly. This also automatically detects changes so it will only recompile when you change something in the code. So just write code and enjoy. You might have noticed there is a target/ folder in your root directory now, this is the standard output folder for Rust binaries so you want to add it to your .gitignore file. You can also add those to your .vercelignore (check the template at the end for reference).

Now that we have everything up and running let's clean it up a bit and run it with the rest of our Next.js codebase. Rust runtime will take care of running our Rust code when deployed to Vercel but for dev, we will modify or npm run dev command a bit.

What we do here is that we run Rust API on different port with new dev:rust script. Then in addition to running next dev we also add it to our standard dev script. Now we can run everything in a single command that we are used to in Next.js projects.

Another thing is that we don't want to go to localhost:3001 for our Rust endpoints which will be a different port than the default localhost:3000 for Next.js. We will fix it by adding a rewrite inside our next.config.js.

We apply rewrite only in development. On Vercel it will all be deployed to the same deployment so no need for a rewrite. Now you can go to http://localhost:3000/api/crab again and it will work the same way even though it is running on a different dev server. Adding rewrite to fallback also makes sure that any of your existing pages/api endpoints do not get overwritten by Rust API endpoints in development.

Now that we have everything up and running let's take a look at some common patterns you might need (or would use) in day-to-day API development.

Caching

Any production-ready API uses some kind of caching. When deploying on Vercel we can use their Edge Network (CDN) to cache API responses. It works with standard cache-control headers and s-maxage directives in response. To cache our API endpoint to 1 hour we can do the following.

Reading request parameters

vercel_runtime allows us to read data from incoming requests. Here are some common examples.

Common utils/libs

With bigger projects, you would probably have a set of utils that are used across different API endpoints. With Rust, we can also register local crates in our Cargo config.

I decided to put all of my shared Rust code inside src/rs directory to separate from other JavaScript files. In this crate I can define functions and macros to be reused in my Rust API endpoints. To use these functions just import them as from any other Rust crate. If you installed rust-analyzer or similar language integration in your IDE just starting to type the function name will allow you to auto-import.

Environment variables

You can access all of your env variables from a local .env file or your Vercel deployment. I recommend dotenv crate which is nice and easy to use.

Error handling

You don't want your JSON APIs to crash with ugly unhandled exceptions. Rust has a specific way of handling errors but to catch or throw structured errors in my API endpoints I added a macro to my utils which I use all around the project.

It accepts message, error, and HTTP status code which then get logged and return a response with the nice error message and status code that was provided. In addition, few macros are available for common cases like throwing generic 500 errors.

Different request methods

You can support different request methods like POST, PUT and DELETE. Each API function can handle multiple request methods so let's see how to validate and limit which methods API endpoint accepts. First, we will define a separate function for each of our request methods. For example for POST method.

Then in our handler, we validate and map correct functions to request methods.

Communicating with other services

API endpoints can communicate with other services over different protocols. Most of the time it is over HTTP and for that, I recommend reqwest crate which works well with other packages we already use. You can see some examples inside my linked template project.

Cold boot and startup

Deploying serverless comes with a cost. Sometimes your users will hit an endpoint without a prepared instance and will have to experience a cold boot delay. One of the factors affecting cold boots is the size. From my testing compiled serverless functions on Vercel with Rust runtime are smaller than Node.js functions with a similar amount of code but larger than Edge functions.

Cold boots vary between Node.js and Edge runtimes on VercelAs for cold boot performance I measured it ranges from 500–1000ms. When hitting warm/ready instance performance is great and if your project gets a lot of requests (together with caching) you will not have any issues with your Rust-implemented API endpoints.

I have been using this setup in some of my projects successfully for the past few months. One of those https://shareimdb.com/ is fully run on Rust runtime in terms of API and I am using JavaScript/React only for pages needing HTML rendering. The project profited a lot by doing underlying scraping and parsing with Rust instead of Node.js. For example, I would have trouble fitting headless browser runtime in a severless function 50mb limit, but with Rust, it was not an issue since the size was smaller. The most important thing is to pick the right tool for the job. Anyone who worked on any kind of long-term production project will know that the "rewrite everything mindset" is not viable. With that in mind this kind of partial integration and usage of different technologies/languages on a single project is perfect because they can co-exist and if picked correctly can make the project better.

Issues

I want to end this by pointing out some issues that I encountered while using Rust in production on Vercel. As said, I have been running this successfully for the last few months but I want to point out some pain points I had and bugs I encountered to hopefully save you some time in decisions and debugging.

One issue that I noticed is that sometimes vercel dev running Rust runtime functions would just crash for no obvious reason. For example, when you add specific dependencies through cargo it might require restart of vercel dev. Sometimes when adding a new file to the cargo config or as a lib I would also need to re-run my dev command or kill node process that is running the server. While this is a community runtime (acknowledged by Vercel) I hope in the future they also add Rust as an official runtime.

Another bigger issue is that there is a problem with Vercel when deploying dynamic routes with any custom runtime (not just Rust). You can read the details on the bug report I opened on Vercel GitHub. There are also some alternatives to how to avoid this inside the ticket.

All in all, I loved that I was able to add Rust to my existing development stack and projects. You can take a look at my Next.js + Rust template created from this blog post for more code and examples. You can also use it as a template for your new projects.

If you enjoyed this post I would appreciate the like or share and if you are interested in learning more about this subject here are some links to continue reading.

Rust runtime for Vercel (community)
Vercel-supported custom runtimes
Rust Book - a crucial resource for learning rust fundamentals
Web Assembly for Rust and Wasm on Vercel
My Next.js + Rust template

And some of my other posts/projects:

https://jawa.sh- Visual scraper interface, exports to puppeteer script which you can run anywhere and scrape anything 🤖
https://medium.com/@capJavert/doing-100-000-push-ups-this-year-with-the-help-of-siri-605c3c8eb18f - How I did 100,000 push-ups in a year 💪
https://get.komfy.app - Stream your favorite video content from your phone directly to your PC or Mac 📺
https://medium.com/@capJavert/re-hacking-wordle-114ba75d1344 - How I hacked the most popular game in the world ⌨️

Seamless emoji creation on Discord with bots and Vercel

Ante Barić — Sat, 06 Jan 2024 10:16:55 +0000

Seamless emoji creation on Discord with bots and Vercel
Photo by Zen Chung from PexelsCustom emojis are a great way to add identity and a sense of fellowship to any Discord server. Users quickly get ideas for new emojis from messages shared in the server but the most common issue is that it takes some work to transfer those ideas and images to emojis. Discord has restrictions on the resolution, type, and size of an image that is required to create an emoji. But, people are lazy, and with those restrictions and additional work required to create them, many cool emojis are left as just ideas…

If you have ever been a part of Discord server you will know that many of them have different bots installed to help with various tasks. Bots can send messages to channels, perform moderation, embed remote content like tweets, and also reply and react to commands from users. The last feature, commands, is especially interesting because it allows users to perform advanced and custom actions inside the Discord server. For example, we could develop a command to create emoji from any image.

Since my (and many other) servers already have a custom bot I decided to try and add a new command that would do just that. The idea sounded great in my head but at first, I was not sure what would take to actually develop it. Manipulating images usually requires some processing power, a dedicated server, and storage. To entertain this idea let's see what would be required for this feature to be added to our server:

Discord bot (my server already had one but if you want to create your own take a look at https://discordjs.guide)
A way for users to input an image and some data like emoji name
Server or application capable of processing the image and adjusting it to given Discord restrictions
Uploading adjusted emoji image to Discord as new emoji

As mentioned above, my Discord bot is implemented in Node.js with the help of Discord.js package. Their guide has an in-depth instructions on how to set up a basic bot, register it for use on Discord, and also some ideas on where and how to deploy it. I won't be going into those details in this blog post but will provide a basic skeleton of it using the above package.

Once I had my Discord bot up and running the first thing was to solve the handling of user input. Users usually have an image they would like to convert to an emoji so we can start with that. Also, each emoji needs a unique name with which it can be used in messages and reactions, it is an alphanumeric phrase (_ is also allowed) without spaces or special characters. If you speak regex this would match:

Luckily Discord command can accept text and images as input so we can collect all the data needed to create an emoji from our user right from Discord interface. Discord.js exports SlashCommandBuilder class which allows us to define what inputs our command accepts.

What we need now is to handle the above input. The most important thing is the image and its conversion. To convert the image we will use the cool feature of Vercel which is called Open Graph (OG) Image Generation. This feature is available as a standalone package and also inside Next.js framework. Since I used Next.js for my general purpose server and API I decided to add a new API endpoint to handle image adjustment. The above image generation package uses JSX to create images from HTML and CSS markup or even custom React components. So to adjust the image all I needed to do was to render <img /> tag inside JSX and set style attribute to correct size and fit values. This is how my request handler looks like:

I also added the fit param to allow further customization of how image should be fitted to square dimension of the emoji. If the image aspect ratio is 16:9 with fit: cover we can cut it in the middle and alternatively with fit: contain we can resize it to fit 1:1 aspect. If you don't use Next.js you should still be able to use @vercel/og package with your Node.js framework of choice. It is a great way to adjust the image per Discord requirements without using any heavy image processing or requiring additional storage.

Now that we have our image ready we can connect it with the rest of the data and create a handler for our slash command.

In the above command, we take options from Interaction object passed from our slash command. We then take the original image URL from Discord and pass it to our API endpoint together with fit option. The response of our endpoint (which is an image) is then parsed as ArrayBuffer and then passed to Discord.js function which registers new emoji in the Discord server that our bot is currently running in. ✅
Here is a sneak peek of how all of this works together inside Discord app:

Done and done.The above is also a good example of how an image with a wide aspect ratio fits perfectly in the square emoji after conversion. 🙌

Thats it! This was a pretty cool usage of different services and technologies to create something new and a great example of what is possible with the tools we have available today. If you want to learn more about Discord and the different integrations it offers for developers you can also check out their developer portal.

P.S. More complete code snippet with validation and additional command for removing an emoji can be found here.

If you are interested in programming, tech or want to see some of my other side projects check below:

https://get.komfy.app - Stream your favorite video content from your phone directly to your PC or Mac 📺
https://medium.com/@capJavert/doing-100-000-push-ups-this-year-with-the-help-of-siri-605c3c8eb18f - How I did 100,000 push-ups in a year 💪
https://jawa.sh- Visual scraper interface, exports to puppeteer script which you can run anywhere and scrape anything 🤖
https://medium.com/@capJavert/re-hacking-wordle-114ba75d1344 - How I hacked the most popular game in the world ⌨️

Taskly - better way to manage your tasks

Ante Barić — Thu, 12 May 2022 21:31:36 +0000

You can try Taskly for free at taskly.kickass.website

Overview of My Submission

Taskly is an app to better manage tasks across all of the different task/todo/issue/project platforms. Developers (especially in open source) often use multiple websites like GitHub and Gitlab to work on their projects and contribute to other. There is also a fair share of project management tools that have their own task/todo lists. With Taskly all of those are merged into single interface for quick access and management. When you start using Taskly it will become a central hub for all of your tasks and give you more time to focus on actually doing your job.

Infrastructure

Taskly is built as a full fledged web application meant to scale and serve many developers working in open source and beyond. At the core of that is Appwrite which we use as a backend for all of Taskly features. As Appwrite is ready for horizontal/vertical scaling Taskly is also ready to scale as its user base grows.

Authentication

As we started building Taskly on Appwrite platform we used the Users module to provide authentication and social features to our users. Users can login with email/password or any of our 3party providers, all of those providers are also available to import/manage tasks from, but more on that later.

Integrating authentication with our Next.js application was easy through Appwrite Web SDK and our custom Authenication Service. Authentication Service is our abstraction which allows Taskly to connect to any auth provider and use it seamlessly with other Taskly features.

Tasks management

The core feature of Taskly is Tasks module. It provides a central list of your tasks with search and filtering methods to quickly organise and find tasks that you need to read, edit or (finally) complete.

Core building blocks for the Taskly interface are NextUI components. It is a great (open source) set of primitive UI components that give Taskly a slick look, also, don't worry, we do support Dark mode (｡▼皿▼).

When you first sign into your Taskly account you will be offered to connect any of the 3party apps that you use to manage your tasks like GitHub, Gitlab, Trello... Currently we only support few of them, but check back soon because more providers will be added in the future (this project was built in less than a month so bear with us), also if you wish to collaborate on that give as a ping on GitHub. After connecting the provider like GitHub you can select which projects issues you wish to import (we only import the things you choose because you are the Boss) and manage through Taskly.

After connecting any of the providers your tasks will just magically start to appear in your Tasks lists. But there is no magic here, just awesome Appwrite Functions doing their thing..

After that you can create, view, edit, update and delete any of your tasks in single place. There is also a handy link to the original task for your enjoyment! Taskly is also optimized for mobile.

All of that is running under the hood with our Data Service which like Auth Service allows Taskly to interact seamlessly with any 3party data source or API. This particular implementation is of course using Appwrite Web SDK.

We also used permissions system from Appwrite Documents where we specifically assign permissions to read/write (Document level) tasks to the users owning them. In the future this will allow us to implement users sharing and collaborating on tasks directly through Taskly.

Importing and syncing Tasks

In the previous section we bragged how after connecting services like GitHub and Gitlab to your account your tasks just seamlessly start to appear in the Tasks list. Here we will talk more about how that is achieved.

Another cool feature Appwrite has are Functions. At Taskly we use them as background job queue. Any time new project is connected to Taskly a new function execution is started in the background. We call it "import job". Import job pulls all of the issues and data from 3party provider to display them inside Taskly interface.

This gives us few things for free:

our users do not need to wait on the loading screen for their tasks to import
they can just continue working and seamlessly their tasks will start appearing as they are being imported through our import job
we save user bandwidth because Appwrite lets us know in realtime when new tasks are imported

After the import job is done there is of course a possibility that someone creates a new task in your project (a pesky project manager) - we do not want you run into problems by missing a new task assigned to you. That is why we use webhooks to continuously sync new tasks and task updates to Taskly from your connected providers. This means that you can focus on your work and let Taskly handle everything else.

Another great thing about Appwrite Functions is that you get logs and errors for free so we can monitor our job executions and potential errors for our users.

Realtime UI

At Taskly we believe in fast user interfaces. We were impressed how quick the Appwrite Documents API was when fetching data for our application but we went step further. We noticed that there is also a Realtime support in Appwrite using Web Sockets. We went ahead and tried it out and it worked as a charm.

Basically from the moment you open Taskly we connect to the realtime socket and listen for changes (from webhooks or async import jobs). Any time new document is added, edited or deleted we refetch those and update our local state.

This also allowed to use more aggressive caching on the client side because we do not need to refetch anything until realtime socket tells us something changed. This automatically makes all of your recent/usual queries instant, Yeeey!!

We were impressed by how good it worked so we also implemented it at some other parts of our UI.

Search and filter

Listing tasks was fine but we also wanted to add some basic search features to Taskly. As we use Appwrite Documents to store all of the user data we used Indexes and filtering options in Appwrite Web SDK to implement those.

We support:

fulltext search of tasks
sorting by title, creation date, status
filtering by task status

With those you can quickly find the relevant tasks. All of the search params are persisted inside the browser URL so until Taskly adds support for "Saved views" (Soon TM) you can for example bookmark the tasks from specific project that are still open.

What is next?

You will notice that there is a lot more to Taskly than the features we mentioned above. That is because even though we want to make something great with Taskly we only created it in less then a month for Appwrite Hackathon. This means that it is a work in progress. If you think this is great and wish to help contribute new features to Taskly give as a ping on GitHub.

Submission Category:

Our submission fits into "Web2 Wizards" category. Taskly is a single page (SPA) web application running with Next.js and React using Appwrite as backend.

Link to Code

kickassCoderz / taskly

Taskly - better way to manage your tasks

Crafted with Appwrite and Next.js ❤️

Created by @Fazla-GroM and @capJavert 🌈

Development

This is a Next.js project bootstrapped with create-next-app.

Getting Started

First, run the development server:

npm run dev
# or
yarn dev

Open http://localhost:3000 with your browser to see the result.

You…

View on GitHub

Additional Resources / Info

Taskly is crafted with ❤️:

Taskly is created by: