DEV Community: Jacob Schatz

The Intriguing Issue of 429 Errors in Cloud Infrastructure

Jacob Schatz — Wed, 26 Apr 2023 16:29:04 +0000

In the realm of cloud infrastructure, unexpected challenges may arise, leading to complex problem-solving scenarios. One such instance occurred in Calendar.dev's Kubernetes deployment, where a peculiar bug caused 429 errors (too many requests) from DigitalOcean's API. This article details the investigative process and shares valuable insights for both learning and amusement.

Upon deploying Calendar.dev via CI/CD, 429 errors were received from DigitalOcean's API, implying over 250 requests per minute. Despite confidence in making fewer requests during deployment, DigitalOcean representatives insisted otherwise. In search of a resolution, the number of replicaSets was reduced, but even with just one, an image pull error emerged from Kubernetes. A careful count revealed no more than 10 requests made to DigitalOcean within a 10-minute window, leaving the situation perplexing.

Further investigation revealed that the rate limit error stemmed from domain checking. The primary issue was traced to a recent domain change to calendar.dev. An outdated Kubernetes (k8s) cert-manager had not been updated to refresh the HTTPS certificate after the domain switch. Consequently, cert-manager attempted to provision new certificates but failed to validate the domain, as it was incorrectly pointing to the external load balancer IP.

DNS was initially dismissed as a potential problem, considering that the SSL certificate functioned without issue, and the error only surfaced when altering the number of replicaSets. The absence of an invalid SSL certificate warning was attributed to Cloudflare, which automatically provided an SSL certificate when recently added to the mix. Unaware of this automatic provisioning, cert-manager continued to run, leading to an unintended overlap of SSL certificates.

In summary, each production deployment was met with 429 errors as Kubernetes attempted to spin up new pods. The excessive API requests to DigitalOcean were a result of Kubernetes trying to renew an improperly configured SSL certificate pointing to the incorrect domain. The lack of a visible error, caused by Cloudflare's automatic SSL certificate provision, masked the issue.

In hindsight, the DNS problem was difficult to identify without any apparent DNS issues, and the focus was directed toward potential Kubernetes configuration mistakes. This was an interesting experience and underscores the importance of understanding interactions between cloud infrastructure components, such as Cloudflare and SSL certificates, to prevent similar challenges.

There's an old saying:
No, it is not a compiler error. It is never a compiler error.
When you're at your wits end trying to solve a bug you can often assume the worst which can stop you from exploring other areas and just saying "what if".

Cron's Request Access Flow

Jacob Schatz — Fri, 24 Jun 2022 11:37:27 +0000

When you're making an app, things are often far more complicated on the inside than meets the eye.

I love Cron! Currently it is invite only, and I was lucky to get early access. Cron has an excellent onboarding flow and I wanted to document it, if only for myself as I build my own apps.

Here's a link to the Lucid Chart itself (sign in required).

From Director to Engineer

Jacob Schatz — Mon, 29 Mar 2021 15:55:35 +0000

I recently made the personal choice to change positions. At MURAL, I joined as Director of Engineering for the Platform Pillar. I've been Director of Engineering at MURAL for just over 9 months and I love the job, and it's super rewarding. My team and I built a whole pillar and everyone is delivering. There is much work to do, but we got some serious shit done. I can't really express it any other way. We got shit done.

To be honest, I'll have to be terse, but if you read this and you have questions feel free to reach out on Twitter.

The reason to join startups vs big co is because you can have a bigger impact. There are bigger companies out there, and I selfishly desire to be a very large part of the success of a something, even if it's small. I also desire to be one of the team, where the task may ambitious and the goals ambiguous, but we all are working our hardest to figure it out. As if others don't do that at Big Co's! I am sure they do.

A Director position is a sought after position, so you might think I'm nuts for switching roles. Ultimately I see things that need to fixing, and I want to have an even more direct impact.

I've been given the opportunity to work on the biggest problems at MURAL as an Engineer. This is really my dream job. To see problems, know the solution and have the freedom to fix them. At other companies, blockers may stand in your way of fixing big problems. It can kill a company if you can't execute quickly with high quality. I've been super lucky to have great mentors over the years. I've gained one often overlooked skill:

It's one thing to know how to solve a problem, it's a completely different and (in my opinion) harder to execute big problems in a way that both brings people together and drives things forward. It's even harder to do all this with urgency. You can always bring people together by moving half the speed of smell, but who has time to move that slow?

My new title is officially title is Principal Engineer, but really I see it as a way to use my skills as a solid dev to get shit done right away.

Anyways. Back to work.

Performant Paintings: Building a Canvas Render Engine

Jacob Schatz — Wed, 17 Mar 2021 18:53:29 +0000

Introduction

We founded MURAL in 2011 with one goal in mind: create a feature-rich digital whiteboard, called a “mural,” for real-time online collaboration. As the company grew, our engineering team focused increasingly on streamlining performance. Then the COVID-19 pandemic hit, forcing companies around the world to rapidly adapt to working remotely. MURAL’s popularity surged, and with it came an influx of new and concurrent users. Increasingly large teams were suddenly coworking in MURAL to create ever more complex and varied content, which was slowing our app down. Performance was now our top priority, and we needed to address it as soon as possible.

The Problem

Initially, we developed MURAL to leverage the DOM; as the base programming interface underlying the HTML, it’s familiar to web developers, making it a practical starting point for our application. However, the DOM was not designed to support dynamically changing interfaces comprising a massive number of graphical components — and MURAL users were creating new elements by the thousands. Each sticky note, image, and text box added to a mural further slowed the app’s performance, eventually leading to a bogged down and frustrating experience.

A collaborative design thinking tool that freezes or lags is a collaborative design thinking tool that won’t be used; we needed to update our approach. To address the limitations of the DOM, we decided to go around it and migrate our application to HTML5 canvas. There was just one problem: we’d designed our entire codebase around the DOM.

A Custom Solution for a Custom Problem

Overnight, MURAL had become massively multiplayer, with concurrency and latency issues. In other words, large teams were working together in MURAL and slowing each other down. We needed to enable many users to collaborate simultaneously within a single mural without affecting the app’s performance, a challenge well traversed by the video game industry. Who better to solve game developer problems than a game developer?

Why performance in game development is important

Performance is often highly prioritized in game development; because games are a time-based medium, developers need to make sure their code doesn't slow down the frame rate.

Enter Fede

Fede, a former game developer, was familiar with building products using existing engines but had never built an engine for an existing product. Excited by the challenge, he began researching and planning his approach, starting with “low-level JavaScript.”

What is an engine?

The definition of an engine can get a bit nebulous. For both the purposes of this piece and game development, an engine can be thought of as a set of code libraries that handles low-level code. This code includes rendering, object management, audio, and user input.

Low-level programming isn’t generally associated with web development. When it is, it often means WebAssembly (WASM) is involved. However, Fede ruled out WASM at the time because of its more limited browser support in comparison to JavaScript. So what did he mean by low-level?

Reminiscent of programming requirements in the financial tech sector, low-level JavaScript describes code in which performance informs every choice. For example, Fede pruned out unnecessary prototypal methods like map and reduce, using for loops instead, and structured the code to avoid using first-class functions while still keeping it readable and maintainable. While taking such an ascetic approach isn’t always necessary in general web development, it’s crucial in the case of a massively concurrently used app like MURAL.

What is "C-like" JavaScript?
JavaScript automatically handles garbage collection. Therefore, web developers typically aren't concerned about memory allocation and the speed of method calls. However, when speed is the highest priority, techniques borrowed from low-level languages can powerfully boost performance. Some of these techniques include removing prototypal methods, avoiding passing functions as arguments, and ruthlessly managing memory (e.g., mutating an array in place rather than creating a new one).

Here's an example. Compare the following two programs, which both identify the first character of every animal type that is exactly three letters long:

     // Common JavaScript approach
     const animals = ['fish', 'dog', 'cat', 'giraffe'];
     const firstCharOfThreeLetterNames = animals
       .filter(animal => animal.length === 3)
       .map(animal => animal[0]);


     // C-like JavaScript approach
     const animals = ['fish', 'dog', 'cat', 'giraffe'];
     const threeLetterAnimalTypes = [];
     for (let i = 0; i < animals.length; i++) {
       if (names[i].length === 3) {
         threeLetterAnimalTypes.push(names[i][0]);
       }
     }

Benchmarking these programs against each other shows the second solution runs roughly 50% faster.

The Engines

Our existing solution started as an experiment to transform user input into graphics and render them to the DOM. While this DOM-based approach served us well until now, it was eventually rendered obsolete by new needs. The existing code was also a monolith designed around painting to the DOM, and the complexity of the existing logic to translate MURAL concepts into renderable content made updating and testing the app a fiddly affair. So, in order to migrate MURAL to canvas in a maintainable way, Fede decided to start anew with a custom system consisting of two engines: one to handle rendering graphical primitives like triangles and circles, and one to translate MURAL objects, like sticky notes and gifs, into those primitives.

The Rendering Engine

The rendering engine has one responsibility: send visuals to a target. Unlike the other engine, this one has no handlers for MURAL-specific concepts.

The rendering engine expects the graphical primitives output by the MURAL engine and renders them to the specified “surface” — anywhere murals are visible to users.

Potential surfaces include:

Canvas: used in the MURAL application
PDF or SVG: used when exporting to these file formats
WebGL: not yet implemented, but presents interesting potential three-dimensional applications

The MURAL Engine

The MURAL engine translates user-facing concepts into graphical primitives, by reducing them down to their most basic characteristics, represented by ASTs.

For example, the aforementioned sticky note could be broken down into a colored rectangle, a border, and a text box.

What is an AST?
An Abstract Syntax Tree, or AST, is a structure that describes how data relates semantically to itself, or in other words, where each datum fits in a hierarchy with the rest. For example, an AST of a sentence might be broken down into verbs, adjectives, and nouns. Read more.

Originally, the dual responsibilities of translating MURAL logic into graphical primitives and sending those primitives to render targets were handled by a single engine. So why did we now divide them into two?

Separating the concerns into two distinct engines enabled us to maintain, update, and build additional support infrastructure around each engine with less overhead going forward. This new design enabled Fede to build a test automation system, a key part of the new approach, alongside the rendering engine to ensure it worked as expected and that future changes didn’t slow down performance.

Challenges

Of course, being the best solution for the job doesn’t mean canvas was simple to integrate. In order to encourage iteration and experimentation, A/B test, and because such a significant change could result in huge merge conflicts, new developments were incrementally merged to production, meaning both the old and new approaches had to work side by side until work was completed.

Additional complications with our use case included canvas' complete lack of built-in accessibility support, which we as a company have committed to addressing; and its suboptimal support for rendering text at the subpixel level across devices and resolutions. Text handling also happens to be under-described in the CSS specification, meaning, for example, that each browser is free to decide when and how line breaks occur. While we could have built workarounds, they would have tacked on significant overhead for testing and maintenance, especially in today’s evergreen browser landscape, and our users create murals on a cornucopia of browsers.

Fede chose to track his development progress with Chrome, because it represents 80% of our user base and leverages the GPU to accelerate canvas renders. To ensure all users had a smooth user experience, he also regularly tested and built across other browsers and devices. These manual tests included devices running Linux, Windows, macOS, Android, and iOS at a variety of hardware specifications, as our latency problem was a hardware issue, not a network one.

By doing so, he discovered and addressed additional browser-specific restrictions, such as Safari’s variable hardware limitation for renderable assets — Fede addressed this by offloading some of the rendering responsibilities to an off-screen canvas whenever the full mural is visible, which slows down scrolling at that zoom level in exchange for enabling the application to continue running smoothly.

Outcomes

After migrating from a monolithic DOM-focused architecture to our custom canvas rendering engine, we improved performance (as measured by the percentage of frames at 30fps or higher) by 80%. When Fede ran into a character painting issue in Chrome during the course of development, he contributed to its resolution by submitting a detailed report with solid reproduction cases.

Because we separated the rendering process from the application-specific logic, they were no longer intertwined. The codebase became more maintainable and testable with automated end-to-end and visual tests, and by creating a system that can render to multiple targets, we opened up new opportunities for both internal and external developers to leverage MURAL features in a variety of exciting ways.

Oh, The Hats You'll Wear

Jacob Schatz — Thu, 10 Oct 2019 16:59:29 +0000

Our first post on Subs prompted an interesting question (which, if you missed it, you can subscribe for updates here.) How does one person juggle the many parts of a startup? Even those outside of their domain? As a founder with many responsibilities, I try to focus on a few goals to achieve great results. My advice has no basis more reliable than my own meandering experience. I will dispense this advice now.

First: I try to work on problems in as small a scope as possible. My time is precious and working on full-blown solutions (instead of small scoped ideas) would only take up more of it. Just like a tool that does too many things, I try to focus my startup on being the master of just one thing for now.

Second: boring solutions always get the job done faster. That one major feature needs to work before anything else. That feature doesn't need to use the latest and greatest tech.

Third: dogfood your product as soon as possible. Subs is currently serving as my dedicated password manager, so I know exactly what I need to fix next.

Keeping those goals in mind, I use tools I'm already productive with and know well. New tools can be great, but it takes time to learn them. Introducing new tools to my workflow takes time. This gets a little more nuanced when it comes to the various tasks, but if it works, it works. Blame the plumber, not the plunger. Anything you practice, you'll get good at. Right now, I am practicing starting a startup, not learning new tools.

Developer

As a developer, I've been using Sublime Text 3 for just about ten years for everything because it's boring and it works. I don't need to worry about anything else, because Sublime just works for me. I want to focus on my code.

Prettier handles all of my JS linting issues for much the same reason: it's boring and it works and it comes with Vue CLI 3. It fixes my code for me! When it comes to JavaScript style preferences, I don't hold myself to many, except semicolons and two spaces, no tabs. Otherwise, I don't care, I've got things to do.

For front-end development, I use VueJS and Vuex. I build with Vue CLI. Everything is super fast, performant, boring, and they work. I know how to solve almost any problem I come across. My backend is usually built with Express with Sequelize. Both are very fast and robust libraries. They are also easy for others to pick up and learn. Anything related to authentication and authorization is done with PassportJS, a very solid and well-tested piece of middleware, that's also driving all OAuth2 authentication with Google in this project. Their documentation is short and to the point.

UX Research and UI Design

Balsamiq Mockups was my tool of choice for anything design-related, but now I just use VueJS with Bulma for prototyping because it's quicker for me to write the code in the first place. Bulma also makes it easy to apply your own styles and doesn't require any JavaScript, which is fantastic. I love the idea of shipping a framework with only SASS/SCSS/CSS.

My user experience research is pretty straightforward: I call a friend and ask them to try it out. Later I will add more steps, but I want to get as much user feedback as possible, but only when it's ready. I want to impress people with an awesome product.

If they can, they'll give it a go and provide me with some feedback. I'll fix things up, then call more people, have them try it out, and get their feedback as well. Every single problem these first sets of users have become top-priority for me to fix. Then I rinse and repeat, collecting as much feedback as I can.

While all of this is going on, I try and remember that data points are not trends. Before throwing time and energy at a fix, confirm that one user's problems are others' problems as well. At this stage, you want to fix things that have the largest impact on the overall experience for your user base.

Marketing

For now, I'll be doing a lot of marketing on Twitter. I'm admittedly not the best at tweeting. If I can find someone to help with writing tweets, I do, but I also make sure that I'm documenting the process of building Subs rather than spitting out memes. Every tweet has to be meaty, which means I have to develop things worth tweeting.

Articles are also a big part of my marketing toolset, so I have someone helping me. We have a call or a video chat to talk about article ideas. They write down some notes and start working on an outline and drafts. We'll meet regularly to go over the drafts together to check for tone and make some changes, and I'll approve of a final draft that gets posted online. This saves me a ton of time and keeps my tone in articles.

Ultimately, when it comes to marketing, I'll do anything that saves time. My goals are to build a very solid version of the product, get it into people's hands, and get feedback. The less I need to worry about drafting and copy editing, the more time I can spend writing excellent code. Which, I think, is something to write about.

Sales and Support

Selling your idea sometimes means ignoring anything that might distract you from getting it out of the door. Even when you're getting started, you might hear suggestions and requests from big companies, if you are lucky. The best thing you can do is to save these requests and ignore them for now. Give the big company a platform to give you feedback so that you have everything saved for later.

Create an issue tracker (like GitLab issues) where everyone can submit feature requests. That way, you can interact with your user base and they'll know their voice is heard, which is what you want! Don't let it distract from your goal of releasing version 1.0, and don't be intimidated by big companies. It's valuable feedback, but as a founder and wearer of many hats, you have limited bandwidth. Your focus should be on the smallest scope possible, on what gets your product built and working.

Opportunity Creation

Building a product on my own is hard work, so something positive I work towards is creating opportunities for others. I enjoy offering people the chance to learn the way I learned. I try to give newer developers the chance to help me out if it helps them out. Letting a junior dev watch you build your product and ask questions, assuming the right non-disclosure agreements are in place, is a great way to keep me accountable and for them to learn.

I'm sure by now you can sense a theme running through the way I work. The latest and greatest in languages, platforms, and frameworks are always exciting, but not when it comes to getting things done. I need to be able to work quickly to gets Subs to you, and that means boring and reliable methods. If you want to see the proof and stay updated, sign up for updates at https://subshq.launchrock.com.

Big Giant Vue Apps

Jacob Schatz — Thu, 03 Oct 2019 18:52:50 +0000

I recently took a huge risk and left Netlify to pursue an opportunity of a lifetime to start a startup. Subs is currently in development. You can read all about it on our Launchrock, and sign up for updates.

Subs makes password management seamless by removing the need to constantly enter a master password. I am joining this startup with someone I deeply respect, and I'm already heads-down working hard on this exciting new project. This isn't my first time writing a new product from idea to production, but it's already introducing me to new and interesting challenges.

For this project, the architecture is going to be a bit different from what I've done in the past. Typically, I've used what I know well: a web app that evolves over time into something bigger. Instead of offering a ton of decent features to a few platforms, the plan is to offer a single, well-built, performant major feature to a multitude of platforms — web, mobile, desktop, and browser extensions.

Architecture and Tooling

The codebase for this startup will be large, using one mono-repo to house eight individual projects covering all of our targeted platforms. We'll also have an additional "common" project to house all of the shared components, services, and stores.

This might seem like a lot in one repo, but there are many benefits to working this way.

I'm putting the building blocks for cross-platform infrastructure in place now, so that future development is as easy as it can be. One thing I've learned: the prototype usually becomes the final code, so do it right the first time. Putting in the building blocks from the beginning allows me to give the major features the attention they need.

The majority of the front end will be driven by Vue and NativeScript. I've spoken about the benefits of Vue in the past. For this project, using the Vue ecosystem to drive the web application, desktop application, and browser extensions affords a level of reusability across the mono-repo and "common" repo, so code won't need to be written twice.

Nativescript with NativeScript-Vue will be used for the iOS and Android applications. While they won't benefit directly from the same shared components, it may let us share a Vuex store across all application instances to handle state, as well as sharing the services that call the API.

Battle-tested libraries are necessary for encryption-related tasks. So, for extra security, I wrote a command-line interface in Python using execa for process execution. This buys us flexibility in the future. If we want to rewrite the CLI in another language like Rust or Go for greater performance, it would be a drop-in replacement.

Project Structure

The diagram might be confusing so let me explain. The project features at least four Vue CLI generated applications: a web application, a desktop application, a browser extension, and the common library.

The common library does not have a build step. It is simply imported into projects. When a project using the common modules is built, the common library gets baked in. I do have to run yarn link initially and link it manually, but it's in my README for the future. This way, any changes I make to the common library immediately benefit from hot module replacement, and I can see the changes immediately.

I can use my shared components in the web app like so:

What's cool about this (besides the fact that I'm using Sublime Text) is that my components library is empty minus an entry point.

I am importing every single component at once which does not take advantage of tree shaking as much as it would in other apps. If that was a concern, I'd restructure my exports in my common library. In my case, my app is very simple, so I can get away with this.

Each app within packages has its own managed node_modules. And I have to run yarn on each of the eight projects. And I have to run yarn link subscommon everywhere I want to use my common library.

Is this a boring solution? It is a marginally longer setup that isn't a big deal to me. I like my setups to be explicit but not tedious. I like less magic and more understanding. The total time to set up the project is about ten minutes. Could it have been three minutes? Maybe.

Sharing State and Services

I also share the Vuex store and my services. Since Vuex is the only part of my app that uses the services, they simply get imported on their own.

When Vuex initializes, it performs a recursive injection of the store into all the child components, so you need to make sure it is using the right Vue instance. Here's my shared Vuex store:

import Vuex from "vuex";
import menu from "./menu";
import user from "./user";

export default function store(Vue) {
  Vue.use(Vuex);
  const store = new Vuex.Store({
    modules: {
      menu,
      user,
      ... more stuff
    }
  });
  return store;
}

I don't import Vue above because I pass Vue into my web app and desktop app like so:

import Vue from "vue";
import App from "./App.vue";
import { store } from "subscommon";

Vue.config.productionTip = false;

new Vue({
  store: store(Vue),
  render: h => h(App)
}).$mount("#app");

Now Vuex can latch onto the app properly and populate any child components with $store.

The Sum of All Parts

This project structure serves its purpose by allowing me to iterate on the primary application quickly, across various target platforms.

How quick? I was able to initialize Vue CLI Electron Builder, import my store, and have an exact running copy of my desktop app in five minutes, with no code changes.

I was able to get my Chrome extension running almost as quickly too, but admittedly it took me a few hours to understand how Chrome extensions work, so some time was spent reading the docs. Using the Vue CLI Plugin Browser Extension got me up and running very quickly, promising minimal code changes between all extensions. I'll believe it when I see it, but I haven't gotten there yet.

Meanwhile, my desktop app is so damn small and fully functional. Here's the file structure:

I kept HelloWorld as the entry point to all my apps because it's the way Vue CLI 3 scaffolds it and it makes sense to me. I'm sure that, at some point, some developers will tell me it's not kosher and to change it. Until then, I'm keeping it.

We hope to get a working proof-of-concept into your hands very soon so we can hear what you think. If you are interested in what I am building and want to know when my beta launches, then sign up on Launchrock and I'll keep you updated.