DEV Community: Burak Yigit Kaya

Marking it Up (and Down)

Burak Yigit Kaya — Wed, 02 Jul 2025 00:00:00 +0000

First, there was plain text

When I first learned about Markdown, I was a bit skeptical. Why use weird punctuation when you can use HTML instead? But as I started using it more, especially on forums etc, I realized the power of it. Unlike HTML, it was way more accessible and easier to type. Even more importantly, it was still readable and expressed meaning without obstructing the text before rendering. And slowly but surely, all major platforms, including WhatsApp adopted it.

The Age of AI

And then ChatGPT happened. Due to the properties I listed above, Markdown was the perfect format for LLMs too. Once the agents hit the scene, they started generating Markdown formatting and they were also more than happy to ingest Markdown formatted text for their context. There was only one problem though: the web revolved around HTML, and some of that being dynamically generated. Even if you could teach or extract HTML, the dynamic JS part of it is still a challenge and usually requires a full browser environment. Sure, there’s Playwright MCP but it’s slow and resource-intensive. These issues lead to the creation of services like Firecrawlwhich I think is awesome, especially when you cannot control the source of the information.

Recently, with a lot of ~~push~~ help from David, I started learning about agentic flows and how to use LLMs more than generating 0-shot responses¹. I wanted to write a bit about these too but Colin Eberhardt already did a great job with Re-implementing LangChain in 100 lines of code. This is the article that made it click for me. Once I read it, I even felt a bit silly for expecting something more complex. It’s deceptively simple: you use the LLM in a loop, parse the responses to trigger “tools”, and feed the results back (part of the loop) until you reach a final result (or the limit of your wallet). Anyway, great article, definitely go read it. Let’s go back to talking about Markdown and its cousins as this is a blog post about that, not LLMs.

Walking back from the X-factor

For my new internal project, I needed to use our docs. Although they were already authored in MDX, it was not pure Markdown. We can strip the MDX parts but Sentry Docs are architected to share certain parts of the content between different pages. This means we actually have to render the MDX to get the full content. As a person who spent some time around parsing out dependencies, building a dependency graph and working over it I had no interest in going down that path unless I really had to. So I decided to look for an existing “HTML to Markdown” solution. This led me to the awesome rehype-remark package which is a part of the unified project. I was already quite familiar with unified and remark which we also used in our docs rendering pipeline. So I simply jumped on this. My initial solution was simply to fetch the page of interest, convert it into markdown, find the relevant header and extract the contents until the next header. The code was also simple:

import type { Root, Heading } from "mdast";
import rehypeParse from "rehype-parse";
import rehypeRemark from "rehype-remark";
import remarkStringify from "remark-stringify";
import { unified } from "unified";

function extractMDSection({ section }: { section?: RegExp }) {
  return (tree: Root) => {
    const headingIdx = tree.children.findIndex((node) => {
      return (
        node.type === "heading" &&
        node.children[0] &&
        node.children[0].type === "link" &&
        section?.test(node.children[0].url)
      );
    });
    const heading = tree.children[headingIdx] as Heading;
    const nextHeadingIdx = tree.children.findIndex(
      (node, idx) =>
        idx > headingIdx &&
        node.type === "heading" &&
        node.depth === heading.depth
    );
    tree.children = tree.children.slice(
      headingIdx,
      nextHeadingIdx === -1 ? undefined : nextHeadingIdx
    );

    return tree;
  };
}

export const getWebpageAsMarkdown = async (url: string, section?: RegExp) => {
  const response = await fetch(url);
  const text = await response.text();
  return String(
    await unified()
      .use(rehypeParse)
      .use(rehypeRemark)
      .use(extractMDSection, { section })
      .use(remarkStringify)
      .process(text)
  );
};

Putting it everywhere, and fast

Then /llms.txt happened. All players in the field who wanted to be more useful in the “age of AI”² started publishing their content accessible to LLMs, in plain text or more commonly, Markdown format. Then a convention emerged: if you add .md at the end of the URL you may get lucky and get the Markdown version of that page. I’m not sure when this kind of convention started but it reminded me of the .patch trick that GitHub offers for their PRs. We wanted this for Sentry Docs! The first approach was to do this on the fly on a specific route. Not only did this prove tricky to implement in NextJS, which our docs are built in, it also had an efficiency problem. Since we cannot go directly from MDX to Markdown, we had to render the HTML from MDX first and then convert it to Markdown, essentially doubling the work. A nice trick Cody came up with was building the Markdown versions from the static HTML files that NextJS generates during pre-rendering, putting them under the public directory, and adding a rewrite rule to NextJS to serve them when the .md extension is requested. This worked beautifully but created another issue: we had to generate the Markdown files for all 8754 pages in Sentry Docs and doing this takes a lot of time, up to 6-7 minutes.

For a one-off job, spending several minutes is more than OK. But for a CI job that runs on every single commit, it is completely unacceptable. So I reached for 2 very old tricks used in every build pipeline: caching and parallelization. The script for Markdown generation was refactored to spawn multiple NodeJS Worker Threads for parallelization. Then I also added a very naive cache which got the MD5 hash of the source HTML file and created a cache file with that name containing the converted Markdown. This allowed me to just do a cp operation if the source file did not change. These worked great on my local environment. The parallelization cut down the processing time by about 6x on my 16-core machine and the caching reduced that time by another 10x. However, when I pushed this to Vercel, our hosting platform for Sentry Docs, it was still very slow. Looking carefully at the build logs I noticed 2 issues:

Vercel build machines usually had 2 or 4 cores, significantly lower than 16.
The cache was not being used at all!

Solving the first one was not possible. During my tuning (local and on CI), I discovered we needed about half of the available cores due to the CPU & I/O intensive nature of the task:

// On a 16-core machine, 8 workers were optimal (and slightly faster than 16)
const numWorkers = Math.max(Math.floor(cpus().length / 2), 2);

Then I started investigating the cache issue and after several hours of digging, I finally realized what was going on. NextJS creates a new “signing secret” for every build which also affects the file names of the JS files it generates as the names are created from file contents. This then causes the HTML files’ MD5 hashes to change although their actual contents were the same. To overcome this in a cheap manner I had to strip the <script> tags (along with their contents) from the HTML files before hash calculation and processing:

const text = (await readFile(source, { encoding: "utf8" }))
  // Remove all script tags, as they are not needed in markdown
  // and they are not stable across builds, causing cache misses
  .replace(/<script[^>]*>[\s\S]*?<\/script>/gi, "");

Surprisingly, this also reduced the processing time by about 2x as the HTML files were significantly smaller without the <script> tags.

We also started uploading these to a special Cloudflare R2 bucket for RAG processing that David started using for a much better search experience.

Can’t Stop the Feeling!

Once I get into optimization mode, I cannot stop until I hit a very hard wall or actually get every ounce of optimization implemented. So I started looking at other places where I can use the same old tricks of caching and parallelization. Turns out our MDX pipeline was not only uncached, it was also mostly using the sync version of the file system APIs in NodeJS. So I made every single file system operation async, used Promise.all to parallelize them and got a huge speed increase. That is until this was shipped to Vercel. This time, it was the “dynamic pages” which used Vercel Functions that caused crashes. These were crashing with an EMFILE file error, indicating that the file descriptor limit was reached. In hindsight, this is very obvious but back at the time I had to dig around as these were not happening locally. It first looked like a silly limitation in AWS Lambda, which is what Vercel Functions are based on, but it turned out to be a legitimate issue as with the top level Promise.all, I was creating all 8600+ promises all at once, which themselves triggered more open file handlers. Again, obviously this is insane so another old friend, p-limit³, came to the rescue. With a limit of 200 concurrent promises, we sailed on smoothly.

Then I moved on to the caching bit which turned out to be a bit more tricky. We are using this other awesome package called mdx-bundler. It takes in an MDX file, discovers all its dependencies, and bundles them together into a single JS file. Easy peasy, right? Just cache the output based on the input MD5 and we’re good! Well, almost. The catch is we ask the bundler to emit the assets (mostly images) into a separate folder. This means we also need to cache these assets too. The solution became a file and a directory, using the cache key as their names⁴where we copy everything in place when we find them. This chopped another 3-4 minutes off the build time when only a few files changed, which is the common case.

Tying it all together

It took about 2 weeks and 12 PRs to tie all the loose ends but now not only do we have .md versions of every single page in Sentry Docs, we also have better RAG-based search (still in-progress), and faster builds (from ~16 minutes down to ~11 minutes). I love these kinds of intense periods where I can focus on a few high-impact things and just punch them out. Hopefully, there will be some more in the coming weeks and months. Here’s a list of the important PRs we made to get here:

feat(ai): Add .md extension to provide pages in markdown for LLMs
ci(md): Add caching to md-export script
ci(build): Parallelize and cache mdx pipeline - fix md cache
ci(md): Upload md files to R2
feat(md): Use page title as the top level title
feat(md): Rewrite URLs to be absolute and to .md versions
Footnotes
1. These are nothing short of amazing but without the agent loop and with 0-shot approaches, they are not very useful for the tasks I have at hand. ↩
2. Yup, let’s cringe together. ↩
3. Btw, I still refuse to believe Sindre Sorhus is a real person. That is alien-level productivity and reach 🙇🏻‍♂️ ↩
4. Well, they cannot be the same name as you cannot have a file and a directory with the same name in the same place. So I just added a suffix to the file name. ↩

Nightmare on Apple Street

Burak Yigit Kaya — Thu, 08 May 2025 00:00:00 +0000

*Clicks fingers, clears throat*

Okay, I have procrastinated on this post for, checks notes, 3 months now. It’s time. Time to let go of all the bad memories and the pain.

See, all I wanted to do was to create a terminal application that you can just download and run on Linux, macOS, and Windows. I also got a bit ambitious and wanted to create this app on, *gasp*, Linux! And you know, it worked on Windows. Yeah, that Windows that every developer loves to hate but secretly uses one way or another. But macOS? No no no no no, tsk tsk tsk, not so fast little boy. You need to sign & notarize your stuff, and you need to do it The Apple Way™.

The Apple Way™

The very first thing Apple wants is ~~your money~~ an Apple Developer account which will set you back $99¹. Every year that is. Oh, and you cannot just create a developer account. You see, you need One Apple Account™. If this is going to be a personal developer account, you just need your name, email, phone number, and your address². If you are trying to enroll your organization, god help you: you need your D-U-N-S number.³

Now that we have warmed up, it is time for you to create an identifier for your application. They recommend using a reverse-domain like name: com.my-company.my-app. I know, you just want to self-distribute a simple binary. Yes, you still need the unique identifier. No, you cannot use asdf or foobar. Okay head over to identifier creation page and get it over with please. I’ll await. I think you leave the capabilities empty.

Switching to the Highway

After this step we need to add a certificate to our account. Now, if you have XCode, there’s a built in UI for this. But remember, we don’t have access to macOS where XCode can only survive in. Hence we will go rogue and will create a certificate signing request (CSR) using the command line. We need a “private key” to create a CSR so we’ll be creating that via the CLI too. Might seem complicated but it is just answering a bunch of questions and shuffling some files around.

For this, we’ll be needing 2 tools:

openssl⁴
rcodesign

Let’s start with the private key:

openssl genrsa -out private.pem 2048

Now that we have generated our private key in private.pem, we can create the CSR using rcodesign:

rcodesign generate-certificate-signing-request --pem-file private.pem --csr-pem-file csr.pem

Okay, we have the signing request in csr.pem. Now we head to the page where you can add a certificateand follow the steps below:

From the gazillion options, select Developer ID Application.
Select G2 Sub-CA (Xcode 11.4.1 or later) for Profile Type.
Upload the csr.pem file we just created.
Now we should arrive at a page saying “Download Your Certificate”
Save this file as pass.cer next to the other ones and keep them safe.
Download Apple’s root certificate and convert to PEM format (Apple Worldwide Developer Relations Certification Authority)
Note down your cert expiration date. You’ll need to do this entire dance again some days before this date:
Now we are going to combine everything into a p12 file. Make sure to replace Company Name in the command line arguments below with your company name or your name.
Before finishing, we need to note down your Team ID:

At this point, you only need the final codesign.p12 file.

Apple’s Sacred Stamp of Approval

To be admitted to Apple’s sacred notarization service, you need to get an App Store Connect API key. If you enjoy a good read from Apple go read their documentation. For the twitchy ones, like myself:

Head to API Key Creation page
Click on the + next to Active at the top of the table.
Enter a name like Code Signing
Put Developer for the Access field
Hit Generate
Notice the Download button in the last column for the key you just created (bottom row)
Download and save the key with the name apikey.p8 next to codesign.p12
Note the Key ID somewhere
Note the Issuer ID somewhere. This is a separate section above the key table.
Now let’s combine all these 3 into a single JSON file so we don’t have to manage them separately:

At this point you only need the codesign_key.json file. This will be used for notarization.

The Entitled Apps

To be able to get your app notarized, it needs to have “entitlements”. This is essentially letting Apple know ahead of time, which sensitive APIs your application will be using. Then Apple’s servers will issue a “ticket” for this specific version of your app and when someone tries to run it, it will be checked and restrained to these limitations.

Since I don’t have cybernetic powers, I cannot (yet) deduce which entitlements your app needs over a blog post. That said I can at least make a recommendation. Since I did this for fossilized Node.js applications, I just copied what Node.js used for itself.

You can use this or create your own by picking and choosing from the vast array of entitlementsthat Apple offers. There’s also more excellent prosefor those to understand deeper and follow the Apple cult even closer.

At the end of this section, I’ll just assume you have an entitlements.plist file that isproperly formatted⁵ next to the binary you want to sign and notarize.

Sign here please⁶

Now that we got everything we need for signing and notarization, we can get to actual business. Signing is quite straightforward but getting the notarization right took a few tries. Let’s start with signing:

rcodesign sign --team-name <your_team_id> --p12-file codesign.p12 --for-notarization -e entitlements.plist <your_binary_path>

If you opted for a password-protected p12 file above, you can add --p12-password <password> or--p12-password-file <password_file_path> at the end of the command above.

Knock Knock Knocking on Notary’s Door

Now that we have a signed binary, we will get it notarized. We already got the prerequisites by using the --for-notarization and-e entitlements.plist parts above so we are in good hands. We still need to zip the file before though⁷.

zip app.zip <path_to_your_app>
rcodesign notary-submit --api-key-file codesign_key.json --wait app.zip
rm app.zip

We’re Done Here

Yup, we really are done. At this point you can start distributing the signed binary. People using a macOS should be able to use it without errors or warnings. If they double click on it (instead of running from a terminal), they may still see a security warning as we cannot “staple” the notarization tickets to plain binaries. To be able to do this you need to package your app as a .pkg or .dmg file but I wasn’t (and still am not) interested in learning more Apple stuff so you’ll need to figure that part out yourself.

If you want to have this process on a CI/CD pipeline you need to remember a few things:

Make sure you don’t do signing and notarization on PR branches as that means anyone who can create a PR can generate and distribute a binary with their potentially malicious changes and with your signature on it.
I don’t think you need to password-protect your p12 file but if you are using a service like GitHub you probably cannot store files as secrets. A quick hack for this is to store the base64 encoded string versions of these 2 files (codesign.p12 and codesign_key.json) as secrets. Then youbase64 decode these into their respective filesand continue business as usual.
Also, don’t forget to store the signed binary as the artifact of your build.

Resources

I’ve used the excellent docs Gregory Szorc created for his amazing apple-codesign project. I essentially summarized these two pages:

I also found this amazing gist for creating pkpass.p12 files from the GitHub userkarnauskas and used parts of it.

Finally, I’ve used this little hack from StackOverflow for providing you with a command for creating password-less p12 files from the get go.

Thanks

I’d like to thank my colleague Daniel Szoke for his help for establishing this entire flow and proof-reading this post. I should have written this before he also got the pain to get sentry-clisigned but hey, better late than never, right? 😅

Footnotes

You need to scroll all the way to the bottom to see this very unimportant detail. ↩
Yep, I’m being snarky. ↩
First time I heard about it. I wish patience to people dealing with Apple. And no, I have no intention of learning more about this but you have that link there. ↩
If you don’t have openssl around, just search for how you can install it. Should be as easy as <package_manager> install openssl where <package_manager> is apt or yum or something akin to those. ↩
Being a bit picky, are we dear Apple? ↩
and here, and here, and here, and here… ↩
Don’t ask me why they cannot be bothered with on-the-fly zipping or HTTP content encoding etc., I don’t know. ↩

Making your Node.js application last centuries

Burak Yigit Kaya — Wed, 12 Feb 2025 00:00:00 +0000

I’ve been working on Sentry Spotlight for the past several months. One of the things I wanted to do was to reduce the friction on trying out and adopting Spotlight. You don’t need to know what Spotlight is (yet!) to enjoy this thriller but if you really must know, it is a local and offline debugging tool leveraging Sentry SDKs. It supports errors, traces, and very soon profiling data 🤞🏻.

One binary to rule them all

Now, where were we? Right, it was a bright San Francisco morning when I decided to create a self-contained binary for Spotlight that you could “just download” and run. Nothing else needed. Without such a binary, you either need to have node & npx or docker on your system. I think we have enough haters for both (rightfully so). Besides, I wanted to make Spotlight accessible to everyone. For instance, if you are an Android developer you probably neither have node nor docker on your system and have no reason to install any of them.

We do have the Electron app, that said we only have it for macOS, and, I don’t really like the idea of shipping an entire browser for an application that has a simple web interface and works over HTTP.¹

Enter Node.js Single Executable Applications

So, I started looking into ways to create a self-contained binary for a NodeJS application. I know tools exist for Python so I was hoping that there would be something for Node.js too. I came by nexe and I was about to give it a shot when I noticed this “Node.js Single Executable Applications (SEA)” entry on Google. Sure enough, Node.js folks were adding exactly what I was looking for into Node.js itself! I quickly tried out the steps listed and started jumping up and down with some hideous dance moves in between when I got a working binary for Spotlight.

It was a bit laborious but OK for a local test. To be able to actually use this in a fully-automated CI system, there were a few things that needed sorting out:

Spotlight server needs to become a single, dependency-free CommonJS file
I need to ship the Spotlight frontend assets with the binary
I need a maintainable script to do all the above and build the binary

Single-file Node.js Application (not a binary)

Creating dependency-free CommonJS files is not something I’m unfamiliar with. I’ve first encountered this technique when I was working on Yarn quite a while ago. Back then, some smart folks at Facebook (nee Meta) realized they can pack a Node.js app into a single file just like a bundled web application². This was using a bundler such as Webpack (remember, this is 2015). I then used this technique on Craft during my first stint at Sentry. This method already makes it easier to distribute and run a Node.js application without needing to install any dependencies. But it still requires node to be installed on the system (and it needs the correct version of it).

Due to my past good memories from Craft, I chose esbuild as my trusty (and swift) bundler for the job. Just as I was thinking this was too easy, I found myself on the sidelines of the great ESM vs CJS war. As an application built in the modern times, Spotlight is using ESM modules all around. This also meant no more pesky __filename and __ dirname globals and using the new import.meta instead. When you compile this into a CommonJS bundle naively, import.meta becomes an empty object, making import.meta.url undefined, making it impossible to determine where your script is running. Thankfully³, I was not the first person to bump into this and there was a simple yet crude solution that I’d happily take.

Packing the frontend assets in

The assets needed for Spotlight’s UI are not much: just an HTML page and an accompanying JS bundle. The first thing I tried was to bake these in with hard-coded names which worked just fine. But I was acutely aware that it was not future-proof at all. It is easy to add more resources to a frontend application: be it split JS chunks, some images, or separate CSS files. I could just pack everything in the dist folder where the assets were generated into, but currently, the Node SEA resources API does not have a discovery mechanism. If you know the name of the resource(s), you can read them but if you don’t GLHF.

Luckily again, all the bundlers produce a manifest.json file that lists all the resources they’ve generated and their relationship with each other. I could just read this file and pack all the resources listed in it along with the manifest file with the well-known name manifest.json. This way, I could read the manifest file and discover all the resources I need to serve the UI. And that is exactly what I did.

Now all that is left was codifying all this logic in a neat little script that I could run on my CI system and get a shiny new binary at the end. Or was it?

A wild boss appears: signing and notarizing on macOS

Of course, if it wasn’t for my arch nemesis, macOS, how could we have fun⁴? Starting from macOS Catalina (circa 2019), Apple requires all applications to be signed and notarized to be able to run without any warnings. The signature is a hard-requirement to be able to run the file at all whereas notarization is to remove the warning and prompt.

Any security-conscious developer would not eschew code signing and maybe even some sort of permission grants. That said since this is Apple, the grand builder and guardian of walled gardens, the Apple-specific way of doing these are quite tyrannical. You need to have an Apple Developer account (only $99/annum!), you need to have a Mac, you need to use XCode and its toolchain, and you need to have a lot of patience. I had none of these. I’m a creature of speed and efficiency and rebellion. I could run the signing portion on a macOS runner on GitHub Actions but I can create all the binaries (including Windows ones) on a Linux machine, with a neat list of target architectures. I just don’t want to split just that part of the process.

After a lot of reading, exploration, and trial & error, I discovered the minimal steps and required files and certificates and secrets you need to get this done⁵. I also remembered the ambitious project from indygreg, opening Apple’s code signing black box to the masses and to other platforms: apple-platform-rs. Now, with the power of rcodesign, I could sign and notarize by bespoke binaries for macOS on the standard Linux CI machines.

Take that, final boss!

A maintainable script tool for all this

With all the stuff built in, my “simple” build script became a ~200-line monster with a few support files around. It was somewhat generalized but not enough for me to share it easily with others to prevent further suffering. This is why I decided to create a tool that would encapsulate all this logic and make it easy for anyone to create a self-contained binary for their Node.js application: presenting fossilize!

Fossilize does all the things above, including macOS signing and auto-discovery of assets through a Vite-compatible manifest.json file. It also caches the Node.js binaries it downloads to speed things up on repeated builds. It supports using different Node.js versions and understands a few simple aliases such as local, latest, and lts.

One irony is fossilize itself cannot be fossilized at the moment due to some of its dependencies requiring dynamically determined native binaries per platform and some obscure issue with postject not being able to postject code containing itself. I’m planning to tackle these with the help of WASM but for now, I think fossilize is in a good place to serve the need.

Onwards 🚀

Footnotes

Yet I happily use VS Code and Slack. Oh the hypocrisy! ↩
They also did even smarter things like code caching to speed up start up times. Node SEA also supports this. ↩
Or unfortunately, depending on how you look at it. ↩
Hoping your definition of fun includes several days of trial & error, reading docs written as if you have to use Apple devices competently with an ambition of reaching Lord of the Rings levels of prose, and some late nights. ↩
A blog post dedicated to this journey is being written as of this writing. ↩

Docker Volume Caching on GitHub Actions

Burak Yigit Kaya — Tue, 14 Jan 2025 00:00:00 +0000

I joined Sentry to exclusively work on their self-hosted product in 2019. Back then, Sentry was just using a few services: Postgres, Memcached, Redis, and Sentry itself. But it was on the cusp of becoming a multi-service application with the introduction of Snuba and along with that Kafka, Relay, Symbolicator and others. Because it was supposed to be simple, self-hosted (or onpremise as it was called back then) did not have any tests or even any automation: just a bunch of instructions and commands to run in the README. With the rapid increase in the number of engineers working on Sentry and the changes being made, it was clear that we needed to automate the testing and setup of the self-hosted repository.

To summarize about a year’s worth of work: we created an install script based in bash (as that was the most common denominator across all platforms), and a very cursory test suite which ran the install script, tried to ingest an event, and read it back. The entire test suite took about 5-6 minutes to run and about half of that time was spent on running Django migrations, from scratch, on a fresh database, over, and over, and over. The thing is we didn’t even add migrations frequently but we still had to run them all to get the service up and running.

The solution was obviously caching but caching Docker volumes was not really a thing that seemed feasible back then. Remember, this is 2019-2020, GitHub Actions was still in its infancy. I was also barely getting comfortable with all that Bash and Docker stuff. Then I got distracted by other things, changed jobs, and eventually came back to Sentry to see that this was still a problem. So I decided to tackle it head-on. I was going to cache the hell out of those Docker volumes for our databases. We already had actions/cache now so how hard could it be? Famous last words.

I have spent about 2 weeks to completely figure this out. About 50% of this was my ignorance about basic Linux tools such as tar, file/directory permissions, and Docker’s way of storing volumes. About 30% was me not trying things locally properly and just pushing to CI and waiting for the results. The remaining 20% was the actual hard parts to figure out, mostly thanks to StackOverflow (yeah, still not on that “ChatGPT for everything” bandwagon¹). I’ll summarize some of the findings here so you don’t have to go through the same pain as I did:

Docker volumes are stored under /var/lib/docker/volumes (by default, and please don’t change it)
You cannot stat a directory or anything under it if you don’t have x permission on the directory itself (╯°□°)╯︵ ┻━┻
tar does preserve permissions and ownership by default but only if you are running it as root (or with sudo) (╯°□°)╯︵ ┻━┻ x 2
tar preserves ownership information as names and not as IDs so if your Docker container uses a user id like 1000, GLHF ² (╯°□°)╯︵ ┻━┻ x 3
Linux (Unix?) fs permissions are not just rwx but there’s also an s you can set on executables to allow them to set ownership of other things³ ＼（〇_ｏ）／
Not only GitHub Actions doesn’t run tar with sudo, and not only it refuses to do this, it also doesn’t allow you to run tar with --same-owner or --numeric-owner (╯°□°)╯︵ ┻━┻ x 4
Bonus: there are these awesome tools called getfacl and setfacl that lets you backup and restore ACLs BUT NOT OWNERSHIP INFORMATION ~~(╯°□°)╯︵ ┻━┻ x 5~~
Bonus 2: mv would happily overwrite your target without even mentioning, especially if you use sudo.

So, with all this information, what is needed to cache Docker volumes on GitHub Actions and restore them properly? Let’s see:

Set +x permission on /var/lib/docker
Set +rx permission on /var/lib/docker/volumes
Set u+s permission on tar
Use tar --numeric-owner to create the archive — oh wait, you can’t because actions/cache doesn’t let you (╯°□°)╯︵ ┻━┻^{(╯°□°)╯︵ ┻━┻^{(╯°□°)╯︵ ┻━┻^{(╯°□°)╯︵ ┻━┻}}}

Side quest: Hacking `tar` on GitHub Actions

Once I realized that I had to change the options passed to tar, I very reluctantly decided to “wrap” the actual tar executable:

sudo cp /usr/bin/tar /usr/bin/tar.orig
sudo echo 'exec tar.orig --numeric-owner -p --same-owner "$@"' > /usr/bin/tar

Oh, but wait, you cannot sudo redirect output to a file as sudo just runs the command and redirection is done by the shell which you are not running as root. Let’s try that again:

sudo cp /usr/bin/tar /usr/bin/tar.orig
echo 'exec /usr/bin/tar.orig --numeric-owner -p --same-owner "$@"' | sudo tee /usr/bin/tar > /dev/null

Once I added this monstrosity, my GitHub Actions runs… started to hang indefinitely. Can you see the issue? ಠಿ_ಠ Well, I couldn’t. I spent about 2 hours trying to figure out why this was happening. I suspected exec might be the culprit and when I removed it, the runs at least started crashing with an error: cannot fork. What? Well, see I was doing this both in my restore and save actions. So, when the restore action ran, it wrapped/replaced tar but then did not restore the original back. After some time, save action ran trying to do the same. Now remember our “Bonus 2” learning from above: when save also backed up tar (which was actually my wrapper script) to /usr/bin/tar.orig, mv didn’t even flinch when tar.orig already existed. Now I had 2 copies of my wrapper script where the second one just execed itself. Nice fork bomb there, me⁴.

Once the fork bomb was defused, I was able to run actions/cache and viola! My volumes were cached and restored properly. Space time is saved Marty!

Final boss

After all this, I was still not very happy as it made all action/cache calls in my workflow doubled, and with the same hack repeated in both parts. So I decided to create a GitHub Action that would contain the chaos, the madness, the fork bomb minefield, and all the other ugliness. Both from my sight and others’. Please enjoy BYK/docker-volume-cache-action and cache responsibly.

Footnotes

That said all images for this article was generated by DeepAI Image Generator ↩
Looking at you confluentinc/cp-kafka ↩
Yes, yes, there are even more. Can you believe it? I couldn’t either. But I digress. ↩
Me when I realized this: mother forking shirt balls! ↩

Having a Good Ol' RSS Feed in Astro

Burak Yigit Kaya — Mon, 30 Dec 2024 00:00:00 +0000

After reviving this blog with Astro, I realized that I didn’t have an RSS feed even though the theme I’m using already has support for that. So I got to work to enable it. For some reason, I just did not have the rss.xml file generated. After much trial and error, I finally figured out that the method name in theendpoint definition should be ALL_CAPS as in GETinstead of get. I’m guessing this was because of a major Astro version upgrade since the demo page for Pacamara has a working RSS feed. Fixed that and problem solved, right? RIGHT?

Sort of.

Yes, I got a feed but I noticed 3 major problems:

The feed did not have the full content of the posts
The feed did not limit the number of entries
The feed did not sort the posts in any way

Although unsorted and uncapped posts was not a big deal as I only had 3 posts at the time, it looked like an easy fix so I started with that:

const MAX_ITEMS = 10;

const posts = (await getCollection("posts"))
  .sort(descDateSort)
  .slice(0, MAX_ITEMS);

Quite straightforward: get all the posts, sort by date in descending order, and slice the first 10. Don’t know why this is not the default or a least is offered through a built-in helper, but let’s move onto the bigger issue.

Getting the full content of the posts in RSS was much trickier as the RSS endpoint was defined as an “endpoint” and was not able to render Astro components. Even the recipe for RSS on Astro docs says this is only possible for Markdown only and it uses a custom Markdown renderer 🤯. But I was determined, and was a devotee of “the search church” so off I went to find a solution.

Although there was this very creative solution, I bumped into a more straightforward one first: https://blog.damato.design/posts/astro-rss-mdx/. This solution uses the new and experimental Astro Containers to be able to render an Astro component in isolation inside the endpoint. I followed the instructions and voilà! I had a working RSS feed with full content indeed. Committed, pushed, and got yelled at by GitHub Actions with the following cryptic error:

cannot test case insensitive FS, CLIENT_ENTRY does not point to an existing file: /home/runner/work/byk.github.io/byk.github.io/dist/client/client.mjs

I ran npm run build on my local terminal immediately, and was able to reproduce the error locally. At least, I was not going to play the “try blind commits to see what the CI says” game.

After searching for this error for about an hour, I realized that something was triggering a client-side render mode in Vite (Astro’s underlying bundler) and I started to remove every single new line of code I added. Indeed, once I disabled the import for both @astrojs/container and @astrojs/mdx the build error disappeared. As to why this was happening, I still had no idea. I kept digging and finally found this random (and very helpful) message on the Astro Containers Stage 3 proposal thread: https://github.com/withastro/roadmap/pull/916#issuecomment-2256059117

// astro.config.mjs -- add the following
{
  vite: {
    ssr: {
      external: ['astro/container', '@astrojs/mdx'],
    },
  },
}

Of course this makes sense! Without the above configuration, Vite tries to put these Astro packages into a client bundle whereas I am strictly operating in a server-side rendering world. Once this is in, the build error went away with MDX rendering still intact. I quickly pushed the code, got my deploy, and had my RSS feed! 🎉

I wanted to test my feed before declaring a complete victory so I loaded it up in Readwise Reader, my RSS reader of choice, and saw that the images were not loading. I quickly realized that I (well, Astro) was using relative paths for the images and that’s simply not how RSS works! I had to make all these image URLs absolute which was supposed be quite straightforward.

For some reason, I couldn’t find that simple answer after much searching. Then I tried to hook into the MDX pipeline to modify the URLs only to be disappointed as the image URLs are generated much later in the process and all I got was a JS identifier for the image source 🤦🏻‍♂️. After more research, I learned all about Astro’s image processing pipeline, found out about its getURL() method, dug into its source code andfinally saw that it uses import.meta.env.BASE_URL as the base!

Easy, I thought: I’ll just set that in the config under vite: {base: '...'}. That didn’t work. Then I tried setting it on the top-level Astro config only to be disappointed again. I also tried some other, sillier things that I don’t want to admit doing here. Finally, like really finally, I found the answer in build.assetsPrefix! Set this to my blog’s main URL, tested in dev mode to make sure it still works, got a full build, checked the rss.xml output and saw that the image URLs were now absolute! 🎉🎉🎉

So, if you ever want the same (Astro blog with full content RSS feed and working images), I hope I can save you some hours with this post.

Oh, by the way, if you want to subscribe to my blog, now you can 😏

Life Lessons from a Rotary Encoder

Burak Yigit Kaya — Thu, 24 Mar 2022 00:00:00 +0000

Recently I got back into an archaic pastime activity of mine: working on hobby electronics. I already had a breadboard lying around but since I am lazy I wanted an Arduino board, complete with all the things I may possibly need: push buttons, LEDs, a 7-segment display, a dot-matrix LCD etc. I bought a TinyLab experiment board based on the recommendation from my poorer self from 6 years ago on Facebook. I know Facebook is an evil machine forced upon us by our alien overlords but it has a tender side surfacing ancient wisdom via its memories feature. Anyway, one of the crucial and interesting components on the board was the rotary encoder. With its infinite, tactile rotation and the new-found popularity among the mechanical keyboard community, this little knob quickly became my new obsession. Trying to read from what’s passing through its contacts would lead to profound realizations about life, universe, and everything — moving our understanding of 42, an inch forward.

My rotary encoder has 5 contacts: VCC, ground, Phase A, Phase B, and push button. It is of the mechanical type. A mechanical encoder means there is mechanical contact between these A and B pins and the rotating disk inside. It is a lot like a pair of buttons being smashed in perfect order tens of times per second. The most interesting part is the rotation direction detection which is roughly done by figuring out which of these buttons get smashed first. The reason these pins are called “phase” contacts is because when the encoder is rotated, you get a square wave out of them which are out of phase by 90 degrees. This is so that you can determine the direction of rotation based on whichever is ahead of the other.

As a self-thought programmer who’s been writing code for 24 years on “ideal” computers, I didn’t even bother to learn much of the above at first. For me, this was simple:

there are two buttons
in each loop you read the values of these
if both are 0 = no rotation
if one is 1 and the other is 0 = encoder is turning in the direction of the pin that is on

Oh the arrogance even at this ripe age of 33! I was so wrong that it took me a whopping 3 days to reliably read this tiny little marvel of electromechanics. Let’s start with the “obvious” issues:

Reading 0 from both pins does not mean we are stopping
It is possible to read 1 from both pins and the one being 1 does not dictate the direction by its own
If you read the values in each loop, you may actually miss values as they don’t sit there and wait for you to read. Life goes on, the encoder keeps rotating, and if your main loop is slow you miss your window of opportunity

So that’s 3 out of 4 from my initial assumptions. At least, I was right about there being two buttons. Sort of.

The solutions were simple but profound. For timing, you prioritize reading rotary encoder inputs by using pin interrupts. An interrupt is a special instruction in micro-controllers that tell them to drop whatever they are doing and attend a special task. It is a lot like when your kid starts screaming: you drop whatever you’re doing and immediately ~~shush~~ soothe her. Luckily, rotary encoders are less demanding than toddlers: they just want to be heard (well, maybe kids want just that too?). So we read and store the state of these A and B pins when there’s a change. Ideally we’d set up the interrupts on both pins but due to the wiring of my board, and some limitations of the Atmega 32u4, I could only listen to one pin. This is very much like hearing only from one of your ears as the other one is gone due to the earlier screaming. Not terrible but you just have to accept the fact that you may not register the initial click in one direction. Again, definitely much less worse than losing your sense of sound directionality along with your will to live after getting yelled at your right ear just because you shut the water faucet off that has been running for the past 187 seconds for the entertainment of a little human’s growing mind.

The solution to detecting the direction of a “click” is also simple: just do some book keeping. Record which pin got triggered first, then on the next cycle compare its value with the new state. For instance if you saw A go high (meaning it switched from 0 to 1), while B is 0 you are rotating in one direction (phase of A is ahead of B). If B was 1 while A was going high, that means it is rotating in the other direction as phase of B is ahead of A (or they are playing a weird version of beer pong). Since this is a bunch of if statements and some variables, I wrote it up and tested quickly. I surely was able to read every single click on the encoder, that said the direction was completely unstable. Just like a toddler learning to ride a scooter, the direction was flipping like crazy. This made no sense at all (except for toddlers and scooters). Computer chips and solid metal disks listen to reason unlike 2-year-old human beings!

I tried blaming the compiler gods but they were too busy torturing my boxed copy who is learning Rust from a borrowed future memory segment¹. Thus, I seized this rare moment where I got to put my computer science knowledge to work. I was going to build a state machine as one wise blog post suggested. The idea is deceptively simple: not all states you can read from the pins are valid states. For instance, if you look at the wave picture above, you’d see that when you are turning in one direction, you should never see pin A from going low to high while B is 0. So you construct a state table, listing all states and valid state transitions you may accept, and ignore everything else. Doing this fixed all the weird direction jumps! The little cost I paid was some skipped clicks very occasionally but that’s a little price to pay for stability.

Although this was a success, I was simply wondering why I had to use actual math and science and how these invalid state transitions could happen in the first place. After some googling, it finally hit me: nothing is ideal, especially mechanical contacts! We can model them and show diagrams like the ones above as their idealized approximations but real world is just messy. It was simply the stuttering of these imperfect copper contacts, sending hysterical signals to my code which expected a perfect square wave. No wonder why it was confused.

In the end, I was quite surprised by getting smacked in the face by a rotary encoder with bitter truths about life:

timing and catching your window of opportunity is very important
life is just messy, no matter how much you try to smooth it out
Footnotes
1. Rust is a newish programming language that has a notoriously high learning-curve but provides great memory safety. ↩

The "Improbable" Truth: Rare Browser Bugs

Burak Yigit Kaya — Thu, 13 Dec 2012 00:00:00 +0000

Over the last two weeks at Disqus we discovered two annoying browser bugs. Both were only happening on iOS, which reminded me the famous quote from Sherlock Holmes: “when you have eliminated the impossible, whatever remains, however improbable, must be the truth”.

The first one was a bad one. We had reports from users not being able to login to our system from the embedded commenting widget, but only on iOS. The symptoms were even stranger:

Users were shown the login page in a popup
Users could login and the cookie was set
The popup stayed open but notified the embed about the logged in user
The embed did not recognize the logged in user until after a refresh

We started debugging and discovered that the embed actually makes a call to get details of the logged in user after it gets the notification from the popup. However, the result of the request was never processed. More so, if you refreshed the popup and then closed it, it worked!

We spent many hours in the iPhone emulator, mostly due to the cumbersome nature of all the emulators, we discovered that if you ever opened a popup and held a reference to it, iOS suspends the events on the parent page but still executes some of the code. So, in our case, the XHR call was being made and the response was received, but the callback was not called. The cause of our bug reports was this callback being responsible for closing the login popup. The embed was dead-locked for waiting for the request to finish to close the popup and the browser was waiting for the popup to close to fire the callback. As you can imagine, there were no visible reports of this behavior, anywhere on the internets.

The other one was even weirder: we were seeing a certain content being repeated exactly three times whereas it should have appended to the DOM only once. The intermittent nature of the problem suggested a hard-to-track race condition but that turned out not to be the case. We were able to mitigate the problem easily by emptying the parent element before appending the content. This was only a symptomatic cure though so we proceeded on our adventure to find the root cause.

After many hours of debugging, which was also very cumbersome due to the specific “ritual” to reproduce it consistently, we traced the problem to the XHR success callback getting called 3 times with readyState == 4 instead of only once. The new information suggested, surprise, a race condition, but tracking the number of XHR objects eliminated that possibility entirely.

It turns out that the onreadystatechange event was getting fired even thoughreadyState did not change at all. Not surprisingly, since this was happening at the completed state, the callback was fired multiple times for the same request. A Google search revealed another poor soul who encountered the same issue:https://github.com/madrobby/zepto/pull/633

There it was: a weird, hard-to-reproduce browser bug breaking our product randomly. After realizing this fact, we went ahead and did what we had to do:patched Reqwest, the XHR library we use at Disqus.

DEV Community: Burak Yigit Kaya

Marking it Up (and Down)

First, there was plain text

The Age of AI

Walking back from the X-factor

Putting it everywhere, and fast

Can’t Stop the Feeling!

Tying it all together

Footnotes

Nightmare on Apple Street

The Apple Way™

Switching to the Highway

Apple’s Sacred Stamp of Approval

The Entitled Apps

Sign here please6

Knock Knock Knocking on Notary’s Door

We’re Done Here

Resources

Thanks

Footnotes

Making your Node.js application last centuries

One binary to rule them all

Enter Node.js Single Executable Applications

Single-file Node.js Application (not a binary)

Packing the frontend assets in

A wild boss appears: signing and notarizing on macOS

A maintainable script tool for all this

Footnotes

Docker Volume Caching on GitHub Actions

Side quest: Hacking tar on GitHub Actions

Final boss

Footnotes

Having a Good Ol' RSS Feed in Astro

Life Lessons from a Rotary Encoder

Footnotes

The "Improbable" Truth: Rare Browser Bugs

Sign here please⁶

Side quest: Hacking `tar` on GitHub Actions