DEV Community: Daniel Sieradski

I'm breaking up with DigitalOcean

Daniel Sieradski — Thu, 03 Mar 2022 17:47:16 +0000

I have used DigitalOcean as my primary cloud provider for the past seven years. Part of that decision was ideological: I don't want to support Amazon because of its abhorrent exploitative practices, nor Microsoft because of its contracts with ICE. Part was economical: Google Cloud, also not the nicest company but with a greener datacenter footprint, is simply far more expensive. And part was pure laziness: DigitalOcean is just really easy to use compared to other, more complex cloud offerings, and has suited my needs as a freelance web developer. Frankly, I was a big fan. Until now.

About a year ago — right around the time they started offering MySQL — I switched from using a third-party database hosting platform to DigitalOcean's managed database service. I set myself up a small cluster in just a few minutes, migrated over my databases, and ever since then, I haven't really had any complaints about its performance. But then yesterday at around 4PM ET, the roughly dozen websites on my droplets (DigitalOcean's cutesy name for VMs) lost database connectivity. I could still connect to the database directly from my local machine and if I SSHed into the droplets. But for some inexplicable reason, none of my websites could reach them.

I immediately opened a trouble ticket, linking to different sites erroring on different droplets and selecting "Something is broken," which supposedly escalates the issue for faster triage. Then I waited for a response. And waited. And waited.

Three hours later, I received:

Hi there,
Thanks for the follow-up
I understand you are having some connection issues between your site and the managed database
In other to better assist here, please can you share with us the screenshot of the error message you are seeing and the time of the connection error

Three hours waiting only to be asked, "Can you send screenshots of the erroring websites to which you provided links I could have just clicked?"

I patiently responded with the requested error messages. And waited again. And waited. Another three hours passed. Then my patience wore thin and I fired off another email, expressing my annoyance. Thirty minutes after that, I finally received a reply from someone on the database team. They told me that beyond a handful of CPU spikes, nothing looked particularly out of the ordinary, and suggested I temporarily disable the IP access restrictions on my database — to no avail.

I decided to go ahead and clone my database cluster and swap it in place of the other to see if the problem would persist. And it did. When I explained to the agent that I am able to connect to the new and old clusters locally and from the droplets, but still my sites would not connect, they declared that there is nothing wrong with the database cluster and passed me off to droplet support.

Hi there,
Thanks for the additional information.
As you are able to connect the managed database cluster, this implies there is no issue with the cluster.
For further investigation, I have forwarded this to our droplet team for further review.

An hour later (now eight hours of downtime), droplet support reached out with a boilerplate message about WordPress database connectivity, RAM usage, and tuning locally installed MySQL databases. Because that's just what I need for my managed MySQL database service — to tune the non-existent MySQL server on my droplet!

At this point, it's 12AM ET and I've had enough. I responded simply:

oh my god read a ticket before you respond for the love of god

And you know what this joker did? They sent me back almost the same exact boilerplate message about tuning a local MySQL database installation.

It took 17 hours for the database connectivity issue to self-correct. It just randomly started working again on its own. DigitalOcean had no follow-up, no explanation, no nothing. My best guess is something happened with the CA certificate (DO's managed databases require SSL even if you're connecting internally via VPC to a server open to only your IP, but okay) but I wouldn't even know how to begin to debug that and shouldn't have to! That's why I paid for managed database services! Because, as DigitalOcean claims, it's supposed to be "Worry free!"

Well, you should worry. Because if your DigitalOcean "managed" database cluster suddenly becomes unreachable by your web applications, you're apparently on your own, SLAs and customer loyalty be damned. Thus, after seven years of paying a not-insignificant sum of money to DigitalOcean, I examined their competitors, selected one, and have started migrating my clients' websites away from their platform.

Look, I have a lot of empathy for tech workers and I know from my own past experience that working in technical support can be unpleasant when dealing with frustrated customers. And I know that the burden on tech support agents has increased because fewer people want to work in such an underpaid and undervalued role and therefore their workload has grown in some cases to an unmanageable degree. But I can't allow my clients' websites to be down for 17 hours because your company isn't adequately staffed and your workers are too busy to read a ticket beyond scanning for keywords let alone to provide an answer that isn't cut-and-pasted from a knowledge base. I can't afford to spend literally thousands of dollars a year only to have to troubleshoot by myself the issues I thought I was paying your company to solve.

It's time to get your shit together, DigitalOcean. Only then can we rekindle what was once a great love.

10 things that would make GitHub Actions even better

Daniel Sieradski — Wed, 09 Feb 2022 19:48:01 +0000

As a DevOps engineer, I spend a great deal of time working on CI/CD pipelines, knee-deep in YAML, API documentation, and web search results, writing and re-writing workflows, running and re-running them over and over again, trying to smooth out the kinks. As such, I've come to develop strong opinions on how these things ought to work for developer's ease. And having spent the last several months working intimately with GitHub Actions specifically, I've come to develop some strong opinions about it in particular.

Overall, I find GitHub Actions to be an excellent service that's incredibly flexible, generally reliable, and has a wonderful, sizable community contributing to its vast catalog of pre-built workflows and workflow steps. However, there are some improvements that I believe should be made to GitHub Actions that will significantly enhance developers' experience.

1. A less disjointed user interface

Currently, if you want to check out the status of a job run, you have to:

Navigate to the corresponding repository in your browser (1-3 page loads if you start from the homepage)
Click the actions tab (+1 page load)
Select the workflow (+1 page load)
Click the run (+1 page load)
Click on the job (+1 page load)
Unfurl the step you want to view

That's 5-7 page loads just to check a job's status.

Sure, GitHub shows the final status of your job runs on the notifications page. But if you have a high velocity repo with a lot of contributors, that page is essentially useless noise.

There has to be a better way.

GitHub could, for example, add a run status view to the GitHub homepage that shows all your currently running jobs. Or, when you click the Actions tab on a repo, there could be a dropdown menu that shows the latest jobs. Anything is better than what it is now.

2. Subscribe to watch specific workflows

GitHub will notify you about the result of any job you initiate, but it won't notify you about the result of scheduled jobs triggered by GitHub's crontab or jobs initiated by other users. GitHub should therefore add the ability for users to subscribe to workflows so that they get notified of their outcome whenever they're run and by whoever.

3. Modular workflows

It's a little frustrating to have to manage multiple copies of the same block of YAML — especially a big block of YAML — when only one or two variables in that block differ between workflows. GitHub should add a way to include YAML files into other YAML files, and to pass variables between the two, so that you can reuse modules across workflows. Yes, you could create other GitHub Actions to include in your workflows, but that's kind of overkill, especially if the workflow is very specific to your development operations. Why not just...

~~A module:~~

name: My module
id: my_module
type: module
inputs: 
  - myVariable1
  - myVariable2

jobs:
  etc.

~~An include:~~

name: My workflow
jobs:
  my_job:
    runs-on: ubuntu-latest
    steps:
      - module: my_module
        inputs:
          - myVariable1: 'my input'
          - myVariable2: 'my other input'

~~This would save so much hassle.~~

As of November 2021, this exists. It's called a Workflow Call.

4. Workflows that trigger other workflows and stay grouped to your P.R.

Let's say you have a monorepo and you run separate workflows specific to each package within your monorepo on every pull request.

Each workflow needs to run the same yarn install from the root when it first starts up. But you've just updated your packages so there's a new yarn.lock and no cache available for your build. So now, each one of your workflows is going to start from scratch downloading and installing everything, adding at least five minutes to each run, unnecessarily eating up billable runtime.

But what if you could have a workflow that runs before your other workflows that detects changes in yarn.lock and pre-caches your node_modules first? You can configure a workflow to trigger other workflows using workflow_run or even Peter Evans' Repository Dispatch action (in which case use a personal access token, which is not ideal for organizations where it's preferable to not tie access keys to individual users' accounts).

However, the problem is that subsequent workflows will get buried under the Actions tab and won't be connected back to their trigger, specifically the P.R. that initiated the first workflow. Therefore, any checks in those workflows won't be applied to the P.R., rendering them essentially useless if you use them for quality control.

Again, there's gotta be a better way, and it could be as simple as modifying the workflow_run trigger to keep triggered workflows connected back to their initiator.

5. Custom check results that go where they belong

GitHub Actions allows you to add custom checks to your workflows — validators that can block the merging of pull requests or the deployment of new builds. But for some bizarre reason, it doesn't attach those custom checks to the workflows that triggered them. It will add them to any workflow triggered by the same P.R.

Per Michael Dorner, author of the Test Reporter Action:

Check Runs are created for specific commit SHA. It's not possible to specify under which workflow test report should belong if more workflows are running for the same SHA. Thanks to this GitHub "feature" it's possible your test report will appear in an unexpected place in GitHub UI.

It's true! And extremely confusing! And annoying! And GitHub should fix it!

6. Automatically capture `stdout/stderr` from any step as an output and add more metadata as outputs

In GitHub Actions, if you want to return the output of a command as a block of text to use in another job or step, you have to do some crazy BASH wizardry like:

- name: Prettier
  id: prettier
  run: |
./node_modules/.bin/prettier --check 2>&1 | tee prettier-report.txt

echo "RESULTS<<EOF" >> $GITHUB_ENV
echo -e "$(cat prettier-report.txt)" >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV

Each job step emits an object that contains user-designated outputs (which have their own issues, see #7), the outcome of the commands that were run — whether they were successful or not, and the conclusion of the step — whether it was successful or not. And...that's it.

Why not include the actual output of what was run on the step object, too, instead of making me go through all that boilerplate each time I need to capture it?

GitHub could also add other useful metadata to the step object, such as the duration and time started and finished. These data are available from the GitHub API and gh CLI, so why not in the workflows themselves?

7. Create an easier way of exporting multiline strings

This jumble right here is how you export a multiline string in GitHub Actions:

echo "RESULTS<<EOF" >> $GITHUB_ENV
echo -e "$(cat my_file.txt)" >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV

You can't use outputs because for some inexplicable reason they're single-line only. (You don't want to know how many failed runs I went through before I figured that out.)

You have to use a global environment variable that gets passed to every job in the workflow. Oh, and be careful that what you've echoed to $GITHUB_ENV isn't too many lines, because it may crash your other jobs with an environment variable is too long error. Super-helpful, right?

Say it with me one more time: There has to be a better way. Maybe it's...

8. Add another variable layer that is neither secrets nor environment variables

Not everything is a secret that needs to be concealed in the logs. And not everything is an environment variable that needs to be exposed to the whole job container. Sometimes you just need a place to jam some text and a way to get it back later. And that text isn't always gonna be one short, convenient string. There needs to be somewhere other than outputs and environment variables that developers can temporarily stick stuff for our workflows. So how about adding some other kind of object that can actually handle things like multiline strings and, gasp, JSON objects gracefully? (And don't you dare say "Use Upload Artifacts.")

9. Change the status indicator to a warning symbol when `continue_on_error` is enabled and a step fails

As mentioned previously, each job step emits an object that contains the outcome of the commands that were run — whether they were successful or not, and the conclusion of the step – whether it was successful or not. The distinction is important.

You can continue a job where the outcome of a step is failure by adding a continue_on_error flag to the step. The step will then succeed even if the command it ran failed.

Sometimes this is useful if you want to, for example, upload test reports before terminating a job. To make the job fail after you complete your final steps, you have to create a validation step at the very end that checks whether the outcome on the actual test was success or failure and then emits an exit code of 0 or 1.

This is fine if you're only validating one test. But if you're validating multiple tests, you need to evaluate multiple steps' outcome to determine the status of the job. Suddenly it becomes more difficult to quickly identify which test failed because there's a big green check next to all the tests, but only one red X at the very end.

What GitHub can do here is change it so that if the conclusion is success but the outcome of a step is failure, the symbol is a yellow warning sign (⚠️) rather than a green checkmark. That way, you'll quickly know which test in the group failed.

10. Better log management

One of the most painful aspects of working with GitHub Actions is reviewing the logs of jobs with a ton of output. Sometimes it's not even worth trying on the website and I go right to the gh CLI. It's just arduously slow and buggy. Sometimes you'll just see thousands of blank lines and have to refresh the page several times before you see anything.

GitHub can address this in a number of ways. One is by sticking the logs in a scrollable window, truncating them, and tailing only the most recent output, making the previous lines available if you scroll up in the log window. They could also offer the option of showing stdout, stderr, or both for each step, so that you can control what gets output to the screen.

**Bonus: Notify users on Slack**

There are several GitHub Actions you can add to your workflows to send notifications to a Slack channel. But not everybody on your team needs to know about every workflow run and there's presently no way to notify individual users directly of the outcomes of the checks on their own P.R.s. Part of this is simply the limitations of Slack's webhooks integration which is channel restricted. But GitHub itself could partner with Slack to develop an integration that allows organizations to connect their Slack workspaces to GitHub and that ties users' identities on GitHub to their identities on Slack. This way, you could create notifications that target individual users on Slack, instead of whole channels.

As you can see, GitHub Actions offers a great deal of functionality and flexibility to developers. But there's always room for improvement, and in the areas I note, those improvements are sorely needed.

If you have any ideas for ways GitHub Actions could be better, please suggest them below. And if you know anyone on the product team at GitHub whose eyes you can get this article in front of, please do!

Render Markdown as HTML in Strapi using controllers

Daniel Sieradski — Sun, 28 Feb 2021 22:56:20 +0000

Strapi is a wonderfully robust headless content management system — one I absolutely adore and believe you will as well — but one shortcoming some users have identified, myself included, is that there's no built-in option to render HTML from the Markdown generated by the default WYSIWYG text editor.

To address this issue, some users opt to replace the default editor with one that saves HTML to the database rather than Markdown, such as CKEditor. Others choose to render the Markdown in their frontend application directly, adding the additional overhead to their site builds, which is not always desirable when working with the JAMstack and every kilobyte in the lambda counts.

There is, however, another way. While Strapi does not parse your Markdown for you, it does provide a powerful interface for modifying outgoing API responses, called controllers, which allow for you to parse the Markdown in the response itself.

It all depends on our dependencies

For this project, I have selected Marked as the Markdown parser we'll be using. Before beginning, install it in your Strapi project's root folder:

yarn add marked

Head of the class

Create a new folder in your Strapi project's root called functions. Fire up your editor and create a new file within the functions folder called md.js.

We'll now create a new class for our Markdown renderer. The class constructor will take three parameters:

model: The model of the collection or single-type whose text fields will be rendered to HTML. From this object, we'll take the value of the attributes key.
types: The field types to convert to HTML. This will be an object consisting of two arrays: One for standard fields to render wrapped in <p> tags and one for inline fields which will not be wrapped.
options: This is an options object with settings corresponding to our chosen Markdown parser.

We'll also specify global defaults and instantiate our Markdown renderer.

const defaults = {
  types: {
    standard: ['richtext'],
    inline: ['string']
  },
  options: {
    smartypants: true,
    headerIds: false,
    breaks: true
  }
}

class StrapiMarkdown {
  constructor(
    model,
    types = defaults.types,
    options = defaults.options
  ) {
      this.model = model.attributes
      this.types = types

      this.marked = require('marked')
      this.marked.setOptions(options)
    }
}

module.exports = StrapiMarkdown

You're valid

Now that the class is defined, we'll add some validation to ensure that any errors in the data passed to our constructor are caught. We'll do this with some basic if/else statements that check whether the necessary fields exist in the passed objects.

For brevity's sake, I've redacted the defaults and module export and will continue in this fashion as we proceed.

class StrapiMarkdown {
  constructor(model, types, options) {
    if (model && model.attributes) {
      this.model = model.attributes
    } else {
      throw new Error('`model` must be valid model object')
    }

    if (types && types.standard && Array.isArray(types.standard) && types.inline && Array.isArray(types.inline)) {
      this.types = types
    } else {
      throw new Error('`types` must be object containing `standard` and `inline` arrays')
    }

    if (options && options.constructor === Object) {
      this.marked = require('marked')
      this.marked.setOptions(options)
    } else {
      throw new Error('`options` must be valid object')
    }
  }
}

The method to the madness

Next, we'll add two methods to our class, in the form of arrow functions, so that they inherit the class' this object.

The first, parse, is the actual data handler. It steps through the model and applies the Markdown parser to the data fields matching those specified in our types configuration object. For each key in the collection/single type's model, we'll check if the corresponding key exists in the incoming response object. If so, we'll check whether its type matches either the standard or inline format, and then apply the Markdown parser.

Because Strapi passes all responses as promises, we'll need to resolve the promise to properly access the data. I prefer the async/await syntax, so that's what I'll be using.

class StrapiMarkdown {
  constructor(model, types, options) { ... }

  parse = async data => {
    try {
      const item = await data

      for (let key in this.model) {
        if (item[key]) {
          if (this.types.standard.includes(this.model[key].type)) {
            item[key] = this.marked(item[key])
          } else if (this.types.inline.includes(this.model[key].type)) {
            item[key] = this.marked.parseInline(item[key])
          }
        }
      }
      return item
    } catch (err) {
      console.error(err)
    }
  }
}

The second method, md, determines whether the response data passed in from our controller is an array or a single object and, in turn, passes it to the data handler accordingly. Because we need to resolve all the promises in the parse method before passing back the data to our controller, we'll use Promise.all to resolve each object as it's mapped over.

class StrapiMarkdown {
  constructor(model, types, options) { ... }
  parse = async data => { ... }

  md = data => {
    try {
      if (Array.isArray(data)) {
        return Promise.all(data.map(obj => this.parse(obj)))
      } else {
        return this.parse(data)
      }
    } catch (err) {
      console.error(err)
    }
  }
}

Bringing it all back home

Our completed md.js file should now look like this:

const defaults = {
  types: {
    standard: ['richtext'],
    inline: ['string']
  },
  options: {
    smartypants: true,
    headerIds: false,
    breaks: true
  }
}

class StrapiMarkdown {
  constructor(
    model,
    types = defaults.types,
    options = defaults.options
  ) {
    if (model && model.attributes) {
      this.model = model.attributes
    } else {
      throw new Error('`model` must be valid model object')
    }

    if (types && types.standard && Array.isArray(types.standard) && types.inline && Array.isArray(types.inline)) {
      this.types = types
    } else {
      throw new Error('`types` must be object containing `standard` and `inline` arrays')
    }

    if (options && options.constructor === Object) {
      this.marked = require('marked')
      this.marked.setOptions(options)
    } else {
      throw new Error('`options` must be valid object')
    }
  }

  parse = async data => {
    try {
      const item = await data

      for (let key in this.model) {
        if (item[key]) {
          if (this.types.standard.includes(this.model[key].type)) {
            item[key] = this.marked(item[key])
          } else if (this.types.inline.includes(this.model[key].type)) {
            item[key] = this.marked.parseInline(item[key])
          }
        }
      }
      return item
    } catch (err) {
      console.error(err)
    }
  }

  md = data => {
    try {
      if (Array.isArray(data)) {
        return Promise.all(data.map(obj => this.parse(obj)))
      } else {
        return this.parse(data)
      }
    } catch (err) {
      console.error(err)
    }
  }
}

module.exports = StrapiMarkdown

Everything is under control

With our class finished, we can now add it to our API's controllers.

Navigate to the controllers folder corresponding to the collection or single type whose output you want to modify and open the controller file in your editor (eg. api/posts/controllers/posts.js).

First, import the class we created, then the model of the collection or single type.

const StrapiMarkdown = require('../../functions/md.js')
const model = require('../models/posts.settings.json')

Next, instantiate the class.

const { md } = new StrapiMarkdown(model)

If you wish to change the fields to be parsed, or adjust the parser options, you can pass those settings in as well.

const types = {
  standard: ['richtext', 'text'],
  inline: []
}

const options = {
  smartypants: false,
  headerIds: true,
  breaks: true
}

const { md } = new StrapiMarkdown(model, types, options)

Finally, we'll create custom find and findOne methods to replace the default methods Strapi generates internally. Each method will await the corresponding Strapi service method invoked with its corresponding default parameters but now wrapped in our Markdown class' md method.

module.exports = {
  async find(ctx) {
    return md(await strapi.services.posts.find(ctx.query))
  },
  async findOne(ctx) {
    const { id } = ctx.params
    return md(await strapi.services.posts.findOne({ id }))
  }
}

Once it's all put together you should have:

const StrapiMarkdown = require('../../functions/md.js')
const model = require('../models/posts.settings.json')

const { md } = new StrapiMarkdown(model)

module.exports = {
  async find(ctx) {
    return md(await strapi.services.posts.find(ctx.query))
  },
  async findOne(ctx) {
    const { id } = ctx.params
    return md(await strapi.services.posts.findOne({ id }))
  }
}

Rinse and repeat for each collection or single type whose output you wish to transform from Markdown to HTML.

Testing, testing, 1-2-3

Start up your Strapi project and give your API a call!

curl http://localhost:1337/your_modified_collection

If all went well, you should now be seeing HTML instead of Markdown in your API's response data.

Wrapping up

You should now have some basic grounding in how to create your own custom Strapi controllers to transform your API response data.

If creating the handler seemed like an inordinate amount of work when you can just use something off-the-shelf in your frontend project, bear in mind, you only need to create such a module once and you can then use it over-and-over again in all your different Strapi projects.

In fact, I've already gone ahead and saved you the trouble, turning this tutorial into an npm module that you can import into your Strapi controllers so that you never have to roll your own!

yarn add strapi-markdown-parser

Now you can skip every other step and jump straight to:

const StrapiMarkdown = require('strapi-markdown-parser')
const model = require('../models/posts.settings.json')

const { md } = new StrapiMarkdown(model)

module.exports = {
  async find(ctx) {
    return md(await strapi.services.posts.find(ctx.query))
  },
  async findOne(ctx) {
    const { id } = ctx.params
    return md(await strapi.services.posts.findOne({ id }))
  }
}

Why didn't I just say that in the first place? Well, you wouldn't have learned much then, would you? 😉