DEV Community: Matt Suhay

Sandwich Shop: A containerless, serverless experiment

Matt Suhay — Sun, 21 Nov 2021 22:33:00 +0000

Some jobs in your career will be pleasant, engaging uses of your waking hours. Challenges are welcome, and the pursuit of bettering yourself ahead of the product is the leading reason you show up every day. However, this was not one of those jobs. Instead, the days felt like they dragged on, with the only way of telling them apart was by weekends, forced birthday celebrations, or the occasional company meeting. The kind of meetings that applause for the senior leadership wasn't only expected but warranted a one-on-one with HR if you didn't provide one. An awkward call to the principal's office where your "company loyalties" were called into question.

HR person: I know we are entirely dissecting your team, putting you in the overflow building on the other side of the city, changing your fundamental job description, and firing a few people you've grown to rely on to do your job. But why weren't you clapping after we announced the half a million-dollar bonus we're giving the CEO instead of the usual 3% raise we offer our warehouse workers every year?

The day began like the rest—the acrid aroma of too cheap coffee brewed for too long in a forever-neglected Black and Decker was my usual welcome. I spent that morning writing serverless functions for a fledgling e-commerce site, a practice that usually involves explaining to the director that the term "serverless" didn't actually mean the absence of a server. Still, I humored him with the burden of explanation. Once completed and with the pitchy coffee cleansed from my mug and bowels, it was time for a lunchtime pilgrimage to Subway.

My co-workers had chosen the topic for the meal—a recount of a game they had recently picked up. Not having much to contribute, I sat quietly and unpacked my sandwich. I'm not sure how long I sat there, not saying anything when one of my co-workers asked what was on my mind.

"Ya know, serverless architecture is a lot like owning a sandwich shop. You put the foundations down, hire the people, stock the kitchen, and put out a menu. Only, with serverless, after the first customer comes in and places an order, we bulldoze the whole thing to the ground and start over when the next customer arrives." I don't believe I have ever seen a table with so many confused faces since my four-year-old nephew tried to tell a joke. At least my nephew got a couple of sympathetic chuckles.

But why do we do it that way? Why do we put our functions inside wrappers or only use specific languages? I want to put my code up somewhere and just use it as-is instead of the extra bits that go with it.

I didn't stay much longer with that company and was soon on my way to the next set of forced birthday celebrations.

suhay / sandwich-shop

Containerless, Serverless experiment using Go and GraphQL

Platform agnostic and Sandwiches (PaaS)

One of the more complex parts of having an idea is finding a way to use it. Logistics, good intentions, and planning aside, solving a problem for someone separates a good idea from a great one. Like the inventors of the cell phone when just starting off, "Who would use something like that?"

In a previous experiment, The Exothermic Project, we discussed what a platform would look like without a front or backend. This pet rock-like creature that neither eats nor poops would solve many problems for would-be pet owners, but it isn't exciting. And let's face it: there are web problems that can't be solved without a backend. I wanted to address this issue by providing a friend our pet rock could talk to. Stack agnostic website, meet the code agnostic platform. Now we had a place where we could write APIs against a database without exposing our passwords or integrating with 3rd parties without distributing our keys. Sandwich Shop will be the gateway, but having it just do one application's heavy lifting seemed like a waste.

With that in mind, we came up with a tenant architecture. By coming up with, of course, I mean we stole the tenant architecture idea from all the other PaaS (Platform as a Service) providers. The idea was to separate our applications by the tenant name. If a site wants to pull something from a database, it would just call https://example.com/shops/theSite/getPage. The webserver would proxy any call for /shops to Sandwich Shop, with the next part of the path being the tenant, and the last part being the function to run (or the Order to place).

Each tenant would have its own workspace. It would define an orders.yml file containing all the essential parts of placing an Order. It'll provide a path to the script, pick the runtime required to run it, set any custom authentication steps it needs to take and add any environment variables.

---
getPage: # order name
  runtime: node14 # shop to run in
  path: getPage.js # path where function lives
  env: # any variables to include
    - NODE_ENV=dev
getNews: # order name
  runtime: go1_17 # shop to run in
  path: news_sync.go # path where function lives
  env: [] # any variables to include

Tenants are also able to determine their own access authorization. By default, a tenant can provide a .key file within its workspace. An authorization will be provided to requests that have an Authorization: Bearer whatever-is-in-the-key-file header. Pretty basic, but it's at least something. Orders can also provide their own header definition, just in case Authorization is not used, and a custom auth function. These functions will be run just before the Order to check if the request is authorized. For example, something that comes from GitHub webhooks:

# orders.yml
---
suhay.dev:
  runtime: python3
  path: parse.py
  auth_header: X-Hub-Signature-256
  auth: auth.py

# auth.py
async def auth(header, data):
    token = os.environ.get("API_TOKEN")
    tokenb = bytes(token, 'utf-8')

    signature = 'sha256=' + hmac.new(tokenb, data, hashlib.sha256).hexdigest()
    if hmac.compare_digest(signature, header):
        return True
    return False

Since we defined a custom auth_header and auth, we will skip the default authorization step and immediately send the request to an available Shop. The Shop will then execute the script specified in auth, passing in the auth_header to be checked. The rest of the execution then waits on whether or not we return true or false from auth.py.

Franchising

Sandwich Shop will define a set of Shops used to broker an Order based upon the Shop's runtimes. A Shop is what actually runs the code associated with the Order and can live anywhere. For this example, we have one defined on localhost and port 4007.

[
  {
    "_id": "ciabatta",
    "name": "Ciabatta",
    "host": "http://localhost",
    "port": 4007,
    "runtimes": [
      "go1_17",
      "node14",
      "node16",
      "python3"
    ]
  }
]

We wanted to keep Sandwich Shop and the Shops as separate processes for scaling purposes. You also might want to specialize your Shops to run certain kinds of code. For instance, you can have one Shop that only does Node.js and lives in one place while another handles Python found in another.

If this, then paywall

I've always loved the idea of connecting my different apps and services together. When I'm researching something or writing an article, grabbing images or stories and sending them to my notes is simply incredible. However, they can be limited without a subscription, and I don't feel I use it enough to justify paying for one.

One of the first things I used Sandwich Shop for was to listen for GitHub webhooks. I can then use a development server to run any build steps and move the readied files to their respective webroots. By eliminating the need to SSH into multiple servers to deploy a change by hand has been lovely.

I also just started opening a few Shops that handle home automation. We can then set up a few Alexa skills and other triggers that you would see in IFTTT or Zapier.

Containing the excitement

One thought was to wrap the Shops in Docker containers. I decided this would be against the idea of being containerless. We should instead deploy them on different ports and across other machines if we need to scale.

What's next?

Something the current version of Sandwich Shop is missing is the ability to check on long-running processes. Right now there is really no way to see if there is an error, or even if it's stuck. You place the Order and then just hope you get what you want.

Standing up another Shop or tenant is something you have to do yourself, today. That's not the best, so allowing an easier way to add either from the command line or even from a web interface would make it a lot easier to scale and add additional projects to the flow.

Although we have added a layer of security to the requests themselves, we need to prevent malicious code from running in a Shop. If this experiment were to move forward to being a PaaS offering, we would want to make sure the code is executed within a rbash or even use something like jailing to prevent code from leaking into another tenant's workspace.

Lastly, comparing Sandwich Shop up against other solutions such as AWS would be interesting to see if we got anywhere close to solving a problem or just something fun to play around with.

The Exothermic Project: Poor design decisions for a new age

Matt Suhay — Mon, 20 Sep 2021 00:07:34 +0000

Since learning React, I've willingly thrown myself into the embrace of starry-eyed fandom. Not quite to the level of getting a tattoo of my obsession, but more than willing to show pictures of my work as if they were photos of my children to strangers. Like a proud parent watching their little ones pick up a Fisher Price hammer for the first time, I encourage my React-lings to treat every web problem as a nail and beam while they smash it into working order. I never once ask if React is the correct answer or even the easiest, so, naturally, when a CMS-shaped nail appeared, we got to work.

We looked at the different aspects that cause a content management system to become bloated in part one. In part two, we will dive into the murky waters of excessive features. We will do a little more experimenting and a lot more thinking—maybe even taking a few pictures of our React-lings along the way to be shared or included in this year's Christmas card.

An idea for a future post: AI Christmas card generator??

suhay / exothermicjs

YAML-based template engine for React

Maestro of many

The learning curve that accompanies React can be a non-starter for some. If you are unfamiliar with the framework, a developer may return to something more comfortable to achieve complicated requirements. Especially when doing client work, we are rarely afforded the time needed to learn something new, and this is precisely when a project can fall into the trap of "Master of None" from part one. When returning to our comfort zone, we start rearranging and renovating what we have into what we want—usually without checking if what we wish will even fit into the space. But, like moving a too-big couch up a too-small staircase, we still attempt to make it work by gouging, scraping, pivoting, and partially disassembling it on the way.

This is called "feature bloat." We fill our programs with more feature furniture than butts to sit on them, and this is a problem and a symptom of an inside-out design pattern. The application assumes that the page it's rendering needs everything until it's told differently. An example of the inside-out approach in WordPress is with their filters and hooks—how it handles injecting work into the render loop. There are over 400 default hooks and filters that do more work than we need for most pages. Even just checking if there is work to be done can waste time and resources.

I worked on a project that had over 100 feature flag checks. It was impossible.

So, we started experimenting with the idea of "feature delegation" instead of bloat. We wanted to take a more outside-in design approach allowing the core to conduct the application while the page dictates what it needs. Our core is mainly a YAML interpreter that stands at about 330kb, and it will stay there regardless of how long the page is or how many different pieces are needed. We made the render loop more efficient by completely eliminating it.

We took a simplified run of WordPress' render loop:

Request -> index.php -> query_posts() -> new WP_Query() -> query() -> get_posts() 
  -> the_post() -> setup_postdata() -> apply_filters() -> wp_reset_query() 
  -> wp_reset_postdata() -> HTML -> Response

and made it into this:

Request -> [url].exo -> useExothermic() -> buildTemplate() -> HTML

Intended side-effects

With our outside-in experiment, one question came up frequently:

"How do we create a dynamic app if the core will never change?"

We needed an abstraction layer that kept track of where our components were. We decided to use YAML and define our components as !custom tags. And since YAML is just a text file, we had the side-effect of manipulating our structure whenever we wanted to without needing to recompile the bundles.

- !section
  class: row w-100
  items:
    - !col
      class: col col-6 align-items-center
      content: |
        Coder<span>&#8226;</span>  
        Wheelman<span>&#8226;</span>  
        Dragon slayer<span>&#8226;</span>
    - !col
      id: me
      class: col col-6 d-flex align-items-end
      content: |
        ![](/images/pxArt.png)

------- Rendered React code -------
<Section className="row w-100">
  <Col className="col col-6 align-items-center">
    Coder<span>&#8226;</span><be />
    Wheelman<span>&#8226;</span><be />
    Dragon slayer<span>&#8226;</span>
  </Col>
  <Col id="me" className="col col-6 d-flex align-items-end">
    <img src="/images/pxArt.png" />
  </Col>
</Section>

Can be turned into:

- !section
  class: row w-100
  items:
    - !col
      class: col align-items-center
      content: |
        Coder<span>&#8226;</span>  
        Wheelman<span>&#8226;</span>  
        Dragon slayer<span>&#8226;</span>
- !section
  id: me
  class: row w-100
  content: |
    ![](/images/pxArt.png)

------- Rendered React code -------
<>
  <Section className="row w-100">
    <Col className="col col-12 align-items-center">
      Coder<span>&#8226;</span><be />
      Wheelman<span>&#8226;</span><be />
      Dragon slayer<span>&#8226;</span>
    </Col>
  </Section>
  <Section id="me" className="row w-100">
    <img src="/images/pxArt.png" />
  </Section >
</>

Effectively re-arranging components with a simple text editor, no Webpack or Babel, and all at runtime.

0 to live

There is always a ramp-up period when starting something fresh. You have to hook up the routes, judge which state management to use, wait for packages to load, and so on. We wanted to focus on getting pages up with as little bootstrapping as possible. The building blocks come predefined yet generic, the routes are handled by the .exo files, and the states are managed for you. Add the core library to a page, define a few blocks in the index.exo file and you're done. You can structure your landing page and subsequent pages with only these parts and without running npm, yarn, webpack, or make.

The great decoupling

Something that always amazes me is how many different flavors a tech stack can come in. Some will say theirs is better than another for reasons, but they all do just about the same thing, although in just about every configuration you can imagine. We wanted to be as agnostic as possible from the schema to the server, backend, microservices, containers, databases—everything. YAML and Markdown will get you started, but if you want to keep your blog authors in WordPress, then keep them in WordPress. Build out a plugin to handle the back and forth, and it's yours. It doesn't make sense to force a workflow change if you don't need to. The rivalries and infighting that result from preferring one thing over another don't make sense in the end when it's all just text on a screen.

By separating the different facets of the application from each other and from the core library, security patches can happen when they need to. Authorization and authentication can be anything you want or nothing at all. Hugely complicated roles and memberships should never have existed in the first place. When content and page layouts become separated, and when Exothermic eventually goes the way of the TRS-80 computer, you can bring your pages to wherever the hell the future has in store next.

Making it more better-er

Plugins will take center stage now that the lights are set and the cast has been—cast. Exothermic is meant to be a launching point, a substrate, for attaching whatever else you want it to do and plugins are how you'll make it happen. There is currently an example of a plugin that handles blog posts that you can reference.

On the roadmap, we've also started putting together what we're calling the Layer Engine, which allows you to create more reusable components across multiple pages. Hopefully reducing any extra copy-paste. This is specifically targeted at how content management systems currently treat templates. The templates enforce the reusable parts of a site by punching holes into what can be changed by the developer. They are restrictive by design and cause significant collections of one-offs to get around their limitations. With the Layer Engine, we want to relook at this and make it better.

I'd also like to see some kind of GUI to manipulate the .exo files, but we'll see what happens there.

Lastly, we want to go further with making everything as agnostic as possible. This started another experiment we're calling Sandwich Shop, where you can turn any piece of code into a microservice—with your choice of cheese, and served hot or cold.

The Exothermic Project: What I learned from 15 years of Content Management Systems

Matt Suhay — Sun, 11 Jul 2021 19:42:39 +0000

Many years ago, a great happening occurred that forever changed my low-stress development job. My day-to-day activities were typically filled with acts of building mundane addons that would live out their short, six-month lives as part of low-traffic ad campaigns. That was all replaced by a massive and excruciating task of moving an entire higher education institution's content management system to a new home. I will take some of the blame. The application we were using showed its age, and I was personally eager for career advancement by working with the newest, shiny thing. So, like any good, and very bored programmer, I began researching how to make our jobs more enjoyable by completely changing everything about them. If there was anything that I learned from all the time we spent moving stacks of content, revamping countless designs, or applying the latest buzzword in any number of three-letter acronyms—every single content management system was the same.

All of them.

I will concede that there is a use case for everything. I’m not here today to throw shade at the idea of a CMS. Instead, I’m hoping to open up the conversation and shed some light on the parts that might be overlooked. Look for any missing pieces that may drive any seasoned developer into a dark room at the end of days spent working with them. To push out the "That's just how it is" culture and replace it with a more welcoming one.

suhay / exothermicjs

YAML-based template engine for React

A master of none

At some point during a CMS's lifetime of feature additions and bug fixes, the core diverged to try and become a one-stop shop for everything, everyone, everywhere, every time, and ever after. Maybe it was to increase the demographic that would want to use the application, or perhaps it was because a client was putting on some added pressure. Whatever it was, at that point, the application should have split instead of continuing down a single-lane road towards a maintenance nightmare.

When the fundamental requirements changed, it should have been embraced and allowed for the two ideas to go off separately. Instead, the project will absorb these additions and mark them as breaking changes. You don’t realize that some of these new features had to be bolted on either by abstracting an already established data contract or by altering underlying core functionality. In most cases, the backend is sporting some kind of hack living freely in production code made to make something like this work. Don't try and deny it—you probably could list off several right now if you were asked to. They are found while stumbling around fixing something else and are surrounded by a big TODO with good intentions of being refactored. Comments that were added several years ago and haven't been touched since.

// TODO: this seems inefficient, will rework in next sprint ---> some_person, 2 years ago via PR #35 • final, final bug fix, for serious this time.

So why is this the rule instead of the exception? What if we removed this burden of effort by introducing a more modular architecture for our features? If you just want a blog, then you should only need to install the blog. There is no need to have the portfolio or e-commerce stuff in the mix when you're only trying to draw buzz for your client's opening day. And, trust me, we've all worked with those people that want everything in one place. It was probably one of them that started the initial fracture in the code.

By keeping things separate, we nurture these very different parts to be developed in parallel and by the people who want to do it. This internal API will talk to the core, should never change, and be nonrestrictive and nondestructive. Free your developers instead of handing them a 200-page document listing all the things you can and cannot do.

I got an integration manual from a fortune 500 company where all the coding examples were written in Silverlight and typed up in purple Comic Sans.

Content, aka the TL;DR

Most mainstream content management systems love shoving all of your content into a database, keeping everything centralized. However, these large TEXT or BLOB types can cause performance hits, especially if you run a search. There are ways you can get around these bottlenecks by using caching, but then you have to start worrying about when you should and shouldn't drop that cache. And then, by doing so, letting everyone know that the cache isn't there anymore so that they stop calling you on the phone about how it looks, "[...] correct in fox-fire, but not in the blue E one."

I should have asked them to check Netscape, just to make sure.

Up to this point in the article, I have written roughly 4500 characters. We are pushing about 7000 symbols into the database with markup and class names, meaning 33% of our storage is being used on—well—nothing. At least nothing a reader is going to see. All this extra data is taking up unnecessary space and locking large institutions into remaining with aging software because of how stuck their content is inside of these things. They have massive archives going back years and years through multiple upgrades and requiring many people to keep everything backward compatible. What if SQL or NoSQL disappears tomorrow? What if a major version release removes something that your architecture needs? At that point, do you just tell your users you can't upgrade even though there's a security issue?

Content should be stored in non-periphery forms, should only be the content that the user wants to see, and should be kept wherever the hell you want it to be. If a content solution has to retain customers through a hostage situation with your data, they probably need to change their business model. And the content that is coming into the core should be just that—content. If it's stored in flat files or cloud drives, it shouldn't matter. The core asks for page 5's content, so the only thing it should be expecting is page 5's content.

Trust your contributors. No, really

One of the site owners I was working with was planning on getting some temporary help to speed up their site move. I said, "Sure, we can get them all set up with accounts. Just let us know their email addresses." But that was when it was made clear that the site owner wanted this temporary help but didn't want them to publish content without it going past their eyes first. And they also shouldn't be allowed to upload new images. Or link to anything just in case the links went somewhere inappropriate. And everything should be time stamped to make sure the helpers are doing their job.

This was a genuine request.

Why would you hire someone to work on your site that you couldn't trust enough to publish a simple page? Or even let you know that a draft is ready to be viewed before going live? Adding granularity to security will reward you with a beach covered in sand. Making a change in the sand will only fill your shoes with it. Set your contributors up for success and show them the tools they will use the most, but not block their ability to do anything requiring more people to be involved. By saving time with temporary hires, you lose it all if they're still waiting on you for approval.

I worked in a system where the roles were: reviewer, author, author2, contrib, contrib2, contrib3, admin, admin2, and superAdmin

Plugin frameworks: wat?

So often developers are expected to learn a new framework. If it's changing jobs or starting a new side gig, there will always be a new way they have to learn to do something they've been doing for years. I can't even begin to count how many plugin frameworks I know, filed away in my mind, next to childhood addresses and that mental list of purchases I wanted to make at the next Scholastic Book Fair from the 3rd grade. A developer has to ingest unnecessarily complex function loops, hook names, and file structures to introduce one thing into one place used only one time and then never spoken about again. We attempt to qualify this effort by making the plugin open source, but who else needs this niche, one-off solution?

A plugin should have a single entry point and be defined through simple metadata. A plugin should be what its name suggests, plug and play, and not just for the consumer. The core only needs to get data into and then out of it, opening up avenues for upgrades and security patching by preventing instances where the plugin is so deeply integrated that it becomes two versions behind the current release.

A templated, template system of templated templates

This is where the developer's true creativity can shine. Building the perfect template requires adding layers upon layers, hooking up dynamic call-outs to bring in generated lists, and fleshing out semantic markup that would make any markup validator proud. But that's not the real world. Content is messy. Authors are fickle and quickly grow bored of the cookie-cutter-shaped baked goods that your masterfully nested templates stamp out.

"But can we put the images on the left?" suddenly, you are using template names like content-main-with-left-image. Bugs begin to appear in odd places, and tracking them down becomes a nightmare as you wind through the patchwork of conditional branches that add hero images on some pages, but not others, and only if they are tagged by informational or temp-tag-4. Your once beautiful markup is now marred by class names that extend further than the content.

I think temp-tag-4 may still be in use today since I introduced it for testing something in 2010.

Templates should be organic and not set in stone. If you have to make a one-off template for more pages than that use the base template, then they aren't one-offs anymore. Content artifacts should work regardless of how their page is structured, and a template should aid in the efficiency of publishing content while staying out of the way.

.page .main-content .content .content-w-tabs .image-con .image-noFloat .image-noFloat_5 .float-left

EOF

These are the main points I plan on focusing on while I revisit the ExothermicJS project in part 2. I like to think of these bullets as ideas to keep in mind while working on the next big thing. Thoughts that will hopefully end your next marketing pitch with applause, instead of an awkward silence followed by the question, "So what did you think?"

This is really how one of our demo calls ended with a potential vendor. I think there was dead air for a good 15 seconds after the sales person was finished before someone from my team forced out, "Seems—neat?"