DEV Community: Clay Murray

The Ultimate Asset Pipeline.

Clay Murray — Mon, 07 Sep 2020 22:50:42 +0000

I have two previous posts about my adventures in creating a pipeline to get my Raw PSD files into a PNG texture atlas.

Well it turned into an entire GUI desktop application, Cute Exporter.

With this newest way of exporting assets I do have it all! Every step in the pipeline can be controlled by the Exporter (and therefore whatever code I write). It makes adding in expanded metadata easy, and works how I want it to. Maybe you can find it useful too.

The exporter can do everything I was doing before through several scripts and programming languages by itself. No need to install any other programs.

Read PSD or PNG or Aseprite files.
Edit and export meta data about each asset file and Layer in that file.
Export each layer in the Asset to a PNG and PNG texture atlas.
Export all metadata to a json file to get info about each entry in the texture atlas and tags and other metadata.

Seriously, You don't need to own Photoshop or Aseprite to export any of those files! It also has a CLI (which I need to document more) included. You can automate your workflow with it pretty easily as well.

The tool is written in a programming language called Odin. It's a lower level programming language, with a small but engaged community. It's a bit of an mix between go and C. If you ever wanted to do some lower level programming it might be one to try out. There is a community discord

For the graphics and windowing stuff I use a library called Raylib. Super beginner friendly and easy to start with for games. But it is in C. However, there are ports of it to many other languages so check those out too. Odin has a great foreign system so I compile the Raylib code as a static library and link it with my odin code. (I should clean up my Odin code for this and make the repo public).

I'm still using sqlite because I think it rules. Similar procedure to Raylib. I compile the sqlite as an object file and link it in Odin. You can check out that code at the repo I made for it

Maybe you don't care about games. Well it still can be useful for you in the web as well.

It can be useful to combine your little assets into one image. Then the end user only needs to download a single file. Using CSS background-image to display the proper graphic on your site.

Ever have raw files from designers that are in PSD format and you want PNGs? You can export them with Cute Exporter. It caches all the individual PNG files to disk before it stitches them into a texture atlas. You can then easily grab them from the cache folder.

But it's mostly cool for games. And making games in HTML5 is really accessible.

Asset pipeline updates

Clay Murray — Wed, 08 Jul 2020 00:35:02 +0000

In a previous post I talked about my asset pipeline.

I have now made a few changes/improvements to the system.

SQLite

SQLite is a pretty neat tool. It's a database system saved to a single file. I am using it to store information about my game assets.

In my previous pipeline I was adding every PDF to a simple text line with a bit more info. It was easy and worked, but I started to want a little more data about assets individually.

Even if certain assets are grouped together in a PDF I wanted to be able to add data to an asset on an individual level. Maybe different assets should have different scaling, or associated tags, or making animations easier.

All this can be done in a simple text file, but it's fairly relational data so why not use a simple database to store the info.

For my tables I made a few

raw_asset, the PDF file for an asset.
game_asset, the individual assets as used in the game.
asset_tag, tags for individual assets.
animation, animation info.

It's pretty simple. raw_assets speficy a name for the group of assets in it and the path to the PDF.
Game assets have a raw_asset they belong to that specifies where in the PDF they are by path eg. Group/layer name.
Then game assets can have tags and what animation they belong to.
Animations are just an id and a name.

Effects

These changes have made the pipeline a bit simpler in some ways and added a bit more complexity to other things, but at the benefit of providing more robust information.

We no longer need to do much inspection of the PDF to get groups and layers in ruby. We can just query the database about the raw assets and for each raw asset get its associated game assets.
Since the game asset tells us in the raw asset where it is, we can just pass that info straight to gimp to export the PNG.

Before the name and path of the asset in the sprites folder gave us metadata about it. Harper/Front/walk_1.png said the group name and the facing direction and the animation and the frame.
While giving some information it's hard to add more.
With the database existing the script doesn't need to save to paths with names that encode data about our asset. The script just saves by the id from sqlite and the group name e.g Harper/26.png (using the group name as a folder purely to make them easier to see in the finder).

When it comes to packing the texture we have all we need to know: the asset id and where it is in the texture atlas. Outputting a much slimmer file that's is pure JSON from texturepacker. It also removes any weirdness with figuring out the frame number from the asset name in the custom formatter in texture packer.


{
    "frames": [

            {
                "filename":"battle/28",
                "frame":{"x":1,"y":3478,"w":500,"h":216},
                "trimmed":true
            }, 

            {
                "filename":"buildings/20",
                "frame":{"x":497,"y":1709,"w":242,"h":500},
                "trimmed":true
            }, 

            {
                "filename":"buildings/21",
                "frame":{"x":1,"y":564,"w":500,"h":412},
                "trimmed":true
            }
    ],
    "meta": {
        "image":"sprites.png"
    }
}

I made a script with Node.js that takes the texture packer json file. Looks up the asset in the db and adds any other metadata we care about to it. Such as the frame number, the tags, the id and group.
It still formats it as lua because I didn't want to change my engine to read json right now...

return {
    frames = {
        {
            filename="battle/28",
            frame={
                x=1,
                y=3478,
                w=500,
                h=216
            },
            trimmed=true,
            tags={
                "battle","banner"
            },
            animationName="",
            id=28,
            name="battle_banner",
            group="battle",
            frameNumber=-1
        },{
            filename="buildings/20",
            frame={
                x=497,
                y=1709,
                w=242,
                h=500
            },
            trimmed=true,
            tags={
                "regular"
            },
            animationName="",
            id=20,
            name="some_building",
            group="buildings",
            frameNumber=-1
        }
    }
}

Since SQLite can give us always unique ids that are consistent across runs for my assets, it makes it easy to build tables about animations in the engine.

Previously when I loaded this data in my engine I was building up animation info and asset info in the same loops. It felt fragile and hacky. The engine now just loads all my assets info from the lua file then builds up animation info as a seperate step.

After the engine loads all the asset info it has a consistent id for an asset that it can always reference in my game and ask the database questions about it if I need it to. (which I do!)

With all the asset info being loaded the engine can build up the animation info. In my engine I just make a call with popen and run sqlite3 database.db "select * from animations".

This will produce a list containing an id and name for all animations. For each animation I can now query about which asset ids are assocated eg sqlite3 database.db "select id from game_asset where animation_id=2".

Now the engine will make up some animation info tables. Essentially a map of animation names and an array of assets ids that are part of the animation.

Later in the gameplay code if the character is walking I can tell the engine just to use the animation named"harper_front_walk".

Pretty fun!

Anway here's a picture of a new things I drew Grambly

Asset Pipeline in my Game

Clay Murray — Tue, 30 Jun 2020 17:58:23 +0000

Every game has assets, and somehow those assets have to get into the game. Here's what I do.

1. Creating the Assets

So first step is obviously creating the assets. What I am doing is different from most people. I draw the assets in a sketchbook, then I scan those images into my computer.
After that I take the scans and mask out all the parts I don't want, rotate stuff, and make everything feel real good.

Photoshop organization

When making an asset for some entities you want animations to have animations on. And one of my goals is keeping the same character's asset data in the same PSD.

So in my PSDs I just groups. With the name of the group being a direction (you can go 4 cardinal directions in my game) and a frame or still image in that direction. Animations cycles are specified via a name_framenumber syntax (eg walk_1)

For other types of entities like buildings I don't really care to have animation frames. In those cases I don't use groups or anything.

2. Exporting the Assets

Now that the assets have been created we want to export them. You could have a script that just looks for PSD files in a directory and exports them all. However, I want a bit more granularity in my process.
I am using a really simple file format to specify the assets I want to use and some other information about them.

assets/raw/harper.psd harper group 75%
assets/raw/some_building.psd buildings single x500
assets/raw/the_store.psd buildings single 500x
assets/raw/robo_dude.psd robots single x400
assets/raw/pointy_boy.psd ui single x50
assets/raw/square_boy.psd robots single x400
assets/raw/roller_boy.psd robots single x400
assets/raw/skelly_boy_blue.psd skeletons single x400
assets/raw/dewey_cheetum_howe.psd buildings single 500x
assets/raw/battle_banner.psd battle single 500x

The format is just space seperated items in a simple text file.

Path to PSD
Grouping to save to
If this PSD uses groups or if it just has a single layer.
imagemagick scale factor.

I wrote a ruby script that takes this info and reads the lines.

Over each line it loads the resulting PSD, finds all the layers we want to export and prepares the directories.

I do a check to see if the PSD is newer than the result PNG or the result PNG doesnt exist. That way we only have to run the export on assets that have changed.

If it's a grouped PSD each group is a sub folder inside the group name and the layer name is the exported png name (eg harper/Back/walk_2.png).

If it's a single layer we save it to the group folder with the PSD file name as the resulting png name (eg buildings/the_store.png).

Saving to PNG

The ruby PSD library I am using is too slow on its own to save PSDs at a reasonable rate. At first I tried using imagemagick but it doesnt accept layer names, only the scene index. Like: convert file.psd[1] out.png

I could never find a way to map between a imagemagick scene and a layer in my ruby PSD stuff but Imagemagick is fine for single layer PSDs and I use it to export those.

GIMP

Well as it turns out gimp has a scripting interface that doesn't even have to show a GUI. I figured out how to use it to export my grouped PSD files.
Here's the process

Ruby passes the group and layer name to gimp along with the path to the PSD and a path to save the final image to.
Gimp finds that layer, makes sure it is visible and saves it to the path we gave.
- this process is somewhat weird. If you just try to save the layer and only the layer it won't include the clipping mask. So you need to make the layer you want visible and hide all the other layers. Then you can merge visible layers and save the result of that.

Here's what my resulting GIMP script looks like:

#!/bin/bash
set -e

# Start gimp with python-fu batch-interpreter
/Applications/GIMP-2.10.app/Contents/MacOS/gimp -i --batch-interpreter=python-fu-eval -b - << EOF
import gimpfu
import traceback

def convert(filename, groupName, layerName, out):
    try:
            print(filename, groupName, layerName, out)
            img = pdb.gimp_file_load(filename, filename)
            if len(groupName) == 0:
                layer = pdb.get_layer_by_name(layerName)
                pdb.gimp_file_save(img, layer, out, out)
            else:
                for layer in img.layers:
                    if pdb.gimp_item_get_name(layer) == groupName:
                        children = pdb.gimp_item_get_children(layer)[1]
                        for child in children:
                            item = gimp.Item.from_id(child)
                            if pdb.gimp_item_get_name(item).split(" ")[0] == layerName:
                                pdb.gimp_item_set_visible(item, True)
                            else:
                                pdb.gimp_item_set_visible(item, False)
                    else:
                        pdb.gimp_item_set_visible(layer, False)

                nLayer = pdb.gimp_image_merge_visible_layers(img, CLIP_TO_IMAGE)
                pdb.gimp_file_save(img, nLayer, out, out)
            pdb.gimp_image_delete(img)
            pdb.gimp_quit(0)
    except RuntimeError as err:
                print(traceback.print_exc())
                pdb.gimp_quit(1)

convert('$1', '$2', '$3', '$4')
EOF

(at one point I was going to use gimp for the group and single layer path so the logic is still there)

Back to ruby

After gimp has our file saved or image magick exported our layer. We write an entry to another txt file this is a list of files that we want imagemagick to resize and the size to make it.

It ends up something like this:

assets/sprites/harper/back/walk_2.png 75%

Imagemagick

With a simple bash script we take each line of the file and resize it according to the sizing info in that line.

It can be passed whatever sizing info imagemagick can accept. Percentages, WIDTHxHEIGHT, xHEIGHT, WIDTHx etc.

Now we have all our images exported to folders.

Texture atlas

Now I want the images packed into one texture atlas PNG. I use TexturePacker Pro. I paid for pro so I could make a custom exporter.

TexturePacker can export from the command line making it easy to include in the automated pipeline.

All texturepacker is doing is looking for files in the sprites directory and adding them to the texture atlas.

TexturePacker exports to a lua file that the engine will read.

Into the Game

Now we have all our stuff exported into a texture atlas with a lua file describing the bounds and names of all the files. I embedded lua in the engine and we read the file and get the resulting data from it into odin. At some point I could reload assets on the fly. I just don't do that yet.

The cool part of all this for me is having a fully automated pipeline. No exporting from photoshop or opening other tools. Just works from one command.

Video of it running.

What do you use for server logging?

Clay Murray — Tue, 03 Dec 2019 20:32:13 +0000

I am looking for a good logging solution for my newest project. For reference I just nodejs and docker. Scanning docker logs isn't the best solution. On previous projects I used cloudwatch from AWS (which is great) but I want to avoid using any AWS services on this project. Some criteria I'd enjoy.

Picks up directly from docker logs no need to change my application code.
Well priced. I won't be generating that many logs.
Mildly searchable. I don't need to look back on logs that often.
30 day retention. Don't need much more than that.
Cloud hosted. I don't want to setup more infrastructure for logging.
Preferably run by a smaller company. I like using services away from huge tech companies. For instance I use postmark for emailing.

What do any of you use for your logging services. What do you like about them?

Incoming email hooks and you!

Clay Murray — Mon, 11 Nov 2019 16:10:31 +0000

Sometimes when you are making your cool app thing you want to accept email. This could be for a myriad of reasons.

Consider the Github use case. When someone comments on an issue you are watching you will receive an email. If you reply to the email it adds a comment to the issue!

Let's learn about some cool email headers and how we might go about building that ourselves.

I'll be using Postmark as the email sender/receiver. Most other email services like them also have this functionality. I just think they are cool and use them for my own projects.

Setup

I'm going to skip a bit past the initial steps of creating an account and a server. Postmark has that well documented.

What we care about is the incoming email hook.

First we go to default inbound stream.

Then in the settings tab the inbound email hook

We'll also want to set an inbound email domain. You'll have to look up How to set that up in your DNS

Okay so we have our quick setup. Postmark has great docs so check those out for more setup info. Though to note it's very important to have a custom inbound domain to accomplish what we want to do!

Sending our email

Let's send our email to the user. What do we want to track how we will know it is them?

Message-ID

The message id is a unique id for every email sent. It's a header that will be sent with any email to or from you. You can see the message id in Gmail by showing the original email. They can be any value but are typically formated as <some-random-id@sender.com> and email I got from twitter had the Message-ID: <42.0B.41170.86A79BB5@twitter.com>

In-Reply-To

This email header is the starting point for how email clients thread emails. When you reply to an email your client puts the Message-ID of the email sent to you in the In-Reply-To header

References

This header contains the Message-IDs of all the emails in the email thread. Every time a new email is send in a thread the Message-ID of the email being replied to is also appened to the References field

Reply-To

When you send an email you can set a reply to address that differs from the to address you sent the email from. These headers are going to enable our workflow.

So let's send our email.

// Require:
const postmark = require("postmark");

// Send an email:
const client = new postmark.Client("POSTMARK-SERVER-API-TOKEN-HERE");
const emailId = await generateAndStoreNewEmailId(commentThread, user);
client.sendEmail({
  "From": "sender@example.com",
  "To": "recipient@example.com",
  "Reply-To": `inbound+${emailId}@inbound.ourdomain.com`
  "Subject": "Test",
  "TextBody": "Hello from Postmark!",
  "Headers": [{Name: "Message-ID", Value: `<${emailId}@inbound.ourdomain.com>`}]
});

Okay so notice we set our Reply-To to an inbound email with an id we generated. We also want to store the user we are sending this email to and what comment thread.

Now we wait for a reply

Handling the inbound email

When any email is sent to our inbound email domain we will get a post request to our webhook. The entire message is quite big you can see it here. We just care about a few fields. We will use them to lookup the context for this email and add the comment.

The reason we used the inbound+${emailId} in our Reply-To was because postmark is nice enough to parse that id off for us and add it to a field called MailboxHash.

Now we can lookup the thread this email is talking about from the emailId.

const emailInfo = await getEmailInfo(inboundEmail.MailboxHash)

Postmark is also nice enough to parse the email body and give us just the reply text. We will use that for the commend body!

const body = inboundEmail.TextBody;

Then we can do something.

await createCommentInThread(
    emailInfo.commentThreadId,
    emailInfo.user,
    body
)

Conclusion

We also stored the email id in the Message-ID so if for some reason we can't find our comment thread via Reply-To we can search the References or In-Reply-To fields to check as well.

This is the general strategy that GitLab uses

This was a pretty high level overview but hopefully it helps you understand how easy it can be to make your app accept incoming emails!

Building JuniperCity.com

Clay Murray — Mon, 04 Nov 2019 18:41:38 +0000

So I started a new thing: Juniper City.

While I also want to shamelessly plug it. I also want to talk about the tech in it and decisions that were made.

The What

Juniper City is a place to post and manage your events. The idea is to get away from having to use Facebook and making it really easy for people not on the platform to use.

There's some core concepts I wanted to keep with.

No account creation.
Invite people where they are.
Usable over email.

No account creation

Technically you do create an account but I wanted it to be easy. We don't have passwords. You put in your email and we send you a one time password to login.

The goal is very little friction for people you invite to use the site. So when you do invite people they get a link with a key in the query params. That key logs them in. They didn't have to think about anything, and no redirect to do what they wanted to do.

Invite people where they are.

I wanted this to be something you could use for your entire family or group. Not everyone has Facebook or any number of social accounts. But almost everyone has an email or can get texts.

You invite people not by a username we have, but their email or phone. In the background if we don't have them on file we will make a user for them. Like mentioned before we have links that auto log them in, but since we made a user and have no passwords, they can log in at anytime without it.

Usable over email

I didn't quite launch with everything I wanted for this part. We currently send out invites over email but I wanted to do more.

So coming soon the future, you will be able to email Juniper City to create your event without ever visiting our site.

The people in the to field of the email get invited. You never have to even copy and paste over emails this way. Your email client is now our fronted client.

The big goal is trying to make useful open tools.

The How

So how was it built?

So the stack for the site is:

Docker
Nodejs
Hapi
Postgres
Redis
Stimulusjs

The entire stack is one monolith. No micro-services and we use server side rendering. It makes it simple to program for and deploy.

Frontend

This is a really simple frontend and we use a really simple framework. I've talked about Stimulusjs in a lot in my other posts so I won't belabor the point.

We use ejs for server side templates. (Though I am thinking of switching to Nunjucks it just seems nicer).

The total JavaScript bundle in production is around 350 kb and 65 kb minified. That's tiny. We don't do a lot in the frontend so why bloat it?

All our frontend is doing is just a few AJAX requests as well as hiding and showing things. Stimulus really does help keep this organized.

Keeping the site server side rendered with minimal JavaScript give a big advantage. Once we respond from the server, we're serving up blazing fast pages. Taking advantage of Turbolinks as well to make it feel single page. Can you get similar results with React or Vue? Yes, but it's not as EASY.

From the styling end we use Tailwindcss it's just awesome. Worth checking out. Makes putting together designs an absolute breeze.

Tools

For bundling we use Parcel. I don't have a very strong opinion on this but parcel is very near it's promise of zero config.

In production to keep the CSS small (tailwind has a lot of unused css) we use PurgeCSS. This pushes our css to 7.5 kb and 2 kb gziped.

Server side.

The stack here is pretty standard and easy.

Nodejs as the application logic.
Postgres as our database.
Redis for job queuing.

We integrate with Postmark for email delivery. For SMS delivery we use Textbelt. Also note that it will only send texts in the US.

I found Textbelt after a large amount of searching.

Some other options I didn't like.

Twilio
- Wow is this complex. Do I have to buy a number? How can I JUST send a text
AWS SNS
- Simple and cheap but I was searching to get away from using AWS.

Queues

We make use of queues and task workers. Take a look at this thing I wrote for an in depth about it.

Anytime we want to send an email or a text, we put it into a queue. Incoming web hooks? Queues. In essence anything you don't want to do right now. Queue it baby! The great thing is our current system has great support for running things at a later date as well. Things like cron jobs fit this use case.

Deploying

We package the server and frontend code in a docker container (Docker is so great).

Right now we server all our assets from the server. For the future we'll likely move static assets (CSS, JS, images) to a CDN. It's good enough right now.

Hosting is on Digital Ocean. Amazon is kinda evil so I wanted to try something else. What Digital Ocean lacks in configuration and power it makes up for in simplicity.

If you couldn't tell simplicity is a core tenant of this entire project.

We Build the project using docker.
Upload the image to a registry.
Deploy with docker compose.
- I learned about this option in compose -H "ssh://user@server". As long as you have SSH access to your remote machine it's the same as running the commands on your own machine. In the past I would copy the env files and compose files over to the remote machine as part of the build step. Using this new method you don't have to do ANY of that.
- docker-compose -H "ssh://user@server" pull image
- docker-compose -H "ssh://user@server" up -d

Again really simple.

Conclusion

I hope you liked that overview of what Juniper City is. Maybe you can find a use for it!

Sending emails with templates using MJML

Clay Murray — Wed, 23 Oct 2019 18:11:33 +0000

Sending emails is something a lot of web apps like to do. Password resets, notifications, promotions etc.

One of the biggest annoyances in sending emails is HTML EMAILS! They are very messy, ugly, and impossible to figure out.

In the past we would design our emails using Mailchimp then export them as an HTML email. This results in a convoluted mess of HTML that no one wants to make minor edits to.

Can we do better? Of course we can that's why I wrote this article!

MJML

Enter MJML. It's a neat little library to make it easier to keep your HTML emails as code without going crazy!

This is just a quick example from their site:

<mjml>
  <mj-body>
    <mj-section>
      <mj-column>
        <mj-image width="100px" src="https://mjml.io/assets/img/logo-small.png"></mj-image>
        <mj-divider border-color="#F45E43"></mj-divider>
        <mj-text font-size="20px" color="#F45E43" font-family="helvetica">Hello World</mj-text>
      </mj-column>
    </mj-section>
  </mj-body>
</mjml>

As you can see, it is very readable and HTML like. Much easier to edit and maintain!

They even have a free online editor to see what your email will look like!

MJML will take the code that you write and transform it into something like this:

<!doctype html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office">

<head>
  <title> </title>
  <!--[if !mso]><!-- -->
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <!--<![endif]-->
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <style type="text/css">
    #outlook a {
      padding: 0;
    }

    .ReadMsgBody {
      width: 100%;
    }

    .ExternalClass {
      width: 100%;
    }

    .ExternalClass * {
      line-height: 100%;
    }

    body {
      margin: 0;
      padding: 0;
      -webkit-text-size-adjust: 100%;
      -ms-text-size-adjust: 100%;
    }

    table,
    td {
      border-collapse: collapse;
      mso-table-lspace: 0pt;
      mso-table-rspace: 0pt;
    }

    img {
      border: 0;
      height: auto;
      line-height: 100%;
      outline: none;
      text-decoration: none;
      -ms-interpolation-mode: bicubic;
    }

    p {
      display: block;
      margin: 13px 0;
    }
  </style>
  <!--[if !mso]><!-->
  <style type="text/css">
    @media only screen and (max-width:480px) {
      @-ms-viewport {
        width: 320px;
      }
      @viewport {
        width: 320px;
      }
    }
  </style>
  <!--<![endif]-->
  <!--[if mso]>
        <xml>
        <o:OfficeDocumentSettings>
          <o:AllowPNG/>
          <o:PixelsPerInch>96</o:PixelsPerInch>
        </o:OfficeDocumentSettings>
        </xml>
        <![endif]-->
  <!--[if lte mso 11]>
        <style type="text/css">
          .outlook-group-fix { width:100% !important; }
        </style>
        <![endif]-->
  <style type="text/css">
    @media only screen and (min-width:480px) {
      .mj-column-per-100 {
        width: 100% !important;
        max-width: 100%;
      }
    }
  </style>
  <style type="text/css">
    @media only screen and (max-width:480px) {
      table.full-width-mobile {
        width: 100% !important;
      }
      td.full-width-mobile {
        width: auto !important;
      }
    }
  </style>
</head>

<body>
  <div style="">
    <!--[if mso | IE]>
      <table
         align="center" border="0" cellpadding="0" cellspacing="0" class="" style="width:600px;" width="600"
      >
        <tr>
          <td style="line-height:0px;font-size:0px;mso-line-height-rule:exactly;">
      <![endif]-->
    <div style="Margin:0px auto;max-width:600px;">
      <table align="center" border="0" cellpadding="0" cellspacing="0" role="presentation" style="width:100%;">
        <tbody>
          <tr>
            <td style="direction:ltr;font-size:0px;padding:20px 0;text-align:center;vertical-align:top;">
              <!--[if mso | IE]>
                  <table role="presentation" border="0" cellpadding="0" cellspacing="0">

        <tr>

            <td
               class="" style="vertical-align:top;width:600px;"
            >
          <![endif]-->
              <div class="mj-column-per-100 outlook-group-fix" style="font-size:13px;text-align:left;direction:ltr;display:inline-block;vertical-align:top;width:100%;">
                <table border="0" cellpadding="0" cellspacing="0" role="presentation" style="vertical-align:top;" width="100%">
                  <tr>
                    <td align="center" style="font-size:0px;padding:10px 25px;word-break:break-word;">
                      <table border="0" cellpadding="0" cellspacing="0" role="presentation" style="border-collapse:collapse;border-spacing:0px;">
                        <tbody>
                          <tr>
                            <td style="width:100px;"> <img height="auto" src="https://mjml.io/assets/img/logo-small.png" style="border:0;display:block;outline:none;text-decoration:none;height:auto;width:100%;" width="100" /> </td>
                          </tr>
                        </tbody>
                      </table>
                    </td>
                  </tr>
                  <tr>
                    <td style="font-size:0px;padding:10px 25px;word-break:break-word;">
                      <p style="border-top:solid 4px #F45E43;font-size:1;margin:0px auto;width:100%;"> </p>
                      <!--[if mso | IE]>
        <table
           align="center" border="0" cellpadding="0" cellspacing="0" style="border-top:solid 4px #F45E43;font-size:1;margin:0px auto;width:550px;" role="presentation" width="550px"
        >
          <tr>
            <td style="height:0;line-height:0;">
              &nbsp;
            </td>
          </tr>
        </table>
      <![endif]-->
                    </td>
                  </tr>
                  <tr>
                    <td align="left" style="font-size:0px;padding:10px 25px;word-break:break-word;">
                      <div style="font-family:helvetica;font-size:20px;line-height:1;text-align:left;color:#F45E43;"> Hello World </div>
                    </td>
                  </tr>
                </table>
              </div>
              <!--[if mso | IE]>
            </td>

        </tr>

                  </table>
                <![endif]-->
            </td>
          </tr>
        </tbody>
      </table>
    </div>
    <!--[if mso | IE]>
          </td>
        </tr>
      </table>
      <![endif]-->
  </div>
</body>

</html>

AHHHHH!!!! That's just terrifying. We thank MJML for fighting the demons for us!

Now we can transform our MJML to HTML but it's currently static.

So what about templating you may ask?

Well...

Templating

We still probably want to be able to use our MJML to make email templates. We want nice things like our user's name and custom links. Good web stuff.

For that I use mustache. It's fairly simple to use:

Our template:

<mjml>
  <mj-body>
    <mj-section>
      <mj-column>
        <mj-image width="100px" src="https://mjml.io/assets/img/logo-small.png"></mj-image>
        <mj-divider border-color="#F45E43"></mj-divider>
        <mj-text font-size="20px" color="#F45E43" font-family="helvetica">Hello {{user}}</mj-text>
      </mj-column>
    </mj-section>
  </mj-body>
</mjml>

Our code:


const mustache = require("mustache");

const templateData = {
    "user": "John"
}

const renderedMJML = mustache.render(mjmlTemplate, templateData);

Now we have filled in our template with mustache. But we are still in the MJML format.

Why did we do that? Well MJML makes huge transformations to the code we hand to it. If we try to run mustache after we convert to HTML we will lose our ability to use mustache.

Luckily mustache doesn't much care what type of document we throw at it. It only cares about {{}}. (Incidentally this allows you to use mustache in many other applications including JSON)

Let's now convert from MJML to HTML.

const mjml = require("mjml");
const html =  mjml(renderedMJML).html;
// don't forget the `.html`

Now we have some HTML but we still need to...

Sending email

Okay we now have an HTML template. We want to send it.
I am going to use the Postmark Api because it's really easy.

const fetch = require("node-fetch");

await fetch("https://api.postmarkapp.com/email", {
    method: "POST",
    headers: {
      "Accept": "application/json",
      "Content-Type": "application/json",
      "X-Postmark-Server-Token": "server token"
    },
    body: JSON.stringify({
        To: "example@example.com",
        From: "example2@example.com",
        Subject: "This is a test",
        HtmlBody: html
    })
})

There you go.

Full Javascript

const fetch = require("node-fetch");
const mustache = require("mustache");
const mjml = require("mjml");

const templateData = {
    "user": "John"
}

const mjmlTemplate = `
<mjml>
  <mj-body>
    <mj-section>
      <mj-column>
        <mj-image width="100px" src="https://mjml.io/assets/img/logo-small.png"></mj-image>
        <mj-divider border-color="#F45E43"></mj-divider>
        <mj-text font-size="20px" color="#F45E43" font-family="helvetica">Hello {{user}}</mj-text>
      </mj-column>
    </mj-section>
  </mj-body>
</mjml>
`

const renderedMJML = mustache.render(mjmlTemplate, templateData);

const html =  mjml(renderedMJML).html;
// don't forget the `.html`


await fetch("https://api.postmarkapp.com/email", {
    method: "POST",
    headers: {
      "Accept": "application/json",
      "Content-Type": "application/json",
      "X-Postmark-Server-Token": "server token"
    },
    body: JSON.stringify({
        To: "example@example.com",
        From: "example2@example.com",
        Subject: "This is a test",
        HtmlBody: html
    })
})

Simplifying my old event bus architecture with Bull

Clay Murray — Wed, 09 Oct 2019 20:12:18 +0000

The why

So before I posted about an event bus with AWS SNS and SQS. I think it works really well and it's pretty cool. It did have some rough edges that bothered me though.

There was a lot of manual setup in production.
- There are scripts etc that can do this but I was having to make the queue and SNS topic and link them for every new queue I wanted to make
Local dev was messy
- There are fairly solid Docker images that imitate SNS and SQS but the setup in code felt very messy.
Our task worker code, while cool, felt fragile.

So how does bull help?

Task worker management:

They have a solid interface for registering and running task workers. Making it much less work on our end to handle that part of the code.

Keeping all the code contained:

All pieces of the flow exist in the same codebase. Queue creation, event passing and task running without external setup for production.

Less moving parts:

In the previous iteration you had SNS, SQS, custom task handler code, and server code. The updated version uses just server code, Bull, and Redis. While only one less system is used in the updated version. We reap many benefits from the battle tested, and frankly much better, Bull code.

Portability:

Our task code now only relies on Redis. This makes it much easier to move to another cloud provider. No code changes required.

The How

So it's pretty simple to implement. (You can refer to the Bull documentation for most of this as well.)

Setting up queues

We make a queue for each type of work we want to do. Such as emailing or sending texts.

const emailQueue = new Queue('email', 'redis://127.0.0.1:6379');
const smsQueue = new Queue('sms', 'redis://127.0.0.1:6379');

Then we have all our queues in an array so when an event comes in we can pass that to all our queues

const queues = [emailQueue, smsQueue];

function onEvent(name, data){
    queues.forEach((queue)=>{
            queue.add({
                name,
                data
            })
    })
}

And we can define our queue processors.

emailQueue.process(emailTaskWorker);

Pretty simple stuff.

Final thoughts.

Why pass events to all queues? It allows any any queue to decide what it wants to do.

For example. If a user commented on a post, we might want to email them, text them, or generate a push notification. If in our code we did:

server.post("/posts/:id/comment, (req) => {
    server.sendEmail(req)
    server.sendText(req)
    server.sendPush(req)
})

That would get really messy. Instead we just make a record of what event happened.

server.createEvent("user-commented-on-post", {post, user});

Now any queue that cares about that event can do whatever it wants in response. It keeps our http handler code very clean as well.

Automating Lambda Deployment With Travis-CI

Clay Murray — Fri, 27 Sep 2019 17:02:41 +0000

This is a post from my old blog. Info in here may be out of date or useless.

January 11, 2017

I really hate doing things myself when I can have a computer do it for me.

So In this post I will describe how I automate lambda deployment with travis-ci.

Create a AWS role that has these permissions.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

I wanted to lock down the permissions more but you basically need them all.

Get an access key for this role.

Second add your repo to travis. It's easy I won't tell you how to do it.

Now we create a .travis.yml file.

language: node_js
node_js:
- '4.3'
env:
  global:
  - AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
  - AWS_DEFAULT_REGION=us-east-1
branches:
  only:
  - master
before_install:
- pip install --user awscli
- export PATH=$PATH:$HOME/.local/bin
script: make build

Here we add our AWS key id to the file. We'll also want to add our secret key but we want to encrypt it. Using the travis CLI

travis encrypt AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY --add env.global

We should be set on the travis side.

Now onto building and uploading

Here is my make file for building

.PHONY: build clean upload
build:
    npm install --only=production
    zip -r code.zip . -x *.git*

clean:
    if [ -a code.zip ]; then rm code.zip; fi

upload: build
    ./upload.sh

here is my upload.sh

#!/bin/bash
aws lambda update-function-code \
--zip-file=fileb://code.zip \
--region=us-east-1 \
--function-name=NAME_OF_YOUR_LAMBDA

Now everytime you push to master it should update your lambda to the newest code!

Making Your Own 2.5D Renderer Thing In C++ For Fun

Clay Murray — Fri, 27 Sep 2019 16:52:58 +0000

I had an old Blog and want to get the posts on here:

From February 2018:

So I like to program. And I guess I like the idea of making a game. I really haven't finised many.
Previously most of my attempts have been in Javacript using canvas I made some library/engine thing even.
I mean it isn't very good is it?

Anyway, I think writing low level code makes you cool. And I want to be cool like Casey Muratori or Jonathan Blow
I also think old school engines like Build, Doom, or Wolfenstien 3D are cool.
Which makes me want to eschew all better ideas and do one similar to those.
However, since I don't hate myself completely. I do use SDL for the platform layer.

In the rest of this post I will go over some cool things I think I do.

Raycasting

I don't really want to explain what ray casting is. I just want to say what I do. It's probably stupid but I like it.

Instead of just sending out a ray and following its path to see what it hits (I tried this). I do something a little different.
I have a list of line segments that are the walls. They contain a start position, end position, texture index, and if they are transparent or not.
For every ray I spit out (on each column of the screen) I also make a line segment the start is the player position and the end is some place like 2000 units away.
Then every ray checks itself against every wall.
Checking if it

Hits the wall
Where it hits the wall

To know if a wall is already occulded I keep a buffer of every column and the largest height of the wall drawn there.
If the height is smaller it doesn't get drawn.
If the wall is transparent the drawing gets deffered until after all the rays have finished their checks then those walls are drawn back to front. Again checking the height buffer.

In terms of rendering that's about what I have so far. Nothing too exciting but I think it's cool I even did it.
I'm working on floors and ceilings now. It's pretty tough.

Hot reloading.

I learned this one from Casey Muratori. On his stream Handmade Hero

If you separate your game into two parts. A platform layer and game layer.
You can compile your platform layer as an EXE and your game layer as a DLL.
As long as the platform layer holds the gamestate and passes it to the game layer.
Anytime the DLL is rebuilt you can reload it in your platform layer and see your code changes on the fly without having to close and reopen the exe.

Here is my shitty SDL code to do that.

Oh and, we copy the DLL to a temp file so our game doesn't keep a file lock on the DLL and keep us from reloading it.

        char *LockFileName = "../build/lock.tmp";
        char *TempDLLName = "../build/temp.seville.dll";
        char *SourceDLLName = "../build/seville.dll";

        stat(SourceDLLName, &buff);
        if(modified < buff.st_mtime){
            if(GameCodeDLL){
                SDL_UnloadObject(GameCodeDLL);
                UpdateAndRender = NULL;
            }
            SDL_RWops *f = SDL_RWFromFile(LockFileName, "rb");

            if(!f)
            {
                if(!stb_copyfile(SourceDLLName, TempDLLName)){
                    err = SDL_GetError();
                }

                GameCodeDLL = SDL_LoadObject(TempDLLName);
                if(GameCodeDLL)
                {
                    UpdateAndRender = (game_update_and_render *)
                        SDL_LoadFunction(GameCodeDLL, "GameUpdateAndRender");

                    modified = buff.st_mtime;
                } else {
                    int var = 1+1;
                    err = SDL_GetError();
                }
            }
            if(f){
                SDL_RWclose(f);
            }
        }
        if(UpdateAndRender){

            UpdateAndRender(Renderer, keysHeld, &state);
        } else {
            err = SDL_GetError();
            int var = 1+1;
        }

So that's where I'm at.

How do you do language translations

Clay Murray — Wed, 25 Sep 2019 02:13:00 +0000

Having worked at a company that supported a large number of languages I always wondered how other developers managed theirs.

The basic idea when I first started was: static JSON files with key value pairs.

So English might look like:

{
    "welcome": "Hello!"
}

Then in the code you would reference the key to fill in the value.

Something like:

<Translate>welcome</Translate>

That's not a bad solution and the basic ideas were pretty alright. However, it has a few problems.

The people who can translate for you don't usually know JSON
It doesn't provide much context on what the translation should be
It requires a code release every time you need to update a language or even fix a typo.

#1 is not too hard. You can convert to CSV for loading into excel which is a tool most people know. You can still run into data conversion issues.

#2 I never had a great solution for. Typically you would have to do a lot of work to provide screenshots and go back and forth for them to have proper context. If you site was already live it was easier since they could just look at the live site.

#3 Can be solved with hosting the JSON in another place such as S3 and have the app pull the data from there. But then you no longer have version control.

Eventually I created an internal app to solve #1 and #3. It was a react app that used firebase as the storage. It integrated with our LDAP so we had simple SSO.
We also had language and environment based permissions. You could say a user could only access French in dev.
It had a way to promote dev to production as well.

Firebase was nice to use in that you can have direct JSON access URLs so the consuming app never had to install their bulky tools. (It wasn't important to have live updating of the translations)
The app worked but was clunky, ugly, buggy and had little buy in from anyone but IT. I could think of many things I might improve on it.

This was a bit of a ramble. But after talking about what I've done and worked on. What do you guys do?

Write your own map filter and reduce

Clay Murray — Tue, 16 Jul 2019 22:05:51 +0000

So most of us know that these functions already exists.

[].map();
[].filter();
[].reduce();

But how do they work internally? It's actually pretty easy. So let's just write them ourselves. Cool

Map

function map(array, cb){
   const result = [];
   for(let i = 0; i < array.length; i++){
      const item = array[i];
      result.push(cb(item, i));
   }
   return result;
}

Map is pretty simple.

Filter

function filter(array, cb){
    const result = [];
    for(let i = 0; i < array.length; i++){
       const item = array[i];
       const check = cb(item, i);
       if(check){
            result.push(item);
       }
    }
    return result;

}

Also not too tough.

Reduce

function reduce(array, cb, init){
    let result = init;
    for(let i = 0; i < array.length; i++){
        const item = array[i];
        result = cb(result, item, i);
    }

    return result;
}

Reduce is a little more involved but it isn't anything too crazy. We just make sure to return what they passed last iteration to the next iteration.

Final note:

Interestingly enough. Reduce can be our base construct for all our other loops and we can use it to implement map and filter

Map using reduce

function map(array, cb){
    return reduce(array, (carry, item, i) => {
        const result = cb(item, i);
        carry.push(result);
        return carry;
    }, [])
}

Filter using reduce

function filter(array, cb){
    return reduce(array, (carry, item, i) => {
        const check = cb(item, i);
        if(check){
            carry.push(result);
        }
        return carry;
    }, []);
}

Anyway don't roll your own reduces and maps unless you have a good reason. But it is nice to know how things work at a lower level. That way you can better utilize them in the future.

Thanks for reading.