DEV Community: DropConfig

Building JuniperCity.com

Clay Murray — Mon, 04 Nov 2019 18:41:38 +0000

So I started a new thing: Juniper City.

While I also want to shamelessly plug it. I also want to talk about the tech in it and decisions that were made.

The What

Juniper City is a place to post and manage your events. The idea is to get away from having to use Facebook and making it really easy for people not on the platform to use.

There's some core concepts I wanted to keep with.

No account creation.
Invite people where they are.
Usable over email.

No account creation

Technically you do create an account but I wanted it to be easy. We don't have passwords. You put in your email and we send you a one time password to login.

The goal is very little friction for people you invite to use the site. So when you do invite people they get a link with a key in the query params. That key logs them in. They didn't have to think about anything, and no redirect to do what they wanted to do.

Invite people where they are.

I wanted this to be something you could use for your entire family or group. Not everyone has Facebook or any number of social accounts. But almost everyone has an email or can get texts.

You invite people not by a username we have, but their email or phone. In the background if we don't have them on file we will make a user for them. Like mentioned before we have links that auto log them in, but since we made a user and have no passwords, they can log in at anytime without it.

Usable over email

I didn't quite launch with everything I wanted for this part. We currently send out invites over email but I wanted to do more.

So coming soon the future, you will be able to email Juniper City to create your event without ever visiting our site.

The people in the to field of the email get invited. You never have to even copy and paste over emails this way. Your email client is now our fronted client.

The big goal is trying to make useful open tools.

The How

So how was it built?

So the stack for the site is:

Docker
Nodejs
Hapi
Postgres
Redis
Stimulusjs

The entire stack is one monolith. No micro-services and we use server side rendering. It makes it simple to program for and deploy.

Frontend

This is a really simple frontend and we use a really simple framework. I've talked about Stimulusjs in a lot in my other posts so I won't belabor the point.

We use ejs for server side templates. (Though I am thinking of switching to Nunjucks it just seems nicer).

The total JavaScript bundle in production is around 350 kb and 65 kb minified. That's tiny. We don't do a lot in the frontend so why bloat it?

All our frontend is doing is just a few AJAX requests as well as hiding and showing things. Stimulus really does help keep this organized.

Keeping the site server side rendered with minimal JavaScript give a big advantage. Once we respond from the server, we're serving up blazing fast pages. Taking advantage of Turbolinks as well to make it feel single page. Can you get similar results with React or Vue? Yes, but it's not as EASY.

From the styling end we use Tailwindcss it's just awesome. Worth checking out. Makes putting together designs an absolute breeze.

Tools

For bundling we use Parcel. I don't have a very strong opinion on this but parcel is very near it's promise of zero config.

In production to keep the CSS small (tailwind has a lot of unused css) we use PurgeCSS. This pushes our css to 7.5 kb and 2 kb gziped.

Server side.

The stack here is pretty standard and easy.

Nodejs as the application logic.
Postgres as our database.
Redis for job queuing.

We integrate with Postmark for email delivery. For SMS delivery we use Textbelt. Also note that it will only send texts in the US.

I found Textbelt after a large amount of searching.

Some other options I didn't like.

Twilio
- Wow is this complex. Do I have to buy a number? How can I JUST send a text
AWS SNS
- Simple and cheap but I was searching to get away from using AWS.

Queues

We make use of queues and task workers. Take a look at this thing I wrote for an in depth about it.

Anytime we want to send an email or a text, we put it into a queue. Incoming web hooks? Queues. In essence anything you don't want to do right now. Queue it baby! The great thing is our current system has great support for running things at a later date as well. Things like cron jobs fit this use case.

Deploying

We package the server and frontend code in a docker container (Docker is so great).

Right now we server all our assets from the server. For the future we'll likely move static assets (CSS, JS, images) to a CDN. It's good enough right now.

Hosting is on Digital Ocean. Amazon is kinda evil so I wanted to try something else. What Digital Ocean lacks in configuration and power it makes up for in simplicity.

If you couldn't tell simplicity is a core tenant of this entire project.

We Build the project using docker.
Upload the image to a registry.
Deploy with docker compose.
- I learned about this option in compose -H "ssh://user@server". As long as you have SSH access to your remote machine it's the same as running the commands on your own machine. In the past I would copy the env files and compose files over to the remote machine as part of the build step. Using this new method you don't have to do ANY of that.
- docker-compose -H "ssh://user@server" pull image
- docker-compose -H "ssh://user@server" up -d

Again really simple.

Conclusion

I hope you liked that overview of what Juniper City is. Maybe you can find a use for it!

Simplifying my old event bus architecture with Bull

Clay Murray — Wed, 09 Oct 2019 20:12:18 +0000

The why

So before I posted about an event bus with AWS SNS and SQS. I think it works really well and it's pretty cool. It did have some rough edges that bothered me though.

There was a lot of manual setup in production.
- There are scripts etc that can do this but I was having to make the queue and SNS topic and link them for every new queue I wanted to make
Local dev was messy
- There are fairly solid Docker images that imitate SNS and SQS but the setup in code felt very messy.
Our task worker code, while cool, felt fragile.

So how does bull help?

Task worker management:

They have a solid interface for registering and running task workers. Making it much less work on our end to handle that part of the code.

Keeping all the code contained:

All pieces of the flow exist in the same codebase. Queue creation, event passing and task running without external setup for production.

Less moving parts:

In the previous iteration you had SNS, SQS, custom task handler code, and server code. The updated version uses just server code, Bull, and Redis. While only one less system is used in the updated version. We reap many benefits from the battle tested, and frankly much better, Bull code.

Portability:

Our task code now only relies on Redis. This makes it much easier to move to another cloud provider. No code changes required.

The How

So it's pretty simple to implement. (You can refer to the Bull documentation for most of this as well.)

Setting up queues

We make a queue for each type of work we want to do. Such as emailing or sending texts.

const emailQueue = new Queue('email', 'redis://127.0.0.1:6379');
const smsQueue = new Queue('sms', 'redis://127.0.0.1:6379');

Then we have all our queues in an array so when an event comes in we can pass that to all our queues

const queues = [emailQueue, smsQueue];

function onEvent(name, data){
    queues.forEach((queue)=>{
            queue.add({
                name,
                data
            })
    })
}

And we can define our queue processors.

emailQueue.process(emailTaskWorker);

Pretty simple stuff.

Final thoughts.

Why pass events to all queues? It allows any any queue to decide what it wants to do.

For example. If a user commented on a post, we might want to email them, text them, or generate a push notification. If in our code we did:

server.post("/posts/:id/comment, (req) => {
    server.sendEmail(req)
    server.sendText(req)
    server.sendPush(req)
})

That would get really messy. Instead we just make a record of what event happened.

server.createEvent("user-commented-on-post", {post, user});

Now any queue that cares about that event can do whatever it wants in response. It keeps our http handler code very clean as well.

How do you do language translations

Clay Murray — Wed, 25 Sep 2019 02:13:00 +0000

Having worked at a company that supported a large number of languages I always wondered how other developers managed theirs.

The basic idea when I first started was: static JSON files with key value pairs.

So English might look like:

{
    "welcome": "Hello!"
}

Then in the code you would reference the key to fill in the value.

Something like:

<Translate>welcome</Translate>

That's not a bad solution and the basic ideas were pretty alright. However, it has a few problems.

The people who can translate for you don't usually know JSON
It doesn't provide much context on what the translation should be
It requires a code release every time you need to update a language or even fix a typo.

#1 is not too hard. You can convert to CSV for loading into excel which is a tool most people know. You can still run into data conversion issues.

#2 I never had a great solution for. Typically you would have to do a lot of work to provide screenshots and go back and forth for them to have proper context. If you site was already live it was easier since they could just look at the live site.

#3 Can be solved with hosting the JSON in another place such as S3 and have the app pull the data from there. But then you no longer have version control.

Eventually I created an internal app to solve #1 and #3. It was a react app that used firebase as the storage. It integrated with our LDAP so we had simple SSO.
We also had language and environment based permissions. You could say a user could only access French in dev.
It had a way to promote dev to production as well.

Firebase was nice to use in that you can have direct JSON access URLs so the consuming app never had to install their bulky tools. (It wasn't important to have live updating of the translations)
The app worked but was clunky, ugly, buggy and had little buy in from anyone but IT. I could think of many things I might improve on it.

This was a bit of a ramble. But after talking about what I've done and worked on. What do you guys do?

Parsing JSON

Clay Murray — Tue, 02 Jul 2019 17:36:13 +0000

Hello friends,

Last time I talked with you about parsing INI files. Now let's parse some JSON.

There is no good reason to do this in JavaScript since it's built in and much faster.
Well there is one good reason. Parsers are super fun. ANYWAY let's dive in.

Lexer

First thing we want to do is take the file string and turn it into an array of tokens.

//Make an enum it's just easier to type this way.
const _types = [
  "LEFT_BRACKET",
  "RIGHT_BRACKET",
  "LEFT_BRACE",
  "RIGHT_BRACE",
  "TRUE",
  "FALSE",
  "NULL",
  "COMMA",
  "STRING",
  "COLON",
  "QUOTE"
];

const TYPES = {};
_types.forEach((type, index)=>{
    TYPES[type] = index;
});
function getString(text, index){
  let str = "";
  let len = 0;
  for(let i = index + 1; i<text.length; i++){
    if(text[i] === `"`){
        return [str, len];
    } else if(text[i] === "\\" && text[i+1] === `"`){
        str += `"`;
      len +=2;
      i++;
    } else{
        len++;
        str += text[i];
    }
  }

  return null;
}


function lex(text) {
  let index = 0;
  const result = [];
  while(index < text.length){
    const at = text[index];
    switch(at){
    case "{":
        result.push({type: TYPES.LEFT_BRACE});
      index++;
      break;
    case "}":
        result.push({type: TYPES.RIGHT_BRACE});
      index++
      break;
    case ":":
        result.push({type: TYPES.COLON});
      index++;
      break;
    case ",":
        result.push({type: TYPES.COMMA});
      index++;
      break;
    case "n":
        if(text.substring(index, index+4) === "null"){
        index+=4
        result.push({type: TYPES.NULL});
      } else {
        throw new Error("unexpected character parsing null");
      } 
      break;
    case "f":
      if(text.substring(index, index+5) === "false"){
        index+=5
        result.push({type: TYPES.FALSE});
      } else {
        throw new Error("unexpected character parsing false");
      } 
      break;
    case "t":
        if(text.substring(index, index+4) === "true"){
        index+=4
        result.push({type: TYPES.TRUE});
      } else {
        console.log(text.substring(index, index+4));
        throw new Error("unexpected character parsing true");
      } 
      break;
    case "\"":
        const str = getString(text, index);
        if(str === null){
        throw new Error("Expected end of string");
      }
        result.push({type: TYPES.STRING, value: str[0]});
      index += str[1] + 2;
      break;
    case "[":
        result.push({type: TYPES.LEFT_BRACKET});
      index++
      break;
    case "]":
        result.push({type: TYPES.RIGHT_BRACKET});
      index++
      break;
    default: 
      index++ 
    }
  }
  return result
}

So it's pretty simple.
We are going over the string and when we encounter certain characters we turn that into a token and push it into an array.
We also validate strings and true, false, null

Tokenizing the string makes it REALLY simple to write the rest of the parser.

SIDE NOTE: I just realized this parser doesn't handle numbers. I'm going to claim I left that as an exercise for the reader. SO do that.

You'll also see we handle the case of escaping quotes. We could escape several other things. But that's way too much work. That's also an exercise for the reader...

Parsing

Now we get to have some fun parsing the tokens we have.

function parse(tokens, index){
    const t = tokens[index];

  switch(t.type){
    case TYPES.STRING:
        return [t.value, index+1];
    case TYPES.FALSE:
        return [false, index+1];
    case TYPES.TRUE:
        return [true, index+1];
    case TYPES.NULL:
        return [null, index+1];
    case TYPES.LEFT_BRACKET:
        return parse_array(tokens, index+1);
    case TYPES.LEFT_BRACE:
        return parse_obj(tokens, index+1);
    default:
     console.log(index, t, tokens)
        throw new Error("unexpected type " + Object.keys(TYPES)[t.type] + " " + t.type); 
  }

}

We don't loop over anything here. Because we will call ourselves recursively in parse_array and parse_obj.

Notice we return an array [value, nextIndex] because we want the value we parsed as well as the next index in the tokens array to look at next.

Parsing an array

function parse_array(tokens, index){
    const result = [];

  let idx = index;

  while(true){
    const token = tokens[idx];
    if(token.type === TYPES.RIGHT_BRACKET){
        return [result, idx+1]
    }

    const [value, newIdx] = parse(tokens, idx);

    result.push(value);
    idx = newIdx;

    if(tokens[idx].type === TYPES.RIGHT_BRACKET){
        return [result, idx+1]
    }

    if(tokens[idx].type !== TYPES.COMMA){
        throw new Error("Expected a comma");
    }
    idx++
  }

  return [result, idx]
}

So if we have an empty array the next token will be TYPES.RIGHT_BRACKET and we can return an empty array.

Else we are going to go through the tokens list until we hit a right bracket. Or an error.
Calling parse will give us back the value from the next token in the list. This is what is the real magic. If we have a nested array we will recursively call our own function again.

We have error checking for a comma between values or closing bracket if it's the last item.

Okay now how do we parse an object?

Parsing an object

function parse_obj(tokens, index){
    const result = {};
  let idx = index;
  while(true){
    const token = tokens[idx];
    if(token.type === TYPES.RIGHT_BRACE){
        return [result, idx+1];
    }

    if(token.type === TYPES.STRING){
        if(tokens[idx+1].type !== TYPES.COLON){
        throw new Error("Expected colon after key");
      }
      const [value, newIdx] = parse(tokens, idx+2);

      result[token.value] = value;
      idx = newIdx;
      if(tokens[idx].type === TYPES.RIGHT_BRACE){
        return [result, idx+1];
      }
      if(tokens[idx].type !== TYPES.COMMA){
        throw new Error("Expected a comma"); 
      }
    } else {
        throw new Error("expected key");
    }
    idx++;
  }

  return [result, idx];
}

As you can see it's really similar to an array. But we need a key value pair. We also check for there to be a colon : between key and value. ("key":"value"). Everything else is about the same as the array.

Wrapping it up.

Okay so what does parse return at the very end?

It will be an array with the first value being the object we parsed from JSON and the second should be the last index in the tokens array.

eg [{}, 2]

To call parse we need to pass it the result of our lex

const tokens = lex(inputString);

const value = parse(tokens, 0);

That's not the most user friendly so let's just wrap up this mess in a simple function that will just return the parsed object.

function parseJSON(text){
    const tokens = lex(inputString);
    const value = parse(tokens, 0);

    return value[0]
}

Easy!

Here is the full source as a JSFiddle: https://jsfiddle.net/pebj04zv/1/

I did have some help with this from an article written for python: http://notes.eatonphil.com/writing-a-simple-json-parser.html

checkout https://dropconfig.com to support me if you want.

Making An INI parser.

Clay Murray — Sat, 29 Jun 2019 23:10:28 +0000

I think writing code to parse stuff is fun.
Sometimes it's nice to get away from regex hell and make our own fun hell.

I want to note: I have no real idea what the best way to do any of this is. There's way smarter people than me that made much better parsers than me.
I'm doing this just because it's fun. And why can't we have a little fun once in a while?

A simple INI file

INI files are pretty simple. Let's parse one in javascript.
INIs are really just a vague concept so we can do whatever we want!

The file to parse

key=value
hello=world
;comment here
[section]
foo=bar

What we want our object to look like after parsing

{
  key: "value",
  hello: "world",
  section: {
    foo: "bar"
  }
}

We are going to go over the file character by character.
Depending on what state we are in we will do different things.

In the default state let's trim whitespace until we get to a comment, section or key.

function isWhiteSpace(at){
    if(at === "\t" || at === " "){
    return true;
  }
}


let state = "default";
while(index < file.length){
   let at = file[index];
  if(state === "default"){
     if(at === "\r" && file[index+1] === "\n"){
        index+=2;
    }else if(at === ";"){
        state = "comment";
    } else if(at === "["){
        state = "section";
    } else if(isWhiteSpace(at){
        //we have to negate the index++ so we don't eat the first character of the key
        index--;
        state = "key";  
    } 

    index++;
  }

Parse some keys

We want to keep track of the key name. And we will keep parsing for a key name until a character =. (We could allow it to be escaped eg \= but I don't want to.)

//...
let keyName = "";
while(/*...*/) {
// ...

    else if(state === "key"){
        if(at !== "\n" && at !== "="){
            keyName += at;
        } else if(at === "\n"){
            state = "default";
            keyName = "";
        } else {
            state = "value";
        }
        index++;
    }
}

So now we have a key. You may have notice we just ignore a line that doesn't have an = at all. You could throw an error or make it an empty string. But I don't want to.

Parse a value

The process for getting a value is much the same as getting a key. We just go until the end of the line. But we also want to set the key to the value when we get to the end of the line.

//...
const result = {};
//at first the section is just the root of result
let section = result;
let value = ""
while(/*...*/){
//... All that other code...

    else if(state = "value"){
        if(at !== "\n"){
          value += at;
        } else {
          //End of the line let's do this
          section[keyName] = value;
          keyName = "";
          state = "default";
        }
        index++;
    }
}

You'll notice how I use section instead of result there because we also ...

Parse a section

When we get to a section we need to nest. That is why we want a final result as well as a reference to the current section we are in

//...
let sectionName = "";
//while(){
//...other stuff
else if(state == "section") {
    if(at === "]" || at === "\n"){
        result[sectionName] = {};
        section = result[sectionName];
        sectionName = "";
        state = "default"
    } else {
        sectionName += at;
    }
    index++;
}
//}

So now we have sections for this. We have no end of section so it's the current section until a new one comes along.

Parsing comments

Comments are easy we just do nothing until the new line.

else if(state === "comment") {
    if(at === "\n"){
        state = "default"
    }
    index++
  }

The entire parser

Okay I broke it down to show you and so here's the entire thing.

const file = `
key=value
hello=world
;comment here
[section]
foo=bar
`

function isWhiteSpace(at){
    if(at === "\t" || at === " "){
    return true;
  }
}


let index = 0;
const result = {};
let section = result;
let state = "default";
let keyName = "";
let value = "";
let sectionName = "";
while(index < file.length){
    let at = file[index];
  if(state === "default"){
    if(at === ";"){
        state = "comment";
    } else if(at === "["){
        state = "section";

    } else if(!isWhiteSpace(at)){
        state = "key";
    index--;
    }
    index++;
  } else if(state === "key") {
    if(at !== "\n" && at !== "="){
        keyName += at;
    } else if(at === "\n"){
         state = "default";
       keyName = "";
    } else {
        state = "value";
    }
    index++;
  } else if(state === "value") {
    if(at !== "\n"){
        value += at;
    } else {
        //End of the line let's do this
      section[keyName] = value;
      keyName = "";
      value = "";
      state = "default";

    }
    index++;
  } else if(state == "section") {
    if(at === "]" || at === "\n"){
        result[sectionName] = {};
        section = result[sectionName];
        sectionName = "";
        state = "default"
    } else {
        sectionName += at;
    }
    index++;
  } else if(state === "comment") {
    if(at === "\n"){
        state = "default"
    }
    index++
  }

}

So parsing stuff can be fun. Try it out here. Give it some other input and see where it breaks. I bet I made a lot of stupid mistakes. But that's part of the fun.

Let me know if you have any questions and checkout out my project: DropConfig

Using Browser Custom Events

Clay Murray — Wed, 26 Jun 2019 18:35:08 +0000

A lot of times when writing things you may want want to react to certain events on your page.

We do this all the time with builtin ones like click or keydown

But we can also make our own custom events and have the browser handle all the work for us! Since it's part of the DOM API we get free event code without installing another lib or rolling our own buggy version.

CustomEvent is what we will be using. We'll wrap it a little to make it a bit neater to use as well.

Making a custom event

It's pretty simple

const event = new CustomEvent("myevent", {details: {some: "data"}});

document.dispatchEvent(event);

Notice we had to put our own custom data in the details key of the event. This is just a quirk of how they work.

Listening for a custom event

function eventHandler(event){
    const data = event.details.data;
    console.lo(data);
}
document.addEventListener("myevent", eventHandler)

Stopping listening

document.removeEventListener("myevent", eventHandler);

Pretty easy stuff.

What's great is, we can also dispatch the event on an element so it doesn't bubble up to the dom. Keeping our code event more modularized. Just replace document with another element you have.

A little wrapper.

Because it's a little bit cumbersome to have to write all that everytime you want to use an event. Let's wrap it just a little.

function publish(eventName, data){
    const event = new CustomEvent(eventName, {details: data});
    document.dispatchEvent(event);
}
const events = [];
function subscribe(eventName, cb){
    const wrapper =  function(event){
        cb(event.details);
    }
    document.addEventListener(eventName, wrapper);

    events.push({
        cb, wrapper, eventName
    })
}

function unsubscribe(eventName, cb){
    events.forEach((event)=>{
        if(event.eventName === eventName && cb === event.cb){
            document.removeEventListener(eventName, event.wrapper);
        }
    })

}

export {subscribe, unsubscribe, publish};

Usage

function eventHandler(data){
    console.log(data);
}
subscribe("myEvent", eventHandler))
publish("myEvent", {some: "data"});
unsubscribe("myEvent", eventHandler);

Et voilà

If you like my stuff please check out my site https://dropconfig.com

Using React with Stimulus JS

Clay Murray — Sat, 22 Jun 2019 02:48:49 +0000

I have written a few articles about how much I like Server Side Rendering (SSR) and using Stimulusjs.

But sometimes you need the power of React to help you. How can we include React when we need it without weighing down our website when we don't?

It's actually not too tough with built in code splitting in Webpack or parceljs. So let's go throught the basics.

Our Stimulus Controller

import {Controller} from "stimulus";

export default class extends Controller {
    //Load the React code when we initialize
    initialize(){
        this.reactComponentPromise = import("./reactComponent");
    }
    connect(){
        this.reactComponent = await this.reactComponentPromise;

        const reactRoot = this.targets.find("reactRoot");
        const props = {
            onChange: this.onChange.bind(this)
        }
        this.reactComponent.render(reactRoot, props);

    }
    onChange(){
        //Some sort of thing to do with the changes.
        //Hit the API?
        //IDK you decide
    }
    disconnect(){
        const reactRoot = this.targets.find("reactRoot");

        this.reactComponent.destroy(reactRoot);
    }
}

It's pretty simple. But now we can load a react component and enjoy code splitting to not weigh down our app!

What does this look like on the React Side?

The React Code

    import ReactDOM from "react-dom";

    import Something from "./Something";

    function render(node, props){
        ReactDOM.render(node, <Something {...props} />      
    }

    function destroy(node){
        ReactDOM.unmountComponentAtNode(node);
    }

    export {
        render,
        destroy
    }

This is really all there is to it.

Keep in mind as well that React is pretty smart. You can call render with new props from your Stimulus controller and it won't break anything.

The HTML

Finally the html to put it together. It's nothing major

<div data-controller="react-controller">
    <div data-target="react-controller.reactRoot"></div>
</div>

Conclusion

There is a great case for mixing technologies in your applications using the right too where you need it.

You can easily mix React where you need to do some heavy lifting and still keep your app mostly SSR for the rest of your pages with Stimulus.

If you have any questions let me know!

SPAs are way too much work

Clay Murray — Wed, 19 Jun 2019 18:08:41 +0000

There is a lot of beauty in a simple HTML + CSS application with a sprinkle of Javascript mixed in.

I'll cover some points I like about this system.

Speed

The HTML that gets sent to your user is the actual content of that page.

Consider Most SPAs

I load the page from the server.
I must download all the JS
I'm shown a loader while the JS fetches the data for the page
The virtual DOM is created and put into the Real DOM
The Browser renders the page

That's a lot of steps and it takes non negligible amount processing power to do. It can be a nightmare on mobile phones.

Consider SSR pages

I load the page from the server.
The browser renders the page
If done well, The javascript is loaded

In the SSR case we get the first render quicker and with much less work on the user's device.

With the added benefit of: The content we wanted the user to see is shown, regardless of our Javascript status.

Of course you can pre-render react and other SPA technologies but I'll get to that.

The Sprinkles

Now we can get to the fun part, the JavaScript! I love JavaScript. I love to write it. I love writing SPAs as well. However, we can rely on it a bit too much in some cases.

So how do you sprinkle in JavaScript? You could just use JQuery or plain JavaScript, but believe it or not there is at least one framework made for this: Stimulusjs. It's made to add functionality to the HTML you already have.

I won't go into details about it too much but it's an incredibly simple, and easy to learn framework. It also doesn't do too much. Basically giving you an easy way to attach to your HTML.

Now you can add your sprinkles. We use it for submitting forms and activating modals etc.

Better flow

If you have designers that know HTML and CSS they can create mockups in HTML instead of photoshop. What this allows is an easy path from designer HTML to HTML template.

Because now that your HTML and CSS are taken care of you don't need to convert it to React. The markup is done! You just have to basically annotate the HTML with your templating language.

If something needs to be changed, at the end of the day it's just HTML your designers can edit the files themselves for tweaks.

This is one of the big ways I think plain HTML beats Server Side Rendered React or Vue. (Although Vue is pretty neat).

Mobile apps

If you see how Basecamp creates their mobile apps SSR and plain HTML is a big key to moving fast with small teams.
You can load the plain HTML page in your app and augment it with Native components. It's way lighter weight and easier to update than React Native. Having used React Native I wouldn't actually wish it on anyway ;) As long as your pages are mobile optimized you have the foundation for a mobile app. You are always just using the same markup.

What do you think?

When do I use React?

There are places React makes sense. Things with heavy data and reactivity (hehe) can really benefit from it. An online image editor or a video site like YouTube and Hulu.

Instead of an entire site SPA we can augment parts of our site with React. Take our video site example. Maybe we only want to load React on the video page but still just SSR the other listing pages.

On the video page our JavaScript sprinkles can load in the React code. Code splitting in Webpack works great for this!

Here's an example using stimulus

//Video React Loader controller

export default class extends Controller {
    static targets = ["react"]
    intialize(){
        this.videoJS = import("./video.js");
    }
    connect(){
        const reactParent = this.reactTarget;

        this.videoJS.init(reactParent);
    }
    disconnect(){
        const reactParent = this.reactTarget;
        this.videoJS.destroy(reactParent);
    }
}

// Video.js
import VideoStuff from "./whatever"

function init(node){
    return ReactDom.render(<VideoStuff />, node);
}

Function destroy(node){
    return ReactDom.unmountComponentAtNode(node);
}

//video.html
<div data-controller="video">
    <div data-target="video.react"></div>
</div>

This way we only load React when we need to and can still SSR render all the other things we want.

What do you think?

A case for Turbolinks, Stimulus and React

Clay Murray — Thu, 06 Jun 2019 15:18:59 +0000

Single page apps are pretty cool. I like them a lot and they are fun to create.
However, they can get quite heavy in terms of code size, and loading time. You also have to jump through many hoops to get server side rendering. These things can be worth the time an effort, but not always.

I think there is a case to be made for a many tool solution.
That is:

Here's what are transitioning to at DropConfig.

We have a lot of pages that just list things. List your organizations, list your repos, list your configs. Why are we bringing the big guns to bear on this? Why bloat a simple listing page with as much react as possible? Previously I'd have to drag in React, and Redux as well as Material-UI. It's a lot of things for a simple listing page.
This is where Stimulus and Turbolinks come in handy.

Turbolinks

On our backend, in node, we use ejs to render listing pages.
The server just returns plain HTML. We then rely on Turbolinks to make the page loads feel more like an SPA. But we never send anything from our Server but plain HTML. This keeps our pages really light and they render fast!

There's a few big wins, in my opinion, in writing plain HTML templates

Making a simple HTML template on your server where the data already lives is far easier than doing it in React.
- No calls to fetch JSON and loading states.
- No Redux for state management etc.
- The same JSON REST APIs we make can we used to render the templates server side.
Better battery life and speed on mobile because we aren't running nearly the same amount of JavaScript.
If a user doesn't have JavaScript enabled they can still view parts of our site.

Stimulus

Stimulus is used for mundane operations on the DOM. Things like menu toggles, modals, etc. It's a really nice and simple too for such things. It's worth a look at least. Basecamp uses it. (And Turbolinks)

React

So where does React come in? Well we have a configuration editor. It has a lot of data and a lot of moving parts. That would be tough to do with Stimulus and Turbolinks so we rely on React to handle it for us. With the ease of code splitting in modern tools like webpack, the user doesnt have to download all that extra bulk until they want to edit a configuration.

In case you don't think there is any precedent for this, consider GitHub. This is more or less how their entire frontend functions. While I don't think they use stimulus and I know they use pjax instead of Turbolinks, it is the same sort of concepts.

Side note: Check out Tailwind CSS it fucking rocks. Maybe I'll write about it someday.

Thanks for listening to me ramble.

Feel free to ask me any questions. I know some things about some other things.

Check out DropConfig you might like it.

Simple blog with DropConfig

Clay Murray — Mon, 03 Jun 2019 17:09:47 +0000

DropConfig is very versatile and with some of the new features we have added it's become very easy to use it for a simple blog with no backend on your part!

I'll be using this codepen for the demo and you can check it out there.

Prerequisites

To do this proper you will want to have a DropConfig org. You can get a 90 day free trial.

After you create your org we will want to create a repo to hold all of our posts.

A template

We want to make a post template that we can easily copy from for new posts.

Make sure to click on Advanced Setup and check Create as Template

Then we'll populate the template with the fields we might like and save it.

Creating a new post

Now that we have a template, we can create a new post.

Create a new config in that same repo. You can name the document whatever you want. Then in Advanced Setup again we are going to select blog_template as our template.

We can see it copied the template fields we had. Now we can go into the Form-UI to edit it easier.

You might notice that the body field is not a textarea. Luckily we can change it to one with the Manage Form Fields button. Then clicking on change to textarea above the Body field.

Fill out the post to your content!

Getting all the posts.

Now we want to list out all the documents in our repo. This will allow us to show them on our homepage for our blog.

We can see our post in that list but the template doesn't also show up!

Note: Check out the final code at The CodePen

The data should look something like this:

Loading the data for the document.

Now that we have the list of documents that exist we need to get the document data.

We can access it from a fast CDN from its access_url.

However, if a document isn't published it won't be on the CDN. This allows you to make drafts that won't show up until you hit publish!

Also since we only get the latest publication you can make edits without them showing live. We also get a url we can link to direct if we decide to have separate pages for posts.

Displaying Posts!

Now for the part we care about. Showing our cool posts.

Final thoughts

As you can see it only takes a few lines of JS to make an entire blog. It's infinitely customizable for your own purposes. Add in react or vue or whatever you want!

Sending Emails With AWS SES

Clay Murray — Wed, 29 May 2019 17:54:25 +0000

So in my last post I mentioned using your event bus to send emails. I skimmed over the implementation though.
I wanted to write a follow up about how you can send emails with SES quite easily. It's how we send all our emails at DropConfig

SES

AWS SES stands for Simple Email Service. The name is fairly accurate.
I'm also not going to go into specifics about setting up your account. AWS has some good guides

Anyway let's get into it.

We break down our email sending into two parts triggered by events.

Fetching and marshaling the data for an email from a given event.
Sending the actual email.

For example.

We may get an event come in USER_COMMENTED_ON_POST first thing we do is lookup an email in our email triggers

const emailTriggers = {
    "USER_COMMENTED_ON_POST": {
        "to": "user.email",
        templateName: "user-commented"
    }
}

Our trigger has a few parts. We have a JSON path to look up in the event data for who to send the email to.
Then we have an email template which might look something like this.

{
    "Subject": "A user commented on your post",
    "Body": "hello {{user.name}} a user commented on your post {{post.link}}"
}

We use mustache for templates in our emails. Also you could make your email body have HTML tags etc.

So we look up in our email trigger list and put together and email

const trigger = emailTriggers[event.type];
if(!trigger){
   //We don't have a trigger so we don't care
   return true
}
//Here we are fetching the template. Maybe we store these in their own DropConfig?
const template = await loadTemplateFromName(trigger.templateName);

//Using lodash here because it can lookup by a path string
const to = _.get(task.data.data, trigger.to);
if(to && template){
    const body = mustache.render(template.Body, task.data.data);
    const subject = mustache.render(template.Subject, task.data.data);

    const params = {
        Destination: {
            toAddresses: [to]
        },
        Source: "sputnik@dropconfig.com",
        Message: {
            Body: {
                Html: {
                    Data: body
                }
            },
            Subject: {
                Data: subject
            }
        }
    }
}

Next we do create a send-email event and put the params in there.

server.createEvent("send-email", params);

Why do we not just send the email?

The big reason is: Imagine you have many different emails to send based off of a single event. If a user comments on a post you might want to send an email to the owner of the post. But also a different email to other commenters. Now if sending to the owner succeeds but sending to the commenters fails we have a problem. Without splitting the sending of emails into two distinct events. We would re-run the event and notify the people we sent to successfully over and over (or as many times as we retry).

If we split it into two steps only the failed emails ever get retried.

Now that we have created the send-email event. It is a breeze to actually send the email.

  if(task.data.type === "send-email"){
    try {
      const params = task.data.data;
      const res = await ses.sendEmail(params).promise();
      return true;
    } catch(e){
      return false;
    }
  }

We don't need much more logic than that!

Putting it all together.

//This is a task runner as explained in my previous post.
exports.handler = async (task, queue, sqs, server) => {
    const emailTriggers = {
        "USER_COMMENTED_ON_POST": {
        "to": "user.email",
        templateName: "user-commented"
        }
    }

    if(task.data.type === "send-email"){
        try {
          const params = task.data.data;
          const res = await ses.sendEmail(params).promise();
          return true;
        } catch(e){
          return false;
        }
      }

    const trigger = emailTriggers[event.type];
    if(!trigger){
       //We don't have a trigger so we don't care
       return true
    }
    //Here we are fetching the template. Maybe we store these in their own DropConfig?
    const template = await loadTemplateFromName(trigger.templateName);

    //Using lodash here because it can lookup by a path string
    const to = _.get(task.data.data, trigger.to);
    if(to && template){
        const body = mustache.render(template.Body, task.data.data);
        const subject = mustache.render(template.Subject, task.data.data);

        const params = {
        Destination: {
            toAddresses: [to]
        },
        Source: "sputnik@dropconfig.com",
        Message: {
            Body: {
            Html: {
                Data: body
            }
            },
            Subject: {
            Data: subject
            }
        }
        }
    }

}

Thanks for reading this far. Checkout https://dropconfig.com for awesome version control and hosting of your config files.

Let me know if you have any questions. I might be able to help!

AWS SNS & SQS Event Bus

Clay Murray — Wed, 29 May 2019 04:19:51 +0000

Even when you have a monolithic architecture like we do at https://dropconfig.com you want to do things asynchronously and be able to respond to events.

For example if a user is created we may want to:

Update billing information
Send the User an email
Send the person who invited them an email

Doing this all in one function is fine but we can do better with an event bus.

SNS

AWS SNS is a service letting you pub sub messages. You can pass messages to HTTP endpoints or AWS lambdas or even SQS. It makes it easy to send a message to one place and have it passed a long to anyone who wants to listen.

SQS

AWS SQS is a dead simple queue. SNS can write messages to it with a click of a button in the AWS console. Then we can read from that queue in whatever fashion we want.

Events

First off, we want to start sending events to SNS so it can forward that message to all of our Queues.

I will be using nodejs for all the code examples

// We are assuming we already created a topic in the AWS console.
const sns = new aws.SNS({region: "us-east-1"});

const params = {
    Message: JSON.stringify({
        type: "some-event-type",
        data: {
            "some": "data"
        }
    }),
    TopicArn: SnsTopic //Create a topic in the SNS console and get the ARN for this.
}
sns.publish(params).promise() //We can asyn await or just do promise stuff here.

As you can see sending a message can be pretty easy.
We might want to set some MessageAttributes but we will cover that later.

Event Handlers

We are setting this up so that SNS publishes these to SQS. We also want to have a queue per task type that we want to run.

E.g. Billing tasks would have a queue separate from Email tasks.

You can set up AWS lambda to read from your queues and AWS will scale them up as needed.

But as mentioned before we want to keep most of our code in our node app so how do we read from SQS and react to events?

We first want something to manage a queue and run workers

//We have a max number of workers that can run at once
//We don't want to melt our server.
const MAX_WORKERS = 10;
let currentWorkers = 0;
async function createTaskRunner(sqs, queue, server) {
  let running = false;

  // we have a copy of this running async for each task-type
  async function start() {
    running = true;
    while (running) {
      // This is soft target, because we may reenter here
      // while async waiting on the tasks. In other words:
      // - workers == 10
      // - we are good, we poll the task
      // - polling takes longer than a second
      // - and workers == MAX_WORKERS
      // - we poll again
      //
      // So this will overshoot a little.
      if (current_workers <= MAX_WORKERS) {
        const task = await checkQueue(sqs, queue);
        if (task) {

          // this is run async here to allow
          // running multiple workers of the same type
          // in parallel
          runTask(task, queue, sqs, server);
        }
      }
      await wait(1000);
    }
  }

  return {
    start,
    stop: () => {
      running = false
    }
  }
}

This function will manage the queue and bring up workers to handle events that show up in a queue next we want to define checkQueue to check if there is even a task to run.

async function checkQueue(sqs, queue) {
  const params = {
    QueueUrl: queue.url,
    MaxNumberOfMessages: 1,
    //WaitTimeSeconds is important. 
    //The `await` will wait until it gets something from the queue or 20 seconds has passed before returning. 
    //This way we don't keep running over and over and over as fast as possible.
    WaitTimeSeconds: 20,
  }
  const res = await sqs.receiveMessage(params).promise();
  if (res.Messages && res.Messages.length) {
    const message = res.Messages[0];
    let messageBody;
    try {
      const data = JSON.parse(message.Body);
      messageBody = JSON.parse(data.Message);
    }
    catch (e) {
      messageBody = message.Body
    }

    const task = {
      id: message.MessageId,
      receipt: message.ReceiptHandle,
      queue: queue.url,
      data: messageBody,
      message: message
    }
    return task;
  } else {
    return null;
  }


}

Now let's check out runTask we put this in a separate function from createTaskRunner so we can have multiple workers on a queue at a time.

async function runTask(task, queue, sqs, server) {
  workers = workers + 1
  const taskSummary = {
    type: queue.type,
    id: task.id
  }

  try {
    const complete = await queue.handler(task, queue, sqs, server)
    if (complete) {

      await sqs.deleteMessage({
        QueueUrl: queue.url,
        ReceiptHandle: task.receipt
      }).promise();
    }
    else {
      //We even create events in our event handlers.
      server.createEvent(TASK_WORKER_FAILED, {
          taskSummary,
          complete
      });
    }
  } catch (e) {
    server.createEvent(TASK_WORKER_FAILED, {
        taskSummary,
        e
    });
  }
  workers = workers - 1

}

A task worker is defined as a stateless function that gets data from the runTask function does its thing and returns if it succeeded on that event or not.

Let's look at an example for sending an email.

exports.handler = (task, queue, sqs, server) => {
   let to;
   let message;

   // We can handle multiple types of events here.
   if(task.data.type === "USER_CREATED"){
        to = task.data.data.user.email;
        message = "WELCOME TO THE SITE!"
   }
   if(task.data.type === "USER_COMMENT"){
        to = task.data.data.post.creator.email;
        message = "SOMEONE COMMENTED ON YOUR POST";
   }

   if(to && message){
        //This is obviously simplified. 
        //Sending emails is not part of this article.
        sendEmail(to, message);

   }

   //If we get a event type we don't care about we just ignore it.
   return true;

}

So now we just have one last piece to put together. Initializing task queues.

const taslQueues = [
    {name: "email", "url": "https://url-to-email-queue.aws.amazon.com/", handler: require("./emailHandler")}
    {name: "billing", "url": "https://url-to-billing-queue.aws.amazon.com", handler: require("./billingHandler")}
]
async function init(sqs, server, tasks) {
  const runners = [];
  taskQueues.forEach(async (queue) => {
    const runner = await createTaskRunner(sqs, queue, server);
    runners.push(runner);
    runner.start();
  })
}

Now we have 2 task runners listening for messages in their respective queues.

Limiting messages to queues

Say you have a lot of messages getting passed around but your email queue only cares about a few types and ignores the rest.
Well luckily SNS has our backs here with the ability to filter messages coming to our queue https://docs.aws.amazon.com/sns/latest/dg/sns-message-filtering.html

We just have to use the MessageAttributes I mentioned earlier to accomplish this.

const sns = new aws.SNS({region: "us-east-1"});

const params = {
    Message: JSON.stringify({
        type: "some-event-type",
        data: {
            "some": "data"
        }
    }),

    // We add in a message attribute to filter on.
    MessageAttributes: {
        "event-type": {
            DataType: "String",
            StringValue: "some-event-type"
         }
    },
    TopicArn: SnsTopic //Create a topic in the SNS console and get the ARN for this.
}
sns.publish(params).promise() //We can asyn await or just do promise stuff here.

And the subscription filter policy for the subscription.
It will only receive events with the listed event types.

{
  "event-type": [
    "some-event-type"
  ]
}

So that's a more high level overview of how you can be passing messages around.

Other Benefits

Datalake

Now that we have an event bus we can store every event we receive into some document store (we just log to cloudwatch actually). Allowing you to query any event that happened and even replay them!

Activity feeds

Say we have some activity feed that shows when a post was commented on.

We could come up with queries for the database but that might start to get complex. Or if we store some of our events in a document store like mongodb we can just ask for events of type USER_COMMENT and give them back in order!

Incoming webhooks

At DropConfig we use stripe. Rather than have to handle the incoming webhooks stripe sends to us at the webhook http handler, we can create events from them. This puts them in the bus and perhaps our billing handler can take it from there.

It's really simple

server.route({
    path: "/api/webhooks/incoming/stripe",
    method: "POST",
    options: {
        handler: (req) => {
            server.createEvent("STRIPE_INCOMING_WEBHOOK", req.payload);
            return true
        }
    }
})

Now anyone who cares can listen for STRIPE_INCOMING_WEBHOOK events and react accordingly.

If you made it this far thanks for reading! Please check out https://dropconfig.com I think you'll like it.

Feel free to comment with any questions!