DEV Community: Jeremy Woertink

Getting Lucky with HTMX

Jeremy Woertink — Sat, 06 May 2023 16:11:06 +0000

A quick intro to HTMX

HTMX is a small, dependency-less javascript library that sprinkles some fancy bindings on to your normal markup without you needing to write javascript. It gives you the ability to have things like asynchronous data fetching and dynamic content updates.

I came across this post by Chris James that sounded right up my alley.

I was following right along, and things were sounding great, but then I read this line

I did not write any JavaScript.

and I was sold!

For me, many of my applications just need minimal javascript for things like search autocomplete, toggling the view of something, or just delay loading hefty content.

I decided to try this out in a Lucky app to see how quick can I get something to show... (hint: it was really quick)

Example code

To get started, you just need to include their package:

<script src="https://unpkg.com/htmx.org@1.3.3"></script>

Then somewhere on your page, you just use HTML attribute designated by their API to augment your markup.

This example taken from the docs site https://htmx.org/examples/click-to-load/

<tr id="replaceMe">
  <td colspan="3">
    <button class='btn' hx-get="/contacts/?page=2" 
                        hx-target="#replaceMe" 
                        hx-swap="outerHTML">
         Load More Agents... <img class="htmx-indicator" src="/img/bars.svg">
    </button>
  </td>
</tr>

This will call back to your server hitting the /contacts/?page=2 path which will return some HTML. Then it takes that markup and replaces the <tr id="replaceMe"> with the content that was fetched.

Using Lucky for the server

Lucky is a full-stack framework written in the Crystal programming language. One of the neat benefits of using Lucky with Crystal is the typesafety you get.

Lucky doesn't use standard HTML, but instead uses Crystal methods that generate HTML. This ensures that your markup is always rendered properly. No misspelling a tag name, or forgetting to close a tag. No passing in the wrong type and accidentally rendering something that looks like [object Object], or worse, rendering an empty string to the page.

Example code

This is an example of a simple HTML page in Lucky.

class Products::ShowPage < MainLayout
  needs product : Product

  def content
    # recognizable HTML tags
    div class: "flex flex-col p-4" do
      h1 product.title, class: "text-xl"
      # use of sharable components
      mount Products::Images, product: product
      # built-in methods for displaying text
      simple_format(product.description)
    end
  end
end

Lucky meets HTMX

Getting these two things together doesn't take too long. I will do a quick demonstration of an app where a user logs in, and on their "dashboard", some numbers are displayed from a timer. You can imagine that these numbers would be fed in to a chart or report that could update without the need of refreshing the browser.

At this point, if you're not familiar with Crystal or Lucky, just do your best to follow along. I'm skipping the setup/install process for brevity.

Check out the Lucky tutorial for later.

Generating the app

From the CLI, we will bootstrap the app.

> lucky init
Project name?: htmx_demo
API only or full support for HTML and Webpack? (api/full): full
Generate authentication? (y/n): y
> cd htmx_demo
> ./script/setup

Adding Report

This is a Report model that just has a count property. It'll be associated to a user.

> lucky gen.model Report

Next we will edit that new migration in db/migrations/ to add our new fields.

def migrate
  create table_for(Report) do
    primary_key id : Int64
    add_timestamps
    add count : Int32
    add_belongs_to user : User, on_delete: :cascade
  end
end

Now update the model in src/models/report.cr

class Report < BaseModel
  table do
    column count : Int32
    belongs_to user : User
  end
end

And lastly, update the User model in src/models/user.cr

table do
  column email : String
  column encrypted_password : String

  # Add this line
  has_many reports : Report
end

Migrate it the DB

> lucky db.migrate

Setting up views

First we will add HTMX to our layout in src/components/shared/layout_head.cr

# add this line to the `head` block
script src: "https://unpkg.com/htmx.org@1.3.3"

For the next part, we will be setting up the user's "dashboard" to render this "report". We will first generate a new component to use for this report.

> lucky gen.component Reports::List

Then update that component in src/components/reports/list.cr

class Reports::List < BaseComponent
  needs reports : Array(Report)

  def render
    div do
      reports.each do |rep|
        para rep.count
      end
    end
  end
end

Now we can update the user's dashboard on src/pages/me/show_page.cr

def content
  h1 "This is your profile"
  h3 "Email:  #{@current_user.email}"

  # the actual HTMX magic
  div hx_get: Stats::Index.path, hx_trigger: "every 10s"
end

Here we create a normal div that tells HTMX to do HTTP GET to the Stats::Index.path (which doesn't exist yet), and we tell it to do that "every 10s".

Setting up the actions

Now we need to define that Stats::Index so it knows where to fetch the data.

> lucky gen.action.browser Stats::Index

We can update that action in src/actions/stats/index.cr

class Stats::Index < BrowserAction
  get "/stats" do
    reports = ReportQuery.new.user_id(current_user.id)

    component Reports::List, reports: reports
  end
end

Wrap it up

That's actually all it takes for the MVP just to see something in action. We can now boot up the app, create an account, and check it out.

Boot the app lucky dev
Visit the app in browser http://localhost:3000
Click the "go to app" button
Create a new account and sign in
See that there's nothing to display

While you're on that page, go back to your terminal, and open a new terminal tab/window. We can jump in to the sql console and add some records manually to watch them show up on the page!

> lucky db.console

# insert into reports (user_id, count) values (1, 1);
# insert into reports (user_id, count) values (1, 9);
# insert into reports (user_id, count) values (1, 4);

I think you get the idea :D

Recap

Every 10 seconds, a call will be made back to the Stats::Index action which will return the Reports::List component markup. We do have an issue here where if someone tries to visit /stats, they'll get some broken markup. We can fix that by detecting HTMX calls.

get "/stats" do
  reports = ReportQuery.new.user_id(current_user.id)

  if request.headers["HX-Request"]? == "true"
    component Reports::List, reports: reports
   else
    html Stats::IndexPage, reports: reports
  end
end

Then we'd create that Stats::IndexPage to render the component

# src/pages/stats/index_page.cr
class Stats::IndexPage < MainLayout
  needs reports : Array(Report)

  def content
    mount Reports::List, reports: reports    
  end
end

Next, you'll notice that when the /me page first loads, there's no data... It takes 10 seconds before we see something. We can make some default data by telling the Me::ShowPage that it needs reports : Array(Report)

div hx_get: Stats::Index.path, hx_trigger: "every 10s" do
  mount Reports::List, reports: reports
end

There's still a lot left to explore with this, and obviously HTMX isn't going to cover every case when it comes to building an interactive site, but if you don't need a ton of interactivity, this is a beautiful solution!

Programming in the Adult Entertainment Industry III

Jeremy Woertink — Thu, 15 Sep 2022 15:53:43 +0000

NOTE The following post talks about the use of adult toys, and though the post only pertains to code, I figured I'd give you a heads up anyway.

The idea

Recently I was asked to add a feature to our site which integrates with the Lovense (adult-toy) API.

How this integrates with the site is through a live chat with a streamer. You as the viewer have the ability to send token tips. When you tip tokens, this makes the streamer's toy activate. It can be activated by strength of vibrations, length of time to vibrate, or in some cases, even control things like rotation. For example, 1 token may vibrate the toy at a low strength for 2 seconds, but 100 tokens may vibrate a high strength for 30 seconds.

A streamer asked if anytime a token was tipped, if the chat could display the strength and length of time. We were already displaying when a tip was made, and we knew what settings the toy had because the API sends this to us. It should be easy!

The setup

Testing the toy isn't that easy. There's quite a bit of setup that goes in to getting this working. You have two primary modes of setup:

The "dongle"
Through mobile

The dongle method requires an additional purchase of a USB-A key. Once you have this, you have to download and install an entire app pack from Lovense. This includes several apps and extensions:

The Lovense Browser. It's a Chromium browser with their extension already installed
Lovense connect. A desktop app for Windows and macOS to connect to the toy through bluetooth.
OBS extensions for displaying integrated animations on your live streams (optional extras)

With the Lovense connect open, you plug in the USB key, and it finds the toy and connects to it.

When using the mobile setup, you open up the browser extension (Chrome, Firefox, or Chromium based), then scan a QR code through the mobile app, and it sets everything up for you.

As a developer, when a streamer says "My toy won't connect", there's a lot that has to be considered.

Are you connected through dongle, or mobile?
If mobile, iOS or Android, and what version?
If dongle, what version of Lovense Connect desktop?
Is the toy charged?
What OS are you on?
What browser are you using?
What version is the browser extension?
Could this be an issue on our side from a recent update?

As you can see, it can be quite a complicated setup making it somewhat hard to test.

Some code

I'm using Crystal Lang on the back-end. One thing about Crystal is that it's statically typed, even for the JSON. (This is important to the story).

Setting up the front-end is fairly straight forward. Here's some javascript:

const camScript = document.createElement("script");
camScript.setAttribute(
  "src",
  "https://api.lovense.com/cam-extension/static/js-sdk/broadcast.js"
);

camScript.onload = () => {
  this.camExtension = new CamExtension("My Site", this.streamID);

  this.camExtension.on("ready", async (ce) => {
    this.updateDeviceSettings(await ce.getSettings());
  });

  this.camExtension.on("postMessage", (message) => {
    // ....        
  });

  this.camExtension.on("toyStatusChange", (toys) => {
    if (toys && toys.length) {
      let mainToy = toys.filter((t) => t.status === "on")[0];
      let status = mainToy ? mainToy.status : "off";

      if (status != this.deviceStatus) {
        this.deviceStatus = status;
        this.updateDeviceStatus(status);
      }
    }
  });

  this.camExtension.on("settingsChange", (data) => {
    this.updateDeviceSettings(data);
  });

NOTE: I'm using VueJS, and there's a lot of stuff going on, so this isn't a copy/paste example.

Basically how this works is, it adds a script that connects to the Lovense API. This assumes you have the Lovense browser extension installed on the browser running this code (i.e. the streamer). When the streamer updates their toy, a callback comes in and fires off an update to store those settings in the DB.

Here's where it gets tricky... The settings are a JSON object that has a levels key, and a special key. Here's an example of a default level:

{"levels":{
{"level1":{"max":9,"min":1,"time":5,"rLevel":0,"vLevel":3}
...
}}

Now, when parsing this with Crystal, it's really easy!

# parse the JSON
device_settings = JSON.parse(lovense_settings)

# iterate over the levels
device_settings["levels"].as_h.each do |name, info|
  name #=> "level1"
  info["min"].as_i #=> 1
  info["max"].as_i #=> 9
end

With access to the data, I just needed a bit more info. Reviewing the original request, what I want is:

userXYZ tipped 1 token. Low vibrations for 5 seconds.

I can see from the JSON that I have a time key which gives me the number of seconds. I can also see the vLevel (vibrate level) is set to 3 which is a "low" number. I was going to need a helper method for that:

private def toy_strength_level_type(level : Int32) : String
  case level
  when .<= 5
    "Low"
  when .<= 10
    "Medium"
  when .<= 15
    "High"
  when .<= 20
    "Ultra"
  else
    "Special"
  end
end

Finding the specific level looked a bit like this:

selected_level = device_settings["levels"].as_h.find do |name, info|
  info["min"].as_i <= token_count && info["max"].as_i >= token_count
end

Putting this together, I got:

"#{context.username} tipped #{pluralize(token_amount, "token")}. #{toy_strength_level_type(selected_level["vLevel"].as_i)} vibrations for #{pluralize(selected_level["time"].as_i, "second")}."

Problem 1

It turns out that when a streamer changes a setting, their extension saves the value as a String. This isn't an issue for dynamic languages like javascript, but with static languages (especially Crystal), this doesn't quite work. Basically, the settings now looked like this:

{"levels":{
{"level1":{"max":9,"min":"3","time":5,"rLevel":0,"vLevel":"5"}
...
}}

Notice min is "3" instead of 3, and vLevel is "5" instead of 5. How this affects Crystal is like this:

# runtime exception can't cast String to Int32
selected_level["min"].as_i

# we can't cast right to integer if it's a string
min = selected_level["min"].as_i? || selected_level["min"].as_s?.try(&.to_i)

but now min is nilable... we can't assume it'll always be an integer, and we can't assume that casting it to a string and converting to an integer will always work. So for now, I just default to 0. This also means we need to do it for max, and vLevel, and time, and rLevel...

min = selected_level["min"].as_i? || selected_level["min"].as_s?.try(&.to_i) || 0
max = selected_level["max"].as_i? || selected_level["max"].as_s?.try(&.to_i) || 0
time = selected_level["time"].as_i? || selected_level["time"].as_s?.try(&.to_i) || 0

yeah, you see where I'm going. It's not pretty. (and yes, I did abstract to a helper method... but still...)

Problem 2

I didn't notice this the first time (or second time) through the API, but max on the last level actually returns the value "infinity" lol. It took deploying to production before I realized that one.. oops!

# it's a string, so it casts, but it's not
# a number, so it doesn't convert.
# runtime exception
selected_level["max"].as_s?.try(&.to_i)

max = selected_level["max"].as_i?
if !max
  if str_val = selected_level["max"].as_s?
    # downcase just in case it comes through as Infinity
    if str_val.downcase == "infinity"
      # if I'm expecting a number, what do I default to?
    else
       str_val.to_i
    end
  end
end

Problem 3

Remember I mentioned there's 2 main keys, levels, and special? The levels are standard for the toy; buzz this hard for this long. The specials, however, are more like "Earthquake" which will buzz in a square wave or sine wave pattern, or "RandomTime" which could be between 1 and 40 seconds. There's several others including "Clear" which will clear out the entire queue blowing out any tips previous users have sent. Devious.... But here's the kicker... None of the specials have min or max. Their structure looks like this:

{"wave":{"time":30,"token":"160"},
"clear":{"time":0,"token":1000},
"twowaves":{"time":"0","token":""}

Now we have a whole new structure. I have to find the special that matches the token count exactly. Also notice that the token key can be a String, Integer, or an empty String. time can also be 0 which doesn't necessarily mean "0 seconds" because randomTime comes through as a time of 0. This also means I need to change how the message displays.

It'll look more like:

userXYZ tipped 160 tokens. Wave vibration for 30 seconds.

userXYZ tipped 1000 tokens. Clear vibration.

Problem 4

At this point, like most devs, we see a time to refactor. This code has become way too messy (and has tanked several production deploys due to not catching runtime edgecases...), so we can make it a lot cleaner.

Crystal comes with a Serializable module that allows you to create an object that receives some JSON, and normalizes all of the data in to easily callable objects.

I took the time, refactored, wrote specs, and smiled at how much nicer all of the code looked. My model had this change:

- column device_settings : JSON::Any = JSON::Any.new({} of String => JSON::Any)
+ column device_settings : LovenseSettingsSerializer = LovenseSettingsSerializer.new, serialize: true

Now with that field being an object I can throw methods on to, it looked more like:

# so clean!
selected_action = device_settings.match?(token_count)

selected_action.name #=> "Wave"
selected_action.time #=> 30

But why is this a problem? Well.... We're using Apollo GraphQL on the front-end. Apollo likes to fire off the mutations/queries (updateDeviceSettings in this case) when it sees that some value/variable has changed. We were checking to see if the settings we got back from Lovense were different from the settings we stored. If they were, then fire off this mutation to store the updated settings.

Guess what? Since we normalized the data, we stored all Integer values, but the data coming from Lovense was a mix.... This meant the values were always different. I'm sure you can see what this problem was. We were running this update basically as fast as the machine could process it in an infinite loop.

Problem 5

As it turns out, if the streamer goes in to their Lovense toy settings, there's a section called "Chat notifications". They have an option to turn on and customize. In the javascript API, there's a postMessage callback which receives a text message from Lovense with these events. When you receive these, you can send them in to your websocket channel or whatever you need.

Once that was figured out, we noticed we had 2 toy notifications. The one solved by all of this complicated code, and the one that Lovense actually does automatically for you with all of (and more) the same information....

The solution

Rip out all the code, undo all the commits, and tell the streamers to just enable it if they want through their settings.

The lesson

Read docs, explore, test test test, and remember that programming is hard.

Using GraphQL with Lucky

Jeremy Woertink — Wed, 07 Apr 2021 01:22:08 +0000

This is how to get GraphQL running with Lucky Framework.

Preface

I have a total of just 1 app that uses GraphQL under my belt, so I'm by no means an expert. Chances are, this setup is "bad" in terms of using GraphQL; however, it's working... so with that said, here's how I got it running.

Setup

We need to get our Lucky app setup first. We can use a quick shortcut and skip the wizard 😬

lucky init.custom lucky_graph
cd lucky_graph
# Edit your config/database.cr if you need

Before we run the setup script, we need to add our dependencies. We will add the GraphQL shard.

# shard.yml
dependencies:
  graphql:
    github: graphql-crystal/graphql
    branch: master

Ok, now we can run our ./script/setup to install our shards, setup the DB, and all that fun stuff. Do that now....

./script/setup

Then require the GraphQL shard require to your ./src/shards.cr

# ...
require "avram"
require "lucky"
# ...
require "graphql"

Lastly, before we go writing some code, let's generate our graph action.

lucky gen.action.api Api::Graphql::Index

This will generate a new action in your ./src/actions/api/graphql/index.cr.

Graph Action

We generated an "index" file, but GraphQL does POST requests... it's not quite "REST", but that's the whole point, right? 😅

Let's open up that new action file, and update to work our GraphQL.

# src/actions/api/graphql/index.cr
class Api::Graphql::Index < ApiAction
  # NOTE: This is only for a test. I'll come back to it later
  include Api::Auth::SkipRequireAuthToken
  param query : String

  post "/api/graphql" do
    send_text_response(
      schema.execute(query, variables, operation_name, Graph::Context.new(current_user?)),
      "application/json",
      200
    )
  end

  private def schema
    GraphQL::Schema.new(Graph::Queries.new, Graph::Mutations.new)
  end

  private def operation_name
    params.from_json["operationName"].as_s
  end

  private def variables
    params.from_json["variables"].as_h
  end
end

There's a few things going on here, so I'll break them down.

send_text_response

It's true Lucky has a json() response method, but that method takes an object and calls to_json on it. In our case, the schema.execute() will return a json string. So passing that in to json() would result in a super escaped json object string "{\"key\":\"val\"}". We can use send_text_response, and tell it to return a json content-type.

param query

When we make our GraphQL call from the front-end, our query will be the full formatted query (or mutation).

operation_name and variables

When you send the GraphQL POST from your client, it might look something like this:

{"operationName":"FeaturedPosts",
 "variables":{"limit":20},
 "query":"query FeaturedPosts($limit: Integer!) {
  posts(featured: true, limit: $limit) {
    title
    slug
    content
    publishedAt
  }
 }"
}

We can pull out the operationName, and the variables allowing the GraphQL shard to do some magic behind the scenes.

A few extra classes

We have a few calls to some classes that don't exist, yet. We will need to add these next.

Graph::Context - A class that will contain access to our current_user
Graph::Queries - A class where we will define what our graphql queries will do
Graph::Mutations - A class where we will define what our graphql mutations will do

Graph objects

In GraphQL, you'll have all kinds of different objects to interact with. It's really its own mini little framework. You might have input objects, outcome objects, or possibly breaking your logic out in to mini bits. We can put all of this in to a new src/graph/ directory.

mkdir ./src/graph

Then make sure to require the new graph/ directory in ./src/app.cr.

# ./src/app.cr
require "./shards"

# ...
require "./app_database"
require "./models/base_model"
# ...
require "./serializers/base_serializer"
require "./serializers/**"

# This should go here
# After your Models, Operations, Queries, Serializers
# but before Actions, Pages, Components, etc...
require "./graph/*"

# ...
require "./actions/**"
# ...
require "./app_server"

Next we will create all of the new Graph objects we will be using.

Graph::Context

Create a new file in ./src/graph/context.cr

# src/graph/context.cr
class Graph::Context < GraphQL::Context
  property current_user : User?

  def initialize(@current_user)
  end
end

Graph::Queries

The Graph::Queries object should contain methods that fetch data from the database. Generally these will use a Query object from your ./src/queries/ directory, or just piggy back off the current_user object as needed.

Create a new file in ./src/graph/queries.cr

# src/graph/queries.cr
@[GraphQL::Object]
class Graph::Queries
  include GraphQL::ObjectType
  include GraphQL::QueryType

  @[GraphQL::Field]
  def me(context : Graph::Context) : UserSerializer?
    if user = context.current_user
      UserSerializer.new(user)
    end
  end
end

This query object starts with a single method me which will return a serialized version of the current_user if there is a current_user. You'll notice all of the annotations. This GraphQL shard LOVES the annotations 😂

For our queries to return a Lucky::Serializer object like UserSerializer, we'll need to update it and tell it that it's a GraphQL object.

Open up ./src/serializers/user_serializer.cr

# src/serializers/user_serializer.cr
+ @[GraphQL::Object]
  class UserSerializer < BaseSerializer
+   include GraphQL::ObjectType

    def initialize(@user : User)
    end

    def render
-     {email: @user.email}
    end

+   @[GraphQL::Field]
+   def email : String
+     @user.email
+   end
  end

That include could probably go in your BaseSerializer if you wanted.

Graph::Mutations

The Graph::Mutations object should contain methods that mutate the data (i.e. create, update, destroy). Generally these will call to your Operation objects from your ./src/operations/ directory.

Create a new file in ./src/graph/mutations.cr

# src/graph/mutations.cr
@[GraphQL::Object]
class Graph::Mutations
  include GraphQL::ObjectType
  include GraphQL::MutationType

  @[GraphQL::Field]
  def login(email : String, password : String) : MutationOutcome
    outcome = MutationOutcome.new(success: false)

    SignInUser.run(
      email: email,
      password: password
    ) do |operation, authenticated_user|
      if authenticated_user
        outcome.success = true
      else
        outcome.errors = operation.errors.to_json
      end
    end

    outcome
  end
end

Notice the MutationOutcome object here. We haven't created this yet, or mentioned it. The GraphQL shard requires that all of the methods have a return type signature, and that type has to be some supported object. This is just an example of what you could do, but really, the return object is up to you. You can have it return a UserSerializer? as well if you wanted.

MutationOutcome

The idea here is that we have some sort of generic object. It has two properties success : Bool and errors : String?.

Create this file in ./src/graph/outcomes/mutation_outcome.cr.

# src/graph/outcomes/mutation_outcome.cr
@[GraphQL::Object]
class MutationOutcome
  include GraphQL::ObjectType

  setter success : Bool = false
  setter errors : String?

  @[GraphQL::Field]
  def success : Bool
    @success
  end

  @[GraphQL::Field]
  def errors : String?
    @errors
  end
end

By putting this in a nested outcomes directory, we can organize other potential outcomes we might want to add. We will need to require this directory right before the rest of the graph.

# update src/app.cr
require "./graph/outcomes/*"
require "./graph/*"
# ...

Checking the code

Before we continue on the client side, let's make sure our app boots and everything is good. We'll need some data in our database to test that our client code works.

Boot the app lucky dev. There shouldn't be any compilation errors, but if there are, work through those, and I'll see you when you get back....

Back? Cool. Now that the app is booted, go to your /sign_up page, and create an account. For this test, just use the email test@test.com, and password password. We will update this /me page with some code to test that the graph works.

The Client

Now that the back-end is all setup, all we need to do is hook up the client side to actually make a call to the Graph.

For this code, I'm going to stick to very bare-bones. Everyone has their own preference as to how they want the client end configured, so I'll leave most of it up to you.

Add a button

Open up ./src/pages/me/show_page.cr, and add a button

# src/pages/me/show_page.cr
class Me::ShowPage < MainLayout
  def content
    h1 "This is your profile"
    h3 "Email:  #{@current_user.email}"

    # Add in this button
    button "Send Test", id: "test-btn"
    helpful_tips
  end
  # ...
end

Adding JS

We will add some setup code to ./src/js/app.js to get the client configured.

// src/js/app.js

require("@rails/ujs").start();
require("turbolinks").start();
// ...

const sendGraphQLTest = ()=> {
  fetch('/api/graphql', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Accept': 'application/json',
    },
    body: JSON.stringify({
      operationName: "Login",
      variables: {email: "test@test.com", password: "password"},
      query: `
        mutation Login($email: String!, $password: String!) {
          login(email: $email, password: $password) {
            success
            errors
          }
        }
      `
    })
  })
  .then(r => r.json())
  .then(data => console.log('data returned:', data));
}


document.addEventListener("turbolinks:load", ()=> {
  const btn = document.querySelector("#test-btn");
  btn.addEventListener("click", sendGraphQLTest);
})

Save that, head over to your browser and click the button. In your JS console, you should see an output showing data.login.success is true!

Next Steps

Ok, we officially have a client side JS calling some GraphQL back in to Lucky. Obviously the client code isn't flexible, and chances are you're going to use something like Apollo anyway.

Before you go complicating the front-end, give this challenge a try:

Remove the include Api::Auth::SkipRequireAuthToken from your Api::Graphql::Index action.
Try to make a query call to me.

query Me {
  me {
    email
  }
}

Notice how you get an error telling you you're unauthorized.

Update the MutationOutcome to include a token : String? property
Set the token property to outcome.token = UserAuthToken.generate(authenticated_user).
Take the outcome token, and pass that back to make an authenticated call to the query Me.

Final thoughts

It's a ton of boilerplate, and setup... I get that, and I also think we can make it a lot better. If you have some ideas on making the Lucky / GraphQL connection better, or you see anything in this tutorial that doesn't quite follow a true graph flow, let me know! Come hop in to the Lucky Discord and we can chat more on how to take this to the next level.

UPDATE: It was brought up to me that the Serializer objects should probably move to Graph Type objects. With the serializers, the render method is required to be defined, but if you don't have a separate API outside of GraphQL, then that render method will never be called. You can remove the inheritence, and the render method, and it should all still work!

Why is X slow?

Jeremy Woertink — Fri, 04 Sep 2020 18:46:17 +0000

Blazing fast

If you program in the JavaScript world, then you're probably really familiar (and maybe sick of) seeing packages say "blazing fast". At some point, we developers decided that we wanted to use dynamic languages like Ruby, Python, PHP, JS, etc... but we were still behind in speed world when compared to compiled languages.

This point started becoming the obsession of many devs. You started seeing a project come up, then another project that did the exact same thing, but it was "faster". In the ruby world we saw this with the faker gem and the ffaker gem which stood for "faster faker". In the JavaScript world, we saw this with Node and IOjs... a fork of node that was being designed to be faster (among other political reasons).

Doing this is sort of messed up, but understandable in some cases. On the one hand, why not just help improve the original project instead of just forking it and writing it your own way? On the other hand, many people do submit PRs to make the original faster, but the owner says something like "I don't like this implementation" or maybe there's extra red-tape in the way. That ends up sparking the "fine, I'll rename my fork and start a new project" which creates a subdivision in the community.

Is having more speed bad?

No, not at all. This is a great thing. But is it the only thing? No. I think what a lot of developers tend to forget about is that there's a lot more that a project can provide than just "speed". For example, what if you had two identical projects, but one benchmarked twice as fast in response time, but the other benchmarked at a quarter the memory consumption? Which project do you choose? Or what about a project that benchmarks at 30% faster, 30% less memory, but the other project has better and easier to understand documentation with a friendlier community?

Lastly, what about that benchmark you're using? What exactly is it benchmarking? Are you benchmarking an app running locally in a Docker container on your 2015 MBP, but your application will run behind a CDN on cloud network? The numbers you get might be a decent gauge to help you filter out what you may or may not want to use. For example, you may benchmark 2 different ways to write a method, and see how much faster one is over the other. Then if you get a 0.5% speed increase, but the implementation is not as readable, is it worth the update?

Benchmark considerations

It's a lot of work to setup a real good benchmark. There's just so many things to consider if you're trying to really compare "apples to apples". Before you blindly look at a benchmark, and make up your mind based on what it says, let's consider a few things first:

What are you looking for initially? Are you currently using something that could be improved by using something else? Is this research for a new project?
How does it compare in speed vs the others? This is usually where we start anyway.
How does it compare in memory consumption vs the others? One project might require 4GB RAM minimum before needing to scale, where another might only need 1GB RAM.
How well is the codebase written? If you can quickly glance through the source, and understand how things are organized, and what things are doing, this may help you contribute to the project faster.
Does the community align with your beliefs? Yes, this is not software related, but it's really important benchmark. Do you feel welcomed, or is it hard to get help? This can make or break your project deadlines within your company.
How is the documentation? Is it all just auto-generated? Is there an easily navigable website? Is it up-to-date? Writing docs is tough, and keeping them up-to-date is tougher, but you don't have time to read the whole codebase just to figure out that the method you need requires 2 arguments.
What other benefits do these projects offer? They might have specific goals, or features like "human readable error messages", or built-in development tooling.
How often is the project updated? It's great if some project is actually "blazing fast", but if it's only updated every few months, what happens when your company runs in to a bug, and you reported it 2 months ago with no fix in sight?

Final thoughts

These are all of the main things I look at when considering some project. Don't shoot down a project just because you read 1 benchmark or reddit post that said "Why is X slow?". If you are curious why a specific project is slow, you need to first ask yourself, "what do I consider slow?". Next, give it a shot anyway and run your own personal test. You may just find out that you've had the best pizza of your life that one person on yelp rated 1 star... oh, I mean code... whatever, I'm hungry, it's lunch time.

Render Component from Action in Lucky

Jeremy Woertink — Wed, 30 Oct 2019 18:43:43 +0000

EDIT: Since this writing, Lucky now has a component method (built-in) you can use in your actions. No need to create a custom macro!

For one of my Lucky apps, I'm using Stimulus which is light-weight. I wanted to avoid doing a ton on the client side so that way I can keep all the nice benefits of Crystal and Lucky.

In a recent instance, I had to load some comments on to a page asynchronously. Normally in this case, you might write some javascript to make an api call, then gather all of your records in a giant JSON object. Then you'd probably iterate over the result, and create some sort of component. Maybe in Vue, or in React, and have those render each comment. With Stimulus, you don't have built-int templating. You're left with writing a lot of document.createElement, and element.setAttribute type stuff. Or you just use lots of innerHTML = '' type calls.

What I wanted was to write my markup in Lucky, then make an API call that returns the markup, and I can just shove that in to an element and be done with it.

By default, the actions in Lucky want you to render an HTML Page. But I didn't want to make a blank layout just for these. So here's what I did:

# src/components/comments_list.cr
class CommentsList < BaseComponent
  needs save_comment : SaveComment
  needs comments : CommentQuery

  # Renders the comments form, and each comment
  def render
    # This could also move to it's own component!
    form_for Comments::Create do
      textarea save_comment.body, class: "field"
      submit "Post Comment"
    end

    div class: "comments" do
      @comments.each do |comment|
        mount Comment.new(comment)
      end
    end
  end
end

Now I just need a handy helper macro in my ApiAction.

# src/actions/api_action.cr
abstract class ApiAction < Lucky::Action
  accepted_formats [:json], default: :json

  macro render_component(component_class, **assigns)
    send_text_response({{ component_class }}.new(
      {% for key, value in assigns %}
        {{ key }}: {{ value }},
      {% end %}
    ).render_to_string, "text/html")
  end
end

Then I can use this new macro in my api actions!

# src/actions/api/comments/index.cr
class Api::Comments::Index < ApiAction
  get "/api/comments" do
    save_comment = SaveComment.new
    comments = CommentQuery.all

    render_component CommentList, save_comment: save_comment, comments: comments
  end
end

Lastly, my javascript is pretty simple now.

import { Controller } from 'stimulus'

export default class extends Controller {

  connect() {
    fetch("/api/comments")
      .then(res => res.text())
      .then(html => element.innerHTML = html)
  }
}

Thoughts

I know I left out several bits like, setting up stimulus, or what the main action / page look like, but all of those should be fairly straight forward. I'll probably do a writeup later on integrating stimulus with Lucky. Hopefully these examples should get you close should you need to use this!

Using custom IDs in Lucky

Jeremy Woertink — Fri, 26 Jul 2019 21:01:56 +0000

If you're using Lucky, and you have a schema that's not default, you may need to create a custom primary key.

I had to do this for an app where my primary keys were defined in postgres like character varying(18). Just doing String wasn't going to be good enough since Avram defines those as just text. Here's how I did it.

# src/models/concerns/custom_id.cr
alias CustomID = String

module Avram::Migrator::Columns::PrimaryKeys
  class CustomIDPrimaryKey < Avram::Migrator::Columns::PrimaryKeys::Base
    def initialize(@name)
    end

    def column_type : String
      "character varying(18)"
    end
  end
end

module Avram::Migrator::Columns
  class CustomIDColumn < Avram::Migrator::Columns::Base
    @default : String? = nil

    def initialize(@name, @nilable, @default)
    end

    def column_type
      "character varying(18)"
    end
  end
end

Then in my model, I can now do this:

# src/models/user.cr
class User < BaseModel
  skip_default_columns
  table do
    primary_key id : CustomID
    column created_on : Time
    column updated_on : Time
  end
end

Deploying Lucky apps to Elastic Beanstalk

Jeremy Woertink — Sat, 29 Jun 2019 17:09:58 +0000

This isn't the prettiest solution, but it's what worked for us. If you have ideas on how to clean this up, I'm all for suggestions.

The setup

Developing locally on macOS
Production using Elastic Beanstalk container
EB container uses Docker

The concept

We had issues getting Crystal's cross compile to work properly, so we had to use a couple extra steps.

Boot Docker locally to compile release build of app
Move release binary, and required files to temp directory
Zip up temp directory to app.zip as per EB deployment instructions
Use eb command locally to push app.zip to EB container

If we happen to get cross-compilation working, we could remove the first two steps in this process.

The code

There's a few files we need to create. I'll define those first, then show the code.

./script/deploy. Be sure to chmod +x ./script/deploy
./docker/BuildDockerfile
./docker/docker_run_build.sh
./docker/Dockerfile
./docker/Dockerrun.aws.json

Deploy script

Used to actually push the code to production.

#!/bin/bash

set -e

# don't execute next commands on error
trap 'exit' ERR

# let echo interpret escape chars (\n)
shopt -s xpg_echo

# 
START=`date +%s`
DATE=`date '+%Y%m%d@%H%M%S'`
BUILD_DIR=build-temp-${DATE}
SHA1=`git rev-parse HEAD`
RANDOM=`awk -v min=5 -v max=1000000 'BEGIN{srand(); print int(min+rand()*(max-min+1))}'`

# deploy to production when on master branch
# deploy to staging when on other branches
branch=$(git branch | sed -n -e 's/^\* \(.*\)/\1/p')
if [ "$branch" == "master" ]
then
  STAGE=production
else
  STAGE=staging
fi
LABEL=$STAGE-$SHA1-$RANDOM-$DATE

# Remove the old build
rm -rf build/app

# Build assets
echo "Building Assets"
rm -rf public/assets/* public/mix-manifest.json
yarn prod

# Build the application
echo "Starting Docker"
docker build -t lucky-app-build -f docker/BuildDockerfile .
docker run -v $(pwd)/build:/app/build lucky-app-build

# Create a temporary directory to stage the files
mkdir ${BUILD_DIR}

# Stage the files
cp .env.${STAGE} ${BUILD_DIR}/.env
cp build/app ${BUILD_DIR}
cp -r public ${BUILD_DIR}
cp docker/Dockerfile ${BUILD_DIR}
cp docker/Dockerrun.aws.json ${BUILD_DIR}
cp -r .ebextensions ${BUILD_DIR}

# Getting them zipped up
cd ${BUILD_DIR}; zip -Xr app.zip * .env .ebextensions; mv app.zip ../; cd ..

# Deploy the application
eb deploy --label $LABEL

# Cleanup
rm -rf ${BUILD_DIR} app.zip

END=`date +%s`

echo Deploy ended with success! Time elapsed: $((END-START)) seconds

BuildDockerfile

Used for building the release binary

FROM crystallang/crystal:0.29.0

ADD . /app
ADD ./docker/docker_run_build.sh /app/docker_run_build.sh

WORKDIR /app

RUN shards update && \
    rm -rf /app/build/app && \
    crystal build src/start_server.cr --release -o app

RUN chmod +x docker_run_build.sh

CMD ["./docker_run_build.sh"]

docker_run_build

I honestly don't know why we have this, but it's here, so here it is.

#!/bin/sh

cp app /app/build/

Dockerfile

The actual Dockerfile used in production on EB

FROM phusion/baseimage

ENV APP_DOMAIN=localhost
ENV SECRET_KEY_BASE=abc123abc123
ENV LUCKY_ENV=production
ENV PORT=8000
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get -q update && \
    apt-get -qy install build-essential libgc-dev libssl-dev libxml2-dev libyaml-dev libevent-dev && \
    apt-get -y install tzdata && \
    apt-get -y autoremove && \
    apt-get -y clean && \
    rm -rf /var/lib/apt/lists/* && \
    rm -rf /tmp/*

RUN mkdir /app

ADD ./app /app
ADD ./.env /app/.env
ADD ./public /app/public

WORKDIR /app

EXPOSE 8000

CMD trap exit TERM; ./app & wait

Dockerrun

This file is used by EB. Learn more on Single Container Docker for AWS EB.

{
  "AWSEBDockerrunVersion": "1",
  "Logging": "/var/log/app.log",
  "Ports": [
    {
      "ContainerPort": "8000"
    }
  ]
}

Final Notes

You'll want to make sure you update your .gitignore file with things like:

/build*
*.zip
.elasticbeanstalk/*
!.elasticbeanstalk/*.cfg.yml
!.elasticbeanstalk/*.global.yml

And lastly, this all assumes you have your elastic beanstalk setup. That would require getting the eb binary installed locally, getting your container setup, and adding in all your config stuff.

Do you really want to learn how to be a programmer?

Jeremy Woertink — Wed, 06 Mar 2019 00:10:48 +0000

So you want to change careers?

You hate your current job/career... You don't make much money, and it's not satisfying. You heard that learning to code is the future, and you can make great money. But is this really the career you want to choose?

I want to talk about what it's like to be a programmer. It's easy to learn how to write code. puts "This is code"; That's a full valid ruby program. That is literally code you might write at some point. Writing code doesn't make you a programmer though. I know plenty of people that know how to write code, but they're not programmers. Being a programmer requires a specific mindset, and hopefully this post will help you to decide if you have that specific mindset or not.

Let's start with an analogy, shall we? What would you consider to be the difference between a "cook" and a "chef"? Well, for me, a cook can follow a recipe and prepare a meal. A chef can create a recipe. How about a classical musician VS a jazz musician? They both can read sheet music, but a jazz musician can think on the fly to create music. This thought process is what separates the two.

Let's look at code

Writing code is similar to learning to speak a new language. There's plenty of spoken languages out there from Spanish, to German, or French, Swahili, Mandarin, and many others. In programming we have Ruby, Javascript, PHP, Python, C++, Rust, Crystal, and a million more. Each language is a tool that aims to have a very specific usage with pros and cons for each. We don't have to know all of them, we only need to the ones we want to use to complete a specific task.

So what does this look like?

# Basic variable assignment in Ruby
a = 1
b = 2
c = a + b

// Basic variable assignment in Javascript
let a = 1
let b = 2
let c = a + b

// Basic variable assignment in Rust
let a: i32 = 1;
let b: i32 = 2;
let c = a + b;

// Basic variable assignment in PHP
$a = 1;
$b = 2;
$c = $a + $b;

As you can see from these examples, we are just doing a little basic math in a few different languages. If you stare at each of these examples close enough like you're looking at one of those Highlights Magazine image difference pictures, you'll see a few similarities, and a few differences between them.

So the first question we must answer is, would you be ok with looking at this code every day? If your first reaction is "that seems really boring", then you'd be right! It is really boring. But staring at code isn't what make being a programmer fun, it's the problem solving. Though, you do stare at a LOT of code, so you have to be ready for that.

Problem solving 101

Let's think of something you know how to do now. Maybe shuffling some playing cards? The problem is that I don't know how to shuffle a deck of cards, so your goal is to teach me. There's a few "gotchas" though. You have to do it over the phone, and you can't see what I'm doing, and I can't see you. I've also never seen a deck of cards shuffled in my life, so I have no point of reference as to what it should even look like. All I know is that I've been tasked with shuffling a deck of cards, and I need you to teach me how.

Where do we begin? Take a moment to think through the steps of how you shuffle cards. There's several different ways that people do it, but how do you do it? Picture your hand movements. Picture the placement of the cards. Now explain to me step by step how to shuffle.

Now let's take this one step further. Imagine we're in this situation, but we don't speak the same language. You have to explain to a translator, then the translator explains to me. I report back to the translator who then tells you what I said. Think of how much longer this will take to teach me now. Remember, we're doing this over the phone, so you can't just show me. This is problem solving for a programmer. Computers only understand 2 things: 0 and 1. That's it, binary. We have to write code similar to what's above and pass that to a translator that in turn translates to the computer. Sometimes you'll say something to the translator with one thing in mind, but the translator interprets it a specific way doing what you said and not what you meant. The computer gets these instructions, and now you have a bug in your app because it's doing something different than what you meant for it to do.

Here's an exercise for you to try on your own. If I have a song to be processed, and it costs $0.03 per minute but only if I go over 100 minutes otherwise it's $0.04 per minute plus $1 for each song. Then I have to process an album of 12 songs which is about 2 hours of music. How much would it cost me to process the music? Don't use a calculator to figure this out. Instead, write out the formula. I should be able to replace the length of the songs and the number of songs to calculate what my next album or someone else's album may cost.

That is called writing an algorithm inside of a function. Did you have fun doing that?

Problem solving 102

Let's imagine we're talking about being a chef, and not a programmer. You need to make a sandwich. You might think to yourself "that shouldn't be too hard". Well, this is what every non-programmer thinks when they come to me asking me to build them a website or app. Yeah, you'd like to think it wouldn't be too difficult, but let's talk about this sandwich.

We're going to make a smoked ham, turkey, and smoked cheddar sandwich on an italian roll with lettuce, tomato, onion, pickle, olive, spicy peppers, vinegar, oil, salt, and oregano. Sounds pretty delicious, right? What, you don't want the peppers on yours? Ok, fine, we'll make a second one that doesn't have that on there for you. Oh, but someone else doesn't want the onion or pickles? Hmm... So I guess we just make a third one like that? Oh great, now someone else wants to add black pepper, and no olives. At this point we have an unknown number of permutations of sandwich combinations, and we haven't even started prep!

Before we start making the sandwich, we have to weigh our options for ingredients. Do we save money and make our own bread? Or do we buy bread to save time? If we make our own, we can really customize it to taste exactly how we want, but it will take a bit more work. If we buy it from a bakery, we are at the mercy of that bakery. What happens if they mess up the bread? What happens if they go out of business? What happens if they run out of the bread we use? What happens if their cost goes up?

Now we need to do this for our meats and cheese. Do we buy cheddar, and ham, then smoke them ourselves? Or buy them pre-smoked? Have you ever made ham? It doesn't fall off the pig ready to eat, you have to cure it.

The duration of the curing process varies by the type of ham, with, for example, Serrano ham curing in 9–12 months (https://en.m.wikipedia.org/wiki/Ham)

If we decide to go this route, we won't have our sandwich for the next year.

If you're following along still, you can see that this 3 minute sandwich could potentially be a year long process, or it could be really expensive. This is not far off from programming. There's so much more that goes in to writing code and building an app. I often hear "well, can't you just use someone else's code? You can't be the first to do this." Well, yes, I can in fact. That is, provided the person that wrote that code has given it a license that allows me to use it. I know we're programmers, but now we have to consider the law. If they wrote the code for all to use for free, can I reuse it and sell it for money legally? Maybe I have to pay them to use their code? Can you afford that cost?

Clearing up some misconceptions

It doesn't take long to build an app.

We need to first determine the type of app. It could take a week, or it could take a year. I have no idea without understanding the entire scope of the project, and the amount of money the project has to spend.

A programmer can build an app alone.

Some can, yes. Of those "some", only some of those know how to design. Have you seen an app that looks ugly? Yeah, they probably didn't hire a designer.

Selling an app is easy.

No. It's not. Someone buying an app will want to get their money back in about 2 years or so. If you want to sell for $1 Million, you better hope your app makes near $500k/year. If not, then it has to be revolutionary technology, or have a few million active users on it. Keep in mind that the more users you have, the more expensive it is to run your app. Lyft is spending $8 Million a month to run their app. That doesn't include employee cost.

Working on a computer all day is easy.

Wrist pains, shoulder pains, back pains, fatigue mentally and eye strain. Have you ever laid in bed so long that you felt tired when you got up, and you felt sore? Our bodies aren't designed to stay in one spot for several hours a day unless we're sleeping. Is it easier than doing construction? Maybe, but then again, I'd say you get fit doing that work so when you get off work, you just relax. With being a programmer, you get off work and need to go be active like hitting the gym and stuff otherwise your health drops real quick.

Programmers make a lot of money

In most cases, yes. However, this isn't always the case. My first programming job was $12/hr. I was coming from a min wage job ($7.50/hr), so this was a bump for me. My next job was $15/hr. I was stoked to have a pay raise. After a few years, I was at a job making $55k/yr but then finding out that other companies were starting their devs out at $50k/yr. People without a degree and less time working were making basically what I was making.

Now we have devs in the silicon valley making anywhere between 45% equity while living out of cardboard boxes to a crazy 7 figure salary both doing the same job. Your amount will be based on so many factors. You may or may not make a lot of money. Sound like every other job?

Everything has been programmed already, just copy others code.

Sure, lots of code has already been written. That doesn't mean we have access to
that code. It also doesn't mean that it's been written in a language I understand. Just because google solved a problem, doesn't mean I can solve that same problem. They have tons of engineers with doctorate degrees in mathematics. I have a degree in playing tuba. This job involves a lot of researching, and digging around, and playing with different bits to see what works and what doesn't.

Becoming a programmer doesn't take much time.

There's a lot of people that go through a 3 month bootcamp and get a job as a jr dev right after. These are the people that have a very decent level knowledge of computers before starting the class. Then they worked their ass off and worked outside of class for those 3 months. These people are usually really inquisitive types that pick up on the concepts rather quick. For me, it took at least a year of doing this full time before things really started to click for me. Concepts don't come naturally to me like they do for some others, so I have to study, and do the rinse/repeat often before I really get something.

Let's Recap

Are you still interested in becoming a programmer? Let's go back over everything real quick to be sure.

You're going to have to learn a lot of technology, and it's a never ending battle of learning. What you learned last week may be out dated next week.
You'll stare at code most of the day, almost every day, and sometimes you won't understand it.
Being a problem solver is the key. You're just trying to solve everyone's problems. Your goal is to work your way out of a job by making things so automated, you don't need to do anything.
By trying to work your way out of a job, you'll create bugs that keep your job, and make you very frustrated.
Everyone will want you to build them an app, but that doesn't mean you'll be able to.
You're not guaranteed to make a ton of money.
You're also not guaranteed to work less than 40 hours in a week. (Especially when you first start out).
Desk jobs can be bad for your health. Plan accordingly
The level of stress you get can be bad for your health. Plan accordingly
Your job may be to handle the data of a billion dollar company. (see #9)
You will make a mistake, and it could potentially cost the company a ton of money. (see #9)
Are you willing to "nerd out" on the latest CVE patch fixing a year long "monkey patch" you've had allowing you to delete 100 LOC?
Does reading #12 make you cringe?

If you're at this point feeling a little nervous, but still thinking it'll be a better move for you in the long run, then I say go for it! It's an amazing job, and you can have a lot of fun with it.

Side note:

If you're already a programmer, leave your thoughts to how your day looks below. Different perspectives always help newcomers.

Programming in the Adult Entertainment Industry II

Jeremy Woertink — Wed, 27 Feb 2019 16:56:00 +0000

When you start programming in the Adult Entertainment industry, things start to get a little tricky. A lot of systems you used and took advantage of with previous apps are no longer available to you. It's mainly different sites TOS prohibiting you using their service when associated with explicit material, or something like cannabis related.

Emailing

Let's take emailing for example! There's tons of email services like MailChimp, MailGun, SendGrid and many others. But the problem is, we can't just choose the best option, or the cheapest option. We have to first see if the service would even allow for sending out any adult related content. I'm not talking about emailing porn, I'm simply talking about transactional emails like a password reset email. A simple all text related email, but it happens to be coming from "some_explicit_domain.com". So in this case, we might use something like "yourfunadultsite.com" or sometimes something even less direct like "6E92119A91B70F8BEDCC2720C69CE1CB.com". Those usually end up leading us to high bounce rates, and spam reports though.

That's before we even start talking about using the service. Once we find a service that allows for sending stuff out, we have to be extra conscious about bounce rates. When people sign up for these type of sites, no one wants their personal email attached. So what do they do? They use a throw away email. This means we now have a member we can't send specials to, we can't email for billing issues, we can't contact to say we need to go down for some server maintenance. We have this person giving us money that we don't have an easy way to contact. Another thing is a lot of people will just use a fake email that doesn't exist. This would raise our bounce rate and potentially get us banned with the service we chose.

CDN and Hosting

Well, you can't just choose any host, that would be too easy. The host has to be cool with what you're selling. Luckily most hosts these days are; however, that wasn't the case just 10 years ago.

The tricky part these days comes with the sheer amount of data. A small company with a few thousand videos could have something ridiculous like 9PB of data... yeah, that's a P... (7 * 10^15). Could you imagine trying to make a backup? Now think of what a larger video company would have with hundreds of thousands of content files...

Amazon now has these devices called AWS Snowball that they ship you. You plug in to your physical server and copy your data, then ship the device to a new location. Unless you're working with massive physical data, this is not something you'd really ever come across.

As well as the hosting, you have to look at the CDN. It's common to have users all across the globe for your sites. You want to make sure latency and connection is as fast as possible because if it takes someone longer than a few seconds to load your videos, they will move on quickly. I mean, there's no shortage out there 😛. A CDN will help that out, but you have to be sure your CDN allows for that. They are basically making a duplicate copy of your site and spreading it all over the world.

Once you find a CDN, you have to consider what your traffic and bandwidth look like. These types of sites are such high traffic and high bandwidth that it gets expensive REAL quick. If you take a look at Cloudflare, you'll see the $200/mo plan (per domain). Since these sites are generally based around multi-tennant apps, you'll run 30+ sites off the same app easily. Oh, then you need Load Balancing, Rate Limiting, and SSL?

How about something more positive?

With so many things that are a challenge with this industry, why even work in it? Well, there are also some really great things (I'm talking about the dev side). For example, learning about Machine Learning stuff which could allow you to build a video recommendation engine. You watch a video, and then based on how long you watched it, which actors are in that video, maybe the types of things (we'll just call them tags) are in the video. Maybe also taking in to account how many people have commented on the video and what the average star rating is. With all of this data, we can now build a system that says "You might enjoy watching these videos as well...". Pretty much any video site you go to like youtube, or twitch, they're going to have some system that says because you watched video X, you may like video Y.

Me personally, I hate the way youtube does it. I checked out one football video to see what people were talking about, and now all the videos recommended are sports related. They must not know me though since I don't watch sports. This is a tricky problem to solve, but not impossible. That's the sort of challenge you get with stuff like this.

Also dealing with videos, you have to be concerned with video encoding. Most companies shoot the content in 4k these days, but not everyone can watch in 4k. Most devices won't really show a difference between 1080p and 4k anyway. So might as well save some bandwidth by showing a different quality version based on the user's device. We take a single video, and turn it in to several different files from a low res 420 to full HD 4k.Then by using HLS streaming, we can stream the right version to you based on things like your connection, screen size, etc...

Some decent video encoding services are Coconut, Zencoder, Bitmovin, and Telestream. So far Coconut seems to be the cheapest as well as providing a very nice API. Nice APIs are hard to come by in these industries.

Now, I've only dealt with the adult video industry, but the cannabis industry is running in to a lot of the same issues. It's legal in a lot of places now. Way more than it was 10 years ago. That means there's a new thriving industry to build awesome new apps. For example, I saw a site that had grow stats for different people. Each day it showed the humidity, temperature, soil saturation, sun light, and tons of other unique data for each day a plant was growing. Building a system that sends data to a site through an API is something a lot of developers never get to work with.

Sadly, because that industry falls in line with the "gray" area, it also gets a lot of the same stigma. When I worked in the poker industry a few years back, it was also in the same line. You could have a poker app, but it had to be hosted in Nevada, and only played by people with a Nevada based IP.

Have you worked in any industry that has some unique challenges? Share your experiences!

Background Jobs with Lucky

Jeremy Woertink — Thu, 14 Feb 2019 23:19:28 +0000

As with a lot of web applications and APIs, there comes a point where you need to process some tasks in the background. To do this, you'll usually use a background job processor. This helps to make some things asynchronous and speed along the users of your app.

An example of this might be logging when your users login to your site. Maybe you store the IP address they used, when they logged in, how many attempts it took to sign in, and the device they used. If you did all this in-line, then the process would be to track all that info, and the user has to wait while you create this new record in your database before they can actually login. Or, you push that info to a background job, and log the user in. Then you can update that record at your own pace since there's nothing relying on the data.

If you're new to Crystal Lang and/or you're new to Lucky, but you're coming from another language like Rails, you may be familiar with Sidekiq. Now, Sidekiq does exist in the Crystal world which is amazing. It even exists by the same guy that created the Ruby version! But at the time of this writing, I spent nearly an hour trying to figure out how to get it integrated with Lucky, and I was running in to quite a few issues. Thankfully, I found an alternative that I had installed and running locally in maybe 2 minutes.

Mosquito

Take a look at Mosquito. The API for it really isn't too far off from what Sidekiq uses, so for me, the concept was quick and easy. This also uses Redis, and since I was going to use Redis with Sidekiq, there's no additional things I needed to install.

Assuming you have your Lucky app all setup and ready to go, you can start by adding Mosquito in:

dependencies:
  mosquito:
    github: robacarp/mosquito

Run your shards install, and you'll be good to go.

Now we just need to setup our mosquito runner. Create a new file ./src/mosquito.cr.

require "./dependencies"
require "./workers/*"

Mosquito::Runner.start

Setting up workers

Next we need to setup the workers. Create a new folder ./src/workers/, and let's make a login_worker.cr file in there.

class LoginWorker < Mosquito::QueuedJob
  params user_id : Int64
  params ip : String

  def perform
    user = UserQuery.find(user_id)
    if user
      # create new Login record from the user
    end
  end
end

Using the worker

At some point we have to kick off the LoginWorker we made, so in one of the actions, we can add this:

post "/login" do
  form = SessionForm.new(params)
  form.submit do |f, user|
    if user
      LoginWorker.new(user_id: user.id, ip: "127.0.0.1").enqueue
      # redirect logged in
    else
      # show your errors
    end
  end
end

Booting the whole thing

Now that you have mosquito installed, and setup, you'll just need to boot the whole app. Since Mosquito runs in it's own process, you have to run it in a separate process.

In your Procfile.dev add in the line worker: crystal src/mosquito.cr.

web: lucky watch --reload-browser
assets: yarn watch
worker: crystal src/mosquito.cr

Now when you run lucky dev, it'll boot mosquito in it's on separate process!

ProTip:

If your app is like mine, and just an api, your Procfile.dev might look more like

web: crystal src/server.cr
worker: crystal src/mosquito.cr

Mosquito needs to know about your stack since it runs in a separate process. If you're trying to slim this down, just keep in mind that whatever you put in your worker file, you'll need to require that separately from your app stack.
By default, Mosquito looks at your REDIS_URL ENV, or pulls just a normal localhost value. Be sure to update that for production, as well as your Procfile for your production.
If you would rather build the targets (maybe for docker), and run those separately, you can update your shard.yml with:

targets:
  server:
    main: src/server.cr
  mosquito:
    main: src/mosquito.cr

Then shards build --production --static --release --no-debug to get your server and mosquito binary files. Just run those however you need, and you should be good.

5 reasons you should learn Lua

Jeremy Woertink — Fri, 14 Dec 2018 17:37:32 +0000

If you haven't heard of Lua, it's a really small programming language. Maybe you have heard of it, but never had a reason to learn it because your programming language of choice does everything you need it to. Well, here are a few reasons you should learn Lua anyway.

5.

Have you heard/used nginx? You can use mini Lua scripts inside of nginx with the openresty setup. This would allow you to do things like dynamically assign SSL certs through Let's Encrypt to different domains.

auto_ssl:set("allow_domain", function(domain)
  return ngx.re.match(domain, "^(mysite.com|yoursite.com)$", "ijo")
end)

Use Apahce instead of nginx? Lua still has you covered.

4.

Another popular tool that is used in the web is Redis. If you use redis to store lists of timeline event IDs or something, you may need to do some crazy sorting on this data. Writing RPOPLPUSH statements in redis can get a little crazy, and sometimes the language you choose may not be as fast as you'd like. For example, if you're using ruby, there's a gem called Wolverine. You write a Lua script that does all the crazy sorting you need, and then pass that data back to your rails app. This gives you a massive performance boost.

3.

If you need lots of performance for your web app, but you don't have the capacity to re-write in something different, then you may employ heavy cache in to your app. One way to do this is to throw Varnish in front of your app. Varnish will handle all the cache, and only proxy back to your app when the cache is expired.

Varnish uses these vcl scripts which can get a little confusing. With a little lua magic you can write some nice programmatic cacheing. Maybe cache different lengths based on the domain in a multi-tenant situation?

2.

Are you big in to playing video games? Maybe you've heard of some of these?

Angry Birds
Civilization V
Far Cry
Garry's Mod
Heroes of Might and Magic V
L.A. Noire
Mafia II
Roblox
Saints Row 2
SimCity 4
Star Wars Battlefront
Stepmania
Warhammer
World of Warcraft

All of these games use Lua in some capacity, and there's MANY more. Most games or game engines will be written in C++ or C#, but using Lua on the front end for scripting allows for rapid development of these games. No need to re-compile each time. Compile your engine, and have it run the lua scripts. You can get started with Love2d or defold

1.

Because Lua is fun to write! If you don't know it, it wouldn't hurt to learn a new language. If you already know how to program, picking up lua should be pretty quick. Read over this 15 min of lua, and you will have a decent understanding of the syntax. At that point it's just a matter of digging in!

Quick start with React on Lucky

Jeremy Woertink — Fri, 30 Nov 2018 17:40:19 +0000

This will be a quick start in to getting some simple React code with minimal setup in to a simple Lucky app also with minimal setup.

Step 1, setup lucky

Make sure you have lucky installed locally. You can check out the installation guide for getting that installed.

Once that's installed, we can generate our app!

$ lucky init
Project name? simple_react_on_lucky

Lucky can generate different types of projects

...

API only or full support for HTML and Webpack? (api/full): full

Lucky can be generated with email and password authentication

...

Generate authentication? (y/n): n

-----------------------

Done generating your Lucky project

  ▸ cd into simple_react_on_lucky
  ▸ check database settings in config/database.cr
  ▸ run bin/setup
  ▸ run lucky dev to start the server

Run those last few steps that lucky tells you to run, and then your app will be setup.

Step 2, add in react

Next we need to add react to our app.

yarn add react
yarn add react-dom

Step 3, make a page to use react

Now we need a page where we can write some code to mount our react app. We will just make a simple home page to override lucky's default home page.

Open up src/actions/home/index.cr

# change
render Lucky::WelcomePage
# to
render Home::IndexPage

Now create a new src/pages/home/index_page.cr file.

class Home::IndexPage < MainLayout
  def content
    div(id: "app")
  end
end

Step 4, write some react

Open up src/js/app.js and add in your react imports. You'll see some code in there already, you can just add this stuff below that. Worry about whether you need that existing code or not later.

import React from 'react';
import ReactDOM from 'react-dom';

const mountElement = document.getElementById('app');

class App extends React.Component {
  render() {
    return React.createElement('h1', null, 'WOOT!');
  }
}

ReactDOM.render(React.createElement(App), mountElement);

Now run lucky dev in your terminal, and provided we didn't screw anything up, you should see "WOOT!" on your home page now!

Obviously you'll want to add a lot more code for a real app, and using the React.createElement functions will get super messy. This should at least give you a jumping point on how you can start working React in to your Lucky applications.

You can also take a look at this post by @mikeeus on some more Lucky and React.