DEV Community: Seth Tucker

An Introduction To App Development With Advanced Vibe Coding

Seth Tucker — Sat, 17 May 2025 15:42:22 +0000

Written Date: 5/17/2025

Why You Need This Process

For the last 2 years I've been using AI to drive most of my development efforts. It wasn't called "vibe coding" then, but it was just as frowned upon. The phrase "AI slop" was coined and tossed around willy nilly. Under every rock and around every corner you would find an expert ready to jump out and say that AI was overhyped. It was tenuous time to be using AI for helping you code.

I will admit that the concerns brought forward were true then and are still mostly true now. However, there were these moments of brilliance that would shine through. These felt like samples of what was to come in the future, something that we would eventually consider to be just plain normal.

I've held on to the inspiration and hope that came from those moments of clarity and brilliance. My wish was to find a way to make them happen consistently. That's a thing you'll come to discover about AI, consistency is the hard part. It's easy to make something incredible happen once. Can you make make it happen two times? Five times? What about nine out of ten times? Can you make AI work consistently 100% of the time?

The five layer design approach is the wisdom I've been able to bring together from thousands of hours of trial and error from the past several years. It is in no way complete. Consider this the first assembly line for the Model T. There will be many improvements and refinements to this process as time goes on. I expect there will be competing schools of theory emerging at some point. We'll see whose idea wins 😉.

So What's The Point?

This question is easy. The point is to create a process the produces a more consistent performance from AI models. If we were talking about people, we would refer to this as "gaining experience and building expertise". However, AI models don't learn the same way people do. They only imitate human behavior. Because of this, we need a process that consistently represents the kind of process that we as humans naturally develop during our experiences.

This is how I came to define the 5 layers. It defines a process that can be performed over and over, with adjustments and customizations, to yield a consistently successful outcome using AI models as the main work horse.

Slaughtering Sacred Cows

From here on out I'll only be discussing the concept of the five layers and how to vibe code your way through them from a high-level perspective. That's because the code does not matter.

Code as art is no longer important. It never really was, but most engineers who come from a pre-AI era have gone through a period of time where their code was their art. The vibe coding era is the beginning of the death of boutique code as art.

I will slaughter many sacred cows as I wrote about this. It's important to do. It's a resetting of the first principles that is required in order for software development to move forward as an industry.

I look forward to seeing what you build. Vibe on my friend.

Advanced Vibe Coding

Seth Tucker — Sun, 11 May 2025 13:44:19 +0000

These days, you can't exist as a developer without hearing about vibe coding. It's the cool new thing that lets "technically dangerous" people write code.

Personally, I've been vibing since November 2023 when GPT 3.5 was launched. Not in the same willy-nilly "throw it against the wall and see what sticks" approach. That's hard no for me.

There's too many valid and real problems with the spaghetti wall approach:

Models don’t understand the big picture, then make design decisions that hurt more later (or spiral into a death loop right now)
When the models start out with a clear objective, they still get lost (dazed and confused) as complexity grows
Models do still succeed, but the code is disorganized. So when you try adding to it, the model starts breaking it’s own work.

There are many more, but these are the biggest points that everyone will experience as they attempt using AI to help with coding. If you're traditionally vibing, you'll feel every point here much sooner than later.

Even though context windows have multiplied in size since GPT 3.5, the same problems are still plaguing us. This means we need new and different tools in our proverbial tool belt in order to effectively build with AI.

Introducing: Advanced Vibe Coding

The premise here is simple: have a structured approach that allows you to maintain the correct context throughout your project as you work through building.

1. Right Model, Right Task

This probably feels obvious, but it's not. Large language models are the first time we can experience tech that can present as being good at something, but it actually doesn't understand what it's doing. Have you ever seen a person work who strongly believed they knew what they were doing, but to your experienced eye they clearly did not know what they were doing? The same thing applies to large language models. Don't stick to a single model for anything, you know have a team. A mixture of experts.

2. Have A Clear Outcome In Mind

Most of your disappointment is going to come from having an expectation that is unmet. You'll dictate something to the model and it'll implement it and you'll quickly realize that what you meant and what it did are not the same, but at the same time the model was still "correct" in what it did. This is a very human problem that we now get to share with our technology. This problem goes away as teams work together and build experience, but models don't build experience like this. You'll grow this skill over time and through experience. This is extremely dependent on your communication style and the context in which you are working. Embrace your uniqueness, have some patience and chill.

3. Work A Process.

You need to work a thought process on behalf of the model. Reasoning models are great, but the experience of working with an LLM is still very much like having an incredibly smart but immature junior. There isn't much thinking things through or planning, it's prompt -> output. Reasoning models are better, but those reasoning tokens take away context from the big picture.

Your job is to guide the model through a process that maintains the bigger from the perspective that is necessary to implement the feature. That perspective varies depending on which point of the development process you are in. As humans, this is what we develop as part of our "expertise". Models don't have that, they have knowledge. This is the first time we are experiencing working with something that is so deeply knowledgable but wildly inexperienced.

Use your wisdom to guide the models use of knowledge.

My Process - 5 Layer App Development

The process I came up with is what I call "5 Layer App Development", or "5 Layers" for short.

If you're an experienced developer, this will not teach you anything that you don't already know. It will organize the way we normally think about working on a feature into a set of clearly purposeful steps so that we can guide a model successfully.

The five layers are as follows:

Jargon Feature Stories
Generic Feature Stories
Technical Feature Stories
Pseudo Code
Actual Code

They are ordered from least descriptive of the technical implementation to the most descriptive of the technical implementation.

Greenfield applications start at Layer 2. As an app matures you'll begin working from Layer 1. This just happens, as it's entirely normal to begin using jargon to describe the larger complex features of your application. I'm not going to give an in-depth explanation of the process in this post, I will do that in a follow-up post. It's complex enough that it's worthy of possibly a series.

Most of your work is going to be during layers 1 & 2. If you properly define the feature you're trying to implement (or the bug you're trying to squash) then your involvement will largely end during layer 3. If you're familiar with agile stories at all, then you'll easily be able to write feature stories in layers 1 & 2.

"Why not call them user stories?"

Because not everything is about a human user anymore. Now that we have MCP and internet capable models, we are going to see a rise of AI users being the "third consumer". I'll write about this later in another post.

Layer 3 is where we get technical. It's where we identify the tech we have, if it's capable of performing the feature we are implementing and how to address any limitations of the tech stack we have.

Layer 4 is basically like using a Chain-of-Thought (CoT) prompting strategy. Feed in the refined feature story, provide the technical capabilities from layer 3 and have your model make the outline of it's implementation plan.

Layer 5 is pretty straight forward... but in case you missed it, have the model read from the layer 4 description and implement the actual code. The larger the implementation steps necessary, the more you'll benefit from breaking this down into modular chunks.

5 Layer Rules

The rules for working through the layers are straightforward (for now). There's a lot more to it, but you don't have to know the nuance to be effective and see benefits.

Start by working "down".

Start at the highest layer that applies. This is always layer 1 or 2.

Focus on 1 feature at a time

Scope creep anyone? You're working from top to bottom and then eventually back up the layers. You're traversing a loop of productivity. Working "up" requires more than just some ideas, so that's outside the scope of this post 😏

Always use git commits (at least) between layers

Commit often. Use a branch, stay organized and commit, commit, commit. You'll naturally find ways to write meaningfully informative commit messages because you can tie it back to the task the model was performing for the layer you were in.

Example: "Refined layer 2 feature story for 'x'. Saving before beginning layer 3 analysis".

Do you feel like a mad scientist yet? 😆

Guide the model

You're job here is to offload the nuanced cognitive load that comes from planning, execution and reflection. Making plans is hard and decision intense. We have a limited amount of decisions we can make in a day. Let the model think for you. Let the model use it's infinite decision making capabilities to handle the mundane.

Have Fun

I shouldn't have to tell you this, but here I am. It's "vibe" coding for a reason. Feel the vibes man! If you're not feeling the vibes, it's a huge red flag that you're not thinking straight. Frustrated, angry or otherwise negative emotions block freeform thinking. If you're stuck, feeling anything other than in the zone, then reset yourself (and probably start a fresh chat with the model).

Good Vibez Only
-- Seth, The Crimson Knight

Integrating Jennifer & Amber v1.0+

Seth Tucker — Mon, 04 Dec 2023 13:09:23 +0000

To get the full experience with Jennifer, we'll first want to install sam. This is Jennifer's CLI tool that has generators for creating models, managing the db creation/resetting, migrations, etc. that aren't currently able to be added quickly and easily into the Amber CLI. So this gives users a way to still have the CLI conveniences of a framework. The nice part is that once sam is installed and working, this also means Amber will work.

First, add the sam shard to your shards.yml, it should be added under the "development dependencies" sections like this:

development_dependencies:
  ameba:
    github: crystal-ameba/ameba
    version: ~> 1.5.0
  sam:
    github: imdrasil/sam.cr
    version: ~> 0.5.0

Next, remove the entry for granite and citrine-i18n, and add the shards for Jennifer and your database adapter of choice. Your dependencies section should look like this on a brand new Amber app:

dependencies:
  amber:
    github: amberframework/amber
    version: 1.4.1 # Make sure to use the latest version of Amber here

  quartz_mailer:
    github: amberframework/quartz-mailer
    version: ~> 0.8.0

  jasper_helpers:
    github: amberframework/jasper-helpers
    version: ~> 1.2.2 

  jennifer:
    github: imdrasil/jennifer.cr
    version: "~> 0.13.0"

  pg:
    github: will/crystal-pg
    version: ~> 0.26.0

  # Uncomment to use MySQL. Note: on MySQL v8, this shard does not support the v8 authentication protocol, you'll want to run:
  # `mysql --default-auth=mysql_native_password` for MySQL to work properly
  #mysql:
    #github: crystal-lang/crystal-mysql
    #version: ~> 0.14.0

  # Uncomment to use sqlite
  #jennifer_sqlite3_adapter:
    #github: imdrasil/jennifer_sqlite3_adapter
    #version: "~> 0.4.0"

Save your shard.yml file at this point.

Next, create a new file in your projects root folder called override.shard.yml and save the following into the file:

i18n:
  github: crimson-knight/i18n.cr
  branch: master

You can now install your shards by running shards install.

You may get something like the warning below:

Shard "ameba" may be incompatible with Crystal 1.10.1

You can usually safely ignore this warning, or adjust the version down until the warning goes away. Eitherway, it's safe to move on.

Next, we are going to change the file contents of the config/database.cr file to the following:

require "jennifer"
require "jennifer/adapter/postgres" # for PostgreSQL
# require "jennifer/adapter/mysql" for MySQL
# require "jennifer/adapter/sqlite" for SQLite

# This allows for easy test db use when running the `crystal spec` command
{% if @top_level.has_constant? "Spec" %}
  APP_ENV = "test"
{% else %}
  APP_ENV = ENV["APP_ENV"]? || "development"
{% end %}

Jennifer::Config.configure do |conf|
  conf.read("config/database.yml", APP_ENV)
  conf.from_uri(ENV["DATABASE_URI"]) if ENV.has_key?("DATABASE_URI")
  conf.pool_size = (ENV["DB_CONNECTION_POOL"] ||= "5").to_i
  conf.logger.level = APP_ENV == "development" ? Log::Severity::Debug : Log::Severity::Error
end

Next, we'll want to create the file containing our database connection details. This will look very similar to the database configuration in a Rails app, but don't be fooled! These are not identical :)

Create the file config/database.yml with the following contents, updated with your apps details. None of the databases need to be created yet, just put what you'd like your app databases to be called. If you use an existing database, it should still connect with the correct details.

default: &default
  host: localhost
  user: user_name # On Mac, this is the results from a fresh terminal session when you run `whoami`
  password: user_password # On Mac, this is blank by default, use empty quotes '' for this.
  adapter: postgres # or `mysql`, or `sqlite` without quotes

development:
  <<: *default
  db: application_database_name_development

test:
  <<: *default
  db: application_database_name_test

production:
  <<: *default
  db: application_database_name_production

Finally, we'll want to modify the sam.cr file found in your projects root after installation the shards.

The contents should look like this:

#!/bin/crystal

require "./config/database.cr" # here load jennifer and all required configurations
require "sam"
load_dependencies "jennifer"

# Here you can define your tasks
# desc "with description to be used by help command"
# task "test" do
# puts "ping"
# end

You should now be able to run: crystal run sam.cr help from the projects root in your command line and see output something like this:

 jennifer_config crystal sam.cr help
Name Description
------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------
help | Prints description for all tasks
generate:makefile | Generates makefile extension. Now command could be executed via `make sam your:command argument`
generate:migration | Generates migration template. Usage - generate:migration <migration_name>
generate:model | Generates model and migrations template. Usage - generate:model <ModelName>
db:migrate | Will call all pending migrations
db:step | Invoke next migration. Usage: db:step [<count>]
db:rollback | Rollback migration. Usage: db:rollback [v=<migration_exclusive_version> | <count_to_rollback>]
db:drop | Drops database
db:create | Creates database
db:seed | Populate database with default entities.
db:setup | Runs db:create, db:migrate and db:seed
db:version | Prints version of the last run migration
db:schema:load | Loads database structure from the structure.sql file

If you've run into any errors, double check that you've saved all of the files with the most recent changes, and that there are no typos.

Important Note

Check in the src/ folder to see if there is already a models folder. As of Amber v1.4 the new command does not create this folder and this is the destination for models being created by generate:model.

Once you create the src/models folder, you should now be able to generate a model:

I like to build sam once and then re-run it. If you change anything in the sam.cr file, you'll need to recompile to pick up any changes outside of yml files.

crysatl build sam.cr


./sam generate:model example_model name:string


./sam db:create


./sam db:migrate

Your database should now be setup and operational with basic generators for Jennifer working.

And, you're done! Your app should now work with Jennifer. The Amber CLI and sam don't yet work together, but the controller and view generators should get you very close to a working initial template with only minor changes to make your app compile.

Make sure to use same for creating/seeding/migrating your database.

Building An AI Automated Print-On-Demand Business: An Experiment

Seth Tucker — Wed, 26 Apr 2023 16:42:49 +0000

Introduction

Last month (March) I was sitting with a friend in our "idea factory" one afternoon while he was showing me Midjourney. I was blown away at how good AI art is getting. Both of us have marketing backgrounds, strongly focused on growth hacking. Even though I have a background in marketing, I'm not naturally a creative or artistic person. My little brother got that talent, I got an analytical mind.

But, what if I could use something like Midjourney to create products?

Could I automate everything?

I dove in head first to start exploring my options and figuring things out. As a Rubyist, I naturally created my online store on Shopify. Just kidding, it's the incredible amount of integrations that Shopify has that made this a no-brainer for me.

So, in true soloprenuer fashion, I also had to dive into YouTube to figure out what needs to be done. There are some fantastic channels out there with loads of content! However, I found one thing commonly true across nearly all of the videos out there: they undersell the level of effort required to start a print-on-demand business.

There are a lot of subtle nuanced steps involved and frankly, a hell of a lot of copy+paste happening. As a developer, this is absolutely MADDENING!

Business Model Outline

Let's dive right into the business model outline here. To frame this model, there are two goals that I am working toward:

100% Automation.
Enough passive income to replace my salary entirely.

My first goal isn't entirely possible. If you're unfamiliar with Midjourney, this is when you learn that the bot you use to interact with to generate images is tied to the specific Discord user who signed up. Pretty smart, but that means I can't write my own bot to interact with it. I'm also not sure I want to program entirely randomized phrases (or full product descriptions) getting turned into images and placed on products for printing that I haven't laid eyes on. There's also some photo editing that has to happen, particularly upscaling the images to maximize the quality of the print.

Here's a nice canvas outline showing, overly simplified, the steps required to create and publish a product:

Step 1 can't really be automated without using techniques that go against Discords TOS. Step 2 has interesting automation aspects. After hours of YouTube videos on how to use Photoshop, I found this YouTube video for bulk-processing image mockups. This guy is using a Javascript automation extension that was created back in the mid-2000s to automate a lot of his mockup process. Cool!

This isn't the kind of automation that I was hoping for, but I'll still take whatever I can get.

For steps 3 and 4 I had to take a closer look at my service providers and determine if there is an API I can use. Shopify has a famous API, but does my print provider?

For this experiment, I decided on using Printful. Of the three most popular, they are the most expensive but they offer the least amount of manual intervention for subtle nuanced customer service tasks. Also, the difference in price doesn't appear to make a meaningful difference until the scale of your store sales is very high. Since my goal is to optimize for automation I think Printful makes the most sense.

Printful appears to have an API that is rather robust. The docs are pretty decent, better than some enterprise API's I've worked on. But, some context from their API is missing. For example, the API documentation shows that I can create a product template by providing the details of the location on a print product and a link to the image. But then how does positioning work? What if I want to use a repeating pattern? 🤔 After doing some searching I found that no one is writing about this API in public. Probably so they don't give away their secret sauce recipe 😏

This product template creation is easily one of the most critical and cumbersome steps in this entire print-on-demand business workflow.

In the next part of this series, I'll go through how I architect the solution and structure my code.

Crystal PCRE2 Upgrade Guide

Seth Tucker — Mon, 06 Mar 2023 21:31:31 +0000

It was announced recently that in Crystal 1.8.0 >= 2.0.0 the default Regex engine is going to be PCRE2.

After doing some digging into this thread on Github it appears that the original developers of PCRE don't remember what was done or aren't around for comment any more.

Sadly, this looks like it's a case of bit rot and an abandoned open source project that was poorly documented. It appears the decision to move forward was with upgrading/rewriting the library is what allowed the current maintainers to fix the known issues while making some improvements. However there wasn't any documentation built along the way, which makes the transition a bit daunting because there is a possibility for undocumented breaking changes.

I'm in a similar situation with the Amber Framework. I'm not one of the original creators of Amber, I just took it over last year (2022). So I have a decent size code base to maintain with potential for breaking changes.

Here's The Plan

Here's how I'm going through and verifying the behavior does not change:

Step 1. Identify all possible regex's used in the code base.

Step 2. Create tests that uniquely test those regular expressions.

Step 3. Switch to using PCRE2 in the tests and see if anything blows up.

Before We Begin

Let's make sure we have PCRE2 installed on our systems. I work on a Mac, so I'll be using Homebrew to install PCRE2.

brew install pcre2

Step 1. Identifying Regexs

My preferred editor is Visual Studio Code which has a handy feature that allows me to search with... regexs! Since it's easy to find the syntaxes for regular expressions in Crystal, I'll summarize them here:

/123/  # Uses the slash regex literal `/` with an enclosing `/`
# Sometimes this syntax will appear with wrapping parentheses, sometimes without.
"abcd".matches? /abcd/
"abcd".matches?(/abcd/)

Regex.new("123") # Converts the string into 

# the percent regex literal and the valid delimiters 
# which are: (), [], {}, <>, ||
%r((/)) # => /(\/)/
%r[[/]] # => /[\/]/
%r{{/}} # => /{\/}/
%r<</>> # => /<\/>/
%r|/|   # => /\//

This gives us 7 combinations of how regular expressions can be found across the code bases.

The reference material for how Visual Studio Code handles regular expressions left me a little frustrated at the lack of specificity or clear exploration path for additional resources. So, through some trial and error I managed to create this regular expression that properly matches all of the above syntaxes for Crystals regular expression syntax:

 \/.*\/ |\(\/.*\/\)|%r\(\(.*\)\)|%r\[\[.*\]\]|%r\{\{.*\}\}|%r<<.*>>|%r\|.*\||Regex.new

Import Note there is a leading space at the beginning of this regex. This is important! Make sure you add that space if it is not present when you copy/paste.

You can verify this is still working as expected by copying the regular expression syntax into a file in your project and using the project search feature with that expression. It should match both the %r syntax and the commented output with the double forward slashes but ignore the # => text, the Regex.new and the more vague double forward slash syntax /.../.

When I ran this regex on the Amber code base (Amber v1.3) I got these results:

31 results in 10 files. Not bad, this turned out to be more manageable than I first expected it would be.

Step 2. Testing The Found Regular Expressions

I can pretty easily grab all of my results here by clicking on the "Open in editor" link just below the search fields in the project search field.

The text document that pops up has just enough info to be dangerous. Let's copy that and make a new spec file. For Amber, I'm going to put this right in the root spec folder.

Now it's time to de-dupe my results. The results aren't large, and I can see the same regex used in a split() multiple times, otherwise everything looks unique. Now I'm down to 27 tests to make.

I'm taking a couple of approaches due to patterns that start to appear in the regex's I'm seeing. We have a group that are unique in the total contents, but they are ultimately testing the beginning and ending of the same regex like this:

# spec/amber/cli/recipes/recipe_fetcher_spec.cr:
template.should match(/.+mydefault\/app$/)
template.should match(/.+mydefault\/controller$/)
template.should match(/.+mydefault\/model$/)
template.should match(/.+mydefault\/scaffold$/)
template.should match(/.+mydefault\/.recipes\/lib\/amber_granite\/app$/)
template.should match(/.+\.recipes\/lib\/amber_granite\/controller$/)
template.should match(/.+\.recipes\/lib\/amber_granite\/model$/)
template.should match(/.+\.recipes\/lib\/amber_granite\/scaffold$/)

I'll narrow this down to a test with /.+\.test\/path$/ which should be testing a string that has "test/path" in it's contents.

As I worked my way through the regexs to test, I noticed a large grouping from a monkey patch module for the String class. Personally, I'm not a fan of monkey patching and these regexs are pretty old with other methods from the std lib now available to do some of these same things. I also noticed there's already a spec specifically for those methods, so I'm going to skip making unique tests for them.

All in all, I ended up with only 9 tests.

require "./spec_helper"

describe "Testing regular expressions" do 
  # spec/amber/cli/commands/exec_spec.cr:
  #  43:         logs = `ls tmp/*_console_result.log`.strip.split(/\s/).sort
  it "verifies the regex splits on a space when using `/\s/`" do
    string_array = "test string".split(/\s/).sort
    string_array.first.should eq("string")
    string_array.last.should eq("test")
  end

  # spec/amber/cli/recipes/recipe_fetcher_spec.cr:
  #   21:           template.should match(/.+mydefault\/app$/)
  #   27:           template.should match(/.+mydefault\/controller$/)
  #   33:           template.should match(/.+mydefault\/model$/)
  #   39:           template.should match(/.+mydefault\/scaffold$/)
  #   57:           template.should match(/.+mydefault\/.recipes\/lib\/amber_granite\/app$/)
  #   77:           template.should match(/.+\.recipes\/lib\/amber_granite\/controller$/)
  #   86:           template.should match(/.+\.recipes\/lib\/amber_granite\/model$/)
  #   95:           template.should match(/.+\.recipes\/lib\/amber_granite\/scaffold$/)
  it "verifies the 1 or more of any starting character, ending with a set specific string" do
   test_string = "blahblah/blah/bblahaaaa1233123412341234123423this/is/a/path"
   test_string.should match(/.+this\/is\/a\/path$/)

    test_string2 = "blahblah4321341234!!!!.this/is/a/test/path"
    test_string2.should match(/.+\.this\/is\/a\/test\/path$/)
  end

  # spec/amber/pipes/static_spec.cr:
  #   57:         response_true.body.should match(/index/)
  it "verfies a basic plain set of characters in a regex works" do
    # This test is so basic is probably could have been skipped, but I kept it for consistency sake
    "has the word index in it".matches?(/index/).should eq(true)
  end

  # spec/support/helpers/cli_helper.cr:
  #   123:     route_table_text.split("\n").reject { |line| line =~ /(─┼─|═╦═|═╩═)/ }
  it "verifies the regex for removing box drawing characters" do
    test_string = "═╩═\nhere is a\ntest string with new lines\n─┼─\nanother line\n═╦═"

    split_array = test_string.split("\n").reject { |line| line =~ /(─┼─|═╦═|═╩═)/ }
    split_array.size.should eq(3)
    split_array.first.should eq("here is a")
    split_array.last.should eq("another line")
  end

  # src/amber/cli/generators.cr:
  #   216:      if name.match(/\A[a-zA-Z]/)
  # src/amber/cli/recipes/recipe.cr:
  #   36:       if name.match(/\A[a-zA-Z]/)
  it "verifies a string starts with A-z (upper and lowercase)" do
    string1 = "Apex Legends123123"
    string2 = "123Googal"
    string3 = "lower case"

    string1.should match(/\A[a-zA-Z]/)
    string2.should_not match(/\A[a-zA-Z]/)
    string3.should match(/\A[a-zA-Z]/)
  end

  # src/amber/cli/commands/pipelines.cr:
  #   91:           pipes = pipes.split(/,\s*/).map(&.gsub(/[:\"]/, ""))
  it "verifies the string is split after a comma with multiple white spaces and removal of colons and quotes from the results" do
    string = ":split,    \"this\",       string, properly"
    final_array = string.split(/,\s*/).map(&.gsub(/[:\"]/, ""))
    final_array.size.should eq(4)
    final_array.first.should eq("split")
    final_array.last.should eq("properly")
    final_array.find { |r| r.matches?(/:/) }.should eq(nil)
    final_array.find { |r| r.matches?(/\"/) }.should eq(nil)
  end


  # src/amber/cli/recipes/file_entries.cr:
  #   50:           if /^(.+)\.lqd$/ =~ filename || /^(.+)\.liquid$/ =~ filename
  it "verifies the line beings with any character and ends up .ldq or .liquid" do
    string1 = "blah_blah blah.lqd"
    string2 = "blah blah blah.liquid"
    string3 = "liquid.blahblah"

    string1.matches?(/^(.+)\.lqd$/).should eq(true)
    string1.matches?(/^(.+)\.liquid$/).should eq(false)

    string2.matches?(/^(.+)\.lqd$/).should eq(false)
    string2.matches?(/^(.+)\.liquid$/).should eq(true)

    string3.matches?(/^(.+)\.lqd$/).should eq(false)
    string3.matches?(/^(.+)\.liquid$/).should eq(false)
  end

  # src/amber/pipes/static.cr:
  #   191:         match = range.match(/bytes=(\d{1,})-(\d{0,})/)
  it "verifies a \"Range\" header has two sets of values separated by a hyphen with 1+ values before the hyphen and 0+ values after the hyphen" do
    range1 = "bytes=1231234-12421341"
    range2 = "bytes=0-1241234"

    range1.should match(/bytes=(\d{1,})-(\d{0,})/)
    range2.should match(/bytes=(\d{1,})-(\d{0,})/)
  end
end

Everything currently passes when running crystal spec spec/pcre2_regex_upgrade_spec.cr.

Now I've ended up with only 9 tests. Not bad!

Step 3 Testing PCRE2 - Does It Explode?

Thankfully this part is pretty easy. If you already have pcre2 installed, all you have to do is add a flag to our test command:

crystal spec -Duse_pcre2 spec/pcre2_regex_upgrade_spec.cr

This is now using the PCRE2 api instead of PCRE.

Everything is still passing, that's great!

Final results

I decided to re-run the tests specifically across the entire code base by customization the bin/amber_spec file to use the -Duse_pcre2 flag and was able to get a clean test run.

So as best I can tell, Amber v1.3 will support the migration from PCRE -> PCRE2 without any hiccups.

Dual-booting Rails 7 & Kemal (a Crystal framework)

Seth Tucker — Fri, 29 Apr 2022 13:42:47 +0000

View the source for this on github.

While working on a start up idea I have an interesting thought occurred to me: could I run Rails and a Crystal framework in parallel?

The "why" behind the idea

My side project idea revolves around a snippet of JS code that triggers a chat bot interaction, so I have two end points this snippet will interact with as part of this. Every time a page loads with this snippet a request gets sent to my app. This means adding a single customer could mean adding hundreds of thousands of requests per month. My first thought was dual-booting Rails with Sinatra or even Roda. However I really want to adopt Crystal as much as possible because I believe the language has enormous potential, so why not try and create a developer experience that is just like working with two ruby frameworks at the same time? Could I use this as a strangler approach to slowly move individual end-point behavior from Rails into Crystal and get the significant performance boost a compiled language has to offer?

For this article I'm going to walk you through step by step how to get Kemal running alongside your Rails app within the same directory to create a developer experience that feels like you're working on a pure Ruby app!

Getting started

Here's what we'll be using for this:

Rails 7 (latest)
Ruby 3.1
Crystal 1.4
Kemal (latest, currently v1.2)
Nginx (latest provided by brew)
Nodemon (an NPM package for watching for file changes)
Foreman (the Ruby gem!)

There's no special compiling requirements for any of these dependencies, so use whatever package manager you have to install each of these. I highly recommend installing Nodemon globally with npm.

Step 1 Create the Rails app

Let's get our rails app started:

rails new crystal_rails
cd && bundle install

Step 2 Create the Crystal app

crystal init app kemal_api

This initializes a minimal Crystal application inside of our Rails root. There's no issue here because Rails does not autoload any files outside of the /app and /lib directories. Now we have a few clean up steps to do here.

Let's open the new kemal_api/shards.yml file and add this snippet at the end of the file.

dependencies:
  kemal:
    github: kemalcr/kemal

You can delete these extra files in the kemal_api directory: - README

license
.editorconfig

You'll also need to remove the git repo that was intialized for this folder. Make sure you've cd'd into the kemal_api root and run rm -rf .git and now your version tracking will be the same for the rails app and this sub-directory.

I recommend copying the contents from the new kemal_api/.gitignore into the .gitingnore file in your root, and modify the paths to be relative to the kemal_api directory like this:

/kemal_api/docs/
/kemal_api/lib/
/kemal_api/bin/
/kelam_api/.shards/
/kemal_api/*.dwarf

Crystal compiles all of the apps dependencies into the /lib folder and crystal can auto generate documentation from your code into the /docs/ folder, so this will prevent adding hundreds of unnecessary files into version control.

Now let's install the shards for Kemal:

cd kemal_api
shards install

Now you'll see a shard.lock file in the crystal apps root directory.

Time to add Kemal and give it a test route. Open the src/kemal_api.cr file and modify it to match:

require "kemal"

# TODO: Write documentation for `KemalApi`
module KemalApi
  VERSION = "0.1.0"

  get "/api/v1/test" do
    # This will be the HTML body in the response.
    "Kemal works! This is our API endpoint :)"
  end

  Kemal.run
end

You'll notice the syntax for declaring a route is nearly identical to Sinatra... because Kemal is basically the Sinatra of Crystal! That's why my mind went to using this framework ;)

Step 3 Configuring Rails

Now we'll need to make a few small modifications in Rails to get both apps booting up at the same time.

When Rails 7 is configured to use importmaps (the default) you won't have the bin/dev command available. That's alright, we'll go ahead and add a Procfile.dev

Back in the rails root, we need to modify or create our Procfile.dev and add a line for the new Kemal app we just setup.

If you are using jsbundling, you'll already have this file and you'll need to modify the web: entry and add the api: entry. Leave the css: entry alone so you can still get your styling! :)

web: rails s -p 3001
api: cd kemal_api && nodemon --exec crystal run src/kemal_api.cr --watch src

Next let's add a basic root path page to load for our rails app.

rails g controller home

I'm going to keep this path the way it is for now, so we can see error pages and the successful path in both Rails and Kemal.

Step 4 Configuring Nginx

To have both apps co-exist on the same domain and port, we'll be setting up Nginx to act as a reverse proxy just like you would in a prod environment.

We'll need to edit the default Nginx server block and configs. If you don't know where this file is, you can run nginx -t and Nginx will output the file path for you, along with doing a syntax check (very handy!)

Here is how I've setup my local Nginx:

worker_processes  1;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;

    keepalive_timeout  65;

    server {
        listen       80;
        server_name  localhost;

        # Paths to the Kemal application
        location /api/v1/ {
                proxy_pass              http://0.0.0.0:3000$request_uri;
        }

        # Everything else going to Rails
        location / {
                proxy_set_header Host   $http_host;
                proxy_pass              http://0.0.0.0:3001/$request_uri;
        }
    }
}

You'll need to restart or reload your Nginx service to get the updated configuration. On a mac with a homebrew installed Nginx this command is brew services restart nginx.

Step 5 Testing our applications

In your terminal you can run foreman start -f Procfile.dev and both apps should now boot up.

In your browser go to : http://localhost/home/index and you'll see how auto generated controller view!

Try going to the root path and you'll see the standard rails error page that the route does not exist.

Then visit http://localhost/api/v1/test and you'll see our Kemal message appearing!

Let's make a change in our kemal app by adding a route.

I'm going to add:

get "/api/v1/app-restart" do
  "Kemal should now automatically restart"
end

As soon as I save the file, I can check the terminal and see the the change was detected and is now being restarted. Excellent!

Note If you have a firewall enabled on Mac, you will have to allow the outside connections every time it restarts. The way around this has been using the bin/dev command that comes with Rails when jsbundling is chosen.

Success!

Now you know how easy it is to start building high performance API's with Crystal right alongside your Rails application. If you found this intriguing please leave a comment!

If there is enough interest, I'll try to write more about using an ORM within the Crystal app for interacting with your database.

Dealing with Postgres - ArgumentError: string contains null byte

Seth Tucker — Wed, 08 Dec 2021 15:22:54 +0000

Recently I was testing using Postgres instead of MySQL on a large Rails 6 application. I had created copies of the schemas and removed the MySQL specific attributes from the tables to load in the schema.

All was well at first! I could run rails db:create and rails db:schema:load and everything would work perfectly. I could even boot up the app and make a user account.

Time to run the tests!

Out of 3000+ tests, nearly 2/3 failed with the same error...

ArgumentError: string contains null byte

Weird. What even is a null byte? A little research showed me a null byte would/should be an escaped character sequence \u0000. I tried searching/grepping through the codebase and didn't find that anywhere.

So what the hell is going on?

Well I found another escaped character pattern, \x00. This specific escaped character sequence is a hexadecimal color, but the double 0 means that the color it should represent has it's bytes set to 0, which is null. So Postgres would see this string coming in and translate it from hexadecimal to escaped unicode which just so happens to turn into \u0000 and raise the string contains null byte error.

Grepping the code base revealed this \x00 pattern used in a few places through model callbacks. This was being used to delineate strings, so I changed the delimiter to \n\n and now I no longer get the null byte error!

Sometimes taking over an existing code base... you find awfully strange things!

Crystal for the curious Ruby on Rails Developer

Seth Tucker — Wed, 11 Aug 2021 14:46:16 +0000

Have you been hearing about the language Crystal lately and you're curious what it's all about? Well, I'm right there with you! Specifically I'm curious how Crystal can be used to create web applications that are highly performant but use less resources.

A little about me

My name is Seth and I'm a self-taught full-stack Ruby on Rails developer. My strong suites are mostly back-end and infrastructure.

The goal of this article

Since I'm self-taught, and because of the way I learn, I rely heavily on video tutorials to help me build my understanding of how Ruby, Javascript, HTML/CSS and infrastructure work together to create functional web applications.

Crystal caught my attention because of it's similarity to Ruby, but with benefits of a compiled language that I think Ruby is naturally trying to move towards. The challenge I find myself having is... a lack of videos helping introduce the different frameworks and the language.

So, I decided I'll take multiple Crystal web frameworks for a test drive and we'll build the same app in each one to explore how each one is different. I plan to personally highlight the differences I notice between the Crystal framework and Rails because most of my app development experience is with Rails.

Crystal Web Frameworks I'll Be Exploring

Here's the list of frameworks I will be making videos for. I'll be creating the same simple blog app on each framework so that anyone following along can compare how each one. I'll update each list item with a link to the article specifically about the framework with links to the videos and the source code for anyone to clone/fork and play around with. So if you don't see a link yet, please check back in the future 😄

The order here is roughly based on the full featured-ness of the framework. I say that because from my current experience Lucky is more fully featured as an out of the box system, but I'm already working with Amber currently and I'm more familiar with it so I plan to start there. Kemal and Athena are much lighter weight and are better compared to a Sinatra type framework.

Blog Project Outline

I'm rarely this structured, but a simple blog site is a good way to explore a frameworks ORM, controller and view layer, routing, simple user authentication and all of the basic CRUD actions.

Since the purpose of this project is to explore the frameworks themselves I'll be using bootstrap for the front-end and making all of the projects look essentially the same with standard bootstrap styling. If I use any Javascript, it'll be through Webpack and using Stimulus or just plain Javascript (clearly I haven't decided yet).

And there you have it!

I'll update this post as I finish the videos and posts with each framework. If this has helped you, or caused you to have more questions that I didn't answer, please let me know. I'd love to get your feedback!

Amber 1.0.0rc2 & Jennifer

Seth Tucker — Wed, 21 Jul 2021 14:16:54 +0000

{UPDATE}

Since writing this post, I became the owner of the i18n.cr repo and I've released a new version that includes some updates to the crystal version and soon you won't need the directions found here!

Back to the original programming...

Hey there! I was recently trying to get up and running with Crystal 1.0.0+ and the new Amber 1.0.0rc2 with Jennifer 0.10.0, and I ran into a curious issue:

shard name (i18n) has ambiguous sources: 'git: https://github.com/dare892/i18n.cr.git' and 'git: https://github.com/techmagister/i18n.cr.git'.

I have a simple work around that will temporarily let you run the shards install or shards update commands to get your app up and running with Jennifer for your ORM!

In your shards.yml file change these lines:

citrine-i18n:
  github: dare892/citrine-i18n
  version: ~> 1.0.0

i18n:
  github: techmagister/i18n.cr
  branch: master

And set your Jennifer shard config to:

jennifer:
  github: imdrasil/jennifer.cr
  branch: master

Now shards can install and update all of your shards, without any Crystal version error warnings or ambiguous source warnings, and amber watch will compile and run the app in development.

How do I know if my server is big enough? DevOps For Beginners

Seth Tucker — Fri, 18 Sep 2020 15:15:06 +0000

As an intermediate level developer who came from an IT background, I've always enjoyed being able to whip up an application in Rails and then quickly deploy to my own server. There are tons of great tutorials on how to publish a Rails app to production on a single server. But when I was first reading through these tutorials, I was always left with a single question. How do I know I have everything setup to handle growth?

Ultimately, that question left me sifting through the documentation for Phusion Passenger for days of my life, only to discover vague optimization advice. The optimization formulas were good, but not great. After taking my first project from a single server with only 3,000 visitors per day and little to no database usage, up to a load-balanced stack of 3 application servers, a separate database server that servers over 8,000 visitors per day that has multiple environments shared across all the servers and now has business-critical applications running, I decided it was time to share some tips and tricks.

First things first, how powerful does each individual server need to be?

Well, like everything in development, it depends. Woof, I hate that vagueness. So here are a few things to help understand server sizing.

Notice I make some assumptions here. First, you're using Phusion Passenger or a similarly non-multithreaded application server (I'm looking at you Unicorn users). If you're using Puma, Nginx Unit, or a similarly multithreaded application server then you may see better performance with smaller safety margins for high demand.

How much memory will you need?

I hated trying to figure this out. It's very frustrating when you are first getting started with your own server because you don't really know. I also found out through plenty of trial and (painful) error that certain services in Rails spin up an entire copy of the application (I'm looking at you ActionCable).

So here's some sizing information I found extremely helpful.

A single Rails app, doing basic CRUD functionality usually takes up 100-150mb of memory, per instance. I've seen that go up to 500mb when under heavy loads (like website scrapers making hundreds or thousands of requests per second).
ActionCable spins up an entire application instance of Rails that is in addition to your main app. Using ActionCable? That means a single app with ActionCable will take up 200mb-300mb of memory to run a "single" instance of your app. Passenger by default runs 3 instances of your app. So that's an easy 600mb-900mb of memory of minimal usage.
A typical Rails instance can handle hundreds or even a few thousand requests per second by itself. I'm throwing this tidbit of information out there because after seeing the numbers above, you may be like me and have a panic of what if I run out of memory because I get a traffic spike? Don't stress about it too much.

Forumula for determining total server memory needed.

Now that you know the rough values, take this handy-dandy formula and apply it to each stage of your application that you want on your server.

TotalApplicationInstances * 225 = TheoreticalApplicationMemoryNeeded

In the above formula, the TotalApplicationInstances should be the number of Rails services that will be running including each ActionCable instance.

For example:
A typical Passenger app will have 3 primary instances, plus 3 additional services for ActionCable. So our formula would now be:
6 * 225 = 1350

That means we should assume Rails needs 1,350mb of memory to function when under a heavy load evenly across all instances on each individual server. That's around 1,000 requests per minute with basic CRUD and view rendering with an average performance-optimized app.

I typically like to have 2x the Rails app's memory available for all of the system processes. So that means I would typically aim for a 4gb Digital Ocean droplet that has 2 virtual CPUs for about $20/month. Not bad!

Now if I'm making a cluster for an actual production site, I'd have this spread across 3 servers, bringing my monthly cost to around $60 for my servers.

In fact, that's exactly how I've set up my latest project. 1 Digital Ocean managed load balancer, that round-robin distributes traffic to my web servers ($15/month, that's a no-brainer). I have 3 application servers that sit right at 30% or less CPU usage and under 40% memory usage during peak hours of the day (that's with around one hundred active users on our site at a time), and then we have a single managed Postgres database that all of the application servers connect to. The database is radically oversized, running at like $60/month.

So for around $150/month, my application serves over 180,000 website visitors, and my servers include both production and staging environments on a single cluster (separated of course). Not bad! It isn't my business, but I can tell you it employs 10 full-time employees with about another dozen near-fulltime contractors.

For anyone trying to wrap their head around infrastructure costs and scaling, I hope this really helps bring to light just how powerful a small server cluster can be! Don't fear your infrastructure, embrace it :)