DEV Community: Matias Salles

Building a NES Emulator from Scratch: The Book (Crystal)

Matias Salles — Sun, 22 Mar 2026 17:23:41 +0000

In my last post I mentioned I'd spent a few months writing a book about building a NES emulator from scratch. Well, it's done. And I want to tell you about it, because I think the journey from "I wonder how emulators work" to "I wrote a 280-page book about it" is kind of ridiculous and worth sharing.

How we got here

It started, as most bad decisions do, at 2 AM. I was playing Mario Bros in a browser emulator, died in world 2-3, and instead of going to sleep like a normal person, I started wondering how the emulator worked. A few weeks later I had a working emulator in Crystal running at 60 FPS. A few months after that, I had a book.

The thing is, while building the emulator I kept thinking: "I wish someone had explained this to me step by step." The NES Wiki is incredible but dense. YouTube tutorials assume you already know C and have opinions about memory allocation strategies. I wanted something that started from zero and built up piece by piece, with code first and theory after.

So I wrote that thing.

What's in the book

You start with a CPU that can't do anything. Literally nothing. Then you teach it to load a number into a register. Then to add. Then to jump. By the end you have all 151 instructions of the 6502 processor implemented, a PPU that renders pixels, an APU that generates audio, and you're playing Super Mario Bros.

Here's the full roadmap:

Chapters 1-2: NES architecture overview + Crystal setup
Chapter 3 (7 sub-chapters): The entire 6502 CPU, all 151 opcodes
Chapter 4: Cartridge parsing, iNES format, Mapper 0
Chapter 5 (6 sub-chapters): PPU, SDL2 GUI, background rendering, sprites, scroll
Chapter 6: Plugging in real games and watching them run
Chapter 7: APU, generating audio with square, triangle and noise waves
Appendix: Mapper 1 (MMC1) for games like Zelda and Mega Man 2

The whole thing is written in Crystal, which if you know Ruby you basically already know. No C, no emulation libraries. Just you, a text editor, and 20,000 virtual transistors.

Let me show you what I mean

The best way to explain the book's approach is to show you some actual code from it. Let's look at how the CPU works.

The CPU has registers (tiny pieces of memory inside the chip), a program counter that tracks where we are in the code, and a step method that fetches the next opcode and executes it:

# src/nes/cpu.cr

getter a      : UInt8   # Accumulator
getter x      : UInt8   # X register
getter y      : UInt8   # Y register
getter sp     : UInt8   # Stack Pointer
property pc   : UInt16  # Program Counter
getter status : UInt8   # Flags (Zero, Negative, Carry, etc.)

def step
  opcode = fetch_byte

  case opcode
  when CODE_LDA_IMMEDIATE   then op_lda_immediate
  when CODE_LDA_ZERO_PAGE   then op_lda_zero_page
  when CODE_LDA_ABSOLUTE    then op_lda_absolute
  when CODE_LDA_ABSOLUTE_X  then op_lda_absolute_x
  # ... STA, LDX, LDY, ADC, SBC, JMP, branches ...
  when CODE_INX             then op_inx
  when CODE_NOP             then op_nop
  else raise UnknownOpcode.new(opcode)
  end

  CYCLES[opcode]
end

That's the whole CPU. Fetch a byte, match it against 151 opcodes, execute the right method, return how many cycles it took. The giant case statement looks intimidating at first, but each instruction is just a few lines.

Let's zoom into one. Every NES instruction has an opcode, a number that tells the CPU what to do. When the CPU reads 0xA9 from memory, it knows it has to run LDA (Load Accumulator) in immediate mode:

# src/nes/cpu/instructions/lda.cr

def lda(value)
  @a = value

  set_z_flag(@a)
  set_n_flag(@a)
end

def op_lda_immediate
  value = fetch_byte
  lda(value)
end

Read a byte, put it in register A, update the flags. That's it. The lda method is reusable across all 8 addressing modes, each one just resolves the address differently:

def op_lda_zero_page
  address = address_zero_page
  value = read_byte(address)
  lda(value)
end

def op_lda_absolute
  address = address_absolute
  value = read_byte(address)
  lda(value)
end

# ... and so on for all 8 modes

See the pattern? Once you implement one instruction family, the rest follow the same structure. The book shows you a few in detail, you implement 10-15 yourself to really internalize how the CPU works, and then you grab the rest from the repo. No one needs to hand-type 151 opcodes.

The part that blew my mind: the PPU

The CPU is satisfying to build, but the PPU is where things get wild. The NES draws an entire screen with 2KB of RAM. Two kilobytes. Your average email is bigger than that.

The PPU (Picture Processing Unit) is a separate chip that runs 3 times faster than the CPU and has its own memory. It draws the screen scanline by scanline, 256x240 pixels, 60 times per second. The book walks you through it step by step: first a black screen, then the background, then sprites, then scroll. Each chapter adds one layer and you can see the progress on screen.

When Mario's title screen showed up for the first time, I just sat there staring at it for a good minute. And then I pressed Start and nothing happened because of a missing feature called sprite 0 hit (in the book I'll tell you all about it). Classic.

The emulation loop

Maybe my favorite part of the whole emulator is how simple the core loop is:

def step
  cycles = @cpu.step
  (cycles * 3).times { @ppu.step }
  @apu.step(cycles)
  cycles
end

Three lines. The CPU executes one instruction and tells you how many cycles it took. The PPU runs 3 times as fast (that's the real hardware ratio). The APU keeps up. That's the entire emulation loop. Everything else is just implementing the details behind each .step.

Play it right now

I compiled the emulator to WebAssembly so you can play it in your browser. No downloads, no setup. Just pick a game and go:

👉 emulator.matiassalles99.codes

Fair warning: you might lose an hour. I sure did while "testing" it.

Get the book

The book is available in both English and Spanish on Leanpub:

🇬🇧 English: Building Your First Emulator
🇪🇸 Español: Construí tu Primer Emulador

Both versions include a free sample that covers the introduction, NES architecture, setup, and the first coding chapter where you build the CPU skeleton and implement your first two instructions. That's enough to know if the book is for you.

If you know Ruby, Python, or any similar language, you can follow along. Crystal reads almost exactly like Ruby, and the book doesn't assume any knowledge of emulation or retro hardware.

Who is this for?

Honestly? Any programmer who's ever been curious about what happens below the abstraction layers we work on every day. You don't need to know anything about emulation, hardware, or assembly.

But it's especially for people like me: web developers who spend all day in Rails or React and sometimes wonder what a CPU actually does when it runs our code. If you're curious about that side of things, I also built a 4-bit CPU on breadboards a while back.

It's also not a reference manual. It's informal, opinionated, and occasionally self-deprecating. I wrote it the way I'd explain things to a friend over coffee (or beer, depending on the chapter).

What's next

I have a few more side projects lined up that I'll be building in Crystal. The language has been a revelation for anything that needs real performance but where I don't want to leave Ruby's syntax behind. I'll keep writing about them here.

If you end up building the emulator, playing the web version, or reading the book, I'd love to hear about it. Drop me a message, open an issue, whatever works.

Now if you'll excuse me, I need to go beat world 2-3.

Crystal: The Language I Didn't Know I Needed

Matias Salles — Sun, 01 Mar 2026 17:23:41 +0000

How I discovered Crystal while building a NES emulator in Ruby, and why this compiled language with Ruby-like syntax became my go-to for performance.

A few months ago I was searching "Mario Bros NES online" at 2 AM. Pure nostalgia. As a kid I used to play that game at my grandparents' house because they had nothing else installed on their PC. I knew all the warps, the infinite lives trick in 3-1, the whole deal.

I found an emulator, played for 15 minutes, died in world 2-3 as always, and closed the tab. But something stuck with me: how hard is it to code one of these?

I'd always been fascinated by the idea of virtual machines, how you can emulate a bunch of machines inside your own hardware. I didn't think it was black magic, but I'd never actually looked at emulator code before. What does it even mean, in code terms, to emulate a console? A lot of assembly? I had no idea.

So after playing I went down a rabbit hole: YouTube tutorials about the NES, reading the NES Wiki, trying to understand the architecture. And then I started coding the CPU, since that's what I felt I understood best from my brief research. It helped that I'd built half of a 4-bit CPU using breadboards back in college, so I had a rough idea of what a CPU involves: registers, ALU, flags, the whole thing.

Starting in Ruby

I haven't touched C or C++ since college, and the YouTube guys building NES emulators are like 100 times smarter than me when it comes to emulation. So I went with Ruby, the language I use every day and know inside out. My goal was to understand how emulation works, not to fight with pointers I haven't thought about in years.

So I started, and as soon as I had 10 instructions implemented I was hooked. There's something about emulation that feels like you're touching hardware without actually touching it. Plus, testing individual instructions by writing tiny 5-instruction assembly programs is a refreshing break from web dev.

After a few days I finished the CPU and moved on to the PPU (Picture Processing Unit). Once that was done (took me about 2 weeks, and that was during the holiday break), I ran Mario, hit Enter to start the game and ... 🥁🥁🥁

Nothing.

It was stuck on the loading screen.

I Spent another 2 days debugging until I gave up and went to cook something. When I came back, there it was. World 1-1 had somehow loaded while I was gone and Mario was just standing there. I pressed the arrow key and...

0.5 FPS. Half a frame per second. Mario was practically a slideshow. That also explained why world 1-1 had taken about 10 minutes to load.

The Search for a Faster Language

So I started looking at alternatives:

Go? Cool, but pretty different from what I'm used to.
Rust? Same deal.
C? The obvious candidate, but that's exactly what I was trying to avoid.

I kept asking ChatGPT for suggestions. At one point it mentioned Crystal, but I brushed it off because I confused it with Carbon (Google's supposed C++ replacement, remember that?).

After asking like 700 times, even the AI got tired of throwing random languages at me and circled back to Crystal, this time really hammering the point that it was very similar to Ruby.

So I looked it up and...

Bam #1: It was made in Argentina 🇦🇷. Hold on, what?

Bam #2: The syntax was basically Ruby!

# This is valid Crystal AND valid Ruby
def fibonacci(n)
  return n if n <= 1
  fibonacci(n - 1) + fibonacci(n - 2)
end

puts fibonacci(35)

Bam #3: I rewrote everything I had in Crystal, ran the emulator, and it was doing 120 FPS. Mario went from slideshow to creatine. Same code, 240 times faster.

I capped it at 60 FPS and it ran perfectly.

So What's Crystal?

Crystal is a compiled, statically typed language with Ruby's syntax. It compiles to native code through LLVM, so you get C-like speeds. Quick rundown:

Static typing with inference

You can add type annotations, but Crystal infers most of them:

name = "Matias"
age = 26

# But you can be explicit when you want
def greet(name : String) : String
  "Hello, #{name}!"
end

The compiler catches type errors before your code runs. No more NoMethodError at 3 AM in production.

Compiled to a single binary

No runtime, no bundle exec, no interpreter. You build it and ship it:

crystal build app.cr --release
./app  # done

C interop out of the box

You can call C libraries directly, which is great for things like SDL2 (graphics), audio libraries, or anything with a C API. I used this a lot for the emulator's GUI and audio.

Macros instead of runtime metaprogramming

Ruby has define_method, method_missing, all those runtime tricks we love. Crystal swaps most of that for compile-time macros:

macro define_getter(name, type)
  def {name} : {type}
    @{name}
  end
end

class User
  define_getter name, String
  define_getter age, Int32
end

Different flavor, same idea. Zero runtime overhead.

What's great

Zero learning curve if you know Ruby. I was productive on day one.
It's fast. My emulator renders frames, processes audio, and emulates a full 6502 CPU at 60 FPS without breaking a sweat.
Error messages are helpful. The compiler tells you exactly what's wrong and where.

What could be better

Compile times can be slow for larger projects. You miss Ruby's instant feedback sometimes.
Smaller ecosystem. No Crystal equivalent for every Ruby gem. You'll sometimes write your own stuff.
Windows support is still catching up. Works great on macOS and Linux though.

Wrapping up

After finishing the emulator I spent a few months writing a book about building one from scratch, so anyone can get a taste of what emulation is about and step away from the web dev grind for a bit. If I could do it, you can too. I'll write about that in another post.

Crystal is a language I plan to keep using for low-level side projects I have coming up, and I'd genuinely recommend it to any Ruby dev who wants to do something closer to the metal, whether that's emulation, audio, graphics, or anything where you need real performance and direct hardware access but don't want to leave Ruby's syntax behind.

Shoutout to the folks at Manas for creating such a cool language, and saving me from having to rewrite everything in C!

You can check out Crystal at crystal-lang.org and their getting started guide. The language reference is excellent.

Save Money on Heroku: Share One Database Across Multiple Apps

Matias Salles — Thu, 07 Aug 2025 17:23:41 +0000

Save money on Heroku by sharing one Postgres database across multiple apps. Step-by-step guide using schema_search_path for Rails developers.

Hey!
This one's going to be a short one, but hopefully useful to some of you out there.

It’s been a while since Heroku announced the end of free dynos and database plans, but it’s still one of the best price/value platforms for deploying proof of concepts, staging environments, and personal projects.

Wait… how?

They introduced a new Eco Dyno plan, a personal subscription for $7/month that gives you 1,000 hours to use across all your Heroku apps. Pretty neat!

But what about the database?

The cheapest plan for Heroku Postgres is $5/month, which doesn’t sound like much… until you have 10 staging apps running. That’s:

$5 * 10 apps + $7 (dyno) = $57/month

For apps you barely use and only access once every few weeks or months? Yeah, not ideal.

What if you could reuse a single DB across all your apps?

This simple trick lets you run as many Heroku apps as you want for just $12/month, and it’s been super helpful for me.

I run several proof of concepts, staging environments for projects I built for friends, my wife’s sports studio staging website, and even an old college final project that my friend and I still maintain to test small changes the company we built it for occasionally requests.

Let’s walk through how to do this. I’ll show the example using Ruby on Rails, but the same idea applies to other frameworks too.

Step-by-step: Share a single Heroku Postgres across apps

Use schema_search_path in your database.yml

In your Rails app, set the schema_search_path in config/database.yml like so:

production:
  <<: *default
  database: <%= ENV["DATABASE_URL"] %>
  schema_search_path: <%= ENV["DATABASE_SCHEMA"] || "public" %>

This tells Rails to use a specific schema for this app, falling back to "public" if nothing is set.

Update your Heroku app’s environment variables

After deploying your app to Heroku, it may automatically provision a new database. Let it finish, then delete it.

Now go to the app where your main Heroku Postgres addon lives, and copy its DATABASE_URL.

Paste that into the DATABASE_URL of your other apps.

Then, add a new environment variable:

DATABASE_SCHEMA=your_schema_name

I usually just set it to the app’s name.

Manually create the schema in the shared database

Your deploy will likely fail at first, that’s expected! The schema doesn’t exist yet.

Open the Heroku Postgres addon in the dashboard, grab the credentials, and connect using a tool like TablePlus or PgAdmin.

From there, just create a new schema with the same name you used in DATABASE_SCHEMA.

Release and deploy!

Once the schema is in place, trigger a new release, and you’re done 🎉

You just saved yourself $5/month, and the next time you do this, it'll be $10/month, then $15/month, and so on 💸

⚠️ Gotchas

If Heroku rotates the DATABASE_URL for maintenance, you’ll need to update it in the other apps that don't have the addon directly attached.
Aside from that, you're good!

🪄 Alternatives: Kamal and self-hosting

If you're a Rails developer, you’ve probably heard about Kamal, it's been out for a few months now, and it’s awesome. It makes it really easy to deploy apps to virtual machines on AWS, Hetzner, DigitalOcean, etc.

If you're not using AWS's 1-year free tier, you're probably looking at $8–12 per app/month, and that’s if you set up the DB as a Docker container yourself.

If you use managed services like RDS, that might cost an extra ~$16/month, even if you reuse the same DB across apps 😬

But here's something cool I might write about soon: You can use Kamal to deploy multiple apps on the same server, reducing costs even further.

Thanks for reading! 😄

Let me know if you end up using this trick — or if you’ve got others to share.

Building a 4-bit CPU with integrated circuits and breadboards

Matias Salles — Tue, 18 Jun 2024 17:23:41 +0000

How I built a 4-bit CPU from scratch with integrated circuits and breadboards, covering the clock, ALU, registers, and RAM.

A few weeks ago, while packing for a move, I stumbled upon an old 4-bit CPU that a friend and I built during our first year of college. For our "Computer System Architecture" class, they didn't give us specific instructions—just that it had to relate to the course topics. My friend and I were fascinated by the inner workings of a CPU and eager to understand how it actually functions. We were learning about Boolean logic, the architecture of a 4x3 RAM chip, and the simplified instruction set of an Intel Pentium chip, but none of this really explained how transistors and logic gates come together to create such a complex piece of engineering like a CPU.

Around that time, electronics engineer and YouTuber Ben Eater posted his first couple of videos on building an 8-bit CPU with integrated circuits and breadboards. The initial videos only covered building a proper clock, but that was enough to hook us. We decided to build our own CPU for the assignment, even though we knew Ben Eater's series wouldn't be finished before our deadline.

Watch the demo video →

Building the CPU

We started by ordering a bunch of integrated circuits from Texas Instruments, since we couldn't find any in Uruguay. These circuits were supposed to help us build the clock. To our surprise, the integrated circuits didn't fit in the breadboard—they were five times smaller than the ones we needed (yeah, and five times smaller than what they showed in the pictures 😐).

After visiting every electronics shop in Uruguay, we finally found a small, old shop that sold the type of chips we needed.

Clock

You might think it would have been smooth sailing from there, but unsurprisingly, they didn't have the exact models we needed. 🤦‍♂️ The interesting part is, this actually led to our first big lesson: you don't need to copy every step and use the exact materials; you can use what you have and think your way through it. The integrated circuits we bought served the same purpose, but since they were a different model, each pin had a different function. This forced us to read through each chip's documentation instead of blindly following Ben's videos (which we did try at first).

A couple of days in, and we had our clock working! I'm not going to dive into the implementation details for the clock because, first of all, I don't remember everything about it 😂, and more importantly, knowing the internals doesn't contribute much to fully understanding how a CPU works. As long as you know that the clock provides regular energy pulses to the rest of the CPU every few milliseconds (each pulse being a clock cycle), and that an operation is executed in each cycle, you should be fine.

We'll dive into the "operations" that the CPU executes later in the post. Meanwhile if you're curious about the internals of a CPU clock, you can watch this video from Ben Eater.

ALU, Registers and a simple BUS

After building the clock, we moved on to the ALU and the registers. The ALU (Arithmetic Logic Unit) is the part of the CPU that performs arithmetic and logical operations, while the registers are memory units that store the data the ALU operates on.

This is where we decided to make our CPU a 4-bit CPU. The old store had limited chip options, and we couldn't afford to order new ones from Texas Instruments without knowing if they would be suitable. We could have chained 4-bit registers and ALUs to create an 8-bit CPU, but we opted to keep things simple and start small, knowing we could always expand later.

Here are some key insights we gained while building the ALU and the registers:

Register Control: You can control when to load data into a register by changing the voltage of a pin. This was our first glimpse into how you can actively manage the flow of data in a CPU, demonstrating that loading data from the BUS isn't just some magical, automatic process.
ALU Operations: For the ALU, you can dictate the operation it performs by altering the voltage of a specific pin. Our simple 4-bit ALU could only handle addition and subtraction, controlled by the voltage at a single pin. If it supported more operations, additional pins would be involved.
BUS Functionality: Data can also be output from the registers and ALU to the BUS. Here, we realized that the BUS acts like the CPU's highway, connecting various components that can read from and output data to it.

After assembling the ALU and the registers, we set them up so that the ALU would output results to the BUS, and register A would read the data from it, enabling us to perform multiplication tables!

Watch the demo video →

RAM

The last component we managed to assemble, which isn’t featured in the video, was the RAM. It comprised several small RAM chips, each capable of holding 4 bits of data. We manually loaded data into these chips using DIP switches.

While the RAM itself might not seem particularly exciting, it’s important to understand its function in a CPU system. RAM is used to store the results of operations performed by the ALU as well as the instructions that the CPU executes. This makes it a critical component, serving as the storage and retrieval point for both data and operational commands.

CU, where the magic happens

The last piece of the puzzle was the CU (Control Unit). Although we attempted to build it, we ran out of time and didn’t have a detailed guide or tutorial to follow, as Ben's series was only beginning to explore the ALU when we reached this stage.

The CU is where terms like "Instructions," "Machine Code," and "Assembly" transition from abstract concepts into practical components of computing. Recall our earlier discussion about the Registers, ALU, and BUS? The CU is responsible for controlling the voltage of specific pins to manage the loading and outputting of data to and from the BUS. This central role in coordinating how the CPU processes instructions makes the CU a fundamental element where much of the computational 'magic' occurs.

But how does it do it?

The CU connects to every "control" pin, thus controlling the operations of the ALU, including whether registers should load data from the BUS. This is where you define your CPU’s instruction set. For example, "0001" might be set to instruct the CPU to "load data from the BUS into Register A," triggering the appropriate voltage change in the register’s control pin to perform the operation. Similarly, "0010" or another binary code could be assigned for different instructions.

This variability is why there are many different Assembly languages, as their instruction sets are defined arbitrarily based on each CPU’s architecture.

By encoding your instruction set in an EEPROM, the CU is programmed to deliver specific control signals to other components based on the instruction codes it receives ("0000", "0001", etc.). This is crucial for directing the precise behavior of the CPU during each operation.

Now, the CU receives instructions from the same BUS that supplies data to other components. In a 4-bit CPU, the limited number of bits presents a challenge: ideally, you use all bits for data or instructions at different times, rather than permanently dividing the bus. Implementing a strict division (2 bits for data, 2 for instructions) severely restricts operational flexibility, limiting you to just 4 distinct operations.

At this point, we decided to conclude our project. While most of our peers had only built simple systems using Arduinos and sensors, we believed our project would earn a good grade. More importantly, we had gained a deep understanding of how a CPU works, satisfying our curiosity.

Conclusion

If you're an electronics engineer, you might find this post a bit basic or lacking in detail—and that's exactly the intention. As software engineers, we rarely get a firsthand look at the hardware we work on. Peeling back some of the "magic" behind a CPU is a fantastic way to spark curiosity and encourage others to explore this fascinating subject further.

I definitely recommend watching Ben Eater's complete series. After our assignment was done, I watched it to see how he addressed the issues with the CU that we couldn't figure out. Let me tell you, he is a genius. If you're interested, you can check out his series here.

Active Record's Magic 🪄: How Rails Defines Getters and Setters from Your Schema

Matias Salles — Fri, 10 May 2024 17:23:41 +0000

A deep dive into how ActiveRecord defines getter and setter methods from your database schema using Ruby metaprogramming and the AttributeMethods module.

Have you ever wondered how ActiveRecord, the ORM behind Ruby on Rails, functions? ORMs are crucial tools that simplify database interactions in web applications, yet many developers use them without understanding their inner workings.

Why is that? Well, as long as it works and gets the job done there is no good reason to dive into such a complex piece of software, ...

Even though I agree on that one, curiosity gets the best of us, and I always used to wonder: How does ActiveRecord manage to define getter/setter methods based on our DB Schema?

This might seem like a basic question, but it's fundamental to how we use ActiveRecord daily. This "harmless" question is a good one to start understanding how ActiveRecord internals work, and how some of those Ruby metaprogramming features we talked about in the previous post come in handy here too 💪.

🏊‍♂️ into ActiveRecord::Base

You may have observed that models/application_record.rb typically contains minimal code and inherits from the ActiveRecord::Base class. Inside ActiveRecord::Base you'll see a lot of include and extend calls on different modules that play a role in how ActiveRecord works:

module ActiveRecord
  class Base
    extend ActiveModel::Naming
    extend ActiveSupport::Benchmarkable
    extend ActiveSupport::DescendantsTracker

    extend ConnectionHandling
    extend QueryCache::ClassMethods
    extend Querying
    extend Translation
    extend DynamicMatchers
    extend DelegatedType
    extend Explain
    extend Enum
    extend Delegation::DelegateCache
    extend Aggregations::ClassMethods

    include Core
    include Persistence
    include ReadonlyAttributes
    include ModelSchema
    include Inheritance
    include Scoping
    include Sanitization
    include AttributeAssignment
    include ActiveModel::Conversion
    include Integration
    include Validations
    include CounterCache
    include Attributes
    include Locking::Optimistic
    include Locking::Pessimistic
    include Encryption::EncryptableRecord
    include AttributeMethods
    include Callbacks
    include Timestamp
    include Associations
    include SecurePassword
    include AutosaveAssociation
    include NestedAttributes
    include Transactions
    include TouchLater
    include NoTouching
    include Reflection
    include Serialization
    include Store
    include SecureToken
    include TokenFor
    include SignedId
    include Suppressor
    include Normalization
    include Marshalling::Methods

    #...
  end
end

That's a lot of modules 😱! We'll only dive into a couple of them though; after skimming through the code, I found that the getter/setter method definitions based on our DB Schema take place in the ModelSchema and AttributeMethods modules.

AttributeMethods Module

Let's first dive into the AttributeMethods module and suppose we already know what the columns in our DB tables are. The first couple of things that caught my eye when digging into this module were the following:

module ActiveRecord  
  module AttributeMethods
    extend ActiveSupport::Concern
    include ActiveModel::AttributeMethods

    included do
      initialize_generated_modules
      include Read
      include Write
      include BeforeTypeCast
      include Query
      include PrimaryKey
      include TimeZoneConversion
      include Dirty
      include Serialization
    end

    class GeneratedAttributeMethods < Module
      include Mutex_m
    end

    #...

    module ClassMethods
      def initialize_generated_modules
        @generated_attribute_methods = const_set(:GeneratedAttributeMethods, GeneratedAttributeMethods.new)
        private_constant :GeneratedAttributeMethods
        @attribute_methods_generated = false
        @alias_attributes_mass_generated = false
        include @generated_attribute_methods

        super
      end

      #...
    end
  end
end

First of all, this module heavily relies on ActiveModel::AttributeMethods, which gives us methods that you might be familiar with, such as: attribute_method_prefix, alias_attribute, and more.

It's important to remember that the ActiveModel's API was extracted from the ActiveRecord component to allow us to create POROs with "model-like" behavior (validations, type casting, etc.).

Then, you can see a bunch of sub-modules included that self-describe their purpose, such as Read (for reader/getter methods), Write (for writer/setter methods), PrimaryKey, Dirty (for tracking changes), etc.

And finally, you can see a method initialize_generated_modules that gets called as soon as the module is included, where it sets up an empty GeneratedAttributeMethods module into the @generated_attribute_methods variable. Inside this GeneratedAttributeMethods module is where all of the getter/setter methods will be defined, and then included in your class.

This pattern of defining an empty module, then dynamically adding methods to it and finally including it is a common approach you'll see in most of Ruby on Rails codebase.

When reading any Ruby open-source code, looking at the "ClassMethods" module within the module you're looking at might be helpful, since usually the "API" or "DSL" methods live in there.

But how does it work!?

Now that we know the basic structure, let's dive into the actual code that defines the getter/setter methods based on our DB Schema.

module ActiveRecord
  module AttributeMethods
    module ClassMethods
      # Generates all the attribute related methods for columns in the database
      def define_attribute_methods
        #...
        super(attribute_names)
      end

      #...

      # Returns an array of column names as strings if it's not an abstract class and
      # table exists. Otherwise it returns an empty array.
      # Person.attribute_names
      # eg: => ["id", "created_at", "updated_at", "name", "age"]
      def attribute_names
        @attribute_names ||= if !abstract_class? && table_exists?
          attribute_types.keys
        else
          []
        end.freeze
      end
    end
  end
end

The super call ends up invoking the define_attribute_methods method that lives within ActiveModel::AttributeMethods, let's take a look at it:

module ActiveModel
  module AttributeMethods
    included do
      class_attribute :attribute_method_patterns, instance_writer: false, default: [ ClassMethods::AttributeMethodPattern.new ]
    end

    #...

    module ClassMethods
      #...

      def define_attribute_methods(*attr_names)
        #...
        attr_names.flatten.each do |attr_name|
          define_attribute_method(attr_name)
        end
      end

      def define_attribute_method(attr_name)
        attribute_method_patterns.each do |pattern|
          method_name = pattern.method_name(attr_name)
          generate_method = "define_method_#{pattern.proxy_target}"

          send(generate_method, attr_name.to_s, owner: owner)
        end
      end

      #...
    end

    #...
  end
end

So what's happening here? The ActiveModel::AttributeMethods uses the class_attribute helper inside the included do block which we saw in the previous post, it defines a class attribute where we'll save the "attribute method patterns", we'll come to this later.

Then, we can find the method we were looking for: define_attribute_methods which goes through each attribute name and calls define_attribute_method for each one of them.

This last method goes through each pattern available in the class attribute defined above and dynamically decides what method to call. This "dynamically deciding" will make sense when looking at the AttributeMethodPattern and how it is used:

class AttributeMethodPattern # :nodoc:
  attr_reader :prefix, :suffix, :proxy_target, :parameters

  def initialize(prefix: "", suffix: "", parameters: nil)
    @prefix = prefix
    @suffix = suffix
    @proxy_target = "#{@prefix}attribute#{@suffix}"
    @method_name = "#{prefix}%s#{suffix}"
    #..,
  end

  def method_name(attr_name)
    @method_name % attr_name
  end

  #...
end

If we only take into account the default AttributeMethodPattern that was added inside ActiveModel::AttributeMethods with no prefix or suffix, when calling generate_method = "define_method_#{pattern.proxy_target}" it will return "define_method_attribute":

module ActiveModel
  module AttributeMethods
    included do
      class_attribute :attribute_method_patterns, instance_writer: false, default: [ ClassMethods::AttributeMethodPattern.new ]
    end

    module ClassMethods
      #...

      def define_attribute_method(attr_name)
        attribute_method_patterns.each do |pattern|
          method_name = pattern.method_name(attr_name)
          generate_method = "define_method_#{pattern.proxy_target}"
          # ============================================================
          # ==== generate_method will be "define_method_attribute" =====
          # ============================================================

          send(generate_method, attr_name)
        end
      end
    end
  end
end

So when we use send to call that method, who will answer to define_method_attribute(attr_name)?

The answer is...... 🥁🥁🥁🥁 The ActiveRecord::AttributeMethods::Read module that we mentioned above!

module ActiveRecord
  module AttributeMethods
    # = Active Record Attribute Methods \Read
    module Read
      extend ActiveSupport::Concern

      module ClassMethods # :nodoc:
        private

        def define_method_attribute(name, owner:)
          "def #{name}" <<
          "  _read_attribute(#{name}) { |n| missing_attribute(n, caller) }" <<
          "end"
        end
      end

      #...

      def _read_attribute(attr_name, &block)
        #...
      end
    end
  end
end

And there we have getter/reader methods defined for each attribute! But what about the setter/writer methods 🤔? It's as simple as adding a new attribute_method_pattern 😃! Let's look at the ActiveRecord::AttributeMethods::Write module:

module ActiveRecord
  module AttributeMethods
    module Write
      extend ActiveSupport::Concern

      included do
        attribute_method_suffix "=", parameters: "value"
      end

      module ClassMethods
        private

        def define_method_attribute=(name)
          "def #{name}(value)" <<
          "  _write_attribute(#{name}, value)" <<
          "end"
        end
      end

      def _write_attribute(attr_name, value)
        #...
      end
    end
  end
end

By adding a new attribute_method_pattern with a suffix of = ActiveModel::AttributeMethods will call define_method_attribute= when iterating over that specific method pattern!

I encourage you to look at the other sub-modules inside ActiveRecord::AttributeMethods since you'll see they follow the same pattern:

module ActiveRecord
  module AttributeMethods
    included do
      initialize_generated_modules
      include Read
      include Write
      include BeforeTypeCast
      include Query
      include PrimaryKey
      include TimeZoneConversion
      include Dirty
      include Serialization
    end
  end
end

When do the getter/setter attribute definitions take place?

If you paid close attention, you may have noticed that we never showed who calls the initial define_attribute_methods method that starts the whole process.

Here's where Railties comes into play, Railties "ties" all of the Rails components together, and it's responsible for orchestrating the "initialization" of a Rails application.

module ActiveRecord
  class Railtie < Rails::Railtie 
    initializer "active_record.define_attribute_methods" do |app|
      config.after_initialize do
        #...

        ActiveSupport.on_load(:active_record) do
          descendants.each do |model|
            model.define_attribute_methods
          end
        end
      end
    end
  end
end

And there you have it! We have now defined all of the getter and setter attributes for our models 💪!

Quick 🏊‍♂️ into ActiveRecord::ModelSchema

So far we supposed we already knew all of the columns in our DB tables, but we'll quickly unravel how this happens under the hood. Let's take a quick look at the load_schema! method inside ActiveRecord::ModelSchema:

module ActiveRecord
  module ModelSchema
    #...

    module ClassMethods
      def load_schema!
        columns_hash = connection.schema_cache.columns_hash(table_name)
        # ...
      end

      #...
    end
  end
end

It's pretty clear that based on a db connection and a table name it will fetch all of the columns in it. At first, I thought loading the schema would involve reading the famous db/schema.rb file, but it turns out the schema is loaded by querying the DB itself! For instance, this is how that query looks like for MySQL:

module ActiveRecord
  module ConnectionAdapters
    class AbstractMysqlAdapter < AbstractAdapter
      #...

      def column_definitions(table_name)
        execute_and_free("SHOW FULL FIELDS FROM #{quote_table_name(table_name)}", "SCHEMA") do |result|
          each_hash(result)
        end
      end

      #...
    end
  end
end

Conclusion

And that brings us full circle! We've explored how ActiveRecord defines getter and setter methods aligned with our database schema and how it dynamically loads this schema.

Ruby on Rails codebase is an awesome place to learn more about Ruby metaprogramming features and how to use it to your advantage; I hope this post has given you a glimpse of that! 🎉

How to Create a DSL (Domain-Specific Language) in Ruby

Matias Salles — Sat, 20 Apr 2024 17:23:41 +0000

Learn how to create a Domain-Specific Language (DSL) in Ruby using metaprogramming, define_method, class_eval, and ActiveSupport::Concern.

After completing the first version of a website for a friend, I dedicated some time to extract an API client I had developed for connecting with dlocalGo's API. dlocalGo is a payment processor built by dlocal, a company like Stripe, but that operates mainly in South America and other emerging markets.

Since this was my first time creating an open-source gem and I wasn't on a rush to publish it, I took the liberty to play around with some Ruby metaprogramming features while at it. After playing around for a while, I realized that creating my own mini-DSL to easily add support for new endpoints with little-to-no code was easier than I initially expected.

I leveraged Ruby's metaprogramming capability, particularly the define_method, which allows for dynamic definition of new methods. Down the road, I also ended up digging into other Ruby features such as class_eval, instance_eval, multiline strings with <<-, Module's lifecycle and more. This also encouraged me to dig into some of ActiveSupport's APIs and discovered some awesome helper methods like class_attribute and how ActiveSupport::Concern actually works

v0.1

When starting the website we only needed to support one-off payments and refunds, so I created a simple class with a method per endpoint:

module DlocalGo
  class Client
    # ...

    def create_payment(params = {})
      uri = "/v1/payments"
      # ...
    end

    def get_payment(payment_id)
      uri = "/v1/payments/#{payment_id}"
      # ...
    end

    def create_refund(params = {})
      uri = "/v1/refunds"
      # ...
    end

    # ...
  end
end

This approach soon proved inadequate as we needed to incorporate subscriptions and recurring payments. By following the same approach we would've ended up with 20 different repetitive methods and a bloated class with 600+ lines of code.

On our way to v1.0 🏎️

The main idea was to create a class method that when being called would generate a method for a specific endpoint based on its URI, HTTP method and response model. It should look something like:

module DlocalGo
  class Client
    # ...

    endpoint :create_payment, uri: "/v1/payments", verb: :post, dto_class: Responses::Payments

    # ...
  end
end

# Usage

client = DlocalGo::Client.new
response = client.create_payment({param1: "1", param2: "2"})
# response is a Responses::Payment object

This is where define_method comes in handy.

I already had experience combining class methods and define_method, but it was always inside Ruby on Rails and using concerns (ActiveSupport::Concern). So when I first tried the following approach in the "dlocal_go" gem I realized "class_methods do ... end" and "included do ... end" don't work! 😭

# This approach doesn't work with plain Ruby

module EndpointGenerator
  class_methods do
    def endpoint(method, uri:, verb:, dto_class:)
      define_method(method) do |params = {}|
        # ...
      end
    end
  end
end

Encountering "method_missing" errors led me to discover that these methods were part of ActiveSupport::Concern. This prompted me to consult the ActiveSupport documentation and delve into the source code for included and class_methods.

It's easy to see in their documentation that class_methods defines class methods from a given block (see docs), but how does it work behind the scenes?

You can navigate to the gem's source code by using gem unpack, gem open or installing extensions in your favorite IDE (although it might not be necessary if you use something like RubyMine). Here you'll find the definition to class_methods

module ActiveSupport
  module Concern
    def append_features(base)
      # ...
      base.extend const_get(:ClassMethods) if const_defined?(:ClassMethods)
      # ...
    end

    def class_methods(&class_methods_module_definition)
      mod = const_defined?(:ClassMethods, false) ?
        const_get(:ClassMethods) :
        const_set(:ClassMethods, Module.new)

      mod.module_eval(&class_methods_module_definition)
    end
  end
end

It checks if there's a constant named ClassMethods defined, if so it keeps it, otherwise it creates a new empty ClassMethods module, then evaluates the block we passed inside the ClassMethods module. Finally ActiveSupport::Concern has its own implementation of the append_features method, a method that gets called when a module is included (see docs). Here, it "extends" base (the class where you included the Module) with all of methods inside the ClassMethods module.

This all makes sense especially when we look how a typical module looks with plain ruby:

module M
  def self.included(base)
    base.extend ClassMethods
  end

  module ClassMethods
    # ...
  end
end

The method included is similar to append_features (see docs).

It's clear that ActiveSupport::Concern gives us a nicer and cleaner API to achieve the same goal. So after getting a better grasp of how modules and their lifecycle works, we can create the first version of our EndpointGenerator Module:

module EndpointGenerator
  def self.included(base)
    base.extend(ClassMethods)
  end

  module ClassMethods
    def endpoint(method, uri:, verb:, dto_class:)
      define_method(method) do |params = {}|
        # ...
      end
    end
  end
end

Parsing the response

🎉 Now we can take a closer look at parsing a succesful response, the first version of the response objects (v0.1) looked more or less like this:

module DlocalGo
  module Responses
    class Payment
      RESPONSE_ATTRIBUTES = %i[id amount etc..].freeze

      attr_reader(*RESPONSE_ATTRIBUTES)

      def initialize(response)
        RESPONSE_ATTRIBUTES.each do |attribute|
          instance_variable_set("@#{attribute}", response.send(attribute))
        end
      end
    end
  end
end

If a response object had an association it would look something like this:

require_relative "subscription_plan"

module DlocalGo
  module Responses
    class Subscription
      RESPONSE_ATTRIBUTES = %i[id country etc...].freeze

      attr_reader(*RESPONSE_ATTRIBUTES)

      def initialize(response)
        (RESPONSE_ATTRIBUTES).each do |attribute|
          instance_variable_set("@#{attribute}", response.send(attribute))
        end

        @plan = SubscriptionPlan.new(OpenStruct.new(response["plan"]))
      end
    end
  end
end

There was clear repetition in all of the response objects, so I thought it would be great if by just defining the attributes and associations with a clean API similar to ActiveRecord's one, all of them could be populated automatically. Something that would look similar to this:

require_relative "subscription_plan"
require_relative "client"

module DlocalGo
  module Responses
    class Subscription < Base
      has_attributes %i[id country etc..]

      has_association :plan, SubscriptionPlan
      has_association :client, Client
    end
  end
end

To achieve this, we needed a way to "save" all of the associations data (attribute name and its corresponding response class) somewhere when has_association was called.

There was a specific method ActiveSupport::Support gives us that I saw all around Rails codebase: class_attribute. It declares a class-level attribute and creates the getter/read and setter/write methods for us. With that in mind, I created a Responses::Base class, which all Response objects will inherit from, and created a new ReponseParser module, in which I planned to include all of the "magic" 🪄

require_relative "response_parser"

module DlocalGo
  module Responses
    class Base
      include ResponseParser
    end
  end
end

module DlocalGo
  module Responses
    module ResponseParser
      def self.included(base)
        base.class_eval { class_attribute :response_attributes, :response_associations }
      end
    end
  end
end

Not sure what class_eval does? Here's a short and interesting article from Stanford that explains class_eval and instance_eval

Now we need to add the ClassMethod module with the methods that will define our second mini-DSL (has_attributes and has_association)

module DlocalGo
  module Responses
    module ResponseParser
      def self.included(base)
        base.class_eval { class_attribute :response_attributes, :response_associations }

        base.extend(ClassMethods)
      end

      module ClassMethods
        def has_attributes(attributes)
          self.response_attributes ||= []
          self.response_attributes << attributes

          class_eval { attr_reader(*attributes) }
        end

        def has_association(attribute, klass)
          self.response_associations ||= {}
          self.response_associations[attribute] = klass

          class_eval { attr_reader attribute }
        end
      end
    end
  end
end

We now save all the information needed to parse the response inside the response_attributes and response_associations class attributes. The only missing piece is to override the initialize method, so we can call a new method that will iterate on the response_attributes and response_associations and populate each one of them.

module DlocalGo
  module Responses
    module ResponseParser
      def self.included(base)
        base.class_eval { class_attribute :response_attributes, :response_associations }

        base.extend(ClassMethods)

        base.class_eval <<-CODE, __FILE__, __LINE__ + 1
          def initialize(response)
            assign_attributes(response)
            assign_associations(response)
          end
        CODE
      end

      module ClassMethods
        def has_attributes(attributes)
          # ...
        end

        def has_association(attribute, klass)
          # ...
        end
      end

      private

      def assign_attributes(response)
        # ...
      end

      def assign_associations(response)
        # ...
      end
    end
  end
end

You might be wondering what does <<- stand for 🤔. It's just a multiline string delimited by the CODE keyword. __FILE__ and __LINE__ + 1 are there for debugging purposes, you can read more about it in Ruby's official documentation.

You'll see this approach in various places inside the Ruby on Rails codebase, class_eval not only works with blocks but also with strings, so instead of defining a method inside of a method (doing def initialize inside self.included(base)), we use class_eval and use a string instead of a block.

Conclusion

That's a wrap! We've explored an extensive range of topics, from ActiveSupport::Concern internals to the lifecycle of modules, along with define_method, class_eval, and multiline strings. We've also looked at helper methods like class_attribute 😄.

So what might be next?

Since we ended up including ActiveSupport as a dependency in order to use the class_attribute method, we might as well refactor these modules into cocerns 😅.
Maybe creating another gem similar to ActiveResource, that would allow you to turn any Ruby class into a HTTP client by just adding endpoint calls like we do inside DlocalGo::Client
I also plan to create a small rails engine in the future were you'll be able to get a simple dlocalGo dashboard with all the payments, subscriptions, and transactions just by mounting it inside your Ruby on Rails app.

Don't know what a rails engine is? That might be a whole new blog post, meanwhile, I encourage you to read the official docs that explains it amazingly