DEV Community: Surya

Never Let Slow Queries Block Your Users: A Smarter Cache Refresh Strategy

Surya — Wed, 02 Jul 2025 16:16:08 +0000

Context

In many systems, some queries are slow. Maybe they hit a large dataset, or they depend on multiple joins or external services. You can cache the results, sure. But when that cache expires, your next unlucky user is stuck waiting for the whole thing to recompute. That’s not ideal.

I had a similar situation. The query was too slow to run live, so I cached the results with a 1-day expiry. But I didn’t want any user even just one to hit the raw query when the cache expired.

So I came up with a simple fix: proactive cache refresh, without background cron jobs or complex scheduling.

The Idea

Instead of setting the cache expiry to 1 day, I set it to 2 days.

But here’s the trick:
When I fetch the cached data, I check how old it is.
If it's close to expiry (say, within 1 day), I kick off a background job to refresh the cache.

So users always get the old response instantly.
And the cache is rebuilt in the background before it expires.

Why This Works

No cron needed: You don't need a separate job scheduler.
No user waits: Even if the cache is stale, users still get a fast response.
Always fresh: The cache is refreshed regularly, just not synchronously.

Example Code (Ruby Version)

class CacheBackedQuery
  TTL = 2.days.to_i
  REFRESH_THRESHOLD = 1.day.to_i

  def initialize(cache_key:, query_proc:)
    @cache_key = cache_key
    @query_proc = query_proc
  end

  def get_data
    cached = Cache.get_key(@cache_key) # Cache can be Redis, Memcached, etc.

    if cached
      parsed = JSON.parse(cached)
      age = Time.now.to_i - parsed['created_at'].to_i

      if age > REFRESH_THRESHOLD
        enqueue_refresh_job
      end

      return parsed['data']
    end

    # This should almost never run
    data = @query_proc.call
    Cache.set_key_and_expiry(@cache_key, { data: data, created_at: Time.now.to_i }.to_json, TTL)
    data
  end

  private

  def enqueue_refresh_job
    RefreshCacheJob.perform_async(@cache_key, Marshal.dump(@query_proc))
  end
end

Usage Example

query = -> {
  User.group(:id)
      .select(:id, 'COUNT(login_count) as login_count')
      .order('login_count DESC')
      .limit(10)
      .map { |u| { id: u.id, count: u.login_count.to_i } }
}

fetcher = CacheBackedQuery.new(
  cache_key: "popular_users:account_#{account.id}",
  query_proc: query
)

result = fetcher.get_data

And in your job file:

class RefreshCacheJob
  def perform(cache_key, serialized_query)
    query_proc = Marshal.load(serialized_query)
    data = query_proc.call
    Cache.set_key_and_expiry(cache_key, { data: data, created_at: Time.now.to_i }.to_json, CacheBackedQuery::TTL)
  end
end

Trade-offs

Slightly stale data: At most, users may see data that’s 1 day old.
Cache churn: You may refresh a few times unnecessarily if traffic is low. Acceptable for most use cases.

When to Use This

✅ Works great for dashboards, analytics, or infrequently changing data.
❌ Not ideal if you need real-time accuracy.

Credits

Big thanks to Shantilal from my team for pairing on this solution. We iterated on the idea together and refined it into a clean, reusable pattern.

Final Thoughts

This pattern is simple, effective, and doesn't need infra changes.
It avoids unnecessary user pain, and doesn’t rely on scheduled background jobs.

Give it a try if you have slow queries that shouldn’t ever run in the foreground.

HTML Escaping & Sanitization: What Belongs in the Backend vs. Frontend?

Surya — Mon, 12 May 2025 12:08:10 +0000

“Should I sanitize user input on the backend, or just escape it in the frontend?”

It’s a deceptively simple question — but one that often separates secure, maintainable apps from brittle, exploitable ones.

In practice, this decision is less about choosing where to handle things and more about being intentional about what needs to be done — and why. Let’s break it down together: when to sanitize, when to escape, and how to divide responsibilities between the frontend and backend.

The Core Idea

Not all user input is created equal. Some values are meant to be raw text (like usernames), while others might intentionally include HTML (like blog posts). How we handle each type depends heavily on context:

Should the input ever be rendered as HTML?
Will the input be stored and reused later?
Is the rendering layer under your control?

With those questions in mind, let’s look at three common cases.

Case 1: Plain Text Only (No HTML Allowed)

📝 Example inputs:

Usernames
Comments (no formatting)
Tags, categories, or metadata
Email subjects or titles

Backend

Store the input as-is — no escaping or sanitization needed.
Keeping data raw helps avoid confusion and over-escaping later.

Frontend

Escape before rendering.
Use your framework’s safe interpolation mechanism:
- {{value}} in Ember
- textContent in plain JS
- JSX in React (escaped by default)

✅ Why?
Escaping on the frontend ensures that even if someone tries to inject HTML or JavaScript into a form field, it’ll be rendered as harmless text, not executable code.

Most modern frameworks default to safe interpolation for a reason — it’s a reliable defense against XSS. You have to explicitly opt in to raw HTML rendering, which helps prevent accidents by default. This default is your friend.

Case 2: Trusted Rich Text (Some HTML Allowed)

📝 Example inputs:

Blog content
WYSIWYG editor output
Markdown rendered to HTML

Backend

Sanitize the HTML before storing it.
Use a whitelist approach — allow only specific tags and attributes like <b>, <i>, <a href="">, etc.
Libraries to consider:
- sanitize-html (Node)
- DOMPurify
- Built-in options in some frameworks

Frontend

Render as raw HTML only if the content has been sanitized upstream.
Use:
- dangerouslySetInnerHTML in React (carefully)
- htmlSafe() in Ember
- v-html in Vue

✅ Why?
Sanitizing at the backend ensures safety before the content enters your system. This way, even if the same HTML is later rendered in an email, mobile app, or web view, it’s already trusted.

Never rely on frontend logic to "catch" bad HTML after the fact — by then, it might be too late.

Case 3: Untrusted HTML (Stored, But Not Meant to Render as HTML)

Sometimes users submit HTML or code — not to render it, but to show it as-is, like in forums or documentation tools.

📝 Example inputs:

Code snippets
Raw HTML examples
Debug or log output

Backend

Store the content raw, unescaped.

Frontend

Escape before rendering: convert <, >, &, " and ' to safe entities.
Wrap the result in <pre> or <code> tags to preserve formatting.

<pre>&lt;div&gt;Hello&lt;/div&gt;</pre>

✅ Why?
You’re not trying to “clean” the HTML here — just make sure it’s displayed as text instead of interpreted as markup. Escaping in the frontend makes that intention clear and safe.

When to Sanitize on the Backend

Use backend sanitization when:

HTML input is allowed and meant to be rendered.
You’re building rich content features (blogs, bios, product descriptions).
You want to catch bad HTML before it ever touches your database.

It’s your first line of defense — sanitize once, reuse safely everywhere.

When Frontend Escaping Is Enough

Rely on frontend escaping when:

The input should never contain HTML.
You’re rendering plain text like usernames or messages.
You want to keep backend logic clean and flexible.

This approach keeps your storage raw and your rendering safe — a good combo in many apps.

🛡️ Golden Rule: Sanitize Early, Escape Late

Layer	Action	Why?
Backend	Sanitize	Remove unsafe HTML before storing it
Frontend	Escape	Prevent unsafe rendering at the UI

Sanitize early → Block malicious input at the source.
Escape late → Safely handle display, even if something slipped through.

This separation of concerns isn’t just about security — it also keeps your codebase predictable and easier to maintain.

Final Thoughts

The “backend vs. frontend” debate isn’t really about turf — it’s about intent and responsibility.

The backend is your gatekeeper. It decides what’s allowed in and ensures safety at the source.
The frontend is your presenter. It shows information to the user and must escape responsibly.

In practice, you'll often need both — sanitize early, escape late. And when in doubt, lean on your frameworks: their defaults are usually built with these concerns in mind.

Have you encountered a different pattern that works well in your stack? I’d love to hear about it.

🍹 Don't Drink the Single Responsibility KoolAid

Surya — Fri, 02 May 2025 12:23:11 +0000

“A class should have only one reason to change.”
You’ve heard it. You’ve probably quoted it. Maybe you even tattooed it on your forearm in solid monospaced font. The Single Responsibility Principle (SRP), one of the SOLID five, is a sacred cow in object-oriented design.

But maybe — just maybe — it’s time to ask the heretical question:

Is SRP always worth following?

Let’s dive into why you shouldn't blindly sip the SRP KoolAid.

☁️ The Theory Sounds Lovely

SRP tells us: a class should do one thing, and do it well.

It’s clean. It’s pure. It gives you a warm feeling of architectural correctness. Your classes become lean little specialists. Your future self will thank you.

But theory has a nasty habit of smacking into the chaos of the real world.

🤹‍♂️ “Responsibility” Is a Vague Word

Let’s get this out of the way: what exactly is a "responsibility"?

In a real-world codebase:

Is generating and exporting a report one responsibility or two?
Is parsing and validating user input one thing or two?
Is managing user authentication and session state too much for one class?

SRP’s core idea hinges on a concept that’s… squishy. Different developers draw the line differently. That makes SRP hard to apply consistently, especially in teams.

🧱 SRP Can Lead to Over-Engineering

Try to rigorously apply SRP and you’ll find yourself drowning in a sea of micro-classes:

ReportFormatter
ReportSaver
ReportMetadataAggregator
ReportFilenameSanitizer
ReportIOManager

You’ve replaced one mildly bloated class with a file system that looks like a Java factory exploded. Reading your code now requires a PhD in class archaeology. Good luck onboarding new developers — or even remembering where anything lives after a week.

Sometimes, it’s better to have one chubby, cohesive class than a scattering of anemic ones.

🛠 YAGNI Says Hello

SRP is often sold as future-proofing. “What if we later need to save the report to the cloud?”
But what if you never do?

You just spent an extra few hours decoupling logic that would’ve been fine living together. In the early phases of a project — when things are changing fast — speed of delivery > architectural purity.

Following SRP too early violates another beloved acronym: YAGNI (You Aren’t Gonna Need It).

🕸️ More SRP, More Glue Code

Ironically, breaking a class apart often means writing more code to stitch things back together.

More interfaces.
More dependency injection.
More orchestration.
More “managers” and “controllers” and “factories.”

Congratulations: you now have smaller classes... and a complex spider web of dependencies. Decoupling doesn’t always make things simpler — it just moves the complexity somewhere else.

🧠 Pragmatism Beats Purity

This isn't a call to throw out SRP entirely. It’s a reminder to apply it with context.

Use it when:

Classes are genuinely doing too much.
Different concerns change at different times.
You’re building a stable, long-lived system that benefits from clean separation.

Ignore it when:

You’re prototyping.
You’re building a small utility or script.
The abstraction adds more overhead than value.

🏁 In Conclusion

SRP is a powerful principle. But like any tool, it can become a blunt instrument when misused. Don’t idolize it. Don’t follow it religiously. And please — don’t drink the SRP KoolAid thinking it’ll solve all your problems.

Code is not art. It’s not math. It’s a messy conversation with the future — and sometimes the clearest message comes from writing code that’s obvious, even if it bends the rules.

Need help refactoring or deciding if you’re overdoing SRP? Ask yourself: if this class changed, what would actually break? And does it matter?

🎭 Save Your Code with Decorators, Proxies, and Delegation

Surya — Fri, 25 Apr 2025 13:51:24 +0000

Introduction

Managing integrations in your app often means dealing with similar objects that have slightly different behaviors. Instead of cluttering your code with conditionals, you can wrap these objects using patterns like decorators, proxies, and delegation.

Let’s walk through a practical example using Ruby and Rails, but the concepts work in any language!

The Problem

Imagine you have a table called installations where you store integrations like Slack, Teams, and SharePoint. Your table might look like this:

CREATE TABLE installations (
  id BIGINT,
  name VARCHAR,
  configs JSON
);

You create records with ActiveRecord like this:

Installation.create(
  name: 'sharepoint',
  configs: {
    tenant_id: '123e4567-e89b-12d3-a456-426614174000',
    sites: ["https://sample.sharepoint.com/sites/welcome"]
  }
)

Installation.create(
  name: 'msteams',
  configs: {
    tenant_id: '123e4567-e89b-12d3-a456-426614174111',
    enabled_channels: ['IT', 'Admin']
  }
)

But now, you need to add specific behavior for each integration. For example, SharePoint needs to handle sites, and Teams needs to handle enabled_channels.

The Bad Solution

You could add these methods directly to Installation, but that creates tight coupling and code duplication:

class Installation < ApplicationRecord
  def sites
    raise NoMethodError if !name.eql? 'sharepoint'
    configs[:sites]
  end

  def sites=(new_sites)
    raise NoMethodError if !name.eql? 'sharepoint'
    configs[:sites] = sites + new_sites
  end

  def enabled_channels
    raise NoMethodError if !name.eql? 'teams'
    configs[:enabled_channels]
  end
end

This approach makes your code hard to maintain and scale.

The Elegant Solution: Decorators, Proxies, and Delegation

Let’s use a cleaner approach: wrapping each integration in a specific class, and delegating shared functionality to the Installation class.

For SharePoint:

class SharepointInstallation
  def initialize(id:)
    @installation = Installation.find_by(id: id)
  end

  def sites
    configs[:sites]
  end

  def sites=(new_sites)
    configs[:sites] = sites + new_sites
  end

  def method_missing(method, *args)
    return @installation.send(method, *args) if @installation.respond_to?(method)
    super
  end
end

Now, you can access and update SharePoint-specific methods:

sharepoint_inst = SharepointInstallation.new(id: 1)
puts sharepoint_inst.sites
sharepoint_inst.sites = ["https://newsite.sharepoint.com"]
sharepoint_inst.save

Adding the Teams Integration

Similarly, create a TeamsInstallation class:

class TeamsInstallation
  def initialize(id:)
    @installation = Installation.find_by(id: id)
  end

  def enabled_channels
    configs[:enabled_channels]
  end

  def enabled_channels=(new_channels)
    configs[:enabled_channels] = enabled_channels + new_channels
  end

  def method_missing(method, *args)
    return @installation.send(method, *args) if @installation.respond_to?(method)
    super
  end
end

Now you can do the same for Teams:

teams_inst = TeamsInstallation.new(id: 1)
puts teams_inst.enabled_channels
teams_inst.enabled_channels = ['Support', 'Marketing']
teams_inst.save

Why This Works

Separation of Concerns: Each integration has its own class.
Easy to Extend: Add new integrations by creating new wrapper classes.
Cleaner Code: No more clutter in the Installation class.
Less Maintenance: New logic for integrations doesn’t affect existing code.

Conclusion

This solution keeps your code clean and scalable. By using decorators, proxies, and delegation, you can add behaviors for different integrations while keeping the core logic simple and maintainable.

Even though this example uses Ruby and Rails, this pattern applies to any object-oriented language, from Python to JavaScript. It’s a universal strategy to wrap complexity and keep your codebase neat!

Generating SQL Insert Statements from ActiveRecord Models in Ruby on Rails

Surya — Mon, 21 Apr 2025 11:08:04 +0000

📌 Overview & Preview

Goal: Convert any ActiveRecord model instance into a SQL INSERT statement — handling hashes, arrays, and enum columns gracefully.

Example Input:

# setup
User.create!(name: "John", role: "admin", preferences: { theme: "dark" }, access_levels: ["read", "write"])

# Get insert statement
User.last.insert_sql

Expected SQL Output:

INSERT INTO users (id, name, role, preferences, access_levels, created_at, updated_at)
VALUES (1, 'John', 1, '{"theme":"dark"}', '["read","write"]', '2025-04-21 10:00:00', '2025-04-21 10:00:00');

📌 Part 1: Generating a Basic SQL Insert Statement for a Specific Model

user = User.last

insert_statement = <<-SQL
  INSERT INTO users (#{user.attributes.keys.join(', ')})
  VALUES (#{user.attributes.values.map { |v| ActiveRecord::Base.connection.quote(v) }.join(', ')})
SQL

puts insert_statement

Problem: If your model has a Hash or Array attribute (like preferences), this raises:

TypeError: can't quote Hash

👉 Next up: handle non-scalar values.

📌 Part 2: Handling Hash and Array Columns

Solution: Serialize them to JSON before quoting.

require 'json'

user = User.last

columns = user.attributes.keys
values  = user.attributes.values.map do |value|
  value = value.to_json if value.is_a?(Hash) || value.is_a?(Array)
  ActiveRecord::Base.connection.quote(value)
end

insert_statement = <<-SQL
  INSERT INTO users (#{columns.join(', ')})
  VALUES (#{values.join(', ')});
SQL

puts insert_statement

✅ Hashes and arrays now safely handled.

👉 Next up: handle enum columns.

📌 Part 3: Handling Enum Columns

Problem: Rails stores enums as integers in the DB but exposes them as strings.

Example:

class User < ApplicationRecord
  enum role: { guest: 0, admin: 1, super_admin: 2 }
end

If user.role is "admin", we need to insert 1 in SQL.

Solution:

enum_columns = User.defined_enums

values = user.attributes.map do |attr_name, value|
  if enum_columns.key?(attr_name)
    enum_mapping = User.send(attr_name.pluralize)
    int_value = enum_mapping[value]
    ActiveRecord::Base.connection.quote(int_value)
  else
    value = value.to_json if value.is_a?(Hash) || value.is_a?(Array)
    ActiveRecord::Base.connection.quote(value)
  end
end

✅ Enum values converted to DB integers.

👉 Next up: make it generic.

📌 Part 4: Building a Generic SQL Insert Generator for Any Model

Solution:

def generate_insert_sql(model_instance)
  model_class = model_instance.class
  table_name  = model_class.table_name
  enum_columns = model_class.defined_enums

  def safe_quote(model_class, attr_name, value, enum_columns)
    if enum_columns.key?(attr_name)
      enum_mapping = model_class.send(attr_name.pluralize)
      int_value = enum_mapping[value]
      ActiveRecord::Base.connection.quote(int_value)
    else
      case value
      when Hash, Array
        ActiveRecord::Base.connection.quote(value.to_json)
      else
        ActiveRecord::Base.connection.quote(value)
      end
    end
  end

  columns = model_instance.attributes.keys
  values  = columns.map do |attr_name|
    value = model_instance.attributes[attr_name]
    safe_quote(model_class, attr_name, value, enum_columns)
  end

  <<-SQL
    INSERT INTO #{table_name} (#{columns.join(', ')})
    VALUES (#{values.join(', ')});
  SQL
end

✅ Works for any model instance.

👉 Next up: make it Rails-native.

📌 Part 5: Extending ActiveRecord::Base for a Reusable Method

Goal: Add an insert_sql instance method to all models via ApplicationRecord.

Solution:

class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true

  def insert_sql
    model_class = self.class
    table_name  = model_class.table_name
    enum_columns = model_class.defined_enums

    def safe_quote(model_class, attr_name, value, enum_columns)
      if enum_columns.key?(attr_name)
        enum_mapping = model_class.send(attr_name.pluralize)
        int_value = enum_mapping[value]
        ActiveRecord::Base.connection.quote(int_value)
      else
        case value
        when Hash, Array
          ActiveRecord::Base.connection.quote(value.to_json)
        else
          ActiveRecord::Base.connection.quote(value)
        end
      end
    end

    columns = attributes.keys
    values  = columns.map do |attr_name|
      value = attributes[attr_name]
      safe_quote(model_class, attr_name, value, enum_columns)
    end

    <<-SQL
      INSERT INTO #{table_name} (#{columns.join(', ')})
      VALUES (#{values.join(', ')});
    SQL
  end
end

✅ Now you can call:

puts User.first.insert_sql

Sample Output:

INSERT INTO users (id, name, role, preferences, access_levels, created_at, updated_at)
VALUES (1, 'John', 1, '{"theme":"dark"}', '["read","write"]', '2025-04-21 10:00:00', '2025-04-21 10:00:00');

puts Department.last.insert_sql

Sample Output:

INSERT INTO departments (id, name, created_at, updated_at)
VALUES (42, 'Engineering', '2025-04-20 12:00:00', '2025-04-20 12:00:00');

Bonus: Works across your entire app — clean, DRY, and Rails-idiomatic.

📌 Final Thoughts

We went from:

🔸 Hardcoding a single model (User)
🔸 Handling hashes and arrays (preferences, access_levels)
🔸 Handling enum columns (role)
🔸 Building a generic function for any model
🔸 Extending ActiveRecord elegantly with insert_sql

Hybrid Pagination in Rails: Paginating Across Two Data Sources

Surya — Mon, 07 Apr 2025 14:22:56 +0000

Pagination is simple—until it isn’t.

In most apps, paginating one dataset is easy with tools like kaminari or will_paginate. But what if you need to paginate across two datasets, giving one higher priority, and use the second only when needed?

That’s where hybrid pagination comes in. Here's how I built a clean, scalable hybrid paginator in Ruby.

The Problem

I had two ActiveRecord scopes:

top_scope: High-priority records (e.g. featured or curated)
bottom_scope: Secondary-priority records (e.g. everything else)

The requirements:

Combine both into a single, paginated feed
Always prioritize top_scope first
If the current page isn't filled, pull the remainder from bottom_scope
Handle datasets of different types cleanly
Keep everything performant (no .to_a, no unnecessary loads)
Return standard pagination metadata

The Solution

We built a lightweight Ruby class to handle this logic, using basic SQL pagination (limit and offset) and a clean API.

# frozen_string_literal: true

class HybridPaginator
  PER_PAGE = 10
  MAX_PER_PAGE = 30

  def initialize(top_scope:, bottom_scope: nil, per_page:, page:)
    raise ArgumentError, 'top_scope must be an ActiveRecord::Relation' unless top_scope.respond_to?(:count)
    raise ArgumentError, 'bottom_scope must be an ActiveRecord::Relation or nil' unless bottom_scope.nil? || bottom_scope.respond_to?(:count)

    @page = page.to_i.positive? ? page.to_i : 1
    @per_page = per_page.to_i.between?(1, MAX_PER_PAGE) ? per_page.to_i : PER_PAGE
    @top_scope = top_scope
    @bottom_scope = bottom_scope
  end

  def call
    top_records = top_scope_paginated
    remaining = @per_page - top_records.size

    if remaining > 0 && @bottom_scope
      bottom_records = bottom_scope_paginated(limit: remaining)
      top_records + bottom_records
    else
      top_records
    end
  end

  def meta
    {
      current_page: @page,
      per_page: @per_page,
      total_pages: total_pages,
      count: total_records
    }
  end

  private

  def total_records
    @top_scope.count + (@bottom_scope ? @bottom_scope.count : 0)
  end

  def total_pages
    (total_records.to_f / @per_page).ceil
  end

  def top_scope_paginated
    @top_scope.limit(@per_page).offset((@page - 1) * @per_page)
  end

  def bottom_scope_paginated(limit:)
    offset = [(@page - 1) * @per_page - @top_scope.count, 0].max
    @bottom_scope.limit(limit).offset(offset)
  end
end

Where This Is Useful

Hybrid pagination is the right solution when you want a single feed, but your records don’t have equal priority—or even the same type.

Here are some real-world scenarios:

📁 Folder View: Subfolders First, Then Files

Say you’re building a file explorer. You want to show contents of a folder, but always list subfolders before files, and paginate the whole thing in a unified list.

Here, subfolders and files are separate models, so this is a good example of paginating across different types of content in a single interface.

HybridPaginator.new(
  top_scope: folder.ordered_subfolders,
  bottom_scope: folder.ordered_files,
  per_page: 20,
  page: params[:page]
)

Now each page:

Fills first with subfolders
Then includes files if there’s space
Returns one clean list, just like a desktop file browser

🛌 Featured vs Regular Products

On an e-commerce page, show sponsored or high-priority products first, then the rest of the catalog.

🔖 Editor Picks + Recent News

In a content platform, blend curated “editor picks” with algorithmically sorted latest posts. Prioritize the human touch, but still show what’s new.

🛠️ Dashboard Alerts

Show high-priority or critical items first, followed by lower-severity data, all in the same list.

Example Data Model: Folders and Files

Here’s how you might model this:

CREATE TABLE folders (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL,
  parent_id INTEGER REFERENCES folders(id),
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE TABLE files (
  id SERIAL PRIMARY KEY,
  folder_id INTEGER NOT NULL REFERENCES folders(id),
  name VARCHAR(255) NOT NULL,
  size INTEGER,
  created_at TIMESTAMP NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_folders_parent_id ON folders(parent_id);
CREATE INDEX idx_files_folder_id ON files(folder_id);

And the Rails associations:

class Folder < ApplicationRecord
  has_many :subfolders, class_name: 'Folder', foreign_key: :parent_id
  belongs_to :parent, class_name: 'Folder', optional: true

  has_many :files

  def ordered_subfolders
    subfolders.order(:name)
  end

  def ordered_files
    files.order(:name)
  end
end

class File < ApplicationRecord
  belongs_to :folder
end

Example Usage

paginator = HybridPaginator.new(
  top_scope: folder.ordered_subfolders,
  bottom_scope: folder.ordered_files,
  per_page: 20,
  page: params[:page]
)

render json: {
  data: paginator.call,
  meta: paginator.meta
}

Credits

This solution was built in collaboration with Anushka Shukla, whose insights and attention to edge cases were instrumental in shaping the final implementation. 👏

Final Thoughts

Hybrid pagination solves a common but under-documented problem: merging priority-ordered data sources—even across different models—without overcomplicating your app or dumping logic into the frontend.

It’s:

SQL-efficient
Easy to test
Simple to integrate
Flexible across use cases

If you're dealing with mixed-priority data or content curation, this approach keeps your stack clean and your UI consistent.

Let me know if you'd like this turned into a gem or Rails concern. We built it lean on purpose.

How I Built a Natural Language to SQL Translator Using Ruby and GPT-4

Surya — Fri, 28 Mar 2025 14:31:48 +0000

Introduction

Building dynamic reports often requires writing SQL queries, which can be inefficient and difficult to scale in a production environment. To streamline this process, I built a solution using Large Language Models (LLMs) that generates reports from natural language queries.

I implemented this in Ruby on Rails 💎, my preferred language, but the approach can be applied in any language like Python 🐍, Java ☕, or JavaScript 📜.

Overall Workflow

Here's the overall workflow:

User Query ➡️ Table Identification ➡️ SQL Generation ➡️ SQL Execution ➡️ Result Delivery

We separate the process into two clear steps:

Table Identification
SQL Generation and Execution

The Problem

Users need to retrieve data insights from databases without writing SQL queries manually. A system should:

Understand user intent 🧐 – Identify the relevant database table.
Generate the appropriate SQL query 📝 – Convert natural language queries into executable SQL.
Fetch and return results in a structured format 📈 – Output the data in an easy-to-consume format.

This approach ensures a seamless experience for non-technical users while leveraging the power of LLMs to dynamically generate reports.

The Solution

I built a modular system that consists of three primary components:

1. Llm::Chat - The OpenAI Chat API Wrapper

This class handles communication with OpenAI's Chat API.

require 'net/http'
require 'json'

class Llm::Chat
  OPENAI_API_URL = "https://api.openai.com/v1/chat/completions"
  API_KEY = ENV['OPENAI_API_KEY']

  def initialize(payload:)
    @payload = payload
  end

  def call
    response = request_openai
    parse_response(response)
  end

  private

  def request_openai
    uri = URI(OPENAI_API_URL)
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    request = Net::HTTP::Post.new(uri.path, headers)
    request.body = @payload.to_json
    http.request(request)
  end

  def headers
    {
      "Content-Type" => "application/json",
      "Authorization" => "Bearer #{API_KEY}"
    }
  end

  def parse_response(response)
    JSON.parse(response.body)["choices"]&.first["message"]["content"].strip
  rescue
    nil
  end
end

Prompts Used

We use predefined prompts to maintain consistency across interactions with the LLM.

module Prompts
  TABLE_IDENTIFICATION = "Given a user query, determine the most relevant table or tables from [users, departments, tickets]. If the query involves multiple tables (e.g., grouping users by department), return a comma-separated list of table names. Only return the table name(s) with no extra text."

  SQL_GENERATION = "Generate a MySQL query based on the table structure: %{table_structure}. Support queries involving multiple tables where applicable (e.g., grouping users by department). Only return the SQL query as plain text with no formatting, explanations, or markdown."
end

2. Identifying the Relevant Table

The first step is to determine the database table that best matches the user's query. This is handled by TableIdentifier.

class TableIdentifier
  def initialize(query:)
    @query = query
  end

  def call
    chat
  end

  private

  def chat
    Llm::Chat.new(payload: chat_payload).call
  end

  def chat_payload
    {
      "model": "gpt-4",
      "messages": [
          { "role": "system", "content": Prompts::TABLE_IDENTIFICATION },
          { "role": "user", "content": @query }
      ],
      "max_tokens": 100
    }
  end
end

Example Table Structures

For reference, here are example table structures:

Users Table 👥

CREATE TABLE users (
  id INT PRIMARY KEY,
  name VARCHAR(255),
  email VARCHAR(255),
  status VARCHAR(50),
  department_id INT,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

Departments Table 🏢

CREATE TABLE departments (
  id INT PRIMARY KEY,
  name VARCHAR(255),
  manager_id INT,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

Tickets Table 🎫

CREATE TABLE tickets (
  id INT PRIMARY KEY,
  user_id INT,
  subject VARCHAR(255),
  status VARCHAR(50),
  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

3. Generating and Executing the SQL Query

Once the table is identified, the next step is generating a valid SQL query. This is done using ReportGenerator.

class ReportGenerator
  require "#{Rails.root}/lib/llm"

  def initialize(query:)
    @query = query
  end

  def call
    report
  end

  private

  def report
    [
      { type: "text", data: "Here is your report" },
      { type: "table", data: ActiveRecord::Base.connection.select_all(query).to_a }
    ]
  end

  def table_structure
    ActiveRecord::Base.connection.execute("SHOW CREATE TABLE #{table_name}").first[1]
  end

  def table_name
    TableIdentifier.new(query: @query).call
  end

  def query
    Llm::Chat.new(payload: query_payload).call
  end

  def query_payload
    {
      "model": "gpt-4",
      "messages": [
          { "role": "system", "content": Prompts::SQL_GENERATION % { table_structure: table_structure } },
          { "role": "user", "content": @query }
      ],
      "max_tokens": 1000
    }
  end
end

Example Usage 🛠️

With this setup, generating a report is as simple as making a method call:

ReportGenerator.new(query: "count of inactive users").call
ReportGenerator.new(query: "list of active users").call
ReportGenerator.new(query: "number of users per department").call

For a query like "count of inactive users", the system will generate:

SELECT COUNT(*) FROM users WHERE status = 'inactive';

For a query like "number of users per department", the system will generate:

SELECT d.name, COUNT(u.id)
FROM users u
JOIN departments d ON u.department_id = d.id
GROUP BY d.name;

Disclaimer ⚠️

The LLM-generated prompts may require tweaking based on trial and error to achieve optimal results. You may need to fine-tune them based on your database schema and specific reporting requirements.

Benefits of This Approach 🚀

No manual SQL writing ✅ – Users can generate reports using natural language.
Adaptability 🔄 – The model can be fine-tuned to support new tables or complex queries.
Security 🔒 – The system ensures only relevant tables and safe queries are executed.
Scalability 📈 – Works across multiple datasets without custom development for each request.

Conclusion 🎯

By leveraging LLMs, we can automate the process of translating user intent into SQL queries, making data retrieval seamless and efficient. This approach eliminates the need for manual query writing while ensuring accuracy and adaptability.

Would you implement a similar solution in your application? Let me know your thoughts!

Force immutability with Ruby

Surya — Mon, 17 Mar 2025 16:11:15 +0000

Immutability is one of the most powerful tools for building predictable, maintainable, and thread-safe applications. However, Ruby is an inherently mutable language — objects are designed to be modified freely, which can lead to subtle bugs and complex state management issues.

In this post, we'll explore:

✅ Why immutability matters

✅ The problems with mutable objects

✅ How to enforce immutability in Ruby using freeze and other patterns

✅ Practical examples to make your Ruby code more robust

🧠 What is Immutability?

An immutable object is an object whose state cannot be changed after it is created. In other words:

You can create it.
You can read from it.
But you can’t modify it.

This contrasts with mutable objects, which allow their state to change after initialization.

🚨 The Problem with Mutable Objects

In Ruby, objects are mutable by default:

person = { name: "John", age: 30 }
person[:age] = 31
puts person # {:name=>"John", :age=>31}

Here’s why this can become a problem:

1. Unintended Side Effects

If an object is passed around and modified by multiple parts of a program, unexpected changes can creep in:

def update_age(person)
  person[:age] += 1
end

person = { name: "John", age: 30 }
update_age(person)
puts person # {:name=>"John", :age=>31}

👉 The function update_age modifies the original object, which could lead to unpredictable behavior if other parts of the code depend on the original state.

2. Thread Safety Issues

Mutable objects can create race conditions in multithreaded environments:

person = { name: "John", age: 30 }

threads = 10.times.map do
  Thread.new do
    person[:age] += 1
  end
end

threads.each(&:join)

puts person # Age is unpredictable!

👉 Since person is shared across threads and modified concurrently, the final value of age is unpredictable and inconsistent.

3. Caching Issues

Mutable objects can make caching tricky since the same object might have a different state between accesses:

person = { name: "John", age: 30 }
cache = { person[:name] => person }

person[:age] = 31
puts cache["John"][:age] # 31 — Cache consistency is broken!

👉 If an object changes after being cached, the cache becomes inconsistent.

🚀 Why Immutability Helps

Predictability – Immutable objects make code easier to reason about because their state is fixed.
Thread Safety – Multiple threads can access immutable objects without risk of race conditions.
Simplicity – No need to track object state across different contexts.
Functional-Style Code – Encourages a functional approach (pure functions, no side effects).

🔒 How to Force Immutability in Ruby

Ruby doesn’t have native support for immutable objects (like final in Java), but you can simulate it using patterns and language features like freeze.

✅ 1. Use `freeze` to Lock Objects

The simplest way to enforce immutability is by calling freeze on an object:

person = { name: "John", age: 30 }.freeze
person[:age] = 31 # => FrozenError (can't modify frozen Hash)

When you freeze an object:

All fields become read-only.
Any attempt to modify it will raise a FrozenError.

👉 Example with Custom Class:

You can freeze the instance inside the initializer to protect it:

class Person
  attr_reader :name, :age

  def initialize(name, age)
    @name = name
    @age = age
    freeze
  end
end

person = Person.new("John", 30)
person.age = 31 # => NoMethodError (undefined method)

Key Behavior:

@name and @age are accessible but read-only.
The object state is now immutable.

✅ 2. Use `Struct` with `freeze`

Struct in Ruby is useful for defining simple data objects. If you freeze a struct, it becomes immutable:

Person = Struct.new(:name, :age) do
  def initialize(*args)
    super(*args)
    freeze
  end
end

person = Person.new("John", 30)
person.age = 31 # => FrozenError (can't modify frozen Struct)

👉 Struct combined with freeze makes it easy to create immutable data objects.

✅ 3. Use `Value Objects` to Create Immutable Objects

You can model state as value objects — objects that represent data without behavior — and make them immutable:

class Money
  attr_reader :amount, :currency

  def initialize(amount, currency)
    @amount = amount
    @currency = currency
    freeze
  end
end

money = Money.new(100, 'USD')
money.amount = 200 # => NoMethodError

👉 Value objects are ideal for representing things like money, dates, and coordinates.

✅ 4. Return New Instances Instead of Mutating State

Instead of modifying an object, return a new one with updated state:

class Money
  attr_reader :amount, :currency

  def initialize(amount, currency)
    @amount = amount
    @currency = currency
    freeze
  end

  def add(other)
    Money.new(amount + other.amount, currency)
  end
end

money1 = Money.new(100, 'USD')
money2 = money1.add(Money.new(50, 'USD'))

puts money1.amount # 100
puts money2.amount # 150

👉 The original object remains unchanged, and the state change is represented by creating a new instance.

🧪 When Should You Use Immutability?

✅ Use immutability when:

The object represents fixed state (e.g., configuration, settings).
You want to avoid unintended side effects.
You need thread safety.
You’re building functional-style code.

❌ Don’t use immutability when:

Performance is critical and object creation is expensive.
The object is designed for short-term mutation.

🎯 Summary

Ruby is mutable by default, which can lead to unpredictable state changes, race conditions, and bugs.
Immutability creates more predictable and maintainable code.
Use freeze to enforce immutability.
Design state as value objects and return new instances instead of modifying existing ones.

🔥 Final Thought

Immutability in Ruby might feel unnatural at first, but once you start working with immutable objects, you’ll notice how much easier it is to reason about your code. Give it a try — your future self (and your bug tracker) will thank you!