Integrating AI into an Existing Rails Application Without Breaking It

Most Rails applications that actually make money are old. Not demo-old. Production-old. Five years, sometimes ten. They have inherited patterns nobody fully remembers. Tests that cover the happy path and not much else. Code that works but that everyone is slightly afraid to touch.

That’s the application a business wants to add AI to. And that’s where most AI integrations fail. Not because the model is wrong. Because the application isn’t ready for the change, and nobody admits it until the feature is three months late.

I’ve been through this. Here’s what I’ve learned.

The Pattern That Repeats

A team builds an LLM classifier. It works perfectly on a laptop. It sits in a branch for five months because it touches a controller nobody wants to modify. The model is fine. The integration is the problem.

This isn’t hypothetical. It’s the default outcome when you try to bolt AI onto a Rails app that hasn’t been prepared for it.

The problem is structural, not technical.

The Sequence You Can’t Skip

There’s a specific order that works. Skip a step and you pay for it later, usually at the worst possible time.

Audit first. Figure out what you can leave alone. Most of the application doesn’t need to know AI exists. The audit tells you which parts do.

Update one major version at a time. Rails upgrade guides exist for a reason. Following them in order is faster than fixing a broken merge six months from now.

Tests aren’t hygiene. They’re a precondition. You cannot safely add AI to a Rails app you’re afraid to change. If the test suite doesn’t give you confidence to refactor, fix that before you write a single line of AI code.

Refactor to service objects. This is where the AI layer will live. Pull business logic out of controllers and models. Create seams. The AI integration goes into those seams, not into the controller that already has forty lines of responsibility.

Integrate last. Only now does the AI code get written. And because of the previous four steps, it has a place to go that isn’t the middle of a fragile controller.

When I audited my own project, I found that three controllers handled most of the requests. The AI feature only needed to touch one of them. That changed the scope entirely.

What the AI Layer Actually Looks Like

Four components. None of them optional if you want this to survive production.

The Provider Client

Wrap your LLM provider behind a single interface. One class. One method. The rest of the application calls that method and doesn’t know whether it’s talking to OpenAI, Anthropic, or a local model.

This matters because providers change. Pricing changes. Rate limits change. A new model comes out that’s cheaper and better. If the vendor is hardcoded into your service objects, switching costs a refactor. If it’s behind a client, it costs a configuration change.

The Service Object

This is where the AI logic lives. One service object per AI feature. It takes input, builds the prompt, calls the provider client, validates the response, and returns a structured result.

The validation step matters more than people expect. LLMs return text. Your application needs data. The service object is responsible for turning one into the other, and for rejecting responses that don’t match the expected shape.

There’s a gem called rails-llm-structured that handles this pattern well. It lets you define fields with types — strings, enums, integers — and validates the LLM output automatically. That’s the kind of structure you want.

The Background Job

Never call the LLM in the request cycle. This is the single most common production mistake.

A typical LLM call takes two to ten seconds. Puma threads are limited. Ten concurrent AI requests can exhaust the thread pool and make the entire application unresponsive, including the parts that have nothing to do with AI.

The fix is straightforward. Put the AI call in a background job. The controller enqueues the job and returns immediately. The job calls the service object and updates the record when it’s done.

Here’s an implementation using Solid Queue, which is Rails 8’s default job backend:

# app/jobs/ai/classify_ticket_job.rb
class Ai::ClassifyTicketJob < ApplicationJob
  queue_as :ai_inference

  retry_on Faraday::TimeoutError, wait: :polynomially_longer, attempts: 3

  def perform(ticket_id)
    ticket = SupportTicket.find(ticket_id)
    return if ticket.ai_classified?

    result = Ai::Classifier.new.call(ticket.body)
    ticket.update!(
      ai_category: result[:category],
      ai_priority: result[:priority],
      ai_confidence: result[:confidence]
    )
  end
end

Note the queue. AI jobs get their own queue so a spike in AI requests doesn’t starve your normal job queue. That single line in config/queue.yml prevents a class of incidents that are hard to debug after the fact.

The Audit Table

Every AI call gets logged. Input. Output. Timestamp. Confidence score. Model version.

You will need this. Not for compliance. For debugging. When the classifier starts returning wrong categories, you need to pull the exact input that produced the wrong output and see what changed. Without an audit table, you’re guessing.

I’d start with columns for the record ID, the input text, the raw response, and the parsed result. That’s enough to trace a problem back to its source.

Where It Breaks in Practice

Three things you have to decide before launch. Not after.

Confidence threshold. The model will sometimes be unsure. What happens then? Two options: fall back to deterministic logic, or push to a human queue. Both are valid. What’s not valid is letting an unsure classification through because you didn’t build the fallback.

Spending limits. AI calls cost money. If your application serves multiple tenants, one tenant’s usage can consume the budget for everyone. Set limits at the job level, where you can actually stop the work. A monthly cap that triggers a 429 is better than a surprise invoice.

Prompt versioning. The prompt is code. It affects output. If you change it, you’ve made a production change. Treat it that way. Version it. Test it. Have a rollback.

I haven’t run into the spending limit problem yet in my own projects, but I’ve seen it described in enough postmortems to know it’s coming.

What This Means for the Business

The business doesn’t care about service objects or background queues. They care about one thing: will this feature work, and will it break what already works?

The sequence I described is how you answer that question with something other than optimism. You audit before you build. You test before you refactor. You put the AI call where it can’t take down the rest of the application. You log everything so you can fix it when it goes wrong.

That’s not exciting. It’s the difference between an AI feature that ships and one that lives in a branch forever.

What I’m Still Figuring Out

I don’t have this fully solved. The confidence threshold is the part I’m least sure about. Where do you draw the line between “confident enough” and “ask a human”? It depends on the cost of a wrong answer, and that varies by feature.

If you’ve dealt with this in production, I’d like to hear how you handled it. The comments are open.