DEV Community: Lucy

Shopify AI Search Optimization: How to Make Your Store Visible Inside ChatGPT, Perplexity, and Google AI Mode

Lucy — Wed, 29 Jul 2026 06:22:47 +0000

Direct answer: Shopify AI search optimization (GEO) means structuring your product and content pages so ChatGPT, Perplexity, Google AI Mode, and Copilot can read, trust, and cite them. In practice that comes down to three things: clean schema.org markup on every product page, a healthy Google/Bing Merchant Center feed, and content written in a self-contained, fact-dense style that AI systems can lift and quote. Below is exactly how the major platforms pull Shopify product data in 2026, and a checklist you can run today.

Why Shopify Merchants Can't Ignore AI Search Anymore

For most of the last decade, "getting found" meant ranking in ten blue links. That's no longer the whole game. Shoppers now ask ChatGPT to compare running shoes, ask Perplexity what the best gift is for a new homeowner, and ask Google AI Mode to shortlist skincare brands before they ever open a browser tab to search normally.

The interesting part is that this isn't a hypothetical shift for Shopify merchants specifically. Shopify has already built the plumbing for it. Shopify Catalog automatically syndicates product data to connected AI platforms including Perplexity, ChatGPT, and Google AI Mode, and when a shopper completes a purchase through one of these assistants, checkout still happens on the merchant's own store using their existing payment setup, according to Shopify's own guide to Perplexity Shopping. In other words, the infrastructure exists. Most stores just aren't feeding it correctly.

That's the gap this post covers: what "AI search optimization" (often called Generative Engine Optimization, or GEO) actually means for a Shopify store, how each AI platform sources its data, and what a merchant or their dev team should fix first.

How Do AI Shopping Assistants Actually Find Your Products?

This is the part most merchants get wrong, because it's easy to assume ChatGPT "crawls your site" the way Googlebot does. It mostly doesn't. Each platform has a different pipeline, and knowing which one matters for you changes where you spend effort.

ChatGPT Shopping draws roughly 83% of its product recommendations from Google Shopping's organic index, based on a March 2026 analysis of 43,000 carousel products that found base64-encoded Google Shopping parameters embedded in ChatGPT's source code, confirming a direct pipeline from Google Merchant Center feeds into ChatGPT's results, according to a feed-optimization breakdown of that research. Shopify merchants in the US get automatic catalog syndication into ChatGPT through Shopify's own Agentic Storefront integration.
Perplexity Shopping is closer to a real-time researcher. It crawls your live HTML and cites the exact page it pulled facts from, and its shoppers convert at a notably higher average order value than other AI referral traffic, per Shopify's Perplexity guide.
Google AI Mode leans on the existing Shopping Graph, meaning your Google Merchant Center feed and on-page schema still do most of the work.
Microsoft Copilot runs on Bing Merchant Center and, as of a January 2026 rollout, offers checkout inside the chat window itself for eligible Shopify merchants, per an ecommerce AI platform comparison.

Platform	Primary data source	What actually moves the needle
ChatGPT Shopping	Google Shopping feed (via Shopify Agentic Storefront)	Clean Merchant Center feed, GTIN, full descriptions
Perplexity Shopping	Live HTML crawl + Shopify Catalog	On-page schema, real prices, return policy fields
Google AI Mode	Shopping Graph	Merchant Center feed + Product/Offer schema
Microsoft Copilot	Bing Merchant Center	Feed hygiene + Copilot Checkout enrollment

The practical takeaway: your Google Merchant Center feed and your on-page schema aren't separate SEO and "AI SEO" tasks anymore. They're the same infrastructure two different systems are reading.

What Structured Data Does an AI Engine Actually Need From a Product Page?

Structured data is how you tell a machine that "$89" is a price and "4.7/5" is a rating, rather than making it guess from surrounding text. Most Shopify themes add basic Product schema automatically, but an audit of 2,400 Shopify product pages found only 9% carried the structured data required for ChatGPT or Perplexity to reliably recommend them, per an analysis of Shopify AI visibility. The gap is usually in the fields nobody thinks about: shipping details, return policy, and review counts.

A reasonably complete Product schema block looks like this:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Example Product Name",
  "image": "https://cdn.shopify.com/example.jpg",
  "description": "A full, specific product description, not a five-word title restated.",
  "brand": { "@type": "Brand", "name": "Your Brand" },
  "sku": "SKU-1234",
  "gtin13": "0012345678905",
  "offers": {
    "@type": "Offer",
    "priceCurrency": "USD",
    "price": "89.00",
    "availability": "https://schema.org/InStock",
    "shippingDetails": {
      "@type": "OfferShippingDetails",
      "shippingRate": { "@type": "MonetaryAmount", "value": "0", "currency": "USD" }
    },
    "hasMerchantReturnPolicy": {
      "@type": "MerchantReturnPolicy",
      "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
      "merchantReturnDays": 30
    }
  },
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.7",
    "reviewCount": "182"
  }
}

Two fields worth flagging specifically because they're the ones most themes skip: hasMerchantReturnPolicy and shippingDetails. Shipping and return terms are now a comparison criterion AI assistants use to decide between two similar products, not just legal boilerplate buried in a footer link.

Does llms.txt Actually Help a Shopify Store Get Cited?

This one deserves an honest answer rather than a hype-cycle one. llms.txt is a proposed standard, introduced in September 2024 by Jeremy Howard of Answer.AI, that puts a curated Markdown map of a site's key pages at the domain root, similar in spirit to robots.txt but aimed at language models instead of search crawlers, as documented in Search Engine Land's coverage of the proposal.

It has real traction in one specific use case: developer documentation for coding assistants and AI agents. It is far shakier as a general AI-search ranking lever. One large-scale analysis of AI bot traffic found that requests to /llms.txt were statistically negligible among the user agents that actually drive citations, such as GPTBot, ClaudeBot, and PerplexityBot, meaning most AI systems answering shopping questions in 2026 simply aren't reading it yet.

The pragmatic move for a Shopify store: add a lightweight llms.txt if it takes an hour, since there's no real downside, but don't treat it as a substitute for the fundamentals above. Structured data and feed hygiene are doing the actual work right now.

How Should Product and Blog Content Be Written So AI Engines Want to Quote It?

Generative engines retrieve short passages, not whole pages, so the unit of content that gets cited is a sentence or two, not a paragraph. Research from Princeton, Georgia Tech, and IIT Delhi found that content optimized for this pattern achieved measurably higher visibility in AI-generated answers, with the biggest single gains coming from adding verifiable statistics to a passage and from leading with a definition-first sentence rather than a scene-setting one, according to a summary of that research.

What that looks like in practice on a Shopify blog or PDP:

Lead with the answer. Put the direct, factual sentence first in every section, something like "A composting bin needs airflow on at least two sides," before the supporting explanation.
Make each paragraph stand alone. Assume an AI system will lift that one paragraph out of context and quote it. Don't rely on "as mentioned above."
Use question-style H2s. "What size composting bin do I need for a family of four?" mirrors how people actually phrase prompts, far more than "Composting Bin Sizes."
Cite something verifiable. A specific number, a named study, or a dated fact gets picked up far more often than a general claim.
Refresh the page. AI engines weigh recency when choosing which source to cite, so a guide last touched in 2023 loses ground to a competitor's 2026 update on the same topic.

A Practical GEO Checklist for Shopify Merchants

Audit every product page for Offer, AggregateRating, MerchantReturnPolicy, and OfferShippingDetails schema, not just basic Product markup.
Check your Google Merchant Center and Bing Merchant Center feeds for stale "in stock" flags. This is one of the fastest ways to get a feed quality-flagged and dropped from AI shopping results.
Confirm your robots.txt isn't accidentally blocking GPTBot, PerplexityBot, or OAI-SearchBot. Cloudflare's default configuration change in 2025 caused a wave of sites to block AI bots without realizing it.
Make sure product pages render server-side. AI crawlers generally don't execute heavy client-side JavaScript the way a human browser does.
Write full, specific product descriptions. A 15-word description gives an AI system almost nothing to reason over when it's matching your product to a conversational query.
Republish or refresh cornerstone guides at least twice a year with a visible "last updated" date.

Who Should Actually Run This Audit?

None of the above is exotic engineering, but it does touch theme templates, feed configuration, and content strategy at the same time, which is exactly where things get missed on a live store. This is the kind of cross-cutting technical and content work a team of dedicated Shopify specialists is set up to audit and implement properly, because it means checking schema output against the live theme, not just the store's settings panel, and tying that back into how product copy is actually written.

If you're not sure where your own store stands on any of the checklist above, that's usually the first thing worth having someone independently look at before you spend a quarter chasing individual AI platforms one by one.

Frequently Asked Questions

Does GEO replace traditional SEO for a Shopify store?
No. GEO builds on the same technical foundation as SEO (crawlability, structured data, fast pages) and adds a layer on top for how AI systems retrieve and cite content.

Which AI shopping platform should a Shopify merchant prioritize first?
Start with your Google Merchant Center feed. It feeds Google Shopping, Google AI Mode, and indirectly a large share of ChatGPT Shopping's results, so fixing it once pays off across multiple platforms.

Is llms.txt required for a Shopify store?
No. It's optional and low-cost to add, but current data suggests the major AI shopping crawlers aren't reading it in meaningful volume yet.

How long does it take to see AI citations after making these changes?
Perplexity, which relies on real-time web retrieval, can pick up freshly published or updated content within days. Feed-based platforms like ChatGPT Shopping and Google AI Mode typically lag behind the next feed refresh cycle, which can take longer.

Have you checked whether your own product pages actually render their schema correctly once the theme's JavaScript runs? Curious what others are finding when they audit their live Shopify stores for this. Drop what you find in the comments.

Shopify Checkout UI Extension Development: A Step-by-Step Guide (2026)

Lucy — Tue, 21 Jul 2026 07:05:17 +0000

Shopify checkout UI extension development means building a small, sandboxed React or Preact component that Shopify renders inside a fixed slot in checkout, such as a block on the Thank you page or a static area next to the cart line items. This guide scaffolds one from a real Shopify CLI project, reads and writes checkout data through the current shopify global object, tests it locally, and deploys it under the platform's actual constraints. By the end you'll have a working delivery-instructions extension and a clear list of the places these extensions tend to fail once real customers start using them.

What You're Building (and What This Guide Skips)

The extension in this walkthrough adds a text field to checkout where a customer can leave a note for the courier, and it shows the current order subtotal above the field. That's deliberately small. It exercises the three things almost every checkout UI extension needs: a target, a read from the shopify object, and a write back to the order.

This guide does not cover Shopify Functions (server-side discount, shipping, and payment logic), the Branding API (colors, fonts, and layout), or post-purchase upsell offers. Those are real parts of checkout extensibility, but they solve different problems and deserve their own walkthroughs. Shopify maintains its own custom fields tutorial covering a similar delivery-instructions pattern, worth a look once you want the officially maintained reference version alongside this one. The next section explains how to tell which one you actually need.

Checkout UI Extensions vs. Shopify Functions: Which One Do You Actually Need?

If the customer needs to see or interact with something, you want a UI extension. If you need to change a price, a discount, a shipping rate, or which payment methods appear, you want a Function. UI extensions render interface inside a sandboxed environment; Functions run server-side with no rendering at all, according to Shopify's documentation on apps in checkout.

	Checkout UI Extensions	Shopify Functions
Renders visible UI	Yes	No
Runs where	Client-side, inside a Web Worker sandbox	Server-side, on Shopify's infrastructure
Typical use	Custom fields, banners, upsell blocks, trust badges	Discount logic, shipping rate changes, payment method ordering
Data access	Reads/writes cart and checkout data through the `shopify` object	Reads structured input, returns structured output, no direct cart mutation
Plan requirement	Available on all plans; the information, shipping, and payment steps still require Shopify Plus	Available broadly, most often paired with Plus-tier checkout customization

Teams often reach for a UI extension when the real problem is a Function, which is why a custom field ends up trying to also silently change a shipping rate through a side channel. Keep the two separate. If your logic doesn't need to be seen, it probably belongs in a Function instead.

It's also worth knowing what these two tools replaced. Shopify Scripts, the old way to write custom discount and shipping logic for Plus stores, stopped executing on June 30, 2026, after running alongside checkout extensions during a transition window that Shopify's own documentation confirms. Anything still living in an old Script needs to move to a Function now, not a UI extension.

Prerequisites

You'll need:

Node.js and a recent Shopify CLI install
A Partner account and a development store with checkout extensibility enabled
An existing custom or public app to attach the extension to (or scaffold a new one with shopify app init)

Set the API version to the latest stable release in your configuration file rather than pinning to whatever version a tutorial used months ago. Each stable version stays supported for a minimum of 12 months, and the CLI blocks deploys against versions older than that window.

Scaffold the Extension

From inside your app directory, generate the extension:

cd my-app
shopify app generate extension --template checkout_ui

This creates a folder with a shopify.extension.toml configuration file and a templated JSX entry point. Edit the TOML to point at a block target, which is the placement type merchants can position themselves using the checkout editor:

api_version = "2026-07"

[[extensions]]
type = "ui_extension"
name = "Delivery Notes"
handle = "delivery-notes"
uid = "generate-this-with-shopify-cli"

  [[extensions.targeting]]
  target = "purchase.checkout.block.render"
  module = "./src/DeliveryNotes.jsx"

Block targets are one of three target types. Static targets render automatically in a fixed spot, like right after the cart line items, and can't be repositioned. Runnable targets don't render anything at all; they fire in response to an event, such as a keystroke in an address field, and return data. Block targets sit in between: merchants place them through the checkout editor, and up to three extensions can share the same slot. The Shopify CLI documentation covers the full set of scaffold and dev-server flags if you need to target a specific store or app configuration.

Read Checkout Data With the `shopify` Global Object

Inside the extension, Shopify injects a global shopify object that exposes checkout data as properties. There's no import needed for it: it's simply available at runtime, similar to how window behaves in a normal browser page, except this one runs inside a Web Worker with no DOM access.

import '@shopify/ui-extensions/preact';
import {render} from 'preact';
import {useState} from 'preact/hooks';

export default function extension() {
  render(<DeliveryNotes />, document.body);
}

function DeliveryNotes() {
  const [note, setNote] = useState('');
  const subtotal = shopify.cost.subtotalAmount.value;

  return (
    <s-stack border="base" padding="base">
      <s-heading>Delivery instructions</s-heading>
      <s-text type="small">
        Subtotal: {subtotal.amount} {subtotal.currencyCode}
      </s-text>
      <s-text-field
        label="Leave a note for the courier"
        value={note}
        onChange={(event) => setNote(event.target.value)}
      />
    </s-stack>
  );
}

shopify.cost.subtotalAmount updates automatically as the cart changes, and reading it doesn't cost you anything in terms of rate limits since it's a read, not a write. The s-stack, s-heading, s-text, and s-text-field elements are web components from Shopify's Polaris-based checkout component library, not raw HTML, so they inherit the store's checkout styling without extra CSS work on your part.

Run shopify app dev at this point to preview the extension live on your dev store before writing a single line of persistence logic. Confirming the field renders and the subtotal updates is the cheapest bug you'll ever catch.

Write Changes Back to the Order

Reading data is half the job. The other half is persisting what the customer typed, which happens through applyAttributeChange:

async function handleChange(event) {
  const value = event.target.value;
  setNote(value);

  await shopify.applyAttributeChange({
    type: 'updateAttribute',
    key: 'deliveryInstructions',
    value,
  });
}

Swap this in for the inline onChange in the component above. applyAttributeChange returns a promise that resolves once Shopify has applied the change and the corresponding property has updated, and the attribute becomes visible on the order in the admin once it's placed.

Two boundaries matter here. First, extensions that fire too many attribute or metafield changes in a short window get rate-limited for the rest of that buyer's session, so batch related writes with Promise.all instead of firing one call per keystroke. Second, an attribute set this way lands on the order object, not inside the payment step; if you need something reflected in pricing or payment method availability, that's a Function's job, not this one.

Test the Extension Before It Touches Real Checkouts

shopify app dev gives you a live preview against a real dev store, and it reloads automatically as you edit. That covers manual testing. As of API version 2026-04, Shopify also ships @shopify/ui-extensions-tester for writing actual unit tests against your extension code, which matters once the component grows past a single field and you want regression coverage without opening a browser every time.

Two checkpoints are worth running before you touch a production store:

Confirm the field appears in the correct slot using the checkout editor's preview, since block target placement is merchant-configurable and easy to get wrong during first setup.
Place a real test order and check that the attribute shows up on the order detail page in the admin. If it's missing, the write silently failed or the promise was never awaited, not a rendering problem.

Deploy It, and Respect the 64 KB Ceiling

Deployment is a single command:

shopify app deploy

The CLI builds your extension bundle and uploads it to Shopify. There's a hard constraint worth planning around early: a compiled UI extension bundle can't exceed 64 KB, and Shopify enforces this at deploy time. A single text field extension won't come close, but adding a chart library or a large icon set will get you there faster than expected. Analyze your bundle size before you're surprised by a failed deploy the day before launch.

If your extension needs more than the default sandbox allows, such as calling your own backend or collecting SMS marketing consent, you declare that explicitly in the TOML under [extensions.capabilities] (network_access, api_access, collect_buyer_consent, block_progress). Requesting a capability you don't end up using is a common reason review takes longer than expected, so keep the list matched to what the code actually does.

Where Do Checkout Extensions Actually Break in Production?

Two failure patterns show up more than any other.

The first is a mismatch between an extension's declared api_version and the version a linked extension expects, most commonly on payment-related targets. One developer on the Shopify community forum hit a crash where checkout failed with a TypeError before their React component even rendered, tied to a payment extension linked to a UI extension through ui_extension_handle. The lesson generalizes: when two extensions are wired together, verify the target string and API version match exactly, and don't assume a typo in a config field will surface as a readable error. It often surfaces as a crash somewhere upstream of your own code.

The second is the sandbox itself. The extension runs inside a Web Worker with no access to window or the DOM, which means a third-party error-reporting tool that assumes a normal browser environment will silently fail to initialize unless you disable its default integrations and attach error and unhandledrejection listeners manually. If you're not seeing errors you know are happening, this is usually why.

Neither failure mode is exotic. Both come from treating a sandboxed, upgrade-safe environment as if it were a regular webpage, which it deliberately isn't.

Build It Yourself or Bring in a Shopify Team?

A single custom field like the one in this guide is a reasonable first project for a developer who's comfortable with React and hasn't touched Shopify's extension model before; budget a day or two including the review cycle. The calculation changes once you're coordinating multiple targets, metafields across international markets, post-purchase offers, or a Scripts-to-Functions migration against a hard deadline. That's less a coding problem than a project-management one, and it's the same reasoning Shopify gives store owners directly in its own upgrade guidance: build the extension yourself, or bring in a service partner to build it for you.

For merchants in that second situation, Lucent Innovation's Shopify development team handles exactly this kind of checkout migration and extension work as part of its Shopify Plus practice. Either path is legitimate. The point of this guide is that you now have enough to make that call with a working extension in front of you, not a guess.

What's Next

You've got a checkout UI extension that reads live cart data and writes an attribute back to the order, scaffolded, tested, and deployed under the real constraints Shopify enforces at build time. The natural next step is a Shopify Function if your use case needs to touch pricing or shipping logic rather than just displaying something.

Have you run into the Web Worker sandbox limitations while wiring up error tracking, or found a cleaner pattern for requesting capabilities without triggering a longer review? Drop it in the comments, it's the kind of detail that doesn't make it into the docs.

From Liquid to Hydrogen: A Developer's Honest Guide to Going Headless on Shopify

Lucy — Wed, 15 Jul 2026 06:26:47 +0000

Most Shopify stores still run on Liquid. That's true today, and it will stay true for a while. It's fast out of the box, cheap to run, and there's nothing extra to host.

Hydrogen is different. It's Shopify's own React framework, built for stores that need more control over the frontend than Liquid can give.

This post walks through both. No sales pitch, just the real trade-offs, some code, and an honest answer to the question people keep asking: when is it actually worth going headless?

What Liquid Gets Right (and Where It Runs Out)

Liquid is Shopify's own template language. Tobias Lütke, one of Shopify's founders, built it, and it's been running real stores since 2006. That's longer than most of us have been writing code for a living.

Liquid is also strict on purpose. Merchants can edit their own theme files, so the language can't do everything a full programming language can do. It has to stay safe, since store owners are the ones typing into it, not developers.

That limit is a good thing, not a flaw. It's why a small business owner can tweak their homepage at 11pm without breaking checkout.

For most stores, Liquid is still the smart choice. It renders right on Shopify's servers. No extra hosting bill. No build step to babysit. No framework update to plan around every few months.

Online Store 2.0 made this even better. JSON templates and drag-and-drop sections gave merchants a lot of the flexibility that used to send people looking for custom builds.

So where does Liquid stop being enough? Usually it's one of these three things:

The frontend needs to act like an app. Think shared cart state across brands, heavy interactive UI, or checkout flows that need custom logic.
The brand needs one design system across more than the store. A marketing site, a mobile app, and the storefront all sharing the same components.
The dev team just knows React better than Liquid, and building in Liquid would slow them down more than it helps anyone.

What "Going Headless" Actually Means

People throw the word "headless" around a lot. Let's slow down and define it properly.

On Shopify, going headless means you split the frontend from the backend. The frontend is what shoppers see and click. The backend holds the product catalog, cart, checkout, and orders. Instead of Liquid rendering pages on Shopify's servers, your own app fetches data through an API and renders it however you want.

That API is called the Storefront API. It's a GraphQL endpoint, and it isn't locked to one framework. You can call it from a plain Node app, from Next.js, or from Shopify's own Hydrogen.

Here's the part most people skip past. Going headless is really three choices bundled into one:

Which framework you build the frontend in
Which API you pull your data from
Where you host the result

These don't have to travel together. Hydrogen can run on hosts other than Shopify's own Oxygen. And Oxygen can host things that have nothing to do with Hydrogen.

Inside Hydrogen: What's New in 2026

Hydrogen has changed shape more times than most frameworks manage to survive.

It launched in 2022. Then Shopify bought the Remix team and rebuilt Hydrogen on top of Remix. Then Remix's routing engine split off into its own project, React Router 7, and Hydrogen moved onto that instead.

Right now, the stable version of Hydrogen runs on React Router v7 and TypeScript. It tracks the Storefront API and Customer Account API on a schedule, and that schedule updates every quarter.

That update schedule is worth planning for. Hydrogen and hydrogen-react are tied to specific versions of these APIs, and since Shopify updates them every three months, breaking changes can show up every three months too. Budget time for that. It's a real, ongoing cost, not a one-time thing you fix and forget.

The bigger news is what comes next. In the middle of 2026, Shopify showed off a rebuilt version of Hydrogen. The old version asked you to adopt the whole framework. The new one is more like a toolkit you bring into whatever stack you already have.

This new preview has a core written in plain JavaScript, with thin adapters that make it feel native to whatever framework your team uses. React support ships first. Vue and Svelte support are on the way.

It also ships with something Shopify calls agent skills. These are structured instructions that let a coding agent set up the Storefront API client, cart, product pages, and checkout hooks for you, in whatever framework you picked. That's a shift from copying boilerplate by hand.

Whatever comes next, the current stable release is what real stores run today, and it holds up well. It uses the open source React Router framework, supports optimistic UI and nested routes, and it deploys for free on Oxygen, Shopify's own hosting network.

Liquid vs. Hydrogen: A Straight Comparison

Neither one wins in every case. Here's how they actually stack up:

	Liquid (Online Store 2.0)	Hydrogen (React Router 7)
Hosting	Included, no extra setup	Free on Oxygen, or any Node/edge host
Language	Liquid plus JSON templates	TypeScript and React
Skills your team needs	Theme development, CSS, a bit of JS	Solid React skills, plus GraphQL
Can merchants edit it themselves?	Yes, through the theme editor	Not easily. You'd need to add a CMS layer
App compatibility	Wide. Most apps just work	Mixed. Some apps built for Liquid themes need custom wiring
Speed out of the box	Fast by default	Fast only if someone tunes it
Best fit	Most D2C and B2B stores	Complex catalogs, custom UX, multi-brand builds, AI-driven shopping

That speed row needs a warning label, because it's the claim people stretch the most. A well-tuned Hydrogen store can absolutely beat a bloated Liquid theme. But "well-tuned" is carrying a lot of weight in that sentence. It means someone who understands edge caching and streaming rendering is actively setting it up, not that Hydrogen wins by default.

Google is pretty clear on this point too. Core Web Vitals are supposed to be measured out in the real world, based on how real users experience a page, and site owners shouldn't have to become performance experts just to know if their site feels fast. That's exactly the bar a rushed, untuned headless build usually misses.

The Same Data Fetch, Two Ways

Let's make this less abstract. Here's the same small task done in Liquid, then in Hydrogen: pull a product's title and price and show them on the page.

In Liquid, the product object is already sitting there, ready to use. No API call needed:

<h1>{{ product.title }}</h1>
<span class="price">{{ product.price | money }}</span>

In Hydrogen, you write a loader that queries the Storefront API's GraphQL endpoint, then you render what comes back:

// routes/products.$handle.jsx
export async function loader({ params, context }) {
  const { product } = await context.storefront.query(
    `#graphql
    query Product($handle: String!) {
      product(handle: $handle) {
        title
        priceRange {
          minVariantPrice { amount currencyCode }
        }
      }
    }`,
    { variables: { handle: params.handle } }
  );
  return { product };
}

export default function ProductPage({ loaderData }) {
  const { product } = loaderData;
  return (
    <>
      <h1>{product.title}</h1>
      <span className="price">
        {product.priceRange.minVariantPrice.amount}{' '}
        {product.priceRange.minVariantPrice.currencyCode}
      </span>
    </>
  );
}

Same end result. Very different amount of code to own.

The Liquid version has no build step and nothing to ship to the browser beyond the rendered HTML. The Hydrogen version gives you full control over that markup and lets you mix it with any React component in your app, but you're the one writing and maintaining the query, the loader, and the rendering logic from here on out.

When Headless Actually Pays Off (and When It's Overkill)

Ask yourself these questions before you commit to a headless rebuild.

Does your catalog or user experience need logic Liquid genuinely can't handle? Product configurators, live inventory pulled from several warehouses feeding one custom availability rule, or a shopping flow that behaves more like an app than a page. These are real reasons to go headless.

Can your team actually maintain a framework that breaks every quarter? Hydrogen moves on the same calendar as Shopify's APIs. Someone on your team needs to own that upgrade cycle as an ongoing job, not a one-off project.

Do the apps you rely on even work headless? A large chunk of the Shopify App Store assumes a Liquid theme is injecting a script tag onto the page. Some apps talk to the Storefront API cleanly. Others need custom work, or don't support headless setups at all. Find this out before you start building, not after.

Is AI-driven shopping actually the goal? If your roadmap includes a shopping assistant or an AI agent that browses your catalog on a customer's behalf, that's a case where Hydrogen's direct connection to the Storefront API earns its complexity. That GraphQL layer is exactly what those systems need to read.

If most of your answers land on "not really," a solid Liquid theme built on Online Store 2.0 is probably the faster, cheaper, and safer path to the same goal.

How Teams Actually Land on an Answer

In practice, this decision rarely comes down to which framework someone likes more. It comes down to how complex the catalog is, how deep the team's React skills go, and how much of the App Store ecosystem the store leans on.

Teams that get this wrong usually make the call based on what's trendy, in the abstract, instead of checking it against their own app stack and how their content team actually works day to day.

When Lucent Innovation's Shopify store experts sit down with a merchant to answer this exact question, the audit almost always starts with app compatibility and how much merchants need to edit the site themselves. Architecture comes later. Those two things alone usually rule out one option before anyone even opens a performance benchmark.

However the decision lands, the same rule holds up: pick the setup your team can actually build and keep running for the next few years. Not the one that sounds best in a slide deck.

Have you shipped a Hydrogen store, or looked into one and decided against it? I'd like to hear what actually tipped the decision for your team.

Shopify Scripts Are Dead — Here's How to Migrate to Shopify Functions Before June 30, 2026

Lucy — Fri, 26 Jun 2026 07:14:33 +0000

If you're running a Shopify Plus store with custom Scripts, you have 4 days left.
June 30, 2026 is Shopify's hard deprecation date for Shopify Scripts — the Ruby-based customization layer powering custom discounts, shipping rules, and payment logic for thousands of merchants. After this date, every Script stops executing. Silently. With no fallback. Your checkout reverts to Shopify's defaults as if your custom logic never existed.

Critical: As of April 15, 2026, the Script Editor is already locked. You can no longer edit or publish Scripts. Any Script still running in production is frozen code — bugs cannot be patched. June 30 is when execution stops entirely. If you haven't started migrating, you are running unmodifiable code in production right now.

This isn't a "we'll get to it eventually" situation. It's a breaking change with a hard deadline days away.
In this guide I'll walk you through:

What's actually changing and why

The direct replacement for each Script type
A step-by-step migration path with corrected CLI commands
Real code examples showing before and after

Let's move fast.

What Are Shopify Scripts?

Shopify Scripts were introduced as a Shopify Plus-only feature that let developers write Ruby code to customize the cart and checkout experience. Three types existed:

Line Item Scripts — modify prices, apply discounts, bundle logic
Shipping Scripts — customize shipping rates, hide or rename options
Payment Scripts — show or hide payment methods based on conditions

They were powerful for their time, but they carried serious limitations: Ruby-only, Plus-only, slow to test, no version control, and fundamentally incompatible with Shopify's Checkout Extensibility architecture that replaced checkout.liquid for Plus merchants in 2023–2024.

Why Shopify Is Deprecating Them

Three reasons, in plain terms:

1. Checkout Extensibility replaced checkout.liquid — Scripts are architecturally incompatible with the new checkout model
2. Performance — Ruby Scripts ran server-side with cold-start delays. Shopify Functions compile to WebAssembly and run under a strict 5ms execution cap. Rust Functions typically execute in 3–5ms; JavaScript Functions run 10–30ms in real-world use and should be used only for simpler logic
3. Platform-wide access — Shopify Functions are available to all plan levels via installed apps, not just Shopify Plus subscribers

Scripts vs. Functions: What Actually Changed

Here's the architectural shift. Scripts were a Plus-only workaround bolted onto an older platform. Functions are native, WebAssembly-powered infrastructure available on every plan.

What Gets Replaced by What?

Enter Shopify Functions

Shopify Functions are the modern replacement. You write them in JavaScript/TypeScript or Rust — the two officially maintained, first-class languages in 2026. Any language that compiles to WebAssembly is technically supported, but Rust and JavaScript are the only paths with active Shopify CLI tooling and official support.

Choose based on your use case:
1. JavaScript/TypeScript — good for prototyping, simpler discount logic, teams without Rust experience. Compiled via Shopify's Javy toolchain.
2. Rust — recommended for complex logic, large carts, public apps, or any Function near the 256KB binary size limit. Runs 3–5ms versus 10–30ms for JavaScript.

Platform limits (as of 2026): Each store can run a maximum of 5 Discount Functions, 1 Cart Transform Function, and 5 Validation Functions. Plan your migration with these caps in mind if you have many Scripts.

Migration: Step by Step

Step 1: Audit your current Scripts

Go to Shopify Admin → Apps → Script Editor. The editor is now read-only (locked since April 15), but you can still view all active Scripts, export the customizations report, and read the source logic.
List every active Script, what business rule it enforces, and every edge case it handles.

Pay attention to:

Is this Script still being used in production?
What customer-facing behaviour does it produce?
Are there conditional rules — customer tags, order thresholds, product exclusions, B2B rules?

Use the Shopify Scripts customizations report (available from the Script Editor page) to export the full list automatically.

Step 2: Set up your Shopify Functions environment

You need Shopify CLI 4.0 or higher:

# Verify CLI version
npm install -g @shopify/cli
shopify version  # confirm 4.0+

Create a new app or add a Function extension to an existing one:

# Create a new app (correct CLI 4.0 syntax)
npm init @shopify/app@latest

# Or add an extension to an existing app
shopify app generate extension
# Choose: Discount, Cart Transform, Delivery Customization, or Payment Customization

Step 3: Scaffold the right Function type

shopify app generate extension
# Select "Discount - Order discounts"    → for order-level discount Scripts
# Select "Discount - Product discounts"  → for variant/product-level logic
# Select "Delivery customization"        → for Shipping Scripts
# Select "Payment customization"         → for Payment Scripts
# Select "Cart transform"                → for bundle/kit Line Item Scripts

Step 4: Rewrite your logic in JavaScript or Rust

Here's a real before/after for the most common migration case — 10% off all orders over $100:

Before (Ruby — Shopify Script)

Input.cart.line_items.each do |line_item|
  if Input.cart.subtotal_price >= Money.new(cents: 100_00)
    line_item.change_line_price(
      line_item.line_price * 0.9,
      message: "10% bulk discount"
    )
  end
end

Output.cart = Input.cart

After (JavaScript — Shopify Order Discount Function)

// run.js — Order Discount Function
export function run(input) {
  const subtotal = parseFloat(
    input.cart.cost.subtotalAmount.amount
  );

  if (subtotal >= 100) {
    return {
      discounts: [
        {
          targets: [
            { orderSubtotal: { excludedVariantIds: [] } }
          ],
          value: { percentage: { value: "10.0" } },
          message: "10% bulk discount"
        }
      ],
      discountApplicationStrategy: "FIRST"
    };
  }

  return {
    discounts: [],
    discountApplicationStrategy: "FIRST"
  };
}

The business logic is the same. The architecture is fundamentally better — compiled to WebAssembly, version-controlled, deployable via CI/CD, and accessible on all Shopify plans.

Using Rust instead? The logic structure is similar. Shopify's CLI scaffolds a full Rust project with the shopify_function crate when you select Rust at the extension type prompt. For complex discount engines with large catalogs, Rust is the safer choice as JavaScript Functions can exceed the 5ms execution cap on heavy carts.

Step 5: Test in a development store

shopify app dev

This deploys your Function to a development store in draft mode. Create a discount in Shopify Admin that uses your new Function, then test it across all cart scenarios — especially every edge case from the original Script.

Step 6: Deploy to production and monitor

shopify app deploy

Link your Function to a discount or customization in Shopify Admin (or via the Admin API). Monitor checkout conversion rate carefully for the first 48 hours after going live. A measurable drop typically signals a missed edge case, not a platform issue.

Common Migration Pitfalls

1. Scripts that relied on execution order

Scripts ran sequentially and could interact with each other. Functions run independently in parallel. If you had two Line Item Scripts that stacked or modified each other's output, refactor the logic into a single Function or control stacking behaviour explicitly with discountApplicationStrategy.

2. Bundle logic mapped to the wrong Function type

Bundle discounts in Line Item Scripts map to Cart Transform Functions, not Discount Functions. These are a completely separate extension type — selecting the wrong one at scaffold time means you're building against the wrong API schema.

3. Assuming Shipping Scripts are low priority

All three Script types — line item, shipping, and payment — stop executing on June 30. Shipping Scripts are often lower revenue impact than discount Scripts, but if your store hides express shipping for fragile items, shows different carrier options by customer tag, or applies shipping discounts conditionally, those rules disappear on the deadline. Migrate them before June 30, not after.

4. Edge cases left uncoded

Ruby Scripts accumulated implicit logic over years — gift card exclusions, B2B pricing tiers, free shipping thresholds, variant-level carve-outs. Functions are explicit. Every condition your business requires must be coded. The audit in Step 1 is your safety net here.

5. JavaScript Functions on complex carts

JavaScript Functions run 10–30ms in real-world use. Shopify's 5ms execution cap means JavaScript Functions can be aborted on heavy carts, leaving the checkout in its default state. For stores with large catalogs or many line items, write in Rust. For simple rules (e.g. flat percentage discount on all orders), JavaScript is fine.

6. Not reading the GraphQL input schema

Each Function type has a defined input schema. You control what data your Function receives by editing run.graphql. If your logic needs metafields, customer tags, or product attributes, add them to the input query — do not assume they arrive automatically.

What If You're Already Behind?

If you're reading this with four days on the clock and active Scripts still in production, here's the prioritisation:

1. Revenue-critical Scripts first — discount Scripts affecting checkout conversion and payment Scripts controlling visible gateways go first
2. Stub before you optimise — a working Function that matches the Script's behaviour is better than a perfectly architected one that isn't deployed
3. Migrate all Script types before June 30 — Line Item, Shipping, and Payment Scripts all stop on the same date. Don't assume Shipping Scripts can wait; schedule them in the same sprint
4. Use Shopify's migration docs at shopify.dev — the customizations report and Function input schemas are well-documented

If you have a complex store with multiple interdependent Scripts and the timeline feels impossible, working with a certified Shopify expert agency is the fastest path. Teams like Lucent Innovation — 10+ years as a Shopify Plus partner, 12+ years in ecommerce, 5,000+ stores delivered — can run a focused Functions migration sprint with proper QA, significantly faster than doing this under pressure in-house.

Quick Recap

Shopify Scripts deprecate June 30, 2026 — no extension, no fallback, no grace period
Script Editor locked since April 15, 2026 — existing Scripts are frozen and cannot be modified
Shopify Functions replace Scripts with WebAssembly execution, all-plan access, and full CI/CD support
Write Functions in JavaScript/TypeScript (simpler logic) or Rust (complex logic, large carts)
Per-store limits apply: 5 Discount Functions, 1 Cart Transform, 5 Validation Functions
Migration path: audit via customizations report → scaffold correct type → rewrite → test → deploy

Start with the audit. Everything else follows from there.

Already migrated? What was the trickiest Script to convert? Drop it in the comments — would love to hear real-world edge cases.

About the author: This post was written by the engineering team at Lucent Innovation, a certified Shopify Plus partner with 10+ years on the platform and 12+ years building and scaling ecommerce stores.

Migrate Your Data to Delta Lake: A Simple Guide for Developers

Lucy — Tue, 16 Jun 2026 09:31:12 +0000

TLDR: Delta Lake is like adding safety rules to your data storage. It stops data accidents, lets you see old versions of your data, and keeps everything organized. If you work with big data and Apache Spark, Delta Lake makes your life easier. It uses ACID transactions (fancy words for "data never breaks") and costs way less than old data warehouses.

What This Article Covers: This post explains what Delta Lake is, why you should move your data to it, and how to start the migration. We look at real problems Delta Lake solves and give you a simple roadmap for getting started. For more in-depth technical details, see Delta Lake Explained on the Lucent Innovation blog.

The Problem With Old Data Lakes

Imagine you have a giant closet to store files. In the old days, data lakes worked like that. You threw all your data files in there, and it was cheap. But there was a big problem.

What happens if someone is reading a file while someone else is writing to it? The file could get messed up. What if the computer crashes while saving? You might lose all your work. There was no good way to fix mistakes or go back to how things were before.

That is the main problem Delta Lake solves.

What Is Delta Lake?

Delta Lake is a layer of software that sits on top of your data files. Think of it like a smart librarian for your data. It keeps track of every change, makes sure nothing gets broken, and lets you fix mistakes fast.

The cool part is that Delta Lake does this without charging you a fortune. It runs on cheap cloud storage like Amazon S3 or Azure Blob Storage, just like regular data lakes. But it adds power that usually costs a lot of money.

The Big Features That Matter

ACID Transactions (No More Broken Data)

ACID is short for Atomicity, Consistency, Isolation, and Durability. That means:

Atomicity: Either the whole write happens or none of it happens. No half-finished work.
Consistency: Data stays clean and organized, no broken records.
Isolation: People can read and write at the same time without getting in each other's way.
Durability: Once data is saved, it stays saved. No losing work if the power goes out.

In plain English: You never have broken data or weird errors from too many people using the system at once.

Time Travel (Go Back in Time)

Did you delete something by accident? Need to check how data looked last week? Delta Lake remembers everything. You can run a query that shows you your data from any point in the past.

This is super helpful for audits, fixing mistakes, or checking when something went wrong.

Schema Enforcement (Keep Data Clean)

A schema is a blueprint that says what columns you have and what type of data goes in each one. Delta Lake watches the door and makes sure bad data never gets in. If someone tries to add data that does not match the blueprint, Delta Lake stops it right there.

Automatic Updates and Deletes

In regular data lakes, updates and deletes are slow and messy. You have to rewrite whole files. Delta Lake makes this fast and easy. You can change or remove records without rewriting everything.

Why Move to Delta Lake?

Cost Savings

A big data warehouse from twenty years ago costs a fortune to run. Delta Lake gives you warehouse-level reliability but uses cheap cloud storage underneath. You can save up to 50 times on computing costs while still getting fast answers to your questions.

Speed and Trust

Your team can trust the data faster. You spend less time fixing problems and more time using the data. No more mysteries about whether a number is right or wrong.

Real-Time and Batch in One Place

Some data comes in one batch per day. Some comes in live streams all day long. Delta Lake handles both in the same place, with the same tools. You do not need different systems for different types of data.

Easy Audits and Compliance

Every change is tracked in a log. This is gold when you need to follow rules like GDPR or show customers that their data is safe. You can prove exactly who changed what and when.

How to Start Your Migration

Step 1: Check Your Current Setup

Before you move anything, understand what you have now. Answer these questions:

What files do you store? (CSV, JSON, Parquet, something else?)
How big is your data? (1 GB or 1 TB?)
Who uses it? (Engineers, analysts, AI models?)
What problems do you hit most? (Slow updates? Broken data? Hard to audit?)

Step 2: Start Small

Do not move everything at once. Pick one table or one folder that is not critical. Move it to Delta Lake and run it for a week. See if your team likes it. Break things on purpose to understand how Delta Lake handles problems.

Step 3: Set Up Your Infrastructure

Delta Lake works with Databricks (a company built for this), or you can run it open-source with Apache Spark. For most projects, using Databricks is easier because everything just works together.

If you want to go full open-source, you will need:

Apache Spark
Storage like S3 or Azure
A way to run the code (like a Linux server)

Step 4: Copy Your Data Over

For small amounts of data, you can copy directly. For huge amounts, break it into chunks and move one chunk at a time. This way if something breaks, you only fix that chunk, not everything.

Step 5: Test Everything

Before you tell everyone to use the new system, run real queries on it. Check that the numbers match the old system. Have your analysts double-check important reports.

Step 6: Switch Over and Watch It

Pick a time when not many people are using the system. Switch everyone to Delta Lake. Have people ready to help if something goes wrong. Watch for problems the first few days.

Step 7: Keep the Old System as Backup

Even after you switch, keep your old data around for a few weeks. If something goes really wrong, you can go back.

Simple Example: Your First Delta Lake Table

If you know Python and Spark, it is super easy:

# Read your data
data = spark.read.csv("old_data.csv", header=True)

# Write it as Delta Lake
data.write.format("delta").mode("overwrite").save("delta_table")

# Now read it back as Delta Lake
df = spark.read.format("delta").load("delta_table")

# See past versions
df_yesterday = spark.read.format("delta").option("versionAsOf", 0).load("delta_table")

That is it. Three lines to switch from regular files to Delta Lake.

Real Costs vs. Old Systems

Let's look at numbers:

System	Cost Per Year	Speed	Broken Data	Ease
Old Data Warehouse	$500K+	Medium	Rare	Hard
Regular Data Lake	$50K	Slow	Common	Easy
Delta Lake	$50K-100K	Fast	Rare	Easy
Databricks Lakehouse	$100K-200K	Very Fast	Very Rare	Very Easy

The exact cost depends on how much data you have and how much you use it. But the pattern is clear: Delta Lake gives you warehouse reliability at lake prices.

Common Questions

Q: Do I need to rewrite all my code?

A: Not really. If you use Spark SQL or Python with Spark, you mostly use the same code. The main change is using "delta" as the format instead of "parquet" or "csv."

Q: What if my company uses a different system like Spark, Flink, or Kafka?

A: Delta Lake works with all of them. It is just a format and a set of rules. Any system that can read Parquet files can work with Delta Lake.

Q: Is Delta Lake production-ready?

A: Yes. Thousands of companies run it in production. It handles petabytes of data every day.

Q: How hard is the migration?

A: It depends on your setup. If you have simple CSV or Parquet files, it is easy. If you have a complex system with lots of custom code, it takes more time. Plan for weeks or months, not days.

Next Steps

Delta Lake is a great investment for any team that works with big data. It solves real problems and saves money at the same time. Start small, test it out, and see if it works for your team.

If you want to learn more about the deep technical details like transaction logs, how Delta Lake picks which files to read, and how schema evolution works, check out Delta Lake Explained on Lucent Innovation's technology blog. It goes much deeper into these topics.

The important thing is to start somewhere. Pick one small project. Try Delta Lake. See how it feels. You will probably wonder why you did not switch earlier.

How to Speed Up Your Shopify Store in 5 Easy Steps for Better Performance

Lucy — Mon, 15 Jun 2026 13:25:34 +0000

Introduction

Most online stores lose customers because of slow loading times. Did you know that if your store takes more than 3 seconds to load, people often leave? A slow Shopify store can cost you money and customers. (Check out more

#ecommerce Follow

strategies on Dev.to)

In this post, we will cover 5 simple things you can do to make your Shopify store faster. These tips work for everyone, from small businesses to large online stores.

What You Will Learn

In this article, you will find out about:

Image optimization methods
Reducing JavaScript and CSS files
Using a content delivery network (CDN)
Caching strategies
Monitoring your store speed

Let's dive in!

Step 1: Optimize Your Images

Large images slow down your store. Images should be as small as possible but still look good.

Why Images Matter:
Images make up most of a website's file size. If you have 20 large product images on one page, your store gets very slow.

How to Fix It:

Use tools like TinyPNG or ImageOptim to make images smaller. These tools remove extra data from images without making them look bad.

You can also use Shopify's built-in image compression features. Just upload your images to Shopify and let it handle the resizing.

Pro Tip: Use WebP format instead of JPG. WebP images are 25 to 35 percent smaller and look just as good.

Step 2: Minify CSS and JavaScript

Minification means removing extra spaces and characters from your code. This makes files smaller and faster to load.

What Gets Removed:

Extra spaces
Line breaks
Comments in the code
Unused characters

Most Shopify themes already do this automatically. But if you are building a custom theme, you should check your code.

Tools You Can Use:

CSS Minifier
JavaScript Minifier
Shopify's built-in minification

These tools take your code and make it shorter without changing what it does.

Step 3: Use a CDN

A CDN is a content delivery network. It stores your images and files in many locations around the world. When someone visits your store, they get files from the closest location.

How It Works:

If you have a customer in Japan and your server is in the USA, they have to download files from far away. With a CDN, a copy of your files sits in Japan too.

Shopify uses Cloudflare as its CDN, which is really good. Most Shopify plans include CDN automatically, so you probably already have this feature.

Step 4: Enable Caching

Caching means saving some information so you do not have to load it again. This makes repeat visits much faster.

Browser Caching:

Your customer's browser can save images, CSS, and JavaScript on their computer. When they visit again, these files load instantly.

Server Caching:

Your Shopify server can save product information and page data. This reduces the work the server has to do.

You can enable caching through Shopify settings or use apps designed for this purpose.

Popular Caching Apps:

Cache Cleaner
Bulk Operations
Speed Booster

Step 5: Test Your Speed

Use Google PageSpeed Insights or GTmetrix to test how fast your store loads. Run tests after you make changes to see what works best.

How to Test:

Go to Google PageSpeed Insights
Enter your Shopify store URL
Click "Analyze"
Read the report
Make changes
Test again

Keep testing until your score gets better. Aim for at least 75 out of 100.

Key Takeaways

Making your Shopify store faster does not have to be hard. Here are the main points to remember:

Optimize images with tools like TinyPNG
Minify your JavaScript and CSS code
Use a CDN like Cloudflare
Enable caching for faster repeat visits
Test your speed regularly

Each of these steps will help you keep customers happy and increase your sales.

Conclusion

A fast Shopify store means more customers and more money. These five steps will help you speed up your store today. Start with image optimization because that gives you the biggest improvement.

What speed tricks do you use on your Shopify store? Share in the comments below.

Related Resources

Google PageSpeed Insights: https://pagespeed.web.dev/
GTmetrix Speed Test: https://gtmetrix.com/
Shopify Performance Guide: https://shopify.dev/
Cloudflare CDN: https://www.cloudflare.com/

Related Dev.to Posts

If you found this helpful, check out these related articles on Dev.to:

#shopify Follow

Learn more about Shopify development and ecommerce solutions
#performance Follow

Tag for content related to software performance.

Explore other web performance optimization techniques
#ecommerce Follow

Discover more ecommerce development best practices
#webdev Follow

Because the internet...

Stay updated with the latest web development trends Have questions about Shopify performance? Drop them in the comments and I will help you out!

What Does an AI Consultant Actually Do? A 2026 Breakdown for Business Leaders

Lucy — Fri, 12 Jun 2026 12:59:04 +0000

Short Answer
An AI consultant helps your business figure out where AI can help, what to build, and how to make it actually work without the guesswork or wasted budget. They bridge the gap between cutting-edge technology and real business outcomes. In 2026, with the global AI consulting market valued at over $11 billion and growing at 26% annually, knowing exactly what you're paying for matters more than ever.

What Even Is an AI Consultant?

Let's be honest. "AI consultant" sounds like one of those titles that could mean anything.

It could mean someone who builds models. Or someone who just makes slide decks. Or someone who helps you figure out which tools to buy. The real answer? All three and more.

An AI consultant is a specialist who helps organizations identify AI opportunities, design solutions, and make sure those solutions actually work in the real world. They are not just coders. They are not just strategists. They sit right in the middle.

Think of them like a general contractor for a home renovation. The contractor doesn't just swing a hammer. They help you design the plan, pick the right materials, manage the work, and make sure the final result matches what you needed, not just what looked good on paper.

According to McKinsey's 2025 State of AI report, 78% of organizations now use AI in at least one business function. But only around 6% achieve significant, enterprise-wide results. That gap between "we're using AI" and "AI is actually helping our business" is exactly where AI consulting services come in.

What Does an AI Consultant Actually Do Day-to-Day?

Here is where most articles get vague. Let's break this down into real phases.

Phase 1: The AI Opportunity Audit

Before building anything, a good consultant spends time understanding your business.

They look at your current workflows, your data, your tools, and your goals. They ask: Where are the bottlenecks? Where is time being wasted? Where could automation or prediction actually add value?

This is not a one-hour meeting. It is typically a multi-week discovery process. It involves talking to department heads, reviewing data pipelines, and mapping out where AI can realistically help vs. where it would just add unnecessary complexity.

Many businesses skip this step and jump straight to building. That is one of the top reasons AI projects fail.

Phase 2: Strategy and Roadmap Building

Once they understand the business, a consultant builds a roadmap. This is a prioritized list of AI projects, ordered by value and feasibility.

Not every AI idea is a good idea. A roadmap helps you focus on what will move the needle first, instead of chasing the flashiest use case.

A solid roadmap answers these questions:

What AI projects do we tackle first?
What data do we need, and is it ready?
How long will each project take?
What does success actually look like?
How does this fit into our existing tech stack?

Phase 3: Picking the Right Tools and Stack

There are thousands of AI tools available in 2026. Large language models, vector databases, MLOps platforms, AutoML tools, fine-tuning services, the list keeps growing.

A good consultant knows which ones are right for your specific problem, not which ones are trending this month.

They look at your cloud provider (AWS, Azure, GCP), your data infrastructure, your team's existing skills, and your budget. Then they recommend a stack that actually fits — not the most expensive or the most popular.

This step alone can save companies from expensive vendor lock-in or over-engineered solutions that nobody ends up using.

Phase 4: Managing the Build and Deployment

This is where the technical work happens. Depending on the team, a consultant might:

Lead a team of data engineers and ML engineers
Review and approve model designs
Oversee integration with production systems
Set up monitoring and alerting for deployed models

They are not just advising from the sidelines. They are in the work — reviewing code, unblocking issues, and making sure the solution is being built correctly.

According to IDC, AI consulting demand grew 40% between 2024 and 2025 — largely because companies realized they needed someone to manage this build process, not just hand over a strategy document and walk away.

Phase 5: Governance, Ethics, and Ongoing Optimization

After a model goes live, the work is not done.

AI systems can drift over time. Their predictions get less accurate as the world changes. They can also produce biased or harmful outputs if they are not monitored carefully.

A consultant puts governance frameworks in place: model monitoring, bias detection, data refresh schedules, and escalation paths for when something goes wrong. In regulated industries like finance, healthcare, or insurance, this step is not optional. It is legally required.

Only 23% of IT leaders are confident their organizations can manage AI governance when rolling out generative AI tools, per a 2025 Gartner survey. This is a massive gap and it is one of the fastest-growing areas of demand in AI consulting right now.

How Is an AI Consultant Different From a Data Scientist or Software Engineer?

This question comes up a lot. Here is the simplest way to think about it:

Data Scientist: Focused on building models. Highly technical. Not always thinking about business outcomes or how the model gets used in practice.
ML/Software Engineer: Builds and deploys systems. Focused on code and infrastructure. Not always involved in strategy or stakeholder communication.
AI Consultant: Connects business goals to technical solutions. Manages the full lifecycle. Communicates across teams from the CEO to the dev team.

A data scientist might build you a great churn prediction model. But a consultant makes sure it connects to your CRM, that the sales team knows how to use it, that it gets updated every quarter, and that someone is watching for problems when it drifts.

Both roles are valuable. But they do very different jobs.

When Does a Business Actually Need AI Consulting Services?

You probably need a consultant if any of these sound familiar:

1."We've been talking about AI for 18 months but haven't shipped anything."
You need someone to cut through the noise and create a real plan with clear milestones.

2."We built something, but it's sitting unused."
You need help with change management, integration, and adoption — not just the model itself.

3."We don't know if our data is ready for AI."
A consultant will run a data readiness audit and tell you honestly what you have to work with.

4."We're worried about AI doing something harmful or non-compliant."
You need governance expertise before you go live not after.

5. "Our team knows how to code, but doesn't know where to start."
Strategy always comes first. A clear roadmap from an experienced team can save months of wasted effort.

For companies in retail, finance, healthcare, or any data-heavy industry, the gap between "wanting AI" and "using AI effectively" is often just a lack of structured guidance. Lucent Innovation's AI consulting experts work through exactly this process, from initial opportunity mapping all the way through to production deployment and ongoing governance.

What Tools Do AI Consultants Actually Use in 2026?

AI consultants are not just using ChatGPT. Here is a practical look at a real toolkit:

Strategy & Discovery

Miro or FigJam for workflow mapping
Notion or Confluence for roadmap documentation
Custom interview frameworks for stakeholder discovery

Data Readiness

Python (pandas, great_expectations) for data audits
dbt for data transformation pipelines
Databricks for large-scale data processing

Model Development

PyTorch or TensorFlow for custom model work
Hugging Face for open-source LLM access
OpenAI or Anthropic APIs for enterprise generative AI

Deployment & Monitoring

MLflow or Weights & Biases for experiment tracking
Kubeflow or AWS SageMaker for production deployment
Grafana or Datadog for monitoring and alerting

Governance

Fairlearn or IBM AI Fairness 360 for bias detection
AWS Macie or Microsoft Purview for data compliance

Here is an example of the kind of simple data readiness check a consultant might run before recommending any ML solution to a client. This is often the very first technical step:

# Basic AI Readiness Check — Data Quality Audit
# Run this before recommending any ML model to a client
# Gives a fast signal on whether the dataset is usable

import pandas as pd

def ai_readiness_check(df: pd.DataFrame) -> dict:
    """
    Quick data quality check before starting an AI project.
    Returns a readiness score and a list of key issues to fix.
    """
    issues = []
    score = 100

    # Check 1: Columns with >20% missing values
    missing_pct = df.isnull().mean() * 100
    high_missing = missing_pct[missing_pct > 20]
    if not high_missing.empty:
        issues.append(f"High missing data in: {list(high_missing.index)}")
        score -= 25

    # Check 2: Minimum row count for a usable ML dataset
    if len(df) < 1000:
        issues.append(
            f"Low row count ({len(df)} rows). "
            "Most ML models need at least 1,000 rows to train reliably."
        )
        score -= 20

    # Check 3: Too many duplicate rows
    dup_pct = df.duplicated().mean() * 100
    if dup_pct > 5:
        issues.append(f"{dup_pct:.1f}% duplicate rows found — clean before training.")
        score -= 15

    # Check 4: No numeric columns (most models need at least some)
    numeric_cols = df.select_dtypes(include="number").columns
    if len(numeric_cols) == 0:
        issues.append("No numeric columns found. Data may need encoding first.")
        score -= 20

    return {
        "readiness_score": max(score, 0),
        "issues_found": issues,
        "rows": len(df),
        "columns": len(df.columns),
        "recommendation": (
            "Good to proceed with model development"
            if score >= 70
            else "Fix data quality issues before building any model"
        ),
    }

# Example usage:
# df = pd.read_csv("your_business_data.csv")
# result = ai_readiness_check(df)
# print(result)

A lot of AI consulting begins exactly here with the data, not the model. Many companies believe they are "AI-ready" when their data tells a very different story. Running something like this before scoping a project saves weeks of rework.

For businesses that need this kind of structured, end-to-end support from data readiness assessments all the way through model governance, Lucent Innovation's AI strategy and implementation consulting covers the full lifecycle across industries including retail, healthcare, and financial services.

How to Pick an AI Consulting Partner in 2026 Without Regret

Lucy — Fri, 05 Jun 2026 06:44:07 +0000

Short answer: Hire an AI consulting partner the way you'd hire a senior engineer. Judge them on what they've shipped, not on the buzzwords in their deck. The good ones say no to bad-fit projects, put working code in front of you early, and tell you straight where AI won't help. The rest is theater.

Fair warning before we get going: I run an AI and data shop, so I've got skin in this. I'll flag it where it matters. This isn't a pitch, though it's the checklist I wish more founders used. Bad engagements are exactly what make this whole field smell like snake oil, and I'm tired of cleaning up after them.

Why does picking wrong hurt so much?

It's not the invoice. It's the quarter you burn, the engineers who quietly stop believing "AI" means anything real, and the brittle demo that folds the second production data touches it.

And the opportunity cost is brutal. While you were untangling someone's over-engineered RAG pipeline, a competitor shipped something boring that just worked. Speed-to-learning beats sophistication here almost every time, and the wrong partner optimizes for the wrong one.

What should I actually look for?

Skip the logo wall. When you're weighing AI consulting services, here's what actually tells you something:

Shipped systems, not slides. Ask to see something running. Real work leaves a trail — repos, dashboards, eval numbers.
Opinionated scoping. A good partner tells you which 80% of your idea to cut for v1. Say-yes-to-everything means they're selling hours, not outcomes.
Data honesty. The first hard question should be about your data: where it lives, how messy it is, who owns it. Nobody asks? Walk.
An exit ramp. You want to own the code, the model choices, the docs. Anyone building you a black box only they can maintain is building themselves a job not solving your problem.

Here's the thing about good AI strategy consulting: it starts from your business constraint, not from a model. If the opening call is about which LLM to pick rather than which decision you're trying to improve, that's a yellow flag.

How do I avoid getting burned?

Three moves that have saved me and people I trust.
Run a paid pilot first. Two to four weeks, tight scope, one real deliverable. And pay for it — free pilots pull the wrong incentives on both sides. You'll learn more in one honest sprint than in five sales calls.

Then ask for a reference who had a project go sideways. Anyone can hand you a happy logo. The question that actually works is, "Tell me about an engagement that didn't go to plan." How they answer tells you how they'll treat you when something breaks. Because something will.

And make them explain their evals. If they can't tell you how they'll measure whether the thing works accuracy, latency, cost per call, hallucination rate they're guessing. Guessing is fine at a hackathon. Not on your budget.

Teams that work this way are happy to scope a small, honest pilot before asking for the big commitment. For transparency, that's roughly how our own AI consulting practice runs but honestly, the principle matters more than the vendor. Hold whoever you're evaluating to it.

Build, buy, or partner at all?

Not every problem needs a consultant. Strong ML engineers and a clear use case? Build it. A SaaS tool already covers 90%? Buy that. Partnering earns its keep when the problem is real, the stakes are high, and you need to move faster than hiring allows or when you want your own engineers learning next to people who've done it before.

If you do bring someone in, treat them like a teammate with an expiry date. Your team should be sharper when they leave, not more dependent. The right AI strategy consulting engagement hands over knowledge on the way out. That's the whole difference between a partner and a crutch.

Takeaway: Judge partners on shipped work, sharp scoping, and data honesty. De-risk with a short paid pilot and a brutal reference check. Insist on owning what gets built. The best one leaves your team stronger — and then leaves.

Batch vs Streaming Pipelines: How I Actually Choose Between Them

Lucy — Fri, 05 Jun 2026 06:43:12 +0000

Every data pipeline starts with one big question before a single line of code gets written.

Should I process data in scheduled chunks? Or should I process it the moment events arrive?

That is the batch vs streaming decision. It sounds simple. But in real projects, it shapes everything: which tools you pick, how much you spend each month, what guarantees you can make about fresh data, and how many nights you spend fixing production incidents.

I have seen teams pick streaming when batch would have worked just fine. I have also seen the opposite. Both mistakes are expensive. This post walks through how I think about it.

What Batch Processing Actually Means

Batch processing collects data over a time window and then processes it all at once when a scheduled trigger fires.

Think about doing laundry. You do not wash one shirt the moment it gets dirty. You wait until you have a full load, then run the machine. The shirts pile up during the week. On Sunday, the machine runs.

Data batch pipelines work the same way. Source data builds up in a staging area. At a set time, usually overnight or hourly, a job picks up everything that arrived, runs the transformations, and loads the results into the destination.

The batch job has a clear start. It has a clear end. When it finishes, the destination has a snapshot of data as of the run time. Between runs, nothing changes.

What batch is great at:

Batch handles complex transformations well because there is zero time pressure per record. A batch job can join across tables with hundreds of millions of rows. It can run expensive multi-level calculations. It can apply feature engineering for machine learning without worrying about processing each event in milliseconds.

Batch pipelines are also much easier to test, debug, and rerun. When a transformation gives wrong results, you fix the logic and reprocess the affected time window. The worst thing that happens is a delayed job, not a production fire.

Where batch falls short:

Batch produces stale data. How stale depends on the schedule. Nightly jobs produce data up to 24 hours old. Hourly jobs produce data up to 60 minutes old.

For use cases where decisions depend on what is happening right now, that staleness is a real problem.

A fraud detection system that runs on a nightly batch schedule is not a fraud detection system. It is a fraud reporting system. The fraud already happened hours ago.

What Streaming Processing Actually Means

Streaming treats data as a continuous flow of individual events. Each event gets processed the moment it arrives, without waiting for others to pile up first.

Think about a moving walkway at an airport. People step onto the walkway as they arrive. Each person moves forward right away. Nobody waits for 500 people to gather before the walkway starts moving. The walkway runs all day whether one person is on it or ten thousand.

A streaming pipeline works the same way. An event source like Apache Kafka, Amazon Kinesis, or Google Pub/Sub delivers events in real time. The stream processor picks up each event, applies the transformation logic, and writes the result downstream within milliseconds to seconds. The pipeline runs 24 hours a day, seven days a week.

What streaming is great at:

Streaming is right when the output of the pipeline needs to trigger an action or update a system in real time.

Fraud detection needs to check whether a transaction looks suspicious before approving it. That decision cannot wait 60 minutes for the next batch run.

An e-commerce recommendation engine that adapts to clicks, cart additions, and browsing behavior as they happen gives a fundamentally different experience than one running on overnight batch data.

Infrastructure health dashboards that catch CPU spikes, error rate increases, or latency anomalies need second-level data, not hourly summaries.

Where streaming falls short:

Streaming infrastructure is a lot more complex to run than batch.

Stream processing introduces distributed processing requirements, state management, and fault tolerance mechanisms that batch engineers rarely deal with. Systems consume compute resources at all times rather than only during defined job windows.

Two failure modes in streaming catch teams off guard. The first is backpressure: incoming events exceed processing capacity, lag builds up, and outputs start describing events from minutes ago instead of seconds ago.

The second is silent correctness drift. Streaming systems often keep running even when data quality issues occur. Duplicate events, missing events, or schema changes can slowly corrupt outputs while dashboards still show active data.

The Comparison at a Glance

Dimension	Batch	Streaming
How data moves	Collects over time, processes in one run	Each event processed the moment it arrives
Latency	Minutes to hours	Milliseconds to seconds
Infrastructure	Compute spins up for the job, shuts down after	Always on, always running
Cost	Lower baseline, pay only when jobs run	Higher baseline, persistent infrastructure
Complexity	Lower, simpler error handling	Higher, state management and fault tolerance required
Failure mode	Delayed job, rerun and recover	Production incident, live intervention needed
Debugging	Rerun the job on the failed time window	Replay events from the message queue checkpoint
Schema change	Pipeline breaks loudly on next run	Can cause silent issues if not monitored

The One Question That Decides It

One question cuts through most of the debate: what happens if the data is one hour old?

If the answer is nothing meaningful, batch is probably the right choice.

If the answer is a real business loss, streaming earns its complexity.

Streaming is justified when the output triggers action. If the output only feeds retrospective analysis, batch is usually sufficient.

Four Questions to Ask Before Picking

1. How fresh does the data need to be to be useful?

Most analytics use cases tolerate data that is a few hours old. A weekly revenue report does not need second-level freshness. A fraud detection engine does. Know the actual freshness requirement before assuming you need streaming.

2. Does stale data cause a real business loss?

If a customer gets a product recommendation based on yesterday's browsing instead of what they clicked five minutes ago, does that cost the business money? If yes, streaming may be worth it. If it is a marginal difference, batch is almost certainly the right choice.

3. What is the operational capacity of your team?

Streaming infrastructure needs engineers who understand state management, checkpointing, exactly-once delivery semantics, and how to respond to backpressure incidents at midnight. If your team is small or your use case does not demand real-time results, that complexity is cost without benefit.

4. Is real-time the actual requirement, or is faster batch enough?

Stakeholders often say they want real-time when what they mean is they want data more current than nightly. A pipeline that runs every 15 minutes often satisfies that requirement at a fraction of the cost and complexity of a true streaming system.

When stakeholders say "real-time" but would accept hourly updates without meaningful business impact, they want faster batch, not streaming.

Real Use Cases: When Each Pattern Wins

When Batch Is the Right Answer

Nightly financial reporting. A bank's end-of-day ledger reconciliation processes every transaction from the day against regulatory limits and account balances. The job needs to run across the full day's dataset, apply complex multi-table joins, and produce a validated snapshot. Batch runs at end of day. Streaming adds nothing here.

ML model training. Training a machine learning model requires a large, static dataset processed multiple times. Streaming the training data adds enormous complexity without improving model quality.

Large-scale historical ETL. Migrating three years of transactional data into a new warehouse schema is a batch workload. The data already exists. There is no real-time requirement. Batch processes it once and moves on.

Compliance reporting. Monthly, quarterly, or annual regulatory reports that pull and aggregate data across long time windows are batch workloads. The business cost of a slightly delayed report is low. The complexity of a streaming system is not justified.

When Streaming Is the Right Answer

Fraud detection. Payment authorization systems need to evaluate whether a transaction is fraudulent before it clears, typically in under 500 milliseconds. A batch pipeline running every 30 minutes would approve or deny transactions without the context of what happened in the last 30 minutes.

Real-time feature serving for ML inference. When a deployed ML model needs features computed from recent user behavior to make a prediction, streaming pipelines update the feature store in real time. A recommendation model running on features from last night's batch is operating blind to today's context.

Live operational dashboards. A supply chain control tower showing current inventory levels, in-transit shipments, and order status across hundreds of warehouses needs second-level freshness. An overnight batch job cannot surface a stockout until the next morning.

IoT and sensor telemetry. In manufacturing, logistics, and energy, IoT devices generate continuous streams of sensor data that batch pipelines were not built to ingest or process. Predictive maintenance models that detect equipment issues before failure require streaming ingestion of live sensor data.

The Middle Ground Teams Often Miss: Micro-Batch

Between batch and streaming sits micro-batch processing. It is the pattern that Apache Spark Structured Streaming uses by default, and it solves most "near real-time" requirements without the full complexity of continuous streaming.

Micro-batch runs the same pipeline logic as streaming but on a very short fixed interval: every 30 seconds, every minute, every 5 minutes. Data builds up for the interval, then the batch processes it. Latency is measured in seconds to low minutes rather than hours.

Most use cases that stakeholders describe as "real-time" actually tolerate micro-batch latency. A dashboard that refreshes every minute looks real-time to every user. A data freshness requirement of "under 5 minutes" is achievable with micro-batch at a fraction of the streaming infrastructure cost.

Here is how the decision tree actually looks in practice:

Hours of latency are fine: standard batch on a schedule
Minutes of latency are fine: micro-batch with short trigger intervals
Sub-minute latency is required and the output triggers action: true streaming with Spark Structured Streaming
Sub-second latency is required: Real-Time Mode on Databricks Spark Structured Streaming

The Real Cost of Streaming: What Teams Underestimate

A simple batch ETL pipeline costs between $15,000 and $50,000 to build. A production streaming pipeline with proper monitoring costs between $50,000 and $200,000 or more. That is a 4x to 10x difference at the build stage alone.

Operational cost compounds on top of that. Streaming systems need always-on compute, persistent state storage, continuous monitoring for lag and backpressure, and engineers who can respond to incidents at any hour.

Three costs teams consistently underestimate:

State management. Streaming pipelines that compute windowed aggregations, sessionization, or joins across event streams must maintain state across every event. State grows with data volume. Managing state storage, checkpointing, and cleanup is a continuous engineering concern with no equivalent in batch.

Exactly-once delivery. Guaranteeing that each event is processed exactly once, not duplicated or dropped, requires careful coordination between the message queue, the stream processor, and the output destination. Getting this wrong means silent duplicate records or missing events in production.

Schema evolution. When a source system changes its event schema, a batch pipeline fails loudly on the next scheduled run. A streaming pipeline may silently accept the new schema, produce corrupt output, and keep running for days before anyone notices.

None of this means streaming is wrong. It means streaming should be chosen when the use case justifies the cost, not because it sounds more modern than batch.

Lambda vs Kappa: Two Ways to Run Both at Once

Many production systems need both patterns. Two architectural approaches define how teams organize that combination.

Lambda Architecture

Lambda runs two parallel pipelines. A batch layer reprocesses the full historical dataset on a schedule and produces accurate, complete results. A speed layer processes real-time events and produces approximate but current results. A serving layer merges outputs from both and delivers whichever is more current and accurate.

The batch layer produces trusted, complete data. The speed layer fills in the gap between now and the last batch run. When the batch layer catches up, it overrides the speed layer's approximate output.

Lambda works well when accuracy matters for historical data but approximate freshness is acceptable for recent data. The real cost is operational: two separate pipelines to build, test, and maintain.

Kappa Architecture

Kappa replaces the dual-pipeline design with a single streaming pipeline that handles everything. All data, historical and real-time, flows through the same stream processor.

Historical reprocessing works by replaying events from a durable message queue like Apache Kafka, which retains events for a configurable window. To reprocess, you replay from the beginning of the queue through the same pipeline code. No separate batch layer needed.

Kappa is simpler to maintain but requires your message queue to retain data long enough to support replays. It also requires that your transformation logic works correctly as a streaming pipeline, which rules out certain types of complex, multi-pass batch transformations.

Quick Reference: Which Pattern for Which Use Case

Use Case	Pattern	Why
Nightly revenue reporting	Batch	Data freshness within hours is fine
ML model training	Batch	Requires full static dataset
Historical data migration	Batch	Data already exists, no real-time constraint
Fraud detection	Streaming	Decision must happen before transaction clears
Real-time ML feature serving	Streaming	Model inference needs current behavioral context
IoT anomaly detection	Streaming	Equipment failure cannot wait for next batch
Live inventory dashboards	Streaming	Stockout response needs current state
Monthly compliance reports	Batch	Fixed window, no freshness urgency

My Rule of Thumb

Before you write a line of code, ask: does the output of this pipeline trigger an action, or does it inform analysis?

If it triggers an action and that action loses value after a few minutes, build streaming.

If it informs analysis and the insights hold up for a few hours, build batch.

And if your stakeholders say "real-time" but can actually accept updates every few minutes, build micro-batch. It gives you most of the freshness at a fraction of the cost.

The goal is not to use the most impressive technology. The goal is to ship the simplest system that meets the actual latency requirement and does not wake anyone up at 3 AM.

This post is part of a series on modern data engineering. For more on how these patterns connect to ETL vs ELT design choices, how Databricks handles both batch and streaming in one platform, and how to design for schema evolution at scale, check out the Modern Data Engineering Guide.

How to Set Up Local Data Engineering Environments with Docker Compose

Lucy — Thu, 28 May 2026 09:17:33 +0000

TL;DR: Docker Compose lets you spin up a full local data stack — Airflow, PostgreSQL, Spark, Redis — with a single YAML file and one command. This guide walks you through the exact setup, real compose configs, and the mistakes most engineers make along the way.

Why Your Local Data Environment Is Probably a Mess

Here's the thing: most data engineers I know have a local setup that technically works — but only on their machine, on a good day, when the stars align.

You install PostgreSQL manually. Pin a Python version. Struggle to get Airflow running without breaking something else. And then a new teammate joins and spends three days just trying to reproduce your environment.

That's not an engineering problem. It's a tooling problem. And Docker Compose solves it.

Docker Compose lets you describe your entire local data stack as code — services, networks, volumes, environment variables — and spin it up or tear it down with one command. No more "works on my machine." No more three-day onboarding nightmares.

This guide covers the full picture: what Docker Compose actually is (and what it's not), the building blocks you need to understand, and a production-quality example stack with Airflow, PostgreSQL, Redis, and Spark.

What Is Docker Compose (and What It's Not)

Docker Compose is an orchestration tool for defining and running multi-container Docker applications on a single machine. You write a compose.yaml file that describes each service — what image it uses, what ports it exposes, how it connects to other services, and where it stores data.

A quick note before we go further: Docker Compose v1 reached end-of-life in July 2023. The old docker-compose binary (with the hyphen) is gone. You should be using Docker Compose v2, which ships as a built-in CLI plugin. If you see docker-compose anywhere in your scripts or tutorials, replace it with docker compose (space, no hyphen).

Also worth knowing: the version: field at the top of your compose file is now officially deprecated. You don't need it. Drop it entirely from any new file you write.

Docker Compose is NOT:

A replacement for Kubernetes in production
A tool for managing distributed multi-machine deployments
A substitute for proper secrets management in prod

But for local development, CI pipelines, and single-machine staging? It's hard to beat.

Sources: Docker official documentation — docs.docker.com/compose; freeCodeCamp Docker Compose v2 guide (2026); Docker Compose specification at compose-spec.io

Prerequisites

Before we write a single line of YAML, make sure you have:

Docker Desktop (v4.0+) or Docker Engine + docker-compose-plugin on Linux
At least 8GB RAM available (data stacks eat memory)
4 CPU cores minimum — Spark in particular needs headroom
Basic familiarity with the command line

Run this to confirm your setup is current:

docker compose version
# Should show v2.24 or later in 2026

If that command fails or shows v1.x, update Docker before continuing.

Understanding the Core Building Blocks

Before jumping into the full stack, you need a mental model of the four things Docker Compose actually manages.

Services

A service is a running container. Each entry under services: in your compose file becomes one or more containers. For a data engineering stack, your services are things like your database, your orchestrator, your message broker, your transformation tool.

Networks

By default, every service in a compose file can talk to every other service using the service name as the hostname. No IP addresses. No manual DNS. This is one of the most underrated features — your Airflow scheduler connects to Postgres by literally using postgres as the hostname.

Volumes

Volumes are how your data survives container restarts. There are two flavors: named volumes (managed by Docker, recommended for databases) and bind mounts (a folder on your host machine mounted into the container, useful for DAGs, scripts, and code you're actively editing).

Environment Variables

Never hardcode credentials in your compose file. Always use a .env file and reference variables with ${VARIABLE_NAME} syntax. Your .env file stays out of version control. Your compose.yaml doesn't.

Building the Stack: A Real Data Engineering Environment

Let's build something real. This stack covers the tools that appear in most data engineering workflows:

Service	Role	Port
PostgreSQL	Metadata DB + data warehouse	5432
Redis	Message broker for Celery tasks	6379
Apache Airflow	Workflow orchestration	8080
Apache Spark	Distributed data processing	4040, 7077
Adminer	Lightweight DB GUI	8085

Step 1 — Project Structure

Start with a clean folder structure. This matters more than most tutorials admit — messy folders create tangled volume mounts and confusing build contexts.

data-eng-local/
├── compose.yaml
├── .env
├── .env.example
├── airflow/
│   ├── dags/
│   ├── logs/
│   ├── plugins/
│   └── config/
├── spark/
│   └── jobs/
├── postgres/
│   └── init/
│       └── 01_create_schemas.sql
└── README.md

Two rules: put your compose.yaml at the root, and never commit .env to git. Add it to .gitignore now, before you forget.

Step 2 — The `.env` File

# .env — DO NOT commit to version control
POSTGRES_USER=dataeng
POSTGRES_PASSWORD=changeme_local
POSTGRES_DB=warehouse

AIRFLOW_UID=50000
AIRFLOW__CORE__FERNET_KEY=your_fernet_key_here
AIRFLOW__WEBSERVER__SECRET_KEY=your_secret_key_here

REDIS_PASSWORD=redis_local_pass

Generate a Fernet key with:

python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Step 3 — The `compose.yaml` File

Here's the full configuration. Read the inline comments — they explain the decisions, not just the syntax.

# compose.yaml — No version field needed (deprecated in Compose v2)

x-airflow-common: &airflow-common
  image: apache/airflow:3.0.4
  environment: &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres/${POSTGRES_DB}
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres/${POSTGRES_DB}
    AIRFLOW__CELERY__BROKER_URL: redis://:${REDIS_PASSWORD}@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ${AIRFLOW__CORE__FERNET_KEY}
    AIRFLOW__WEBSERVER__SECRET_KEY: ${AIRFLOW__WEBSERVER__SECRET_KEY}
    AIRFLOW_UID: ${AIRFLOW_UID}
  env_file:
    - .env
  volumes:
    - ./airflow/dags:/opt/airflow/dags        # Bind mount — edit DAGs without rebuilding
    - ./airflow/logs:/opt/airflow/logs
    - ./airflow/plugins:/opt/airflow/plugins
    - ./airflow/config:/opt/airflow/config
  depends_on:
    postgres:
      condition: service_healthy              # Wait for postgres to be ready — not just started
    redis:
      condition: service_healthy

services:
  # ─── DATABASE ──────────────────────────────────────────────────────────────
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data          # Named volume — data survives restarts
      - ./postgres/init:/docker-entrypoint-initdb.d     # SQL files run on first startup
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

  # ─── MESSAGE BROKER ────────────────────────────────────────────────────────
  redis:
    image: redis:7-alpine                    # Alpine = smaller image, same functionality
    command: redis-server --requirepass ${REDIS_PASSWORD}
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

  # ─── AIRFLOW ───────────────────────────────────────────────────────────────
  airflow-init:
    <<: *airflow-common
    entrypoint: /bin/bash
    command:
      - -c
      - |
        airflow db migrate &&
        airflow users create \
          --username admin \
          --password admin \
          --firstname Admin \
          --lastname User \
          --role Admin \
          --email admin@example.com
    restart: "no"                            # Run once and exit — not a long-running service

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 5
    restart: unless-stopped

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    restart: unless-stopped

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    restart: unless-stopped

  # ─── SPARK ─────────────────────────────────────────────────────────────────
  spark-master:
    image: bitnami/spark:3.5
    environment:
      - SPARK_MODE=master
    ports:
      - "4040:4040"                          # Spark UI
      - "7077:7077"                          # Spark master port
    volumes:
      - ./spark/jobs:/opt/spark-jobs
    restart: unless-stopped

  spark-worker:
    image: bitnami/spark:3.5
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=2G
      - SPARK_WORKER_CORES=2
    depends_on:
      - spark-master
    restart: unless-stopped

  # ─── DB GUI ────────────────────────────────────────────────────────────────
  adminer:
    image: adminer:latest
    ports:
      - "8085:8080"
    depends_on:
      - postgres
    restart: unless-stopped

# ─── VOLUMES ───────────────────────────────────────────────────────────────
volumes:
  postgres_data:        # Docker-managed, persists across down/up cycles
  redis_data:

Step 4 — Start the Stack

# First time — initialize Airflow's database and create the admin user
docker compose up airflow-init

# Then start everything else in detached mode
docker compose up -d

# Watch it come up
docker compose ps

# Stream logs for a specific service (useful for debugging)
docker compose logs -f airflow-scheduler

That's it. Once everything is green, your services are at:

Airflow UI: http://localhost:8080 (admin / admin)
Spark UI: http://localhost:4040
Adminer: http://localhost:8085 (server: postgres, user: dataeng)

Health Checks: The Feature Most People Skip

This is the part most tutorials skip — and it's genuinely important.

Without health checks, depends_on is almost useless. By default, Docker considers a container "started" the moment the process launches, not when the service inside it is actually ready to accept connections. PostgreSQL needs a few seconds to initialize. Redis needs a moment to come up. If Airflow tries to connect before they're ready, it crashes — and you end up with a confusing pile of restart loops.

The condition: service_healthy syntax in depends_on fixes this. It tells Docker: don't start this service until that other service's health check is passing. Pair it with a proper healthcheck block on the dependency, and your stack starts in the right order every time.

# This is how you do it properly
depends_on:
  postgres:
    condition: service_healthy  # ← This is the key

Without this, you're relying on timing. Timing is not a strategy.

Common Mistakes and How to Avoid Them

Hardcoding credentials in compose.yaml — Don't. Use .env files. It takes 30 seconds to set up and prevents you from accidentally committing passwords to a public repo. It happens more than you'd think.

Using docker-compose (v1) instead of docker compose (v2) — The old binary is dead. If you're copying configs from tutorials older than mid-2023, check for this.

Including version: in your compose file — This field is obsolete as of Docker Compose v2 and now triggers deprecation warnings in Docker Desktop. Remove it from any file you write or maintain.

Not pinning image versions — postgres:latest is a trap. One upgrade and your init SQL might fail, your connection string might change, your extension might not exist. Pin to postgres:16. Always.

Forgetting to add .env to .gitignore — Seriously. Do this first.

Running docker compose down -v when you meant docker compose down — The -v flag deletes named volumes. That means your database data is gone. There's no undo. Be very intentional with that flag.

Working With Your Stack Day-to-Day

Once it's running, these are the commands you'll reach for most:

# Check what's running and the health status
docker compose ps

# Tail logs from everything
docker compose logs -f

# Tail logs from one service only
docker compose logs -f airflow-worker

# Restart a single service without touching the rest
docker compose restart airflow-scheduler

# Run a one-off command inside a running container
docker compose exec postgres psql -U dataeng -d warehouse

# Open a shell in a container for debugging
docker compose exec airflow-webserver bash

# Stop everything (preserves volumes — safe)
docker compose down

# Nuclear option — stops everything AND deletes all volumes
docker compose down -v

Managing Multiple Environments with Profiles

Here's something worth knowing once your stack gets more complex: Docker Compose Profiles let you define services that only start in certain contexts. You tag a service with profiles: [dev] or profiles: [monitoring] and it only runs when you explicitly request that profile.

services:
  # This only starts when you run: docker compose --profile monitoring up
  prometheus:
    image: prom/prometheus:latest
    profiles:
      - monitoring
    ports:
      - "9090:9090"

This is how you keep a single compose.yaml that works for all environments — local dev, CI, staging — without maintaining multiple files. It's one of the features that quietly makes Compose much more production-capable than its reputation suggests.

E-E-A-T Reference: Tools, Versions, and Sources

Here's a quick reference table with the verified tool versions used in this guide — and where to find official documentation.

Tool	Version Used	Official Docs
Docker Compose	v2.24+	docs.docker.com/compose
Apache Airflow	3.0.4	airflow.apache.org
PostgreSQL	16	postgresql.org/docs
Redis	7 (Alpine)	redis.io/docs
Apache Spark	3.5 (Bitnami)	spark.apache.org/docs

Further reading:

Docker Compose Specification: compose-spec.io — the authoritative reference for all YAML syntax
Apache Airflow official Docker Compose setup: airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html
Data Engineering Zoomcamp (DataTalks.Club): practical hands-on course that covers Docker + Airflow + dbt workflows — datatalks.club/courses/data-engineering-zoomcamp.html
freeCodeCamp — "How to Use Docker Compose for Production Workloads" (March 2026): covers profiles, watch mode, and GPU support in depth

Wrapping Up

Setting up a reliable local data engineering environment used to be a day-long exercise in frustration. With Docker Compose, it's a one-time investment: write the compose.yaml once, commit it to your repo, and everyone on your team gets the exact same environment with docker compose up.

A few things to remember as you go:

Drop the version: field — it's deprecated
Always use condition: service_healthy in depends_on
Pin your image versions, not latest
Keep credentials in .env, never in the compose file
Name your volumes — data you can't recover isn't worth much

The config in this guide is a starting point. As your stack grows, you'll want to look at override files (compose.override.yaml) for environment-specific tweaks, and Compose profiles for toggling monitoring tools, debug containers, or test databases without polluting your main setup.

The whole point is reproducibility. When a teammate clones your repo and runs docker compose up, they should get exactly what you have. That's not a nice-to-have — it's the baseline for any serious data engineering workflow.

Need a Data Engineer Who Already Knows This Stack?

Honestly, this is where a lot of projects stall. The setup is one thing — building production-grade pipelines on top of it, handling schema drift, optimizing Spark jobs, writing reliable Airflow DAGs that don't silently fail at 2am — that's the real work.

If your team is scaling and you need someone who has done this before (not just read about it), that's what we do.

Lucent Innovation helps product teams and growing companies hire experienced data engineers — people who can own the full stack, from local containerized environments to cloud-scale data infrastructure on AWS, GCP, or Azure.

Whether you need to augment your current team with a specialist or are looking to build a data engineering function from scratch, we can help you find and place the right person fast — usually within 2–3 weeks.

💬 Talk to us about hiring a data engineer →

No long intake forms. Just a conversation about what you're building and what you need.

Have questions about this setup or running into issues with a specific service? The comments are open. Also worth checking: the official Apache Airflow Docker Compose documentation gets updated with each release and is often more current than any tutorial you'll find.

Why Your In-House Databricks Team Is Probably Losing You Money

Lucy — Wed, 27 May 2026 10:44:20 +0000

60% of enterprise AI projects get abandoned because of data readiness and infrastructure issues.

Not because of bad ideas. Not because of wrong tooling. Because the foundation wasn't built right and by the time anyone noticed, the cost of fixing it was higher than starting over.

If you're running Databricks in-house, there's a decent chance you're heading toward one of four failure modes. I've seen each of them play out, sometimes in the same org.

1. The "unicorn engineer" job post

You know the one. It asks for someone who can handle platform architecture, complex ETL pipeline design, MLOps, and data governance. Maybe Unity Catalog experience preferred. Definitely Spark optimization. Oh, and some Python.

That person doesn't exist. Or if they do, they're already at a FAANG and not answering your recruiter.

What actually happens: you hire someone capable, and they spend most of their time on operational noise that manually partitioning tables, babysitting cluster configs, debugging integration issues that have nothing to do with your actual data problems.

Databricks has gotten genuinely complex. Delta Lake, Lakeflow Declarative Pipelines, Unity Catalog- these aren't plug-and-play. A generalist data engineer in 2026 is not the same as a Databricks platform specialist.

A consulting partner brings people who've already built this across multiple clients. You're not buying hours. You're buying what they learned the hard way somewhere else multi-cloud workspace topology, Liquid Clustering, private endpoint configs without waiting for your team to acquire those scars.

2. The cloud bill no one is watching

Here's one I've seen kill otherwise solid data platforms quietly.

In-house team gets the pipelines working. Everyone moves on. Nobody sets up auto-termination. Nobody enforces cluster policies. Clusters run indefinitely. Variable workloads stay on always-on compute when they should be hitting Serverless SQL.

[Traditional In-House Setup] ---> Over-provisioned Clusters ---> High Idle Waste & Skyrocketing Bills
[Consulting-Led Framework] ---> Serverless SQL + Cluster Policies ---> Automated Auto-Termination & Controlled Spend

The bill climbs slowly, and then suddenly it's a boardroom conversation.

A proper FinOps setup isn't exciting work, but it has a direct, measurable line to your cloud costs. Things like mandatory auto_termination_minutes, enforced instance pool configs, and routing the right workloads away from always-on clusters. This is table stakes, it just often doesn't get done when you're underwater on pipeline work.

3. Governance that gets bolted on after the fact

The pattern is almost universal:

Build the pipelines
Ship the dashboards
Deal with governance "later"

By the time "later" arrives, you've got fragmented data silos, ML models stuck in sandbox environments, inconsistent access controls, and no data lineage. Then someone asks about compliance.

Unity Catalog isn't an afterthought, it's the thing you configure before the pipelines, not after. Role-based access controls, automated data quality monitoring, end-to-end lineage tracking. If these aren't in the foundation, your downstream reports are unreliable by design.

The uncomfortable truth: A lot of teams treat governance like a documentation task. It's not. It's infrastructure.

4. The hiring timeline nobody accounts for

Realistic timeline from job post to a team that's onboarded, trained on Databricks, and actually productive:

6–9 months.

That's not pessimism, that's just recruiting + onboarding + platform ramp-up. Most orgs don't factor this in when they're comparing in-house costs against consulting rates.

A consulting firm gets there faster because they're not starting from scratch. Pre-built IaC templates, established Bronze/Silver/Gold ingestion patterns, CI/CD already wired up. Deployment that takes your internal team six months can happen in weeks.

That gap matters if your competitors are already running predictive analytics in production.

So what actually works?

It's not a binary choice, and framing it that way is usually how you end up making the wrong call.

The companies that handle this well use a hybrid model:

Bring in specialists for the hard setup — architecture, Unity Catalog, cluster optimization, MLOps scaffolding
Keep internal team focused on domain knowledge, custom data products, and the business problems that actually need context to solve

Your internal engineers understand your data, your customers, and your edge cases. That's valuable and hard to transfer. But asking them to also be platform infrastructure experts is how you end up with both things done poorly.

TL;DR

Problem	In-house default	What fixes it
Skill gaps	Overhire, underdeliver	Consulting for platform-specific work
Cloud costs	Idle compute, no policies	FinOps framework from day one
Governance	Bolted on later	Unity Catalog before pipelines
Speed	6–9 months to productivity	Pre-built templates + IaC

The architecture decisions you make in the first few months of a Databricks deployment are surprisingly hard to undo. Getting them right upfront — even with outside help — is almost always cheaper than refactoring a broken foundation at scale.

Have you gone through a Databricks migration or build-out? Curious what broke first — drop it in the comments.

RAG or Fine-Tuning? How We Decide for Our AI Consulting Clients

Lucy — Thu, 21 May 2026 07:22:26 +0000

Choosing the right architecture for an artificial intelligence product is one of the most expensive decisions a business can make. When clients come to Lucent Innovation for AI consulting, they often ask the same core question: should we use RAG or fine-tuning?

Many teams assume they need to train a custom model from scratch to make an AI understand their business. However, making the wrong choice can lead to hundreds of thousands of dollars in wasted cloud computing bills and months of lost development time.

This guide breaks down the choice in simple, plain English. Whether you are a software engineer building the pipeline or a business leader managing the budget, this framework will help you make the right architectural choice.

What is RAG in AI?

To understand your choices, we must begin with the basics of Retrieval-Augmented Generation.

What does RAG stand for in AI?

RAG stands for Retrieval-Augmented Generation. In simple terms, it is an architectural approach that gives a generative AI model an open-book exam.

Instead of relying solely on what the model learned during its initial training, a RAG AI system looks up real-time information from an external database before it answers a user query.

[User Query] ──> [Search External Database] ──> [Retrieve Relevant Text] ──> [Feed into RAG LLM] ──> [Final Accurate Answer]

How does RAG improve the accuracy of generative AI models?

Standard Large Language Models (LLMs) are frozen in time. They only know the data they were trained on. If you ask a standard model about a customer invoice from yesterday, it will either admit it does not know or confidently make up a false answer. This false answer is called a hallucination.

A RAG LLM setup solves this problem by executing a simple multi-step process:

The Retrieval Step: When a user asks a question, the system searches a private corporate database or vector store for matching documents.
The Augmentation Step: The system takes those matching documents and pastes them directly into the hidden prompt background.
The Generation Step: The model reads the question and the pasted documents together, synthesizing a perfectly accurate answer based strictly on the provided facts.

By grounding the model in verified data, you eliminate guessing and ensure that the system can access real-time, constantly changing information.

The Core Battle: RAG vs Fine Tuning

While RAG gives the model a library card, LLM fine tuning is completely different. Fine-tuning actually changes the internal brain structure of the model.

Understanding LLM Fine Tuning

When you fine tune LLM models, you take an existing base model and expose it to a highly specialized dataset for intensive training. This process adjusts the internal weights of the neural network. You are not giving the model an open-book exam: you are sending it back to school to learn a specific style, dialect, or structural format.

Here is an engineering visual to help conceptualize the foundational pathways:

RAG vs LLM: The Core Differences

To see why this matters for your engineering budget, consider this comparison table of operational trade-offs:

Evaluation Feature	RAG AI Systems	LLM Fine Tuning
Knowledge Base Type	Dynamic and real-time external data	Static snapshot baked into the model
Primary Use Case	Finding specific facts and text chunks	Learning a specific style, tone, or format
Hallucination Control	Very high: sources can be cited directly	Low: can still invent facts if prompt is weak
Upfront Setup Cost	Low to moderate developer hours	High compute costs and specialized data engineering
Data Privacy Boundaries	Easy to restrict data via database permissions	Difficult to restrict access once data is baked in

When to Use Fine Tuning vs RAG?

The choice between fine tuning vs RAG comes down to a simple engineering rule: Use RAG for knowledge, and use fine-tuning for behavior.

The Unique Lucent Innovation Point of View: The Data Lifecycle Reality

Most online guides tell you to evaluate your choice based purely on accuracy. At Lucent Innovation, we tell our enterprise clients to look at something completely different: look at who owns the data and how fast it changes.

If your data changes every hour, every day, or every week, fine tuning LLMs is a terrible operational trap. The moment your business updates a pricing sheet or changes a product feature, your fine-tuned model becomes obsolete. You would have to spend thousands of dollars to retrain it again.

RAG fine tuning decisions should follow these strict operational guidelines:

Choose RAG when:

You need to connect your AI to live business documents, customer support wikis, or internal Slack logs.
You must show users exactly where the information came from by providing source citations and links.
You need to build your product quickly without renting expensive GPU clusters for training cycles.

Choose Fine-Tuning when:

You need the model to output perfect, strict JSON code structures every single time without fail.
You want the AI to perfectly mimic a specific person's copywriting style, voice, or industry jargon.
You are working with an ultra-niche domain (like advanced medical pathology reports or ancient legal statutes) that the base model cannot comprehend.

RAG vs Fine Tuning vs Prompt Engineering?

Before jumping into a complex software architecture, engineers should always evaluate the entire spectrum of optimization. This brings us to a three-way comparison: RAG vs fine tuning vs prompt engineering.

[Prompt Engineering] ──> Simple instructions in the text box (Minutes to set up)
[RAG Architecture]   ──> Hooking up a search engine to the text box (Days to set up)
[Fine-Tuning]        ──> Re-wiring the underlying engine itself (Weeks to set up)

Prompt engineering is the foundation. It involves writing clever, descriptive instructions directly inside your system prompt. For instance, telling a model to "act like a professional accountant" is prompt engineering.

The Decision Spectrum

Prompt Engineering: Best for fast prototyping, basic text transformations, and setting up initial rules.
RAG vs Prompt Engineering: When your system prompt gets too full of information, it hits a wall. Standard context windows can become slow and expensive. That is when you step up to RAG, which selectively feeds only the relevant data chunks into the prompt instead of dumping the entire database.
Fine-Tuning: The final step. Once your RAG system knows what to say, you can use fine-tuning to perfect how it says it, shrinking your prompt sizes and reducing latency.

Real World Client Scenario: How We Consult

To make this practical, let us look at a real architecture challenge we solved for one of our enterprise consulting clients.

The client wanted an AI assistant to help their customer success team look up technical product specifications and write email responses in the company's precise tone of voice.

Instead of picking just one path, we deployed a hybrid strategy:

The RAG Layer: We hooked up their product documentation manuals to a vector database pipeline. This ensured that the AI always retrieved 100 percent accurate product specifications, eliminating hallucinations.
The Fine-Tuning Layer: We took the base open-source model and fine-tuned it on 5,000 historical customer service emails that were manually approved by their marketing team. This taught the model's brain to always write responses with a helpful, warm, and structured corporate tone.

By combining the open-book access of RAG with the behavioral habits of fine-tuning, the client achieved a 40 percent reduction in average ticket handling time while keeping errors at absolute zero.

Conclusion: Designing Your AI Roadmap

There is no single winner in the battle of RAG vs fine tuning. They are complementary tools designed for completely different software problems.

If your product goals require access to fresh facts, internal knowledge bases, and clear data source tracking, building a RAG framework is your optimal choice. If your product demands strict adherence to complex code layouts or deep alignment with a specific brand persona, investing in custom weights is the right path forward.

Get Expert Engineering Guidance

Navigating these architectural decisions requires deep hands-on experience. Making a mistake early in your development cycle can result in severe technical debt and bloated maintenance costs.

At Lucent Innovation, we specialize in helping businesses design, build, and optimize high-performance AI systems that drive real business outcomes. We analyze your data dynamics, security requirements, and budget constraints to engineer the perfect pipeline for your platform.

Are you unsure which approach fits your upcoming product? This is exactly what our engineering team helps clients figure out every day. Let us protect your runway and accelerate your deployment timeline. Book a free discovery call with the Lucent Innovation AI consulting team today.

Foundational Sources & Technical Reading

Learn more about the mechanics of Retrieval-Augmented Generation on the Databricks Lakehouse Platform Architecture.
Review foundational research and code guidelines on Large Language Model Fine-Tuning via OpenAI Developer Documentation.
Explore semantic indexing protocols via the Pinecone Vector Database Engineering Blog.

DEV Community: Lucy

Shopify AI Search Optimization: How to Make Your Store Visible Inside ChatGPT, Perplexity, and Google AI Mode

Why Shopify Merchants Can't Ignore AI Search Anymore

How Do AI Shopping Assistants Actually Find Your Products?

What Structured Data Does an AI Engine Actually Need From a Product Page?

Does llms.txt Actually Help a Shopify Store Get Cited?

How Should Product and Blog Content Be Written So AI Engines Want to Quote It?

A Practical GEO Checklist for Shopify Merchants

Who Should Actually Run This Audit?

Frequently Asked Questions

Shopify Checkout UI Extension Development: A Step-by-Step Guide (2026)

What You're Building (and What This Guide Skips)

Checkout UI Extensions vs. Shopify Functions: Which One Do You Actually Need?

Prerequisites

Scaffold the Extension

Read Checkout Data With the shopify Global Object

Write Changes Back to the Order

Test the Extension Before It Touches Real Checkouts

Deploy It, and Respect the 64 KB Ceiling

Where Do Checkout Extensions Actually Break in Production?

Build It Yourself or Bring in a Shopify Team?

What's Next

From Liquid to Hydrogen: A Developer's Honest Guide to Going Headless on Shopify

What Liquid Gets Right (and Where It Runs Out)

What "Going Headless" Actually Means

Inside Hydrogen: What's New in 2026

Liquid vs. Hydrogen: A Straight Comparison

The Same Data Fetch, Two Ways

When Headless Actually Pays Off (and When It's Overkill)

How Teams Actually Land on an Answer

Shopify Scripts Are Dead — Here's How to Migrate to Shopify Functions Before June 30, 2026

What's actually changing and why

What Are Shopify Scripts?

Why Shopify Is Deprecating Them

Scripts vs. Functions: What Actually Changed

What Gets Replaced by What?

Enter Shopify Functions

Migration: Step by Step

Step 1: Audit your current Scripts

Step 2: Set up your Shopify Functions environment

Step 3: Scaffold the right Function type

Step 4: Rewrite your logic in JavaScript or Rust

Step 5: Test in a development store

Step 6: Deploy to production and monitor

Common Migration Pitfalls

1. Scripts that relied on execution order

2. Bundle logic mapped to the wrong Function type

3. Assuming Shipping Scripts are low priority

4. Edge cases left uncoded

5. JavaScript Functions on complex carts

6. Not reading the GraphQL input schema

What If You're Already Behind?

Quick Recap

Migrate Your Data to Delta Lake: A Simple Guide for Developers

The Problem With Old Data Lakes

What Is Delta Lake?

The Big Features That Matter

ACID Transactions (No More Broken Data)

Time Travel (Go Back in Time)

Schema Enforcement (Keep Data Clean)

Automatic Updates and Deletes

Why Move to Delta Lake?

Cost Savings

Speed and Trust

Real-Time and Batch in One Place

Easy Audits and Compliance

How to Start Your Migration

Step 1: Check Your Current Setup

Step 2: Start Small

Step 3: Set Up Your Infrastructure

Step 4: Copy Your Data Over

Step 5: Test Everything

Step 6: Switch Over and Watch It

Step 7: Keep the Old System as Backup

Simple Example: Your First Delta Lake Table

Real Costs vs. Old Systems

Common Questions

Next Steps

How to Speed Up Your Shopify Store in 5 Easy Steps for Better Performance

Introduction

Read Checkout Data With the `shopify` Global Object

Step 2 — The `.env` File

Step 3 — The `compose.yaml` File