DEV Community: Anand Pawar

3 Tests That Pass in LangFlow But Fail in n8n Production

Anand Pawar — Mon, 22 Jun 2026 18:50:41 +0000

You built a LangFlow prototype. Every test passed. You exported the flow, dropped it into n8n, and the first production run broke.

This is not a bug report. It is a pattern.

The three-year SDET who has built LangFlow prototypes but hit mysterious failures when deploying the same logic in n8n production already knows the feeling. The prototype felt solid. The production pipeline felt like a different language.

It is not. The difference is execution context. LangFlow runs in a notebook-like environment where state is forgiving and retries are invisible. n8n runs in a workflow engine where every node is a transaction boundary and every failure is final unless you explicitly handle it.

Here are the three tests that pass in LangFlow but fail in n8n production, and what they teach about building reliable AI pipelines.

Test 1: The "LLM Returns Valid JSON" Test

What passes in LangFlow: You send a prompt asking the model to return JSON. The response comes back as a string. You parse it with json.loads(). It works. You move on.

What fails in n8n: The model returns a string that starts with a code block. Or a trailing comma. Or a markdown fence. Or a preamble sentence before the JSON. Or nothing at all because the context window was exceeded.

Why the difference: LangFlow's Python node silently tolerates malformed output. If json.loads() fails, you see the error in the output panel and fix the prompt. n8n's JSON node does not retry. It does not fall back. It throws a structured error that stops the entire workflow.

The fix is not a better prompt. The fix is a validation layer that normalizes LLM output before parsing.

import json
import re

def extract_json(raw: str) -> dict:
    # Strip markdown fences
    cleaned = re.sub(r'^```

(?:json)?\s*', '', raw.strip())
    cleaned = re.sub(r'\s*

```$', '', cleaned)

    # Find the first { and last }
    start = cleaned.find('{')
    end = cleaned.rfind('}')

    if start == -1 or end == -1:
        raise ValueError("No JSON object found in response")

    candidate = cleaned[start:end+1]

    # Attempt parse
    try:
        return json.loads(candidate)
    except json.JSONDecodeError:
        # Try fixing trailing commas
        candidate = re.sub(r',\s*}', '}', candidate)
        return json.loads(candidate)

This function lives in a shared utility node in n8n. Every LLM call routes through it. It catches the edge cases that LangFlow silently absorbed.

What this teaches: Prototype environments hide fragility. The test that passes in LangFlow is not testing the output format. It is testing whether the model usually returns something parseable. Production requires a contract, not a hope.

Test 2: The "Context Window Fits" Test

What passes in LangFlow: You feed a document into a prompt. The document is 8,000 tokens. The model handles it. You test again with 12,000 tokens. Still fine. You declare the pipeline ready.

What fails in n8n: The production document is 18,000 tokens. The model truncates silently. Or the context window fills with system prompts and conversation history, leaving no room for the actual input. The output becomes generic. The test that passed now produces garbage.

Why the difference: LangFlow runs each prompt in isolation. You control exactly what goes in. n8n workflows accumulate state. A node that prepends conversation history, a sub-workflow that adds metadata, a loop that concatenates previous outputs — each one eats context. The prototype never tested this because the prototype never ran the full chain.

The fix is not a larger model. The fix is a context budget that is measured and enforced.

// n8n node: Context Budget Checker
// Place this before any LLM call

interface ContextBudget {
  maxTokens: number;
  reservedForOutput: number;
  systemPromptTokens: number;
  historyTokens: number;
}

function checkBudget(input: string, budget: ContextBudget): boolean {
  const available = budget.maxTokens - budget.reservedForOutput;
  const used = budget.systemPromptTokens + budget.historyTokens + estimateTokens(input);

  if (used > available) {
    throw new Error(
      `Context budget exceeded: ${used} tokens needed, ${available} available. ` +
      `Truncate input or increase model capacity.`
    );
  }

  return true;
}

function estimateTokens(text: string): number {
  // Rough estimate: 4 characters per token
  return Math.ceil(text.length / 4);
}

This node fails fast. It does not let the LLM run with an overstuffed context. The error message tells the operator exactly what to fix.

What this teaches: A test that passes in isolation is not a test of the system. The system includes every node that touches the context. LangFlow tests the prompt. n8n tests the pipeline. They are different things.

Test 3: The "Retry Is Free" Test

What passes in LangFlow: The LLM call fails. You click "Run" again. It works. You assume the transient error was a network blip. You do not write a retry handler.

What fails in n8n: The LLM call fails at 2 AM. The workflow stops. No one notices until morning. The data that should have been processed is stuck in an error queue. The test that passed in LangFlow never ran at 2 AM.

Why the difference: LangFlow is a development environment. You are the retry handler. You see the error, you decide what to do, you click the button. n8n is a production environment. It runs unattended. If you did not tell it what to do on failure, it does nothing.

The fix is not a simple retry. The fix is a retry with exponential backoff and a dead-letter queue.

// n8n sub-workflow: LLM Call with Retry
// Wraps the LLM node

const MAX_RETRIES = 3;
const BASE_DELAY_MS = 1000;

async function callWithRetry(prompt: string, attempt: number = 1): Promise<string> {
  try {
    // This is the actual LLM call
    return await $node["LLM"].call(prompt);
  } catch (error) {
    if (attempt >= MAX_RETRIES) {
      // Send to dead-letter queue
      await $node["Dead Letter Queue"].send({
        prompt,
        error: error.message,
        attempt
      });
      throw new Error(`LLM call failed after ${MAX_RETRIES} attempts`);
    }

    const delay = BASE_DELAY_MS * Math.pow(2, attempt - 1);
    await new Promise(resolve => setTimeout(resolve, delay));
    return callWithRetry(prompt, attempt + 1);
  }
}

The dead-letter queue is critical. It preserves the failed input so you can replay it after fixing the issue. Without it, the error is a black hole.

What this teaches: Prototypes do not have failure modes. Production does. The test that passes in LangFlow is testing the happy path. The test that matters in n8n is testing the unhappy path. If you only test the happy path, you are not testing.

What These Three Tests Teach

Each of these failures shares a root cause: the prototype environment and the production environment have different assumptions about reliability.

LangFlow assumes you are watching. It assumes you will handle edge cases manually. It assumes the input is clean and the model is cooperative.

n8n assumes you are not watching. It assumes every edge case must be handled explicitly. It assumes the input is dirty and the model is unreliable.

The three-year SDET who built LangFlow prototypes and hit mysterious failures in n8n production is not making a mistake. They are learning the difference between a prototype and a production system. That difference is not the tool. It is the contract.

LangFlow says: "I will show you what the model returns."

n8n says: "I will do exactly what you told me, every time, without asking."

The second one is harder. It is also the one that runs at 2 AM.

What to Do Next

If you are moving a LangFlow prototype to n8n, do not export the flow and hope. Do this instead:

Add a validation layer for every LLM output. Normalize before parsing.
Measure context usage before every LLM call. Fail fast, not silently.
Implement retry with backoff and a dead-letter queue. Assume every call will fail at least once.

These three changes will catch the tests that pass in LangFlow but fail in n8n. They will also teach you something about your pipeline that the prototype never revealed.

The prototype is a sketch. Production is the building. Do not confuse the two.

Which of your LangFlow tests have you never run at 2 AM?

3 Things Nobody Tells You About BDD Before Your Cucumber Suite Becomes a Maintenance Nightmare

Anand Pawar — Sat, 20 Jun 2026 21:17:07 +0000

You have six years of automation experience and a Cucumber suite that takes forty minutes to run, and you still can't tell me what a single feature file actually tests without reading three layers of glue code.

I've been there. You're the senior automation engineer on a team that adopted BDD because "business stakeholders need to read the tests." Except those stakeholders stopped reading them after sprint two. Now you're maintaining a suite where every scenario is a small novel, every step definition calls three other step definitions, and the person who wrote the original framework left the company eighteen months ago.

Here's what nobody tells you about BDD before your Cucumber suite becomes a maintenance nightmare.

Thing One: Gherkin Is Not a Programming Language, But You're Treating It Like One

Your feature files look like this:

Scenario: User completes checkout with valid payment
  Given the user is logged in with email "test@example.com" and password "Password123!"
  And the user has 3 items in their cart
  And the user's shipping address is "123 Main St, Springfield, IL 62701"
  And the user's billing address matches their shipping address
  When the user selects "Credit Card" as payment method
  And the user enters card number "4111111111111111"
  And the user enters expiry date "12/28"
  And the user enters CVV "123"
  And the user clicks "Place Order"
  Then the order confirmation page is displayed
  And the order number is shown
  And the user receives an email confirmation

This is not BDD. This is a test script written in a markup language that happens to parse into Cucumber.

The original promise of BDD was that scenarios describe behavior, not implementation. But somewhere along the way, teams started treating Gherkin as a readable scripting language. Every parameter gets inlined. Every step becomes a concrete action. The feature file becomes a play-by-play of the UI interaction.

The result? When the checkout flow changes — and it will — you have to update every scenario that touches checkout. Not the step definitions. The feature files themselves. Because the business logic is embedded in the Gherkin, not abstracted behind it.

Here's what a better version looks like:

Scenario: User completes checkout with valid payment
  Given the user has items ready to purchase
  When the user pays with a valid credit card
  Then the order is confirmed

That's it. Three lines. The "how" lives in the step definition. The "what" lives in the feature file. When the payment provider changes, you update one step definition, not twenty scenarios.

Your step definition file should be the place where complexity lives. Your feature files should be so simple that a product manager could read them in a standup and nod along. If your Gherkin looks like a test script, you've already lost.

Thing Two: Your Step Definitions Are a Dependency Graph, Not a Library

Here's the pattern I see in every struggling Cucumber suite:

// step_definitions/checkout_steps.ts
import { Given, When, Then } from '@cucumber/cucumber';
import { loginAsUser } from './auth_steps';
import { addItemsToCart } from './cart_steps';
import { fillShippingAddress } from './shipping_steps';

Given('the user has items ready to purchase', async function () {
  await loginAsUser.call(this, 'test@example.com', 'Password123!');
  await addItemsToCart.call(this, 3);
  await fillShippingAddress.call(this, '123 Main St');
});

This is a dependency graph masquerading as a test framework. Every step definition imports other step definitions. The call order matters. The shared this context accumulates state across steps. If one step fails, the entire scenario is poisoned, and you can't tell whether the failure is in the step that failed or in the accumulated state from the three steps before it.

The counterargument: "But we need to reuse steps across scenarios. That's the whole point of BDD."

I agree with reuse. I disagree with how you're doing it.

The fix is to separate your page objects or service clients from your step definitions entirely. Your step definitions should be thin wrappers that call a shared domain layer. Not other step definitions.

// domain/checkout.ts
export class CheckoutFlow {
  constructor(private page: Page) {}

  async prepareItemsForPurchase(): Promise<void> {
    await this.page.goto('/login');
    await this.page.fill('#email', 'test@example.com');
    await this.page.fill('#password', 'Password123!');
    await this.page.click('#login-button');
    await this.page.waitForURL('/dashboard');
    // ... add items, set address
  }

  async payWithCreditCard(): Promise<void> {
    await this.page.click('#credit-card-option');
    await this.page.fill('#card-number', '4111111111111111');
    await this.page.click('#place-order');
  }
}

// step_definitions/checkout_steps.ts
import { CheckoutFlow } from '../domain/checkout';

Given('the user has items ready to purchase', async function () {
  this.checkout = new CheckoutFlow(this.page);
  await this.checkout.prepareItemsForPurchase();
});

When('the user pays with a valid credit card', async function () {
  await this.checkout.payWithCreditCard();
});

Now your step definitions are stateless. They don't import each other. They don't share mutable context. Each step creates or calls a domain object that encapsulates the behavior. If a step fails, the failure is isolated to that step's domain operation.

Your step definition file becomes a routing table, not a tangled web of imports. You can delete a step definition without worrying about breaking three other scenarios that depend on it.

Thing Three: Your Cucumber Suite Is a Documentation Project That You're Treating as a Test Project

This is the one nobody wants to say out loud.

BDD was never primarily about testing. It was about communication. The original vision was that scenarios would serve as living documentation — a shared language between developers, testers, and business stakeholders. The tests were a side effect.

But somewhere in the last decade, teams started measuring BDD success by test coverage. "We have 500 Cucumber scenarios." "Our BDD suite runs in CI." "We achieved 90% Gherkin coverage."

None of those metrics tell you whether your scenarios are readable. None of them tell you whether a new team member can open a feature file and understand what the system does without running the tests.

Here's the uncomfortable truth: if your Cucumber suite is a maintenance nightmare, it's because you optimized for the wrong thing. You optimized for test execution speed, for step reuse, for parameterization. You forgot to optimize for reading.

I learned this the hard way. I had a suite with beautiful step definitions. Clean abstractions. Zero duplication. But when a product manager asked me what the checkout flow looked like, I had to open three feature files and trace through five scenario outlines to explain it. The documentation was technically there, but it was useless.

The fix is brutal but simple: every time you add a scenario, ask yourself whether someone who has never seen your codebase can understand it in under thirty seconds. If the answer is no, rewrite the scenario. Not the step definition. The scenario.

Your feature files should be the first thing a new engineer reads when they join your team. Not the README. Not the wiki. The feature files. If they can't understand the system from those files alone, your BDD suite has failed its primary purpose.

What to Do Tomorrow

Pick one feature file. The worst one. The one with the most steps, the most parameters, the most inline data.

Rewrite it to three to five lines per scenario. Move every concrete value into a step definition or a test data factory. Delete every step definition that imports another step definition. Replace them with domain objects.

Run the suite. If it passes, you've just made your documentation better and your tests more maintainable in one move.

Then do it again for the next file.

A Question for You

Your Cucumber suite runs in CI. It passes. Your team is shipping. But if you had to explain the system's behavior to a new hire using only your feature files, could you do it in under five minutes?

If the answer is no, you know where to start.

The Tester Who Had 10 Certifications But Couldn't Write a Single Test That Caught a Bug

Anand Pawar — Sat, 20 Jun 2026 09:29:18 +0000

You have ISTQB Foundation. ISTQB Advanced. Certified ScrumMaster. A cloud cert. A security testing cert. Maybe a Python for Testers badge from a platform.

And you still cannot write a test that finds a real bug.

I interviewed someone like you last quarter. The resume was a wall of acronyms. The conversation was a wall of theory. "I follow the V-model." "I use equivalence partitioning." "I believe in shift-left."

Then I asked: "Show me one test you wrote that caught something the developer missed."

Silence.

Not because they were nervous. Because they had never written a test that found a bug. They had written tests that passed. They had written tests that covered requirements. They had never written a test that broke something.

That is the difference between a certification holder and a tester.

Certifications test your memory. Bugs test your thinking.

Let me show you what I mean.

The Certification Trap

Certifications are not useless. They give you vocabulary. They give you structure. They give you something to put on LinkedIn so recruiters stop asking if you know what a test case is.

But they do not teach you how to find bugs.

Here is why. Every certification exam tests known knowledge. You study a syllabus. You memorize definitions. You answer multiple-choice questions about boundary value analysis. You pass.

Then you sit in front of an application. The application does not have a syllabus. It does not have a boundary value analysis section in the documentation. It has a login form that sometimes lets you in with a password that is clearly wrong, but only on Tuesdays, and only if the server clock is behind by exactly four minutes.

No certification prepares you for that.

The tester with 10 certifications treats testing like a checklist. They write test cases from requirements. They execute them. They mark pass or fail. They report coverage metrics.

The tester who finds bugs treats testing like an investigation. They start with a hypothesis. They try to prove the application wrong. They follow the weirdness.

What a Real Bug-Finding Test Looks Like

Let me show you the difference in code.

Here is a typical test written by someone who studied for certifications but never learned to hunt bugs:

import { test, expect } from '@playwright/test';

test('user can complete checkout with valid credit card', async ({ page }) => {
  await page.goto('/products');
  await page.click('[data-test="add-to-cart"]');
  await page.click('[data-test="checkout"]');
  await page.fill('[data-test="card-number"]', '4111111111111111');
  await page.fill('[data-test="expiry"]', '12/28');
  await page.fill('[data-test="cvc"]', '123');
  await page.click('[data-test="submit-payment"]');
  await expect(page.locator('[data-test="order-confirmation"]')).toBeVisible();
});

This test passes. It covers the happy path. It proves the feature works when everything is perfect.

It will never find a bug.

Now here is a test written by someone who thinks like a bug hunter:

import { test, expect } from '@playwright/test';

test('checkout rejects card number with typo in first digit', async ({ page }) => {
  await page.goto('/products');
  await page.click('[data-test="add-to-cart"]');
  await page.click('[data-test="checkout"]');

  // Card number with first digit changed from 4 to 5
  await page.fill('[data-test="card-number"]', '5111111111111111');
  await page.fill('[data-test="expiry"]', '12/28');
  await page.fill('[data-test="cvc"]', '123');
  await page.click('[data-test="submit-payment"]');

  // The application should reject this card
  // But does it show an error, or does it silently fail?
  await expect(page.locator('[data-test="error-message"]')).toBeVisible();

  // Also check: did the order confirmation accidentally appear anyway?
  const confirmationCount = await page.locator('[data-test="order-confirmation"]').count();
  expect(confirmationCount).toBe(0);
});

The second test is not more complex. It is more suspicious. It assumes the application will fail. It checks for the error. It also checks that the success state did not accidentally appear.

This is the mindset. Not "does it work?" but "where does it break?"

The Real Gap: You Have Never Failed a Test

Here is the honest truth for the reader I am speaking to directly. You, the tester with three years of experience and a stack of certs. You have never written a test that failed for a reason you did not expect.

You have seen tests fail because the environment was down. You have seen tests fail because the data changed. You have seen tests fail because someone updated the locator.

But you have never written a test that failed because you found a real, previously unknown defect in the application logic.

That is your career gap. Not a gap in your resume. A gap in your experience.

And no certification can fill it.

How to Close the Gap

You close it by writing tests that are designed to fail.

Not flaky tests. Not tests that fail because of timing. Tests that fail because the application has a bug.

Here is the exercise I give every tester I mentor. Pick one feature in your current application. Spend one hour writing tests that try to break it. Not tests that verify it works. Tests that verify it breaks correctly.

For a search feature, do not test that "search returns results for a valid query." Test that "search returns zero results for gibberish" and "search handles SQL injection characters without crashing" and "search with an empty string shows a helpful message instead of a 500 error."

For a file upload feature, do not test that "user can upload a valid PDF." Test that "uploading a file named ../../../etc/passwd does not overwrite system files" and "uploading a 2GB file shows a size error before the upload starts" and "uploading a file with no extension does not crash the preview generator."

Write these tests. Run them. When they fail, you have found a bug. When they pass, you have proven the application handles edge cases correctly. Either way, you learn something.

The Code That Changed My Mind

I was wrong about something for years. I thought good test automation meant covering every requirement with a passing test. I measured success by green builds.

Then I worked on a payment system for a fintech team. We had 95% test coverage. Every test passed. The system went to production and lost money for three days before anyone noticed.

The bug was in a rounding calculation. When the total was something like $10.005, the system rounded down for the customer but rounded up for the merchant. The difference was pennies per transaction. Over thousands of transactions, it was real money.

Our tests covered the happy path. They covered the sad path. They did not cover the "what if the decimal math is wrong" path because nobody thought to write that test.

After that, I changed how I write tests. Now every test starts with a question: "What would have to be true for this feature to fail silently?"

Here is a Python example of that mindset:

import pytest
from decimal import Decimal

def test_rounding_does_not_favor_any_party():
    """Verify that rounding is consistent between customer and merchant."""
    amounts = [
        Decimal("10.005"),
        Decimal("19.995"),
        Decimal("100.005"),
        Decimal("0.015"),
    ]

    for amount in amounts:
        customer_charge = round_to_cents(amount, party="customer")
        merchant_payout = round_to_cents(amount, party="merchant")

        # The total should be the same regardless of who we round for
        assert customer_charge == merchant_payout, (
            f"Rounding mismatch for {amount}: "
            f"customer pays {customer_charge}, merchant gets {merchant_payout}"
        )

This test found nothing in the first version. The developers had written correct rounding. But when we added a new currency with different decimal places, this test caught the bug in the first hour. It was already written. It was already running. It failed immediately.

That is the value of tests that are designed to find bugs. They sit there, waiting for the code to change, and they catch the regression before it reaches production.

What to Do This Week

You have the certs. You have the years. You do not have the proof.

Here is one action. Do not build a portfolio site. Do not update your LinkedIn. Do not study for another certification.

Write one test that is designed to fail. Not a test that verifies something works. A test that tries to prove something is broken.

Run it. If it passes, you have documented a safety guarantee. If it fails, you have found a bug. Either way, you have something real to show.

Write it up. Three paragraphs. What you tested. How you tested it. What you found. That is your portfolio. That is proof that you can think.

The next time someone asks you to show them a test that caught a bug, you will have an answer.

Not a certification. An answer.

What is the one test you would write this week to prove you can find bugs?