DEV Community: Amaan Khan

One Test Case, Every Layer: REST, OpenAI, AWS, Parameters, and Variables Together

Amaan Khan — Wed, 12 Aug 2026 01:45:20 +0000

Every step in this walkthrough was generated by describing the scenario in plain English to AutomationCodeless's AI Test Step Assistant — not written by hand. The AI drafted the seven steps, chained the references between them, and picked reasonable field values; from there it was just a matter of reviewing the output and adjusting the parameter IDs and bucket names to match a real environment. Want to see it for yourself? Sign in at automationcodeless.com and you can execute these exact steps.

Most integration bugs live at the boundary between two systems — the place where a unit test can't see, because unit tests mock the boundary away instead of testing it. Add an AI step into the pipeline and you get a second kind of boundary: not just "did the two systems talk to each other correctly," but "did the model's output actually make sense, in a form the rest of the pipeline can use." This walkthrough builds one test case that exercises both — REST, AI classification, and AWS — end-to-end, using Parameters and a Variable to keep it from turning into a hardcoded mess.

The scenario

A support ticket comes in through a REST API. An OpenAI step classifies its severity. If the classification is critical, a downstream Lambda (outside the test — already deployed) writes an alert record to S3. We'll test that this entire chain actually works, not just that each piece works in isolation.

Step 1 — fetch a stored credential instead of hardcoding it

The ticketing API needs a bearer token. Rather than pasting it into every step that needs it, it's stored once as a Parameter and pulled in at runtime:

{
  "step_id": 1,
  "description": "Fetch the internal service token stored as a Parameter",
  "step_type": "core",
  "step_name": "get_parameter",
  "parameter_id": 21
}

Step 2 — create the ticket

{
  "step_id": 2,
  "description": "Create a new support ticket",
  "step_type": "rest",
  "step_name": "post",
  "value": {
    "url": "https://api.example.com/tickets",
    "headers": {
      "Content-Type": "application/json",
      "Authorization": "Bearer ${step[1].result_value}"
    },
    "requestBody": {
      "customerId": 8842,
      "subject": "Payment charged twice for order #5521",
      "description": "I was charged $129.00 twice for the same order. Please refund the duplicate charge immediately, this is affecting my business."
    }
  }
}

Typical response:

{
  "id": "TCK-90142",
  "status": "open",
  "createdAt": "2026-08-02T10:12:04Z"
}

Step 3 — store the ticket ID as a variable

The ticket ID gets referenced by several later steps. Rather than repeating ${step[2].result_value.id} everywhere, save it once as a named Variable:

{
  "step_id": 3,
  "description": "Store the new ticket ID for reuse in later steps",
  "step_type": "core",
  "step_name": "set_variable",
  "variable_name": "ticket_id",
  "value": "${step[2].result_value.id}",
  "data_type": "string"
}

Step 4 — classify severity with OpenAI

{
  "step_id": 4,
  "description": "Classify the ticket's severity from its description",
  "step_type": "open_ai",
  "step_name": "gpt-4-turbo",
  "parameter_id": 22,
  "value": "Classify this support ticket's severity as exactly one word — low, medium, high, or critical: ${step[2].result_value.description}"
}

For this ticket, the model returns critical.

Step 5 — assert the AI actually returned something usable

AI output isn't strongly typed the way a database column is. Before anything downstream trusts it, assert it's actually one of the values the rest of the pipeline knows how to handle:

{
  "step_id": 5,
  "description": "Assert the AI returned a recognized severity level",
  "step_type": "assert",
  "step_name": "oneOf",
  "value": "${step[4].result_value}",
  "expected_value": ["low", "medium", "high", "critical"]
}

Step 6 — confirm the downstream alert actually fired

A critical classification should trigger a Lambda (already deployed, outside this test) that writes an alert record to S3. Check that it actually did:

{
  "step_id": 6,
  "description": "Confirm a critical-alert record was written for this ticket",
  "step_type": "aws",
  "step_name": "s3_get_object",
  "value": {
    "bucket": "support-critical-alerts",
    "key": "${ticket_id}.json",
    "region": "us-east-1",
    "wait_for_seconds": 10
  }
}

Step 7 — assert the alert points back to the right ticket

{
  "step_id": 7,
  "description": "Assert the alert record references the same ticket we created",
  "step_type": "assert",
  "step_name": "equal",
  "value": "${step[6].result_value.ticketId}",
  "expected_value": "${ticket_id}"
}

Why chain all of this into one test case

Each of these pieces could be tested in isolation — REST endpoint returns 200, OpenAI returns a string, S3 object exists. None of that proves the pipeline works. The bugs that actually reach production live in the connections: the Lambda trigger that silently stopped firing after a permissions change, the AI prompt that used to return "Critical" (capitalized) until a model update changed its formatting and broke every downstream string match, the ticket ID that gets truncated somewhere between systems. A single chained test case is what catches those — each step only passes if the step before it actually produced something usable.

Common pitfalls

Don't skip validating AI output. Treat a model's response as untrusted input, the same as anything from the network — assert its shape and allowed values before anything downstream depends on it.
Use variables for anything referenced more than twice. ${step[2].result_value.id} works fine once; repeating it across five steps makes the test fragile to reorder and harder to read. Save it once as a variable instead.
Add a wait for asynchronous side effects. The S3 alert in step 6 is written by a Lambda triggered asynchronously — checking too early is a false failure, not a real one.
Never hardcode credentials into a step. Pull them from Parameters, the same way step 1 does — it's the difference between updating one value when a token rotates and hunting through every test case that uses it.

Testing an AI-powered order flow end to end: API, database, AWS, OpenAI, and assertion.

Amaan Khan — Wed, 12 Aug 2026 01:32:42 +0000

I've spent 26 years in enterprise software — full stack, cloud architecture, DevOps across Java, Node, React, and AWS. Financial services, healthcare, big enterprise systems. And across every one of those environments, the same wall kept showing up, so I want to put it to the dev community here and see if your experience matches mine.

Unit and API testing are basically solved. Postman for APIs, Cypress/Playwright for UI, plenty of frameworks for unit tests. But integration testing — where you call an API, capture a value from the response, use it to query the database, confirm the AWS service reacted, then maybe validate what an AI model returned, all as one flow — falls into a gap. To do that today you're usually stitching together multiple tools plus custom scripts, which means it lands back on developers.

And that "lands back on developers" part is the bit I keep chewing on. The person who best understands the business logic being tested is often the BA or the QA analyst — not the dev who ends up writing the chain. So I've been wondering whether AI step generation actually changes this: if you can describe the flow in plain English ("create a user, check they're in the DB, confirm the welcome-email Lambda fired, validate the record with an AI check") and have the steps generated for you to review, does that finally let a non-coder own the integration test end to end? Or does it just move the problem — now they're reviewing generated steps they don't fully understand?

Why the chain is the hard part

A real integration test isn't a series of isolated calls — it's a chain where data flows between steps:

You need variables to capture an ID or token from step 1 and feed it into step 3.
You need assertions at every hop, not just at the end — did the API return 201, and did the row actually land in the DB, and did the Lambda return 200.
Sometimes a step needs real logic — transforming a payload, generating a signature, decoding a JWT — which is where a JavaScript step matters, because no visual builder covers every case.
And increasingly there's an LLM in the flow — you call OpenAI as a step and assert on what comes back, so you can actually test the AI-powered parts of your product alongside the deterministic ones instead of leaving them untested.

That combination — API + DB + AWS + JS + LLM, chained with variables and asserted at each step — is what nobody's really packaged for people who aren't developers. And the QA engineers, BAs, and product folks who understand the business logic best are exactly the ones locked out of building it themselves.

Making it concrete

Rather than hand-wave, here's the actual flow I keep coming back to as the test case: create a user → verify it landed in the DB → confirm the welcome-email Lambda fired → transform the result → AI-validate → assert at every hop → clean up. Each step captures a value the next one uses.

Worth stating up front, because it's the whole point of the earlier question: I didn't hand-write these 13 steps. I described the flow in plain English — roughly the sentence above — and the AI assistant drafted the steps, which I then reviewed and adjusted. So the JSON below isn't a hand-authored artifact I'm showing off; it's generated output. That's the experiment in practice — whether "describe it, then review" is enough to let someone who understands the business logic own the test without writing the chain by hand.

It starts with a REST call that creates the user and captures the response:

{
  "step_id": 1,
  "description": "Create a new user via the API",
  "step_type": "rest",
  "step_name": "post",
  "value": {
    "url": "https://api.example.com/users",
    "requestBody": "{\"name\": \"Alice Smith\", \"email\": \"alice@example.com\", \"role\": \"viewer\"}",
    "auth": { "type": "bearer", "token": "eyJhbGciOiJIUzI1NiI..." }
  },
  "result_status": "P",
  "result_value": "{\"status\":201,\"data\":{\"id\":456,\"email\":\"alice@example.com\"}}"
}

The next step captures the new ID into a variable so the rest of the chain can reach it, and an assertion confirms the API actually returned 201:

{
  "step_id": 2,
  "step_type": "core",
  "step_name": "set_variable",
  "variable_name": "newUserId",
  "value": "${step[1].result_value.data.id}"
},
{
  "step_id": 3,
  "step_type": "assert_2_value",
  "step_name": "equal",
  "value": "${step[1].result_value.status}",
  "expected_value": 201
}

Now the interesting hop — a 201 means the API said it wrote the user. The database step confirms it actually persisted, and the next assertion cross-checks the DB email against the one the API returned. Same newUserId, two systems, one consistent record:

{
  "step_id": 4,
  "step_type": "database",
  "step_name": "select",
  "value": "SELECT id, name, email, role FROM users WHERE id = ${newUserId}",
  "result_value": "[{\"id\":456,\"email\":\"alice@example.com\"}]"
},
{
  "step_id": 5,
  "step_type": "assert_2_value",
  "step_name": "equal",
  "value": "${step[4].result_value[0].email}",
  "expected_value": "${step[1].result_value.data.email}"
}

Then the async side effect everyone forgets to test — did the welcome-email Lambda actually fire? The AWS step invokes it and an assertion checks the status code:

{
  "step_id": 6,
  "step_type": "aws",
  "step_name": "lambda",
  "value": {
    "command": "invoke",
    "parameter": { "jsonValue": "{\"--function-name\": \"send-welcome-email\", \"--payload\": \"{\\\"userId\\\":\\\"${newUserId}\\\"}\"}" }
  },
  "result_value": "{\"StatusCode\":200,\"Payload\":\"{\\\"emailSent\\\":true}\"}"
}

When a step needs real logic that no visual builder covers, the JavaScript step handles it — here, parsing the DB rows and pulling the name out:

{
  "step_id": 8,
  "step_type": "java_script",
  "step_name": "execute_code",
  "value": "const rows = JSON.parse('${step[4].result_value}'); return rows.length ? rows[0].name : 'NOT_FOUND';",
  "result_value": "Alice Smith"
}

And the part I think is genuinely new — putting the LLM in the test. An OpenAI step validates the record and an assertion turns the model's answer into a hard pass/fail:

{
  "step_id": 10,
  "step_type": "open_ai",
  "step_name": "gpt-4-turbo",
  "value": "Is this a validly formatted email address: ${step[4].result_value[0].email}? Reply with only 'yes' or 'no'.",
  "result_value": "yes"
},
{
  "step_id": 11,
  "step_type": "assert_2_value",
  "step_name": "equal",
  "value": "${step[10].result_value}",
  "expected_value": "yes"
}

Finally the chain cleans up after itself with a wait and a delete, so the test is repeatable and leaves no orphaned row behind.

The through-line: one newUserId threads an HTTP response into a SQL row, into a Lambda payload, into a JS transform, into an AI check — and every hop has its own assertion. That's the "chain with visibility into every step" idea in practice, and it's the class of bug that never shows up in unit tests and always shows up in production.

Full 13-step workflow JSON (click to expand)

[
  {
    "step_id": 1,
    "screen": 1,
    "description": "Create a new user via the API",
    "step_type": "rest",
    "step_name": "post",
    "value": {
      "url": "https://api.example.com/users",
      "headers": "{\"Content-Type\": \"application/json\"}",
      "requestBody": "{\"name\": \"Alice Smith\", \"email\": \"alice@example.com\", \"role\": \"viewer\"}",
      "auth": {
        "type": "bearer",
        "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
      },
      "timeout": 5000
    },
    "execution_start": "2026-07-29 13:12:44",
    "execution_end": "2026-07-29 13:12:44",
    "result_duration": 420,
    "result_status": "P",
    "result_value": "{\"status\":201,\"data\":{\"id\":456,\"name\":\"Alice Smith\",\"email\":\"alice@example.com\",\"role\":\"viewer\"}}"
  },
  {
    "step_id": 2,
    "screen": 1,
    "description": "Capture the new user's ID from the API response",
    "step_type": "core",
    "step_name": "set_variable",
    "variable_name": "newUserId",
    "data_type": "string",
    "value": "${step[1].result_value.data.id}",
    "execution_start": "2026-07-29 13:12:44",
    "execution_end": "2026-07-29 13:12:44",
    "result_duration": 0,
    "result_status": "P",
    "result_value": "Variable string set successfully"
  },
  {
    "step_id": 3,
    "screen": 1,
    "description": "Assert the API returned HTTP 201 Created",
    "step_type": "assert_2_value",
    "step_name": "equal",
    "value": "${step[1].result_value.status}",
    "data_type": "number",
    "expected_value": 201,
    "execution_start": "2026-07-29 13:12:44",
    "execution_end": "2026-07-29 13:12:44",
    "result_duration": 1,
    "result_status": "P",
    "result_value": "201 equal 201"
  },
  {
    "step_id": 4,
    "screen": 1,
    "description": "Verify the user row actually landed in the database",
    "step_type": "database",
    "step_name": "select",
    "parameter_id": 1,
    "value": "SELECT id, name, email, role FROM users WHERE id = ${newUserId}",
    "execution_start": "2026-07-29 13:12:44",
    "execution_end": "2026-07-29 13:12:45",
    "result_duration": 640,
    "result_status": "P",
    "result_value": "[{\"id\":456,\"name\":\"Alice Smith\",\"email\":\"alice@example.com\",\"role\":\"viewer\"}]"
  },
  {
    "step_id": 5,
    "screen": 1,
    "description": "Assert the DB email matches the email returned by the API",
    "step_type": "assert_2_value",
    "step_name": "equal",
    "value": "${step[4].result_value[0].email}",
    "data_type": "string",
    "expected_value": "${step[1].result_value.data.email}",
    "execution_start": "2026-07-29 13:12:45",
    "execution_end": "2026-07-29 13:12:45",
    "result_duration": 1,
    "result_status": "P",
    "result_value": "\"alice@example.com\" equal \"alice@example.com\""
  },
  {
    "step_id": 6,
    "screen": 1,
    "description": "Invoke the welcome-email Lambda that fires when a user is created",
    "step_type": "aws",
    "step_name": "lambda",
    "parameter_id": 3,
    "value": {
      "command": "invoke",
      "parameter": {
        "jsonValue": "{\"--function-name\": \"send-welcome-email\", \"--payload\": \"{\\\"userId\\\":\\\"${newUserId}\\\"}\"}"
      }
    },
    "execution_start": "2026-07-29 13:12:45",
    "execution_end": "2026-07-29 13:12:46",
    "result_duration": 1800,
    "result_status": "P",
    "result_value": "{\"StatusCode\":200,\"Payload\":\"{\\\"result\\\":\\\"ok\\\",\\\"emailSent\\\":true}\"}"
  },
  {
    "step_id": 7,
    "screen": 1,
    "description": "Assert the Lambda returned StatusCode 200",
    "step_type": "assert_2_value",
    "step_name": "equal",
    "value": "${step[6].result_value.StatusCode}",
    "data_type": "number",
    "expected_value": 200,
    "execution_start": "2026-07-29 13:12:46",
    "execution_end": "2026-07-29 13:12:46",
    "result_duration": 1,
    "result_status": "P",
    "result_value": "200 equal 200"
  },
  {
    "step_id": 8,
    "screen": 1,
    "description": "Transform the DB result: extract the user's name with custom JS logic",
    "step_type": "java_script",
    "step_name": "execute_code",
    "value": "const rows = JSON.parse('${step[4].result_value}'); return rows.length ? rows[0].name : 'NOT_FOUND';",
    "execution_start": "2026-07-29 13:12:46",
    "execution_end": "2026-07-29 13:12:46",
    "result_duration": 6,
    "result_status": "P",
    "result_value": "Alice Smith"
  },
  {
    "step_id": 9,
    "screen": 1,
    "description": "Assert the transformed name matches what we created",
    "step_type": "assert_2_value",
    "step_name": "equal",
    "value": "${step[8].result_value}",
    "data_type": "string",
    "expected_value": "Alice Smith",
    "execution_start": "2026-07-29 13:12:46",
    "execution_end": "2026-07-29 13:12:46",
    "result_duration": 1,
    "result_status": "P",
    "result_value": "\"Alice Smith\" equal \"Alice Smith\""
  },
  {
    "step_id": 10,
    "screen": 1,
    "description": "Ask OpenAI to validate the email format from the DB record",
    "step_type": "open_ai",
    "step_name": "gpt-4-turbo",
    "parameter_id": 2,
    "value": "Is this a validly formatted email address: ${step[4].result_value[0].email}? Reply with only 'yes' or 'no'.",
    "execution_start": "2026-07-29 13:12:46",
    "execution_end": "2026-07-29 13:12:48",
    "result_duration": 1900,
    "result_status": "P",
    "result_value": "yes"
  },
  {
    "step_id": 11,
    "screen": 1,
    "description": "Assert the AI confirmed the email is valid",
    "step_type": "assert_2_value",
    "step_name": "equal",
    "value": "${step[10].result_value}",
    "data_type": "string",
    "expected_value": "yes",
    "execution_start": "2026-07-29 13:12:48",
    "execution_end": "2026-07-29 13:12:48",
    "result_duration": 1,
    "result_status": "P",
    "result_value": "\"yes\" equal \"yes\""
  },
  {
    "step_id": 12,
    "screen": 1,
    "description": "Wait briefly before cleanup",
    "step_type": "core",
    "step_name": "wait",
    "value": 1000,
    "execution_start": "2026-07-29 13:12:48",
    "execution_end": "2026-07-29 13:12:49",
    "result_duration": 1000,
    "result_status": "P",
    "result_value": "Waited 1000ms"
  },
  {
    "step_id": 13,
    "screen": 1,
    "description": "Clean up: delete the test user from the database",
    "step_type": "database",
    "step_name": "delete",
    "parameter_id": 1,
    "value": "DELETE FROM users WHERE id = ${newUserId}",
    "execution_start": "2026-07-29 13:12:49",
    "execution_end": "2026-07-29 13:12:50",
    "result_duration": 1000,
    "result_status": "P",
    "result_value": "{\"affectedRows\":1}"
  }
]

What I actually want to know

A few things I'm curious whether this community agrees with:

Is "no-code testing" a dirty word, or does it have a place? I know the instinct — no-code tools generate brittle junk and you lose control. But I think there's a real distinction between no-code UI testing (genuinely hard, flaky) and no-code integration testing (API + DB + cloud assertions with variables passing between steps), where the steps are far more deterministic. Do you draw that line differently?
How are you testing the AI/LLM parts of your product? If your app calls OpenAI or similar, is that under test at all, or is it the untested corner everyone avoids because the output isn't deterministic? Genuinely want to know what's working here.
How is your team handling cross-system integration tests today? Custom framework? A pile of Postman collections plus SQL scripts run by hand? And do your QA folks write their own automation, or is it gated behind engineering — and what breaks when it's gated?

Full disclosure: this frustration is what pushed me to build a tool for it — automationcodeless.com. No-code workflows that chain REST APIs, databases, AWS, and OpenAI calls together, passing variables between steps with assertions at each one, plus JavaScript steps for custom logic and an AI assistant that drafts the steps from plain English so you review rather than write them. Runs on a cloud runner by default, with an optional local agent when you need those JavaScript steps or access to internal systems so data stays on your network. Free tier, no card, if anyone wants to poke holes in it. But I'm more interested in the discussion than the plug — the integration-testing-for-non-coders gap is real regardless of whether my tool is the answer. How are you all solving it?