I've spent 26 years in enterprise software — full stack, cloud architecture, DevOps across Java, Node, React, and AWS. Financial services, healthcare, big enterprise systems. And across every one of those environments, the same wall kept showing up, so I want to put it to the dev community here and see if your experience matches mine.
Unit and API testing are basically solved. Postman for APIs, Cypress/Playwright for UI, plenty of frameworks for unit tests. But integration testing — where you call an API, capture a value from the response, use it to query the database, confirm the AWS service reacted, then maybe validate what an AI model returned, all as one flow — falls into a gap. To do that today you're usually stitching together multiple tools plus custom scripts, which means it lands back on developers.
And that "lands back on developers" part is the bit I keep chewing on. The person who best understands the business logic being tested is often the BA or the QA analyst — not the dev who ends up writing the chain. So I've been wondering whether AI step generation actually changes this: if you can describe the flow in plain English ("create a user, check they're in the DB, confirm the welcome-email Lambda fired, validate the record with an AI check") and have the steps generated for you to review, does that finally let a non-coder own the integration test end to end? Or does it just move the problem — now they're reviewing generated steps they don't fully understand?
Why the chain is the hard part
A real integration test isn't a series of isolated calls — it's a chain where data flows between steps:
- You need variables to capture an ID or token from step 1 and feed it into step 3.
- You need assertions at every hop, not just at the end — did the API return 201, and did the row actually land in the DB, and did the Lambda return 200.
- Sometimes a step needs real logic — transforming a payload, generating a signature, decoding a JWT — which is where a JavaScript step matters, because no visual builder covers every case.
- And increasingly there's an LLM in the flow — you call OpenAI as a step and assert on what comes back, so you can actually test the AI-powered parts of your product alongside the deterministic ones instead of leaving them untested.
That combination — API + DB + AWS + JS + LLM, chained with variables and asserted at each step — is what nobody's really packaged for people who aren't developers. And the QA engineers, BAs, and product folks who understand the business logic best are exactly the ones locked out of building it themselves.
Making it concrete
Rather than hand-wave, here's the actual flow I keep coming back to as the test case: create a user → verify it landed in the DB → confirm the welcome-email Lambda fired → transform the result → AI-validate → assert at every hop → clean up. Each step captures a value the next one uses.
Worth stating up front, because it's the whole point of the earlier question: I didn't hand-write these 13 steps. I described the flow in plain English — roughly the sentence above — and the AI assistant drafted the steps, which I then reviewed and adjusted. So the JSON below isn't a hand-authored artifact I'm showing off; it's generated output. That's the experiment in practice — whether "describe it, then review" is enough to let someone who understands the business logic own the test without writing the chain by hand.
It starts with a REST call that creates the user and captures the response:
{
"step_id": 1,
"description": "Create a new user via the API",
"step_type": "rest",
"step_name": "post",
"value": {
"url": "https://api.example.com/users",
"requestBody": "{\"name\": \"Alice Smith\", \"email\": \"alice@example.com\", \"role\": \"viewer\"}",
"auth": { "type": "bearer", "token": "eyJhbGciOiJIUzI1NiI..." }
},
"result_status": "P",
"result_value": "{\"status\":201,\"data\":{\"id\":456,\"email\":\"alice@example.com\"}}"
}
The next step captures the new ID into a variable so the rest of the chain can reach it, and an assertion confirms the API actually returned 201:
{
"step_id": 2,
"step_type": "core",
"step_name": "set_variable",
"variable_name": "newUserId",
"value": "${step[1].result_value.data.id}"
},
{
"step_id": 3,
"step_type": "assert_2_value",
"step_name": "equal",
"value": "${step[1].result_value.status}",
"expected_value": 201
}
Now the interesting hop — a 201 means the API said it wrote the user. The database step confirms it actually persisted, and the next assertion cross-checks the DB email against the one the API returned. Same newUserId, two systems, one consistent record:
{
"step_id": 4,
"step_type": "database",
"step_name": "select",
"value": "SELECT id, name, email, role FROM users WHERE id = ${newUserId}",
"result_value": "[{\"id\":456,\"email\":\"alice@example.com\"}]"
},
{
"step_id": 5,
"step_type": "assert_2_value",
"step_name": "equal",
"value": "${step[4].result_value[0].email}",
"expected_value": "${step[1].result_value.data.email}"
}
Then the async side effect everyone forgets to test — did the welcome-email Lambda actually fire? The AWS step invokes it and an assertion checks the status code:
{
"step_id": 6,
"step_type": "aws",
"step_name": "lambda",
"value": {
"command": "invoke",
"parameter": { "jsonValue": "{\"--function-name\": \"send-welcome-email\", \"--payload\": \"{\\\"userId\\\":\\\"${newUserId}\\\"}\"}" }
},
"result_value": "{\"StatusCode\":200,\"Payload\":\"{\\\"emailSent\\\":true}\"}"
}
When a step needs real logic that no visual builder covers, the JavaScript step handles it — here, parsing the DB rows and pulling the name out:
{
"step_id": 8,
"step_type": "java_script",
"step_name": "execute_code",
"value": "const rows = JSON.parse('${step[4].result_value}'); return rows.length ? rows[0].name : 'NOT_FOUND';",
"result_value": "Alice Smith"
}
And the part I think is genuinely new — putting the LLM in the test. An OpenAI step validates the record and an assertion turns the model's answer into a hard pass/fail:
{
"step_id": 10,
"step_type": "open_ai",
"step_name": "gpt-4-turbo",
"value": "Is this a validly formatted email address: ${step[4].result_value[0].email}? Reply with only 'yes' or 'no'.",
"result_value": "yes"
},
{
"step_id": 11,
"step_type": "assert_2_value",
"step_name": "equal",
"value": "${step[10].result_value}",
"expected_value": "yes"
}
Finally the chain cleans up after itself with a wait and a delete, so the test is repeatable and leaves no orphaned row behind.
The through-line: one newUserId threads an HTTP response into a SQL row, into a Lambda payload, into a JS transform, into an AI check — and every hop has its own assertion. That's the "chain with visibility into every step" idea in practice, and it's the class of bug that never shows up in unit tests and always shows up in production.
Full 13-step workflow JSON (click to expand)
[
{
"step_id": 1,
"screen": 1,
"description": "Create a new user via the API",
"step_type": "rest",
"step_name": "post",
"value": {
"url": "https://api.example.com/users",
"headers": "{\"Content-Type\": \"application/json\"}",
"requestBody": "{\"name\": \"Alice Smith\", \"email\": \"alice@example.com\", \"role\": \"viewer\"}",
"auth": {
"type": "bearer",
"token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
},
"timeout": 5000
},
"execution_start": "2026-07-29 13:12:44",
"execution_end": "2026-07-29 13:12:44",
"result_duration": 420,
"result_status": "P",
"result_value": "{\"status\":201,\"data\":{\"id\":456,\"name\":\"Alice Smith\",\"email\":\"alice@example.com\",\"role\":\"viewer\"}}"
},
{
"step_id": 2,
"screen": 1,
"description": "Capture the new user's ID from the API response",
"step_type": "core",
"step_name": "set_variable",
"variable_name": "newUserId",
"data_type": "string",
"value": "${step[1].result_value.data.id}",
"execution_start": "2026-07-29 13:12:44",
"execution_end": "2026-07-29 13:12:44",
"result_duration": 0,
"result_status": "P",
"result_value": "Variable string set successfully"
},
{
"step_id": 3,
"screen": 1,
"description": "Assert the API returned HTTP 201 Created",
"step_type": "assert_2_value",
"step_name": "equal",
"value": "${step[1].result_value.status}",
"data_type": "number",
"expected_value": 201,
"execution_start": "2026-07-29 13:12:44",
"execution_end": "2026-07-29 13:12:44",
"result_duration": 1,
"result_status": "P",
"result_value": "201 equal 201"
},
{
"step_id": 4,
"screen": 1,
"description": "Verify the user row actually landed in the database",
"step_type": "database",
"step_name": "select",
"parameter_id": 1,
"value": "SELECT id, name, email, role FROM users WHERE id = ${newUserId}",
"execution_start": "2026-07-29 13:12:44",
"execution_end": "2026-07-29 13:12:45",
"result_duration": 640,
"result_status": "P",
"result_value": "[{\"id\":456,\"name\":\"Alice Smith\",\"email\":\"alice@example.com\",\"role\":\"viewer\"}]"
},
{
"step_id": 5,
"screen": 1,
"description": "Assert the DB email matches the email returned by the API",
"step_type": "assert_2_value",
"step_name": "equal",
"value": "${step[4].result_value[0].email}",
"data_type": "string",
"expected_value": "${step[1].result_value.data.email}",
"execution_start": "2026-07-29 13:12:45",
"execution_end": "2026-07-29 13:12:45",
"result_duration": 1,
"result_status": "P",
"result_value": "\"alice@example.com\" equal \"alice@example.com\""
},
{
"step_id": 6,
"screen": 1,
"description": "Invoke the welcome-email Lambda that fires when a user is created",
"step_type": "aws",
"step_name": "lambda",
"parameter_id": 3,
"value": {
"command": "invoke",
"parameter": {
"jsonValue": "{\"--function-name\": \"send-welcome-email\", \"--payload\": \"{\\\"userId\\\":\\\"${newUserId}\\\"}\"}"
}
},
"execution_start": "2026-07-29 13:12:45",
"execution_end": "2026-07-29 13:12:46",
"result_duration": 1800,
"result_status": "P",
"result_value": "{\"StatusCode\":200,\"Payload\":\"{\\\"result\\\":\\\"ok\\\",\\\"emailSent\\\":true}\"}"
},
{
"step_id": 7,
"screen": 1,
"description": "Assert the Lambda returned StatusCode 200",
"step_type": "assert_2_value",
"step_name": "equal",
"value": "${step[6].result_value.StatusCode}",
"data_type": "number",
"expected_value": 200,
"execution_start": "2026-07-29 13:12:46",
"execution_end": "2026-07-29 13:12:46",
"result_duration": 1,
"result_status": "P",
"result_value": "200 equal 200"
},
{
"step_id": 8,
"screen": 1,
"description": "Transform the DB result: extract the user's name with custom JS logic",
"step_type": "java_script",
"step_name": "execute_code",
"value": "const rows = JSON.parse('${step[4].result_value}'); return rows.length ? rows[0].name : 'NOT_FOUND';",
"execution_start": "2026-07-29 13:12:46",
"execution_end": "2026-07-29 13:12:46",
"result_duration": 6,
"result_status": "P",
"result_value": "Alice Smith"
},
{
"step_id": 9,
"screen": 1,
"description": "Assert the transformed name matches what we created",
"step_type": "assert_2_value",
"step_name": "equal",
"value": "${step[8].result_value}",
"data_type": "string",
"expected_value": "Alice Smith",
"execution_start": "2026-07-29 13:12:46",
"execution_end": "2026-07-29 13:12:46",
"result_duration": 1,
"result_status": "P",
"result_value": "\"Alice Smith\" equal \"Alice Smith\""
},
{
"step_id": 10,
"screen": 1,
"description": "Ask OpenAI to validate the email format from the DB record",
"step_type": "open_ai",
"step_name": "gpt-4-turbo",
"parameter_id": 2,
"value": "Is this a validly formatted email address: ${step[4].result_value[0].email}? Reply with only 'yes' or 'no'.",
"execution_start": "2026-07-29 13:12:46",
"execution_end": "2026-07-29 13:12:48",
"result_duration": 1900,
"result_status": "P",
"result_value": "yes"
},
{
"step_id": 11,
"screen": 1,
"description": "Assert the AI confirmed the email is valid",
"step_type": "assert_2_value",
"step_name": "equal",
"value": "${step[10].result_value}",
"data_type": "string",
"expected_value": "yes",
"execution_start": "2026-07-29 13:12:48",
"execution_end": "2026-07-29 13:12:48",
"result_duration": 1,
"result_status": "P",
"result_value": "\"yes\" equal \"yes\""
},
{
"step_id": 12,
"screen": 1,
"description": "Wait briefly before cleanup",
"step_type": "core",
"step_name": "wait",
"value": 1000,
"execution_start": "2026-07-29 13:12:48",
"execution_end": "2026-07-29 13:12:49",
"result_duration": 1000,
"result_status": "P",
"result_value": "Waited 1000ms"
},
{
"step_id": 13,
"screen": 1,
"description": "Clean up: delete the test user from the database",
"step_type": "database",
"step_name": "delete",
"parameter_id": 1,
"value": "DELETE FROM users WHERE id = ${newUserId}",
"execution_start": "2026-07-29 13:12:49",
"execution_end": "2026-07-29 13:12:50",
"result_duration": 1000,
"result_status": "P",
"result_value": "{\"affectedRows\":1}"
}
]
What I actually want to know
A few things I'm curious whether this community agrees with:
Is "no-code testing" a dirty word, or does it have a place? I know the instinct — no-code tools generate brittle junk and you lose control. But I think there's a real distinction between no-code UI testing (genuinely hard, flaky) and no-code integration testing (API + DB + cloud assertions with variables passing between steps), where the steps are far more deterministic. Do you draw that line differently?
How are you testing the AI/LLM parts of your product? If your app calls OpenAI or similar, is that under test at all, or is it the untested corner everyone avoids because the output isn't deterministic? Genuinely want to know what's working here.
How is your team handling cross-system integration tests today? Custom framework? A pile of Postman collections plus SQL scripts run by hand? And do your QA folks write their own automation, or is it gated behind engineering — and what breaks when it's gated?
Full disclosure: this frustration is what pushed me to build a tool for it — automationcodeless.com. No-code workflows that chain REST APIs, databases, AWS, and OpenAI calls together, passing variables between steps with assertions at each one, plus JavaScript steps for custom logic and an AI assistant that drafts the steps from plain English so you review rather than write them. Runs on a cloud runner by default, with an optional local agent when you need those JavaScript steps or access to internal systems so data stays on your network. Free tier, no card, if anyone wants to poke holes in it. But I'm more interested in the discussion than the plug — the integration-testing-for-non-coders gap is real regardless of whether my tool is the answer. How are you all solving it?
Top comments (0)