A security check that runs once is evidence, while a security check that keeps running, has a controlled scope, produces reviewable artifacts, and fails visibly can become an engineering control.
We learned this while preparing WP Suite Deployment Access for the AWS Foundational Technical Review.
The final FTR validation was fast. AWS Partner Central now accepts AWS Well-Architected Framework Review reports and automatically checks whether they satisfy the required validation controls.
The work that mattered happened before we uploaded the report.
Answering the Well-Architected questions honestly forced us to compare the architecture we described with the system we had actually deployed. That process exposed a recurring problem: many of our security and recovery statements were true at a point in time, but we did not yet have a repeatable way to prove that they would remain true after the next release.
A successful scanner run could support one review. It could not tell us whether a future API change weakened an authorization boundary, whether a browser-dependent reCAPTCHA flow still worked on the CI runner, or whether an AWS Backup recovery point could actually be restored.
So we turned the evidence work into two scheduled GitLab pipelines:
- a controlled security-assurance pipeline;
- a separate reliability and disaster-recovery pipeline.
This article explains the structure, the mistakes we found while automating it, and why neither pipeline is allowed to approve itself.
The product boundary came first
WP Suite Deployment Access launches optional backend infrastructure into the buyer's AWS account through AWS Marketplace and CloudFormation.
The supported architecture can include components for:
- Amazon Cognito identity;
- AI and knowledge-base APIs;
- form and workflow processing;
- protected static delivery;
- scheduled backup and recovery resources.
The public CloudFormation templates show buyers and reviewers what the deployment intends to create. Private runtime artifacts are staged only during an authorized deployment.
That architecture creates an important testing constraint.
Our assurance jobs must never treat customer production accounts as scanner targets.
Before selecting any testing tool, we defined the control boundary:
- all automated tests run against a dedicated, production-like test deployment;
- targets are explicitly allowlisted;
- test data is disposable;
- destructive operations are limited to pipeline-created fixtures;
- denial-of-service, stress, flooding, password spraying, and uncontrolled destructive testing are excluded;
- availability degradation, unexpected cost, unintended data access, or secret exposure are stop conditions.
This boundary matters more than the scanner configuration.
A technically capable security tool pointed at the wrong environment is not an assurance control. It is an operational risk.
The security pipeline shape
The resulting security pipeline runs on a schedule and can also be started through controlled manual or evidence-producing paths.
At a high level, the stages look like this:
prepare
↓
application security regression
↓
browser-dependent validation
↓
API scan
↓
fixture cleanup
↓
evidence generation
↓
human review
The real pipeline contains several jobs, but this simplified sequence captures the responsibility of each phase.
1. Prepare
The preparation stage validates the approved targets and creates only the disposable resources required by that run.
For workflow tests, for example, the pipeline creates data with identifiers derived from the pipeline run. Cleanup can therefore distinguish test fixtures from pre-existing data.
Preparation also records the tool versions and relevant test configuration required to interpret the results later.
2. Application-specific security regression
Generic scanners are useful, but they do not understand all of the invariants that matter to an application.
We use Node.js regression tests for controls such as:
- expected Cognito authorization boundaries;
- denial of administrative operations to non-administrative identities;
- reCAPTCHA enforcement and replay behavior;
- rejection of tokens generated for the wrong action;
- webhook-signature validation;
- mutation resistance for presigned upload URLs;
- signed-cookie behavior for protected static routes;
- the rule that credentialed CORS must never be combined with a wildcard origin.
These tests express product-specific expectations directly.
If an endpoint begins returning a valid HTTP response while silently ignoring an authorization rule, a generic scanner may not recognize the business impact. An explicit regression assertion can.
3. Browser-dependent validation
Some security paths cannot be tested accurately with a plain HTTP client.
reCAPTCHA is a good example. A meaningful test may require a token produced by a real browser context, followed by a backend request that verifies the token's action, validity, and replay behavior.
We use Playwright for these browser-dependent cases.
This layer also taught us an important lesson about incomplete automation.
During an early run, the reCAPTCHA script behaved differently on the CI runner than it did in our local and deployed browser tests. The affected checks remained pending.
The wrong response would have been to treat the rest of the green pipeline as a pass.
Instead, the evidence report had to preserve the incomplete state until the runner-specific problem was understood and corrected.
A missing test is not a successful test.
4. API scanning
The pipeline exports the current API definitions from the deployed test environment and uses OWASP ZAP to scan the intended API surface.
ZAP adds a different type of coverage from the application-specific tests:
- passive response analysis;
- API-driven exploration;
- detection of common web and API weaknesses;
- an independent view of the deployed interface.
The scan is deliberately bounded.
Targets are allowlisted, concurrency is limited, and testing classes that could create availability or cost risk are excluded.
ZAP is one layer of the process, not the approval authority.
5. Cleanup
Cleanup is a first-class pipeline phase rather than a shell command appended to a test job.
The pipeline verifies that temporary workflow records, uploaded objects, and other run-specific fixtures are removed.
Cleanup evidence matters for two reasons.
First, disposable data should not gradually turn a test environment into an unreviewable collection of stale records.
Second, cleanup behavior is itself part of operational correctness. A test that succeeds but leaves unexpected resources behind has found another problem.
6. Evidence generation
The pipeline produces private artifacts containing the summaries, structured results, logs, tool versions, and cleanup evidence needed for review.
Automating this stage produced one of the more instructive failures in the project.
An early summarizer confused a scanner's confidence classification with its risk classification.
The scan itself had run correctly. The generated evidence represented part of the result incorrectly.
That is a serious defect because reviewers usually consume the summary before they inspect every raw artifact.
We corrected the summarizer and added validation around the evidence transformation.
The lesson was simple:
Evidence generation is production code.
A pipeline can execute every test correctly and still produce a misleading assurance record if its reporting layer is wrong.
Why the pipeline does not approve itself
A green CI pipeline is not the final approval.
After every evidence-producing run, a separate review item is created for a human reviewer.
The reviewer checks:
- security findings and their actual risk;
- failed, skipped, pending, or manually validated controls;
- the boundaries and exclusions of the run;
- cleanup evidence;
- tool and target versions;
- whether the result supports the intended assurance claim.
The final disposition is recorded separately from the pipeline status.
This separation prevents several common problems.
A script cannot silently redefine its own acceptance criteria. A newly skipped test cannot disappear inside an otherwise green result. A scanner warning cannot automatically become either a release blocker or an accepted risk without context.
Automation collects and organizes evidence, a responsible person remains accountable for interpreting it.
Recovery required a different pipeline
Security testing and disaster-recovery validation have different operational characteristics, so we do not run them as one large assurance job.
The security pipeline runs more frequently.
The reliability and DR path runs monthly and performs slower, more stateful validation.
Its most important rule is:
Do not test only whether backups exist. Test whether selected recovery points can be restored.
A backup job can report success while recovery is still blocked by permissions, encryption configuration, missing dependencies, an invalid procedure, or assumptions that have never been tested.
Our monthly process therefore works with actual AWS Backup recovery points.
At a high level, it:
- identifies fresh recovery points for the protected test resources;
- selects a bounded restore sample;
- restores DynamoDB and S3 data into isolated resources;
- validates known canary data or checksums;
- records the observed recovery behavior;
- removes the restored resources;
- generates evidence for human review.
The restore sample is intentionally bounded.
It does not claim that every protected resource is restored during every monthly run. Backup coverage and restore sampling are reported as separate concepts so a successful sample cannot accidentally be presented as proof that every resource was restored.
This distinction made the report more honest and more useful.
From internal control to product capability
The recovery work did not remain only inside our assurance repository.
WP Suite Deployment Access now includes an optional backup and disaster-recovery component in the deployment model.
When selected, it can configure scheduled AWS Backup recovery points for supported, deployment-tagged S3 and DynamoDB resources and copy them to a second AWS Region chosen by the buyer.
This is a recovery capability, not an active-active architecture.
It should not be described as:
- automatic regional failover;
- continuous replication;
- point-in-time recovery for every resource;
- a permanently running secondary production stack.
The intended model is simpler and more explicit: a separate regional copy remains available so selected data can be restored if the primary Region experiences a serious outage or the primary recovery points are unavailable.
The infrastructure definition is deployed into the buyer's AWS account, alongside the other selected WP Suite components.
The recurring assurance pipelines remain vendor-side controls against our dedicated test deployment. They do not scan customer production environments.
What the AWS review changed
The AWS Well-Architected review began as a validation requirement, but it became a way to find places where architectural claims were not yet represented by repeatable operational behavior.
By the time we submitted the WAFR report:
- security testing had become scheduled and reviewable;
- browser-dependent and application-specific controls complemented API scanning;
- incomplete automation remained visible rather than being counted as a pass;
- evidence generation had its own validation requirements;
- recovery testing used real backup restores;
- human approval was separated from CI success;
- a useful recovery control had become a buyer-selectable deployment capability.
WP Suite Deployment Access subsequently passed AWS Foundational Technical Review.
That statement applies to the named solution and the submitted review scope.
It does not mean AWS audited our source code, performed our penetration test, or guaranteed that the product has no risk.
The durable result is not the badge. The durable result is that the questions asked during the review continue to be answered after the review is over.
Practical takeaways
For teams building similar assurance workflows, these are the lessons I would carry forward.
Define the boundary before choosing the tools
Decide what can be tested, where it can be tested, and which activities are excluded before configuring a scanner.
Combine generic and application-specific tests
A scanner can identify broad classes of weaknesses. It cannot automatically understand every authorization, workflow, tenancy, or business invariant in your product.
Use a browser when the control depends on browser behavior
Do not replace browser-generated security context with a mock and then claim that the complete production path was tested.
Make incomplete controls visible
Pending, skipped, and manual checks belong in the evidence summary. They must not disappear behind the overall pipeline status.
Treat reporting code as production code
A wrong summary can invalidate otherwise correct test execution.
Test restores, not backup configuration
The existence of a vault, plan, or recovery point is not proof that your recovery procedure works.
Keep a human approval step
Automation should make review faster and more consistent, not eliminate accountability.
Productize controls only when the boundary remains clear
A useful internal control can become a customer-facing capability, but its limits must remain explicit. Scheduled cross-Region backups are valuable without being presented as automatic failover.
Top comments (0)