DEV Community

Fernando Azevedo
Fernando Azevedo

Posted on Edited on Originally published at fernando.moretes.com

Glue native SaaS connectors vs AppFlow, zero-ETL and custom code

#dataplatforms #awsglue #saasingestion #dataplatform

On November 9, 2026, AWS announced 100 new native connectors in AWS Glue for SaaS applications and REST APIs: payments (Adyen, Square, Checkout.com), accounting (Xero, QuickBooks), billing (Chargebee, Recurly, Zuora), operations (PagerDuty, Sentry, Jenkins, Vercel) and even usage and cost data from Anthropic. You create the connection once in the Glue Data Catalog and any job can reuse it under the same Glue permissions model, on Glue 5.0 and later, in every commercial Region. I've written and maintained acquirer API extractors by hand. They never broke at deploy time. They broke the day the vendor changed pagination and nobody noticed for a week. So this piece doesn't ask whether the connector works. It asks which of the four paths available today costs the least to maintain.

The real choice: four paths to get data out of a SaaS

When a finance team asks for "Adyen transactions in the lake by tomorrow", there are four answers on AWS today.

Glue native connector: a Spark job on Glue 5.0 reads the entity through the SaaS connection_type, with ENTITY_NAME, SELECTED_FIELDS, FILTER_PREDICATE in Spark SQL, and partitioning through PARTITION_FIELD, LOWER_BOUND, UPPER_BOUND and NUM_PARTITIONS. The secret lives in Secrets Manager, one per connection.

Glue zero-ETL: managed replication, no job, into Redshift, S3 or S3 Tables through the SageMaker lakehouse architecture. The problem is coverage: in the current documentation, the SaaS sources are Salesforce, Salesforce Marketing Cloud Account Engagement, SAP OData, ServiceNow, Zendesk, Zoho CRM, Facebook Ads and Instagram Ads. None of the payment connectors in the announcement are on that list.

Amazon AppFlow: a managed flow billed per run and per GB processed, good for scheduled copies with no transformation.

Custom extractor: Lambda or Step Functions calling the API and writing to S3. Full control, and full debt.

The announcement shifts the balance because the first path now covers the long tail that only the fourth used to cover. For sources outside the 100, there is still the Glue REST connector (RegisterConnectionType), with OAuth2 client credentials, authorization code, JWT, basic or custom auth, and cursor or offset pagination. So even the "my SaaS isn't on the list" case has a declarative way out before you reach for code.

The four paths side by side

Criterion Glue native connector Glue zero-ETL AppFlow Custom extractor
SaaS coverage 100 new + earlier ones; REST for the rest 8 SaaS applications today Its own connector catalog Any API
Transformation at ingestion Full Spark, QUERY and FILTER_PREDICATE None, it replicates Simple mapping and filtering Whatever you write
Cost model US$0.44/DPU-hour (Flex: US$0.29) By ingested volume, 1 MB minimum per request US$0.001/run + US$0.02/GB Lambda cents + engineering hours
Incremental You own the watermark via filter Managed Managed where the connector supports it You write and test it
Who absorbs API changes AWS (connector); you (schema contract) AWS AWS Your on-call

The incremental load is yours, and it's where payments break

The Stripe connector documentation spells out a detail that applies to much of the long tail: only field-based partitioning exists. There's no record-based partitioning (the connector can't retrieve a total count) and no ID-based partitioning, because the primary key is a string. In practice, parallelism and incremental loads depend on a time field you pick, and owning the watermark is your job.

The design I use is simple. A DynamoDB table with partition key source#entity stores the last processed created. The job reads that value, builds FILTER_PREDICATE with a 15 to 30 minute overlap window backwards, sets LOWER_BOUND and UPPER_BOUND on the same field and writes into an Iceberg staging table. A MERGE on the transaction's natural key (the acquirer's payment ID) makes reruns idempotent. Only after the commit does the job advance the watermark, with a ConditionExpression on the previous value so two concurrent runs can't both move the pointer.

The overlap exists because payment APIs aren't append-only. Refunds, chargebacks and status changes rewrite old records, and an updated field isn't always available as a filter. Without the window, you lose the update. Without the MERGE, you duplicate the transaction. Both bugs look the same: reconciliation against the bank statement doesn't close, and someone finds out at month-end close, not on the day.

Payment SaaS ingestion with a native connector and your own watermark

The connector handles auth and pagination; the watermark, the idempotent MERGE and publishing to Redshift are still yours. The dashed line is the watermark update, which only happens after the Iceberg commit. Simplified: one SaaS, one entity, no Lake Formation in the picture.

SaaS

  • Adyen API REST, paginated (external)

Control plane

  • Secrets Manager one secret per connection (security)
  • Glue connection Data Catalog (data)
  • Watermark DynamoDB source#entity (data)

Lake

  • Glue 5.0 job FILTER_PREDICATE (compute)
  • S3 Iceberg MERGE by payment ID (storage)
  • Redshift reconciliation (data)

Flows

  • conn -> secret: reads credential
  • job -> conn: uses connection
  • job -> wm: reads watermark
  • job -> saas: HTTPS, time window
  • job -> s3: idempotent MERGE
  • job -> wm: advances after commit
  • s3 -> rs: publishes

Cost: the DPU is the cheap part

Let me run the math for a payment source pulled hourly. A job with 2 G.1X workers (2 DPU) running 10 minutes consumes about 0.33 DPU-hours: roughly US$0.15 per run at US$0.44, or close to US$105 a month at 24 runs a day. On Flex, at US$0.29, it drops to about US$70, at the cost of a non-guaranteed start. The same volume on AppFlow, with 720 runs and 1 GB per run, lands around US$15: US$0.72 in runs and US$14.40 in data.

Looking only at the bill, AppFlow wins. But the bill isn't the cost. AppFlow charges for runs that found no new data and bills runs that failed on a misconfiguration as successful, and it doesn't transform: someone still has to deduplicate, type and MERGE afterwards. If that step is already a Glue job, you pay the DPU anyway and gain a second tool to operate.

The custom extractor has the lowest bill of all, Lambda cents. Its cost shows up elsewhere: credential rotation, API version changes, a new rate limit, a renamed field. On platforms I've maintained, one acquirer extractor ate something like two to three engineering days per quarter just to keep working, and that number never made it into the business case.

Rule I use: if the cadence is hourly or slower, start on Flex and move to standard only when the freshness SLO requires it.

Security and governance: what changes on a regulated platform

The most concrete governance gain is the connection as a Data Catalog resource. Before, every extractor had its own way of storing an API key, almost always an environment variable someone copied across accounts. Now there's one secret per connection in Secrets Manager, and access goes through IAM.

Three adjustments I make before releasing to production:

Role scope: the documentation example grants secretsmanager:GetSecretValue, PutSecretValue and the ENI actions on "Resource": "*". In an environment audited under BACEN 4.893 or PCI-DSS, I narrow the resource to the connection's secret ARN and use aws:ResourceTag/data-domain so the finance job can't read the Gusto secret, which holds payroll data.

Minimization at the source: SELECTED_FIELDS isn't only an optimization. If the acquirer's transaction report carries cardholder data the analysis doesn't need, don't pull it. What never enters the lake never enters audit scope.

Network: the connection accepts its own network options. Running the job in a private subnet with egress through a fixed-IP NAT makes the SaaS-side allowlist easier, and some acquirers require it.

Finally, observability: an alarm on glue.driver.aggregate.numCompletedTasks equal to zero isn't enough. The signal that matters is ingested volume against the moving average for the same hour, because the typical failure mode isn't an error, it's zero rows with a success status.

Decision matrix

Glue native connector

Pros

  • Covers the long tail of payments, billing and operations
  • Reusable connection, per-connection secret, Glue IAM
  • Transformation and MERGE in the same job

Cons

  • Watermark and overlap window are yours
  • Field-only partitioning on several connectors
  • DPU is expensive for small volume at high frequency

Verdict: Default for finance sources outside zero-ETL

Glue zero-ETL

Pros

  • No job to operate, managed incremental
  • Targets Redshift, S3 or S3 Tables

Cons

  • Only 8 SaaS sources, no payment ones
  • No transformation on the way in

Verdict: Use it whenever the source is on the list

AppFlow

Pros

  • Lowest bill for scheduled copies
  • No code

Cons

  • Bills empty runs and misconfiguration failures
  • Needs a second step to deduplicate

Verdict: Fine for marketing and CRM with no reconciliation SLO

Custom extractor

Pros

  • Full control over rate limits and webhooks
  • A bill measured in cents

Cons

  • Every API breaking change lands on your on-call
  • Secrets and retries reinvented per source

Verdict: Only for near real-time via webhook or exotic APIs

The connector removes pagination, not the contract: The native connector hands AWS the part that broke the most: authentication, pagination and endpoint changes. What stays yours is the data contract. If Adyen renames a field, the job doesn't fail, it writes nulls. A Glue Data Quality check on null rates and volume in the staging layer is what turns that silent failure into an alarm.

What I'd do: I'd start by inventorying the custom extractors that already exist and mapping each one against the list of 100. The ones on the list become Glue jobs with a DynamoDB watermark and an Iceberg MERGE, and the old code gets deleted in the same sprint, not kept as a fallback. The lesson I paid for: a homegrown acquirer extractor never gets retired by decision, it gets retired by incident. Swapping it while it still works costs far less than swapping it during month-end close.

Recommendation

Use zero-ETL when the source is one of the 8 supported ones and you don't need to transform on the way in. Use the Glue native connector when the source is payments, billing or accounting, reconciliation is involved and you can own the watermark and the idempotent MERGE: today it's the default I'd adopt for Adyen, Square, Xero and Chargebee. Use AppFlow for scheduled marketing and CRM copies, where a duplicated row doesn't break a balance sheet. Keep a custom extractor only for near real-time webhooks or APIs that even the REST connector can't cover. In every case, the alarm that matters is volume against the average, not job status.

Rating: Glue native connector for finance source

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

Top comments (0)