DEV Community: Sakura Sky

Activation Is the Hard Part

Andrew Stevens — Tue, 30 Jun 2026 13:21:16 +0000

The customer data platform went live nine months ago. It was the centrepiece of last year's marketing technology budget, and on its own terms it works. An analyst can write a segment query and get back, in seconds, every customer who bought twice last quarter, opened the last three emails, and carries a lifetime value above a chosen threshold. The queries are fast. The segments are precise. The dashboards are, by any fair measure, beautiful. And in nine months, not one of those segments has reached a single channel. The save offer that the at-risk segment was built to trigger has never been sent to a customer.

This is the most common and least discussed failure in marketing technology. The money goes to the layer that produces insight and skips the layer that acts on it. Insight tells you who. Activation does something about it. The two are not the same system, they rarely have the same owner, and the gap between them is where most of the spent budget quietly evaporates. The rest of this post is about that gap: what sits in it, why it is harder than it looks, and how to tell whether yours is working.

The CDP that went live and stayed quiet

Insight platforms get funded because they demonstrate well. A segment count is a slide. A cohort chart is a slide. A leadership review can absorb "we have built a 360-degree view of forty thousand at-risk customers" in one breath, and it sounds like progress because it is legible. You can point at it.

Activation is invisible until it fails. Nobody opens a quarterly review with "our reverse ETL sync to the email platform held a 99.4 percent delivery rate against the suppression list." It is plumbing, and plumbing only becomes a topic when the floor is wet. So the budget tilts toward the visible half. The CDP, the business intelligence stack, and the segmentation tooling get the capital and the headcount, while the path that carries a segment to a channel gets whatever attention is left over, which is usually a junior operations hire and a tangle of manual CSV exports.

The result is the quiet CDP. It holds a complete, queryable, genuinely useful picture of the customer base, and that picture sits behind glass. The distance between "we know this customer is about to churn" and "this customer received a save offer on Tuesday" is not a small last step. It is most of the work, and it is the half that was never properly resourced. A segment that never leaves the warehouse has the same revenue impact as a segment that was never built.

Three layers of activation

Activation is not one thing, and conflating its layers is part of why teams underestimate it. It helps to separate three.

The first is audience sync. This is the batch movement of a segment from the warehouse out to the tools that can act on it: the email platform, the ad networks, the support desk, the CRM. The mechanism is usually reverse ETL, which is simply the warehouse-to-application direction of data movement. Normal ETL pulls data in from source systems; reverse ETL pushes the modelled, cleaned result back out to where people work. A churn-risk audience computed in the warehouse becomes a live suppression or targeting list inside Meta or Braze.

The second is journey orchestration. A synced audience is static. Orchestration adds time and sequence: send this message, wait three days, branch on whether they opened it, suppress anyone who has bought in the meantime, cap the total contacts per week. This is the layer that turns a list into an experience, and it carries most of the logic that marketers actually think of as the campaign.

The third is real-time activation. Some signals are worthless by tomorrow. A cart abandonment, a pricing-page visit, a support escalation: these have to reach the surface the customer is on within seconds, while intent is still live. This layer trades the comfort of a nightly batch for the demands of a streaming path, and it is the one most organisations have not built at all. Each layer is a different engineering problem with a different latency budget, and a platform that does the first well can be hopeless at the third.

Why activation is hard

The difficulty is not conceptual. Everyone agrees the save offer should go out. The difficulty is that the activation path is a write path, and write paths into other people's systems are unforgiving.

Every destination has its own identity model, its own schema, and its own rate limits. The warehouse knows a customer by an internal key; the ad network knows them by a hashed email, the email tool by a subscriber ID, the CRM by an account. Pushing one audience to four channels means resolving one customer to four different identities and accepting that some will not match. A reverse ETL job is not a query you can rerun for free. It mutates state in a live system, and when it gets something wrong, the failure mode is not an empty dashboard cell. It is a real person receiving the wrong message, or the same message five times, or a win-back discount sent to someone who renewed yesterday.

Consent makes this sharper. The permission state that governs whether a customer can be contacted on a given channel has to travel with the audience at the moment of activation, not as a setting somebody checked at segment-build time. If consent is enforced one step too early or one step too late, the system sends mail it was not allowed to send, and regulators treat consent as the controller's responsibility regardless of which system held the stale state. Suppression, frequency capping, and quiet hours all live in the same path, and all of them are stateful, which is the property warehouses are worst at handling. None of this shows up in a segment preview. It only shows up at the moment of delivery, which is exactly the moment the insight half of the stack has stopped paying attention.

The activation health metric

If insight is measured by what you can see and activation by what reaches a customer, the organisation needs a number for the second half. One useful pair: the activation rate and the time to activation.

The activation rate is the share of built segments that actually reach at least one channel. A team that builds two hundred segments a quarter and activates twelve of them does not have an insight problem. It has an activation problem wearing an insight costume, and the rate makes that legible to the people holding the budget. Time to activation is the lag from a signal being detected to a message being delivered against it. For a quarterly campaign, days may be fine. For a cart abandonment, anything over minutes means the metric is telling you the real-time layer does not exist.

Both numbers point at the same destination, which is closed-loop measurement: the ability to trace a delivered message backward to the signal that triggered it and forward to the outcome it produced. Most marketing measurement breaks because the loop is open. The insight layer records who was in the segment, the channel records what was sent, and nothing joins the two, so attribution becomes an argument rather than a lookup. Instrumenting the activation path to emit its own events, segment entered, message sent, suppression applied, outcome observed, is what closes the loop and turns measurement from a debate into a query.

What good looks like

In a healthy setup, activation is treated as infrastructure rather than as the afterthought to a platform purchase. It has an owner, a service level, and monitoring, in the same way the data pipelines feeding the warehouse do. When a sync to a channel fails, somebody is paged, because a silent failure here means real customers are silently not being contacted.

The reverse ETL jobs are observable: every run reports how many records were sent, how many matched, how many were suppressed, and why. Consent is enforced inside the activation path, evaluated at the moment of send, so the question "were we allowed to send this" is answered by the system and not by a spreadsheet. There is an explicit latency budget per layer, so nobody pretends a nightly batch can carry a real-time signal. And the activation path emits the events that make closed-loop measurement possible, so the team can answer "did the save offer reduce churn" without a six-week reconciliation project.

None of this is glamorous, and that is the point. The organisations pulling ahead in 2026 are not the ones with the most sophisticated segmentation. They are the ones that resourced the unglamorous half: the path that carries a known signal to a delivered action, reliably, with consent intact and the outcome measurable. Insight without activation is a very expensive way to know things you never act on.

The series in one paragraph

The frame is that every marketing question is eventually a data question (Part 1). The substrate that answer depends on is a single customer view that actually ships rather than one that stalls in committee (Part 2). The reporting built on top of it stops lying the moment the underlying data is observable and the metrics mean one thing (Part 3). The consent state that governs all of it has to run through every flow at runtime, not sit in a policy document (Part 4). And the value of every one of those investments only becomes real in the activation path, where a known signal turns into a delivered action. Insight is the half that demos. Activation is the half that pays.

Building that path is engineering work, which is why it tends to land with the team that owns the warehouse rather than the agency that runs the campaign. Sakura's Data & AI practice builds and instruments the activation layer underneath the marketing stack, and its Managed Services team keeps those pipelines running once they are live.

Data Costs: Are You a Victim of Your Own Success?

Andrew Stevens — Tue, 30 Jun 2026 12:12:54 +0000

Even in an era of AI and automation, the most valuable asset we have is judgment. We talk often about saving money in the cloud, but what does that actually mean in practice? This post is about small architectural decisions that compound cost at scale, and the structural fix when they do. If you have any pipelines running today against raw object storage on a fixed cadence, there is a good chance one of them is in here.

A case study in insight unit economics

Imagine the scenario. Your high-traffic e-commerce platform has just survived a record-breaking sales weekend. Traffic peaked, transactions hit an all-time high, and the security team flagged every bot and DDoS attempt cleanly using real-time HTTP log monitoring. The executive suite is celebrating.

Thirty days later, the cloud invoice arrives.

Instead of the economies of scale the hyperscalers all promise, your data infrastructure bill has grown linearly with your analytical data. Worst of all, you are paying ten times more to compute the exact same 15-minute analysis the team has been looking at for months.

This is the classic data platform problem: being penalised by your own operational history. It is not a query problem. It is the direct result of a high-frequency, stateless execution loop sitting on top of a suboptimal storage abstraction. Architectural decisions that look trivial on paper produce orders-of-magnitude differences on the cloud invoice.

Monitoring is a data problem at scale

In enterprise e-commerce, downtime or a security breach is millions in lost revenue and a long PR tail. To catch DDoS attacks, credential stuffing, and bot traffic before they do real damage, you have to monitor real-time HTTP logs continuously.

Most enterprises rely on high-end SIEM (Security Information and Event Management) and SOAR platforms for security orchestration. The catch is that these tools almost universally charge based on data ingestion volume, which becomes unsustainably expensive at HTTP-log volumes. Ingesting 100% of raw, noisy, unfiltered HTTP logs directly into a SIEM is neither economical nor sensible.

The pipeline approach

To preserve capital, the architectural plan is simple:

Land raw, compressed log data each day into inexpensive object storage (Google Cloud Storage, AWS S3).
Analyse it in near real-time, on a 15-minute cadence, inside a cloud data warehouse.
Push only the high-value, actionable alerts to the expensive SIEM.

To prove the business case quickly and without burning a lot of engineering time, teams almost always set up a quick-and-dirty proof of concept at 10% log volume. That is a fast, cheap way to get stakeholders aligned on the data they want to see and what it means.

Where the convenience pattern ends

To get the proof of concept live in a single afternoon, the path of least resistance is to point data warehouse external tables (Schema-on-Read) directly at the raw storage bucket. No ETL pipelines, no data orchestration, no upfront schema optimisation. At 10% data volume in initial testing, this setup runs exceptionally fast and costs pennies.

So you flip the switch to 100% production volume. Within a few hours, the linear cost curve starts to break the unit economics on every insight the pipeline produces.

The zero-ETL convenience hits its limit because the cost scales linearly with the full scan it has to do, and security monitoring needs frequent checking. If insights are produced every 15 minutes, the pipeline executes the exact same query 96 times a day. The architecture has no state management layer, so it has no memory of what it has already processed. Every run re-scans every log file that has ever landed in the bucket.

This low-friction convenience behaves the same way across every major hyperscaler:

Google Cloud (BigQuery). External tables over GCS force BigQuery to scan unoptimised, non-columnar files on every run, billing at the full on-demand rate (currently $6.25 per TiB scanned in US multi-region) instead of leveraging compressed, native columnar formats (Google Cloud, 2026).
AWS (Athena). Works on an identical on-demand model, charging $5.00 per TB scanned from S3 (Amazon Web Services, 2026). As historical logs accumulate in the bucket, Athena hits a literal pricing wall.
Snowflake. External tables bypass the platform's native micro-partitioning, proprietary sorting, and localised metadata caching (Snowflake, 2026). Queries take significantly longer to execute, forcing virtual warehouses to stay active longer and burn through credit hours.

By coupling a low-overhead onboarding abstraction to a high-frequency production system, you tie the cost of a static, identical insight to an exponentially growing historical dataset.

The math: scaling penalty vs flat-line

The analytical value of a 15-minute window does not change. The threat profile does not change. Yet look at how the unit economics decay when you couple a fixed insight to an unpartitioned, historical dataset. Over 100 days, with a small 50GB-per-day log volume and a standardised data warehouse cost model:

The wrong way: the stateless external scan

The system runs a full scan of the growing historical storage 96 times a day just to look at the latest drop in the bucket.

Day 1. 50GB in storage. You scan 4.8 TB/day. Cost: $24/day (1x base cost).
Day 30. 1.5TB in storage. You scan 144 TB/day. Cost: $720/day (30x base cost).
Day 100. 5TB in storage. You scan 480 TB/day. Cost: $2,400/day (100x base cost).

By Day 100, you are paying 100x more for the exact same 15-minute alert you produced on Day 1. The infrastructure bill is actively penalising you for your own history.

The right way: stateful incremental design

With a state management layer or strict partition pruning, the compute engine isolates only the new data that arrived in the 15-minute window (about 520MB per run). History is stored for deep audits but is never scanned for operational alerts.

Day 1. 50GB in storage. You scan 50 GB/day. Cost: $0.25/day (1x base cost).
Day 30. 1.5TB in storage. You scan 50 GB/day. Cost: $0.25/day (1x base cost).
Day 100. 5TB in storage. You scan 50 GB/day. Cost: $0.25/day (1x base cost).

On Day 100, the cost is exactly 1x. It is flat and identical to Day 1. The business growth and data history have been successfully decoupled from the cloud compute invoice.

[The wrong way: stateless external scan]
Day 1:    $24/day      (1x)
Day 30:   $720/day     (30x)    Cost is bleeding meaningfully
Day 100:  $2,400/day   (100x)   Cost is a structural problem

[The right way: stateful incremental design]
Day 1:    $0.25/day    (1x)
Day 30:   $0.25/day    (1x)
Day 100:  $0.25/day    (1x)     Flat across the full window

The hidden multiplier: it is never just one pipeline

If this were the only pipeline running in the enterprise, a $2,400/day bleed would be a painful but isolated leak.

It is never just one.

Modern enterprise data platforms easily run 100+ pipelines concurrently: customer behavioural tracking, transactional updates, inventory syncs, third-party API ingestions. When a data organisation lacks a strict framework for intentional storage design and state management, the quick-and-dirty external-table pattern gets copy-pasted across the estate. It becomes the unwritten standard for speed.

If just 20% of production pipelines carry this exact structural oversight, you are not looking at a minor cloud overage. You are looking at a compounding, multi-million-dollar architectural deficit that quietly erodes company margins behind the scenes.

The CXO lens: system latency vs decision latency

When cloud data invoices spike, the traditional executive response is to reach for blunt accounting instruments. Budget caps, restricted compute privileges, multi-layered approval chains. These tactics are counterproductive. They introduce friction, kill developer velocity, and ultimately cost the business more in lost time and delayed features than they save in cloud credits.

To manage cloud costs without stalling growth, leadership has to understand the structural disconnect between two latencies:

System latency. How fast the technical infrastructure can process a batch of logs and trigger an alert (10 seconds).
Decision latency. How long it takes a human analyst to see that alert, log into a console, verify the threat, and actually change a firewall rule (20 minutes).

If the organisation's human decision latency is 20 minutes, paying an exponential financial premium to maintain a 10-second stateless system latency is not a competitive business strategy. It is an expensive vanity project. You are subsidising a hyperscaler's profit margin.

The leadership pivot: FinOps enablement

The executive job is not to micro-manage queries. It is to foster a culture where engineering teams align technical velocity with unit economics:

Democratise cost visibility. Engineers cannot optimise what they cannot see. Pipe query-cost metrics directly into developer environments or internal communication channels. When an engineer can see that a newly deployed pipeline cost $4,000 to run yesterday, optimisation happens naturally.
Treat cloud efficiency as a core product metric. Cloud compute is a direct variable cost of your product. If an infrastructure team deploys a hyper-fast real-time system that completely erodes product gross margins because of lazy storage abstractions, that deployment is an architectural failure, not a business win.
Brief leaders to look for linear cost. Architects and engineering leaders should investigate anything that presents as a linear cost. Linear in a fixed-output system is a smell.

The architect lens: Medallion isolation strategy

For the system designer, the fix for the proof-of-concept trap is a tailored Medallion/Lambda approach. Instead of pointing high-frequency analytics directly at a raw object storage dump, the system is broken into distinct, isolated operational layers.

The ephemeral Bronze landing zone

Instead of leaving data in object storage and relying on high-overhead Schema-on-Read external tables, execute a targeted, isolated ingestion step. Programmatically isolate the storage search path down to a rolling two-hour file window using dynamic wildcards, then execute a native LOAD DATA OVERWRITE statement into an ephemeral, partitioned, clustered Bronze staging table. This shifts the raw ingestion cost from an $O(n)$ historical file scan to a flat, deterministic window overhead.

The stateful Silver watermark

To persist data cleanly without introducing high-overhead third-party state orchestration tools, let the data warehouse manage state natively. Running a localised MAX() watermark calculation over a strict lookback window (trailing 24 hours, for instance) lets the architecture instantly determine where the previous execution ended. The compute engine processes only the delta, applies inline transformations, deduplicates records via native window functions, and appends the clean data into the persistent Silver table.

The engineer lens: production implementation

At the implementation layer, the theoretical optimisations turn into concrete BigQuery procedural scripts that break the cost curve.

Bronze window ingestion

Every 15 minutes, this routine runs a dynamic string evaluation to calculate the current and previous hour's file layout in Google Cloud Storage. It overwrites the staging table, loading only the necessary files into a native, hourly-partitioned columnar environment.

-- Step 1: Isolate the file scan path and execute a native load overwrite
BEGIN
  DECLARE current_hour STRING;
  DECLARE prev_hour STRING;

  SET current_hour = FORMAT_TIMESTAMP('gs://enterprise-cdn-logs/%Y%m%dT%H*.log.gz', CURRENT_TIMESTAMP());
  SET prev_hour    = FORMAT_TIMESTAMP('gs://enterprise-cdn-logs/%Y%m%dT%H*.log.gz', TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR));

  EXECUTE IMMEDIATE FORMAT("""
    LOAD DATA OVERWRITE `your_bronze_project.your_dataset.your_table`
    (
      [__insert schema__]
    )
    PARTITION BY TIMESTAMP_TRUNC(StartTimestamp, HOUR)
    CLUSTER BY [__fields__]
    FROM FILES (
      format = 'JSON',
      uris = ['%s', '%s'],
      compression = 'GZIP',
      ignore_unknown_values = true
    )
  """, current_hour, prev_hour);
END;

Stateful Silver ingestion and deduplication

Once the raw logs are materialised inside the Bronze layer, the pipeline reads the highest timestamp from the persistent Silver table to determine the watermark. Note the deliberate constraint on the watermark lookback: WHERE StartTimestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY). This ensures BigQuery never scans the entire historical Silver table to compute the state, keeping the watermark discovery scan virtually free.

-- Step 2: Calculate the watermark delta, transform, deduplicate, and insert
INSERT INTO `your_silver_project.your_dataset.logs_persistent`
WITH watermark AS (
  -- Bounded lookback keeps the watermark calculation query cheap
  SELECT
    COALESCE(MAX(StartTimestamp), TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR)) AS max_ts
  FROM `your_silver_project.your_dataset.logs_persistent`
  WHERE StartTimestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
)
SELECT
  [__add your schema__]
FROM `your_bronze_project.your_dataset.your_table` AS s
CROSS JOIN watermark
WHERE s.StartTimestamp > watermark.max_ts
-- Inline deduplication filters out late-arriving network duplicates on the fly
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY [your_unique_id], StartTimestamp
  ORDER BY StartTimestamp DESC
) = 1;

The takeaway: trivial code, order-of-magnitude results

This pipeline layout completely isolates the variable cost. Because the system transitioned from a stateless table read on raw files to an event-driven ingestion window and a stateful insert loop, the processing metrics flat-line.

The next time an engineering or architecture team brings a "minor" infrastructure modification proposal to sprint planning, asking to adjust an ingestion path or isolate a staging layer, remember this code. On paper it looks like a standard code change. In reality it is the structural difference between a flat-line, predictable infrastructure footprint and a 100x compounding financial problem.

This is the kind of work Sakura Sky's Data & AI practice does inside client estates, and the kind of pattern our Managed Services team runs in production for clients who would rather buy the outcome than staff the rotation.

References

Amazon Web Services, 2026. Amazon Athena pricing. Amazon Web Services. Available at: https://aws.amazon.com/athena/pricing/ [Accessed 29 June 2026].

Google Cloud, 2026. BigQuery pricing. Google Cloud documentation. Available at: https://cloud.google.com/bigquery/pricing [Accessed 29 June 2026].

Snowflake, 2026. External tables. Snowflake documentation. Available at: https://docs.snowflake.com/en/user-guide/tables-external-intro [Accessed 29 June 2026].