DEV Community: Guilherme Vassoller Daros

Airflow Scheduling: Assets vs. Cron | Which One Should You Use?

Guilherme Vassoller Daros — Sat, 29 Aug 2026 20:25:17 +0000

Sometimes, a change that looks simple on the surface is not actually that simple.

Imagine that you need to replace the source table feeding a refined or trusted table in a data pipeline. At first, it might look like a one-line change: update the table name, deploy the code, and move on.

But in a real data platform, there is usually much more behind that change.

There are dependencies, scheduling rules, upstream and downstream processes, resource consumption, concurrency, data lineage, and, sometimes, assumptions that were not immediately obvious when the pipeline was first created.

I recently had to look into exactly this kind of situation in an Apache Airflow project, and one of the questions that came up was:

Should this DAG be scheduled using a cron expression, or should it be triggered based on an Asset?

The answer, as usual in software engineering, is: it depends.

And understanding why it depends is much more important than simply knowing how to configure either option.

Cron: the familiar way of scheduling a DAG

Let's start with the simplest and most familiar option: a time-based schedule.

With Airflow, we can define a DAG to run according to a cron expression:

with DAG(
    dag_id="my_pipeline",
    schedule="0 13 * * 0",
    catchup=False,
):
    ...

In this example, the DAG is scheduled to run every Sunday at 1 PM.

In a real environment, we might have different schedules for different environments.

For example:

Environment	Schedule
Development	Saturday at 1 PM
Homologation	Sunday at 1 PM
Production	Monday–Friday at 1 PM

The important characteristic here is that the schedule is based on time.

If the DAG is configured to run at 1 PM every Sunday, Airflow will try to run it at that time, regardless of whether the data it depends on has actually changed.

This is not necessarily a bad thing.

In fact, sometimes this is exactly what we want.

But there is another approach.

When data becomes part of the schedule

Modern data pipelines often have dependencies that are better described in terms of data rather than time.

For example:

Raw table
    ↓
Trusted table
    ↓
Refined table
    ↓
Analytics / ML / BI

Instead of saying:

"Run the refined pipeline every Sunday at 1 PM."

we might want to say:

"Run the refined pipeline whenever the trusted data has been successfully updated."

This is where Airflow's Assets become interesting.

Assets were introduced in Airflow 2.4 under the name Datasets. In Airflow 3.0, the concept was renamed from Dataset to Asset, aligning the terminology with the broader data ecosystem.

The idea is simple:

the availability or update of data can become a scheduling event.

Inlets, outlets, and data lineage

To understand this concept, it helps to look at two terms: inlets and outlets.

An inlet represents something a task consumes.

An outlet represents something a task produces or updates.

Conceptually:

             inlet
               ↓
        ┌─────────────┐
        │   Task      │
        │             │
        └─────────────┘
               ↓
             outlet

For example, imagine a task that reads a raw table and produces a trusted table:

raw.customers
      │
      │ inlet
      ▼
┌─────────────────────┐
│ transform_customers │
└─────────────────────┘
      │
      │ outlet
      ▼
trusted.customers

The upstream table is the input.

The trusted table is the output.

This relationship is useful not only for scheduling, but also for understanding data lineage.

Airflow describes this idea as a way to track where data comes from, what happens to it, and where it moves over time. This can support audit trails, data governance, and debugging of data flows.

Airflow's documentation summarizes the idea nicely:

"Airflow tracks data by means of inlets and outlets of the tasks."

In other words, the relationship between what a task consumes and what it produces becomes part of the workflow's metadata.

And this is where things start getting interesting.

A simple Asset example

Suppose we have a task that produces a table represented as an Asset:

from airflow.sdk import Asset, task

trusted_customers = Asset(
    "postgres://warehouse/trusted/customers"
)

@task(outlets=[trusted_customers])
def build_trusted_customers():
    # Transform raw data
    ...

Now another DAG can depend on that Asset:

from airflow.sdk import Asset

trusted_customers = Asset(
    "postgres://warehouse/trusted/customers"
)

with DAG(
    dag_id="refined_customers",
    schedule=[trusted_customers],
    catchup=False,
):
    ...

Conceptually, the dependency becomes:

Producer DAG
     │
     │ updates
     ▼
trusted.customers
     │
     │ triggers
     ▼
Consumer DAG

The consumer does not need to guess when the upstream pipeline will finish.

It reacts to the data event.

This is what makes Asset-based scheduling particularly interesting for data-driven architectures.

So, Asset or Cron?

This was the part that made the problem more interesting for me.

At first, it is tempting to think:

"If Assets are more dynamic and data-aware, shouldn't we just use Assets everywhere?"

I don't think so.

There is no universally better option.

Cron and Assets solve different problems.

A useful way to think about it is:

Cron answers "when?"
Assets answer "after what?"

And depending on the pipeline, one question may be much more important than the other.

Cron is predictable

Let's say we have a pipeline that should run every Sunday at 1 PM.

With Cron:

schedule="0 13 * * 0"

the intention is extremely clear.

Every Sunday at 1 PM, the DAG should run.

It doesn't matter whether the upstream table was updated once, twice, or ten times during the week.

The schedule defines the cadence.

This can be very useful when:

the processing is inherently time-based;
the source system follows a fixed delivery schedule;
running more frequently would provide no benefit;
you want strict control over execution frequency;
the downstream processing is expensive.

Predictability can be a feature.

Assets are data-aware

Now imagine that our trusted table is updated whenever new raw data arrives.

We could make the downstream DAG depend on that Asset.

That sounds great.

And often, it is.

But there is an important detail:

How often is that Asset updated?

Let's say we have a large table that takes two hours to process.

It consumes a significant amount of cluster resources.

Previously, the pipeline ran twice a week, and that was enough.

Now someone decides to "modernize" the architecture:

"Let's make this task Asset-driven."

The upstream raw table, however, is updated three times a day.

Now the dependency looks like this:

Raw table
   │
   ├── update #1 ──► Asset event ──► expensive pipeline
   │
   ├── update #2 ──► Asset event ──► expensive pipeline
   │
   └── update #3 ──► Asset event ──► expensive pipeline

The downstream pipeline can now run three times a day.

That might be exactly what we want.

Or it might be a disaster.

If each execution takes two hours and heavily consumes the cluster, we may suddenly have:

unnecessary executions;
increased cluster utilization;
task concurrency;
longer queues;
contention with other pipelines;
higher infrastructure costs;
potentially overlapping runs.

Nothing is technically wrong with the Asset configuration.

The problem is that the data frequency and the processing frequency are not necessarily the same thing.

Data frequency is not always processing frequency

This is probably the most important lesson I took from this experience.

Just because data changes does not mean that the downstream pipeline should process that change immediately.

Consider this:

Source
  │
  ├── 08:00 update
  ├── 12:00 update
  └── 18:00 update

An Asset-driven DAG could potentially react to all three events.

But perhaps the business only needs the transformation to happen once per day.

In that case:

Source updates
  │
  ├── 08:00
  ├── 12:00
  └── 18:00
          │
          ▼
    Daily processing
          │
          ▼
      Final table

A Cron schedule might actually be the better design.

On the other hand, if every update needs to be available downstream as quickly as possible, an Asset-based approach could be much more appropriate.

The point is not to choose the more modern feature.

The point is to choose the scheduling model that matches the actual behavior of the system.

A simple decision framework

When deciding between Cron and Assets, I like to think about a few questions.

1. Is time the real dependency?

If the pipeline needs to run at a specific cadence regardless of upstream changes, Cron is probably a good fit.

Every Sunday
     ↓
   Run DAG

2. Is data availability the real dependency?

If the downstream pipeline should only run after upstream data has been successfully produced, Assets may be a better fit.

Data updated
     ↓
Asset event
     ↓
Run DAG

3. How frequently can the Asset be updated?

This is easy to overlook.

Before switching from Cron to Assets, ask:

"How many times can this Asset emit an event?"

A source that updates once a day is very different from a source that updates every few minutes.

4. How expensive is the downstream processing?

A lightweight task running frequently may be perfectly fine.

A two-hour transformation consuming a large cluster is a different story.

5. Do we actually need immediate processing?

Sometimes the answer is no.

A pipeline can be data-dependent without being required to react to every single data update.

Asset-driven does not automatically mean better

There is a tendency in engineering to associate newer or more dynamic features with better architecture.

But adding a dependency-aware scheduler to a pipeline that was perfectly suited to a fixed schedule can actually make the system more complex.

The important question is not:

"Can we use Assets here?"

It is:

"Should this pipeline react to every upstream data event?"

Those are very different questions.

Assets can provide a much clearer representation of data dependencies and can make workflows more event-driven.

But they also introduce a different execution model.

And that model needs to be understood before changing an existing pipeline.

The bigger picture: scheduling is part of architecture

What initially looks like a small scheduling configuration can actually be an architectural decision.

Changing:

schedule="0 13 * * 0"

to:

schedule=[some_asset]

is not simply changing syntax.

You are changing what causes the pipeline to execute.

With Cron, the cause is time.

With Assets, the cause is a data event.

That difference can propagate through the entire platform.

It can affect resource utilization, execution frequency, concurrency, downstream dependencies, observability, and even operational costs.

This is why I think scheduling deserves more attention when designing data pipelines.

Final thoughts

After looking into this, my conclusion was surprisingly simple:

There is no "best" scheduling strategy.

There is only the strategy that best matches the behavior and requirements of a particular pipeline.

Cron is not outdated just because Assets are more data-aware.

Assets are not automatically better just because they provide dynamic dependencies.

Sometimes you want:

Time → Pipeline

And sometimes you want:

Data → Pipeline

The important part is understanding which one represents the real dependency.

Before replacing a Cron schedule with an Asset, I would ask:

How often does the upstream data change?
How often should the downstream process actually run?
And how much does each execution cost?

If those three answers don't align, blindly switching to Asset-based scheduling may create more problems than it solves.

And perhaps that is the broader lesson:

In data engineering, making a pipeline more dynamic does not necessarily make it better. Sometimes, being explicit and predictable is exactly what the system needs.

How I Built a Intelligent AgTech Risk Monitoring System: Architecture, Technical Decisions, and Key Learnings

Guilherme Vassoller Daros — Thu, 15 Jan 2026 01:04:34 +0000

Introduction

This project was developed as part of the Hardware Architecture course at my university. Our team set out to build a simple yet powerful system where we could experiment with different technologies, sensors, and hardware components. It was our first hands-on experience working with sensors and embedded systems, so we aimed to create a solution that was fast, scalable, and accessible.

In this article, I walk through the project’s architecture, the technologies used, the challenges we faced, and the key lessons learned throughout the development process.

Project Overview

Main Features
- Real-time monitoring: Continuously collects environmental data (temperature, humidity, and luminosity) from sensors connected to an Arduino.
- Asynchronous data pipeline: Uses a message queue (RabbitMQ) to reliably transmit sensor readings for analysis and storage.
- Risk analysis engine: Processes sensor data to compute risk levels for pest outbreaks, with multi-tier alert levels.
- Dashboard interface: Interactive web dashboard built with Next.js that displays real-time and historical visualizations.
- Scalable architecture: Designed with distributed components that can scale independently and adapt to multiple crop types.
Repository:

https://github.com/guiDaros/project_agtech_hdwach

Tech Stack

Hardware
- Arduino Uno with environmental sensors (DHT11, HW080, LDR)
- Raspberry Pi (for backend services)
Backend
- Python 3.8+ with the Flask framework
- RabbitMQ (CloudAMQP) for asynchronous messaging
- Redis (Upstash) for near real-time data caching
- SQLite for historical data storage
- PySerial to communicate with the Arduino
- Pandas for data analysis
Frontend
- Next.js (React framework) with TypeScript
- Tailwind CSS and shadcn/ui for components
- Recharts for interactive data visualizations
Tools
- Git for version control
- VSCode as the main editor

System Architecture

High-level system flow

Sensors → Arduino → Message Queue → Backend → Database → Frontend

Development setup during testing and validation.

Why this architecture was chosen

Asynchronous Messaging

RabbitMQ is used to decouple producers and consumers through asynchronous, message-driven communication. This allows system components to operate independently, remain resilient to failures, and support multiple processing paths such as real-time streaming, historical storage, and analytics. The same structure also enables future extensions, including machine learning services, without changing the ingestion layer.

Separation of System Layers

The system is organized into hardware, backend, and frontend layers, responsible for data acquisition, processing and storage, and visualization. This separation of concerns improves maintainability, simplifies testing, and allows components to evolve and scale independently.

Why RabbitMQ Instead of Direct HTTP

RabbitMQ was chosen over direct HTTP to handle the realities of a distributed agricultural environment, including network latency, intermittent connectivity, and partial failures. By providing buffering, reliable delivery, and asynchronous processing, the message broker ensures that sensor data is not lost and can be consumed at different rates by downstream services.

How data moves through the system

The system follows an event-driven, ELT-oriented data pipeline.
Sensor readings are collected by Arduino-connected devices and sent to the backend via a serial connection, where they are published to RabbitMQ as raw events. The data remains unprocessed at this stage to preserve its original form and avoid coupling ingestion with transformation.
RabbitMQ acts as the ingestion layer, routing messages to dedicated queues for different consumers. Raw data is then loaded into storage, with SQLite used for historical persistence and Redis for fast access to recent readings.
Downstream analysis services consume messages from the queues, process sensor data, and compute derived metrics such as environmental risk indicators for pest and fungus development. These refined results are then delivered to a React and Next.js frontend, which renders real-time and historical dashboards.

Challenges and How We Addressed Them

Ensuring Reliable Sensor Data Ingestion

One of the main challenges was designing a reliable ingestion mechanism for sensor data in a distributed environment. Sensor readings are generated continuously and originate from hardware components that are inherently prone to noise, temporary failures, and unstable communication.
To address this, the system was designed around asynchronous messaging. Instead of tightly coupling data producers and consumers through direct communication, sensor readings are published as events to RabbitMQ. This approach allows data to be buffered, retried, and processed independently of ingestion, increasing fault tolerance and system resilience.
By decoupling hardware data acquisition from downstream processing, the system remains operational even when individual services become temporarily unavailable.

Balancing Real-Time Processing and Analytical Flexibility

Another challenge was designing a solution that could handle both immediate real-time visualization and future analytical needs without major changes.

By preserving raw data and separating different types of consumption, the system ensures flexibility. Dashboards, historical analysis, or predictive models can be added or improved independently, without disrupting existing processes.

Results

The system successfully integrated hardware sensors, asynchronous data ingestion, backend processing, and a web-based dashboard into a single working solution.

Sensor readings were collected, transmitted, processed, and visualized in near real time, enabling continuous monitoring of environmental conditions. Historical data was also stored and accessed for retrospective analysis.
The project was demonstrated in a live presentation setting, where the complete data flow—from sensor acquisition to dashboard visualization—was executed in real time. This validation confirmed the correctness of the system integration and the architectural decisions made during development.

Future Work

Future iterations of this project will focus on extending analytical capabilities and improving data processing quality. One of the next planned steps is the development of an initial predictive model to analyze historical sensor data and estimate the likelihood of pest or disease outbreaks. This first model will serve as an experimental foundation for more advanced predictive approaches.
In parallel, ongoing work is being done to improve the data pipeline itself, with a focus on increasing processing capacity, filtering noisy sensor readings, and producing more reliable inputs for analysis and visualization. These improvements aim to enhance the quality of results without altering the overall system architecture.
Additional enhancements may include integrating new sensors, refining alert mechanisms, and adapting the system for larger-scale or more distributed deployments.

Conclusion

This project provided practical experience in designing and implementing a distributed, event-driven system that integrates hardware, backend services, and a modern web interface.

Beyond the technical implementation, the project reinforced the importance of architectural decisions such as decoupling, data flow design, and system modularity. Working with real sensor data highlighted the challenges of handling data at the boundary between hardware and software.

Overall, the project served as a valuable learning experience in applied system design, bridging concepts from hardware architecture, data engineering, and web development.