DEV Community: Sanjeet Singh Jagdev

Building a Declarative JSON Extraction Engine with Python’s Annotated Types, jq and Pydantic

Sanjeet Singh Jagdev — Mon, 09 Mar 2026 14:10:26 +0000

Inspiration

I recently ran into a problem where I needed to process a JSON payload, extract a few fields, transform some values, compute a few derived ones, and produce a new JSON structure.

My first instinct was the usual approach: manually parse the JSON, dig through nested keys, and apply transformations along the way. But it quickly started to feel repetitive and brittle. Every new field meant more extraction logic and more edge cases to handle.

I knew about tools like jq for expression-based JSON querying and Pydantic for schema validation, so I tried combining them. While this helped a bit, I still found myself writing a lot of glue code just to move data from one place to another.

At that point a simple idea occurred to me:

What if I could just declare what I want from the JSON, and let the rest happen automatically?

I looked around for a library that did this out of the box but couldn’t find something that quite fit. So, naturally, I decided to try building one.

Understanding the Problem

Say you have a JSON structure of an Order

{
  "id": "order_789",
  "created_at": "2024-02-10T14:21:00Z",
  "customer": {
    "first_name": "Alice",
    "last_name": "Smith",
    "contact": {
      "email": "ALICE@EXAMPLE.COM"
    }
  },
  "items": [
    { "name": "Keyboard", "category": "premium", "price": 120 },
    { "name": "Mouse", "category": "standard", "price": 40 },
    { "name": "Monitor", "category": "premium", "price": 300 }
  ]
}

Now I want to transform this into an OrderSummary that looks like

{
  "order_id": "order_789",
  "customer_email": "alice@example.com",
  "item_count": 3,
  "premium_total": 420,
  "order_label": "ORDER-order_789"
}

Naive Approach

To produce the same transformed JSON using a traditional approach, we would typically write something like this:

with open("orders.json") as f:
    data = json.loads(f.read())

order_id = data["id"]

customer_email = data["customer"]["contact"]["email"].lower()

items = data.get("items", [])
item_count = len(items)

premium_total = sum(
    item["price"]
    for item in items
    if item.get("category") == "premium"
)

order_label = f"ORDER-{data['id']}"

order_summary = {
    "order_id": order_id,
    "customer_email": customer_email,
    "item_count": item_count,
    "premium_total": premium_total,
    "order_label": order_label
}

While this works, the extraction logic, transformations, computed fields are all mixed together and the schema is implicit. As the JSON structure grows, this approach quickly becomes harder to maintain.

Declarative Solution

The idea is to declare a model that describes the OrderSummary like

from typing import Annotated

# I named the library as "jresolve"
from jresolve import (
    JqModel,
    Jq,
    JqMode,
    Transform,
    Computed
)


class OrderSummary(JqModel):
    order_id: Annotated[
        str,
        Jq(".id")
    ]

    customer_email: Annotated[
        str,
        Jq(".customer.contact.email"),
        Transform(str.lower)
    ]

    item_count: Annotated[
        int,
        Jq(".items"),
        Transform(len)
    ]

    premium_total: Annotated[
        float,
        Jq(
            ".items[] | select(.category == \"premium\") | .price", 
            mode=JqMode.MANY
        ),
        Transform(sum)
    ]

    order_label: Annotated[
        str,
        Computed(lambda d: f"ORDER-{d['id']}")
    ]

The usage for this would be

order_summary = OrderSummary.from_json(data)

The key idea here is

The model declares how fields are extracted and transformed directly in the type annotation.

High Level Architecture

Input JSON
    ↓
Resolver (Jq / Computed)
    ↓
Transform Pipeline
    ↓
Collected Field Values
    ↓
Pydantic Model Construction
    ↓
Typed Output

Diving Deeper

Now that we have seen how the declarative model looks from the outside, let's briefly look at the core ideas that make this work internally.

Analyze a field

customer_email: Annotated[
        str,
        Jq(".customer.contact.email"),
        Transform(str.lower)
    ]

Just by looking at it I can already tell that:

The type of the field is str
Its value is extracted using the jq expression ".customer.contact.email"
Once I have the value I want to apply a transformation to lowercase

The intent of the field becomes immediately obvious. And since we are using Pydantic you get schema validation for free.

The backbone `Annotated`

The Annotated type acts as the glue that binds all the declarations together.

If we examine the type

order_id: Annotated[
    str,
    Jq(".id"),
    Transform(str.upper)
]

It tells us

Type: 
  → str

Metadata:
  Resolvers:
    → Jq(".id")

  Transforms:
    → str.upper

Since Pydantic allows us to access the metadata stored in Annotated, we can interpret those declarations and execute them against the JSON input.

So the field effectively becomes

JSON
  ↓
Jq Resolver
  ↓
Transform Pipeline
  ↓
Typed Field

A clean pipeline to reason and add more operations to.

Resolver

While Annotated does all the heavy-lifting in the interface, the Resolver does so for the core implementation.

It is an Abstract class that provides an interface for different types of Resolvers

class Resolver(ABC):
    @abstractmethod
    def resolve(self, data: dict) -> Result[Any, ResolutionError]:
        ...

NOTE: A Rust styled Result type is used to return results or errors throughout the implementation. Take a look at result.

Currently there are 3 implementations of Resolver:

Jq to extract values using the jq expression syntax
Computed to generate a value using a function
Pipeline used internally that works on a base Resolver and a list[Transform]

How the Model Is Executed

The current execution model looks like

resolver output (Jq / Computed)
      ↓
transform 1
      ↓
transform 2
      ↓
final value

Which translates to

Inspect model annotations
Extract resolvers and transforms from Annotated metadata
Build a pipeline
Execute pipeline on input JSON
Construct the Pydantic model which roughly looks like

for field_name, field in cls.model_fields.items():
    resolver = build_pipeline_from_field(field)

    if resolver:
        result = resolver.resolve(data)
        values[field_name] = result.ok()

Nested Models

Real JSON structures are rarely flat. Fortunately, JqModels can be nested and are resolved recursively.

class OrderSummary(JqModel):
    # previous fields

    customer: Customer

Where Customer can look like

from typing import Annotated

class Customer(JqModel):
    name: Annotated[
        str,
        Jq(".profile.name.last + ', ' + .profile.name.first")
    ]
    email: Annotated[
        str,
        Jq(".customer.email"),
        Transform(str.lower)
    ]

Error Handling

Resolvers return a Rust-style Result type that either contains a value or a structured error.

Ok(value)
Err(ResolutionError)

The pipeline inspects the result and short-circuits if an error occurs, allowing failures to propagate cleanly without raising exceptions.

Closing Thoughts

This pattern works particularly well when dealing with:

Complex API responses
ETL pipelines
Data normalization layers
Event payload transformations

Instead of scattering extraction logic across the codebase, transformations become declarative and centralized in the model definition.
Using Annotated types allows us to build a small DSL directly inside Python’s type system, while still benefiting from Pydantic’s validation and typing support.

The result is a system that is:

Declarative
Type-safe
Composable
Easy to extend

If you’re interested in the full implementation, the code is available in my Github Repo.

I would love to hear your thoughts about this and what I could have done better!

Designing an Either in Java: Making Failure Part of the Type System

Sanjeet Singh Jagdev — Sun, 15 Feb 2026 16:59:12 +0000

Inspiration

Error handling in Java is powerful — but it is also architectural.

In small programs, exceptions are straightforward. In larger systems, they quietly influence structure: where logic lives, how layers depend on each other, and how responsibilities are divided.

While refactoring some database initialization code, I noticed something uncomfortable: most of my try/catch blocks weren’t protecting against rare failures. They were handling completely expected outcomes — missing configuration, invalid values, connection problems.

Those aren’t exceptional situations. They are part of the domain.

That observation led me to experiment with a minimal Either<L, R> container — not only as a functional programming exercise, but also as a design exploration:

What happens when failure becomes data instead of control flow?

This article walks through:

The architectural problem with nested exception handling
Designing a minimal Either<L, R>
Refactoring real code to use it
The trade-offs this introduces

An obvious question would be, "Why not use Vavr?"

Short answer, We can! But then I also wanted to learn how to make composable containers.

Problem: Exceptions as Hidden Control Flow

Consider a simplified database setup workflow:

Read configuration from file
Parse configuration
Connect to database
Execute schema
Insert data

public class DbTestImperative {

    public static void main(String[] args) {
        performDatabaseOperation();
    }

    public static void performDatabaseOperation() {
        try {
            Properties props = readDatabaseConfig("db_config.properties");

            String url = props.getProperty("db.url");
            String user = props.getProperty("db.user");
            String password = props.getProperty("db.password");

            try (
                    Connection conn = DriverManager.getConnection(url, user, password);
                    Statement stmt = conn.createStatement()
            ) {
                String schema = readTableSchema("schema.sql");
                stmt.execute(schema);
                stmt.executeUpdate("INSERT INTO users (name, email) VALUES ('John', 'john@example.com')");
            }
        } catch (IOException e) {
            System.err.println("Configuration error: " + e.getMessage());
        } catch (SQLException e) {
            System.err.println("Database error: " + e.getMessage());
        }
    }

    public static String readTableSchema(String fileName) throws IOException {
        try (
                InputStream in = DbTestImperative.class
                        .getClassLoader()
                        .getResourceAsStream("schema.sql")
        ) {
            if (in == null) {
                throw new IOException("schema.sql not found");
            }
            return new String(in.readAllBytes(), StandardCharsets.UTF_8);
        }
    }

    public static Properties readDatabaseConfig(String fileName) throws IOException {
        Properties props = new Properties();
        try (InputStream in = DbTestImperative.class.getClassLoader().getResourceAsStream(fileName)) {
            props.load(in);
            System.out.println("Configuration file read successfully.");
        }
        return props;
    }
}

This code works. But architecturally, several things are happening:

Error propagation is implicit.
Business logic and error handling are interleaved.
Each layer must decide whether to catch, wrap, or rethrow.
Method signatures do not communicate failure modes clearly.

In this flow, an IOException is not an anomaly. It is part of the normal behavior of the system. The type system, however, does not reflect that.

The result is control flow that is partially hidden in exception mechanics.

Modeling Failures Explicitly

Instead of throwing exceptions, what if methods returned:

Either<AppError, DbConfig>

That type says:

Left → Something went wrong (AppError)
Right → Success (DbConfig)

Failure is no longer an implicit side effect. It is a first-class value.

The caller cannot ignore it. The method signature forces acknowledgment.

Designing a Minimal `Either<L, R>`

The goal was not to build a full functional library, but to design the smallest useful container that supports:

Mapping over success
Chaining fallible operations
Explicit boundary handling

This is not meant to compete with libraries such as Vavr. The purpose here is architectural clarity — understanding the mechanics by building the abstraction ourselves.

public interface Either<L, R> {
    boolean isLeft();
    boolean isRight();

    L getLeft();
    R getRight();

    Optional<L> left();
    Optional<R> right();

    static <L, R> Either<L, R> left(L value) {
        return new Left<>(value);
    }

    static <L, R> Either<L, R> right(R value) {
        return new Right<>(value);
    }

    @SuppressWarnings("unchecked")
    default <L2> Either<L2, R> mapLeft(Function<L, L2> mapper) {
        Objects.requireNonNull(mapper);
        return isLeft() ? left(mapper.apply(getLeft())) : (Either<L2, R>) this;
    }

    @SuppressWarnings("unchecked")
    default <R2> Either<L, R2> mapRight(Function<R, R2> mapper) {
        Objects.requireNonNull(mapper);
        return isRight() ? right(mapper.apply(getRight())) : (Either<L, R2>) this;
    }

    @SuppressWarnings("unchecked")
    default <L2> Either<L2, R> flatMapLeft(Function<L, Either<L2, R>> mapper) {
        return isLeft() ? mapper.apply(getLeft()) : (Either<L2, R>) this;
    }

    @SuppressWarnings("unchecked")
    default <R2> Either<L, R2> flatMapRight(Function<R, Either<L, R2>> mapper) {
        return isRight() ? mapper.apply(getRight()) : (Either<L, R2>) this;
    }

    default <L2, R2> Either<L2, R2> biMap(Function<L, L2> leftMapper, Function<R, R2> rightMapper) {
        Objects.requireNonNull(leftMapper);
        Objects.requireNonNull(rightMapper);

        return isRight() ?
                right(rightMapper.apply(getRight())) :
                left(leftMapper.apply(getLeft()));
    }

    default <T> T fold(Function<L, T> leftFold, Function<R, T> rightFold) {
        Objects.requireNonNull(leftFold);
        Objects.requireNonNull(rightFold);

        return isRight() ?
                rightFold.apply(getRight()) :
                leftFold.apply(getLeft());
    }

    default void fold(Consumer<L> leftFold, Consumer<R> rightFold) {
        Objects.requireNonNull(leftFold);
        Objects.requireNonNull(rightFold);

        if (isRight()) rightFold.accept(getRight());
        else leftFold.accept(getLeft());
    }

    final class Left<L, R> implements Either<L, R> {
        private final L value;

        public Left(L value) { this.value = value; }

        public boolean isLeft() { return true; }
        public boolean isRight() { return false; }
        public L getLeft() { return value; }
        public R getRight() { throw new NoSuchElementException("getRight() on Left<L, R>"); }
        public Optional<L> left() { return Optional.ofNullable(value); }
        public Optional<R> right() { return Optional.empty(); }
    }

    final class Right<L, R> implements Either<L, R> {
        private final R value;

        public Right(R value) { this.value = value;}

        public boolean isLeft() { return false; }
        public boolean isRight() { return true; }
        public L getLeft() { throw new NoSuchElementException("getLeft() on Right<L, R>"); }
        public R getRight() { return value; }
        public Optional<L> left() { return Optional.empty(); }
        public Optional<R> right() { return Optional.ofNullable(value); }
    }
}

Three operations carry most of the weight:

mapRight → transform success
flatMapRight → chain fallible operations
fold → exit the abstraction

The rest supports ergonomics and symmetry.

Refactoring the First Layer: Configuration

Instead of throwing IOException, we return Either.

public static Either<AppError, Properties> readDatabaseConfig(String fileName) {

    Properties props = new Properties();

    try (InputStream in = DbTestEither.class
            .getClassLoader()
            .getResourceAsStream(fileName)) {

        if (in == null)
            return Either.left(new AppError(fileName + " not found"));

        props.load(in);
        return Either.right(props);

    } catch (IOException e) {
        return Either.left(new AppError(e.getMessage()));
    }
}

The method now communicates explicitly:

It may fail.
Failure is part of its contract.
Callers must handle it.

No checked exceptions. No hidden propagation.

Adding Validation as Composition

Parsing configuration is another fallible step.

public static Either<AppError, DbConfig> parseConfig(Properties properties) {

    String url = properties.getProperty("db.url");
    String user = properties.getProperty("db.user");
    String password = properties.getProperty("db.password");

    if (url == null || url.isBlank())
        return Either.left(new AppError("db.url is not provided"));

    if (user == null || user.isBlank())
        return Either.left(new AppError("db.user is not provided"));

    if (password == null || password.isBlank())
        return Either.left(new AppError("db.password is not provided"));

    return Either.right(new DbConfig(url, user, password));
}

Composition becomes straightforward:

public static Either<AppError, DbConfig> loadConfig(String fileName) {
    return readDatabaseConfig(fileName)
            .flatMapRight(DbTestEither::parseConfig);
}

This is where the architectural shift happens.

If reading fails → the error propagates.
If parsing fails → the error propagates.
If both succeed → we get DbConfig.

No nested conditionals.
No rethrowing.
No duplicated catch logic.

There is no branching logic here. The control flow is encoded in the type.

Handling at the System Boundary

Eventually, you must handle the result. That’s what fold is for:

public static void performDatabaseOperation() {
    loadConfig("db_config.properties").fold(
                err -> System.err.println("Error Occurred: " + err.errorMsg),
                DbTestEither::insertRecord
        );
}

Architecturally, this separation matters.

Inside the system, logic remains composable and side-effect free.

At the boundary, you translate values into effects: logging, retries, termination.

That distinction scales well.

Architectural Implications

Using Either changes design in subtle but important ways:

Failure is part of the type system. Method signatures now document failure explicitly.
Control flow becomes declarative. `flatMapRight` encodes propagation rules structurally.
Boundaries become clear, fold marks the edge between pure logic and side effects.
Exceptions become truly Exceptional, you reserve exceptions for unrecoverable system failures.

Trade-offs

This approach is not universally superior.

You lose:

Automatic stack trace propagation unless you explicitly model it
Simplicity for very small programs
Immediate familiarity for some teams

You gain:

Explicit failure modeling
Better composability
Clearer architectural boundaries
Improved reasoning about control flow

For business logic, validation, parsing, and external calls, this model scales better than nested exceptions.

Closing Thoughts

Designing this minimal Either wasn’t about importing functional programming into Java.

It was about making failure visible.

When failure becomes data instead of control flow:

Pipelines become natural.
Types tell the story.
Architecture becomes clearer.

And that changes how you structure systems.

You can find the entire code for this article in this Github Gist

Building an Executable Chain to Understand Composable Systems

Sanjeet Singh Jagdev — Sun, 18 Jan 2026 17:44:42 +0000

Inspiration

While exploring frameworks like LangChain, I noticed a recurring idea: instead of writing monolithic logic, execution is broken into composable steps that can be chained, reused, and rearranged.

This article is not about LangChain itself, but about understanding that underlying idea by building a minimal executable chain abstraction from scratch.

Why do this?

When working without a mental model for execution chains, one often runs into the pain of:

Deeply nested conditionals
Hardcoded execution order
Monolithic "do-everything" methods

As a result, the entire chain of operations becomes brittle, hard to test, and difficult to rearrange.

If we instead think of each step as a unit that does exactly one thing, we arrive at the basic building block of a chain.

With that mental model in place, let’s design the smallest possible abstraction to support it.

What is an Executable Chain?

As the name suggests, an executable chain is a sequence of small, focused units of logic that can be arranged or rearranged to achieve a larger unit of work.

Take a look at a simple Order Processing chain below.

Each step handles one clear process and the chain is executed sequentially.

An Executable is something that:

Takes input
Performs work
Returns output

And it is:

Composable
Testable
Predictable

Designing the Core Abstraction

It starts with `Execution.java`

@FunctionalInterface
public interface Executable {
    ChainAction execute(ChainContext context) throws ChainException;
}

This is the building block of a chain. We wrap our logic inside the execute method and use the ChainContext to track the current state of the chain. The method returns a ChainAction, which determines whether the chain should proceed to the next step or stop execution.

Returning a ChainAction makes control flow explicit and keeps individual executables unaware of the larger chain structure.

Determining flow with `ChainAction.java`

public enum ChainAction {
    CONTINUE, STOP
}

This is a self-explanatory enum that declares two possible actions after a step completes:

Go to next step
Stop the current chain as is

A custom wrapper `ChainException.java`

public class ChainException extends RuntimeException {
    // Other fields can be added for metadata
}

Remembering everything with `ChainContext.java`

public class ChainContext {
    private final Map<String, Object> values;

    public ChainContext() {
        this.values = new HashMap<>();
    }

    public <T> void setValue(String key, T value) {
        values.put(key, value);
    }

    public Object getValue(String key) {
        return values.get(key);
    }

    @SuppressWarnings("unchecked")
    public <T> T getValue(String key, Class<T> clazz) {
        Objects.requireNonNull(clazz);
        return (T) values.get(key);
    }
}

This class is a crucial core abstraction that generalises what enters an Executable and what leaves it.

We pass inputs by setting them as key-value pairs and similarly handle outputs.

NOTE: Using a Map<String, Object> trades type safety for flexibility. This is a deliberate choice to keep the abstraction simple and generic.

Controlling it all with `Chain.java`

Now that we have defined the building blocks of the chain, we can define a class that orchestrates the entire execution.

public class Chain {
    private final String name;
    private final List<Executable> executables;
    private ChainContext context;

    private Chain(String name) {
        this.name = name;
        this.executables = new ArrayList<>();
        this.context = new ChainContext();
    }

    public static Chain of(String name) {
        Objects.requireNonNull(name);
        return new Chain(name);
    }

    public Chain next(Executable executable) {
        executables.add(executable);
        return this;
    }

    public Chain context(ChainContext context) {
        this.context = context;
        return this;
    }

    public void run() throws ChainException {
        System.out.println("Executing Chain : " + name);
        for (var e : executables) {
            ChainAction state = e.execute(context);
            if (state == ChainAction.STOP) break;
        }
    }

    public void printChain() {
        var s = executables.stream()
                .map(e -> e.getClass().getSimpleName())
                .collect(Collectors.joining(" --> "));

        System.out.println(s);
    }
}

Let's break it down:

We have a name for the chain and a List<Executable> to store all the executables in a sequence.
We define a Fluent API for creating a chain using the static of(String name) and next(Executable executable) and context(ChainContext context) methods
We can then run this chain using the run() method, which iterates over all the executables and executes them
We also have a helper method to visualise the chain with all its steps

Notice that each Executable knows nothing about what comes before or after it. The chain is responsible for orchestration; executables focus purely on their logic.

And that's it! That's all it takes to implement the most barebones Chain Execution Engine.

Putting it Together

Defining a constants class

public class GenericConstants {
    public static final String ORDER_ID = "orderId";
    public static final String CURRENT_STATE = "currentState";
}

Defining the steps


public class ValidateOrder implements Executable {

    @Override
    public ChainAction execute(ChainContext context) throws ChainException {
        System.out.println("Executing ValidateOrder");
        context.setValue(GenericConstants.CURRENT_STATE, "ValidateOrder Done!");
        return ChainAction.CONTINUE;
    }
}

public class CheckInventory implements Executable {

    @Override
    public ChainAction execute(ChainContext context) throws ChainException {
        System.out.println("Executing CheckInventory");
        System.out.println("    Previous State : " + context.getValue(GenericConstants.CURRENT_STATE));
        context.setValue(GenericConstants.CURRENT_STATE, "CheckInventory Done!");
        return ChainAction.CONTINUE;
    }
}

public class ReserveStock implements Executable {

    @Override
    public ChainAction execute(ChainContext context) throws ChainException {
        System.out.println("Executing ReserveStock");
        System.out.println("    Previous State : " + context.getValue(GenericConstants.CURRENT_STATE));
        context.setValue(GenericConstants.CURRENT_STATE, "ReserveStock Done!");
        return ChainAction.CONTINUE;
    }
}

public class TakePayment implements Executable {

    @Override
    public ChainAction execute(ChainContext context) throws ChainException {
        System.out.println("Executing TakePayment");
        System.out.println("    Previous State : " + context.getValue(GenericConstants.CURRENT_STATE));
        context.setValue(GenericConstants.CURRENT_STATE, "TakePayment Done!");
        return ChainAction.CONTINUE;
    }
}

public class SendNotification implements Executable {

    @Override
    public ChainAction execute(ChainContext context) throws ChainException {
        System.out.println("Executing SendNotification");
        System.out.println("    Previous State : " + context.getValue(GenericConstants.CURRENT_STATE));
        System.out.println("    Order with Id: " + context.getValue(GenericConstants.ORDER_ID) + " processed!");
        context.setValue(GenericConstants.CURRENT_STATE, "SendNotification Done!");
        return ChainAction.CONTINUE;
    }
}

Running the Chain

public class Main {
    public static void main(String[] args) {
        Chain chain = Chain.of("order-process-chain")
                .next(new ValidateOrder())
                .next(new CheckInventory())
                .next(new ReserveStock())
                .next(new TakePayment())
                .next(new SendNotification());

        ChainContext context = new ChainContext();
        context.setValue(ORDER_ID, "ORD-123");
        chain.context(context);
        chain.run();

        System.out.println();
        chain.printChain();
    }
}

//    Output:
//    Executing Chain : order-process-chain
//    Executing ValidateOrder
//    Executing CheckInventory
//        Previous State : ValidateOrder Done!
//    Executing ReserveStock
//        Previous State : CheckInventory Done!
//    Executing TakePayment
//        Previous State : ReserveStock Done!
//    Executing SendNotification
//        Previous State : TakePayment Done!
//        Order with Id: ORD-123 processed!
//
//    ValidateOrder --> CheckInventory --> ReserveStock --> TakePayment --> SendNotification

Notice how the ChainContext is being used to transfer data between Executables within the chain.

What this implementation does NOT handle

Other chain actions like SKIP, ROLLBACK, RETRY. These would offer more functionality

Lifecycle hooks for chain and executables, e.g. onFailure(), before(), after()

Parallel execution of steps

A Chain Registry which stores chains based on their names

Final Words

Creating simple abstractions is both fun and educational. They force you to confront trade-offs that are often hidden behind polished frameworks.

Building this executable chain made it clear that many modern systems — from workflow engines to AI pipelines — rely on the same underlying ideas. Once you understand the abstraction, the framework feels far less magical.

I Re-implemented Java Streams to Understand Lazy Evaluation

Sanjeet Singh Jagdev — Sat, 03 Jan 2026 14:18:28 +0000

Why do this?

I have always been very fascinated by how some libraries that I use day-to-day provide us with such easy to use interfaces, all while elegantly hiding the complexity and using efficient techniques.

I wanted to learn this art by implementing something simple yet challenging!

What is Lazy Evaluation?

In simple terms, lazy evaluation means deferring the execution of an expression or piece of code until its result is actually needed.

The best example of this would be the Java Streams API, where we can define certain operations on a collection and then apply a terminal operation on it to collect the result(s) while applying those operations.

E.g.

List<Integer> r = List.of(1, 2, 3, 4, 4, 5, 6, 6, 7, 8, 9, 10)
   .stream()
   .filter(n -> n % 2 == 0)
   .map(n -> n * 2)
   .toList();

Here, filter(...) and map(...) merely describe what should happen. The actual computation happens only when toList() is invoked.

So how do we apply this paradigm to our own code?

To understand this I decided to create a basic implementation of Java Streams from scratch!

Note: This article focuses on understanding lazy evaluation, not on building a production-ready Streams implementation.

Defining the `Flow.java`

For the sake of differentiating from Streams let's call this class Flow.

For simplicity this class will accept only a List<T>, this way we can focus on the Lazy aspect rather than interoperability with all possible java collections.

Let's define the class

class Flow<T> {
private final List<T> collection;
    private final List<T> list;
    private final List<Function<T, T>> ops;

    private Flow(List<T> list) {
        this.list = list;
        this.ops = new ArrayList<>();
    }

    // static method to create a "Flow"
    public static <U> Flow<U> of(List<U> c) {
        return new Flow<>(c);
    }
}

We add a static method of() to create the Flow object (A constructor would work fine as well).

Now in order to perform action Lazily we need to store them so that they can referenced later and evaluated. That is where the List<Function<T, T>> ops comes into the picture.

Why the List<Function<T, T>> ops ?

By forcing every intermediate operation to conform to T → T, we can chain operations uniformly and apply them in sequence for each element. This simplification makes the lazy pipeline easy to reason about, even though it introduces limitations.
We will collect all operations inside this list and later apply them.

Intermediate Operations

Just like Streams API we will introduce a couple of intermediate operations that can be applied to the items in the list.

In this example we will implement the following operations:

map
filter
peek

map()

@SuppressWarnings("unchecked")
public <R> Flow<R> map(Function<T, R> fn) {
    ops.add(t -> (T) fn.apply(t));
    return (Flow<R>) this;
}

This is the core principle behind lazy evaluation. Storing the operation/expression. In the ops list we store the operation as a wrapper function that simply calls the original function and returns its result. (Yes I am aware that this involves an unsafe cast and this implementation might break in some cases).

filter()

public Flow<T> filter(Predicate<T> p) {
    ops.add(t -> p.test(t) ? t : null);
    return this;
}

Again, we wrap the predicate call as function which simply tests the condition and returns the original value or null in case the test fails.

Note: This implementation uses null as a signal to indicate that an element should be dropped. While this works for demonstration purposes, real stream implementations avoid this approach due to safety and composability concerns.

peek()

public Flow<T> peek(Consumer<T> fn) {
    ops.add(t -> {
        fn.accept(t);
        return t;
    });
    return this;
}

For peek there is no side effect and we just want to run the function and return the original value.

Hopefully now the List<Function<T, T>> ops is starting to make sense as we can express each type of operation using this wrapper function.

Note: This trick with Function<T, T> quickly fails when we want to apply operations like limit(n) or skip(n), as it will require creating a new sub-list with these values, hence we won't truly iterate the list just once.

Terminal Operations

For terminal operations we will implement the following:

forEach
toList
reduce

forEach()

public void forEach(Consumer<T> fn) {
    for (T item : list) {
        for (var op : ops) {
            if (item != null) {
                item = op.apply(item);
            }
        }

        // Added to handle the 'false' evaluation of the predicate()
        if (item != null) {
            fn.accept(item);
        }
    }
}

This right here is Lazy Evaluation. Notice that each element flows through the entire pipeline before the next element is processed. This allows the collection to be traversed only once, which is a defining characteristic of lazy stream processing.

The intermediate operations (map, filter, peek) do not execute immediately. They only record intent by storing functions. Execution is deferred until a terminal operation iterates the source collection.

Moving on to other operations.

toList()

public List<T> toList() {
    List<T> list = new ArrayList<>();

    for (T item : this.list) {
        for (var op : ops) {
            if (item != null) {
                item = op.apply(item);
            }
        }

        // Added to handle the 'false' evaluation of the predicate()
        if (item != null) {
            list.add(item);
        }
    }
    return list;
}

Same structure but in this case we collect the final items to a list.

reduce()

public T reduce(T initial, BinaryOperator<T> accumulator) {
    T finalVal = initial;

    for (T item : list) {
        for (var op : ops) {
            if (item != null) {
                item = op.apply(item);
            }
        }

        // Added to handle the 'false' evaluation of the predicate()
        if (item != null) {
            finalVal = accumulator.apply(finalVal, item);
        }
    }

    return finalVal;
}

Finally with the sample principle yet again, we perform the reduce operation like we have in the Streams API.

An Example with `Flow`

class FlowTest {
    public static void main(String[] args) {
        var intList = List.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);

        var sumOfTwiceOfEven = Flow.of(intList)
                .filter(n -> n % 2 == 0)
                .peek(n -> System.out.println("Saw : " + n))
                .map(n -> n * 2)
                .reduce(0, (sum, i) -> sum + i);

        System.out.println(sumOfTwiceOfEven);
    }
}

// Output:
// Saw : 2
// Saw : 4
// Saw : 6
// Saw : 8
// Saw : 10
// 60

As you can see this closely resembles how the Streams API looks and with just a few lines of code we implemented Lazy Evaluation.

What this implementation does NOT handle

Short-circuiting operations like findFirst
Stateful operations like limit or skip
Type safety across transformations

What did I learn?

Implementing even a simplified version of a familiar API made one thing clear: good abstractions are hard-earned. Lazy evaluation is not magic — it’s the result of carefully controlling when and how work is performed. Building this helped me appreciate both the power and the complexity hidden behind everyday libraries.

The entire code for this example can be found in this Github Gist

How would you redesign this pipeline to avoid the shortcomings like

Using null for predicate
Adding short-circuiting operations
Ensuring type safety across transformations

DEV Community: Sanjeet Singh Jagdev

Building a Declarative JSON Extraction Engine with Python’s Annotated Types, jq and Pydantic

Inspiration

Understanding the Problem

Naive Approach

Declarative Solution

High Level Architecture

Diving Deeper

Analyze a field

The backbone Annotated

Resolver

How the Model Is Executed

Nested Models

Error Handling

Closing Thoughts

Designing an Either in Java: Making Failure Part of the Type System

Inspiration

Problem: Exceptions as Hidden Control Flow

Modeling Failures Explicitly

Designing a Minimal Either<L, R>

Refactoring the First Layer: Configuration

Adding Validation as Composition

Handling at the System Boundary

Architectural Implications

Trade-offs

Closing Thoughts

Building an Executable Chain to Understand Composable Systems

Inspiration

Why do this?

What is an Executable Chain?

Designing the Core Abstraction

It starts with Execution.java

Determining flow with ChainAction.java

A custom wrapper ChainException.java

Remembering everything with ChainContext.java

Controlling it all with Chain.java

Putting it Together

Defining a constants class

Defining the steps

Running the Chain

What this implementation does NOT handle

Final Words

I Re-implemented Java Streams to Understand Lazy Evaluation

Why do this?

What is Lazy Evaluation?

Defining the Flow.java

Intermediate Operations

map()

filter()

peek()

Terminal Operations

forEach()

toList()

reduce()

An Example with Flow

What this implementation does NOT handle

What did I learn?

The backbone `Annotated`

Designing a Minimal `Either<L, R>`

It starts with `Execution.java`

Determining flow with `ChainAction.java`

A custom wrapper `ChainException.java`

Remembering everything with `ChainContext.java`

Controlling it all with `Chain.java`

Defining the `Flow.java`

An Example with `Flow`