DEV Community: Zack Siri

Synthetic Benchmarks Are Clean. Agent Workflows Are Not.

Zack Siri — Tue, 24 Feb 2026 13:08:48 +0000

Most model evaluations happen in controlled environments: single prompts, tidy inputs, and clear expected outputs.

Production agents don’t operate like that.

They route between capabilities.

They call tools.

They maintain state.

They operate under constraints.

They break in subtle, non-obvious ways.

In this post, I’ll walk through the evaluation framework I built to test LLMs inside a structured, stateful agent workflow.

To make the discussion concrete, the walkthrough uses Mistral models as the implementation example — demonstrating the kind of behavior and outcomes you can expect when running this evaluation in practice.

The goal is simple:

Evaluate how models behave inside a real execution system, not how they perform on isolated prompts.

What I’m Testing

The framework evaluates five core capabilities required for practical agent systems:

1. Routing

Can the model correctly identify intent and select the appropriate execution path?

Agent systems depend heavily on correct internal routing. Misclassification at this layer cascades into downstream failures.

2. Tool Use

Does the model call tools correctly with valid structured arguments?

This includes:

Schema adherence
Proper parameter formatting
Calling the correct tool
Avoiding hallucinated fields

Tool misuse is one of the most common failure modes in agent systems.

3. Basic Decision Making

Given system instructions and available tools, can the model take reasonable next steps?

This tests whether the model:

Understands procedural expectations
Chooses logical actions
Avoids unnecessary tool calls

4. Resolving Constraints

Does the model respect hard rules?

Examples:

Forbidden actions
Required preconditions
Deterministic constraint layers overriding reasoning

Real systems include guardrails. Models must operate within them.

5. Handling Multi-Turn Conversation

Can the model maintain state and coherence across turns?

This evaluates:

Context retention
Correct updates to system state
Avoiding contradiction
Consistent identity and task awareness

Single-turn correctness does not guarantee multi-step stability.

How the Test Is Structured

This is not a prompt benchmark. It is a workflow simulation.

The framework includes:

Multi-step task execution (not single-prompt evaluation)
Structured tool interfaces with defined schemas
Deterministic constraint layers applied over model reasoning
Explicit state tracking across conversation turns
Clear evaluation criteria per capability
Repeatable, controlled scenarios

Each scenario is designed to simulate how LLMs are actually deployed inside agent systems — where planning, execution, constraint handling, and state management are all interacting simultaneously.

Why This Matters

Synthetic benchmarks are optimized for clean scoring.

Agent workflows are optimized for messy reality.

When models operate inside structured systems, failure modes shift:

Partial tool calls
Invalid arguments
Broken routing
State drift
Constraint violations

These issues rarely show up in leaderboard-style benchmarks.

What This Post Covers

This post (and accompanying video) focuses on:

The architecture of the evaluation framework
The capability breakdown
The workflow simulation model
A practical walkthrough using Mistral models
The kind of results this methodology surfaces in real agent scenarios

If you're building agent systems or evaluating models beyond chat use cases, this framework is directly applicable to real-world deployments.

Benchmarks tell you how a model performs in isolation.

Agent evaluations tell you how it behaves inside a system.

Core Banking: Complete Guide

Zack Siri — Wed, 14 Dec 2022 09:00:41 +0000

The foundational piece that is needed by any institution that works with financial products. Whether a Bank or a FinTech company.

This post as a third part in the series of posts. I have to say this part is the part I'm most excited to write about. In my journey of building a fintech company this part was the most fun to build. I think during the toughest of times the excitement of building the core banking was what kept me going.

Let's first answer the question of why core banking is needed in banks and fintech companies.

Why Core Banking?

As humans we have multiple ways of handling money. We are evolving from using physical objects and transitioning into more abstract ways of representing money. The modern economy has multiple versions of money, physical cash, cards, and more recently mobile / digital wallets on our devices.

Cash is one way of doing accounting. It is using physical representation to do accounting of how much value one person has in physical form in their wallet. The wallet exists only to hold physical cash. Imagine you have 10 $100 notes in your physical wallet, you go and buy something for $500, you will physically hand over that 5 $100 notes and be left with 5 $100 notes leaving you with $500 in total balance in your wallet. I would say physical cash is a representation of an abstract concept of "accounting".

Now let's imagine the transaction above of purchasing items happening 10,000 times, each with different values and with different people, happening all at once. That's basically how our economy works! When you make a purchase, you are only responsible for how much you spend, the merchant is responsible for how much she has collected. Each party in the transaction is doing their own accounting.

Given that we're now transitioning to a more 'digital' economy. Money will become once again a more 'abstract' concept. You will only experience money in digital form. In fact there is no wallet. The concept of digital wallet exists only on your device to make it easy for the users to comprehend. Naturally if you wanted to have digital systems that keeps track of all the transactions and the ledgers of all the members in the economy everything will need to be digitized! This is the purpose behind core banking systems. Digitizing money enables many more possibilities! Your money is shapeless and formless. They can be accessed on any of your devices and you can send money to anyone no matter how far away they are! You can use your money to purchase goods on online stores, pay for services you use all without ever leaving your house.

How does Core Banking Work?

One of the most basic component of the core banking is the ability to handle transactions. Transactions represent intention to move funds from one place to another. That's all it is. In the above example of buying something. All you are doing is moving $500 from your wallet into the merchant's cash register. Eventually the merchant will 'move' that money collected into some kind of 'bank account' for safe keeping.

Now that we've established what a transaction is. This means you will need some kind of 'source' and 'destination' to keep track of this flow of funds. This also means that there has to be some kind of ledger system within the core banking. Let's take a moment to understand how transactions and general ledgers work.

Let's start with Deposit

Let's imagine you have $1000 physical cash and you wanted to digitize it. In the modern day and age, you would go to the bank open an account and deposit your cash. Once you deposit your cash, you can then access your money using your smartphone or even your watch. Let's take a look at that step-by-step starting with depositing your cash.

You can probably understand the image above. The question may arise "how does this look digitally?". Usually banks will use some sort of centralized database to store this information. Let's take a look at how this would look in the database. Do note that this is a simplified example to illustrate a point. In reality the design may be much more complex.

We'll start with just the account opening. In this case the account opening means the balance will start at zero. The 'deposit' transaction will have to be represented separately. We'll call the table of entities that has the ability to 'hold' an account 'holders'. The actual record that keeps the balance is called the 'accounts'. Given that 1 holder may have many accounts it's better to have this sort of structure.

Next we'll take a look at how a transaction is represented. Given that we've established that a transaction is the 'movement' of fund from source to destination. Cash has to be represented in our system somehow. The transition from physical to digital has to be created in some way in our system. Let's take a look at how that would work.

What we can do is add a 'system' entry with it's own 'account'. The balance is zero but we'll make the 'minimum_balance' infinity since we do not know the limit of cash 🤣. In systems you can represent infinity in many ways. However I won't go into that in this post. We now have a system level 'holder' and has an account with zero balance but infinite 'minimum_balance'.

All we have to do now is create a transaction that represents the movement of this 'Cash' to 'John's' account. We'll introduce a new tables into the mix let's call it 'transactions' and 'changes'.

We use the 'transaction' record as the main record that 'encapsulates' the 2 changes that happened on 2 accounts. The change on each account is recorded separately. We can also use the 'transactions' table to record the 'maker_id' if we wanted, however to keep focus on the actual funds movement in this case we will keep it simple.

You can see that in this case we are removing $1000 from the 'Cash' account. Both the changes are grouped by the same 'transaction_id'. This means 1 transaction will be able to show exactly what happened.

In this case we moved $1000 from the 'Cash' account in the system which has the account_id 2 to 'John's' account which has the account_id 1.

All the entries in the database should represent a complete picture of how the deposit happened when you open an account in a core banking system. This is just one design. There can be many other designs based on requirements.

At this point one may wonder, "how does this get displayed on my devices?". That's a great question. This is where APIs come in!

What is an API?

APIs in full form are "application programming interfaces". In this post I won't go into the full detail, however APIs are essentially a way to expose functionality of some systems to the outside world. They're usually secured with some authentication. This is why when you use your mobile banking application you have to login using your password and your OTP token of some kind.

APIs enable your mobile device or web interfaces, to talk to the core banking in a secure way, and enables your device to retrieve your information from the core banking system.

Making Transfers

Now that we understand APIs and core banking and how they keep track of funds movements inside the system, we can now take this to the next level.

Let's imagine you have a friend who also happens to have an account in this same bank and has deposited. The database would look something like this.

I've also added the maker_id field to track who the "maker" for a given transaction is.

Let's assume that John wants to transfer $500 to Dave. You could assume some kind of UI like this.

Finally in the database a new transaction would be created with the changes to move 500 from John's account to Dave's account.

This is a simple example illustrating the core mechanics of what one possibility of the main components of a core banking system would look like. There are many possibilities depending on requirements.

Complexities

Generally when building core banking, though it may look simple in reality there are many challenges that can cause difficulties in developing such a system. For example. Since creating these transactions can take time to fully execute, you cannot always afford to 'do everything all at once'. Imagine you had 1000s of transactions coming in, you would have to make sure the system can operate asynchronously. Which means you break operations into small parts and put them inside a some sort of queue and distribute the workload.

There are also other problems like race conditions which can crop up. This means you will have to take into account things like locking accounts when a transfer is in progress so double deductions do not happen. It would be terrible for an account holder to have $1000 but due to lack of proper operations enabling the holder to spend $2000 accidentally.

You'll also want to use databases that support transactions of some kind to ensure that all the necessary updates and insertions happen in a single database transaction. If one of those entry fails, it should rollback.

There are also issues of handling double transactions. How do you prevent customers from accidentally creating 2 of the same transactions.

These are just the beginning of the list of common issues that can crop up in core banking systems. Proper engineering and due process need to be put in place to ensure the integrity of the system.

Keeping things secure is another important factor. Since everything is stored in centralized database, ensuring proper procedures are in place to restrict database access is critical. Imagine a hacker hacks into the database and replaces the value of an account from 1000 to 1,000,000. That would not be good would it!

Conclusion

There are many ways to build Core Banking. The same mechanics can be used to power lending products, cross border payments, digital wallet systems and many other types of financial products. Having access to such technologies would have been extremely difficult many years ago. There are platforms like this for sale however they are very expensive and since it's an off the shelf product it may be difficult to customize to serve unique business requirements.

Having the ability to build and maintain a reliable core banking system can be a huge advantage for any organization providing financial services.

There are many more things to explore with the concept of core banking. In this post we only covered account to account transfer within the same bank. How would this work in a cross-bank transfer. We will explore such topics and much more in future posts!

If you are interested in learning more please subscribe. I'm also very versatile when it comes to building such complex systems. I have experience in various industries if you are interested in working with me you can reach me from my consultancy business artellectual.com

Concurrency vs Parallelism

Zack Siri — Wed, 21 Jul 2021 13:37:20 +0000

In modern computing the only way to scale up to increase productivity and output of the system is to increase the core count on the CPU and GPU.

As a developer it's important to understand how this impacts the software that we write. It's also important to understand how things work for us as developer to be able to take advantage of the multi-core nature of the future of computing.

Whether we're handling large amount of web traffic, render demanding 3d movies, streaming video games live on social network, these tasks require multi core processors to get the work done.

In this episode we discuss the difference between concurrency and parallelism.

Ruby on Rails API Authentication

Zack Siri — Tue, 12 Feb 2019 07:14:19 +0000

If you are building any kind of API with Ruby on Rails you are going to have to handle authentication if you want to secure your endpoints. If you're using devise with it's pre-built authentication you'll see that when it comes to managing the Authentication of an API endpoint devise falls slightly short. In this post we will show you some of the options you have with securing your Rails API endpoints and how you can integrate them into devise.

Simple Token Authentication

If you have a simple requirement where you just want basic authentication via supplying a token to the client then look no further. This gem is as simple as the name itself. You can check it out here.

The way it works is you run a simple command that pretty much generates the migration file for the user model.

rails g migration add_authentication_token_to_users "authentication_token:string{30}:uniq"

This essentially adds a authentication_token field yo your user model. You then add a macro to your user model like so.

class User < ApplicationRecord
  acts_as_token_authenticatable
  # ...

end

Then you add another macro to your ApplicationController

acts_as_token_authentication_handler_for User, fallback: :none

Then any endpoint inheriting from the ApplicationController will work with token authentication.

On the client to authenticate with the endpoint the user has to pass the X-User-Email and X-User-Token in the header. That's pretty much it for the simple token authentication gem. Since the gem already works with devise all your devise utilities should work great.

If you are interested in learning more about JWT authentication check out our free episode below.

JWT Authentication with Devise

JSON Web Token is all the rage when it comes to api authentication. This is because it makes life convenient when it comes to securing your API. How? Well If you are building your API for multiple clients, by using JWT you have 1 method of authentication that will work across all platforms whether you are authenticating a client in the browser, a mobile application, or any kind of server side service consuming data from your api.

It supports a ton of features and allows you to generate a token for the client on the fly without having anything stored in the database, because the token itself contains the session information. You can generate a token and embed any kind of information into it. For example

{
  "user_id": 1,
  "exp": 1516239022
}

Gets turned into

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2lkIjoxLCJleHAiOjE1MTYyMzkwMjJ9.UCTyi_gPUNhfeelaWxMTfnPtRtNByDUMxKTRpkyewQI

This is a cryptographically secure token generated using a secret key on the server. Which means it can only be decrypted by any party with the secret key. Which means if this secret key gets compromised your authentication token method will be compromised.

The client then embeds this token in the header usually called Authorization. Each request needs to be authenticated on the server using this token. Which means the token sent by the client needs to be decoded on the server and the data can be verified on the server side. You can check out more about the basics of JWT in our free video.

In the video we show you how to generate a basic JWT token using the jwt gem. We event show you how to fulling develop an authentication solution using JWT.

Once you start working with JWT in your Rails API you'll realize that it doesn't work with your existing devise setup. Which is why we recently released 2 episodes showing you how to integrate your JWT solution with devise by creating a custom devise strategy. You can check them out below. They're member's exclusive so you'll need to become a member for $9 / month, however if you just want the code example it's available for free by clicking the image below.

If you are building an API with Ruby on Rails and don't know where to start, I would recommend going for JWT. Since JWT is more future proof and there is a standard that you can follow. Having a standard can be useful because you have 1 authentication system that works in many platforms / languages. So if you are building micro-services in multiple languages you can use the libraries for the language of your service and have different services use 1 authentication system that work with one another.

JWT as a solution can grow as your application grows in complexity. You can start off with a database-less JWT solution and later on add the ability to store the token in the database so you can expire tokens at will.

On our site we use JWT for many things, authenticating our users, service-to-service communication, and authenticating video plays. Since we can embed data into the token itself this makes it very flexible for many kinds of things.

Rails Routes, HTTP Methods and SQL Queries

Zack Siri — Tue, 29 Jan 2019 12:52:42 +0000

Learning how the rails routes are generated and how they map to HTTP methods and SQL queries is a foundational piece of knowledge that is missed by many developers when starting out. In this post we will explore some of the basics that will help fill in the gap so that students can understand Rails routes in the least amount of time.

Rails convention depict that a controller should only have 7 actions most of the time. Generally speaking if you are creating a controller or defining routes and you feel like you need something other than the 7 standard action you should consider creating a new controller.

In the rest of this post we uncover how Rails Routes map to HTTP Methods and SQL Queries

Ruby Symbols

Zack Siri — Fri, 18 Jan 2019 08:58:14 +0000

Symbols are immutable data structure in ruby, it can come in handy in many instances. We also explore the difference between ruby's Strings and Symbols in this episode.

This video is a part of the Ruby From Scratch Series on Codemy.net

Create a Custom Plug

Zack Siri — Fri, 21 Dec 2018 10:41:45 +0000

We show you how to create your own elixir plug in 3 parts. Check out the free code example.

Learn React on Rails

Zack Siri — Wed, 26 Sep 2018 08:49:00 +0000

React and Ruby on Rails is a powerful combination. This collection is a combination of the React Foundation, Rails API and React Intermediate series. It is organized in sequence of how the invoiced application is created. This is a great way to learn how to build React apps that get their data from a Rails API backend.

Upgrading Legacy Rails App Guide

Zack Siri — Tue, 11 Sep 2018 12:21:01 +0000

Upgrading a Legacy Rails App Part 1 - Upgrading a legacy rails app is a chore for every rails developer. Knowing the right strategy can save you a lot of time. We'll be showing you exactly how to go about it in this episode!

Upgrading a Legacy Rails App Part 2 - We continue the upgrade process from the previous episode. Now that our application is running on the stable version of 4.2.10 . We can start the process of upgrading to 5.2.1 . We will use the railsdiff.org extensively to guide us through the upgrade process.

Upgrading a Legacy Rails App Part 3 - We finalize the upgrade process by patching up a few files. Once all the changes are in place we should be able to run bundle update rails