DEV Community: kenkondo

Hermes Lakehouse Engineer: An Agentic Databricks Data Engineering Teammate

kenkondo — Thu, 28 May 2026 06:56:09 +0000

What I Built

I built Hermes Lakehouse Engineer, an agentic Databricks data engineering experience. It turns a source contract and sample feed into a repeatable lakehouse delivery run: implementation planning, Databricks bundle artifacts, live Delta tables, quality checks, Unity Catalog-style governance metadata, lineage, release notes, and workspace evidence.

The project has a local-first fallback for demo reliability, and the live Databricks path creates real bronze, silver, and gold Delta tables in a serverless workspace job. The point of the build is to show Hermes Agent behaving like a practical data engineering teammate: it carries the work from intake to proof instead of stopping at a generated notebook.

Demo

The demo walks through a new vendor sales feed becoming a governed Lakehouse data product:

Demo video: docs/demo_assets/hermes_lakehouse_engineer_demo.mp4

Recommended screenshots:

Databricks job run showing TERMINATED SUCCESS
Live table list under workspace.sales
Local Hermes delivery loop output
Governance documentation packet

The local demo command is:

python scripts/run_agentic_delivery.py

It writes a timestamped local evidence packet under runs/ so reviewers can inspect exactly what the agent validated, documented, and prepared. The generated packet is ignored by git; the curated live proof is in docs/live_databricks_evidence.md.

The live Databricks demo is:

cd databricks
databricks bundle validate --target dev
databricks bundle deploy --target dev
databricks bundle run vendor_sales_release --target dev

Hermes reads the source contract and sample CSV.
Hermes plans the data engineering work.
Hermes prepares Databricks bundle, job, quality, and governance artifacts.
Hermes deploys a serverless Databricks job.
Hermes creates live bronze, silver, and gold Delta tables.
Hermes validates quality checks against the live tables.
Hermes prints data product documentation evidence for review.

Latest captured live proof:

Job result: TERMINATED SUCCESS
Tables: workspace.sales.bronze_vendor_sales_orders, workspace.sales.silver_sales_orders, workspace.sales.gold_sales_order_metrics
Quality checks: all pass with zero failing rows
Evidence packet: docs/live_databricks_evidence.md

Code

Repository: https://github.com/kenkondo/hermes-lakehouse-engineer

My Tech Stack

Hermes Agent
Databricks Asset Bundles
Databricks serverless jobs
Unity Catalog tables, comments, and governance conventions
Python
SQL
YAML
Markdown

How I Used Hermes Agent

Hermes Agent powers the project as the autonomous data engineering operator. It reads the source contract, reasons through the implementation plan, prepares pipeline and governance artifacts, deploys the Databricks job, validates live tables, and summarizes the release.

The most important Hermes capabilities are planning, tool use, multi-step reasoning, and file-aware execution. Data engineering is not a single code-generation step; it requires understanding the source, designing target tables, enforcing quality gates, documenting governance, and communicating readiness. Hermes is a strong fit because it can carry that workflow across the full local delivery loop.

The Unity Catalog documentation packet is a key part of the project. Hermes acts as both data engineer and data steward by generating table comments, column comments, data classifications, ownership metadata, lineage, and quality expectations before the asset is promoted.

What I liked most about using Hermes here is that the agentic loop matches the real work. A data engineer does not simply write Spark code. They read the contract, make tradeoffs, create deployment artifacts, validate the result, explain what changed, and leave behind evidence that another person can trust. That is the experience this project tries to make concrete.

Team HackReddy

kenkondo — Tue, 27 Oct 2020 18:27:43 +0000

Mulesoft Haikuthon
Every day, API day
Get Reddy, Set Go

I remember the days when I was a child and I used to take paper numbers and watch a red LED light uptick while waiting for my number to come up. I'd secretly hope that my fellow patrons would succumb to their thoughts and abandon their paper ticket spot in line. There were no cell phones back then; you could only people watch or go over the various things in your sight cyclically until you are finally able to do what you came to do.

I know, you're wondering if the semicolon is right in that last sentence… me too.

The next thing I can remember was someone greeting me with a tablet, asking me what I was there for and then seeing my name pop up on a screen in a table of many. This was a game changer because waiting in a line that snakes around or worse many lines in parallel, where you wonder if you chose the right one as Sally in front of you does that one thing that takes forever.

Fast forward to something to the App Economy where every single store has an App and Push notifications and Deals on Guac and French Fries… your hair is long from the shutdown, but now you can get a virtual place in line and show up to get a mediocre haircut without waiting. You can go to the mall and sample Bath and Body Works smells until you get notified that your time to look into the mirror and question your decision to be at the mall for a hair cut and further whether or not that snip was the right one to get you to look like that virtual persona in your head.

Now, let's talk about this year called 2()2(). I write it leetly because it's mere mention speaks to bad JuJu and Team Hack Reddy wants a Dub. (This is slang for saying that I replaced the zeroes with parentheses because writing it out would be bad luck and the team wants to win) In this year, the notion of a line has been magnified 6X, or 3X if we're counting the guy that was behind me, smoking a cigarette, not wearing a max at the DMV.

If you've been to a bank or credit union this year, you've seen what would appear to be opening night at The Apple Store after Tim Cook gave a presentation in a black t-shirt 2 months ago unveiling a mini rounded corner device that you MUST HAVE. It's November now and the temperatures have fallen and in the Bay Area there is this funny thing called rain falling from the sky that hasn't shown it's beautiful grace in 7 months.

These poor people! I mean we need the rain but standing in line in the cold rain in November is not my cup of tea. However, it is completely necessary for Muledred who doesn’t like taking images of checks and then not trusting that she'll get the $8.55 from the book club breakfast check split.

Team HackReddy was called to the scene to solve this most difficult problem: Keep Muledred dry and happy, and make sure she doesn't miss her place in line while she plays scrabble on her phone.

Now we are wondering about the colon, right?

The Team brainstorms and comes up with User Experience Flows and User Stories.

Then, they came up with the design.

The Member could send a text to Twilio which would then send a message to a Mulesoft Webhook that create a Solace Topic that would put the Member into a Solace Queue. A Mulesoft Twilio API subscribed to the Solace Topic would send a text back to the Member letting them know where they are in line and that they could play Jeopareddy while waiting via text.

The Member could also go to a website to check in and get number that way. They can then monitor the queue with the app or get alerts by text as to when they can brave the cold rain and check in.

We would also need a check-in and remove from queue microapp for members that neither texted or went to the website. This same app would check people in when their number was up and they were served.

React opening / onboarding screen:

React waiting list page, polls every 10 seconds:

React Game Page:

Mulesoft Trivia Game Flow:

External Messaging Flow:

Incoming SMS Flow:

Messaging Callback Flow:

Solace System Checkin Service:

React Check-in App (Tablet):

Solace Event Queue:
Taxonomy: redwood/{desc}/{method}/{branch-id}/{action}
Examples: redwood/check-in/text/21/enable
redwood/check-in/web/23/enable
{desc} will always be ‘check-in’ as this is how we distinguish our app from other Solace events.

Queue: redwood/check-in queue listens on the topic redwood/check-in/> to get all events related to check-in.

Mulesoft service is the only method to pull out of the queue, and can be the only listener on the queue, so other apps interested in changes to the queue will need to listen on the topics below. Look to the description to get the result desired.

Topics:
Subscription Description
redwood/check-in/> Gets all events
redwood/check-in///enable Wildcards: Gets all enable or disable events from all branches
redwood/check-in///disable
redwood/check-in/text//enable Method: Gets all text or web or 1mv initiated additions to the queue from all branches.
redwood/check-in/web//enable
redwood/check-in/*/21/enable Branch: Gets all new additions to the queue for a specific branch (21 in this case) regardless of entry method

Reference

Mule application source code:
https://github.com/kenkondo/VirtualTicketQueue
*Note: we did not include the Twilio External Messaging flow as that was an existing service. All others are new.

Team HackReddy
Astro Ken
Cloudy Jeff
Einstein Sean
Codey Adrian
Bot Jake
Appy Tanner
Hootie Rio
Blaze Dean
Trailhead John

This Hack is dedicated to Alex Trebek, may he rest in peace…