DEV Community: TC Ricks

Why AI agents fail without a Data Layer

TC Ricks — Wed, 29 Oct 2025 18:01:07 +0000

AI agents can reason. They can plan, summarize, and even write SQL.

But they can’t fix messy data.

That gap between reasoning and reality is where most AI projects stall, not because the models don’t work, but because the data underneath them is too fragmented to think with.

The hidden bottleneck

Every company wants to give their business users an AI assistant that can answer questions like:

"Which campaigns drove the most revenue this quarter?"

Or even more ambitious:

"If we increase ad spend by 10% across platforms, what happens to conversion cost?"

The problem isn’t the prompt.

It’s that the model has to reason over conflicting definitions of “campaign,” multiple “revenue” columns, and data scattered across CRMs, ad tools, and spreadsheets.

The result is confident answers that are wrong.

Large language models can generate logic, but they can’t repair the structure that logic depends on.

They don’t know which system is right when numbers disagree, and even if you’re able to give them guidance, it’s forgotten the moment you start a fresh conversation.

Reasoning requires ground truth

Think of an LLM as a sharp analyst dropped into a disorganized office.

They can ask smart questions, make connections, and even propose insights, but if every folder has its own version of “customer_data_final_v2,” they’ll still get lost.

LLMs need a consistent, unified layer underneath them to reason safely.

That layer provides:

Entity resolution (so “user,” “lead,” and “customer” point to the same person)
Normalization (so “spend” always means the same thing)
Proof and lineage (so you can see why the AI answered the way it did)

Without it, the reasoning is just pattern-matching on chaos.

Example: Why AI struggles with marketing data

Consider a simple question:

"Which campaigns had the highest ROI this month?"

The data for that question usually lives across multiple tools:

LinkedIn Ads: campaign performance data such as impressions, clicks, conversions, and spend
Reddit Ads: campaign performance data that captures similar metrics but uses different naming conventions, attribution logic, and data structures
Google Sheets: manually tracked pipeline and revenue data that connects campaign activity to actual business outcomes

At first glance, this looks easy to join. Both ad platforms are reporting the same kind of metrics.

In practice, the schema, field names, and even metric logic differ enough that an LLM can’t cleanly reconcile them.

For example:

Reddit might call a column “Total Spent (USD)” while LinkedIn calls it “Amount Spent.”
Reddit’s conversions might be attributed by “View” and “Click,” while LinkedIn uses “Post Click” and “Post View.”
The underlying campaign IDs differ across both systems, and neither directly ties back to the deal IDs tracked in Sheets.

So the problem isn’t that the data types are different.

It’s that the definitions, identifiers, and relationships are inconsistent across tools, and no AI model can reliably resolve that structure automatically.

What that inconsistency looks like in practice

Problem Example:

Mismatched identifiers: LinkedIn uses “Campaign ID,” Reddit uses “Campaign Name,” and the Sheet uses free-text entries like “Spring Promo.” The AI can’t reliably join them.
Inconsistent schemas: LinkedIn and Reddit measure engagement differently. One uses CTR, the other uses CPE. The model treats them as comparable.
Misaligned time frames: Reddit reports in UTC, LinkedIn in local time, and the Sheet updates once a week. The AI merges mismatched periods.
Different metric logic: The Sheet’s “revenue” column includes both “won” and “lost” deals, while ad platforms count all spend as active.
No persistent schema: Even if the AI fixes these issues once, it doesn’t remember them for the next query.

The output looks intelligent. It returns numbers and charts, but the logic underneath is incoherent.

What looks like a small naming issue becomes a silent failure mode when these systems feed an LLM.

The model isn’t lying; it’s improvising with incomplete context.

To answer that question correctly, several structural steps need to happen first:

Entity resolution: connect “Spring Promo” across Sheets, LinkedIn, and Reddit even when IDs or names differ.
Normalization: unify currencies, time zones, and date formats.
Semantic alignment: define what “revenue” and “ROI” actually mean across all systems.
Proof and lineage: show exactly how those numbers were derived.

None of this can be done reliably by an LLM alone.

It requires a persistent data layer that understands relationships, definitions, and meaning.

Once that layer exists, the AI can reason instead of guess.

The same prompt, “Which campaigns had the highest ROI this month?”, now produces consistent, explainable results across every source.

In AstroBee, you can literally see every system mapped to the same entity definitions so “customer,” “lead,” and “user” finally mean the same thing.

And this story repeats everywhere

Marketing is only one example.

The same pattern plays out across every function in a business:

Product managers want to connect feature usage data with revenue outcomes, but product analytics and billing rarely share entity definitions.
Sales teams track leads in CRMs that don’t match marketing definitions of “qualified,” making pipeline reports unreliable.
Operations and finance rely on metrics from systems that calculate costs differently across regions or business units.

Each role faces the same challenge: they know what questions to ask, but not how to unify the context to get trustworthy answers.

Without a data layer to standardize meaning, every query becomes a one-off cleanup job instead of a reusable insight.

Business users can’t build this infrastructure themselves

Before AstroBee, the only people who could create this kind of foundation were data engineers and analytics teams.

That meant every marketing manager, product lead, or ops analyst had to file tickets and wait weeks just to get clean data to work with.

It’s the same bottleneck data scientists faced years ago.

They could train models, but couldn’t deploy them without infra engineers.

Now business users face the same wall.

They understand the business context better than anyone, but don’t have the tools to access or unify their data effectively.

They don’t need another dashboard or prompt template.

They need a system that builds the data foundation for them, so AI can actually reason with the business context they know best.

The foundation matters more than the model

A great model sitting on messy data is like a Ferrari on gravel.

No matter how powerful it is, it’s not going anywhere fast.

No matter how powerful the LLM is, if the data foundation is incomplete, automated reasoning will fail.

The real innovation isn’t another smarter LLM.

It’s the layer that makes your company’s data understandable, traceable, and ready for reasoning.

That’s what AstroBee focuses on, creating a unified data layer that lets business users query, explore, and automate decisions confidently.

Because the future of AI in business isn’t about bigger models.

It’s about giving every user solid ground to stand on.

Closing thought

AI doesn’t fail because it’s not smart enough.

It fails because it doesn’t understand your business.

The fix isn’t a better chatbot.

It’s a better foundation.

If this challenge sounds familiar, we’re always open to comparing notes.

Feel free to reach out at tc@astrobee.ai to talk through what a cleaner data foundation could look like.

Where MCP falls short: data integration in the AI world

TC Ricks — Mon, 11 Aug 2025 18:21:39 +0000

Connecting to disparate source systems is a solved problem. Tools like FiveTran, AirByte, and merge.dev sink thousands of hours of engineering work into supporting connectors to every software system on the planet, and most teams who supporting 1,000s of connectors can use one of these tools under the hood to get there.

Once data is connected, you still have the problem of data representation. Each of your source systems (like HubSpot and Intercom, or your billing system and product database), will represent data in their own way. That’s what you get when you connect to APIs, you must speak the language of the API provider. It’s what you get when you connect to MCP servers, too. It’s the same structure, and presents data the same way.

If you connect an LLM to 5 different MCP servers and then ask for information regarding your customer, say Acme Incorporated , your answer will be degraded by the fact that each source system might represent customers differently. Maybe in HubSpot, companies are represented by their primary domain, getacme.com . Maybe in your billing system, they’re represented by the “Doing-business-as” name; Acme. And then maybe in your product, there are three subsidiaries that have engaged and are submitted support requests in Intercom under the names of those subsidiaries.

For those who are dead-set on AI enablement, there are two options available to solving this problem:

Resolve Entities on Inference: Trust the LLM to expend enough tokens in “thought” to scan through all of the data available in each source system via its MCP connections to exhaustively search around the Acme Incorporated concept to figure out where data related to Acme lives in your systems, and then report on it
Resolve Entities Ahead of Time: Use traditional data processing techniques to reveal gaps in entity resolution prior to inference, and then use inputs from business stakeholders and data experts to resolve entities in an integrated “source of truth” layer. LLMs refer to this layer on inference, instead of raw source systems.

If you just upload small amounts of data to Chat-GPT or Claude, it seems like the first option will work great. LLMs are very good at this semantic search and conceptual construction work. Your CSVs fit entirely into the context window of the LLM, and every token emitted is constructively contributing to grouping like elements together.

The moment you start working on larger datasets, the first option falls apart. For large scale search-and-compute, LLM-driven crawling is painfully slow compared to traditional data processing techniques. Semantic search works great at larger scales, for what it does, but is different altogether from reasoning occurring within one context window.

You could view this as an issue of context window limitations, but I think this is short-sighted. The context window is what it is. In the medium term, at least, it seems like it’ll always be a factor we need to design around, in the same way that RAM is still a factor to design around no matter how much memory we manage to get on our laptops. I think the context window is just the wrong tool for the job. Use token-based compute for what its good at, and use classical compute for what it’s good at: large scale data processing.

At AstroBee, we don’t foresee a world where MCP alone is the primary way to get strong AI capabilities across your enterprise. We see MCP working great for specific tool calling, and faltering when data needs to be reconciled across tools in order to make good decisions.

AstroBee integrates your data for you, so that entity resolution across your different systems happens before inference, and with the right tools: a mixture of SQL-driven integration, semantic search, and a touch of LLM-driven reasoning. Then, when AI agents work over your enterprise data, they work with pre-integrated data, delivering significantly better results and making significantly better decisions than before.

To see an alpha research preview of AstroBee, visit us at https://astrobee.ai. If you’d like to discuss our approach and give feedback or special requests, contact me directly at support@astrobee.ai.

What the heck is an Ontology?

TC Ricks — Fri, 08 Aug 2025 15:50:39 +0000

Is Palantir onto something, or are they just blowing hot air?

You may have heard the term Ontology, and why leading businesses rely on them to build trustworthy data models. Recent advances in LLMs have made it possible for startups and enterprises alike to build flexible, reliable semantic layers faster than ever before.

What do we mean by “ontology”?

The ontology has its roots in philosophy, where it represents a formal catalog of objects, the properties that describe them, and the relationships that connect them. Here’s a simple example:

You might notice that this begins to resemble a data model. The Ontology has become popularized by Palantir as a semantic layer solution, which, to be reductive, is essentially a relational database with a semantic structure (different from the web 2.0 semantic ontology). This Ontology acts as an abstraction between raw data and the business concepts they represent. There are many implementations of semantic layers under a variety of names, but the bottom line is they’re working to make data represent our real world as we understand it.

Why do people want semantic layers anyway?

Every term you’ve heard — source of truth, golden tables, semantic layer — is chasing the same promise: let anyone ask and receive answers that faithfully reflect how the business works. An analyst should be able to ask, “How many premium users bought espresso last month?” and trust the answer. Well-implemented Ontologies deliver on that promise by encoding business logic and enforcing queries adhere to that model.

Ontologies in the Real World

As you might guess, this isn’t a new idea.

The star schema, introduced in the 1990s, puts a wide fact table at its center and fan‑outs to dimension tables on the edges. It celebrates speed and simplicity; query planners can predict access patterns, and BI tools understand the layout instinctively. This modeling approach forces data into either “dimensions” or “facts”. Facts generally have many, many rows with a few columns. The idea is to represent that which is large and highly repeatable row-over-row.

However, this hard divide between facts (immutable events) and dimensions (reference data) falls apart as the business evolves-say, a new pricing plan blurs the line between a transaction and a subscription, those early modeling choices become concrete shoes.

Snowflake schemas attempt to tame redundancy by further normalizing dimension tables. Storage usage drops, but the price is paid in join complexity. What once took a single hop now requires a small expedition through intermediary tables, and each extra join is another surface for error or performance regressions.

Both designs inherit the fact‑dimension mindset. It has served analytics well for decades, but it assumes the world stays still long enough for engineers to carve data into neat shapes. Modern businesses rarely stay that sessile.

Palantir’s Ontology: turning the model into a graph

Palantir threw out the fact‑dimension wall and treated every table as a node in a directed graph. Links between nodes come with explicit types, one‑to‑one, one‑to‑many, and so forth, so analysts can traverse the graph without guessing how joins should behave. The model is also iterative. If tomorrow’s reality demands a new object or a new relationship, you add a node or an edge and keep moving. Query optimizers still have plenty to chew on, but modelers are no longer forced to contort reality to fit a star pattern decided months earlier when the business looked dramatically different.

Palantir’s Ontology reflects current semantic logic while remaining fluid enough to reflect the changing reality of the business.

Why volatility hurts everyone-enterprises and startups alike

Unfortunately, it is incredibly challenging to build a solid, trustworthy source of truth that is also flexible enough to reflect a fast-moving environment.

Consider an enterprise that suddenly faces a new regulatory regime or an unexpected cybersecurity threat. Core schemas that once changed quarterly may now shift weekly as compliance teams add tables, columns, or entire data sources. Meanwhile, a fast‑growing startup is rewriting its own playbook-pivoting products, experimenting with pricing, or integrating a surge of user‑generated events. In both cases, pipelines that were handcrafted for yesterday break under the weight of today’s questions. The cost is measured in stale dashboards, duplicate metrics, and lost decisions.

Palantir gets around this problem by using the forward-deployed engineer model; embedded analysts who work to deeply understand how a business operates across industries, then task themselves with manually, and often painstakingly, maintaining an Ontology for a Fortune 500 company. Meanwhile, startups simply accept analytical debt as the price of speed.

LLMs have changed the game: automatic ontology generation

Large‑language models have changed the economics of generating and maintaining Ontologies. Point them at your data warehouses and they can take a massive snapshot of the current state of your data. They can then:

Scan thousands of objects, notice common naming conventions, and summarize as a data model
Spot tables that share primary keys or semantic similarities in values, and propose relationships
Propose relationships between objects with cardinality

What previously required weeks of an expensive embedded analyst working to understand every nook and cranny of your data warehouse, an LLM can gain a version of that understanding in moments. The challenge is that LLMs, unlike a human analyst, don’t innately know that your “customer” excludes free‑trial users or the specific way your company calculates key metrics. Without guidance, they hallucinate, generalize, or miss edge cases. That’s where human context and Astrobee come in.

Astrobee, the collaboration layer

Astrobee is the collaboration layer between subject‑matter experts and the LLM runtime.

Extract -We ingest your warehouse and scrape lineage to give the LLM its raw material.
Draft — The model proposes objects, links, tests, plus the SQL and pipeline code to populate them.
Review — Stakeholders approve, comment, or override in a Git‑style diff. Astrobee records every decision.
Democratize Data — Anyone across your business can query your data, and rest assured they’re all referencing a single source of truth, Ontology

As more questions flow through Astrobee, it spots patterns and optimizes itself. Repeated joins become reusable generated pipelines. Expensive ad‑hoc queries trigger recommendations to use specific measures org wide.

The result is compound leverage: each inquiry refines the ontology, and consequently, business insights, for subsequent users. Enterprises manage schema drift without large data teams, and startups achieve enterprise-grade modeling agility at seed-stage budgets.

If this sounds interesting to you, we’d love to chat. Drop us a line at hi@astrobee.ai

Kimball Star Schema vs Palantir’s Ontology

TC Ricks — Wed, 06 Aug 2025 15:38:34 +0000

Here’s a joke I thought of:

Q: What does the father of data warehousing have in common with a $300B analytics powerhouse named after a Lord of the Rings spying device?

A: Both of them defy the star schema paradigm and encourage you to model your data top-down, in a highly normalized format.

…maybe you didn’t think that was funny, but I did. Not funny in a “ha-ha” way, but in a glitch-in-the-matrix kind of way. Like “huh, that’s funny”.

Pretty much everyone in data today uses some variation of the star schema paradigm popularized by Ralph Kimball. If you’re familiar with data warehousing or business intelligence, you’ve probably seen “facts” and “dimensions” tables floating around in your tools. Those are Kimball’s work and they really are the industry standard for modeling data for BI purposes. They’re everywhere.

Palantir, a $300B data analytics company, makes a ton of money selling services that model enterprise data in a way that runs directly counter to the Kimball industry standard. They center their offering around the ontology, a highly normalized data model that represents the “objects” a business user would think about. These objects have links between them characterized as 1-to-1, 1-to-many, or many-to-many. It’s a highly intuitive structure.

The weird thing is that most of the industry doesn’t do that. Palantir’s roaring success in deploying teams of forward deployed engineers to build out ontologies in their platform is a testament to the power of this modeling technique. But usually there’s nothing new under the sun. So why isn’t anyone else doing this?

To me, as a former Palantir forward deployed engineer, it felt like a glitch in the matrix. It didn’t quite make sense.

I would come across customer data warehouses which were highly denormalized and structured in “facts” and “dimensions”. I asked myself why on earth they adopted this strategy, instead of the more intuitive, straightforward object-property approach. Following this train of thought, I uncovered a fascinating story of a passionate but respectful technological debate between titans of the data industry, Bill Inmon and Ralph Kimball.

Bill Inmon, from the beginning, advocated for a top-down approach to data modeling. He felt enterprises should establish what they need from their data warehouse in a single source of truth, and then do the integration work upfront to map raw data from source systems into the source of truth in a highly consistent way. He advocated for a normalized layer (many tables, not pre-joined), pushing joins until the very end of the analytics process. This maximizes data availability, ensuring information is not aggregated away or dropped in “join” steps before it reaches the end user.

Ralph Kimball, on the other hand, is largely responsible for the “dimensional modeling” approach that most people use today. You may know this as the “star schema”. The approach has a lot of great features that I love: it’s incremental, decentralized, simpler up-front, and returns user-facing results quickly. It’s a “bottoms up” methodology; you can incrementally deliver analytics from each source system in your organization before you create a centralized, consistent source of truth. As a result of these really great properties, Kimball’s approach has achieved full ideological capture of the data warehousing community.

So what’s going on with Palantir? Why did they go so off-track of the consensus?

I don’t think they’re contrarian for the sake of being contrarian. Palantir sells contracts on business results and the price tag is high. High enough to be able to deploy specialized, highly skilled resources with the sole goal of developing a data model that achieves 7 or 8 figure business outcomes. In this environment, Palantir is able to pay the heavy up-front human cost of the Inmon, top-down approach: integrate all the data you can, pay down the effort to achieve strong consistency across disparate data sources, and deliver a high-level model that fully delivers on data availability in a human-understandable way (the ontology).

Most data teams don’t have the luxury of the budget, focus, and firepower of Palantir to build and maintain an ontology. Kimball’s approach is far easier, but comes with the downside of a lack of data visibility and coverage for the business user. Usually people close the gap with fancy dashboards and on-call schedules for analysts who respond to ad-hoc requests from business users.

AI is rapidly changing what people can do with a limited amount of budget and human firepower. In my opinion, now is the time to invest in technology that cuts the expense of ontology curation by 90%. That’s what we’re working on at AstroBee. We’re using LLM-based agents to do the upfront integration and data consistency work required to bring a full-coverage ontology to your team at an affordable price point. You could say that AstroBee brings the firepower of Palantir to the little guy. We’re gearing up to launch soon, but if you’d like to take part in early pre-launch user testing, please reach out to support@astrobee.ai!

Develop on a branch of your remote cluster with Tilt and Kardinal

TC Ricks — Fri, 11 Oct 2024 17:45:07 +0000

Tilt is a powerful tool for microservice development, automating many steps including watching files, building images, and keeping an up to date environment. However, its effectiveness is often limited to local Kubernetes clusters. For stacks that require remote development clusters, the operational overhead with using Tilt can become a significant hurdle.

Enter Kardinal, an open-source solution for spinning up extremely lightweight ephemeral development environments within a shared Kubernetes cluster. By leveraging request-level routing, Kardinal maximizes the reuse of microservices and databases while maintaining isolation. An environment in Kardinal is like a “branch” of a deployed application on Kubernetes. You only deploy what you need, streamlining overhead and reducing cloud costs by over 90%—especially beneficial for larger teams and complex microservice architectures.

In this tutorial, we'll combine the strengths of developing on Tilt, and the efficient service routing of Kardinal. You'll learn how to effectively deploy your applications, manage resources, and build containers while ensuring a smooth and efficient development experience.

Curious to dive deeper? Check out our GitHub repo for more info on Kardinal, and don’t miss the Tiltfile in the Kardinal Boutique demo app for a hands-on experience.

Ready to transform your development process? Let’s get started.

Prerequisites

Before we dive in, make sure you have the following set up first:

Kardinal CLI
Tilt
A local Kubernetes (K8s) cluster (you can use Minikube or Docker Desktop)
- ⛳ flag: If you’re new to Kardinal, we suggest trying it locally first, before using it with your remote clusters.
Istio
- 💡flag: It’s not important that your cluster uses Istio. Kardinal will manage everything related to Istio on your development cluster.
Distributed tracing on your application
- For a full list of supported distributed tracing systems, check out our docs here.

Setting up Kardinal for your application

Getting started with Kardinal involves deploying Kardinal Manager to your cluster, and annotating your kubernetes manifest. Don’t worry, this should only take few minutes!

If you haven’t installed Kardinal yet, run the following command:

curl get.kardinal.dev -sL | sh

Lets set up a local development cluster for testing first. All you need is a cluster with Istio installed, with kubectl pointing to your cluster. If you don’t have one handy, not to worry!We can get set up by installing the below on your machine:Then running the following:
- Docker
- Minikube
- Kubectl
- Istio

minikube start --driver=docker --cpus=10 --memory 8192 --disk-size 32g
minikube addons enable ingress
minikube addons enable metrics-server
istioctl install --set profile=default -y

This is normally when you would deploy Kardinal Manager to your cluster, however in this case, we’ll be using Tilt for this so you can skip for now.
We’ll then get started by having Kardinal recognize your application’s entrypoint. Kardinal supports managing access to your services either through the Kubernetes Ingress API or the Gateway API.All you need is to add the kardinal.dev.service/ingress or kardinal.dev.service/gateway annotations to your Ingress or Gateway manifest to mark this entrypoint.You can find more information and example annotations in our docs here.

You’re all set! Now to move onto the fun part.

Deploy your application

Option 1: deploy your current Kardinal topology

If you've already deployed your application's manifest using the kardinal deploy command, your cluster topology is ready for deployment with Tilt. If you haven’t, check out our getting started docs here.

Run the command:

sudo tilt up

Create a Tiltfile with the following content:

kardinal_topology_yaml = local(['kardinal', 'topology', 'print-manifest', '--add-trace-router'], quiet=True)
kardinal_topology_yaml_str = str(kardinal_topology_yaml)

if kardinal_topology_yaml_str != '':
    k8s_yaml(kardinal_topology_yaml, allow_duplicates=True)

local_resource(
    name='ingress-gateway-port-forward',
    serve_cmd=['kubectl', 'port-forward', 'service/istio-ingressgateway', '80:80', '-n', 'istio-system']
)

In this setup, the first local call retrieves the cluster topology from Kardinal Kontrol using the kardinal topology command. This generates a multi-resource manifest that Tilt applies with the k8s_yaml command. The local_resource function then executes the port forwarding command, allowing the Ingress Gateway to handle browser requests on port 80.

Option 2: deploy your manifest file with Kardinal annotations

You can also incorporate the kardinal deploy command within Tilt to manage the deployment flow seamlessly. Here’s how:

Run the command:


sudo tilt up

Create a Tiltfile like this, replacing the placeholder data:

local(['kardinal', 'deploy', '-k', '{your-kardinal-manifest-yaml-filepath}'])

kardinal_topology_yaml = local(['kardinal', 'topology', 'print-manifest', '--add-trace-router'], quiet=True)
kardinal_topology_yaml_str = str(kardinal_topology_yaml)

if kardinal_topology_yaml_str != '':
    k8s_yaml(kardinal_topology_yaml, allow_duplicates=True)

local_resource(
    name='ingress-gateway-port-forward',
    serve_cmd=['kubectl', 'port-forward', 'service/istio-ingressgateway', '80:80', '-n', 'istio-system']
)

In this example, the kardinal deploy command is used at the beginning to deploy your multi-resource manifest file directly to Kardinal Kontrol.

Build and deploy your application

Integrate Kardinal with Tilt to build your application containers and set up workflows that automatically reflect changes when you save your files. Follow these steps:

Run the command:

sudo tilt up

Create a Tiltfile like this, replacing the placeholder data:

local(['kardinal', 'deploy', '-k', '{your-kardinal-manifest-yaml-filepath}'])

local(['kardinal', 'flow', 'create', '{service}', '{service-dev-image}'])

docker_build(
    '{service-dev-image}',
    context='{./src/}',
    dockerfile='{./src/Dockerfile}',
)

kardinal_topology_yaml = local(['kardinal', 'topology', 'print-manifest', '--add-trace-router'], quiet=True)
kardinal_topology_yaml_str = str(kardinal_topology_yaml)

if kardinal_topology_yaml_str != '':
    k8s_yaml(kardinal_topology_yaml, allow_duplicates=True)

local_resource(
    name='tunnel',
    serve_cmd=['minikube', 'tunnel']
)

In this setup, we introduce two new elements: the kardinal flow local execution for creating the development flow and the docker_build function to link the container being built with the service in the development flow. This setup triggers Tilt's hot reload mechanism whenever changes are made to files within the specified context.

Accessing PROD and DEV flows

Once your Tiltfile execution is complete, you’ll see URLs for both the prod and dev applications printed in the logs. Click on these links to access the respective flows in your browser.

Cleanup

After finishing your development cycle, you can remove all created resources from the cluster by running the following command:

tilt down --delete-namespaces

The --delete-namespaces flag will remove your application’s namespace, but it won’t affect the default namespace.

Once you feel comfortable with running this setup locally, you can deploy to remote clusters as needed, enjoying the benefits of both Tilt’s devex and Kardinal’s efficient dev flows.

Helpful resources

If you want to learn more about Kardinal, get additional support, or test it out without installing anything-here are some helpful links:

Explore our docs to learn more about how Kardinal works.
Ask questions and get help in our community forum.
Test Kardinal without installing via our playground.
For additional examples of configuring Kardinal with Tilt, check out the Tiltfile in the Kardinal Boutique demo app.

Happy coding! 💻

Building the lightest-weight Kubernetes dev ephemeral environments

TC Ricks — Thu, 22 Aug 2024 15:17:03 +0000

We’re excited to introduce Kardinal, an open-source tool that’s designed to make development and test environments for Kubernetes-deployed applications as lightweight as possible. If you manage multiple application deploys across dev, test, and QA, or you’re spinning up expensive development sandboxes, Kardinal can cut down resource usage and time-to-test by over 90%.

Kardinal makes development easier

Testing new features often means spinning up dedicated application environments, which can be both costly and resource-draining. Kardinal flips the script by letting you create temporary, tailored environments that are dynamically managed. This means you get to focus on what really matters without the hassle of managing extra overhead.

Kardinal redefines development environments by introducing the concept of "flows": dedicated paths through your Kubernetes cluster that act as ephemeral environments. These flows are created on-demand to deploy and test the specific versions of services relevant to feature development. In traditional development setups, managing and testing new features often involves deploying numerous resources that other developers have already deployed in some form or fashion, leading to inefficiencies and high resource usage.

These inefficiencies are addressed by deploying the minimum necessary resources related to the feature under development, and reusing any other resource that’s already been deployed in another workflow. This strategy optimizes resource usage. Whether you need a single-service test or a full application flow with isolated state, Kardinal’s flexible architecture ensures you have the tools to test and develop efficiently.

Some of the benefits of this approach:

Ephemeral environments: Spin up a new environment exactly when you need it, and just as quickly spin it down when you’re done.
Minimal resource usage: Only deploy the services you’re actively working on. Kardinal takes care of the rest, so you don’t waste resources.
Flexible flow types: Whether you need to test a single service or an entire application, Kardinal has you covered:
Single-service flows: Perfect for when you’re tweaking just one service.
- Multi-service flows: Ideal for when your feature involves multiple services.
- State-isolated flows: Great for features that need their own databases or caches.
- Full application flows: For those times when you need end-to-end testing with full isolation.
Cost savings: Kardinal can help you save big by avoiding unnecessary resource duplication. Check out the cost calculator here.
Open source: Kardinal is open source, so use it however fits best within your workflows

How Kardinal works

Kardinal is designed to streamline dev and testing environments for your Kubernetes-deployed application.

Here’s a quick look at how it operates:

Kardinal manager + Kardinal kontrol (control plane)

At the heart of Kardinal are Kardinal kontrol and Kardinal manager. Kardinal kontrol is a cloud-hosted control plane that maintains information about topology and resources in your cluster. Kardinal manager is deployed as a service inside your cluster. It maintains a live cluster topology from the control plane and uses routing rules with Istio, to ensure that traffic is routed properly within your environment.

Managing state with plugins

To handle state in your cluster, Kardinal uses a flexible plugin system. These plugins help manage and share state across different environments or set up new “dev” versions of stateful services as needed. Here’s a breakdown:

Stateful Service Plugins: These are for managing state in services running on Kubernetes, like Postgres databases.
External Service Plugins: These handle state for external services not managed by Kubernetes, such as Stripe or Twilio.

By using these plugins, Kardinal can efficiently set up and tear down flows, reusing existing state or creating new instances as needed.

Plugins operate by applying patches on top of your baseline Kubernetes manifest, which appropriately modify the deployment of the stateful service, or services that depend on external services, for the new flow. For example:

You can modify a microservice deployment spec to change an API key environment variable to access a separate dev account for an external service.
You can also run scripts within plugins to do things like provision new state, for example to deploy a dev database seeded with data for testing purpose.

Visualizing a cluster

Kardinal includes a frontend that provides a visual representation of your cluster. This frontend connects to the Kardinal manager to retrieve info, show service dependencies, and manage multi-user scenarios. Currently, the control plane is cloud-hosted, but an on-premise option is in the works.

You can read more about our high level concepts and architecture in our documentation.

How to use Kardinal

Getting started with Kardinal is straightforward.

Step 1: Install Kardinal

To install Kardinal, run the following command:

curl get.kardinal.dev -sL | sh

Step 2: Set up a development Kubernetes cluster

All you need is a Kubernetes cluster with Istio enabled, and kubectl installed on your machine, pointing to your cluster. If you need help with this, read more here

Step 3: Deploy the Kardinal Manager to your cluster

Make sure that kubectl is pointing to your cluster, and then run the following command:

kardinal manager deploy kloud-kontrol

Once installed, follow the demo below, or check out our docs on creating your first flow.

Let’s dive into a demo

If you want to see Kardinal in action, here’s how to get started with a demo environment using an online boutique store as an example.

Step 1: Deploy the demo app

In a new terminal window, deploy the demo app via Kardinal:

curl https://raw.githubusercontent.com/kurtosis-tech/new-obd/main/release/obd-kardinal.yaml > ./obd-kardinal.yaml
kardinal deploy -k ./obd-kardinal.yaml

Set up the gateway:

kardinal gateway prod

You can view the frontend of the demo app by going to the local host link that the gateway prints.

Feel free to click around, add items to your cart, and shop!

If you want to see the architecture of your application, the Kardinal dashboard will show you it along with any logical environments (flows) you create on top of it.

To view the dashboard, run:

kardinal dashboard

Click "Traffic configuration" in the sidebar to see the architecture.

Step 2: Create a lightweight development environment

Let's deploy a dev version of the frontend that's adding bolder style to the website, this modification is contained into a single frontend image using an image we've prepared for this demo:

kardinal flow create frontend kurtosistech/frontend:demo-frontend

This command will output a URL that you can use to access the frontend of the development environment. You can view the frontend of the application by going to the URL provided.

To interact with the dev version, first stop your previous gateway (if it's still running)--currently you can only run one gateway at a time in this demo, and run the following to forward the dev demo application port to a URL you can access:

 kardinal gateway <flow-id>

Notice that there are already items in your cart in the development environment. We've configured the development "flow" in this demo to run with it's own database which is seeded with test data.

To inspect the resources in your cluster, and see how Kardinal is reusing resources from your stable environment in the dev environment, go to the dashboard again:

kardinal dashboard

and click on the "Traffic configuration" sidebar item.

Step 3: Clean up your development flow

The flow_id is in the output of the kardinal flow create command, but if you've lost it, you can get it again by running:

kardinal flow ls

When you're done with your development flow, you can delete it by running:

kardinal flow delete <flow_id>

And you're done! 🥳

Helpful resources

If you want to learn more about Kardinal, get additional support, or test it out without installing anything—here are some helpful links:

Explore our docs to learn more about how Kardinal works.
Ask questions and get help in our community forum.
Test Kardinal without installing via our playground.

Introducing Streamlit Components

TC Ricks — Fri, 17 Jul 2020 19:06:15 +0000

This originally appeared on July 14th, 2020 by Adrien Treuille. Click here to see the original content.

In the ten months since Streamlit was released, the community has created over 250,000 apps for everything from analyzing soccer games to measuring organoids, and from COVID-19 tracking to zero-shot topic classification. Inspired by your creativity, we added file uploaders, color pickers, date ranges, and other features. But as the complexity of serving the community grew, we realized that we needed a more scalable way to grow Streamlit's functionality. So we're turning to Streamlit's best source of ideas: you!

Today, we are excited to announce Streamlit Components, the culmination of a multi-month project to enable the Streamlit community to create and share bits of functionality. Starting in Streamlit version 0.63, you can tap into the component ecosystems of React, Vue, and other frameworks. Create new widgets with custom styles and behaviors or add new visualizations and charting types. The possibilities are endless!

The Streamlit Components Gallery

The first thing you should do is check out the Streamlit Components Gallery to see what others have created and shared.

Each component can be installed with just a single line of code:

pip install some_cool_component

If you don't find a component that works for you, you can make your own!

Building your own components

Streamlit has a unique, functional style which lets you create rich, interactive experiences in very few lines of code. For example, let's check out this simple Streamlit app:

import streamlit as st

x = st.slider('x')st.markdown(f'`{x}` squared is `{x * x}`')

Looking at this code, you can see that Streamlit calls come in two flavors: static components like st.markdown are stateless and only send data to the browser, whereas bidirectional components like st.slider have internal state and send data back from the browser.

Our challenge was to provide an API that embraces Streamlit's functional style while capturing these use-cases as simply as possible. A few months ago, two amazing Streamlit engineers, Tim Conkling and Henrikh Kantuni, tackled this challenge and came up with a super elegant solution. The result is the new streamlit.components.v1 package which comprises three functions. For static components, we added:

html(...), which lets you build components out of HTML, Javascript, and CSS
iframe(...) , which lets you embed external websites

For bidirectional components, we added:

declare_component(...), which lets you build interactive widgets which bidirectionally communicate between Streamlit and the browser.

Let's dive into how it works!

Static Components

Let's start with a simple static component to embed snippets of code called Github gists in your app. Ideally, adding the component should just be a single function call:

# Render a gist
github_gist('tc87', '9382eafdb6eebde0bca0c33080d54b58')

which would render a gist like this:

To create such a component, we start by importing the Streamlit Components library:

import streamlit.components.v1 as components

This somewhat wordy import statement serves two purposes:

It versions the components API so that future changes don't break existing components.
It reminds us that we're starting to use deep magic which we should hide from the user.

Now let's use the html(...) method to serve up the gist:

def github_gist(gist_creator, gist_id, height=600, scrolling=True):
    components.html(
        f"""
      <script src="https://gist.github.com/{gist_creator}/{gist_id}.js">
      </script>
    """,
        height=height,
        scrolling=scrolling,
    )

This approach has a few awesome properties. First, it's really simple and functional. In fact, this pattern lets you hide the ugly-looking HTML and wrap it into a pretty, Pythonic function call, github_gist(...). You can wrap code in a function and reuse it throughout your project. (Better yet, put it in a package and share it with the community in the gallery.) Second, note that we can add arbitrary HTML in this component --- divs, spans, and yes, scripts! We can do this safely because the component is sandboxed in an iframe which lets us include external scripts without worrying about security problems.

Getting widget with it!

What if you want to create a stateful bidirectional component that passes information back to Python from the browser, or in other words, a widget? You can do this too! For example, let's create a counter component:

count = counter(name="Fanilo")st.write('The count is', count)

which creates this:

Note that this code follows Streamlit's unique functional style and captures the counter state embedded in the component. How did we achieve this? Happily, a single function call, declare_component, does all the work to enable bidirectional communication with Streamlit.

# Declare a simple counter component.import 
streamlit.components.v1 as componentscounter = 
components.declare_component("counter", path=BUILD_PATH)

Nice! Under the hood, BUILD_PATH points to a component built using React, Vue, or any frontend technology you like. For this example we decided to use React and Typescript giving us this render function:

public render = (): ReactNode => {
  return (
    <span>
      Hello, {this.props.args["name"]}! &nbsp;
      <button onClick={this.onClicked} disabled={this.props.disabled}>
        Click Me!
      </button>
    </span>
  )
}

and this callback:

private onClicked = (): void => {
  this.setState(
    prevState => ({ numClicks: prevState.numClicks + 1 }),
    () => Streamlit.setComponentValue(this.state.numClicks)
  )
}

Donezo! You've now created a simple, stateful component which "feels like React" on the website, and "feels like Streamlit" on the Python side. Information is passed back to Python using Streamlit.setComponentValue(...). Because we're using React in this case, the component's state is stored in this.state. For more details on this example, see our component template.

A neat benefit of this architecture is that you're not limited to React. You can use any language or framework which compiles for the web. Here is the same counter component written in ClojureScript.

(defonce counter (atom 0))

(defn increment-counter []
  (swap! counter inc)
  (send-message-to-streamlit :set-component-value {:value @counter}))

(defn app []
  [:button {:on-click increment-counter} "Click Me!"]

Another cool feature of this API is that you can do hot-reloading as you develop your component like this:

components.declare_component(name, url="http://localhost:3001")

Here, the url parameter lets you specify a dev server for the component created with [npm run start](https://github.com/streamlit/component-template#quickstart).

What we've shown you so far just scratches the surface. For more details, please check our documentation.

Sharing with the world

Did you create something broadly useful for the Streamlit community? Sure, you could keep that superpower for yourself, but it would be even cooler to share it! Get community feedback and praise. 😇 You can easily wrap your component in a PyPi package and submit it to our Gallery by following these instructions.

Try it out and let us know what you think!

We're excited to unlock for the community a new way to plug-and-play functionality into Streamlit. Streamlit Components let you write simple HTML extensions or tap into the whole ecosystem provided by React, Vue, and other frameworks. Your feedback drives innovation in Streamlit. Please tell us what you think and what you'd like next. Show off your shiny new components and share them with the world. We can't wait to see what you build! 🎈

Special thanks to Fanilo Andrianasolo, Daniel Haziza, Synode, and the entire Streamlit Components beta community who helped refine this architecture and inspired us with their feedback and ideas. Thanks also to TC Ricks, Amanda Kelly, Thiago Teixeira, Beverly Treuille, Regan Carey, and Cullan Carey for their input on this article.