DEV Community: AI Data Legends

Netflix Conductor 4.0: Scaling Workflows to 30,000 Tasks with Asynchronous Evaluation

AI Data Legends — Wed, 28 Oct 2026 14:38:35 +0000

A workflow orchestrator can coordinate thousands of operations and still hit a scalability ceiling—not because individual tasks are too expensive, but because the engine spends too much time deciding what should happen next.

That was the problem Netflix faced with Conductor as its internal adoption grew. The platform now supports roughly 200,000 workflow definitions across about 150 applications and executes around 420 million workflows in a typical month, according to the Netflix Technology Blog. Its workloads span content and studio production, advertising, games, and other business systems. Source: Netflix Technology Blog

Netflix's response was not simply to add more compute. It redesigned the evaluation engine and the data model underneath it.

The distinction between releases matters. Conductor 3.0 modernized the data plane; Conductor 4.0 rebuilt the workflow evaluation engine. Netflix says the 4.0 rewrite increased the supported workflow size from approximately 2,500 tasks to 30,000 and reduced p99 workflow-evaluation latency by roughly 40% in production. Those are Netflix-reported results, not universal performance guarantees.

For engineers building data pipelines, feature-engineering systems, media workflows, or multi-step AI applications, the architectural lessons are highly relevant: keep workflow metadata lightweight, avoid loading state that an evaluation does not need, and move contention-prone decision-making out of a synchronous request path.

Why it matters

A workflow engine is responsible for more than executing tasks. Whenever a task completes, fails, or reports progress, the engine must update state, evaluate dependencies, and determine which tasks can run next.

As the graph grows, that coordination can become the bottleneck.

In Netflix's earlier engine, evaluating a workflow meant loading its entire state into memory. Workflow metadata and task payloads also shared a Cassandra partition, creating large rows and increasing the complexity of concurrent updates. Even if only one task changed, the engine could still need to process far more state than that decision required.

The second problem was synchronization. Workflow evaluation ran inline when updates arrived. Concurrent updates to the same workflow competed for a lock, and unsuccessful attempts could be deferred to a repair process. Under peak load, that caused unpredictable tail latency.

These problems have direct parallels in data and AI infrastructure:

A feature-engineering workflow with thousands of dependent steps can spend more time coordinating than executing useful work.
An AI agent that fans out to many tools needs a reliable way to track task status and resume after failures.
A batch orchestration system can become slow when many workers update a shared execution record.
Long-running workflows need recovery mechanisms that reconcile missed or delayed updates without corrupting state.

The important performance metric is therefore not just average task duration. It is how efficiently the orchestrator evaluates state, handles concurrent updates, and schedules the next piece of work when the system is under pressure.

What changed: from Conductor 3.0 to 4.0

Netflix describes the evolution as several related changes rather than one isolated rewrite.

Stage	Main architectural development
Conductor 1.0	Dynomite and DynoQueues supported metadata and queueing, with Elasticsearch for indexing
Conductor 2.x	Cassandra became the primary execution datastore, large task inputs and outputs moved to S3, and Timestone replaced DynoQueues
Conductor 3.0	The modernized data plane was consolidated; Dynomite was retired, while Cassandra, Timestone, a Kafka-fed indexer, Elasticsearch, and Iceberg served different roles
Conductor 4.0	The evaluation engine and task data model were redesigned, and evaluation became asynchronous through Timestone

Conductor 3.0 also introduced automated workflow repair and multi-cluster deployment, allowing workflows to run in isolated contexts. These were important foundations, but the engine still carried an older evaluation design.

The 4.0 work addressed that remaining bottleneck. Netflix's engineering post explains the rewrite in three connected areas: a partitioned task model, safer reconciliation of competing updates, and asynchronous workflow evaluation.

This is an excerpt. The full article continues with:

1. Separate workflow metadata from individual task records
2. Make concurrent task updates safe without relying on conflicting write paths
3. Move workflow evaluation off the synchronous request path
What the results mean
4. Scale task polling as well as workflow evaluation
5. Improve concurrency controls and task execution options
What data and AI engineers should do
Limitations and open questions

Read the full article on AI Data Legends, with diagrams, code and sources.

OpenAI Dots: A Persistent Cloud Agent for 4,000+ Apps

AI Data Legends — Mon, 26 Oct 2026 16:37:02 +0000

OpenAI says Dots are always-on agents running on GPT-6 Astra with a dedicated cloud computer, and that they reach over 4,000 apps through a plugin ecosystem. For engineers, the interesting question is whether a persistent agent that idles and polls thousands of integrations around the clock remains economically viable at Astra's price point, or whether a cheaper successor model shifts the unit economics enough to make always-on workloads practical.

Why it matters

For data and AI engineers, the most immediate consequence is how Dots reshape cost modeling inside a team.

OpenAI confirms that the initial dot is included at no additional charge for Pro and Business Premium subscribers in eligible regions. For engineers assessing the cost of persistent agents on GPT-6 Astra against scheduled pipelines, this complimentary access offers a low-risk opportunity to test the system.

What happened

OpenAI held DevDay 2026 on September 29 at Fort Mason in San Francisco. Simon Willison's live blog also highlights GPT-6.1 Sol, which offers near-Astra performance at one-fifth the price, alongside an Ultrafast tier that runs eight times faster at six times the cost.

On availability, Willison notes Dots went live for Pro and Enterprise subscribers on launch day itself.

How it works

Under the hood, a Dots instance is a long-lived process hosted in OpenAI's cloud rather than a local CLI session. For enterprises, Microsoft Agent 365 supplies the governance and security perimeter around those persistent sessions.

The economics are where the design decisions get interesting. Simon Willison reports that GPT-6.1 Sol delivers near-Astra intelligence at roughly one-fifth the per-token cost, which changes the unit economics of keeping a dot alive for hours. A Pro 500 subscription bundles Ultrafast access with twenty-five times the usage allowance of a Plus plan, effectively pricing sustained agent workloads into a flat monthly fee. The trade-off is straightforward: Astra-backed dots get the strongest reasoning and widest tool coverage, but at a premium that Sol largely erases for non-latency-critical tasks.

OpenAI's real-time stack transmits audio and JSON control events over a single WebRTC channel, while an optional sideband WebSocket handles server-side integrations. Client delegation allows connections to arbitrary backends, though this mode is fixed at session creation and cannot be changed during the conversation. Partners including LiveKit, Twilio, Telnyx, and Daily/Pipecat supply the necessary telephony and media infrastructure for these server-side paths.

This is an excerpt. The full article continues with:

What data and AI engineers should do
Limitations and open questions

Read the full article on AI Data Legends, with diagrams, code and sources.

Grok 4.7 on Amazon Bedrock: Routing, reasoning effort, and cost with OpenAI-compatible APIs

AI Data Legends — Mon, 26 Oct 2026 13:16:29 +0000

xAI's Grok 4.7 is now available on Amazon Bedrock, bringing a 500K-token context window, configurable reasoning, image input, and support for long-running agentic workloads.

At first glance, this looks like another model joining the growing Bedrock catalog.

But there is a more interesting engineering story underneath.

Grok 4.7 gives you several knobs that directly affect how an application behaves in production:

Geo vs Global cross-Region routing
Four reasoning-effort levels
Standard, Priority, and Flex service tiers
OpenAI-compatible Responses and Chat Completions APIs
Bedrock's native Converse API

Those aren't just configuration options. They affect cost, latency, throughput, data residency, and how much reasoning context your agent can carry between turns.

For anyone building agentic pipelines, that's where Grok 4.7 gets interesting.

Why this matters for AI engineers

The interesting change isn't simply that another frontier model is available through Bedrock.

It's that reasoning has become an explicit production-control problem.

AWS exposes four reasoning levels:

low → medium → high → xhigh

And reasoning is enabled by default, with high as the default effort level.

That means a production application shouldn't blindly send every request using the default.

A simple classification request doesn't necessarily need the same amount of reasoning as a multi-step coding agent.

Consider a pipeline like this:

User request
     │
     ▼
Intent classification
     │
     ▼
Planning agent
     │
     ▼
Tool execution
     │
     ▼
Verification
     │
     ▼
Final response

Using xhigh everywhere could be wasteful.

A more deliberate strategy might look like:

Classification       → low
Simple extraction    → low
Normal reasoning     → medium
Planning             → high
Complex agent loop   → xhigh

The exact mapping should be benchmarked against your workload, but the principle is important:

Reasoning effort should become part of your routing policy, not a value you leave at its default.

AWS recommends benchmarking the reasoning levels against your workload to determine where additional reasoning stops paying for itself.

Grok 4.7 is also more expensive when it thinks longer

This becomes particularly important for agentic systems.

Artificial Analysis reports an Intelligence Index of 46 for Grok 4.7 versus 44 for Grok 4.6, while its Coding Agent score increased from 47 to 56.

But there's a catch.

The evaluation also reports approximately 81K output tokens per Intelligence Index task for Grok 4.7 versus roughly 36K for Grok 4.6, with Grok 4.7 measured at xhigh reasoning effort and Grok 4.6 at high.

That is more than twice the output tokens per task.

And this is exactly why reasoning effort matters in production.

A single additional token might not matter much.

An agent making 50 model calls absolutely can.

If every step becomes more expensive, the economics of the entire agent changes.

Geo vs Global: you're choosing a routing policy

One subtle point about Grok 4.7 on Bedrock is that it isn't exposed through a normal in-Region model invocation.

AWS currently exposes two cross-Region inference profiles:

us.xai.grok-4.7
global.xai.grok-4.7

The us profile is the Geo option, while global is the Global option.

The distinction matters.

Geo

Geo routing keeps inference within the supported geographic boundary.

For Grok 4.7, AWS currently provides a US Geo profile.

Its Standard pricing is:

Input: $2.20 / 1M tokens
Output: $6.60 / 1M tokens
Cache read: $0.55 / 1M tokens

Global

Global routing can use supported commercial AWS Regions worldwide.

Its Standard pricing is:

Input: $2.00 / 1M tokens
Output: $6.00 / 1M tokens
Cache read: $0.50 / 1M tokens

So Global is about 9% cheaper than Geo on both input and output under Standard pricing.

The trade-off is control.

With Global routing, you get access to a larger pool of capacity, but you have less control over exactly where an individual request is processed.

That makes the choice less about:

"Which AWS Region should I use?"

and more about:

"What routing and residency policy does this workload require?"

That's a much more useful way to think about it.

This is an excerpt. The full article continues with:

There is another cost knob: Service Tier
OpenAI-compatible APIs make migration easier
Controlling reasoning effort
Responses API vs Chat Completions
Where I would actually use each setting

Read the full article on AI Data Legends, with diagrams, code and sources.

GPT-6.1 Sol on Amazon Bedrock: Near-Astra performance at a fraction of the cost

AI Data Legends — Mon, 26 Oct 2026 13:16:08 +0000

OpenAI has introduced GPT-6.1 Sol, a new model designed to bring much of GPT-6 Astra's capability to workloads where cost matters.

The headline is simple: OpenAI says GPT-6.1 Sol approaches GPT-6 Astra on agentic coding, computer use, and professional work at one-fifth of Astra's standard API input and output token prices.

That alone is interesting.

But the more important question for engineers is what happens when you put that model inside a production agent.

A real agent doesn't make one model call and disappear. It reads files, searches for information, calls tools, writes code, tests the result, recovers from failures, and sometimes repeats the whole process.

Every additional step costs money.

So a model that can deliver near-frontier capability at a much lower token price can change the economics of the entire workflow.

And GPT-6.1 Sol is now generally available through Amazon Bedrock, with AWS identity, networking, auditing, and security controls around the inference layer.

That combination makes Sol much more interesting than just another model release.

Why GPT-6.1 Sol matters for AI agents

Think about a software-engineering agent working on an unfamiliar repository.

The agent may have to:

Understand the repository
        ↓
Find relevant files
        ↓
Trace dependencies
        ↓
Plan the change
        ↓
Write code
        ↓
Run tests
        ↓
Inspect failures
        ↓
Fix the implementation
        ↓
Run tests again

A small difference in model capability or pricing can become a large difference across the full task.

OpenAI reports that GPT-6.1 Sol matches GPT-6 Astra on DeepSWE v1.1, a benchmark for long-horizon software-engineering tasks, while exceeding GPT-6 Sol's best result by 6.4 percentage points at a lower reasoning effort and cost.

The same pattern appears in other evaluations.

On AutomationBench, GPT-6.1 Sol scores 2.2 percentage points above Opus 5.5 at medium reasoning effort, while costing roughly one-third as much on the evaluated task. OpenAI also reports a 4.8-point improvement over GPT-6 Sol at the same reasoning setting.

On OSWorld 2.0, GPT-6.1 Sol improves by seven percentage points over GPT-6 Sol at maximum reasoning effort and comes within 2.1 percentage points of Astra's score, according to OpenAI.

For AI engineers, that creates an interesting question:

How much of your agent's workload really needs your most expensive model?

GPT-6.1 Sol is effectively an answer to that question.

GPT-6.1 Sol pricing: the cost difference is the real story

OpenAI's standard API pricing compares the GPT-6 family like this:

Model	Input / 1M tokens	Cached input / 1M tokens	Output / 1M tokens
GPT-6 Astra	$10.00	$1.00	$50.00
GPT-6.1 Sol	$2.00	$0.10	$10.00
GPT-6 Luna	$0.10	$0.01	$0.50

That makes GPT-6.1 Sol approximately one-fifth the standard input and output token price of GPT-6 Astra.

Cached input is even more interesting.

GPT-6.1 Sol's cached input price is $0.10 per million tokens, which OpenAI describes as 95% below its standard input price and 50% below GPT-6 Sol's cached input price.

For an agent repeatedly working against the same repository, instructions, documents, or context, that difference can become significant.

A ten-token prompt isn't where economics get interesting.

A million-token workload repeated across hundreds of agent interactions is.

Amazon Bedrock changes the deployment story

GPT-6.1 Sol became generally available on Amazon Bedrock on September 29, 2026.

AWS describes the Bedrock deployment as running on an inference engine designed for performance, security, and reliability at scale.

For enterprises already operating inside AWS, the important part isn't just the model.

It's everything around it.

Bedrock lets organizations control model access through AWS IAM, audit invocation activity through CloudTrail, and keep traffic inside their network boundaries using VPC endpoints powered by AWS PrivateLink.

AWS also states that inference runs on hardware-isolated infrastructure with zero-operator access, meaning AWS operators cannot access customer prompts and completions during inference. AWS says inference data isn't used for model training, and using GPT-6.1 Sol doesn't require customers to opt into sharing data with OpenAI.

There is one important nuance.

For automated abuse detection, AWS says classifier-flagged traffic can be retained for up to 30 days and processed programmatically. Customers can request zero data retention through their AWS account team.

For regulated or sensitive workloads, that distinction matters.

This is an excerpt. The full article continues with:

A 1M-token context window changes the type of workload you can build
How GPT-6.1 Sol is exposed through Bedrock
OpenAI-compatible APIs make the integration familiar
Prompt caching: useful, but understand the distinction
Where GPT-6.1 Sol fits in a model-routing layer
The Agent Toolkit for AWS makes the AWS side easier
Getting started with Agent Toolkit for AWS
The biggest limitation: Sol isn't Astra

Read the full article on AI Data Legends, with diagrams, code and sources.

Databricks Lakehouse Platform for Real-Time, Agentic Data and AI Workloads

AI Data Legends — Mon, 26 Oct 2026 13:15:50 +0000

For years, data engineering has often meant connecting systems that were never designed to work together.

ERP data lives in one place. Manufacturing data lives somewhere else. Application events arrive through streaming systems. Operational databases serve applications. Vector databases power AI search. Machine-learning systems maintain their own features and serving layers.

Then someone asks a deceptively simple question:

"Can we connect all of this and let an AI agent actually use it?"

That question is where the modern lakehouse architecture gets interesting.

Databricks is increasingly positioning its Data and AI Platform as a common foundation for analytical, transactional, real-time, and agentic workloads.

The idea isn't that every byte of data has to be physically moved into one database.

Instead, the platform combines data that is copied into the lakehouse with data that can remain in its source system and still be queried through capabilities such as Lakehouse Federation and Open Sharing, while Unity Catalog provides a common governance layer.

On top of that foundation sit Lakeflow for data engineering, low-latency ingestion through technologies such as Zerobus, Lakebase for operational serving, AI Search for retrieval, and agentic capabilities such as Genie and Agent Bricks.

For data and AI engineers, the interesting question isn't whether Databricks has all of these products.

It's whether bringing them together actually changes how we build production AI systems.

I think it does.

The whole platform on one page: ingest and connect, govern and prepare, serve and search, then build agentic applications on the same governed foundation.

Why the lakehouse matters to data engineers

Consider a manufacturing quality investigation.

A quality engineer might need to correlate:

MES records
machine and process-historian data
supplier information
QMS history
8D corrective-action records
logistics data
plant-level operational information

The question might be:

Which supplier lot reached the products affected by this defect, and has the same problem appeared before?

Databricks points out that questions like this cross organizational and system boundaries. Historically, answering them can mean tickets, exports, specialist knowledge, and manual reconciliation.

The architectural alternative is to make the relevant data accessible through a governed platform.

With Lakehouse Federation and zero-copy Open Sharing, data doesn't necessarily need to be copied into another ETL pipeline simply because a new analytical question appeared. Data can remain in its source system while being queried through Databricks, while other data can be mirrored into the lakehouse when that makes more sense.

That changes the role of the data engineer.

Instead of continuously building point-to-point pipelines for every new question, more of the engineering effort can move toward:

Source systems
      │
      ▼
Governed data
      │
      ├── Analytics
      ├── BI
      ├── ML
      ├── AI Search
      └── Agents

The goal isn't "no ETL."

The goal is less unnecessary data movement and fewer disconnected copies created solely to answer another question.

The same fragmentation problem exists in AI systems

AI systems make the problem even harder.

An agent may need access to:

Customer data
     +
Operational data
     +
Product catalog
     +
Documents
     +
Real-time events
     +
Vector embeddings
     +
Business rules
     +
Model predictions

If every AI application builds its own copy of these assets, organizations quickly end up with another version of the data-platform problem.

One team maintains a warehouse.

Another maintains a feature store.

Another maintains a vector database.

Another maintains an operational database.

Another builds an agent-specific retrieval pipeline.

The result can look like:

                    ┌── Data warehouse
                    │
                    ├── Data lake
                    │
Applications ───────┼── Feature store
                    │
                    ├── Vector DB
                    │
                    ├── Operational DB
                    │
                    └── Agent / RAG stack

Every arrow between those systems is another integration to maintain.

Databricks' lakehouse approach is essentially an attempt to reduce that fragmentation.

This is an excerpt. The full article continues with:

What the Databricks platform actually brings together
Real-time data changes the architecture
Two serving paths beat pretending everything is real-time
Graceful degradation matters just as much
Where Databricks AI Search fits
But AI Search isn't magic
The embedding lifecycle matters too
The reference architecture I would take away

Read the full article on AI Data Legends, with diagrams, code and sources.

Google Research Proposes Contextual Integrity Policy Engines and Multi-Agent Benchmarks for Agentic Privacy

AI Data Legends — Mon, 26 Oct 2026 13:15:25 +0000

AI agents are becoming increasingly capable of doing things on our behalf.

They can read documents, access databases, call APIs, use external tools, communicate with other agents, and execute long-running workflows with considerably less human intervention than traditional software.

That capability creates a problem that traditional permission systems weren't designed to solve.

An agent may have permission to access a piece of information, but that doesn't necessarily mean it is appropriate for the agent to use or transmit that information in a particular context.

That distinction is at the center of a recent Google Research workshop report, Open and Emergent Problems in Agentic Privacy and Security: A Contextual Angle, introduced on the Google Research blog on October 5, 2026 by Eugene Bagdasarian and Marco Gruteser.

The report came out of the Google Contextual Agent Privacy and Security (CAPS) Workshop, held in New York in late 2025 with more than 50 academic and industry leaders. It explores how contextual integrity could provide a foundation for privacy and security policies in increasingly autonomous agentic systems. (Google Research)

It also calls for standardized environments for evaluating the safety of multi-agent systems.

The important part isn't that Google has released a production policy engine.

It hasn't.

The interesting part is the architectural direction being proposed:

Instead of asking only whether an agent is technically allowed to access information, ask whether that information flow is appropriate in the current context.

For engineers building systems that allow agents to interact with personal, enterprise, or sensitive data, that is a significant shift in thinking.

The proposal on one page: a contextual policy engine between agents and the data and tools they use, plus open "Agent Gym" environments to test how multiple agents behave together.

Why traditional permissions aren't enough for AI agents

Traditional access control generally answers a relatively straightforward question:

Can this identity access this resource?

For example:

User → Database
       │
       └── SELECT permission? → YES

That works well when the software behavior is relatively predictable.

An autonomous agent is different.

Imagine an agent has legitimate access to a customer's account information.

The agent could potentially:

read the customer's address
send that address to another service
include it in a generated report
share it with another agent
use it to make a decision
store it in a different system

The fact that the agent can access the information doesn't answer whether every one of those actions is appropriate.

The missing dimension is context.

Google Research's report argues that traditional manual permissions and expert-written policies struggle to scale when agents can dynamically discover tools, interact with other agents, and execute long-running tasks.

That leads to a different question:

Is this information flow appropriate here, for this purpose, between these parties, under these conditions?

Contextual integrity gives us a different model

The idea comes from contextual integrity, a privacy theory developed by Helen Nissenbaum around the principle that privacy depends on whether information flows appropriately within a particular social context.

Instead of treating privacy as simply:

allowed / denied

the system considers the circumstances surrounding an information transfer.

The Google Research report applies this idea to agentic systems and discusses three important components:

Actors
   +
Information type
   +
Transmission principle

Actors

Who is involved?

For example:

User
Agent
Organization
Third-party service
Another agent

Information type

What information is being transferred?

For example:

Personal data
Financial data
Enterprise data
Documents
Operational data
Credentials

Transmission principle

Under what conditions is the information being transferred?

For example:

User consent
Specific task
Business purpose
Legal requirement
Security requirement

Together, these provide a much richer policy description than a simple access-control rule.

From access control to contextual policy

Consider an employee asking an AI agent:

"Find the latest status of my company's supplier contracts."

A traditional permission system might determine:

Employee
   ↓
Contract database
   ↓
Access allowed

A contextual policy engine could reason about something more specific:

Actor:
Employee + Agent

Information:
Supplier contracts

Purpose:
Contract-status investigation

Destination:
Internal workspace

Transmission:
Allowed for this task

Now change one variable.

The agent wants to send the same contract information to an external summarization service.

The underlying database permission hasn't changed.

But the context has.

A contextual policy could therefore produce:

Internal retrieval       → ALLOW
External transmission    → BLOCK

That is the conceptual leap.

The policy isn't merely attached to the database.

It evaluates the information flow.

This is an excerpt. The full article continues with:

What Google Research is proposing: a contextual policy engine
The dynamic policy problem
Why this matters for multi-agent systems
The benchmark problem
Agent Gym: evaluating agents in environments
The engineering trade-off
This is where data engineers should pay attention
What AI engineers should do today

Read the full article on AI Data Legends, with diagrams, code and sources.

Databricks Runtime Introduces NEAREST BY: Native SQL Vector Search at Scale

AI Data Legends — Mon, 26 Oct 2026 13:15:03 +0000

Vector search has become a foundational building block for retrieval-augmented generation (RAG), recommendation systems, semantic matching, and AI-powered data applications.

But there is an architectural detail that is easy to overlook: not every vector-search workload is an online serving problem.

Sometimes you need to match millions of query vectors against a large collection of embeddings in a scheduled job. You might be matching transactions to merchants, finding similar products, resolving duplicate records, or generating candidate pairs for a machine-learning pipeline.

In those cases, routing every query through a separate real-time vector-search endpoint can introduce an unnecessary dependency.

Databricks is addressing this problem with NEAREST BY, a native SQL join for batch vector search in Databricks Runtime. The implementation, described in a Databricks engineering article published on October 5, 2026, combines a fused Photon operator, a specialized General Matrix Multiply (GEMM) kernel, and an approximate nearest-neighbor index stored in a liquid-clustered Delta table.

The idea is straightforward: instead of treating every vector lookup as a remote service request, bring the search operation into the analytical query engine.

For data engineers and AI engineers, that could simplify an important class of workloads.

NEAREST BY on one page: query vectors and a Delta vector table meet in a SQL join, Photon's fused GEMM scores them, and APPROX or EXACT decides how the top-k is found.

Why batch vector search matters

Databricks describes a payments company that matches more than 100 million daily transactions against 140 million merchant embeddings for entity resolution.

Or picture a nightly job that compares millions of new product embeddings against an existing catalog.

The conventional architecture might look like this:

Query vectors
     |
     v
Batch orchestration
     |
     v
External vector-search endpoint
     |
     v
Results returned to Spark
     |
     v
Downstream transformations

This can work well, particularly when the application needs low-latency online retrieval. But for large batch workloads, it introduces a separate service, request orchestration, network overhead, and another component to monitor.

With NEAREST BY, the architecture can instead look like this:

Query vectors                 Base vectors
      |                             |
      v                             v
  Query table                Delta vector table
      |                             |
      +-------------+---------------+
                    |
                    v
             NEAREST BY JOIN
                    |
                    v
          Top-k matching vectors
                    |
                    v
          Downstream SQL / Spark

Databricks describes this as bringing batch vector search directly into the lakehouse, with no separate vector-serving system required for the supported query pattern: "Your Lakehouse is also your vector store."

The practical benefit is architectural simplicity: query vectors, indexed vectors, and downstream processing can participate in the same SQL workflow.

That doesn't eliminate every external vector database use case. Interactive applications with strict online latency requirements may still need a dedicated serving architecture, such as the online AI Search path we covered in our Databricks lakehouse breakdown. The important distinction is choosing the right search architecture for the workload.

What is NEAREST BY?

NEAREST BY extends SQL joins with a top-k nearest-neighbor operation.

For each row on the left side of a join, it finds up to k matching rows on the right, ranked by a distance or similarity expression.

The syntax supports two key choices:

EXACT: returns the exact top-k results according to the ranking expression.
APPROX: allows approximate search strategies to return a top-k set that approximates the exact ranking.

The ranking expression can use vector similarity or distance functions, including cosine similarity, L2 distance, and inner product. A query can therefore express nearest-neighbor retrieval as a relational operation rather than manually orchestrating a table-valued function or an external API.

The official NEAREST BY SQL reference documents the join semantics and supported ranking expressions.

A simplified example illustrates the idea:

SELECT
    q.query_id,
    b.item_id,
    vector_cosine_similarity(
        q.embedding,
        b.embedding
    ) AS similarity
FROM query_vectors AS q
JOIN base_vectors AS b
    APPROX NEAREST 10
    BY SIMILARITY vector_cosine_similarity(
        q.embedding,
        b.embedding
    );

This example illustrates the intended query shape; check the current Databricks SQL reference and your runtime version for exact syntax and supported expressions before deploying it.

The important point is that top-k matching becomes part of the SQL plan. The runtime can choose how to distribute the work and execute the ranking, instead of requiring the application to issue one network request per query vector.

This is an excerpt. The full article continues with:

How Databricks makes batch vector search fast
APPROX versus EXACT is an important contract
What Databricks reports about scale
What data and AI engineers should do
Limitations and open questions

Read the full article on AI Data Legends, with diagrams, code and sources.

n8n Bridges Workflow Automation and API Management, With Important Trade-offs

AI Data Legends — Mon, 26 Oct 2026 13:14:34 +0000

Enterprise integration has an awkward middle ground. Traditional integration platforms make it easier to connect applications and move data, while API management platforms focus on securing, governing, and operating APIs throughout their lifecycle. Engineering teams that need both workflow logic and controlled endpoints can end up stitching together multiple systems.

In an engineering article published on October 1, 2026, n8n argues that this gap leaves teams looking for a more flexible approach. The company cites an estimate that the average enterprise operates around 1,000 applications but integrates only 27% of them. Those figures are cited by n8n; treat them as industry context rather than a universal measurement of every organization.

n8n's proposition is straightforward: combine visual workflow orchestration with code-level extensibility, webhook-based endpoints, integrations, and AI capabilities, while allowing teams to self-host.

For data and AI engineers, this creates an interesting option. A single workflow can receive an event, validate data, call several APIs, invoke an AI model, branch on the result, and return a response. But being able to expose an endpoint is not the same as operating a fully governed API platform. Understanding that distinction is the key to using n8n well.

Workflow layer plus API layer: n8n orchestrates the steps; a gateway in front adds the governance an external API needs.

Why the iPaaS and API management gap matters

Integration is not just about moving information between systems. Production workflows must also handle authentication, validation, branching, retries, observability, and failure recovery. When the logic becomes complex, a collection of basic connectors may no longer be enough. When an endpoint becomes business-critical, workflow functionality alone may not provide all the controls expected from an API management platform.

n8n sits between these concerns. It provides a visual environment for connecting systems and managing execution, with a Code node for custom logic and webhook nodes for request-driven workflows. A self-hosted deployment gives an engineering team more control over its runtime and deployment environment.

That combination is particularly relevant for AI agents. An agent often needs to retrieve context from one system, call a model, invoke a tool or downstream API, apply business rules, and route the outcome to another service. n8n can coordinate those steps without requiring every integration to be implemented as a separate custom service.

However, fewer components do not automatically mean fewer risks. If a workflow exposes an endpoint to external callers, the team still needs to think about authentication, authorization, input validation, request limits, secrets, logging, and recovery. The platform can simplify orchestration; it does not remove the need for an appropriate security and operational architecture.

What n8n brings to the table

n8n describes itself as a source-available, AI-native automation platform, and advertises over 500 pre-built integrations. Its documented building blocks include:

Webhook and Respond to Webhook nodes to receive requests and return responses.
HTTP Request node to call external REST APIs.
Code node to implement custom transformations and logic in supported JavaScript or Python configurations.
Credential management to configure supported authentication methods without hardcoding secrets into workflow logic.
Execution history to inspect runs and investigate failures.
Error handling and retries to help workflows recover from transient problems.

These capabilities can cover a surprising amount of integration work. A workflow might receive an order event, validate the payload, retrieve customer information, call a fraud-scoring API, route the result for review, and return an acknowledgement.

The important architectural benefit is that connectivity, branching, and operational execution can live in one orchestrated workflow. The important limitation is that the workflow runtime and an API gateway solve overlapping but different problems.

This is an excerpt. The full article continues with:

How it works: from webhook to orchestrated response
An API-like endpoint is not a complete API management platform
Execution visibility helps, but it is not the whole observability stack
What data and AI engineers should do
Limitations and open questions

Read the full article on AI Data Legends, with diagrams, code and sources.

Meta Open-Sources Stateless NTS: Authenticated Time Without Server-Side Cookie State

AI Data Legends — Mon, 26 Oct 2026 13:13:14 +0000

Time synchronization is easy to overlook in a distributed system, until a clock is wrong.

A data pipeline may process events correctly but order them incorrectly. A distributed database may struggle to reconstruct the sequence of writes. A certificate can appear expired or not yet valid, while a short-lived token can be accepted or rejected at the wrong moment. In AI infrastructure, these problems can affect job coordination, logs, credentials, and the reproducibility of operational workflows.

The Network Time Protocol (NTP) has long provided the mechanism for synchronizing clocks across networks. The security problem is that traditional NTP does not, by itself, authenticate the source of a time response. An attacker positioned on the network path may be able to forge responses and influence a client's clock.

Meta's latest contribution addresses that problem. In an October 6, 2026 engineering post, the company announced that its public time service supports Network Time Security (NTS), specified in RFC 8915, and that its protocol, server, and client implementations are available through the Meta Time libraries on GitHub. The public NTS key-establishment endpoint is nts.meta.com. Source: Meta Engineering

The most interesting engineering detail is not simply that Meta supports authenticated time. It is how the service scales: the servers can validate cookies without maintaining a per-client session table or replicating a cookie-sealing key ring across the fleet.

Why authenticated time matters

A machine clock is an input to much more than timestamps.

TLS certificate validation: Clients compare certificate validity windows with their local clocks.
Credential expiry: Short-lived tokens and credentials rely on time-based validity checks.
Replay protection: Security controls often reject requests that fall outside an allowed time window.
Distributed event processing: Timestamps help systems correlate records and reconstruct timelines.
Observability: Logs from machines with inconsistent clocks can make the order of events difficult to establish.

Traditional NTP provides time synchronization, but an unauthenticated response does not cryptographically prove which server supplied it or whether an intermediary modified it. A malicious response can therefore be more than a logging nuisance: shifting a clock can interfere with controls that depend on expiration or validity windows.

NTS adds cryptographic authentication to the NTP client-server exchange. It helps a client verify that a response belongs to an authenticated exchange and has not been modified in transit. It does not magically guarantee that the authenticated server has the correct time, nor can it prevent every network-based timing attack. Those distinctions matter when designing a time service for production infrastructure. Source: RFC 8915

What Meta released

Meta's announcement brings together three components:

NTS protocol support for authenticated NTP exchanges.
Server implementation to provide the service.
Client implementation in the company's open-source Time libraries.

The repository contains more than NTS. It includes NTP and Precision Time Protocol (PTP) code, hardware-clock-related components, timestamping libraries, and utilities for working with leap-second information. The repository identifies its license as Apache 2.0. Teams should still review the repository's current license and any dependencies before incorporating it into their own products. Source: Meta Time repository

The public service is available at nts.meta.com, but the endpoint has a specific role: it performs NTS Key Establishment (NTS-KE). During that handshake, it advertises the NTP responder the client should use for authenticated time packets. The advertised responder is separate from the key-establishment endpoint.

That separation is important because it allows the handshake service and the time-serving infrastructure to scale independently.

This is an excerpt. The full article continues with:

How NTS works: two phases, two different responsibilities
The key design: stateless cookies with daily key derivation
Why statelessness matters at scale
How to try it with chrony
What data and AI engineers should do
Limitations and open questions

Read the full article on AI Data Legends, with diagrams, code and sources.

Databricks SQL Serverless Cost Governance: Budgets, Alerts, and Query-Level Attribution

AI Data Legends — Mon, 12 Oct 2026 17:58:59 +0000

Serverless compute can simplify warehouse operations, but it does not make cloud spending self-governing. A warehouse can scale to meet demand, a dashboard can trigger a burst of queries, and a poorly timed refresh can consume resources long after its value has diminished.

Databricks is addressing the visibility side of this problem with account-level budgets, alerts, billing system tables, and usage dashboards. Its recommended SQL Serverless warehouse experience is designed to reduce operational overhead through rapid startup and automatic scaling. Yet teams still need to understand what they are spending, which workloads are responsible, and what happens when spending exceeds expectations.

That last point matters: a budget is a monitoring target, not a hard spending limit. Databricks says it plainly: budgets "don't stop usage or prevent charges." An alert can tell your team that costs are rising, but the response still depends on the controls and workflows you put around the platform.

For data and AI engineers, the practical question is not simply how to view a bill. It is how to build a cost-governance system that connects resource usage to ownership, workload behavior, and corrective action.

Visibility is built in; enforcement is yours: budgets, dashboards and tags show where SQL Serverless spend comes from, while guardrails stop it.

Why serverless cost governance matters

SQL warehouses often serve unpredictable, interactive workloads: BI dashboards, ad hoc analysis, embedded analytics, scheduled reporting, and data applications. A busy reporting period or a newly deployed dashboard can create a sharp increase in query volume. A warehouse may be configured correctly and still process far more work than the team anticipated.

Without attribution, the resulting bill becomes a detective exercise. Engineers have to work backward from aggregate usage to identify which warehouse, team, dashboard, or query drove the increase. By then, the workload may have completed and the opportunity to intervene may have passed.

Databricks' cost-management guidance brings several capabilities together:

Attribution: use resource tags and billing metadata to associate usage with teams, projects, and workloads.
Budgets: define spending targets for an account or selected groups of resources.
Alerts: notify owners when spending crosses a threshold or when a custom query detects a concerning trend.
Dashboards: inspect usage patterns and drill down into cost drivers.
Optimization: reduce unnecessary execution, data scanning, and refresh activity.

The first four improve observability. Optimization and operational controls are what turn that visibility into lower costs.

What Databricks is adding to the cost-governance workflow

Databricks' guidance, published on October 7, 2026 by Shweta Verma and Lingeshwaran Kanniappan, describes a more proactive approach to SQL warehouse cost management. Instead of waiting for a monthly invoice, teams can combine budgets, usage records, alerts, and dashboards to identify spending patterns earlier. The post recommends tagging every SQL warehouse with team, cost centre, and use case at creation, monitoring attribution through system.billing.usage to drive unattributed spend to zero, and setting layered budgets: per-team for accountability, account-wide as a safety net. (Databricks cost-governance guidance)

The accompanying usage dashboards provide account- and workspace-level views of usage, including breakdowns by product, SKU, and custom tags. Dashboard version 2.0 is marked Preview in the documentation and adds cost forecasting and object-level cost drill-down. Access requires the relevant Unity Catalog-enabled workspace and SELECT permissions on the billing usage and list-prices system tables. (Databricks usage dashboards documentation)

These capabilities improve the feedback loop, but they do not eliminate the need for engineering controls. A forecast is not a guarantee, a budget alert is not an automatic shutdown, and a dashboard cannot attribute usage reliably when the metadata needed to connect costs to ownership is missing.

This is an excerpt. The full article continues with:

1. Understand what budgets can and cannot do
2. Build attribution from resource tags and billing records
3. Use alerts to detect spending before month-end
4. Read usage dashboards with the right caveats
5. Include costs that do not appear where you expect
6. Turn visibility into engineering action
A practical implementation checklist
Limitations and open questions

Read the full article on AI Data Legends, with diagrams, code and sources.