<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Sangkyoon Nam</title>
    <description>The latest articles on DEV Community by Sangkyoon Nam (@sangkyoonnam).</description>
    <link>https://dev.to/sangkyoonnam</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4216627%2F1eaa7d89-790f-4c23-a383-d4fa444ce5a0.png</url>
      <title>DEV Community: Sangkyoon Nam</title>
      <link>https://dev.to/sangkyoonnam</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/sangkyoonnam"/>
    <language>en</language>
    <item>
      <title>What Changed in Airflow 3.4.0</title>
      <dc:creator>Sangkyoon Nam</dc:creator>
      <pubDate>Sat, 07 Nov 2026 02:16:31 +0000</pubDate>
      <link>https://dev.to/sangkyoonnam/what-changed-in-airflow-340-1i6m</link>
      <guid>https://dev.to/sangkyoonnam/what-changed-in-airflow-340-1i6m</guid>
      <description>&lt;p&gt;Airflow 3.4.0 arrived while I was preparing a talk for Apache Airflow &amp;amp; Apache Magpie Meetup. I've been adding an LLM to a product catalog enrichment pipeline, so the changes to repeated execution and model call records caught my attention first.&lt;/p&gt;

&lt;p&gt;This release changes both how workflows are written and how they are operated. Task Loops can repeat a task group until a condition is met, and complete DAGs can now be written in TypeScript, Java, and Go as well as Python. There is also YAML-based DAG authoring and a folder navigation panel, alongside changes to defaults that affect existing workflows, including Asset event handling. I went through the &lt;a href="https://airflow.apache.org/blog/airflow-3.4.0/" rel="noopener noreferrer"&gt;official announcement&lt;/a&gt; and versioned documentation to put together this overview.&lt;/p&gt;

&lt;h2&gt;
  
  
  The main changes at a glance
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4fre3x9qhkzf7resqnqf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4fre3x9qhkzf7resqnqf.png" alt="Main changes in Airflow 3.4.0" width="800" height="481"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://blog.namsang.co/en/posts/airflow-3-4-0-release/#the-main-changes-at-a-glance" rel="noopener noreferrer"&gt;Text version of this table in the original article&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The AI usage features in the table come from the Common AI provider, which is released separately from Airflow core. &lt;code&gt;@task.llm&lt;/code&gt;, agent tool calls, and human approval were already available; in 3.4, they can be combined with Task Loops to build iterative workflows. The provider's AI Model tab is also available on Airflow 3.4 and later, making model usage easier to inspect in the UI.&lt;/p&gt;

&lt;h2&gt;
  
  
  Managing iteration with Task Loops
&lt;/h2&gt;

&lt;p&gt;Some workflows cannot know their iteration count in advance. An API response may contain the URL of the next page, or validation feedback may require another revision to generated output. &lt;a href="https://airflow.apache.org/docs/apache-airflow/3.4.0/authoring-and-scheduling/loops.html" rel="noopener noreferrer"&gt;Task Loops&lt;/a&gt; let you put that work in a &lt;code&gt;@task_group&lt;/code&gt; and repeat it with &lt;code&gt;.loop()&lt;/code&gt;. A &lt;code&gt;max_iterations&lt;/code&gt; limit is required. An optional &lt;code&gt;until&lt;/code&gt; condition stops the loop as soon as it becomes true, even before that limit is reached.&lt;/p&gt;

&lt;p&gt;The diagram shows a generation and review workflow. Feedback from the first review becomes input to the next generation step, and the second review satisfies the exit condition. Airflow keeps task states and logs for each iteration, so the results can be inspected separately.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7sjspm4kirox28ozxxxh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7sjspm4kirox28ozxxxh.png" alt="Task Loops pass results to the next iteration and retain a separate execution history for each iteration." width="799" height="465"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The loop body runs before the exit condition is checked, so it always runs at least once. From the second iteration onward, tasks can read the previous result through &lt;code&gt;loop.previous&lt;/code&gt;, while the exit condition examines the current result through &lt;code&gt;loop.result&lt;/code&gt;. Airflow creates task instances for another iteration only when the decision requires one. The repetition is managed at runtime without introducing a cycle into the DAG's dependencies.&lt;/p&gt;

&lt;p&gt;Reaching the limit has different consequences depending on how the loop is configured. If an &lt;code&gt;until&lt;/code&gt; condition remains unsatisfied, the final exit-decision task fails. A loop configured only with an iteration count completes normally after that many iterations. Returning a &lt;code&gt;LoopDecision&lt;/code&gt; from the exit condition can also record a reason, which the execution view can show when explaining why the loop continued or stopped.&lt;/p&gt;

&lt;p&gt;Passing feedback into subsequent work serves a different purpose from retries or dynamic task mapping. The choice depends on the work being done.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjcvye78or7kjzyoy1osk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjcvye78or7kjzyoy1osk.png" alt="When to use retries, dynamic task mapping, or Task Loops" width="800" height="188"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://blog.namsang.co/en/posts/airflow-3-4-0-release/#managing-iteration-with-task-loops" rel="noopener noreferrer"&gt;Text version of this table in the original article&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Iterations run sequentially, while independent tasks within an iteration can run in parallel or be dynamically mapped. Nested loops, loops inside mapped task groups, and dynamic mapping of an entire loop are not supported. For example, mapping a separate generation-and-review loop over every product in a list is not currently supported.&lt;/p&gt;

&lt;h2&gt;
  
  
  Writing complete DAGs in TypeScript, Java, and Go
&lt;/h2&gt;

&lt;p&gt;The Java and Go SDKs introduced in Airflow 3.3 let developers implement individual tasks in other languages inside a DAG declared in Python. In 3.4, that scope expands to the whole DAG: the schedule, task definitions, and dependencies can be written in TypeScript, Java, or Go. A separate Python file to declare the DAG is no longer necessary in this setup.&lt;/p&gt;

&lt;p&gt;For a Java pipeline that reads, transforms, and stores data, Java code can now define not only each operation but also its daily schedule and execution order. Task groups and conditional branches can be expressed through the SDK too. This is useful for teams that want to maintain their services and workflows in the same language. The &lt;a href="https://speakerdeck.com/leew/airflow-beyond-python-new-languages-for-tasks-and-workflows" rel="noopener noreferrer"&gt;Airflow Beyond Python talk&lt;/a&gt; also describes teams running DAGs written in different languages within a single Airflow deployment.&lt;/p&gt;

&lt;p&gt;The connection to Airflow is provided by &lt;a href="https://airflow.apache.org/docs/apache-airflow/3.4.0/administration-and-deployment/dag-importers.html" rel="noopener noreferrer"&gt;Dag Importer&lt;/a&gt;, an extension point for reading formats other than Python files. The language SDKs and the YAML importer use this mechanism.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5sxcffu4swia62ipai7a.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5sxcffu4swia62ipai7a.png" alt="Dag Importer reads definitions in different formats into DAGs managed by Airflow." width="799" height="417"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For deployment, the artifacts produced by each SDK go into a DAG bundle. The DAG processor executes them to read the DAG definitions, which the scheduler and UI then manage in the same way as Python DAGs. The artifact and runtime requirements depend on the language.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F34ypmohnlhvrt4hcwg2o.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F34ypmohnlhvrt4hcwg2o.png" alt="Artifacts and runtime requirements for TypeScript, Java, and Go DAGs" width="800" height="188"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://blog.namsang.co/en/posts/airflow-3-4-0-release/#writing-complete-dags-in-typescript-java-and-go" rel="noopener noreferrer"&gt;Text version of this table in the original article&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;After deploying the files, you need to register a coordinator for the language and associate the DAG's task queue with it. This configuration is needed on both the DAG processor and workers. Airflow still manages schedules and state, while the deployment must provide the environment needed to read the code and execute its tasks.&lt;/p&gt;

&lt;p&gt;YAML is another option if you want to declare a DAG in a configuration file. The schedule and dependencies are defined in YAML, with Python operators or code tasks attached to them. The YAML definition itself requires no build step, but its importer must be registered separately. The 3.4.0 release notes mark Dag Importer and the language SDKs as experimental, so API changes remain a consideration when adopting them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Finding DAGs and inspecting their execution
&lt;/h2&gt;

&lt;p&gt;The UI changes make sources and execution flows easier to navigate. The new &lt;strong&gt;Folders panel&lt;/strong&gt; in the DAG list shows the directory structure within bundles. Selecting a folder filters the list to DAGs under that path, letting you narrow down a large collection by its existing structure.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy16l0k807m01na99n6uu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy16l0k807m01na99n6uu.png" alt="The DAG Folders panel in Airflow 3.4.0. Selecting a path in the left-hand tree filters the DAG list." width="800" height="463"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Screenshot: &lt;a href="https://airflow.apache.org/blog/airflow-3.4.0/" rel="noopener noreferrer"&gt;Apache Airflow 3.4.0 announcement&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Inside a DAG, the code view can show source in its authoring language. Task Loops also gain views for individual iterations and task history, making it easier to locate a failure in a particular iteration. If the exit condition records a reason for its decision, that reason is visible too.&lt;/p&gt;

&lt;p&gt;The Common AI provider's LLM and agent tasks offer an &lt;a href="https://airflow.apache.org/docs/apache-airflow-providers-common-ai/1.1.0/observability.html" rel="noopener noreferrer"&gt;AI Model tab&lt;/a&gt; on Airflow 3.4 and later. It shows the model name, token usage, and estimated cost. These values come from the most recent attempt's usage stored in XCom; they are not a total across all retries and iterations. The provider's OpenTelemetry integration can trace individual model calls and tool executions. That instrumentation is experimental.&lt;/p&gt;

&lt;p&gt;There is also a new &lt;code&gt;draining&lt;/code&gt; state for pausing a DAG after its current work finishes. It prevents new scheduled and Asset-triggered runs while letting queued and running DAG runs continue. Once the remaining runs finish, the DAG becomes &lt;code&gt;paused&lt;/code&gt; automatically. Explicitly requested manual runs are still allowed while it is draining.&lt;/p&gt;

&lt;h2&gt;
  
  
  Asset handling and upgrade implications
&lt;/h2&gt;

&lt;p&gt;For an existing deployment, &lt;strong&gt;the unit of Asset event processing&lt;/strong&gt; deserves an early check. An Asset represents a data resource produced or awaited by a DAG, and updates to that resource can trigger another DAG. In 3.4, the default for a single Asset or an OR expression changes from consuming accumulated events together to creating a DAG run for each event. Three pending events for one Asset therefore produce three runs by default.&lt;/p&gt;

&lt;p&gt;To keep the previous batching behavior, set &lt;code&gt;batch_asset_events=True&lt;/code&gt; in the timetable or scheduler configuration and reparse the DAG. AND expressions that wait for multiple Assets, among other cases, retain batching by default, so the impact depends on the schedule expression in use.&lt;/p&gt;

&lt;p&gt;Runtime and storage changes also need to be checked against the deployment's configuration. These are the main compatibility changes I picked out from the &lt;a href="https://airflow.apache.org/docs/apache-airflow/3.4.0/release_notes.html" rel="noopener noreferrer"&gt;3.4.0 release notes&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5g8zk2w9gxb0yiq2s45h.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5g8zk2w9gxb0yiq2s45h.png" alt="Airflow 3.4.0 upgrade compatibility checks" width="800" height="506"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://blog.namsang.co/en/posts/airflow-3-4-0-release/#asset-handling-and-upgrade-implications" rel="noopener noreferrer"&gt;Text version of this table in the original article&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The release gives me a few more things to investigate in the product catalog enrichment experiment. I plan to upgrade the environment to 3.4.0, check the behavior in detail, and test the changes that affect the pipeline for my talk. In particular, I want to express the handoff of validation feedback to the next iteration with Task Loops and compare its execution records with the current implementation.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Astronomer is also preparing a &lt;a href="https://www.astronomer.io/events/webinars/whats-new-in-apache-airflow-3-4-video/" rel="noopener noreferrer"&gt;What's New in Apache Airflow 3.4&lt;/a&gt; webinar for December 10. I plan to cover my hands-on results and what I learn from the webinar in a separate post.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://airflow.apache.org/blog/airflow-3.4.0/" rel="noopener noreferrer"&gt;Apache Airflow 3.4.0 announcement&lt;/a&gt;, Rahul Vats, November 6, 2026. Introduces Task Loops, multi-language DAGs, and UI changes with examples.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://airflow.apache.org/docs/apache-airflow/3.4.0/release_notes.html" rel="noopener noreferrer"&gt;Airflow 3.4.0 release notes&lt;/a&gt;. Documents feature support and compatibility changes affecting existing workflows.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://speakerdeck.com/leew/airflow-beyond-python-new-languages-for-tasks-and-workflows" rel="noopener noreferrer"&gt;Airflow Beyond Python&lt;/a&gt;, Jason Liu and Wei Lee, October 11, 2026. A talk on DAG authoring and deployment in multiple languages ahead of 3.4.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://airflow.apache.org/docs/apache-airflow-providers-common-ai/1.1.0/observability.html" rel="noopener noreferrer"&gt;Common AI provider observability&lt;/a&gt;. Explains usage values in the AI Model tab and the scope of OpenTelemetry tracing.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://airflow.apache.org/docs/apache-airflow-providers-common-io/1.11.0/changelog.html" rel="noopener noreferrer"&gt;Common IO 1.11.0 changelog&lt;/a&gt;. Describes the object-storage XCom backend changes for Task Loops.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;I write about building and leading large-scale platforms, from ads and CRM to AI agents, at &lt;a href="https://blog.namsang.co/en/?utm_source=devto&amp;amp;utm_medium=crosspost" rel="noopener noreferrer"&gt;Namsang Labs Tech Blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>airflow</category>
      <category>agents</category>
      <category>workflow</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Streaming Agents with Flink Agents 0.4.0</title>
      <dc:creator>Sangkyoon Nam</dc:creator>
      <pubDate>Sun, 01 Nov 2026 17:35:22 +0000</pubDate>
      <link>https://dev.to/sangkyoonnam/streaming-agents-with-flink-agents-040-12dj</link>
      <guid>https://dev.to/sangkyoonnam/streaming-agents-with-flink-agents-040-12dj</guid>
      <description>&lt;p&gt;Apache Flink Agents 0.4.0 was released on October 30. It's a preview release that refines the features and APIs ahead of a production-ready 1.0. I contributed to 0.4.0. The release prompted me to write about what Flink Agents does and how it runs and recovers agents.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why run agents on Flink
&lt;/h2&gt;

&lt;p&gt;When a payment fails or a sensor reports an unusual reading, the system needs to determine what happened and take the next step. It can't wait for someone to ask in a chat window. &lt;a href="https://flink.apache.org/2025/10/15/apache-flink-agents-0.1.0-release-announcement/" rel="noopener noreferrer"&gt;Flink Agents started as a project for this kind of work&lt;/a&gt;: agents process a continuous stream of events, make decisions, and call tools to act on them.&lt;/p&gt;

&lt;p&gt;As event volume grows, the work has to be distributed across servers, with state kept for each customer or device. A system that runs continuously also needs to resume after a server fails or execution stops. Code that calls a model doesn't provide that runtime on its own.&lt;/p&gt;

&lt;p&gt;Flink already distributes stream processing, manages state by key, and recovers from checkpoints. Flink Agents is an Apache Flink sub-project that adds an execution model for agents using models, tools, and memory. Event ingestion, agent decisions, and downstream processing all connect within one Flink job.&lt;/p&gt;

&lt;h2&gt;
  
  
  From 0.1.0 to 0.4.0
&lt;/h2&gt;

&lt;p&gt;The first preview, &lt;a href="https://flink.apache.org/2025/10/15/apache-flink-agents-0.1.0-release-announcement/" rel="noopener noreferrer"&gt;0.1.0&lt;/a&gt;, was released in October 2025. It established the execution model: divide work into actions, connect them through events, and call models and tools from within Flink. An event log helped developers follow execution, and saved action records could be reused during recovery. The release ran on Flink 1.20.3 and provided both Python and Java APIs, with more features implemented in Python.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://flink.apache.org/2026/02/06/apache-flink-agents-0.2.0-release-announcement/" rel="noopener noreferrer"&gt;0.2.0&lt;/a&gt;, released in February 2026, filled gaps in the Java API and expanded Flink support from 1.20 through 2.2. Agents could also use resources implemented in the other language. The memory system grew to distinguish data needed within one run from data kept for later runs. Recovery could now reuse saved results of individual calls inside an unfinished action, as well as records of completed actions.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://flink.apache.org/2026/06/19/apache-flink-agents-0.3.0-release-announcement/" rel="noopener noreferrer"&gt;0.3.0&lt;/a&gt; followed in June 2026. It added YAML declarations for models and tools, along with support for running actions written in the other language. Long-term memory moved to Mem0 for extracting and organizing information, and Agent Skills provided task instructions and reference material. Apache Fluss became an option for storing execution records, and event logs became available in the web UI. The update also added reconcilers to check external calls whose outcomes weren't known during recovery. Earlier versions could reuse saved call results; now an application could also check the external system's state when execution had stopped without saving a result.&lt;/p&gt;

&lt;p&gt;The newly released &lt;a href="https://flink.apache.org/2026/10/30/apache-flink-agents-0.4.0-release-announcement/" rel="noopener noreferrer"&gt;0.4.0&lt;/a&gt; builds on that foundation with sub-agents and model routing. The new &lt;code&gt;ctx.chat&lt;/code&gt; API lets you write the logic after a model call in the same action, while tracing now covers model and tool calls as well as events. Existing APIs and default behavior also changed ahead of the API compatibility commitment planned for 1.0.&lt;/p&gt;

&lt;h2&gt;
  
  
  How an agent runs
&lt;/h2&gt;

&lt;p&gt;An agent runs as an operator inside a Flink job. It receives data from upstream, performs its work, and passes the result to the next operator. Within the agent, events connect the steps. The &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/development/create-agent/overview/" rel="noopener noreferrer"&gt;building blocks&lt;/a&gt; are events, actions, and resources. An event triggers matching actions, which do their work and can send more events. One event can trigger several actions, and the events they send can lead to branches or another iteration of a loop. Shared components such as models, tools, and prompts are registered as resources and referenced by name. You can also connect tools and prompts from MCP servers. Since 0.3.0, resources can be declared in YAML.&lt;/p&gt;

&lt;p&gt;For an agent that summarizes product reviews, an input record arrives as an &lt;code&gt;InputEvent&lt;/code&gt;. An action responds by sending a &lt;code&gt;ChatRequestEvent&lt;/code&gt; to request a summary. The built-in chat action calls the model and sends a &lt;code&gt;ChatResponseEvent&lt;/code&gt;, then another action puts the summary in an &lt;code&gt;OutputEvent&lt;/code&gt; for the next Flink operator. The events connect the steps, so their records let you follow an input through the decisions and outputs it produced.&lt;/p&gt;

&lt;p&gt;You can define the workflow in code or let a model choose what to do next. With the built-in &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/development/react_agent/" rel="noopener noreferrer"&gt;&lt;code&gt;ReActAgent&lt;/code&gt;&lt;/a&gt;, you supply a model and can add tools and a prompt. The model chooses a tool, examines its result, and decides the next step toward its goal. You can also combine the two approaches, defining the overall workflow in code and delegating an investigation to a ReActAgent.&lt;/p&gt;

&lt;p&gt;Once the agent is defined, attach it to a Flink job. This excerpt from the Python example in the 0.4.0 docs shows that part.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;env&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;StreamExecutionEnvironment&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_execution_environment&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;agents_env&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;AgentsExecutionEnvironment&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_execution_environment&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;output_stream&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;agents_env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_datastream&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;input_stream&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key_selector&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;apply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;your_agent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;to_datastream&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Input is partitioned by key, and each record from upstream starts a new execution, which the docs call an agent run. If &lt;code&gt;x.id&lt;/code&gt; in this example is a customer ID, runs for the same customer can share memory. An explicit key selector takes precedence; if you omit it, the input &lt;code&gt;DataStream&lt;/code&gt; must already be keyed. Without either, the call fails. &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/development/integrate_with_flink/" rel="noopener noreferrer"&gt;&lt;code&gt;Table&lt;/code&gt;&lt;/a&gt; input is also supported, but it must be insert-only. Aggregations that update or delete earlier rows can't be connected directly.&lt;/p&gt;

&lt;p&gt;Memory types differ in how long they last and how you retrieve their contents. &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/development/memory/overview/" rel="noopener noreferrer"&gt;Sensory and short-term memory&lt;/a&gt; both live in Flink state. Sensory memory is cleared when a run ends; short-term memory remains available to later runs with the same key. For a customer-specific agent, temporary values needed to handle the current request belong in sensory memory, while values needed for later requests belong in short-term memory.&lt;/p&gt;

&lt;p&gt;Long-term memory organizes information collected across runs and retrieves it through semantic search. Since 0.3.0, &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/development/memory/long_term_memory/" rel="noopener noreferrer"&gt;Mem0&lt;/a&gt; handles extraction and consolidation, while an external vector store holds the data. You need to configure a chat model, an embedding model, and a vector store. Searches are scoped to the same job and input key by default.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/development/calling-llms/skills/" rel="noopener noreferrer"&gt;Agent Skills&lt;/a&gt; provide task instructions for the model. A skill bundles instructions in &lt;code&gt;SKILL.md&lt;/code&gt; with related scripts and reference material, which the agent can read as needed.&lt;/p&gt;

&lt;h2&gt;
  
  
  How recovery works
&lt;/h2&gt;

&lt;p&gt;An agent may call models and external tools several times before finishing a task. Along the way, calls can time out or hit rate limits, and the process running the agent can stop. Retry policies handle call failures; recovering from an interrupted execution also requires a way to continue the remaining work without repeating what has already finished. Starting over may produce a different result. A model can answer the same input differently, and values queried from an external system can change between attempts. Repeating an email or payment request can also duplicate an external operation. Recovery needs to distinguish completed work from work that must run again.&lt;/p&gt;

&lt;p&gt;After a failure, Flink restores state and input positions from the last checkpoint, then reprocesses subsequent input. Transactional or idempotent sinks can keep those replayed outputs from being applied twice to an external store. But while reprocessing input, an agent may call models and tools again, and a checkpoint can't undo an external operation that has already happened.&lt;/p&gt;

&lt;p&gt;To reuse completed action records during recovery, configure an Action State Store alongside Flink checkpointing. It stores action execution records in Kafka or Apache Fluss; reusing completed action records has been supported since 0.1.0. In the recovery flow described in the &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/operations/consistency/" rel="noopener noreferrer"&gt;0.4.0 consistency docs&lt;/a&gt;, an action with a saved completion skips its body, and the framework replays its saved sensory and short-term memory updates and output events. It does not reproduce the order in which asynchronous actions read and wrote shared memory, though. If the result depends on that order, you have to make it explicit with event dependencies.&lt;/p&gt;

&lt;p&gt;0.2.0 narrowed what gets skipped on recovery down to individual calls inside an action. Even if the action didn't finish, recovery skips any durable call that completed and had its result saved. The &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/development/create-agent/actions/#durable-execution" rel="noopener noreferrer"&gt;action docs&lt;/a&gt; use a profile lookup as the example.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ProfileAgent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Agent&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nd"&gt;@action&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EventType&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;InputEvent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nd"&gt;@staticmethod&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;process_input&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;RunnerContext&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;user_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;InputEvent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_event&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nb"&gt;input&lt;/span&gt;
        &lt;span class="n"&gt;profile&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;durable_execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;profile_service&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get_profile&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;durable_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;profile:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;user_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send_event&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;OutputEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;profile&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the action fails after the lookup finished and its result was saved, the recovered action reads the saved profile instead of calling the service again. As the docs note, code outside the durable call can still run again. Completions and call results are written to the external store synchronously, and you have to retain both the checkpoints you may restore and the records that recovery needs. For recovery to match those records to the same work, the per-key input order, key types, serializer configuration, and action names have to stay the same. Durable calls also need the same inputs, the same order, and stable identifiers, and the saved values must still be readable at restore time.&lt;/p&gt;

&lt;p&gt;If execution stops after a call starts but before its result is saved, the record alone can't tell you whether the call succeeded. If the external service handles repeated requests without duplicating the operation, you can retry. Otherwise, you need to check whether the request was processed. The &lt;a href="https://zhuanlan.zhihu.com/p/2029555366648653207" rel="noopener noreferrer"&gt;0.3 roadmap walkthrough&lt;/a&gt; raised this problem and proposed a function through which application code could check the external state.&lt;/p&gt;

&lt;p&gt;Version 0.3.0 added a reconciler for that check. When recovery finds an in-flight call record with no saved result, it invokes the registered reconciler. You implement the function to check the external system's state and return the original result or report a failure. That outcome is recorded as well. Without a reconciler, the original call runs again.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcmlr3rzlwb36ukpnfu5o.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcmlr3rzlwb36ukpnfu5o.png" alt="What recovery re-runs depends on what the store held at the time of failure. A saved completion is replayed, a saved call result is reused, and an in-flight call is checked through a reconciler or run again" width="800" height="480"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The Action State Store can't commit an external operation and its saved result in one transaction. Design any call that changes external state on the assumption that it may repeat. Use the external service's idempotency support, or give the durable call a reconciler that can find the original result.&lt;/p&gt;

&lt;h2&gt;
  
  
  New features
&lt;/h2&gt;

&lt;p&gt;This release adds ways to delegate work to other agents and select a model for each request. You can write a model call and its response handling in one action, and inspect records of model and tool calls during operation.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj5awhcrzz79jp7ti55py.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj5awhcrzz79jp7ti55py.png" alt="Flink Agents 0.4.0 features: sub-agents, decision models, model routing, multimodal input, structured output, ctx.chat, conditional triggers, tool parameter injection, trace logs, and metrics" width="800" height="971"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://blog.namsang.co/en/posts/flink-agents-0-4-0/#new-features" rel="noopener noreferrer"&gt;Read the table as text in the original article&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;On September 15, 2026, after the 0.3.0 release, TypeSafe introduced &lt;a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev" rel="noopener noreferrer"&gt;Jev&lt;/a&gt;. It evaluates the supplied context against defined criteria and returns probabilities, a choice, or a score. Flink Agents adds a &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/development/decision_models/" rel="noopener noreferrer"&gt;decision model API&lt;/a&gt; in this version, connecting Jev, OpenAI, and Alibaba Cloud Model Studio through the same interface. For example, you can decide whether a customer inquiry belongs with billing or technical support, or assess an alert's risk against a defined scale. The application supplies the facts and criteria and decides what to do with the result.&lt;/p&gt;

&lt;p&gt;Developer tooling and model integrations were expanded too. Java asynchronous execution now supports JDK 11 and 17 alongside JDK 21. New chat model integrations are Gemini for Java and vLLM and IBM watsonx.ai for Java and Python, and Java also gets OpenAI embeddings. The new &lt;code&gt;flink-agents-dev&lt;/code&gt; skill provides guidance for coding agents building and validating Flink Agents applications.&lt;/p&gt;

&lt;p&gt;Consider an agent that handles only severe alerts. A conditional trigger runs the action only for high-severity alerts. Inside that action, &lt;code&gt;ctx.chat&lt;/code&gt; calls the model and lets the code handle its response directly. If the model asks for a registered tool, the framework runs it.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/development/calling-llms/chat/" rel="noopener noreferrer"&gt;event-based chat API&lt;/a&gt; uses separate actions to send the request and handle the response. It remains supported in 0.4.0. Use &lt;code&gt;ctx.chat&lt;/code&gt; when you want to write the logic after the model call in the same action. With durable execution enabled, recorded &lt;code&gt;ctx.chat&lt;/code&gt; responses can also be reused on recovery.&lt;/p&gt;

&lt;p&gt;Further investigation can be delegated to a sub-agent, and those calls support durable execution too. To score risk against fixed criteria, use a Decision Model. Model Routing lets you select a model for each request, sending simple requests to a lightweight model and reserving a larger one for complex tasks.&lt;/p&gt;

&lt;p&gt;In operation, Trace Log lets you follow the steps and calls that turned an input into a result. When the built-in metrics show a failure or a latency spike, the log can help narrow down the step where it went wrong. &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/operations/monitoring/" rel="noopener noreferrer"&gt;By default it records only events&lt;/a&gt;, so seeing steps and calls means setting &lt;code&gt;trace-log.targets&lt;/code&gt;. A tool that rebuilds the causal tree of events also ships. It reads only event records and skips the step and call records. Event processing and the job continue even if a Trace Log write fails. The trace alone can't prove that every execution was recorded.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to upgrade to 0.4.0
&lt;/h2&gt;

&lt;p&gt;The 0.x releases are still previews, so APIs and configuration can change incompatibly. This release addresses issues found through feedback from early production users and an API review, bringing more breaking changes than any earlier release. The plan is to stabilize 0.4.0, make small adjustments for 1.0, and commit to API compatibility from that point. The detailed compatibility policy will be announced separately. If you're upgrading from 0.3.x to 0.4.0, check these changes first.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Action declarations: Java &lt;code&gt;@Action(listenEventTypes = ...)&lt;/code&gt; becomes &lt;code&gt;@Action(...)&lt;/code&gt;, and Python &lt;code&gt;add_action(events=...)&lt;/code&gt; and YAML &lt;code&gt;listen_to&lt;/code&gt; become &lt;code&gt;trigger_conditions&lt;/code&gt;. The common Python &lt;code&gt;@action("event_type")&lt;/code&gt; form still works.&lt;/li&gt;
&lt;li&gt;Python execution: the path that ran without Flink through &lt;code&gt;from_list()&lt;/code&gt; and &lt;code&gt;to_list()&lt;/code&gt; is gone. Connect to a Flink &lt;code&gt;StreamExecutionEnvironment&lt;/code&gt; with DataStream or Table inputs and outputs, and use a local Flink runtime for local development.&lt;/li&gt;
&lt;li&gt;Retry defaults: the framework's &lt;code&gt;max-retries&lt;/code&gt; default drops from 3 to 0, and &lt;code&gt;error-handling-strategy&lt;/code&gt; is removed. Handle failed chat responses in application code. Provider SDK retries are configured separately.&lt;/li&gt;
&lt;li&gt;Concurrency defaults: under the default async settings, independent tool calls may run concurrently. Split order-dependent tools into separate steps or limit parallelism. For plans with only Java actions on JDK 11 through 20, experimental parallel action execution is on by default and can be turned off with &lt;code&gt;parallel-execution.enabled&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Tool argument validation: unknown fields, wrong types, and invalid values fail before execution, and numeric strings are no longer converted to numbers.&lt;/li&gt;
&lt;li&gt;Logging configuration: the event log is replaced by Trace Log, so old keys such as &lt;code&gt;event-log.*&lt;/code&gt; are rejected. Move to the &lt;code&gt;trace-log.*&lt;/code&gt; keys.&lt;/li&gt;
&lt;li&gt;Execution records: Action State Store data written by 0.3.x is not compatible with 0.4.0, so plan the upgrade without reusing old execution records. Whether Flink checkpoints and savepoints can be restored is a separate compatibility check.&lt;/li&gt;
&lt;li&gt;Pending actions: operator snapshots that contain a pending action may fail to restore when upgrading to 0.4.0. Earlier releases had the same limitation, so test a restore with representative state first.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Message formats changed too. &lt;code&gt;ChatMessage&lt;/code&gt; holds content in &lt;code&gt;blocks&lt;/code&gt; and extra attributes in &lt;code&gt;metadata&lt;/code&gt;. Model calls return a &lt;code&gt;ChatResult&lt;/code&gt; that separates the response message from model information and usage, and tool results are unified as &lt;code&gt;ToolResponse&lt;/code&gt;. If you wrote your own model integration or store messages as JSON, start there.&lt;/p&gt;

&lt;h2&gt;
  
  
  0.4.0 from a contributor's seat
&lt;/h2&gt;

&lt;p&gt;I started contributing in this release and am one of the 38 people listed in its announcement. Five of my 23 PRs fixed the durable execution replay path described above. Replaying a recorded &lt;code&gt;null&lt;/code&gt; result threw an exception &lt;a href="https://github.com/apache/flink-agents/pull/1171" rel="noopener noreferrer"&gt;#1171&lt;/a&gt;, and a Python exception that could be pickled but not rebuilt replayed as an unrelated &lt;code&gt;TypeError&lt;/code&gt; &lt;a href="https://github.com/apache/flink-agents/pull/1182" rel="noopener noreferrer"&gt;#1182&lt;/a&gt;. A Python durable call that records only its completion, with no reconciler, didn't tell the caller when the Java side failed to save the result &lt;a href="https://github.com/apache/flink-agents/pull/1211" rel="noopener noreferrer"&gt;#1211&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;When ordinary and durable calls were awaited together asynchronously, canceling an ordinary call could prevent results from being saved for durable calls that had already finished and came before it in input order &lt;a href="https://github.com/apache/flink-agents/pull/1239" rel="noopener noreferrer"&gt;#1239&lt;/a&gt;. I also fixed a built-in Python chat call identifier that could differ from process to process &lt;a href="https://github.com/apache/flink-agents/pull/1240" rel="noopener noreferrer"&gt;#1240&lt;/a&gt;. The rest of my PRs covered model routing validation, memory references, runtime queues, metrics, and docs.&lt;/p&gt;

&lt;p&gt;Just before the release, I verified the first release candidate and voted. I wrote about that in &lt;a href="https://blog.namsang.co/en/posts/open-source-has-votes/" rel="noopener noreferrer"&gt;There Are Votes in Open Source Too&lt;/a&gt;. During verification, I found a problem restoring existing snapshots after a version upgrade, and the official announcement included it in the upgrade notes. I then took on &lt;a href="https://github.com/apache/flink-agents/issues/1332" rel="noopener noreferrer"&gt;building test infrastructure for state restoration across versions&lt;/a&gt;. I hope 1.0 brings a stable API and stronger upgrade testing, making agents in production easier to maintain.&lt;/p&gt;

&lt;h2&gt;
  
  
  Getting started
&lt;/h2&gt;

&lt;p&gt;Check the supported Flink, Java, and Python versions in the &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/get-started/setup/" rel="noopener noreferrer"&gt;setup guide&lt;/a&gt;, then run the &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/get-started/quickstart/create-agent/single-agent/" rel="noopener noreferrer"&gt;first agent example&lt;/a&gt;. If you want to start by letting a model choose tools, use the &lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/docs/get-started/quickstart/react_agent/" rel="noopener noreferrer"&gt;ReActAgent quickstart&lt;/a&gt;. If you develop with a coding agent, you can add the &lt;code&gt;flink-agents-dev&lt;/code&gt; skill. It gives the coding agent API guidance and examples; install Flink and the Flink Agents runtime separately by following the setup guide above.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx skills add &lt;span class="s1"&gt;'apache/flink-agents#release-0.4@flink-agents-dev'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Bugs and feature requests go to &lt;a href="https://github.com/apache/flink-agents/issues" rel="noopener noreferrer"&gt;GitHub Issues&lt;/a&gt;. Use cases and ideas are discussed in &lt;a href="https://github.com/apache/flink-agents/discussions" rel="noopener noreferrer"&gt;GitHub Discussions&lt;/a&gt; and the &lt;code&gt;#flink-agents-user&lt;/code&gt; channel on the Apache Flink Slack.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://flink.apache.org/2026/10/30/apache-flink-agents-0.4.0-release-announcement/" rel="noopener noreferrer"&gt;Apache Flink Agents 0.4.0 Release Announcement&lt;/a&gt;, Wenjin Xie, 2026: the new features in 0.4.0, the breaking changes, and the expectation that 0.4.0 will be the last release before 1.0.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://flink.apache.org/2026/06/19/apache-flink-agents-0.3.0-release-announcement/" rel="noopener noreferrer"&gt;Apache Flink Agents 0.3.0 Release Announcement&lt;/a&gt;, 2026: why agents run on Flink, and the API compatibility plan through 0.4.0 and 1.0.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://flink.apache.org/2026/02/06/apache-flink-agents-0.2.0-release-announcement/" rel="noopener noreferrer"&gt;Apache Flink Agents 0.2.0 Release Announcement&lt;/a&gt;, 2026: the three kinds of memory, call-level durable execution, and support for multiple Flink versions.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://flink.apache.org/2025/10/15/apache-flink-agents-0.1.0-release-announcement/" rel="noopener noreferrer"&gt;Apache Flink Agents 0.1.0 Release Announcement&lt;/a&gt;, 2025: why the project exists and the core features of the first preview.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://zhuanlan.zhihu.com/p/2029555366648653207" rel="noopener noreferrer"&gt;Flink Agents 0.3 Roadmap解读&lt;/a&gt;, Flink Chinese Community, 2026: a contributor's walkthrough of the 0.3.0 roadmap, including the remaining gap in recovery and a possible fix. The article in that community has not yet been updated to cover 0.4.0.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://nightlies.apache.org/flink/flink-agents-docs-release-0.4/" rel="noopener noreferrer"&gt;Flink Agents 0.4.0 documentation&lt;/a&gt;: getting started, the development guide, and an operations section on consistency and the recovery flow.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;I write about building and leading large-scale platforms, from ads and CRM to AI agents, at &lt;a href="https://blog.namsang.co/en/?utm_source=devto&amp;amp;utm_medium=crosspost" rel="noopener noreferrer"&gt;Namsang Labs Tech Blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>flink</category>
      <category>flinkagents</category>
      <category>ai</category>
      <category>opensource</category>
    </item>
    <item>
      <title>There Are Votes in Open Source Too</title>
      <dc:creator>Sangkyoon Nam</dc:creator>
      <pubDate>Thu, 29 Oct 2026 15:05:00 +0000</pubDate>
      <link>https://dev.to/sangkyoonnam/there-are-votes-in-open-source-too-10e9</link>
      <guid>https://dev.to/sangkyoonnam/there-are-votes-in-open-source-too-10e9</guid>
      <description>&lt;p&gt;I started taking part in open source seriously in the second half of this year. It hasn’t been long, but I knew which areas I cared about, so I read a lot of issues and sent a fair number of PRs. Then in October I got an email inviting me to vote in the OpenTelemetry Governance Committee election. That was when I learned that an open source project, which is neither a company nor a country, holds elections, and that I had a vote even though I had only just started, because I had kept showing up.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkmusknmbhxl8nq4fry3a.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkmusknmbhxl8nq4fry3a.png" alt="An email notification of a comment on the election issue in the OpenTelemetry community repository. Below a sentence inviting everyone with more than 20 contributions on GitHub in the past year, my username sits among other contributors" width="800" height="278"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The invitation I got on October 22. It calls on people who made more than 20 contributions on GitHub in the past year, counting comments, code reviews and PRs.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Once I looked, open source turned out to have far more votes than I expected, and in that one month I encountered three kinds: a vote on a new committer, a vote on whether to ship a release, and an election for the people who run the project. I only saw the committer vote through its result announcement. I cast a ballot in the other two, and the Governance Committee election was the first time I had voted in one.&lt;/p&gt;

&lt;h2&gt;
  
  
  #The vote that makes a committer
&lt;/h2&gt;

&lt;p&gt;In Apache Software Foundation projects such as Flink, a committer is someone with the right to merge code directly, and who becomes one is also decided by vote. That vote happens out of sight, though. Under the Flink bylaws, the Project Management Committee (PMC) votes on a private mailing list for at least three days. It passes with three or more yes votes from active PMC members and no justified objection from any of them. One such objection is enough to act as a veto.&lt;/p&gt;

&lt;p&gt;From the outside, all you see is the result. On October 21 an &lt;code&gt;[ANNOUNCE]&lt;/code&gt; email about a new committer arrived on the Flink dev mailing list, and people replied with congratulations one after another. I looked at how the earlier replies were written and sent a short one with my name.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmtog85n4bnm3x4a5vd59.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmtog85n4bnm3x4a5vd59.png" alt="The new committer announcement thread on the Flink dev mailing list. Under the first message, which announces on behalf of the PMC that Ramin Gharib has accepted the invitation to become a committer, congratulations follow, with my short reply at the bottom" width="799" height="311"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The new committer announcement on October 21 and the congratulations I sent early the next morning.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  #The vote that ships a release
&lt;/h2&gt;

&lt;p&gt;The vote on whether to ship a release happens in the open. The release manager uploads a release candidate, the build to be checked before the official release, and sends an email whose subject starts with &lt;code&gt;[VOTE]&lt;/code&gt; to the dev mailing list. People download the candidate, check it themselves and reply with +1 or -1. The Flink bylaws keep the vote open for at least three days, and only the votes of active PMC members count toward the result. These are called binding votes. A release passes with at least three binding yes votes and more binding yes than binding no, and others can also vote and report what they verified.&lt;/p&gt;

&lt;p&gt;On October 21, while checking things ahead of the Flink Agents 0.4 release, I found and fixed a bug where Java chat calls failed in the build made with JDK 17. On the 27th I checked the signatures and checksums of the first release candidate, compared the source against the tag and built it myself, and I also checked whether the classes my fix needed were in the release files. Then on the 28th I found that the vote email itself had landed in my spam folder.&lt;/p&gt;

&lt;p&gt;In my reply I gave a +1 marked as non-binding, meaning it doesn’t count toward the result, listed what I had checked, and said I had not run an end-to-end functional test. I had also found a case where the state of in-flight work was not restored when upgrading from the previous version, but since the problem was not new in this release, I did not raise it as a blocker. I suggested adding a line to the upgrade notes and said I would try to fix it in the next version.&lt;/p&gt;

&lt;p&gt;The vote passed with no objections. The result email listed seven yes votes, four of them binding, and my name was on the non-binding side.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxot9jfkpsuj2vrmxaqwn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxot9jfkpsuj2vrmxaqwn.png" alt="The result email for the first Flink Agents 0.4.0 release candidate. Below a sentence saying there were seven yes votes, four binding, and no objections, voter names are split into binding and non-binding" width="800" height="336"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The result email for the first Flink Agents 0.4.0 release candidate, received on October 29.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  #The vote that picks the Governance Committee
&lt;/h2&gt;

&lt;p&gt;OpenTelemetry’s Governance Committee reviews proposals for new projects, sets priorities and maintains the public roadmap, and the vote for its members was the most election-like of the three. I hadn’t applied for voting rights, either. Anyone with 20 or more contributions to the OpenTelemetry GitHub organization in the past year, counting PRs, issues and comments, gets one. Approvers, who review and approve contributions in their area, and maintainers, who set the technical direction of subprojects, also get a vote. The list is generated automatically, and contributors who fall short can ask for an exception.&lt;/p&gt;

&lt;p&gt;The public voter list as of October 21 had 837 accounts, and my username was on it. The count next to it said 29 contributions.&lt;/p&gt;

&lt;p&gt;All ten candidates nominated themselves. Self-nomination alone was not enough: each needed endorsement comments from voters at two other companies, and a cap keeps people from one company from filling the committee. During the election, one candidate even messaged me directly to ask what I thought.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9zhe3t5wui6vqe2joiry.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9zhe3t5wui6vqe2joiry.jpg" alt="The OpenTelemetry Governance Committee election page on the Helios voting system. It shows when voting opens and closes and a button to vote" width="750" height="381"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Voting ran on an online system called Helios from 9 p.m. on October 26 to 8:59 p.m. on October 29, Korea time.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;We could pick up to four candidates for four seats, and by a few hours before the deadline I had read all ten platforms. I had expected governance to mean rules and votes, but the platforms ranged much wider, from install packages and the demo app to moderation policy, events and relations with neighboring projects.&lt;/p&gt;

&lt;p&gt;AI stood out first. By my count, six of the ten candidates mentioned AI or agents in their bios or platforms, and by my reading five promised to protect maintainers’ time. Several platforms promised to help end users adopt OpenTelemetry, and a candidate who works at the American Red Cross wrote that OpenTelemetry is a core part of the blood services that supply roughly 40% of the US blood supply.&lt;/p&gt;

&lt;p&gt;What stayed with me longest was what the candidates said about code review. One candidate wrote that monthly PRs in the group they lead nearly quadrupled this summer and review could not keep up, and another, who put maintainer burnout on the first line, proposed cutting repetitive work with shared tools and automated checks.&lt;/p&gt;

&lt;h2&gt;
  
  
  #After the vote
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://opentelemetry.io/blog/2026/gc-elections-results/" rel="noopener noreferrer"&gt;official results&lt;/a&gt; for the OpenTelemetry Governance Committee election went up on October 29 in the US, and I saw them on the morning of the 30th, Korea time. 41% of eligible voters cast a ballot, and the top four vote-getters won new terms. That doesn’t mean seats simply go in order of votes. The charter allows at most two committee members from the same company at any time, and the cap counts members whose terms are still running. A candidate over the cap is skipped even with more votes, and the next candidate in line takes the seat. This time the top four came from four different companies, and none of those companies matched the companies of members whose terms run through 2027, so the seats went straight by vote count.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgchc8ozwhsv126kr48p8.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgchc8ozwhsv126kr48p8.jpg" alt="The 2026 OpenTelemetry Governance Committee election results graphic. It names Daniel Gomez Blanco, Josh Suereth, and the re-elected Pablo Baeyens Fernández and Trask Stalnaker" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The winners graphic OpenTelemetry shared with the results. All four serve two-year terms.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Whoever won, the review problem in those platforms is something many open source projects share right now. A post on the Kubernetes blog in June made the same point: AI has made generating code fast, but there has been very little improvement in maintaining code bases. In OpenTelemetry, one candidate wrote that review can’t keep pace with the growth in PRs, and the project’s generative AI policy calls maintainer and approver time a very scarce resource. The policy doesn’t ban AI. It says AI can be a useful tool for writing code, docs and tests when used well, and asks contributors to share their own thinking, even in imperfect English, rather than just passing along what a model generated.&lt;/p&gt;

&lt;p&gt;I use AI to contribute too. It makes writing issues in English and reading large, unfamiliar codebases much easier. When I open a PR, I write in the description what I actually ran and what I couldn’t verify, and splitting my release vote reply into what I checked and what I didn’t came from the same habit. Reading the platforms made me think this habit matters even more in open source. At a company you share context with your reviewers and can ask the person next to you, but an open source reviewer gives their own time to read a PR from someone they’ve never met. So the code and the PR description have to explain the intent of the change on their own, and they have to be written for the reviewer from the start.&lt;/p&gt;

&lt;h2&gt;
  
  
  #References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws" rel="noopener noreferrer"&gt;Flink Bylaws&lt;/a&gt;: Defines who votes, for how long and what it takes to pass, for release votes and committer votes.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://opentelemetry.io/blog/2026/gc-candidates/" rel="noopener noreferrer"&gt;Announcing the 2026 OpenTelemetry Governance Committee Election Candidates&lt;/a&gt;: Announces the ten candidates, the voting schedule, and how to check eligibility.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/open-telemetry/community/blob/main/elections/2026/governance-committee-candidates.md" rel="noopener noreferrer"&gt;2026 OpenTelemetry Governance Committee candidates&lt;/a&gt;: Collects the bios and platforms the ten candidates wrote themselves.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://opentelemetry.io/blog/2026/gc-elections-results/" rel="noopener noreferrer"&gt;Announcing the 2026 Governance Committee Election Results&lt;/a&gt;: Reports that 345 of 836 eligible contributors voted, a 41% turnout, and names the four winners.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/open-telemetry/community/blob/main/governance-charter.md" rel="noopener noreferrer"&gt;OpenTelemetry governance charter&lt;/a&gt;: Defines the committee’s role, voter eligibility, endorsement requirements for candidates and the per-company cap.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://kubernetes.io/blog/2026/06/26/open-source-maintainership-in-the-age-of-ai/" rel="noopener noreferrer"&gt;Open source maintainership in the age of AI&lt;/a&gt;: Describes how the Kubernetes community has changed its policies and review practices as AI-driven contributions grew.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/open-telemetry/community/blob/main/policies/genai.md" rel="noopener noreferrer"&gt;OpenTelemetry generative AI contribution policy&lt;/a&gt;: Sets out maintainer discretion and contributor responsibility for contributions made with AI.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;I write about building and leading large-scale platforms, from ads and CRM to AI agents, at &lt;a href="https://blog.namsang.co/en/?utm_source=devto&amp;amp;utm_medium=crosspost" rel="noopener noreferrer"&gt;Namsang Labs Tech Blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>opensource</category>
      <category>opentelemetry</category>
      <category>flink</category>
      <category>governance</category>
    </item>
    <item>
      <title>ObservabilityCON 2026 Keynote, Grafana Has Started Opening PRs</title>
      <dc:creator>Sangkyoon Nam</dc:creator>
      <pubDate>Tue, 20 Oct 2026 15:05:00 +0000</pubDate>
      <link>https://dev.to/sangkyoonnam/observabilitycon-2026-keynote-grafana-has-started-opening-prs-4df6</link>
      <guid>https://dev.to/sangkyoonnam/observabilitycon-2026-keynote-grafana-has-started-opening-prs-4df6</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxwud3sxzqvhiohv9gcxp.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxwud3sxzqvhiohv9gcxp.jpg" alt="Opening of the ObservabilityCON 2026 keynote. Co-founder Raj Dutt stands on stage with his name and title on the screen to the left" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;1:15 into the keynote recording. Source: &lt;a href="https://www.youtube.com/watch?v=_ASFsH_LIKE&amp;amp;t=75s" rel="noopener noreferrer"&gt;Grafana on YouTube&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The company best known for dashboards spent an hour barely talking about dashboards. That’s my takeaway from watching the recording of the ObservabilityCON 2026 keynote, held October 20 at Pier 27 in San Francisco. What held my attention was the demo where an agent investigates an incident and opens a fix PR, and the fact that the instrumentation Grafana builds itself is now built inside OpenTelemetry.&lt;/p&gt;

&lt;h2&gt;
  
  
  #Agents instead of dashboards
&lt;/h2&gt;

&lt;p&gt;Co-founder Raj Dutt introduced the company with numbers. A remote company of 1,700 people spread across more than 40 countries had passed 10,000 paying customers and $600M in annualized revenue, he said. He spent more time, though, on the claim that we’re at the start of a shift bigger than the mainframe or the cloud.&lt;/p&gt;

&lt;p&gt;The edge he pointed to is that Grafana, Prometheus and Loki are open source. The stack runs monitoring at a huge number of companies worldwide, and the internet is full of docs and posts on how to run it. That, in his telling, is why frontier models know this stack better than proprietary vendors’ products. So when Assistant went GA earlier this year, it became a usable tool much faster than they expected, and that surprised even them. Public knowledge is training data. For an open source company in the AI era, I can’t think of a more convincing story, and it matched what I’ve felt contributing to OpenTelemetry recently.&lt;/p&gt;

&lt;p&gt;The back half of the keynote was about Assistant taking over the work from watching to fixing. Watchers spend tokens during calibration, then run lightweight checks against the telemetry they selected. Deploy a new feature, let a watcher run for a few weeks, and it learns what normal looks like and alerts you when the feature drifts from it. In Investigations, one agent forms hypotheses while other agents concurrently try to disprove them. The presenter said only that letting the agents fight raises quality. My read is that the design targets a model’s habit of committing to one plausible cause too early. Managed Sandboxes take the investigation’s findings, change the code in your repository and open a PR. In the &lt;a href="https://www.youtube.com/watch?v=_ASFsH_LIKE&amp;amp;t=2887s" rel="noopener noreferrer"&gt;demo&lt;/a&gt;, Assistant stepped into a GitHub CI stage and commented that the PR apparently would break an SLO, and once an incident investigation finished it produced a fix PR from the result. They said they run a loop of their own that prepares a PR every Monday to cut CPU usage at the hotspots profiling found.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5gdjsuwhtc8dr6rig367.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5gdjsuwhtc8dr6rig367.jpg" alt="The Assistant workspace creating a fix PR straight from an incident investigation. The code change and the PR link sit next to the findings" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Slide at 49:05 of the keynote recording. Source: &lt;a href="https://www.youtube.com/watch?v=_ASFsH_LIKE&amp;amp;t=2945s" rel="noopener noreferrer"&gt;Grafana on YouTube&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I was more interested in the controls on code changes than in the fixing itself. Each task gets its own MicroVM, and policy decides which networks it can reach, whether it may push branches, and when it creates a PR. Pages, which builds a screen from a single sentence, offers a different safeguard. You can inspect the queries behind its output, and the presenter added that the model can make things up, so trust but verify. The Pages &lt;a href="https://www.youtube.com/watch?v=_ASFsH_LIKE&amp;amp;t=3334s" rel="noopener noreferrer"&gt;demo&lt;/a&gt; had its moment when zooming out wouldn’t work and two people stood there briefly flustered.&lt;/p&gt;

&lt;p&gt;Grafana has &lt;a href="https://grafana.com/docs/grafana-cloud/send-data/otlp/" rel="noopener noreferrer"&gt;accepted OpenTelemetry data&lt;/a&gt; for years. The instrumentation announced here is OpenTelemetry on the producing side as well. Agent Observability &lt;a href="https://grafana.com/press/2026/10/20/observabilitycon-2026-ai-updates/" rel="noopener noreferrer"&gt;went GA&lt;/a&gt; with OpenTelemetry instrumentation support. This is what the keynote called observability for AI, and the OpenTelemetry GenAI semantic conventions I’ve been working on lately are the spec for exactly this. Instead of a proprietary agent, Mobile Observability adds iOS and Android support directly to the upstream OpenTelemetry SDKs from the start. The presenter who then ran the Android demo also said Grafana is now the second largest contributor to OpenTelemetry. Logs you’d rather not throw away get shipped to your own bucket in OpenTelemetry format too.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fumhy75xlz954kn25s4n0.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fumhy75xlz954kn25s4n0.jpg" alt="Mobile Observability slide. Three pillars, consolidated experience, full platform support and OpenTelemetry native, with a private preview badge" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Slide at 27:20 of the keynote recording. Source: &lt;a href="https://www.youtube.com/watch?v=_ASFsH_LIKE&amp;amp;t=1640s" rel="noopener noreferrer"&gt;Grafana on YouTube&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Still, if my team adopts agent operations, the order has to start with context, not PRs. The model knows the software stack but not our architecture, and without context it burns tokens re-deriving the structure on every question. Grafana’s Knowledge Graph is that context. It pulls how infrastructure, runtimes and applications connect out of telemetry, automatically and continuously. The graph is GA, and the gcx CLI and an MCP server let an outside agent like Claude Code read the same graph. The 25% token reduction on the slide comes from Grafana’s &lt;a href="https://grafana.com/blog/knowledge-graph-as-context-for-llms-demonstrating-decisive-rca-and-faster-production-performance/" rel="noopener noreferrer"&gt;own measurement&lt;/a&gt;, the median over the pairs that reached the same conclusion out of 553 pairs where an agent with the graph and one without investigated the same alert separately. What we’d check first on our side is whether the service name carries the same value on every signal, and whether calls between services connect in traces. If names disagree, the graph can count one service as two, and if calls break, there’s less ground for tying a metric anomaly and a log error to the same incident. The agent then fills that gap with inference, and a Watcher on a sloppily instrumented system is likely to learn noise rather than normal.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffowozbp7kro70xmj80ht.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffowozbp7kro70xmj80ht.jpg" alt="Knowledge Graph slide. Internal measurement showing 25% fewer tokens to reach the same answer" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Slide at 22:10 of the keynote recording. Source: &lt;a href="https://www.youtube.com/watch?v=_ASFsH_LIKE&amp;amp;t=1330s" rel="noopener noreferrer"&gt;Grafana on YouTube&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I only run self-managed Grafana, so I checked whether any of this applies to us. The features in this keynote, from Assistant through Watchers and Managed Sandboxes to Pages, need a Grafana Cloud account. You &lt;a href="https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/get-started/self-managed/" rel="noopener noreferrer"&gt;can use Assistant&lt;/a&gt; on self-managed Grafana, but it has to connect to a Grafana Cloud stack. Investigations and Grafana Cloud MCP connections stay off in self-managed even with a Cloud stack attached. What self-managed does get is Loki full-text search and the OpenTelemetry SDKs, and they said the Loki side will reach an open source release shortly. For someone who has held out on self-managed, this is the first time there’s a reason to use the cloud. I haven’t used Assistant yet, so this judgment rests on the keynote alone. I plan to verify it with a PoC when I get the chance. The baseline is the record a human left investigating the same incident, and I’ll look at whether it got the cause right and whether the fix PR was mergeable without edits. (We could also build this structure ourselves, of course, inside the agent harness we’ve been putting together.) Either way, the direction is clear. Observability tools are moving from only showing you things to resolving incidents. Grafana starting to open PRs is what that means. And then telemetry is no longer a record you keep. Which telemetry you send becomes the input that decides what the agent changes and how.&lt;/p&gt;

&lt;h2&gt;
  
  
  #Two announcements outside the main thread
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://grafana.com/blog/observabilitycon-2026-announcements/" rel="noopener noreferrer"&gt;official guide&lt;/a&gt; sorts everything into GA, public preview and private preview, so this section adds just two things the main thread didn’t cover. If you’re short on time, I’d start at &lt;a href="https://www.youtube.com/watch?v=_ASFsH_LIKE&amp;amp;t=1947s" rel="noopener noreferrer"&gt;32 minutes&lt;/a&gt; into the video.&lt;/p&gt;

&lt;p&gt;One is user-side observability, which Grafana files under digital experience monitoring. Frontend Observability grew Session Replay and Critical User Journeys, and Mobile Observability landed next to it in private preview. The presenter said Mobile has been one of their most requested features.&lt;/p&gt;

&lt;p&gt;The other is cost, and the message was that they’d absorb the telemetry explosion with technology rather than price. The evidence was running entire regions inside the customer’s cloud account, a bare-metal cluster being built in Ireland, and Mimir’s 10 billion active series, up tenfold in a year. Loki full-text search went GA on indexing technology they acquired in April, with average queries 5x faster and 75% less data scanned, they said. That it comes at no additional cost is in the &lt;a href="https://grafana.com/blog/observabilitycon-2026-announcements/" rel="noopener noreferrer"&gt;official guide&lt;/a&gt;. Sampling policies get pushed down to Alloy and OpenTelemetry collectors so data you’d drop is dropped before it’s sent. For now that applies to logs only, with other signals to follow, they said.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foj61tvlitc942b5wwei1.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foj61tvlitc942b5wwei1.jpg" alt="Adaptive Telemetry slide. An edge sampling pipeline that pushes sampling rules down to the collectors in customer infrastructure, and a potential 4 PB per month in egress savings on logs alone" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Slide at 17:10 of the keynote recording. Source: &lt;a href="https://www.youtube.com/watch?v=_ASFsH_LIKE&amp;amp;t=1030s" rel="noopener noreferrer"&gt;Grafana on YouTube&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The event is still underway in US time. Once the session recordings are up, I’ll go through them and add the ones worth watching here.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  #References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://www.youtube.com/watch?v=_ASFsH_LIKE" rel="noopener noreferrer"&gt;ObservabilityCON 2026 Keynote (YouTube, Grafana)&lt;/a&gt;. The one-hour keynote in which eight presenters, Raj Dutt and Tom Wilkie among them, walk through cloud economics, context, agent operations and Pages in turn.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://grafana.com/blog/observabilitycon-2026-announcements/" rel="noopener noreferrer"&gt;ObservabilityCON 2026: A guide to all the announcements from Grafana Labs&lt;/a&gt;. The official guide that sorts the announced features into GA, public preview and private preview.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://grafana.com/press/2026/10/20/observabilitycon-2026-ai-updates/" rel="noopener noreferrer"&gt;Grafana Labs Charts the Path to Agentic Operations (press release, Oct 20, 2026)&lt;/a&gt;. The press release announcing OpenTelemetry support in Agent Observability, Watchers going GA, and the public previews of Pages and Managed Sandboxes.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/get-started/self-managed/" rel="noopener noreferrer"&gt;Grafana Assistant for self-managed Grafana&lt;/a&gt;. The official doc on what it takes to use Assistant on self-managed Grafana.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://grafana.com/events/observabilitycon/2026/agenda/open-the-black-box-why-observability-is-the-key-to-trusting-your-AI-agents" rel="noopener noreferrer"&gt;Why observability is the key to trusting your AI agents&lt;/a&gt;. The session that treats agent sessions as a signal to view alongside metrics, logs, traces and profiles. Worth watching right after the keynote once the recording is up.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;I write about building and leading large-scale platforms, from ads and CRM to AI agents, at &lt;a href="https://blog.namsang.co/en/?utm_source=devto&amp;amp;utm_medium=crosspost" rel="noopener noreferrer"&gt;Namsang Labs Tech Blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>observability</category>
      <category>grafana</category>
      <category>opentelemetry</category>
      <category>agents</category>
    </item>
  </channel>
</rss>
