DEV Community: Span Observer

Understanding Dynatrace Managed architecture: nodes, storage and data flow

Span Observer — Fri, 06 Nov 2026 21:30:17 +0000

When you inherit a Dynatrace Managed cluster, the first challenge is not reading the dashboards. It is understanding what actually runs on the servers: in other words, the Dynatrace Managed architecture. A disk incident, a Cassandra alert or a node that will not restart forces you to know which components live where, what data they store and which path OneAgent measurements take to get there.

This question is specific to Managed. In the hosted model, the vendor absorbs all of this complexity. With Managed, your team owns the operating system, the disks, the network, the backups and the update schedule. A good mental map of the architecture is therefore a prerequisite for running it with confidence.

This article describes the building blocks of a Managed cluster and how data flows between agents, ActiveGates and nodes, then walks through a short hands-on routine to inventory your own installation.

Dynatrace Managed architecture overview: a cluster, nodes and environments

A Managed cluster is a set of Linux servers, called nodes, that all run the same software stack. There is no fixed « master » role: every node receives traffic, processes data, stores part of it and serves the web UI. This symmetry makes adding capacity simple, but it requires all nodes to be sized identically.

On top of this foundation, the cluster hosts one or more environments (tenants). Each one is reachable at https://{your-domain}/e/{environment-id} and has its own entities, alerting rules, users and quotas. Cluster-wide administration (nodes, licences, backups, updates, environments) is done in the Cluster Management Console (CMC).

What runs on each node

A typical node runs several cooperating processes:

The Dynatrace server: the core application (JVM) that receives agent data, correlates entities, runs causation analysis and exposes the UI and APIs.
A front-end web proxy: it terminates HTTPS connections from browsers and agents, then distributes requests to the cluster’s servers.
Cassandra: the distributed database for time series (metrics) and part of the configuration.
Elasticsearch: the index for user-oriented data, notably RUM sessions and actions.
Transaction storage: local files holding distributed traces (PurePaths) and their details.

By default, the binaries live in /opt/dynatrace-managed and the data in /var/opt/dynatrace-managed. It is common, and recommended, to put the different stores on dedicated volumes so that unexpected growth in one does not fill up the others.

Storage types and how they behave

Store	Content	Replication	Watch out for
Cassandra	Metrics, timeseries data	Replicated across nodes (factor 3 is typical from 3 nodes)	Sensitive to disk and network latency
Elasticsearch	User sessions, actions, RUM data	Replicated across nodes	Growth tied to session volume
Transactions	Distributed traces, request details	Distributed, usually not replicated	Losing a node loses part of the recent traces
Backups	Copies of Cassandra, Elasticsearch, configuration	On shared storage (NFS)	Must be mounted at the same path on every node

The practical consequence matters: high availability for metrics and configuration relies on replication, and therefore on having enough nodes. A single-node cluster works, but it offers no fault tolerance. In a multi-node cluster the metrics store is kept in three copies, while transaction storage is simply distributed, without redundancy: the documentation requires at least three nodes in production.

End-to-end data flow

Collection: the OneAgent installed on a host instruments processes, services and infrastructure.
Transport: the agent sends its data over HTTPS either directly to the cluster nodes or to an ActiveGate, which concentrates, compresses and forwards it.
Reception: a node’s front-end proxy accepts the connection and passes it to the Dynatrace server.
Processing: the server correlates the data with the entity model (topology), computes baselines and detects anomalies.
Persistence: metrics go to Cassandra, sessions to Elasticsearch, traces to transaction storage.
Consumption: the web UI, the environment API (…/e/{id}/api/v2/…) and integrations use this data.

Agents receive the list of available endpoints and fail over automatically if a node or an ActiveGate becomes unavailable. That is why you should avoid hard-coding a single node in network configurations.

Network flows you need to know

Agents and ActiveGates to the cluster: HTTPS on port 443 (redirected locally to 8022 by an iptables rule). Port 8443 also receives OneAgent data and carries node-to-node traffic; it no longer needs to be exposed outside the cluster, except on clusters older than version 1.166 or upgraded from them.
Between nodes: a set of ports for Hazelcast (5701–5711), Cassandra (7000, 7001, 9042), Elasticsearch (9200, 9300) and server-to-server communication. These ports must be open between all nodes, and only between them.
Cluster to Mission Control: outbound HTTPS for licensing, updates and support, except in offline mode where these operations are manual.
Nodes to NFS: for backups.

Step-by-step tutorial: inventory your cluster

Here is a simple routine to check, on an existing cluster, that reality matches the picture above.

Step 1 — List the nodes in the CMC

Log in to the CMC and open the nodes overview page. For each node, note its IP address, status, version and the disk usage shown. Check that all versions are identical.

Step 2 — Identify the services on a node

On each server, as root:

systemctl list-units --type=service | grep -i dynatrace
ps -eo pid,user,rss,cmd --sort=-rss | grep -i -E "dynatrace|cassandra|elasticsearch" | head -20

You should see the server, Cassandra, Elasticsearch and front-end proxy processes.

Step 3 — Map the storage volumes

df -h /opt/dynatrace-managed /var/opt/dynatrace-managed
du -sh /var/opt/dynatrace-managed/* 2>/dev/null | sort -h
findmnt -t nfs,nfs4

These commands show where the data actually lives, which directory grows fastest and whether the backup share is properly mounted.

Step 4 — Check the listening ports

ss -tlnp | grep -E ":443|:8443|:9042|:9200|:9300"

Compare the output with the port matrix in the documentation for your version, and review the firewall rules between nodes.

Step 5 — Query the Cluster API

Create a Cluster API token with the Service Provider API permission in the CMC, then list the environments:

curl -s -H "Authorization: Api-Token {token}" \
  "https://{your-domain}/api/cluster/v2/environments" | python3 -m json.tool

This output becomes your reference for documenting which tenants live on the cluster.

Step 6 — Check the logs

journalctl -u "dynatrace*" --since "1 hour ago" --no-pager | tail -50
ls -lh /var/opt/dynatrace-managed/log/ 2>/dev/null

The exact location of application logs can vary with the version and installation choices.

Best practices

Aim for at least three nodes in production to benefit from data replication.
Keep nodes strictly identical (CPU, RAM, disks): the weakest one sets the cluster’s performance.
Split the stores across separate volumes and monitor how full each one is independently.
Place nodes on the same low-latency network; for multiple sites, look at Premium High Availability.
Use ActiveGates to concentrate traffic from remote network zones rather than opening the cluster everywhere (see Environment ActiveGate or Cluster ActiveGate).
Document the flow matrix and version it alongside your firewall configuration.

Frequently asked questions

Is a single-node Managed cluster enough?

It works, but it offers no fault tolerance: Cassandra and Elasticsearch replication requires several nodes. In production, aim for at least three identical nodes.

Where is the data stored on a node?

By default under /var/opt/dynatrace-managed: metrics in Cassandra, RUM sessions and actions in Elasticsearch, distributed traces in local transaction storage. Backups go to an NFS share mounted at the same path on every node (see backing up and restoring a cluster).

Should OneAgents be pointed at a specific node?

No. Agents receive the list of available endpoints and fail over automatically if a node or an ActiveGate becomes unavailable. Hard-coding a single node in the network configuration throws that benefit away.

Conclusion

A Managed cluster is a complete distributed system: an application server, a proxy, two distributed databases, local trace storage and a shared backup mechanism. Knowing what stores what and where the data travels helps you anticipate incidents, size correctly and talk to support effectively. The next articles in this series build on this map to cover sizing, installation and cluster expansion. To practise on a test cluster, follow module 2 of the course on the Dynatrace Managed cluster, then keep it under watch with cluster health monitoring.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Always check values (ports, versions, sizes) in the official documentation for your version of Managed.

Originally published on span.observer. French version: https://span.observer/fr/architecture-dynatrace-managed/.

OpenChoreo Kubernetes developer platform, open source

Span Observer — Fri, 06 Nov 2026 19:30:15 +0000

OpenChoreo Kubernetes developer platform: the openchoreo/openchoreo repository describes itself as « a complete, open-source developer platform for Kubernetes ». It targets teams that build an internal developer platform (IDP) by gluing together a portal, CI, GitOps and observability. The project is released under the Apache 2.0 licence and, according to its README, belongs to the CNCF Sandbox.

The README states that WSO2 developed OpenChoreo from its experience with the Choreo SaaS platform, and that it is a complete rewrite, not a fork. For an SRE, the appeal comes down to one idea: logs, metrics and traces are surfaced « through the abstractions your developers already understand » (component, project, environment). What does the repository contain, how can it be tried, and what can be monitored?

A note on honesty: span.observer has not installed or run OpenChoreo. The commands and files quoted here come from the README, the repository’s samples/ folder and the official documentation on openchoreo.dev, all read on 5 November 2026. The screenshots below reproduce those texts. The star count could not be re-read at the source that day, so it is not quoted.

OpenChoreo Kubernetes developer platform: what the repository says

Licence: the LICENSE file is the Apache License 2.0; the README confirms « Apache 2.0 ».
Version: the VERSION file on the main branch reads 1.4.0-m.2, a milestone build. The installation guides in the documentation target v1.3.3, which is the version used here.
Plane architecture: the README describes an experience plane (Backstage portal, CLI, GitOps, AI agents), a control plane that translates abstractions into Kubernetes resources, a data plane where workloads run, an observability plane and, optionally, a CI plane based on Buildpacks and Argo Workflows.
Abstractions: developers work with components, endpoints and dependencies; platform teams define component types, traits and workflows through Kubernetes resources (CRDs), with no custom controller.

Another useful point: the documentation groups the ideas into four concept pages: developer abstractions, platform abstractions, runtime model and resource relationships.

Step-by-step installation (local trial)

These steps follow the « on k3d locally » guide in the official documentation. They create a local Kubernetes cluster inside Docker: this is a trial, not a production install.

Step 1: check the prerequisites

The documentation asks for Docker v26 or later (with 8 GB of RAM and 4 CPUs), k3d v5.8 or later, kubectl v1.34 or later and Helm v3.12 or later. It suggests checking them with the four commands in the screenshot below.

Step 2: run the one-command install

The script creates the cluster and installs all four planes. The --with-build option adds the workflow (CI) plane and --with-observability adds the observability plane.

The guide also offers a step-by-step Helm installation to understand each component. A shorter alternative exists: the quick start launches a development container (ghcr.io/openchoreo/quick-start:v1.3.3), then runs ./install.sh --version v1.3.3 inside it; adding --with-build --with-observability works the same way.

Step 3: open the portal

The Backstage portal is served at http://openchoreo.localhost:8080; the Observer answers at http://observer.openchoreo.localhost:11080, and deployed application traffic uses port 19080. The default trial credentials are listed in the documentation: never reuse them outside a local machine.

Service observability: what the documentation describes

The installation guide deploys the observability plane with several Helm charts: a logs module and a tracing module based on OpenSearch, a metrics module based on Prometheus, Fluent Bit to collect logs and an OpenTelemetry collector for Kubernetes events. The events are exported to an OpenSearch index named k8s-events.

The plane is modular. The « tracing adapter API » page explains that the Observer queries an adapter to read traces, spans and span details; the backend is whatever the module provides. That page names no backend other than those in the installation, so do not assume any.

Alerts declared as code

According to the documentation, developers attach an alert rule to a component as a trait (observability-alert-rule). A condition has a window, an interval, an operator (gt, lt, gte, lte, eq) and a threshold. The source is a log pattern or a resource metric; the repository example names cpu_usage and memory_usage.

This example fires a critical alert if more than five occurrences of the error appear within five minutes, evaluated every minute. It is exactly the reasoning of an error-rate SLI; to frame it, see our SLO and SLA learning path.

Notification channels

Alerts leave through one channel per environment, either email (SMTP) or webhook. According to the documentation, the first channel created in an environment becomes the default. The next excerpt comes from the sample file; the webhook address is an example to replace.

A root cause analysis agent

The repository also contains an RCA agent. According to its README, it receives an alert through a REST API, investigates with a ReAct loop through two MCP servers (observability, and OpenChoreo resources) and produces a structured report. Only a set of read-only tools is exposed to it. To judge this kind of agent in production, read the guardrails for AI agents and how to secure an MCP server.

Limits and open questions

Weight of the stack: a full installation assembles Gateway API, cert-manager, External Secrets, an OpenBao vault, an identity provider, OpenSearch and Prometheus. Plan for operating time, not just installation time.
Local trial: the documentation examples use default credentials, and the events collector is configured with insecure_skip_verify: true towards OpenSearch. These settings do not suit production.
Pace of change: the main branch is a 1.4.0 milestone; the changelog mentions, for example, release lines and secret management for the observability plane. Read the release notes before each upgrade.
Application instrumentation: we did not read a page describing how an application sends its traces to the observability plane. This point is unverified; check the documentation before promising it to a team.
Dynatrace: the project documents no Dynatrace integration, and we do not invent one.

Monitoring the platform itself

An internal platform is a service like any other: if the control plane goes down, nobody deploys. OpenChoreo’s documentation describes observability for the deployed workloads; for the platform, you still have to monitor the Kubernetes components that carry it. Here are common-sense pointers, not specific to the project:

Availability: watch the deployments in the openchoreo-control-plane, openchoreo-data-plane and openchoreo-observability-plane namespaces created by the installation, using alerts on pods and quotas.
Metrics: if your cluster already runs Prometheus, see how to connect Prometheus to Dynatrace; the principle holds for any observability backend.
OpenTelemetry signals: the installation’s events collector is an OpenTelemetry collector; our notes on Collector contrib components help to read this kind of configuration.

Next in the series: Jan, the open source ChatGPT alternative, another project read the same way.

Best practices

Try it on k3d first with the observability plane, then compare with your current stack before any migration.
Keep alert rules in Git like other resources, and tie each threshold to an SLO.
Replace every example credential and secret; use the intended secrets vault rather than plain-text values.
Pin chart versions, as the documentation does with --version.
Restrict agents’ MCP tools to read-only, as the repository’s RCA agent does.

Frequently asked questions

Does OpenChoreo replace Backstage?

No: the README describes a « Backstage-powered » portal. OpenChoreo adds the abstractions, control plane, CI, GitOps and observability around that portal.

Do I need to install the observability plane?

It is optional according to the documentation: the --with-observability option adds it. You therefore need to enable it to get the logs, metrics and traces described above.

Which licence, and who maintains the project?

The code is under Apache 2.0. The README states initial development by WSO2 and CNCF Sandbox status; read the repository’s governance page (GOVERNANCE.md) before an enterprise decision.

Conclusion

OpenChoreo offers a complete developer platform for Kubernetes, with built-in observability and alerts declared in YAML. The project looks active, but the installation is heavy and several points remain to be checked in the documentation: application instrumentation and long-term operation. Try it locally, measure what it replaces in your toolchain, then decide.

span.observer is an independent community blog, not affiliated with OpenChoreo, WSO2 or Dynatrace LLC. Brands mentioned belong to their owners. Tools change quickly: check commands and options in their official documentation.

Originally published on span.observer. French version: https://span.observer/fr/openchoreo-plateforme-developpeur-kubernetes/.

Jan open source ChatGPT alternative: offline local AI

Span Observer — Fri, 06 Nov 2026 16:20:23 +0000

Jan open source ChatGPT alternative: the janhq/jan repository calls itself an « Open-source ChatGPT replacement », a desktop app that downloads and runs language models on your own machine « with full control and privacy ». It is released under the Apache 2.0 licence by Menlo Research. The README lists Windows, macOS and Linux, and the documentation speaks of full offline support for the desktop app.

For an SRE or a developer, the appeal goes beyond chat. Jan bundles an OpenAI-compatible server on localhost:1337, an MCP host and a command-line tool. That makes it a local inference endpoint for your scripts and agents. What does the repository actually say, how do you install it, and what can you measure on a tool that runs on a workstation?

A note on honesty: span.observer has not installed or run Jan. The commands and settings come from the README, from the official documentation on jan.ai and from the repository’s docs/ files, read on 31 October 2026. The screenshots below reproduce their text. We could not read the star count from the source that day, so we do not quote it.

Jan open source ChatGPT alternative: what the repository says

Licence: the repository’s LICENSE file is the Apache License 2.0, with the line « Copyright 2025 Menlo Research ». The README confirms « Apache 2.0 ».
Version: the main branch carries version 0.8.8 in src-tauri/tauri.conf.json. The repository’s changelog dates 0.8.8 to 28 October 2026. It is a security release: chat links open in your browser, and the local server checks the origin of browser requests.
Engine: the README thanks llama.cpp, Tauri and Scalar. The documentation says the llama.cpp engine is built into the app and updates with it. On Apple Silicon Macs, MLX models are supported too.
Listed features: local models from Hugging Face (Llama, Gemma, Qwen, GPT-oss), connections to cloud models (OpenAI, Anthropic, Mistral, Groq, MiniMax), custom assistants, an OpenAI-compatible API and MCP integration.

One more point: « offline » applies to local models. The documentation states that requests routed to a remote provider are not offline and do not stay on your computer.

A tour of the repository and its documentation

The first screenshot reproduces, word for word, the README’s « Features » section and its licence. It shows the scope of the product in six lines.

Source : github.com

The documentation then describes the local server. The defaults are host 127.0.0.1, port 1337 and prefix /v1. An optional API key goes in the Authorization: Bearer header. The second screenshot gathers the sample request and these settings.

Source : jan.ai

Step-by-step installation

Step 1: download the app

The README offers one installer per platform (jan.exe for Windows 10 and later, jan.dmg for macOS, jan.deb and jan.AppImage for Linux) from jan.ai or the GitHub Releases page. Windows is also available through the Microsoft Store.

On Linux, the documentation recommends Flatpak:

flatpak install flathub ai.jan.Jan
flatpak run ai.jan.Jan

For a .deb package, the documentation gives sudo apt-get install ./jan-linux-amd64-{version}.deb, where {version} stands for the number you downloaded. For the AppImage, run chmod +x on the file, then execute it directly.

Step 2: download a model

Open the Hub in the sidebar, search for a model, click Download, then pick it in a conversation’s model selector. For macOS, the README gives a memory guide: 8 GB of RAM for 3-billion-parameter models, 16 GB for 7 billion, 32 GB for 13 billion.

Step 3: start the OpenAI-compatible server

Go to Settings > Local API Server.
Choose the Default Model, a llama.cpp or MLX model you have already downloaded.
Click Start Server, then wait until the status shows the server is running.
Test it with the curl request in the screenshot above, replacing YOUR_MODEL_ID.

According to the documentation, you read the list of available models with GET /v1/models. If no key is configured, omit the Authorization header.

Step 4 (optional): the command line

The documentation says the jan CLI is installed automatically the first time you launch the app. It serves a model without the interface, with jan serve, or starts an agent wired to a local model, with jan launch. Note that the CLI exposes the model on localhost:6767/v1, not on the app server’s port 1337. The third screenshot shows the CLI help and the layout of the data folder.

Source : jan.ai

Limits and open questions

Hardware: the documentation says the bundled llama.cpp engine does not load models on Intel Macs since 0.8.5; MLX also requires Apple Silicon.
Network exposure: choosing host 0.0.0.0 makes the server reachable from the local network. The documentation then recommends an API key and advises against leaving it empty. Trusted Hosts add a further layer of control.
Tool execution: the « Execute Tools on Server » option, disabled by default, runs MCP tool calls received by the API on the server side. Leave it off unless the client needs it.
Models and MCP: the documentation warns that not every model handles tool calls well. MCP-specific risks are covered in our guide to securing an MCP server.
Telemetry: the privacy page describes usage statistics subject to your choice at first launch, changeable later in settings, sent to PostHog (EU hosting). It states that conversation content is not tracked. Read it again before an enterprise rollout.
Pace of change: three releases in three weeks (0.8.6, 0.8.7, 0.8.8, going by the changelog dates). Test each update before rolling it out widely.

Monitoring Jan on a workstation or test server

Jan is a desktop app, not a service designed for Kubernetes. The project documents neither Prometheus metrics nor OpenTelemetry export for the app, so we assume none. What the documentation does provide:

Logs: logs/app.log in the data folder, with the [APP], [SERVER] and [SPECS] categories. « Verbose Server Logs » are enabled by default and detail the requests the local server receives.
Availability: a synthetic check on GET /v1/models confirms the server answers and the expected model is present. A 401 means a missing or wrong key, a 404 an unknown model ID, and « Connection Refused » a stopped server or wrong port. The documentation lists these three cases.
Resources: memory and CPU or GPU use of the process can be measured with host tools. That is where you see a model too large for the machine. The « Request Timeout » setting bounds the wait for a model response.

For client-side calls, instrument the script that queries the endpoint: one span per request, with the model name and the duration. The approach is detailed in monitoring a local LLM with Ollama, and it applies to any local OpenAI-compatible server. To choose the model itself, see the operational criteria for an LLM.

And Dynatrace?

Jan documents no Dynatrace integration, and we do not invent one. OpenTelemetry instrumentation of the client that calls the local server remains possible with any OTLP-compatible backend, Dynatrace SaaS included; check the setup in the Dynatrace documentation.

Best practices

Keep the host at 127.0.0.1 while no other machine needs the API, and set an API key as soon as you go beyond that.
Leave « Execute Tools on Server » off unless you have a specific need, and limit connected MCP servers. See also guardrails for AI agents in production.
Keep Jan up to date: version 0.8.8 fixes issues reported through GitHub Security Advisories.
Watch the app.log file after every model or version change.
If you need an engine for production, compare it with vLLM and its metrics.

Frequently asked questions

Does Jan really work offline?

Yes for local models, once downloaded: the quickstart describes local models as « private, offline, no API key ». Cloud models and requests routed to a remote provider need a connection.

Can Jan replace the OpenAI API for my scripts?

The documentation presents it as a drop-in replacement for private applications. Point your client at http://127.0.0.1:1337/v1. span.observer has not tested the exact compatibility of each parameter, so check the ones your code depends on.

Which licence, and can I use it at work?

The code is under Apache 2.0, a permissive licence. Each downloaded model has its own licence: read it before any professional use.

Conclusion

Jan is a simple way into local AI. Its OpenAI-compatible server and CLI make it useful for trials, agent prototypes and environments without a cloud. Monitoring remains yours to build around it: logs, synthetic checks and host measurements. The project moves fast, so reread the documentation before each rollout.

span.observer is an independent community blog, not affiliated with Jan, Menlo Research or Dynatrace LLC. Trademarks belong to their owners. Tools change quickly: check commands and options in their official documentation.

Originally published on span.observer. French version: https://span.observer/fr/jan-alternative-chatgpt-open-source/.