<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Linhao Liu</title>
    <description>The latest articles on DEV Community by Linhao Liu (@p311).</description>
    <link>https://dev.to/p311</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1373431%2F048a4787-cd00-446b-b07b-0afaca0c2ef8.png</url>
      <title>DEV Community: Linhao Liu</title>
      <link>https://dev.to/p311</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/p311"/>
    <language>en</language>
    <item>
      <title>Who owns the LLM? 28 AIOps products compared</title>
      <dc:creator>Linhao Liu</dc:creator>
      <pubDate>Wed, 04 Nov 2026 04:35:48 +0000</pubDate>
      <link>https://dev.to/p311/who-owns-the-llm-20-aiops-products-compared-2jnl</link>
      <guid>https://dev.to/p311/who-owns-the-llm-20-aiops-products-compared-2jnl</guid>
      <description>&lt;p&gt;If you build AI features for an AIOps product, a customer will ask "can we use our own LLM?" sooner or later. What they want is clear: keep the data private and under their control, and still get useful answers from the LLM. &lt;/p&gt;

&lt;p&gt;The strongest models mostly sit behind public APIs, and that is also where vendors can test and evaluate their features properly. A model running inside the customer's own network keeps every alert and log inside their walls, and it even works at air-gapped sites. But it is hard for the vendor to manage, and it tends to struggle more with multi-step tool calls. Some vendors try a middle path: They host an open model in their own cloud instead of calling a public API. Data no longer goes to a third-party model provider, and the vendor tests the exact model it runs. Open models usually are weaker than the top API models, while the vendor now pays for the GPUs.&lt;/p&gt;

&lt;p&gt;I went through public docs for 28 products to see who chose what. AIOps here means event correlation and incident intelligence platforms, plus the newer AI SRE agents that investigate incidents on the same data.&lt;/p&gt;

&lt;p&gt;Short answer: most of them run the model for you. 13 give you no real say in it. 10 ship a default and let you plug in your own. 5 expect you to bring it. Three of those are open source, and the other two are commercial products that still leave the model to you.&lt;/p&gt;

&lt;h2&gt;
  
  
  The tools
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Vendor-managed&lt;/strong&gt;&lt;strong&gt;:&lt;/strong&gt; the vendor picks and runs the model. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Hybrid&lt;/strong&gt;&lt;strong&gt;:&lt;/strong&gt; there is a vendor default and you can swap in your own.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Customer-provided&lt;/strong&gt;&lt;strong&gt;:&lt;/strong&gt; means the AI features stay off until you connect a model.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Product&lt;/th&gt;
&lt;th&gt;Strategy&lt;/th&gt;
&lt;th&gt;What the docs say&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://docs.dynatrace.com/docs/discover-dynatrace/platform/davis-ai/copilot/copilot-conv" rel="noopener noreferrer"&gt;Dynatrace Davis CoPilot&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;Runs on Azure OpenAI. &lt;a href="https://www.dynatrace.com/company/trust-center/capability-specific-terms/" rel="noopener noreferrer"&gt;Terms&lt;/a&gt; say prompt logs are kept 35 days in Azure US East.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://newrelic.com/blog/how-to-relic/protecting-data-privacy-and-security-with-generative-ai" rel="noopener noreferrer"&gt;New Relic AI&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;Runs on Azure OpenAI. Queries are inspected and responses redacted before going to the LLM.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.pagerduty.com/blog/news-announcements/aws-reinvent-2024-announcements-2/" rel="noopener noreferrer"&gt;PagerDuty Advance&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;Powered by Amazon Bedrock models, with &lt;a href="https://aws.amazon.com/isv/case-studies/how-pagerduty-and-aws-transform-digital-operations-with-ai" rel="noopener noreferrer"&gt;Anthropic Claude&lt;/a&gt; named by PagerDuty. Billed as &lt;a href="https://www.pagerduty.com/fr/product-specific-terms/" rel="noopener noreferrer"&gt;AI Actions&lt;/a&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.logicmonitor.com/?p=590579" rel="noopener noreferrer"&gt;LogicMonitor Edwin AI&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;Addendum says customers can't choose the model. Azure, OpenAI and Bedrock are subprocessors.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.datadoghq.com/legal/subprocessors/" rel="noopener noreferrer"&gt;Datadog Bits AI SRE&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;OpenAI is listed as an AI subprocessor, and Nebius was added in Oct 2026. No BYO option found.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://learn.microsoft.com/en-us/azure/sre-agent/data-privacy" rel="noopener noreferrer"&gt;Azure SRE Agent&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;Microsoft runs it. You choose Anthropic or Azure OpenAI, and the default depends on region.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://aws.amazon.com/devops-agent/faqs" rel="noopener noreferrer"&gt;AWS DevOps Agent&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;AWS FAQ says it runs on Amazon Bedrock foundation models.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://docs.incident.io/admin/ai-governance.md" rel="noopener noreferrer"&gt;incident.io AI SRE&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;OpenAI, Anthropic and Google Vertex. FAQ says no BYO model or key, because features mix models per task.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://resolve.ai/" rel="noopener noreferrer"&gt;Resolve AI&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;Its own domain models plus frontier models. A &lt;a href="https://resolve.ai/security" rel="noopener noreferrer"&gt;dedicated single-tenant option&lt;/a&gt; comes with bring-your-own keys.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://cleric.ai/resources/webinars/first-self-learning-ai-sre-agent" rel="noopener noreferrer"&gt;Cleric&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;Hosted. The CTO said they're open to customer-brought models, but nobody has asked for it yet.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://help.komodor.com/hc/en-us/articles/22447540595218" rel="noopener noreferrer"&gt;Komodor Klaudia&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;Kubernetes AI SRE running on the AWS Bedrock API.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://sciencelogic.com/press_release/sciencelogic-expands-secure-ai-deployment-options-with-skylar-ai-2-5" rel="noopener noreferrer"&gt;ScienceLogic Skylar AI&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;Version 2.5 (Aug 2026) runs the AI tier in sovereign cloud, on-prem or secure cloud, and tracks token usage of its own agents.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://selector.ai/?p=5891" rel="noopener noreferrer"&gt;Selector&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Vendor-managed&lt;/td&gt;
&lt;td&gt;Its own network language model, built on an open base model. Fine-tuned per customer, on-prem at the customer.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://docs.bigpanda.io/en/manage-ai-incident-assistant-configuration" rel="noopener noreferrer"&gt;BigPanda Biggy AI&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hybrid&lt;/td&gt;
&lt;td&gt;Default is BigPanda's OpenAI service with zero data retention. Azure OpenAI can be BigPanda-hosted or customer-hosted.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://servicenow.com/community/now-assist-articles/using-external-llms-with-now-assist-including-byok-byollm/ta-p/3218103" rel="noopener noreferrer"&gt;ServiceNow Now Assist&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hybrid&lt;/td&gt;
&lt;td&gt;ServiceNow-managed by default. BYOK through your Azure, Google AI Studio or Bedrock. BYOLLM for uncertified models.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://docs.helixops.ai/bin/Service-Management/Employee-Digital-Workplace/BMC-HelixGPT/helixgpt262/Getting-started/Key-concepts/BMC-HelixGPT-architecture/" rel="noopener noreferrer"&gt;BMC Helix AIOps&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hybrid&lt;/td&gt;
&lt;td&gt;BMC's own fine-tuned &lt;a href="https://docs.helixops.ai/bin/Service-Management/Employee-Digital-Workplace/BMC-HelixGPT/helixgpt263/Getting-started/Key-concepts/Models-in-BMC-HelixGPT/" rel="noopener noreferrer"&gt;Helix LLM&lt;/a&gt; on Vertex AI and Azure, or your own Azure, OpenAI or Google model.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://community.ibm.com/community/user/blogs/neil-boyette/2025/10/01/how-agents-and-llms-are-accelerating-ops-teams?hlmlt=VT" rel="noopener noreferrer"&gt;IBM Cloud Pak for AIOps&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hybrid&lt;/td&gt;
&lt;td&gt;Assistant runs on a mix of Llama and Granite models. Incident summary (tech preview) uses the customer's own watsonx.ai instance.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.ibm.com/docs/en/SSE1JP5/src/pages/incident_ai_summarization/index.html" rel="noopener noreferrer"&gt;IBM Instana&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hybrid&lt;/td&gt;
&lt;td&gt;SaaS uses default LLM gateways provided by Instana. Self-hosted installs &lt;a href="https://www.ibm.com/docs/en/SSE1JP5/src/pages/ai_gateway/llm_gateways.html" rel="noopener noreferrer"&gt;connect your own&lt;/a&gt; watsonx.ai or vLLM.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.traversal.com/blog/what-is-byoc" rel="noopener noreferrer"&gt;Traversal&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hybrid&lt;/td&gt;
&lt;td&gt;Runs in your cloud account (BYOC). BYOM can point it at your self-hosted model.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://docs.neubird.ai/platform/introduction/amazon-bedrockllm/" rel="noopener noreferrer"&gt;NeuBird&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hybrid&lt;/td&gt;
&lt;td&gt;SaaS by default. VPC installs need specific Claude and Llama models enabled in the customer's own Bedrock account.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://metoro.io/docs/on-premises/11.x/installation/metoro-ai.md" rel="noopener noreferrer"&gt;Metoro Guardian&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hybrid&lt;/td&gt;
&lt;td&gt;SaaS by default. On-prem installs point it at your OpenAI-compatible endpoint, a LiteLLM gateway included.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://docs.rootly.com/integrations/anthropic.md" rel="noopener noreferrer"&gt;Rootly&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hybrid&lt;/td&gt;
&lt;td&gt;AI incident workflows can use your own Anthropic or &lt;a href="https://docs.rootly.com/integrations/azure-openai" rel="noopener noreferrer"&gt;Azure OpenAI&lt;/a&gt; key. For the AI SRE agent itself, BYOK is &lt;a href="https://rootly.com/ai-sre-guide" rel="noopener noreferrer"&gt;on the roadmap&lt;/a&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://home.robusta.dev/" rel="noopener noreferrer"&gt;Robusta&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Hybrid&lt;/td&gt;
&lt;td&gt;Robusta-hosted models by default. You can add your own models and keys, and turn the hosted ones off completely.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://docs.opsramp.com/platform-features/feature-guides/copilot/getting-started/bring-your-own-llm/" rel="noopener noreferrer"&gt;HPE OpsRamp OpsPilot&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Customer-provided&lt;/td&gt;
&lt;td&gt;SaaS, but off until an admin adds LLM credentials. Supports Vertex AI, Bedrock, Azure Foundry and OpenAI, and recommends Claude Sonnet 4.6.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://openobserve.ai/ai-sre/" rel="noopener noreferrer"&gt;OpenObserve AI SRE&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Customer-provided&lt;/td&gt;
&lt;td&gt;Has a cloud SaaS, but the AI SRE runs on your own provider and key. Markets this as the difference from bundled assistants.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://holmesgpt.dev/" rel="noopener noreferrer"&gt;HolmesGPT&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Customer-provided&lt;/td&gt;
&lt;td&gt;Open source AI SRE agent (CNCF Sandbox). You configure the provider. Ollama is marked experimental with limited tool calling.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/Arvo-AI/aurora/wiki" rel="noopener noreferrer"&gt;Aurora&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Customer-provided&lt;/td&gt;
&lt;td&gt;Open source AI SRE agent. Hosted providers or Ollama, with an air-gapped install bundle.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/Tracer-Cloud/opensre" rel="noopener noreferrer"&gt;OpenSRE&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Customer-provided&lt;/td&gt;
&lt;td&gt;Open source AI SRE framework. Set the provider by env var, Anthropic by default, Ollama supported.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Observations
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Vendor-managed wins, and it follows the billing.&lt;/strong&gt; PagerDuty sells AI Actions and Datadog draws Bits from a credit pool, so running the model is part of the product they sell. HolmesGPT, Aurora and OpenSRE are open source with no inference bill to charge, so you must bring the model. Customers may also ask for their own model less often than you'd think. Cleric's CTO said they'd accept customer-brought models, but no customer has asked yet.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AIOps vendors with on-prem customers ship their own model instead of asking for yours.&lt;/strong&gt; Selector fine-tunes its network model on-prem at each customer. ScienceLogic runs its AI tier in sovereign cloud or on-prem. BMC has its fine-tuned Helix LLM. This makes sense for AIOps, where telcos, banks and government still run a lot on-prem and refuse to send events to a public API.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Bring your own" here mostly means bring your own cloud contract.&lt;/strong&gt; BigPanda's customer-hosted Azure, ServiceNow's BYOK, NeuBird's VPC install on your Bedrock, and every OpsRamp option are cloud providers. So the model still runs in a public cloud, just under the customer's account. Running the model on your own hardware is the exception, and it shows up mostly in on-prem installs like Metoro and Traversal and in open source agents that support Ollama.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Even required BYO comes with a model recommendation.&lt;/strong&gt; OpsRamp won't turn OpsPilot on without your credentials, but it still recommends Claude Sonnet 4.6. NeuBird lists exact model IDs to enable. OpenObserve is the one that turns bring-your-own into a selling point. Vendors let you hold the contract. They still want a say in which model sits behind their agent.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bring your own agent, not your own model
&lt;/h2&gt;

&lt;p&gt;A growing number of vendors skip the model question. They open their data to the customer's own agent over MCP, and the customer brings the agent and the model together. Most big names in the table now ship an MCP server. What differs is what they put behind it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Opening the raw data
&lt;/h3&gt;

&lt;p&gt;The simplest version gives the agent query access and lets it complete its tasks. Datadog's MCP server is now &lt;a href="https://aws.amazon.com/blogs/devops/production-ready-autonomous-incident-resolution-with-aws-devops-agent-now-ga-and-datadog-mcp-server/" rel="noopener noreferrer"&gt;generally available&lt;/a&gt;, and AWS DevOps Agent already plugs it in as a telemetry source. In that setup, the reasoning is done by AWS agent, and Datadog only supplies the data. It gives agents &lt;a href="https://docs.aigateway.cequence.ai/docs/remote-mcp-servers/datadog" rel="noopener noreferrer"&gt;logs, metrics, traces, monitors and incidents&lt;/a&gt;. But a few tools still call models hosted by Datadog's own providers, and you have to ask the support to turn those off.&lt;/p&gt;

&lt;p&gt;Splunk went further on control. &lt;a href="https://www.splunk.com/en_us/blog/artificial-intelligence/splunk-mcp-server-2-0.html" rel="noopener noreferrer"&gt;MCP server 2.0&lt;/a&gt; comes preinstalled in Splunk Cloud 10.6, admins get guardrails like an SPL allow list, and every agent search carries a provenance label into the audit index. Dynatrace offers a &lt;a href="https://docs.dynatrace.com/docs/dynatrace-intelligence/dynatrace-intelligence-integrations/dynatrace-mcp" rel="noopener noreferrer"&gt;remote MCP server&lt;/a&gt; too. Its open source local version warns that Grail queries are &lt;a href="https://llmbase.ai/mcp-servers/dynatrace" rel="noopener noreferrer"&gt;billed by GB scanned&lt;/a&gt;, and ships a per-session budget that defaults to 1,000 GB.&lt;/p&gt;

&lt;h3&gt;
  
  
  Opening the vendor's intelligence as a tool
&lt;/h3&gt;

&lt;p&gt;Others expose their AI results, not just data. &lt;a href="https://www.logicmonitor.com/platform/edwin-ai-mcp-model-context-protocol" rel="noopener noreferrer"&gt;LogicMonitor MCP for Edwin AI&lt;/a&gt; puts monitoring tools and Edwin's investigation tools behind one endpoint, with inherited RBAC, rate limits, audit logs, bounded time windows and capped payloads. LogicMonitor describes MCP as the &lt;a href="https://www.wwt.com/article/partner-pov-logicmonitor-mcp-for-edwin-ai-unified-governed-context-for-ai-agents" rel="noopener noreferrer"&gt;interoperability layer&lt;/a&gt;, while Edwin stays the intelligence layer. So your agent gets an investigation that LogicMonitor's model already produced.&lt;/p&gt;

&lt;p&gt;ServiceNow's &lt;a href="https://www.servicenow.com/community/now-assist-articles/action-fabric-mcp-server-mcp-client-and-a2a-explained/ta-p/3557794" rel="noopener noreferrer"&gt;Action Fabric&lt;/a&gt; does the same with Now Assist skills, the knowledge graph, subflows and scripted REST APIs. The MCP server went GA at Knowledge 2026 and is &lt;a href="https://www.techzine.eu/blogs/analytics/141272/the-ui-is-dead-long-live-the-agent-servicenow-goes-headless-and-opens-its-platform/" rel="noopener noreferrer"&gt;included in every Now Assist SKU&lt;/a&gt;, but headless calls burn the same Assist credits as the built-in assistant. BigPanda ships a &lt;a href="https://apis.io/providers/bigpanda/" rel="noopener noreferrer"&gt;remote MCP server&lt;/a&gt; for Biggy's action plans, plus an A2A endpoint.&lt;/p&gt;

&lt;h3&gt;
  
  
  Going both ways
&lt;/h3&gt;

&lt;p&gt;Some vendors also let their own agent call out. ServiceNow agents can act as &lt;a href="https://www.servicenow.com/community/now-assist-articles/enable-mcp-and-a2a-for-your-agentic-workflows-with-faqs-updated/ta-p/3373907" rel="noopener noreferrer"&gt;MCP clients&lt;/a&gt;, and over A2A they can lead an external agent or take orders from one. Komodor lets customers plug their own tools and agents into Klaudia &lt;a href="https://www.helpnetsecurity.com/2026/03/19/komodor-klaudia-ai-extensibility-framework/" rel="noopener noreferrer"&gt;via MCP or OpenAPI&lt;/a&gt;, next to its 50+ built-in agents. LogicMonitor lists collaboration with ServiceNow ITOM, Microsoft Copilot and customer MCP servers.&lt;/p&gt;

&lt;h3&gt;
  
  
  What this changes
&lt;/h3&gt;

&lt;p&gt;Bring-your-own-agent solves the model question only for raw data tools. Once your agent calls a vendor skill, like Edwin's investigation or a Now Assist skill, the vendor's model runs again, and your data goes with it. The meter doesn't go away either. It moves, to GB scanned at Dynatrace and to Assist credits at ServiceNow.&lt;/p&gt;

&lt;p&gt;It may also explain the Cleric quote above. Maybe customers rarely ask for their own model because what they really want is their own agent.&lt;/p&gt;

&lt;h2&gt;
  
  
  Vendor-managed
&lt;/h2&gt;

&lt;p&gt;Good for the customer who wants it to just work. Bad for the one with strict data rules.&lt;/p&gt;

&lt;p&gt;Pros:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Works the day you turn it on. No API key, no endpoint, no GPU.&lt;/li&gt;
&lt;li&gt;The vendor tests prompts and tool calls against models it knows, and can route each task to the model that does it best.&lt;/li&gt;
&lt;li&gt;One contract, one DPA. Zero data retention with the LLM providers is the vendor's job (BigPanda states it for its default OpenAI service).&lt;/li&gt;
&lt;li&gt;Pricing is a credit or an action count, not a token bill you have to forecast.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cons:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Your data goes to the vendor's subprocessor on the vendor's terms. Dynatrace, for example, keeps prompt logs in Azure US East for 35 days.&lt;/li&gt;
&lt;li&gt;No model choice. One LogicMonitor user &lt;a href="https://www.peerspot.com/questions/what-advice-do-you-have-for-others-considering-logicmonitor-570358" rel="noopener noreferrer"&gt;put it bluntly&lt;/a&gt;: why pay for more tokens when we already have Bedrock or Copilot?&lt;/li&gt;
&lt;li&gt;You can't decide who sees your data. &lt;a href="https://learn.microsoft.com/en-us/azure/sre-agent/data-privacy" rel="noopener noreferrer"&gt;Azure SRE Agent&lt;/a&gt; defaults to Anthropic outside the EU, and Anthropic is not covered by Microsoft's EU Data Boundary.&lt;/li&gt;
&lt;li&gt;Air-gapped or fully on-prem is basically off the table.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Customer-provided
&lt;/h2&gt;

&lt;p&gt;Bringing your own API key is not the same as self-hosting. With the key, your data still goes to AI companies, just under your contract instead of the vendor's. Only a model running inside your own network can keeps the data in. &lt;a href="https://www.alation.com/blog/bring-your-own-model-byom/" rel="noopener noreferrer"&gt;Alation's BYOM write-up&lt;/a&gt; makes the same point.&lt;/p&gt;

&lt;p&gt;Pros:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The data path sits under your own contract, region and retention settings.&lt;/li&gt;
&lt;li&gt;You reuse LLM spend you already committed, instead of paying for tokens twice.&lt;/li&gt;
&lt;li&gt;Self-hosting makes air-gapped setups possible, which some regulated sites require.&lt;/li&gt;
&lt;li&gt;You pick the model, and you decide when to upgrade it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cons:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Quality depends on your pick now. HolmesGPT says Ollama tool calling is limited and can give inconsistent results. OpsRamp names one recommended model for a reason.&lt;/li&gt;
&lt;li&gt;Vendor support gets thin past the defaults. HolmesGPT tells you to try a hosted model first and make sure everything works before switching to Ollama.&lt;/li&gt;
&lt;li&gt;The vendor tuned its prompts and evals on its own model. Your model might pass simple chat and still break multi-step agent features.&lt;/li&gt;
&lt;li&gt;Capacity, latency and GPU cost become your problem.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That third con is the one I'd watch most. A summary feature degrades gently with a weaker model. An agent that has to pick the right tool five times in a row does not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;p&gt;Everything comes from public docs, trust pages and vendor blogs, checked in early November 2026. "No BYO option found" means I didn't find one, not mean the vendor doesn't offer it. LogicMonitor and incident.io are the exceptions, since their docs say no outright. Some products have more than one AI feature, and I looked at the main assistant or agent only. I left out general observability and ITSM tools that don't sell event correlation or incident investigation. Docs in this space change every few months, so check the vendor page before you quote a row. If I got one wrong, tell me in the comments and I'll fix it.&lt;/p&gt;

&lt;p&gt;Each row in the table links to the page it comes from. Other pages used, by section:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What the split tells you and the trade-offs&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.dynatrace.com/company/trust-center/capability-specific-terms/" rel="noopener noreferrer"&gt;Dynatrace capability-specific terms&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.pagerduty.com/fr/product-specific-terms/" rel="noopener noreferrer"&gt;PagerDuty product-specific terms&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.dash0.com/comparisons/datadog-bits-ai-alternatives" rel="noopener noreferrer"&gt;Dash0: Datadog Bits AI alternatives&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.helixops.ai/bin/Service-Management/Employee-Digital-Workplace/BMC-HelixGPT/helixgpt263/Getting-started/Key-concepts/Models-in-BMC-HelixGPT/" rel="noopener noreferrer"&gt;Models in BMC HelixGPT&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://learn.microsoft.com/en-us/azure/sre-agent/data-privacy" rel="noopener noreferrer"&gt;Azure SRE Agent data privacy&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.peerspot.com/questions/what-advice-do-you-have-for-others-considering-logicmonitor-570358" rel="noopener noreferrer"&gt;PeerSpot: LogicMonitor user review&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.alation.com/blog/bring-your-own-model-byom/" rel="noopener noreferrer"&gt;Alation: Bring Your Own Model&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Bring your own agent&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://aws.amazon.com/blogs/devops/production-ready-autonomous-incident-resolution-with-aws-devops-agent-now-ga-and-datadog-mcp-server/" rel="noopener noreferrer"&gt;AWS: DevOps Agent GA with Datadog MCP Server&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.aigateway.cequence.ai/docs/remote-mcp-servers/datadog" rel="noopener noreferrer"&gt;Cequence AI Gateway: Datadog MCP&lt;/a&gt; (third-party)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.splunk.com/en_us/blog/artificial-intelligence/splunk-mcp-server-2-0.html" rel="noopener noreferrer"&gt;Splunk MCP Server 2.0&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.dynatrace.com/docs/dynatrace-intelligence/dynatrace-intelligence-integrations/dynatrace-mcp" rel="noopener noreferrer"&gt;Dynatrace MCP server docs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://llmbase.ai/mcp-servers/dynatrace" rel="noopener noreferrer"&gt;Dynatrace local MCP server README&lt;/a&gt; (third-party mirror)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.logicmonitor.com/platform/edwin-ai-mcp-model-context-protocol" rel="noopener noreferrer"&gt;LogicMonitor MCP for Edwin AI&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.wwt.com/article/partner-pov-logicmonitor-mcp-for-edwin-ai-unified-governed-context-for-ai-agents" rel="noopener noreferrer"&gt;WWT: LogicMonitor MCP partner POV&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.servicenow.com/community/now-assist-articles/action-fabric-mcp-server-mcp-client-and-a2a-explained/ta-p/3557794" rel="noopener noreferrer"&gt;ServiceNow: Action Fabric explained&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.servicenow.com/community/now-assist-articles/enable-mcp-and-a2a-for-your-agentic-workflows-with-faqs-updated/ta-p/3373907" rel="noopener noreferrer"&gt;ServiceNow: MCP and A2A FAQ&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.techzine.eu/blogs/analytics/141272/the-ui-is-dead-long-live-the-agent-servicenow-goes-headless-and-opens-its-platform/" rel="noopener noreferrer"&gt;Techzine: ServiceNow goes headless&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://apis.io/providers/bigpanda/" rel="noopener noreferrer"&gt;apis.io: BigPanda&lt;/a&gt; (third-party)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.helpnetsecurity.com/2026/03/19/komodor-klaudia-ai-extensibility-framework/" rel="noopener noreferrer"&gt;Help Net Security: Komodor Klaudia extensibility&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>aiops</category>
      <category>llm</category>
      <category>agents</category>
    </item>
    <item>
      <title>Keyword queries were where keyword search lost</title>
      <dc:creator>Linhao Liu</dc:creator>
      <pubDate>Thu, 20 Aug 2026 01:35:08 +0000</pubDate>
      <link>https://dev.to/p311/keyword-queries-were-where-keyword-search-lost-55l8</link>
      <guid>https://dev.to/p311/keyword-queries-were-where-keyword-search-lost-55l8</guid>
      <description>&lt;p&gt;&lt;em&gt;A "lexical query" is not the same thing as a query that needs lexical matching.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;My retrieval benchmark sorts every query into two groups: ones that contain an identifier — a file path, a version number, an error code — and ones that don't. Queries with identifiers are supposed to be the easy case for keyword search. They were the case where it lost by the widest margin.&lt;/p&gt;

&lt;p&gt;The reason: &lt;strong&gt;for 54% of those queries, the identifier never appears in any of the documents that answer them.&lt;/strong&gt; There is nothing for exact matching to match against.&lt;/p&gt;

&lt;p&gt;The usual advice says keyword search handles keyword queries, dense retrieval handles the rest, and hybrid gives you both. That advice is why most people run hybrid search. None of it held up here, and the part I was confident about is the part that broke worst.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;BEIR / CQADupStack, the &lt;code&gt;unix&lt;/code&gt; and &lt;code&gt;android&lt;/code&gt; subforums: 70,380 documents and 1,771 judged queries. The task is finding duplicate questions — given a Stack Exchange question, find the earlier question that asks the same thing.&lt;/p&gt;

&lt;p&gt;Before any retrieval runs, a classifier labels each query lexical or semantic. It only looks for lexical signals: something shaped like an identifier, or text in backticks. Semantic is just everything left over. There's no equally reliable way to look at a query and tell that it needs meaning-based matching, the way you can tell that a string looks like a hex literal. 214 queries end up labeled lexical, 1,557 semantic.&lt;/p&gt;

&lt;p&gt;The BM25 side uses Porter stemming and identifier-aware tokenization. Anserini's Lucene analyzer, which produced the published BEIR numbers I compare against, stems by default, and the dense side gets that kind of robustness for free from how it tokenizes. The unstemmed numbers are in the repo as a control.&lt;/p&gt;

&lt;h2&gt;
  
  
  The result
&lt;/h2&gt;

&lt;p&gt;Hybrid belongs in that table too, since the advice this post argues with is partly about hybrid:&lt;/p&gt;

&lt;p&gt;Recall@3, macro-averaged&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                              overall   lexical   semantic
BM25 (identifier-aware
      + Porter)                0.2255    0.3070     0.2143
Hybrid RRF (equal weight)      0.2964    0.3952     0.2828
Hybrid RRF + Porter            0.3173    0.4076     0.3049
Dense (all-MiniLM-L6-v2)       0.3834    0.4842     0.3695
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;Equal-weight RRF lands between its two inputs and below dense on its own. Dense beats the stemmed fusion by +0.0660 R@3 overall — 257 queries better against 107 worse, p = 2.3e-15 — and it wins both layers (+0.0766 lexical, +0.0646 semantic). Blending a 0.23 system with a 0.38 system on equal terms costs more than it gains.&lt;/p&gt;

&lt;p&gt;The direction is stable when the two subforums are run as separate datasets: all 108 signs hold. Two confidence intervals don't clear zero, both for dense against stemmed fusion on android's lexical layer, where n drops to 96 (R@3 +0.0568, p = 0.093; MRR +0.0478, p = 0.281). That's a power limit rather than a reversal — the same cell was already borderline before stemming — but it's the one place the merged result isn't independently confirmed.&lt;/p&gt;

&lt;p&gt;Weighting fixes most of the fusion gap. Giving BM25 only 10-20% of the score edges past dense (R@3 0.3894 vs 0.3834), and the lexical layer gains the most (0.5003 vs 0.4842). But that alpha was chosen on the same queries the number is reported on, so it's an optimistic ceiling rather than a validated gain. The defensible reading: tuned hybrid is roughly a wash against dense on this corpus, and equal-weight RRF is a bad default.&lt;/p&gt;

&lt;p&gt;Dense wins overall, which is no surprise — that roughly matches the published BEIR ordering, and the dense nDCG@10 of 0.4071 is close enough to the reference number for this model that I trust the harness.&lt;/p&gt;

&lt;p&gt;The surprise is where it wins. Dense beats BM25 by &lt;strong&gt;+0.18 on the lexical layer&lt;/strong&gt; and +0.16 on the semantic one. The gap is bigger on exactly the queries full of file paths and version strings. A paired bootstrap over 10,000 resamples keeps the lexical interval clear of zero, and a sign test on the overall comparison gives 429 wins against 74 losses, p = 8.1e-62. It also holds when I run the two subforums as separate datasets instead of one merged pool (unix +0.2202, p = 3.4e-07; android +0.1940, p = 2.2e-05), so it isn't an artifact of merging them.&lt;/p&gt;
&lt;h2&gt;
  
  
  Why
&lt;/h2&gt;

&lt;p&gt;That 54% explains the whole thing. BM25 can only match &lt;code&gt;xorg.conf&lt;/code&gt; against documents that literally contain &lt;code&gt;xorg.conf&lt;/code&gt;, and in a corpus of duplicate questions, most documents that do are about somebody else's unrelated problem. The question that actually duplicates yours describes the same situation but never types the path — or writes &lt;code&gt;/etc/X11/xorg.conf&lt;/code&gt;, or&lt;br&gt;
a different version number, or leaves it out. Worse, a rare token gets a very high IDF, so on the rare occasion it does match, it dominates the score and pushes up a document whose only connection to the query is that one string. The bi-encoder ignores the identifier entirely and matches on the problem description around it, which the duplicate really does&lt;br&gt;
share.&lt;/p&gt;

&lt;p&gt;So the classifier does exactly what it was written to do, and still measures the wrong thing. It tells you whether a query &lt;em&gt;contains&lt;/em&gt; something identifier-shaped. I had been reading it as "this query needs exact matching." Those turn out to be different questions,&lt;br&gt;
and on this task the answers differ more than half the time.&lt;/p&gt;

&lt;p&gt;How far this goes: finding duplicate questions is close to the worst case for the assumption that identifiers are shared, because the query and the target are two different people describing one problem in their own words. In log search or code search the identifier is shared by construction, and I'd expect the usual advice to hold there. So the finding isn't "dense beats BM25 on lexical queries." It's that a query &lt;em&gt;containing&lt;/em&gt; keywords and a query &lt;em&gt;helped by&lt;/em&gt; keyword matching are two different things — and if you're routing between retrievers based on what a query looks like, you're assuming they're the same thing.&lt;/p&gt;

&lt;p&gt;The useful version: &lt;strong&gt;how often a query's identifiers show up in its own correct answers is something you can measure on your corpus before building anything&lt;/strong&gt;, and it tells you whether a keyword arm is worth having.&lt;/p&gt;
&lt;h2&gt;
  
  
  The check that made this believable
&lt;/h2&gt;

&lt;p&gt;A benchmark that calls every difference real is worthless. So I pointed the same statistics at a comparison where I expected to find nothing.&lt;/p&gt;

&lt;p&gt;That comparison was my identifier-aware tokenizer. It emits everything the plain one does, plus the full identifier — strictly more information, so it should only ever help, but in fact it didn't. On the lexical layer it landed 0.0125 &lt;em&gt;below&lt;/em&gt; the plain tokenizer, with a confidence interval spanning zero and only 10 of 214 queries changing at all. p = 0.75. No&lt;br&gt;
effect, and the same non-result shows up in both subforums separately.&lt;/p&gt;

&lt;p&gt;That's what makes the p = 8.1e-62 above worth anything. The tests can tell a real difference from an imagined one.&lt;/p&gt;

&lt;p&gt;Three limits on all of this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The bootstrap resamples &lt;strong&gt;queries&lt;/strong&gt;. It covers variation in which questions I happened to test, not in the model, the preprocessing, or the relevance judgments.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One embedding model, one corpus, one task.&lt;/strong&gt; The 54% is a fact about the dataset that no choice of model would change, but everything built on top of it was measured on a single setup.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;These comparisons came after the fact.&lt;/strong&gt; They show the ordering here isn't luck. They don't show it transfers anywhere else.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The repo has the rest of it: why equal-weight RRF ends up worse than dense alone, a tokenizer change that quietly cost recall, a leakage audit of CQADupStack, and the per-subforum replication. Code, full per-layer metrics, and the significance script&lt;br&gt;
&lt;/p&gt;
&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/P311" rel="noopener noreferrer"&gt;
        P311
      &lt;/a&gt; / &lt;a href="https://github.com/P311/hybrid-rag-eval" rel="noopener noreferrer"&gt;
        hybrid-rag-eval
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;hybrid-rag-eval&lt;/h1&gt;
&lt;/div&gt;

&lt;p&gt;This project compares BM25, dense, and hybrid retrieval on CQADupStack, with every metric split by query type — lexical (identifier-bearing) vs semantic. BEIR already shows dense beats BM25 overall; splitting by layer shows it wins by an even wider margin on identifier-bearing queries.&lt;/p&gt;

&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;Setup&lt;/h2&gt;
&lt;/div&gt;

&lt;div class="highlight highlight-source-shell notranslate position-relative overflow-auto js-code-highlight"&gt;
&lt;pre&gt;./scripts/download_data.sh     &lt;span class="pl-c"&gt;&lt;span class="pl-c"&gt;#&lt;/span&gt; ~65 MB into data/cqadupstack/&lt;/span&gt;
python -m pytest tests/ -q
python run_eval.py             &lt;span class="pl-c"&gt;&lt;span class="pl-c"&gt;#&lt;/span&gt; BM25 rows only, no dependencies&lt;/span&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;p&gt;The BM25 rows, fusion and the metrics are standard library only. Dense
retrieval and reranking import &lt;code&gt;sentence-transformers&lt;/code&gt; optionally; when it is
unavailable those three rows are reported as skipped and the BM25 rows still
complete. To run the full table:&lt;/p&gt;
&lt;div class="highlight highlight-source-shell notranslate position-relative overflow-auto js-code-highlight"&gt;
&lt;pre&gt;python -m venv .venv
.venv/bin/pip install --index-url https://download.pytorch.org/whl/cpu torch
.venv/bin/pip install sentence-transformers pytest
.venv/bin/python run_eval.py&lt;/pre&gt;

&lt;/div&gt;
&lt;p&gt;To reproduce the significance tables below (paired bootstrap + sign test on the
paired comparisons, permutation test on the layer gap, all metrics, all layers)
run with the venv interpreter so…&lt;/p&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/P311/hybrid-rag-eval" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;

&lt;p&gt;.&lt;/p&gt;

</description>
      <category>rag</category>
      <category>ai</category>
      <category>nlp</category>
      <category>programming</category>
    </item>
  </channel>
</rss>
