DEV Community: Michelle Sebek

Two cats, four vendors, and the model the AI couldn't find

Michelle Sebek — Mon, 10 Aug 2026 00:14:31 +0000

Tech Talks went dark for a few months, and on episode 13 I finally got to ask why. Mathias Palmersheim's answer, delivered completely straight, was that his users were unhappy with the availability and usability of their feeders and their litter box, and he wasn't allowed back on stream until that got fixed.

The users are two dogs and two cats, and they have titles. Maisie, a Shiba Inu who came to him through a rescue, is the recently promoted chief executive pawofficer. Millie is head of /dev/null as a service, on the grounds that she's a void that consumes everything and somehow never gains an ounce. Max is doing his best. Moxxie is the chaos engineer, and she's the one who matters later.

This was my first tech talk since joining VictoriaMetrics. I came in expecting a cute demo and got a better argument about alert design than I've heard from most people talking about production systems.

The solutions engineering team lives in Home Assistant
Mathias's counterpart Zakhar has a coffee maker wired into Home Assistant, and Home Assistant knows when he's actually awake. Most coffee makers let you set an alarm, so 7:45 on the dot, every day, including Saturday, and by the time you get downstairs the burner has been cooking it for an hour. Zakhar's coffee maker brews when he's up and moving instead. Stephen, another teammate, went further and built himself an Iron Man wake-up routine, where the lights come up gradually as a gentle alarm and he gets a report on the day when he opens his eyes. Nobody on that team is getting jolted awake by the same song every morning.

I mention this because it explains the rest of the talk. These are people who use this stuff at home for reasons that have nothing to do with work, which is why the pet project isn't really a toy.
Many vendors, one pane of glass, and a hard rule about the cloud
The device shopping came with a constraint that'll be familiar to anyone who's worked a support rotation through a cloud outage. Mathias wanted his lights to come on when us-east-1 is having a bad day, and he was blunt about why: that's exactly when he's busiest helping customers whose own systems are down. So every device had to be controllable over the LAN, which knocks out a lot of otherwise nice hardware.

No single vendor makes sensors for everything, either. PetLibro does the fountains and feeders, Whisker makes the Litter-Robot, Roborock cleans up afterward, and TP-Link and Aqara handle the lights and switches. Four vendors, four apps, four data models, and a homeowner who does not want to open four apps to find out if the kids left the lights or if someone forgot to take of the animals..

Home Assistant is the glue that holds all of this together. Sensors push to it, or it polls a cloud API for the few devices that only speak cloud. Then comes the part I didn't expect. Mathias called Home Assistant his favorite Prometheus exporter , and he meant that literally: the Prometheus integration publishes everything Home Assistant knows at /api/prometheus, behind a long-lived access token.

After that it's an ordinary scrape. He uses Telegraf rather than a scrape job, which needs the format declared explicitly because Telegraf speaks several, plus a field drop filter doing the same job as a relabeling rule. Home Assistant emits state-change counters and metadata he has no use for, and that metadata is high cardinality, so he drops it at the source rather than storing it and filtering later. Telegraf pushes into a single-node VictoriaMetrics, Grafana reads from there.

Why bother, when Home Assistant already has dashboards? Because long ranges are awkward in them, and comparing a few metrics over the same window is worse. In Grafana he pulled up 90 days of cat weight, and Moxxie's line is drifting up. She seems to be getting ready for winter despite it being 100 degrees outside where she lives.
The alert that clears itself
The second reason is a design argument, and it's the part of the talk I'd repeat to people building alerting for a team.

Home Assistant has a powerful automation engine and it is amazing for things like turning on the lights when a motion sensor is tripped, but for being notified of an issue that requires human intervention like filling an empty water bowl or a dealing with a full litterbox Mathias wanted something similar to an incident response tool that update or delete notifications once they are resolved so he didn’t have to check dashboard to see if someone else had already resolved the issue. The other thing that home assistant notifications were lacking was the ALERTS metric which is how vmalert tracks which rules are in a pending or firing state over time. This metric is the backbone of the alert statistics dashboard maintained by Victoriametrics which Mathias uses to show to his stakeholders/pets to prove they are being well taken care of so they will allow him to continue doing tech talks

So the metrics live in VictoriaMetrics, vmalert evaluates alerting rules and sends alerts, to Alertmanager, Alertmanager posts a webhook to a self-hosted ntfy server, and everyone in the house gets one push. He admitted to using AI to write a small service for the deleting notifications out of ntfy after alertmanager resolves them. If you'd rather not write that glue, Alertmanager also supports email, Discord, Telegram, and more or less every incident response tool there is.

The alert we watched was called Thicc Kitty. Mathias had quietly moved the threshold down to 10.5 pounds so it would trigger during the demo, and I caught him, because the dashboard right next to it said 11. He copped to it. The notification lands on his phone with a link straight back to the Grafana panel, so at least the alert about his cat's weight is actionable.
Thresholds in a git repo, edited by someone who doesn't use an IDE
This is the trick I'd steal outright, and it's one flag.

vmalert loads rules from plain files, Kubernetes resources, or an HTTP URL. Mathias points it at a git repo over HTTP. That means the threshold for Thicc Kitty is one line in a file with a web editor sitting in front of it, and his wife doesn't have to live in an IDE to change it. She opens the repo in a browser, changes 10.5 to 11.25, commitsit, and vmalert picks it up on its next poll, once a minute in his setup, and we sat through that minute on stream twice waiting for the alert to resolve. There was a request for dramatic music.

Grafana shows these as data source managed alerts by querying the vmalert API through the data source config. If you'd rather skip Grafana, setting -vmalert.proxyURL also surfaces the same rules and states in VMUI. Stephen described the route he prefers in chat. He applies the operator's VMRule resource to the cluster, which writes the rules out as a plain file with the same hot reload, and Argo CD pulls every minute. Browser tab or GitOps, same idea.

And the idea is worth saying plainly, because it generalizes past cats. The threshold is the piece of an alerting rule most likely to be wrong and least likely to need an engineer to fix. Put it somewhere the person who actually knows the right number can reach it. In this house that person is his wife, relaying what the vet said. At work it's probably your application/service owner and either way you've taken yourself out of the loop for the change that happens most often.
Where static thresholds Fall short
The vet gave him a number for weight. Above 11.25 pounds, Moxxie needs more play, less food, or different food. That's the easy case, and it's rarer than you'd think.

The vet did not give him a number for litter box visits. Too few can mean the cat is backed up or can't get to the box. Too many can point at an infection. Both directions are bad, neither has a published figure, and the normal range depends on the individual animal. Mathias's summary was that cats are too chaotic for static thresholds most of the time, which is how Moxxie ended up with the chaos engineer title.

He built the query as a 24-hour max, averaged to smooth it, one series per cat. Just eyeballing the graph, Millie had a couple of multi-day stretches with no visits at all, and Moxxie had a spike. Catch both of those with a rule that doesn't also catch normal variation and you've written the kind of interconnected rule set nobody on the team will touch a year from now.

vmanomaly is the answer here, and Fred Navruzov, who leads it, sat down with Mathias before the stream to tune the model. They used Temporal Envelope, an in-house model added in v1.30.0 that learns an expected range from trend, calendar patterns and holidays, and keeps adapting during inference without letting one spike bend it permanently. Z-score and median absolute deviation don't handle seasonality at all. Prophet handles seasonality, but does not support online updates to the model, but Temporal envelope supports both seasonality and online model updates.

Tuning is where your own knowledge goes in, and the parameters stay readable even when the statistics behind them don't. The ones they reached for:

detection_direction, set to both here, since either extreme means something. For request latency you'd set it to above only. Mathias put it better: nobody has ever been upset that a website loaded too fast.
clip_predictions pins the model to a plausible range. The floor is 0.001, because you can't use a litter box a negative number of times, but using the litter box 0 times for 2 days should still be flagged as an anomaly
min_deviation_from_expected throws out deviations too small to care about.
Seasonality presets handle hour-of-day and day-of-week shapes, and you can encode holidays directly, which is how you stop Black Friday from reading as an incident.
A streaks setting makes a condition persist before it counts, so one lonely spike doesn't wake anybody up. (Check the exact parameter name in the docs before you copy that.)

The UI plots the model against your real data while you turn these knobs, so you watch the expected range move. Once it looked right they exported it, either as a full standalone config with server info and queries or as just the model block. Mathias took the model block, since he already had vmanomaly running.
The uncomfortable part
Fred and Mathias needed about an hour. I asked how long the whole build took, and he raised the harder question himself: what do you do if you don't have a Fred (our product lead)?

vmanomaly ships an assistant in the UI for exactly that. It takes any OpenAI-compatible endpoint, Anthropic compatible endpoints, Open router, or Amazon Bedrock, and the model behind those endpoints can drive the vmnaomaly ui. In this case Mathias and Fred were running Claude Fable 5. He typed a plain-English prompt asking for a model that uses streaks to spot litter box use that's too frequent or not frequent enough, approved the tool calls, and we watched it drive the interface with results streaming back.

It picked the Prophet modefor everything, and Mathias recognized the failure on sight because he and Fred had already hit it while building the agent skills. The context the model was reading from pushed it toward the Prophet model no matter what the data looked like, and he'd forgotten to update his skills before going live. Mathias also had forgotten to add in the MCP server URL for the vnanomaly assistant so the assistant was working with one hand tied behind it’s back. This misconfiguration lead to the vmanomaly assistant believing it didn’t have access to the new temporal_envelope model so the assistant settled for the seasonal quantile model instead. The seasonal quantile model landed reasonably close, caught the spike and the trough, and never found Temporal Envelope at all an issue that might be addressed in next months tech talk
Here is what I did not expect on a vendor livestream. Mathias said it out loud: our expertise is better than what you can get with just Claude, and we confirmed that today.

Even after Mathias had fixed the MCP issue, and updated his skills after the stream, the temporal envelope model that Fable came up with was still a bit worse than what Mathias and Fred had built without the assistant. Chat sharpened it further. Mark pointed out you should define the agent's role and its guard rails before you ask it anything, which is the fix for a model confidently grabbing the wrong tool. Stephen pointed out that a UI showing you what the model is doing makes the model usable by almost anyone. He's right, and it's why the wrong answer was obvious inside of a minute instead of after a week of bad alerts. An assistant that hands you a config you can't inspect is a much riskier proposition than one whose output you can see drawn over your own data.
What it costs to run
Metrics, logs, traces, vmanomaly, and all the glue between them run on two CPUs and four to six gigabytes of RAM in his lab. Machine learning without a rack of H100s in the basement, as he put it. If you do have a rack of servers in the basement or more likely a datacenter that needs watching, the same stack scales with Kubernetes operator and the cluster version.

Most of what's above is the open source core. vmanomaly is part of our enterprise offering and needs a license key, and it reads from and writes to whichever VictoriaMetrics you're already running, open source or enterprise, with vmalert and Grafana untouched. There's a trial license if you want to try the litter box thing at home.

The whole thing, tuned and shipped, took him under 90 minutes.

We'll have a booth at We Are Developers at the end of September, All Things Open in Raleigh in October, and KubeCon in November. There's a Google Form in the video description for suggesting topics, and I read everything that comes to michelle@victoriametrics.com. If you've found a use for VictoriaMetrics we haven't thought of, send it.

The most expensive half-hour of an incident

Michelle Sebek — Mon, 10 Aug 2026 00:00:24 +0000

It's not the outage, it's the stretch before you know what actually broke

In short: VictoriaMetrics Enterprise support is expertise, not a ticket queue. It's reactive by design (you reach engineers who know the stack when something breaks), with one proactive service, Monitoring of Monitoring, that watches the health of your VictoriaMetrics observability stack (metrics, logs, and traces). Because reliability and cost are usually the same event, the same expertise that shortens an incident also helps you run leaner.
The difference between a stack that runs calm and one that runs hot usually isn’t the software. It’s knowledge. Why VictoriaMetrics Enterprise support is the most underrated line item in your budget.

It’s 3 am and your phone won’t quit. By the time you’re at the laptop, you already know how the night will unfold: something’s wrong, it’s yours to find, and every minute you spend finding it is costing something. Ask an SRE what they dread, and you’ll hear two things in the same breath: that call, and the cap.
The call, everyone gets. The cap takes more explaining. Most SREs don’t actually lose sleep over the invoice, because the invoice is the boss’s number. What lands on them is the ceiling: a hard spending limit someone above them set, with a standing order not to cross it. So they do the quiet, unglamorous things that keep them under it. They sample, storing one datapoint every few minutes instead of every few seconds. They trim retention, tossing last quarter’s history to make room for this week’s. It works. It also blinds the system they’re on call to keep alive.
That ceiling gets more attention every year. Observability spend is its own FinOps line now, tracked like cloud compute, and on some teams it’s quietly outgrown the infrastructure it’s watching. It’s on the short list of things engineering leaders worry about, right next to complexity and noise. So where does all that money go? High-water-mark billing runs two to three times over plan. Usage-based pricing turns a clean budget into a 30 to 50% surprise at renewal. And nearly every major vendor now sells a “cost-control” add-on to manage a problem of their own making.
We think about this differently, starting with what we even mean by “support.”

Picture two teams the same size, running the same VictoriaMetrics on the same kind of workload. One stays calm and barely thinks about its monitoring. The other firefights, over-buys capacity, and watches the bill climb. Same software. So what actually separates them? Knowledge. And that’s really what you’re buying from us: not the support contract, but the knowledge that decides whether the same stack runs calm or runs hot.

How does expert support shorten an incident?
Think back to your last bad incident. Afterward, your team did the responsible thing and built a dashboard for exactly this scenario. Clean panels, careful labels, everything in its place. Now it’s happening again; it’s your turn on call, and you’ve been staring at that dashboard for half an hour. Can you tell whether the five unhealthy panels are five problems or one problem showing up five ways? Not yet. You’ll get there. You always do. But that half hour is the expensive part.
There’s a tell in here nobody says out loud. After every incident, the dashboard grows. We keep bolting on panels because the dashboard is standing in for knowledge we didn’t have when it counted.
Someone who knows the internals can often narrow the problem faster, because they know which signals to check first. The dashboard isn't the problem; the same panels read differently to the person who has seen this failure mode a hundred times before. The more data, metrics, and dashboards you have in place, the faster that person gets you to the answer.
So that’s the product. It’s people who know how VictoriaMetrics behaves under real load and at your scale, and who shorten the ugly part of an incident: the stretch where you’re still figuring out what actually broke. To be clear, it isn’t us watching your production for you. Nobody’s staring at your dashboards at 3 am but you. What you get is the person who knows what to look for when you call, and everything you've instrumented makes that call shorter.

Why are observability, reliability, and cost the same problem?
Why does this matter for both the 3 am call and the cap? Because they’re usually the same event. A cardinality blowup is an incident and a cost overrun at once. A retention setup that was fine at 100 nodes but falls over at 1,000 is a reliability and budget risk. A bad query during an incident burns your SLO and your compute in one go.
So the same knowledge cuts both ways. Bring our engineers in for an architecture review, a capacity plan, or a cardinality pass, and that one engagement works both sides at once: the failure modes that would wake you up, and the waste that shows up on the invoice. (Those reviews cover your VictoriaMetrics components, and the proactive, telemetry-driven ones depend on your tier and on having that telemetry in place.)
One honest caveat. We can’t promise you a number. What a deployment costs comes down to your retention, cardinality, workload, and architecture, and those are yours to own. What we can do is make sure the expertise that shortens an investigation is the same expertise that keeps you lean. Much of what gets labeled “capacity planning” is really uncertainty: resources provisioned “just in case.” People who know the system can size it with less guesswork, so you're not paying for headroom you'll never touch.

Efficiency is built in, not bolted on
One distinction is worth being precise about. Most of the market sells cost control as a premium layer on top of an already-expensive base: Adaptive Metrics, Streama, consumption-tier discounts. You pay more to stop overpaying.
VictoriaMetrics runs the other way. The efficiency is in the open-source core by design, [up to 7x less RAM] and up to 7x less storage than Prometheus, Thanos, or Cortex once you’re into millions of active time series, and we don’t lock it behind a paid tier. It isn’t a slide-deck number, either. It’s why Grammarly cut its observability cost 10x by moving to VictoriaMetrics, and why DreamHost cut its memory footprint 80% while scaling to 76 million active series. It holds at the top end too: Roblox runs it behind 200M+ monthly users, and Spotify at 78 million data points a second. Whatever you’re growing into, someone has already run it there, and the people who’d support you have run it there too.

Independence is cost stability
One more thing the last two years made obvious: durability is a feature now. Splunk went to Cisco. New Relic and Sumo Logic went to private equity. In January 2026, Palo Alto closed a $3.35 billion deal for Chronosphere. Run down the shortlist you’d have built two years ago, and almost every name has a new owner, and new owners re-price. There’s always someone upstairs deciding your renewal should be a more “strategic” number.
VictoriaMetrics is self-funded, profitable, genuinely open source, and it runs anywhere. You work with a stable, independent vendor whose incentives are far less likely to be reset by an acquisition. So there's less exposure to the usual post-acquisition playbook: a new owner re-pricing your renewal, pushing a migration, or tightening the lock-in. If you're defending a budget a year out to a CFO who's already watching this line, that kind of stability is worth as much as the efficiency.

Who monitors the monitoring?
Everything so far is reactive by design, and we said so on purpose. But there's one thing we do watch, and it's the thing that scares people most. What happens when the monitoring itself goes down? When an incident is bad and your time series database picks that moment to stop answering, you're blind exactly when you can least afford to be.
So we built a service for it. Monitoring of Monitoring (MoM) is a paid option where our core team keeps an eye on the health of your VictoriaMetrics components and flags trouble early, before it turns into an outage. Setup is one line in your metrics pipeline: you send us the health metrics of your VM components, and that is all it ever touches. No business data, no customer data, just the monitoring's own vital signs.
From there we watch for anomalies, catch setups drifting toward trouble, and reach you over email, Slack, or your usual alerting when something needs attention or could simply run better. This is the part most "premium support" tiers only gesture at. For us it's a real service with a real boundary: MoM covers your VictoriaMetrics components, not your whole production estate, and it's open to open-source users and Enterprise customers alike. Think of it as the proactive complement to everything above. One service keeps watch on the monitoring; the rest of support is there the second anything else breaks.

Net-net
We’re not the biggest vendor in this space, and we’re not trying to outspend the ones who are. We compete on cost-per-value, because that’s where this market is most exposed and where our efficiency is most real.
So think of the support less as a safety net and more as the reason the software keeps its promises. It’s the deepest VictoriaMetrics expertise anywhere, working your problem fast and helping you run the thing more reliably and leaner at the same time.
By engineers, for engineers. Simple, reliable, and efficient observability for everyone.
Want to see the math on your own deployment? Talk to our team about what Enterprise expertise and a leaner and more predictable bill could look like for you.

*Common questions *

Is VictoriaMetrics Enterprise support proactive or reactive?
Reactive by design. You reach the team through the portal or email when something breaks or a bill spikes. The one proactive service is Monitoring of Monitoring (MoM).

What is Monitoring of Monitoring (MoM)?
A paid service where the VictoriaMetrics core team monitors the health of your VictoriaMetrics components, using health metrics you send, and flags issues early. It covers your VM components, not your whole production estate.

Does Enterprise support guarantee a lower bill?
No. What a deployment costs depends on your retention, cardinality, workload, and architecture. Support gives you the expertise to run leaner; the efficiency itself is built into the open-source core.