When Sentiment Dashboards Lie: Tracking the Product Anomalies That Averages Mask

Umaima Fatima — Tue, 29 Sep 2026 12:06:41 +0000

Three days after the 4.2 release, login complaints spiked threefold.
The support tickets started slow on Tuesday morning, but by Friday, the entire queue was consumed with issues. "Can't sign in after the update" was a refrain that appeared dozens of times in a single day's tickets. However, at the same time, our standard sentiment dashboard showed only a minor drop from 0.74 to 0.66 - a still-positive score that masked the severity of the problem.
While the average metrics suggested stability, our AI agent was able to zero in on the relevant subset of tickets and identify the authentication middleware change as the root cause
The key difference was that we built an agent capable of retaining a chronological memory of relevant feedback and detecting anomalous sentiment spikes associated with specific product surface areas.
Let's explore the problems with the status quo and how we built our temporal memory backbone.

The Noise Overload and the Signal Swamps

The product and data teams receive an enormous volume of signals from tweets, Zendesk tickets, in-app NPS comments, and Reddit posts. Over a few months, this creates a chaotic signal noise that obscures important events.
A large regression will create a cluster of similar negative signals from affected users, but since the remainder of your users are happy, these negative signals will appear as an inconsequential "blip" in your overall metrics.
By the time a human analyst notices the cluster, the relevant release notes are old history, and the engineering team has moved on to other priorities.
This is why the ideal agent needs to preserve a chronological memory of relevant feedback, be able to detect anomalous topic density increases, and understand how these relate to the product's deployment history.

Introducing the Hindsight Temporal Memory

We instrumented our agent with a temporal memory module built on top of Hindsight.
Hindsight differs from standard vector databases in that it provides first-class support for temporal and entity-based organization of information, which we leverage to create chronological memory banks for different product surfaces:
from hindsight_client import Hindsight
client = Hindsight(base_url="http://localhost:8888")
client.create_bank(
bank_id="feedback-login-surface",
name="Login & Auth Feedback",
mission="Chronologically log authentication-related feedback and identify the root causes behind abrupt negative shifts."
)
Every cleaned feedback item is stored in the bank with its original timestamp:
def store_feedback(item):
client.retain(
bank_id="feedback-login-surface",
content=item["body"],
timestamp=item["created_at"],
context=f"channel={item['channel']}",
metadata={
"source_id": item["id"],
"user_segment": item.get("plan"),
"channel": item["channel"]
},
document_id=item["id"]
)
When we need to investigate a potential issue, we can ask the memory bank for relevant past items:
memories = client.recall(
bank_id="feedback-login-surface",
query="login or sign-in failures after the 4.2 release",

budget="high"
)
For more involved investigations, we use the reflect primitive and provide our deployment history as context:
diagnosis = client.reflect(
bank_id="feedback-login-surface",
query="Has sentiment around login changed since deploy 4.2? What is the most probable cause?",
context="Deploy 4.2 (2025-09-02) included auth-middleware-refactor and session-token-rotation changes."
)
By collapsing similar user complaints into evidence clusters while retaining the original timestamps and source references, we enable the agent to determine both when and why an issue emerged, rather than simply what the issue is.

Time Series Sentiment Analysis for Spikes Detection

Rather than relying on simple polarity scores, we analyze the evolution of sentiment over time. Specifically, we use a HDBSCAN clustering algorithm to detect abrupt increases in the density of similar feedback items that occurred within a short time window.
Here is a highly simplified version of our spiky sentiment detector:
from sentence_transformers import SentenceTransformerimport hdbscanimport numpy as np
embedder = SentenceTransformer("all-MiniLM-L6-v2")
def find_spikes(weekly_texts, min_size=6):
anomalies = []
prev_sizes = {}
for week, texts in sorted(weekly_texts.items()):
if len(texts) < min_size:
continue
emb = embedder.encode(texts)
clusterer = hdbscan.HDBSCAN(min_cluster_size=min_size)
labels = clusterer.fit_predict(emb)
for label in set(labels) - {-1}:
members = [t for t, l in zip(texts, labels) if l == label]
size = len(members)
growth = size / max(prev_sizes.get(label, 1), 1)

Detect sharp increase with negative sentiment

if growth >= 2.5 and average_sentiment(members) < 0.45:
anomalies.append({
"week": week,
"size": size,
"growth": growth,
"samples": members[:5]
})
prev_sizes[label] = size
return anomalies
When we detect a spike that meets our criteria, we write it as an official observation to the Hindsight bank along with the corresponding Git tags and release notes.

The Attribution Engine

Our attribution engine works in a very specific way. At a high level, the process is:

Determine the timeline of the spike and find intersecting deployments
Extract the most representative quotes from the affected user feedback
Perform an entity-level comparison between the quotes and release notes
Provide a confidence score and point to the most likely broken component The chronological memory context enables us to transform vague user complaints into a precise engineering task. ## Signal Cleaning Pipeline Social media and support ticket data contains a lot of noise. Before we store any item in the Hindsight bank, we perform a set of preprocessing steps: Near-duplicate detection with MinHash/LSH Simple bot and spam filters Language and length validation A classifier to remove empty praise or ranting This ensures that our observations are of high enough quality to avoid polluting the memory bank. ## The Before and After Comparison The Classic Pipeline Input: 50 mixed tickets and tweets from a 3 month period Output: "Sentiment is 69% positive. Frequent terms: login, error, timeout." No dates, no context, no action item The Temporal Memory Agent Input: Same data stream as above Output: Specific date reference (September 3rd) The reflect query gives us a complete diagnosis: Negative feedback regarding logins spiked 3.1× starting 18 hours after the 4.2 deploy. Users are reporting sudden session invalidation and forced re-authentication. This correlates with the auth-middleware-refactor and session-token-rotation changes in the release notes. Most likely root cause: The new token rotation is too aggressive and lacks migration path for existing sessions. Confidence: High

Instead of an ambiguous complaint about the "login being broken", the engineering team received a single, focused ticket with a specific hypothesis.

A Word of Caution

The described system is not without its flaws. Our HDBSCAN clustering is not sensitive enough to "slow-burn" issues that do not form a dense enough cluster to trigger an alert.
The system can also be confused by vague release notes. In one test, we falsely associated a front-end deployment with a bug that appeared two weeks later. The release notes contained sufficiently vague language that the model assumed a connection where there was none. As a result, we added code to treat release notes as weak evidence and explicitly mention when the confidence is low.
Without reducing the signal-to-noise ratio in our data, a simple vector database would be unable to separate occasional complaints from large-scale issues. We needed to create a system that could track changes over time and understand the relationship between specific product surfaces and their deployment history. Hindsight's temporal memory capabilities were crucial in this architecture. Most importantly, the combination enabled us to answer the question that is actually important to the product teams: "not just are users unhappy, but when did it happen and what recently changed?