Imagine that a wildfire breaks out near a populated area.
In a matter of minutes, social media fills with evacuation warnings, photographs, eyewitness reports, speculation, and conflicting updates. Some posts contain critical information. Others repeat outdated claims. A few may be misleading.
An AI system detects a sudden change in posting activity and flags several messages as anomalies.
What has the system actually discovered?
It has identified unusual patterns in the data. It has not necessarily established which messages are false, which describe a real emergency, or which deserve immediate human attention.
That distinction is central to a problem I explore in my 2026 MSc thesis research, Anomaly Detection (Fact-Checking) of Social Media Data in Crisis Management.
It also motivates a broader question: how can we build AI systems that help us investigate information without giving their outputs more authority than the evidence warrants?
- Anomaly detection is not fact-checking
Anomaly detection identifies observations that deviate from an expected pattern. In social media, this might mean unusual language, a rare combination of terms, a sudden change in activity, or a post that differs substantially from its surrounding data.
These observations can be useful. During an emergency, a monitoring system cannot investigate every post with equal depth. An anomaly detector can help prioritize material that warrants closer examination.
However, unusual does not mean false.
A more defensible process is:
Anomaly detected → Prioritize for investigation → Examine evidence → Assess claim → Review when necessary
The anomaly score helps determine what to inspect. Verification requires a different kind of reasoning.
This separation reflects an established distinction in anomaly detection and automated fact-checking research: detecting unusual observations and assessing whether a claim is supported by evidence are different tasks (Chandola, Banerjee, & Kumar; Shu et al., 2017).
- A modular architecture for crisis-related social media
My thesis investigated how anomaly detection could be integrated with automated fact-checking and natural language processing (NLP) in a crisis-monitoring framework, without treating unusual activity as proof of misinformation.
Rather than relying on a single model to make every decision, I designed a modular pipeline.
Stage 1: Collect and normalize social media data
The framework uses a data-collection component and a preprocessing stage to prepare social media posts for analysis.
The controlled experiment used a disaster-related tweet dataset containing 2,559 entries and 13 original fields. A second experiment used a Mastodon API prototype to explore how public posts from a federated social platform could enter the same processing pipeline.
Normalization matters because different platforms expose different fields, metadata, and content formats. A reusable internal representation makes it easier to apply downstream processing consistently.
Stage 2: Identify unusual content
The framework evaluated several approaches to anomaly detection, including:
K-Means to organize observations into clusters.
DBSCAN to identify density-based clusters and potential outliers.
Spectral autoencoder-based detection using reconstruction-based deviations
A hybrid DT-SVMNB model combining decision-tree, support-vector-machine, and naïve Bayes components.
The experiments also examined different feature representations, including document-term matrices (DTM), TF-IDF vectors, and BERT embeddings.
These representations capture different properties of the text. DTM and TF-IDF emphasize lexical patterns, whereas BERT embeddings provide contextual semantic representations.
The important engineering lesson is that model performance depends not only on the algorithm, but also on how the data is represented and what task the model is expected to perform.
Stage 3: Add semantic and verification-oriented analysis
Posts identified for further examination can then enter downstream components.
The framework includes NLP components for sentiment and emotion analysis, alongside an automated fact-checking verifier that classifies claims using labels such as True, False, Partially Supported, Partially Not Supported, and Unclear.
These outputs answer different questions.
Sentiment analysis can provide information about the tone of a message. Emotion classification can offer additional context. A verification component attempts to assess a factual claim.
Neither sentiment nor emotional intensity establishes whether a claim is true. A frightened eyewitness may be accurate, and a confidently written message may be completely unsupported.
- What did the experiments show?
One useful finding was that performance varied with both the model and the feature representations.
For example, in the controlled dataset experiment, K-Means using BERT embeddings achieved a ROC-AUC of 0.59 and a PR-AUC of 0.30. These results indicate a limited discriminatory ability in that experimental setting.
This is important because a technically functioning model is not necessarily a sufficiently reliable operational detector.
A pipeline can collect posts, transform text into features, calculate scores, and display suspicious messages while still struggling to distinguish the cases that matter most.
The Mastodon experiment demonstrated a different kind of result: the framework could ingest, normalize, and process public posts through the existing pipeline.
That was a deployment-oriented feasibility test, not proof that the system could reliably determine the truth of claims during a live crisis.
The distinction matters. The Mastodon sample did not have manually verified labels for the relevant fact-checking and NLP evaluations, so simulated reference labels were used to test the evaluation workflow. The resulting scores should not be interpreted as definitive evidence of real-world verification accuracy.
These limitations are not incidental details. They define what can responsibly be concluded from the research.
- Turning the architecture into a trustworthy system
The experiments suggest a practical design principle: store detection, verification, and review as separate parts of an assessment.
For example, a developer might represent the output of a monitoring system with a structure like this:
from dataclasses import dataclass, field
from typing import Literal
VerificationStatus = Literal[
"not_assessed",
"supported",
"partially_supported",
"unclear",
"partially_not_supported",
"not_supported",
]
@dataclass
class ClaimAssessment:
post_id: str
anomaly_score: float | None = None
verification_status: VerificationStatus = "not_assessed"
evidence_refs: list[str] = field(default_factory=list)
needs_human_review: bool = True
This is an illustrative design, not code copied from the thesis.
Its purpose is to make an important boundary explicit: an anomaly score does not automatically change a claim's verification status. The system can flag a post for investigation while leaving its factual status unassessed.
The same design also makes room for uncertainty. A claim can remain unclear when the available evidence is insufficient, rather than being forced into a binary category.
For consequential decisions, I would extend this representation with timestamps, source provenance, the basis for each assessment, and a record of subsequent corrections.
I would also treat human review as an explicit operational stage for ambiguous or high-impact claims. Although the thesis includes a feedback mechanism, it does not implement a complete expert-review process. That remains an important direction for future development.
Finally, evaluation should reflect the task's practical requirements. When relevant positive cases are relatively rare, PR-AUC and precision-recall behaviour can be particularly informative. A high-level accuracy figure alone may conceal how many important cases are missed or how many ordinary posts are incorrectly flagged.
- The larger lesson: useful signals are not final judgments
There is a broader implication for AI-assisted decision-making.
A machine can detect a pattern, generate a classification, summarize several sources, or produce a plausible explanation. Each capability can be valuable. None, by itself, guarantees that the resulting conclusion is justified.
In crisis management, this distinction can influence how information is prioritized and communicated. In other settings, it can affect moderation decisions, reputation assessments, public services, and the interpretation of AI-generated answers.
The practical objective should not be to eliminate uncertainty by giving every message a confident label. It should be to organize uncertainty so that people can investigate the right questions with the evidence available.
That means asking:
What exactly did the model detect?
What evidence supports the resulting claim assessment?
Are the sources independent, current, and relevant?
What remains uncertain?
Who can review or challenge the decision?
These questions turn transparency from a general aspiration into a set of engineering requirements.
Conclusion
My thesis explored a modular approach in which anomaly detection prioritizes unusual social media content, while downstream fact-checking and NLP components provide additional context and claim assessment.
The main lesson is not that one particular algorithm solves misinformation detection. It is that different models should perform clearly defined tasks, and their outputs should not be mistaken for conclusions they have not established.
The next research steps include larger-scale live validation, broader data sources, multimodal and graph-based analysis, and evaluation with crisis-management practitioners. Human oversight, evidence provenance, and responsible handling of public social media data also remain essential considerations.
These technical questions eventually lead to a more fundamental one: what should we be entitled to conclude when a machine gives us an answer?
That question is the subject of my book, Beyond True or False: Artificial Intelligence, Uncertainty, and the Future of Human Judgment.
It develops the broader implications of AI-assisted assessment, including evidence, uncertainty, human oversight, and the responsibilities of institutions that rely on algorithmic judgments.
Read more:
My research thesis: https://www.researchgate.net/publication/415288464_Anomaly_Fact-Checking_Detection_of_Social_Media_Data_in_Crisis_Management
My book, Beyond True or False: https://www.lulu.com/shop/dimitrios-ioannidis/beyond-true-or-false/paperback/product-7k5pyp7.html?page=1&pageSize=4
I would be interested to hear how other developers approach this boundary between anomaly detection and verification. In your own systems, how do you prevent a model's signal from being treated as proof?
References
Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly Detection: A Survey. ACM Computing Surveys, 41(3).
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT.
Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017). Fake News Detection on Social Media: A Data Mining Perspective. ACM SIGKDD Explorations Newsletter, 19(1), 22–36.
Top comments (0)