DEV Community: Dimitrios Ioannidis

ROC-AUC vs PR-AUC: Why Rare Events Need More Than One Metric

Dimitrios Ioannidis — Mon, 19 Oct 2026 11:03:51 +0000

Imagine you are building a system to monitor social media during a wildfire.

Thousands of posts appear every hour. Most are ordinary discussions, but a small number may contain important information about an evacuation, a blocked road, or an emerging hazard.

Your classifier is designed to flag potentially relevant posts for further review.

You evaluate it and obtain a ROC-AUC of 0.95.

That sounds excellent.

But when you examine the alerts, you discover that most of the flagged posts are irrelevant.

How can a model have a high ROC-AUC and still generate so many false alarms?

The answer becomes clearer when we compare ROC-AUC with precision-recall metrics, especially when the positive class is rare.

In this tutorial, we will construct a small, reproducible Python example, calculate both metrics, and explore what they tell us about the same classifier.

The problem with rare events

Consider a dataset containing 10,000 social media posts.

Only 100 contain the event we want to identify. The remaining 9,900 do not.

Class	Number of posts	Percentage
Relevant events	100	1%
Other posts	9,900	99%
Total	10,000	100%

This is an imbalanced classification problem.

The positive class is the event we want to detect. It represents only 1% of the dataset.

This situation appears in a plethora of applications:

Fraud detection, where fraudulent transactions are uncommon.
Security monitoring, where genuine attacks are rare.
Predictive maintenance, where equipment failures occur infrequently.
Crisis monitoring, where important event reports are a small fraction of all posts.

A model can achieve 99% accuracy simply by predicting that every post is negative.

It would identify none of the 100 relevant events.

This illustrates an important lesson: a metric is only useful when it reflects the problem we are trying to solve.

What do ROC-AUC and PR-AUC measure?

Before writing code, let us distinguish the metrics.

ROC-AUC: How well does the model rank positive examples?

A Receiver Operating Characteristic (ROC) curve plots:

True Positive Rate (TPR): the proportion of positive examples correctly identified.
False Positive Rate (FPR): the proportion of negative examples incorrectly flagged.

The true positive rate is also called recall or sensitivity.

The true positive rate is also called recall or sensitivity:

TPR = TP / (TP + FN)

FPR = FP / (FP + TN)

ROC-AUC summarizes the ROC curve across classification thresholds. It measures how well the model ranks positive examples above negative examples.

A score of 1.0 represents perfect ranking; a score near 0.5 indicates performance similar to a random ranking under the usual binary-classification interpretation.

However, the false positive rate uses all negative examples as its denominator. When there are thousands of negatives, a seemingly small rate can still produce a large number of false alarms.

Precision-recall: How useful are the positive predictions?

A precision-recall (PR) curve plots precision against recall at different thresholds.

Precision answers the question:

Of all the posts flagged by the model, how many are actually relevant?

Precision = TP / (TP + FP)

Recall answers a different question:

Of all the relevant posts, how many did the model find?

Recall = TP / (TP + FN)

The area under the precision-recall curve summarizes the trade-off between these two quantities.

In scikit-learn, we will use Average Precision (AP) as our summary of the PR curve. AP is a common PR-curve summary, but it is not the same as calculating the trapezoidal area under the curve.

For imbalanced datasets, precision-recall analysis often reveals the practical consequences of false positives more clearly than ROC analysis alone (Saito & Rehmsmeier, 2015).

Build a deliberately simple example

Instead of training a complicated model, let us construct a synthetic dataset with known outcomes.

This is intentional: the goal is to isolate the behaviour of the metrics, not to claim that we have built a production-ready detector.

Our classifier assigns the following scores:

Actual class	Score	Number of posts
Negative	0.90	495
Positive	0.80	80
Positive	0.60	20
Negative	0.10	9,405

Higher scores indicate that the model considers a post more likely to be positive.

Notice something interesting: the 495 negative posts with the highest scores rank ahead of the positive posts.

This is a useful teaching example because it lets us see how a low false positive rate can coexist with a substantial number of false alerts.

Let us implement it.

import numpy as np

from sklearn.metrics import (
    average_precision_score,
    confusion_matrix,
    precision_score,
    recall_score,
    roc_auc_score,
)

# 10,000 posts: 100 positive and 9,900 negative.
y_true = np.concatenate([
    np.zeros(9_900, dtype=int),
    np.ones(100, dtype=int),
])

# Scores aligned with the labels above.
# Negative posts: 495 high scores, then 9,405 low scores.
# Positive posts: 80 medium-high scores, then 20 lower scores.
y_score = np.concatenate([
    np.full(495, 0.90),
    np.full(9_405, 0.10),
    np.full(80, 0.80),
    np.full(20, 0.60),
])

# Threshold for converting scores into binary decisions.
threshold = 0.50
y_pred = (y_score >= threshold).astype(int)

# Ranking metrics.
roc_auc = roc_auc_score(y_true, y_score)
average_precision = average_precision_score(y_true, y_score)

# Metrics at the selected threshold.
precision = precision_score(y_true, y_pred, zero_division=0)
recall = recall_score(y_true, y_pred, zero_division=0)

tn, fp, fn, tp = confusion_matrix(
    y_true, y_pred, labels=[0, 1]
).ravel()

accuracy = (tp + tn) / len(y_true)

print(f"ROC-AUC: {roc_auc:.4f}")
print(f"Average Precision: {average_precision:.4f}")
print(f"Precision at threshold {threshold}: {precision:.4f}")
print(f"Recall at threshold {threshold}: {recall:.4f}")
print(f"Accuracy at threshold {threshold}: {accuracy:.4f}")

print(f"True positives: {tp}")
print(f"False positives: {fp}")
print(f"True negatives: {tn}")
print(f"False negatives: {fn}")

The expected output is:

ROC-AUC: 0.9500
Average Precision: 0.1449
Precision at threshold 0.5: 0.1681
Recall at threshold 0.5: 1.0000
Accuracy at threshold 0.5: 0.9505

True positives: 100
False positives: 495
True negatives: 9405
False negatives: 0

The contrast is striking.

The ROC-AUC is 0.95, which indicates strong overall ranking performance on this synthetic dataset. Yet the average precision is only about 0.145, and at the selected threshold, fewer than 17% of flagged posts are genuinely positive.

The example demonstrates why these metrics answer different questions.

	Predicted positive	Predicted negative
Actually positive	100	0
Actually negative	495	9,405

We found every positive case.

That gives us 100% recall.

But we also flagged 495 negative posts.

Our precision is:

Precision: = 100 (100 + 495), approx 0.1681

In other words, out of 595 flagged posts, only 100 are positive.

The model has achieved perfect recall at this threshold, but the alert workload is substantial.

Imagine a crisis-response team that must manually investigate every alert. Reviewing 595 posts to find 100 relevant ones could be expensive, especially if the system operates continuously.

That is the operational side of the metric.

ROC-AUC tells us about ranking across thresholds. Precision tells us how much confidence we should place in the positive predictions produced at a particular threshold.

Neither should be interpreted in isolation.

Visualize both curves

import matplotlib.pyplot as plt

from sklearn.metrics import (
    average_precision_score,
    precision_recall_curve,
    roc_auc_score,
    roc_curve,
)

# ROC curve.
fpr, tpr, _ = roc_curve(y_true, y_score)

# Precision-recall curve.
precision_values, recall_values, _ = precision_recall_curve(
    y_true, y_score
)

roc_auc = roc_auc_score(y_true, y_score)
average_precision = average_precision_score(y_true, y_score)
prevalence = y_true.mean()

fig, (ax_roc, ax_pr) = plt.subplots(1, 2, figsize=(12, 5))

# Left: ROC.
ax_roc.plot(fpr, tpr, label=f"ROC-AUC = {roc_auc:.3f}")
ax_roc.plot([0, 1], [0, 1], "--", label="Random ranking")
ax_roc.set_xlabel("False Positive Rate")
ax_roc.set_ylabel("True Positive Rate (Recall)")
ax_roc.set_title("ROC Curve")
ax_roc.legend()
ax_roc.grid(True, alpha=0.3)

# Right: Precision-recall.
ax_pr.plot(
    recall_values,
    precision_values,
    label=f"Average Precision = {average_precision:.3f}",
)
ax_pr.axhline(
    prevalence,
    linestyle="--",
    label=f"Positive prevalence = {prevalence:.1%}",
)
ax_pr.set_xlabel("Recall")
ax_pr.set_ylabel("Precision")
ax_pr.set_title("Precision-Recall Curve")
ax_pr.legend()
ax_pr.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

The ROC curve shows strong overall ranking performance, with a ROC-AUC of 0.95.

However, this metric can obscure the impact of false positives when the dataset is highly imbalanced.

The precision-recall curve highlights this issue: 495 negative posts receive higher scores than all the positive posts, reducing precision despite the high ROC-AUC.

The dashed line in the PR plot represents positive-class prevalence. With random rankings, expected Average Precision is approximately equal to this prevalence, or 0.01 (1%) in this example.

One caution: this dataset uses intentionally constructed score groups. Real models typically produce different score distributions, so their curves may have different shapes.

Does Pr-AUC always matter more?

Not necessarily.

ROC-AUC remains useful when comparing ranking behaviour across thresholds, and it may be particularly valuable when both classes matter to the task.

Precision-recall metrics are especially informative when the positive class is rare, and the quality of positive predictions matters operationally. This is why they are widely considered alongside ROC analysis in imbalanced classification research (Davis & Goadrich, 2006; Saito & Rehmsmeier, 2015).

The choice ultimately depends on the cost of the errors.

Consider two scenarios.

Scenario A: Missing an event is extremely costly.

In early crisis detection, you may prefer a lower threshold to capture as many potentially important posts as possible. This increases recall but can also increase the number of false alarms.

Scenario B: Human review capacity is limited.

If a small team can inspect only a limited number of posts, precision and precision-recall behaviour become particularly important. A model that flags too many irrelevant posts may overwhelm the people who are supposed to investigate them.

The threshold should therefore reflect the task, available resources, and relative cost of false positives and false negatives.

AUC values alone do not choose the threshold for you.

How should we apply this to anomaly detection and fact-checking?

This distinction is especially important in social-media analysis during emergencies.

An anomaly detector might identify an unusual cluster of posts, a sudden change in message volume, or content that differs from typical patterns.

That output is a signal for further investigation. It is not evidence that a post is false.

The system still needs to examine the claim, assess available evidence, and account for uncertainty.

Evaluation should reflect those separate responsibilities. Anomaly detection should be tested for its ability to prioritize relevant cases, while verification should be evaluated against suitable reference evidence and labels.

This is one of the design principles I explored in my MSc thesis on anomaly detection and fact-checking of social-media data in crisis management: unusual content should be prioritized for analysis without automatically treating it as misinformation.

The distinction between detection and verification is not merely philosophical. It changes what the system measures, how its outputs are interpreted, and which errors matter.

Conclusion

When positive cases are rare, a model can look impressive under one metric while exposing weaknesses under another.

ROC-AUC summarizes ranking behaviour across thresholds. Precision-recall analysis makes the relationship between retrieved positives and false alarms more visible. Precision, recall, and the confusion matrix then help us understand what happens at the operating threshold we actually intend to use.

My practical checklist is:

Inspect the class distribution. Understand how rare the positive class is before interpreting scores.
Report more than accuracy. Consider ROC-AUC, average precision, precision, recall, and the confusion matrix.
Choose thresholds based on consequences. Think about the workload created by false positives and the harm caused by missed events.
Validate on representative data. A model's performance on a controlled dataset does not automatically establish its usefulness in a live environment.

Most importantly, evaluate the model against the task it is intended to support.

In AI-assisted decision-making, the goal is not simply to produce a high score. It is to understand what that score establishes—and what remains uncertain.

References and further reading

Saito, T., & Rehmsmeier, M. (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE, 10(3), e0118432.
Davis, J., & Goadrich, M. (2006). The Relationship Between Precision-Recall and ROC Curves. Proceedings of the 23rd International Conference on Machine Learning, 233–240.
scikit-learn. ROC-AUC documentation: https://scikit-learn.org/stable/modules/model_evaluation.html#receiver-operating-characteristic-roc
scikit-learn. Average Precision documentation: https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-and-f-measures

Enjoyed this breakdown? Feel free to check out my website at ioannidis.dev for more machine learning guides and articles.

An Anomaly Is Not Misinformation: Building AI-Assisted Fact-Checking for Crisis Social Media

Dimitrios Ioannidis — Sat, 17 Oct 2026 11:49:11 +0000

Imagine that a wildfire breaks out near a populated area.

In a matter of minutes, social media fills with evacuation warnings, photographs, eyewitness reports, speculation, and conflicting updates. Some posts contain critical information. Others repeat outdated claims. A few may be misleading.

An AI system detects a sudden change in posting activity and flags several messages as anomalies.

What has the system actually discovered?

It has identified unusual patterns in the data. It has not necessarily established which messages are false, which describe a real emergency, or which deserve immediate human attention.

That distinction is central to a problem I explore in my 2026 MSc thesis research, Anomaly Detection (Fact-Checking) of Social Media Data in Crisis Management.

It also motivates a broader question: how can we build AI systems that help us investigate information without giving their outputs more authority than the evidence warrants?

Anomaly detection is not fact-checking

Anomaly detection identifies observations that deviate from an expected pattern. In social media, this might mean unusual language, a rare combination of terms, a sudden change in activity, or a post that differs substantially from its surrounding data.

These observations can be useful. During an emergency, a monitoring system cannot investigate every post with equal depth. An anomaly detector can help prioritize material that warrants closer examination.

However, unusual does not mean false.

A more defensible process is:

Anomaly detected → Prioritize for investigation → Examine evidence → Assess claim → Review when necessary

The anomaly score helps determine what to inspect. Verification requires a different kind of reasoning.

This separation reflects an established distinction in anomaly detection and automated fact-checking research: detecting unusual observations and assessing whether a claim is supported by evidence are different tasks (Chandola, Banerjee, & Kumar; Shu et al., 2017).

A modular architecture for crisis-related social media

My thesis investigated how anomaly detection could be integrated with automated fact-checking and natural language processing (NLP) in a crisis-monitoring framework, without treating unusual activity as proof of misinformation.

Rather than relying on a single model to make every decision, I designed a modular pipeline.

Stage 1: Collect and normalize social media data

The framework uses a data-collection component and a preprocessing stage to prepare social media posts for analysis.

The controlled experiment used a disaster-related tweet dataset containing 2,559 entries and 13 original fields. A second experiment used a Mastodon API prototype to explore how public posts from a federated social platform could enter the same processing pipeline.

Normalization matters because different platforms expose different fields, metadata, and content formats. A reusable internal representation makes it easier to apply downstream processing consistently.

Stage 2: Identify unusual content

The framework evaluated several approaches to anomaly detection, including:

K-Means to organize observations into clusters.
DBSCAN to identify density-based clusters and potential outliers.
Spectral autoencoder-based detection using reconstruction-based deviations
A hybrid DT-SVMNB model combining decision-tree, support-vector-machine, and naïve Bayes components.

The experiments also examined different feature representations, including document-term matrices (DTM), TF-IDF vectors, and BERT embeddings.

These representations capture different properties of the text. DTM and TF-IDF emphasize lexical patterns, whereas BERT embeddings provide contextual semantic representations.

The important engineering lesson is that model performance depends not only on the algorithm, but also on how the data is represented and what task the model is expected to perform.

Stage 3: Add semantic and verification-oriented analysis

Posts identified for further examination can then enter downstream components.

The framework includes NLP components for sentiment and emotion analysis, alongside an automated fact-checking verifier that classifies claims using labels such as True, False, Partially Supported, Partially Not Supported, and Unclear.

These outputs answer different questions.

Sentiment analysis can provide information about the tone of a message. Emotion classification can offer additional context. A verification component attempts to assess a factual claim.

Neither sentiment nor emotional intensity establishes whether a claim is true. A frightened eyewitness may be accurate, and a confidently written message may be completely unsupported.

What did the experiments show?

One useful finding was that performance varied with both the model and the feature representations.

For example, in the controlled dataset experiment, K-Means using BERT embeddings achieved a ROC-AUC of 0.59 and a PR-AUC of 0.30. These results indicate a limited discriminatory ability in that experimental setting.

This is important because a technically functioning model is not necessarily a sufficiently reliable operational detector.

A pipeline can collect posts, transform text into features, calculate scores, and display suspicious messages while still struggling to distinguish the cases that matter most.

The Mastodon experiment demonstrated a different kind of result: the framework could ingest, normalize, and process public posts through the existing pipeline.

That was a deployment-oriented feasibility test, not proof that the system could reliably determine the truth of claims during a live crisis.

The distinction matters. The Mastodon sample did not have manually verified labels for the relevant fact-checking and NLP evaluations, so simulated reference labels were used to test the evaluation workflow. The resulting scores should not be interpreted as definitive evidence of real-world verification accuracy.

These limitations are not incidental details. They define what can responsibly be concluded from the research.

Turning the architecture into a trustworthy system

The experiments suggest a practical design principle: store detection, verification, and review as separate parts of an assessment.

For example, a developer might represent the output of a monitoring system with a structure like this:

from dataclasses import dataclass, field
from typing import Literal

VerificationStatus = Literal[
 "not_assessed",
 "supported",
 "partially_supported",
 "unclear",
 "partially_not_supported",
 "not_supported",
]

@dataclass
class ClaimAssessment:
 post_id: str
 anomaly_score: float | None = None
 verification_status: VerificationStatus =   "not_assessed"
 evidence_refs: list[str] = field(default_factory=list)
 needs_human_review: bool = True

This is an illustrative design, not code copied from the thesis.

Its purpose is to make an important boundary explicit: an anomaly score does not automatically change a claim's verification status. The system can flag a post for investigation while leaving its factual status unassessed.

The same design also makes room for uncertainty. A claim can remain unclear when the available evidence is insufficient, rather than being forced into a binary category.

For consequential decisions, I would extend this representation with timestamps, source provenance, the basis for each assessment, and a record of subsequent corrections.

I would also treat human review as an explicit operational stage for ambiguous or high-impact claims. Although the thesis includes a feedback mechanism, it does not implement a complete expert-review process. That remains an important direction for future development.

Finally, evaluation should reflect the task's practical requirements. When relevant positive cases are relatively rare, PR-AUC and precision-recall behaviour can be particularly informative. A high-level accuracy figure alone may conceal how many important cases are missed or how many ordinary posts are incorrectly flagged.

The larger lesson: useful signals are not final judgments

There is a broader implication for AI-assisted decision-making.

A machine can detect a pattern, generate a classification, summarize several sources, or produce a plausible explanation. Each capability can be valuable. None, by itself, guarantees that the resulting conclusion is justified.

In crisis management, this distinction can influence how information is prioritized and communicated. In other settings, it can affect moderation decisions, reputation assessments, public services, and the interpretation of AI-generated answers.

The practical objective should not be to eliminate uncertainty by giving every message a confident label. It should be to organize uncertainty so that people can investigate the right questions with the evidence available.

That means asking:

What exactly did the model detect?
What evidence supports the resulting claim assessment?
Are the sources independent, current, and relevant?
What remains uncertain?
Who can review or challenge the decision?

These questions turn transparency from a general aspiration into a set of engineering requirements.

Conclusion

My thesis explored a modular approach in which anomaly detection prioritizes unusual social media content, while downstream fact-checking and NLP components provide additional context and claim assessment.

The main lesson is not that one particular algorithm solves misinformation detection. It is that different models should perform clearly defined tasks, and their outputs should not be mistaken for conclusions they have not established.

The next research steps include larger-scale live validation, broader data sources, multimodal and graph-based analysis, and evaluation with crisis-management practitioners. Human oversight, evidence provenance, and responsible handling of public social media data also remain essential considerations.

These technical questions eventually lead to a more fundamental one: what should we be entitled to conclude when a machine gives us an answer?

That question is the subject of my book, Beyond True or False: Artificial Intelligence, Uncertainty, and the Future of Human Judgment.
It develops the broader implications of AI-assisted assessment, including evidence, uncertainty, human oversight, and the responsibilities of institutions that rely on algorithmic judgments.

Read more:

My research thesis: https://www.researchgate.net/publication/415288464_Anomaly_Fact-Checking_Detection_of_Social_Media_Data_in_Crisis_Management
My book, Beyond True or False: https://www.lulu.com/shop/dimitrios-ioannidis/beyond-true-or-false/paperback/product-7k5pyp7.html?page=1&pageSize=4
https://www.amazon.com/dp/B0HN5WZ2V4

I would be interested to hear how other developers approach this boundary between anomaly detection and verification. In your own systems, how do you prevent a model's signal from being treated as proof?

References

Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly Detection: A Survey. ACM Computing Surveys, 41(3).
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT.
Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017). Fake News Detection on Social Media: A Data Mining Perspective. ACM SIGKDD Explorations Newsletter, 19(1), 22–36.