DEV Community

Cover image for Source Score: Continuing Exploration of LLM Usage in Automated Workflows
Amit Singh
Amit Singh

Posted on

Source Score: Continuing Exploration of LLM Usage in Automated Workflows

#python #automation #devops #ai

TL;DR – I built a weekly pipeline that pulls the ten newest articles from a news outlet, picks the one that actually makes a falsifiable claim, fills in any missing summary, finds two independent proofs, and opens PRs with ready‑to‑ingest claims/ and proofs/ files. All of this runs on a couple of Python scripts, 2 custom OpenRouter skills, and two GitHub Actions.

In my last post I covered how I automated ingestion of top news sources by combining Firecrawl, OpenRouter API, and Github Action workflows. In this post I'll implement the same pattern for news source claims and their proofs.

The first challenge I ran into was to figure out a way to fetch recent articles published by a news source. Luckily, I found NewsData.IO which provides an API to search, collect and track worldwide news. The NewsData.io free tier gives me 200 API credits per day, more than enough for a weekly run across 12 sources (for now at least 😌).

1️⃣ Fetch the latest articles - newsdata_io.py

The first step is to write a thin wrapper around the NewsData.io latest endpoint. It pulls the 10 most recent articles for a given domain.

# Free‑tier‑friendly query params
CATEGORY = "environment,technology,world"
LANGUAGE = "en"
REMOVE_DUPLICATE = "1"
SIZE = "10"
DATATYPE = "news,research,analysis,pressRelease"

NEWSDATA_API_BASE_URL = os.getenv(
    "NEWSDATA_API_BASE_URL", "https://newsdata.io/api/1"
)
NEWSDATA_API_KEY = os.environ["NEWSDATA_API_KEY"]

def get_claims(src_domain_url: str):
    """
    Call the NewsData.io `/latest` endpoint for a specific domain.
    Returns a list of article dicts (or None on error).
    """
    endpoint = f"{NEWSDATA_API_BASE_URL}/latest"
    params = {
        "category": CATEGORY,
        "language": LANGUAGE,
        "removeduplicate": REMOVE_DUPLICATE,
        "size": SIZE,
        "datatype": DATATYPE,
        "apikey": NEWSDATA_API_KEY,
        "domainurl": src_domain_url,
    }
    response = requests.get(endpoint, params=params, timeout=10)
    if response.status_code != 200:
        print(
            f"Error: couldn't fetch claims for {src_domain_url}: {response.status_code}"
        )
        # Show suggestions if the API knows a better domain
        resp_body = response.json()
        if (
            resp_body.get("results") is not None
            and resp_body.get("results")[0].get("suggestion") is not None
        ):
            print(
                f"Suggested domain url(s) for {src_domain_url}: {resp_body['results'][0]['suggestion']}"
            )
        return None
    return response.json()["results"]
Enter fullscreen mode Exit fullscreen mode

Why it matters: The free tier only gives 200 credits per day, so we keep the request lightweight: single domain, ten results, and a narrow set of categories. The DATATYPE filter is intentionally broad; we prune non‑falsifiable items later.

2️⃣ Filter for a falsifiable claim - the claim‑verification skill

Given the way I'm currently calculating scores for a news source, only claims that are falsifiable matter. So out of all the claims returned by NewsData API I'm keeping only one that matches the falsifiable claim criteria best.

I'm using LLMs to perform this classification, to be specific, I'm forwarding an array of objects returned by NewsData API to OpenRouter API.

[{
"article_id": "40305aa160787297dd3f9cc15faa8637",
"link": "https://www.theguardian.com/us-news/2026/may/22/kansas-bird-nest-truck",
"title": "Federally protected bird’s nest holds up sale of Ford truck in Kansas",
"description": "A robin built a nest on a Ford-F-250’s tire and laid its eggs in it; a law prohibits removing it while inhabited by bird brood A truck sold by a Kansas dealership cannot be taken from the lot by its new owner because a family of robins is living atop one of the vehicle’s tires. The relatively novel situation has gained widespread attention after the dealership in the Kansas community of Olathe wrote about it on its Facebook page – and it perhaps taught many that active robin nests are protected by federal law from the US. Continue reading...",
"keywords": [
"birds",
"kansas",
"wildlife",
"ford",
"animals",
"us news",
"law (us)"
],
"creator": [
"josé olivares"
],
"language": "english",
"country": [
"united states of america"
],
"category": [
"top",
"environment"
],
"datatype": "news",
"pubDate": "2026-05-22 19:03:07",
"pubDateTZ": "UTC",
"fetched_at": "2026-05-22 19:32:47",
"image_url": "https://i.guim.co.uk/img/media/c9e972eb2d494c4a9c713a7b5550f0fa9efcae1f/0_503_1536_1229/master/1536.jpg?width=140&quality=85&auto=format&fit=max&s=ad95c6dcdf71df9bc3461b683effe424",
"video_url": null,
"source_id": "theguardian",
"source_name": "The Guardian",
"source_priority": 106,
"source_url": "https://www.theguardian.com",
"source_icon": "https://n.bytvi.com/theguardian.jpg",
"duplicate": false
}]
Enter fullscreen mode Exit fullscreen mode

To help with the classification I wrote an AI agent skill. It tells the LLM to:

  1. Visit each article URL with the web‑search tool.
  2. Detect whether the article contains a claim that can be objectively proven true or false.
  3. Return exactly one JSON object that matches the criteria. 
CLAIM_FILTER_PROMPT = (
    "Use web search tool to visit the link for each article, access the content and then assess if it is a falsifiable claim."
    "Out of these 10 articles, only return 1 article that best fits the falsifiable claim criterion."
    "Prefer claims that have been made by the news source directly."
    "Keep the json structure of the claims the same as the original schema in the input. Do not add, remove, or modify any key or value."
    "Only output the plain json array string that I can safely unmarshal."
    "Do not format the string. Do not output anything else."
)

req_content = (
    "Following is a list of 10 articles published by the same news outlet. Each article is represented by a json string type element in the array"
    f"\n\n{claims}\n\n"
    f"{CLAIM_FILTER_PROMPT}"
)

filtered_claims = openrouter.req_w_addons(
    req_content, skill=falsifiable_claim_skill, tools=[openrouter.WEB_SEARCH_TOOL]
)
Enter fullscreen mode Exit fullscreen mode

Result: A single, well‑structured claim with all the fields required to create a claim ingestion doc.

I'm only selecting one claim for now, once I've verified the stability of ingestion workflows and the reliability of OpenRouter responses I'll the increase the count and the ingestion frequency.

3️⃣ Fill in missing descriptions - a quick summarization pass

NewsData.io sometimes returns "null" for the description field. When that happens I'm asking OpenRouter to summarize the article in under 500 characters.

CLAIM_SUMMARY_PROMPT = (
    "Use web search tool to visit the link to the article and access its content."
    "Summarize the article in under 500 characters."
    "Return only the summary without any additional text."
)

req_content = (
    "Following is the url to an article published by a news media outlet."
    f"\n\n{claim['link']}\n\n"
    f"{CLAIM_SUMMARY_PROMPT}"
)

claim_summary = openrouter.req_w_addons(
    req_content, tools=[openrouter.WEB_SEARCH_TOOL]
)

claim["description"] = claim_summary
Enter fullscreen mode Exit fullscreen mode

Now every claim has a concise, human‑readable description, even when the source API left it blank.

4️⃣ Weekly claim ingestion - GitHub Actions workflow

The whole thing runs on a GitHub Actions schedule once a week. The workflow checks out the repo, installs dependencies, runs ingest_claims.py, and opens a PR.

workflow execution

Outcome: A PR appears every Sunday with fresh claim documents, ready for review.

5️⃣ Fetch proofs - ingest_proofs.py + workflow

Once a claim has been ingested, I need proofs that either support or refute it. The proof‑verification prompt asks the LLM to find two independent sources and label each with a boolean supports_claim.

To help LLMs search for supporting or refuting proofs for a given claim using web search tool, I wrote another skill, claim-verification, which performs the following:

  1. Extracts and validates falsifiable claims: Takes a URL to a media article/post and identifies the core, testable claim from its content, ensuring it's concrete and can be proven true or false.
  2. Performs targeted web searches: Uses time-aware search queries to find up to two high-quality external documents that directly support or refute the claim, with strict attention to temporal relevance (matching the claim's timeframe).
  3. Returns verifiable evidence URLs: Outputs a JSON array of retrieved URLs with boolean indicators (supports_claim: true/false) showing whether each document confirms or contradicts the original claim, prioritizing official/authoritative sources over opinion pieces. 
CLAIM_VERIFICATION_PROMPT = (
    "Use web search tool to access the claim link, fetch the content and process it."
    "Use the web search tool again to look for proofs in the form of official statements, press releases, or reports from reputable sources to prove the claim right or wrong conclusively."
    "Ensure that the proofs belong to the same timeline as the claim. Do not include outdated sources."
    "Output links to the 2 sources that prove the claim right or wrong and specify as a boolean whether they support the claim or not."
    "The output format should be a json array with each element being a json object corresponding to a source supporting or refuting the claim."
    "Each json element should follow the following schema: {\"uri\": \"string\", \"supports_claim\": boolean}"
)

req_content = (
    "Following is a link to a falsifiable claim by a news media outlet as an article"
    f"\n\n{claim['uri']}\n\n"
    f"{CLAIM_VERIFICATION_PROMPT}"
)

claim_proofs = openrouter.req_w_addons(
    req_content, skill=claim_verification_skill, tools=[openrouter.WEB_SEARCH_TOOL]
)
Enter fullscreen mode Exit fullscreen mode

Proof workflow: Just like claim ingestion workflow, I'm running proof ingestion workflow once a week. It runs the ingest_proofs.py script, creates the proof documents in a new branch, then creates a PR from this branch to the main branch.

workflow execution example

Outcome: A PR appears every Sunday with fresh proof documents for ingested claims, ready for review.

6️⃣ OpenRouter reliability improvements

Since my OpenRouter API usage has been increasing, my list of free tier models is shortened to the following:

FREE_MODELS_DOC = [
    "google/gemma-4-31b-it:free",
    "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free",
    "openrouter/free"
]
Enter fullscreen mode Exit fullscreen mode

These three models consistently give good results while staying within the free tier.

Back‑off retry loop (openrouter.py)

I also added incremental delays between retries to reduce the odds of running into server side errors.

for i in range(1, OPENROUTER_MAX_RETRIES + 1):
    status, body = helper.post_request(...)
    if status in (0, 429) or 500 <= status < 600:
        print(f"OpenRouter API returned status {status}, retrying...", file=sys.stderr)
        time.sleep(10 * i)
        continue
    # success handling...
Enter fullscreen mode Exit fullscreen mode

Impact: 500‑error rate dropped dramatically, and weekly API spend stayed well under the free‑tier limits.

I really like this OpenRouter dashboard btw.

Token usage dashboard

7️⃣ Trade‑offs & limitations

Aspect Trade‑off
Free‑tier limits 200 NewsData.io credits/day restricts us to a single domain per run. Scaling to many outlets will need a paid plan or smarter batching.
Single claim per source I deliberately return only the “best” falsifiable claim to keep the workflow simple. Future work will support multiple high‑quality claims per article.
LLM hallucinations Even with the claim‑verification skill, the model can surface outdated links. Temporal awareness helps, but it’s not a silver bullet.
Proof quality Proofs are fetched automatically; a human review is still recommended for high‑stakes claims.

8️⃣ Conclusion & next steps

By stitching together NewsData.io, custom falsifiable-claim and claim‑verification skills, OpenRouter (with incremental back‑off and free‑tier model rotation), and GitHub Actions, I built a cost‑effective, fully automated pipeline that turns raw news into structured, falsifiable claim documents and their supporting proofs.

What’s next?

  1. Add confidence scores to claims and proofs so downstream consumers can weigh evidence.
  2. Move from weekly to daily ingestion for high‑volume outlets once the retry logic proves rock‑solid.
  3. Negative tests because I'm yet to see a case where a claim is proven wrong, I guess the LLMs are playing it safe 🫣

If you’re curious, the full code lives in the SatyaLens/sources repo. If you have any suggestions or questions for me, feel free to drop them in the comment section.

Top comments (0)