DEV Community: Ali

We Ran Our Search Engine Backwards

Ali — Mon, 24 Aug 2026 06:05:00 +0000

Run the query rate cuts against our REST API: it finds "Fed cuts rates again."
Register the same query on a WebSocket subscription: the article sails right past.
Silence. Same product, same query syntax, same article, two different answers to
"does this match?"

We shipped that for a year.

Here's the engineering problem underneath. We ingest around ten thousand news articles
a day, in bursts, because news doesn't believe in uniform distribution. A few hundred
standing queries like rate cuts AND source:www.reuters.com are listening over
WebSockets and webhooks. Within seconds of an article entering the system, it has to
reach exactly the subscribers whose query matches it. No misses, no spam.

So: for every incoming article, how do you compute who gets notified?

It looks like a search problem, and it almost is, just mirrored. Search is one query
against many documents. This is one document against many queries.

A subscription is a search query that never stops running

Quick context so the rest makes sense. I run finlight, a
financial news API. Articles flow through an ingestion pipeline: scraped, enriched
with sentiment and company tags, then indexed into OpenSearch (the Elasticsearch
fork; everything here applies to both). Customers consume them in two ways:

Pull: a REST API with full-text search over that OpenSearch index.
Push: WebSocket streams and webhooks. You register a query once, in the same query syntax the REST API uses, and every future article that matches is pushed to you within seconds of ingestion.

A subscription, in other words, is just a search query that never stops running.
Because both modes share one query syntax, our docs have always given users a simple
tip: test your query against the REST API first, because what search returns is what
your socket will deliver. File that sentence away. It's going to come back with a
knife.

Somewhere between ingestion and delivery, then, a matching step has to answer: which
of the N registered queries does this one article satisfy? Everything that follows is
about that one step.

Wrong answer #1: query the search index for every article

The naive approach: we already run OpenSearch for our REST search. When an article
arrives, loop over all subscriptions and run each stored query against the article
index, then check if the new article is in the results.

At the time we were ingesting 4,000-5,000 articles per day. With N live subscriptions,
that's N × 4-5k full-text queries a day, not spread out nicely but clumped into spikes
whenever a news wave hits, hammering the same cluster that serves customer search
traffic.

You can refine it: filter every stored query down to the one new document's ID, so
each check touches almost nothing. That variant is a legitimate, battle-tested
pattern, and it would have limped along fine at our size. But it still runs N queries
against the live cluster for every article, all to answer a yes/no question about
one document you're already holding in your hand: does this document match this
query?

So no, not fatal. Wasteful. And wasteful was enough to send us looking for something
cheaper. The expensive lesson was waiting on the cheaper path.

Wrong answer #2: match in application code

So we did what almost every alerting feature in history does: we matched in application
code.

You've probably written this exact thing. Load the active subscriptions, and for every
incoming item, run a filter function:

function matches(article, sub) {
  const text = (article.title + ' ' + article.content).toLowerCase();
  return sub.keywords.every((k) => text.includes(k.toLowerCase())) &&
    (sub.sources.length === 0 || sub.sources.includes(article.source));
}

Keyword alerts in a Slack bot, price alerts in a shop, job alerts on a listings site:
under the hood it's nearly always this shape. toLowerCase(), includes(), maybe a
regex on an ambitious day. And it's genuinely attractive: microseconds per check,
zero extra infrastructure, zero load on your search cluster.

Our version was an industrialized flavor of the same idea. We already had a parser that
compiled our query syntax into OpenSearch DSL for the REST API, so we gave it a second
backend that emitted JavaScript predicates as strings, stored those in DynamoDB next to
each subscription, and revived them with new Function() per article. (Yes, eval-ing
strings from a database. They came from our own compiler, not from users, so injection
wasn't the issue. But it remained the kind of architecture you explain to another
engineer in a slightly lowered voice.)

The flavor doesn't matter, though. Hand-rolled includes() chain or compiled
predicate, it's the same category: matching in application code, next to a search
engine that matches differently. Ours ran in production for about a year.

The drift

Here's what we underestimated: not a bug, but a category difference.

When OpenSearch matches rate cuts, the text has been through our analyzer chain:
tokenized, lowercased, unicode-normalized, stemmed. "cuts", "cut" and
"cutting" collapse into the same root. Word order and punctuation are handled by query
semantics, not by luck.

An application-code matcher does none of that. Whatever the flavor, it's a string
check. Which means:

REST API, query rate cuts → finds the article "Fed cuts rates again", the exact pair from the top of this post. Stemming maps cuts→cut, rates→rate, and multi-word queries compile to AND over stemmed terms; trivial match for a search engine.
WebSocket, same query → the literal substring rate cuts appears nowhere in "Fed cuts rates again". Silence.

Then the support messages started:

"Why didn't I receive this article over my WebSocket? When I run the exact same query
against your REST API, it's right there."

This one hurt more than a normal bug report. Remember the tip from our docs, the one
about validating your query against the REST API first? Customers were doing exactly
that. The validation workflow we ourselves recommended was structurally incapable of
working.

We sat down and catalogued the differences between the two matchers, and found 13
distinct classes of divergence. A sample (a few of these depend on how we configured
our analyzers; not all of it is out-of-the-box behavior):

Divergence	REST search (OpenSearch)	Push matcher (string check)
Multi-word queries	`apple earnings` matches "Apple Reports Strong Earnings", any order, any field	phrase-as-substring in one field: no match
Phrase vs. loose	quoted `"apple earnings"` requires the words adjacent and in order; unquoted doesn't	both compile to the identical `includes()`
Stemming	`rate cuts` matches "Fed cuts rates"	nothing
Unicode width	`Apple` matches full-width `Ａｐｐｌｅ` in Japanese text	different bytes: no match
Composed vs. decomposed	`café` (U+00E9) matches `café` (e + combining accent)	no match
Accent folding	`Erdogan` finds `Erdoğan`	no match
Arabic normalization	alef variants and diacritics folded	raw comparison
CJK segmentation	dictionary-driven tokenizers	there are no spaces to split on

Row three is the support ticket. The other twelve were waiting their turn.

The five not shown: nested company queries, wildcards, stopwords, a filter bug,
scoring. We had already been patching these one at a time, each fix small, correct,
and doomed, before we sat down and made the full list. Thirteen individually
patchable divergences added up to one conclusion: we were rebuilding a search
engine's analysis chain in application code, badly, and it would never converge.
The drift wasn't a defect. It was the architecture.

And here's the uncomfortable generalization:

If your product has a search bar backed by a real engine and alerts matched by
includes(), you have this bug. Your users just haven't diffed the two channels
yet.

The breaking point: multilingual support

For English-only, you can almost live in denial. Then we decided to support more
languages, including Chinese, Japanese and Korean.

CJK languages don't put spaces between words. Tokenizing them isn't a regex problem;
it's dictionary- and model-driven segmentation. In OpenSearch you reach for dedicated
analyzers: kuromoji for Japanese, nori for Korean, smartcn for Chinese, plugins all of
them, and each a serious piece of engineering in its own right. And every language
brings a bag of quirks: compound-word splitting, reading forms, stopword curation,
width normalization. Japanese alone will humble you.

Now look at that from the predicate matcher's perspective. Our options:

Rebuild all of that in Node, inside a Lambda, and keep it bit-compatible with OpenSearch forever. A hobby project's worth of computational linguistics per language, with "stays identical to the real engine" as an eternal, manual QA problem.
Embed a different search library in the matcher. But a different engine means different analyzers and different semantics. We'd be institutionalizing the drift instead of eliminating it.

Spot the common flaw: both options mean tediously aligning a second matching engine
with OpenSearch, and never being done. Consistency across channels was the actual
product requirement, the one thing that support message demanded, and neither option
could deliver it. We genuinely evaluated alternative engines and walked away every
time with the same bad taste.

And then our prayers were answered

I wish I could say we designed our way out. The truth is humbler. While digging through
OpenSearch's documentation for something else entirely, we found a feature we had never
heard of:

The percolator.

We had spent weeks agonizing over how to keep two matching engines consistent, and
the tool we were already running in production could natively do the thing we
needed. Not a plugin. Not a sidecar. A built-in field type sitting in the docs,
shipping in core since 2016 with ancestors back to Elasticsearch 0.15 in 2011,
waiting for us to scroll far enough.

The pattern even has a name: prospective search, sometimes reverse search. It's
the pattern behind Google Alerts-style systems; Google even shipped a Prospective
Search API on App Engine once. Lucene has an entire library for it (Luwak, since
absorbed into Lucene as the Monitor module). We had simply never needed the words, so
we had never searched for them.

A normal index stores documents and you throw queries at it. A percolator index stores
queries, as documents in a dedicated percolator field type, and you throw a
document at it:

PUT /subscriptions
{
  "mappings": {
    "properties": {
      "query":  { "type": "percolator" },
      "title":  { "type": "text", "analyzer": "english" },
      "source": { "type": "keyword" }
    }
  }
}

PUT /subscriptions/_doc/sub_42
{
  "query": {
    "bool": {
      "must": [{
        "match": {
          "title": { "query": "rate cuts", "operator": "and" }
        }
      }],
      "filter": [
        { "term": { "source": "www.reuters.com" } }
      ]
    }
  }
}

GET /subscriptions/_search
{
  "size": 1000,
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "title": "Fed cuts rates again",
        "source": "www.reuters.com"
      }
    }
  }
}

Three details are load-bearing. The analyzer: the stemming that started this whole
story lives in that line. The operator: and: every term must match, which is exactly
when stemming matters, because without stemming rate never matches rates, and one
failed term sinks the whole query. And the size: the default is 10, and with enough
subscriptions a truncated hit list is silently missed notifications.

The response is the set of stored queries matching this document. Which is literally
the question from Wrong Answer #1, "does this document match this query?", asked N
times at once, natively, by the engine itself. The mirror image from the top of this
article isn't an analogy. It's the feature.

Why this killed the alignment problem dead

The mechanism matters: when you percolate a document, OpenSearch builds a tiny
in-memory index from it using the percolator index's own mapping and analyzers
(which mirror the article index's), then runs the stored queries against it. Same
tokenizers, same stemmers, same normalization as our article indices. Not "kept
compatible with" OpenSearch, but OpenSearch itself. There is no second engine to
align. What remains is config-level: each percolator index must keep mirroring its
article index's analyzers. A chore, but greppable and testable, unlike behavioral
drift.

Which means all the language machinery is defined once and inherited by both
directions: the kuromoji tokenization chain for Japanese, nori's compound-splitting
modes for Korean, smartcn plus a curated stopword list for Chinese, the stemmers and
unicode normalizers for everything else. Onboarding a new language means writing its
analyzer config once, and interactive search and real-time matching both speak it,
guaranteed to agree on what matches, from day one. Under the old architecture, every
piece of that machinery would have needed a hand-written, forever-maintained
JavaScript twin.

Because our REST API and our subscription registration already shared one query parser,
both directions now compile to the same DSL and execute on the same engine. The REST
validation workflow we recommend to users went from aspirationally true to
guaranteed by construction (with one scoring caveat, covered in the trade-offs
below).

Production: 10,000 articles a day, 5-20 ms a batch

The migration itself was pleasantly boring, the best kind. Today's shape:

One percolator index per language, each mirroring the analyzer configuration of its article index. The subscription's language decides where its query is stored; the article's language decides which index it's percolated against.
Subscriptions sync via change streams. Registering a WebSocket query or webhook writes to DynamoDB exactly as before; a small Lambda tails the table streams and upserts or deletes the corresponding percolator document. Subscription ID = document ID, so the sync is idempotent, and the realtime API's write path never touches OpenSearch directly. (The stream is eventually consistent; a fresh subscription starts matching within the sync lag, typically well under a second.)
Batched matching. Articles leave the enrichment pipeline in small batches; a dispatcher Lambda percolates each batch in a single _msearch: one sub-request per article, routed to that article's language index. A typical percolate batch completes in 5-20 ms, and the whole matching workload sits well under one percent of cluster load.

Part of why it's so cheap is a lovely detail of the percolator itself: at registration
time the engine extracts terms from every stored query and uses them to pre-select
candidates, so only queries that could plausibly match ever run against the in-memory
document. Matching an article against hundreds of stored queries, a corpus totaling a
few megabytes, is nothing like querying millions of articles.

The new Function() matcher (the second parser backend, the predicate strings, the
matcher loop) was deleted. Nobody held a funeral.

Ingest volume has roughly doubled since the migration, to around 10,000 articles a
day, and the matching layer is the one part of the pipeline that has never needed a
second thought.

The recipe, if you're building this

Stripped of our war story, here's the general pattern for real-time alerts on any
OpenSearch/Elasticsearch-backed product:

Create a percolator index that mirrors your document index: same analyzers, same field mappings for everything queryable. Add a field of type percolator for the query itself, plus whatever metadata you need (subscription ID, channel, owner).
Sync subscriptions into it. On create/update/delete, upsert or remove one document: the compiled query plus metadata, with the subscription ID as _id so the sync is idempotent. Tail your subscription store's change stream (we use DynamoDB Streams) or write through an outbox; just don't couple your API's write path to the cluster.
Percolate every incoming document. Batch with _msearch; each response is the list of subscription IDs to notify. Set size well above your worst-case match count: the default is 10, and truncated hits are missed notifications. Hand the result to your delivery fan-out.
Keep non-search concerns out of the stored queries. Authorization, quotas, dedup: apply them in application code after matching, or every policy change becomes a reindex.
Going multilingual? One percolator index per language, each mirroring that language's analyzer config. Route by the subscription's language when storing, by the document's language when percolating.

That's the entire architecture.

The honest trade-offs

The percolator isn't magic, and you should hear the sharp edges from me:

Matching is effectively binary. Percolation does score its matches, but against a single document there's no meaningful relevance cutoff. Our REST search drops low-scoring hits below a quality threshold; percolation has no equivalent. For alerting that's usually what you want, but it is a semantic difference between pull and push, accepted consciously.
Mapping changes fan out. Every article field that should be queryable in subscriptions must be added to every language's percolator index. Adding two metadata fields once touched over twenty index definitions. The consistency guarantee has a maintenance tax.
Language switches need care. A subscription lives in exactly one language index. Change a subscription's language and you must delete-then-register, or you leak an orphaned query that matches forever.
Authorization stays outside. Which sources a customer may receive depends on their plan, not their query. Bake that into stored queries and you'll be reindexing customers every time pricing changes. Percolation stays about search semantics; access control runs in application code afterwards.

What I'd tell past-me

Don't rebuild your search engine's semantics in application code, not even with an innocent includes(). Analyzers, stemming, CJK segmentation: that's not "string matching with extra steps". Your reimplementation starts drifting the moment you ship it.
Consistency has to be a property of the architecture, not of your diligence. If two code paths must agree, make them the same code path. "We'll keep our clone aligned with the real engine" is a promise you have to re-keep after every single upgrade, and one day you won't.
Read your tools' feature lists, all the way down. We evaluated entire alternative engines while the answer sat in the docs of the database we already operated. Sometimes the answer to your prayers has been shipping since before your product existed.

Same parser, same DSL, same analyzers, same engine. Just run backwards. The support
messages stopped.

finlight.me is a financial news API with full-text search and real-time delivery over
WebSockets and webhooks. If you want to watch the percolator's output land on a socket,
the live demo is the fastest way.

Integrate Real-Time Financial and Geopolitical News into Make.com Workflows with finlight

Ali — Tue, 12 Aug 2025 08:00:00 +0000

Introduction

In this post, you’ll learn how to integrate real-time financial and geopolitical news into your Make.com workflows using the finlight API.

We’ll cover how to:

Trigger automations instantly when market events happen
Run scheduled news searches for reporting or research
Filter results at the source so only relevant data reaches your workflows

The Problem

Until now, Make.com scenarios had no direct and reliable feed for structured financial or geopolitical events.

If you wanted an alert for a corporate earnings report or a major political development, you had to use RSS feeds, scrapers, or delayed APIs — messy and often too slow.

The Solution: finlight + Make.com

The finlight Make.com integration adds two key modules:

Webhook Trigger (Pro tier) – Fires instantly when a new article matches your query.
Search Module (All tiers) – Retrieves targeted news articles on demand or on a schedule.

You can filter by ticker, exchange, source, keywords, and more, all server-side, so your Make.com scenario only processes what’s relevant.

📄 Full API and query documentation: docs.finlight.me
🔗 Official Make.com integration page: make.com/integrations/finlight

Example 1: Real-Time Tesla and Apple Breaking News Alert

Trigger: Webhook with advanced query:

(+ticker:TSLA OR +ticker:AAPL) AND ("earnings" OR "quarterly results") AND NOT crypto

Actions:

Send a formatted message to Slack.
Log the event in Google Sheets.

Use cases:

Monitor portfolio companies in real time
Alert marketing or sales teams about client news
Track competitor announcements

Screenshots:

finlight node configuration
Full Make.com scenario view

Example 2: Weekly Market Trends Report

Trigger: Schedule every Monday at 9 AM

Search Query:

("mergers and acquisitions" OR "market trends" OR "economic outlook") AND (exchange:NASDAQ OR exchange:NYSE)

Actions:

Aggregate articles from the past week and sort by relevance.
Create a Google Doc or PDF with headlines and summaries.
Email the document to marketing, research, or strategy teams.

Use cases:

Weekly market briefings
Research content for blogs or newsletters
Automated client update reports

Screenshots:

finlight node configuration
Full Make.com scenario view

How to Try It

Create a finlight account and check the documentation.
Connect finlight to Make.com via the integration page.
Build your first scenario using either the Webhook trigger (Pro tier) or Search module (all tiers).

💬 Have you built something cool with this? Share your scenarios in the comments — I’d love to see how you’re using real-time market and geopolitical data in your workflows.

Streaming Financial Data in Real-Time with Finlight’s WebSocket API

Ali — Wed, 30 Apr 2025 10:02:21 +0000

Looking to supercharge your financial applications with real-time news updates? Whether you’re building trading dashboards , market monitoring tools , or alert systems , Finlight’s WebSocket API has you covered.

Finlight is a powerful API platform for delivering real-time financial, market, and geopolitical news , designed for developers who want fast, filtered, and flexible access to the stories that move markets.

✨ What You’ll Learn

In this guide, you’ll learn how to:

Connect to Finlight’s WebSocket API
Authenticate securely
Subscribe to live news streams
Keep your connection healthy with ping-pong
Understand how it compares to REST

By the end, you’ll have a working real-time stream of relevant financial news — perfect for trading, analytics, or alerts.

Setup & Requirements

Before we dive in, here’s what you need:

✅ A Finlight API Key — grab one from your dashboard at app.finlight.me

✅ Basic JavaScript or WebSocket knowledge

✅ Node.js (if using the SDK) or your preferred WebSocket-compatible language

You can use our official SDKs or roll your own integration:

npm : finlight-client
Python : finlight-client

🔍 REST API vs WebSocket

Here’s how the two options compare — choose the one that fits your real-time needs best:

REST API

Data Access : Fetch articles on demand via HTTP requests
Sorting : Articles are returned sorted by their publishDate (i.e., when the news provider originally published them)
Delayed Articles : Late-arriving stories (e.g., from aggregators like Yahoo) appear in the past based on their original publish date
Real-Time Capable? Yes — if polled frequently and tracked carefully
Best For : Custom polling systems, historical queries, or hybrid real-time setups

WebSocket API

Data Access : Stream new articles in real time as they are ingested into Finlight
Sorting : No manual sorting needed — articles are pushed in the order they enter the system
Delayed Articles : Even late stories appear immediately upon arrival (regardless of publish date)
Real-Time Capable? Yes — native real-time by design, no polling required
Best For : Dashboards, trading bots, alert systems, or anything needing instant updates

💡 Key Insight : REST is great if you need on-demand control or history.

But if you want true push-based real-time without manual effort, WebSocket is the way to go.

🔐 Authentication

Every WebSocket connection must include your API key in the header:

headers: {
  'x-api-key': '<your-api-key>',
}

💡 You can find your API key at https://app.finlight.me

WebSocket Endpoint

Connect to:

wss://wss.finlight.me

Real-Time News in Action (Node.js Example)

Here’s how easy it is to stream live financial news using our npm client:

import { FinlightApi } from "finlight-client";

const client = new FinlightApi({
  apiKey: "<your-api-key>",
});
client.websocket.connect(
  {
    // 'query' is optional – if omitted, you receive all incoming articles
    query: "Tesla",
    language: "en",
    extended: true,
  },
  article => {
    console.log(article);
  }
);

Available Query Parameters

You can filter the incoming stream by these optional parameters:

{
  query?: string; // e.g., "interest rates" – omit to receive all articles
  sources?: string[]; // e.g., ["www.reuters.com", "www.ft.com"]
  language?: string; // default: "en"
  extended?: boolean; // full article text if true
}

Highlights:

query is optional — leave it out to stream everything
sources supports multiple domains
extended: true gives you full article text (if available), summaries are provided in both versions

Keeping the Connection Alive (Ping & Reconnect)

Finlight’s WebSocket API is powered by AWS API Gateway WebSockets , which means:

Connections are automatically closed after 2 hours
The server will disconnect you after 10 minutes of inactivity

Send Pings Every ~8 Minutes

Send this JSON message to keep your connection alive:

{ "action": "ping" }

You’ll receive:

{ "action": "pong" }

Example:

setInterval(() => {
  if (socket.readyState === WebSocket.OPEN) {
    socket.send(JSON.stringify({ action: "ping" }));
  }
}, 8 * 60 * 1000);

Reconnect Gracefully

If the server closes the connection (after 2h or inactivity), reconnect:

socket.onclose = () => {
  console.log("Connection closed. Reconnecting...");
  setTimeout(() => {
    connectWebSocket(); // your connection logic here
  }, 1000);
};

Make sure to restart your ping interval after reconnecting.

Example Article Response

{
  "title": "Trump tariffs could lead to a summer drop-off...",
  "source": "www.cnbc.com",
  "publishDate": "2025-04-20T18:02:08.000Z",
  "language": "en",
  "confidence": 0.9999,
  "sentiment": "negative",
  "summary": "...",
  "content": "Full article content if `extended` is true..."
}

💳 WebSocket Pricing & Connection Limits

Your Finlight subscription tier determines how many simultaneous WebSocket connections you can use:

Pro Standard — Includes 1 WebSocket connection
Pro Scale — Includes 3 WebSocket connections
Custom Tier — Supports custom limits , ideal for enterprise or high-volume setups

Final Thoughts & Use Cases

With Finlight’s WebSocket API, you can build:

📊 Live trading dashboards
🚨 Real-time alert systems
🧠 News sentiment analytics
🤖 Automated trading or LLM agents

Explore more:

Got questions? Curious to see more examples? Want to share what you’re building?

Tell me here or at our Discord.

Scaling Search at finlight.me: From Postgres Full-Text to Real-Time OpenSearch

Ali — Wed, 30 Apr 2025 10:00:00 +0000

Scaling search isn't just about adding bigger servers — sometimes you need the right tools.

When we first launched finlight.me, our real-time financial news API, Postgres full-text search was more than enough. It was fast, easy to set up, and fit perfectly into our simple early architecture. But as the number of articles grew and search demands became more complex, cracks started to appear. In this article, I'll share how we moved from Postgres to OpenSearch, the challenges we faced along the way, and why keeping Postgres as our source of truth turned out to be one of our best decisions.

It all started with a simple full-text search setup inside Postgres that worked surprisingly well — until it didn’t.

First Architecture: Postgres Full-Text Search

In the early days, we used Postgres’ built-in full-text search to power article queries. Titles and content were combined into a single tsvector field, allowing us to search efficiently without worrying about casing, suffixes, or keyword order — limitations that basic %query% searches would have struggled with. Incoming search queries were also converted into vectors, and Postgres did a solid job of ranking and returning relevant results. For a while, this setup handled our needs with fast response times and minimal overhead. It was simple, integrated, and worked well alongside the rest of our ingestion and storage system.

But as our article volume started to grow and user queries became more complex, we began to notice cracks forming beneath the surface.

Pain Points with Scaling Postgres

At first, Postgres full-text search handled our growing dataset reasonably well. But as article counts climbed into the hundreds of thousands, search performance started to noticeably degrade. The major issue wasn’t just the full-text search itself — it was how users combined free-text queries with additional filters like publish date ranges, specific sources, or metadata fields. Postgres was strong at indexing individual fields, and its GIN index accelerated full-text search. However, we quickly ran into a hard limitation: Postgres doesn’t allow combining a GIN index with a regular B-Tree index in a composite index. This meant we couldn’t optimize both kinds of queries at the same time, forcing the database to either pick a suboptimal plan or fall back to sequential scans — both of which became painfully slow as the dataset grew.

Index Management Nightmare: Optional Parameters and Growing Complexity

The flexibility of our API — allowing users to combine any subset of filters like publish date, source, free-text search and more — introduced another layer of scaling challenges. Since every search parameter was optional, we faced a combinatorial explosion of possible query patterns. To maintain acceptable performance, we had to create different indexes to support the most common combinations of parameters. Each time we added a new searchable field, it required designing new indexes, analyzing query plans with EXPLAIN ANALYZE, and validating performance manually. This constant index tuning became tedious and unsustainable. Worse, despite all the effort, the core limitation remained: we still couldn’t efficiently optimize full-text search combined with metadata filters in a single query.

Pagination Collapse and User-Visible Slowness

As article volumes continued to grow, another performance bottleneck surfaced: pagination. Our API exposed a page parameter that mapped directly to SQL OFFSET behavior behind the scenes. While this worked fine at low offsets, performance deteriorated rapidly as users requested deeper pages. Ironically, the very feature that should have made searches faster — returning just a small slice of results — ended up making things slower. Each paginated request forced Postgres to scan, count, and skip thousands of rows before it could even start returning results, recalculating large parts of the query plan every time. Queries that once took under a second ballooned to tens of seconds. At that point, it was no longer just an infrastructure problem — it became a user experience failure. We realized that even a well-tuned Postgres setup wouldn't be enough to support fast, flexible search at the scale we were growing toward.

Why We Chose OpenSearch

It was clear that we needed a system built specifically for search — something optimized for free-text queries, filtering, and fast pagination at scale. Having worked with Elasticsearch during previous freelance projects, we were already familiar with the strengths of dedicated search engines: inverted indexes, efficient scoring algorithms, and powerful query flexibility. We decided to adopt OpenSearch, the community-driven fork of Elasticsearch, both for its strong technical capabilities and its more favorable licensing model.

At the same time, we made an important architectural decision: to separate the write path from the read path. Postgres would remain our single source of truth for ingested and processed articles, ensuring data integrity and consistency. OpenSearch would serve as the read-optimized layer, delivering fast and flexible search without overloading our ingestion pipeline. This allowed us to use the best tool for each requirement — a reliable, relational, normalized database for storage and ingestion, and a high-performance search engine for querying — instead of trying to force one system to do everything.

Testing Phase: Starting Small and Learning Fast

Before fully committing to production, we rolled out OpenSearch in a minimal-resource testing setup: a single-node cluster with limited RAM, intended purely for evaluation and tuning. In this environment, we quickly encountered behaviors that hinted at the system's scaling needs. Over time, we noticed missing indexes and degraded search performance — symptoms likely caused by memory pressure and resource eviction events on the hosting side. Far from being a setback, these early tests validated an important lesson: while OpenSearch could deliver the performance we needed, it demanded production-grade resources to do so reliably. Testing lean allowed us to tune index mappings, validate query performance, and plan capacity based on real behavior. It also reinforced our architectural choice to keep Postgres as the source of truth, ensuring that even if the search layer needed recovery or rebuilding, the core data remained safe and consistent.

Scaling OpenSearch for Production

Armed with insights from our testing phase, we moved to a production-grade OpenSearch deployment with the resources needed to match our growth. We added multiple nodes to the cluster, allocated sufficient RAM, and tuned index mappings to optimize both write and query performance. With the new setup, search response times dropped dramatically — even complex queries with deep pagination returned results in milliseconds instead of seconds.

The overall data flow evolved as well: after articles pass through our real-time article processing pipeline — where they are collected, enriched, and analyzed — they are immediately fed into OpenSearch for fast retrieval. Postgres remains the single source of truth, storing all raw and processed data reliably, while OpenSearch acts as the read-optimized layer tuned for search performance.

We also introduced regular snapshotting of the OpenSearch indices, ensuring that even as the article base grew, we could recover quickly from failures or rebuild indexes without downtime. Treating OpenSearch as an advanced cache rather than the primary database gave us flexibility: we could evolve search schemas, rebuild indexes, or adjust mappings without putting core data integrity at risk. Over time, as traffic increased and our dataset expanded, the new architecture continued to perform reliably under load.

Today’s Architecture: Resilient, Real-Time Search

Today, our system cleanly separates responsibilities between ingestion, storage, and retrieval. Postgres continues to act as the single source of truth, reliably storing all raw and processed article data in a normalized relational structure. Articles flow through our real-time article processing pipeline — where they are scraped, enriched, and analyzed — before being fed into OpenSearch for optimized search performance. OpenSearch handles all user-facing search queries, allowing us to deliver fast, flexible results even under high load. Regular snapshotting, thoughtful index management, and a multi-node deployment ensure that our search infrastructure remains resilient and scalable as our data set grows. By decoupling the write and read sides of the architecture and choosing the best tool for each need, we've built a system that is fast, reliable, and ready for continued growth.

Lessons Learned: Advice for Builders

Start simple, but design with scale in mind. Postgres full-text search served us well early on — but flexibility in design made migration possible later without major pain.
Separate read and write paths as early as practical. Trying to make a single database handle everything becomes exponentially harder as complexity grows.
Use the right tool for the job. A relational database excels at storage and consistency; a search engine excels at flexible retrieval and ranking.
Don’t underestimate optional query complexity. Supporting flexible API filters sounds simple until you have to index every possible combination.
Test lean, scale smart. Early testing with minimal resources taught us what production-grade OpenSearch really needed — and avoided costly surprises.
Keep a reliable source of truth. Having Postgres behind OpenSearch allowed us to rebuild, heal, and extend our search infrastructure without risking core data integrity.

If you're building scalable APIs or working with large search datasets, I'd love to hear how you're approaching similar challenges. Feel free to share your thoughts or experiences in the comments!

How I Built a Multi-Agent AI Analyst Bot Using GPT, LangGraph & Market News APIs

Ali — Wed, 23 Apr 2025 10:01:28 +0000

Yes, I know — this title sounds like one of those overly long anime names. “That Time I Built a GPT Bot That Read the Financial News So I Didn’t Have To” 🧙‍♂️📉

1. Intro: The Pain

I used to wake up and check four different sites, scroll Twitter, and still feel behind on what actually happened in the markets.

Now, I get one clean email every morning with exactly what I need: key headlines, a few bullet points of summary, and a tone check on the overall market mood — powered by GPT and a financial news API.

Here’s how I built it in under an hour — and how you can too.

2. The System

🛠 Stack_:_

Finlight.me — Financial news API for clean, market-relevant headlines
LangGraph — Multi-agent GPT orchestration framework (built on LangChain)
Python — With a cron job or AWS Lambda for scheduling
SMTP — To deliver the final email briefing

🧠 Agents_:_

Analyst Agent Pulls news for a given subject → summarizes → classifies tone (bullish, bearish, neutral)
Composer Agent Takes all outputs from the Analyst and assembles the final morning email

🔄 Flow_:_

⏰ Script runs daily at 6:45am
For each subject (e.g., Inflation, AI Stocks) the Analyst Agent :
Queries relevant news via API
Summarizes key points
Assesses sentiment (bullish/bearish/neutral)
After all subjects are processed:
The Composer Agent creates the full morning briefing
📬 Email lands in your inbox before 7:00am

3. What the resulting email looks like

🗓 April 13, 2025 – Market Briefing

• Trump Tariffs Escalate 🇺🇸🇨🇳
   - The U.S. imposed a 145% tariff on Chinese imports, while China retaliated with a 125% tariff, urging a complete removal of these measures.
   - U.S. Commerce Secretary hinted at upcoming tariffs on semiconductors to promote domestic production.
   - Despite concerns of inflation, layoffs, and supply chain disruptions, exemptions on smartphones and laptops provided a temporary boost for tech giants like Apple and Nvidia.
   - Market sentiment remains highly volatile with fears of a U.S. recession and worsening global trade dynamics.

• China Tightens Its Stance 🇨🇳
   - China halted critical rare earth exports, exacerbating the trade war and affecting key industries like semiconductors and aerospace.
   - Beijing dismissed U.S. tariff exemptions as insufficient and continued to push for cancellation of all reciprocal tariffs.
   - On the global stage, India’s and Brazil’s industries are reaping benefits as alternatives in electronics and agriculture.
   - Allegations of Chinese cyberattacks targeting U.S. infrastructure have further strained relations.
   - Domestically, Hong Kong’s last major opposition party moved towards disbandment, while Britain rejected Chinese involvement in its steel sector, fuelling political tensions.

📊 Sentiment: Uncertain and risk-averse, with increasing global market volatility. Investors are eyeing alternative regions like India for stability.

This is what lands in my inbox every day. 30 seconds to understand the market mood.

4. Inside the System: The Multi-Agent Morning Briefing

Now that you’ve seen the output, let’s unpack how it works under the hood.

At a high level, this is a multi-agent workflow that runs on a schedule. Each agent handles a specific task — fetching news, summarizing, analyzing tone, composing a message — and they all pass info via shared state.

Everything is built in Python using LangGraph, a framework for agent orchestration.

🔌 4.1: Pulling News with the Finlight API

We use a small wrapper around a financial news API — Finlight.me — to get clean, focused headlines for a given subject.

Why Finlight? Because it gives you way less noise than other general news APIs. And also… because I built it. 😎

Here’s the tool I use inside the system — it wraps the Finlight SDK as a LangChain-compatible StructuredTool:

# tools/finlight.py
from langchain.tools import StructuredTool
from finlight_client import FinlightApi
from finlight_client.models import BasicArticleResponse, ApiConfig, GetArticlesParams
from pydantic import BaseModel, Field, field_validator

from market_briefing.config import FINLIGHT_API_KEY

class GetBasicArticle(BaseModel):
    query: str = Field(..., description="Search term, e.g., 'Nvidia'")
    from_: str = Field(..., alias="from", description="ISO 8601 start time")
    to: str = Field(..., description="ISO 8601 end time")
    pageSize: int = Field(100, description="Number of articles per page")
    page: int = Field(1, description="Page number")

    model_config = {"populate_by_name": True}

def search_finlight_articles(params: GetBasicArticle) -> str:
    client = FinlightApi(config=ApiConfig(api_key=FINLIGHT_API_KEY))

    api_response: BasicArticleResponse = client.articles.get_basic_articles(
        params=params
    )

    if not api_response:
        return "No articles found."

    return "\n".join(
        f"Title: {a['title']}\nDate: {a['publishDate']}\nSummary: {a['summary'] or 'No summary.'}\n{'-'*30}"
        for a in api_response["articles"]
    )

search_tool = StructuredTool.from_function(
    func=search_finlight_articles,
    name="search_finlight_articles",
    description="Search Finlight for news by keyword and date range",
)

This allows agents in your system to call Finlight dynamically using structured input and output — with proper validation via Pydantic, and easy chaining through LangChain or LangGraph.

🧠 Note: This input will later be fed into GPT for summarization and sentiment.

4.2: The Shared State

All agents share one data structure to read/write context. It’s a simple Python TypedDict, but it powers the whole pipeline.

from typing import List, Dict, TypedDict

class BriefingState(TypedDict):
    subjects: List[str]
    current_index: int
    analyst_outputs: Dict[str, str]
    briefing: str

subjects: list of user-defined topics (like ["inflation", "oil", "semiconductors"])
current_index: used for looping through one subject at a time
analyst_outputs: filled in by the Analyst Agent and consumed by the Composer
briefing: filled in by the Composer Agent with the final result

🧠 Why this matters: it allows us to loop over multiple topics with a single reusable agent_, and conditionally switch to the next phase when done._

4.3: The LangGraph Logic

The diagram shows the core agent flow:

Analyst Agent is the workhorse. It loops through each subject (like “Inflation” or “AI Stocks”), pulling and analyzing news one at a time.
After each subject, it either:
🔁 Loops back to process the next subject
➡️ Passes control to the Composer Agent once all subjects are processed
Composer Agent then compiles the final email briefing.

This loop-until-done + handoff pattern is a core use case for LangGraph: small, focused agents collaborating via shared state.

from langgraph.graph import StateGraph, END
from market_briefing.workflow.state import BriefingState
from market_briefing.agents.analyst import analyst_agent_node
from market_briefing.agents.composer import compose_briefing_node

def increment_index_node(state: BriefingState) -> dict:
    return {"current_index": state["current_index"] + 1}

def should_continue(state: BriefingState) -> str:
    return (
        "continue" if state["current_index"] + 1 < len(state["subjects"]) else "format"
    )

def build_graph():
    graph = StateGraph(BriefingState)

    graph.add_node("analyst", analyst_agent_node)
    graph.add_node("increment_index", increment_index_node)
    graph.add_node("composer", compose_briefing_node)

    graph.add_conditional_edges(
        "analyst",
        should_continue,
        {"continue": "increment_index", "format": "composer"},
    )

    graph.add_edge("increment_index", "analyst")
    graph.set_entry_point("analyst")
    graph.add_edge("composer", END)

    return graph.compile()

This defines a clean LangGraph flow:

Start with the Analyst Agent
If there are more topics, increment index and repeat
When all topics are done, switch to Composer Agent
Finish at END

🧠 Modular and readable. Each agent focuses on one job — no giant monolith functions.

4.4: The Analyst Agent

This is the workhorse. For each subject, it:

Pulls relevant news from Finlight
Passes it to GPT for summarization
Extracts sentiment
Stores the result in the state

Here’s the full code:

from datetime import datetime, timezone
from langchain_core.messages import HumanMessage
from market_briefing.llm.executor import agent_executor
from market_briefing.workflow.state import BriefingState
import logging

logger = logging.getLogger( __name__ )

def analyst_agent_node(state: BriefingState) -> dict:
    subject = state["subjects"][state["current_index"]]

    now = datetime.now(timezone.utc)
    from_time = now.replace(hour=0, minute=0, second=0, microsecond=0)
    to_time = from_time.replace(day=from_time.day + 1)

    iso_from = from_time.isoformat() + "Z"
    iso_to = to_time.isoformat() + "Z"

    prompt = f"""
    Summarize financial/political developments on '{subject}' in the last 24h (from {iso_from} to {iso_to}).
    Include what happened, market sentiment, and confidence if available. Keep it short.
    """

    logger.info(f"📤 Prompt to analyst agent:\n{prompt}")

    result = agent_executor.invoke({"messages": [HumanMessage(content=prompt)]})
    outputs = state.get("analyst_outputs", {})
    outputs[subject] = result["messages"][-1].content

    logger.info(f"📝 Final formatted briefing:\n{outputs[subject]}")

    return {"analyst_outputs": outputs}

🧠 Prompt Design Tip: Keep structure consistent so parsing is easy. You want predictable outputs if you ever want to use this downstream.

4.5: The Composer Agent

Once the loop is done, the Composer takes over. It assembles the full email — Markdown-style — by summarizing all the analyst outputs into a single readable message.

Here’s the code:

from datetime import datetime, timezone
from langchain_core.messages import HumanMessage
from market_briefing.llm.executor import agent_executor
import logging

logger = logging.getLogger( __name__ )

def compose_briefing_node(state: dict) -> dict:
    now = datetime.now(timezone.utc).strftime("%B %d, %Y")

    logger.info("🧩 Composing final morning market briefing...")
    logger.info(f"📦 Current subjects and summaries: {state['analyst_outputs']}")

    joined_summaries = "\n\n".join(
        f"🔹 {subj}\n{summary.strip()}"
        for subj, summary in state["analyst_outputs"].items()
        if summary and isinstance(summary, str)
    )

    formatting_prompt = f"""
You are a professional financial news editor. Format the following topic summaries into a clean, well-presented morning market briefing.

------------

✅ Do:
- Use headlines
- Add emojis and good spacing
- Improve clarity where needed
- Skip sections that have no information available
- Dont repeat yourself

🚫 Do not:
- Guess or fill in missing content
- Say anything unrelated to the actual summaries

🗓️ Date: {now}

------------

Use this template:

🗓 <Date in Month, Day - Year> – Market Briefing

• <Bullet Point 1>
• <Bullet Point 2>
...
• <Bullet Point n>

📊 Sentiment: <General sentiment>

------------

Here are the summaries:

{joined_summaries}
"""

    logger.info(f"📤 Prompt to composer agent:\n{formatting_prompt}")

    result = agent_executor.invoke(
        {"messages": [HumanMessage(content=formatting_prompt)]}
    )
    final_message = result["messages"][-1].content

    logger.info(f"📝 Final formatted briefing:\n{final_message}")

    return {"briefing": final_message}

💡 This is where tone and readability matter. The analyst agent gives raw data — the composer agent gives it polish.

4.6: Running the Whole Thing

You trigger the whole workflow from a handler script. This is where subjects are defined and the LangGraph pipeline is run:

from market_briefing.config import SUBJECTS
from market_briefing.utils.html_formatter import markdown_to_html
from market_briefing.sender.email_sender import send_daily_email
from market_briefing.workflow.graph import build_graph
import logging

logging.basicConfig(level=logging.INFO) 

def main(event=None, context=None):
    state = {
        "subjects": SUBJECTS,
        "current_index": 0,
        "analyst_outputs": {},
    }

    graph = build_graph()
    result = graph.invoke(state)

    print("✅ Morning Briefing Generated:\n", result["briefing"])

    briefing = result["briefing"]
    html = markdown_to_html(briefing)
    send_daily_email(html)

    return {"statusCode": 200, "briefing": result["briefing"]}

Simple, clean, and callable from cron, Lambda, or a notebook.

4.7: Sending the Email

Once the Composer Agent builds the final message, we need to send it as an email. Here’s the full implementation:

import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

from market_briefing.config import EMAIL_PASS, EMAIL_SUBJECT, EMAIL_TO, EMAIL_USER

def send_daily_email(html_content: str):
    # Create MIME message
    msg = MIMEMultipart("alternative")
    msg["From"] = EMAIL_USER
    msg["To"] = EMAIL_TO
    msg["Subject"] = EMAIL_SUBJECT

    # Attach HTML content
    msg.attach(MIMEText(html_content, "html"))

    # Send email
    with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server:
        server.login(EMAIL_USER, EMAIL_PASS)
        server.send_message(msg)

This uses standard smtplib + MIMEText to format and send HTML emails via Gmail.

📫 Pro Tip: You can easily swap in any other provider — Mailgun, SendGrid, Outlook — or even push to Slack or Telegram.

4.8: Going Serverless with AWS Lambda

Want your system to run every morning, even while you sleep? Here’s how I deployed it with the Serverless Framework:

service: market-briefing

plugins:
  - serverless-python-requirements

provider:
  name: aws
  runtime: python3.11
  region: us-east-1
  timeout: 60
  memorySize: 512
  environment:
    PYTHONPATH: /var/task/
    OPEN_AI_API_KEY: ${env:OPEN_AI_API_KEY}
    FINLIGHT_API_KEY: ${env:FINLIGHT_API_KEY}
    EMAIL_USER: ${env:EMAIL_USER}
    EMAIL_PASS: ${env:EMAIL_PASS}
    EMAIL_TO: ${env:EMAIL_TO}
    EMAIL_SUBJECT: ${env:EMAIL_SUBJECT}
  iam:
    role:
      statements:
        - Effect: Allow
          Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
          Resource: "*"

functions:
  briefer:
    handler: market_briefing/handler.main
    events:
      - schedule:
          rate: cron(0 7 * * ? *) # every day at 07:00 UTC
          enabled: true

package:
  patterns:
    - '!**'
    - market_briefing/**

custom:
  pythonRequirements:
    dockerizePip: true
    slim: false
    strip: false
    fileName: requirements.txt

This will:

Deploy your Python app as a Lambda function
Schedule it to run daily (cron-style)
Keep your environment variables safe with .env or AWS Secrets Manager

☁️ Tip: If you’re using Ollama or local models, you can skip this and run it via a cron job on your own machine.

Bonus: The Config File

from dotenv import load_dotenv
import os

load_dotenv()

OPEN_AI_API_KEY = os.environ["OPEN_AI_API_KEY"]
FINLIGHT_API_KEY = os.environ["FINLIGHT_API_KEY"]
EMAIL_USER = os.getenv("EMAIL_USER")
EMAIL_PASS = os.getenv("EMAIL_PASS")
EMAIL_TO = os.getenv("EMAIL_TO")
EMAIL_SUBJECT = os.getenv("EMAIL_SUBJECT", "Daily Email")

SUBJECTS = ["Trump tariffs", "Nvidia", "China"]

You can easily:

Switch out topics
Plug in new LLM providers
Redirect output to different email addresses

Recap: How It All Comes Together

Let’s tie it all together:

You define what you care about (in SUBJECTS)
Each subject is processed by the Analyst Agent , which pulls news, summarizes it, and scores sentiment
Results are stored in a shared state
When all subjects are processed, the Composer Agent assembles a clean, skimmable morning email
The Email Sender delivers it to your inbox
All of this runs daily via cron or Lambda

It’s modular, clean, and surprisingly easy to extend.

🚀 What’s Next?

Right now, this works great with a static list of subjects. But there’s so much potential to go further.

Here’s what I’m exploring next:

Smart Topic Detection

Instead of passing predefined subjects, let an LLM scan the entire news feed , detect key themes, and generate briefings dynamically.

Portfolio-Aware Briefings

Pull in your portfolio holdings or watchlist, and prioritize news that impacts your assets.

Local Execution

Swap OpenAI for Mistral , Mixtral , or Gemma running locally via Ollama, for:

Full data privacy
No API costs
Offline support

More Data Sources

Add financial calendars, earnings transcripts, Twitter trends, Reddit sentiment, or macro indicators.

💻 Want the full code?

I’m open-sourcing the entire repo. Just drop a comment if you want early access to the GitHub link. You can run it locally or in the cloud — whatever fits your setup.