DEV Community

Cover image for Turn any newsletter into a database
Yaobuilds
Yaobuilds

Posted on

Turn any newsletter into a database

#ai #javascript #tutorial #api

Newsletters are the strangest data feed on the internet. Airlines, deal hunters, job boards, real estate brokers, security teams: thousands of organizations run pipelines that end in a beautifully formatted email, and nowhere else. There is no API. There is no RSS. There is a designer-approved HTML table in your inbox, and that is the product.

Which means your inbox is quietly one of the best free data sources you have. You just cannot query it.

This tutorial fixes that. By the end, a flight-deals newsletter will be landing in a Postgres-queryable event stream as typed JSON, and the whole thing takes about fifteen minutes. Disclosure up front: I built the service used in the middle step (Inbin), after running the DIY version inside my own product first. The pattern works with any extraction layer, including one you build yourself.

The architecture 

Newsletter (Going, TLDR, anything)
        ↓  subscribes to
inbox address (yours-a3f2c8@in.inbin.dev)
        ↓  extraction against your schema
typed JSON event
        ↓  HMAC-signed webhook
your app / database
Enter fullscreen mode Exit fullscreen mode

The trick is the middle: instead of your personal email address, you subscribe the newsletter to a machine address, and every edition that arrives is parsed against a schema you declare once.

Step 1: create an inbox 

import { Inbin } from "@inbin/core";

const inbin = new Inbin({ apiKey: process.env.INBIN_API_KEY });

const inbox = await inbin.inboxes.create({ name: "flight-deals" });
console.log(inbox.address);
// flight-deals-a3f2c8@in.inbin.dev
Enter fullscreen mode Exit fullscreen mode

That address is permanent. It is also receive-only, which matters: the worst a leaked newsletter subscription can do is send you more newsletters.

Step 2: declare what every edition should become

This is the part that replaces the parser you would otherwise write. Describe the fields; do not describe how to find them.

await inbin.schemas.put({
  extract: {
    deals: {
      type: "array",
      items: {
        destination_city: { type: "string", required: true },
        origin_iata: { type: "string" },
        price_usd: { type: "number", required: true },
        airline: { type: "string" },
        travel_window: { type: "string" },
      },
    },
  },
  hallucination_guard: true,
});
Enter fullscreen mode Exit fullscreen mode

Two design notes:

Arrays are the right default for newsletters. One edition usually carries many records (five deals, ten stories, three listings). Declaring deals as an array means one email becomes five rows, not one blob.

Turn the hallucination guard on. Extraction here is LLM-based, and language models will confidently invent a price if you let them. The guard checks that every extracted string appears verbatim in the source email, and every number in its digits; anything it cannot back comes back null. Missing data is recoverable. Invented data is poison in a database.

Step 3: subscribe the newsletter

Go to the newsletter's signup page and enter the inbox address. That is the whole step. Some senders require confirming the subscription; the confirmation email also arrives as an event, so you can grab the confirm link from your dashboard or the API.

Step 4: receive rows, not emails

Point your webhook at an endpoint and verify the signature against the raw body:

import { verifyWebhook } from "@inbin/core";

export async function POST(request) {
  const raw = await request.text();
  const event = verifyWebhook(
    raw,
    request.headers,
    process.env.INBIN_WEBHOOK_SECRET,
  );

  for (const deal of event.extracted.deals ?? []) {
    await sql`
      INSERT INTO deals (city, price_usd, airline, received_at)
      VALUES (${deal.destination_city}, ${deal.price_usd},
              ${deal.airline}, ${event.received_at})
    `;
  }
  return new Response("ok");
}
Enter fullscreen mode Exit fullscreen mode

Every edition of the newsletter now becomes rows in your table, minutes after it is sent.

Step 5: skip the table entirely (optional)

If you do not want to run a database for this, the events are already queryable where they land:

curl -X POST https://api.inbin.dev/v1/query \
  -H "Authorization: Bearer $INBIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "flatten": "deals",
    "where": [{ "field": "price_usd", "op": "lt", "value": 300 }],
    "group_by": "destination_city",
    "aggregate": [{ "fn": "avg", "field": "price_usd", "as": "avg_price" }]
  }'
Enter fullscreen mode Exit fullscreen mode 
{ "rows": [ { "destination_city": "Lisbon", "avg_price": 274.5 } ] }
Enter fullscreen mode Exit fullscreen mode

Filter, sort, group, aggregate: the inbox is the database.

What people actually build with this

  • Price watchers: alert when any deal to a saved city drops under a threshold.
  • A personal news API: three newsletters in, one deduplicated JSON feed out.
  • Market history: newsletters are ephemeral; a table of every deal ever sent is not. Six months of data tells you what a genuinely good price looks like.
  • Agent food: the same events are exposed over MCP, so an AI agent can query them natively. That one gets its own article.

The general lesson holds beyond newsletters: any email your business receives on a schedule (invoices, shipping updates, alerts) is a feed pretending to be correspondence. Declare a schema and it stops pretending.

If you want to try the hosted version, it is free while in beta at inbin.dev.

Top comments (0)