DEV Community: saint-james-fr

Build a No-Code Tech Newsletter Intelligence Agent with n8n and Vector Database

saint-james-fr — Sun, 04 May 2025 21:26:09 +0000

Fighting mental fatigue...

Do you ever feel overwhelmed by tech newsletters? As a fullstack developer, you need to stay up to date across multiple areas — frontend, backend, databases, infrastructure, and more.

Personally, I’m subscribed to at least 15 newsletters, and I love them all — spoiler alert: I list most of them at the end of the article. But if I miss a day or two, my “DEV” folder starts overflowing. I often spend hours on the weekend catching up — reading inspiring articles, testing snippets, and discovering new tools. It’s one of the best ways to grow and stay sharp.

But let’s be honest — keeping up with everything is exhausting.

So last weekend, I gave myself a challenge: build a system to manage the overflow using as little code as possible, combining AI and automation. I’m no expert— I had only heard of embeddings and we use some at work— but I pulled it off. And you can too. 🙂

What Are We Building?

We’ll build a system that pulls content from your favorite tech newsletters, stores it in a vector database, and lets you query them conversationally — all in (almost) no-code using n8n, Pinecone, a bit of JavaScript, and of course OpenAI - but you can use an other AI provider of course.

We’ll need three workflows but one of them is optional:

Gmail Trigger: It is going to poll our gmail account and if there are new emails, it will get their full details, including html, create the embeddings and store them in the vectorized database.
Chat Interface: This will leverage n8n’s chat interface to prompt ChatGPT with results from the vector database in an agentic way.
(Optional) Webhook for Historical Ingestion: This workflow processes older newsletters in bulk and required some code. I'll treat it last.

Services we’ll use:

Gmail
OpenAI API
Pinecone (free tier)
n8n (free trial)

I’ll assume that, like me, you’re using Gmail, and that you can set up forwarding rules.Also, you’ll need Node.js installed to run a small script if you go for the optional step.

Oh and just before we go, let's just get a quick reminder about a concept we gonna use: what is an embedding ?

An embedding converts text (or images, audio, etc.) into a vector — a list of numbers — that represents its meaning in a way a machine can understand. We store it in a vectorized database.
Example:

Text: "How to build an agent?"

Embedding: [0.12, -0.98, 1.45, ...]

Let's synchronize emails

Create a Gmail account

I think it's easier not to mix everything as it only takes 3min, I recommend you create a new email adress for this exercise.

Forward your newsletters

I already had a rule that filtered all tech newsletters into a “DEV” folder. I just added a forward rule to send them to the new address.

Connect Gmail to n8n

Ok let's dive in, you need a free trial n8n subscription and create a new workflow from scratch. You select the "on app event" trigger, choose Gmail and "Message received". From there you can create a credential using Oauth and connect to your newly created account. For testing, you can send yourself an email.

One of the feature I liked the most on n8n, just like my previous experience with Integromat, is the possiblity to listen for an input (for example a webhook or an event) and analyze the input received. Immensely useful, given the fact that you can pin it for the next step also.

Get the full email content

The trigger doesn’t give you the full HTML body. You need to add a Get Message node (still using the Gmail integration) and uncheck “Simplify” to get the raw HTML.

Handling multiple emails?

I remember having to create loop and handle array in my previous no-code experience with Make Integromat. But with n8n it's much more simpler. Thanks to their "multiple items" processing model, n8n automatically handles arrays of items by processing each one through the entire workflow. It's like every node is accepting an array - even if there is only a single element inside. This is brillant!

Convert to Embeddings & Store in Pinecone

Now, we need to convert our html piece of code into a more LLM-friendly data format and this is why we need to convert it to an embedding. Embeddings come in different formats, depending on the model we use.

Create a Pinecone index

We’ll use the OpenAI text-embedding-3-small model (1536 dimensions), which is more than enough for newsletter HTML.

Thanks to Pinecone serverless architecture, creating the index is very straightforward, I didn't touch to any of settings. The only thing we're going to need is its id, which we set up in the first input fields.

Set up the Pinecone node with embeddings

Create the node, enter your index ID and link it to the OpenAI embeddings node. This is very easily done with n8n, when a module needs another module, it just appears below, you click it and can navigate in a sub-menu to set everything up. Here you just have to click first on Embedding, choose the same model and enter your OpenApi API key.

Set up the Pinecone node with document

We set up a Default data loader to access the html field but we need to set what is called a separator. We choosed a basic setting as it was mentioned it will cover most use case: the recursive character text splitter. Why? In n8n, you're loading html content and need to break it into smart chunks for further processing. The separator defines how you split it. I'm sure there is era for improvement here as I choose the default and set options for split code to html.

You've just created your first AI pipeline!

Query the Database via Chat

Here I was a bit lost but thanks to n8n great automation examples I just had a look at how someone analyzed a PDF and then chat with the result of it. That was close enough to our use-case. Let's create another workflow.

Use n8n's Chat Interface

Basically, instead of relying of an app's event, here we choose to have a chat integration. n8n really made a difference here. You can keep your chat private, in the workflow, but also make it public, and even put it behind a login step (username/password or just your n8n credentials). Out-of-the-box!

Create an AI Agent in n8n

The AI agent will use our tool when needed and attach it to ChatGPT when building prompts. This is simply done again by filling up the modules. That's why our agent needs:

A connection to ChatGPT (the model)
Access to the Pinecone vector DB (the tool)

n8n connects these easily via built-in modules. Just fill the needed informations and credentials.

Craft a good system prompt

A well-written system message boosts the answer quality. Here’s mine:

  You are an expert assistant specialized in technologies and web development.
  You have access to a curated database of newsletters from top-tier sources, each provided with full HTML source code.
  When answering, you must:
  - Extract insights directly from the HTML content
  - Identify the original sender by examining the HTML metadata
  - Clearly cite the newsletter source and reception date
  - Always include the original link when available

  Your responses should be accurate, concise, and well-structured.
  Prioritize verifiable information from the newsletter content itself.

Backfill Old Emails (optional)

I wanted to ingest my backlog of newsletters too. The idea was simple : get my data from Google, transform it into JSON, send it via HTTP to a n8n worklow to store it in database. Let's do it!

Export from Gmail via Google Takeout

Actually, it's quite simple to get your data from Google. There is a service for this called Google Takeout. I choosed the Gmail service, my DEV folder and... kabooom! A few moments later I got a zip with my data.

I tried to open it and found inside a html page with a mbox data folder... I was not familiar with this format but it's used by many email clients to exchange data. But my n8n workflow don't speak MBOX.

Convert MBOX to JSON

A little research led to me to this repository. Just clone the code and run the command mentioned in the README to get your emails as Json. Easy peasy!

  # Instal dependencies
  npm i
  # Run the script (no need to use node, script uses a shebang)
  ./mbox-to-json.js --input my_emails.mbox --output my_emails.json

Ok we finally have our JSON. What's the idea? Maybe we can build a little JS script to parse it, and send it via a POST request to another n8n workflow with a webhook that can receive data.

Send JSON to n8n via Webhook

Basically, I just copied our Gmail trigger workflow and changed the initial step to a webhook. You can chose an authentification method, just buid a token using openssl and put in your header of your choice.

  # It's easy to build a simple token using openssl
  openssl rand -hex 16

Analyzing answer

Humm... Something is wrong. the body of our request contains an array of data, we need to map them before passing them to our Pinecode node - the same we used before. And here I must say I fell in love for real with n8n : you can write javascript code very easily inside the node. With the help of my good friend Claude, I got the syntax right (it's not that easy at first to understand why you need $input.first()) and then it's just basic JS.

Here comes the trouble...

But here I encoutered many difficulties that I'll resume here

First, my data was quite big, around 200Mo of JSON, this was way too much of a payload. I need to batch it.

Then, even after batching it in slice of 10 elements, I've realized that the limits of my free instance of n8n : it does not have much memory, some operations are difficult to parallelize and making 10 async calls in JS with 5 workflow running didn't work for me.

What worked was batching it and adding a delay. As creating embeddings and inserting in Pinecode database was a 20-30s job, I used this delay amount in my script to make sure only one workflow was running at a time.

Even with this, some failed so I decided to record which index of the batches was failing to retry later in a specific retry mode in my script. Also, because of the delay, handling 282 batches of 10 elements was quite long so I also decided to store what should be the next batch to start if I had to stop.

I won't bother you with the code here, just take it from this repository and make it yours - all informations are also resumed on the README.

After a few hours....all newsletters are indexed. My agent was ready !

Test & Use

Let’s see it in action.

Query: “Do you have any ressources about building components in Vue.js”

→ GPT scans the vector DB and responds with summarized insights, linked to the source newsletter and sender.

If you did the optional part, you saw that n8n's cloud instance has memory and parallelism limits. But it’s manageable...and they even give you advises for this.

Next step for me would probably be to self-host n8n on a VPS to have more RAM and CPU available hat as it'a possibility offered by them and that seems quite well documented. It was my first try with this tool and vectorized database and I must admit this is amazingly powerful. Can't wait to try other scenarios....

I hope this tutorial was helpful! As I said at the start — I’m still learning in both the no-code and AI space, and if I can build this, you definitely can too. I'm coming from a code background so using these tools requires me to adopt a specific logic but after this first step, I must admit I loved the experience.

Also, if I had to code this from scratch, I wouldn’t be writing this article — I'd still be debugging regex filters 😄

Please reach out if you have feedback or ideas on how to improve the flow or integrations.

Bonus: My favorite newsletters

ByteByteGo
Node Weekly
Javascript Weekly
Tyler from ui.dev
TLDR / TLDR Web Dev
Neo Kim
Frontend Focus
Smashing Magazine
Weekly Vue News
Codepen
WizardZines / Julia Evans
Trevor I Lasn
Quastor
Modern CSS Newletter
Sequoia
Engineering Leadership
Dev Community Newsletter
pganalyze
The New Stack

Integrate Strapi in Rails or how I decided to build my first gem strapi_ruby

saint-james-fr — Fri, 20 Oct 2023 09:47:06 +0000

TL;DR:

This article introduces the strapi_ruby gem, a solution for integrating headless CMS Strapi with Ruby on Rails seamlessly. Strapi is an ideal choice for a CMS, but using it effectively in a Rails environment and keeping your code DRY can be tricky. If you're a Rails developer willing to use Strapi, this is for you!

Strapi: modern, easy and flexible

I came across a problem. I'm a Fullstack developer and I have worked with Ruby On Rails since a year, and despite the joy I try to transfer to my clients for using Rails, I often heard the need for a CMS like Wordpress. In a world dominated by content strategies, articles, videos, tutorials or interviews often lay the foundation for better referencing within a business.

In the Rails world, some gems exist but nothing actually gets high enough to compare to the Wordpress standards needed by the client or really combine in Rails.

I discovered the Strapi headless CMS solution and I just loved how good it is. Users and developers get a proper UI, lot of tweaking possibilities and nobody will mess up some Wordpress configuration or introduce nasty plugins, or worse, forget to update Wordpress, theme and plugins. Forget about all that.

Using a headless solution means end-users will have a separate interface to manage content, and actually, that's what everybody wants as a real-world solution.

Using Strapi in a Rails environment means you decided to deliver the content using a Server-Side Rendering strategy - which means having your Rails controller prepare the articles to the views. This could be a better SEO strategy than fetching it client-side with Javascript. But this is clearly post-rationalisation: I was doing front-end for a week now on the project and I wanted to write some Ruby.

Let's get motivated

So, that's it, let's use Strapi. Having in mind some vague architectural concepts, I coded my best - which was not that good - and tried to have classes and modules organized but this ended up a bit too close to the needs of the client and not enough modular.

Looking at the code six months later and waiting for work to show up, I decided to externalise the service and its ideas into a gem, maybe influenced by the Hacktoberfest which is happening right now. I love the idea that freelance status also offers more time to participate in Open-Source projects or build stuff for the community.

I set up some objectives like having an API as clean and smooth as possible, writing tests, better organizing my code, and of course offering all options offered by Strapi REST API parameters.

I did love the rush of dopamine when starting the project but sooner or later, you have to make choices about your project. I set myself a deadline of 6 days and I must say, the final form and architecture became good enough at day 3. It helped me understand a few things :

You don't have to get it right immediately. It's okay to change the design if you see what could go wrong. Better late than never!
I thought it was an amazing thing to extract some code you wrote months before you can see how you improved and that's motivating - and it's for the good cause.
Looking for inspiration, I did check some gems repos (not related to Strapi), and also some gems that would interact with Strapi and Jekyll. I would highly recommend this if you wanna start to build a gem. Just open one and see how it's set up. Of course, I often heard this advice but I thought it was really interesting to do it while building a gem.

Back to the gem

Anyway created the strapi_ruby gem, so it could be as easy as passing options to a simple API.

Also, I wanted my gem to even more ease the situation by converting what selected fields from Markdown to HTML or converting ISO formatted time to regular Ruby DateTime instances.

So let's dive in!

Installation

First, we need to download, install and configure the gem

# Gemfile

# ...
gem 'strapi_ruby'

bundle
bundle exec rake strapi_ruby:config

It will generate a config file in an initializer. Fill your Strapi server URI and token, using environment variables or Rails credentials.

# config/initializer/strapi_ruby.rb

StrapiRuby.configure do |config|
  config.strapi_server_uri = ENV["STRAPI_SERVER_URI"]
  config.strapi_token = ENV["STRAPI_SERVER_TOKEN"]
end

That's it, I'm ready to fetch some data! Let's get some articles for our home page.

Fetching articles collection

# pages_controller.rb
def home
@articles = StrapiRuby.get(resource: :articles)
end

What we get is an OpenStruct instance similar to the JSON answer of Strapi, you can access data by dot notation. Each sub fields have also been converted to Openstruct so you can navigate easily.

# home.html.erb

<%= @articles.data.first.attributes.title %>

Gracefully degrading when errors happen

But we'd rather access these in our views. And there's a simple reason for that. Let's say you reference a specific article and this one is deleted. This will trigger an StrapiRuby::ClientError and you don't want this to block your application. We need to escape an empty answer.

# home.html.erb

<% StrapiRuby.escape_empty_answer(@articles) do %>
  <ul>
    <% @articles.data.each do |article| %>
      <li>
        <%= article.attributes.title %>
      </li>
    <% end %>
  </ul>
<% end %>

The block will return an empty string and the error will be logged in the console if there is any error.

Handling errors should be an essential part of your gem if you know it will trigger runtime exceptions. I found it to be almost half of the job, especially in the last days of my challenge.

Debugging

Same thing about debugging. This is what we do every day at some point.

Outside the block, you can still access the answer and get information about the error. Imagine requesting a non-existing resource because you typoed it.

# pages_controller.rb

def home
@articles = StrapiRuby.get(resource: :articless)

@articles.data # => nil
@articleS.meta # => nil
@articles .error.message # => "The Strapi servers answer with an error status 404. Not Found"
@article.endpoint # => "URL_OF_MY_SERVER/articless"
end

This is what happens in the gem.

Begin
# some code for getting the answer ...
Rescue ClientError e=>
# return OpenStruct.new(data: nil, meta: nil, error: OpenStruct.new(message: e.message), endpoint: @endpoint)
end

What do you think about this solution? I would love to know more how someone else would handle this and the graceful degradation.

Build queries with ease

You can access all Strapi V4 parameters jut as populate, fields, sort, filters, locale, and pagination...

Using it in Strapi with Javascript is easy enough using Strapi interactive query builder. Or have a look at the Strapi excellent documentation.

Strapi use the qs javascript library which traverses a JS object to build a query. strapi_ruby makes it as easy with an equivalent function.

# Using $in operator to match multiples values
StrapiRuby.get(resource: :restaurants,
               populate: :*,
               filters: {
                 id: {
                   "$in" => ["3", "6", "8"],
                 },
               })
# StrapiRuby.endpoint
"YOUR_SERVER/restaurants?populate=*&filters[id][$in][0]=3&filters[id][$in][1]=6&filters[id][$in][2]=8"

As you'd expect, some JS syntax must be translated into Ruby though.
We need to use strings or symbols as values, we can pack them into arrays that's no issue. For operators of filters, we need to pass them as strings because of the $ character.

# Complex filtering with $and and $or
StrapiRuby.get(resource: :books,
               filters: {
                 "$or" => [
                   {
                     date: {
                       "$eq" => "2020-01-01",
                     },
                   },
                   {
                     date: {
                       "$eq" => "2020-01-02",
                     },
                   },
                 ],
                 author: {
                   name: {
                     "$eq" => "Kai doe",
                   },
                 },
               })
# StrapiRuby.endpoint
"YOUR_SERVER/books?filters[$or][0][date][$eq]=2020-01-01&filters[$or][1][date][$eq]=2020-01-02&filters[author][name][$eq]=Kai%20doe"

Convert Markdown to HTML

Strapi is transferring Rich Content as Markdown so you can fully leverage its CMS potential. Convert with ease the fields you want using this option in the config file or in your options.

# This way
StrapiRuby.configure do |config|
  config.convert_to_html: [:body]
end

# Or this way

StrapiRuby.get(resouce: :restaurants, convert_to_html: [:menu, :description])

Real-World Scenarios

Let's consider a few scenarios:

Blog Websites: For a blog site built with Ruby on Rails, using Strapi as the content management system is a wise choice. With strapi_ruby, you can quickly fetch, render, and display blog posts in a Rails application while keeping your codebase clean.
E-commerce Stores: In e-commerce applications, product details and descriptions are critical. The gem simplifies the retrieval of product information from Strapi, ensuring an up-to-date and error-free product catalogue.
Actually, any services that need separation from content and views. Let the creators create content!

Conclusion

I hope you found this article informative and that it encourages you to build yourself your own gem or participate in Open-Source projects.

Don't hesitate to visit the repo for the full code, I'd love some advice. Feel free to connect, comment, and help me improve as you just read my first article - and this was my first gem!

Thank you for reading, and happy coding 😎