I have always liked RSS because it gives me a easy way to decide for myself which sources I want to follow, without relying on social networks or (commercial) recommendation platforms to decide what appears in front of me. I want to be in control of the content I see myself.
Years ago, I built a small application called RSSMonster to collect those feeds in one place. It started as a fairly simple, Google Reader-inspired project: fetch articles, organize them into feeds and folders, mark them as read, and keep track of the things I wanted to return to later.
For a long time, that was enough.
Over the last few months, however, I have been revisiting the project much more seriously. Coding assistants such as Claude Code and Codex made it practical to modernize an older codebase, revisit decisions, improve the look and feel, and experiment with features that had been sitting on my backlog for years.
In one of my medium.com articles, I described how AI changed the way I develop RSSMonster.
The part that I would like to share here are the usage of small language models and whether they should be considered within local project. Some background: I initially used OpenAI for RSSMonster’s semantic features, including embeddings and other AI-related processing. That worked well, but as the amount of content being processed increased, I also saw the API-related costs going up. In my own case to around €10 per month. That is not an enormous amount, but for a self-hosted RSS reader it made me question whether commercial inference really needed to be the default.
This led to a different experiment: could small language models running locally take over (some of) this work? Could they provide useful embeddings, generate summaries, and classify articles while running on common hardware? And how would their results compare with the commercial OpenAI models I was already using?
To investigate this, I added two local small language models to RSSMonster: Qwen and ModernBERT. For those who are not familiar with Qwen and ModernBERT.
Qwen is a family of compact language and embedding models developed by Alibaba, with smaller variants that are practical to run locally on CPU. In RSSMonster I use Qwen for embeddings, summaries, tags, and semantic labels.
ModernBERT is a modernized BERT-style encoder model that is well suited to classification tasks, which I use for things such as information quality, promotional content, and emotional tone.
To measure the performance of these models, I used the existing semantic regression suite that I had developed to analyze embedding results. Next, I asked Codex to analyze the log files, identify differences, and explain what those differences meant for the application, and how to optimize the usage and performance.
For the comparison, I asked Codex to analyze the two generated log files. For example below you see the embedding process that is benchmarked. Both models passed all tests, but the resulting semantic structures were noticeably different:
| Metric | OpenAI | Local Qwen |
|---|---|---|
| Articles evaluated | 778 | 778 |
| Articles assigned to events | 48 | 54 |
| Articles assigned to topics | 46 | 50 |
| Active events | 20 | 22 |
| Active topics | 16 | 17 |
| Topic-island article paths | 5 | 4 |
| Vector-fallback island paths | 28 | 45 |
| Standalone articles | 702 | 679 |
| Incremental articles assigned to events | 18 | 19 |
| Duplicate groups | 1 | 1 |
| Duplicate articles | 2 | 2 |
I also created a small diagnostic fixture containing 23 manually labeled article-like inputs and 23 taxonomy candidates, including closely related concepts such as content I like to read (Football, Tech, F1, etc.).
The experiment compared different ways of preparing the embedding inputs:
| Strategy | Correct first-choice matches | Average separation margin |
|---|---|---|
| Existing symmetric representation | 22/23 | 0.1278 |
| Taxonomy as instructed query | 22/23 | 0.1617 |
| Article as instructed query | 23/23 | 0.1306 |
The most interesting outcome of the embedding benchmark is that no model clearly won. Both OpenAI and Qwen passed the regression tests, but they behaved differently inside the RSSMonster content processing pipeline.
RSSMonster uses embeddings to organize articles at several levels:
- Events group articles that cover the same real-world story.
- Topics connect related events into broader subjects.
- Interest Islands are longer-lived clusters inferred from what a user tends to read.
OpenAI was generally a little more conservative, creating fewer event and topic associations. Qwen grouped more articles together and often found broader relationships, which was useful for event detection but required more care when assigning content to longer-lived Interest Islands.
Small models are credible, but not drop-in replacements
That was the point where it became clear that switching embedding models was not a drop-in replacement. The differences forced me to make a few adjustments to the semantic processing engine itself. In particular, I had to treat similarity thresholds and fallback behavior as model-specific rather than assuming that one set of values would work equally well across providers.
Qwen performed better than I expected. In several cases it created more complete event and topic groupings than OpenAI, while still passing the same end-to-end tests.
But switching embedding models is not just changing one configuration value.
The smaller taxonomy experiment reinforced this point: changing how the input text was prepared sometimes had as much impact as changing the model itself.
So if you are evaluating embeddings, I would test:
- the model;
- how much and what kind of content you embed;
- the similarity thresholds;
- the actual downstream behavior of your application.
One useful lesson for me was that more input text is not always better. We improved the results by feeding the embedding model less content, but making that content more focused and meaningful. Removing noisy or redundant text made the representation cleaner and helped the semantic matching behave more consistently.
A quality score starts with a sentence
One surprising lesson from working with ModernBERT was how much of the classification behavior starts with the input I give it.
For RSSMonster, I wanted to assess promotional content, emotional tone, and writing quality. Using the NLI variant of ModernBERT, I could express those tasks as candidate descriptions. For promotional content, these include “purely editorial,” “partly promotional,” and “strongly promotional.” The model evaluates how well the article supports each description, and RSSMonster converts the resulting probabilities into score buckets, which can be used for ranking the articles in the list. For example, promotional content scores lower in the list.
This allows a small local model to perform classification without first training it on a collection of manually labeled RSSMonster articles. The task is defined through language, while the output remains structured enough for the application to use.
Local models move the trade-off
The original motivation was partly cost. OpenAI was easy to use, but my RSSMonster instance had reached around €10 per month in API usage.
Running locally removes the per-request cost and keeps article content and reading behavior on the host, but it introduces other costs: CPU, memory, model storage, startup time, and more operational complexity.
That means the choice is not simply cloud versus free local inference.
It is more like:
Hosted models: easier to operate, but with an external dependency and recurring API costs.
Local models: more control and data locality, but also more responsibility for runtime stability and hardware. For example, I had to implement retry logic for local inference, whereas OpenAI's API has generally been much more stable.
For a self-hosted application, I think that trade-off is worth considering seriously.
Different jobs need different models
Another conclusion was that there is little reason to force one model to do everything.
In RSSMonster, the workloads are quite different:
| Capability | Model |
|---|---|
| Embeddings | OpenAI or Qwen3 Embedding |
| Summaries and tags | OpenAI or local Qwen3.5 |
| Quality, tone, promotional classification | ModernBERT |
| Conversational assistant | OpenAI |
This hybrid approach works better because each model gets a narrow, well-defined responsibility.
That is probably the main takeaway I would give other developers:
Small local models become much more useful when you stop asking them to be general-purpose AI systems and give them one specific job.
For RSSMonster, the experiment did not prove that Qwen or ModernBERT are better than OpenAI. It showed something more useful: small local models can be good enough to support a real semantic pipeline, provided you evaluate them against the behavior your application actually needs.
For me, that is the real takeaway. The question is not whether a local model wins a benchmark, but whether it performs the specific job well enough to justify the operational trade-offs.
Top comments (0)