DEV Community: Nox

I built a privacy-focused search engine from scratch (and learned why search is hard)

Nox — Fri, 24 Jul 2026 19:18:20 +0000

I built Slick because I wanted a search engine that actually felt like it belonged to the user.

Not another search wrapper.

Not another app that just sends your query somewhere else.

A real search engine.

At first, I thought:

"How hard could it be?"

You crawl some websites, put them in an index, add a search box, and you're done... right?

Turns out, not even close.

Search is one of those things that looks simple until you try to build it yourself.

What is Slick?

Slick is an independent, privacy-focused search engine that I'm building from scratch.

The goal is simple:

Give users more control over how they search the web.

Features include:

Privacy-focused searching
Customizable ranking
Independent web indexing
Semantic search
Multiple search providers
Personalized ranking controls
A Google-like experience without tracking

The project started as an experiment, but quickly became a deep dive into crawling, information retrieval, machine learning, and infrastructure.

The first problem: getting data

A search engine is useless without an index.

The first major component I built was the crawler.

It needs to:

discover pages
download content
extract useful information
remove duplicates
handle dynamic websites
prepare documents for indexing

The goal isn't to blindly crawl the entire internet.

The goal is building a useful, high-quality index.

The search engine itself

For indexing, Slick uses Elasticsearch. By default we use DuckDuckGo's index, as Slick is still maturing.

Each document contains information like:

title
URL
description
page content
metadata
keywords

The initial search system uses traditional information retrieval techniques like BM25.

But keyword matching has a problem.

A user doesn't always search using the exact words found on a page.

For example:

"how do I stop websites tracking me"

and:

"privacy tools that prevent online surveillance"

mean almost the same thing.

A keyword-only search engine sees two different queries.

Humans don't.

Adding AI to search

To improve this, Slick uses semantic search.

Pages and queries can be converted into embeddings that represent meaning instead of just text.

The ranking pipeline combines:

traditional keyword relevance
semantic similarity
reranking models
quality signals

The idea is to combine the strengths of old-school search with modern machine learning.

The hardest part: ranking

Getting results is easy.

Getting the right results is hard.

A search engine can return thousands of pages that technically match a query.

But users don't want thousands of pages.

They want the one that answers their question.

Ranking involves constantly asking:

Is this result actually relevant?
Is this page high quality?
Does this match the user's intent?
Should another result be ranked higher?

This is where most of the engineering effort goes.

Building it as a solo developer

The biggest surprise was how much infrastructure search requires.

The UI is honestly the easy part.

The difficult parts are:

crawling
indexing
storage
ranking
optimization
handling bad data
keeping everything affordable

Large search engines have entire teams dedicated to one small part of this.

Building even a small one makes you appreciate how much work happens behind every search box.

Some things I'm experimenting with

Slick is still actively being developed.

Some areas I'm working on:

growing the independent index
improving ranking quality
better semantic retrieval
more customization options
better search categories
improving crawler reliability

The long-term goal is not to make "another Google clone."

It's to explore what search looks like when users have more control.

Try Slick

If you want to try it:
https://slicksearchhq.com/

The project is still early, so any feedback, suggestions, or ideas are greatly appreciated.

Thanks for checking it out!

Building a Search Engine Changed How I See the Web

Nox — Thu, 23 Jul 2026 13:05:00 +0000

When people use a search engine, they usually only see the final step.

A search box.

A query.

A list of results.

The complicated parts are hidden behind that interface.

Once you try to build a search engine yourself, you quickly realize that the internet is not a clean database. It is a constantly changing collection of websites created by millions of different people, companies, and systems.

Some pages are written to answer questions. Some are created to sell products. Some exist mainly because someone wanted to rank higher in search results.

A search engine has to figure out the difference.

That is the problem I started working on while building Slick, an independent search engine focused on customization, privacy, and giving users more control over their search experience.

The first challenge is finding the web

Before ranking results, before machine learning, and before semantic search, there is a simpler problem:

How do you collect the documents?

A search engine needs its own view of the web. That requires systems that can discover pages, fetch content, process information, remove duplicates, and keep everything updated.

Crawling sounds simple at first. You request a page, save the content, and move on.

In reality, the web constantly creates problems.

Pages disappear.

Servers block requests.

Content loads dynamically with JavaScript.

Multiple URLs can point to almost identical pages.

Some websites are designed for humans, not machines.

A crawler is not just downloading pages. It is trying to build a useful representation of the internet.

A bigger index does not automatically create better search

One of the easiest mistakes to make is assuming that more documents means better results.

It does not.

A search engine can index millions of pages and still provide poor answers.

The problem is not finding documents. The problem is deciding which ones are worth showing.

For example, a search for how to configure a Linux server might return:

Official documentation
A forum post from someone who solved the exact issue
A copied tutorial
An outdated blog post
An SEO page designed to attract traffic

All of these pages may contain similar keywords.

Only some of them are actually useful.

That is where ranking becomes difficult.

Search ranking requires multiple signals

Keyword matching is still one of the most useful tools in search.

Algorithms like BM25 are extremely effective at finding documents that match a query.

However, keywords alone do not always understand intent.

Someone searching for "best privacy browser" might not want pages that repeat those words the most. They may want comparisons, technical discussions, or opinions from people who have actually used different browsers.

Modern search systems combine many different signals:

Traditional information retrieval
Semantic search
Machine learning reranking
Document quality signals
User preferences

Each approach solves a different problem.

There is no single algorithm that understands every type of search.

AI makes search more important, not less

There is a common idea that AI will replace search engines.

The reality is more complicated.

AI systems are very good at transforming information into answers, but they still depend on finding reliable information first.

A model can summarize a document, but it does not automatically know if that document is accurate, outdated, or misleading.

Retrieval is still a fundamental challenge.

As AI systems become more common, the ability to find high-quality information becomes even more important.

Search should give users more control

Traditional search engines usually work in one direction.

The search engine decides how results are ranked, and users accept the output.

But different people have different needs.

A developer looking for documentation wants different results than someone researching a topic. A researcher may prefer academic sources, while someone debugging an issue may prefer forum discussions.

Search does not have to be identical for everyone.

Users should have more control over how results are ranked and what sources they prefer.

Privacy does not mean removing features

Privacy-focused products often face a difficult balance.

Collecting more data can improve personalization, but it also creates questions about how that data is stored and used.

A search engine should be able to provide useful features without requiring users to give up control of their data.

That is one of the challenges I have been exploring while building Slick.

What building search taught me

Building a search engine changed how I think about the internet.

The web looks organized when you browse it normally.

But when you try to index it, the complexity becomes obvious.

Every search result represents many decisions:

Should this page be indexed?
Is it relevant?
Is it trustworthy?
Is there a better source?
Should it rank higher?

The search box is the simplest part.

Everything behind it is where the real engineering happens.

Building search is not just about collecting more pages. It is about helping people find useful information in a constantly changing environment.

That problem is still worth solving.