As a developer, I am pretty sure I spend more time searching for solutions to coding problems than actually coding. Having reliable search results is incredibly important to me, so I don't waste time paging through useless results before finding that StackOverflow post or blog post from a developer that has the solution I am looking for.
For years, I have used Google as my main search engine, but the results have definitely been going down hill. Most search results come back with AI generated spam or something completely irrelevant to what I am looking for.
A large part of this is due to the way Google is funded. Google is an advertising company so they make their money from promoting the results from whoever has paid the most. Even if you ignore the sponsored results, can you really trust the rest of the results to be impartial? After all a large part of the web is usually running a combination of Google Analytics and Google AdSense so it is in Google's interest to promote those sites over others.
For a while I switched to DuckDuckGo as my main search engine, but I often found the results lacking and found myself adding !g to my search to revert to Google.
I heard a lot of people saying good things about Kagi, which is a paid search engine which promotes “No ads. No tracking. No compromise. Just deep, powerful search”. The results from Kagi are actually very good, and it has a number of different features such as being about down rank or block certain domains. I particularly like their Small Web “search” which helps promote smaller websites that might not appear in search results.
For those that want a no fuss solution for better private search then I would definitely give Kagi a try.
Despite it's benefits I cancelled my Kagi subscription this week in favour of my own self-hosted private search with searXNG (pronounced searching).
There was nothing particularly wrong with Kagi, I just can't bring myself to pay $10 a month for something as basic as search. Plus I always like a self-hosted solution if there is one! Even though Kagi says there is no tracking they could still link all your search history with your account if they wanted to.
SearXNG is a search aggregator so it relies on results from other search engines, but there are over 200 search engines you can add into the mix so you get the best results from all of them.
It also protects your privacy to stop companies like Google from profiling you for advertising. This is what it says on the website about tracking:
SearXNG protects the privacy of its users in multiple ways, regardless of the type of the instance (private or public). Removal of private data from search requests comes in three forms:
- Removing private data from requests going to search services
- Not forwarding anything from third party services through search services (e.g. advertisement)
- Removing private data from requests going to the results pages
Removing private data means not sending cookies to external search engines and generating a random browser profile for every request. Thus, it does not matter if a public or private instance handles the request, because it is anonymized in both cases. The IP address used will be the IP of the instance, but SearXNG can also be configured to use proxy or Tor. Result proxy is supported, too.
SearXNG does not serve ads or tracking content, unlike most search services. Therefore, private data is not forwarded to third parties who might monetize it. Besides protecting users from search services, both the referring page and search query are hidden from the results pages being visited.
You don't have to set up SearXNG yourself, you can use one of the many public instances. Of course if you opt for a public instance than you are putting your trust in the admin of the instance not to do anything with your data.
As I like self-hosting anyway it was a no-brainer to self-host my own instance on my server. There is separate repository for the SearXNG docker setup which makes it really simple to set up.
If you host SearXNG on your own network then it will be your public IP address that will be sent to the search engines. If you want to avoid this you have a few options:
- Host it in the cloud on something like Digital Ocean, AWS, or Railway (affiliate link, get $20 free credit)
- Configure it with a Proxy or Tor
- Run the docker container through a VPN
I opted for the last option as I already have a container set up running a VPN.
I have a gluetun container set up with my VPN:
version: "3"
services:
gluetun:
image: qmcgaw/gluetun
container_name: vpn
cap_add:
- NET_ADMIN
devices:
- /dev/net/tun:/dev/net/tun
ports:
- 8888:8888/tcp # HTTP proxy
- 8388:8388/tcp # Shadowsocks
- 8388:8388/udp # Shadowsocks
volumes:
- ./config:/gluetun
# https://github.com/qdm12/gluetun-wiki/tree/main/setup/providers
env_file:
- .env
networks:
- caddy
networks:
caddy:
external: true
This is then set up on the same network as my caddy container. My SearXNG instance is then setup to route all traffic through the VPN with network_mode: 'container:vpn'. If your containers are in the same docker-compose file you can use network_mode: 'service:vpn' instead.
version: "3.7"
services:
searxng:
container_name: searxng
image: docker.io/searxng/searxng:latest
network_mode: 'container:vpn'
restart: always
volumes:
- ./config:/etc/searxng:rw
- ./cache:/var/cache/searxng:rw
environment:
- SEARXNG_BASE_URL=https://search.mydomain.com/
- SEARXNG_SECRET=${SEARXNG_SECRET}
- UWSGI_WORKERS=${SEARXNG_UWSGI_WORKERS:-4}
- UWSGI_THREADS=${SEARXNG_UWSGI_THREADS:-4}
cap_drop:
- ALL
cap_add:
- CHOWN
- SETGID
- SETUID
logging:
driver: "json-file"
options:
max-size: "1m"
max-file: "1"
Finally, my Caddyfile has the following so that I can access SearXNG through the VPN container:
# Searxng
search.mydomain.com {
encode gzip
reverse_proxy vpn:8080
tls {
protocols tls1.3
}
}
The only other tweak I made to SearXNG was setting up the favicon resolver which is set in the SearXNG settings.yml file with:
search:
favicon_resolver: "duckduckgo"
Once this is set up you can pick duckduckgo from the settings in SearXNG. Your settings are saved in a cookie in your browser, but you copy then to another browser using a link found in the Cookie tab in settings.
I have been really pleased with SearXNG over the last week. The results come back in only a few seconds at most, they are on par with Kagi and I have peace of mind that I am not feeding even more data to Google or any other company. The fact I am also saving myself $10 a month is just an added bonus.
It would be nice if SearXNG had the option to downvote and upvote results like you can do in Kagi. There is an option to block certain domains using the hostnames plugin, but it looks a bit hacky.
It looks like it is possible to create plugins and custom themes for SearXNG so I am going to add that to my never ending project list!
❤️ Picks of the Week #
I didn't post anything last week as I was off spending time with the family. So there are 2 weeks worth of interesting finds this week!
📝 Article — Care Doesn't Scale — I often fall into this trap. Is there something I could be doing which has more impact? Why spend the time helping one person when I could be writing a blog post or video and help many. Care might not scale but helping one person is just as important and often more fulfilling than helping lots.
📝 Article — We can now fix McDonald's ice cream machines — I never knew this was why I could never get a milkshake at McDonald's when I was a kid. Now I know!
📝 Article — Before you buy a domain name, first check to see if it's haunted — I have always done this but never thought to write an article about it. If you find a good domain name now chances are someone has registered it before and let it lapse. Especially if it is for a new business site the last thing you want is to be associated with any negative reputation tied to the domain.
📝 Article — How 'Factorio' seduced Silicon Valley and me — I have so far managed to avoid playing Factorio, but it has been on my Steam wish list for a while. I can see why this game appeals to developers though. My current vice is Oxygen Not Included which I think scratches the same itch. It has a decent amount of complexity and I often need to refactor my bases into something more efficient.
📝 Article — Character amnesia in China — I always wondered how Chinese people manage to remember the 13,000 characters in their language, turns out they don't!
🛠️ Tool — You-get: Dumb downloader that scrapes the web — This looks like a useful too if you often find yourself downloading YouTube videos for viewing offline. I also recently came across Pinchflat which is a YouTube media manager that looks good for this too!
💥 Comic — Wells — I have always wondered about this. I ended up going down a bit of a rabbit hole on how wells work after reading this too. I wonder, did they know they were going to get water by digging deep enough or did someone just dig a really deep hole one day?
🛠️ Tools — Using Setapp — I generally try to avoid subscriptions if you hadn't guessed by this post. For between $10 and $15 a month however you can get access to a whole range of applications. Given I have a Mac Mini and a MacBook Air I would need the top plan, so I have avoided it so far. They have some useful apps on there though and if you are already paying a subscription for a few of them you could save yourself some money.
📝 Article — HTML Form Validation is underused — This week I have been adding validation to a complex form using react-hook-form. I must admit I didn't know about a lot of these native HTML validations.
📝 Article — How I write code using Cursor — I haven't used Cursor yet and given it is $20 a month I probably won't. It is interesting to see how other people are using it though. I do however have a pay as you go account for Claude which I have been using to help me with frontend code.
📝 Article — How to get the whole planet to send abuse complaints to your best friends — It is amazing that this is possible and there doesn't seem to be a way around it. Luckily you are only likely to be targeted if you run a Tor node but something to be aware of anyway.
💻 Tech — New Mac Mini with M4 — I have only had my Mac Mini for a few years, so I won't be upgrading anytime soon, but I love how small these are!
👾 Game — NandGame: Build a computer from scratch — I think it is important to know how computers work for software developers, but I must admit I have never gone into this much depth. This looks like it could be a fun way to learn though.
📚 Book — Make It Yourself — I am not a big fan of the format of this book. It is a PDF of wireframe images that when clicked linked to the particular project website. It would have been much better as a searchable website but oh well. It features some interesting projects that people have made.
📝 Article — Apple acquires Pixelmator — I have never used Pixelmator, but it looked like a good alternative to some of the Adobe suite. I wonder what this acquisition will mean for the Apple apps.
📝 Article — Unsaid — They seem to be putting AI into everything and not enough companies cover where the data is coming from and the ethics behind that. We already know that models from OpenAI and Midjourney are trained on data scrapped from the internet without permission. I think if they want AI to be really embraced it needs to be on legally sourced data and use a fraction of the current computing power.
📝 Article — Hertz-dev, the first open-source base model for conversational audio — So far a lot of the voice AI models have been speech → text → speech. So your voice is converted to text for the model to understand, the model replies with text and then TTS (text to speech) is used for the response. This model instead is speech → speech. If you listen to some examples it generally starts responding with non-coherent gibberish, but it does sound quite realistic!
📝 Article — Blog Writing for Developers (2023) — I need to give this a read it looks like it might be helpful for my own writing!
🛠️ Tool — Self-hosted web browser bookmarks syncing — At the moment I am just using Firefox to sync all my bookmarks between my phone, laptop, and desktop. It is good there is a self-hosted option too though.
📝 Article — Hacking 700M Electronic Arts accounts — It is always interesting how these attacks are done. In nearly all cases I have seen it is a case of developers handling authentication but forgetting about authorisation.
🛠️ Tool — 98.css: A design system for building faithful recreations of old UIs — I spent most of my childhood using either Windows 3.1, Windows 95 or Windows 98. Some of my early programming experience was with Visual Basic 6, so a lot of my programs looked like this. Even when I moved to XP I often default back to the old styling for performance. This is quite fun and nostalgic. Makes me want to code something using it just for the fun of it!
📝 Article — Tracker Beeper (2022) — As I mentioned in this post, Google tracking is everywhere especially given the use of Google Analytics on most websites. I think if your browser beeped every time it sent data to Google people would be more privacy conscious.
📝 Article — Useful built-in macOS command-line utilities — I really need to remember these! uuidgen is a good one. The amount of times I find my self searching “new guid” when I need one is embarrassing!
📝 Article — Monorepo‚ Our Experience — At work we use vertical sliced monorepos. Our system is made up of separate domains that shouldn't really mix, so this makes sense. Being able to do work on the frontend and backend API within the same PR is a godsend though!
📝 Article — What have you done, America? — I try to avoid politics in this newsletter. Mostly because it doesn't interest me all that much. Still given the recent election I am glad I don't live in the US. I think Will Wheaton sums things up pretty well.
💬 Quote of the Week #
We now trust our senses, memories, and gut feelings less and put all our faith in Google. These bad habits rot the discipline necessary for contemplation or study and rob us of the grit necessary to follow through with long-term goals.
From Feck Perfuction by James Victore.
Top comments (0)