Out of curiosity - what's the doc store? Or is the search being done in-memory by the backend? I used to work at a non-Google search company, and we had a lot of success using ElasticSearch as our doc store. I've even been able to write reasonable searches for the Vietnamese language with it. Handling human language is something it does really well, and something that is quite a pain to cobble together by hand. Then again, not sure that level of complexity is necessarily needed for just searching blog posts :)
Good question. We actually use Algolia, which is a hosted search. It specializes in low latency search which you can use for stuff like autocomplete, and I'm really drawn to that because I figured it would take years before we'd have a great search index, but we could quickly get up and running with a fast search index. That gives people the chance to make a few quick searches if we don't give them the right answers the first time around. We use Algolia's global distribution so it's fast everywhere on earth. We wanted to be globally performant with the whole site from the get-go and the path to getting there is a bit more complicated with some of the other routes we could have chosen.
The last 10% of the work is going to be the hardest, so we went with a user-friendly solution that would get us most of the way there. I've used a few of the other search indexes and I'm pretty happy with the direction we chose for this project so far.
Totally off topic, but I <3 Algolia. I use their WordPress plugin on my own site, for type-to-search and autocomplete. They've been responsive and helpful, and their templating is easy to pick up.
Interesting, that is a solution I am not familiar with - I'll definitely check it out. Setting up ElasticSearch (or any other type of search) is always a lot of trial and error so if someone can take out the infrastructure tweaking part and just let you fiddle with the search part, that sounds ideal.
Yeah, it's kind of a whole different direction to go. I'd definitely recommend checking it out. You're not going to get all the flexibility, but you'll get a powerful product.
I'd love to see some kind of flowchat or diagram of dev.to's internals. Every time something like this comes up I become more and more fascinated by how every aspect of this site is fast, and how everything is built.
Good job on picking Algolia. Elasticsearch under the covers I believe. I'm recently designing the architecture for a client's website and recommended Algolia to them it's a neat product and saves you having to roll your own lucene engine. I particularly liked the automated indexing of your site if you are willing to pay a little for it.
Thanks for recommending Algolia ImTheDeveloper! If you don't mind, I'll ping you on twitter about sending you a t-shirt :)
In fact, Algolia is new search technology built from scratch, not based on Lucene or Elasticsearch. If you want to read about the design of our engine, I recommend the "Inside the Algolia Engine" series written by our CTO. Here's a link to the first part: blog.algolia.com/inside-the-algoli...
Just had a read through the blog post including some of the previous ones. Great descriptions and really bring to life all of the design decisions that had to be made. Brave usage of Zookeeper, I know it sometimes can be met with gasps due to being a bit tough to configure and manage but it appears to fit the use case well.
Good information on nginx and using redis for counters. I've used a similar architecture in the past and the key/value store is perfect. Is there any further details on the search algorithms themselves? The blogs elude to the applications being written in pure C for speed and also that there are some specialist techniques used for relevance and the like.
I've asked a few of the team who are closer to the engine about the algorithms used. I'll post back the response. Also, there will be more posts in the Inside the Engine series next year, so stay tuned! :)
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Out of curiosity - what's the doc store? Or is the search being done in-memory by the backend? I used to work at a non-Google search company, and we had a lot of success using ElasticSearch as our doc store. I've even been able to write reasonable searches for the Vietnamese language with it. Handling human language is something it does really well, and something that is quite a pain to cobble together by hand. Then again, not sure that level of complexity is necessarily needed for just searching blog posts :)
Good question. We actually use Algolia, which is a hosted search. It specializes in low latency search which you can use for stuff like autocomplete, and I'm really drawn to that because I figured it would take years before we'd have a great search index, but we could quickly get up and running with a fast search index. That gives people the chance to make a few quick searches if we don't give them the right answers the first time around. We use Algolia's global distribution so it's fast everywhere on earth. We wanted to be globally performant with the whole site from the get-go and the path to getting there is a bit more complicated with some of the other routes we could have chosen.
The last 10% of the work is going to be the hardest, so we went with a user-friendly solution that would get us most of the way there. I've used a few of the other search indexes and I'm pretty happy with the direction we chose for this project so far.
Totally off topic, but I <3 Algolia. I use their WordPress plugin on my own site, for type-to-search and autocomplete. They've been responsive and helpful, and their templating is easy to pick up.
Interesting, that is a solution I am not familiar with - I'll definitely check it out. Setting up ElasticSearch (or any other type of search) is always a lot of trial and error so if someone can take out the infrastructure tweaking part and just let you fiddle with the search part, that sounds ideal.
Yeah, it's kind of a whole different direction to go. I'd definitely recommend checking it out. You're not going to get all the flexibility, but you'll get a powerful product.
I'd love to see some kind of flowchat or diagram of dev.to's internals. Every time something like this comes up I become more and more fascinated by how every aspect of this site is fast, and how everything is built.
Thanks for the feedback. I've written about it a bit before, but there's definitely more ways to describe it.
Good job on picking Algolia. Elasticsearch under the covers I believe. I'm recently designing the architecture for a client's website and recommended Algolia to them it's a neat product and saves you having to roll your own lucene engine. I particularly liked the automated indexing of your site if you are willing to pay a little for it.
Thanks for recommending Algolia ImTheDeveloper! If you don't mind, I'll ping you on twitter about sending you a t-shirt :)
In fact, Algolia is new search technology built from scratch, not based on Lucene or Elasticsearch. If you want to read about the design of our engine, I recommend the "Inside the Algolia Engine" series written by our CTO. Here's a link to the first part: blog.algolia.com/inside-the-algoli...
Just had a read through the blog post including some of the previous ones. Great descriptions and really bring to life all of the design decisions that had to be made. Brave usage of Zookeeper, I know it sometimes can be met with gasps due to being a bit tough to configure and manage but it appears to fit the use case well.
Good information on nginx and using redis for counters. I've used a similar architecture in the past and the key/value store is perfect. Is there any further details on the search algorithms themselves? The blogs elude to the applications being written in pure C for speed and also that there are some specialist techniques used for relevance and the like.
I've asked a few of the team who are closer to the engine about the algorithms used. I'll post back the response. Also, there will be more posts in the Inside the Engine series next year, so stay tuned! :)