Planning a feature enhancement.... continued

Building off last weeks blog, there are four things we want to implement:

1. Continue to build off the previously created UI for the advanced search.
2. I want to implement the search by date, once I have this down I should understand how to further leverage the implementation of Elasticsearch to implement more.
3. Let's try to implement search by Semester, or date range.
4. I explored some queries on the Elasticsearch container in docker, and realized we can pass more than just one parameter

Knocking off #2

Last week, I was able to put up a PR for the date search. What this accomplished was that we now had a way to search posts by date.

Unfortunately, I quickly realized this is not the direction I wanted to head in for my solution, as it was not practical. Why would anyone want to search just by date? The point of an advanced search should be that you can search multiple fields and refine the search to that. So from that PR, I was able to get some feedback about the UI and enhancements to make for the users.

Although this PR should not be merged in for the solution, it provided me more insight on how the search works. I was able to understand Elasticsearch a bit better. So from here, the implementation was not completely useless. What I needed to do now was implement the date search as a part of the fields for an advanced search.

Back to our journey

The Repository now contained the post/title search, author search, and the date search. Additionally, I thought, why search by date? Let's do date range. This way, if you want just one day, we can use this, but we can also expand it to be a range.

So what we needed to do now was take all the data fields, and come up with a query for Elasticsearch. This was by far the most difficult part of the implementation. I would think of this as "diving into the unknown". I had no idea how difficult, something so flexible could be. The queries have absolutely no limits on Elasticsearch, you can try and do almost everything - which makes writing a query a little difficult when learning.

Knocking off #3 and #4 together

The original query was simple:

query: {
 query: {
      simple_query_string: {
        query: textToSearch,
        default_operator: 'and',
        fields: createFieldsFromFilter(filter),
      },
    },
}

What I ended up with was this:

query: {
 query: {
      bool: {
        must: [
          {
            match: {
              author: {
                query: authorText,
                zero_terms_query: 'all',
              },
            },
          },
          {
            match: {
              text: {
                query: postText,
                zero_terms_query: 'all',
              },
            },
          },
          {
            match: {
              title: {
                query: titleText,
                zero_terms_query: 'all',
              },
            },
          },
          {
            range: {
              published: {
                gte: dateStart,
                lte: dateEnd,
              },
            },
          },
        ],
      },
    },

So to start off with, we changed a few things. I used a bool query, which would allow us to search multiple fields for matches. What you need to do next is decide do we use should or must. Simply put must is like the AND operator, should is OR, and then there's one more must_not which is the NOR operator. This was helpful, because my brain operates better on logical terms than explanations.

Problems? Yes!!!

Now that we are using must we need to provide all the fields... So if a field was missing it would return 0 results. I thought this was not good - however, after some exploration zero_terms_query: 'all' covers scenarios where that term is not provided. This allowed us to bypass that.

Lastly, we had to do our date implementation. For this one, had already seen the documentation for the ranges, and thought it wouldn't be too bad - and it wasn't. I only had some difficulty with where to place it. I believe I just stuck it in the wrong braces and took 4 minutes to figure that out. We want it so that it MUST be in that range also. I default set the dateStart and dateEnd in the query, so if it's not provided it will search from 2010-01-01 to date.now.

Next Steps:

So now that we have all the backend implementation, we can do the front end right? I was wrong. I didn't realize that we go through both Elasticsearch AND search, two containers on docker. I need to further explore how to implement this so that we can access both search types and not break either.