DEV Community: Roxanne Lee

ElasticSearch Continued #1

Roxanne Lee — Sat, 18 Mar 2023 13:14:49 +0000

Introduction

My previous blog posts have chronicled my journey with ElasticSearch, starting from my early experiences using it in the Telescope open-source project. It's amazing to think that after almost a year, I still have the privilege of working with this powerful search and analytics engine as part of my job.

This month, I am fortunate enough to attend specialized ElasticSearch courses that are fully funded by my workplace. The courses are led by industry experts, giving me a unique opportunity to gain an in-depth understanding of ElasticSearch and its many features.

Although ElasticSearch is planned to be removed from Telescope, it seems fitting to share some of the key takeaways and new knowledge that I have gained. While I'm not sure how many posts there will be, I am excited to continue exploring and expanding my understanding of this powerful tool, especially in revising my explanation of how auto-complete indexing works.

Text Analysis

Text analysis is utilized in two scenarios: indexing text and querying search with text. This can be done through analyzers.

Anatomy of an Analyzer

In general, an analyzer can be considered as a bundled configuration for character filters, tokenizer, and token filters.

Sequential flow of text analysis:
          ___________        ___________        _________  tokens 
         |           |      |           |      |         | ------>  Index 
Text --> | Character | ---> | Tokenizer | ---> |  Token  |          
         |  Filters  |      |           |      | Filters |  query       
         |___________|      |___________|      |_________| ------>  Search

Character Filters

Are used for pre-processing text before it is sent to the tokenizer.
Some built in character filters include

HTML Strip Character Filter to strip out HTML elements
Mapping Character Filter to replace based on specified strings
Pattern Replace Character Filter to replace based on regex patterns

Tokenizer

A tokenizer is the component that is in charge of separating a sequence of characters into distinct tokens (commonly words), and produce a stream of tokens as the output. It is worth noting that an analyzer must have and can only have a single tokenizer.

Besides being able to create Custom Tokenizers, commonly used built-in tokenizers include

Standard Tokenizer for grammar-based tokenization
Keyword Tokenizer which outputs the whole text as a single token
Whitespace Tokenizer breaks text by whitespace
Pattern Tokenizer splits text based on Java regex
UAX URL Email Tokenizer the standard tokenizer while also being able to recognize URLs and emails.

Token Filters

Token filters continue to process the token stream, and may remove, edit or add new tokens based on the filter.
Some filters include:

lowercase - to switch all to characters to lower case
ngram/edge-ngram - for splitting the token into ngram chunks
stemmer/snowball - stemming text to reduce words to its root form
hunspell - dictionary-based text stemming
stop removes stop words from the stream such as "a", "and", "the", etc.

What we have in Telescope

PUT /posts
{
  "settings": {
    "analysis": {
      "analyzer": {
        "autocomplete_analyzer": {
          "tokenizer": "autocomplete",
          "filter": ["lowercase", "remove_duplicates"]
        },
        "autocomplete_search_analyzer": {
          "tokenizer": "lowercase"
        }
      },
     "tokenizer": {
        "autocomplete": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 20,
          "token_chars": ["letter", "digit"]
        }
      }
    }
  },
  "mappings": {
    ...
  }
}

Breaking down the Posts index in Telescope, we have

Two custom analyzers
One named autocomplete_search_analyzer that doesn't have any character filters or extra token filters, as it uses the built-in lowercase tokenizer
One named autocomplete_analyzer, which uses our custom tokenizer, autocomplete, which includes a customized setting for the edge_ngram type. Our analyzer also includes built-in lowercase and remove duplicates token filter

To be continued

For now, this is all I have time for. In later blogs, I'll continue to try and share my findings on text analysis for auto-completion, including insights on ngrams/edge-ngrams and index mappings.

Release 3.0 - OSD700 Recap

Roxanne Lee — Sat, 23 Apr 2022 07:02:29 +0000

This blog is not really technical.

So, the release happened, and also not during a final, so I got to witness it. Its not perfect yet, but fixes will happen, and its not really what this blog is about.

Take OSD700

To get straight to the point. If you, dear reader, happened to stumble here cause you're still trying to decide if you should take this course or not. Short answer is "just take it".

Here's my metaphor of OSD700. You come in, you're thrust with a bunch of people, but no worries, you'll get to know them soon enough cause these are your co-parents for the next few months. "Telescope", or whichever project for your class, is now your new adopted child, imperfect, but beautiful in its own way. It's your job to take care of it, and your responsibility to make sure it stays alive and healthy. Obviously, as a kid, Telescope is very fragile and needs a lot of attention, and if you ever feed it anything bad, be sure to be prepared for all the tantrums that follows.

This course is like no other. All throughout the semester there is this sense of comradeship. You're all in this together, you don't have to worry about keeping your code away from one another, and its more about sharing your research and experiences with each other. This gets stuff done more efficiently, and you'll never have to feel like you're alone.

Fair warning though. A team can only carry you so far. You, yourself, will have to break out of your comfort zone and get passed that initial feeling of inconfidence and awkwardness of reaching out for help. And, even though there might always be some people who are seemingly on a whole other level, in the end, all that matters is how much you've learned and have grown.

My Change

At the beginning of the semester I was utterly clueless to what "Telescope" is, and immensely frightened. Before the grades came out I had thought I had flunked OSD600, and me coming back for more in 700 just felt like going for something way out of my league. Not to mention all the confusion and all these questions in my head. Like, what in the world was "Docker", or "Nginx", or "Redis", or "Traefik", or "Elasticsearch"? What about all these microservices and what do they do, and how do they even work with each other? And, whoa, there's also something called a "Satellite"?

But now after 14 weeks of experimenting and sorting out the threads, I've come out a whole new person. I know the insides and out of the Search service, made friends with Jest, and became intimate with Elasticsearch on a level I would never have imagined. Along the way I became familiar with the nuts and bolts of our back-end (Parser), and got comfortable in working with tools like Docker, Nginx, Redis, and Traefik. I even poked a bit in Satellite, and had a taste of working with the front-end. There was also that time when I was Sheriff... I might not have enjoyed it as much as others, but it was a different experience and it kept me on my toes. Not to mention the large amounts and variety of PRs that I had a chance to take a look at and learn from.

Undoubtedly, I was lucky to get to work with a huge group of talented people. You can read all their blogs on Telescope itself, or you can check them out selectively on the repo wiki. We also got a lot of support from alumni, and of course the Prof (humphd) himself. To this day I am still amazed how he can still keep on top of things with so much going on.

Conclusion

For me, working on Telescope is a roller coaster of emotions. Sometimes its fun, then the next second, tortuous. Finishing the code could be exhilarating, but not breaking through could be agonizing. Sometimes I'd look forward to the next issues to work on, but there are also times when I want to pretend the project doesn't exist. Undoubtedly, the scale definitely tips more towards the positive side. Plus, the amount of fluidity and uncertainty in open source projects like these is just another aspect to the uniqueness of this course.

I'll probably stick around especially for Elasticsearch related things, but I might try dip my toes in other areas without the pressure of trying to get things done for the next release.

All in all, its the best course that Seneca could ever offer, and I'm glad I stuck with it. Even you, dear reader, student or not, Telescope is always there with open arms. Consider contributing today!

Release 3.0 Pre

Roxanne Lee — Thu, 21 Apr 2022 04:36:30 +0000

The last two weeks for the last release, everyone's busy with other assignments and finals, and including me as well. When I envisioned 3.0 I had thought that we'd have something stable and it'd be more about putting in tests, smoothing out docs and maybe dealing with a few critical bug fixes. But, its seems that we're still trying to cram a lot of major stuff in this release. Changes that make it seem like I have to know Telescope again, as if I've never worked on it this semester. But, maybe more on this later.

Indexer tests

The last set of issues for me adding in autocomplete indexing is to add unit tests for our Indexer. The tests might not seem like much, but ElasticSearch-mock was at it again, playing around with me for "old times sake". As I had probably mentioned blogs ago, ES-mock is rather strict with its mocking. If it doesn't like the request that it's detected, it'll complain.

When for indexing it was trying to translate console commands to javascript, for mocking it was the other way around. Sometimes the ElasticSearch documentation aren't really that helpful either. Like the create api, for example, there's four possible requests.

Anyways, I eventually found a way to deal with the horrifying mock-not-found error. For any future playmates of ES-mock that happen to stumble upon this blog, if you ever get that error, just console log error.meta.meta.request, and you'll see the correct request path that its trying to call.

It was also neat I got to play with spying on logger for these tests. Just remember to also jest.clearAllMocks(); after each unit test, or the log spies will interfere with one another.

Search service README

A smaller documentation PR I did was to update the Search readme. Technically its partially my fault for not having kept it up to date when I made all those changes in Search.

I had initially wanted to do a bit more for this Search README, and maybe set the service up with a dev script and document in the README how it can be used. Kind of like how I had written it in a previous blog. In order to make it useful I'd have to know which docker images needed to be run for Search to fully function (with results). But, as I had mentioned above, quite a bit of new stuff has been merged into this last release, that I am not even sure anymore. So, maybe when things are more stable and clear, I can get back to it.

On a side note, a rather cool PR that I reviewed was on a CloudFormation template. I think it might be nice for future students who also work on potatoes for Telescope. Just please do remember to delete the stack when done working.

Conclusion

I have a final during the time when the release is scheduled to happen. So, all I can say now is, fingers crossed, I hope everything goes smoothly.

3.0-alpha Autocomplete front-end

Roxanne Lee — Thu, 21 Apr 2022 03:37:03 +0000

In 2.9 a few things had happened.

Authors autocomplete indexing was added in the back-end.
A new authors autocomplete query was created for Search.
Parser service booted up.

So in 3.0 alpha I had a few things I needed to deal with.

#2 and #3 I wasn't too fazed. For #3, it was basically just a copy- paste. As long as Parser was working correctly before, adding the indexing would work. For Search tests, well, at this point ES-mock and I are already quite familiar with each other, in terms of mocking the ES search query.

As for front-end. I honestly wasn't all that prepared to do it. I thought I had swore myself off of React and front-end on week 2 of this course, but I guess, since I put in the back-end, it was sort of all part of a package deal.

Front End Logic

The logic was very simple. Each time (or rather, every few seconds) the search text has changed, we get the results from our Search service API, and list the results for the user to choose from.

I'd just like to take note of a fancy trick that the OSD600 me didn't know about but had also ran into with useEffect, it looks something like this:

  useEffect(() => {
    // debounce so it searches every 0.5 seconds, instead of on every stroke
    const debounce = setTimeout(() => {
      (async () => {
        const prepareUrl = () => `${searchServiceUrl}/authors/autocomplete/?author=${text}`;
        // Do the request if there is something to search for
        const shouldFetch = () => text.length > 0;
        const data = await fetchResults(shouldFetch() ? prepareUrl() : null);
        if (data) {
          setOptions(data);
        }
      })();
    }, 500);

I mostly just want to point out the (func)(); part, which I honestly have no idea what the name of is. But back in OSD600 I had ran into a similar problem with trying to call a fetch function in useEffect. I'm just really amazed that a trick like this works.

For my version I had tried to use useSWR, cause that's what all the other components were using. But, it was acting too weird for me. For starters, it wouldn't try to get the API results on the right change. For example if I typed "Ro", it would only look for results starting with "R". There also seemed to be some weird debounce problems where deleting a search text with backspace would fire the fetch with every keystroke instead of every 0.5 seconds. So in the end, I looked for an alternative route with the old fetch and useEffect.

MUI Autocomplete Component

I chose to use this component since its supposed to be the fallback if I never got autocomplete indexing in. The original idea was we could probably feed it a static list of authors, and let it do its version of an autocomplete thing. So now it'll also serve as a fallback if one day ElasticSearch gets ripped out for some reason.

Conclusion

In all honestly, I can't decide if its front-end in general or Telescope front-end in particular that I'm not really too fond of tackling. However, besides styling, which I admit I am just horrible at, I felt my code did what it needed to do. Mostly proud that I got it working so it doesn't search on the wrong text and it also doesn't fire with every keystroke, including backspace.

2.9 - ElasticSearch Autocomplete Indexing

Roxanne Lee — Sat, 02 Apr 2022 01:54:10 +0000

Release 2.9 was a partial success on my part, cause in the end I did manage to get Author autocomplete indexing into the back-end.

Admittedly, I've put this blog off a little too long. Partly, because my other courses needed my explicit attention lest I want to fail them all, but also, because I'm rather afraid of having to explain just exactly how I got it working.

A long story short, lots of time playing around in the Kibana dev tools, lots of time pouring into the oddly structured ElasticSearch documentation, and lots of time waiting for docker images to be rebuilt and posts to be re-indexed. All in all, many many trial and errors.

But why does it work? Well, that's a whole other story. I can not guarantee complete accuracy, and can only explain it to how I understand it.

Explaining the code

To begin with, here's the code, and in case this ever gets lost, the permalink to the code with a huge blob of comment consisting of many links to various documentation.

const setupPostsIndex = async () => {
  try {
    const response = await client.indices.exists({ index });
    // If the index doesn't exist, 404 statusCode is returned
    if (response.statusCode === 404) {
      await client.indices.create({
        index,
        body: {
          settings: {
            analysis: {
              analyzer: {
                autocomplete_analyzer: {
                  tokenizer: 'autocomplete',
                  filter: ['lowercase', 'remove_duplicates'],
                },
                autocomplete_search_analyzer: {
                  tokenizer: 'lowercase',
                },
              },
              tokenizer: {
                autocomplete: {
                  type: 'edge_ngram',
                  min_gram: 1,
                  max_gram: 20,
                  token_chars: ['letter', 'digit'],
                },
              },
            },
          },
          mappings: {
            properties: {
              author: {
                type: 'text',
                fields: {
                  autocomplete: {
                    type: 'text',
                    analyzer: 'autocomplete_analyzer',
                    search_analyzer: 'autocomplete_search_analyzer',
                  },
                },
                analyzer: 'standard',
              },
            },
          },
        },
      });
    }
  } catch (error) {
    logger.error({ error }, `Error setting up ${index} index`);
  }
};

For starters, ElasticSearch will complain if we try to create an index if it already exists. To save ourselves from the error spam in case ElasticSearch data was stored and the index already exists, an extra if-statement was added.

It was decided to use the Edge N-gram tokenizer. To make use of that we'd need to set up a custom analyzer which would be set to use our custom tokenizer of the Edge N-gram type and with our chosen settings. All of this is to be done in the analysis process inside of index settings.

After that, we'd need to specifically map which properties (in this case, author) are to use the custom analyzer. For Telescope, I used a trick to designate a custom field inside the author property to use the custom analyzers instead. This way, our original search query on the index wouldn't be affected.

The decision to create an index instead of updating the already existing index is because

The index.create() function we use to index posts will create the index (if it doesn't already exist), and add a document to this index. If there weren't any previous settings and mappings, it'd only apply the standard options.
We can only update an index setting on a closed index, this would involve closing, updating, and re-opening the index.
Furthermore, the Update Settings API hasn't been implemented yet.

Now with this new indexing, while searching, instead of querying on the author property for an exact match, we can now search on the author.autocomplete field for more detailed, possible matches.

    query: {
      match: {
        'author.autocomplete': {
          query: author,
          operator: 'and',
        },
      },
    },
    highlight: {
      fields: {
        'author.autocomplete': {},
      },
    },

highlight is an ES Search option to show where the query matches are in the search results of our designated field.

Since each author could have more than one post, we had a problem with multiples of the same author showing up in the queries. Josue (alumni) introduced me to the reduce() function as a possibility to deal with it.

Now a query on the new authors/autocomplete/author=r l will look something like this.

Conclusion

This may have looked easy, but one thing I learned the hard way was that we had to specify settings and mappings in the body in JavaScript mode, while this wasn't needed in console mode. Hopefully this knowledge will save time for anyone else implementing autocomplete stuff in the future.

Amusingly, before I got a chance to break Telescope, I also encountered Telescope seemingly breaking my WSL, during the many times of re-indexing and re-building docker images.

During my local runs I like to run

docker-compose --env-file ./config/env.development up --build (whatever images I want to build)

and

docker-compose --env-file ./config/env.development down

to build and stop images.
Sometimes, my WSL trips up and returns a message such as this.

Traceback (most recent call last):
  File "docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 81, in main
  File "compose/cli/main.py", line 200, in perform_command  
  File "compose/cli/command.py", line 40, in project_from_options
  File "compose/config/environment.py", line 67, in from_env_file
  File "compose/config/environment.py", line 57, in _initialize
FileNotFoundError: [Errno 2] No such file or directory      
[22287] Failed to execute script docker-compose

When this happens, my solution is to close all the existing terminals, open up WSl, and run the commands again. Doing it this way always works, so my only conclusion is that somehow my WSL got confused and messed up in between running all those commands.

The next part is to try get this indexing into Parser, which surprisingly giving me some problems. Then, implementing autocomplete in the front-end. Finally, it'd be to add some tests so we can catch errors when we try to update to ES 8.0+.

2.9 Pre - Owning ElasticSearch

Roxanne Lee — Thu, 17 Mar 2022 01:18:29 +0000

I'm Sheriff this week. During the weekend we received an email from the Prof that we were nearing the end of our term, and release 2.9 would be the last release for new features. The meeting on Tuesday was supposed to help people answer the following questions:
1. Which part of Telescope 3.0 do I own?
2. Which exsting Issues are part of this? Which new Issues do I need to file in order to properly describe the work? When will those issues be filed?
3. Who can I depend on for support in development, debugging, testing, and reviews? I can't do this alone.
4. What are the risks that I see, which could prevent me from getting this shipped?
5. How will I mitigate these risks?

Since it didn't really go as planned. As Sheriff, it is only right that I write this blog as an example, so I can encourage my fellow classmates to write a blog to answer these questions as well.

1. Which part of Telescope 3.0 do I own?

The Search Service. Most specifically ElasticSearch. For 2.9 I want to get some Autocomplete features in.

2. Which exsting Issues are part of this? Which new Issues do I need to file in order to properly describe the work? When will those issues be filed?

The Autocomplete will require many parts, including touching code within the legacy backend or the new Parser, the Search Service, and the front-end Search Bar. Tests will also need to be included, and the following issues will need to be involved.

Backend - #1417-Add author suggestions to Elasticsearch indexer backend
Backend tests - #3224-Tests for the ElasticSearch Indexer in the backend and Parser
Search Service and tests - #3225-Updating Search Service and tests to implement Autocomplete for Authors
Front-end - #3226.
Meta issue - #1260- Add autocomplete to search for Authors, which seemingly has some front-end support for autocomplete.

3. Who can I depend on for support in development, debugging, testing, and reviews? I can't do this alone.

Besides the Prof, anyone who's worked on, written blogs about for, or have shown a keen interest in the Search service. Off the top of my head, but not limiting to Amasia, Jia, and Josue (alumni)

4. What are the risks that I see, which could prevent me from getting this shipped?

Besides a limiting time factor, the swap from legacy backend to the Parser service could greatly complicate things. It may also take a long time for ElasticSearch indexing to work as planned.

5. How will I mitigate these risks?

Keeping a close eye on the Parser service. Splitting things off in smaller issues (already done). As well as doing lots of tests and asking for reviews for the autocomplete implementation. As mentioned in my last blog, making use of the dev tools in Kibana could greatly help with testing indexing and search requests.

2.8 Update - ElasticSearch Research

Roxanne Lee — Thu, 17 Mar 2022 00:49:00 +0000

This blog is part of blog debt, which should have been posted on Friday.

Release 2.8.0

We had our 2.8 release in week 8, and it was quite a big one, including the merge of Satellite into Telescope. Furthermore, in between 2.8.0 and 2.8.1, Telescope also had its own Docker Registry set up, and we have even started using the feed-discovery image from there.

For me, I didn't get much new code into 2.8. Instead, I had to do a lot of reading and experimenting on ElasticSearch indexing and autocomplete. Starting on Friday, I also managed to find some really weird bugs following the Satellite merge.

PNPM, Jest and Satellite.

Let's start with the bugs first.

The background was that on Friday, ElasticSearch-mock had a 2.0.0 release that would allow its new version to work with ElasticSearch v8.0+. Based on my blog from last week, this was the main thing left that was stopping us from updating to ElasticSearch v8.0+. So I got excited and had plans for ES to be updated in Telescope by the end of the day. Motto of this story, software development is never what you expect.

Much too many Satellite tests

I began with updating ES and ES-mock in the Satellite inside of Telescope. However, I quickly ran into these errors in my tests:

TypeError: Class extends value undefined is not a constructor or null

       9 |   const client = new Client({
      10 |     node: 'http://localhost:9200',
    > 11 |     Connection: mock.getConnection(),
         |                      ^
      12 |   });
      13 |   // Provide a fake health check
      14 |   client.cluster.health = () => Promise.resolve();

      at buildConnectionClass (node_modules/.pnpm/@elastic+elasticsearch-mock@2.0.0/node_modules/@elastic/elasticsearch-mock/index.js:121:32)
      at Mocker.getConnection (node_modules/.pnpm/@elastic+elasticsearch-mock@2.0.0/node_modules/@elastic/elasticsearch-mock/index.js:116:12)
      at new MockClient (src/satellite/src/elastic.js:11:22)
      at Elastic (src/satellite/src/elastic.js:32:14)
      at Object.<anonymous> (src/satellite/test.js:999:14)

By this point I'm already too familiar with this error. This meant that the ES client and the ES-mock weren't compatible. One of them was using the older version, while the other one was using the new one. But for me, this didn't make sense since I had updated both in Satellite. So I went to my fork of Satellite, with it being its own repo, and did the same changes.

All the tests passed.

While this stumped me even further, I also noticed some really odd numbers occurring when I ran tests on both sides. In Telescope, the Satellite test suite was running more times than it should be.

I filed the issue, and Prof set up a PR to fix it. Only, it wasn't as quick a fix as we had thought. It lead into an afternoon of code sleuthing for me, along with the Prof (humphd) and cindyledev, one of our alumni. This all happened during a pretty hectic afternoon, as in this time, Telescope 2.8.0 was being released, and the Docker Registry was being set up.

Tweaking Jest Config

Once that PR was merged, I still hadn't solved my mystery of failing Satellite tests. After quite a long while of pouring into the Jest Config Documentation, I believed I had found the culprit, moduleDirectories, and added this line in the Satellite jest config.

moduleDirectories: ['<rootDir>/src/satellite/node_modules', 'node_modules'],

From what I gather, it meant jest will look at the Satellite node_modules to find the dependencies it needs first, and if it doesn't find them, it'll look into the root node_modules.

PNPM and breaking the Search Service

I was ecstatic to get the Satellite tests working, only to find in horror that my Search tests were now failing.

I would encourage you, the reader, to read the comments in the issue and the PR. I hope it is detailed enough to explain the issue involved. Josue (alumni) and I tried to tackle this way into the late night, but to no avail.

A "simple" summary of this issue is that something is up with PNPM and its making it that our Services are using the local Satellite instead of the Satellite package they're supposed to. It broke Search because now, Search wants to use the updated ES inside of the local Satellite.

Learning to run the Search Service locally

In between this frustration and headache, I learned how to test the Search service locally. Not with building a docker image, but really locally.

Inside of the Search service, download env-cmd as a dev dependency.
Still in the Search service, create a script

"dev": "env-cmd -f env.local nodemon src/server.js",

In the Search service, create a env.local with the following:

ELASTIC_URL=http://localhost
ELASTIC_PORT=9200

Run pnpm install to get everything updated
In root (WSL for windows), delete redis cache and build the other docker images

rm -rf redis-data
docker-compose --env-file ./config/env.development up --build redis elasticsearch posts traefik nginx planet

Wait for all these images to be built. Then in another terminal run pnpm start
Wait for a bit for post data to be filled up again. Head Search, cd src/api/search and run pnpm dev
The Search service will be running locally on http://localhost:4445

ElasticSearch Autocomplete Research

I did a series of reading, to understand ES Autocomplete. Not just limited to these below:

Basically there are 3 main ways to implement autocomplete with ElasticSearch

Prefix queries are ruled out cause they run during search query time and are slow. Completion suggestors don't seem to work with multiple queries (in our case, advanced search), so that is ruled out as well.

Kibana

I soon realized that it was kind of painful to change the ElasticSearch indexing in the backend, and testing out those queries with the Search service. Perhaps I wasn't doing things right, but each time I did any changes in the Backend or Search, I'd have to rebuild the Docker Images over and over again. My potato of a laptop did not like it. Sometimes the errors returned aren't even helpful at all, and I get errors with only the message ResponseError: ElasticSearch Error but no context.

After a long while I finally caved and created an account with Kibana, which is advertised as a free interface to navigate the Elastic Stack cloud.

For me, I was looking to use the console in "dev tools".

Now, I can more easily send requests and test the correct indexing and search queries. After that, hopefully, all it will take is to translate these console commands into JavaScript.

Conclusion

In all honesty, it was a frustrating weekend for me. There was a very coincidental timing of all sorts, that ES-mock had to be updated on a Friday, and just after Satellite was merged into Telescope. I think my morale had been hit a bit, cause I still believe that if it hadn't been for the merge, I would've had ElasticSearch updated to 8.1.0 in all of Telescope by now. The merge had exposed a lot of bugs that I currently don't fully understand and don't really know how to tackle. All in all, a part of the fun and frustrations of software development.

Week 7 - 2.8.0 Pre

Roxanne Lee — Sat, 26 Feb 2022 03:31:15 +0000

Week 7 for me is a mid-term week as well as a pre-release week, so this blog probably won't be as long.

Dealing with ES Errors for `createError`

My PR last week on Satellite, for refactoring our exported createError to deal with ElasticSearch errors, was merged this week. Of course, it was only after a few reviews and fix ups. I'm continuing to relearn small things like remembering to name files in lower case, because not all filesystems are case sensitive.

Update to ElasticSearch 8.0?

This week, I tried to see what would've happened if we updated to ES 8.0. I hadn't initially planned to tackle this so soon, mostly cause I had mid-terms and originally, the possible Docker-related problems made this issue less appealing. But I was lucky that Josue, one of our alumni, poked me on Slack about it. Through his help, I was able to get Telescope running in dev mode with ES 8.0. Unfortunately, even though ES 8.0 seemed like it worked fine, ElasticSearch-Mock doesn't support ES 8.0 yet. If we updated now, we'd have to rip out the tests that took us weeks to get in, and we'd be moving forward without tests for a while.

Getting Satellite Updates Semi-automatically

I've already mentioned this issue last week. This week, I managed to get a hold of Duke, our resident Renovate bot guru (just, one of the many titles that he has). He directed me to a possible solution, by changing to rangeStrategy="bump" for Satellite in the bot's configs. The PR was a quick fix, and has been merged. Hopefully, next time Satellite has an update, Renovate will be able to detect it, and we can just get the bot to create an update PR, instead of running the code ourselves.

Parser Tests

Tue had gotten his Parser service PR merged, and is now writing tests on it. I figured I'd try help out since ElasticSearch is used in this service. I've created a PR to his PR, with replacing some (actually only one) of the tests with using ES-mock. But, the Parser service is huge, with many tests involved. So, currently this is an ongoing project.

The agenda for next week

These are the open issues currently assigned to me. It might look like a lot, but I'm sure its not as daunting as it seems. Some of the issues I've already mentioned above; Some, I won't be working alone, and would have help while learning to tackle it; Some, I'm not sure if I can proceed just quite yet.

Overall, next week would be interesting, but, my priority would be getting the autocomplete in. It is an issue that has surprisingly been open for two years, so hopefully, it can get closed soon.

Conclusion

It was not exactly an OSD focused week for me, but I'm glad I got some coding in. Hopefully next week will be very fruitful.

On a final note, my HacktoberFest t-shirt (and many stickers) has finally arrived, yay! I wasn't blogging on dev.to back then, and I realized I missed the date to claim the badge on dev.to. Oh well, I'm happy the shirt arrived at least.

Release 2.7.0

Roxanne Lee — Sat, 19 Feb 2022 08:04:06 +0000

I never planned for these late night posts, yet, it seems to becoming a trend. It's always on Friday nights when something pops up that I just want to get over with before I start enjoying the weekend and pretend Telescope doesn't exist. But really though, we all know students don't have real weekends.

So, to ensure this doesn't turn from a late night post to an extremely early morning blog, I better get started.

Release 2.7

This week was release week and we're now on 2.7.0! This release was really busy with over 80 issues and PRs that were addressed. Prof has a blog post on this with more detail.

At the time of this post, I think there's a scheduled 2.7.1 release some time during the weekend. To sum it up, something broke, its now fixed, just waiting for a mini release.

I managed to sneak into the later parts of the initial release meeting to see the Prof, our Sheriffs (Tue and Francesco), and some other classmates knocking their heads to get things working again. The causes and fixes are out of my league, but maybe they will be revealed in the blog posts on Telescope.

`pnpm` Problems

All throughout the meeting, I had my fingers crossed that I didn't cause any of the breaks. Which, shouldn't even be a concern cause I didn't do all that much this release. I had my advanced search tests merged, so yay! But, they were stalled mostly because of issues with getting the right Satellite version.

Somehow running pnpm install wouldn't update Satellite to its latest version. I had thought the solution was to change the version to "latest" in package.json, and even closed the issue with a PR, thinking that was the right fix. But nope, it was just an illusion. Before I even got to researching how to automatically get Satellite updates, the problem showed up again.

It seems that the pnpm-lock.yaml won't update if nothing in the package.json was changed. Currently, the fix is to manually update with pnpm update -r --latest @senecacdot/satellite.

Coming up next

Here are some of the issues that I'm currently assigned to.

Documentation

One for Auth docs, because I joined the meeting on Supabase to get an idea of what it's about. Originally I had figured I needed to know Supabase since it might eventually take over ElasticSearch in Telescope. But it actually seems to be really neat and I'm still getting to know it and exploring the possibility of using it in my personal project.

Another one for translating the About page. During our triage meeting we were exploring the possibility of translations. I think if eventually set up correctly, it might create some easy good-first-issues to attract future contributors.

`pnpm` and maybe CI

As mentioned above, currently, to get the latest Satellite version, we do things manually. The issue is to explore the possibility of doing it automatically. I am not too sure if this is possible, since any pnpm-lock.yaml change requires merging a commit. So, maybe we can do it semi-automatically, kind of how renovate does it for the other dependencies. I'll have to explore more on this and also discuss if this is still needed.

Docker

I kept myself on the team for creating our own Docker registry. I am still catching up to the cloud computing classes, but since I made all that effort to "sneak" in, I might as well find a place to get handsy with it.

More ElasticSearch

This comes in three parts, fixing a bug, refactoring and updating to ES 8.0, getting in some Autocomplete.

Part 1: Dealing with ElasticSearch errors

This late night post is a result of working on the Satellite related issue. Though, I want to emphasize how nice it is to have tests so I sort of know what to look for.

It started with Prof running into a crash related with our Search code. Here's where the crash happened when we try to createError (from http-errors)

// query.js
router.get('/', validateQuery, async (req, res, next) => {
  try {
    const { text, filter, page, perPage } = req.query;
    res.send(await search(text, filter, page, perPage));
  } catch (error) {
    console.log({ error }, 'query error');
    next(createError(503, error));   // <-- crash is here
  }
});

It could've been our problem, ElasticSearch's, or http-errors'. I'll give the conclusion here, so you could skip the next parts if it is too lengthy. "ElasticSearch" has their own custom errors that couldn't be recognized by "http-errors", which we tried to pass in as a regular Error in "our" code. I'd say, it's probably mostly our fault for not having tests to catch it sooner.

I figured it out thanks to my own tests that I landed.

I did what Prof had suggested, and added a test in Satellite with Error objects. It passed, so I was a bit more confident that it was our fault, the way I assumed and written last week.

The fact that Search seemingly wasn't handling errors correctly was still bothering me. Therefore, I tried to refactor our Search code to handle errors, and also wrote tests to check the results. But, the error was still occurring. Eventually, it gave me the idea to add tests with ElasticSearch errors, and the tests results confirmed it was indeed related to ElasticSearch's fancy errors.

test('should fail when directly creating Error from ElasticSearch error object', () => {
    // { errors } is imported from ElasticSearch
    const elasticError = new errors.ResponseError({
      body: { error: 'testing ElasticSearch Error' },
      statusCode: 404,
    });

    try {
      createError(503, elasticError);
    } catch (err) {
      expect(err instanceof TypeError).toBe(true);
      expect(err.message).toBe(
        'Cannot set property statusCode of [object Object] which has only a getter'
      );
    }
  });

The cause might be from ElasticSearch's change with errors in 7.16. It was also probably not noticed much because we don't get that many errors from ElasticSearch?

Anyways. As a follow up, Prof asked to explore the possibility of handling this internally within Satellite, since we export createError from there.

I was initially worried about createError being immutable from being imported, and that we might have to change the function name. Luckily, Prof came to the rescue.

I am so glad I am still at school so I can still ask questions like this that make me seem dumb. But I'm really glad I know this trick now.

Following Prof's tip, I have made a PR to take care of the ElasticSearch issue internally. It might have been a bad time now that I think of it, since its the weekend and Renovate is going to try to make many dependency update PRs. Nonetheless, I hope for my PR, after reviews and tweaks, will be able to fix the issue.

Part 2: Updating to ES 8.0

Rumor has it that every time we update ElasticSearch, Telescope breaks. I've been reading the release notes, and I've already came up with a list of features that would possibly break our code.

Some other things to note in the API changes (will have to scroll down all the way to Search for the whole list).

Fix range query on date fields for number inputs #63692 (issue: #63680)
Remove type query #47207 (issue: #41059)

Part 3: Autocomplete

I've been trying to read more on ElasticSearch, and I realized that our Search only uses a teeny tip of what ES has to offer. I also found, to my surprise, that even GitHub uses ElasticSearch.

Up till now, I've mostly been debugging, so I'm hoping I can actually get some more ES features in before Supabase takes over. Autocomplete could be a nice place to start out.

I think we might need the Parser service in first so I can get the indexes in. I still need to get reviews in for that. Also, before I can tackle getting in new features, we have to get parts 1 and 2 completed. Furthermore, if I had understood correctly, Amasia, who's been working on the search bar redesign, has also changed some of the Search code, and might have a PR up soon as well.

Conclusion

At this point of writing, when it might actually be categorized as an early morning post, my head is just thinking, "Oh gosh, there's so much to do", and yelling at me to sleep. But I know really, that I'm enjoying the work that I've done, and might still get to do for this course.

On a side note. Just last night I had thought maybe my Hacktoberfest T-shirt is forever lost in the abyss. Just checking now though, it finally has updates. So, maybe there's still hope.

2.7.0 Pre

Roxanne Lee — Sat, 12 Feb 2022 06:10:24 +0000

Week 5, not a release week, but I was still quite busy. I must've cursed myself when I was worried about things picking up for other courses last week. And they sure did! I didn't get to spend as much time as I'd hope for OSD700.

However, on a note that's slightly related to OSD. In one of my other classes they want us to work on our project in GitHub, with issues, PRs, and milestones all established and set up. Thing is, even though everything seems second nature to me now, no one else in my group knew how to do those things. I basically had to do mini OSD600 lessons throughout the week, while tossing Prof's videos at my teammates and hoping that they will watch them. Conclusion is, take OSD if you can. It is so worth it.

Moving on to actual OSD700 related things.

Search Tests!

There's finally been progress on tests for advanced search! I have a PR up and waiting for tweaks and reviews. I think I'm just happy that ElasticSearch-Mock is cooperating with me.

I am not totally sure why the mocks weren't clearing for me last week. I changed up how I did the tests a bit, but I think it was related to me trying to play around with both specific paths and the * for paths. It seems that probably the * path is just too generic and that maybe it caused the mocked client to not be able to find the correct mock. So not really the fault ofmock.clearAll(), but just me not knowing how to use it well enough.

Other Issues

I'm looking at the other issues that are assigned to me. First thing is that I wasn't aware I was assigned to doing React Native Documentation. Probably cause I had mentioned trying it out in the first week, then quickly scratched it off my priorities in the second week. But, its just documentation, might be fun to take a look at.

After finishing up the tests for advanced search today, I have a feeling the issue on the Satellite test is on us. Last week I had mentioned I had encountered the 404 Mock Not Found error many times when I was trying to do mocks. However I was only able to do detect those errors after I modified the search.js locally with error tracking for the client.search() function. My code was modified from this example below from the documents:

// callback API
client.search({
  index: 'my-index',
  from: 20,
  size: 10,
  body: { foo: 'bar' }
}, {
  ignore: [404],
  maxRetries: 3
}, (err, result) => {
  if (err) console.log(err)
})

Even though I managed to log and find the error this way, tests for search was still basically stuck because it didn't seem like it knew what to do with the errors. Probably, we just need a way to handle errors from the client.search() function.

However, I'll still try to add the test in for Satellite. Just to be safe. The more tests the merrier?

Docker Issues

Compared to a lot of people in this class, I am very slow at getting to learning Docker. However, this week I had managed to sneak (ask to get added) into Prof's other lectures on Docker, and just the first class had cleared up some of my personal confusions.

As for these two Docker related issues "Improve size of web Dockerfile", Make Telescope project usable by a Docker Registry. I admit I am still mostly dumbfounded by it all even though there has been a lot of discussion. Hopefully by catching up with the lectures and doing a bit more research, I'll know more how to approach them.

Conclusion

I am glad I stuck around in OSD. My teammates from the group I mentioned earlier had been asking me where I knew all this stuff with Git and GitHub, and I've told them that it was all OSD. Then we'd laugh and try not to think about what would've happened if none of us knew how to use Git and GitHub.

I'm also quite excited in the possibility of getting to move on from ElasticSearch, and starting to learn something new (Docker). While I don't need to worry about labs and assignments for the class that I had snuck into, I'm going to count it as part of the time I'm dedicating to researching for OSD700 and I'm definitely looking forward to future lectures, and catching up on the recordings.

Release 2.6.0

Roxanne Lee — Sat, 05 Feb 2022 04:53:29 +0000

In this week for OSD700, we had our 2.6.0 release for Telescope. As what seems to becoming a pattern for all release weeks, many PRs are getting merged, and things are moving quickly. There was a lot of new things popping up, like Turborepo and Supabase, and I wish I had some time to get to know them better.

For me, my focus this week was to get to creating tests for Search, specifically advanced search. Unfortunately, it didn't go as smoothly as I had hoped for. But, there was progress, and ElasticSearch-mock and I seem to be gradually getting along.

ElasticSearch Mock

If there was a type of friend that ElasticSearch Mock (ES Mock) would be, it'd probably be one of those that you seem to know them, but actually don't. They're picky and stubborn, and like to give you surprises and tease you to no end. This about sums up the relationship I've had with ES Mock this week. Every time I thought I might've have had a breakthrough and could move on, it seems to drag me back while yelling, "No! Play with me more!", and then comes up with more hurdles that I'll need to jump through.

Getting ES Mock exported from Satellite

"But, Roxanne" you say, "The article says clearly, Smooth mocking."

Under normal circumstances, it might be smooth. ES mock is a library provided by Elastic.co, who are the ones that created the regular ElasticSearch. How I understand that they have it set up is that during mocking, a mocked API would establish a connection that an ES client could connect to instead of using the regular connection.

Except Telescope is a little special. Thanks to my PR last week, Telescope doesn't use ElasticSearch directly anymore, and instead uses a client packaged inside of Satellite. So, in order to get this special client mocked in our tests, Satellite will also need give us access to its mock.

There were a lot of experiments, a couple versions, and many tweaks. In between I realized just how picky and strict ES mock could get with the APIs. Here is an example from the ES Mock npm registry README

mock.add({
  method: 'GET',
  path: '/_cat/health'
  querystring: { pretty: 'true' }
}, () => {
  return { status: 'ok' }
})

client.cat.health(console.log) // => 404 error
client.cat.health({ pretty: true }, console.log) // => { status: 'ok' }

Basically, the more field that are specified, the stricter the mock gets. There are some fields that if they exist in the API, you had to specify in the query, and vice versa, otherwise, ES Mock would just think the right mock doesn't exist, and return a 404 Mock not Found error.

At last, a final version was merged into Satellite. Prof helped a lot with this final version, and it was fascinating to see him code. There was a few coding tricks that I wasn't aware of. Especially the line

client.mock = mock

Now, we could not only get a client instance, we can also get a mock object { mock } as well. I'm just fascinated cause I think the original way I had it in my head wouldn't be as condensed.

ES Mock Strikes Again

Finally, with the changes in Satellite released, and updated in the Search service, it seemed that maybe, just maybe it would be smooth sailing onward, and I could finally get to really mocking Search. But nope, this time around, the mocks weren't clearing for me and was messing with my tests. This got me a little frustrated, cause it was clearing for the tests written for Satellite.

At this point, last night of writing this blog, I decided I needed a break from playing with ES Mock. It was getting to a point where, even taking a small nap, I'm dreaming of coding and just getting the Mock not Found error. A break might be working cause as I am writing this blog, I'm forming some theories that I'd like to test out to make it right. Hopefully I'll get to write about this next week.

Conclusion

It doesn't seem like I got much accomplished this week, but I'd say I tried. I had spent a lot of time on OSD this week, and I wish I had more, or that I was smarter so my time could be more efficient. Like I mentioned, I would really like to have the time to explore more of what's going on in Telescope, and branch off and maybe do reviews in other areas.

It's lucky that most of the meetings during off hours are recorded (a little sad about the lost Supabase meeting), and that there's lots of information in the weekly blogs to read about (Supabase meeting blog please? *Wink *Wink).

I'm also lucky that, since its the start of the semester, my other courses are rather light. I shudder to think what would happen later on when things pick up. Regardless, OSD doesn't seem like coursework. Things are much more dynamic, and I'm pretty sure the more time that I put into it, the more I'd gain.

2.6.0 Pre

Roxanne Lee — Sat, 29 Jan 2022 06:49:17 +0000

This blog is unfortunately going to come out a bit later than midnight. Unforeseen merge conflicts happened that didn't go as smoothly as I had hoped for (more on that later).

So much has happened in week 3 for OSD700, and I had quite enjoyed it all.

Sheriff-lead meetings

Our two sheriffs ran both in-class meetings this week, and they were both really awesome. They worked well together, pushed out quite a few releases, and showed an extensive knowledge of the goings-on in Telescope and Satellite. I won't be able to do it to the level that they did, so I admire them a lot for that.

Extracting Elasticsearch from Satellite for Search

I had mentioned this issue last week that I had for release 2.6.0. Instead of just for Search it was targeted for all the microservices that used Elasticsearch.

At this time last week my brain was still thinking that this issue meant using Elasticsearch on Telescope without Satellite. Which had me really confused, because it "seemed" already implemented in Telescope. Luckily, I had a quick crash course with the Prof on how to mock Elasticsearch, and had this confusion diffused.

That didn't stop me from making things seem more difficult than they actually were. A lot of wasted time was spent thinking I needed to take Elasticsearch completely out of Telescope and diving into too much research on areas that wasn't even related to what the issue intended.

Elasticsearch is also one of the dependencies for the Parser service, but I couldn't find where it was being used since the service is still a work in progress. I reached out to Tue to get more insight on it. He also helped me out with removing dependencies in the services. Apparently, taking out the dependency in the relevant package.json then running pnpm install is the trick.

In the end I decided to just make a PR on just the Search service, and work from there. I think Tue must've worked out how they'd implement Elasticsearch for Parser with his mentors, because the issue was closed when the PR merged.

Getting ready for tests on Search

The issue on tests for search I had mentioned last week has been bumped to 2.6.0 as well. The tests are blocking implementations for advanced search, so we have to try to get things started at least.

I'm working with Jia-Hua on this issue, and we came to an agreement that I'd make sure using Elasticsearch from Satellite wouldn't break Search, while he creates a PR for the initial setup for tests. This PR quite amusingly led to a couple more PRs.

Fixing setup files for Posts tests

In the setup for Search tests, we used Posts tests as a template. Prof spotted that the config files for tests correctly. According to the docs this meant that MOCK_REDIS in the Posts test setup file shouldn't have been set to the value we wanted. I'm pretty sure there had to be better ways to debug, but console.log had always been a trusty friend, and I had used it to find out we've only been tricked to thinking the tests were set up correctly. I sent up a PR for setting it up correctly for Posts.

Removing cross-env dev dependency

While working on the PR for Posts I noticed some scripts for tests in the main package.json had defined MOCK_REDIS=1 in their lines. I removed them, and this led to another PR with the intention of removing cross-env, which is needed to define these config values.

This current PR is still open and is the main reason why I'm late in submitting this blog post.

Removing the dependencies also involves touching the pnpm-lock.yaml file. In our current master branch there are many long lines like
which prettier doesn't like and auto-commits to change to below:
. Trying to rebase got me a merge conflict, and sorting it out meant 8,000+ lines were changed. I had thought I had messed up until I figured out it was prettier all along.

This PR still needs tweaks, and maybe I did mess up on setting up prettier, but I'm sure Prof will tell me what to do to fix it.

Moving forwards with testing

I didn't have enough time to play around with mocking Elasticsearch just yet. Jia-hua has been kind enough to share some reading material. Doing this would be my main priority for 2.6.0.

Conclusion

All in all, a very eventful week. Things are moving quickly in Telescope, I'd like to try keep up with them.

If I'd have re-done things this week, I would've saved time in just getting the PRs out and stop trying to overthink things. It is incredible how fast Prof responds and how knowledgeable some of the students in the class are. So, I'm sure if my PR needs tweaking they'll let me know in the reviews.

DEV Community: Roxanne Lee

ElasticSearch Continued #1

Introduction

Text Analysis

Anatomy of an Analyzer

Character Filters

Tokenizer

Token Filters

What we have in Telescope

To be continued

Release 3.0 - OSD700 Recap

Take OSD700

My Change

Conclusion

Release 3.0 Pre

Indexer tests

Search service README

Conclusion

3.0-alpha Autocomplete front-end

Front End Logic

MUI Autocomplete Component

Conclusion

2.9 - ElasticSearch Autocomplete Indexing

Explaining the code

Conclusion

2.9 Pre - Owning ElasticSearch

1. Which part of Telescope 3.0 do I own?

2. Which exsting Issues are part of this? Which new Issues do I need to file in order to properly describe the work? When will those issues be filed?

3. Who can I depend on for support in development, debugging, testing, and reviews? I can't do this alone.

4. What are the risks that I see, which could prevent me from getting this shipped?

5. How will I mitigate these risks?

2.8 Update - ElasticSearch Research

Release 2.8.0

PNPM, Jest and Satellite.

Much too many Satellite tests

Tweaking Jest Config

PNPM and breaking the Search Service

Learning to run the Search Service locally

ElasticSearch Autocomplete Research

Kibana

Conclusion

Week 7 - 2.8.0 Pre

Dealing with ES Errors for createError

Update to ElasticSearch 8.0?

Getting Satellite Updates Semi-automatically

Parser Tests

The agenda for next week

Conclusion

Release 2.7.0

Release 2.7

pnpm Problems

Coming up next

Documentation

pnpm and maybe CI

Docker

More ElasticSearch

Part 1: Dealing with ElasticSearch errors

Part 2: Updating to ES 8.0

Part 3: Autocomplete

Conclusion

2.7.0 Pre

Search Tests!

Other Issues

Docker Issues

Conclusion

Release 2.6.0

ElasticSearch Mock

Getting ES Mock exported from Satellite

ES Mock Strikes Again

Conclusion

2.6.0 Pre

Sheriff-lead meetings

Extracting Elasticsearch from Satellite for Search

Getting ready for tests on Search

Fixing setup files for Posts tests

Removing cross-env dev dependency

Moving forwards with testing

Conclusion

Dealing with ES Errors for `createError`

`pnpm` Problems

`pnpm` and maybe CI