DEV Community: FDiaz

Scaling Zensearch's capabilities to query the whole database

FDiaz — Fri, 08 Nov 2024 15:46:30 +0000

Previously I've been able to crawl and index web pages for my search engine without a problem, until my database grew more than what RabbitMQ's message queue was capable of holding. If a message in a message queue exceeds its default size, RabbitMQ will throw an error and panic, I could change the default size but that would not scale if my database grows, so in order for users to crawl web pages without having to worry about the message broker crashing.

Creating Segments

I've implemented a function to create segments with a maximum segment size or MSS from the same principles from TCP when creating segments, the segment contains an 8 byte header where each 4 byte of the 8 byte header is the sequence number and the total segment count, and the rest of the body is the payload of the segmented database.

// MSS is number in bytes
function createSegments(
  webpages: Array<Webpage>, // webpages queried from database
  MSS: number,
): Array<ArrayBufferLike> {
  const text_encoder = new TextEncoder();
  const encoded_text = text_encoder.encode(JSON.stringify(webpages));
  const data_length = encoded_text.byteLength;
  let currentIndex = 0;
  let segmentCount = Math.trunc(data_length / MSS) + 1; // + 1 to store the remainder
  let segments: Array<ArrayBufferLike> = [];
  let pointerPosition = MSS;

  for (let i = 0; i < segmentCount; i++) {
    let currentDataLength = Math.abs(currentIndex - data_length);

    let slicedArray = encoded_text.slice(currentIndex, pointerPosition);

    currentIndex += slicedArray.byteLength;
    // Add to offset MSS to point to the next segment in the array
    // manipulate pointerPosition to adjust to lower values using Math.min()

    // Is current data length enough to fit MSS?
    // if so add from current position + MSS
    // else get remaining of the currentDataLength
    pointerPosition += Math.min(MSS, currentDataLength);
    const payload = new Uint8Array(slicedArray.length);
    payload.set(slicedArray);
    segments.push(newSegment(i, segmentCount, Buffer.from(payload)));
  }
  return segments;
}

function newSegment(
  sequenceNum: number,
  segmentCount: number,
  payload: Buffer,
): ArrayBufferLike {
  // 4 bytes for sequenceNum 4 bytes for totalSegmentsCount
  const sequenceNumBuffer = convertIntToBuffer(sequenceNum);
  const segmentCountBuffer = convertIntToBuffer(segmentCount);
  const headerBuffer = new ArrayBuffer(8);
  const header = new Uint8Array(headerBuffer);
  header.set(Buffer.concat([sequenceNumBuffer, segmentCountBuffer]));
  return Buffer.concat([header, payload]);
}

function convertIntToBuffer(int: number): Buffer {
  const bytes = Buffer.alloc(4);
  bytes.writeIntLE(int, 0, 4);
  console.log(bytes);
  return bytes;
}

Parsing incoming segments

This method of creating small segments of a large dataset would help scale the database query even if the database grows.

Now how does the search engine parse the buffer and transform each segments into a web page array?

Reading from segment buffers

First extract the segment header, since the header contains 2 properties namely Sequence number and Total Segments,

func GetSegmentHeader(buf []byte) (*SegmentHeader, error) {
    byteReader := bytes.NewBuffer(buf)
    headerOffsets := []int{0, 4}
    newSegmentHeader := SegmentHeader{}

    for i := range headerOffsets {
        buffer := make([]byte, 4)
        _, err := byteReader.Read(buffer)
        if err != nil {
            return &SegmentHeader{}, err
        }
        value := binary.LittleEndian.Uint32(buffer)

        // this feels disgusting but i dont feel like bothering with this
        if i == 0 {
            newSegmentHeader.SequenceNum = value
            continue
        }
        newSegmentHeader.TotalSegments = value
    }
    return &newSegmentHeader, nil
}

func GetSegmentPayload(buf []byte) ([]byte, error) {
    headerOffset := 8
    byteReader := bytes.NewBuffer(buf[headerOffset:])
    return byteReader.Bytes(), nil

}

Handling retransmission and requeuing of segments

The sequence number will be used for retransmission/requeuing of the segments, so if the expected sequence number is not what was received then re-queue every segment starting from the current one.

    // for retransmission/requeuing
        if segmentHeader.SequenceNum != expectedSequenceNum {
            ch.Nack(data.DeliveryTag, true, true)
            log.Printf("Expected Sequence number %d, got %d\n",
                expectedSequenceNum, segmentHeader.SequenceNum)
            continue
        }

Appending segment payloads

The total segment will be used for breaking out of listening to the producer (database service) if the total number of segments received by the search engine is equal to the length of the total segments that is to be sent by the database service then break out and parse the aggregated segment buffer, if not the keep listening and append the segment payload buffer to a web page buffer to hold bytes from all of the incoming segments.

        segmentCounter++
        fmt.Printf("Total Segments : %d\n", segmentHeader.TotalSegments)
        fmt.Printf("current segments : %d\n", segmentCounter)
        expectedSequenceNum++
        ch.Ack(data.DeliveryTag, false)
        webpageBytes = append(webpageBytes, segmentPayload...)
        fmt.Printf("Byte Length: %d\n", len(webpageBytes))

        if segmentCounter == segmentHeader.TotalSegments {
            log.Printf("Got all segments from Database %d", segmentCounter)
            break
        }

I use vim btw

Thank you for coming to my ted talk, I will be implementing more features and fixes for zensearch.

I built my own search engine

FDiaz — Fri, 01 Nov 2024 15:02:50 +0000

The Beninging

From building react applications to building my own search engine and web crawler for indexing. I’m happy to introduce to you to Zensearch, a search engine where you as a user have more control over what you want your searches to be, you can create entries to crawl different websites and continue using the search engine functionality if you have existing indexed data in the database while it does its work, now I know this may not be the most complex or sophisticated search engine in the world like google or brave search but I built this thing to gauge how much I can do on my own and learn as much as I can while doing it, and oh boy I’ve learned a lot.

It all started when I was building my React web application, a sort of commonplace book for inserting your favorite quotes or adding notes to that specific page as if you’re trying to converse with the author or typing down what you were thinking at that moment in time on a page that corresponds to the page of your physical book, its not a bad project but I just got so bored of building Reactjs applications, not that it’s bad but it felt like I was not going anywhere with it, there’s no technical depth into what I was doing and I was not learning anything from building those ReactJs projects.

so I tried to study about computer networking, Operating systems, Computer architecture and so on then after a few months of studying and building my own application layer protocol like a websocket where I can handle multiple users and each user can join these different rooms or namespaces where they can communicate with each other and I felt ecstatic, alive even. I felt like I could do so many things as long as I understood how the computer works eg: threads, semaphores, process, memory layout, interrupt signals etc, So I thought to myself, what projects can I do to utilize some of things that I’ve learned?

oh and I'm a self-taught btw and I used The Odin Project to learn programming and web development so shout out to those guys because they taught me how to become independent to study and refused to hand hold programmers throughout the curriculum.

Challenges

I've only been able to program using Nodejs, that was my bread and butter along with typescript, so I built the web crawler using Nodejs... pretty stupid right? I mean the plan was to create a crawler that can crawl an array of source URLs from the front-end and let each crawler send these extracted data to the database, and as we all know Yabascript is single-threaded and every asynchronous task is handled by the environment where Yavascript is running eg: browser's apis, node, deno, bun and done.

so doing multi-tasking operations using Nodejs was a suicide mission and it was, from converting the webpage object to be encoded to an 8-bit buffer but then the shared array buffer can only transport 64-bit array buffer due to data alignment so I had to convert from 8-bit buffer to 64-bit by adding some offset paddings and then back from 64-bit buffer to 8-bit buffer after sending data from the crawler to the main thread and then finally parsing it to a vajascript Object... wow that was fun, there is another way for message passing but that creates a copy of the same data that is in the crawler to the main thread so I didn't want to that since it would take so much memory.

I had to handle race conditions using nodejs' atomics module and to this day I still don't understand how that module even works to be honest and annoyed me so much so I had to turn to Golang. I love this language so much, it's so easy to create threads handle race conditions, using semaphores and wait groups, I haven't had the need to use mutex yet and I'm excited to learn it so maybe in the future, along with context would be fun to learn.

Let's move on to front-end shall we? has any of you read this article from frontend masters? You might not need that framework, remember that I said that I got bored of ReactJs? well, this made me appreciate frameworks because of their reusability and their data binding mechanisms.

I don't want to get into too much details about the front-end but I used a PubSub pattern to update any UI changes when data changes and used web components along with the shadow dom to create reusable components, the shadow dom was a pain to access in javascript and style since it is isolated from entire dom tree so accessing it using CSS and DOM API won't work, so yeah those were the only challenges I had but it was fun.. it was fun when I was migrating the crawler from Nodejs to Go.

Things to consider

There are some functionalities that I have not yet been implemented because I was so eager to show off the project but that doesn't matter to me that much even if this is an ongoing project, this won't be a one and done project I will keep improving zensearch in the future so for now here are some key things that are missing:

Implementing a list of already indexed websites to be displayed to the users on the front-end.
Save the most recently crawled web page for continuation.
Create cancellation for crawling but still save the indexed pages up to that point.
Rabbitmq's message size limitation for scaling, if a database contains more than the default size that is set in rabbitmq, the message broker will throw an error and crash, so to avoid this I will try to implement a window frame algorithm used in tcp by creating a pipelining mechanism where the array of webpages will be broken into segments and sent to the search engine by N size where N is the size of the window, I still need to think about this.
Give users the ability to remove their Indexed websites.

Epilogue

I would like to write more about what I learned and some nuances of my development journey but I think this would be too long, so for now I want to show off my greatest project, and I would be happy to get some feedback from you guys if you have the time and let me know if there are any problems and improvements I could do to make Zensearch better, oh and this is all thanks to theprimeagean this guy inspired me to go deeper into things and to learn the fundamentals instead of just running npm create vite@latest my-vue-app -- --template react-ts in the terminal, which admittedly made me insecure about myself as a programmer and the things that I know but because of that insecurity I've learned new things now I'm always striving to learn more things and would be happy to learn from YOUR feedback so thank you listening to my ted talk.

Github repository for Zensearch

What I've Learned This Week with TOP(The Odin Project)

FDiaz — Sat, 03 Sep 2022 11:12:17 +0000

Learning Webpack And SOLID Principles

This is my First ever post in dev.to and Im not very particular at formatting so you have to forgive me.

So I've been studying WebDev for 5 months now using The Odin Project as my guide for a Structured program and I've taught myself how to become more independent in looking up things and this made me more curious and made me want to learn more about web development thanks to TOP, concepts are the things that i look forward to learning and backend development, I've started this journey because i wanted to do something with my life and i don't want to cause anymore trouble or become a hindrance for my mother who's always supporting me throughout this whole journey of self-studying Web Development.

this is not a success story but more of a journal to see my progression and blog to get people's opinion and perspective on the topics that I've learned and if I've interpreted it or understood them correctly, so please do tell me if I understood some of it wrong and correct me, i don't want to be ignorant and oblivious about important concepts.

I've only used dev.to recently and was amazed of how abundant the resources you could learn from it, its crazy, so anyways this is the last 2 weeks of August 2022 of what i've learned.

What I Learned This week

19th - 20th week of TOP:

19th Week

I've just finished working on the Restaurant Page Provided by TOP, I've learned a lot about webpack5, how to bundle up modules so that we can decrease the size of our project (minifying it), how to structure our project using webpack eg: the dist (distribution) folder is where all of our modules are bundled up together in one output called 'main.js'(for my case, you can name your ouput whatever you want) and images or resources are also outputted there in the dist folder this will take our script at 'src/index.js' as the 'entry point', there are still gaps in my knowledge about webpack, i don't know why the resources' local url that are being used in our modules are being converted to different names after outputting it, i might look it up in the future.

In the Asset Management section in webpack documentations where if we want to use different types of file besides javascipt we could use a 'loader' or the built-in Asset Modules, understanding it is pretty straight forward(i think), the 'test:' property looks for a specific type eg: .css, .png or .jpeg and the second property "use:" uses the 'loader' that we installed using npm for that specific type of file or 'type' that doesn't take in a loader but instead uses the built-in asset module in Webpack eg: 'type': 'asset/resources'.

Parsing different file types into Json

We can use Json files and read other types of files (toml,yaml or Json5) and convert it into Json using a custom parser of a specific webpack loader.

first we install them and then inside webpack.config we use the require function on each file eg:'require('yaml')', then we do the same thing when we want to use different file types, "use" and "type" looks for the specific type extension, then use its loader, and now we use the 'parser' object with a 'parse:' property for each file type and in that 'parse' property we type in whatever that file type is with a '.parse' at the end to use the parse method, and then just import it to wherever you want to use it.

Script Automation

Script Automation, everytime we try to run our project after every change we need to build our wepack or project and referesh the page, but this task is tedious and no one wants to do a very small task just to see if a code actually works, 'watch' and 'web-dev-server' are helpful tools for that problem, watch build our projects whenever we save it without having to type in 'npx webpack build', it watches all the files within our dependency graph for change and if one of those files are changed, the 'watch' will recompile it automatically so we dont have to build it everytime.

In the package.json scripts property object we can make a property inside of it called "watch" and the value of it "webpack --watch" now we can do "npm run watch" from the command line.

Now for Web-dev-server this one provides a web server and has the ability to use a live reloading, (because watch doesnt have live reloading), first we need to install the module '--save-dev webpack-dev-server', after that we need to choose what file we want to serve (in this case the dist folder) so inside webpack config we add a devServer object and add in a property called 'static' with a value of './dist' this tells webpack to serve the files from the dist directory on localhost:8080 webpack server serves bundled files from the directory defined in output.path, then after that we go to our package.json to add a script called "start" with a value of "webpack serve --open" now we can just type in the command 'npm start' to run the webpack dev server.

20th Week

OOP Principles:
Currently im still trying to build up my knowledge to understand this concept which is pretty simple but some articles and blogs are using too much techy and abstract words and makes it more harder for me to understand some of it, but im slowly understanding the implementation of the SOLID principle:

Single Responsibility
Tells us that every module or class or function should only have one responsibility which is very self explanatory and it should not do anything else aside from that one thing, for example: if i make a Person class that makes a person then that class should just MAKE a person, not make person run jump or walk, Another one is a Book Class which just makes Books OR a basic calculator function that takes in 2 numbers and outputs something based on what operation the user uses, this function calculator SHOULD ONLY calculate things not display it to the user, you could do that in a separate function to achieve that.

Open/Closed
This one is easy to understand too, this one is used for extensibility of a Class or module or Function, so that we dont have to tinker the innards of a fully functioning Class or Module or function because modifying a built class or module would probably break the code and would need refactoring to cater different functionalities in the future.

Liskov Substitution
This tells us about having our dervied class from a parent class, to be able to be a substitute of a parent class, this one is achieved by using inheritance
--this is all that i understood from this principle because i still haven't been able to apply this to any of my projects YET but i think i'd be able to understand this one easily and apply this to one of my projects with further research, there's probably more to this or this is maybe all there is to it.

Interface Segregation
An easy one, this basically means that our Classes or subclasses should not have any methods that they dont need and shouldn't be forced to use any of those methods, one of the fundamentals of composition over inheritance.
these extra methods should be removed or tucked away, it should only execute actions that the Class requires.

Dependency Inversion
This part is still pretty tough for me to comprehend as i said in the Liskov substitution, i haven't been able to implement this principle yet, but i would love to explain it from how i understood it.

So what this one means is that if there are 2 classes or modules that need to work together for example a Class A, the has data of a customer and Class B that wants to output that data of the customer, but in order for Class B to output that data from Class A, we need to reference the Class A in Class B which violates this principle, because what if we want to have a different data from Class D(different :D) then this would cause problems in our program, because Class B only references the Class A, and if we want to use Class D we'd have to modify our Class B in order to cater different data that we want to import and use in Class B, so in order for us to avoid this conundrum, we would have to have a mediator between Classes A and D and Class B, or an interface if you will, this interface will connect the data from Classes A,D to B without having to disrupt or modify the functionality or behavior of Class B.

Summary
I've learned a lot in 2 weeks and have a long way to go since there are still a few gaps of my understanding in Webpacks and the SOLID Principles.
I'm hoping this community would help me understand more concepts in the future and i could really learn a lot from this community.

yeet

FDiaz — Tue, 30 Aug 2022 04:02:03 +0000

yeet