DEV Community: Kunal Desai

Measuring Performance of Long Running Tasks in the Browser

Kunal Desai — Sat, 22 Apr 2023 17:16:37 +0000

I was looking at some performance metrics around file upload the other day and noticed some really large outliers. It was being reported that some operations were taking over 20 hours! After digging through some of the data, I noticed that the there were several data points at 20 hours, and they were all coming from the same user.

This was suspicious to me - as if all operations had halted on the users computer and restarted at around the same time. One possible explanation was that the computer fell asleep, so the CPU halted, and upon restart, the CPU restarted and finished the operation. We were using performance.now() to measure how long operations took, and I expected it to measure CPU time (since DateTime measures Unix time).

Turns out that is not the case! According to the docs, performance.now() measures “time elapsed since Performance.timeOrigin which is the time when navigation has started in window contexts.” There’s a specific call out in the documentation that the performance.now() specification requires that performance.now() also ticks during sleep.

💡 You might be wondering, why use performance.now() instead of Date.now() ? The difference is precision. performance.now() has floating point numbers with up to microsecond precision, whereas Date.now() measures at the one millisecond resolution.

What do I do now? My goal is to get a high signal metric on how long these operations are taking while the CPU is running at full speed. My first guess was to eliminate samples that were “way too high,” but it was difficult to figure out a threshold that didn’t eliminate numbers that were representative of the user experience. I was reading online about performance.now() and ran into a blog post by Conrad Irwin (co-founder of Superhuman) which mentions you can ignore metrics where, in between the beginning and end of the metric calculation, the tab or window becomes backgrounded (user switches tab, computer goes to sleep). Note that browsers also lower the priority of operations of a non-foreground tab, and as a result, the operations appear to run slower. The code snippet from the blog post:

let lastVisibilityChange = 0
window.addEventListener('visibilitychange', () => {
  lastVisibilityChange = performance.now()
})

// don’t log any metrics started before the last visibility change
// or if the page is hidden

if (metric.start < lastVisibilityChange || document.hidden) return

I decided for our implementation, I would continue emitting all the metrics, but tag metrics that had occurred while the CPU wasn’t running at full speed so we could see the difference and filter by that metric. This indeed fixed the issue we were seeing with outliers and I was able to get a much higher signal metric that we could depend on.

Learnings from Advent Of Code Day 1 as a Rust Newbie

Kunal Desai — Sat, 10 Dec 2022 22:44:20 +0000

I've always been curious to learn a little bit more about Rust and its properties, so I decided to take part in the Advent Of Code challenge and do it in Rust! In this post, I'm going to share my learnings about Rust as a newbie going through the first challenge in Advent Of Code. Here is the day1 challenge: https://adventofcode.com/2022/day/1. I'd encourage looking through it so that you can understand what the code below is intended to do.

First, here is the code that you can look over:

use std::collections::BinaryHeap;
use std::fs::File;
use std::path::Path;

/*
    BufRead isn't explicitly mentioned anywhere in the file, why do we need it imported?
    Additionally, if it's not included, then `lines` method throws an error
    I see that this has something to do with traits...
    Reading: https://doc.rust-lang.org/book/ch10-00-generics.html
    - Traits are similar to interfaces in other languages, but there are some differences - according to Rust Book
    - They can have default implementations
    - Traits can be parameters (any object that implement's that trait can be passed as a paramter)

    BufReader implements the BufRead trait:
    - https://doc.rust-lang.org/std/io/struct.BufReader.html#impl-BufReader%3CR%3E
    - https://doc.rust-lang.org/src/std/io/buffered/bufreader.rs.html#55-96

    Why do traits need to be imported in Rust?
    - https://stackoverflow.com/questions/25273816/why-do-i-need-to-import-a-trait-to-use-the-methods-it-defines-for-a-type
*/
use std::io::{BufRead, BufReader};

fn main() -> Result<(), std::io::Error> {
    /*
    What is the '?' operator in Rust?
    - https://stackoverflow.com/questions/42917566/what-is-this-question-mark-operator-about
    - https://www.becomebetterprogrammer.com/rust-question-mark-operator/#:~:text=operator%20in%20Rust%20is%20used,or%20Option%20in%20a%20function.
    */

    /*
    Related: Exceptions in Rust
    https://doc.rust-lang.org/book/ch09-00-error-handling.html

    Two types of errors: recoverable and unrecoverable.
    - Recoverable errors are where you just want to report the error to the user and retry the operation
    - Unrecoverable errors are things like accessing beyond the end of an arraya and you immediately want to stop the program

    For recoverable errors, there is type Result<T, E> and panic! macro for non-recoverable errors
    Result is a type that represents either success or failure: https://doc.rust-lang.org/std/result/enum.Result.html#:~:text=Result%20is%20a%20type%20that,the%20module%20documentation%20for%20details.
    T contains the success type and E contains the Error type.
    enum Result<T, E> {
        Ok(T),
        Err(E),
    }
    When handling the Result<T, E> return type, it's common to use the match keyword like so:

    match result {
        Ok(success) => success,
        Err(error) => panic!(error),
    }

    For unrecoverable errors, there is type panic!. You can either call this explicitly in the code, or by doing something bad like accessing the end of an array.
    You can have Rust show a stack trace of the panic by setting an environment variable RUST_BACKTRACE = 1
    */

    let path = Path::new("day1.txt");
    let file = File::open(&path)?;
    let reader = BufReader::new(file);

    let mut curr_sum = 0;
    let mut heap = BinaryHeap::<i32>::new();
    /*
    lines() is a function implemented by the BufRead trait. BufReader implements the BufRead trait.
    */
    for line_result in reader.lines() {
        /*
        Calling a function on an object changes the object's ownership
        Assignment leads to ownership and re-assignment also moves ownership, read here to understand why: https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html#ways-variables-and-data-interact-move
        However, if a value implements the Copy trait, then the value is copied

        https://depth-first.com/articles/2020/01/27/rust-ownership-by-example/
        String ownership examples: https://doc.rust-lang.org/book/ch04-01-what-is-ownership.html#the-string-type
        */

        /*
        References:
        If you don't want to copy, but you also don't want to change ownership, you can borrow with the & character
        If a function takes in a reference, the function doesn't own the value so it won't drop it upon completion of the function's execution
        References are immutable by default, but adding the `mut` keyword makes it so that the reference is modifiable.
        There is one big caveat which is that if you have a mutable reference to a value, you can't have any other references to that value.


        https://doc.rust-lang.org/book/ch04-02-references-and-borrowing.html
        */

        let line = line_result?;

        /*
        Why can we call this function but still access `line` in the else statement?
        Because .len takes in a string reference, so line is borrowed, not moved.
        Ownership does not change as a result
        */
        let is_new_line = line.len() == 0;
        if is_new_line {
            /*
                What is the ! operator on println and why don't I need to import println via use?
            */
            heap.push(curr_sum);
            curr_sum = 0;
        } else {
            /*
            What is unwrap? Give me the result and if there's an error, then panic. It's somewhat equivalent to:
            match line {
                Ok(line) => line,
                Err(e) => panic!("Error"),
            }
            */
            curr_sum += line.parse::<i32>().unwrap();
        }
    }
    const TOP_K_ELVES: i32 = 3;
    let mut total_calories = 0;
    for _ in 0..TOP_K_ELVES {
        total_calories += heap.pop().unwrap();
    }
    println!(
        "{:?} total calories were retrieved by the top {:?} elves",
        total_calories, TOP_K_ELVES,
    );
    Ok(())
}

I've included comments in various sections describing some of the things I learned and was curious about.

Let's walk through the code, highlighting the interesting parts.

Main function

Line 24 declares the main function. This is the entry point to a Rust program. The keywords after -> indicates the return value of the function. In this case, the return value is of type Result. It looks like this:

enum Result<T, E> {
   Ok(T),
   Err(E),
}

This is a very common type in Rust that you'll see everywhere. I'll explain why in a following section.

In the Result enum, Ok implies the success response and Err implies the Error type. In our case, the success return type is () which is known as the unit primitive: https://doc.rust-lang.org/std/primitive.unit.html. It's similar to the void type in other languages. The error type is of type std::io::Error.

Reading A File

To solve the challenge, we must first read in a file of information. I downloaded the file and saved it in my repo under day1.txt. The next step is to use Rust to read the file.

I use the std::path::Path module in the Rust standard library to do this. To do that, I first reference the module as a use declaration above to tell the compiler that I'm going to be using symbols from that module. The Path::new function returns a reference to a path object that can be passed into File::open for reading. I'll talk about references in a different post.

Next, I pass in the Path reference to the File::open function which, if you look at the docs, returns a std::io::Result type. This type is just a shortcut for the std::result::Result type that we saw earlier in the main function - it's a very common return type in Rust.

? Operator

The interesting thing about this line of code is the ? mark operator at the end of the File::open function call. What does it do?

The ? operator is specific to Result or Option function return types. It causes the function to return with the error if there is any error in the File::open call. If there isn't an error, in this case, it'll return the success type of the Result enum which is a File object.

When used on the Result type, it operates similarly to the following code:

let file = match File::open(&path) {
    Ok(file) => file,
    Error(error) => return Error(),
}

This code returns a File object if File::open succeeds. If there is an error in it, then the main function returns with the std::io::Error that File::open may throw. When we use the ? operator, the function must return a Result with that Error type.

Reading Line-By-Line

Now that we have the plumbing to read the file, let's go through it line-by-line to solve the challenge. We will use the BufReader struct to do this. The struct implements a trait called BufRead and this trait has a method lines() which we can use to iterate through each line of the file. A trait in Rust is similar to interfaces in other languages. The interesting thing is that we need to call use std::io::BufRead to tell the compiler that we want to use this trait. Otherwise, it'll throw an error saying that the lines() method does not exist on the struct BufReader.

The reader.lines() call returns a Result<String, std::io::Error> type. To get the String value out of the result, we can, just like we did previously, use the ? operator. Note that the error type in Result<String, std::io::Error> is the same as the error type when we used the ? operator on the File::open call which is why we can use the ? operator here as well.

Now that we're able to read each line, we implement the logic to solve the challenge. I'm not going to talk too much about the logic as this post is intended to talk about Rust, not the challenge.

Converting a String to an i32

line.parse::<i32>() is a function you can use to convert a String to an i32. Its return type is Result<i32, ParseIntError>. I initially tried to pull the i32 out of the Result return type using the ? operator, but the compiler wouldn't let me. The reason is because, when using the ? operator, upon an error of the parse method, the main function would return with error ParseIntError. Right now, the main function is returning an error of type std::io::Error which is a different type from ParseIntError.

So, what is another way that we can pull the i32 out of the Result<i32, ParseIntError> type? Similar to what we discussed earlier, we can use the match keyword like so:

curr_sum += match line.parse::<i32>() {
    Ok(val) => val,
    Err(e) => panic!("Error trying to turn the string to an int"),
}

This snippet will return the i32 if it's able to convert the String to an i32. If not, it will panic!. What is a panic? A panic is an unrecoverable error that will cause the program to exit immediatly.A short-hand of the above match snippet is to use the unwrap method.

Conclusion

These are some of the interesting things I learned about Rust in the first challenge in Advent Of Code. I also learned a bunch about ownership, references, and borrowing which I'll save for a different post since those are much longer topics.

Understanding Startup Equity

Kunal Desai — Thu, 08 Dec 2022 15:13:47 +0000

Understanding startup equity can be confusing, this post attempts to explain what I've learned about it over the years.

Fundraising

To kick off the discussion on fundraising, I'm going to first illustrate a basic example of what a fundraising round might look like. Company A has 1000 shares. The founder of the company owns 100% of the shares (1000 shares). Now, the founder is seeking an investment to expand the business. She finds an investor that is willing to purchase 100 shares of the company at $100 each. To accommodate this investor, the business will increase the number of available shares to 1,100 and give 100 of these shares to the new investor. In exchange, the company will receive $10,000 (100 shares * $100 each). The company now has a post-money valuation of $110,000 (# of shares * price per share).

Dilution

The founder of the company in the example got diluted as a result of a new fundraise. Prior to the fundraise, she had owned 100% of the company. After the fundraise, she owns about 90% of the company and the investor that invested in the company owns the remaining 10%. This is known as dilution during a fundraise. Because the shares issued to investors are added to the pool of existing shares in the company, the percent ownership of existing shareholders is reduced.

A typical fundraise has between 10 - 20% dilution and multiple fundraises continues to reduce the percent owned by a company over time. If the founder of the company in the example decided to raise money again, she would likely need to issue 110 - 200 new shares of the company, which would bring her ownership percentage down from 90% after the first fundraise to 82% after the second fundraise. All existing shareholders (existing investors, employees) get diluted in a similar way.

Valuation and Stock Class

After the fundraise, Company A now has a valuation of $110,000. The valuation is the value of the company that investors have assigned it based on what they are willing to pay per share. The amount the investors are willing to pay per preferred share multiplied by the number of shares in the company is equal to the company's valuation.

Investors in a startup are typically given a special class of shares called preferred shares, as opposed to common shares which founders and employees generally get. What is the difference? The difference is mostly liquidation preference. When a startup exits, the preferred share holders can choose to either take back all the money they have invested or convert their preferred stock into common stock and receive their share of the exit based on the percentage of the company they own. Here are a couple examples illustrating when this liquidation preference matters:

Say Company A is not doing well and gets sold for $10,000. The investor in Company A, as a preferred share owner, has the option of recouping her initial investment in the company ($10,000) or converting her preferred shares to common stock and receiving a percentage of the sale price. In this case, the investor had purchased 100 shares which is about 10% of the company. If the investor decides to convert her preferred shares to common stock, she would receive 10% of the sale price in the sale which is $1000. In this case, the investor would choose to recoup her initial investment and would get all $10,000 back. The founder of the company, although owning 90% of the shares, would get $0.
Say Company A sold for $50,000. The investor in Company A has the option of recouping the $10,000 investment or getting 10% of the sale price ($5,000). The investor would choose to recoup the $10,000 investment and the founder would receive the remaining $40,000. Typically, the founder is not the only other shareholder in a company. If there was another shareholder, the remaining $40,000 would get split among the shareholders appropriately based on their equity percentage.
Say Company A does great and sells for $200,000. The investor in Company A has the option of recouping the $10,000 investment or getting 10% of the sale price ($20,000). The investor would choose to convert their preferred stock to common stock and receive the $20,000. The founder would receive the remaining 90% of the share price which is $180,000. As you can see, owning 90% of a company as a founder / employee (or other common stock shareholder) doesn't always translate to 90% of the exit price. It only does when the exit price exceeds the company valuation.

There is also another class of share that companies can give investors called participating preferred shares. These shares have even more power than classic preferred shares in that participating preferred shares not only give holders the ability to recoup their initial investment, but also receive their share of the remaining exit price. Using example #3 above, the investor that holds participating preferred shares would make $29,000 ($10,000 to recoup the initial investment and $19,000 which is 10% of the remaining exit price). The founder would receive 90% of the remaining exit price which is $171,000. Make sure you know what class of shares your company offers to investors, it can greatly impact your financial outcome as a common shareholder.

Given the distinction between the different stock classes (common, preferred, participating preferred), it seems that there should be a price difference between them! In fact, there is. When company A receives a post-money valuation of $100,000 after the investment, it doesn't mean that the founder now has equity worth $90,000. This is because the founder has common share while the investor has preferred shares. For an early stage startup, the price of common stock is heavily discounted from the preferred share price (can be 80% in some cases) since there is still a lot of risk involved and likely no sign of an exit soon. In other words, if you own 10% of a startup valued at $10M, you're not a millionaire...yet. The common stock price will eventually converge to the preferred share price assuming the company is doing well and is getting close to an exit.

Stock Options

As an employee or founder, you'll typically get stock options as part of your compensation package. Stock options are the ability to purchase a stock at a specific price (often known as the strike price). Below are some important tips that will help you understand how things work.

Tips for Founders / Early Employees

Remember to exercise your stock options early if you can afford it! You may be allowed to exercise even unvested stock options ahead of time. However, upon termination of employment at the company, you will need to give back unvested stock options.

If you choose to exercise your stock options, remember to file an 83b election. This is a form you send to the IRS which inform the IRS that you are electing to pay taxes on the fair market value on the date of the stock grant instead of on the fair market value of when your stock options vest. Remember, you need to file the 83b days within 30 days of when you exercise your stock options!

You should also be aware of QSBS (Qualified Small Business Stock). The stock a company gives you is QSBS if at the time you receive your stock (time you exercise your stock options) the company's assets total less than $50 million and you hold the stock for 5 years after. The benefit of having QSBS is that federal capital gains tax doesn't apply on gains up to $10 million. This can save you 20%! There's no actionable item to take advantage of this apart from exercising your options early and holding it for at least five years.

Tips for All Stock Option Holders

If you're evaluating a compensation package that includes stock options, it is important to understand your strike price, tax implications of exercising stock options, and how long you have after termination of employment to exercise your vested stock options.

Your strike price is the amount of money it costs to turn your stock option into common stock. Typically, your strike price isn't determined at the time you sign the offer, or even your first day on the job. It is determined by the next company valuation after your start date. You want a low strike price because you will likely be able to exercise your options earlier (taking on less risk by exercising cheaper options) and don't have to pay as much money to exercise them.

When exercising stock options, you should also understand the tax implications. Just like there are different classes of stock, there are also different classes of stock options, ISOs and NSOs. What's the difference? The difference is that, if you have NSOs, you are taxed on the difference between the strike price and fair market value price of the stock at the time of exercise. If you have ISOs, you are not taxed at all at the time of exercise. Once you decide to sell the stock, if you have NSOs, you pay taxes on the difference between the value at exercise and the current share value. If you have ISOs, you pay taxes on the difference between the strike price and the current share value. This typically matters a lot if you plan to exercise the options before the stock is liquid (you can sell the stock for money) since you may be paying a lot in taxes before you are able to make cash gains from the stock.

Finally, as an employee evaluating a compensation package with stock options, it is important to know how long the company allows you to exercise your vested stock options for in the case that your employment at the company is terminated. I have seen some companies only offer 3 months of an exercise window while others have offered a several year window. If your strike price is high and you aren't able to afford the stock options immediately, then, if you don't want to give up your vested stock options, you may be stuck at the company till you are able to afford exercising your options.

Conclusion

There are a lot of details involved when understanding startup equity and I hope that this post demystifies some of it for you. This is mostly information I wish I had when deciding how to evaluate stock options as part of a compensation package. I hope it was helpful!

Revisiting Postgres

Kunal Desai — Tue, 06 Dec 2022 08:00:00 +0000

At Robinhood (my previous company), most of the interesting technical challenges I worked on involved a database. But at Mighty, my first 18 months didn't involve anything with databases! After having worked for months on Mighty's streaming team, I recently switched to work on the infrastructure team to help the company scale to serve many more users. The job of the infrastructure team at Mighty is very similar to a subset of the service AWS, GCP, and Azure provide: provide users with secure and performant virtual machines while giving them an easy way to manage them. Part of this job has involved using databases to store virtual machine state.

Race Conditions

One of the trickiest parts of using databases, as I'm sure most people will tell you, is managing race conditions at scale. Race conditions typically occur when you have a read, modify, then write operation. For example:

def assign_user_to_machine(user, machine):
    execute_sql("BEGIN")
    machine = execute_sql(f"SELECT * FROM machines WHERE id = {machine.id}")
    if (machine.state != "open"):
        raise Exception("Cant assign a user to non-open machine")
    execute_sql("UPDATE users SET machine_id = {machine.id} WHERE id = {user.id}")
    execute_sql("UPDATE machines SET state = 'assigned' WHERE id = {machine.id}")
    execute_sql("COMMIT")

Say this database transaction is being run on two separate threads of execution. At first glance, it may seem impossible for two users to be assigned to the same machine. However, in an environment with multiple simultaneous threads of execution, it is possible. Say both these functions are executed simultaneously:

Execution #1: assign_user_to_machine(User1, Machine1)

Execution #2: assign_user_to_machine(User2, Machine1)

Say Execution #1 and Execution #2 both get to right before this line:

execute_sql("UPDATE users SET machine_id = {machine.id} WHERE id = {user.id}")

To both executions, it appears that the machine is the open state. Now, once execution of both the threads completes, two users will have machine_id = [Machine1.id](http://Machine1.id) and the machine will be put in the assigned state. This is really bad!

Concurrency Control

There are two ways that I'm aware of to prevent race conditions: pessimistic concurrency control and optimistic concurrency control.

Pessimistic concurrency control is pre-emptively using explicit locking on database tables and rows before modifying the objects. Optimistic concurrency control does not use explicit locking, but rather checks that the row hasn't been altered by a separate concurrent thread of execution when it needs to update it. If it has been altered, then abort the transaction. I will demonstrate some examples of both these methods below.

Pessimistic Concurrency Control

At Robinhood, the primary method of concurrency control was to use pessimistic concurrency control. When we had to solve this problem at Mighty, my gut said to use the same technique. Pessimistic concurrency control involves explicit locking. The example from above has been modified to be race condition free by using a SELECT FOR UPDATE SQL statement. SELECT FOR UPDATE acquires a lock explicitly on all rows in the specified table that match the WHERE condition.

def assign_user_to_machine(user, machine):
    execute_sql("BEGIN")
    machine = execute_sql(f"SELECT FOR UPDATE * FROM machines WHERE id = {machine.id}")
    if (machine.state != "open"):
        raise Exception("Cant assign a user to non-open machine")
    execute_sql("UPDATE users SET machine_id = {machine.id} WHERE id = {user.id}")
    execute_sql("UPDATE machines SET state = 'assigned' WHERE id = {machine.id}")
    execute_sql("COMMIT")

Now Execution #1 and Execution #2 of this function must occur serially rather than in parallel. If Execution #1 acquires the lock explicitly via SELECT FOR UPDATE on Machine1, Execution #2 will be blocked on the SELECT FOR UPDATE statement till Execution #1 commits the transaction, and as a result, releases the lock. The lock is held till the transaction has either been committed or rolled back, at which point the lock is released. Once the transaction completes, Execution #2 can continue to execute. Execution #2 will notice that the machine in question is now in the assigned state, so it will throw an exception Can't assign a user to a a non-open machine. This new code prevents the race condition and resulted in the expected behavior!

Optimistic Concurrency Control

Implementing optimistic concurrency control is best explained by showing an example. The code would look like the following to implement optimistic concurrency control on our initial example:

def assign_user_to_machine(user, machine):
    execute_sql("BEGIN")
    machine = execute_sql(f"SELECT * FROM machines WHERE id = {machine.id}")
    if (machine.state != "open"):
        raise Exception("Cant assign a user to non-open machine")
    execute_sql("UPDATE users SET machine_id = {machine.id} WHERE id = {user.id}")
    num_updated = execute_sql("UPDATE machines SET state = 'assigned' WHERE id = {machine.id} AND state = 'open'")
    if num_updated == 0:
        execute_sql("ROLLBACK")
        raise Exception("Another transaction concurrently modified this machine")
    execute_sql("COMMIT")

Execution #1 and Execution #2 can still occur concurrently, however, two users will never be assigned to the same machine. Let's run through our initial example of how a race condition may happen to show that it will no longer happen with our new code. First, say that Execution #1 and Execution #2 both execute up until the following line:

num_updated = execute_sql("UPDATE machines SET state = 'assigned' WHERE id = {machine.id} AND state = 'open'")

Because the UPDATE statement acquires lock implicitly on the Machine1 row, only one of these statements can occur at a single point in time. Say Execution #2 executes it first. After Execution #2 completes executing the statement, the machine's state is now assigned. Note that Execution #2 must complete the transaction till Execution #1 can continue. This is because the lock implicitly acquired by the UPDATE statement in Execution #2 is held from the UPDATE statement till the transaction is either committed or rolled back. Execution #2 continues, num_updated is 1 so the transaction commits and the lock on Machine1 is released. Execution #1 can continue since the lock on Machine1 has been released. When executing the UPDATE statement, Execution #1 finds that no machines match the WHERE clause of the UPDATE statement since the machine with id 1 (Machine1) is not in the state open. num_updated is 0 and an Exception is raised and the entire transaction is rolled back. This includes a revert of the UPDATE users SET machine_id = ... statement! Now you can see that optimistic concurrency control can also prevent race conditions.

When should I use optimistic vs. pessimistic concurrency control?

Optimistic concurrency control is better when there isn't a lot of contention. If you expect that race conditions will be very rare, then optimistic concurrency control may be the most performant since you are minimizing critical sections of your application by not using any explicit locking. At Mighty, we chose to prevent race conditions by using pessimistic concurrency control. Why? Two reasons.

Firstly, I feel that a hidden cost of optimistic concurrency control is maintaining and reasoning about application logic is harder in this world. Because of this, it impedes developer velocity and increases the probability of bugs. Pessimistic concurrency control, on the other hand, is easier to write and reason about. This is because there is a simple heuristic we can use when updating data: explicitly lock everything that you are updating in a pre-defined order (I explain why this is important in the Deadlocks section).

Secondly, I believe in not solving performance problems before they are an issue! As a small startup, Mighty is likely very far from running into performance problems due to lock contention any time soon.

Database Connections and Holding onto Locks

One concern that we had when using pessimistic concurrency control was: if we are using locks more, does that increase the likelihood that we can hold onto a lock forever in the case of application crashes or network blips? I had a limited understanding of how database connections and transactions worked, and this question forced me to dive a bit deeper into it.

One possible scenario that is helpful to examine is: A client initiates a TCP connection to Postgres, begun a transaction, acquired a lock, and then goes offline. In this situation, Postgres is still waiting for the transaction to terminate and so still has not released the lock in question. Eventually, Postgres will notice that the client has gone offline. How? It will wait for the client TCP connection to terminate. TCP uses keepalives messages from the database to the client (Postgres settings here: https://www.postgresql.org/docs/9.5/runtime-config-connection.html) to determine if the connection to the client is still healthy. When sufficient keepalive messages go un-acked by the offline client, TCP will determine the client is no longer connected and will terminate the connection. When Postgres notices the connection has terminated, all transactions that the specific client connection has been executing will end (rolled back, not committed) and, as a result, all locks that the transaction held will be released.

That answered our question! Our database was not at risk of holding onto locks forever due to application crashes or network blips.

The only risk we hold now is if an active and online client held onto a lock longer than expected. Because we are in control of all clients connecting to our database, this was deemed to be lower priority since it is more easily preventable. Our rule of thumb for reducing the time locks are held are to minimize the amount of time it takes to execute a transaction that holds onto locks. Among other things, this means ensuring that no network I/O happens inside a transaction.

Deadlocks

Another concern the team had about increasing the amount of explicit locking in our application is deadlocks. Deadlock is possible even without explicit locking in our application, this is illustrated in the Deadlock section in Postgres documentation. As discussed earlier, this is because UPDATE statements acquire locks implicitly and don't release it till the transaction is complete.

To prevent this form of deadlock, as the documentation mentions, it is important to run UPDATE queries, always in a consistent order, in all transactions, across all your clients.

Another form of deadlock can happen in a similar way, but when explicit locks (vs. implicit locks acquired by UPDATE) are called simultaneously in an inconsistent order.

Explicit locking in a consistent manner can prevent both cases of deadlock. Additionally, it turns out that this can be very simple by using the following heuristic across all transactions, across all clients:

If a row is being updated in a transaction, then acquire an explicit lock for the row at the beginning of the transaction.
If you are acquiring a lock: First, always lock rows in the users table in the order of their id. Then, always lock rows in the machines table in the same order of their id. Follow this pattern for all other tables that may require a lock. We made this very simple in our application by adding one function that is used for all Postgres locking. That function always obeys the consistent lock ordering that we want.

This heuristic should help us eliminate all deadlocks in our codebase! Explicit locking is no longer an issue since it is always done in a consistent order. Implicit locking from UPDATE statements is also not an issue given the heuristic because the locks are explicitly acquired at the beginning of the transaction in the correct order.

Woo!

It was a lot of fun looking into this over the last day or two. It made me feel like I've really leveled up over the last almost three years of being a software engineer. I remember it'd take me a week or more to understand topics of similar difficulty a year ago, but now it only takes a couple hours to have an understanding I feel confident in. Looking back and feeling progress is always motivating!

Setting Up a Virtual Desk

Kunal Desai — Mon, 05 Dec 2022 08:00:00 +0000

I recently noticed that the base Oculus Quest 2 model was being sold for $300. Shocked that it was so cheap, I decided to buy one and try it out. 8 years prior, I remember attending a Meetup where some engineers were pitching their startup: an IDE in VR. I remember trying their demo and not being very impressed. I was curious how much has improved since then. It's safe to say, I am shocked.

I ordered the Oculus Quest 2 128GB model and it was shipped to my house in some nice packaging. It came with the headset, a charger, two handheld controllers, and some accessories to fit the headset better on your face.

I turned on the Oculus, charged it, and finally got past the setup instructions. I was greeted by a beautiful backdrop with some menu options for using a browser, messaging people (there is an integration with Facebook Messenger), and an app store. Like most, the first thing I downloaded was Beat Saber.

The game as ok. I played it a couple times and it reminded me of Guitar Hero back in the day. I can see how people really enjoy it — I'm just not a huge gamer. The experience was really cool though. I was very happy with the latency in terms of responsiveness to my actions. Things actually felt real!

After that, I decided I was ready to see how I could go about setting up a coding environment. I found Oculus Workroom which is an application built by Meta aimed at simulating a virtual work environment. It goes beyond desks and monitors, but I was particularly interested in the monitor set up. As someone that enjoys traveling while working (a "digital nomad" I guess?) one of the biggest pain points for me was having to pack light and not being able to carry a monitor around with me. As a software engineer, this is a critical piece of my productivity! Buying the Oculus Quest 2, my hope was that I could create many screens of many different sizes without any additional monitors or hardware. That way, I could travel while working, and still have access to the same amount of screen real estate I did at home.

To set this up, I needed to download an Oculus Workroom app on my MacBook and install the Horizon Workroom Beta App on my Oculus Quest 2. Once that was ready, I connected my Oculus Quest 2 to my laptop and it just. worked. I was shocked. I was able to connect to my monitor, see my keyboard, and write code on my normal IDE while in VR. The experience was seamless.

One feature that I was mesmerized by was the hand gestures. Instead of using the controllers provided, there is an option to just use your hands. I kid you not, I spent at least 10 minutes just moving my hands around in the space and feeling how imperceptible latency was between when I told my hand to move and when it moved in virtual reality. On top of that, I was able to click and pan with my hands and fingers rather than the controller.

What's missing?

There has clearly been massive improvements in virtual reality technology over the last 8 years. I'm in awe at the engineering that went on behind it. Nonetheless, there are still some areas of improvement. Here is what is on my bucket list:

Solving my remote work monitor problem - The Oculus Quest 2 doesn't yet solve my remote work monitor problem. The screen size is fixed to two resolutions and I'm only able to view one screen at a time. For me to have the same screen real estate I have at home, I need to be able to see multiple screens simultaneously and would like larger resolutions.
Keyboard recognition - I use an ergonomic Microsoft keyboard at home and my Oculus wasn't able to recognize it. So when I look down at my hands, it looks like I'm typing in space.
Text blurriness - I experienced some text blurriness while using the Oculus. It seems that when my headset is in some positions, the text is less blurry than in other positions. I'd like if there was a way for me to keep the headset more stable or be able to toggle how the text appears to me with software.
Hand gestures - The hand gestures were incredibly low latency, but it wasn't always super accurate. There were distances from my headset at which my VR hands wouldn't accurately represent what I was doing with my hands. Additionally, at times, the Oculus would fail to correctly identify my hand and finger positions when my hands or fingers were close together. For example, sometimes I would touch my fingers together in real life, but my VR hands wouldn't touch. Man all these improvements in VR get me so excited for its future. It has definitely sparked a craving to do some engineering work in this space. Similar to Mighty, it scratches my itch of deep, low level software challenges while being a consumer facing product with a big mission.

My rule of thumb when making software architecture decisions

Kunal Desai — Sun, 04 Dec 2022 14:13:18 +0000

If you have a lot of confidence and/or experience in what an ideal end state would be for your use case, then:

Be explicit about identifying that end state and what it would look like.
Incrementally, as-needed, solve the short term problems biasing for solutions that will get you closer to the ideal end state.

If you don’t have a lot of confidence and/or experience in what an ideal end state would be:

Ship incrementally till you gain confidence in what your ideal end state would be.

Counting to 1 Billion (Node vs. Bun)

Kunal Desai — Mon, 07 Nov 2022 00:02:54 +0000

I was inspired by Ishaan Sheikh's post here on counting to 1 billion: https://dev.to/sheikh_ishaan/count-to-1-billion-20de.

I wanted to see if there was a sizable performance difference in counting to 1 Billion in Javascript when using Node vs. when using Bun.

I ran this program and timed it using the MacOS time utility:

for (let i = 0; i < 1_000_000_000; i++) {
}

I ran it with Node and got:

node counter.js  0.46s user 0.03s system 84% cpu 0.578 total

Then, I ran it with Bun and got:

bun counter.js  0.90s user 0.01s system 99% cpu 0.912 total

I was surprised to see in something as simple as this the time difference was double for Bun vs. Node! Especially since Bun claimed that it was faster than Node since it uses Javascript Core instead of V8.

To verify this was the case, I tested it on 10,000,000,000. Here were the results:

node counter.js  7.87s user 0.06s system 97% cpu 8.118 total

bun counter.js  8.79s user 0.02s system 99% cpu 8.820 total

Clearly, it seems that there may be some bun startup/initialization cost that is causing the counting to be slower than Node. I'm curious to know where this is coming from.

Thanks to writethrough.io for helping me write this article.

Building a GPT-3 Powered Discord Support Bot

Kunal Desai — Sun, 06 Nov 2022 22:17:18 +0000

Discord is becoming a popular platform for open source projects and companies to manage their developer communities. These Discord servers are where developers can go to get their questions answered quickly. Some support channels are extremely busy with the same questions being asked and answered over and over again. I figured that answering these questions might be something that GPT-3 could do really well! So I decided to spend a couple hours to build the bot - this article talks about how the bot was built. If you want to add the Discord bot to your server, use this URL. I’d also like to thank writethrough.io for helping me write this article.

The Tech Stack?

I decided to use Python to build this bot since neural nets are involved and Python has great libraries like PyTorch for neural nets. I also used the discord.py library to make it easy to interact with the Discord API. To deploy the bot, I created a small GCP Compute Engine instance, Prisma, PostgresDB, and Pinecone.

Setting up the Discord Bot

The first thing I needed to do to create this bot was to create a Discord account and application. You can do that here. Under Settings on the left navbar, click Bot and then hit Add Bot.

Set the name and icon of your bot to something fun! Next, you’ll need to generate a token for your bot by hitting Reset Token. Save this token somewhere private because we’ll need it in a bit. Additionally, you’ll need to turn on the Message Content Intent for your Discord bot which is further down the Bot page.

This will allow your bot to read the contents inside the messages. Finally, you’re going to add the Discord bot to your Discord server so you can start testing things. Go back to the Settings on the left navbar, click OAuth2 and click URL Generator in the submenu.

In this step, we’re generating the URL that’s going to allow anyone to add the bot to any server with the correct permissions. We’re going to add the bot scope and Send Messages, Create Public Threads, and Read Message History bot permissions. On the bottom of the page, it will have generated a URL, copy that URL and navigate to it. Follow the steps to add your bot to your server.

Setting up the environment for our Python Bot

The next thing we need to do is to set up the functionality for our bot! We’ll set up our Python bot with some good software hygiene. The only pre-requisites are that you have Python and pip installed.

First, install virtualenv package with python -m pip install virtualenv. This package will help us create a virtual environment with a Python instance that’s local to our repository. That way our environment is reproducible and we don’t mess with the system Python.

Why python -m pip install virtualenv and not just pip install virtualenv? The former ensures that we install virtualenv in the correct Python installation.

Once virtualenv is installed, we’ll create our project folder and virtual environment with mkdir discord-bot , cd discord-bot, and python -m virtualenv venv. This will create a folder in our project directory called venv. To activate the virtual environment, run source venv/bin/activate. Now, if you type which python, it should point to an installation of Python inside discord-bot/venv/bin/.

Reading messages from your Discord server

Now, create the bot.py file where we will write the code for the Discord bot. First, we’ll install the discord Python client with pip install discord.py. Next, we’ll set up the scaffolding for our Discord bot in bot.py:

import discord

class Client(discord.Client):

    async def on_ready(self):
        print(f'{self.user} has connected to Discord!')

    async def on_message(self, message: discord.Message):
        print(message.content)

intents = discord.Intents.default()
intents.message_content = True
client = Client(intents=intents)
client.run('<TOKEN_HERE>')

Replace <TOKEN_HERE> with your Discord bot token from the steps above. This code creates the Discord bot and prints the contents of the Discord message whenever a message is sent in your Discord server. Give it a shot! Run python bot.py to run your bot, then start sending some Discord messages and watch the logs of your bot.

Saving Discord Messages to a Database

Next, we’ll need to start saving our messages to a database so that we can query for them later. For this step, we’re going to use a Postgres database and interact with it using the Python Prisma Client. If you haven’t heard of Prisma, I’d highly recommend checking it out for Node.js projects! It’s amazing. The community has built a version for Python and I wanted to try it out.

Run pip install prisma to get the Python Prisma CLI and library. Next, create a schema.prisma file with the contents:

generator client {
  provider             = "prisma-client-py"
  interface            = "sync"
  recursive_type_depth = 5
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model DiscordMessage {
  id        String   @id @default(cuid())
  channelId String
  guildId   String
  discordId String   @unique
  isThread  Boolean
  content   String
  position  Int
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
}

The important part here is the model section which specifies a table in our Postgres database (which we’re about to set up!). If you’re on MacOS, here is a good guide on starting an instance of Postgres: https://www.robinwieruch.de/postgres-sql-macos-setup/.

Once you’ve gotten your Postgres database set up, you’ll need to identify your DATABASE_URL. It’ll usually be something along the lines of postgresql://<username>@localhost:5432/discord (where username is usually the output of whoami when typed in the terminal). Create the discord database by running psql <DATABASE_URL> and typing in CREATE DATABASE discord.

Next, we’re going to set up our Prisma Python client by running prisma generate --schema=schema.prisma. Additionally, we’re going to migrate our database to include the schema defined in schema.prisma by running DATABASE_URL=<DATABASE_URL> prisma migrate dev.

Now that the database is set up properly and we’ve generated our Python client to interact with the database, lets actually save the messages to a database:

import discord
from prisma import Prisma

db = Prisma()
db.connect()


class Client(discord.Client):

    async def on_ready(self):
        print(f'{self.user} has connected to Discord!')

    async def on_message(self, message: discord.Message):
        is_thread = message.channel.type == discord.ChannelType.public_thread
        discord_message = db.discordmessage.create(data={
            'channelId': str(message.channel.id),
            'guildId': str(message.guild.id),
            'discordId': str(message.id),
            'isThread': is_thread,
            'content': message.content,
            'position': message.channel.message_count if is_thread else message.channel.position or 0,
        })


intents = discord.Intents.default()
intents.message_content = True
client = Client(intents=intents)
client.run('<TOKEN_HERE>')

Run the bot with DATABASE_URL=<DATABASE_URL> python bot.py. It’ll become clear why these fields are necessary later down the tutorial. You can verify that the messages are being saved correctly by hopping into a psql session and querying the "DiscordMessage" table.

Using AI to Answer Questions

This step, in my opinion, is the most exciting step. We’re going to take the messages, and if they’re a question, then respond to them automatically if our AI can. OpenAI provides a GPT-3 model called Davinci which can help us answer questions given a prompt. I’d encourage you to play around with it so you can get a better understanding of it: https://beta.openai.com/playground.

The first thing we’re going to do is integrate OpenAI into our bot. You’ll need to create an OpenAI account and generate an API key here: https://beta.openai.com/account/api-keys. Remember to keep it a secret!

Here is the code to integrate OpenAI’s GPT-3 Completion model:

import discord
from prisma import Prisma
import openai
openai.api_key = "<API_KEY>"

COMPLETIONS_MODEL = "text-davinci-002"
COMPLETIONS_API_PARAMS = {
    # We use temperature of 0.0 because it gives the most predictable, factual answer.
    "temperature": 0.0,
    "max_tokens": 300,
    "model": COMPLETIONS_MODEL,
}
END_PROMPT = "Answer the question as truthfully as possible, if you don't know the answer, say I don't know"

db = Prisma()
db.connect()


class Client(discord.Client):

    async def on_ready(self):
        print(f'{self.user} has connected to Discord!')

    async def on_message(self, message: discord.Message):
        is_thread = message.channel.type == discord.ChannelType.public_thread
        discord_message = db.discordmessage.create(data={
            'channelId': str(message.channel.id),
            'guildId': str(message.guild.id),
            'discordId': str(message.id),
            'isThread': is_thread,
            'content': message.content,
            'position': message.channel.message_count if is_thread else message.channel.position or 0,
        })

        if not is_thread:
            prompt = f"{END_PROMPT}\n{message.content}"
            response = openai.Completion.create(
                prompt=prompt,
                **COMPLETIONS_API_PARAMS,
            )
            answer = response["choices"][0]["text"].strip(" \n")
            print(prompt)
            print(answer)
            if "I don't know" in answer:
                return

            thread = await message.create_thread(name=f"Answer to {message.content}")
            await thread.send(answer)


intents = discord.Intents.default()
intents.message_content = True
client = Client(intents=intents)
client.run('<TOKEN_HERE>')

This code takes every message that’s not in a thread and attempts to use GPT-3 to answer the question. It adds “Answer the question as truthfully as possible, if you don’t know the answer, say I don’t know” before each message and then asks GPT-3 to complete the answer. If GPT-3 doesn’t know the answer, then the bot will stay silent, otherwise it will create a thread and respond with the answer.

Great! Now we have a silent bot that does nothing since GPT-3 doesn’t know anything about your Discord server and the types of questions that have been asked or answered in the past. It will always respond with “I don’t know.” Now is the fun part - we’re going to add context to the GPT-3 prompt so that the AI can actually give us some useful answers to our questions.

The biggest question we have to answer is: how do we find out what context is relevant to include in the GPT-3 prompt? We can’t include all of it - GPT-3 has a 4000 token limit. One way to decide what context is relevant is by turning the question into a vector, converting all other messages in the database to a vector and then pulling the top 10 similar messages. If you’re curious about text embeddings, this wikipedia article is a good place to learn: https://en.wikipedia.org/wiki/Word_embedding.

First, we need to turn our messages into an embedding. For that, we’re going to use the CLIP model. CLIP is a model that can help turn text or images into the same latent space so that text and images can be compared - it’s really exciting technology! In this tutorial, we’re just going to use it for text embeddings. The CLIP GitHub repo is a great place to learn more: https://github.com/openai/CLIP

Let’s integrate CLIP into our bot so that we can generate embeddings for every message.

from typing import List
import discord
from prisma import Prisma
import torch
import clip
import openai
openai.api_key = "<API_KEY>"

COMPLETIONS_MODEL = "text-davinci-002"
COMPLETIONS_API_PARAMS = {
    # We use temperature of 0.0 because it gives the most predictable, factual answer.
    "temperature": 0.0,
    "max_tokens": 300,
    "model": COMPLETIONS_MODEL,
}
END_PROMPT = "Answer the question as truthfully as possible, if you don't know the answer, say I don't know"

db = Prisma()
db.connect()

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)


def generate_clip_embedding(text) -> List[torch.FloatType]:
    tokens = clip.tokenize([text])
    with torch.no_grad():
        text_features = model.encode_text(tokens)
        return text_features.cpu().detach().numpy().tolist()[0]


class Client(discord.Client):

    async def on_ready(self):
        print(f'{self.user} has connected to Discord!')

    async def on_message(self, message: discord.Message):
        is_thread = message.channel.type == discord.ChannelType.public_thread
        discord_message = db.discordmessage.create(data={
            'channelId': str(message.channel.id),
            'guildId': str(message.guild.id),
            'discordId': str(message.id),
            'isThread': is_thread,
            'content': message.content,
            'position': message.channel.message_count if is_thread else message.channel.position or 0,
        })

        clip_embedding = list(
            map(str, generate_clip_embedding(message.content)))
        if not is_thread:
            prompt = f"{END_PROMPT}\n{message.content}"
            response = openai.Completion.create(
                prompt=prompt,
                **COMPLETIONS_API_PARAMS,
            )
            answer = response["choices"][0]["text"].strip(" \n")
            print(prompt)
            print(answer)
            if "I don't know" in answer:
                return

            thread = await message.create_thread(name=f"Answer to {message.content}")
            await thread.send(answer)


intents = discord.Intents.default()
intents.message_content = True
client = Client(intents=intents)
client.run('<TOKEN_HERE>')

Now that we’ve generated the CLIP embeddings, we need to store it somewhere. Our goal is to be able to get the top 10 closest messages to the current message, so we have to store it somewhere where that query will be really fast. Unfortunately, a Postgres database is not going to be the best storage option to calculate top 10 closest embeddings to a vector. Luckily, there are some great solutions out there like Pinecone.

To get started with Pinecone, follow the Quickstart instructions here. Here is how we integrate Pinecone in our bot:

import discord
from prisma import Prisma
import pinecone
import torch
import clip
import openai
openai.api_key = "<API_KEY>"

COMPLETIONS_MODEL = "text-davinci-002"
COMPLETIONS_API_PARAMS = {
    # We use temperature of 0.0 because it gives the most predictable, factual answer.
    "temperature": 0.0,
    "max_tokens": 300,
    "model": COMPLETIONS_MODEL,
}
END_PROMPT = "Answer the question as truthfully as possible, if you don't know the answer, say I don't know"

db = Prisma()
db.connect()

pinecone.init(api_key="<API_KEY>",
              environment="us-west1-gcp")
pinecone_index = pinecone.Index("message-embedding")

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)


def generate_clip_embedding(text) -> List[torch.FloatType]:
    tokens = clip.tokenize([text])
    with torch.no_grad():
        text_features = model.encode_text(tokens)
        return text_features.cpu().detach().numpy().tolist()[0]


class Client(discord.Client):

    async def on_ready(self):
        print(f'{self.user} has connected to Discord!')

    async def on_message(self, message: discord.Message):
        is_thread = message.channel.type == discord.ChannelType.public_thread
        discord_message = db.discordmessage.create(data={
            'channelId': str(message.channel.id),
            'guildId': str(message.guild.id),
            'discordId': str(message.id),
            'isThread': is_thread,
            'content': message.content,
            'position': message.channel.message_count if is_thread else message.channel.position or 0,
        })

        clip_embedding = list(
            map(str, generate_clip_embedding(message.content)))
        pinecone_index.upsert([(discord_message.id, clip_embedding, {
            'guildId': str(message.guild.id), 'id': discord_message.id})])
        if not is_thread:
            prompt = f"{END_PROMPT}\n{message.content}"
            response = openai.Completion.create(
                prompt=prompt,
                **COMPLETIONS_API_PARAMS,
            )
            answer = response["choices"][0]["text"].strip(" \n")
            print(prompt)
            print(answer)
            if "I don't know" in answer:
                return

            thread = await message.create_thread(name=f"Answer to {message.content}")
            await thread.send(answer)


intents = discord.Intents.default()
intents.message_content = True
client = Client(intents=intents)
client.run('<TOKEN_HERE>')

In this tutorial, we created a Pinecone index called message-embedding and upload all of our embeddings to that index. Notice that we also upload the guildId and an id as metadata to our index. We do that so we can filter by the guildId when searching for similar embeddings (we wouldn’t want to pull messages from other servers/guilds). The id is stored so that we have a mapping of vector to id in our Postgres database. It’ll help us get the actual text of the similar messages.

Next, we need to add the similar messages to our prompt.

import discord
from prisma import Prisma
import pinecone
import torch
import clip
import openai
openai.api_key = "<API_KEY>"

COMPLETIONS_MODEL = "text-davinci-002"
COMPLETIONS_API_PARAMS = {
    # We use temperature of 0.0 because it gives the most predictable, factual answer.
    "temperature": 0.0,
    "max_tokens": 300,
    "model": COMPLETIONS_MODEL,
}
END_PROMPT = "Answer the question as truthfully as possible, if you don't know the answer, say I don't know"

db = Prisma()
db.connect()

pinecone.init(api_key="<API_KEY>",
              environment="us-west1-gcp")
pinecone_index = pinecone.Index("message-embedding")

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)


def generate_clip_embedding(text) -> List[torch.FloatType]:
    tokens = clip.tokenize([text])
    with torch.no_grad():
        text_features = model.encode_text(tokens)
        return text_features.cpu().detach().numpy().tolist()[0]


class Client(discord.Client):

    async def on_ready(self):
        print(f'{self.user} has connected to Discord!')

    async def on_message(self, message: discord.Message):
        is_thread = message.channel.type == discord.ChannelType.public_thread
        discord_message = db.discordmessage.create(data={
            'channelId': str(message.channel.id),
            'guildId': str(message.guild.id),
            'discordId': str(message.id),
            'isThread': is_thread,
            'content': message.content,
            'position': message.channel.message_count if is_thread else message.channel.position or 0,
        })

        clip_embedding = list(
            map(str, generate_clip_embedding(message.content)))
        pinecone_index.upsert([(discord_message.id, clip_embedding, {
                              'guildId': str(message.guild.id), 'id': discord_message.id})])
        if not is_thread:
            response = pinecone_index.query(vector=clip_embedding, top_k=10, filter={
                'guildId': str(message.guild.id)}, include_metadata=True)
            prompt = ""
            for match in response['matches']:
                score = match['score']
                is_match_relevant = score < 0.6
                if is_match_relevant:
                    break
                id = match['metadata']['id']
                if id == discord_message.id:
                    continue
                matching_message = db.discordmessage.find_unique(where={
                                                                 'id': id})
                if matching_message:
                    prompt += f"* {matching_message.content}\n"
            # limit the context in the prompt to 2,000 characters
            prompt = prompt[:2000]

            prompt += f"\n{END_PROMPT}\n{message.content}"
            response = openai.Completion.create(
                prompt=prompt,
                **COMPLETIONS_API_PARAMS,
            )
            answer = response["choices"][0]["text"].strip(" \n")
            print(prompt)
            print(answer)
            if "I don't know" in answer:
                return

            thread = await message.create_thread(name=f"Answer to {message.content}")
            await thread.send(answer)


intents = discord.Intents.default()
intents.message_content = True
client = Client(intents=intents)
client.run('<TOKEN_HERE>')

Finally, our Discord bot is able to pull in relevant past context and include it in our prompt. It turns out, this works pretty well! You can see how it works for you by asking a question in Discord and then answering it yourself. Ask the same question again, and the Discord bot should be able to answer it for you!

In the next step, we’re going to productionize our bot. To do that, save your virtualenv dependenceis to a requirements.txt file with pip freeze > requirements.txt. Push your bot.py, requirements.txt, and schema.prisma files to a GitHub repo.

Productionizing the Bot

The final step is to productionize this bot so others can use it. I didn’t find many great options for productionizing a Discord bot, so I ended up going the old fashion route. The first thing you need to do is create a Linux GCP Compute Engine instance. You can follow the docs here to create one: https://cloud.google.com/compute/docs/instances. Next, you’ll need to create your Postgres instance, I used CloudSQL from GCP: https://cloud.google.com/sql/docs/postgres/connect-instance-private-ip.

Once you’ve got those set up, pull the GitHub repo which has the bot.py and requirements.txt file onto your GCP Compute Engine instance. Make sure you have Python installed on the Compute Engine along with pip. Next, run python -m pip install -r requirements.txt to install all the dependencies and run prisma generate --schema=schema.prisma to create the Python client to interact with your DB.

Finally, we’re going to run the bot with a systemd process. Systemd is a process manager in Linux. To use it, first create a systemd service file and put it in /etc/systemd/user/discord-bot.service. The contents of the file are:

[Unit]
Description=Discord Bot
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=5
WorkingDirectory=~/
Environment=DATABASE_URL="<DATABASE_URL>"
ExecStart=python bot.py

[Install]
WantedBy=multi-user.target

Make sure to use your production database URL here. To start the bot, reload the systemd daemon with systemd --user daemon-reload and start the bot with systemd --user start discord-bot. Your bot should be running now! You can tail the logs by running journalctl -f --user -u discord-bot

Conclusion

I hope you enjoyed learning how to build this bot, I definitely did! Subscribe for more content like this and please comment if you have any questions or suggestions on how to improve this tutorial.

This article was written with the help of writethrough.io