DEV Community: Kevin Alemán

Got featured :noice:

Kevin Alemán — Tue, 10 Mar 2026 16:45:59 +0000

Jess Lee for The DEV Team

Mar 10

🙌🏽

Kevin Alemán — Sun, 08 Mar 2026 18:51:54 +0000

Kevin Alemán

Mar 4

In the AI Era, Code Is Cheap. Reputation Isn’t.

#ai #opensource #programming #software

Comments 15

6 min read

In the AI Era, Code Is Cheap. Reputation Isn’t.

Kevin Alemán — Wed, 04 Mar 2026 21:01:29 +0000

In the era of AI it's easier than ever to be an Open Source contributor!

But, at the same time, and quite paradoxically, it's harder than ever.

Why? Because it's now mechanically easier, but reputationally harder

Previously, you competed against other incredibly smart volunteers, or maybe against employed developers that contributed their expertise and free time to open source, or against newcomers. Now, you compete against all of that and an army of AI agents generating pull requests, issues, refactors and vulnerability reports faster than what any human can do.

It's pretty tempting. Using an LLM can reduce significantly the cognitive work you have to do to understand a codebase, and even to produce a valuable change request. Things that were previously "very hard" to do are now a few prompts away, and that lowered the entry barrier for contributions, which is great! But also created a massive increase in volume, which is flooding repositories.

What's the problem?
Maintainer capacity did not increase
Review cost did not decrease

These two truths have already led maintainer to resort to drastic measures, like what happened with curl:

Death by a thousand slops | daniel.haxx.se

daniel.haxx.se

So, how do you become a valuable Open Source contributor in the era of infinite code generation? Here are a few tips learned from dealing with a large volume of contributions:

1. Think. Then think again

AI can produce great code, but it can also produce the worst code ever seen by the humanity. This depends a lot on:

The prompt that you use
The quality of the LLM
The context being shared

And the most important one:

Your own coding abilities.

Why do you need to know how to code before using LLMs to contribute to open source?

Well, because an LLM cannot know everything.
For example, we had a PR that attempted to change this:

        } catch (error) {
            console.error('Some error:', error);
        }

After

        } catch (error) {
            logger.error('Some error:', error);
        }

You would say: that is a valid change, we want to use logger instead of console! What's the problem?

The problem was that the LLM didn't know (and didn't check, cause it wasn't asked to) that we recently changed the typings on our logger instances, rendering the change invalid.

What a valid change would have looked like:

        } catch (err) {
            logger.error({ msg: 'Some error', err });
        }

But, the LLM didn't know this, and it would have required quite some context for it to know. Good LLMs (Claude, Codex, etc) would have caught, but others would not, unless you asked them. How would you ask something that you didn't know? There's where your coding abilities enter into play.

Here's the key point: You cannot prompt what you don't know exists

Someone that wants to be a contributor doesn't trust blindly on what the LLM produces. Knows how to use the tooling (typecheck, in this case) to catch those issues, reads the code to understand current conventions, caveats, etc.
For this programmers, LLMs become a powerful tool, cause it helps you with the mechanical task of coding and gives you enough time to think.

This also protects your reputation. If maintainers see a pattern of low-signal PRs from you, they’ll eventually stop reviewing them. Or worse.

LLMs make contribution faster, but they don’t replace your brain unless you let them.

2. Respect volume. Review is not free.

You are happy. We're also happy that you want to contribute to open source. It's a noble endeavor and it's something that helps the repository to be safer & you to be better. It's a win-win!

But, maintainers have:

Limited capacity.
Limited time.
Limited patience.

We understand, you are happy you created your new PR, and we understand the joy in you of doing that. We've all being there. You want, for sure, your PR to be merged so you can say with honor: "I contributed to X!"

But, be reasonable:

The cost of opening a PR is now near zero.
The cost of reviewing it is not.

This asymmetry is the real tension of the AI era.

If you:

comment “please review” repeatedly,
ping maintainers aggressively,
demand merging timelines,

You’re increasing the cognitive cost of your contribution. Even if your PR is valid.

And here’s the harsh truth:

Maintainers are human.

If interacting with you feels expensive, they may subconsciously deprioritize your work. That means fewer reviews, fewer merges, fewer release notes mentions.

This is specially important when you use LLMs or Agents as they are fast, way faster than humans, so you could end up doing 100 PRs that you think are valid, and maybe they are! But if the maintainers don't have the capacity to review them all, they just won't.

That's the definition of a DoS (Denial of Service): you send more requests than what the server (maintainers) can process.

3. Quality that doesn't cut corners (pls don't sue me Wendy's)

Maintainer capacity is low. We closed around 700 issues (stale, invalid, fixed) in the past month, but at the same time, we received 300 new issues. Some of those issues were valid, some of them were not. The problem is we don't know beforehand, we have to check them all. You cannot trust an LLM to confirm if something is slop... cause the check may be slop too!

Related link, curl fighting against AI slop vulnerability reports:

HackerOne Curl report

As you see, maintainers won't be happy if you waste their time, cause they have stuff to do. The link above, as well as this beautiful list

have something in common: they look legit, until they don't. However, for realising something is slop you have to check manually, and that is time/effort wasted.

So you have to be human, respect the times maintainers have and respect their work. How you do that? Helping the maintainers reduce the cost of reviewing your code:

Before submitting:

Did you reproduce the issue yourself?
Did you run CI locally?
Did you include screenshots for UI changes?
Did you provide benchmarks for performance claims?
Did you include a PoC for security reports?
Did you check if a feature request already exists?

If you claim a performance improvement, show numbers.
If you fix a bug, explain the root cause.
If you propose a feature, ask first.

High quality contributions get reviewed faster.
Low signal contributions accumulate.

In the AI era, reputation compounds. If maintainers can trust you, your PRs will be reviewed faster, and you will feel happier. Win-win!

4. Bundle trivial changes.

Typos happen. Smol mistakes happen. For example

Before:

        if (!sub) {
            throw new Error('subsciption not found');
        }

After

        if (!sub) {
            throw new Error('subscription not found');
        }

Do you spot the bug? Yep, a missing r on subscription. The contribution is valid, yes, we want to write correct words in english.

Does it deserve a PR? Not by itself, maybe if you combine a few more. Think if you find 10 of those misspellings. What's easier for maintainers? 1 PR with 10 changes or 10 PRs with 1 change? Hehe.

If you were the maintainer, what would you prefer?

Every PR has overhead: CI runs, notifications, review time, merge time, mental context switch, etc. Reduce the overhead by batching trivial work.

AI can find an enormous list of fixes like this in a codebase, and if you use an agent like Claude or Clawd you can tell "Hey for each issue create a PR" and then become the most hated contributor on the project :(

Don't do that. Please. Be mindful. Be human.

LLMs are fun, but human work is better.

We're on the AI era, we cannot fight AIs anymore, they're here to stay. What we can fight is how we use them.

For GSoC for example, we value human contributions. That's the point of the program: more humans converting to open source contributors!

We need new generations of maintainers, of smart people that will use LLMs wisely. You can do amazing stuff with them if you use them properly, but nothing will replace your ability to do amazing stuff as a human. You have to remember that.

It's easier to contribute, but harder to stand out from the crowd.
Reputation matters.
Trust is earned.
You build it by doing good work.

As a final note, this is pretty much personal experience. Some projects may have different rules. Some may like to receive 1000 PRs a day (?) idk.

But, the main point is: AI is not human, but maintainers and contributors are. So remember that.

Welcome to Open Source!

If u want to read more:

AI is destroying Open Source, and it's not even good yet - Jeff Geerling

Over the weekend Ars Technica retracted an article because the AI a writer used hallucinated quotes from an open source library maintainer. The irony here is the maintainer in question, Scott Shambaugh, was harassed by someone's AI agent over not merging its AI slop code. It's likely the bot was running through someone's local 'agentic AI' instance (likely using OpenClaw). The guy who built OpenClaw was just hired by OpenAI to "work on bringing agents to everyone." You'll have to forgive me if I'm not enthusastic about that.

jeffgeerling.com

You're doing great, and you need to hear that :)

Kevin Alemán — Wed, 15 Sep 2021 16:09:03 +0000

I'm not the kind of person that likes to be "thanked" for something. I like to do the things, and that's it. And later I realized it was not only me who felt this way. But, is receiving recognition something reserved for super special situations?

Everyone needs to be "praised" for doing a great work

Being praised releases dopamine, that's why you feel happy when someone recognizes your work and says that it's good. Everyone likes that feeling of "doing something amazing" by receiving a simple "thanks for helping me". But, it doesn't stop there.

Your motivation is also increased. It's important to note that everyone has a different set of motivators, so what may work for someone probably won't have the same effect for others. But there are some common motivators in a workplace, like the compensation. Receiving a considerable wage and a good compensation plan on your company may motivate you to work better. Same as having a clean workstation, or having a good team composed of capable people. Recognition is also a key motivator, that means, surprisingly, that people feel good when they're recognized :)

However, as any other dopamine-releaser, the effect of recognition is transient, and, it may decrese the effect over time when it's done too much. So, recognition should be given timely.

Timely recognition

So, how do we achieve a recognition scheme that's not sporadic but is not rare at the same time? Balance, as always, is the key.

This is more an HR topic, probably at your company there's a lot of people trying to fix this same issue, because recognition is not as easy as it may look. Fairness plays a really important role. What if another employee who does the same thing I do receives recognition every week, but I receive it every year? Then, recognition becomes a problem and a demotivator, so it's important to watch out how recognition will be conducted.

But, let's return to the frequency. From this BI Article, a weekly recognition scheme is good. Probably you don't need it to be too frequent. Think for example, on sprints. If your sprints are 2 weeks long, why not dedicating some time at the end to thank your colleagues for their work?

The same article says that this type of recognition helps the employee to be engaged, and trying to get that recognition again, because people like that. Try to avoid recognizing your employees just with the normal performance review cycle, because that may be too late.

Again, if your colleague did something really good for you or the team, don't wait to say thanks. Recognition has a "shelf life", that means, it can expire if not used at the right moment!

So, key points for timely recognition:

Do it before it expires :)
Don't wait to normal performance review cycles to recognize people
Be fair

How to create a recognition culture?

Make it easy
Your team should feel that it's easy to give recognition. If people find a lot of requirements or an extremely formal process for doing so, they won't do it.
Be specific
Saying "thanks" is good, but it's even better when you know the reason for that. So saying "thanks for helping me with the last presentation" would be much more valuable than just saying thanks, because that means you're recognizing a specific effort, and not just saying a canned response.
Connect the dots :)
Relate the recognition with your company's values. This makes easier for your team to know how they're contributing to the company's objectives and culture, and that creates a strong culture :)
Make it for everyone
Avoid having only managers giving recognition. Foster p2p recognition, since that'll create bounds between your team.

(Key points taken from this article

So, as conclusion, thanks for what you're doing to improve the community :) and thanks for sharing your knowledge with the rest of the people using this platform.

Now, go there and recognize people for what they're doing :)

The beauty of Streams in Node

Kevin Alemán — Fri, 02 Oct 2020 21:06:17 +0000

Probably you have heard of them, probably you don't. But they have been around for a while. We're talking about streams, an interesting and often ignored functionality of Node.JS.

What are streams?

To make things easier, we will define a stream as a sequence of data that flows freely. Think of streams like rivers. Rivers flow from one point to another in a constant way. The receiver doesn't know when the river will stop flowing but it's always there for receiving more water.

In Node, streams are very similar. They are a constant sequence of data. They flow from one point (emitter) to another (receiver).

The receiver can decide if it wants to receive data or not. Also it decides what to do with the data received. It can ignore the data, pipe it to another receiver, parse it before receiving...

Why are streams useful?

This is a really good question. If you have lived without streams your whole life, you will think that you don't need them. And, depending on your use case, this could be true.

But the reality is that we need to use streams for some operations that otherwise will kill us. Let's put an example

Dealing with files

Let's say we want to create a compressed copy of a file on our disk with a Node application. Usually, we will end up with something like this:

fs.readFile(file, (err, buffer) => {
  zlib.gzip(buffer, (err, buffer) => {
    fs.writeFile(file + '.gz', buffer, err => {
      console.log('Done!!');
    });
  });
});

Imports removed for brevity

Here, we're doing 4 things:

We're reading the entire file and saving the data to a variable called buffer
We're using the zlib.gzip to compress the file into a gzip. We're passing the buffer to the function call. This will return a new compressed buffer
We write the compressed buffer to a new location
A console.log indicating that the operation finished

Well, this works. What's the problem with this approach? You may be wondering. Well, look again at the first 2 steps. In this approach, we're reading the whole file before starting to process it. Additionally, we're storing the contents of that file in memory. This is not a problem if the file size is in MB. But, if the file size is in the order of GB? Hundreds of GB? Will your computer have all that RAM available to hold the file in it? Probably no.

So, this approach, even when enough for more simple tasks, represents a problem when we're looking for performance and scalability, or simply we want to support larger files.

The problem can be solved by using streams. Let's see how:

fs.createReadStream(file)
  .pipe(zlib.createGzip())
  .pipe(fs.createWriteStream(file + '.gz'))
  .on('finish', () => console.log('wooh!'));

We're doing some things different here:

We're creating a read stream. This will return us chunks of the file until it reaches the end.
We're piping the chunks to zlib for compression. What's important here is that we are not waiting for the whole file to be read before starting the compression.
We're creating a write stream in which we're passing chunks of data so Node can write them to the file.
After all, we listen for the finish event, which will be triggered when there's nothing more to do.

There's a vast set of details & quirks & functions related to streams which will be covered in other blog posts.

Hope you liked it!

Building a new feature: talks preview

Kevin Alemán — Fri, 11 Sep 2020 06:59:14 +0000

I'm a backend developer, so most of the time I'm doing backend things. I feel stressed most of the time when I need to do something at the front.

Then, I started to build my own personal site. And it was great! It's a blog site located temporarily at Netlify where you can contact me, read what I wrote, share my content with a suggested tweet, and now you can see the talks I've given in my career.

The talks are mostly about basic topics, one of them is in progress yet, but I wanted to showcase that I made them. And this was when the trouble begins

How?

The first thing was to define what I wanted: I wanted to showcase my presentations. This was easy (in my mind), but, it wasn't easy in real life.

To give you more context, I store all my presentations in Google Drive, so I can work on them wherever I go. This makes things easier for working on them, but a little bit hard if you want to show them offline.

Why? Because first, you have to download the file. This is a no-brain task with a UI, just right-click and download. GG. The task gets more complex when you want your code to perform that process. So I walked a lonely road, the only one that I have ever known: looking for someone who tried the same and succeeds.

I found multiple Gatsby plugins by just typing drive or google drive into the search box. I tried every single one, with the hope of finding what I wanted. Here's a list of the highlighted ones if you are trying to do something similar:

gatsby-source-google-docs

The problem: the use case I had in mind was very different from what the library did

gatsby-plugin-drive

The problem: it only allowed me to download DOC files (not presentations, which was what I wanted)

gatsby-source-drive

The problem: it didn't work. Maybe I misconfigured something or so.

gatsby-plugin-docs-sheets

The problem: it only allowed me to download Sheets. I'm pretty sure this was based on gatsby-plugin-drive)

So, since no plugin was able to fulfill my needs, I had to make a difficult decision:

Give up on my feature
Create my own plugin

I took the second, and this plugin was the result: Gatsby-plugin-googledrive

The entire plugin was based on a single idea: given a Google Drive's folderID, download all files from the folder, and traverse the subfolders recursively to create the same structure on your selected destination. And download the files for each folder.

The technical details can be found at the GitHub of the project and, it's open-source, so if you want to build something around it feel free to do so.

How? Part 2

After creating my plugin, I "plugged" it and the download started. I finally had the files on my local, and this was one step closer to my goal.

I had the PDFs on my local, now, I wanted to do 3 things:

Get the URL from the PDF in my filesystem (and application!)
Get the first page of the PDF (the "cover page")
Convert that page to an image, and show it on a nice grid

Soo, I needed 3 things:

One way to get the PDF from my filesystem
One way to read the PDF
One way to convert the read PDF into images

For the first item, since I'm using Gatsby, I used the gatsby-source-filesystem to read my folder and get the actual asset URL pointing to my file. It was really easy, just added this to my gatsby-config.js file:

    {
      resolve: "gatsby-source-filesystem",
      options: {
        name: "talks",
        path: "${__dirname}/src/talks/",
      },
    }

And that's it! Then, to query my filesystem and get the nodes with data, some graphql was needed:

  query {
    allFile (filter: {sourceInstanceName: { eq: "talks" } }) {
      edges {
        node {
          name
          publicURL
        }
      }
    }
  }

That query will do 2 things:

Get all the files under the path of the instance name called talks (which is the name property you pass to the plugin definition)
For each node aka file, get its name and publicURL (which is the URL that the asset will have after the building)

For the second item, I had to do some research. After some looking, I finally found PDF.JS which is like the standard for managing PDFs in JavaScript. It was written by Mozilla and has a ton of useful functions. I just used the basic API to get my feature working in a few LoC.

PDFJS.getDocument(talk.publicURL).promise.then((pdf) => {
      pdf.getPage(1).then((page) => {
          // some code goes here
      })
})

The introduction to the library in its GitHub page was simple, maybe too small for my taste, but it worked. With this, the only thing left was to find a way to convert the PDF page's data into an actual image.

I found a pretty nice approach to accomplish that. It worked like this:

Read the PDF page
Scale the page's width and height (by using the viewport of the page)
Create a <canvas> element
Set the canvas context to 2d (since we'll draw an image)
Set the scaled page as the new data context for the canvas

The full code was linked in this StackOverflow's answer which even had an example of how it worked. Nice.

With all of that in place, the final result:

Hope you liked the article, feel free to also visit this post on my own website

Algos in real life: binary search

Kevin Alemán — Fri, 04 Sep 2020 20:14:12 +0000

You probably know about algorithms. You probably don't. So here's a basic definition: an algorithm is a chain of steps for moving from A to B.

If you want to make a sandwich, you probably have an algorithm for that. Mine is:

Get the ingredients
Separate bread and put it on a plate
Put some dressing on both halves
Put lettuce, tomatoes, pickles (if any) on one half
Put ham, cheese, ham again on one half
Add extra things (this is, probably whatever I can find that fits my sandwich)
Put the empty half over the other ingredients
Eat

Whatever your process is, the point is you use an algorithm for that.

Algos are really useful, in programming and real life. Here, I'll show you an example of how I used one of those algos to make my life (and work) easier.

The task

Suppose that you were assigned to solve an issue at work. Depending on the kind of issue this may be easy peasy lemon squeezy or you can end stressed depressed lemon zest.
The issue in this example is a simple one: a function was working at some time in the past, but now it's not. Easy.

You fix the function, upload your changes, and create a merge request. Everything fine at this point.

Then, someone from above asks the hard question: "Hey, this worked in the past, could you tell me when it stopped working?". If like mine, your project has a ton of commits, then you'll have fun, more if your commit history looks like this:

The solution

You have 3 options here:

Cry in a corner
Traverse all your commits until you find one where it was working
Do something good with the knowledge you have

Ok, after doing the first we can start thinking on a real solution. And here's where algorithms come handy. Let's remember one of them!

The binary search algo, or the principle of divide and conquer.

You may remember that algorithm from a class, a course, some reading, or any other place. In case you haven't, I'll leave a brief explanation here:

The main idea of this algorithm is to, given a sorted list and a value to find as inputs, traverse the list to find the value you're looking for.

The main point of difference between this and other search algos is how the list is traversed.

In traditional searching, you go from the start until the end (in the worst case) to find what you're looking for. If you reach the end and you haven't found the value, then it's not there. Something like this:

In a binary search, however, you "divide" the list in halves, then compare the middle value with what you're looking for. If the value is greater, then you repeat the process in the left half, otherwise you use the right half. You repeat this until you cannot divide your input list anymore. Something like this:

To use a binary search, you just have to follow some rules:

Your list must be sorted
Your list must be sorted
Your list must be sorted

Ok. You got this, the list must be sorted. Let's continue with the solution!

The solution (part 2)

Since our commit history is sorted (by the date of the commit) we can perform a binary search on the commit history!

I selected some old commit (like 20 days before) as my start point and my current commit as the last item.
I, then, selected the middle one (this was made visually) and checkouted to that commit.

Was functionality working? Yes? Select a more recent one! Else, use an older one.

You know how the story ends. I was able to verify the whole commit history in like 20 minutes, when it may have taken 2 hours due to the number of commits in the pool.

Moral: if you know how to do something fast, do it! And, algos are also useful in real life!

Hope you liked this history, and let me know if it helped you! Thanks for reading 🎉

Developing JS/TS with VIM

Kevin Alemán — Thu, 18 Jun 2020 06:48:58 +0000

In the beginning, there were editors. And VIM was an editor. And VIM was the editor.

I started to code 3 years ago. When I started to code, I remember having one coworker that was using Emacs. He just stared at its screen without using the mouse, and all he was doing was using the keyboard to do what he needed. He tried to convince me to use Emacs as well, but something didn't feel so good when using it.

I, then, started to use VSCode. The good VSCode. I don't have anything to say against him. VSCode is an excellent editor. The only thing that stressed me out was its insane memory usage! After all, it's built in the top of Chromium, so it will use the RAM accordingly. VSCode is a perfect editor for almost anything you want to do. Do you want to write Java? There's a plugin. Do you want to test your API? There's a plugin. Do you want to manage your database? There. Is. A. Plugin. It's not an IDE because they haven't given the title to it!

Even when that rich ecosystem is something beautiful by itself, it can led to some issues. The memory usage may go insane as you install more extensions. And disabling them based on the workspace (which you have to configure manually) instead of filetype stressed me more.

So, there was a time, when I was working in a project that used a lot of docker containers, they used a lot of RAM by themselves, and I was using VSCode too. The RAM cake was starting to finish. Also, I had Chrome and another applications that I use to do my work (like Slack, which is also on top of Chromium) trying to get a piece of cake too... My computer didn't like that, so it crashed. This happened consistently for one month. I tried to change browser, to use Slack in my phone, to do some kind of black magic to get the most of my PC, even I tried to download more RAM (I really expected this one to work... :( ). But nothing happened.

Then, the illumination came to me.

I took a course, and one of the instructors was using VIM, but it was like an IDE. It was perfect. He moved so quickly between tabs (buffers) and commands were intuitive. The color scheme was great, the speed was really impressive. He had autocompletion and listing... everything was perfect.

Motivated by this, I went to my laptop and installed VIM. But...

IT WAS NOT LIKE THE VIDEO. WHY? I didn't have the autocomplete, neither the status bar nor the file explorer... it was just an empty console. Like Emacs was. And also I wasn't able to get out of it.

I realized that Vim was not pretty out-of-the-box. You have to work on it before using it to write code. So I spent a few days reading VIM tutorials and contents. I looked for online resources, and found these:

After finishing 2 of those (obviously the first 2...) I was ready to start to use VIM.

The shorcuts

This is a list of the basic shortcuts I learnt before using VIM. These ones allowed me to at least know how to use it.

To exit vim: :q
To enter in edit mode: i (yeah, just type i and you will get into edit mode)
To exit edit mode (and all other modes): esc
To "save a file": :w
To close without "saving": :q!
To split "tabs" vertically: Ctrl + wv
To split "tabs" horizontally: Ctrl + ws
To change between "tabs": Ctrl + ww

The plugins

First of all, like every person looking for knowledge, I typed How to add plugins to vim in Google. I followed some links and one of them took me to Pathogen, which is a package manager that uses Git (just Git) to get the plugins. I also read about VimPlug, but Pathogen was my choice.

Why? Well, I liked the way it should be enabled:

execute pathogen#infect()
syntax on
filetype plugin indent on

Do you see? Pathogen "infects" VIM. Isn't it nice? Well, I probably lol'ed so hard when reading that, more than I should. That convinced me to use it instead of VimPlug.

Later, I discovered VIM 8 had native plugin support...

The plugin list

Now that I had a plugin manager, I needed plugins. Where did I find them? I used Vim Awesome to get a list of most used plugins and filter the ones that were related to JS/TS/Go. Then, I installed them. Here's the list of the modules I have downloaded for my current development:

(Images taken from Vimawesome)

The NERDTree (to interact with the filesystem)
Vim Airline (to get nice look and feel)
Vim Fugitive (this plugin is so awesome that it should be illegal)
Vim Polyglot (for really good syntax highlight)
ALE (An asynchronous linting engine. This is very useful to configure ESLint and Prettier in Vim)
COC.vim (the ultimate code completion plugin)

These aren't all the plugins I use. I just wrote there the most important ones. The ones without its help I wouldn't be able to write code in VIM.

The final product, using the plugins and one theme that I found (it's called onehalfdark) was this:

Hope you liked my intro to the Vim world, and maybe this motivates you to start getting into Vim!

Fuzzy Searching with PostgreSQL

Kevin Alemán — Mon, 11 May 2020 17:12:11 +0000

What's fuzzy search?

A fuzzy search is a type of search where the items are returned even when they are not an exact match. A fuzzy search is performed by a fuzzy algorithm that evaluates the likeliness between the search query and the values, even when the search query is misspelled or the order of the words changed.

For example, let's assume that we're using Google to find information about washin machines. As you can see, the word washing is misspelled. But even when Google knows that it'll return results that are somewhat similar to what you typed. Sometimes it even tries to correct you!

Well, that's a fuzzy search. Returning elements that are just similar one to another.

How's searching occasionally implemented?

Let's take a typical requirement: allow a user to search other users by its name. To accomplish that, you may end up building a query like this one:

SELECT * FROM users WHERE full_name ilike '%query_param%';

Assuming you have a user named Jonh Doe in your database, looking for Jonh, JonH, jOnh, jonh doe, Jonh Doe and so on will return the value you're looking for. But, what if the user types Doe Jonh?

Well, for sure your app won't crash with that input. But I'm pretty sure it won't return anything! Why? Because the order of the words is different than what's stored on the database, and ilike won't do that rearrange of the words to match. One way to overcome this is to query also for the words in a different order, but that's not elegant.

Apart from that, what happens if now the user (yes, always the user, ugh) types john doe? There are some ways to handle this with ilikes but, why doing that amount of work when you can let PostgreSQL do that for you?

Let's introduce pg_trgm. A PostgreSQL extension based on something called a trigram that will be very useful for this! But, first of all, what's a trigram?

PostgreSQL trigrams. What are they?

The basic definition of trigram is: "A group of 3 consecutive written units, such as words, letters, etc". The pg_trgm extension allows us to divide a word in such.

How does a trigram look like?

To see how a trigram look like, let's enable the PostgreSQL extension first!

CREATE EXTENSION pg_trgm;

After that, run:

postgres=# select show_trgm('abcdefghijk');
                        show_trgm                        
---------------------------------------------------------
 {"  a"," ab",abc,bcd,cde,def,efg,fgh,ghi,hij,ijk,"jk "}
(1 row)

As you see, it'll divide the word into groups of 3 characters. Why there are spaces in the first and last trigrams? Well, that's because PostgreSQL assumes that every word to be converted into a trigram has 2 spaces before and one space after. Why? This allows PostgreSQL to create trigrams for words with less than 3 characters (we, for example). Also, this makes the similarity to take into account the first and last characters, which otherwise would have less weight when the trigrams are generated.

How to use them to find similar words?

If we select the trigrams for 2 words, we should be able to find how similar those strings are. How? By counting the number of trigrams they share!

postgres=# select show_trgm('Friend');
              show_trgm              
-------------------------------------
 {"  f"," fr",end,fri,ien,"nd ",rie}
(1 row)

postgres=# select show_trgm('Friendship');
                      show_trgm                      
-----------------------------------------------------
 {"  f"," fr",dsh,end,fri,hip,ien,"ip ",nds,rie,shi}11
(1 row)

How we find those similar trigrams between the 2 sets? Correct, an intersection!

postgres=# select a                                                 
from (select unnest(show_trgm('friends')) as a) as a
intersect select b
from ( select unnest(show_trgm('friendship')) as b) as a;
  a  
-----
 rie
 fri
  fr
   f
 end
 ien
 nds
(7 rows)

As you can see, there are 7 trigrams common between the 2 sets. Now, how do we know how similar they are? We can do that by calculating a similarity coefficient between the 2 words. How?

word_similarity = common_trigrams / first_word_total_trigrams

Why we use the first word? Well, because we're comparing the first against the second word. We don't care about the excedent trigrams from the second word because we know they won't correlate to the first one.

Now, let's apply the formula. We have 7.00 common trigrams, and we have 8.00 total trigrams in the first word. Let's use PostgreSQL to determine the coefficient:

postgres=# select 7.00 / 8.00;
        ?column?        
------------------------
 0.87500000000000000000
(1 row)

Here, we use decimals because otherwise, PostgreSQL will ignore the decimal part of the operation, thus returning us 0.

Now, we know that our words have a similarity of 0.875. That means they are very similar! But, having to do all of this manually may take time... there should be a better way.

Well, there's a better way! Alongside the trigram extension, we enabled a set of functions that came in the same packages. For example, the whole process we've done for calculating the similarity is abstracted away into a single function: word_similarity. By using the function, we can get all of this process into a single line:

postgres=# select word_similarity('friends', 'friendship');
 word_similarity 
-----------------
           0.875
(1 row)

And even better, there's a set of useful operators that are built into the same package we're using! For example, we may let PostgreSQL decide if two words are similar or not by using the <% operator:

postgres=# select 'friends' <% 'friendship';
 ?column? 
----------
 t
(1 row)

That operator returns if 2 words are similar or not by using a word_similarity_threshold, which we can query too:

postgres=# select show_limit();
 show_limit 
------------
        0.3
(1 row)

Note that show_limit() function is deprecated, but still works.

That limit is, the minimum coefficient a pair of words can have before marking them as different. You can modify that threshold using the set_limit() function and provide new value, but for most use cases, the default one should work!

Using trigrams, we can also query for the "distance" between 2 words, which is calculated by taking the similarity of the words from 1.

distance = 1 - similarity(word_1, word_2);

With the <-> operator. The lower the number the more similar (or less distance) they have.

postgres=# select 'friends' <-> 'friendship';
 ?column? 
----------
 0.416667
(1 row)

We can use this operator to build something like Google's did you mean functionality, which is probably done with n-grams.

Building a `Did you mean` engine

First, let's create a table and populate it with data. For this example, I'll be using the departments of my country, El Salvador.

postgres=# create table t_departments(name text);
CREATE TABLE

And now, some data:

postgres=# insert into t_departments(name) 
values ('San Salvador'), ('Chalatenango'), ('Ahuachapan'), ('Cabanas'), ('Sonsonate'), ('La Libertad'), ('Santa Ana'), ('La Union'), ('Cuscatlan'), ('San Vicente'), ('La Paz'), ('Usulutan'), ('Morazan'), ('San Miguel');
INSERT 0 14

postgres=# select * from t_departments;
     name     
--------------
 San Salvador
 Chalatenango
 Ahuachapan
 Cabanas
 Sonsonate
 La Libertad
 Santa Ana
 La Union
 Cuscatlan
 San Vicente
 La Paz
 Usulutan
 Morazan
 San Miguel
(14 rows)

Now, what happens if the user is looking for San Salvador but it misspells it and write sn salvador? Our system should recommend the user to write it properly! How? Let's see:

postgres=# select * from t_departments order by 'sn salvador' <-> name limit 3;
     name     
--------------
 San Salvador
 Santa Ana
 San Miguel
(3 rows)

And 🎉. By returning the first result, we can tell the user that we know what he was trying to write, and still, return results related to its search, as Google does!

Solving our initial problem

Well, our initial problem was the user, right? Jk, our initial problem was to handle misspellings when users type a name, but they still want a result.
It's now trivial to do this with our new knowledge, by querying either the word similarity or the distance between the words.

postgres=# select * from ( select unnest(array['jonh doe', 'jane doe', 'mark twain', 'some user', 'a person', 'tommy shelby']) as names) as a order by 'doe jonh' <-> a.names limit 3;
   names    
------------
 jonh doe
 jane doe
 mark twain
(3 rows)

Ignore Mark Twain...

I hope you liked the article, and if you are curious and want to delve more, you can read the whole documentation for trigrams and all the operators you can. In my next blog post, I'll share how to index this kind of data so the queries are faster!

Getting query performance stats with PostgreSQL

Kevin Alemán — Wed, 29 Apr 2020 16:11:51 +0000

In this article, you will see how to use some hidden features of PostgreSQL to get useful insight into your running queries in PostgreSQL.

The problem

Have you tried to spot performance problems in your application? Maybe some of them live in the code (a map over thousands of elements maybe...) or maybe the performance issue is caused by another factor: badly defined SQL queries.

As a developer, you will have to deal with SQL someday, maybe sooner than later. And probably you will have to work with queries that other people made, or even with queries that the you from the past created.

The problem is that, without the right tool and the right information, is very difficult to identify a slow query. Why?

Some queries are slower with more data

For example, imagine a simple query that joins multiple tables. In your local, with probably 10 users the query won't perform bad (and if it is, it is easier to spot it!).

Some queries needs an index

Indexing is probably the root cause of performance issues. The lack of them, as well as the presence of them, can cause problems. With a small set of data, you won't be able to see if a query needs an index or not. Even worse (or better, it depends) PostgreSQL can leave the index out of the query if the dataset is small enough to do a sequential scan (this is, row by row).

Without having the issue happening in a production environment, is very difficult to spot issues like this one, and there's a big possibility of the end-user spotting them before you.

This approach (waiting for the user to tell you the app is slow) is very reactive. You have to wait until the problem happens before working on a fix for the issue. But, what about being able to have this information before the issue happens?

For that scenario is the reason why some PostgreSQL views exist. Those maintenance views are a gold mine for developers that want to track the performance of their queries. Let's talk more about them!

The solution: PostgreSQL maintenance views

PostgreSQL has a lot of views for this purpose. Some of them give us stats about Disk I/O and network statistics. Others allow us to see replication stats and things like that. Here, we'll talk about 3 views that can help you nail down query issues: pg_stat_user_tables, pg_stat_user_indexes and pg_stat_statements.

`pg_stat_user_tables`

This view shows you statistics about each table per schema (there's one row per table) and gives you information like the number of sequential scans that PG has performed in the table, how much select/insert operations are done in it and so on

As you can see here, for the first row, it has performed 1 sequential scan and that scan returned 939 rows. There were 2 index scans and they returned 2 rows. Numbers are low because I'm using a local database, but these numbers should be larger in a production database.

From this view, apart from all the useful information, we can answer something really interesting: which of my tables need an index? You can easily answer this question by querying the seq_scan and seq_tup_read columns!

select 
  schemaname,
  relname,
  seq_scan,
  seq_tup_read,
  seq_tup_read / seq_scan as avg,
  idx_scan
from pg_stat_user_tables
where seq_scan > 0
order by seq_tup_read desc limit 25;

Running that query will return us the following

As you can see, it is suggesting me to add an index to those tables since they were used recently in sequential scans. With more data and more runtime, this query will give you a good insight on how your tables are behaving.

`pg_stat_user_indexes`

Even when adding indexes solves many problems, they are not the holy grail, and they come with a cost: space. Indexes are good, yes, we all concord in that. But something worse than not having an index is to have a useless one. Why? Well, first of all, it will take space from your DB server. Indexes on big tables can be very expensive, and get very very big. The second reason is that the index should be recalculated each time you write to the table. And of course, recalculating a useless index is like paying for food you won't eat!

So, every time you add an index, make sure it makes sense.

But, what happens if you work on a codebase & DB schema that you didn't design? Is this the world's end? Absolutely! PostgreSQL views to the rescue again! The pg_stat_user_indexes table is capable of showing you the frequency of use of your indexes, alongside with the space they are taking.

As you can see from the image above, some of my primary keys weren't used yet. But, this doesn't give us too much detail yet. Because we don't know what's the space in disk our index is using! We can get that information by using the pg_relation_size function with the indexrelid from our results.

select schemaname, relname, indexrelname, idx_scan, 
pg_size_pretty(pg_relation_size(indexrelid)) as idx_size,
pg_size_pretty(sum(pg_relation_size(indexrelid))
over (order by idx_scan, indexrelid)) as total
from pg_stat_user_indexes
order by 6;

The result of this query shows you indexes that hadn't been used in a while alongside with its space consumption. This can give you an idea of which indexes to take care of.

As a note, the result of this query doesn't mean that you should drop all of those indexes that hadn't been used. You should always investigate why the index isn't being used before deleting it!

`pg_stat_statements`

This is probably the most useful one. It's hard for me to understand why this view isn't enabled by default! This view should be enabled in the PostgreSQL conf to be used by you.

Activate

To enable this view, we should include it into the shared_preload_libraries list. Since I'm using Docker & Docker Compose to manage my database, I can easily add an option to the start command to be like this:

  postgres:
    container_name: postgres
    image: postgres:10
    restart: always
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_PASSWORD=${PG_PASSWORD:-postgres}
      - PGDATA='/var/lib/postgresql/data'
    command:
      - "postgres"
      - "-c"
      - "shared_preload_libraries=pg_stat_statements"

After this, when you start PostgreSQL again the library will be loaded with the DBMS

Create extension

After enabling the library, you should enable it as an extension. You can do this by running the following query

create extension pg_stat_statements;

If that query doesn't give you an error, you are good go! Let's confirm this by running:

select * from pg_stat_statements;

From this view, we can get very good information about how our queries perform. For example, we have the number of calls a given query had. The mean_time of execution between all the calls and even the stddev_time (Standard Deviation) of the calls, to see if the queries have a consistent runtime or they vary and how much they do.

From this view, you can even know how many rows a query returned if those rows came from the cache or from disk and so on!

With all of this information, it's easy to get a list of the most expensive queries and get to know why

select round(( 100 * total_time / sum(total_time) over ())::numeric, 2) percent,
             round(total_time::numeric, 2) as total,
             calls,
             round(mean_time::numeric, 2) as mean,
             stddev_time,
             substring(query, 1, 40) as query
from pg_stat_statements
order by total_time DESC
limit 10;

With that query, you now have a list of the 10 most expensive queries, how much time they used, how many times they have been called and the deviation from the mean-time those queries have.

Using this, you can track what queries are the ones that take more time and try to fix them (or understand at least why they perform like that).

Conclusion

Using PostgreSQL to monitor PostgreSQL is very useful and can point you in the right place to understand the performance of your application and the pain points it has.

I hope you liked the article and learned something from it!

Note: This article was also published in my blog. I'm still trying to find a good domain name for it lol

How-to: Handle schedules with PostgreSQL

Kevin Alemán — Wed, 01 Apr 2020 22:32:48 +0000

How many times you tried to build a scheduling system and found too complex to arrange the dates in a way there are no collisions? Nobody? Only me?
Well, in this article you'll learn an easy way to handle scheduling by using some of the hidden features of PostgreSQL!

In case you don't know, PostgreSQL is a fully-featured and open source DB Engine written mostly in C. PostgreSQL contains a ton of features that can make our lives easier, I'll try to cover some of them in other articles.

In this article, we'll talk about 3 of those powerful features: GiST indexes, Exclusion operators & ts-ranges.

Let's talk about them a little to have context:

GiST indexes

A GiST index, or a Generalized Search Tree is a very important type of index. They are a "balanced, tree-structured access method that acts as a base template in which to implement arbitrary indexing schemes" according to the Postgres documentation about them. They are like the "b-trees" that you may have heard of. The difference between them is the operations they support. For example, a "b-tree" supports the standard comparisons between elements in the tree (<,>,=), but, for some types of data (geospatial, text, images) this kind of comparisons doesn't make sense!
This is when GiST saves the day. They allow us to customize the way the data is stored & balanced in the tree so we can perform another kind of operation over the indexed data, for example, calculate overlapping between the data, intersections, containment, etc.

However, they are not enabled by default (at least in my compilation) so, you have to enable them by "creating an extension". Create an extension is basically install it from a predefined source. You can create your own extensions but for this example, we'll be using a predefined one:

CREATE EXTENSION btree_gist;

You can delve into how they work in this blog post that explains the topic really well!

TS-Ranges

A ts-range or Timestamp Range is a kind of Range datatype in Postgres. This datatype will create a "range" (surprise!) between the 2 dates passed.

SELECT tsrange('2020-01-01', '2020-01-02');

                    tsrange                    
-----------------------------------------------
 ["2020-01-01 00:00:00","2020-01-02 00:00:00")
(1 row)

If curious about what the [ ) on the range means, you can look at Postgres' official documentation about ranges.

tsranges doesn't have a timezone by default. If you want to include the timezone in your ranges you can use the tszrange datatype.

Exclusion operators

Indexes are widely used to speed things up and for uniqueness. However, do you know that you can use indexes to define additional constraints? For example, scheduling constraints. As I mentioned earlier, you can determine if 2 pieces of data overlap using GiST indexes. What we'll be doing is to use that in our favor!
Exclusion operators are easy to define, you just state you want to use them:

CREATE TABLE reservations (
  room_id int,
  from_to tsrange,
  EXCLUDE USING GiST (room_id with =, from_to with &&)
);

Basically, what we're saying is: "hey Postgres, create an exclusion rule for this table. And, when adding new elements use the = operator to compare room_ids and use && (overlap) operator to compare the from_to tsrange."

Now that we have the table created, It's time to test!!

Create the test table we're going to use (in case you didn't)

CREATE TABLE reservations (
  room_id int,
  from_to tsrange,
  EXCLUDE USING GiST (room_id with =, from_to with &&)
);

Let's book some rooms!

INSERT INTO reservations VALUES (1, '["2020-01-01", "2020-01-10"]');
INSERT INTO reservations VALUES (2, '["2020-01-01", "2020-01-15"]');
INSERT INTO reservations VALUES (3, '["2020-01-06", "2020-01-12"]');

Test our exclusion rule by inserting a reservation on a scheduled period. Visually, this is what we're trying to do when inserting the new schedule: As you can see, the period in red overlaps with the current booked period, so it should fail. Let's try it!

INSERT INTO reservations VALUES (1, '["2020-01-06", "2020-01-15"]');

And what happens when we try to insert that scheduling?

ERROR:  conflicting key value violates exclusion constraint "reservations_room_id_from_to_excl"
DETAIL:  Key (room_id, from_to)=(1, ["2020-01-06 00:00:00","2020-01-15 00:00:00"]) conflicts with existing key (room_id, from_to)=(1, ["2020-01-01 00:00:00","2020-01-10 00:00:00"]).

That means our implementation is correct! 🎉
Since we're using timestamps, you can be even more specific by specifying minutes, seconds and so on and Postgres will also take care of them!

I hope you learned something new! And, if you already knew about those features but didn't know you could use them for this, there you have it!

Hope you liked the post, bye!

Disclaimer: this may not work for all the scenarios.

A tale of Candy Crush & Technical debt

Kevin Alemán — Mon, 30 Mar 2020 03:50:17 +0000

One day, when I was playing Candy Crush (Soda Saga, to be specific) I came across a very interesting but difficult level. Personally, I hate all the Chocolate-related levels on the game, but, well, a game is a game.
In case you had never played Candy Crush before, this is what I'm talking about:

The objective in those levels is to destroy all the chocolate on the screen with the movements you have. This is not so difficult until you discover that when in a move you don't destroy any chocolate block, they multiply leaving you with less space to move, converting a peaceful level in a kind of race where you have to keep destroying or end up flooded in chocolate.

This situation made me think about a real-life scenario: Agile projects & technical Debt. So, how does technical debt assimilates to one of those levels? Let's discover!

Chocolate multiplies as well as Technical Debt

First of all, let's state something: it's almost impossible to have an agile project without having technical debt at some point.
Having said that, let's think about chocolate. You start a level with a predefined amount of chocolate. If you don't do anything to clear them quickly, they will increase in number to the point where all on your screen is chocolate.
The same happens with tech debt. If your project doesn't have a plan to reduce the amount of TD, your entire project will convert into it and this will leave you in a point where it is easier to rebuild everything from scratch than fix the existing code base.

If you don't have a strategy, you will lose.

That kind of levels have something in common: you should think your movements before doing them.
When debating about tech debt the same happens: You should know what you're doing. For example, if someone tells you that to reduce tech debt you should refactor every piece of code, yes, that could reduce tech debt, but at the same time, you are removing battle-tested code with something that probably won't work for all cases the old code did, introducing new tech debt & bugs. On the other hand, if the tech debt is not in the code per se, but on the way the project is architectured, you can end with more troubles than solutions by blindly changing current architecture. So, you should have a plan to fix tech debt, or you'll end up in a worse state.

Good strategies to reduce the Tech debt in a project are described here in this DZone's blog post

There are pieces of chocolate harder than others

As in real life, there are pieces of code, architecture decisions, product deadlines that are very difficult (or nearly impossible) to change! Let's write a dummy piece of code that will represent a tech debt:

async function evaluateUser(user) {
  if(user.act && !user.isNew && !user.company.name === 'fancy company') { 
     /* do something really difficult here */
  }
}

Maybe there was a project deadline that the team had to meet so they decided to hard-code the company's name. And this works! Until the time the client says: I want users from the company"bla"to be excluded too, but only new users. As easy as this change may look, the team should be careful not to affect how the users from "fancy company" are evaluated and, if possible, make changes to this piece of code easier in the future (pay the tech debt). This situation makes changes to take longer and to be more error-prone.

Chocolate is not always bad

When talking about tech debt, some people tend to touch it with a stick from the distance, avoiding all contact like if tech debt was the plague. But, that's not realistic. We all have tech debt. In this topic, I really like Squarespace's blog post regarding good types of tech debt. In summary, tech debt is good when you know it is there. When it's intentional. Because that kind of tech debt can be paid, the one you are aware of. You can't pay what you don't know you owe!!
So, evergreen advice for dealing with this is to always document what your tech debt is, keep that document updated & involve your management team in this!

In conclusion, if I was able to pass that level, you can pay your technical debt!!

DEV Community: Kevin Alemán

Got featured :noice:

Top 7 Featured DEV Posts of the Week

🙌🏽

In the AI Era, Code Is Cheap. Reputation Isn’t.

In the AI Era, Code Is Cheap. Reputation Isn’t.

Death by a thousand slops | daniel.haxx.se

1. Think. Then think again

2. Respect volume. Review is not free.

3. Quality that doesn't cut corners (pls don't sue me Wendy's)

4. Bundle trivial changes.

LLMs are fun, but human work is better.

AI is destroying Open Source, and it's not even good yet - Jeff Geerling

You're doing great, and you need to hear that :)

Everyone needs to be "praised" for doing a great work

Timely recognition

How to create a recognition culture?

The beauty of Streams in Node

What are streams?

Why are streams useful?

Dealing with files

Building a new feature: talks preview

How?

How? Part 2

Algos in real life: binary search

The task

The solution

The binary search algo, or the principle of divide and conquer.

The solution (part 2)

Developing JS/TS with VIM

The shorcuts

The plugins

The plugin list

Fuzzy Searching with PostgreSQL

What's fuzzy search?

How's searching occasionally implemented?

PostgreSQL trigrams. What are they?

How does a trigram look like?

How to use them to find similar words?

Building a Did you mean engine

Solving our initial problem

Getting query performance stats with PostgreSQL

The problem

Some queries are slower with more data

Some queries needs an index

The solution: PostgreSQL maintenance views

pg_stat_user_tables

pg_stat_user_indexes

pg_stat_statements

Activate

Create extension

Conclusion

How-to: Handle schedules with PostgreSQL

GiST indexes

TS-Ranges

Exclusion operators

A tale of Candy Crush & Technical debt

Chocolate multiplies as well as Technical Debt

If you don't have a strategy, you will lose.

There are pieces of chocolate harder than others

Chocolate is not always bad

Building a `Did you mean` engine

`pg_stat_user_tables`

`pg_stat_user_indexes`

`pg_stat_statements`