DEV Community: Dmitry Kudryavtsev

Why engineers should focus on writing

Dmitry Kudryavtsev — Mon, 24 Jul 2023 22:27:16 +0000

This essay was originally published in my blog - yield code(); I'm a big believer in owning your content, instead of publishing it to corporations who might charge their readers for it. Please consider subscribing to my blog. Not all the articles I post there, will be reshared on other platforms.

All engineers are good writers… of code. But I believe that in order to a become better engineer–you should improve your writing skills.

From the dawn of times, people were writing. We have written using symbols, like in Ancient Egypt. And we have written using letters, like in Renaissance times. And all of us, got at least one writing assignment in school, without the “Why?” And yet, today writing is so underrated, that most people want to avoid it. But the truth is–you will have to write. Comments, documentation, design documents, presentations. Whether you like it or not. So why not become better at it?

Why write?

Writing is a way to organize your brain

For some reason we, humans, think that knowledge equals reading stuff. I’ve read the starting guide for Rust, so now I know this topic. I’ve read a book about how TCP/IP works–so now I know this stuff. But it’s not true. If it was true, we’d all be super-stars.

What helps us cement our knowledge–is writing. That’s why I believe in writing code, as opposed to copying code snippets. Because when you type it–you cement that knowledge.

Writing is a way to learn something

If you want to learn a new topic–write about it. When I want to learn a new programming language, I write a short program using that programming language. When I want to understand how something works–I write an article on this blog about it.

Real writing, where you try to dig the truth, will require you to try and experiment. I’ve learned more by writing articles on this blog, than by reading programming books.

Writing helps you identify mistakes

How many times we are asked to prepare a design document, while we think to ourselves, “What for? The design is so simple, I can hold it in my head easily.” This is a big misconception, we all fall for. If we would actually write the design document, we would have identified so many problems with our “simple” design. Things like–inconsistencies, missing details, or simply “lazy” thinking our brain did, which in reality make no sense.

How to write more?

Remember–reading is a habit, writing is a skill. And in order to prefect your skill, you need to write more. One simple way to write more–is to approach design reviews differently. Instead of hating them, and doing them like a homework assignment, try to approach them with enthusiasm. Each time you write a design review, try to improve something. Make it shorter–without missing the point. Make it longer–in order to cover more use-cases.

Writing design reviews, and documentations–at your workplace–is an easy way to get into writing. You will have to do it anyway, so why not improve while writing? However, if you want to improve your writing further, outside your workplace, consider starting a blog. If writing a blog is intimidating for you, consider answering questions on Stackoverflow–but focus on providing textual content, rather than copy-pasting code snippets.

One last advice–abolish the copy-paste. So many developers, whom I mentored, simply copy-paste everything. Code snippets, function declarations, etc. I know how to initialize a git repository, because I do it by hand every time. Most people simply copy the instructions from GitHub or Google. And if you are scared about being unproductive–remember that you are not judged by how much code you write, or how fast you complete the assignment.

Writing is unavoidable. Whether you like it or not, if you want to become a 10x engineer, you will have to write. Write documentations, write design reviews, write presentations. The sooner you get better at writing, the faster you will become a better engineer.

Cursor based pagination

Dmitry Kudryavtsev — Sat, 21 Jan 2023 13:56:20 +0000

This article was originally published in my Personal Blog

Traditional pagination existed for many years. But it’s not useful for platforms with huge amount of user generated content. Luckily, there is an alternative solution for such systems.

Traditional pagination

Before we jump into the solution, let’s look at traditional pagination and it’s drawbacks. Let’s say we have a database table for forum posts that looks like this:
| ID | Title | Content | LastActiveDate |
| --- | --------- | ------------ | ------------------- |
| 1 | Some post | Some content | 2022-11-26T14:00:00 |

In this case, LastActiveDate—indicates the date when the post was last active (i.e., a comment was published).

We want to display a front page with all the posts sorted by LastActiveDate. Since we don’t know how much data we can have, we decide on an arbitrary page size of 10. And then, in order to display the posts on a particular page, we will use the following query:

SELECT * FROM posts
ORDER BY LastActiveDate
LIMIT ${pageSize}
OFFSET ${(currentPage - 1) * pageSize}

The math is very simple. Given a specific page, say 2, we will select all items starting from offset (2 - 1) * 10 = 10, limiting the selection to 10 elements, and the result will be all the items in group of [10,20).

Usually, with traditional pagination, we also need to have another query in order to generate the visual pagination itself. And for that we need to know the total amount of pages we have, which can be done by:

const totalItems = await sql("SELECT count(ID) FROM posts") as number;
const totalPages = Math.ceil(totalItems / pageSize);

Drawbacks

This pagination style, however, has one major drawback. Let’s say I’ve navigated to page 1 of the forum and then decided to go make some tea. After coming back and seeing that no posts on the first page are of interest to me, I navigate to the second page and find out that it includes 2 posts from the first page.

This happens because during the time I was making and drinking my tea, some older posts that were not among the 10 posts on the first page — were updated, hence changing their LastActiveDate to be the most recent. Upon navigating to the second page, I now see 10 more posts, but because of the shift in order, 2 posts from the first 10 — were pushed into the second 10 on the second page.

Usually, on platforms with low update rates, like a small forum — this drawback is not that critical. Occasionally users will see content on the next page, which appeared on the previous page — but the frequency during which it happens — is low. We can increase the page size to say 15, 20, 25, or even 30 posts per page — and since forums often have activity concentrated in few popular or new posts on the first page, the chance of seeing the same data on next pages — is small, and we can live with that.

However, for high throughput platforms like Twitter, or machine-to-machine streaming services — this case is unacceptable. Image navigating Twitter only to see the same content over and over again, because the rate at which content is created on Twitter is rapid, and thus content from the previous page is always pushed down to the next page.

In order to solve this problem, let me introduce — Cursor based pagination.

Cursor based pagination

Let’s say we want our forum pagination to never display data from previous pages. When a user navigates to the second page — she will always see data that suppose to be on that page. This can be achieved with cursor based pagination.

How cursor based pagination works

In cursor based pagination, we do not have logical pages. Instead, each page returns the data that suppose to be displayed on that page as well as a token to retrieve the next page. The token is usually opaque, and for the end user it has no meaning — but our system knows to read it and generate the next page based on it.

This brings me to the first drawback of cursor based pagination:

[!attention]
Cursor based pagination is forward. It does not support going back (i.e. navigating from second page back to first, or from third back to second); nor it supports skipping pages (i.e. going to page 5 from page 2).

On a platform like forum, this might not be very useful. Forums are typically meant to be browsed, so users always can jump to a specific page, navigate backwards and forwards, or event jump to the last page. However, — on platforms with infinite scrolling navigation, like feeds, this solution is perfect.

Cursor based pagination example

Let’s say our product manager came up with the terrible idea of turning our 90s looking forum with traditional pagination, to 2000s looking forum with infinite scroll feed. We’ve tried to warn him that this is a bad idea, but he insisted that users wanted it. And so let’s turn our forums’ traditional pagination — to cursor based instead.

First, we need to select the cursor. Since forum posts are often sorted in chronological order from newest to oldest post, it makes sense to use the LastActiveDate as a cursor. Hence, the retrieval of a particular pages’ data can be done like this:

async function getPosts(cursor: null | Date) {
    const posts = await table("posts").select("*").orderBy("LastActiveDate", "DESC").limit(PAGE_SIZE).where(q => {
        if(cursor) {
            q.where("LastActiveDate", "<", cursor);
        }
    });

    return {posts, nextPageCursor: posts.length ? posts[posts.length - 1]["LastActiveDate"] : null}
}

Disclaimer: The query builder is a pseudocode, but can be achieved with tools like KnexJS

For every call, we fetch the first PAGE_SIZE posts and provide a cursor to the next page, which is equal to the LastActiveDate of the last post in our current batch (or null if no posts exist). If the cursor is null — we start from the first page.

Since our posts are always sorted in chronologically descending order, our pagination becomes consistent.

Let’s try to do a visualization. Let’s imagine we have the following database:
| ID | Title | …. | LastActiveDate |
| --- | ------- | --- | ------------------- |
| 1 | Title 1 | | 2022-01-10T00:00:00 |
| 2 | Title 2 | | 2022-01-09T00:00:00 |
| 3 | Title 3 | | 2022-01-08T00:00:00 |
| 4 | Title 4 | | 2022-01-07T00:00:00 |
| 5 | Title 5 | | 2022-01-06T00:00:00 |

[!disclaimer]
I’ve intentionally sorted the posts by LastActiveDate to easier visualize the process. Obviously, the data does not need to be sorted in advance, since SQLs SORT BY LastActiveDate DESC handles the sorting.

For the sake of simplicity, let’s set our page size to 2. Upon first execution, we will get posts 1 and 2 and our nextPageCursor will be equal to 2022-01-09T00:00:00 (which is the LastActiveDate of the second post).

As we would like to retrieve the next page, we will provide the nextPageCursor to our function, and since the query looks at posts older than 2022-01-09T00:00:00 we will get two more posts — IDs 3 and 4.

Now — let's come back to our tea scenario. Imagine I’m looking at the forum's first page with posts 1 and 2, and before clicking next, I go to drink tea. During that time, someone posts a new comment in post ID 3 — thus altering its LastActiveDate to be today (let’s say today is 2022-01-11T00:00:00). The table will now look like this:
| ID | Title | …. | LastActiveDate |
| --- | ------- | --- | ------------------- |
| 3 | Title 3 | | 2022-01-11T00:00:00 |
| 1 | Title 1 | | 2022-01-10T00:00:00 |
| 2 | Title 2 | | 2022-01-09T00:00:00 |
| 4 | Title 4 | | 2022-01-07T00:00:00 |
| 5 | Title 5 | | 2022-01-06T00:00:00 |

When I come back to the computer, I click next page. In our traditional navigation — my action would have returned posts with IDs 2 and 4, thus “leaking” the second post from the first page to the second.

Cursor based navigation, with the cursor of 2022-01-09T00:00:00, would however return posts with IDs 4 and 5 as expected. Notice how post ID 3 is skipped and will never be seen by me, unless I start over from page 1. It’s important to point out that this flaw is not a problem of the cursor based pagination — but rather a problem with the way we’ve implemented cursor based pagination. Different use cases — have different implementations. It is possible to implement cursor base pagination which takes into account newer content as well. It, however, makes little sense to do so in a context of a forum, since in forums — going forward in pagination — means seeing older content.

As I’ve said before, we do lose some functionality — namely: going backwards (from page 2 to page 1 for example) and skipping pages (from page 2 to page 7 for example). So this approach is not useable for browsing content in a paginated manner, but it is a good approach to implement infinite scrolling without encountering duplicate content.

Bonus—don’t use dates and or timestamps as cursor

I could have ended the article here, but I’m pretty sure if I did so — I’d get criticized for using date as a cursor. And this will be a well deserved criticism.

You see — the entire point of cursor based navigation — is to create a consistent stream of data without repetitions. Yes — you give away some features, but in exchange you get fresh data for each page. As I’ve said — some systems might benefit from it.

However, — by using dates as cursors — you run into the same problem. You can encounter content repetition. Let’s assume we have a high traffic forum with a very active community. Let’s look at a subset of such forums posts table:
| ID | Title | …. | LastActiveDate |
| --- | ------- | --- | ------------------- |
| 675 | Title 1 | … | 2022-01-10T12:31:22 |
| 123 | Title 2 | … | 2022-01-10T12:30:48 |
| 534 | Title 3 | … | 2022-01-10T12:30:23 |
| 301 | Title 4 | …| 2022-01-10T12:30:23 |
| 231 | Title 5 | …| 2022-01-10T12:30:23 |
| 945 | Title 6 | …| 2022-01-10T12:30:17 |

Disclaimer: as earlier, the data is sorted by LastActiveDate for demonstration purposes only.

For the sake of example, let’s say our page size is 2. And since we are looking at a subset of data, let’s say we hold a nextPageCursor of 2022-01-10T12:31:22. What will be the result of the invocation of getPosts("2022-01-10T12:31:22");? If you’ve read through the article and understood it well, you would be correct to answer: posts 123 and 534. And what will be the nextPageCursor value for that invocation? The correct answer is 2022-01-10T12:30:23.

Now, here’s the problem. Let’s say you want to navigate to the next page using nextPageCursor = '2022-01-10T12:30:23'. What would be the result? That’s true — post ID 945. Since the query we’ve written earlier, translates to:

SELECT * FROM posts
ORDER BY LastActiveDate DESC
WHERE LastActiveDate < '2022-01-10T12:30:23'
LIMIT 2

We are skipping posts ID 301 and ID 231. The hasty among you might just run and replace < to <=, but this introduces another issue. Now, instead of getting post ID 945 you get posts ID 534 and ID 301. So not only you are getting post ID 534 again, your nextPageCursor is set to 2022-01-10T12:30:23 so you're essentially stuck in an endless loop (unless someone decides to update one of the posts).

Dates, although provide chronological order, are not providers of uniqueness. In a system with high rate of content, you might end up with collisions. And I know what you are thinking about: I’ll just add milliseconds! While milliseconds might help you to prevent this issue in human generated content (although there is still a chance of collision, depending on the traffic and the nature of the system) — they are not a solution for machine generated content in systems with high rate of content generation. There are other solutions, like content versioning, to provide uniqueness of the cursor.

Also keep in mind that cursors do not have to be bound to one specific dimension. I once designed a system where the cursor was a JSON that held multiple vectors for data retrieval. This JSON was then base64 encoded and was used as a cursor for navigation. So you could combine multiple fields, and any other data you would want, inside the cursor. As long as you can pass the cursor in the URL and your system can decode it — it’s a viable solution.

Why you should adopt Makefile in all of your projects

Dmitry Kudryavtsev — Mon, 19 Dec 2022 09:48:28 +0000

GNU Make. A software that is, most likely, older than you. It’s so simple, so standard, and so ignored. I’m here to provide a case in favor of make and Makefiles.
What is make and Makefile?

Before I even start to build my case, I need to explain to you what is make and Makefile.

What is make and Makefile?

make - is a build automation tool. It is written in C and was first released in April 1976. On Linux, make is usually available as part of the build tools dependencies that includes stuff like gcc - the C compiler etc. On Ubuntu, it's under build-essentials; on Arch Linux, under base-devel. MacOS users can install it with brew. And on Windows... I honestly have to clue what is going-on on Windows.

But make does not work by itself. It needs instructions. And those instructions are stored in a Makefile.

I believe in owning my content, in the same way I own my computer or my car. Therefore, if you want to continue reading this article, for free, you can follow this link to my personal blog.

4 Ways to Minimize Your Dependencies in Node.js

Dmitry Kudryavtsev — Wed, 14 Sep 2022 11:29:46 +0000

We all know the joke about how node_modules is the heaviest object in the universe.

For example, a project that uses only fastify, knex, typescript, and uuid generates an 83MB node_modules folder! That's huge! And those four packages are far from a complete set for a relatively small back-end. A more realistic size for node_modules is north of 100MB, in some cases reaching 1GB.

In this post, we'll explore four methods to minimize your code dependencies, resulting in faster CI/CD execution and safer code.

But first, let's touch on some of the problems with heavy node_modules.

The Issues with Heavy `node_modules` in Node.js

Heavy node_modules can cause slower CI/CD pipelines, as dependencies are usually installed during those pipelines, and require network calls to a registry (whether npm or your mirror of it). This affects your development experience.

Moreover, package creep can introduce serious issues like security vulnerabilities, as you don't own the code that resides inside the packages.

Let's jump into how to minimize the dependencies in your Node.js code.

Method One: Check the Age of Node.js Packages

Imagine a situation where you have a function that looks like this:

function findSomethingByIds(ids: number | number[]) {
  ...
}

This function's purpose is to either find a single entity or a list of entities by their id. It's common practice with different repository patterns.

However, let's say you want to know whether you've requested a single entity or a list of entities. A quick npm search leads us to npm/isarray, an insanely popular package with over 62 million weekly downloads! You know the drill:

npm i isarray

Don't forget:

npm i -D @types/isarray

But wait! Before you do that, have you noticed that this package is three years old? And if you do a better search, you will discover that isArray is now part of the JavaScript core, can be invoked using Array.isArray, and is perfectly supported in Node.js version 10 and up.

Package age is a good first indicator. Many old (2+ years) packages might have security vulnerabilities or be outdated. Some of the outdated packages are also merged into the official JavaScript spec.

Another example is the trim package. It has more than 4 million downloads, and while it's not that old (only 1 year), a native solution exists in JavaScript core: String.prototype.trim().

So always try to find a native solution first, as JavaScript evolves quickly and its standard library is always expanding. Don't be fooled by the weekly downloads counter in npm.

Method Two: Using 'One-liner' Node.js Packages

Many 'simple' packages are actually what I call 'one-liners'. A one-liner is a package that contains very few lines of code.

If we continue with our isArray example from above, by examining the content of index.js we can see that it's a simple one-line function:

var toString = {}.toString;

module.exports =
  Array.isArray ||
  function (arr) {
    return toString.call(arr) === "[object Array]";
  };

Source: GitHub

I always recommend you look at the source code of packages, at least simple ones because they can teach you a lot. Looking at the above source code, we discover one important thing: that Array.isArray exists (the same conclusion we came to with method one).

However, even if we did not have a native is-array method, the entire function is a simple if statement. And instead of requiring this package as a dependency, we can simply write the code as part of our project. It's worth stopping for a minute and discussing the pros and cons of such an approach.

Using Npm vs. Writing Your Own Code

Npm is a public registry, and the code contributed to npm comes from people worldwide. It is amazing that we have such a big repository of free and open-source code. However, this also comes with some disadvantages.

Npm, unfortunately, is known to be at high risk of attack. Packages might get compromised, whether by third parties or by developers themselves.

We all remember the scandal around left-pad and the recent rise in crypto-mining and crypto-stealing code that resides inside popular npm packages.

While it's impossible to audit every single package we install, we can lower the attack vector by using fewer packages in our projects. A great way to reduce the number of packages we depend on is to write trivial packages ourselves.

On the other hand, by writing some code ourselves, we deprive ourselves of the community's sheer knowledge. Multiple people maintain popular packages, so they can quickly react to new vulnerabilities (for example, by monitoring the GitHub issues page of their package). This is something you, as a sole developer or part of a small organization, might lack the resources to do.

So the next time you are eager to install a package, check its source code. It might be a one-liner that you can write instead of introducing a potential attack risk and slowing your CI/CD pipeline.

But don't go too far with this method: you don't want to reinvent the wheel or entirely deprive yourself of the community's support.

Method Three: Extract Sub-packages with `lodash`

Imagine we have the following object interface:

interface SomeObject {
  foo?: {
    bar?: {
      baz?: string;
    };
  };
}

And we have a function that accepts an object with that interface.

function getBazOrDefault(obj: SomeObject, defaultValue: string);

As you've guessed from the function's name, it will give us the value of baz, or defaultValue, if the path to baz is undefined. Here is one implementation of that function:

function getBazOrDefault(obj: SomeObject, defaultValue: string) {
  if (!obj.foo || !obj.foo.bar || !obj.foo.bar.baz) return defaultValue;
  return obj.foo.bar.baz;
}

Ugly, right? It will get uglier if you need to deal with arrays. Luckily, we can use the popular lodash library!

Install lodash, and the code becomes nice and easy to read:

import _ from "lodash";

function getBazOrDefault(obj: SomeObject, defaultValue: string) {
  return _.get(obj, "foo.bar.baz", defaultValue);
}

Neat!

However, if we install lodash:

npm i lodash

And its type definitions (because we use TypeScript):

npm i -D @types/lodash

We will introduce 8.4MB of dependencies to our node_modules. That's a lot!

❯ du -sh node_modules
8.4M    node_modules

We have some experience now, so let's put it to practice!

Package age - lodash is relatively old — it was published a year ago. I couldn't find any JavaScript core functionality to get a nested value from an object by a string key. Let's move on.
One-liner package - lodash is not a one-liner. It has tens of files and lots of tests. Even looking at the functionality of _.get, it's not exactly a one-liner. While it's a simple two-line function, it has an internal dependency on ./internals/baseGet.js, which in turn depends on ./castPath.js and ./toKey.js — and each depends on more files! It's too much to be a candidate for extracting standalone code.

So it seems we are stuck with lodash then. But wait! There is another trick I want to show you! Sub-package extraction. If we read the lodash readme carefully, we will notice that every single functionality lodash provides is extracted into its own sub-package, including the _.get function!

Instead of installing the entire lodash library, we can install lodash.get for just the _.get function (of course, don't forget the type definitions). This reduces the node_modules size from 8.4MB to 3.6MB. It's still a large folder, but a 57% reduction from the original size! I'll happily take such a percentage reduction.

❯ npm i lodash.get
❯ npm i -D @types/lodash.get
❯ du -sh node_modules
3.6M    node_modules

The getBazOrDefault code will look like this:

import get from "lodash.get";

function getBazOrDefault(obj: SomeObject, defaultValue: string) {
  return get(obj, "foo.bar.baz", defaultValue);
}

In conclusion, be aware that many packages, especially collections of utilities like lodash, can be published to npm as individual packages. Instead of installing the entire 312 different methods from lodash, we can install the ones we actually need, reducing the weight of our node_modules.

Method Four: Do It Yourself

Last but not least is the DIY method.

Let's say we need a method to capitalize the first letter of each word in a given string. We can run an npm search for the term capitalize and be presented with 266 different packages (not all relevant).

We can spend some time looking for a relevant, relatively maintained package and add it as a dependency to our project. Or we can write it ourselves!

Why, you ask? Software engineering is the art of problem-solving. To become a great software engineer, you need to be able to solve software engineering problems.

If you only know how to use third-party packages and copy code from Stack Overflow, you will be a software engineer, but not a great one. And you do want to be great, don't you? Then let's write our own capitalize function!

Let's start with a signature:

function capitalize(str: string);

First, let's identify words from our string. We can do that by splitting the string with whitespace:

const words = str.split(" ");

Then we need to capitalize the first letter of each word:

for (let i = 0; i < words.length; ++i) {
  words[i] = words[i].charAt(0).toUpperCase() + words[i].substring(1);
}

And lastly, we need to join the words back into a string:

const capitalizedString = words.join(" ");

Here's the entire function:

function capitalize(str: string) {
  const words = str.split(" ");
  for (let i = 0; i < words.length; ++i) {
    words[i] = words[i].charAt(0).toUpperCase() + words[i].substring(1);
  }
  return words.join(" ");
}

If you run this with some example strings:

console.log(capitalize("hello world")); // Hello World
console.log(capitalize("how are you?")); // How Are You?

Great success!

Note - this function is for demonstration purposes only. It does not consider scenarios like quoted words — for example, hello "world" will be capitalized incorrectly. However, this is a great opportunity to up your problem-solving skills! Go on and figure out how to capitalize quoted words as well!

As with the second method, you'll want to avoid 'reinventing the wheel' here. On the one hand, you don't want to write the same functionality over and over. On the other hand, you don't want to introduce a dependency for every small functionality you need.

Always analyze and evaluate each solution. Many packages do way more than you need, so it's better to implement the code yourself. However, there are good quality packages for common utils like slugify, which is well-maintained and provides basic functionality you probably wouldn't want to implement yourself in a production environment.

Wrap Up

In this post, we ran through four methods to analyze packages and shrink your node_modules:

Checking the age of packages
Using one-liner packages
Extracting sub-packages
Doing it yourself

These methods should make your development process faster, reducing the number of packages you need to pull every time you run your CI/CD pipeline.

All in all, the npm ecosystem is great! It is the biggest package registry as of today. But installing packages shouldn't be a panacea. Our project is not only the code we write — it's also the packages it consists of. Knowing what goes into our node_modules makes our projects more resilient and makes us better developers.

Until next time, happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

DEV Community: Dmitry Kudryavtsev

Why engineers should focus on writing

Why write?

Writing is a way to organize your brain

Writing is a way to learn something

Writing helps you identify mistakes

How to write more?

Cursor based pagination

Traditional pagination

Drawbacks

Cursor based pagination

How cursor based pagination works

Cursor based pagination example

Bonus—don’t use dates and or timestamps as cursor

Why you should adopt Makefile in all of your projects

What is make and Makefile?

4 Ways to Minimize Your Dependencies in Node.js

The Issues with Heavy node_modules in Node.js

Method One: Check the Age of Node.js Packages

Method Two: Using 'One-liner' Node.js Packages

Using Npm vs. Writing Your Own Code

Method Three: Extract Sub-packages with lodash

Method Four: Do It Yourself

Wrap Up

The Issues with Heavy `node_modules` in Node.js

Method Three: Extract Sub-packages with `lodash`