DEV Community: Alex Zherdev

What's confusing you about npm?

Alex Zherdev — Tue, 02 Mar 2021 19:49:10 +0000

Photo by Emily Morter on Unsplash

Let’s have an open floor to vent about things we don’t quite get about npm when using it in our daily lives (as well as alternatives like yarn).

Are you not sure why we need lockfiles? Do peerDependency warnings scare you? We don’t judge here. Do note that the purpose is not to bash the tool—I’m doing independent research to identify gaps and opportunities for better education.

Anything goes, but one scenario I’m particularly interested in is application developers trying to create their first reusable libraries, which seems to be a common source of confusion. Whether that’s you or not though, I’m grateful for any and all responses 🙇‍♂️

Async Generators in Use Pt. 2: Multiple DynamoDB Queries in Parallel

Alex Zherdev — Wed, 30 Dec 2020 22:06:26 +0000

Photo by Cameron Venti on Unsplash

Recap

This is a follow-up to my previous post on using async generators in conjunction with DynamoDB pagination. As we saw last time, async generators can be a handy way to iterate over ”eventually available” collections:

async function* getPaginatedResults(dynamoClient, params) {
  let lastEvaluatedKey;
  do {
    const queryResult = await dynamoClient.query(params).promise();

    lastEvaluatedKey = queryResult.LastEvaluatedKey;
    params.ExclusiveStartKey = lastEvaluatedKey;

    yield queryResult.Items;
  } while (lastEvaluatedKey);
}

for await (const pageOfItems of getPaginatedResults(dynamoClient, someQueryParams)) {
  // do something with the page of items
}

This works great as is for making a single query that can potentially be paginated. What if we need to run multiple such queries in parallel? Let’s try a naïve approach first.

👎 Stacking `for-await-of` Loops

The for-await-of loop (like the regular for loop) doesn’t allow the code below itself to execute until it’s done iterating:

for await (const pageOfItems of getPaginatedResults(dynamoClient, someQueryParams)) {
  // do something with the page of items
}

// <-- Can’t get to here until the loop is over

This can present a problem in cases like ours. If we just put two for-await-of loops adjacent to each other, the second query won’t start until the first one loops through all of its pages:

for await (const pageOfItems of getPaginatedResults(dynamoClient, someQueryParams)) {
  // do something with the page of items
}

// Only when the first query is done can we start the next one.
// If the second query doesn’t depend on the first one’s results,
// running them sequentially is suboptimal.
for await (const pageOfItems of getPaginatedResults(dynamoClient, someOtherQueryParams)) {
  // do something with the page of items
}

Looks like just putting two for-await-of loops together isn’t achieving what we want, so we need to look for a different approach.

👍 Wrapping `for-await-of` Loop in an Async Function

Before proceeding, let’s assume we have two parallel queries for simplicity (to be clear, since each query can get paginated, we’re potentially talking about two parallel series of queries).

If we want two parallel asynchronous tasks to complete before continuing, we can use Promise.all, passing in the promises representing the completion of each task:

async function task1() {}
async function task2() {}

const p1 = task1();
const p2 = task2();
await Promise.all([p1, p2]);
// ...continue execution

In our case, each task is a paginated query. If we can somehow get a promise that indicates all pages of a query have been processed, we’ll be able to use the above scheme. How do we get a promise like that? Well, there’s a hint in the example above: if we put our old for-await-of loop into an async function, the promise returned from that function will essentially represent the completion of that loop. Let’s do just that:

async function paginateQuery(dynamoClient, queryParams) {
  for await (const pageOfItems of getPaginatedResults(dynamoClient, queryParams)) {
    // do something with the page of items
  }
}

Now, for this to become a truly reusable helper, we need to parameterize it with the body of the loop. Here’s an example for querying users and posts in parallel:

async function paginateQuery(dynamoClient, queryParams, callback) {
  for await (const pageOfItems of getPaginatedResults(dynamoClient, queryParams)) {
    await callback(pageOfItems); // optionally break from the loop if callback returns false
  }
}

const usersPromise = paginateQuery(client, usersParams, async (pageOfUsers) => { /* ... */ });
const postsPromise = paginateQuery(client, postsParams, async (pageOfPosts) => { /* ... */ });
await Promise.all([usersPromise, postsPromise]);
// ...continue execution

Wait, we’re back to dealing with callbacks now? A bit anticlimactic, but it seems that we have to in this case. Note that you only have to use this helper where you need parallel queries, although if you decide to use it everywhere, that’s totally reasonable too 😄.

Conclusion

As we’ve seen, directly using a for-await-of loop with an async generator can be pretty convenient but has its limitations. Do let me know if you can think of a more elegant solution!

Async Generators in Use: DynamoDB Pagination

Alex Zherdev — Thu, 29 Oct 2020 17:32:35 +0000

Async Iteration and Async Generators: Recap

Asynchronous iteration was added to the ECMAScript standard in its 2018 edition (TypeScript has supported it since version 2.3). In layman’s terms, it means iterating over a collection where you have to wait for each item to become available:

// Compare:

const collection = [1, 2, 3];

for (const item of collection) {
  console.log(item);
}

// and

const collection = [Promise.resolve(1), Promise.resolve(2), Promise.resolve(3)];

for await (const item of collection) {
  console.log(item);
}

Just like with regular, synchronous iteration, asynchronous iteration is compatible with asynchronous generators. In both cases you iterate over values yielded from a generator:

// Compare:

function* gen() {
  yield 1;
  yield 2;
  yield 3;
}

for (const item of gen()) {
  console.log(item);
}

// and

async function* asyncGen() {
  const one = await getOne();
  yield one;
  const two = await getTwo();
  yield two;
  const three = await getThree();
  yield three;
}

for await (const item of asyncGen()) {
  console.log(item);
}

You can read more on the topic in this article by the ever scrupulous Dr. Axel Rauschmayer.

Asynchronous generators seem like a neat idea, but maybe not something the average developer will make a lot of use of. And indeed, it took a couple of years for me to encounter a nice real-world application for async generators, and now I’m happy to tell you all about it. (Credit goes to my colleague Peter Smith for the original idea.)

Async Generators: A Real Use Case

When you query a DynamoDB table, the result set can get paginated if the amount of data exceeds a certain size. It looks a little bit like this:

const queryResult = await dynamoClient.query(params).promise();

// If queryResult.LastEvaluatedKey is present, the query was paginated.
// queryResult.Items contains a page of results, but not the entire result set.
// To load the next page, we need make another query, passing LastEvaluatedKey
// as the start key in the params for that query.
params.ExclusiveStartKey = queryResult.LastEvaluatedKey;
const nextQueryResult = await dynamoClient.query(params).promise();

// ...Repeat until queryResult.LastEvaluatedKey is empty.

In a real application you would do this in a loop. You might also want to create a helper function with this logic so it can be reused for different types of queries throughout the application. A simple approach would be to combine results from all pages into an array:

async function getPaginatedResults(dynamoClient, params) {
  let lastEvaluatedKey;
  const results = [];

  do {
    const queryResult = await dynamoClient.query(params).promise();

    lastEvaluatedKey = queryResult.LastEvaluatedKey;
    params.ExclusiveStartKey = lastEvaluatedKey;

    results.push(...queryResult.Items);
  } while (lastEvaluatedKey);

  return results;
}

const allItems = await getPaginatedResults(dynamoClient, someQueryParams);

Depending on the context, this may be perfectly reasonable. What if, however, you want to further process each page of results as soon as it’s available, without waiting for the rest to come in? The simplest implementation might accept a callback with the processing logic:

async function forEachPage(dynamoClient, params, callback) {
  let lastEvaluatedKey;

  do {
    const queryResult = await dynamoClient.query(params).promise();

    lastEvaluatedKey = queryResult.LastEvaluatedKey;
    params.ExclusiveStartKey = lastEvaluatedKey;

    await callback(queryResult.Items);
  } while (lastEvaluatedKey);
}

await forEachPage(dynamoClient, someQueryParams, async (pageOfItems) => {
  // do something with the page of items
});

This is workable, but callbacks are known to be clunky in use. For example, you may need to make the callback return false to indicate that the loop needs to stop. What if, instead of this foreach-style iteration, you want to move to a for...of style? Enter asynchronous generators.

async function* getPaginatedResults(dynamoClient, params) {
  let lastEvaluatedKey;
  do {
    const queryResult = await dynamoClient.query(params).promise();

    lastEvaluatedKey = queryResult.LastEvaluatedKey;
    params.ExclusiveStartKey = lastEvaluatedKey;

    yield queryResult.Items;
  } while (lastEvaluatedKey);
}

for await (const pageOfItems of getPaginatedResults(dynamoClient, someQueryParams)) {
  // do something with the page of items
}

Every time a new page of items is loaded, the async generator yields it back into the for-await-of loop. Neat.

That last example highlights one of the key aspects of generators (both sync and async). If you look at the for-await-of loop, we only invoke getPaginatedResults once right at the start of the loop. And at that moment, it’s not known how many pages we’ll get. However, we can still conveniently run a for loop over this ”eventually known” collection, as if it were a plain old array.

Conclusion

I hope this practical example helps illustrate the usefulness of asynchronous generators. Perhaps now you can more easily spot places in your own code where they might be handy.

Three Reasons Why I Love Reviewing Code

Alex Zherdev — Fri, 28 Aug 2020 16:46:32 +0000

Introduction: Who Benefits From Code Review?

Code review is probably one of the most well-researched software development practices. There’s strong evidence that the code review process has a positive impact on software quality. And pretty much everyone agrees that having another pair of eyes on your code is a great way to level up as a developer.

Anecdotally, however, I haven’t seen much discussion around the benefits of being a code reviewer. I find this curious because I attribute a good part of my growth in recent years to doing a ton of code reviews. So I thought I’d share the top three ways in which reviewing code has made me a better engineer.

1. ”Nanoblogging”

Teaching others is one of the best ways to learn. You may have heard the advice: to get a solid grasp on a subject, write a blog post about it.

Well, oftentimes writing code review comments is akin to blogging. Commenting on a non-trivial issue you’ve noticed in a pull request can involve finding articles to support your points and double-checking your own understanding of the problem at hand. The comment itself becomes a short article, and even though your audience is small, all the benefits of teaching kick in. I would call this process ”microblogging” but that term is already taken. ”Nanoblogging” perhaps? 😄

2. ”I Know Kung Fu!”

In The Matrix, Neo (Keanu Reeves) instantaneously becomes a martial arts master after a machine dumps data directly into his brain. The data essentially simulates thousands of hours of training, so Neo is able to master kung fu without actually stepping into the dojo. Something similar happens when you review code.

The amount of coding situations you’ll find yourself in over the course of your developer career is finite. All those situations and the ways in which you respond to them ultimately form your experience. Reviewing someone else’s code expands that set—your exposure is no longer limited by the tasks you work on yourself. Code review gives you the opportunity to put yourself in the author’s shoes and ask yourself, ”How would I implement this? Could I improve on this solution?” In doing so, you’re simulating in your mind a situation you may not have encountered otherwise.

Granted, it’s not quite the same as actually working on a task and writing code on your own, and your thinking is biased by seeing one possible solution. But if you work through enough of these simulations, I’m sure you’ll start to reap the benefits. I can’t promise that you’ll learn any martial arts that way, though.

3. Expanding Reputation

In organizations with more than a couple of engineers it’s not realistic for a single person to review every change being made to the codebase. If there is more than one team, you usually end up reviewing code your immediate team is responsible for. Naturally, providing valuable feedback in a tactful manner helps you gain recognition within your team.

As I got more senior, I’ve been able to step this up a notch. I follow a few repos other teams work on. Then whenever I have some downtime, I skim the PR notifications from those repos and see whether I could be of help to the folks there. The time I can dedicate is limited, so I can usually respond to just a few of those notifications at a time and I need to be picking them strategically. This process almost deserves a post of its own, but here I’ll list a few rules of thumb I apply when choosing which notifications to open and comment on:

If the author is struggling with something, is the problem small enough that I can help them without needing a ton of context I may be missing?
Say I noticed a problematic part in the PR. Is it likely that the author’s teammates may miss it? In case the author has a question that I can answer, is it likely to be left unanswered if I don’t chime in?
Do I have enough time to leave an informative and considerate comment?

If the answer to all of these questions is ”yes”, I usually go in and leave a comment. Examples of issues I often help with include tricky JavaScript/TypeScript questions, tooling issues (Babel/Webpack/Jest/etc.), and npm gotchas.

This approach is not without caveats and I wouldn’t blindly recommend it to everyone. I strive to make sure that my coworkers are okay with receiving unsolicited feedback on their PRs, and that the time of both parties in these interactions is used effectively—hence the rules of thumb above. But so far the outcome has been overwhelmingly positive: by helping out many of my peers, I’ve been able to build reputation and relationships beyond my own team. If you decide to try this out, let me know how it goes!

Conclusion

There’s more to love about reviewing code but I feel like these three aspects are the most shareworthy. I hope this has been helpful and I’m curious what everyone’s experiences are with being a code reviewer!

DEV Community: Alex Zherdev

What's confusing you about npm?

Async Generators in Use Pt. 2: Multiple DynamoDB Queries in Parallel

Recap

👎 Stacking for-await-of Loops

👍 Wrapping for-await-of Loop in an Async Function

Conclusion

Async Generators in Use: DynamoDB Pagination

Async Iteration and Async Generators: Recap

Async Generators: A Real Use Case

Conclusion

Three Reasons Why I Love Reviewing Code

Introduction: Who Benefits From Code Review?

1. ”Nanoblogging”

2. ”I Know Kung Fu!”

3. Expanding Reputation

Conclusion

👎 Stacking `for-await-of` Loops

👍 Wrapping `for-await-of` Loop in an Async Function