DEV Community: Maciej Wakuła

JavaScript turbo startup

Maciej Wakuła — Mon, 23 Oct 2023 05:36:14 +0000

Note: Some topics might be intentionally skipped or not giving full picture to speed you up into JavaScript.

JavaScript language can run in a browser (ex. chrome, firefox, safari) or server (ex. node, bun). Your code runs command after command (serially) with no multithreading. Common mistake is to run long method (ex. "FOR" iteration loop) too long in a row blocking page or server.

Most often used IDE is currently visual studio code due to low price (free), great ecosystem and integration, plenty of features out of the box and plugins.

JavaScript has plenty of frameworks and modules (libraries) hosted on npmjs.com. Most cases you pick one framework for the front-end (browser), one for the back-end (server) and stick to them. It is important to separate front-end from back-end as modules are loaded and executed differently what in the end leads to different specialization of the tools.

Few projects to start: express web server, socket.io browser-server communication, passport.js authentication to login to your page, next.js framework, nest.js framework.

Language versions confusion.
JavaScript evolves dynamically and many features are being constantly introduced. Developers usually can use a transpiler, that is an application to translate code from whatever language (newest version of JavaScript, Typescript, coffeescript, etc.) into an older version of JavaScript supported by their browser/server. The most important distinction is to JavaScript, or rather EcmaScript with different versions (having a number of an equivalent year) and other languages. Things get much more complicated when you read about the module loading, code execution and more complex topics.

Git commit messages

Maciej Wakuła — Thu, 22 Jun 2023 16:40:27 +0000

Hints for writing git commit messages.

commit message has a header, then optional empty line followed by longer description
- header is a 1-line abstract summary
- header can start with a prefix, ex. "fix:", "feat:", "BREAKING" - see https://conventionalcommits.org
- keep it short, avoid plural, "the", "an", dot, etc.
- examples: "add lib X", "fix #123", "add default cfg"
- longer description can be added after an empty separator line - use it to provide details for whoever might need to check this commitin the future, link to tickets, features, links, files
use git log --oneline to show only the commit headers
as long as it is on your local branch - use rebase to keep your commits on the top of other changes
after someone might have used it (ex. you merged it to develop), avoid rebase

Asynchronous processing in JavaScript

Maciej Wakuła — Sat, 15 Apr 2023 13:48:36 +0000

This is a continuation of Introduction to node and npm

1. Introduction

JavaScript engines are generally single-threaded. Long call processing would normally block any other call but we leverage asynchronous processing to interrupt currently executed procedure allowing engine to switch between the jobs. Behind the scenes Input/Output is multi-threaded.
There are many ways JS developers could achieve effects similar to multi-threading.

2. How asynchronous processing works

You send a message (ex. a request to process a resource like "add a new user to the system") and usually get a confirmation that the request was received. In REST this might be 202 (Accepted) and "location" header, in messaging systems a valid ID of the request.
But the message is only queued for processing, not yet processed.

Whenever system has free resources (ex. is not doing anything more important or queued before your request) then your request could be processed. This takes usually between milliseconds and weeks. It could be for example a request to add a new bank account that must be verified by someone.

When requesting something, often a feedback (confirmation and/or return data) is expected. This is usually done using message sent by the processing system to a place you can access. Often it would contain "correlation-id" that is the ID of the request to indicate it is a response to previous request. Simply saying: We cannot send response (not yet) but we can send instruction (ex. URL) to check if it was already finished.

3. Several attempts to use asynchronous processing

3.1. How the code execution gets postponed

In JavaScript calls could be postponed whenever you use a promise (or async/await which is using promises behind the scene).

async function log1(msg) {
  await console.log(msg);
}
async function log2(msg) {
  console.log(msg);
}
function log3(msg) {
  console.log(msg);
}
function log4(msg) {
  return new Promise(resolve=>console.log(msg));
}

setTimeout(()=>log1(1), 0);
setTimeout(()=>log2(2), 0);
setTimeout(()=>log3(3), 0);
setTimeout(()=>log4(4), 0);

Usually prints 1,2,3,4 as expected.
Internally those are 4 different calls scheduled to happen with 0ms delay (no delay). And the order of execution doesn't need to keep the sequence.

log1 is async (returns a promise) and stops execution when calling console.log because there is await. Once called, the promise is not yet fulfilled - it would be after console.log returned,then log1 could continue, and finally it returns. You are sure that console.log returns before log1 gets fulfilled (only because there is "await").

log2 is async (returns a promise) and triggers console.log and returns. You have no guarantee that console.log prints output before log2 gets fulfilled.

log3 is just a function. The console.log might be scheduled but log3 returns and continues to execute your code. The JavaScript thread is blocked until it returns (though it can schedule some promises).

log4 returns a promise. Its execution could be delayed (then you see that returned promise is "pending" until it gets "fulfilled").

Think about it thoroughly and you could also omit setTimeout in 3 out of 4 cases - then call is made and a "pending promise" is returned. In fact the case of log3 is probably most dangerous.

If you are using node, then internally libuv is used for calls and also for anything like input/output (ex. to print something onto the console).

3.2. Calling other applications

You can sent a request to another system, service or application. You can send a REST request and should wait for the response (even "Accepted" when task is only queued). Or publish this to a queue yourself.

3.3. Workers

Node can run parallel instances of node called "workers". Workers can act as a parallel process to perform heavy work without blocking the main thread.

4. What to look at

4.1. Intervals

You can schedule a method to be called with a fixed interval:

let counter = 0;
let intervalHandle = undefined;
const intervalInMilliseconds = 10;

function myMethod() {
  counter++;
  for(let i = 0; i < 9999; i++){console.log(i);}

  if(counter > 10) {
    clearInterval(intervalHandle);
    intervalHandle = null;
  }
}

intervalHandle = setInterval(myMethod, intervalInMilliseconds);

You could expect that it prints number from 0 to 9999 every 10ms until 10 iterations are made within 100ms.
The issue is that you have no guarantee that methods would be called every 10ms. This could lead to 2 problems:

Your method could be called more than once at the same time (if it is using asynchronous processing internally)
Your method could be executed with longer interval (it needs to print the numbers first)

A bit corrected code:

let counter = 0;
const minimumIntervalInMilliseconds = 10;
function myMethod() {
  counter++;
  for(let i = 0; i < 9999; i++){console.log(i);}

  if(counter <= 10) {
    setTimeout(myMethod, minimumIntervalInMilliseconds);
  }
}

setTimeout(myMethod, minimumIntervalInMilliseconds);

4. Dangers of single-threaded app

If your application runs inside a docker container that has a healthcheck and restarts whenever healthcheck is not responded (ex. timeout 1s, interval 1s), then any job blocking main thread for 1000ms would result in container being restarted.

If your app runs on kubernetes and your is blocked over the timeout threshold then your application might be "disconnected" (readiness probe) or restarted (liveness probe).

If you are connected to a queue (ex. kafka or RabbitMQ) and your application is busy for the timeout (ex. 3000ms for kafka subscriber) then your application will be kicked-out and considered "dead". I have seen this scenario where developers tried to update RabbitMQ and its libraries searchign for an issue in the server - where the cause was in the node code.

node --save-*

Maciej Wakuła — Tue, 14 Mar 2023 21:32:10 +0000

Content

This article roughly describes node package management concepts especially focusing on dependencies management.

Introduction

Node

The node is a single-threaded asynchronous server-side runtime environment for JavaScript (you can run JavaScript without a web browser) using "V8" JavaScript runtime written in C++ by Google.

Node packages

JavaScript is very (too much?) flexible resulting in millions of libraries (modules) being created and published (over 1.3 million at the moment this article was created).

Project often include many modules - which then include others. This creates a very complex web of dependencies that needs to be solved in order to run the software.

NPM - Node Package Manager

Node comes with its own solution to manage, publish, download, install and also share modules (packages). There are many solid alternatives but npm became a part of the node (while npm is an installable module itself).

Types of dependencies

Node currently defines following types of dependencies:

normal dependencies - meant to run on production environment
- you install them using npm install --save {dependencyName}
- keep them as least as possible
- whenever someone installs your module - they would also install those dependencies
development dependencies - to be used during development only (but it is used also for testing)
- whatever you need for your development and testing
- assume they are not present on production environment
- you install them using npm install --save-dev {packageName}
- when installing on non-production environment those gets installed together with your module
peer dependencies - which are not installed for the module but must be provided by your application
- you expect them to be provided, not installed by your module
- often developers start with dependencies and devDependencies optimizing them later

Each module (package) has a version. NPM is using SemVer (Semantic Versioning). Version has 3 numbers and optionally extra part:
{major}.{minor}.{patch}, ex. 1.2.3 where

major should be increased whenever it can break existing code (increase it to indicate that update is not strightforward and might require code change)
minor is to indicate new features (with backward compatibility)
patch is to indicate non-functional changes (logic remains the same but for example bugfix is applied) Version can have an optional suffix: {major}.{minor}.{patch}-{suffix}
suffix is often either
- {name}.{number} allowing you to have multiple "branches" of builds
- {number} to just indicate build number

When installing a package dependency to your application it installs by default "latest". When updating it tries to use same major version.

Traps

Developers often release new breaking change with patch version or publish updated version with incorrect SemVer.

Tag pre-id

When releasing new version of package to npmjs.org you always publish it under a "tag". The main one is "latest". The only hint for alternative names is that "next" is often used for development/unstable releases. Often used tags are used:

to indicate a version line "npm" (ex. "latest" for current version, "next" for the unstable one in development, "latest-2" to indicate version 2.x which might still receive updates)
to indicate flavor (ex. "stable", "unstable" but also "latest")

By default most recent "latest" version is installed and later updated to newer "latest" with same major number.

Different versions in same project

Imagine using libA@1.0.1 and libB@1.0.0 but then libB depends on libA@1.0.0.
Assuming libA@1.0.1 is backward compatible - npm can install libA@1.0.1 and libB@1.0.0.

But let's change it a bit: use libA@2.0.0 and libB@1.0.0 (we know it requires libA@1.0.1). NPM would install both versions of libA.

Some packages (ex. react) would cause multiple problems when different versions are installed all together. Others might create a cross-dependency hell (libA depends on libB and libB depends on libA) what is an implementation bug (but works so often happens).

Checking your dependencies

npm outdated shows outdated dependencies: whether any newer compatible exists, and also any newer (non-compatible)
npm audit to see if your dependencies contain any known vulnerabilities
npm update is a risky operation updating your dependencies to "newer versions that **shouldBB be compatible"

My personal opinion

Huge risk of package marked as compatible (while it is not) discourage developers from updates increasing likeliness of outdated software and technical debt creation.
Having development dependencies leads to scenario where test build differs from target artifact (not reliable tests).
Together the approach is risky and dirty but you must deal with it when working with node. Node has a huge win of being single-threaded but asynchronous nicely fitting to contenerization approach on cloud. Kotlin (soon Java with loom) attempts to offer same functionality but still generally requires more powerful hardware. Node is still a good option for microservices and frontend.

This is just an introduction to the dependencies management hell in NPM.

Introduction to node and npm

Maciej Wakuła — Sun, 30 Oct 2022 19:54:05 +0000

1. Introduction

1.1. Target audience

This post is created especially for beginners and people who know some programming but are not familiar with node or npm.

1.2. Topics (not) covered

This is merely an introduction to node and npm (node package manager). It does not mention alternatives.

2. Node

2.1. About node

Node is a server-side runtime built on top of Google Chrome V8 javascript engine.
Node simply runs your javascript code on the server (without any browser window).

2.2. Usage

Simple usage: node myscript.js
Usually main file is called index.js

2.3. Setup

You can download node from https://nodejs.org/en/
You could also run it within a container, ex. by running docker run --rm -it node:lts bash (assuming you have docker installed).

2.4. Important notes about threading for developer not familiar with JavaScript

JavaScript is single threaded by design. It means that all the commands are executed sequentially but every chunk can be executed out of order.
Input/Output operations are multi-threaded though.
Knowledge of multi-threading from other languages is making things harder for you.
Try to divide your code into smaller chunks that end quickly. Example trap waiting for you is a "for" iteration which executed time consuming operation - you would block entire node process until iteration is completed.

Node is using either promises (a method call that is postponed but you could plan further calls based on its result) or async/await (which internally works on promises). See code example at the end.

If you are thinking that single-threading is bad then think again. Developer has possibly better control over the process than with multithreading one and you don't need to create a separate thread. Java is currently trying to implement similar mechanism with project loom. This solution brings some dangers though as developer can easily block the thread so your critical handlers (which are excepted to handle ex. keepalive requests) could not be called. Common unwanted effect is that your application gets disconnected from servers (database, rabbitMQ, kafka, etc.) or killed (ex. docker health checks).

3. Node Package Manager (npm)

https://www.npmjs.com/

3.1. What is npm

It is a manager application used to install, remove, update, check, ... your code but also an repository of immense amount of libraries and frameworks.

3.2. Npm packages

You could create own projects and publish them for free onto npmjs.org with public access (but also private access is available for paid accounts).
Projects that consist of several packages often are grouped into organizations like @babel. Organizations are prefixed with "@" character.

The npm ecosystem is very rich and has many packages. Amount of them is often used in jokes like with the npm drinking game.

3.3. Npm package versioning

Packages are released with tags, default tag is "latest". Any custom tag can be created but often tag "next" is used for any "unstable" releases.

Package versions should follow SemVer. Example number is 1.2.3 (major.minor.patch).
You should increase major number when introducing breaking changes (anything that could break apps using your package).
Minor number should be used when adding new features without breaking existing functionality.
Patch number is used for small bugfixes that do not break logic nor add new features.

By default when installing a package you would install most recently released version from "latest" tag (even if its number is lower than previously released one).
"Unstable" releases should be released under alternative tag and have additional suffix, ex. 1.2.4-next.0 to indicate that this is a pre-release version of upcoming 1.2.4 version under tag next, candidate number 0.
When auto-updating packages npm would try to stick with current major version (meaning "non-breaking changes only") but pick highest available "stable" version.
In fact there are no strict rules for the tags or versions.

3.4. Npm version update

You can use command npm version to update your version.
Example flow:

npm version 0.0.1  # versions starting with 0 are often considered to be not yet completed
npm version major  # bumps to 1.0.0 (breaking change)
npm version minor  # bumps to 1.1.0 (new feature)
npm version patch  # bumps to 1.1.1 (bugfix)
npm version prerelease --preid next  # bumps to 1.1.2-next.0
npm version prerelease --preid next  # bumps to 1.1.2-next.1
npm version minor  # bumps to 1.2.0
npm version prerelease --preid next  # bumps to 1.2.1-next.0
npm version patch  # bumps to 1.2.1

Command npm version has a special logic when everuted from within a git repository - then a new commit with version update is created and a new annotated tag is created.

Keeping version in the source code has pros and cons. Most important cons is probably that version must be known "in advance" while many might prefer to have a build result (an artifact) which is tested and given its number after tests are passed.

3.5. Npm code scan

Npm has built-in commands for code scan:
npm outdated shows any packages having newer versions (beware of packages like stack-trace which has non-standard pre-release published on latest tag).
npm audit scans your code for known vulnerabilities.

4. Code examples

I might update the examples in the future

4.1. Async/Await example

async function myMethod(){
  const result1 = readFileAsync('/filename');
  console.log('This gets immediately printed BEFORE file is read');
  console.log(result1); // Prints "Promise" and is not yet resolved
  const result2 = await readFileAsync('/filename');
  console.log('This is not executed until result2 promise gets fulfilled').
  console.log(result2); // Prints result as it was already read
}

myMethod(); // Here I can use promises but not async/await - but I could "run" the async method "in the background"

5. IDE (Integrated Development Environment)

Paid one https://www.jetbrains.com/webstorm/ (IntelliJ ultimate includes all its features).

Free one and providing very similar experience: https://code.visualstudio.com/

There are many other IDE's available (ex. https://atom.io/ which is written using node, javascript and npm).

Please let me know if you can find any issues with this post, do I can fix it. Also if you would like any topic to be added in this or another post.

Kafka vs RabbitMQ

Maciej Wakuła — Sat, 29 Oct 2022 23:35:31 +0000

This is a response to the article an article with same name

Introduction

Both Kafka and RabbitMQ are used to relay messages between message producers and message consumers. RabbitMQ can handle large amount of messages (dozens up to thousands per second) but Kafka can handle much higher traffic. Unless you are working with very high volume of messages, the kafka would be an overkill.

Distribudion

RabbitMQ (in "new" version 3.7 or newer) consists of "durable" instances (working and saving data) and "working instances" (not saving data). Works well with 1+ instances. Works with exchanges and queues.

Kafka spreads all messages through the instances (works well with 3+ instances, preferably 5+). Can work with single instance (for development/test purpose). You must set "partitioning key" to guarantee that messages are processed in order (for this key). It is not a queue.

High availability

RabbitMQ works well but you should not assume that it is 100% available. Under some rare circumstances messages could be lost.

Kafka is using multiple servers and quorum. Much has changed between versions 1, 2 and 3. Internally Kafka 2 and 3 are fully transactional to ensure that no messages are lost. Its "exactly once delivery" ensures that a message is delivered at least once (to the library, not necessary processed) OR delivered at least once (be prepared for same messages delivered multiple times).
Kafka is using either zookeeper or its own kraft protocol (available since kafka 2.80).

Performance

Kafka can scale much - use it for REALLY high messages volume.

RabbitMQ works well with high load but expect dead-end when dealing with HUGE load.

Replication

In RabbitMQ messages gets copied, consumed or dequeued (when time to live reached). Or lost (worst nightmare as you have no idea what happened).

In Kafka messages are stored for some time, never lost. Even if consumed you can still access then to see what happened.
You can replicate all the messages to secondary data center (and delay is not an issue).

Multi subscriber

In RabbitMQ you must control who should receive the messages and handle when message is not consumed in time.

Kafka is just a journal of messages. Same messages can be consumed by many consumer groups and you can always re-consume them (unless timed out).

Message Protocols

RabbitMQ is a "Standard" in terms of protocol but it is quite complex (and examples and libraries are not easy to understand). If you dig into the libs then all are just barely proof-of-concept's.

With enterprise level kafka is easier to feel but you must feel also asynchronous message processing. Rabbit works well for synchronous one.

Message Ordering

RabbitMQ if an advanced queue. Message ordering is "guaranteed" (but messages not processed are re-queued what means they could get to the end of queue).

Kafka is NOT an queue but you can use partitioning key to ensure those are processed in order. Expect issues when your consumer is not responding and same messages are passed to another consumer. Expect issues when single partitioning keys contians VERY HIGH amount of messages (it could block an partition).

Message lifetime

In RabbitMQ message consumed is gone.

In Kafka message stays until removed.

Architecture

RabbitMQ is handling message delivery and logic "what to do if delivery fails". This pushes responsibility to the message producer (sender). Your service must handle multiple delivery failure scenarios.

Kafka is rather "fire ans forget". You produce a message and don't care if it is consumed by anyone. This helps is decoupling of services. You must handle problems in consumers so producers are more simple.

Use Cases

Kafka is meant for asynchronous message delivery (fire and forget).

RabbitMQ is a message send and handle delivery failures. You have better control over the delivery but you must handle all the scenarios.

Transactions

Kafka has transactions. RabbitMQ does not. If you want reliable transactions then more complex kafka can provide this.

Language

RabbitMQ is written in Erlang (older versions are using erlang config files).
Kafka is written in scala (based on java).

Routing Support

Both options are OK but handle your routing differently (kafka pushes it to the consumers while rabbit depends much on the producer)...

Developer Experience

Rabbit seems simple at first glance but fails when issues appear.

Kafka is much more challenging at first but then does not fail once you understand how it works. Ir requires much more knowledge and experience.

Kafka can be scaled to much higher volume.
Kafka requires more knowledge.
Kafka provides better disaster recovery.

Disk space

I decided to add this one extra as it is important but not straightforward to notice or understand.
Kafka keeps messages for a while, message is kept only once but can be consumed multiple times. It consumes much disk space because you keep them for a while just in case a consumer is not available for a while. Messages should be small (preferably kilobytes). Consumer outage is not that scary anymore. You often want to outsource Kafka management to a company focused on that (having know-how and well established tools).
RabbitMQ can handle larger messages but copies them for every subscriber. Once consumed, the message is gone. As a result only few messages are kept and you need less disk space. If message is not consumed in time, it gets to a dead letter queue. Consumer outage is a pain and often forces someone to handle it manually. Your nightmare is a message that gets consumed but not processed as you end up digging the logs. System management is often very bound to domain knowledge so outsourcing might be difficult.

How this document was created

It was created in relation to the https://dev.to/rakeshkr2/kafka-vs-rabbitmq-4ioj