DEV Community: Sergey Podgorny

💾 Why I Chose SQLite for My Startup — The Most Underrated Database You're Probably Ignoring

Sergey Podgorny — Mon, 20 Oct 2025 16:15:00 +0000

For years, I ignored SQLite. I assumed it was only good for toy apps, quick experiments, or maybe some desktop utilities. Like many developers, I immediately jumped to Postgres or MySQL for any "serious" project. I even paid for a managed AWS RDS instance, believing I was preparing for scale.

But over time, I learned something that changed how I build small and medium-sized systems:

👉 For 90% of applications, SQLite is not only enough — it's often better.

My entire application runs on a single VPS, serving just a few requests per second. Most startups never reach the mythical "millions of requests per minute". For this scale, running a full database server is like renting a truck to deliver a pizza.

🧩 Why SQLite Is So Powerful

SQLite is a serverless, file-based, self-contained database engine. It's just a single file (e.g., data.db) that your app reads and writes to — no network connections, no daemons, no setup.

Yet it supports:

Full ACID transactions
Foreign keys, indexes, views, triggers
Concurrency (especially in WAL mode)
Gigabytes of data and millions of rows

And it's blazing fast when used correctly.

⚡ The Secret Sauce: WAL Mode

By default, SQLite stores changes using a rollback journal — it writes updates directly to the main database file and keeps a small backup journal during each transaction for safety.

WAL (Write-Ahead Logging) changes this strategy completely.

Instead of touching the main DB directly, all writes are appended to a separate .db-wal file. Readers keep reading from the main file while writers add to the WAL log. Later, the changes are merged automatically.

That means:

Readers and writers don't block each other.
Concurrent operations become much faster.
Crashes don't corrupt your main data file.

WAL mode can significantly boost performance in multi-threaded or multi-request apps, like Go web servers, where reads and writes happen at the same time.

But if your app is tiny and writes only occasionally, the default mode (DELETE) is perfectly fine. You can always switch later with:

PRAGMA journal_mode = WAL;

Other journal modes include:

DELETE (default on many builds): makes a rollback file, then deletes it after writing. When you start a transaction, SQLite writes a rollback journal file (data.db-journal). After committing, it deletes that file. It's the safest and most widely compatible default. This is what you get if you don't set journal_mode at all.
TRUNCATE: same as DELETE, but instead of removing the journal file, it just truncates it to zero bytes and reuses it.
PERSIST: same idea, but keeps the journal file contents and just overwrites a header.
MEMORY: journal is only in RAM → faster, but data can be lost on crash.
OFF: no journaling at all → faster, but unsafe if the app or OS crashes, the database can be corrupted on crash.
WAL: the special one we just discussed.

⚙️ What Are PRAGMAs?

In SQLite, PRAGMAs are configuration switches that control database behavior — kind of like settings in a config file, but stored inside the DB engine itself.

You can enable them either by running SQL commands:

PRAGMA foreign_keys = ON;
PRAGMA synchronous = NORMAL;

Or pass them directly in the Go connection string:

dsn := "file:data.db?_pragma=busy_timeout(10000)&_pragma=foreign_keys(ON)"

Here are a few useful ones:

busy_timeout(10000) tells SQLite: if the database is locked, wait up to 10 seconds before giving up (instead of failing immediately with database is locked). Useful when multiple goroutines or processes may try to write at the same time. It is good for web apps; safer than retrying manually.
journal_mode(WAL) enables Write-Ahead Logging instead of the default rollback journaling. It makes reads/writes more concurrent.
journal_size_limit(200000000) sets a cap on the WAL file size (here: ~200 MB). Normally, the WAL file can grow until checkpointed (merged back into the main DB). This prevents it from growing forever. If your DB is small, it will never hit this size anyway.
foreign_keys(ON) enables foreign key constraints (OFF by default in SQLite). If you define relations like user_id REFERENCES users(id), this ensures referential integrity. It is always good practice if you use relationships.
temp_store(MEMORY) tells SQLite to keep temporary data (like for ORDER BY, GROUP BY, and indexes while querying) in RAM instead of on disk. Faster for queries, but uses more memory.
cache_size(-16000) sets the page cache size. The negative number means "KB" instead of pages. Here: -16000 = about 16 MB of cache. This is how much SQLite will keep in memory for speeding up queries.
synchronous(NORMAL) controls how careful SQLite is about flushing data to disk.

FULL (default): every write is forced to disk immediately → safest, but slowest.
NORMAL: doesn't flush on every step, but still durable enough for most apps (data is safe unless the OS/hardware crashes at a very unlucky moment).
OFF: fastest, but risk of corruption on crash.

ℹ️ NORMAL is a common compromise in web apps: faster inserts, still quite safe.

Not all are required — start simple and tune only when needed.

🧠 When You Should (and Shouldn't) Use SQLite

SQLite is perfect for:

Small to medium web apps
APIs on a single server
Prototypes, MVPs, side projects
Command-line or desktop tools
Local caching layers or analytics snapshots

But you'll hit limits if:

You need multiple servers writing to the same DB file
You expect thousands of writes per second
You require fine-grained access control
You need replication or clustering

That's when Postgres or MySQL makes sense. Until then, SQLite saves you time, money, and complexity.

🐹 Example: Using SQLite with Go

Let's set up a connection in Go using the excellent cgo-free driver modernc.org/sqlite:

$ go get modernc.org/sqlite

package main

import (
    "context"
    "database/sql"
    "fmt"
    "log"
    "time"

    _ "modernc.org/sqlite"
)

func main() {
    // Open a database file (or ":memory:" for in-memory)
    dsn := "file:data.db?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)"
    db, err := sql.Open("sqlite", dsn)
    if err != nil {
        log.Fatalf("open: %v", err)
    }
    // Configure pool: keep a small pool for sqlite; tune for your workload
    db.SetMaxOpenConns(10)  // readers can have several
    db.SetMaxIdleConns(5)
    db.SetConnMaxLifetime(0)

    // Ping to verify connection
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
    defer cancel()
    if err := db.PingContext(ctx); err != nil {
        log.Fatalf("ping: %v", err)
    }

    fmt.Println("DB open, WAL enabled and busy_timeout set (via DSN).")
}

SQLite allows many concurrent readers but only one writer at a time. A common pattern:

Use one DB instance restricted to a single writer (SetMaxOpenConns(1)) for all writes.
Use a separate DB instance with a larger pool for readers.

// writer
writerDSN := "file:test.db?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)"
writerDB, _ := sql.Open("sqlite", writerDSN)
writerDB.SetMaxOpenConns(1) // single writer connection
writerDB.SetMaxIdleConns(1)

// readers
readerDSN := "file:test.db?_pragma=busy_timeout(5000)"
readerDB, _ := sql.Open("sqlite", readerDSN)
readerDB.SetMaxOpenConns(20) // allow concurrent readers
readerDB.SetMaxIdleConns(10)

This reduces SQLITE_BUSY when multiple goroutines try to write. Practical guides and community posts recommend separating writer/reader pools.

🧭 Navigating SQLite from the Terminal

SQLite also comes with a lightweight console client: sqlite3.

Open your database:

sqlite3 data.db

Inside, these dot-commands make life easier:

.help             # list all available commands
.databases        # show loaded databases (usually just `main` pointing to your file)
.tables           # list all tables
.schema           # show schema of all tables
.schema table     # show schema of a specific table
.headers on       # turn on column headers in query output
.mode column      # format results in a nicely aligned table
.mode line        # show each row vertically, one field per line (great for wide tables)
.mode list         # results as plain text separated by | (default mode)
.width 20 30 15   # set column widths when using .mode column
.nullvalue NULL   # choose how NULLs are displayed
.quit or .exit    # leave the CLI

Example session:

sqlite> .headers on
sqlite> .mode column
sqlite> SELECT * FROM users;
id  email             created_at
--  ----------------  ---------------------
1   test@example.com  2025-10-07 14:30:00

Once you learn .headers on and .mode column, the CLI feels surprisingly pleasant.

If you prefer certain settings to be enabled by default, then it makes sense to customize the configuration file to suit your needs. If the initialization file ~/.sqliterc exists, sqlite3 will read it to set the configuration of the interactive environment. This file should generally only contain meta-commands.

.headers on
.mode column
.nullvalue NULL

🧩 Conclusion

SQLite is like the pocketknife of databases — tiny, portable, and incredibly capable when used properly. For many apps, it's a smarter default than running a full database server.

For Kinsly's current stage, SQLite is the perfect choice:

It's blazing fast, even on cheap VPS hardware
It simplifies deployment
It requires zero configuration
It saves money (no RDS bills)
It's easy to move or back up (just copy the file)

When the time comes to scale, migration will be straightforward.
Until then, SQLite lets me focus on what matters: building the product, not managing infrastructure.

So before spinning up another managed Postgres instance, consider starting with SQLite. You might be surprised how far it takes you.

🚀 How I Deployed My Startup's Server Without Kubernetes or Docker (Yet)

Sergey Podgorny — Tue, 07 Oct 2025 14:00:00 +0000

Almost every article these days talks about setting up Kubernetes clusters, writing Terraform scripts, or diving deep into AWS-managed services. While those tools are powerful, I believe that for most small projects, they are unnecessary overhead.

For Kinsly, my still-small project, I chose a simpler and more old-school approach. It's lighter, uses less memory, has fewer layers of abstraction, and remains secure. In this post, I'll walk through how I deployed my server, why I avoided AWS/GCP at this stage, and the exact setup I used, from SSH security to deployment automation.

🌍 Choosing the Right Infrastructure

For years, I favored DigitalOcean: simple, affordable, and reliable. But for this first phase, I went even more cost-effective and chose Contabo VPS. Although their SLA looks low on paper, performance in practice has been surprisingly stable.

Unlike AWS or GCP, where you can easily drown in unnecessary configuration, here I get just what I need: a straightforward server that I fully control. And importantly, I can move it anywhere at any time, without being locked into provider-specific services.

💡 Lesson: Keep deployments provider-agnostic so you can move quickly when needed.

🔐 Securing Access with SSH

The first thing I did was lock down access. By default, servers allow SSH login with a password. That's a major attack vector: bots constantly attempt brute-force logins with common credentials. An SSH key is far more secure.

Secure SSH key was created locally using this command:

$ ssh-keygen -t ed25519 -f ~/.ssh/deploy_server.pem -C "deploy@gitlab"

And then the local public key was uploaded to the server:

# This command is the easiest way to install your key.
$ ssh-copy-id -i ~/.ssh/deploy_server.pem deploy@[target-ip]

Next, I hardened the SSH configuration on the server.

$ sudo vim /etc/ssh/sshd_config

Inside this file, I made sure the following lines were set. This completely disables password logins and ensures only key-based access is allowed:

PermitRootLogin prohibit-password
# Disallow password authentication entirely.
PasswordAuthentication no
# Ensure public key authentication is enabled.
PubkeyAuthentication yes
# Also disable challenge-response auth, which can sometimes be a password fallback.
ChallengeResponseAuthentication no

Finally, I applied the changes by restarting the SSH service:

$ sudo systemctl restart sshd

From now on, only valid SSH keys can connect. This instantly shuts down most automated attacks.

⚠️ IMPORTANT: Before you close your current terminal session, open a new terminal and confirm you can still log in with your SSH key. If you don't, you could lock yourself out of your own server!

🌐 Cloudflare for Domains & Security

I registered my domain through Cloudflare. Unlike GoDaddy or Namecheap, Cloudflare sells domains at cost price (no markup) and forces you to use its DNS — this turned out to be a blessing. DNS hosting is free and comes with Cloudflare Proxy.

With just the free plan, I got:

Hidden server IP (Cloudflare Proxy)
Free DDoS protection
Global caching across regions for speed
Basic analytics
Built-in security filtering

👉 In the past, I even paid AWS money just to register DNS records. Cloudflare gives more for free.

⚔️ Hardening the Server Firewall

As soon as I installed nginx, bots started probing for vulnerabilities (checking for /wp-login.php, .git folders, etc.). To prevent direct access to my VPS, I restricted requests to Cloudflare IPs only.

To prevent this, ufw (Uncomplicated Firewall) method is the best practice. It provides the highest level of security and efficiency. It is a one-time setup, though you should plan to update the IP list once or twice a year, as Cloudflare occasionally adds new ranges.

First, always allow SSH, or you'll lock yourself out.

$ sudo ufw allow ssh

$ sudo ufw allow 22/tcp

Then allow traffic from all of Cloudflare's IPs on port 443 (HTTPS). You can get the latest lists from cloudflare.com/ips. You can run a simple loop to add all the current IPv4 and IPv6 addresses:

# For IPv4
$ for ip in $(curl -s https://www.cloudflare.com/ips-v4); do sudo ufw allow from $ip to any port 443 proto tcp; done

# For IPv6
$ for ip in $(curl -s https://www.cloudflare.com/ips-v6); do sudo ufw allow from $ip to any port 443 proto tcp; done

After specifically allowing Cloudflare IPs, you can now safely deny all other traffic on ports 80 and 443.

$ sudo ufw deny http
$ sudo ufw deny https

This forces all web traffic through Cloudflare's secure proxy, effectively locking the back door to the server.

Then, if it's not already enabled, turn on the firewall. It will ask for confirmation.

$ sudo ufw enable

Verify your rules are in place.

$ sudo ufw status verbose

You should see a list showing that port 22 is allowed from anywhere, port 443 is allowed from Cloudflare's IPs, and ports 80/443 are denied from anywhere else.

You can also do this within nginx itself, though it's slightly less efficient as nginx has to process the connection before denying it.

You would create a file (/etc/nginx/snippets/cloudflare-ips.conf) containing allow rules for all of Cloudflare's IPs, and then include it in your server block, followed by deny all;.

# IPv4 Ranges
allow 173.245.48.0/20;
allow 103.21.244.0/22;
allow 103.22.200.0/22;
allow 103.31.4.0/22;
allow 141.101.64.0/18;
allow 108.162.192.0/18;
allow 190.93.240.0/20;
allow 188.114.96.0/20;
allow 197.234.240.0/22;
allow 198.41.128.0/17;
allow 162.158.0.0/15;
allow 104.16.0.0/13;
allow 104.24.0.0/14;
allow 172.64.0.0/13;
allow 131.0.72.0/22;

# IPv6 Ranges
allow 2400:cb00::/32;
allow 2606:4700::/32;
allow 2803:f800::/32;
allow 2405:b500::/32;
allow 2405:8100::/32;
allow 2c0f:f248::/32;
allow 2a06:98c0::/29;

deny all;

This snippet will then need to be included separately in the corresponding server directive.

server {
    # ...
    include /etc/nginx/snippets/cloudflare-ips.conf;
    # ...
}

🏗️ A Simple, Minimal Stack

My architecture is intentionally simple: instead of React or Angular SSR, the first version of Kinsly runs on a static HTML page. Form submissions go to a small Go service that listens only on the local loopback interface (127.0.0.1) and it's not exposed to the outside world. Nginx acts as a reverse proxy, forwarding public requests from api.getkinsly.com to this private port. This is a secure and standard pattern.

I also considered using Unix sockets, but TCP on localhost was good enough.

Here's a look at the core of my Nginx config for the API:

# /etc/nginx/sites-available/api.getkinsly.com
server {
    server_name api.getkinsly.com;

    listen 443 ssl http2;
    # ... (all my SSL and security headers go here) ...

    location / {
        # These two lines enable efficient keep-alive connections to the backend.
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Pass essential client information to the Go application.
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # The actual proxy pass to the local Go service.
        proxy_pass http://127.0.0.1:8001;
    }
}

For now, Kinsly just needs to collect form data. Deploying a full Postgres or MySQL server at this stage would be pure over-engineering, so I deliberately chose SQLite - it's file-based, requires zero configuration, incredibly fast, and perfect for the simple task of collecting waiting list emails.

Later, when scaling requires it, migrating to Postgres or MySQL will be straightforward.

⚙️ Keeping the App Alive with systemd

My Go application runs as a systemd service, ensuring it's always on. The service is configured to run as the non-privileged www-data user for security.

Example service file (/etc/systemd/system/kinsly-api.service):

[Unit]
Description=Kinsly API Service
# Start this service only after the network is available.
After=network.target

[Service]
# The user and group the service will run as.
# Running as a dedicated, non-sudo user is a critical security best practice.
User=www-data
Group=www-data

# The command to start the application. Make sure the binary is executable (chmod +x binary)
ExecStart=/opt/kinsly/current/kinsly-api
Restart=always
# Wait 5 seconds before restarting to prevent rapid-fire restarts.
RestartSec=5

# Set environment variables if your application needs them.
Environment="APPLICATION_MODE=prod"

[Install]
# This allows the service to be enabled to start on boot.
WantedBy=multi-user.target

With this, if the app crashes or the server reboots, systemd automatically restarts it.

👤 Creating a Safe Deploy User

My GitLab CI/CD pipeline, however, connects as a separate deploy user that does not have full sudo access. This creates a problem: how does the CI/CD pipeline restart the service after a new deployment?

The solution is to grant the deploy user permission to run only that one specific command. This is done via the sudoers file.

The ONLY safe way to edit the sudoers file is with visudo. It performs a syntax check on save to prevent you from breaking sudo.

$ sudo visudo -f /etc/sudoers.d/deploy-app

I added this single line to the bottom of the file. This allows the deploy user to restart the kinsly-api service without a password.

deploy ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart kinsly-api.service

This is a secure and granular way to enable deployment automation without giving away root access. The CI/CD pipeline can now run sudo systemctl restart kinsly-api.service, and it's the only sudo command it's allowed to execute.

ℹ️ Make sure the directive @includedir /etc/sudoers.d is enabled in the main /etc/sudoers configuration file. If it's not there, you should add it using sudo visudo command (without -f argument).

Also, ensure correct file ownership & permissions:

$ sudo chown root:root /etc/sudoers.d/deploy-app
$ sudo chmod 0440 /etc/sudoers.d/deploy-app

Validate sudoers syntax, so you don't lock yourself out:

$ sudo visudo -c

🔄 Deployment via GitLab CI

Deployment is automated:

GitLab CI builds a Go binary.
It packages the binary + static HTML into a .tar.gz archive.
The archive is uploaded to the server and unpacks under new release folder /opt/kinsly/releases/<timestamp>.
A symlink current → <latest_release> is updated.
The service is restarted.
Old releases are pruned (keep the last 5 for rollback).

This is simple, reliable, and avoids the need for Docker at this stage.

🍪 Frontend Challenges: Analytics, Consent & Captcha

Even with a simple site, there are complexities. To use Google Analytics, I needed a cookie consent banner to comply with EU law. I chose axept.io, a free and Google-certified service.

I also added a free captcha (Altcha) to prevent bots from spamming my form with fake emails. Spam protection was essential. Without it, malicious bots could flood forms, causing emails to be marked as spam. While not perfect, it filters out the majority of malicious requests.

🛡️ Final Security Checks

Before going live, I checked open ports with nmap and online scanners to ensure only nginx was exposed. My Go app, database, and system processes remain inaccessible from outside.

$ nmap -sS -Pn -T5 -p- [target-ip]

✅ Conclusion

This setup may sound "basic" compared to a Kubernetes cluster on AWS, but for an early-stage project, it's exactly what I need:

Minimal cost
Maximum control
Easy portability
Strong enough security

As the project grows, I will likely containerize services and introduce Docker orchestration, but for now, this lean approach lets me move fast without unnecessary complexity.

It reminds me that sometimes the most elegant solution is the simplest one.

Will AI steal software engineering job?

Sergey Podgorny — Thu, 29 Feb 2024 11:08:00 +0000

The role of artificial intelligence (AI) in software engineering has sparked both curiosity and concern. The question on many minds is: Will AI steal software engineering jobs? Let's delve into this topic, explore the nuances, and decipher what the future might hold for software developers. Spoiler alert: it's not all doom and gloom.

The Power of AI in Code Generation

AI has proven its ability to generate code swiftly and efficiently. It may help developers to be more efficient and to contribute more features with less effort.

While concerns about job security arise, it's essential to note that the demand for skilled software engineers is far from diminishing. In fact, there's a surplus of open positions waiting to be filled, indicating that AI could complement, rather than replace, human developers.

Surviving Layoff Rollercoasters

Did you hear about those big tech layoffs recently? Yeah, it stung. But before you start blaming AI, let's think for a second. It happens every decade or so, and is likely influenced by external factors, such as the aftermath of the Covid-19 pandemic.

However, junior positions will most likely be more difficult to obtain, since AI solves many problems that have long been left to newcomers, but the overall demand for experienced developers remains high.

The Evolution of Open-Source Culture

Remember the dark ages when we had to code everything from scratch? Thanks to open source, there are millions of lines of code out there and the amount is growing. Before open-source culture became popular, all this code developers would need to write on their own, and now just need to use.

As a result, we do not have a decrease in the number of developers, but quite the opposite, there are more and more software developers.

However, the emphasis is shifting toward a deeper knowledge of programming and application architecture so that AI cannot easily replace the developer. Basic tasks may be automated, but developers with profound expertise will always be in demand. Creating HTML forms and being able to change the color of a button will soon not be enough.

The Fate of Stack Overflow

Will AI make platforms like Stack Overflow obsolete? Not quite. While AI may streamline certain processes, the essence of platforms like Stack Overflow as a community-driven hub for problem-solving is likely to persist. Trivial questions like "What's wrong with my code?" might decrease, but the platform will evolve into a more robust forum, fostering in-depth discussions and collaborative problem-solving.

Prompt Engineering and Programming Languages

Prompt engineering sounds like the cool kid in town, right? It is a rising trend, that could redefine certain developer roles, especially in content management system (CMS) development. However, it's unlikely to replace programming languages entirely. I believe that prompt engineering will become a skill necessary for a good and effective developer.

The industry's shift towards languages with simpler syntax, such as Go, Kotlin, and Dart, doesn't signal the demise of traditional languages. Instead, it emphasizes the importance of staying adaptable and acquiring diverse skill sets.

The Future of Software Developers and Conclusion

In conclusion, the fear of being replaced by AI is not unfounded, but it's crucial to recognize the positive aspects. The role of a software developer is unlikely to vanish in the next 5-10 years. To stay relevant, developers should aspire to be as proficient as machines while possessing in-depth knowledge of algorithms, data structures, and system design. Soft skills, including effective communication and understanding of business needs, will remain pivotal for success in the evolving landscape of software engineering.

Change is inevitable, and AI is part of our coding journey. Let's roll with it, stay curious, and keep our dev skills sharp. The future's uncertain, but one thing's for sure: we're not going anywhere if we embrace the change and bring our A-game. 💻✨

How We Transformed Our Daily Meetings for the Better

Sergey Podgorny — Mon, 13 Nov 2023 10:00:17 +0000

In the world of teamwork, our daily stand-ups play a crucial role in our collective success. We wanted to make things more efficient, and that's when we found something awesome in the videos from "Development That Pays": Walking the Board.

The Problem

We've all been there: the endless cycle of daily stand-ups where the team updates on what they did yesterday, what they're working on today, and any blockers they're facing.

While it seems like a straightforward approach, it has its drawbacks. The individual spotlight often overshadows the collaborative nature of the team event, with team members more focused on crafting their own stories than actively listening to others. This realization prompted us to seek a more effective alternative.

The Discovery

While browsing YouTube, I discovered "Development That Pays". It proved to be a goldmine of helpful videos. Two videos, in particular, caught my attention: Daily Stand-up: You're Doing It Wrong! and Agile Daily Standup - How To Walk the Board (aka Walk the Wall). These videos challenged our traditional approach and introduced an alternative — Walking the Board.

The "Walking the Board" Concept

Walking the Board, also known as Walking the Wall, offered a refreshing perspective. Instead of individual updates, the team starts with the ticket on the top right of the board. The team member assigned to that ticket provides an update, and we move on to the next ticket. This method shifts the focus from individuals to the work itself and changes the stand-up from a chain of individual updates to a team event.

This approach transforms the stand-up into a collaborative journey across the board, ensuring that the spotlight remains on the work itself, not on individual narratives. No more struggling to come up with a good story, the cards on the board provide the agenda. It's a game-changer that keeps everyone focused on the work. It's a shift from "What did I do?" to "What is the status of the work on the board?"

How To Walk the Board

Starting at the top right of the board makes financial sense — the items closest to being live are discussed first. If you're familiar with the concept of net present value, you'll understand that income now is more valuable than income later, and income tomorrow is more valuable than income next week.

The second reason for starting at the right is purely practical: we are going to move the cards across the board from left to right. By starting at the right, we create space for cards to move into.

Also, the "Development That Pays" video emphasized the importance of moments of glory — allowing team members to move their own cards and take pride in their progress. It's not just about updating the board; it's about actively participating in the collective journey.

At the end of the meeting, the board and the team's understanding of its current "shape are up to date. Team members can also share non-board related topics or updates, fo example tasks not listed on the board.

Success Story: Implementing Walking the Board

Inspired by these videos, we decided to give "Walking the Board" a shot. The transformation was remarkable! Our daily stand-ups became more than just updates — they turned into collaborative sessions centered around the work on the board.

ELMO Rule

To keep discussions concise, we introduced the "ELMO" rule. ELMO stands for Enough, Let's Move On. "ELMO" is a word that the guide and travelers may use to indicate a conversation is either off-topic or takes too long. If a discussion is going off track, anyone can simply say "ELMO". This signals that we'll discuss it later, outside our daily stand-up.

Secret Order

We established a specific order for leading the board and other meetings. This system not only enhances a sense of responsibility but also encourages shared leadership among the team.

The Result and Summary

Since adopting Walking the Board in the summer of 2020, our meetings have changed a lot. We have shifted away from giving individual updates. Instead, our focus is entirely on the work on the board. This change has made our stand-ups more productive and collaborative, as we're now centered on the tasks at hand rather than individual narratives.

We switched from traditional stand-ups to "Walking the Board" because we wanted our meetings to be more efficient. The videos from "Development That Pays" played a key role in inspiring this change, showing us what wasn't working with the old way and the benefits of a more team-focused approach. Now, Walking the Board is a regular part of our daily routine, making our meetings more focused, productive, and collaborative. If you're looking to improve your stand-ups, the insights from "Development That Pays" are definitely worth exploring.

Migrating from Self-Managed RabbitMQ to Cloud-Native AWS Amazon MQ: A Technical Odyssey

Sergey Podgorny — Sat, 14 Oct 2023 15:04:00 +0000

In the ever-evolving world of cloud-native solutions, it can be a daunting task to maintain message brokers. For a while, our team was responsible for a self-managed RabbitMQ instance. While this worked well initially, we encountered challenges in terms of maintenance, version updates, and data recovery. This led us to explore Amazon MQ, a fully managed message broker service offered by AWS.

In this article, we'll discuss the advantages of both self-managed RabbitMQ and Amazon MQ, the reasons behind our migration, and the hurdles we faced during the transition. Our journey offers insights for other developers, who consider a similar migration path.

The Self-Managed RabbitMQ Era

Our experience with self-managed RabbitMQ was characterized by control, high availability, and the responsibility to ensure data integrity. Here are some of the advantages of this approach:

Total Control
Running your own RabbitMQ server gives you complete control over configuration, security, and updates. You can fine-tune the setup to meet your specific requirements: ideal for organizations with complex or unique messaging needs.
High Availability
It's worth noting that our entity was running on AWS EC2, whose SLA guarantees only 99.99%, but de-facto we achieved a remarkable uptime rate of 99.999% with our self-managed RabbitMQ setup. The downtime was almost non-existent, which ensured a reliable message flow through our system. High availability is crucial for many mission-critical applications.
Data Recovery
Ironically, data recovery was a challenge with our self-managed RabbitMQ. In the event of a crash, we lacked confidence in our ability to restore data fully. This vulnerability urged us to consider Amazon MQ, a fully managed solution.

The Shift to Amazon MQ

As time passed, it became apparent that managing RabbitMQ was no longer sustainable for our team. Here are the primary reasons that drove us to explore Amazon MQ as an alternative:

Skills Gap
Our team lacked in-house experts dedicated to managing RabbitMQ, which posed a risk to our operations. As RabbitMQ versions evolved, staying up-to-date became increasingly challenging. This skill gap urged us to consider Amazon MQ, a fully managed solution.
AWS Integration
As an AWS service, Amazon MQ seamlessly integrated with our existing AWS infrastructure, providing us with a more cohesive and consistent cloud environment. It allowed us to leverage existing AWS services and tools, which resulted in a smooth migration process.
Managed Service
The promise of offloading the operational burden to AWS was enticing. Amazon MQ handles tasks like patching, maintenance, and scaling. This allows our team to focus on more strategic initiatives.
Enhanced Security
One key advantage of switching to AmazonMQ is its strong foundation on AWS infrastructure. This not only ensures robust security practices but also regular updates are integrated into the system. So, it gives us confidence, as we know that any potential vulnerabilities are under active monitoring and management.

The Amazon MQ Experience

While the move to Amazon MQ presented numerous benefits, we also encountered some challenges that are worth noting:

SLA Guarantees
Amazon MQ's service level agreement (SLA) guarantees 99.9% availability. This is generally acceptable for many businesses but was a step down from our self-managed RabbitMQ's 99.999% uptime. While the difference might seem small, it translated into more downtime. A trade-off we had to accept.
Limited Configuration
Amazon MQ abstracts many configuration details. This simplifies management for the most users. However, this simplicity comes at the cost of fine-grained control. For organizations with highly specialized requirements, this might be a drawback.
Cost Considerations
Amazon MQ is a managed service, which means there are associated costs. While the managed service helps reduce operational overhead, it's crucial to factor in the cost implications when migrating.

What do three nines (99.9) really mean?

Here are my calculations according to Amazon MQ SLA:

if the monthly downtime is lower than ~43 minutes, they will charge 100% of the costs
if the monthly downtime is between ~43 minutes to ~7 hours, they will charge 90% of the costs of this downtime
if the monthly downtime is between ~7 hours to ~1day, they will charge 75% of the costs of this downtime
and if the monthly downtime is higher than ~1 day, they won't charge any costs for this downtime

Conclusion

Our migration from self-managed RabbitMQ to Amazon MQ represented a shift in the way we approach message brokers. While Amazon MQ offered many benefits, such as reduced operational burden and seamless AWS integration, it came with some trade-offs, including a lower SLA guarantee and less granular control.

Ultimately, the decision to migrate should be based on your organization's specific needs, resources, and objectives. For us, the trade-offs were acceptable given the advantages of a managed service within our AWS ecosystem.

The path to a cloud-native solution isn't always straightforward, but it can lead to more streamlined operations and a greater focus on innovation rather than infrastructure management. Understanding the pros and cons of both approaches is vital for an informed decision about your messaging infrastructure.

As technology continues to evolve, it's essential to stay adaptable and leverage the right tools and services to meet your business needs. In our case, the migration to Amazon MQ allowed us to do just that.

Understanding GraphQL: A Guide for Backend Developers

Sergey Podgorny — Wed, 11 Oct 2023 09:56:00 +0000

GraphQL is a powerful query language for APIs that provides a flexible and efficient way to request and manipulate data from a server. If you're a backend developer looking to learn about GraphQL, this guide will walk you through the basics, implementation, and its relevance to your work.

What Is GraphQL?

GraphQL was developed by Facebook and has gained popularity as an alternative to RESTful APIs. It provides the following key features:

Query Language:
GraphQL uses a schema to define the types of data available and the relationships between them. Clients can send queries to request exactly the data they need, and they receive a JSON response containing only that data.
Single Endpoint:
GraphQL typically has a single endpoint for all data operations, simplifying the API surface and allowing clients to request various data in a single request.
Strongly Typed:
GraphQL APIs are strongly typed, meaning they have a defined schema with clear data types. Clients can introspect the schema to understand what data is available and how to query it.
Real-time Data:
GraphQL can be used to request real-time data through subscriptions, allowing clients to receive updates when data changes.

Implementing GraphQL

Here's a step-by-step guide to implementing GraphQL in your backend:

Define a Schema:

Create a schema that describes the types of data your API will expose. You'll specify object types, queries, mutations (for write operations), and potentially subscriptions.

type User {
  id: ID!
  name: String
  email: String
}

type Query {
  getUser(userId: ID!): User
}

type Mutation {
  updateUser(userId: ID!, input: UserInput): User
}

Set Up Resolvers:

For each field in your schema, implement resolver functions. These resolver functions determine how to fetch the data for each field. These can be as simple as fetching data from a database or involve more complex operations.

const resolvers = {
  Query: {
    getUser: (parent, args, context, info) => {
      // Fetch user data based on 'args.userId'
      // Return the user data
    },
  },
  Mutation: {
    updateUser: (parent, args, context, info) => {
      // Update user data based on 'args.userId' and 'args.input'
      // Return the updated user data
    },
  },
  User: {
    name: (user) => user.name, // Resolver for the "name" field
    email: (user) => user.email, // Resolver for the "email" field
  },
};

Middleware:

Set up middleware for handling tasks like authentication, authorization, and caching.

Server Implementation:

Choose a GraphQL server library or framework that fits your tech stack. Popular options include Apollo Server, Express GraphQL, and GraphQL Yoga for JavaScript/Node.js.

Testing and Documentation:

Write unit tests for your resolvers and document your GraphQL schema so that clients can understand how to use it.

Integration with Frontend:

Implement the GraphQL client on your frontend to send queries and mutations to the GraphQL server.

Resolvers in GraphQL

Resolvers are central to GraphQL, as they determine how to fetch the data for specific fields. Each field in your schema typically has its resolver function. These resolver functions can optimize data fetching.

If multiple fields can be efficiently fetched in a single query from the same data source, you can use a single resolver to retrieve all the necessary data.
GraphQL clients request multiple fields in a single query, and the resolver functions are invoked only for the fields requested in that query.
Resolvers allow you to tailor the database query to fetch only the data that corresponds to the requested fields.

Do You Need GraphQL?

The decision to implement GraphQL depends on your project's requirements. You might benefit from GraphQL if:

Your project involves complex data fetching, with data from multiple sources.
You want to avoid versioning your API, as GraphQL allows clients to request only the data they need.
Real-time data updates are necessary through subscriptions.
Frontend and backend teams work in parallel, as GraphQL enables frontend developers to specify their data requirements.
Mobile apps are part of your project, as GraphQL's ability to fetch only the necessary data is advantageous for mobile apps.

However, if your project has simple data retrieval needs or is small in scope, you may not necessarily need to implement GraphQL.

Conclusion

GraphQL is a powerful tool for backend developers to create flexible and efficient APIs. Its ability to allow clients to request exactly the data they need and its strong typing make it a valuable addition to your API toolkit. By understanding GraphQL's principles and implementing it effectively, you can improve data retrieval and enhance your development process.

Start exploring GraphQL today, and you'll find it to be a valuable addition to your backend development toolkit.

10 Lesser-Known Tools and Websites to Spice Up Your Developer Toolbox

Sergey Podgorny — Tue, 10 Oct 2023 14:51:00 +0000

As a software engineer, you're no stranger to the tried-and-true tools and platforms that are essential for your day-to-day work. But what if you're looking for something a bit more unconventional, something to add a touch of creativity or a spark of inspiration to your coding journey? Look no further! We've curated a list of 10 lesser-known tools and websites that can make your job more interesting and enjoyable.

1. Carbon

Carbon allows you to create stunning and customizable code screenshots with syntax highlighting. Whether you want to share code snippets on social media or enhance your documentation, Carbon is a handy tool to have in your arsenal.

2. DevHints

DevHints is your cheat sheet and quick reference repository for various programming languages, frameworks, and tools. It's the perfect resource for quick syntax lookups without the need to dive deep into documentation.

3. RegExr

RegExr simplifies working with regular expressions. This online tool provides a visual interface for building and testing regex patterns in real-time, making regex less intimidating.

But that's not all! Here are a couple of bonus resources to complement your regex journey:

- Regexper

Regexper takes your regular expressions to the next level. It generates interactive, visually appealing diagrams that help you understand your regex patterns. With Regexper, you can see your regex patterns come to life, making complex expressions easier to grasp.

- iHateRegex

iHateRegex is your ally in conquering regular expressions. It offers a collection of regex patterns for common use cases and provides explanations and examples for each. Whether you're a regex novice or a seasoned pro, iHateRegex can save you time and frustration by offering pre-built solutions and guidance.

With RegExr, Regexper, and iHateRegex in your toolkit, you'll have everything you need to master regular expressions efficiently and effectively.

4. Excalidraw

Excalidraw is a collaborative virtual whiteboard where you can sketch diagrams, flowcharts, and wireframes. It's perfect for brainstorming and discussing design ideas with your team.

5. JQ

JQ is a lightweight and powerful command-line JSON processor. It's a time-saving tool for manipulating and extracting data from JSON files effortlessly.

6. Quicktype

Quicktype automates the generation of code from JSON data. It's a real timesaver when dealing with complex JSON structures in your applications.

7. Wappalyzer

Wappalyzer is a browser extension that identifies the technologies and frameworks used on a website. It's a valuable tool for competitive analysis and staying up-to-date with web technologies.

8. Hacker Typer

Hacker Typer is pure fun! It simulates typing like a Hollywood hacker. Use it to entertain your colleagues during presentations or meetings.

9. The Useless Web

The Useless Web takes you on a journey to random, quirky, and entirely useless websites. It's a delightful way to take a quick break and have a laugh when you need to reset your mind.

10. A Soft Murmur

A Soft Murmur lets you create custom ambient sounds to improve your focus or relax while working. Mix sounds like rain, thunder, and birdsong to create your perfect background noise.

Bonus Tools:

Lorem Ipsum

Lorem Ipsum is the classic Lorem Ipsum text generator, perfect for generating placeholder text in your projects.

You may also want to consider alternatives:

Bacon Ipsum adds a delicious twist to placeholder text. Instead of the standard Lorem Ipsum, it generates text with a meaty theme. If you're feeling hungry for creative content, give Bacon Ipsum a try.
Cupcake Ipsum serves up a sweet alternative to Lorem Ipsum. It generates text with a confectionery theme, perfect for adding a touch of whimsy to your design mockups. Cupcake Ipsum is a delightful way to sprinkle some fun into your projects.

With Lorem Ipsum, Bacon Ipsum, and Cupcake Ipsum at your disposal, you can choose the perfect placeholder text to match the mood of your design or project.

Lorem Picsum

Lorem Picsum provides random placeholder images, adding visual appeal to your designs.

Summary

These tools and websites may not be the most conventional choices for developers, but they can certainly add a touch of creativity, convenience, or fun to your work. So go ahead, explore, and enjoy the journey of discovering new and exciting ways to enhance your development experience!

CPU Cache Basics

Sergey Podgorny — Mon, 09 Oct 2023 09:21:00 +0000

CPU caches are the unsung heroes of modern computing, silently speeding up your computer's performance. These small but incredibly fast memory storage areas play a vital role in ensuring that your CPU can access frequently used data and instructions with lightning speed. In this article, we'll explore the world of CPU caches, uncovering their design, optimization strategies, and their indispensable role in enhancing the performance of software and systems.

Here's what you should know as a software engineer:

L1 Cache (Level 1 Cache):

L1 cache is the smallest but fastest cache located closest to the CPU cores.
It's divided into two parts: L1i (instruction cache) and L1d (data cache). L1i stores instructions, and L1d stores data.
The purpose of L1 cache is to store the most frequently used instructions and data to speed up the CPU's operations.
It has low latency (the time it takes to access data) and is usually separate for each CPU core.

L2 Cache (Level 2 Cache):

L2 cache is larger than L1 cache but slightly slower.
It is shared among CPU cores in many multi-core processors.
Its role is to store additional frequently used data and instructions that couldn't fit in L1 cache.
L2 cache is still faster than accessing data from RAM (main memory).

L3 Cache (Level 3 Cache, if available):

L3 cache is even larger but slightly slower than L2 cache.
It is shared across all CPU cores in a multi-core processor.
L3 cache acts as a backup storage for frequently used data and instructions that couldn't fit in L1 or L2 cache.
Having an L3 cache can help reduce bottlenecks when multiple CPU cores are accessing memory simultaneously.

How it works:

When the CPU needs data or instructions, it first checks if they are in the L1 cache.
If the needed information is in the L1 cache (or L2 cache if not found in L1), it's called a cache hit, and the CPU can quickly retrieve it.
If the data is not in the cache, it's called a cache miss. In this case, the CPU has to fetch the data from slower main memory (RAM), which takes more time.
The goal of the cache hierarchy is to reduce the number of cache misses by storing the most frequently used data and instructions in the faster, smaller caches.

When it is used:

L1, L2, and L3 caches are used constantly as the CPU executes programs.
They are especially beneficial for speeding up frequently executed code and data access patterns.
The cache hardware manages what gets stored in the cache, so as a software engineer, you generally don't interact with it directly.

Optimization principles

Caches like L1, L2, and L3 are managed by the hardware, and as a software engineer, you don't have direct control over which programs or data are stored in them. However, you can follow certain programming and optimization principles to increase the likelihood that your program benefits from cache usage. Here's how:

Locality of Reference: Caches work best when your program exhibits good locality of reference. There are two types of locality:
- Temporal Locality: This means that if you access a piece of data once, you're likely to access it again in the near future. To leverage temporal locality, try to reuse data that you've recently accessed.
- Spatial Locality: This refers to the tendency to access data located near recently accessed data. To benefit from spatial locality, try to access data in a sequential or predictable pattern.
Cache-Friendly Data Structures: Use data structures and algorithms that are cache-friendly. For example, when iterating over an array, processing elements that are stored close to each other in memory is more cache-efficient than jumping around in memory.
Cache Line Awareness: Cache systems typically work with fixed-size cache lines (e.g., 64 bytes). Be aware of this when designing your data structures. If you only need a small portion of a cache line, avoid loading the entire line to reduce cache pollution.
Compiler and Compiler Flags: Compilers can optimize code to improve cache locality. Use compiler flags (e.g., -O2 or -O3 in GCC) to enable optimizations. Additionally, understand how your compiler optimizes code for your target architecture.
Profiling and Benchmarking: Use profiling tools to analyze cache behavior in your program. Tools like perf (on Linux) or performance analyzers in integrated development environments (IDEs) can help you identify cache-related issues.
Thread Affinity: If you're working with multi-threaded programs, consider using thread affinity techniques to bind threads to specific CPU cores. This can help minimize cache contention between threads.

Cache sizes

Regarding the sizes of cache levels, they can vary widely depending on the CPU architecture. However, here's a rough estimate:

L1 Cache: Typically ranges from 16KB to 128KB per core.
L2 Cache: Can range from 256KB to 1MB per core or be shared among multiple cores.
L3 Cache: Usually shared among multiple cores and can range from 2MB to 32MB or more in high-end processors.

Keep in mind that these numbers can change with different CPU models and generations. You can usually find the specific cache sizes for your CPU in its documentation or by checking the manufacturer's website. Understanding cache sizes can help you make informed decisions when optimizing your code for specific hardware.

Cache latencies

Let's compare the latencies of different memory levels, including CPU caches and RAM (main memory):

L1 Cache Latency:

L1 cache is the fastest and has the lowest latency among all memory levels.
Typical latency ranges from 1 to 3 cycles, which is extremely fast.
Accessing data from L1 cache is significantly faster than any other memory level.

L2 Cache Latency:

L2 cache has slightly higher latency compared to L1 cache.
Typical latency ranges from 4 to 10 cycles, depending on the CPU architecture.
It is still much faster than accessing RAM.

L3 Cache Latency:

L3 cache has higher latency compared to L2 and L1 caches.
Typical latency ranges from 10 to 40 cycles, depending on the CPU and cache design.
While slower than L1 and L2, it is still much faster than RAM.

RAM (Main Memory) Latency:

Accessing data from RAM is significantly slower than accessing any level of cache.
RAM latency can vary widely, but it typically ranges from 60 to 100 cycles or more.
RAM access times are several orders of magnitude slower than L1 cache.

Assuming a CPU clock speed of 3 GHz (3 billion cycles per second):

L1 cache access time:
- Fastest case (1 cycle): 1 / 3 * 10^9 = 0.33 nanoseconds
- Slowest case (3 cycles): 3 / 3 * 10^9 = 1 nanosecond
L2 cache access time:
- Fastest case (4 cycles): 4 / 3 * 10^9 = 1.33 nanoseconds
- Slowest case (10 cycles): 10 / 3 * 10^9 = 3.33 nanoseconds
L3 cache access time:
- Fastest case (10 cycles): 10 / 3 * 10^9 = 3.33 nanoseconds
- Slowest case (40 cycles): 40 / 3 * 10^9 = 13.33 nanoseconds
RAM access time:
- Fastest case (60 cycles): 60 / 3 * 10^9 = 20 nanoseconds
- Slowest case (100 cycles): 100 / 3 * 10^9 = 33.33 nanoseconds

To put these numbers into perspective, accessing data from L1 cache can be over 10 times faster than accessing the same data from L2 cache, and it can be more than 100 times faster than accessing it from RAM.

Efficient use of CPU caches is crucial for optimizing software performance because minimizing cache misses and utilizing cache-friendly algorithms can help reduce the impact of slower RAM access times. This is why understanding cache behavior and optimizing for cache locality is a key consideration in high-performance computing and software development.

Summary

The importance of CPU caches in modern computing cannot be overstated. These small, high-speed memory storage areas play a pivotal role in enhancing software performance by reducing the latency of data access. Cache-aware programming, which involves optimizing code and data structures to maximize cache utilization, has a profound impact on software performance.

In summary, caches like L1, L2, and L3 are crucial for optimizing CPU performance by reducing memory access times. As a software engineer, understanding the basics of how caches work can help you write more efficient code, such as optimizing data access patterns and minimizing cache thrashing (when cache contents change frequently). However, the specifics of cache management are typically handled by the hardware and the operating system.

Dealing with RabbitMQ exchange types

Sergey Podgorny — Tue, 20 Apr 2021 14:53:05 +0000

Exchanges control the routing of messages to queues. Each exchange type defines a specific routing algorithm which the server uses to determine which bound queues a published message should be routed to.

RabbitMQ provides four types of exchanges: Direct, Fanout, Topic, and Headers.

Fanout Exchanges

The Fanout exchange type routes messages to all bound queues indiscriminately. If a routing key is provided, it will simply be ignored. The following illustrates how the fanout exchange type works:

The Fanout exchange type is useful for facilitating the publish-subscribe pattern. When using the fanout exchange type, different queues can be declared to handle messages in different ways. For instance, a message indicating a customer order has been placed might be received by one queue whose consumers fulfil the order, another whose consumers update a read-only history of orders, and yet another whose consumers record the order for reporting purposes.

Direct Exchanges

The Direct exchange type routes messages with a routing key equal to the routing key declared by the binding queue. The following illustrates how the direct exchange type works:

The Direct exchange type is useful when you would like to distinguish messages published to the same exchange using a simple string identifier. This is the type of exchange that was used in our Hello World example. As discussed in part 3 of our series, every queue is automatically bound to a default exchange using a routing key equal to the queue name. This default exchange is declared as a Direct exchange. In our example, the queue named "hello-world-queue" was bound to the default exchange with a routing key of "hello-world-queue", so publishing a message to the default exchange (identified with an empty string) routed the message to the queue named "hello-world-queue".

Topic Exchanges

The Topic exchange type routes messages to queues whose routing key matches all, or a portion of a routing key. With topic exchanges, messages are published with routing keys containing a series of words separated by a dot (e.g. "word1.word2.word3"). Queues binding to a topic exchange supply a matching pattern for the server to use when routing the message. Patterns may contain an asterisk ("*") to match a word in a specific position of the routing key, or a hash ("#") to match zero or more words. For example, a message published with a routing key of "honda.civic.navy" would match queues bound with "honda.civic.navy", "*.civic.*", "honda.#", or "#", but would not match "honda.accord.navy", "honda.accord.silver", "*.accord.*", or "ford.#". The following illustrates how the fanout exchange type works:

The topic exchange type is useful for directing messages based on multiple categories (e.g. product type and shipping preference), or for routing messages originating from multiple sources (e.g. logs containing an application name and severity level).

Headers Exchanges

The Headers exchange type routes messages based upon a matching of message headers to the expected headers specified by the binding queue. The headers exchange type is similar to the topic exchange type in that more than one criteria can be specified as a filter, but the headers exchange differs in that its criteria is expressed in the message headers as opposed to the routing key, may occur in any order, and may be specified as matching any or all of the specified headers. The following illustrates how the headers exchange type works:

The Headers exchange type is useful for directing messages which may contain a subset of known criteria where the order is not established and provides a more convenient way of matching based upon the use of complex types as the matching criteria (i.e. a serialized object).

Conclusion

That wraps up our introduction to each of the exchange types. Next time, we’ll walk through an example which demonstrates declaring a direct exchange explicitly and take a look at the push API.

Interesting in reading more about digital literacy? Follow this link to find more articles 😉

Event-driven architecture over standard client-server aproach

Sergey Podgorny — Fri, 16 Apr 2021 14:32:08 +0000

Just 20 years ago, most of the business was offline and web development was just starting to gain traction. But gradually everything develops, the whole business is trying to go online. And along with it, web development is actively developing, there are more and more services. Web development is an integral part of any modern business.

To keep our business running fast, we need to consider all the third-party services on which our business depends. First, let's take a look at the standard architecture we're used to.

Standard Client-Server architecture

With a standard synchronous architecture, most changes have a large number of dependencies, sometimes on slow and unpredictable systems. This slows down interactions and increases the error rate. Maintainability is troublesome, as there is usually a high amount of possible side effects to consider.

👍 Advantages:

Simple to understand
Easier to debug and test
Better data consistency

👎 Disadvantages:

High rate of client-visible errors
Slow
One thing fails, everything fails - does not work well for complex business processes

Event-Driven architecture

In an event-driven architecture, when a service performs some piece of work that other services might be interested in, that service produces an event — a record of the completed action. Other services consume those events so that they can perform any of their tasks needed as a result of the event.

By decoupling the core change from the side effects performance is greatly increased and error rates go down.

Secondary concerns can be handled independently, this allows easier maintenance, and even involving different teams and/or technologies that are best suited for the respective tasks.

👍 Advantages:

Responsive
Resilient
Scales as a whole as well as only where needed
Development can be distributed between multiple teams

👎 Disadvantages:

Extra work needed to keep data consistency
Higher learning curve

Summary

Event-driven architecture has been gaining momentum in popularity lately, and there are reasons for this: services grow and develop, become larger and acquire new dependencies. Decoupling of responsibility can be a solution in this case and improve performance, but you shouldn't abuse the architecture without a detailed understanding of all its pros and cons. For small and simple projects, a standard synchronous architecture may be a much better solution.

To read more, follow these links:

Interesting in reading more about APIs? Follow this link to find more articles 😉

Top commands I am using in my daily work

Sergey Podgorny — Wed, 09 Dec 2020 15:37:17 +0000

Nowadays developers should handle a lot of different operations, especially on their own machine. To simplify the work and monitoring of processes in the system and in the network, I picked up several tools to make work in the terminal more comfortable.

Disclaimer: this article is not about my comprehensive list of tools I am using every day, it is about useful top-like tools that help me do my job better and improve performance.

`htop`: Linux Process Monitoring

htop is a much advanced interactive and real-time Linux process monitoring tool. This is much similar to Linux top command, but it has more features like user-friendly interface to manage processes, shortcut keys, vertical and horizontal view of the processes and much more. htop is a third-party tool and isn't included in Linux systems, so you need to install it manually.

Features

it shows a frequently updated list of the processes running on a computer, normally ordered by the amount of CPU usage;
it provides a full list of processes running, instead of the top resource-consuming processes;
it uses colour and gives visual information about processor, swap and memory status;
it can also display the processes as a tree;
it provides a convenient, visual, cursor-controlled interface for sending signals to processes.

`ctop`: interface for container metrics

ctop provides a concise and condensed overview of real-time metrics for multiple containers, as well as a single container view for inspecting a specific container.

It mostly works same as docker stats, but has a better interface and single container view and logging pages.

Features

it shows a frequently updated list of the containers with metrics running on a computer;
it comes with built-in support for Docker and runC;
it uses colour and gives visual information about CPU, memory and network usage;
it shows logs per instance;
it has a full-screen, menu-driven operation.

`mytop`: real-time threads and performance monitoring tools

mytop is an open-source, command-line tool used for monitoring MySQL performance. mytop connects to a MySQL server and periodically runs the show processlist and show global status commands. It then summarizes the information in a useful format. Using mytop, we can in real-time monitor MySQL threads, queries, and uptime as well as see which user is running queries on which database, which are the slow queries, and more. All this information can be used to optimize the MySQL server performance.

Features

mytop provides a command-line shell interface to monitor real-time MySQL/MariaDB threads, queries per second, process list and performance of databases and gives an idea for the database administrator to better optimize the server to handle the heavy load.

mytop display screen is really broken into two parts. The top 4 lines (header) contain summary information about your MySQL server:

The first line identified the hostname of the server and the version of MySQL it is running. The right side shows the uptime of the MySQL server process in days+hours:minutes:seconds format (much like FreeBSD's top) as well as the current time.
The second line displays the total number of queries the server has processed, the average number of queries per second, the real-time number of queries per second, and the number of slow queries.
The third line deals with threads. Versions of MySQL before 3.23.x didn't give out this information, so you'll see all zeros.
And the fourth line displays key buffer efficiency (how often keys are read from the buffer rather than disk) and the number of bytes that MySQL has sent and received.

The second part of the display lists as many threads as can fit on screen. By default, they are sorted according to their idle time (least idle first).

Interesting in reading more about Development Tools? Follow this link to find more articles 😉

How and why we updated RabbitMQ queues on production

Sergey Podgorny — Sat, 05 Dec 2020 13:16:49 +0000

In this article, I would like to share with you and the whole internet our experience of dealing with RabbitMQ Live updates. You will learn some details about our architecture and use cases. Let's start from the simplest... Why do we need RabbitMQ in our business?

Our Architecture

As a health insurance company, our business depends on many different third-party services to analyze risks, process claimable documents, charge monthly payments etc. All these processes take some time to be processed, so to keep our services fast and autonomous from each other, we are using asynchronous processing of tasks that can be done in the background. This approach speeds up responses and allows to do more in the background, ie. email sending, policy creation, acceptance verification etc.

Whenever a client expresses some intent to the API by making a request to it, this intent can create follow-up tasks. These tasks do not need to be handled synchronously, i.e. they do not need to be handled while processing the initial request. Instead, we put a message about this intent onto the message queue where it can be picked up asynchronously by another process and handled independently from the original request.

Problem

But with great opportunities comes great responsibility. Message processing is very important and critical for our business. Some messages could expire without being consumed or inconsistent with queue restricted arguments. In theory, this should not happen or might happen in a very rare case. But as we are working with customers data, we do not want to lose important messages. To keep dead messages saved in the message broker and do not stuck them in the original queue, we are using dead-letter feature.

Messages are published to exchange and can be sent to multiple queues depending on the routing key. As you can see from the image above, we used the same dead-letter scheme as for the original queues, so dead messages may end up in the wrong dead-letter queues. It is not very critical if you pick up dead messages manually (considering that they are rare), but nevertheless, it is still strange to find these messages in the wrong place.

To solve this problem, we need to add a new argument to the properties of the queues, it is x-dead-letter-routing-key and it should be unique. As a unique value for the routing key, we can use the queue name itself. This idea brought our team one step closer to a good solution: we don't need a dead-letter exchange anymore 🎉. To simplify it, we can use default nameless exchange "" with the dead-letter queue as the routing key and it will forward the message directly to the proper queue.

Unfortunately, doing everything is not as easy as writing or talking about it 😒. To maintain the consistency and stability of the message broker, the RabbitMQ does not allow changing the arguments of already existing queues.

Deployment preparation

So, RabbitMQ does not allow you to change queue arguments in the runtime, so the only possible way to do it by removing queues and re-creating them again with updated arguments. But it is not possible in production, as we might lose some messages when they already removed, but new ones still do not exist. To solve this problem we need to introduce temporary queues to handle these messages, while old queues will be removed. For a simple system, this will be possible with 4 releases:

Create temporary queues, but do not handle messages from them for now.
Switch to the new queues and remove old queues. At this step, we already have a properly configured queues, but names are different. To return to old names, we need to do the same steps again.
Create new queues with old names, but with updated arguments. Do not consume messages from them for now.
Switch to the new queues with updated arguments.

4 releases, not a few, right? This requires not only a lot of small work, but also attention to make sure everything went right every time. How can we reduce them? 🤔

The simplest thing we can do is agree to rename the queues. This will reduce the number of releases by 2 times, since we will not need to rename them back. This was acceptable to us, and we even got more of it as we improved the message handling process. But that's a completely different story 😉.

What else can you do? Enabling consumers and message handling in the new queues right away will reduce release count to only one, but we should accept the risk of duplicated messages when new queues already created but old ones are still processing.

At this point, I was stopped by the teammate, because I did not take into account the process of our deployment. We have blue-green deployment process, it's when you have multiple instances of the same thing. And when you deploy, you take one down, upgrade, then put it up, then take the other one down to upgrade. This guarantees there is something always up. In our case, this means there is always a consumer there.

So, messages can definitely be duplicated if deployed during business hours. Deployment takes several minutes, which means that both old and new queues will be active for several minutes.

Time to analyze and decide whether it is safe to deploy the application at night (and do we really want to do it 🙂) when the message flow is low, or it is worth implementing a third-party service like a Redis to check if the message has already been processed by some consumer, old or new.

Release

The easiest way to check the load on our message broker is to check the number of logs by day of the week and time. Since we are a highly focused company working only in Germany, we have a very low message load from late evening to early morning.

It is not such a big highload as it could be, so we can accept the risk that some messages may be duplicated, but even if this happens, their number will be extremely small and we can manually solve them. This will save the resources and time that would be required for two releases.

After trying to release after midnight we found out that we couldn't do it at night. Some of our third-party services are not available, so the container simply cannot be booted. Well, it was worth trying once, now we know it for sure. Nighttime for sleeping 😴.

But we can still do it late in the evening or early in the morning. One has only to pay attention to the RabbitMQ load.

Late in the evening:

Early in the morning:

We made the decision to press the release button early in the morning after a good night's sleep. This time everything went fine and there were no duplicates.

It was not an easy way to solve this problem, but it was worth it. Solving this problem, our team and I learned a lot of interesting things about message consuming and deployment processes. Now it is even better than before, with correct queue settings and decoupled message handling 😎.

TL;DR

RabbitMQ does not allow to rename queues or change queue arguments;
to change something in the queue, you have to remove it and re-create;
to re-create it safe, you need to use temporary queues;
stable system could be run under multiple instances, so be aware of duplicated messages between old queues and new queues;
if your business is tied to one timezone and is not high loaded at night, it is acceptable to have duplicated messages instead of over-engineering your consumers.