DEV Community: Rishabh Agarwal

Some VSCode Extensions Were Mining Crypto — Here’s What You Need to Know

Rishabh Agarwal — Fri, 06 Jun 2025 12:12:55 +0000

I’m a big fan of customizing my VSCode setup — like most devs, I’ve got a bunch of extensions installed to boost productivity and improve my workflow.

But a recent report really caught my attention: several extensions that looked completely legit were secretly running malicious scripts behind the scenes.

What Happened?

Researchers found 10 extensions on the VSCode Marketplace that:

Disabled Windows Defender
Gained admin privileges via PowerShell
Installed XMRig to mine Monero in the background
Faked install numbers to look popular
Disguised themselves as trusted tools like Prettier and Discord Presence

Some of these had over 900K installs — clearly meant to build false trust.

What This Means for Developers

It’s a serious wake-up call for all of us who rely heavily on third-party tools. Even trusted environments like VSCode aren’t immune to abuse.

Here’s how to stay safe:

Don’t trust install counts alone — they can be manipulated
Verify the publisher — check for an established history
Avoid newly published extensions unless vetted
Keep your system’s security settings enabled

Microsoft has removed the malicious extensions and blocked the accounts behind them. But this incident highlights a bigger issue: the need for better vetting and more awareness on our end.

👉 I wrote a full breakdown, including the list of affected extensions and how the attack worked.

Read it here

I believe in learning from real-world case studies. In software engineering, we have valuable insights through tech company blogs. Here are 5 of my top picks: Uber, Canva, Pinterest, Airbnb, and Netflix Engineering Blogs. What are Yours?

Rishabh Agarwal — Tue, 18 Mar 2025 13:49:56 +0000

How Pinterest uses Kafka for Long-Term Data Storage

Rishabh Agarwal — Wed, 15 Jan 2025 07:37:43 +0000

I spent hours diving into this so you don’t have to!

Here is what I learned:

Pinterest doesn't store all data on Kafka brokers forever.
Older data is moved to a remote storage like Amazon S3.
They built a tool called Segment Uploader to automate this process.
The Segment Uploader periodically transfers older data from Kafka brokers to remote storage.
Segment Uploader runs as a sidecar alongside the Kafka broker.
They also developed a specialized Consumer Library to fetch data intelligently.
The library fetches old data directly from remote storage and new data from Kafka brokers.

By combining Kafka’s real-time capabilities with cost-efficient remote storage, Pinterest ensures scalability, reliability, and efficient long-term data management.

PS - I recently published an article on my free Newsletter covering this case study in-depth with visuals: https://designsystemsweekly.substack.com/p/how-pinterest-leverages-kafka-for

15 Databases, 15 Use Cases — The Ultimate Guide That No One Asked For (But Everyone Needs)

Rishabh Agarwal — Sun, 05 Jan 2025 16:14:23 +0000

1. Relational Databases

Examples: Postgres, MySQL, Oracle
Features: Organized into tables with fixed columns; uses foreign keys to establish relationships.
Use Cases: E-commerce, finance, healthcare, and applications with structured data and well-defined relationships.

2. Wide-Column Databases

Examples: Cassandra, ScyllaDB, DynamoDB
Features: NoSQL databases with flexible columns that can vary across rows; optimized for high scalability and low-latency querying.
Use Cases: Big data, analytics, high-write throughput.

3. Time-Series Databases (TSDB)

Examples: InfluxDB, Prometheus, Kdb+
Features: Optimized for time-indexed data like metrics and events.
Use Cases: Financial trading, performance monitoring, IoT sensor data.

4. Ledger Databases

Examples: Amazon Quantum
Features: Designed for financial data with immutability and cryptographic verification; validated by a central authority.
Use Cases: Financial applications, supply chain management, voting systems.

5. Graph Databases

Examples: Neo4j, ArangoDB, Amazon Neptune
Features: Store data as nodes, relationships, and properties; ideal for managing interconnected data.
Use Cases: Social networks, knowledge graphs, recommendation systems.

6. Object-Oriented Databases (ODBMS)

Examples: ObjectDB, db4o
Features: Store data as objects, similar to object-oriented programming languages; support inheritance and polymorphism.
Use Cases: Object-oriented applications, multimedia databases.

7. Hierarchical Databases

Examples: IBM IMS, Windows Registry
Features: Organize data in a tree-like structure with parent-child relationships.
Use Cases: File systems, legacy systems.

8. Document Databases

Examples: MongoDB, CouchDB, ArangoDB
Features: Store semi-structured data as JSON-like documents; flexible and hierarchical.
Use Cases: Content management systems, e-commerce platforms.

9. Key-Value Databases

Examples: Redis, Couchbase, DataStax
Features: Store data in key-value pairs; highly scalable and fast.
Use Cases: Caching, session storage.

10. Blob Databases

Examples: Amazon S3
Features: Store large unstructured binary data like media files.
Use Cases: Multimedia storage, content delivery networks.

11. In-Memory Databases

Examples: Redis, Memcached, Apache Ignite
Features: Store data in memory for ultra-low latency and high throughput.
Use Cases: Caching, real-time bidding, gaming leaderboards.

12. Text Search Databases

Examples: Elasticsearch
Features: Optimized for storage, retrieval, and analysis of large volumes of text data.
Use Cases: Web searches, auto-complete, filtering.

13. Spatial Databases

Examples: PostGIS, Oracle Spatial
Features: Manage spatial data types (e.g., points, polygons) and related relationships.
Use Cases: GIS, location-based services, spatial analysis.

14. Vector Databases

Examples: Pinecone, Chroma
Features: Store, index, and search high-dimensional data; optimized for AI and machine learning tasks.
Use Cases: Image and video search, recommendation systems, AI model embeddings.

15. Embedded Databases

Examples: SQLite, RocksDB, BerkeleyDB
Features: Lightweight databases integrated within applications; offer fast data access with a small footprint.
Use Cases: Desktop applications, quick proofs of concept, resource-constrained environments.

Checkout the full article on Medium!

Conquer Tedious Tasks with These 10 Python Automation Scripts

Rishabh Agarwal — Tue, 31 Dec 2024 12:57:51 +0000

Juggling repetitive tasks can significantly slow down your development workflow. Thankfully, Python offers a rich ecosystem for crafting clever automation scripts that streamline these processes and free you to focus on more creative coding endeavors.

This article highlights 10 such scripts that can significantly enhance your developer experience:

AFK No More: Prevent your computer from going to sleep with a script that keeps your mouse moving, mimicking activity and avoiding those pesky screen locks.
News at Your Fingertips: Stay informed with a script that delivers news updates directly to your desktop, eliminating the need for constant web browsing.
Dictation Dynamo: Feeling the typing fatigue? This script empowers you to convert speech to text, allowing you to relax and verbally dictate your code or notes.
Cartoon Break! Take a fun break by transforming photos into cartoon avatars using this script. It's a great way to add a touch of humor or personalize your online presence.
Bulk Email Blaster: Streamline email marketing campaigns or mass communication with a script that facilitates sending emails to a large list of recipients efficiently.
GIF-tastic Videos: Liven up your projects or social media presence by converting video clips into cool GIFs with this easy-to-use script.
Voice-Activated Browsing: Ditch the keyboard and navigate websites using voice commands! This script lets you speak the URL you want to visit and opens it automatically.
Hydration Hero: Never forget to stay hydrated again! This script acts as a personal reminder system, sending notifications throughout the day to nudge you to drink water.
Stack Overflow Savior: Stuck on a coding error? This script comes to the rescue by automatically searching Stack Overflow for relevant solutions based on the error message you're encountering.
Screen Recorder on Demand: Need to capture your screen for a tutorial or demo? This script allows for quick and easy screen recording, saving the session as a video file.

Intrigued? Dive Deeper!

This is just a taste of what's possible with Python automation. To explore detailed explanations, code snippets, and practical use cases for these 10 scripts, head over to the full article on Medium.

I hope this summary piques your interest in leveraging Python for automation! By incorporating these scripts into your workflow, you can significantly boost your productivity and free up valuable time for the aspects of development you enjoy most.

My Notes on Canva’s Real-Time Mouse Pointer Implementation

Rishabh Agarwal — Fri, 27 Dec 2024 03:47:32 +0000

Canva's engineering team embarked on developing a solution to handle real-time mouse pointer interactions for users working on designs simultaneously. The solution needed to support 3 updates per second per user, for up to 50 users on a single design, while ensuring scalability and performance.

Short-Term Solution: Backend-Centric Architecture

Initially, Canva's team relied on WebSockets and Redis to facilitate real-time updates. The backend was designed to be stateless, with Redis acting as a message broker. The system scaled horizontally by adding new server instances as load increased, with each instance managing WebSocket connections and communicating through Redis. However, the challenge was to balance communication across these stateless instances.

To optimize Redis, they used Redis Streams for session control information and PubSub for real-time mouse pointer updates, creating a dedicated Redis instance for PubSub messages. Despite some limitations, this architecture successfully supported up to 100,000 mouse pointer updates per second, with the system handling the load efficiently even at peak usage. Key optimizations included binary encoding of pointer data and batch commits to Redis, reducing backend and Redis CPU load by 30%.

However, to meet the requirement of 60 updates per second per user, a new solution was necessary.

Long-Term Solution: WebRTC Implementation

The long-term vision was to move towards a WebRTC-based solution to reduce backend load and increase scalability. WebRTC enables peer-to-peer communication, eliminating the need for server roundtrips. Canva’s team decided to use WebRTC for sending serialized mouse pointer data directly between clients, using WebSockets for signaling and NAT traversal via STUN/TURN servers.

The system employed a Mesh topology for WebRTC connections, where each user directly connects with others, reducing backend dependency. About 50% of connections were direct peer-to-peer, while the rest utilized TURN servers when a direct connection wasn’t possible. The WebRTC solution, though complex to implement, provided better scalability and efficiency for handling high-frequency updates.

Key Insights:

Moving to WebRTC allowed Canva to scale to 60 updates per second per user.
WebRTC's peer-to-peer connections significantly reduced backend load.
A hybrid approach, using WebSockets for signaling and TURN servers for connectivity, ensured smooth communication between clients. Conclusion:
By combining WebSockets, Redis, and WebRTC, Canva’s engineering team successfully built a scalable solution that can handle real-time mouse pointer interactions at scale, allowing users to work on designs with minimal lag and high responsiveness.

To dive deeper into the technical implementation, challenges, and optimization strategies used by Canva, read the full blog on Medium!

My Notes on Canva’s Real-Time Mouse Pointer Implementation | by Rishabh Agarwal | ILLUMINATION

Rishabh Agarwal ・ Dec 27, 2024 ・
Medium

5 Caching Patterns Every Developer Should Know

Rishabh Agarwal — Mon, 23 Dec 2024 06:54:40 +0000

Caching is a crucial technique for enhancing the performance of applications by temporarily storing data for quick access. In this post, we'll briefly explore five popular caching patterns: Read Through, Write Back, Write Around, Write Through, and Cache Aside.

1. Read Through

The Read Through caching pattern allows your application to retrieve data directly from the cache. If the data isn't available in the cache, it fetches it from the database and stores it in the cache for future access. This pattern reduces the number of read operations on the database and speeds up data retrieval.

2. Write Back

With the Write Back pattern, updates to the cached data are written back to the cache first and then asynchronously persisted to the database later. This approach can improve write performance but requires careful handling to ensure data consistency and avoid potential data loss.

3. Write Around

The Write Around pattern processes writes directly to the database, bypassing the cache. This means that when data is requested afterward, it will be pulled into the cache. This method is beneficial when writes are infrequent compared to reads, minimizing cache pollution with rarely-accessed data.

4. Write Through

In the Write Through caching pattern, every write operation is performed on both the cache and the database simultaneously. This ensures that the cache always contains the most up-to-date data, providing consistency at the cost of some performance overhead during writes.

5. Cache Aside

The Cache Aside pattern places the responsibility for managing the cache on the application. When the app requests data, it first checks the cache; if the data isn’t present, it retrieves it from the database and stores it in the cache. This pattern allows fine-grained control over caching but requires more complex logic in the application.

These caching patterns serve different use cases and have distinct advantages and trade-offs. Understanding them can help you optimize your application for better performance and user experience.

Want to dive deeper?

For a detailed discussion on these caching patterns and how to implement them effectively, check out my full blog here: 5 Caching Strategy Every Developer Keeps in Their Back Pocket

Happy coding! 🚀

Relational Databases: PostgreSQL Vs. MariaDB Vs. MySQL Vs. SQLite

Rishabh Agarwal — Sat, 08 Jun 2024 06:37:54 +0000

Relational databases first appeared in the 1970s—that was more than half a century ago! They have stood what one would call the “test of time” and remained the go-to persistence solution for software applications. Other database technologies like NoSQL have challenged this dominance of RDBMS, but their sheer versatility has kept them at the top.

In a landscape filled with open-source and commercial relational databases, this article focuses on the four most prominent open-source databases - PostgreSQL, MySQL, MariaDB, and SQLite. These DBMS are the most preferred databases per the SO’s 2023 survey.

When developers face many database options, it can feel like drowning in a sea of choices. But the real danger lies in the aftermath of a poor database selection. It's not just a ticking time bomb; it's a full-blown explosion in the advanced stages of application development, turning a switch to a more suitable database technology into a living nightmare.

This article aims to empower developers with enough knowledge to make informed decisions about various popular relational databases.

Core Feature Comparison: PostgreSQL Vs. MariaDB Vs. MySQL Vs. SQLite

The core features of a database often emerge as the pivotal factor in your choice of database technology. Whether it's the architecture or the feature set implemented by a database, these properties can significantly sway your decision. Given the importance of these core features, we begin our discussion with an extensive comparison of these across the four databases.

1. Database Architecture

While all of the databases we are comparing are relational databases, there are some significant architectural differences among them. Let's explore these differences.

PostgreSQL: PostgreSQL's unique architecture as an Object-Relational DBMS (ORDBMS) is a game-changer. It not only caters to the needs of a relational database but also extends its support to classes, objects, and inheritance. This powerful capability of PostgreSQL effectively bridges the gap between relational databases and object-oriented databases, opening up a world of possibilities for a wide range of applications.

Example: This object-oriented perspective of PostgreSQL is evident in several places. Every table in PostgreSQL has a corresponding data type created, and each row of a table can be seen as an instance of that data type.

CREATE TYPE car_type AS (id integer, maker text, model text);
CREATE TABLE car OF car_type;

You can also create tables inherited from other tables.

CREATE TABLE ev_car (battery_capacity integer) INHERITS (car);

Additionally, PostgreSQL is fully ACID-compliant in all of its configurations. This is a desired guarantee for many enterprise-grade applications! Moreover, it is the closest database to complete conformance with the SQL standards.

PostgreSQL's implementation of MVCC (Multi-Version Concurrency Control) is a testament to its efficiency. Unlike other databases that rely on locks or undo logs, PostgreSQL maintains versioned copies of DB objects. This approach is not just efficient, it's highly efficient, particularly when dealing with a large number of concurrent users and long running transactions. This reassures you of its performance in handling your database's needs.

MySQL: This is a pure relational database with an architecture optimized for read-heavy workloads. It uses a single process for multiple users, resulting in better read performance overall compared to other databases!

MySQL offers a flexible architecture, supporting around 15 DB engines in addition to its default InnoDB engine. MySQL allows you to specify database engines at the table level using the TYPE parameter.

Example: Using ISAM engine for a table in MySQL.

CREATE TABLE car (
       id INT NOT NULL AUTO_INCREMENT,
       PRIMARY KEY (id),
       maker TINYTEXT,
       model TINYTEXT
) TYPE=ISAM

MariaDB: This is a fork of MySQL and thus shares many architectural features with MySQL. The project was initiated by MySQL's developers after its acquisition by Oracle, but since then, both databases have been developed independently.

MariaDB's architecture offers improved scalability and performance. It has a larger thread pool for better concurrency, only offered in MySQL's enterprise version. It also provides better performance in terms of the number of queries served per second.

Columnar storages have proven to outperform their row-based counterparts for workloads heavy on analytics. MariaDB sets itself apart by offering a dedicated DB engine for columnar storage, known as ColumnStore. This unique feature positions MariaDB as an exceptional choice for OLAP workloads, further enhancing its appeal for specific use cases.

Example: Using ColumnStore in MariaDB

CREATE TABLE `experiment` (
  `time` datetime NOT NULL,
  `scientist` varchar(1024) NOT NULL,
  `content` text NOT NULL
) ENGINE=Columnstore

SQLite: Unlike the other three databases, SQLite is a serverless database! A single-file database, it is lightweight, simple, and quickly embeddable. This unique architecture makes it a great choice for embedded applications.

SQLite, a client-side database, operates on a relational data storage model. However, it does come with some limitations when compared to the other three databases. Its unique architecture, while offering benefits, also means that SQLite has limited transaction and concurrency support.

2. Data Types & Unique Functionality

All four databases encompass a comprehensive set of basic datatypes, including text, number, boolean, and date, for regular use cases. However, differences start to appear when dealing with advanced data types. Let us compare the support offered by various databases for advanced data types.

JSON - With the increased adaption of JSON in web applications, databases are often required to persist JSON data directly. Both MySQL and PostgreSQL provide support for JSON datatypes. This support is absent for MariaDB and SQLite, and the best we can do is serialise JSON as a string.
Arrays - PostgreSQL is the only database among the four that supports storing arrays in database tables. Arrays can be persisted as JSON in MySQL, but there is no support for storing arrays natively. Again, MariaDB and SQLite do not provide support for persisting arrays natively.
Geospatial Data - Barring SQLite, all the other databases support Geospatial Data. PostgreSQL's PostGIS extension supports over 600 spatial operators. MySQL, with its Spatial extension and MariaDB support over 160 operators.

Each database offers unique features that may be suitable for specific use cases. Here is a list of popular features provided by each database.

Database	Features
PostgreSQL	Materialized Views, Pub/Sub Notifications
MySQL/MariaDB	Invisible Columns, Temporary Tablespaces
SQLite	Virtual Table

3. Performance & Scalability

The performance of a database depends on several factors, such as data volume, mix of operations, number of concurrent users, and hardware capability. Therefore, a statement like “Database system X is more performant than database system Y” is incomplete without mentioning all of these variables!

MySQL/MariaDB have generally been found to provide better performance for a read-heavy workload. On the other hand, when the workload consists of both read and write operations, PostgreSQL outperforms other databases. For very simple queries on small to medium-sized databases, SQLite will be the most efficient since there is no overhead of maintaining a server and heavy concurrency machinery.

NOTE: It is very important to characterize your workloads and compare database performance on each workload instead of using a general performance comparison!

Talking about scalability, all databases, except SQLite, offer replication and horizontal scalability. Between MariaDB and MySQL, MariaDB is known to be better scalable.

Development Experience

Besides the core features, general development experience is another significant factor when deciding on a database. This section will explore key factors influencing your experience working with each database option.

1. Ease of Use & Learning Curve

Installation and Setup Complexity

Regarding setup and installation, SQLite wins the race by miles. With its server-less architecture and zero-configuration setup, it is the simplest to spin up and use. Everything needed to use SQLite comes bundled with the application!

Both MariaDB and MySQL offer a relatively straightforward configuration and usage experience. This ease of installation is a key factor in their popularity for quick Proof of Concept (PoC) and ideation, as it allows developers to focus on their projects rather than the setup process.

PostgreSQL has the most complex configuration of all four. Installing it can be challenging, especially for beginners. While efforts are being made to simplify its installation, it still lacks MariaDB and MySQL in terms of ease of installation.

Learning Curve

Without the complexities of large database systems, SQLite has the gentlest learning curve. With some basic knowledge of SQL and data persistence, anyone can start using and working with SQLite.

Both MySQL and MariaDB, with their shared architecture, offer a smooth learning curve for beginners. The knowledge you gain from one database can be easily applied to the other, making the learning process even more efficient.

With its extensive set of features, PostgreSQL has the steepest learning curve. Before they can start using it, users should be familiar with the ORDBMS architecture and several database configurations.

2. Community & Support

All four databases are open-source and highly popular, so they provide good community support. However, MySQL also provides an enterprise version, and some feature requests might be denied in the community version in favor of the enterprise version.

3. Third-Party Tools and Integrations

All four databases enjoy a rich repository of third-party tools and integrations. These tools simplify day-to-day database management and usage.

PostgreSQL

PostgreSQL has a large collection of extensions that add a number of features to it. Some of the popular ones are PostGIS, LTree, and HStore. All four databases are open-source and highly popular, so they provide good community support. However, MySQL also provides an enterprise version, and some feature requests might be denied in the community version in favor of the enterprise version.

Management Tools - psql and pgAdmin.
Backup & Restore - pg_dump and pg_restore.
Monitoring - pg_stat_activity and pg_stat_statement

MySQL and MariaDB

MySQL and MariaDB being interoperable shares most of the tools in common.

Management Tools - mysql and phpMyAdmin
Backup & Restore - mysqldump and mysqlimport

SQLite

SQLite too offer tools like sqlite3 to manage database from command line.

Database Integration with Strapi

Strapi allows you to use either of these four databases for your application development purposes. Once you decide on the most suitable database technology for your application, you can configure Strapi to persist your application data inside that database. Let us understand the configurations required in Strapi for each database. Check out the documentation for database configurations in Strapi.

SQLite Integration with Strapi

SQLite is the default (quickstart) and the recommended database to quickly create an app locally. You can use the quickstart flag to automatically configure the SQLite database.

yarn create strapi-app my-project --quickstart

This should automatically open up the admin page on localhost:1337/admin. Once logged in, you can create collections and SQLite database will be used to persist all data.

By default the database file (data.db) for SQLite will be placed inside .tmp folder at the root of your Strapi project. This can be configured in the database.js file inside the config directory.

module.exports = ({ env }) => ({
  connection: {
    client: 'sqlite',
    connection: {
      filename: path.join(__dirname, '..', env('DATABASE_FILENAME', '.tmp/data.db')),
    },
    useNullAsDefault: true,
  },
});

You can either set DATABASE_FILENAME environment variable or explicitly provide database file path in database.js. As discussed, SQLite is the easiest way to get you started when working on applications.

PostgreSQL Integration with Strapi

We can use the custom installation method for creating Strapi projects that uses PostgreSQL. For this to work, you must already have a running PostgreSQL instance on your machine.

Here is how this can be done. Specify the correct database name, host, port, username, and password for your Postgres database and Strapi will do the rest for you!

npx create-strapi-app strapi-with-postgres           
? Choose your installation type Custom (manual settings)
? Choose your preferred language JavaScript
? Choose your default database client postgres
? Database name: postgres
? Host: 127.0.0.1
? Port: 5432
? Username: postgres
? Password: ********
? Enable SSL connection: No

Creating a project with custom database options.
Creating a new Strapi application at /home/strapi-with-postgres.
Creating files.
Dependencies installed successfully.
Initialized a git repository.


Your application was created at /home/strapi-with-postgres.

Available commands in your project:

  yarn develop
  Start Strapi in watch mode. (Changes in Strapi project files will trigger a server restart)

  yarn start
  Start Strapi without watch mode.

  yarn build
  Build Strapi admin panel.

  yarn strapi
  Display all available commands.

You can start by doing:

  cd /home/strapi-with-postgres
  yarn develop

As suggested in the output of the npx command, we can start the server and our Strapi application will now use the Postgres database. You might see errors like below if the PostgreSQL database is not running or is not configured properly.

┌───────────────────────────────────────────────────────────────────────────┐
│                                                                           │
│   Error: connect ECONNREFUSED 127.0.0.1:5432                              │
│       at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1195:16)   │
│                                                                           │
└───────────────────────────────────────────────────────────────────────────┘

Again, you can find all the database configuration inside the config/database.js file.

postgres: {
  connection: {
    connectionString: env('DATABASE_URL'),
    host: env('DATABASE_HOST', 'localhost'),
    port: env.int('DATABASE_PORT', 5432),
    database: env('DATABASE_NAME', 'strapi'),
    user: env('DATABASE_USERNAME', 'strapi'),
    password: env('DATABASE_PASSWORD', 'strapi'),
    ssl: env.bool('DATABASE_SSL', false) && {
      key: env('DATABASE_SSL_KEY', undefined),
      cert: env('DATABASE_SSL_CERT', undefined),
      ca: env('DATABASE_SSL_CA', undefined),
      capath: env('DATABASE_SSL_CAPATH', undefined),
      cipher: env('DATABASE_SSL_CIPHER', undefined),
      rejectUnauthorized: env.bool(
        'DATABASE_SSL_REJECT_UNAUTHORIZED',
        true
      ),
    },
    schema: env('DATABASE_SCHEMA', 'public'),
  },
  pool: { min: env.int('DATABASE_POOL_MIN', 2), max: env.int('DATABASE_POOL_MAX', 10) },
}

For a comprehensive tutorial on setting up Strapi with Postgres, refer to this article.

MySQL/MariaDB Integration with Strapi

The configuration process for MySQL/MariaDB is similar to that of PostgreSQL. We can use custom installation method for starting a MySQL/MariaDB based Strapi application.

npx create-strapi-app strapi-with-mysql   
? Choose your installation type Custom (manual settings)
? Choose your preferred language JavaScript
? Choose your default database client mysql
? Database name: strapi-with-mysql
? Host: 127.0.0.1
? Port: 3306
? Username: mysql
? Password: *****
? Enable SSL connection: No

Creating a project with custom database options.
Creating a new Strapi application at /home/strapi-with-mysql.
Creating files.
Dependencies installed successfully.
Initialized a git repository.


Your application was created at /home/strapi-with-mysql.

Available commands in your project:

  yarn develop
  Start Strapi in watch mode. (Changes in Strapi project files will trigger a server restart)

  yarn start
  Start Strapi without watch mode.

  yarn build
  Build Strapi admin panel.

  yarn strapi
  Display all available commands.

You can start by doing:

  cd /home/strapi-with-mysql
  yarn develop

This will add the following configuration inside the database.js file. It lists out the client, host, password, database, and user to be used for connecting.

mysql: {
  connection: {
    connectionString: env('DATABASE_URL'),
    host: env('DATABASE_HOST', 'localhost'),
    port: env.int('DATABASE_PORT', 3306),
    database: env('DATABASE_NAME', 'strapi'),
    user: env('DATABASE_USERNAME', 'strapi'),
    password: env('DATABASE_PASSWORD', 'strapi'),
    ssl: env.bool('DATABASE_SSL', false) && {
      key: env('DATABASE_SSL_KEY', undefined),
      cert: env('DATABASE_SSL_CERT', undefined),
      ca: env('DATABASE_SSL_CA', undefined),
      capath: env('DATABASE_SSL_CAPATH', undefined),
      cipher: env('DATABASE_SSL_CIPHER', undefined),
      rejectUnauthorized: env.bool(
        'DATABASE_SSL_REJECT_UNAUTHORIZED',
        true
      ),
    },
  },
  pool: { min: env.int('DATABASE_POOL_MIN', 2), max: env.int('DATABASE_POOL_MAX', 10) },
}

For a more comprehensive tutorial on using MySQL/MariaDB with Strapi, refer to this article!

Use Cases & Recommendations

Equipped with the knowledge of core features and development experience for each of the database, it is time for us to identify what scenarios makes one database preferrable over the other.

SQLite: This is most suited for embedded applications. It can also be a viable option if you are working with low volumes of data without much concern for data concurrency.
MySQL: MySQL is a good choice for quick prototyping. Additionally, if your workload consists of bulk reads and a comparatively smaller number of writes, MySQL is a better candidate.
MariaDB: The fact that MariaDB and MySQL share the same architecture implies that both databases are good candidates for many everyday use cases. But if you are looking for a scalable database that offers high query speed, then MariaDB is the way to go.
PostgreSQL: PostgreSQL becomes the best option when building large and complex enterprise-grade applications. Its rich feature set and SQL standard conformance offer everything you need from a relational database. Strapi allows you to use either of these four databases for your application development purposes. Once you decide on the most suitable database technology for your application, you can configure Strapi to persist your application data inside that database. Let us understand the configurations required in Strapi for each database.

Comparison Matrix: PostgreSQL Vs. MariaDB Vs. MySQL Vs. SQLite

The image below shows some summarized differences between the top relational databases discussed so far.

Conclusion

Each database has its own strengths and weaknesses. While one might be easy to install and use, the other might offer more features. When dealing with large volumes of data requiring complex queries, PostgreSQL is the way to go. For embedded applications or low volume-low concurrency use cases, SQLite offers the best solution. MariaDB/MySQL is a better option for medium data volumes and read-heavy workloads.

Choosing the correct database for your particular use case is challenging. Finding one that best aligns with your needs takes a lot of research and experimentation. A thorough understanding of core database features facilitates making informed decisions, as explored in this comparison of PostgreSQL, MySQL, MariaDB, and SQLite. While this article covered the four most popular relational databases, a similar exercise can be done for any database.

What are Content Delivery Networks (CDNs)?

Rishabh Agarwal — Sat, 25 May 2024 08:23:54 +0000

Imagine running an application that serve latest news in the form of 100-words articles. News in the form of such short articles allowed your application’s users to consume lot more news in a short time.

With growing popularity of your application, the load on your application server started growing exponentially. Initially you increased your server’s capability to serve more requests, but you soon realised that this was not sustainable. You dig deep to find a better solution for this problem. You found the following usage pattern for your application ~

Users that were geographically closer to each other almost consumed the same set of news articles
While the number of users were large at any point in time, the news articles that they consumed in-together wasn’t that large. In other words, people read almost the same set of articles during a given interval.

All this made you to think of caching news articles. However, this cache needs to be different for different geographical locations. You researched further and found a perfect solution called Content Delivery Networks (CDNs).

This is how Cloudflare defines CDN ~

A content delivery network (CDN) is a geographically distributed group of servers that caches content close to end users. A CDN allows for the quick transfer of assets needed for loading Internet content, including HTML pages, JavaScript files, stylesheets, images, and videos.

So how does CDN works? There are the following three crucial components to it.

➙ Origin Server: This is the application server that contains the source of truth. Any updates are made to the contents of the origin server.

➙ Edge Servers: These servers are located in multiple geographical locations that are called “Points-of-Presence” or “PoP”. Content from the origin server is cached here and is served to the clients. In case of missing data, origin server is used to respond to the client’s query and for simultaneously updating edge server’s cache.

➙ DNS Servers: Domain Name Systems (DNS) servers keep track of the origin and edge server. On a DNS lookup, IPs of both the origin and the edge server.

While caching is one of the biggest advantages of CDNs, it is not the only one. There are several other advantages like decreased server loads, improved site performance, and protection against cyber-attacks. All these benefits are the reason why most of the internet today is being served up by CDNs.

Types of CDNs

Based on how the cache in edge servers is updated, two types of CDNs exists: push and pull.

Push CDNs push contents from origin server to the edge servers. Updated content is pushed periodically which is then served up by the CDNs. The cache data from edge server is not removed until the data is explicitly deleted or overridden by the updated data.

Pull CDNs is the complete opposite of push CDNs. Here, the edge servers are responsible for pulling in data from the origin server. The process works like this: as the website owner, you maintain the content on the origin server and adjust the URLs to point to the CDN. Subsequently, when a request is made for a web page, the CDN fetches the necessary elements and files from the origin server and presents them to the visitor.

Popular CDNs

Almost always you would use services of a CDN provider instead of setting up your own CDN. There are some very good options out there, some of them being ~

Akamai
Cloudfront
Azure CDN
Cloudflare

With this we reach the end of this blog. Give yourself a pat on the back if you make it this far!

Database Replication Encyclopaedia — Single Leader Replication (1/3)

Rishabh Agarwal — Fri, 24 May 2024 05:55:42 +0000

Replicated databases are the new reality of data based software applications. Gone are those days when databases were confined to a single machine in a data centre. With the rise of distributed systems, database replication has become one of those topics which every developers should know.

If you are hearing the term ‘Database replication’ for the first time, let me summarise it for you in a single line as follows,

Database replication is the process of maintaining several copies of data across several machines that are interconnected via a network.

Why replicate database you may ask? There are several reasons to why distributed systems use replication. Some of the important ones are the following —

To keep data geographically closer to the users for reduced latency.
To increase fault tolerance and increase overall availability of the system.
To be able to scale out the number of machines that can server the read queries.

If the replicated data is constant and does not change over time, there is nothing to worry about. Simply replicate your data across nodes and you are done. Almost all of the complexities of database replication arise because of changing data!

To tackle these difficulties there are three popular approaches to database replications — Single Leader Replication, Multi-Leader Replication, and Leaderless replication. Almost all of the database systems out there use one of these three to achieve database replication. In this blog, we will be discussing the Single Leader Replication method.

Single Leader Replication

Each machine that maintains a copy of the data is called a replica.

In single-leader replication, one of the replica is designated the Leader. The Leader is the one responsible for processing write operations received from the clients. It is also responsible for propagating the write operations to other replicas called follower.

While writes are only served by the leader, read requests can be served by both the leader and followers.

On receiving a write request from the client, the leader first updates its own local storage. It then sends the update request to followers, which on receiving the message from the leader update their own local storages. In this way, all write operations are ensured across replicas.

Synchronous Vs Asynchronous Replication

Depending on how the updates from leader propagates to the follower, the replication is of two types — Synchronous and Asynchronous.

Under Synchronous replication, leader waits for followers to acknowledge the update. Once the leader ensures that the update has been successfully written to all the synchronous replicas and its own local storage as well, it sends success message in response to the client’s request. This replication method ensures that all of the replicas remain in-sync with the leader. Synchronous replication, however, is slow since all replicas must commit changes before the operation can be deemed successful. It is also susceptible to downtime owing to a single node failure.

Under Asynchronous replication, leader does not wait for followers to process the write operation. Once the leader updates its own local storage, it sends a success message back to the client. In this setting, the system may not be consistent immediately but will be eventually consistent. Since the leader does not wait on followers to complete their work, updates are faster when compared to synchronous replication.

In practice, synchronous replication is characterised by a single follower being replicated synchronously while the other replicas are still replicated asynchronously. This ensures that there is always a follower with up-to-date data that can be made the leader in case the current leader fails. This type of replication is called semi-synchronous replication.

Adding new Followers

Adding a new follower replica to the cluster requires some careful considerations. A copy operation on the leader’s database might not give us the correct result since leader is actively accepting write requests and its state is constantly evolving. This could result in corrupted data getting copied. Another option is to take a lock on the leader and then perform the copying operation. This is also not acceptable since it will cause downtime anytime a new follower has to be added.

Instead, we can take a snapshot of the leader’s database. Most database provide this option for the purpose of backup. Restore the snapshot on the new follower’s database. Once the snapshot data is restored, follower can start asking the leader for changes that have occurred since the snapshot was taken. They can use Log Sequence Number (or LSN) to get that data from the leader.

Once the follower has done the catching-up process, it can continue working like the other followers.

Handling Node Outages

It is common for nodes to drop out of the cluster. It can be because of network or machine failure, or it could be because of scheduled maintenance. Let us understand how node outages are handled in Single-Leader replication.

When a follower node goes down, the system can continue to work as usual. Read requests that were earlier served by the impacted node will now be served by other replicas. Later when the follower joins back, it can ask for the updates from the leader and can continue working normally.

Things become interesting in case of a leader failure. To keep the system in a working state, a new leader has to be elected. There are several algorithm by which the cluster can select the new leader replica. Once the new leader is elected, the client needs to be configured to send all write request to this new leader and all the followers should now accept update requests from this new leader.

Issues arising from Replication Lag

Replication lag between the follower and the leader can lead to issues that would otherwise not occur in a single-machine database. These issues require careful considerations, especially when the database does not provide a mechanism to solve them.

Read Your Own Writes

Consider the following scenario, a client sends an update request to the leader. As soon as the client receive a success response from the leader, it makes a read request to the cluster and receive a response from an asynchronous follower. Interestingly, the client will see its updates reverted. Therefore we must design applications that provide read-after-write consistency.

There are several possible ways in which we can provide read-after-write consistency in our application. Some of them are as follows —

We can always use leader to respond for the data that a client may have updated. This obviously depends on being able to identify all the data that could be updated by a client.
Another way is to make all reads from the leader until a certain time after a client makes any update. We can also monitor the replication lag on the followers and prevent queries on followers that lags more than a certain amount of time.
Client may keep the timestamp of the update and this can then be used by the cluster to respond from replicas that are up-to-date until that point in time.

Same user accessing your cluster from different device poses new challenges. In such cases, we want to provide cross-device read-after-write consistency.

Monotonic Reads

With multiple replicas having different replication lags, it is possible that a client, after making request to an up-to-date replica, makes a request to a replica with larger lag. In this situation, client would see an older state and for it the time would appear to go backward.

Monotonic reads is a guarantee that this kind of anomaly does not happen. One way of achieving it is to ensure that a client always reads from a particular replica. Though this is not the ideal solution since a replica may go down at any point in time.

Single leader replication is one of the most used replication techniques. When working with read-heavy workloads, single leader replication becomes a good solution since reads are distributes across replicas. This however becomes an issue when working with write heavy-workloads that can cause your leader to bottleneck.

With this we reach the end of this blog. If you learned something new, then follow me up for more such interesting reads!

Object Sharing in Multi-threaded Environments

Rishabh Agarwal — Sat, 30 Mar 2024 10:42:50 +0000

All the intricacies of a multi-threaded environment arises from sharing objects across threads. As discussed here, managing access to shared objects is the base of thread safety. If there are no shared objects, our program is inherently thread-safe!

But it is not always possible to design a threaded application without making some objects visible across threads. In this article, we will talk about some standard practices for sharing objects. These practices will help us building more resilient and robust multi-threaded systems!

Escaped Objects

When we publish an object, we make it available to code outside of its current scope. This can be useful in many situations, but there are also cases where we want our objects to remain unpublished. However, there are times when we may need to publish objects, and it’s important to ensure that this is done in a thread-safe manner using proper synchronisation.

An accidentally published object is said to have been escaped. There are several way in which an object may escape!

➥ Making objects visible globally ~ Storing references to objects such that any executing thread can access it causes objects to ‘escape’.

class Escape1 {
  // ...
  public static List<Integer> activeUserIds = new ArrayList<>();
  // ...
}

In the example above, any thread can access activeUserIds and it is technically escaped!

➥ Returning objects from non-private methods ~ Returning objects from non-private methods also publishes objects. This is because any executing thread can get hold of internal states of the object.

class Escape2 {
  // ...
  private static List<Integer> activeUserIds = new ArrayList<>();
  // ...
  public List<Integer> getActiveUserIds() { return activeUserIds; }
  public void addActiveUser(Integer userId) {
    boolean userExists = activeUserIds.stream().anyMatch(id -> id == userId);
    if(!userExists) {
      activeUserIds.add(userId);
    }
  }
}

In the above example, the activeUserIds variable, intended to be private to Escape2 class, has escaped its intended scope due to the accessibility of the getActiveUserIds() method. This situation poses a risk of breaking the invariant that an ID should only appear once in the activeUserIds list.

➥ Passing object as parameter to alien methods ~ When we pass objects as parameter to an alien method, there is a possibility of that method using the object in a way that breaks the bounded invariants. Even worse, that method may store a reference to the object and modify it later from another thread. It therefore causes objects to escape!

➥ Publishing inner class instance ~ When an object of inner class is published, an implicit reference to the outer class’ object is published. This leads to escaping!

Confining Objects to Thread

When objects are used only from a single thread, there is no requirement of extra management to make your code thread safe.

Utilising an object from a single thread is an implementation decision, and although the language provides mechanisms to enforce these constraints, it ultimately falls on the programmer to guarantee that such objects remain within the intended scope of a single thread.

Use of local variables helps us confine objects to thread. This is because each thread maintains its own set of local variables on stack. This stack is inaccessible to other threads. However, we need to take special care of not publishing reference to such local variables.

Another way of confining object to a thread is to use Java’s ThreadLocal class. Objects of this class hold separate values for separate executing threads.

public class ThreadLocalTesting {
    final static ThreadLocal<Integer> threadLocal = ThreadLocal.withInitial(() -> 0);
    public static void main(String[] args) throws InterruptedException {
        // Setting threadLocal to 5 from manin thread
        threadLocal.set(5);

        // Creating a new thread
        final Thread thread = new Thread(() -> {
            System.out.println("::: Thread => " + threadLocal.get());
            // Setting threadLocal to 10 from new thread
            threadLocal.set(10);
            System.out.println("::: Thread => " + threadLocal.get());
        });

        thread.start();
        System.out.println("::: Main => " + threadLocal.get());
        thread.join();
    }
}

Make your objects Immutable

Need for thread-safety arise when mutable states are shared across threads. We just saw how can we ensure objects are confined to a single thread. Let us consider the other end, objects with immutable states.

Since an immutable object can always be in a single state, it is inherently thread-safe! But how do you define immutability? An immutable object —

Cannot have its state modified after construction
All its fields are final
Properly constructed i.e., this reference does not escape during construction

Immutable objects can be safely accessed across threads even when synchronisation is not used!

Note ~ It is considered a best practice to mark all fields as private unless they require greater visibility. Similarly, it is advisable to make all fields final unless there is a specific requirement for them to be mutable.

Safely Publishing Objects

Sometimes, it’s necessary to share objects across threads, but it’s essential to ensure that this is done in a thread-safe manner. When dealing with immutable shared objects, no extra synchronization is required. However, when working with mutable objects, it’s crucial to ensure they're safe publication to avoid concurrency issues.

To safely publish an object, we can,

➯ initialise object reference from static initialiser
➯ store object reference in a volatile field or AtomicReference
➯ mark the object with final keyword
➯ use proper locking when accessing the object

After an object has been safely published and is no longer subject to modification, no additional measures are necessary to guarantee its correctness in a multi-threaded environment. However, if the object can be modified after publication, it’s crucial to either implement proper locking mechanisms or ensure that the object is inherently thread-safe to maintain thread safety.

A proper understanding of when and how to share objects in a multi-threaded environment is really necessary to ensure thread-safety of your application. Avoiding sharing objects or making them immutable wherever possible is a great rule-of-thumb to ensure there are less things to fail!

With this we reach the end of this blog. If you enjoy reading this blog, consider exploring these other blogs ~

Java’s Synchronized Collections

Rishabh Agarwal — Sat, 30 Mar 2024 06:16:14 +0000

Java’s Collection framework is one of the foundational components of Java platform. Collections in Java represent a group of objects and the framework provides several interfaces and implementations to work with these collections. With such a rich framework at disposal, developers can focus on what matters the most!

With the advent of multi-threaded programming, the need for thread-safe collections grew. It is when Synchronized Collections were added to Java’s Collection framework. Let us understand what makes Synchronized Collections thread-safe and what are some things we should keep in-mind while using them.

How does Synchronized Collections ensure thread-safety?

Synchronized collections achieves thread-safety by enforcing synchronization on each of its publicly available method. In addition to that, it ensures that its internal state is never published. Thus, the only way to modify a collection is via its public synchronized methods!

Think of synchronized collections as plain unsynchronised collections plus state encapsulation and synchronized public methods.

Since every public method of a synchronized collection is guarded by the same (intrinsic) lock, no two threads can modify/read the collection at once. This ensures that the collection always maintain its variants and hence become thread-safe.

How to create Synchronized Collections?

The Collections class expose several static methods for creating synchronized collections. These static method are named in the following format — synchronizedXxx. Here is a list of these methods:

synchronizedCollection(Collection c)
synchronizedList(List list)
synchronizedMap(Map m)
synchronizedNavigableMap(NavigableMap m)
synchronizedNavigableSet(NavigableSet s)
synchronizedSet(Set s)
synchronizedSortedMap(SortedMap m)
synchronizedSortedSet(SortedSet s)

This is how you will create a synchronized list using the Collections class!

List<Integer> synchronizedIntegerList = Collections.synchronizedList(new ArrayList<>());

Pitfall of Compound Actions on Synchronized Collections

While methods exposed from synchronized collections are thread-safe, compound actions on client-side still require proper locking. Consider the following method —

public void putIfAbsent(List<Integer> synchronizedList, Integer elem) {
  if(!synchronizedList.contains()) {
    synchronizedList.add(elem);
  }
}

This innocent looking method is actually thread-unsafe! Even with thread-safe contains and add methods, the putIfAbsent method does not achieve what it intended to. Between the ‘check’ and ‘act’ steps of our method, another thread can add the elem to the list. Guarding such compound actions is the responsibility of the client.

Synchronized collections allow client-side locking to ensure that such compound actions are atomic with respect to other operations. Each public method of the synchronized collection is guarded by its own intrinsic lock. It is thus possible for client to create atomic operation by acquiring the same lock.

Here is how we will fix putIfAbsent using client-side locking.

public void putIfAbsent(List<Integer> synchronizedList, Integer elem) {
  synchronized(synchronizedList) {
    if(!synchronizedList.contains()) {
      synchronizedList.add(elem);
    }
  }
}

Pitfalls of Iterating on Synchronized Collections

Some extra care is needed to be taken when iterating over a synchronized collection. When iterating over a synchronized collection using iterators, any concurrent modifications to the collection will cause the iterator to fail-fast. On detecting any concurrent modification, the iterator will throw a ConcurrentModificationException.

final Set<String> syncStringSet = Collections.synchronizedSet(new HashSet<>());

// Might throw ConcurrentModificationException
for(String s: syncStringSet) {
  doSomething(s);
}

To avoid getting a ConcurrentModificationException, we can hold a lock on the synchronized collection for the entire duration of iteration. Clearly this is not the most performant approach since the iteration can take a long time and it will block other executing threads from access to the collection!

Synchronized classes provides an easy mechanism to make your collection classes thread-safe. A thorough understanding of these would help you avoid common pitfalls associated with it.

With this we reach the end of this blog. I hope you enjoyed reading this piece.

If you like reading such articles, here are some other interesting reads from this series —