DEV Community: Hrishikesh Dalal

Drupal x Google Summer of Code - Week 3: Reuse & Previews

Hrishikesh Dalal — Tue, 07 Jul 2026 07:17:42 +0000

WELLLLCOMMEE to Week 3 of this series!

In the last week made quite a few structural changes that will help us. Have created the YAML extractor and parser which can extract the YAML component form any response of the LLM using Symfony to validate it. Made a Utility for this thereby making the code reusable.

Official Issue Tracker: Week 3 Tracker

Let me now give me glimpse for this.

https://www.youtube.com/watch?v=7sH6pv2Vyns

The Output Preview & Validator

Tiny UI Changes

Time to Dive Deep into this

YAML Validator & Extractor

Created a Utility folder under src and created a file named Yaml Processor. The concept of this file was simple, when you get an output from LLM, from whichever Service or Function you pull this from this will extract the YAML and give it to you.

Here it uses this : use Drupal\Component\Serialization\Yaml

This under the hood uses Symfony which helps to validate and extraction. Symfony is the underlying PHP framework that powers modern Drupal ( this is just one of the usecase)

<?php

namespace Drupal\ai_recipe_generator\Utility;

use Drupal\Component\Serialization\Yaml;

/**
 * Utility class to parse and extract YAML from AI responses.
 */
class YamlProcessor {

  /**
   * Cleans a response down to YAML text when possible.
   *
   * @param string $response
   *   The raw model response.
   *
   * @return string
   *   A normalized YAML string or the response with code fences removed.
   */
  public static function normalize(string $response): string {
    $parsed = self::extractAndParse($response);

    if (is_array($parsed)) {
      return trim(Yaml::encode($parsed));
    }

    $cleaned = preg_replace('/```

(?:yaml|yml)?\s*|

```/i', '', $response) ?? $response;
    return trim($cleaned);
  }

  /**
   * Attempts to extract and validate YAML from a given string.
   *
   * @param string $response
   *   The raw string response.
   *
   * @return array|null
   *   The parsed YAML as an array, or null if no valid YAML could be extracted.
   */
  public static function extractAndParse(string $response): ?array {
    // First try parsing the entire response as YAML.
    try {
      $parsed = Yaml::decode($response);

      if (is_array($parsed)) {
        return $parsed;
      }
    }
    catch (\Exception $e) {
      // Not valid YAML, continue with extraction.
    }

    // Extract YAML from markdown code blocks.
    $pattern = '/```

(?:yaml|yml)?\s*(.*?)

```/is';

    if (preg_match($pattern, $response, $matches)) {
      $yaml = trim($matches[1]);

      try {
        $parsed = Yaml::decode($yaml);

        if (is_array($parsed)) {
          return $parsed;
        }
      }
      catch (\Exception $e) {
        return NULL;
      }
    }

    return NULL;
  }

}

Reusable YAML Previewer

If you have seen my previous blog, the older YAML previewers they were not upto the mark. So improved that with having better parsers and lines. What I used for this was a template, using a html.twig format which helped to render properly anywhere, anyhow!

This is slightly difficult to think and then implement (tedious slightly) but since highly reusable, highly useful!

{#
/**
 * @file
 * Template for the fluid YAML previewer component with line numbers.
 */
#}
<div class="ai-yaml-preview-container" {{ attributes }}>
  <div class="ai-yaml-preview-header">
    <span class="ai-yaml-preview-title">{{ title|default('YAML Output Preview') }}</span>

    <button class="ai-yaml-copy-btn" onclick="
      const container = this.closest('.ai-yaml-preview-container');
      const code = container.querySelector('code').innerText;
      navigator.clipboard.writeText(code);
      const originalText = this.innerText;
      this.innerText = 'Copied!';
      setTimeout(() => this.innerText = originalText, 2000);
    ">Copy YAML</button>
  </div>

  <div class="ai-yaml-preview-body">
    {# This empty div will be filled with numbers by the JS below #}
    <div class="ai-yaml-line-numbers" aria-hidden="true"></div>

    <pre class="ai-yaml-preview-content"><code class="language-yaml">{{ yaml_string }}</code></pre>
  </div>
</div>

{# Inline script to generate line numbers based on the code length #}
<script>
  (function() {
    // Find all preview containers on the page
    const containers = document.querySelectorAll('.ai-yaml-preview-container');

    containers.forEach(container => {
      const codeBlock = container.querySelector('code');
      const linesContainer = container.querySelector('.ai-yaml-line-numbers');

      // Only process if it hasn't been filled yet
      if (codeBlock && linesContainer && linesContainer.innerHTML === '') {
        // Split by newline to get total lines
        const lines = codeBlock.innerText.split('\n');

        // Handle the case where the AI leaves a trailing empty line at the end
        const lineCount = lines[lines.length - 1] === '' ? lines.length - 1 : lines.length;

        let numbersHTML = '';
        for (let i = 1; i <= Math.max(1, lineCount); i++) {
          numbersHTML += i + '<br>';
        }
        linesContainer.innerHTML = numbersHTML;
      }
    });
  })();
</script>

Code Reduction

Let's understand the problem, we had 3 files for calling the LLM models & providers. All of them had separate functions. So if I ever wanted to change anything, I would have to change in all the three files : (

So we changed the calling part. for 2 (ArenaService & RecipeGenerationService) of them as they were the same replicas the third one at BenchmarkDashboardService was a different one. Clubbing them was fairly simple, just had to change the function calls.

Started fixing the Pipeline

Did the cspell checking and then the phpcs checking which was initially failing but then started fixing the same. For fixing the cspell introduced a file called: .cspell.json. This needed to have all the words in the document which I wanted to exclude form the checking.

Will write a separate blog on testing!

{
    "version": "0.2",
    "language": "en",
    "words": [
        "ai_recipe_generator",
        "chartjs",
        "crosshairs",
        "dalal",
        "fontawesome",
        "hrishikesh",
        "llm",
        "llms",
        "multichat",
        "onclick",
        "Starshot",
        "tempstore",
        "unconfigured",  
        "xaxis",
        "xmark",
        "yaml",
        "yaxis"
    ],
    "ignorePaths": [
        "vendor/**",
        "node_modules/**",
        "*.min.js",
        "*.min.css"
    ]
}

Thank you & catch you up next time!

Follow me at https://www.linkedin.com/in/hrishikeshdalal/
Portfolio: https://www.hrishikeshdalal.dev/
Alt Portfolio: https://hrishikesh-dalal.vercel.app/

Google Summer of Code (GSoC) x Drupal: Week 2 - Mapping the Dashboards

Hrishikesh Dalal — Tue, 07 Jul 2026 07:09:11 +0000

WELCOMMMEEEE to another episode of Weekly Recap of my project

This week we have built two things: LLM Benchmarking Dashboard & LLM Arena to show you which ones are the best. Both of these will be super useful while building our final product.

The Offical Issue Tracker: Week 2 Issues Tackled
Let me give you a glipmse of what our output will look like.

https://www.youtube.com/watch?v=FWhz2f81urQ

Dashboard (Part 1)

Dashboard (Part 2)

The Benchmarking Dashboard

In order to capture the true essence of which model performs the best in the long run, had made a LLM benchmarking dashboard. Someone just by looking at charts can capture which model is most suitable for their usecase.

Metrics that provided the overall picture

1). The Top Banner: It had the total runs so far, along with the average response time and the average accuracy for the recipe generated.

2). The Visuals: Any dashboard is incomplete without a good visual. The first graph which shows the latency for the previous runs mapped across models is useful for finding the trend for model hits. The second graph is for using a bar graph for visualising the model avg. latency to compare which model is the faster as compared to others. Used the Chart module for the same.

3). Key Metric Table: This is the detailed table for per prompt summary. It covers: the provider, the model, the speed, whether the YAML is valid or not, whether the Schema proper or not as per Drupal Symfony, & finally tells us about the accuracy along with timestamps.

What is useful in this is that you can filter it along with sort the columns in ascending or descending order. Is if further colour coded so that you can quickly view the page at a glance.

Things I used to build this:

Chart.js / Drupal Chart Module: To render the responsive, interactive graphs.
Custom Database Schema (.install): To reliably save and query the historical benchmarking data.
Drupal AJAX API: To ensure the table filters and dynamic sorting update seamlessly.
Font Awesome Library: To have cool icons in the dashboard

LLM Arena

The LLM Arena

This was particularly fun to build as this was for comparing the different responses. You get to type the custom prompt and then fire it across all the different models and providers.

You can have all the different models & providers, there is NO LIMIT!!

Further these is also an option to store the responses in the database so that they reflected in the schema. There is also a delete button for the same. If you want to remove some model-provider. Further you will see a very familiar screen that, while firing the prompts, it will look like new module install screen, this is because of the Batch API Screen.

Sources I used:

AJAX API: Model - Provider dropdown
Batch API: For firing all the prompts together
Font Awesome Icons: For the icons there

Laying out the Structure

*System Diagram *

Just like a good Drupal Practice (have to mention), have used AI to generate the image above.

1. Client (User Browser) Layer

Handles the user interface and visual presentation.

User (Arena/Dashboard Interface): Renders the comparison forms, interactive data tables, and metrics dashboards.
Frontend Components:
JavaScript: Uses Chart.js to render interactive line and bar charts for model latency trends.
Icons: Integrates Font Awesome for immediate visual cues (success/failure statuses).
CSS: Uses css/arena.css and css/dashboard.css via .libraries.yml for custom, polished layouts.
Client-Server Communication:
AJAX API: Controls non-blocking form updates, dynamic filtering, and column sorting without page reloads.
Batch API (Client-Side): Drives the real-time progress bar (similar to the module installation screen) during execution.

2. Drupal Application Logic Layer

The PHP backend that processes requests and orchestrates module logic.

Form API (ArenaForm.php, BenchmarkDashboardForm.php): Handles form submissions and handles backend execution triggers.
Services Layer (ArenaService.php, BenchmarkDashboardService.php): Houses decoupled, reusable core logic registered via ai_recipe_generator.services.yml.
API Integration Engine:
Drupal Batch API: Splits concurrent LLM requests into isolated chunks to prevent gateway timeouts and server lockups.
Drupal AJAX API: Manages seamless, lightweight data passing between backend services and frontend components.

3. Data & External Integrations Layer

Manages persistent storage and outbound network requests.

Local Database (.install): Uses a custom schema defined in the install file to save historical run stats, latency, timestamps, and schema validation flags.
External LLM APIs: Handles outbound calls to external models (OpenAI, Anthropic, Gemini). Isolated by the Batch API so a timeout on one provider doesn't crash the session.

Thanks a lot for reading!!

Follow me at https://www.linkedin.com/in/hrishikeshdalal/
Portfolio: https://www.hrishikeshdalal.dev/
Alt Portfolio: https://hrishikesh-dalal.vercel.app/

Drupal x GSoC: Week 1 - Building the Base

Hrishikesh Dalal — Tue, 07 Jul 2026 07:02:24 +0000

WELLLLLLCOMMEE To Week 1 of GSoC x Drupal.

Over the first week my focus was on establishing a proper base for the project. This was particularly fun as you can see your project come to life. These are some other chapters of the journal that you should definitely read!

I created a project: Creating a Module Project on Drupal (NO CODE) (With BONUS Section)

I learnt about PIRLS - Drupal's Core .yml files

Before you proceed any further, you should have a drupal project set up. If not, you can check it out here: Installing Drupal

Have added a bit more changes too, you can check it our here: Week 1 Official Tracker

https://www.youtube.com/watch?v=2IcKTPBoyqc

Week 1 Issues Tackled

Let's zoom out and see what we have built. UI might not be that great yet but that was not the focus of the first week.

You should clone the basic repo of the module and paste it in: web/modules/custom/ai_recipe_generator.

You might need to create this folder & add all the files. We use custom as we haven't made a deployment of the project yet. So once I did this I made. Now I have added ai_recipe_generator.info.yml, ai_recipe_generator_services.yml, ai_recipe_generator.routing.yml. I have made a blog on these files and what do these mean, you can check that out too.

I then created 2 things in src folder - Form & Services.

So we have two pages as you saw above. Let's understand the routing or the file urls

Our File Structure

ai_recipe_generator.test_form:
  path: '/admin/config/development/ai-recipe-test'
  defaults:
    _form: '\Drupal\ai_recipe_generator\Form\RecipeGeneratorTestForm'
    _title: 'AI Recipe Generator (Test UI)'
  requirements:
    _permission: 'administer site configuration'

ai_recipe_generator.settings:
  path: '/admin/config/ai-recipe/settings'
  defaults:
    _form: '\Drupal\ai_recipe_generator\Form\SettingsForm'
    _title: 'AI Recipe Generator Settings'
  requirements:
    _permission: 'administer site configuration'

Each entry has the following

1 The URL Path

2 The Form File which it should render

3 The title of the Page

4 Permission Requirements

We have dependecies as well. This module as of now depends on the AI Module, Gemini Provider Module as well as the Key Module for managing all the keys. You have to install and enable all these modules for the AI Recipe Generator to function. This is actually cool as we are using the Drupal Ecosystem to do our tasks.

Flow Diagram

Phase 1: The Setup (Settings Form)

Before a user can generate a recipe, an administrator sets the rules.

Setting the Ground Rules: An admin visits the "AI Recipe Generator Settings" form and enters the System Instructions (e.g., "You are a senior Drupal developer...").
Saving to the Database: When the admin clicks "Save configuration," Drupal uses its configuration service (@config.factory) to safely store these instructions in the database for later use.

Phase 2: The User Request (Test UI Form)

This is where the actual generation happens.

User Input: A user opens the "AI Recipe Generator (Test UI)" form. They select their preferred AI Provider (like Gemini) and type exactly what they want to build into the Recipe Prompt box.
Triggering the AJAX Call: The user clicks the Generate Recipe button. Instead of reloading the whole webpage, the browser sends a quiet, background message (an AJAX Request) to the Drupal server containing the user's prompt and their chosen AI provider.

Phase 3: The Backend Brain (Drupal Services)

Once Drupal receives the AJAX request, it hands the work over to the custom services defined in your .yml file.

Gathering the Pieces: The main service (RecipeGenerationService) goes to work collecting three crucial pieces of information:
The Rules: It pulls the saved "System Instructions" using @config.factory.
The Context: It asks the ContextExtractor service to pull any relevant background information about your specific Drupal site.
The Request: It takes the user's specific "Recipe Prompt" from the AJAX call.
Talking to the AI: The RecipeGenerationService combines the rules, the context, and the user's request into one massive prompt. It then uses the @ai.provider service to send this combined package over the internet to the external AI (e.g., Gemini).

Phase 4: The Result (AJAX Response)

Receiving the Recipe: The external AI processes the prompt, generates the Starshot Recipe YAML file, and sends it back to your Drupal server.
Updating the Screen: Drupal packages this generated recipe into an AJAX Response and sends it back to the user's browser. The browser reads this response and instantly displays the new recipe on the screen, completing the process without ever forcing the user to refresh the page.

Adding your first piece of code to Drupal Project!

Hrishikesh Dalal — Tue, 09 Jun 2026 02:33:54 +0000

Setting up the codebase on Drupal is EASY! Only you need a little bit of patience. Let's dive deep into it.

So previously we had set up a project. Now we are going to initialise the git repo and add a few changes. First if you have created your project, you can move on into the Version Control section. This will contain instructions of what we should do further.

This was my link: Version Control

Version Control

mkdir ai_recipe_generator

cd ai_recipe_generator

git init

git checkout -b 1.0.x

echo "AI Recipe Generator" > README.txt

git add README.txt

git commit -m "Initial commit."

git remote add origin git@git.drupal.org:project/ai_recipe_generator.git

git push origin 1.0.x

They are the explicit instructions of what we are going to do.

Copying without understanding is useless. Let's go through each step one by one.

Command	What it does	Why we do it
`mkdir ai_recipe_generator`	Creates a new directory named `ai_recipe_generator`.	Allocates a dedicated local space for your project files.
`cd ai_recipe_generator`	Moves your terminal path into the new folder.	Ensures all future commands execute inside the project context.
`git init`	Initializes a hidden local `.git` repository.	Turns the plain folder into a version-controlled project.
`git checkout -b 1.0.x`	Creates and switches to a new branch named `1.0.x`.	Bypasses standard `main`/`master` to match Drupal's release branch rules.
`echo "AI..." > README.txt`	Generates a `README.txt` file with project text.	Creates an initial asset to track, as Git cannot track empty folders.
`git add README.txt`	Stages the file to the Git index.	Explicitly marks the new file to be included in the next snapshot.
`git commit -m "Text"`	Saves the staged file with a historic log message.	Establishes the official baseline/first snapshot of your codebase.
`git remote add origin <url>`	Links your local Git setup to the Drupal.org repository.	Establishes the secure target destination path on Drupal's servers.
`git push origin 1.0.x`	Uploads the local branch and commit to the server.	Publishes your code online so your mentors can access and review it.

Post this we had committed 3 more files of out own, which are equally crucial for the module development.

1 ai_recipe_generator.module.info

2 composer.json

3 .gitignore

Resources

Module Creation & .info.yml Documentation Creating Custom Modules (Main Hub): This is the central landing page for all module development tutorials in modern Drupal, covering everything from routing to creating custom blocks.

Defining the .info.yml File: This page provides the exact specifications, required properties, and core version requirement syntax for your module's metadata.

Composer Documentation for Modules Adding a composer.json File to a Module: This explains exactly why and how to add a composer.json file to your custom or contributed module, including the composer init steps specifically tailored for Drupal package types.

name: AI Recipe Generator
description: Creates recipes using AI.
package: AI

type: module
core_version_requirement: ^10.5 || ^11.2

dependencies:
  - ai:ai

{
    "name": "drupal/ai_recipe_generator",
    "description": "Creates recipes using AI.",
    "type": "drupal-module",
    "license": "GPL-2.0-or-later",
    "authors": [
        {
            "name": "Hrishikesh Dalal",
            "email": "204620-hrishikesh-dalal@users.noreply.drupalcode.org"
        }
    ],
    "require": {}
}

Somewhere where I struggled and made mistakes

If you have seen the video, you would know I had made a mistake initializing the git repo 😅. I made a commit to the main brand before setting up the 1.0.x which is crucial for creating forks and deployments. Well it sucked but it was a great source of learning.

Understanding the Concept of Remote in Git

Git Remote

Think of Git Remotes as a Cloud Backup or a Shared Office.

Imagine you are writing a book on your own private desk at home (Local Repository). You can write, delete, and organize pages however you want.

However, your publisher and co-authors are in a different building. To show them your work, you have a Shared Drafting Table (Remote Repository) in their office.

1. A "Remote": It's just a bookmark or a "saved contact" in your computer's address book that tells Git exactly where that Shared Table is located on the internet.

2. Origin: This is just the default nickname for that Shared Table. (But you could call it anything, backup, publish etc)

Managing Remotes: The Commands

Sometimes you need to change where your code is going (e.g., commit to a different fork or repository).

1. Check your current remotes

Before changing anything, see where your project is currently "talking" to.

The -v stands for "verbose." It shows you the URLs for fetching (downloading) and pushing (uploading).

git remote -v

2. Change a remote's URL

If the server location changes, but you want to keep the same nickname (origin), use this command:

git remote set-url origin https://git.drupalcode.org/issue/ai_recipe_generator-3592994.git

When to use: Use this if you accidentally typed the wrong URL during setup or if the project moved to a new server.

3. Add a second remote

You can have more than one! You might have one for a private backup and one for the public Drupal project.

git remote add backup https://git.drupalcode.org/issue/ai_recipe_generator-000001.git

Rename a remote

If you don't like the name origin and want to call it something else: (here drupal_server)

git remote rename origin drupal_server

5. Remove a remote

If you want to cut ties with a specific server:

git remote remove backup

Pro-Tip: If you ever get an error saying "Remote origin already exists," it means your project is already linked to a server. Use set-url to update it instead of add.

Eventually....

So after taking a short detour, we finally pushed our initial code to gitlab! 🎉🎉

Final Code Push

Follow me: https://www.linkedin.com/in/hrishikeshdalal/

My portfolio: https://hrishikeshdalal.tech/

Creating a Module Project on Drupal (NO CODE) (With BONUS Section)

Hrishikesh Dalal — Mon, 08 Jun 2026 07:51:00 +0000

Hey there!

Welcome to my GSoC journey where in I am going to take you through an adventure of events. This blog is for the Level 1. Or rather I should saying learning about how to "make the dish". We are going to create our very own Drupal Module Project!!

I will also append a Youtube link for the same if it does suit you better. Lets get into it!

Step 1: Understanding what all types of Projects you can create

When you head over to this site for project creation - Drupal Project Create. You will have 4 options.

Drupal offers a few different project types:

1. Module project: Contains PHP code that extends Drupal or adds new features.

2. Theme project: Changes the look and feel of a Drupal site.

3. General project: Contains code, but isn't a module, theme, or specific type.

4. Community project: Non-code projects used to manage groups or camps.

Since our aim is to build an AI Recipe Generator that adds brand-new functionality to the site, selecting the Module project was the clear choice.

Types of Drupal Projects

Step 2: Have Git on your computer

The absolute bare minimum as all the code changes everything should be done by this. But if you reading this and are a techie, you must already be having git : )

Step 3: Creating a project

Now when you go over this link: Project Creation, you will have a ton of options. Let us go through each one of them one by one.

Drupal core: The foundational software that runs the entire Drupal platform.
Distribution project: An all-in-one starter kit that bundles Drupal with pre-selected features to quickly launch a specific type of website.
Community project: An organizational space for managing Drupal groups, events, and people rather than website code.
General project: A catch-all category for miscellaneous code, tools, and libraries that don't fit into the standard module or theme categories.
Module project: A functional plugin that adds new features or changes how your website works.
Theme project: The design package that controls the colors, layout, and overall visual appearance of your website.
Theme Engine project: The behind-the-scenes system that translates your theme's visual design into something the Drupal platform can understand.
Translation project: An outdated archive for language translations, which are now managed on a separate, dedicated website.

Step 4: The Nitty Gritties

All the things you can do in a snap

Now when you go to the next page, there is this huge form that you need to fill. Let me take you across each filed one by one.

The Basic Identity

Name: AI Recipe Generator

What it is: The project's human-readable title.
Why I chose it: It clearly tells users exactly what the tool does right away.

Short name: ai_recipe_generator

What it is: The machine-readable system name used for URLs and codebases.
Why I chose it: It matches the readable name using standard Drupal convention (lowercase and underscores) for consistency.

Project type: Full project

What it is: Defines if it's a temporary "Sandbox" or an official "Full project."
Why I chose it: To get end-to-end control and official release packaging for this GSoC module.

Status and Classification

Maintenance status: Actively maintained

What it is: Shows the community how much attention the project gets.
Why I chose it: I will be actively monitoring, updating, and managing this code for the next 12 weeks and beyond.

Development status: Under active development

What it is: Indicates the codebase's current maturity.
Why I chose it: The module is currently being built from the ground up and evolving daily.

Module categories & Ecosystem: Artificial Intelligence (AI)

What it is: Tags used to organize and filter Drupal projects.
Why I chose it: Since the module relies on generative LLMs, this ensures the right audience finds it.

Dependencies: AI module

What it is: Other modules required for this to work.
Why I chose it: We are leveraging the existing Drupal AI core module to securely handle API configurations and process prompts.

Issue Tracking and Releases

Components: Code, Documentation, Miscellaneous, User interface

What it is: Categories for issue queue tickets.
Why I chose it: Granular tags keep the project organized and make bug triage much easier.

Enable issue tracker: Checked

What it is: Allows the community to report bugs and submit patches.
Why I chose it: Open source thrives on collaboration, and this is essential for community contributions.

Version options: Tagged releases, Development branch releases

What it is: How Drupal packages the code for download via Composer.
Why I chose it: It gives users the choice between a stable "Tagged" release or the bleeding-edge "Development" version.

The Description that I put is below.

<p>The AI Recipe Generator is a module designed to accelerate site building by utilizing Artificial Intelligence to generate Drupal Recipes dynamically.</p>

<p>By integrating with the Drupal AI ecosystem, this tool allows developers and site builders to input natural language prompts and automatically generate the necessary configuration, module dependencies, and structure required for a functional Drupal Recipe.</p>

<h3 id="module-project--features">Features</h3>
<ul>
  <li>Integrates directly with the Drupal AI module to process prompts.</li>
  <li>Dynamically generates <code>recipe.yml</code> files and associated configurations.</li>
  <li>Streamlines the creation of standard site features (e.g., blog setups, event content types) without manual YAML writing.</li>
</ul>

<h3 id="module-project--post-installation">Post-Installation</h3>
<p>Once installed, you will configure the generator through the AI module's interface. You must have an active AI provider configured via the core AI module to process prompts and begin generating recipes.</p>

<h3 id="module-project--additional-requirements">Additional Requirements</h3>
<p>This module requires the following dependencies:</p>
<ul>
  <li>AI (Artificial Intelligence) module ecosystem</li>
</ul>

<h3 id="module-project--recommended-libraries">Recommended modules/libraries</h3>
<p>There are currently no additional recommended modules beyond the required AI ecosystem dependencies.</p>

<h3 id="module-project--similar-projects">Similar projects</h3>
<p>While standard Drupal site-building tools exist, this project uniquely leverages generative LLMs and the new Drupal Recipe API to architect configurations via natural language.</p>

<h3 id="module-project--support">Supporting this Module</h3>
<p>This project is currently being developed as part of <strong>Google Summer of Code 2026</strong>.</p>

<h3 id="module-project--community-documentation">Community Documentation</h3>
<p>Weekly progress, architectural breakdowns, and documentation for this GSoC 2026 project are being documented under the <em>Engineering in the Wild</em> series.</p>

Further you can also add your logo here. It was also confusing for me to find out where the logo is.

Where to put Drupal Logo

You are almost done!!! You can review and see the details what they are. And then finally you can save and your project is successfully created! You can view my module here:AI Recipe Generator

Bonus: How to Add Maintainers

Most probably, you will be working with someone else. And it will be necessary to add someone there. Head over to the Maintainers tab and select whom you want to add in the search box. You can select the necessary changes to be made and then press enter!

For adding Maintainers

Follow Along for my next journey

Hrishikesh Dalal | Portfolio | Engineer, Innovator, Hackathon Winner

Official website of Hrishikesh Dalal – Computer Engineering student, Hackathon winner, and tech enthusiast.

hrishikeshdalal.tech

The AI Ghostwriter Experiment

Hrishikesh Dalal — Sun, 17 May 2026 11:48:08 +0000

Over the past few months, I ran an experiment on this blog.

Balancing engineering projects, open-source contributions, and university means deep-focus writing is a luxury I rarely have. I wanted to test a hypothesis: Can a developer leverage LLMs to fully automate content creation without losing technical depth?

I launched my "Engineering in the Wild" & "System Design" series to find out. Here is what I learned from letting AI ghostwrite my blog, and why I’m officially pivoting my strategy.

The Workflow

It wasn’t just mindless prompting. I built a specific pipeline to ground the AI in reality:

Input: Raw notes on architectural decisions (like sync APIs vs. Pub/Sub) or details from my ongoing projects.
Generation: Prompting the LLM to structure an engaging, DEV.to-optimized article.
Output: Flawless markdown, clean code snippets, and accessible analogies.

On paper, it worked perfectly. I shipped consistently, and the articles read like polished, textbook overviews that explained complex concepts exceptionally well.

The Flaw: Sanitized Engineering

While the explanations were solid, the experiment revealed a fatal flaw in fully automated content: it completely sanitizes the reality of writing code.

Reviewing the generated articles made a few things glaringly obvious:

Loss of Signal: AI outputs the statistical average of human knowledge. It produces clean, pristine ideals. But real engineering is chaotic and messy.
Erasure of Friction: The real value of a dev blog isn't just the final architecture—it’s the friction it took to get there. AI erased the edge-case geofencing bugs in my attendance platform, the legacy codebase constraints at my enterprise internship, and the brutal code reviews from merging open-source AI modules.
The Commodity Trap: A flawless explanation of a protocol is useful, but if anyone can generate it in ten seconds, it loses its unique edge. It didn’t show how I think; it showed how the model thinks.

The Pivot: Authentic Grit

The foundational overviews the AI helped me build serve their purpose, and AI remains a phenomenal tool. But going forward, the workflow for "Engineering in the Wild" & "System Design" is evolving.

The best documentation isn’t just a flawless essay; it’s a collection of genuine experiences from the trenches of development. It’s time to bring the grit back to this blog.

Even this meta-analysis was generated by AI :) but the pivot is 100% human.

EP 15: Pub/Sub - Stop Chaining Your Services

Hrishikesh Dalal — Wed, 18 Feb 2026 11:20:06 +0000

You just launched your dream startup, "DevEats"—a food delivery app for developers who need serious fuel while debugging. Your signature item? The "Triple Stack Smash Burger with Truffle Fries."

In the beginning, life was simple. You had one backend service (a monolith). A user ordered a burger, your code saved it to the database, and life was good.

But you grew fast. Now, YOU are the owner of a scaling platform, and your code is turning into a tangled mess of spaghetti (pun intended).

Here is why you need the Pub/Sub pattern, explained through the lens of your own burger empire.

1. What is the Need? (The "Smash Burger Panic")

Let’s look at your current codebase. When a user clicks "Order Burger", your backend does this strictly in order:

Payment Service: Charge the credit card $25.
Order Service: Save the order to the database.
Notification Service: Send a receipt email.
Kitchen Service: Print the ticket for the chef.
Analytics Service: Update your admin dashboard sales graph.

The Problem:
One Friday night, your Email Provider goes down.
Because your code is sequential (A -> B -> C -> D), the entire process crashes at Step 3.

The payment went through (Step 1), but the kitchen printer never got the ticket (Step 4).

Now you have a hangry developer who paid for a Triple Stack Burger that isn't being made. They are angry, and your support inbox is flooding.

The Need:
You need a way for your system to say, "Hey, a burger was just ordered!" without caring who is listening or if the email service is currently online. You need to decouple the sender from the receivers.

2. The Solution: Pub/Sub Architecture

In a Publish/Subscribe (Pub/Sub) model, we split your architecture into three parts:

The Publisher (You/The Order Service): You simply shout a message ("Order #505: Triple Smash Burger!") into a channel. You don't know who is listening. You don't care.
The Topic (The Channel): A dedicated lane for specific types of messages (e.g., events.burger_ordered).
The Subscribers (The Listeners): Independent services that have signed up to listen to that topic.

The New Flow:

User orders the burger.
You (Order Service) publish a message to the events.burger_ordered topic.
Done. You return "Success" to the user immediately.

Meanwhile, in the background:

The Kitchen Service hears the message and instantly prints the ticket on the grill station.
The Notification Service hears the message and tries to send an email. If it fails (because the provider is down), it retries later. Crucially, it does not stop the kitchen from cooking.
The Rewards Service hears the message and adds 50 points to the user's account.

3. When to Use It?

You shouldn't use Pub/Sub for everything. Use it when:

1-to-Many Relationships: One event (Burger Ordered) triggers multiple, unrelated actions (Email, Kitchen Display, Inventory Check).
Decoupling is Vital: You want to add a new "SMS Service" next week without rewriting your existing Order Service code.
Asynchronous Processing: You don't want the user staring at a loading spinner while your server generates a PDF invoice.

4. The Pros and Cons

✅ The Pros

Scalability: You can add 50 new subscribers (e.g., a "Surge Pricing Bot") without touching the original Publisher code.
Reliability: If one subscriber crashes (e.g., the Email service), the others (Kitchen, Analytics) keep working perfectly. The burger still gets made.
Speed: The Publisher fires the event and moves on. The user gets a "Your Order is Placed!" screen instantly.

❌ The Cons

Complexity: You are introducing a new component (a Broker like Redis, Kafka, or Google Pub/Sub) to manage.
Debugging is a Nightmare: You can't just follow a linear stack trace anymore. You have to trace messages flying across a network.
Consistency Issues: What if the message is delivered twice? (Does the customer get charged twice? Does the chef make two burgers?). You have to write code to handle these edge cases.

5. Pub/Sub vs. Message Queue (The Confusion)

This is where most devs get tripped up. Both use "Brokers," but the intent is different.

Scenario A: Pub/Sub (The "Megaphone")

Use Case: You launch a New limited-edition Ghost Pepper Burger.
Logic: 1 Message -> Many Receivers.

Publisher: The Menu Manager Service.
Subscribers:
App Notification Service: Pushes "New Burger Alert!" to all users.
Inventory Service: Reserves spicy peppers in the warehouse.
Marketing Service: Posts automatically to Twitter/X.
Why? Everyone needs to know about this event to do their own specific job.

Scenario B: Message Queue (The "Grill Line")

Use Case: It's lunch rush. You have 500 burger orders coming in.
Logic: 1 Message -> 1 Receiver (Load Balancing).

Producer: The Order Service pushes 500 tickets to a Queue.
Consumers (Chefs): You have 5 Chefs (Chef A, B, C, D, E).
The Flow: Chef A grabs Ticket #1. Chef B grabs Ticket #2.
Why? If Chef A is grilling the burger for Ticket #1, you do not want Chef B to grill the same burger. That’s a waste of meat. You want to distribute the work, not broadcast it.

The Cheat Sheet

Feature	Pub/Sub	Message Queue
Philosophy	"Everyone needs to know this happened."	"Someone please do this job."
Distribution	Broadcast (One-to-All)	Point-to-Point (One-to-One)
Burger Example	"New Menu Item Launched!" (Tell everyone)	"Grill this patty" (One chef does it)

Final Words

As the owner of DevEats, switching to Pub/Sub saved your Friday night rush. When your Email Service crashed, nobody noticed. The orders kept flowing, the smash burgers kept sizzling, and the emails were just queued up and sent an hour later.

That is the power of decoupling. 🍔 code responsibly!

The Jungle Protocol: Turning my Professional Journey into an Adaptive Quest

Hrishikesh Dalal — Sun, 01 Feb 2026 13:58:07 +0000

This is a submission for the New Year, New You Portfolio Challenge Presented by Google AI

About Me

I’m Hrishikesh Dalal, a Computer Engineering student at Sardar Patel Institute of Technology (SPIT), Mumbai. I don't just write code; I build experiences. Currently, I’m engineering a Jumanji-themed gamified portfolio that turns the standard resume into an adventure.

My focus lies in Full Stack Development (Next.js, React) and creating developer tools—I recently published envalyze and env-fault to the NPM registry to help devs manage environments better. When I'm not pushing commits to Open Source projects like Drupal or contributing to MDN Web Docs, I'm exploring the intersection of AI and law with projects like VerdictAI.

I write about system design, open source, and the reality of building software.

Portfolio

Link: https://hrishikesh-dalal-portfolio-571902073238.europe-west1.run.app

How I Built It

My Tech Stack

To build an experience this immersive, I needed a stack that balanced performance with high-end visuals:

Frontend: Next.js & Tailwind CSS
Animations: GSAP (for that cinematic feel)
3D Elements: Three.js (for the interactive globe and hero patterns)
AI:Gemini API (powering the NPC-style chatbots)

The Process: From Chaos to Clarity

Building a portfolio is a trap. You want to show off everything, but if you're not careful, you end up with a bloated mess.

1. The Affinity Mapping Phase

Initially, I was drowning in ideas from my UI/UX course. I had a laundry list: a full 3D world, a terminal, hidden easter eggs, business-centric project stats, and a Jumanji-themed game. I used Affinity Mapping to dump every "cool" idea onto a board and then ruthlessly segregated them.

2. The "Reality Check"

I had to ask the most important question: Why is someone here?
Hiring managers are busy. They might be tech-savvy, or they might not be. They might want to play a game, or they might just want to see my resume and leave in 30 seconds.

The Decision: A "Clean & Clear" landing page with a defined Call to Action (CTA) is non-negotiable. The "wow" factor should enhance the info, not hide it.

Design Breakdown: Section by Section

The Hero (The First 5 Seconds)

I treated the Hero section like a movie opening. I implemented a preloader with a cinematic transition that drops you into an interactive dot pattern. It’s meant to grab attention immediately.

The Impact (Big Numbers & Open Source)

Instead of just saying "I build tools," I used big, bold typography to showcase my NPM stats. I wanted the real-world impact of envalyze and env-fault to be unavoidable. I also dedicated space to my Open Source journey—showing my Hacktoberfest badges and my role as a project admin.

The Experience Timeline

I used a scroll-triggered timeline with GSAP micro-interactions. When you hover over an entry, there’s a slight movement—just enough to feel alive, but not enough to be distracting.

The Terminal (For the Geeks)

I built a fully functional terminal for my projects. You can change themes, run an echo, or see animations.

Pro Tip: Try typing sudo. You’ve been warned.

The Accessibility Switch: If you aren't a terminal person, there's a toggle to switch back to a standard grid. No one gets left behind.

The "Jungle" Experience

This was the hardest part to get right. My first version was just a collection of games, and the feedback from friends was brutal: "It’s disconnected. I don't see YOU in this."

I went back to the drawing board and turned it into a narrative. I added:

A Prologue & Epilogue: To give the "quest" a purpose.
Level 2 (The Jumper): While you play, the character narrates my tech stack and experience journey.
Secret Cheat Codes: Because what's a game without them?

Bringing it to life with AI

My main chatbot you have different answers queries, sample questions that you can ask to make it better.

While the game chatbot is like NPC character answering you.

The Architecture

The diagram below shows how the game chatbot is integrated to handle context-aware queries, ensuring it knows where you are in the game and who I am as a developer.

The User Interaction Layer
User Sends Message: The process begins with the user interacting with the interface.
Frontend App (NEXT JS): The core UI is built using React. It features a React Markdown Renderer, which ensures that the AI's responses—often containing code snippets or formatted text—are displayed cleanly and professionally to the user.
Security and Traffic Management
Backend API Gateway & Security Layer: Instead of calling the Gemini API directly from the frontend (which would expose your API keys), the app calls a secure backend gateway. This layer handles authentication and protects your credentials.
Load Balancer: To prevent any single server from becoming overwhelmed, a load balancer sits in the middle. It analyzes incoming requests and routes them efficiently across multiple active instances.
The Processing Core
Chat Model Instances: The system uses multiple parallel instances (Instance 1, 2, and 3) to process requests.
Least Used Routing: The load balancer is shown routing a request specifically to "Instance 1" because it is currently the "Least Used," ensuring optimal performance and low latency.
Google's Gemini API: These instances act as the bridge to Google’s infrastructure. They send the processed user prompt to the Gemini API and receive the raw AI generation in return.
The Response Loop
Forwards Response: Once a chat instance receives the data from Google, it forwards that response back through the secure backend.
Final Display: The message is sent back to the NEXT JS app, where it is rendered as Markdown, completing the loop and appearing on the user's screen.

Easter Eggs

I have a game at the start where the user can press the "?" button and play that simple.

"Stay Hungry, Stay Foolish." My portfolio ends with this quote because it sums up my approach to engineering. Whether it's deep-diving into the Drupal source code or building a terminal from scratch, I’m always looking for the next thing that feels slightly impossible to build.

What I'm Most Proud Of

If you’re a developer, you know the feeling: you spend 10% of your time on the features people see and 90% of your time on the "over-engineered" systems that make you feel like a wizard. Here are the pillars of this portfolio that I obsessed over.

1. The Terminal (Because real devs use a CLI)

I didn't want a static "Projects" page; I wanted something that felt like home. I built a fully functional, web-based Terminal where you can actually interact with my work.

The Experience: It’s a working CLI where you can navigate projects, change themes, or trigger animations.
The "Sudo" Trap: I’ve hidden easter eggs for people who like to poke around—try typing sudo.
The Hybrid UI: I know recruiters aren't always looking to type commands, so I built a "safety net" toggle that instantly switches the terminal into a standard project grid.

2. Projects with a "Business First" Logic

Most portfolios just show a screenshot and a GitHub link. I decided to treat my projects like real products.

Beyond the Code: When you click into a project, I don't just list the tech stack. I break down the Business Perspective, the Impact, and the Stats.
The "Why" Factor: I included the reasoning behind my architectural choices so that a non-technical stakeholder can understand the value I created, not just the code I wrote.
Deep Dives: Each project page is designed to show the journey from a pain point to a scalable solution.

3. The Jumanji Game Engine

I turned my professional experience into a narrative quest. This wasn't just about putting a game on a website; it was about making the resume playable.

The Narrative: I added a full prologue and epilogue to make sure it felt like a real quest, not just a random mini-game.
The Career Jumper: In Level 2, the character literally narrates my tech stack and my journey through different roles while you play.
Recruiter Mode: I even made the game adaptive—a recruiter can upload a job description, and the game will fine-tune the information it highlights to match what they need.

4. The Chatbot Architecture

I’m most proud of what’s happening "under the hood" here. I built a production-ready pipeline for the Gemini-powered Chatbot.

The Load Balancer: My system uses three different chat model instances. I implemented a load balancer that routes traffic to the "least used" instance to keep things snappy, no matter the traffic.
Security & Scalability: I kept the logic on the backend. The React frontend talks to a secure API Gateway, which handles the Gemini API calls to keep my keys safe.
The NPC Persona: I used the Gemini API to give the bot a specific "In-game NPC" persona—it knows it’s in the Jungle, but it also knows my GitHub stats.

System Design: EP 14 - The Coffee Shop Guide to Understanding Message Brokers

Hrishikesh Dalal — Wed, 21 Jan 2026 11:17:47 +0000

If you are a developer moving from a monolith to microservices, you’ve probably hit The Problem. Service A calls Service B, but Service B is slow, or down, or just busy. Suddenly, Service A hangs, the user sees a spinning wheel of death, and you get paged at 3 AM.

The solution? Stop making your services talk directly. Introduce a Message Broker.

If that sounds abstract, let's talk about coffee.

The "Bad" Coffee Shop (Synchronous)

Imagine a coffee shop where the person taking your order is also the only person allowed to make your coffee.

You (The User) walk in and order a latte.
The Cashier (Service A) takes your money.
The Cashier then walks over to the espresso machine, steams the milk, pulls the shot, pours the art, and hands it to you.
Only then do they return to the register to take the next person's order.

The Result: A massive line out the door. If the espresso machine breaks, the cashier is stuck waiting, and nobody can even place an order. This is Synchronous Communication (like HTTP REST calls). It creates tight coupling and bottlenecks.

The "Good" Coffee Shop (Asynchronous with a Broker)

Now, let's look at how a Starbucks or a high-volume cafe actually works.

You (The User) order a latte.
The Cashier (Service A / Producer) takes your money and writes your order on a cup (or a ticket).
The Cashier places the cup on the Counter/Ticket Rail (The Message Broker).
The Cashier immediately turns back to you and says, "Next please!"
The Barista (Service B / Consumer) sees the cup on the rail, picks it up, and makes the coffee.

The Result: The cashier never stops working. They can take 50 orders in the time it takes the barista to make 5 drinks. The "Ticket Rail" acts as a buffer.

The Technical Translation

That "Ticket Rail" is your Message Broker (like RabbitMQ, Kafka, or AWS SQS). Here is the mapping for your next system design interview:

1. The Producer (The Cashier)

This is the service that sends the data. In the coffee shop, the cashier "produces" the order. They don't care who makes the coffee or when it gets made; they just need to know the order has been safely placed on the rail.

Dev Term: Publish

2. The Message Broker (The Ticket Rail)

This is the middleware that sits between your services. It holds the messages (orders) until a consumer is ready to take them.

The Queue: The line of cups waiting on the machine. If the baristas get overwhelmed, the queue grows, but the cups aren't lost. They are safely stored until the backlog clears.

3. The Consumer (The Barista)

This is the service that processes the data. The barista "consumes" the order from the rail.

Dev Term: Subscribe
Scaling: If the line of cups gets too long, what do you do? You don't clone the cashier; you add a second barista. Message brokers allow you to scale your processing power (Consumers) independently of your intake power (Producers).

Why use a Message Broker?

Using the coffee shop model, the benefits become obvious:

Decoupling: The cashier doesn't need to know which barista is working today. If the barista quits (Service B crashes), the cashier can keep taking orders. The cups just pile up on the rail until a new barista is hired (Service B restarts). No data is lost.
Throttling (Load Leveling): If 100 people rush the store at once, the cashier takes all the orders quickly. The baristas continue working at their normal, safe speed. They don't explode from stress; they just work through the backlog.
Asynchronous Processing: Heavy tasks (like roasting beans or baking pastries) don't block the customer from paying and leaving.

Summary

Direct HTTP calls are like a cashier making every drink themselves. It works for a lemonade stand, but it fails at scale.

A Message Broker is the ticket rail that lets your services work at their own speeds, independently and reliably.

So, the next time you're architecting a system, ask yourself: Are we building a lemonade stand, or are we building a coffee empire?

System Design - EP 13: Content Delivery Networks (CDNs)

Hrishikesh Dalal — Fri, 16 Jan 2026 16:11:48 +0000

We’ve all been there. You click a link, and the page hangs. You stare at the loading spinner, your patience draining with every second. In 2026, a slow website isn't just annoying; it’s a dealbreaker.

As developers, we know Speed = Revenue. But how do you make a website fast for a user in Tokyo when your server is sitting in a basement in New York?

Enter the CDN (Content Delivery Network).

If you've ever nodded along while someone said "just put it behind Cloudflare" but were secretly fuzzy on the details, this article is for you. Let's break down CDNs using a scenario we can all understand: The Global Pizza Empire.

The Scenario: "Uncle Tony's Pizza"

Imagine you run the world's best pizza shop, Uncle Tony's, located in New York City. Your pizza is legendary. People fly in from all over the world just to get a slice.

In technical terms, your New York shop is the Origin Server. It is the single source of truth where the pizza (content) is created.

The Problem: The Latency Delivery

One day, you decide to offer global delivery.

Customer A lives in Brooklyn (5 miles away). They get their pizza hot and fresh in 20 minutes.
Customer B lives in London (3,500 miles away). You have to bake the pizza in NY, put it on a supersonic jet, and fly it to London. It arrives 6 hours later, cold and soggy.

This delay is Latency. In the web world, even if your server (New York Kitchen) is super fast, the physical distance to the user (London Customer) creates unavoidable lag.

The Solution: The Franchise Model (The CDN)

You realize you can't defy physics. You can't make the jet faster. So, you change your strategy. You open small, reheating stations in major cities around the world: London, Tokyo, Mumbai, and Sydney.

These stations are your Edge Servers. They don't have the full kitchen setup of the NYC headquarters, but they are capable of holding inventory.

How the "Caching" Works

You can't bake every single unique pizza at these small stations. Instead, you look at your data. You realize 80% of people just order Pepperoni or Cheese.

The First Request (Cache Miss): A guy in London orders a Pepperoni pizza. The London station is empty. They call the NY HQ, get a Pepperoni pizza flown over, and deliver it. It’s slow, but now the London station keeps a stash of 50 frozen Pepperoni pizzas in their freezer.
The Second Request (Cache Hit): Ten minutes later, a girl in London orders a Pepperoni pizza. The London station checks its freezer. Bingo! They grab one, heat it up, and deliver it in 10 minutes. They didn't even call New York.

The Theory: Key Concepts in CDNs

Now that we understand the pizza model, let's look at the actual technical terms you'll need for your system design interviews or cloud configs.

1. Edge Servers

These are geographically distributed servers where CDN providers (like Cloudflare, Akamai, or Fastly) cache your content. Users are routed to the nearest edge server for faster delivery.

Pizza Translation: The local reheating stations in London or Tokyo.

2. Origin Server

This is your main web server (e.g., an AWS EC2 instance or S3 bucket). The CDN fetches content from here only if it’s not already cached at the edge server.

Pizza Translation: The main Uncle Tony's Kitchen in New York.

3. Caching

CDNs store copies of your static content (e.g., images, videos, CSS, HTML) in their edge servers. Cached content is served directly to users, reducing the need for frequent requests to the origin server.

Pizza Translation: Storing frozen Pepperoni pizzas in the local freezer so you don't have to bake a fresh one every time.

4. TTL (Time to Live)

The duration for which a file is cached on a CDN edge server.
Example: An image might have a TTL of 24 hours. This means it stays cached for 24 hours; after that, the Edge Server assumes it might be stale, deletes it, and fetches a fresh copy from the Origin.

Pizza Translation: The expiration date on the frozen pizza. You throw it out after 24 hours to ensure quality.

5. GeoDNS

CDNs use GeoDNS to route users to the nearest edge server based on their geographic location. When a user types in your URL, the DNS resolver looks at their IP address and says, "You're in Germany? Okay, talk to the Frankfurt server, not the New York one."

Pizza Translation: The call center routing your order to the nearest branch automatically.

Why this Architecture Wins

1. Speed (Performance)

Your users in Tokyo aren't downloading images from New York anymore; they are downloading them from a server in Shinjuku. The data travels fewer miles, reducing the Time to First Byte (TTFB).

2. Scalability (Traffic Spikes)

Imagine it's Super Bowl Sunday. Everyone calls New York at once. The phone lines jam. The kitchen catches fire.
With the Franchise model (CDN), the New York kitchen only handles the weird custom orders. The millions of standard Pepperoni orders are handled by the thousands of local branches. Your main kitchen survives.

3. Security (DDoS Protection)

Imagine a competitor hires a million bots to prank call your New York Pizza shop so no real customers can get through.
With a CDN, these prank calls hit the thousands of local branches first. The branches are smart—they realize it's a prank and hang up before the call ever reaches the New York HQ.

System Design EP: 12 - Why Your Database Hates Your Images: A Guide to BLOBs

Hrishikesh Dalal — Wed, 14 Jan 2026 14:34:27 +0000

Ummm, BLOBS. Everyone has heard that term somewhere sometime across their development journey, but what are those?

In system design, a BLOB (Binary Large Object) is any unstructured data that doesn't fit neatly into a database row. Unlike integers or strings, the database engine doesn't "understand" this data,it just sees a massive chunk of bytes.

Examples: User avatars (JPEG), video files (MP4), PDFs, Application Logs, Backup files.

Size Range: From a few KBs (thumbnails) to TBs (genome sequencing data).

So basically storing mp4 file directly as a file is difficult, how to store it say in MySQL but it becomes easier when we store in Blobs using 0 & 1s.

When looking for the "definitive" resource on Blobs (Binary Large Objects) in System Design, most engineers end up piecing together chapters from different sources.

Since a single perfect article is hard to find, I have synthesized the industry-standard approach below—essentially the article you are looking for—followed by links to the best external deep-dives.

The Great Debate: Database vs. Object Storage

The most common interview question and real-world decision is: "Should I store user uploads in my SQL database or on disk?"

Option A: Storing BLOBs in the Database

You store the image directly in a column (e.g., VARBINARY in MySQL or BYTEA in PostgreSQL).

Pros: ACID compliance (the image and the user profile update commit together), easy backups (one dump file).
Cons:
Performance Suicide: Databases are optimized for small, random reads/writes. Reading a 10MB image consumes the same I/O as reading thousands of user rows.
Cost: Block storage (SSD for DBs) is 3-5x more expensive than Object Storage.
Scalability: You cannot easily cache database responses at the edge (CDN) compared to static URLs.

Option B: Storing BLOBs in Object Storage (The Standard)

You store the actual file in a service like Amazon S3, Google Cloud Storage, or Azure Blob, and store only the reference URL in your database.

Pros: Infinite scalability, cheaper storage tiers, built-in redundancy, easy integration with CDNs.
Cons: Loose consistency (you might delete the database row but forget to delete the S3 file, creating "orphan" data).

The Verdict: 99% of the time, use Object Storage (Option B). Only use Option A if your files are tiny (<20KB), strictly transactional, and security is paramount (e.g., encryption keys or sensitive legal docs that must never leak via a public URL).

The Architecture Pattern

The standard design pattern for handling blobs involves decoupling metadata from storage.

The Workflow:

Client Request: User uploads a profile picture.
API Gateway: Authenticates the user.
Presigned URL: Instead of uploading to your server (which blocks your server's threads), your backend generates a "Presigned URL" from S3. This authorizes the client to upload directly to the bucket for a limited time.
Direct Upload: Client uploads the binary data directly to S3/Blob Storage.
Confirmation: S3 returns a success code. The client notifies your backend: "Upload Complete."
Metadata Save: Your backend saves the file path (e.g., s3://my-bucket/users/123/avatar.png) into the SQL database users table.

EP 11: The Legend of ShopStream: The In-Memory Revolution

Hrishikesh Dalal — Tue, 13 Jan 2026 10:15:59 +0000

Akash had fixed his database latency problems by using Redis as a cache. ShopStream was fast. But Akash was about to learn a hard lesson: Redis is not just a "dumb" cache. It is a Ferrari engine that he was using to drive to the grocery store.

Chapter 1: The Day the Lights Went Out (Persistence)

One stormy night, the power failed at the data center. The Redis server rebooted.

When the lights came back on, Akash checked the dashboard.

User Sessions: Gone. Everyone was logged out.
Shopping Carts: Empty.
Revenue Impact: Massive.

Akash panicked. "I thought Redis stores data in RAM! If RAM loses power, data is gone!"
He realized he needed Persistence. He had two options:

Option A: The Photographer (RDB - Redis Database Backup)

Akash configured Redis to take a "Snapshot" every hour.

How it works: Every hour, Redis forks a background process to save all data to a file (dump.rdb) on the hard disk.
The Trade-off: It’s compact and fast to restore. But, if the server crashes at 4:59 PM, he loses 59 minutes of data since the last snapshot at 4:00 PM.

Option B: The Stenographer (AOF - Append Only File)

This wasn't enough for "Shopping Carts." So Akash turned on AOF.

How it works: Every time a command runs (e.g., SET cart:123 "Apple"), Redis logs that command to a file on the disk immediately.
The Trade-off: No data loss! But the file grows huge over time because it records every change.

The Solution: Akash used a hybrid approach. He used AOF for critical data (carts/sessions) and RDB for less critical cache data to balance speed and safety.

Chapter 2: The Data Structure Buffet (Beyond Strings)

ShopStream was growing. Akash was building a "Live Leaderboard" for the most active shoppers.

The "Newbie" Way:
Akash was storing the leaderboard as a JSON string in Redis:
"[{user: 'Alice', score: 10}, {user: 'Bob', score: 5}]"
Every time Alice bought something, Akash had to:

GET the whole JSON string.
Decode it in his backend code.
Update Alice’s score.
Sort the array again.
SET the whole JSON string back to Redis.

This was slow and caused "Race Conditions" (two users updating at once).

The "Pro" Way (Redis Data Structures):
Akash realized Redis isn't just a Key-Value store; it's a Data Structure Server.

Sorted Sets (The Leaderboard Fix): He used the ZSET data type.
Command: ZADD leaderboard 10 "Alice"
Command: ZINCRBY leaderboard 5 "Alice"
Magic: Redis automatically kept the list sorted in RAM. To get the top 3 users, Akash just ran ZREVRANGE 0 2. No application code needed.
Hashes (The User Profile):
Instead of storing a user as a big blob of text, he used HASH.
Command: HSET user:101 name "Akash" email "a@test.com"
Benefit: He could update just the email without reading/writing the whole user object.
Lists (The Job Queue):
When a user bought an item, the system needed to send an email. Instead of making the user wait, Akash pushed the "Send Email" task into a Redis LIST.
Command: LPUSH email_queue {user_id: 101}
A background worker simply monitored the list (BRPOP) and processed emails instantly.

Chapter 3: The Traffic Jam (Single Threaded Nature)

One day, the app froze. CPU usage on the Redis server was at 100%.

Akash looked at the logs. A junior developer had run a command: KEYS * (Give me ALL keys).
Because ShopStream had 10 million keys, Redis had to iterate through every single one.

The Critical Lesson:

Redis is Single-Threaded. It processes one command at a time. It is incredibly fast (handling 100,000+ ops/sec), but if one command takes 1 second (like KEYS *), all other 99,999 requests are blocked waiting in line.

The Fix: Akash banned the KEYS command and used SCAN (which reads keys in small batches) to prevent blocking the main thread.

Chapter 4: Too Big to Fail (Sentinel vs. Cluster)

The data grew to 500GB. But the server only had 64GB of RAM. The server crashed with an OOM (Out of Memory) error.

Akash faced the ultimate architectural choice: Scale Up or Scale Out?

Scenario A: High Availability (Sentinel)

Problem: "If my main Redis server dies, the app dies."
Solution: Akash set up Redis Sentinel.
Architecture: 1 Master Node (Write), 2 Slave Nodes (Read).
How it works: Sentinel watches the Master. If the Master dies, Sentinel automatically votes for one of the Slaves to become the new Master.
Limit: The total data size is still limited to the RAM of one node (64GB).

Scenario B: Infinite Scale (Redis Cluster)

Problem: "I have 500GB of data. I need more RAM."
Solution: Akash migrated to Redis Cluster.
How it works: He bought 10 servers (nodes). Redis automatically split the data across them using Sharding.
Keys A-M go to Node 1.
Keys N-Z go to Node 2.
The Magic: The client app doesn't need to know which node has the data. It asks the cluster, and the cluster routes the request to the right node.

The Moral of the Story

Akash learned that Redis is the "Swiss Army Knife" of backend engineering.

Persistence: Use AOF if you can't lose data; RDB for backups.
Logic: Don't pull data to your code to sort it. Push your data into Sorted Sets or Lists and let Redis do the work.
Performance: Never block the single thread.
Scaling: Use Sentinel for reliability, use Cluster for massive data size.

Summary Cheat Sheet

Feature	Concept	Use Case
String	Basic Key-Value	Caching HTML pages, Sessions.
List	Linked List	Message Queues, Timelines (Twitter feed).
Set	Unordered Unique	Storing Followers, IP Whitelists (fast lookups).
Sorted Set (ZSet)	Sorted by Score	Leaderboards, Priority Queues.
Hash	Field-Value Map	Storing Objects (User Profiles, Product Details).
Pub/Sub	Radio Station	Real-time Chat, Notification Systems.
TTL (Time to Live)	Expiry	Auto-deleting OTPs or Cache data.