DEV Community: QURBAN AHMAD

Crowd Safety at Scale: Lessons from San Fermín and the Mahakumbh Mela Tragedy

QURBAN AHMAD — Sun, 16 Feb 2025 07:26:29 +0000

Large gatherings—whether religious festivals, concerts, or sporting events—are a testament to humanity’s ability to come together in celebration. But as we’ve seen time and again, they can also turn tragic when crowd management fails. The recent stampede at the Mahakumbh Mela in Prayagraj, Uttar Pradesh, which claimed 30 lives and left many injured, is a heartbreaking reminder of this reality.

But how can we prevent such tragedies in the future? A fascinating study published in Nature (link to the paper) offers some groundbreaking insights. By analyzing crowd behavior at the San Fermín festival in Pamplona, Spain, researchers have uncovered patterns that could revolutionize how we manage large-scale events.

In this post, we’ll dive into the study’s findings, explore their implications, and discuss how technology and data-driven strategies can make large gatherings safer.

The Science of Crowd Behavior: What the Study Revealed

Researchers spent four years studying the San Fermín festival, which attracts thousands of participants for its famous “running of the bulls.” Using strategically placed cameras and sensors, they monitored crowd density and movement in a 50m x 20m plaza. Here’s what they found:

Crowd Density Matters
- At 2-6 people per square meter, crowds move in a relatively orderly fashion.
- When density reaches 6-8 people per square meter, movement becomes more predictable but also more constrained.
- At a critical threshold of 9 people per square meter, something extraordinary happens: the crowd begins to oscillate in a rhythmic, wave-like pattern every 18 seconds, even without any external triggers.
Why Does This Happen?

This phenomenon is an example of emergent behavior—a situation where individuals unconsciously synchronize their actions, creating a collective pattern that isn’t directed by any single person or external factor. Here’s how it works:
- At 9 people per square meter, individuals are packed so tightly that their movements are heavily constrained.
- Even small shifts in position—like someone adjusting their footing or leaning slightly—create a ripple effect that propagates through the crowd.
- Over time, these ripples synchronize into a rhythmic, wave-like motion, much like a pendulum swinging back and forth.
What Does It Look Like?

Imagine standing in a tightly packed crowd. Suddenly, you feel a gentle push from behind. A few seconds later, the push comes again, and then again, like a wave. This isn’t because someone is intentionally pushing you—it’s because the entire crowd is moving in unison, almost like a fluid. These waves can travel through the crowd, creating a back-and-forth motion.
Why Is This Dangerous?

While this rhythmic movement might sound almost poetic, it’s actually a warning sign of potential danger. Here’s why:
- Increased Pressure: The wave-like motion creates pressure within the crowd. As people are pushed back and forth, the force can build up, making it hard to breathe or move.
- Loss of Control: Individuals lose the ability to move independently. If someone falls or stumbles, the waves can make it nearly impossible for them to get back up, increasing the risk of trampling.
- Risk of Stampedes: If the pressure becomes too great, the crowd can panic, leading to a stampede. This is especially dangerous in confined spaces where there’s no easy way to escape.

What This Means for Large Gatherings in India

The Mahakumbh Mela, which attracts millions of devotees, is one of the largest gatherings in the world. Managing such a massive crowd is a monumental task, and the recent stampede underscores the urgent need for better crowd management strategies.

The insights from the San Fermín study are particularly relevant here. By understanding how crowds behave at different densities, we can anticipate risks and take proactive measures to prevent disasters.

How Technology Can Help: A Data-Driven Approach

Here’s where technology and data science come into play. By leveraging real-time monitoring, predictive modeling, and smart design, we can make large gatherings safer. Here are some actionable strategies:

1. Real-Time Crowd Monitoring

High-Resolution Cameras and Sensors: Deploying cameras and IoT sensors can help track crowd density and movement in real time.
AI-Powered Analytics: Machine learning algorithms can analyze video feeds to detect anomalies, such as sudden increases in density or unusual movement patterns.

2. Predictive Modeling

Mathematical Models: Using equations derived from fluid dynamics, we can predict how crowds will behave as density increases.
Simulations: Running simulations based on historical data can help organizers identify potential trouble spots and plan interventions.

3. Controlled Access and Flow Management

Timed Entry: Limiting the number of people entering a space at any given time can prevent overcrowding.
Dynamic Barriers: Deploying retractable barriers or digital signage to redirect crowds in real time can help maintain safe densities.

4. Public Awareness and Communication

Mobile Alerts: Sending real-time alerts to attendees’ phones can inform them about crowded areas and suggest alternative routes.
Clear Signage: Well-designed signs and announcements can guide people and reduce confusion.

The Bigger Picture: Designing Safer Spaces

The study also emphasizes the importance of space design in crowd management. Here are some key takeaways:

Wider Pathways: Ensuring that walkways are wide enough to accommodate large crowds can reduce the risk of bottlenecks.
Multiple Exit Points: Having multiple, clearly marked exits can help disperse crowds quickly in case of an emergency.
Open Spaces: Designing venues with open areas where crowds can spread out can prevent dangerous densities from forming.

A Call to Action: Collaboration and Innovation

The tragedy at the Mahakumbh Mela is a wake-up call. But it’s also an opportunity—to learn, innovate, and collaborate. By combining insights from research, the power of technology, and thoughtful design, we can create safer, more inclusive experiences for everyone.

If you’re working on crowd management solutions, data analytics tools, or IoT devices, I’d love to hear from you.

Final Thoughts

Large gatherings are a celebration of our shared humanity. But they also come with risks that we can’t afford to ignore. By understanding crowd behavior, leveraging technology, and designing smarter spaces, we can ensure that these events remain joyful and safe for all.

What are your thoughts on using technology for crowd management? Have you worked on similar projects? Let’s discuss in the comments!

ChamaleonLLM: Dynamic Adaptation for Large Language Models During Inference

QURBAN AHMAD — Sun, 09 Feb 2025 08:48:08 +0000

Hey everyone! 👋 I recently came across an exciting research paper titled ChamaleonLLM: Batch-Aware Dynamic Low-Rank Adaptation via Inference-Time Clusters by Kamer Ali Yuksel and Hassan Sawaf from aiXplain Inc., and I wanted to share my learnings with you all. The paper introduces a novel framework that enables dynamic adaptation of large language models (LLMs) during inference, which is a game-changer for improving their flexibility and efficiency. Let’s dive into the details!

The Problem with Static LLMs

Large language models like GPT-3 and GPT-4 have revolutionized natural language processing (NLP) with their ability to generate human-like text, summarize documents, translate languages, and more. However, these models are typically deployed with fixed weights, meaning they cannot adapt to new or varying data during inference (the phase where the model generates predictions). This static nature can lead to suboptimal performance when the input data differs from what the model was trained on.

For example, if a model is trained on formal text but encounters informal or noisy data during inference, it may struggle to generate accurate or coherent responses. Traditional fine-tuning methods like Low-Rank Adaptation (LoRA) help by introducing small, efficient updates to the model's weights, but these updates are still static during inference. This is where ChamaleonLLM comes in!

What is ChamaleonLLM?

ChamaleonLLM is a framework that enables dynamic adaptation of LLMs during inference. Instead of using fixed weights or pre-learned updates, ChamaleonLLM adapts the model's behavior on-the-fly based on the statistics of the input batch. Here’s how it works:

Key Innovations

Batch-Aware Clustering:
- Inputs in a batch are grouped into clusters based on their token embeddings (numerical representations of words or sentences).
- This clustering ensures that similar inputs are processed together, allowing the model to capture shared context and reduce noise.
Dynamic Low-Rank Updates:
- A hyper-network (a smaller neural network) generates low-rank updates (small adjustments to the model's weights) tailored to the statistics of each cluster.
- These updates are computed in real-time, enabling the model to adapt dynamically to the specific characteristics of the input batch.
Efficiency:
- Unlike traditional methods that require storing multiple expert models or masks, ChamaleonLLM generates updates on-the-fly, reducing memory and computational overhead.

How Does ChamaleonLLM Work?

The framework is built on a pre-trained causal language model (e.g., GPT-2) and consists of two main components:

1. Batch-Aware Clustering

Inputs are tokenized and converted into token embeddings.
These embeddings are normalized and grouped into clusters using k-means clustering, a simple algorithm that minimizes the distance between points and cluster centroids.
Each mini-batch contains inputs from the same cluster, ensuring that the model processes contextually similar data together.

2. Adaptive Low-Rank Update Generation

A hyper-network takes the mean token embeddings of each cluster as input and generates low-rank update parameters.
These updates are applied to the model's weights, allowing it to adapt to the specific characteristics of the cluster.
The hyper-network is trained to produce updates that improve the model's performance on the given batch.

Why is ChamaleonLLM Better?

The authors compare ChamaleonLLM with traditional LoRA and unadapted GPT-2 models on the WikiText-2 dataset, a benchmark for language modeling. Here are the key results:

Adaptation Regime	Parameters	Validation Loss	Validation Perplexity
Unadapted GPT-2	124,439,808	10.2513	28,319
Traditional LoRA	204,100	1.3528	3.8683
ChamaleonLLM	6,786,596	0.3753	1.4554

ChamaleonLLM achieves significantly lower validation loss and perplexity compared to traditional LoRA and unadapted GPT-2.
The dynamic adaptation mechanism allows the model to generalize better and handle diverse input distributions.

Key Takeaways

Dynamic Adaptation: ChamaleonLLM enables LLMs to adapt dynamically during inference, improving their performance on diverse and novel data.
Batch-Aware Clustering: By grouping similar inputs, the model can capture shared context and reduce noise.
Efficiency: The hyper-network generates low-rank updates on-the-fly, eliminating the need for storing multiple expert models or masks.
Versatility: ChamaleonLLM can adapt to a wide range of tasks and data distributions without requiring predefined task embeddings.

Why This Matters

ChamaleonLLM represents a significant step toward making LLMs more flexible and efficient in real-world applications. By enabling dynamic adaptation during inference, this framework can improve the performance of LLMs in scenarios where input data is highly variable or noisy. It also reduces the computational and memory overhead associated with traditional fine-tuning methods.

Open Source and Reproducibility

The authors have open-sourced the code for ChamaleonLLM, ensuring that the research community can reproduce and build upon their work. You can find the code and additional details in the paper.

Final Thoughts

ChamaleonLLM is a promising framework that addresses a critical limitation of current LLMs: their inability to adapt dynamically during inference. By leveraging batch-aware clustering and dynamic low-rank updates, this approach opens up new possibilities for improving the flexibility and efficiency of language models. I’m excited to see how this research evolves and how it will be applied in real-world NLP applications!

If you’re interested in learning more, I highly recommend reading the full paper. Let me know your thoughts in the comments below! 🚀

References

Paper: ChamaleonLLM: Batch-Aware Dynamic Low-Rank Adaptation via Inference-Time Clusters
Authors: Kamer Ali Yuksel & Hassan Sawaf (aiXplain Inc.)
Code: Open-sourced for reproducibility.

🔥 𝗧𝗵𝗲 𝟭𝟬𝘅 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 𝗶𝘀 𝗮 𝗠𝘆𝘁𝗵 (𝗔𝗻𝗱 𝗧𝗵𝗮𝘁’𝘀 𝗮 𝗚𝗼𝗼𝗱 𝗧𝗵𝗶𝗻𝗴!)

QURBAN AHMAD — Wed, 29 Jan 2025 14:04:40 +0000

𝑊𝑒’𝑣𝑒 𝑎𝑙𝑙 ℎ𝑒𝑎𝑟𝑑 𝑜𝑓 𝑡ℎ𝑒 𝗹𝗲𝗴𝗲𝗻𝗱𝗮𝗿𝘆 “𝟭𝟬𝘅 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿”—𝑡ℎ𝑎𝑡 𝑜𝑛𝑒 𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑟 𝑤ℎ𝑜 𝑠𝑢𝑝𝑝𝑜𝑠𝑒𝑑𝑙𝑦 𝑤𝑟𝑖𝑡𝑒𝑠 𝑚𝑜𝑟𝑒 𝑐𝑜𝑑𝑒 𝑡ℎ𝑎𝑛 10 𝑒𝑛𝑔𝑖𝑛𝑒𝑒𝑟𝑠 𝑐𝑜𝑚𝑏𝑖𝑛𝑒𝑑, 𝑛𝑒𝑣𝑒𝑟 𝑛𝑒𝑒𝑑𝑠 ℎ𝑒𝑙𝑝, 𝑎𝑛𝑑 𝑠𝑖𝑛𝑔𝑙𝑒-ℎ𝑎𝑛𝑑𝑒𝑑𝑙𝑦 𝑏𝑢𝑖𝑙𝑑𝑠 𝑏𝑖𝑙𝑙𝑖𝑜𝑛-𝑑𝑜𝑙𝑙𝑎𝑟 𝑠𝑦𝑠𝑡𝑒𝑚𝑠 𝑤ℎ𝑖𝑙𝑒 𝑑𝑟𝑖𝑛𝑘𝑖𝑛𝑔 𝑛𝑜𝑡ℎ𝑖𝑛𝑔 𝑏𝑢𝑡 𝑐𝑜𝑓𝑓𝑒𝑒 𝑎𝑛𝑑 𝑤𝑟𝑖𝑡𝑖𝑛𝑔 𝑖𝑛 𝑉𝑖𝑚.

𝑆𝑜𝑢𝑛𝑑𝑠 𝑖𝑚𝑝𝑟𝑒𝑠𝑠𝑖𝑣𝑒, 𝑟𝑖𝑔ℎ𝑡? 𝐵𝑢𝑡 ℎ𝑒𝑟𝑒’𝑠 𝑡ℎ𝑒 ℎ𝑎𝑟𝑑 𝑡𝑟𝑢𝑡ℎ: 𝗧𝗵𝗲 𝟭𝟬𝘅 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 𝗶𝘀 𝗮 𝗺𝘆𝘁𝗵. 𝐴𝑛𝑑 𝑤𝑜𝑟𝑠𝑒—𝑖𝑡’𝑠 𝑎 𝑑𝑎𝑛𝑔𝑒𝑟𝑜𝑢𝑠 𝑜𝑛𝑒.

𝗪𝗵𝘆 𝗖𝗵𝗮𝘀𝗶𝗻𝗴 𝘁𝗵𝗲 “𝟭𝟬𝘅 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿” 𝗶𝘀 𝗮 𝗠𝗶𝘀𝘁𝗮𝗸𝗲

🚨 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗶𝘀 𝗮 𝘁𝗲𝗮𝗺 𝘀𝗽𝗼𝗿𝘁 – 𝑁𝑜 𝑚𝑎𝑡𝑡𝑒𝑟 ℎ𝑜𝑤 𝑠𝑘𝑖𝑙𝑙𝑒𝑑 𝑎𝑛 𝑒𝑛𝑔𝑖𝑛𝑒𝑒𝑟 𝑖𝑠, 𝑠𝑜𝑓𝑡𝑤𝑎𝑟𝑒 𝑑𝑒𝑣𝑒𝑙𝑜𝑝𝑚𝑒𝑛𝑡 𝑖𝑠 𝑟𝑎𝑟𝑒𝑙𝑦 𝑎 𝑠𝑜𝑙𝑜 𝑚𝑖𝑠𝑠𝑖𝑜𝑛. 𝐴 𝑔𝑟𝑒𝑎𝑡 𝑒𝑛𝑔𝑖𝑛𝑒𝑒𝑟 𝑖𝑠𝑛’𝑡 𝑡ℎ𝑒 𝑜𝑛𝑒 𝑤ℎ𝑜 𝑤𝑟𝑖𝑡𝑒𝑠 𝑡ℎ𝑒 𝑚𝑜𝑠𝑡 𝑐𝑜𝑑𝑒; 𝑖𝑡’𝑠 𝑡ℎ𝑒 𝑜𝑛𝑒 𝑤ℎ𝑜 𝑚𝑎𝑘𝑒𝑠 𝑡ℎ𝑒 𝘄𝗵𝗼𝗹𝗲 𝘁𝗲𝗮𝗺 𝗺𝗼𝗿𝗲 𝗲𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲.

🚨 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗶𝘀𝗻’𝘁 𝗮𝗯𝗼𝘂𝘁 𝘀𝗽𝗲𝗲𝗱 – 𝐴 𝑠𝑜-𝑐𝑎𝑙𝑙𝑒𝑑 10𝑥 𝑒𝑛𝑔𝑖𝑛𝑒𝑒𝑟 𝑤ℎ𝑜 𝑟𝑒𝑓𝑢𝑠𝑒𝑠 𝑡𝑜 𝑑𝑜𝑐𝑢𝑚𝑒𝑛𝑡, 𝑐𝑜𝑙𝑙𝑎𝑏𝑜𝑟𝑎𝑡𝑒, 𝑜𝑟 𝑚𝑒𝑛𝑡𝑜𝑟 𝑜𝑡ℎ𝑒𝑟𝑠 𝑖𝑠𝑛’𝑡 𝑎𝑛 𝑎𝑠𝑠𝑒𝑡—𝑡ℎ𝑒𝑦’𝑟𝑒 𝑎 𝑏𝑜𝑡𝑡𝑙𝑒𝑛𝑒𝑐𝑘. 𝑊ℎ𝑒𝑛 𝑡ℎ𝑒𝑦 𝑙𝑒𝑎𝑣𝑒, 𝑡ℎ𝑒𝑦 𝑡𝑎𝑘𝑒 𝑡ℎ𝑒 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒 𝑤𝑖𝑡ℎ 𝑡ℎ𝑒𝑚, 𝑙𝑒𝑎𝑣𝑖𝑛𝑔 𝑏𝑒ℎ𝑖𝑛𝑑 𝑎 𝑚𝑒𝑠𝑠 𝑡ℎ𝑎𝑡 𝑜𝑡ℎ𝑒𝑟𝑠 𝑠𝑡𝑟𝑢𝑔𝑔𝑙𝑒 𝑡𝑜 𝑚𝑎𝑖𝑛𝑡𝑎𝑖𝑛.

🚨 𝗖𝗼𝗱𝗲 𝗶𝘀 𝗰𝗵𝗲𝗮𝗽, 𝗶𝗺𝗽𝗮𝗰𝘁 𝗶𝘀 𝗲𝘃𝗲𝗿𝘆𝘁𝗵𝗶𝗻𝗴 – 𝑊𝑟𝑖𝑡𝑖𝑛𝑔 𝑚𝑜𝑟𝑒 𝑐𝑜𝑑𝑒 𝑑𝑜𝑒𝑠𝑛’𝑡 𝑎𝑙𝑤𝑎𝑦𝑠 𝑚𝑒𝑎𝑛 𝑑𝑒𝑙𝑖𝑣𝑒𝑟𝑖𝑛𝑔 𝑚𝑜𝑟𝑒 𝑣𝑎𝑙𝑢𝑒. 𝑆𝑜𝑚𝑒𝑡𝑖𝑚𝑒𝑠, 𝑡ℎ𝑒 𝑏𝑒𝑠𝑡 𝑒𝑛𝑔𝑖𝑛𝑒𝑒𝑟𝑠 𝑤𝑟𝑖𝑡𝑒 𝑙𝑒𝑠𝑠—𝑡ℎ𝑒𝑦 𝑚𝑎𝑘𝑒 𝑏𝑒𝑡𝑡𝑒𝑟 𝑎𝑟𝑐ℎ𝑖𝑡𝑒𝑐𝑡𝑢𝑟𝑎𝑙 𝑐ℎ𝑜𝑖𝑐𝑒𝑠, 𝑒𝑙𝑖𝑚𝑖𝑛𝑎𝑡𝑒 𝑢𝑛𝑛𝑒𝑐𝑒𝑠𝑠𝑎𝑟𝑦 𝑐𝑜𝑚𝑝𝑙𝑒𝑥𝑖𝑡𝑦, 𝑎𝑛𝑑 ℎ𝑒𝑙𝑝 𝑡ℎ𝑒 𝑏𝑢𝑠𝑖𝑛𝑒𝑠𝑠 𝑚𝑜𝑣𝑒 𝑓𝑜𝑟𝑤𝑎𝑟𝑑 𝑒𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑡𝑙𝑦.

🚨 𝗧𝗵𝗲 𝗶𝗹𝗹𝘂𝘀𝗶𝗼𝗻 𝗼𝗳 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝘃𝗶𝘁𝘆 – 𝑀𝑎𝑛𝑦 𝑠𝑒𝑙𝑓-𝑝𝑟𝑜𝑐𝑙𝑎𝑖𝑚𝑒𝑑 “10𝑥 𝑒𝑛𝑔𝑖𝑛𝑒𝑒𝑟𝑠” 𝑜𝑝𝑡𝑖𝑚𝑖𝑧𝑒 𝑓𝑜𝑟 𝑠ℎ𝑜𝑟𝑡-𝑡𝑒𝑟𝑚 𝑠𝑝𝑒𝑒𝑑 𝑎𝑡 𝑡ℎ𝑒 𝑐𝑜𝑠𝑡 𝑜𝑓 𝑙𝑜𝑛𝑔-𝑡𝑒𝑟𝑚 𝑚𝑎𝑖𝑛𝑡𝑎𝑖𝑛𝑎𝑏𝑖𝑙𝑖𝑡𝑦. 𝑇ℎ𝑒𝑦 𝑚𝑖𝑔ℎ𝑡 𝑑𝑒𝑙𝑖𝑣𝑒𝑟 𝑓𝑒𝑎𝑡𝑢𝑟𝑒𝑠 𝑞𝑢𝑖𝑐𝑘𝑙𝑦, 𝑏𝑢𝑡 𝑖𝑓 𝑡ℎ𝑜𝑠𝑒 𝑓𝑒𝑎𝑡𝑢𝑟𝑒𝑠 𝑎𝑟𝑒 𝑓𝑢𝑙𝑙 𝑜𝑓 𝑡𝑒𝑐ℎ 𝑑𝑒𝑏𝑡, 𝑡ℎ𝑒𝑦 𝑐𝑟𝑒𝑎𝑡𝑒 𝑚𝑜𝑟𝑒 𝑤𝑜𝑟𝑘 𝑓𝑜𝑟 𝑒𝑣𝑒𝑟𝑦𝑜𝑛𝑒 𝑒𝑙𝑠𝑒.

𝑆𝑜 𝑊ℎ𝑜 𝐴𝑟𝑒 𝑡ℎ𝑒 𝑅𝑒𝑎𝑙 10𝑥 𝐸𝑛𝑔𝑖𝑛𝑒𝑒𝑟𝑠?

𝑇ℎ𝑒 𝑏𝑒𝑠𝑡 𝑒𝑛𝑔𝑖𝑛𝑒𝑒𝑟𝑠 𝑎𝑟𝑒𝑛’𝑡 𝑗𝑢𝑠𝑡 𝑓𝑎𝑠𝑡 𝑐𝑜𝑑𝑒𝑟𝑠—𝑡ℎ𝑒𝑦’𝑟𝑒 10𝑥 𝑝𝑟𝑜𝑏𝑙𝑒𝑚 𝑠𝑜𝑙𝑣𝑒𝑟𝑠. 𝑇ℎ𝑒𝑦:

✅ 𝐸𝑛𝑎𝑏𝑙𝑒 𝑜𝑡ℎ𝑒𝑟𝑠 – 𝑇ℎ𝑒𝑦 𝑚𝑒𝑛𝑡𝑜𝑟, 𝑠ℎ𝑎𝑟𝑒 𝑘𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒, 𝑎𝑛𝑑 ℎ𝑒𝑙𝑝 𝑡ℎ𝑒 𝑡𝑒𝑎𝑚 𝑔𝑟𝑜𝑤.

✅ 𝑇ℎ𝑖𝑛𝑘 𝑙𝑜𝑛𝑔-𝑡𝑒𝑟𝑚 – 𝑇ℎ𝑒𝑦 𝑓𝑜𝑐𝑢𝑠 𝑜𝑛 𝑤𝑟𝑖𝑡𝑖𝑛𝑔 𝑚𝑎𝑖𝑛𝑡𝑎𝑖𝑛𝑎𝑏𝑙𝑒, 𝑠𝑐𝑎𝑙𝑎𝑏𝑙𝑒 𝑠𝑦𝑠𝑡𝑒𝑚𝑠 𝑖𝑛𝑠𝑡𝑒𝑎𝑑 𝑜𝑓 𝑟𝑢𝑠ℎ𝑖𝑛𝑔 𝑐𝑜𝑑𝑒 𝑜𝑢𝑡 𝑡ℎ𝑒 𝑑𝑜𝑜𝑟.

✅ 𝐾𝑛𝑜𝑤 𝑤ℎ𝑒𝑛 𝑡𝑜 𝑠𝑎𝑦 “𝑛𝑜” – 𝑇ℎ𝑒𝑦 𝑑𝑜𝑛’𝑡 𝑏𝑢𝑖𝑙𝑑 𝑓𝑒𝑎𝑡𝑢𝑟𝑒𝑠 𝑓𝑜𝑟 𝑡ℎ𝑒 𝑠𝑎𝑘𝑒 𝑜𝑓 𝑖𝑡; 𝑡ℎ𝑒𝑦 𝑎𝑠𝑘, “𝐼𝑠 𝑡ℎ𝑖𝑠 𝑟𝑒𝑎𝑙𝑙𝑦 𝑛𝑒𝑒𝑑𝑒𝑑?”

✅ 𝐶𝑜𝑚𝑚𝑢𝑛𝑖𝑐𝑎𝑡𝑒 𝑒𝑓𝑓𝑒𝑐𝑡𝑖𝑣𝑒𝑙𝑦 – 𝑇ℎ𝑒𝑦 𝑏𝑟𝑒𝑎𝑘 𝑑𝑜𝑤𝑛 𝑐𝑜𝑚𝑝𝑙𝑒𝑥 𝑝𝑟𝑜𝑏𝑙𝑒𝑚𝑠, 𝑚𝑎𝑘𝑖𝑛𝑔 𝑖𝑡 𝑒𝑎𝑠𝑖𝑒𝑟 𝑓𝑜𝑟 𝑜𝑡ℎ𝑒𝑟𝑠 𝑡𝑜 𝑐𝑜𝑛𝑡𝑟𝑖𝑏𝑢𝑡𝑒.

𝐹𝑖𝑛𝑎𝑙 𝑇ℎ𝑜𝑢𝑔ℎ𝑡

𝑇ℎ𝑒 𝑡𝑒𝑐ℎ 𝑖𝑛𝑑𝑢𝑠𝑡𝑟𝑦 𝑛𝑒𝑒𝑑𝑠 𝑓𝑒𝑤𝑒𝑟 “10𝑥 𝑐𝑜𝑑𝑒𝑟𝑠” 𝑎𝑛𝑑 𝑚𝑜𝑟𝑒 “10𝑥 𝑡𝑒𝑎𝑚𝑚𝑎𝑡𝑒𝑠.” 𝐵𝑒𝑐𝑎𝑢𝑠𝑒 𝑎𝑡 𝑡ℎ𝑒 𝑒𝑛𝑑 𝑜𝑓 𝑡ℎ𝑒 𝑑𝑎𝑦, 𝑔𝑟𝑒𝑎𝑡 𝑠𝑜𝑓𝑡𝑤𝑎𝑟𝑒 𝑖𝑠𝑛’𝑡 𝑏𝑢𝑖𝑙𝑡 𝑏𝑦 ℎ𝑒𝑟𝑜𝑒𝑠—𝑖𝑡’𝑠 𝑏𝑢𝑖𝑙𝑡 𝑏𝑦 𝑡𝑒𝑎𝑚𝑠.

𝑊ℎ𝑎𝑡’𝑠 𝑦𝑜𝑢𝑟 𝑒𝑥𝑝𝑒𝑟𝑖𝑒𝑛𝑐𝑒? 𝐻𝑎𝑣𝑒 𝑦𝑜𝑢 𝑤𝑜𝑟𝑘𝑒𝑑 𝑤𝑖𝑡ℎ 𝑠𝑜𝑚𝑒𝑜𝑛𝑒 𝑤ℎ𝑜 𝑠𝑒𝑒𝑚𝑒𝑑 𝑙𝑖𝑘𝑒 𝑎 “10𝑥 𝑒𝑛𝑔𝑖𝑛𝑒𝑒𝑟” 𝑏𝑢𝑡 𝑎𝑐𝑡𝑢𝑎𝑙𝑙𝑦 𝑠𝑙𝑜𝑤𝑒𝑑 𝑡ℎ𝑒 𝑡𝑒𝑎𝑚 𝑑𝑜𝑤𝑛? 𝑂𝑟 𝑚𝑎𝑦𝑏𝑒 𝑠𝑜𝑚𝑒𝑜𝑛𝑒 𝑤ℎ𝑜 𝑚𝑎𝑑𝑒 𝑒𝑣𝑒𝑟𝑦𝑜𝑛𝑒 𝑎𝑟𝑜𝑢𝑛𝑑 𝑡ℎ𝑒𝑚 𝑏𝑒𝑡𝑡𝑒𝑟? 𝐿𝑒𝑡’𝑠 𝑑𝑖𝑠𝑐𝑢𝑠𝑠! 👇

Node.js Event Loop: A Comprehensive Guide

QURBAN AHMAD — Sat, 18 Nov 2023 17:11:17 +0000

Node.js's asynchronous nature and event-driven architecture often lead to questions about its underlying mechanism, particularly the event loop. Understanding how Node.js manages tasks and handles asynchronous operations empowers developers to write efficient and responsive applications.

Understanding the Event Loop

The event loop in Node.js enables it to do many things at once, even though JavaScript runs on a single thread. It does this by handing off tasks to the computer's core operating system, which can handle multiple tasks simultaneously. When one of these tasks finishes, Node.js gets a signal from the system, allowing it to run the associated callback with that task.

When Node.js starts, it initializes the event loop, processes the provided input script which may make async API calls, schedule timers, or call process.nextTick(), then begins processing the event loop.

The event loop is the heart of Node.js, managing the execution of asynchronous operations in a single-threaded environment. It operates by continuously checking a series of phases to handle different tasks efficiently.

The Event Loop Phases:

Timers: Handles callbacks scheduled via setTimeout() or setInterval().
Pending Callbacks: Manages specific system operation callbacks deferred to the next loop iteration.
Idle & Prepare: Only used internally.
Poll: Retrieve new I/O events; execute I/O related callbacks (almost all with the exception of close callbacks, the ones scheduled by timers, and setImmediate()); node will block here when appropriate.
Check: Executes setImmediate() callbacks after the poll phase ends.
Close Callbacks: Deals with callbacks related to closed handles or sockets, like the 'close' event.

Between each loop iteration, Node.js checks for pending asynchronous I/O or timers to ensure a clean shutdown if none exist.

1. Timers Phase:

With the Timers phase, we provide the threshold time after which the associated callback runs.

For ex:

const fs = require('fs');

function asyncFunction(callback) {
  // Let's assume this 'fs.readFile' takes 90ms to read the whole 'abc.txt' file here.
  fs.readFile("abc.txt", callback);
}

const startTime = Date.now();

setTimeout(() => {
  const setTimeoutExecutionDuration = Date.now() - startTime;
  console.log(`${setTimeoutExecutionDuration}ms has passed since I was scheduled.`);
}, 100);

asyncFunction(() => {
 // Let's assume it takes 20ms
})

Now as we can see above, the timer that we have specified for the 'setTimeout' function is 100ms, and assuming the 'fs.readFile' called inside 'asyncFunction' takes 90ms and its callback takes 20ms to run.

Even though the 'setTimeout' callback has been scheduled to be executed after 100ms it will run after 110ms.

Explanation: In the above, scenario event loop will first enter the Timers phase will execute the setTimeout function, and will go into the Poll phase. In the pole phase, it will wait for other callbacks to be queued upon, as soon as fs.readFile completes its execution after 90ms, the event loop will put its callback into the poll queue and execute which takes 20ms. Now, since there are no other callbacks left, it will go into the Timers queue and execute the setTimeout callback. As we can see, the setTimeout has taken total of 110ms to run instead of its actual duration which was 100ms.

2. Pending Phase:

Handles specific system operation callbacks, like TCP errors.

3. Poll Phase:

In the event loop, the poll phase handles I/O operations and queued events.

Tasks in Poll Phase:

If no timers are set, and the poll queue isn't empty, the event loop processes queued callbacks one by one until it finishes all or reaches a system-defined limit.
If the poll queue is empty:
- If setImmediate() scripts are scheduled, the event loop moves to the check phase to run those scheduled scripts.
- If no setImmediate() scripts are scheduled, the event loop waits for new callbacks to be added to the queue and executes them immediately.

Transition after Poll Phase:

Once the poll queue empties, the event loop checks for timers that have reached their set thresholds.

If any timers are ready, the event loop shifts back to the timers phase to execute their associated callbacks.

4. Check Phase:

The check phase in the event loop runs callbacks right after the poll phase finishes its tasks. If the poll phase is inactive and there are pending scripts scheduled with setImmediate(), the event loop moves directly to the check phase instead of waiting.

Typically, as code executes, the event loop reaches the poll phase, awaiting incoming connections or requests. However, if a setImmediate() callback is in line and the poll phase pauses, the loop swiftly transitions to the check phase without waiting for new poll events.

5. Close Phase:

If a socket or handle is closed abruptly (e.g. socket.destroy()), the 'close' event will be emitted in this phase.

Conclusion

Understanding the Node.js event loop is pivotal for writing scalable and performant applications. Leveraging its phases effectively can enhance code reliability and efficiency.

By diving into practical examples and insights, we demystified the event loop, shedding light on its functionality and offering best practices for optimal code execution in Node.js.

If you want to dive deeper into the intricacies of Node.js event loop, timers, and process.nextTick(), you can explore the official Node.js documentation on Event Loop, Timers, and process.nextTick().

Photo by Pixabay: https://www.pexels.com/photo/ferris-wheel-in-city-315499/

Revolutionizing React with Server Components: A Game Changer for Performance

QURBAN AHMAD — Sat, 04 Nov 2023 07:57:09 +0000

In the ever-evolving world of web development, React Server Components is a groundbreaking addition that promises to reshape how we build and optimize web applications. React Server Components tackle the limitations of client-side rendering, creating an enhanced user experience, particularly in scenarios where content frequently changes.

The Problem with Client-Side Rendering

The Blank Screen Blues:

Traditional React client-side rendering can be a frustrating experience for users. When an app loads, users are often greeted with a blank screen, leaving them staring into the void of nothingness.

The Savior: Server-Side Rendering (SSR)

The Power of SSR:

Server-side rendering (SSR) comes to the rescue. With SSR, React generates the initial HTML of your app and sends it to the client on request. This means that even before React starts its magic, users can see content on the screen.

The Data Fetching Dilemma

Data Fetching Dilemmas:

Data fetching in React apps can be a bit of a waiting game:

In client-side rendering (CSR), data fetching typically occurs after the app has loaded. Users are left staring at loading screens until the data arrives.
In server-side rendering (SSR), data can be fetched on the server, improving loading times. However, traditional React lacks official support for data fetching within components during SSR.

Meet the Game-Changer: React Server Components

Introducing React Server Components:

Enter the hero of our story: React Server Components. These components run directly on the server, keeping your JavaScript bundle lean and mean. But:

Unlike regular React components, React Server Components don't re-render on the client side. This means you can't use features like useState and useEffect, but it also means they're not hydrated on the client.
However, to unleash their power, React Server Components need to play nice with external elements like the bundler, the server, and the router. And currently, the only way Currently is with Next.js 14.0+ and their re-architected "App Router."

Next.js Integration:

If you're a Next.js enthusiast, you're in for a treat. In Next.js, every component is, by default, a React Server Component. To add interactivity to a server component, you can use the "use client" directive provided by React.

The Advantages

Why You Should Be Excited:

React Server Components bring a host of advantages to the table:

Performance Boost: With server components, you reduce the amount of JavaScript that needs to be downloaded and the number of components that need to be hydrated. It's a performance win!
Optimized Bundles: Say goodbye to bundling large library files if they're only used within server components. The final HTML generated on the server is sent to the client, resulting in significant reductions in your JS bundle size.
Simplified Data Fetching: Forget the hassle of async data fetching using hooks like useEffect. You can now integrate data-fetching code directly into your React Server Components.

import db from 'abc-db';

async function Page() {
  const link = db.connect('localhost', 'root', 'password');
  const data = await db.query(link, 'SELECT * FROM customers');
  return (
    <>
      <h1>Customers</h1>
      {data.map((user) => (
        <article key={user.id}>
          <h2>{user.name}</h2>
          <p>{user.age}</p>
        </article>
      ))}
    </>
  );
}
export default Page;

React Server Components marks an exciting milestone in React's evolution, enabling us to run server-exclusive code within our components. It's a game changer for performance, bundle size, and the overall developer experience. So, whether you're building the next big thing or optimizing your current app, React Server Components are here to revolutionize your web development journey. Embrace the change, and let your apps shine! 🚀🌟

Photo by Christopher Farrugia: https://www.pexels.com/photo/street-cafe-employees-working-at-counter-at-night-3755849/

Understanding Source Maps: Simplifying Debugging

QURBAN AHMAD — Wed, 12 Jul 2023 18:05:38 +0000

Photo by cottonbro studio

Introduction

Today, we are talking about source maps, a crucial tool in modern web development that makes debugging significantly easier.

What is a Source Map?

Definition of the Source Map from Firefox official docs:

A source map is a file that maps from the transformed source to the original source, enabling the browser to reconstruct the original source and present the reconstructed original in the debugger.

Debugging with Source Maps

Let's dive into two common debugging scenarios and see how source maps can simplify the process.

Debugging Transpiled code

In the past, websites/web applications used to be created with simple HTML, CSS, and Javascript. However, in the present day, we are usually building these web applications with various development tools such as:

JavaScript frameworks: Angular, React, Vue, Svelte, etc.
High-level programming languages: TypeScript, Flow, etc.
CSS preprocessors: SCSS, LESS, PostCSS, etc.

These tools then transpile our code into standard HTML, JavaScript, and CSS that browsers can understand.

Without source maps, debugging the transpiled code can be cumbersome. However, with source maps, the debugging experience becomes seamless.

Debugging minified code:

Minifying is a common practice for optimizing production code. (for example, using Terser to minify and mangle JavaScript).

However, minification often obfuscates the code, making it challenging to debug. Source maps come to the rescue again by mapping the minified code back to its original, unminified form.

Generating source maps

Source maps are files with names ending with .map (for example, example.min.js.map and styles.css.map). They can be generated by most build tools, for example, Typescript, Webpack, Rollup, etc.

Some tools include source maps by default, while others may need additional configuration to produce them.

Demo

Here, we will be creating a web page with Typescript code transpiled into Javascript to log "Hello World" in the browser console when a button is clicked. To generate an error, button has not been included in the HTML file.

This is the Typesript configuration file (tsconfig.json) of the project and as we can see we have enabled the source map generation option in line 6.

The complete code of this project can be found here

qur786 / Introduction-to-source-map

Introduction to Source Map

View on GitHub

When we transpile Typescript file index.ts into Javascript, besides index.js file it also creates one more file called index.js.map. This is a source map that a browser can use to recreate index.ts file from index.js.

When we render the webpage in the browser using a server (Live Server VSCode extension has been used here to server the index.html file in localhost), we see an error in the browser console, it contains the stack trace of the error. As we can see it is pointing at index.ts even though the browser in no way has access to our index.ts file or using it in any way.

If we check the browser Sources tab, we find we have both our Typescript and Javascript files.

Thus, from here we can also debug the code using our source Typescript code via the browser debugger. This may not seem to have a huge impact on debugging since the file transpiled here is very short. But eventually, for a larger and more complex project, it can ease a developer's job of debugging.

Conclusion:

Generating a source map for your transpiled/minified code allows you to simplify debugging and trace errors back to the original source. Incorporate source maps into your development workflow to enhance your debugging experience and streamline the error-tracing process.

Happy coding!