DEV Community: Pannaga Perumal

When the Kiosk Remembered Wrong

Pannaga Perumal — Sat, 22 Aug 2026 13:42:28 +0000

The face that forgot how to count

Mosaic is the UI running on a robot's face and kiosk display — GLTF facial animation, tool results, conversational state, all streamed over NATS and rendered live. Most of the time it's seamless. But every so often, after a dropped Wi-Fi connection reconnected, something would look... off. A response would appear twice. A tool result that had already been shown would echo back into the UI a second time, like the robot was repeating itself for no reason.

Nothing was wrong in the steady state. The bug only ever showed up right after the network had a hiccup and recovered — which meant it was almost invisible in normal testing, and only ever bit in exactly the moment reliability mattered most: right when something had already gone a little wrong and the system was trying to recover.

Event-sourced UI, and a guarantee I hadn't questioned

Mosaic's state isn't just rendered directly — it's built from a stream of events coming over NATS: a tool call result here, a UI fragment there, a state update from the conversation pipeline. The store applies each event as it arrives and that's what ends up on screen.

That model works great — until you actually think about what "arrives" means over a message bus. NATS gives you at-least-once delivery. That's a deliberate, sensible guarantee: after a dropped connection, NATS will redeliver messages you might have missed, rather than silently lose them. Good default. But it means the same event can legitimately arrive twice.

My store didn't know that. It applied every incoming event as if "arrived" meant "new." Which is true almost all the time — until a reconnect happens, NATS does exactly what it's designed to do and redelivers a message you'd already processed, and the store dutifully applies it again.

Finding the gap between two different guarantees

This is the kind of bug that's obvious once you say it out loud and completely invisible while you're staring at the UI code, because the UI code isn't wrong — Svelte's rendering the state correctly. The state itself is wrong.

The real insight was separating two things that had been quietly conflated:

A stream of updates — what NATS gives you: a sequence of events, possibly including redeliveries.
The current true state — what the UI actually needs to render correctly.

I'd been treating the first as if it were automatically the second. It isn't. A replay stream tells you things happened; it doesn't promise you're hearing about each one exactly once, and it doesn't promise the order you get them in after a reconnect matches what you'd expect if nothing had ever dropped.

The fix: make "again" a no-op, and don't trust the stream blindly

Two changes, and they mattered in different ways:

1. Made event application idempotent. Every event now has an identity, and the store dedupes on that identity before it's allowed to mutate anything. A redelivered event, or an echoed tool result, becomes a no-op instead of a double-apply.

before:  event arrives -> apply to store   (always)
after:   event arrives -> seen before? -> skip
                        -> not seen?    -> apply + remember id

2. Added an authoritative resync path. After a reconnect specifically, the kiosk doesn't just trust whatever the replay stream sends first — it pulls the true current state directly. The event stream is great for staying in sync once you're already there; it's the wrong tool for re-establishing sync after you've been offline. Those are different problems, and I'd been using one mechanism for both.

The result

State stays correct through drops and redelivery, full stop. The kiosk can lose its NATS connection, reconnect, and the display doesn't quietly lie about what state it's in.

That matters more for a robot face than it might for a typical dashboard — a duplicated toast notification is mildly annoying; a robot's face flickering into a stale or doubled expression right in front of someone reads as broken, immediately, to anyone watching. UI bugs on embodied systems don't get to hide in a support ticket. They happen live, in front of a person.

Takeaways

At-least-once delivery pushes idempotency onto you, the consumer. If your message bus can redeliver — and most production-grade ones can — "I received this event" and "this event is new" are two different claims. Conflate them and you will eventually double-apply something.
A stream of updates isn't the same as a source of current truth. Replaying an event log is great for staying in sync; it's the wrong primitive for re-establishing sync after a gap. Sometimes the right move after a reconnect is to ask "what's actually true right now?" instead of "what did I miss?"
Test the unhappy path on purpose. The steady-state path will always work in a demo. The reconnect/redelivery path is the one that's actually worth writing a deliberate test for, because it's the one nobody exercises by accident.
Bugs on visible, embodied systems have zero grace period. A backend duplicate-processing bug might sit quietly in a log for a week. The same bug on a robot's face is a bug someone's watching happen. Prioritize accordingly.

Over to you

If you're building anything event-sourced on top of a message bus with at-least-once semantics — NATS, Kafka, SQS, whatever — I'd bet you have a version of this bug somewhere, even if it hasn't bitten you yet. Have you hit redelivery duplication before, and how did you handle dedup — event IDs, content hashing, something else?

I write about the systems-level bugs that show up when real-time UI meets distributed messaging — if that's your kind of problem too, follow along, and let me know what you're building.

The Mic That Argued With Itself

Pannaga Perumal — Sat, 22 Aug 2026 13:41:09 +0000

The robot that couldn't hold a conversation

Picture this: a social robot standing in a lobby, listening for its wake word, chatting with whoever walks up. It's supposed to feel alive — responsive, natural, always listening. Instead, every so often, mid-sentence, it would just... die. SIGSEGV. The process gone, the conversation cut off like someone pulled the plug.

And the worst part? I couldn't reproduce it on demand. It happened more when the robot was busier — more people walking up, more wake words triggering, more conversations overlapping. Classic intermittent bug energy: the kind that makes you doubt your own sanity a little.

Sometimes it wasn't even a crash. The robot would just... not hear you. You'd say the wake word, nothing would happen, and you'd be left wondering if it was ignoring you or if something deeper was broken. Turns out — both, in a way.

The system: two ears, one microphone

Voxa, the robot's speech-to-speech pipeline, has two jobs running more or less independently:

Wake-word detection — always listening in the background for its name.
The conversation loop — once triggered, opens a live capture session to actually record and process what you're saying.

Both of these needed audio. Both of these, it turned out, were opening their own independent PortAudio capture stream to get it — two separate handles onto the same physical microphone.

Most of the time this was fine. Wake-word stream closes, conversation stream opens, no overlap, no problem. The failure only showed up in the gap between those two states — when a new utterance kicked off before the previous stream had actually finished tearing down.

Chasing a bug that only bites under load

The first instinct with a crash like this is to look at the crash site — the stack trace at the moment of SIGSEGV. But that only tells you where the memory got corrupted, not why two completely different code paths were touching the same resource at the same time.

The real signal was in the pattern, not the stack trace: this only got worse under load. That's usually not a logic bug — your code isn't suddenly wrong when there are more users. That's a race condition: code that's individually correct but breaks when two copies of it run concurrently and neither one knows the other exists.

Once I started looking at it through that lens, the pattern became obvious: wake-word detection and the conversation loop each owned a PortAudio stream, and neither one knew the other existed. There was no coordination — no lock, no shared state, nothing telling one "wait, the other one still has the mic." When a new utterance started fast enough, both streams ended up holding the device simultaneously. Sometimes that just corrupted the audio buffer silently. Sometimes it corrupted memory badly enough to take the whole process down with it.

The fix: give the microphone exactly one owner

The fix wasn't clever — it was structural. Instead of two components each managing their own capture stream, I unified audio capture onto a single PortAudio stream, with one explicit owner responsible for its entire lifecycle: opening it, closing it, and deciding who gets to consume from it.

Wake-word detection and the conversation loop still both need audio — but now they read from the one stream that owns the device, instead of each independently owning a competing one. There's no window anymore where two teardown paths can race each other, because there's only one teardown path.

before:
  wake-word  -> opens PortAudio stream A -> mic
  convo loop -> opens PortAudio stream B -> mic
  (both can hold the device at once — race)

after:
  single owner -> opens PortAudio stream -> mic
  wake-word  -> reads from owned stream
  convo loop -> reads from owned stream
  (exactly one lifecycle, no race)

The result — and the feature it unlocked

The crash class disappeared completely. No more SIGSEGVs, no more leaked stream handles, no more mysterious "why didn't it hear me" reports.

But the more interesting payoff was what came after the fix. With a stable, contention-free capture path in place, I could finally tune barge-in detection — letting someone interrupt the robot mid-sentence, the way you'd naturally interrupt a person. That feature had been effectively unbuildable before, because you can't reliably detect an interruption on a microphone that occasionally forgets it's supposed to be listening.

Fixing the "boring" stability bug is what made the actually delightful feature possible.

Takeaways

A shared hardware resource needs exactly one owner for its lifecycle. The moment two components each independently manage the same device, you have a race condition waiting for the right timing to expose it — not a convenience, a liability.
Bugs that scale with load are usually contention, not logic. If a bug gets worse the busier your system is, stop looking for a wrong if statement and start looking for two things touching the same resource without knowing about each other.
Stability work often pays for itself in features you didn't plan for. Barge-in wasn't the goal when I started debugging crashes — it was a side effect of finally having a capture path solid enough to build on.
Reproducing "under load" bugs on purpose beats waiting for them. If you suspect contention, write the thing that hammers the two code paths back-to-back instead of hoping it happens again in front of you.

Over to you

Race conditions are sneaky precisely because the code looks right in isolation — it's only wrong in relation to something else running at the same time. If you've hunted one of these before, I'd love to hear it: what was the tell that made you stop looking at logic and start looking at timing?

If you're building anything with real-time audio, sensor fusion, or just systems that are supposed to feel alive and responsive, I write about this stuff regularly — follow along, and feel free to reach out if you're wrestling with something similar.

Rust vs. Other Languages: Why Rust Stands Out in Modern Development

Pannaga Perumal — Tue, 18 Feb 2025 11:52:45 +0000

Introduction
In the ever-changing world of programming languages, Rust has quickly gained attention. Why? It solves many headaches developers face, like memory bugs and complex concurrency, all while offering top-notch performance. As someone diving deep into Rust backend development, I’ve experienced firsthand how it compares to languages I’ve used before, like Python and Java. Let’s break down how Rust stacks up against C, C++, Python, and Go.

Memory Safety Without Garbage Collection

Memory bugs can be a nightmare.

C/C++: Blazing fast but managing memory manually is tricky. One mistake can cause major crashes.

Rust: Rust’s ownership system and borrow checker ensure memory safety. You can’t even compile code with memory issues. No garbage collector means no unexpected pauses.

Code Snippet: Safe Memory in Rust

fn main() {
    let x = String::from("Hello");
    let y = x;  // Ownership moves to y
    // println!("{}", x); // This line won’t compile 
                          //because x is no longer valid
}

My Take: Coming from Python, where memory is managed for you, Rust felt strict at first. But soon, I realized how much safer my code became.

Performance and Concurrency

Concurrency is where Rust shines.

Python: Great for scripts but the GIL often slows down multi-threaded tasks.

Go: Lightweight goroutines make concurrency simple.

Rust: Rust ensures no data races at compile time.

Code Snippet: Concurrency in Rust

use std::thread;
fn main() {
    let handle = thread::spawn(|| {
        println!("Hello from a thread!");
    });
    handle.join().unwrap();
}

My Experience: Implementing concurrency in Rust felt safe. I wasn’t constantly worrying about race conditions, unlike in Python.

Developer Experience and Ecosystem

Tools matter as much as the language.

Python: Simple syntax and tons of libraries.

Go: Fast builds and easy syntax.

Rust: Cargo makes dependency management easy, and documentation is top-notch.

Code Snippet: Using Cargo

$ cargo new my_project
$ cd my_project
$ cargo run

My Take: Cargo made setting up projects super easy. The community is welcoming, and docs are clear.

Why Rust for Backend Development?

Rust’s performance and safety are game-changers for backend systems.
My Journey: Learning Rust has been challenging but rewarding. Each project teaches me something new, and I’m excited to build high-performance backends.

Conclusion

Rust isn’t just another language; it’s a new way of thinking. Its safety, speed, and modern tools make it stand out. My Rust journey is just starting, but I’m already seeing how it shapes better code.

What do you think about Rust? Let’s chat below!

What is Dark Web?Hidden World Revealed

Pannaga Perumal — Fri, 21 Apr 2023 17:29:31 +0000

To begin with, the internet is a large network of computers communicating with each other. There are various versions of the internet, which are clear web, dark web and deep web.
You may have heard about the dark web many times, but the misconception of people is that the dark web is illegal and its all about hacking, weapons, drugs and stuff.
In this blog, you understand the dark and deep web in a much detailed manner...

What is Clear Web?

Before we move on, let's see about the clear web. To be simple, the Web pages that can be accessed using any search engine are clear web. The pages that we visit on a day-to-day basis are the clear web. It may be Google, Instagram, Amazon etc. This website contains information that can be accessed by everyone. In contrast, you will have your own identity on the clear internet..

Is dark web really Dark?

To be simple, the dark web is that part of the internet that cannot be searched using a search engine. This includes forums, websites and resources that are hidden from the rest of the internet.

The concept of the dark web started in 2006. It was created to provide users anonymity while browsing the web. Initially, it was used by government organizations and political groups for communication.

The Tor browser gave the dark web access to normal users, thereby providing anonymity.

Coming to the main question. The dark web is actually not illegal. Let's say one wants to read an article on a particular topic that is banned in a certain nation, he/she can find that on the dark web and can read it anonymously. This does not mean it's illegal.

There are few parts of the dark web that are actually dark because the crimes that are done on the dark web can't be traced due to its layers. Let's discuss that latter.
The governments of some countries also have their eyes on the dark web in order to peep into the conversations that are going on in the forums on the dark web.

Why is Dark web users Untraceable?

Let's say you are using a laptop. As soon as you connect to the internet you will be assigned an IP address on the network. Each device on the internet will have a unique IP address. One can be easily tracked using their IP.

But in the case of the dark web or while using Tor(to be specific), your IP will be encrypted and data packets will be, hoped through many routers before reaching the main network that means that your IP will not be visible to anyone, thereby it will be impossible to find the person behind the device. The disadvantage is that, due to latency, the network speed decreases.

Activities on Dark Web

Illegal Market place
It is a place where individuals can carry out illicit activities, including the buying and selling of drugs, weapons, passports and stolen information. Cybercriminals often use these marketplace to sell stolen data, such as credit card information, login credentials, and personal data for fraud etc. Usually all these transactions will be made using cryptocurrencies.

Forums
There are various forums on dark web where one can express their points on any of the topic anonymously. Users are not allowed in such forums unless they are referred by the existing users inside the forum.

Hire a Hacker
It is one of the new services on dark web ,here one can rent a hacker to perform his or her work. This is so easily available
that one can hire them as freelancers. For example if I need to hack a Instagram account I need to just send him the account and he will be back with the data in turn one needs to pay him.

News Pages
This is the place where one gets a unbiased news about each and every information from all over the world the people. The users here publish there opinion freely without any avoidance from the government and other entities.

Red Room
Some places on the dark web are particularly disturbing, such as the operation of so-called "red rooms." These are live-streamed events where individuals pay to watch extreme acts of violence, torture, and sexual abuse. Red rooms are illegal and unethical, and those who participate in them are often involved in criminal activity. The use of the dark web and red rooms in particular is a growing concern for law enforcement agencies around the world, as it can be difficult to track down those who are involved in these activities.

Censor Violation
There are also platforms where movies are streamed for free of cost in order to reduce the distribution on the particular media. This may include books, movies, documentaries and even pictures that are banned in specific country.

Child P*rn*grapy
This is very unfortunate that there are a few forums that stream child p*rn*graphy. It is completely unethical and illegal, It is a growing concern for law enforcement agencies around the world, as it can be difficult to track down those who are involved in these activities.

Conclusion

In conclusion, the dark web is a hidden part of the internet that is often associated with illegal and unethical activities. While there are legitimate uses for the anonymity it provides, such as protecting the privacy of dissidents and journalists in repressive regimes, the majority of activities carried out on the dark web are highly problematic.

As such, it is important for individuals and law enforcement agencies to be aware of the risks associated with the dark web and to take steps to mitigate them. Ultimately, the dark web is a reminder that the internet is a powerful tool that can be used for good or ill, and that we must remain vigilant in our efforts to ensure that it is used responsibly.

Thank you for reading please share it with your friends
You can checkout my other blogs here

Test post

Pannaga Perumal — Mon, 13 Mar 2023 08:48:45 +0000

pannaga

Go to Example.com

Top 3 Tech apps to try today

Pannaga Perumal — Thu, 09 Mar 2023 19:10:54 +0000

Tech news is an important part of staying informed about the latest developments in the ever-evolving technology industry. With so many news sources available, it can be difficult to keep track of everything that’s happening.That’s where tech news apps come in. These apps provide a convenient and efficient way to stay up-to-date on the latest tech news, whether you’re on the go or just want to quickly catch up on what’s happening in the tech world. In this article, we’ll take a closer look at some of the best tech news apps available today.

DevBytes

Devbytes covers the most recent developments, updates, and viewpoints in the tech sector. The app is made to assist developers to stay up to date with the newest trends and technology in the software development industry. We'll examine It's features and advantages in more detail in this post, along with the reasons developers and tech aficionados should give it some thought.

It has a tailored news stream is one of its best features. The software gathers content from a variety of sources, such as renowned tech blogs and newspapers, and presents it in a feed that is customised to your preferences.With a clear structure and straightforward design, the app enables users to easily obtain the news and information they require.It is a terrific choice for developers who want to remain up to date on the most recent tech news without spending too much time navigating through multiple sources thanks to its short articles of around 60 words.
Users can choose the subjects and sources that most appeal to them and they will receive news updates that are specifically tailored to their areas of expertise.

The app provides articles on a variety of subjects such as cybersecurity, artificial intelligence, and software development. Experts in their professions who have written the content offer insightful and unique viewpoints on the most recent changes in the market.
The cryptowatch in the app keeps the user updated with the current prices of the cryptocurrencies.It provides push alerts to keep consumers updated on crucial tech sector news and breaking updates.It has a user-friendly interface that makes it easy to navigate and find what you're looking for. The app's clean design and intuitive layout allow you to quickly access the news, articles, and tutorials that you're interested in. The app is also available on both Android and iOS, so you can access it on your phone or tablet.

One of the standout features of Devbytes is its expert analysis and insights on various tech topics. The app provides detailed articles, videos, and podcasts on subjects like programming languages, AI, and cybersecurity. These resources are created by experienced professionals in the tech industry, giving readers a valuable perspective on the latest trends and innovations.
Devbytes is a great tech news app that provides valuable insights and updates on the latest trends and technologies in the tech industry. With its personalized news feed, in-depth analysis and insights, push notifications, user-friendly interface, and accessibility on multiple platforms.

Devbytes is a great option for developers who want to stay informed and up-to-date in their field including the job openings. Whether you’re a seasoned developer or just getting started,It is definitely worth considering as your go-to source for tech news and insights.

Download the app here

TechCrunch

TechCrunch is a popular tech news app that offers comprehensive coverage of the latest developments in the tech industry. The app provides articles, videos, and podcasts on a range of topics, including startups, gadgets, and social media. TechCrunch is known for its in-depth reporting and analysis, making it an essential source for tech enthusiasts and industry professionals alike.

Wired

Wired is a tech news app that provides coverage of the latest tech news, analysis, and commentary. The app offers articles, videos, and podcasts on a range of topics, including gadgets, science, and culture. Wired is known for its in-depth reporting and analysis, making it a great resource for those who want to stay informed about the latest developments in the tech industry.

In conclusion, there are many great tech news apps available today that provide a convenient and efficient way to stay up-to-date on the latest developments in the tech industry. Whether you’re a tech enthusiast or a professional in the industry, these apps offer a wealth of information and insights that can help you stay informed and ahead of the curve.

Thank you for reading please share it with your friends

For such technology-related blogs visit click here

Demystifying Block chain and web 3.0

Pannaga Perumal — Sun, 05 Mar 2023 18:44:12 +0000

Let’s start with a very basic understanding of the internet i.e web 1.0 and web 2.0 before diving into web 3.0. The topic of blockchain has been in the news for quite a few years. It is tech that will completely change the way we look at the internet. It is a state of the computer application of cryptography and computer power…

What is Web 3.0?

During the early days of the internet, it was created so that two computers could communicate with each other. First, web 1.0 gave the capability for devices to communicate on a network where one host can disperse information and other devices can see the information. Web 2.0 leaped and gave access to interact seamlessly with other devices all over the globe thanks to the tech giants out there who made the internet accessible to all. Web 2.0 made its growth mainly through the enormous amount of data collected, assuring so-called privacy. A lar ge portion of the data on the internet is owned by big tech companies out there.

Web 3.0 is mainly developed from blockchain technology that gives users ownership of their data on the internet. It is just that simple, making the internet owned by everyone and ensuring privacy through complex cryptographic algorithms. It is aimed at changing almost every field that includes finance, healthcare, media, etc

Blockchain:Tech of the future
Now that you know about the basic of web 3 let's look into the technical aspects of blockchain technology. The blockchain simply means a chain of blocks let me explain each block will have a set of transactions that will be recorded the transaction may even be a search or a payment that you made on the network. Each network will have its cryptocurrency that is used like bitcoin, Ethereum, etc. The network will have nodes (Devices) that validate the transaction and record it into the block the user then he/she will be rewarded in the cryptocurrency of that network. One who is on the network can either be a normal user or a node. You may have heard of the term crypto mining it simply means validating the transactions by running cryptography algorithms thereby getting rewarded.

In simple terms, when you are making a payment on web 2 the bank validates the payment by checking your bank balance but on web 3 fellow users of the network validate your payments.

You may question how it secures our data, the answer is your data will not be saved in the centralized server but will be theoretically present on every device on the network in the form of cipher. Every device will have a copy of the blockchain each copy needs to be updated after every single transaction. When it comes to data security hacking is technically impossible as your data will be on millions of devices all across the globe. The identity of the user on the network will be a hexadecimal cipher code so you are just a number traveling across devices.

Applications of Block Chain

Crypto currency
The finance sector will be affected to a large extent due to the rise of cryptocurrencies such as bitcoin, Ethereum, Matic, etc. This revolution will replace large financial organizations and provides authority to everyone. Just imagine you need not obey the Terms & Conditions of some company blindly.

Medicine and Healthcare
The Health care sector will have quite impactful effect. It can help hospitals store their patient's data on a decentralized network in a more secure way that includes all the details of the patient that includes his/her medications etc. As it is decentralized any doctor can easily access the patients' data without the need to look through all the records. Just imagine even before the patient arrives in Infront of the doctor he will have the complete medical record of the patient.

It has a variety of applications in almost every field that we come across in our data day life. I would recommend every one of you to try out the web 3 space because the blockchain will be a mainstream tech a few years from now. So anyone willing to build a business then blockchain can be your choice.

Decentralized Apps
The idea of decentralized apps is purely based on the concept of a blockchain that distributes the user’s data across all devices to ensure security. This gives users freedom of speech where they can express their views and also surf the web without revealing their data to some XYZ company.

NFTs and Digital Media
The media space has seen a rise in a new sector called the NFTs which are the digital assets that are sold online. NFTs can be of any sort of audio, video, blog post, or even an image. Many popular organizations have sold their logos as an NFT. There are a variety of platforms that allows one to sell their artwork online for cryptocurrency one can check out Opensea

Conclusion

This blog was to provide you guys with the basics of web 3.0 and blockchain, the major takeaway from this was that web 3.0 is not only for the developers and content creators but it's for everyone. Just take a few minutes to appreciate the whole development and developers who are responsible for providing a secure and entirely new approach to the internet. This is just an introduction to blockchain in the successive blog posts I would be posting a detailed blog on cryptocurrencies and explaining to you guys how one can get into the web3 by developing NFTs. Special note I will also be posting the blog on how can one develop their first decentralized application.

Thank you for reading please share it with your friends
You can checkout my other blogs here

Top 3 Tech apps to try today

Pannaga Perumal ・ Mar 9 '23

For such technology-related blogs visit click here

Hate speech recognition using python.

Pannaga Perumal — Sat, 07 Jan 2023 14:37:35 +0000

In the world of today's internet the people all around the globe have the right of open speech through online forums.There are a few topics on which one cannot post or write on online forms we call it Hate speech.
So this blog gives you info about how to build an AI model that recognises hate speech on your forum.

Before diving into the actual technical aspects. Artificial intelligence has evolved to a new form when it comes to language models for example chatgpt,DallE2 etc.So these products use natural language processing and language understanding to provide results.
The model that we are building also uses language understanding which understands the text prompt and gives the negativity score that classifies whether it is hate speech or not.

We will be using python module called Tensorflow to train our language model.
Importing modules

import matplotlib.pyplot as plt
import os
import re
import shutil
import string
import tensorflow as tf
from tensorflow.keras import layers
from tensorflow.keras import losses
print(tf.__version__)

Downloading dataset
We will be using sentiment analysis data from Stanford University this is a dataset of movie reviews along with the labels labeling them as postive and negative review.

url = "https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"
dataset = tf.keras.utils.get_file("aclImdb_v1",url,untar=True,cache_dir='.',cache_subdir='')
dataset_dir=os.path.join(os.path.dirname(dataset),'aclImdb')

train_dir= os.path.join(dataset_dir,'train')
os.listdir(train_dir)

remove_dir =os.path.join(train_dir, 'unsup')
shutil.rmtree(remove_dir)
os.listdir(dataset_dir)

batch_size=32
seed = 42
raw_train_ds = tf.keras.utils.text_dataset_from_directory(
    'aclImdb/train',batch_size=batch_size,validation_split = 0.2,subset='training',seed=seed
)

The above code downloads the dataset from the website and divides it into train and test data.

Preprocessing the data
You can view the dataset using the below code

for text_batch, label_batch in raw_train_ds.take(1):
    for i in range(3):
        print("Review",text_batch.numpy()[i])
        print("Label",label_batch.numpy()[i])

we will group out the train and test datasets into variables

raw_val_ds = tf.keras.utils.text_dataset_from_directory(
    'aclImdb/train',batch_size=batch_size,
    validation_split=0.2,
    subset='validation',
    seed = seed
)
raw_test_ds=tf.keras.utils.text_dataset_from_directory('aclImdb/test',
batch_size=batch_size)

Then as we derive data from webpages we need to remove html tags and other symbols like emojis etc so we define a custom standardization function that standardizes the data. Also we implement the function as a vectorization layer so that it can be used in our model.

def custom_standardisation(input_data):
    lowercase=tf.strings.lower(input_data)
    stripped_html= tf.strings.regex_replace(lowercase,'<br />',' ')
    return tf.strings.regex_replace(stripped_html,'[%s]' %re.escape(string.punctuation),
    '')        

max_features = 10000
sequence_length=250

#creating a layer that applies custom standardization
vectorize_layer=layers.TextVectorization(
    standardize=custom_standardisation, max_tokens=max_features,
    output_mode='int',
    output_sequence_length=sequence_length
)

you can view the results of vector layer using below code.(optional)

print("Review",first_review)
print("label", raw_train_ds.class_names[first_label])
print("vectorized Review", vectorize_text(first_review, first_label))
print("1287 ----->",vectorize_layer.get_vocabulary()[1287])
print(" 313 ---->",vectorize_layer.get_vocabulary()[313])
print("vocabulary size:{}".format(len(vectorize_layer.get_vocabulary())))

Defining the model
Now we define our actual language model by taking the standardized input by using custom layer that automates the pre processing.

train_ds = raw_train_ds.map(vectorize_text)
val_ds = raw_val_ds.map(vectorize_text)
test_ds=raw_test_ds.map(vectorize_text)

AUTOTUNE= tf.data.AUTOTUNE

train_ds=train_ds.cache().prefetch(buffer_size=AUTOTUNE)
val_ds=val_ds.cache().prefetch(buffer_size=AUTOTUNE)
test_ds = test_ds.cache().prefetch(buffer_size=AUTOTUNE)

#building the model with custom layer
embedding= 16
model = tf.keras.Sequential([
   #vectorize_layer,
    layers.Embedding(max_features + 1,embedding),
    layers.Dropout(0.2),
    layers.GlobalAvgPool1D(),
    layers.Dropout(0.2),
    layers.Dense(1),
    #layers.Activation('sigmoid')
    ])
model.summary()

We compile the model with the loss function optimizer and the number of epochs.

model.compile(
        optimizer='adam',
loss=losses.BinaryCrossentropy(from_logits=True),
metrics=['BinaryAccuracy'])
epochs=10

history= model.fit(
train_ds,
validation_data=val_ds,
epochs=epochs
)

loss , accuracy= model.evaluate(test_ds)
print("loss: ",loss)
print("Accuracy: ",accuracy)

We will modify and finalize the model.

final_model = tf.keras.Sequential([
    vectorize_layer,
    model,
    layers.Activation('sigmoid')
])
final_model.compile(loss=losses.BinaryCrossentropy(from_logits=False),optimizer='adam',metrics=['accuracy'])

Hurray you have created a model that automates the process of detecting hate speech.
You can view the learning curve of the model through the below code.

history_dict=history.history
history_dict.keys()
acc = history_dict['binary_accuracy']
val_acc= history_dict['val_binary_accuracy']
loss=history_dict['loss']
val_loss=history_dict['val_loss']
epochs=range(1, len(acc)+1)
#bo for blue dot
plt.plot(epochs, loss, 'bo', label='Training loss')
#b for blue solid line
plt.plot(epochs, val_loss,'b',label='Validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.show()
#bo for blue dot
plt.plot(epochs, acc, 'bo', label='Training loss')
#b for blue solid line
plt.plot(epochs, val_acc,'b',label='Validation loss')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.show()

User Interface
In this blog we have developed a basic user interface that asks for the prompt by the user and returns the negativity score.Below code is the implementation of the same.

import numpy as np
examples = []
q=input("enter a review: \n")
examples.append(q)
p=final_model.predict(examples)
print('negativity score',end=' ')
print(p[-1])

You can further deploy the model in your website that provides you a alert whenever it detects hate speech on your forum.

Thank you for reading the blog.
For more such informative blogs do follow.
share it with your friends and family.😁

What makes "MAANG special?

Pannaga Perumal — Fri, 25 Nov 2022 05:53:36 +0000

For all who don't know what MAANG is it's an abbreviation which stands for
Meta,Apple,Amazon,Netflix and Google. Have you wondered why these big tech companies are so successfull.Let us try to go a little deeper and understand.

In a conventional organization ther will be a hierarchy of employees like CEO,manger,team lead,team member.This hirarchy will sometime create a non friendly working atmosphere to it's employees this reduces the productivity of the employees.
Here's where the MAANG companies takes over by understanding the problem they have aimed to create a best working atmosphere to its employees.

These big tech companies creates a bond between the employees and the company to create a homely atmosphere.The work culture here will be lot more than just getting paid for the job they do.
They give their employees a identity one who works in google is googler And also they provide dedicated hours of their work for the employees where they can work on the project of their intrest this feature is one of the crowd major reason why MAANG is dream company for many techies.
The book 'The culture code' explain how these big teams work.The book explains the in and out of all the successful and well performing teams.

Also they try to keep their work places creative as possible so as to boost their performance.Also these workplaces will be covered with lush greenery as staying with the nature gives positive energy.

The above picture is the Amazon headquarters in silicon valley the lush green surrounding makes the place beautiful.These workplace has replaced the old cubicle format with small discussion rooms where employees can collaborate on their projects.Every corner of the place will have a white board where people can display their thoughts and ideas. Even the smallest thing that we make in our work culture may affect the company's growth.

By taking these MAANG companies as their inspiration the start ups are incorporating the changes and are creating a healthy work atmosphere for its employees.

how to build one?

Here are few important points that you should consider.
Actually building does not mean building the next tech empire.The principles can be applied to almost every team and a community whether it may be a football team or a Tesla production line.To establish such a organisation you need to create a feeling of oneness among them.
Few points that must be considered,

Create a vulnerability
Develop a feeling where the companies problem is his problem the company's success will be his success that motivates him to work.
Respect every opinion
The lead must hear to everyone's suggestions,as people tend to stay where they are valued.
Give them me time
Every member must be given his me time where he has the liberty to do or work on the things of his/her intrest this makes rejoice.

So,big tech empires are not only built on big projects but also in small and creative changes that makes them stand out and create value for both it's customers and employees.

For more such inspiring blogs
Follow
Linkedin

Are we living inside Our Mobiles?

Pannaga Perumal — Wed, 11 May 2022 19:01:00 +0000

Have you ever thought how frequently do you check your phones? How many hours do you spend on social media daily? I guess most of you will be spending more than half of your day on social media. What makes social media so addictive...(have you ever thought)
Answer of some may be No.

The reason behind these platforms becoming so addictive is the developers working all day long behind the screens you see.

Few days ago I read a book named "Hooked: How to build habit building products" written by Neir Eyal(professor at Stanford business school.
After reading the book I got answers to my questions. In this book Neir explains the "hooked model" that is used by the almost all the social media platform that you see around. The main motive behind these platforms are user attention span, and make them spend more time on their platform.
This is accomplished by giving the user rewards(not the material ones) but the pleasure.
For example Every time you visit Instagram you will never get bored because it provides with the photos that your friends have shared,likes,comments etc. You are much unlikely to stop at one photo(post) you will start scrolling, as you go on the posts will keep on loading make you spend hours together on the platform.

As a add on the data that the platform receives uses that to understand their users better and hence keeping them hooked.
You may have noticed people saying "I'll google it" instead of "I'll do some re-search on it" there is no dictionary that say google and research are synonyms but, the amount of convenience, pleasure and usability that google provides, has made that change. The developers who build these habit building product focus on more on their user psychology in order to make them keep hooked. The platform focuses on the insecurity that the user is facing they solve that in security there giving them pleasure.

Before the rise of online shopping people used to buy the things in the offline store, the people faced problem in going to far away shops to buy even a small thing. Online shopping gave the user the convenience of buying the stuff and delivered to their door steps.
The convenience that it provide makes the user to the site again to buy things, in return they use your data to make you buy from their store itself thereby making it a habit.

Thanks for your support!

PS PANNAGA