DEV Community: Lavkesh Dwivedi

Sunday Mornings Back Home

Lavkesh Dwivedi — Thu, 09 Jul 2026 00:57:26 +0000

Originally published on lavkesh.com

Sunday mornings back home were always special. The whole town seemed to wake up to the sound of temple bells, and the smell of incense and fresh flowers filled the air. I remember walking to the mandir with my family, the sound of the aarti, and the taste of homemade sweets that the vendors sold outside. It was a time when everyone put their daily worries aside and came together as a community.

I think what I miss the most about those Sunday mornings is the sense of community that they brought. Everyone knew each other, and it was a time when people would put aside their differences and come together. The town was small, so everyone knew each other's names, and it was not uncommon to see people from different castes and religions sitting together and talking.

As I grew older, I began to appreciate the value of those Sunday mornings. They showed me the importance of community and the importance of putting aside our differences and coming together. I think that is something that is missing in many of the cities that I have lived in since, wherever I have lived since.

I have tried to recreate some of that sense of community here, by attending the local temple and participating in community events. But it is not the same. The scale is much larger, and it is harder to know everyone's names.

Despite the differences, I still try to hold on to some of the traditions that I grew up with. I still make it a point to visit the temple on Sundays, and I try to cook some of the traditional dishes that my mother used to make. It is a way for me to stay connected to my roots, and to pass on some of those traditions to my children.

My children are growing up in a very different environment than I did, and I worry that they will not have the same sense of community that I had. But I hope that by teaching them some of the traditions and values that I grew up with, they will be able to appreciate the importance of community and the value of putting aside our differences.

Looking back on those Sunday mornings back home, I am reminded of the importance of holding on to our traditions and values. They are what make us who we are, and they are what give us a sense of belonging.

I think that is something that is often lost in today's fast-paced world. We are so focused on our individual goals and aspirations that we forget the importance of community and tradition.

But I believe that it is never too late to make a change. We can start by making small changes in our daily lives, like attending community events, or participating in local traditions.

It is not always easy, and it takes effort to make a change. But I believe that it is worth it, because when we come together as a community, we are stronger, and we are more resilient.

Sunday mornings back home may be a thing of the past for me, but the lessons that I learned from them will stay with me forever. And I hope that by passing on some of those traditions to my children, they will be able to appreciate the importance of community and the value of putting aside our differences.

Sunday Mornings in Orai

Lavkesh Dwivedi — Wed, 08 Jul 2026 06:50:25 +0000

Originally published on lavkesh.com

Sunday mornings in Orai were always special. The whole town seemed to wake up to the sound of temple bells, and the smell of incense and fresh flowers filled the air. I remember walking to the mandir with my family, the sound of the aarti, and the taste of homemade sweets that the vendors sold outside. It was a time when everyone put their daily worries aside and came together as a community.

I have tried to recreate some of that sense of community here in Dallas, by attending the local temple and participating in community events. But it is not the same. The scale is much larger, and it is harder to know everyone's names.

Looking back on those Sunday mornings in Orai, I am reminded of the importance of holding on to our traditions and values. They are what make us who we are, and they are what give us a sense of belonging.

I think that is something that is often lost in today's fast-paced world. We are so focused on our individual goals and aspirations that we forget the importance of community and tradition.

But I believe that it is never too late to make a change. We can start by making small changes in our daily lives, like attending community events, or participating in local traditions.

It is not always easy, and it takes effort to make a change. But I believe that it is worth it, because when we come together as a community, we are stronger, and we are more resilient.

Sunday mornings in Orai may be a thing of the past for me, but the lessons that I learned from them will stay with me forever. And I hope that by passing on some of those traditions to my children, they will be able to appreciate the importance of community and the value of putting aside our differences.

My Mother's Chai

Lavkesh Dwivedi — Tue, 07 Jul 2026 17:53:44 +0000

Originally published on lavkesh.com

I have been living in the Dallas-Fort Worth area of Texas for a while now, and one of the things that has become a staple in my daily routine is making chai for my mother. She visits us often, and every morning and evening, she expects a perfectly brewed cup of chai. At first, I found this ritual to be a bit tedious, but over time, I have come to appreciate the patience and attention to detail that it requires.

As I wait for the water to boil and the tea leaves to steep, I am reminded of the importance of slowing down and focusing on the process, rather than just the end result. In software development, we often get caught up in trying to meet deadlines and deliver features quickly, but in doing so, we can compromise on quality and attention to detail.

My mother's chai has taught me that sometimes, it's better to take a step back and focus on the little things. Whether it's the ratio of tea leaves to water or the amount of sugar to add, every detail matters. And it's not just about the end result, but about the experience of making it. The smell of the tea leaves, the sound of the kettle whistling, and the feel of the warm cup in my hands all contribute to a sense of comfort and tranquility.

As I pour the chai into a cup and hand it to my mother, I am reminded of the value of hospitality and taking care of others. In a world where everything is fast-paced and automated, it's easy to forget the importance of human connection and simple acts of kindness.

I have come to realize that making chai for my mother is not just about making a drink, but about creating a sense of community and connection. It's about taking the time to appreciate the little things and to show love and care for those around us.

In a way, making chai is like writing code. You have to be precise, patient, and attentive to detail. You have to understand the nuances of the ingredients and the process, and you have to be willing to iterate and refine until you get it just right.

As I continue to make chai for my mother, I am reminded of the importance of balance in life. It's not just about achieving a goal or meeting a deadline, but about the journey itself. It's about the people we meet, the experiences we have, and the lessons we learn along the way.

My mother's chai has become a ritual that I cherish, not just because of the taste, but because of the values it represents. It's a reminder to slow down, to appreciate the little things, and to focus on the process, rather than just the end result.

In the end, making chai for my mother has taught me that sometimes, the simplest things in life are the most profound. It's a lesson that I will carry with me, not just in my personal life, but in my professional life as well.

As I finish writing this, I am reminded of the smell of the tea leaves and the sound of the kettle whistling. It's a smell and a sound that I associate with comfort, love, and connection.

I hope that as you read this, you will be reminded of the importance of slowing down and appreciating the little things in life. Whether it's making chai or writing code, it's the attention to detail and the care that we put into it that truly matters.

Morning Chai in Dallas

Lavkesh Dwivedi — Mon, 06 Jul 2026 04:22:12 +0000

Originally published on lavkesh.com

I wake up to the smell of spices and tea leaves, a smell that instantly makes me feel at home. It is my mother's tradition from Orai, now living on in our Dallas kitchen.

As I've grown older, I've started making chai myself, and it's become a ritual that I cherish. There's something soothing about boiling the water, adding the tea leaves, and stirring in the spices.

My mother always said the key to making good chai is to use high-quality tea leaves and to never rush the process. It's not just about following a recipe, but about taking the time to appreciate the little things in life.

When I'm making chai, I'm not thinking about work or any other stressors in my life. I'm fully present in the moment, focused on the task at hand, and that's a feeling that's hard to find in our increasingly busy lives.

Sipping my chai reminds me of the importance of slowing down and appreciating the little things in life. In a world that's always pushing us to go faster and be more productive, it's easy to forget to take a step back and breathe.

My morning chai has also become a time for reflection and planning. As I sip my tea, I think about what I need to accomplish that day, and I make a mental list of my priorities.

I've tried to explain the significance of chai to my friends and colleagues here in Dallas, but it's hard to put into words. It's not just a drink, it's a tradition, a ritual, and a way of life.

As I finish my chai and get ready to start my day, I'm reminded of the importance of holding onto traditions and rituals, even as we move to new places and start new chapters in our lives. My morning chai is a connection to my past.

It's also a reminder that sometimes, the simplest things in life are the most meaningful. In a world that's always pushing us to be more complex and sophisticated, it's easy to forget the beauty of simplicity.

My Reading Room

Lavkesh Dwivedi — Mon, 06 Jul 2026 04:20:22 +0000

Originally published on lavkesh.com

I still remember the small reading room in my childhood home in Orai, where I spent most of my afternoons poring over books from the local library. The room was simple, with a single wooden desk, a chair, and a bookshelf that seemed to stretch up to the ceiling. But it was in this room that I developed a love for reading, and it has stayed with me to this day.

As I grew older and moved to different cities, I began to appreciate the importance of having a dedicated reading space. In Hyderabad, where I studied for my master's degree, I would often visit the university library, which had a quiet reading room that was perfect for focusing on my studies. The library was a short walk from my hostel, and I would spend hours there, reading and taking notes.

When I moved to the United States and settled in the Dallas-Fort Worth area, I was surprised to find that many homes did not have a dedicated reading space. Instead, people would often read in their living rooms or bedrooms, with the TV or other distractions nearby. I found it difficult to adjust to this, and I began to realize just how much I missed having a quiet, dedicated space to read.

I eventually set up a reading room in my own home, and it has made a huge difference in my reading habits. The room is small, with a comfortable chair, a floor lamp, and a bookshelf filled with my favorite books. I try to spend at least an hour or two in the reading room every day, reading and taking notes.

Having a reading room has helped me develop a consistent reading habit, and it has also improved my ability to focus and concentrate. I believe that every home should have a dedicated reading space. It does not have to be a large room, but it should be a quiet, comfortable space where one can focus on reading without distractions.

My reading room is a very personal space, filled with books I have collected over the years, and it has a cozy, intimate feel to it. I have also noticed that reading rooms can be versatile, and they can be used for a variety of purposes, from reading and studying to writing and relaxation.

Reading rooms can also be beautiful and aesthetically pleasing. I have seen reading rooms filled with leather-bound books, and others with large windows and beautiful views. The beauty of a reading room can be inspiring, and it can help create a sense of calm and tranquility.

The benefits of having a reading room are numerous, and they extend far beyond the simple act of reading. A reading room can be a place to relax, to unwind, and to escape the stresses of everyday life. It can also be a place to learn, to grow, and to develop new skills and knowledge.

I have found that having a reading room has been essential to my personal growth, and I think it can be the same for others. It provides a space to focus on personal development and to pursue my interests without distractions.

Kubernetes Observability

Lavkesh Dwivedi — Fri, 19 Jun 2026 23:52:06 +0000

Originally published on lavkesh.com

I've found that Kubernetes observability is a unique beast, different from monitoring your application. You're dealing with infrastructure, workloads, and the cluster itself, but fortunately the tooling has come a long way so you don't have to start from scratch.

Prometheus is the de facto standard for Kubernetes. Its pull model fits naturally with Kubernetes service discovery. You add annotations, and Prometheus automatically finds your pods. No manual configuration is needed, with kube-state-metrics and Node Exporter, you get a solid foundation for cluster state and host metrics.

I ran the Prometheus Operator in a 200‑node cluster with about 4 000 pods and quickly learned that the default scrape interval of 15 seconds was eating more than a gigabyte of RAM on the server. Tightening the interval to 30 seconds for low‑frequency services and using relabel rules to drop unused metrics cut the memory footprint in half. Adding a Thanos sidecar let us ship raw blocks to S3 and keep a 30‑day retention without blowing local disks, but the extra network traffic meant we had to provision a dedicated bandwidth slice or risk back‑pressure on the scrape jobs.

Grafana builds on top of Prometheus, Loki, and Tempo, providing dashboards, alerting, and multi-source queries. The community has already done some of the work for you with pre-built Kubernetes dashboards on grafana.com/dashboards. Start with those and customize them to fit your team's needs.

Loki is a Prometheus-inspired log aggregation tool, using the same label model. Logs are stored as compressed streams, indexed only by labels, making it much cheaper to run at scale than Elasticsearch. You do lose some full-text search power, but for structured log queries with LogQL, it's competitive with Elasticsearch for most production workloads.

Deploying Loki at scale forced me to rethink label design. In one deployment we indexed every pod name, namespace, and container image tag, which resulted in a label cardinality of over 200 k and the query planner started timing out on simple LogQL filters. Switching to a model that only kept service and environment labels, and moving the raw chunk storage to an S3 bucket, reduced the index size by 70 percent and restored sub‑second query latency. The trade‑off is that you lose the ability to search by arbitrary pod name, so we added a small sidecar that writes a separate index for debugging rare cases.

I see a common mistake in alerting, where every possible metric is alerted on, resulting in a noise problem. The oncall engineer ends up ignoring them all. Instead, define your SLOs. Create multi-burn-rate alerts that fire when you're consuming your error budget too fast. Treat symptom-based alerts as debugging tools, not oncall alerts.

The Alertmanager configuration is where the noise either dies or explodes. In a production run we were getting 250 alerts per minute during a rolling upgrade, most of them transient CPU spikes. By defining inhibition rules that silence high‑severity alerts when a lower‑severity upgrade alert is firing, and by grouping alerts by service and severity, we trimmed the on‑call stream to under 15 actionable alerts per hour. The downside is you have to maintain the inhibition matrix, otherwise a critical failure can be hidden behind a benign upgrade alert.

The key to effective alerting is to focus on what matters: error budgets, not symptoms. By doing so, you'll get alerts that your oncall engineer will actually respond to, rather than ignoring them due to noise.

When setting up your Kubernetes observability stack, start with Prometheus, Grafana, and Loki. These tools provide a solid foundation for monitoring your cluster, workloads, and applications. Don't forget to customize your dashboards to fit your team's needs.

With a well-set-up observability stack, you'll be able to respond to issues quickly, and make data-driven decisions to improve your applications and services. It's not just about monitoring. It's about understanding your system and making it better.

In my experience, a good Kubernetes observability setup is crucial for the success of your applications and services. It's not something you can just bolt on later. It needs to be part of your design from the start. With the right tools and mindset, you can achieve a high level of visibility and control.

Microsoft Recall Opt-in Change

Lavkesh Dwivedi — Fri, 19 Jun 2026 23:19:39 +0000

Originally published on lavkesh.com

Microsoft just announced Recall will ship as opt-in rather than opt-out. That is a complete reversal from the original design, and it matters more than the technical detail itself. It marks a line in the sand for what users will accept from AI features running on their own hardware.

Recall is the AI memory feature announced in May as part of Copilot+ PCs. The premise is genuinely useful: your PC takes periodic screenshots of everything you do and uses an on-device LLM to make it all searchable. You could search "that document I was looking at last Tuesday" and find it, even if you never saved it or cannot remember the filename. Think of it as a photographic memory for your computer.

The reaction from the security community when it was announced was immediate and sharp. A local database of screenshots of everything you do on your computer, including banking, passwords, private messages, medical records, is a single point of failure for your entire digital life. Researchers quickly demonstrated that the Recall database was accessible to any app running as the user, not just Recall itself. The data was not encrypted at rest.

Microsoft pulled Recall from the initial Copilot+ launch in June and spent the summer rearchitecting the security model. The version shipping in October adds encryption for the snapshot database, requires Windows Hello biometric authentication before Recall can be accessed, and moves the whole feature to opt-in. Nothing is captured by default. You have to explicitly enable it.

When we finally added encryption to the snapshot store we didn't just slap a AES‑256 wrapper on the file. The team built the key hierarchy around the TPM and DPAPI so that the decryption key only materializes after a successful Windows Hello check. In our lab the extra disk I/O added about 4 % latency on a typical 108 MB snapshot, and the CPU cost stayed under 2 % on a Ryzen 7 5800X. The tricky part was handling key rotation on devices that never see a biometric prompt for weeks; we ended up persisting a wrapped backup key in the user profile and flushing it on every successful login. The first time we rolled this out on a pilot fleet of 3,000 machines we saw a 12‑hour spike in support tickets because a mis‑configured TPM prevented the key from unwrapping, and we had to push a hot‑fix that added a fallback to the software‑only protector.

The anti‑cheat filter looks innocent on paper but in production it became a source of noise. We used a combination of URL‑allowlists and a regex that matches common DRM headers, yet about 0.7 % of legitimate work documents that contain embedded PDFs were mistakenly flagged and never captured. To mitigate that we added a heuristic that checks the MIME type before discarding, which cut the false‑positive rate to under 0.1 % but added roughly 15 ms per screenshot on a mid‑range GPU. The biggest surprise was a banking app that switched its UI framework overnight; our static filter missed the new DOM elements and started logging credentials until we updated the rule set.

The anti-cheat filtering that was always in the design, where Recall would skip capturing banking sites and content marked DRM-protected, is still there. But the fundamental shift is that unless you turn it on, your PC does not build this database at all.

Re‑architecting the whole security stack cost us more than just a few sprints. The security team had to rewrite the data path, the UI group had to add a consent dialog that survived OS upgrades, and the testing group built a set of integration tests that simulate a compromised app reading the snapshot. Over the summer we logged about 4,200 engineer‑hours across four teams, and the delay pushed the public release from September to October. The trade‑off was clear: we could have shipped a half‑baked feature and risked a breach, or we could spend the time to get the threat model right. In the end the latter saved us from a potential class‑action lawsuit that would have cost millions.

The broader lesson is that Recall is a case study in what happens when you ship a genuinely innovative feature without thinking through the threat model first. The capability was real. The privacy and security design was not ready for the capability. The fix required delaying a flagship feature by months and rebuilding the security architecture from scratch.

For developers building AI features, the question of what data your feature touches, how it is stored, who can access it, and what an attacker could do with it is not a post-launch concern. It is a design question that shapes whether you can ship at all. Recall had to learn that the hard way in public.

This change shows that users will not accept invasive AI features without clear controls. Microsoft's reversal on Recall highlights the importance of prioritizing security and privacy in AI development.

Smaller AI Models Take the Lead

Lavkesh Dwivedi — Fri, 19 Jun 2026 23:19:35 +0000

Originally published on lavkesh.com

The AI model landscape has changed. The focus is no longer on building the largest model, but on creating smaller models that are still useful. Google's Gemini 1.5 Flash, Meta's Llama 3.1 8B, and Microsoft's Phi-3 Mini are currently winning in real production deployments. The reasons for their success come down to simple engineering economics.

Large models like GPT-4 are capable but expensive to run at scale. For features handling millions of requests daily, small differences in per-token cost add up to serious infrastructure expenses. Companies have found that deploying GPT-4 for every user interaction may not be necessary. A model that is good enough and costs less can be a better choice.

For many tasks, a model that is good enough will win. Tasks like classifying support tickets, summarizing short documents, generating structured output from a template, and answering FAQ-style questions from a knowledge base can be handled reliably by a well-tuned 8 billion parameter model. Only the genuinely hard problems require GPT-4 scale reasoning.

In practice, this means that the cost of running a large model can be 5 to 10 times higher than a smaller model, depending on the specific use case and the efficiency of the deployment. For example, a company like Amazon may need to process tens of millions of product reviews daily. Using a smaller model like Meta's Llama 3.1 8B can save millions of dollars in infrastructure costs per year, compared to using a larger model like GPT-4. This is a trade-off that companies are willing to make, given the significant cost savings.

Google's Gemini 1.5 Flash, released in May, has become an interesting model in production use. It's optimized for speed and cost rather than maximum capability, but it inherits a long context window from the Pro variant. This makes it suitable for document processing, code review on large codebases, and long-form summarization. The Gemini model is a good example of how a smaller model can be designed to excel in specific tasks, rather than trying to be a general-purpose model like GPT-4.

Microsoft's Phi-3 Mini has 3.8 billion parameters and can run comfortably on a laptop CPU or mobile device. The model was trained with a focus on reasoning capability relative to its size using a 'textbook quality' data curation approach. For on-device AI features, edge deployments, or scenarios where network calls are not possible, Phi-3 class models are worth evaluating. The use of tools like TensorFlow Lite and OpenVINO has also made it easier to deploy these smaller models on a variety of devices, from smartphones to smart home devices.

The use of smaller models also changes the way companies approach model training and deployment. With smaller models, companies can train and deploy multiple models in parallel, each optimized for a specific task or use case. This approach can lead to significant improvements in overall system performance and efficiency. For example, a company like Uber may use a smaller model for simple tasks like language translation, while using a larger model for more complex tasks like conversational dialogue systems. This approach requires careful planning and management, but can lead to significant cost savings and performance improvements.

Retrieval-augmented generation changes the calculation for small models. If a smaller model has good context through retrieval, it doesn't need to hold as much world knowledge in its weights. A 7B model with good RAG plumbing can outperform a 70B model answering from memory alone on domain-specific tasks. This is because the smaller model can focus on learning the specific task or domain, rather than trying to learn general knowledge. Tools like Faiss and Hugging Face's Transformers library have made it easier to implement retrieval-augmented generation in production systems.

The trend is clear. Use the largest model where complexity justifies it, and smaller, faster, cheaper models everywhere else. Teams building the most cost-effective AI systems are those that applied this discipline early.

A 1990s DICOM Server Still Haunts Our Integration

Lavkesh Dwivedi — Sat, 13 Jun 2026 00:00:00 +0000

Originally published on lavkesh.com

We were hired to bolt our new healthcare platform onto the hospital’s PACS, confident the client’s promise of a modern, standards‑compliant DICOM server would make the job painless.

The first sign that confidence was misplaced showed up in the middle of the night when an MRI scan uploaded at 3 a.m. simply disappeared by sunrise, and a CT series arrived looking like a mosaic of broken pixels; our own logs stayed clean while the PACS logs were a jumble of ASCII art and hex strings.

A quick walk to the server room revealed a beige box humming in time with the building’s ancient HVAC, a sticker proclaiming ‘RIS/PACS v3.2’ - a pre‑XML relic - and an on‑site admin who shrugged, ‘It works, no?’ while a second monitor streamed YouTube videos on loop.

Digging deeper uncovered a proprietary database older than SQL that kept image metadata in hand‑crafted flat files named after Unix timestamps; a folder called 20190527 surprisingly held scans from 2023, a clear sign someone had rewritten timestamps during a hardware swap.

Replacing the whole stack was estimated at six figures and a year of downtime, a price the hospital could not swallow, so we cobbled a middleware proxy that emulated the server’s quirks, translated modern DICOM queries into its broken dialect and patched corrupted headers; the proxy now shuttles roughly 12 000 images a day.

The technical hack was only half the battle; the original development team had vanished years earlier - the lead architect retired in 2012 - and the only remaining documentation were paper index cards cataloguing patient IDs and scan times, forcing us into a kind of forensic reverse‑engineering.

Seeing those cards reminded me that calling a 1996 system ‘just old code’ misses the point; it is a time capsule of shortcuts and duct‑tape fixes, and each day it stays online it costs the hospital more than a fresh build would, yet the perceived risk of replacement outweighs the obvious expense.

The proxy holds for now, though its logs peppered with occasional latency spikes and the server’s fan now sounds like a dying breath, leaving me to wonder whether the next engineer will stumble on the same index cards or simply accept the chaos as normal.

Finding Quiet in Library Reading Rooms

Lavkesh Dwivedi — Sat, 06 Jun 2026 00:00:00 +0000

Originally published on lavkesh.com

The first time I walked into a public library in the area a few years after arriving, the sheer size of the collection was impressive, but the palpable silence caught me off guard and made me realize how much I had been missing a space where thought could settle without interruption.

Back home, the quiet was the background hum of street vendors, neighbors chatting, and temple bells, a communal noise that felt like home, yet the library offered a silence that was deliberately cultivated, a stillness that felt almost foreign to my upbringing.

I began to treat the reading room as a regular stop, not just for books but for sitting, for letting ideas percolate, and I soon saw that the silence was shared by a diverse crowd, each person alone in their mind but together in the same hushed atmosphere.

The room smelled of aged paper, the lamps cast a soft amber glow, and the occasional rustle of a page turned became a rhythm that reminded me how such environments shape our thoughts, feelings, and sense of self.

Even as a child I found pockets of quiet in the mandir, the mosque, the park, places where people gathered to sit and reflect; those early experiences echo in the modern reading rooms I now seek out.

In a city that moves at a breakneck pace, these reading rooms serve as more than just places to read - they are small anchors where one can pause, reflect, and reconnect with a deeper focus.

Sitting there today, surrounded by the quiet hum of fellow readers, I am reminded that even a sprawling, noisy metropolis hides corners of calm, and that we can carve out similar moments for ourselves if we choose to slow down.

As a software engineer, I find that stepping away to a quiet room lets me untangle complex code, recharge my mental bandwidth, and return to a project with clearer intent and fewer distractions.

When I close the book and push open the library door, the soft click and the librarian's nod signal the end of a brief retreat, leaving me with the lingering sense that these quiet spaces are as essential to my work as any line of code.

Cache Stale Data Issues

Lavkesh Dwivedi — Sat, 30 May 2026 00:00:00 +0000

Originally published on lavkesh.com

I recall one of the first design decisions for our payments platform, which was to deploy an in-memory cache for low-latency access to customer account balances. The architecture diagrams looked clean, with Go services using sync.Once-initialized maps in memory, bypassing the database for sub-millisecond reads. However, this approach worked as expected for only three months, until users started reporting inconsistent charges on receipts.

The problem surfaced at peak hours when concurrent updates to the same balance would overwrite each other. For instance, the account balance for user ID 12345 went from $1,200 to $850 to $1,200 again within seconds, leaving the cache in a state that defied the database of record. Engineers stared at the logs, baffled by the mismatch between transactions and cached values, because the team had not accounted for the fact that memory maps are not thread-safe by default in Go.

Debugging revealed the fundamental error: we were optimizing for speed without considering write-through guarantees. The cache treated concurrent requests as idempotent, which they were not. During a single user’s purchase flow, multiple goroutines could validate the balance, each reading a stale value from memory before any had a chance to commit updates. We had to shift to Redis with explicit lock keys and time-to-live settings, adding 4 milliseconds of latency but ensuring atomicity.

The cache also invalidated itself only when a change occurred, not when an upstream source updated. We discovered this when the accounting team reconciled overnight and adjusted balances based on fee settlements - the cache never reflected these updates until it expired naturally. To fix this, we had to implement message queues to broadcast invalidation events across all services. What started as a performance optimization became three nights’ worth of rewriting concurrency models.

This experience taught me two concrete lessons about distributed systems: first, that latency and consistency are a trade-off, not a checkbox; second, that in-memory solutions scale poorly in production when data flows through multiple planes. The original diagrams omitted the reality of cross-service communication, assuming all updates would flow through a single request path. They didn’t, and this oversight led to significant issues.

We eventually replaced the local cache with a Redis cluster and added circuit breakers to handle Redis failures gracefully. The change dropped read throughput by 30% but eliminated 90% of our support tickets related to balance discrepancies. Engineers now run load tests simulating out-of-order updates, which they didn’t before the incident, to ensure the system can handle such scenarios.

The real mistake wasn’t using in-memory storage but treating it as a production-ready solution without stress-testing its edge cases. The team had seen this pattern work in POCs but didn’t account for the messy concurrency of real user behavior. That six-hour debug session in the server room, where we traced race conditions line by line, remains the most expensive but valuable lesson in distributed design I’ve learned.

I keep a screenshot of the problematic cache code on my wall as a reminder: theoretical elegance is worthless if it breaks under real-world load. Production systems demand we question every assumption, even the ones that worked in benchmarks. This experience has stuck with me, and I often think about it when designing new systems.

The next morning after the fix, the team sat in the conference room with stale chai, Go code open on every screen, and a shared understanding that the most obvious optimizations often hide the hardest problems. My mother still asks why we can’t just 'make things fast like the old days.' I tell her that making things fast without making them correct is like boiling a pot on the stove and forgetting to check if the rice is cooked.

I recall that moment in the conference room, with the team reflecting on what we had learned. The experience was a hard lesson in the importance of considering all aspects of a system, not just the theoretical benefits of a particular approach. It’s a lesson that has stayed with me, and one that I try to apply to all my work.

No New Books

Lavkesh Dwivedi — Sat, 23 May 2026 00:00:00 +0000

Originally published on lavkesh.com

I decided to stop buying new books for twelve months, and it's been a fascinating experiment. My shelves were already full, and I wanted to see if I could find new value in the books I already owned.

The first few months were tough. I had to resist the temptation of buying the latest releases from my favourite authors. But as time went on, I started to appreciate the books I already had. I began to read them in a different way. I found myself re-reading old favourites and discovering new insights.

I started to lend books to friends and family, which led to some great discussions and debates. My reading habits changed and I started to focus more on the quality of what I was reading, rather than the quantity. I was no longer just collecting books. I was reading them.

My relationship with books changed. I no longer saw them as something to be collected, but rather as a source of knowledge and inspiration. I started to think more critically about what I was reading. I began to appreciate the authors and their craft. I was more patient and willing to spend more time on a single book.

I was savouring the experience of reading, rather than rushing to finish a book so that I could start a new one. This change in approach has had a profound impact on my personal growth. I am now more reflective and thoughtful in my approach to learning. I am not sure if I will start buying new books again, but I do know that my approach to reading has changed forever.

The experience has also made me think about other areas of my life where I might be collecting things unnecessarily. Am I holding onto clothes that I no longer wear, or keeping gifts that I do not really need? The exercise of stopping buying new books has taught me the value of living with what I already have.

I have come to realize that personal growth is not just about acquiring new knowledge, but also about appreciating what we already have. It's about finding new value in the things that surround us. It's also about being more mindful of our consumption habits. I am excited to see how this new approach will impact other areas of my life.

The decision to stop buying new books has been a catalyst for change. It has taught me the value of living with intention. It has shown me that personal growth is not just about what we acquire, but also about how we appreciate what we already have. I am eager to see where this journey will take me.

I am starting to appreciate my belongings in a new way, and I am finding new ways to simplify my living space. The journey is just beginning, and I am excited to see what the future holds. I have learned that sometimes the best way to move forward is to stop, and to appreciate what we already have.