DEV Community: Padmaraj Nidagundi

What Happens When 15+ Public APIs Keep Breaking? I Built a Fault-Tolerant Data Platform for India

Padmaraj Nidagundi — Sat, 22 Aug 2026 08:28:27 +0000

I started IndiaRealTime with a fairly simple idea:

What if people could check useful, frequently changing information about India without opening five or ten different websites?

Mandi prices. Fuel prices. Gold and silver rates. AQI. Weather alerts. Earthquakes. Bank holidays. Currency conversion. Cricket scores. And a growing list of other data.

Most of this information already exists.

The problem isn't necessarily finding the data.

The problem is making all of those different sources work together reliably.

That turned out to be a much more interesting engineering problem than I expected.

Today, IndiaRealTime is built around roughly 20 independent WordPress plugins that consume around 15 public data sources and feed a single site.

The most important design decision wasn't how to fetch the data.

It was deciding what the website should do when the data source doesn't work.

Live data is easy when every API is healthy.

The interesting engineering starts when it isn't.

The problem: public data doesn't mean reliable data

India has a huge amount of publicly available information.

Agricultural commodity prices are available through public systems.

Air-quality information is published by government sources.

Fuel prices are published by oil marketing companies.

Weather and seismic information comes from different feeds.

Financial information comes from exchanges and other providers.

But these sources don't behave like one clean API.

They have different:

response formats
update schedules
authentication requirements
rate limits
naming conventions
failure modes
geographic coverage
data freshness
maintenance schedules

One source might return JSON.

Another might change a field name.

Another might temporarily return an empty response.

Another might be unavailable for several hours.

Another might return HTTP 200 while the actual data is unusable.

So the problem became:

How do you build a website that depends on many external data sources without allowing one broken source to break the entire site?

That question shaped most of IndiaRealTime's architecture.

Why I chose one plugin per data source

Instead of building one giant ingestion pipeline, I chose a deliberately modular architecture.

Each major data source gets its own WordPress plugin.

Conceptually:

Public data sources
        │
        ├── Source A
        ├── Source B
        ├── Source C
        ├── Source D
        └── ...
             │
             ▼
      Independent plugins
             │
      fetch → validate
             │
          normalize
             │
             ▼
        local cache
             │
             ▼
       WordPress theme
             │
             ▼
       IndiaRealTime

Each plugin owns its own:

fetching logic
validation
normalization
schedule
cache
failure handling

That means a failure in one source doesn't have to become a failure for the entire site.

If a commodity API changes its response format, I want one component to have a problem.

I don't want the weather pages, fuel pages, AQI pages and everything else to become collateral damage.

This is essentially fault isolation at the application architecture level.

But then I discovered the real problem

Suppose a visitor opens a fuel-price page.

The site has a cached value from the previous successful fetch.

Then the upstream source goes down.

What should happen?

Option 1: Show an error

Unable to fetch current fuel price.

Technically honest.

Terrible user experience.

Option 2: Show nothing

Even worse.

Option 3: Keep showing the last successful value

Now the user gets useful information, but it might be stale.

For this project, I chose option 3.

That led to one of the most important patterns in the system:

stale-while-revalidate style fallback.

When the API fails, keep the last good value

The basic flow is:

Fetch new data
     │
     ▼
Is response valid?
   /       \
 yes       no
  │         │
  ▼         ▼
Normalize   Keep previous
  │         cached value
  ▼
Update cache

The important part is that a temporary upstream failure does not immediately become a broken page.

If yesterday's fuel price was successfully retrieved and today's request fails, the system can continue serving the last known good value rather than replacing it with an error.

There is also a short-lived failure flag so the application doesn't repeatedly hammer an upstream source that is already failing.

This sounds simple.

But it changes the reliability model considerably.

The site doesn't have to choose between:

"live but broken"

and

"working but empty."

It can instead say:

"Here is the last known good data while we try again."

Of course, that means freshness becomes part of the data model.

A cached value isn't magically current.

So the system needs to be honest about what it knows and when it last successfully obtained it.

That distinction is important for any application that claims to provide "real-time" information.

Real-time is really a freshness problem

I think the phrase "real-time data" can be misleading.

Fetching something every few minutes doesn't automatically make it real-time.

What matters is:

When was the source updated?
When did we retrieve it?
Did the retrieval succeed?
Is the value valid?
When should we try again?
What do we display if the source fails?

Once you have multiple external sources, "real-time" becomes less about speed and more about freshness management.

For example, different datasets don't need identical schedules.

Some information can be refreshed several times a day.

Some information changes daily.

Some feeds need much more frequent updates.

So each plugin has its own schedule rather than forcing every source into one global polling system.

Why the API health monitor exists

Once you have one plugin, checking whether it works isn't particularly difficult.

With around 20 independent plugins, manually checking logs becomes annoying.

So IndiaRealTime has a separate API health-monitoring component.

Conceptually:

Plugin A ──┐
Plugin B ──┤
Plugin C ──┤
Plugin D ──┤──> API Health Monitor
Plugin E ──┤
Plugin F ──┘

The goal is simple:

One place to see which external integrations are healthy and which ones need attention.

This is one of those engineering decisions that doesn't make the homepage look more impressive.

But it makes operating the system much easier.

Another challenge: PHP and JavaScript don't always agree

One of the more unusual problems appeared in author attribution.

The backend computes a hash in PHP.

The frontend also needs to compute the corresponding value in JavaScript.

That sounds trivial until integer behavior becomes relevant.

PHP and JavaScript don't handle integer arithmetic in exactly the same way.

In particular, reproducing 32-bit wraparound required explicit handling on the JavaScript side.

The solution uses Math.imul() so the JavaScript calculation can reproduce the intended 32-bit multiplication behavior.

Without that, the backend and frontend can silently calculate different hashes.

There isn't necessarily an obvious error on the page.

The problem only appears when you compare the two results.

This was a good reminder that cross-language compatibility isn't only about syntax or APIs.

Sometimes it is about arithmetic semantics.

Location routing was another interesting problem

IndiaRealTime has location-based URLs.

The site needs to represent things like:

/state/city/category/

rather than maintaining a completely separate routing implementation for every state and city.

The application parses location information through shared WordPress query variables such as:

ir_state
ir_sub
ir_sub2

That lets the same architecture handle many different locations.

But flexible routing creates its own problems.

For example:

Is this segment a city?
Is it a category?
Is there a trailing slash?
Does this URL represent a national page or a local page?
What happens when a location name overlaps with another route?

Those are not glamorous problems.

They're the kind of problems that show up after the architecture looks finished.

So routing edge cases need dedicated tests rather than relying on manually opening a few URLs in a browser.

Accessibility isn't just a CSS detail

Another thing I wanted to take seriously was color.

A data website naturally uses semantic colors.

Green might mean good.

Yellow might mean caution.

Red might mean dangerous.

AQI makes this especially important.

If a page is communicating air-quality severity, color isn't merely decoration.

It carries information.

So the semantic colors are checked for WCAG contrast and considered against color-vision deficiencies.

The AQI bands also follow the relevant CPCB scale rather than using an arbitrary visual gradient.

That creates a useful engineering principle:

If color communicates data, color is part of the data presentation layer.

It deserves testing just like an API response does.

Structured data is part of the architecture too

IndiaRealTime isn't only rendering HTML for humans.

The project also generates machine-readable structured data through Schema.org JSON-LD.

That gives search engines and other systems a more explicit representation of what a page contains.

The architecture therefore looks roughly like:

             External sources
                    │
                    ▼
             Data fetchers
                    │
                    ▼
              Validation
                    │
                    ▼
               Normalize
                    │
                    ▼
              Cache/store
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
     HTML pages          Schema.org JSON-LD
          │
          ▼
      Human users

And when fresh data is successfully updated, the system can use IndexNow to signal that updated content is available for crawling.

Again, the idea isn't to make search engines the architecture.

The goal is to make fresh data discoverable without waiting indefinitely for a crawler to happen to revisit the page.

Why WordPress?

This is probably the question developers will ask.

Why build a data aggregation system with WordPress?

The answer is that WordPress wasn't being used as a page builder.

The project uses:

a custom WordPress theme
PHP templates
vanilla JavaScript/CSS
MySQL
custom plugins
WP-Cron
WordPress's existing routing and content infrastructure

There is no page-builder dependency at the center of the architecture.

WordPress provides the application framework, while the custom plugins provide the data-ingestion layer.

That separation works reasonably well for this project.

It also gives me a useful deployment and content-management environment without having to build an entire CMS from scratch.

But ~20 plugins isn't free

The modular architecture has a clear tradeoff.

Fault isolation is good.

Maintainability is harder.

With one plugin per source, shared conventions have to remain consistent.

For example:

transient naming
cache durations
failure flags
error handling
scheduling
validation
data normalization

A monolith gives you more opportunities to centralize those conventions.

A collection of independent plugins gives you more isolation.

So the architecture deliberately chooses:

more components + stronger isolation

over:

one central ingestion system + tighter coupling.

I'm still interested in whether that is the correct long-term decision.

What IndiaRealTime currently covers

The project currently brings together data and utilities across several categories:

Prices

Mandi/agricultural commodity prices
Fuel prices
LPG
Precious metals
Currency conversion
Mutual fund NAVs
FD rates
Toll rates

Environment

Air quality
Weather alerts
Earthquake alerts

Civic and time

Bank holidays
Pincode lookups
Timezone conversion

Culture

Panchang
Muhurat
Vrat calendar
Rashifal

Sport

Live cricket scores

The exact coverage depends on what the underlying sources make available.

For example, fuel prices can be represented at city level rather than just showing a national average.

The part I like most: the site is actually a test of the architecture

IndiaRealTime started as a website.

But the longer I worked on it, the more it became an experiment in a different question:

How do you build a useful application on top of external systems that you don't control?

That question applies far beyond India.

It applies to:

payment providers
shipping APIs
weather services
financial APIs
government data
mapping services
social APIs
SaaS integrations
public datasets

The upstream provider can change.

The network can fail.

The response can be malformed.

The service can be rate-limited.

The data can become stale.

Your application still has to behave sensibly.

That is why the stale-cache mechanism ended up being more important than some of the visible features.

What I would change if I started again

There are several things I'd approach differently.

1. Historical data earlier

The project currently has historical tracking for some datasets such as fuel and AQI.

Extending that to mandi and commodity prices is one of the next useful steps.

A current value tells you what is happening.

A history tells you why it matters.

2. More automated contract testing

External APIs change.

Ideally, each source should have stronger automated checks that detect:

missing fields
changed field types
unexpected empty responses
invalid values
schema changes

before those changes reach production pages.

3. More complete accessibility testing

Color contrast is only one part of accessibility.

The next step is broader testing for:

keyboard navigation
focus states
semantic HTML
screen readers
form controls
dynamic content

4. More granular geographic coverage

The architecture can support more city-level data as the underlying sources permit it.

That is especially useful for prices that vary geographically.

What I learned building it

The biggest lesson wasn't about WordPress.

It wasn't about APIs.

It wasn't even about caching.

It was this:

Reliability is often about deciding what to do when something goes wrong.

Anyone can build the happy path:

API works
   ↓
Fetch data
   ↓
Display data

Production systems need the other diagram:

API works
   ↓
Fetch
   ↓
Validate
   ↓
Cache
   ↓
Display


API fails
   ↓
Detect failure
   ↓
Keep last known good value
   ↓
Record failure
   ↓
Retry later
   ↓
Continue serving

That second path is where a lot of the engineering lives.

I'd like other developers to challenge this architecture

IndiaRealTime is a working project, but I don't consider the architecture finished.

I'm particularly interested in how other developers would approach:

stale data versus unavailable data
API failure detection
cache invalidation
per-source scheduling
contract testing for unstable APIs
multi-source data normalization
plugin-per-source architecture
accessibility for data-heavy interfaces
cross-language hashing
large-scale geographic routing

If you've built something similar, I'd genuinely like to compare approaches.

The project overview is on GitHub:

GitHub: https://github.com/padmarajnidagundi/indiarealtime-com

And the live application is:

IndiaRealTime: https://www.indiarealtime.com

I'm especially interested in hearing from developers who have had the same experience:

the API works perfectly in development, and then production teaches you otherwise.

Testing the New HTTP QUERY Method (RFC 10008): A Guide for Test Engineers and Developers

Padmaraj Nidagundi — Wed, 08 Jul 2026 07:24:12 +0000

HTTP has a new method. RFC 10008, published by the IETF as a Proposed Standard in June 2026, defines QUERY — a method that's safe and idempotent like GET, but carries a request body like POST.

If you've ever encoded a huge filter object into a query string because GET couldn't carry a body, or used POST for a read-only search and lost cacheability in the process, QUERY closes that gap.

This post is a practical breakdown of how to test it — manually, with Postman, with automation, and with AI in the loop.

TL;DR on the Spec

Safe + idempotent — same guarantees as GET. Clients and proxies can retry it freely.
Has a body — same shape as POST. No more 8000-octet URL limits (RFC 9110's recommended cap) for complex queries.
Cacheable, but the cache key includes the body, not just the URL. This is the part most infra hasn't caught up to.
Not CORS-safelisted — browser calls to QUERY always trigger a preflight OPTIONS, same as PUT/DELETE.
New Accept-Query header — lets a resource advertise supported query formats, e.g.:

Accept-Query: "application/jsonpath", application/sql;charset="UTF-8"

Now, testing.

1. Manual Testing

Before any tooling, walk through these by hand:

Method support: Does the endpoint even accept QUERY? Expect 405 Method Not Allowed on a lot of stacks right now — frameworks, gateways, and WAFs are still catching up.
Idempotency: Fire the identical request 5-10 times manually. No side effects, no state changes, same result every time.
Body edge cases: empty body, malformed JSON, oversized payload, wrong Content-Type. Confirm you get 400/415 where expected, not a silent 200.
Discovery: Send OPTIONS or HEAD and check whether Allow: QUERY and Accept-Query show up.
Caching: Repeat the same QUERY through your CDN/proxy — check for cache hits. Then vary the body slightly and confirm the cache key changes accordingly (mismatches here = cache poisoning risk).
CORS preflight: From a browser console, confirm an OPTIONS preflight actually fires before the QUERY request.

2. API Testing with Postman

Postman doesn't have a built-in QUERY button yet, but it's easy to add:

Method dropdown → "Custom method" → type QUERY.
Set the body (raw JSON, or whatever your API expects) exactly like you would for POST.
Explicitly set Content-Type — QUERY has no default assumption.
Build a small test collection:

javascript// pm.test examples for a QUERY request

Use Collection Runner (or Newman in CI) to fire the same QUERY N times in a loop and assert consistent responses — a fast, scriptable idempotency check.

If your Postman version won't let you type a custom verb in the dropdown, switch to raw request mode or override the method via a pre-request script.

3. Automation Testing

Most HTTP libraries don't have a .query() shortcut yet. Treat it as a custom verb with a body, same pattern you'd use for any non-standard method.

Java (REST-assured):

Python (requests):

JavaScript (fetch, Node 18+):

Scenarios worth automating and running in CI:

Idempotency regression — same QUERY 5x, assert byte-identical (or semantically identical) responses.
Cache key correctness — two near-identical bodies shouldn't collide in cache; two identical bodies should hit cache.
Contract tests — response schema stays stable regardless of query body shape (nested vs. flat filters).
Negative tests — oversized bodies, bad Accept-Query values, missing Content-Type.
Infra checks — confirm your WAF, API gateway, and load balancer actually pass QUERY through instead of blocking or silently rewriting it to POST.

4. Where AI Fits In

This is the part that saves the most time day-to-day:

Edge case generation: paste the RFC summary + your endpoint spec into an LLM and ask for edge cases you haven't thought of — malformed Accept-Query values, boundary body sizes, cache-key collision attempts.
Test script drafting: ask AI to draft Postman pm.test blocks or REST-assured/pytest scaffolding for idempotency and caching checks, then review and tighten them yourself.
Response diffing: have AI compare payloads across repeated QUERY calls and flag subtle non-idempotent behavior — a stray timestamp, reordered array, inconsistent pagination cursor.
Security review assist: since WAF/CSRF rules are often written before QUERY existed, AI can help draft candidate test payloads (oversized bodies, unusual Accept-Query values) for your authorized security testing — always within your org's scope and rules of engagement.
Doc gap-checking: feed your API docs to AI and ask it to flag places where Accept-Query support isn't documented or where QUERY behavior isn't clearly distinguished from GET/POST.

AI doesn't replace reading the spec — but it's genuinely good at generating breadth of coverage for a method this new, where nobody has a mature checklist yet.

Quick Reference

QUERY support is inconsistent across frameworks and infra right now — that's exactly the window where testing it thoroughly matters most, before assumptions get baked in.

*QUERY is still fresh and newborn *👶🏻 - expect inconsistent support across frameworks, proxies, and tools for a while. That's exactly why testing it thoroughly now, before it's baked into every stack, is worth the effort.

Have you run QUERY against your own stack yet? #testing #http #postman #automation #ai Curious what broke first — WAF, cache, or CORS. Drop it in the comments. Or reach me on email padmaraj (dot) nidagundi (at) gmail.com

How test automation engineers survive and thrive in the AI 2025 Era.

Padmaraj Nidagundi — Mon, 17 Feb 2025 19:02:28 +0000

In 2025, test automation engineers are adapting and thriving in the AI era by leveraging new technologies and methodologies. Here are some key strategies and trends that are helping them succeed:

Upskilling and Continuous Learning:

AI and Machine Learning: Engineers are learning AI and machine learning concepts to understand how to integrate these technologies into their testing frameworks.
New Tools and Technologies: Staying updated with the latest testing tools and frameworks that incorporate AI, such as AI-driven test automation platforms.

Example: An engineer might take online courses on platforms like Coursera or Udacity to learn about AI and machine learning. They could also attend industry conferences like the AI Summit or the Selenium Conference to stay updated on the latest trends and tools

AI-Driven Test Automation:

Intelligent Test Generation: Using AI to generate test cases based on historical data and patterns, reducing the manual effort required to create test scripts.
Predictive Analytics: Leveraging AI to predict potential defects and areas of the application that are more likely to fail, allowing for more targeted testing.

Example: Using a tool like Applitools, which employs AI to visually validate UI components and automatically detect differences, even those that might be missed by traditional pixel-based comparisons.

Enhanced Test Coverage:

Automated Exploratory Testing: AI can help in exploring the application in ways that manual testers might not think of, increasing the coverage and finding edge cases.
Continuous Testing: Integrating AI-driven testing into CI/CD pipelines to ensure continuous and automated testing throughout the development lifecycle.

Example: Implementing a tool like Mabl, which uses AI to automatically explore an application and generate test cases based on user behavior, ensuring that even edge cases are covered.

Improved Test Maintenance:

Self-Healing Scripts: AI can help in identifying and fixing broken test scripts automatically, reducing the maintenance effort.
Automated Test Data Management: Using AI to generate and manage test data, ensuring that tests are run with relevant and valid data.

Example: Utilizing Testim, an AI-driven test automation platform that offers self-healing test scripts. If a UI element changes, Testim can automatically update the test script to reflect the new element, reducing maintenance effort.

Collaboration with AI Tools:

AI-Assisted Debugging: Tools that use AI to help identify the root cause of failures more quickly and accurately.
Natural Language Processing (NLP): Using NLP to convert human language requirements into automated test scripts, making the testing process more intuitive.

Example: Using a tool like Functionize, which employs NLP to convert plain English test cases into executable test scripts. This makes it easier for non-technical stakeholders to contribute to the testing process.

Focus on Quality Assurance:

Shift-Left Testing: Incorporating testing earlier in the development process to catch defects sooner, reducing the cost and effort of fixing them later.
User Experience Testing: Using AI to simulate user behavior and test the application from a user's perspective, ensuring a better user experience.

Example: Integrating AI-driven testing tools like Tricentis Tosca into the development process from the beginning. This tool can help in creating and maintaining test cases early in the development cycle, ensuring that defects are caught sooner.

Adoption of DevOps Practices:

Integration with CI/CD: Ensuring that AI-driven testing is seamlessly integrated into the continuous integration and continuous deployment pipelines.
Collaboration with Developers: Working closely with developers to ensure that testing is an integral part of the development process, not an afterthought.

Example: Using Jenkins with AI-driven testing plugins to integrate automated testing into the CI/CD pipeline. This ensures that every code change is automatically tested, and any issues are flagged immediately.

Ethical and Security Considerations:

AI Ethics: Ensuring that AI-driven testing is ethical and unbiased, avoiding any potential issues related to data privacy and security.
Security Testing: Using AI to identify vulnerabilities and potential security threats in the application.

Example: Employing AI tools like Veracode to perform static and dynamic application security testing (SAST and DAST). These tools can identify vulnerabilities in the code and provide insights into potential security threats.

By embracing these strategies and leveraging AI, test automation engineers in 2025 are not only surviving but thriving, delivering higher quality software more efficiently and effectively.

The Next CTO

Padmaraj Nidagundi — Fri, 07 Oct 2022 07:16:41 +0000

You want to become a CTO or enhance your skills as a Visionary Chief Technology Officer, but you don't know where to start.

You want to be able to lead your team to success and be at the forefront of the latest technologies.

THE NEXT CTO book will help you understand what it takes to be a great CTO, from building an engineering culture to choosing the right technologies.

Who is book for

Any wanting to become a CTO. Aspiring Software engineers, startup owners, VP engineers, tech leads, testers, product owners, analysts, business consultants, engineers, and business owners.

If you have ever thought to yourself, “I want to become a CTO or I want to enhance my skills as a Visionary Chief Technology Officer” then this is best book for you. In it, the author comes up with some important technical and leadership themes: CTO in Agile era, Lean mindset for CTO, R&D lab, CTO opportunities in the year 2030 and so on.

Chapter 1: Steps to get into the CTO role, connection between technology and the CTO, how CTO get into innovation and become visionary CTO.

Chapter 2: CTO Building and debugging engineering culture, factors to consider when choosing technologies and different CTO personalities.

Chapter 3: Professional skills, qualities and competence for CTO, Types of CTO in the industry, Budgets management, technology management and people management, Agile and Lean mindset for creating lean organization, product, and technology. CTO and POC, Prototype, MVP, Usability.

Chapter 4: Coding skills for CTO, code, low code and no code skills for CTO, technology trends to build strategy, avoid over engineering, lead research, technical and business due diligence, product development (R&D) and CTO role in technical and business due diligence.

Chapter 5: Learning from mistakes and failure, how CTO avoid and manage failures, startup vs corporate journey and learning lessons, CTO one step ahead and employee journey.

Chapter 6: CTO opportunities in the year 2030 with Quantum computers, Data science, AI, ML, Smart cities, EV, IOT, OPENAI GPT 4, NEO, DALL·E and M2M-100

BONUS: A New CTO Three Month Toolkit, SaaS CTOs Check List, Mind map for CTO and Self Q&A

Search on amazon "THE NEXT CTO"
Pre-order book it via Amazon

How chief technology officer CTO avoid over engineering

CTO and Employee Journey

It's Time to Take a Stand - #COVID19 Ergonomics Design Challenge

Padmaraj Nidagundi — Tue, 14 Apr 2020 08:35:30 +0000

Due to #COVID19 maximum people working from home with laptop and computers. Today I challenge everyone you to create your own standing desk in the home using hashtags #MyStandingDesk #Jugada #ReverseEngineering

Creating own Standing Desk in the home using existing home tables or furniture is not an impossible task. Just look around in your home, and you get many small tables and furniture items that help you to set up your own Standing Desk.

COVID19 Ergonomics Design Challenge ( Desi Jugada )

Benefits of Using a Standing Desk

Standing Lowers Your Risk Of Weight Gain And Obesity
Using A Standing Desk May Lower Blood Sugar Levels
Standing May Lower Your Risk Of Heart Disease
Standing Desks Appear To Reduce Back Pain
Standing Desks Help Improve Mood And Energy Levels
Standing Desks May Even Boost Productivity
Standing More May Help You Live Longer
Keep Your Back Correctly Straight While Raising Your Neck To Prevent Neck Pain
More Calories Burned
Less Back Pain
More Productive
Improve Mood And Gain Energy

Work Smarter. Feel Better. https://www.linkedin.com/pulse/create-your-own-standing-desk-in-home-challenge-padmaraj-nidagundi/