I maintain Village Finder, an open-source mapping project tracking over 78,000 Indian villages. It works by pulling daily raw updates straight from official public administration feeds.
That means my GitHub Actions CI environment hits Indian government web infrastructure every single day. Over time, you accumulate quite a few war stories.
If you're building systems that rely on open data pipelines, these five hard-learned lessons will save you days of baseline debugging.
1. The WAF That Lies to Python-Requests (HTTP 502)
One week, my daily cron job started throwing a wall of HTTP 502 Bad Gateway errors after its eighth automatic retry. Oddly, copying and pasting the exact same endpoint URLs into a desktop browser loaded the JSON data instantly.
Rate limits? IP ranges blocked by geographical firewalls? A sudden mid day server maintenance window? I spent hours drafting a complicated time of day query handler before stepping back to run the simplest sanity test:
# Same host machine, same target API gateway URL:
curl -A "Mozilla/5.0" https://api.data.gov.in/... ➡️ 200 OK
python3 -c "import requests; print(requests.get('...').status_code)" ➡️ 502 Bad Gateway
The Catch: The Web Application Firewall (WAF) protecting the data.gov.in gateway explicitly fingerprints incoming request headers. If it catches the default python-requests or urllib3 signature, it drops the connection and purposefully returns a misleading 502 instead of a classic 403 Forbidden.
The fix is a single, honest line of code:
session.headers["User-Agent"] = "my-civic-project/1.0 (+https://github.com/me/my-project)"
Takeaway: When an endpoint drops connection only when executed from automation code, isolate the
User-Agentimmediately. Beware of firewalls that mask blocks behind 502 server errors—they will send you chasing transient infrastructure ghosts that don't exist.
2. Treat Upstream Outages as a Contract, Not an Exception
Public data infrastructure has bad days. If your nightly cron job fires an angry email alert every single time an official site drops connection for an hour, your alerts turn into background noise and real codebase bugs go unnoticed.
To manage this cleanly, adopt the classic Unix EX_TEMPFAIL standard. When data fetchers hit a total upstream failure after exhaustive retries, the script catches the issue and explicitly exits with a status code of 75.
# GitHub Actions contract snippet
- name: Fetch Remote Assets
run: python scraper/pipeline.py
continue-on-error: true
id: fetch_step
- name: Evaluate Exit Code
run: |
if [ ${{ steps.fetch_step.outputs.exit_code }} -eq 75 ]; then
echo "Upstream is down. Skipping cleanly, maintaining previous snapshot."
exit 0
fi
By teaching the pipeline that a code 75 indicates a clean, non-breaking skip, the build stays green, a brief status note is logged, and yesterday's cached data snapshot is safely retained. Any other non-zero exit status code remains a true code failure and triggers an alert.
3. Handling Undocumented Frontends (The myScheme Blueprint)
India's central directory for welfare discovery, myScheme, doesn't offer a formal developer portal or public API keys. However, the modern web application is a client-rendered Next.js application that regularly hits a private backend query path: api.myscheme.gov.in/search/v6/schemes.
If you inspect the client networking requests, you can uncover the internal parameters:
-
The Guard: Passing the plain client application key alone triggers a
401 Unauthorized. The gateway explicitly cross-checks headers—you must provide exact, matchingOriginandRefererstrings pointing back to their landing page. -
The Multilingual Gift: Passing regional language ISO parameters (e.g.,
lang=te,lang=kn, orlang=ta) responds with beautifully localized scheme descriptions, perfect for regional user interfaces. - The Trap: Client access keys rotate dynamically with new production releases of the main frontend. Design your fetch engine with this rotation in mind from day one. When the key breaks, ensure the workflow surfaces a specific failure type so a maintainer can update the configuration token instantly without breaking the surrounding data structures.
4. Knowing When to Walk Away (And Finding Alternatives)
Data engineering isn't just about building functional scripts; it's about documenting exactly why certain avenues are structural dead ends so future maintainers don't retread old steps.
During development, several highly requested public portals proved completely impossible to automate reliably:
-
India-WRIS / CGWB (Groundwater Data): Suffered weeks of deep network drops, prolonged
NXDOMAINrouting issues, and permanent connection timeouts. - Urvarak (Fertilizer Stocks): Hard-resets TCP connections instantly when queried by headless HTTP clients. It is structurally isolated for portal usage only.
- Soil Health Card Directory: Implements aggressive client-side validation barriers and relies on undocumented, dynamic internal state transformations.
The Solution: Walk away and look for open alternatives. Instead of scraping fragile state portals, we integrated ISRIC SoilGrids’ open geospatial point API for deep soil profiles, transitioned to ISRO’s Bhuvan WMS vector layers for groundwater prospects, and hardcoded regional baseline pricing sheets directly into configuration files.
5. Expect Zero Cross-System Naming Conventions
Do not assume different departments within the same government speak the exact same dialect. The market monitoring network (Agmarknet) might report trades under the district name "Chittor" or "Dr.B.R.A.Konaseema", while the administrative census directory logs those same regions as "Chittoor" and "Dr. B.R. Ambedkar Konaseema".
Because shared national metadata codes are rarely integrated universally across distinct microservices, you must build a translation layer.
A lightweight string normalizer paired with a fallback token-matching threshold resolves these regional spelling variances seamlessly:
def normalize_name(text):
return "".join(c for c in text.lower() if c.isalnum())
# Match token tokens: ["dr", "br", "ambedkar", "konaseema"]
Budget for structural string reconciliation inside your normalization layers right from the start.
🚀 The End Result
Wrestling through these API barriers means we can provide a lightning-fast, static utility where a rural user can click a single map location to pull up local market pricing, applicable welfare benefits, and detailed topsoil compositions—all instantly translated into their local script.
The complete automated architecture, testing blueprints, and deep decision records are open to everyone:
📦 Explore the Pipeline on GitHub: mchittineni/india-village-finder
Top comments (0)