DEV Community: Manideep Chittineni

GitOps for Geospatial Data: Building a Self-Healing, Zero-Cost Data Pipeline with GitHub Actions

Manideep Chittineni — Mon, 13 Jul 2026 15:21:37 +0000

Most data engineering and geospatial projects follow a predictable infrastructure blueprint: an ingestion cron job, an enterprise database (like PostGIS), a hosted API layer, a dynamic tile server, and a frontend client. Before you write a single line of business logic, your cloud architectural diagram already carries a fixed monthly overhead.

For Village Finder an open-source interactive mapping application tracking over 50,000 Indian villages, live market rates, and land records I have decided to reject the traditional stack.

Instead, I applied core DevOps, GitOps, and DataOps principles to build an entirely serverless, self-updating data platform. The operational infrastructure bill? Exactly $0.

Here is the architectural blueprint of how I shifted left on geospatial data infrastructure.

🏗️ 1. Infrastructure as Code (IaC) is Good. Data as Code is Better.

The foundational principle of GitOps is that Git is the single source of truth. If your infrastructure states can live in a declarative YAML file, your application data can too.

In Village Finder, the core dataset of record (the structural relationship between Districts -> Mandals -> Villages) is treated exactly like code.

[Upstream API] ➡️ [Daily Ingestion Run] ➡️ [Automated Code Generation] ➡️ [PR Matrix]

When changes occur upstream in the Government of India’s Local Government Directory, a daily scheduled GitHub Actions pipeline handles the processing. But instead of updating an active runtime database, it generates normalized, flat-file static JSON and CSV tables and writes them directly back into a version-controlled Git Pull Request.

The Core Benefits:

Auditable Log Streams: The repository’s commit history doubles as an exact, immutable audit trail of physical data mutations over time (e.g., catching exactly when local authorities reclassified or shifted thousands of village codes overnight).
Zero-Downtime rollbacks: If an upstream data feed introduces corrupt schema transformations, rolling back production to a prior valid data state is as simple as a git revert.

🔀 2. Separating State: PR Pipelines vs. Historyless Data Branches

If you commit highly volatile, real-time data metrics (like daily market prices or weekly welfare shifts) directly to your master branch, your primary git tree history will explode in size within months.

To solve this, I implemented a decoupled lifecycle strategy separating Structural Data from Volatile Metrics:

Structural Data Pipeline (The PR Path)

Changes to core geographic boundaries trigger a standard code-review loop. An automated workflow creates a Pull Request accompanied by a dynamically generated markdown changelog detailing exactly which entries were modified. Consecutive nightly runs update the same open PR branch, eliminating bot spam.

Volatile Metrics Pipeline (The Flat Data Branch CDN)

Daily commodity prices change constantly. For these, our automated runners bypass the Pull Request loop entirely and push directly to standalone, isolated git tracking branches (e.g., data/mandi-prices) using an history-less force-push contract:

# Compile volatile snapshots into a completely history-less single-commit branch
work="$(mktemp -d)" && cp out/*.json "$work" && cd "$work"
git init -q -b data/mandi-prices
git add . && git commit -q -m "data: snapshot $(date -u +%FT%RZ)"
git push --force "https://x-access-token:${TOKEN}@github.com/${REPO}.git" data/mandi-prices

GitHub’s global static asset delivery network (raw.githubusercontent.com) serves public files with open CORS permissions out-of-the-box. The browser frontend handles fetches directly from the head of these flat data branches at runtime. I get a high-availability, zero-latency data delivery network with no active server management.

🧪 3. Shifting Left on Data Validation (DataOps Testing)

In a traditional DevOps loop, Continuous Integration (CI) validates code compilation and unit tests. In a DataOps loop, CI must validate data integrity.

A single missing sub-district mapping or a broken polygon geometry will crash a client-side map canvas. To prevent this, our pipeline treats raw data updates with the same rigor as code changes. Every automated Pull Request must pass an exhaustive suite of over 90 validation rules orchestrated via pytest:

def test_referential_integrity():
    # Assert that every single village maps cleanly to a valid structural parent code
    assert missing_parents == 0

def test_geojson_geometries():
    # Enforce polygon boundary validity before deploying static vector tile coordinates
    assert geometries_are_valid == True

If the upstream government registry outputs an incomplete schema variant, the automated testing matrix fails loud, blocks the integration path, and prevents the deployment runner from shipping broken maps into production.

☔ 4. Designing for Fragile Upstreams: The `EX_TEMPFAIL` Contract

Public data networks encounter high rates of transient downtime, network timeouts, and sudden firewall shifts. If your continuous deployment platform fires a critical failure alarm every single time a remote service faces a routine network drop, your team experiences alert fatigue. You begin to ignore notifications, and true application bugs slip past unnoticed.

To build an infrastructure that tolerates flaky external gateways, I adopted the classic Unix EX_TEMPFAIL status contract.

# Handle flaky connections inside ingestion layers
except RequestException as e:
    logger.error(f"Transient upstream network drop: {e}")
    sys.exit(75) # EX_TEMPFAIL

When our data acquisition routines encounter a terminal connection timeout after multiple retries, they explicitly exit with a status code of 75.

Our orchestration runner identifies code 75 not as a breaking error, but as a clean skip request. The workflow execution concludes gracefully, retains the previous day's working data cache, logs a brief summary update, and keeps the overall build pipeline completely green. Alarms only sound if something is truly broken inside our internal code logic.

📐 5. Progressive Build Aggregations (Branch Overlays)

To keep the platform highly decoupled, I use a modular architecture where separate workflows generate independent data layers (like complex cadastral survey boundaries or village point catalogs) entirely in parallel.

At deployment time, instead of running heavy multi-hour calculation steps sequentially, a highly optimized compilation worker overlays these pre-rendered data blocks straight into the target production directory using shallow checkouts:

- name: Overlay Pre-Rendered Vector Boundaries
  run: |
    git fetch -q --depth 1 origin "data/boundary-tiles"
    git checkout -q FETCH_HEAD -- ./tiles/boundaries.pmtiles

If a developer forks the project locally to build a minor feature patch, absent data branches skip missing files quietly. This design ensures the entire local system remains lightweight, instantly reproducible, and highly modular for open-source contributors.

🎯 The Big Takeaway

DevOps is frequently treated as an enterprise enterprise cost center—a tool setup meant for managing Kubernetes nodes or provisioning massive cloud databases.

Village Finder proves that by applying core DevOps mental models—immutability, continuous data validation gates, decoupled architecture, and treating external inputs as a version-controlled pipeline—you can deliver fast public software with exceptional durability and zero maintenance costs.

🌐 Live App: mchittineni.github.io/india-village-finder
📦 Explore the Actions & Code: github.com/mchittineni/india-village-finder

🚀 India Village Finder Introduces Farmer Schemes & Farm-Input Insights

Manideep Chittineni — Mon, 13 Jul 2026 15:14:11 +0000

Village Finder is an open-source, serverless, and multilingual map tracking over 50,000 villages across Andhra Pradesh, Telangana, Karnataka, and Tamil Nadu. By transforming the Government of India's raw Local Government Directory (LGD) data into fluid, searchable maps, it connects rural locations to live mandi prices, land parcels, weather, and soil data all maintained by fully automated, validated CI/CD pipelines.

Here is what’s new in our latest release.

🔥 New Features & Capabilities

🏛️ Integrated Farmer Schemes Panel

Cultivators can now discover exactly which Central and State welfare benefits they qualify for (focused on the "Agriculture, Rural & Environment" sectors).

True Localization: Scheme names are completely translated across all six UI scripts (English, Telugu, Kannada, Tamil, Hindi, and Urdu).
Seamless Action: The list is fully searchable, grouped cleanly by Central vs. State jurisdiction, and every single entry features a direct link to its formal "how-to-apply" portal on the national myScheme platform.
Under the Hood: A new automated worker (scraper/fetch_farmer_schemes.py orchestrated by update-farmer-schemes.yml) snapshots the myScheme discovery API weekly, pushing compressed tracking arrays to a dedicated data/farmer-schemes branch. The frontend client (web/schemes.js) queries this lightweight data branch directly at runtime with zero server overhead.

🧪 Static Farm-Inputs Reference Directory

To protect farmers from localized retail overcharging, the utility panel now surfaces official government-notified commodity benchmarks:

Displays statutory Urea Maximum Retail Prices (MRP) alongside NBS-subsidized DAP reference baselines.
Provides verified web paths straight to the official live fertilizer-stock tracking networks (iFMS/urvarak) and national Soil Health Card registries.

🛠️ Engineering Note on Data Integrity: Live structural API integrations for real-time stock levels and individual soil cards were evaluated and intentionally skipped. The urvarak.nic.in gateway actively drops automated HTTP requests, and the primary SHC portal employs strict automation blocks. Building a fragile web-scraper that fails silently is inherently worse than providing clean, human-verifiable web links.

📦 Open Code, Open Data

Every asset fueling this platform is built to be reused. The core structural application logic is published under the flexible MIT License, while all associated geographic, pricing, and administrative datasets are distributed freely under open public frameworks (GODL-India / CC0 / CC BY).

Flat-file data splits (.csv and .json files mapped with clean hierarchical keys, PIN codes, and localized scripts) are readily available in the repository's Releases tab.

🌐 Live Map: mchittineni.github.io/india-village-finder
📦 Source Code & Data Drops: github.com/mchittineni/india-village-finder

Whether you want to test out the search filters, incorporate regional datasets into your own agritech models, or help us scale up coverage by adding a new state, contributions are highly encouraged. Check out our repository's contributing guide to get started!

Translating 50,000+ Indian village names into six scripts (when your data feed is only in English)

Manideep Chittineni — Mon, 13 Jul 2026 15:10:58 +0000

A mapping tool designed for rural India that only displays location data in English completely fails its target audience.

When building Village Finder which tracks over 50,000 villages across South India rendering the platform in regional languages wasn't a nice-to-have feature; it was a core functional requirement.

The engineering problem? The official Local Government Directory (LGD) open data feeds I consume do not contain native-script columns. They are entirely in English.

Here is how I built a highly reliable, multi-tiered localization pipeline to map English strings into Telugu, Kannada, Tamil, Hindi, and Urdu without bloating the frontend or sacrificing data integrity.

🏗️ The Multi-Tiered Fallback Architecture

To handle thousands of names without introducing massive runtime overhead, every single village name goes through a four-layer resolution matrix at compile time. The highest-trust layer wins:

[ Tier 1: Authoritative LGD Script ] (Future-proof hook)
               ⬇️ (Fallback)
[ Tier 2: Crowdsourced Seed Data ]   (OpenStreetMap name:* tags)
               ⬇️ (Fallback)
[ Tier 3: Offline Neural Inference ] (AI4Bharat IndicXlit Model)
               ⬇️ (Fallback)
[ Tier 4: Client-Side Rule Engine ]  (Morpheme-aware suffix parser)

1. Authoritative Government Scripts (Future-Proofing)

The LGD database schema includes provisions for a Village Name (In Local) field. The upstream open data API feed leaves this blank for our target regions today, but I explicitly place this at the front of the resolution chain. If the feed updates tomorrow, the system automatically self-corrects and upgrades to official spellings.

2. Crowdsourced Verification Seeds

Next, the builder harvests verified local script data straight from OpenStreetMap data assets (name:te, name:kn, name:ta tags) paired with a hand-curated localized exception map. It is crowd-sourced, but human-verified.

3. Offline Neural Transliteration

For the thousands of remaining villages lacking public records, I lean on AI4Bharat's IndicXlit a deep learning model trained specifically for English-to-Indic script transliteration.

Crucially, this model runs completely offline. A weekly batch job processes English strings and compiles them into highly compressed static JSON structures (names_translit.json). The client browser and core CI pipelines never load PyTorch, weights, or neural dependencies.

4. Morpheme-Aware Client Rule Engine

As a final safety net, a lightweight client-side script acts as a backup. Naive letter-by-letter transliteration engines frequently break on common regional geographic suffixes. Our engine is morpheme-aware, identifying distinct place-name components like -pur, -palli, -puram, or -halli and mapping them from their canonical phonetic spellings rather than raw character-by-character translation.

Note: The canonical English string is always preserved on hover and used for background fuzzy search engines the localized script enhances the experience without replacing indexing paths.

📈 If You Don't Metric It, It Will Rot

Transliteration accuracy can easily devolve into guesswork without baseline targets. To ensure changes to our engines didn't introduce hidden regressions, I built an internal evaluation harness (scraper/translit_eval.mjs).

The pipeline cross-checks the outputs of both the neural model and the fallback rule engine against "gold standard" government name arrays available in separate state datasets. The script calculates exact-match percentages alongside character-level accuracy bounds

Our GitHub Actions CI suite treats these evaluations as strict integration gates—if an algorithm optimization causes script accuracy to drop below a predefined floor, the build fails instantly.

🧹 Managing Upstream Churn & Orphans

Administrative data is surprisingly volatile. During routine updates, regional authorities frequently drop, merge, or renumber village indexes in massive waves often altering thousands of structural IDs in a single week.

When your data dependencies rely on sidecar mapping files (like our coordinates and neural text tables), upstream structural changes cause silent dataset rot. Orphaned records accumulate, and new villages display blank titles.

To handle this, our data pipelines enforce a strict pruning invariant:

Every daily build cross-references all sidecar dictionaries against the fresh list of valid master LGD village codes.
Any orphaned key is instantly purged from the committed JSON tracking maps.
Slower weekly workflows detect the resulting gaps and query the machine learning engine to backfill missing entries.

By encoding validation limits into the pipeline, structural changes cause the automation to self-heal instead of generating bloated, broken releases.

🌍 Localization is a Core System Discipline

Building a multi-script application means treating localization as an architectural pillar rather than an internationalization wrapper. Beyond translating basic buttons, it requires tailoring structural technical summaries to real world usage.

An agricultural soil analysis framework that reports a location's profile as a scientific Vertisols class provides zero utility to a rural farmer. True localization means embedding native agronomic terms directly into your core dictionary tables:

Script	Local Soil Translation	Common Meaning
Telugu	నల్లరేగడి	Black Cotton Soil
Hindi	काली रेगुर	Black Cotton Soil
Tamil	கரிசல்	Black Cotton Soil

The code, translation maps, and verification rules are completely open source:

🌐 Live Application: mchittineni.github.io/india-village-finder
📦 Codebase & Translation Pipelines: github.com/mchittineni/india-village-finder

What should a village map show a farmer? Prices, schemes, weather, and soil from sources that actually work

Manideep Chittineni — Mon, 13 Jul 2026 15:06:43 +0000

Village Finder started as an open-source engineering exercise: clean, structure, and index the raw administrative hierarchies of over 50,000 villages across Andhra Pradesh, Telangana, Karnataka, and Tamil Nadu.

But once a user drill downs through the maps, drops a pin, and finds their village, the real product question emerges: What value does this page actually deliver to their day-to-day life?

For the vast majority of rural India, the answer is tied directly to agriculture. A pin on a map shouldn't just be an administrative marker; it should be a gateway to actionable, local insights.

Here is what our village profiles look like today, and more importantly the reality of where this public data actually comes from.

🌾 Real-Time Mandi Market Quotes

Selecting a village instantly pulls up the current day's APMC market quotes for its parent district. Users can search and filter through specific agricultural commodities, distinct varieties, minimum-maximum price bounds, and overall modal rates (measured in ₹/quintal), neatly grouped by active local markets.

[Target Village Pin] ➡️ [District Lookup] ➡️ [Fuzzy Name Matcher] ➡️ [Agmarknet API Snapshot]

The Data Reality: This data is pulled from the Ministry of Agriculture's official Agmarknet feed via the data.gov.in gateway, snapshotted daily by our background workflows.

The biggest hurdle? Inter-departmental naming inconsistencies. The market network reports figures using its own localized spelling variations (e.g., "Chittor" or "Dr.B.R.A.Konaseema"), while the central census registry lists them as "Chittoor" and "Dr. B.R. Ambedkar Konaseema". A dedicated token-matching text parser bridges these discrepancies under the hood to ensure seamless client-side joins.

🏛️ Government Scheme Discovery, In the Local Language

The schemes component aggregates every Central and state-level agricultural welfare benefit a cultivator qualifies for ranging from universal income support (PM-KISAN) and crop insurance frameworks (PMFBY) to specialized credit lines (Kisan Credit Card) and unique regional state subsidies. Crucially, every entry links directly to its respective execution page on the national myScheme platform.

To ensure true accessibility, these benefits are fully localized across all six interface scripts (English, Telugu, Kannada, Tamil, Hindi, and Urdu) directly from the underlying JSON bundles.

⚠️ An Honest Caveat: Regional onboarding data quality across the national catalog varies wildly. For instance, the platform lists dozens of hyper local state-level initiatives for Tamil Nadu, but displays very few for neighboring Andhra Pradesh. The application's directory can only ever be as exhaustive as the national index feeding it.

🧪 Organic Soil Profiles with Candid Safeguards

When a user triggers the specialized "Soil & Fertilizer" panel, the interface returns a contextual data breakdown:

📊 Soil Type: Vertisols — Black Cotton Soil / Clay Loam
Topsoil Metrics: pH 6.9 (Neutral) · Organic Carbon: 1.9%
Nutrient Baseline (All-India N-P-K Guide): 4:2:1
💡 Alkaline conditions detected. Zinc solubility drops significantly in this range, elevating regional Zn-deficiency risks. Confirm exact levels with an official laboratory test before applying zinc sulphate amendments.

[Scientific Group: Vertisols] ➡️ [Vernacular Script Mapping] ➡️ ನಲ್ಲరేగడి / काली रेगुर / கரிசல்

The Engineering Choice:

Vernacular Translation: Farmers don't talk in terms of abstract scientific classifications like Vertisols or Alpisols. The application explicitly maps global soil taxonomies back to their traditional names across regional scripts (e.g., నల్లరేగడి, काली रेगुर, ಎರೆ ಮಣ್ಣು, or கரிசல்).
Responsible Limits: The data is pulled directly from the ISRIC SoilGrids 250 meter global computational model. Because it relies on predictive spatial modeling rather than real-time sampling, the user interface explicitly tags these metrics as model estimates, framing them alongside active links to acquire a verified, state-issued Soil Health Card. A software model is an educational baseline, not a definitive agronomic prescription pretending otherwise could actively harm crop yields.

Similarly, fertilizer pricing displays the official government notified statutory reference limits (such as standard Urea MRP and NBS-subsidized DAP baselines) so individuals can identify localized market overcharging. I provide direct structural links to official state portals for inventory lookups rather than scraping them, because fragile backend scrapers that break silently are inherently worse than a clean, direct web redirect.

🌤️ Weather, Groundwater, and Cadastral Layouts

To wrap up the regional assessment, each village view includes:

A keyless 7-day agricultural weather forecast via the open Open-Meteo engine.
Dynamic regional lithology overlays detailing groundwater prospects streamed directly from ISRO’s Bhuvan servers.
High-resolution cadastral vector map layers (CC0 datasets extracted from state GIS agencies) allowing users to select individual survey plot geometries and seamlessly bridge over to official databases to grab formal FMB sketches.

📝 The Product Takeaway

Building software for rural use cases requires a shift in engineering philosophy: Leverage official public endpoints wherever they are open, clearly acknowledge the boundaries of your datasets when they aren't, and never present an algorithmic model estimate as an absolute physical measurement.

The complete architecture, sanitized geo-datasets, and our comprehensive data source decision ledger are completely transparent and open to the community:

🌐 Live Platform: mchittineni.github.io/india-village-finder
📦 Codebase & Engineering Logs: github.com/mchittineni/india-village-finder

A self-updating open-data site with zero servers: GitHub Actions, Pages, and branches as a CDN

Manideep Chittineni — Mon, 13 Jul 2026 15:04:11 +0000

Village Finder serves interactive maps for over 50,000 Indian villages, refreshes its own geographic datasets daily, and streams live agricultural market prices.

It does all of this with no dedicated backend, no database server, and a monthly infrastructure bill of exactly ₹0.

Here is the exact architecture behind the project, including a highly effective pattern for zero-cost static architectures: weaponizing transient Git branches as a free, high-availability, CORS-enabled data CDN.

🛠️ The Serverless Toolkit

To achieve massive scale at absolute zero cost, the stack relies strictly on edge infrastructure:

GitHub Pages: Serves the frontend mapping applications (Vanilla JS + Leaflet.js, intentionally bypassing complex frontend frameworks for zero build-step latency).
GitHub Actions: Handles the entire continuous data integration workflow: daily upstream ingestion, polygon boundary simplification, release artifact compilation, and deployments.
GitHub Releases: Serves as the distribution network for versioned dataset downloads (.csv, .json, and .geojson splits).
Cloudflare R2: The single architecture exception. I stream gigabyte-scale cadastral land parcel maps that require absolute zero-egress pricing combined with reliable HTTP range-request support for geospatial vector formats.

🔀 The Core Insight: Two Classes of Data, Two Pipelines

When your application is completely database-less, every piece of incoming information has to live somewhere inside version control. Trying to treat volatile, real-time metrics the exact same way you treat structural administrative data will quickly ruin your git history.

I separated the data pipeline into two isolated lifecycles:

1. The Dataset of Record -> Reviewed Pull Requests

Core administrative village registries change infrequently but demand total structural integrity. When our scraper detects an upstream boundary change, the automation opens a structured Pull Request.

A comprehensive suite running over 90 validation checks via pytest validates referential tracking, and a custom changelog utility summarizes exactly which villages were shifted or reclassified. Consecutive nightly runs update the same PR branch in place, preventing bot spam.

When merged, the commit history acts as a transparent, version-controlled audit trail of government data changes.

2. Regenerable Artifacts -> Flat Data Branches

Daily agricultural market quotes (mandi prices), weekly welfare scheme updates, and monthly boundary vector tiles are highly volatile machine outputs. Having humans code-review a daily 10,000-row JSON diff is a waste of time.

Instead, the automation bypasses the PR loop entirely and writes directly to dedicated, isolated data branches using this isolated script routine:

# Compile and switch to an isolated, history-less data branch
work="$(mktemp -d)" 
cp out/*.json "$work" && cd "$work"
git init -q -b data/mandi-prices
git add . 
git commit -q -m "data: mandi prices $(date -u +%FT%RZ)"
git push --force "https://x-access-token:${TOKEN}@github.com/${REPO}.git" data/mandi-prices

By using git push --force with a history-less initial commit, the branch size never grows. I get infinite, automated data updates without inflating the base repository's .git footprint.

🌐 Branch Assets as a Live CDN

Here is where the magic happens: GitHub's asset proxy (raw.githubusercontent.com) serves any public repository file with an explicit Access-Control-Allow-Origin: * header.

Because of this, the frontend application doesn't need to redeploy when daily market figures shift. The client-side browser simply sends an asynchronous fetch directly to the head of the data branch:

// Fetch real-time market data straight from the branch asset
const targetStateUrl = `https://raw.githubusercontent.com/mchittineni/india-village-finder/refs/heads/data/mandi-prices/${stateName}.json`;
const response = await fetch(targetStateUrl);

For large spatial files that must match same-origin restrictions (like PMTiles requires for rendering vector graphics), I use a tiny GitHub Action step to overlay the static data branch files straight into the deployment workspace during the core production build window:

- name: Overlay Spatial Tile Artifacts
  run: git fetch -q --depth 1 origin "data/boundary-tiles" && git checkout -q FETCH_HEAD -- .

If the branch is completely absent on a developer's local fork, the step simply skips cleanly—ensuring the localized environment remains easy to run out-of-the-box.

☔ Designing to Survive Flaky Upstreams

Public sector APIs drop connections frequently. If a scheduled cron job fires an unexpected error alert every single time an upstream government gateway encounters an hour of maintenance downtime, it trains your engineering team to ignore CI alerts.

To resolve this, I mapped all connection timeouts and API drops to the standard Unix temporary failure contract: EX_TEMPFAIL (Status Code 75).

When our fetchers encounter a structural gateway issue after exhaustive retries, they exit with 75. The master GitHub Actions workflow catches this code, stops execution cleanly, keeps yesterday's working data snapshot intact, and writes a brief summary notice. The build stays green, preventing alert fatigue, while true codebase failures continue to ring alarms loudly.

💡 Hard-Learned Takeaways

If you are planning to leverage GitHub as an active data delivery network, watch out for these execution edge-cases:

Pathspecs Aren't Shell Globs: Inside Git operations, path formatting rules differ from traditional bash syntax. Writing */web/data maps to nothing, whereas explicitly writing */web/data/ targets nested subdirectories accurately. This single variance can silently eat data commits while keeping your workflows completely green.
Untracked Diffs: The standard utility git diff --quiet is blind to untracked files on initial initialization passes. Always verify pipeline variations by reading raw output lines from git status --porcelain.
Isolate Through Composites: Bundle identical setups, validation guards, and branch publication routines into modular custom Composite Actions. Keep your root workflow files incredibly readable, focusing only on core state execution loops.

📈 Open Code, Open Data

By decoupling static presentation logic from isolated data branches, you can build remarkably resilient, scalable open-data utilities that cost nothing to run.

The complete open-source pipeline architecture, testing frameworks, and spatial datasets are entirely free to explore and copy:

🌐 Live Application: mchittineni.github.io/india-village-finder
📦 Codebase & Architecture Logs: github.com/mchittineni/india-village-finder

Scraping Indian government open data in 2026: what actually works

Manideep Chittineni — Mon, 13 Jul 2026 15:02:38 +0000

I maintain Village Finder, an open-source mapping project tracking over 50,000 Indian villages. It works by pulling daily raw updates straight from official public administration feeds.

That means my GitHub Actions CI environment hits Indian government web infrastructure every single day. Over time, you accumulate quite a few war stories.

If you're building systems that rely on open data pipelines, these five hard-learned lessons will save you days of baseline debugging.

1. The WAF That Lies to Python-Requests (`HTTP 502`)

One week, my daily cron job started throwing a wall of HTTP 502 Bad Gateway errors after its eighth automatic retry. Oddly, copying and pasting the exact same endpoint URLs into a desktop browser loaded the JSON data instantly.

Rate limits? IP ranges blocked by geographical firewalls? A sudden mid day server maintenance window? I spent hours drafting a complicated time of day query handler before stepping back to run the simplest sanity test:

# Same host machine, same target API gateway URL:
curl -A "Mozilla/5.0" https://api.data.gov.in/... ➡️ 200 OK
python3 -c "import requests; print(requests.get('...').status_code)" ➡️ 502 Bad Gateway

The Catch: The Web Application Firewall (WAF) protecting the data.gov.in gateway explicitly fingerprints incoming request headers. If it catches the default python-requests or urllib3 signature, it drops the connection and purposefully returns a misleading 502 instead of a classic 403 Forbidden.

The fix is a single, honest line of code:

session.headers["User-Agent"] = "my-civic-project/1.0 (+https://github.com/me/my-project)"

Takeaway: When an endpoint drops connection only when executed from automation code, isolate the User-Agent immediately. Beware of firewalls that mask blocks behind 502 server errors—they will send you chasing transient infrastructure ghosts that don't exist.

2. Treat Upstream Outages as a Contract, Not an Exception

Public data infrastructure has bad days. If your nightly cron job fires an angry email alert every single time an official site drops connection for an hour, your alerts turn into background noise and real codebase bugs go unnoticed.

To manage this cleanly, adopt the classic Unix EX_TEMPFAIL standard. When data fetchers hit a total upstream failure after exhaustive retries, the script catches the issue and explicitly exits with a status code of 75.

# GitHub Actions contract snippet
- name: Fetch Remote Assets
  run: python scraper/pipeline.py
  continue-on-error: true
  id: fetch_step

- name: Evaluate Exit Code
  run: |
    if [ ${{ steps.fetch_step.outputs.exit_code }} -eq 75 ]; then
      echo "Upstream is down. Skipping cleanly, maintaining previous snapshot."
      exit 0
    fi

By teaching the pipeline that a code 75 indicates a clean, non-breaking skip, the build stays green, a brief status note is logged, and yesterday's cached data snapshot is safely retained. Any other non-zero exit status code remains a true code failure and triggers an alert.

3. Handling Undocumented Frontends (The `myScheme` Blueprint)

India's central directory for welfare discovery, myScheme, doesn't offer a formal developer portal or public API keys. However, the modern web application is a client-rendered Next.js application that regularly hits a private backend query path: api.myscheme.gov.in/search/v6/schemes.

If you inspect the client networking requests, you can uncover the internal parameters:

The Guard: Passing the plain client application key alone triggers a 401 Unauthorized. The gateway explicitly cross-checks headers—you must provide exact, matching Origin and Referer strings pointing back to their landing page.
The Multilingual Gift: Passing regional language ISO parameters (e.g., lang=te, lang=kn, or lang=ta) responds with beautifully localized scheme descriptions, perfect for regional user interfaces.
The Trap: Client access keys rotate dynamically with new production releases of the main frontend. Design your fetch engine with this rotation in mind from day one. When the key breaks, ensure the workflow surfaces a specific failure type so a maintainer can update the configuration token instantly without breaking the surrounding data structures.

4. Knowing When to Walk Away (And Finding Alternatives)

Data engineering isn't just about building functional scripts; it's about documenting exactly why certain avenues are structural dead ends so future maintainers don't retread old steps.

During development, several highly requested public portals proved completely impossible to automate reliably:

India-WRIS / CGWB (Groundwater Data): Suffered weeks of deep network drops, prolonged NXDOMAIN routing issues, and permanent connection timeouts.
Urvarak (Fertilizer Stocks): Hard-resets TCP connections instantly when queried by headless HTTP clients. It is structurally isolated for portal usage only.
Soil Health Card Directory: Implements aggressive client-side validation barriers and relies on undocumented, dynamic internal state transformations.

The Solution: Walk away and look for open alternatives. Instead of scraping fragile state portals, I integrated ISRIC SoilGrids’ open geospatial point API for deep soil profiles, transitioned to ISRO’s Bhuvan WMS vector layers for groundwater prospects, and hardcoded regional baseline pricing sheets directly into configuration files.

5. Expect Zero Cross-System Naming Conventions

Do not assume different departments within the same government speak the exact same dialect. The market monitoring network (Agmarknet) might report trades under the district name "Chittor" or "Dr.B.R.A.Konaseema", while the administrative census directory logs those same regions as "Chittoor" and "Dr. B.R. Ambedkar Konaseema".

Because shared national metadata codes are rarely integrated universally across distinct microservices, you must build a translation layer.

A lightweight string normalizer paired with a fallback token-matching threshold resolves these regional spelling variances seamlessly:

def normalize_name(text):
    return "".join(c for c in text.lower() if c.isalnum())

# Match token tokens: ["dr", "br", "ambedkar", "konaseema"]

Budget for structural string reconciliation inside your normalization layers right from the start.

🚀 The End Result

Wrestling through these API barriers means I can provide a lightning-fast, static utility where a rural user can click a single map location to pull up local market pricing, applicable welfare benefits, and detailed topsoil compositions—all instantly translated into their local script.

The complete automated architecture, testing blueprints, and deep decision records are open to everyone:

📦 Explore the Pipeline on GitHub: mchittineni/india-village-finder

Indian villages with live market prices, government schemes, and soil data

Manideep Chittineni — Mon, 13 Jul 2026 14:52:53 +0000

I mapped 78,000+ Indian villages with live market prices, government schemes, and soil data (Serverless)

If you've ever tried to query rural geographic data in India, you know the workflow: archaic government portals with deeply nested dropdowns, session timeouts, aggressive captchas, and names spelled three different ways.

The Ministry of Panchayati Raj actually maintains an authoritative registry called the Local Government Directory (LGD). It contains every administrative unit in the country, perfectly indexed with unique codes. The catch? There is no map, no global fuzzy search, and the data is locked inside fractured tabular feeds.

To solve this, I built Village Finder: a fully open-source, interactive, multilingual map tracking Andhra Pradesh, Telangana, Karnataka, and Tamil Nadu indexing over 78,000 villages across 130 districts with an automated pipeline that keeps the entire platform fresh.

⚡ What It Does

Visual Administrative Hierarchy: An interactive choropleth map that drills down from District -> Mandal/Taluk -> Village, backed by instantaneous client-side fuzzy search (including by PIN code).
Localized Neural Scripts: Fully translated into 6 languages: English, Telugu, Kannada, Tamil, Hindi, and Urdu (with native RTL layout support). Because the raw API feeds lacked local script fields, the build pipeline hooks into an offline neural transliteration model (AI4Bharat IndicXlit) to generate accurate native names.
Vector Land Parcels: Streams millions of raw cadastral survey plots (CC0 maps from state GIS agencies) directly on the map. You can tap any plot, copy its unique geometric identifiers, and seamlessly jump to official land portals.
Live Mandi Quotes: Aggregates real-time APMC market quotes (commodity variety, price ranges, modal rates) for the village's district via the national Agmarknet feed, refreshed daily.
Dynamic Agriculture Profiles: Delivers an hyper local experience per village, pulling 7-day agromet forecasts (Open-Meteo), organic soil profiles (pH, texture, and organic carbon from ISRIC SoilGrids), and active farmer welfare schemes matched weekly from the national myScheme platform.

🏗️ The Engineering Hook: $0 Infrastructure

The most exciting constraint of this project? Zero server costs. The entire architecture operates natively on top of the GitHub ecosystem.

[ data.gov.in API ] ──(Daily GitHub Action)──> [ Pytest Validation Suite ]
                                                      │
                                          (Static File Compilation)
                                                      │
                                                      ▼
[ Browser Client ] <──(Byte-Range Requests)── [ Git Branches / Pages CDN ]

1. Vector Streaming via PMTiles

To render millions of highly complex land parcel polygons without spinning up an expensive PostGIS database and tile server (like Tegola or Martin), I compressed the spatial data into PMTiles a single file, cloud-optimized vector archive. Using MapLibre GL on the frontend, the client browser sends standard HTTP range requests to read only the specific bytes needed for their exact map viewport.

2. Git Branches as a Free CDN

Because public government APIs can be flaky, I designed a pipeline that runs daily via GitHub Actions. Instead of polluting the main branch commit history with volatile daily changes (like shifting mandi prices or scheme details), the workflows push isolated data snapshots to dedicated standalone data branches (e.g., data/mandi-prices). The runtime web app treats these raw file paths on GitHub as a high-availability, CORS-enabled static CDN.

3. CI/CD as a Data Audit Trail

For core LGD administrative data, the daily workflow opens an automated Pull Request. Before it can merge, a strict pytest suite running over 90 validation checks ensures that zero broken codes or invalid geometries leak into production.

Once approved, the commit history becomes a transparent, version-controlled audit log of exactly when the government updates rural boundaries such as when a routine refresh reclassified and renumbered nearly 1,600 villages overnight.

📦 The Data is Yours

A primary goal of this project was to liberate this data for other engineers. The code is under the MIT License, and the processed datasets are published under the open GODL-India framework.

You can download clean, flat data drops (.csv and .json splits mapped with pristine structural hierarchies, PIN codes, and localized scripts) directly from the repository's Releases tab.

🌐 Live App: mchittineni.github.io/india-village-finder
📦 Source Code & Data: github.com/mchittineni/india-village-finder

If you are working in civic tech, agritech, logistics, or just love messing around with serverless geospatial architectures, I'd love to hear your thoughts. Issues, ideas, and Pull Requests to add new states are always welcome!

Village Finder!

Manideep Chittineni — Mon, 13 Jul 2026 14:43:49 +0000

Finding accurate rural data in India shouldn't mean wrestling with clunky legacy portals.

Meet Village Finder an open-source project mapping 78,000+ villages across South India in a unified District ➡️ Mandal/Taluk ➡️ Village hierarchy.

What's inside?
🔹 Instant fuzzy search by village name or PIN code
🔹 On-demand lookups for nearby hospitals, police stations & post offices
🔹 Real-time district mandi prices & Open-Meteo forecasts
🔹 Interactive cadastral layers showing survey plot lines

Best part? Hosting costs are exactly $0. The entire pipeline runs via GitHub Actions, compiling vector layers into cloud-optimized PMTiles for serverless streaming.

100% free and open-source. Explore the map or download the flat data splits:
👉 https://mchittineni.github.io/india-village-finder/

OpenData #GIS #BuildInPublic #Serverless #CivicTech

🗺️ Empowering Rural Analytics: Announcing India Village Finder

Manideep Chittineni — Mon, 13 Jul 2026 14:43:06 +0000

Accessing accurate, localized information for rural India shouldn't require jumping through hoops. Whether you are a logistics provider optimizing last-mile delivery networks, an agricultural researcher evaluating regional crop patterns, or a citizen checking today's market rates, data accessibility matters.

Today, we are highlighting Village Finder an open-source interactive mapping application that cleans, organizes, and visualizes geographical data across Andhra Pradesh, Telangana, Karnataka, and Tamil Nadu using the official District -> Mandal/Taluk -> Village hierarchy.

⚡ What Can You Do with Village Finder?

🔍 Instant Fuzzy Search: Instantly locate any of the 78,000+ indexed villages by name, region, or PIN code, completely bypassing complex government directory navigation.
🌾 Live Mandi Prices: View the day's real-time APMC market quotes (commodity variety, price ranges, and modal ₹/quintal) pulled directly from the Government of India’s Agmarknet feed, neatly aggregated at the district level.
🌱 Agricultural & Soil Insights: Access hyper-local 7-day weather forecasts alongside dynamic groundwater prospects (Bhuvan/NRSC) and WRB soil classification profiles (ISRIC SoilGrids) to evaluate localized farming conditions instantly.
🏥 Civic Infrastructure Mapping: Run on-demand, point-radius lookups for nearby critical services including hospitals, clinics, police stations, post offices, and fire stations powered by open-source OpenStreetMap integrations.
📐 Cadastral & Land Record Linkage: Toggle optional, high-resolution cadastral survey plot layers on the map. One tap copies a plot's unique boundaries and GPS coordinates, allowing seamless transition to official state portals for fetching FMB sketches.

🤝 Open Data for Real-World Impact

Open data has the power to transform local governance, agricultural planning, and business operations. To ensure it is completely accessible to everyone, Village Finder is built entirely out in the open.

The application logic is shared under the flexible MIT License, while the processed data assets are published under the Government Open Data License (GODL-India) framework.

Whether you need a quick visual tool or flat .csv / .json data splits to fuel your own data science models, the platform is ready for you. Explore the live map, download the raw data datasets, or join the community to help expand coverage to the remaining states!

🌐 Live Web Application: Explore the Map
💻 Open Source Codebase: GitHub Repository

OpenData #CivicTech #GIS #RuralDevelopment #Logistics #AgTech

🛠️ How I Built a Serverless Interactive Map for 50,000+ Indian Villages with Zero Hosting Costs

Manideep Chittineni — Mon, 13 Jul 2026 14:42:22 +0000

Navigating government data platforms to extract precise administrative boundaries or village-level coordinates is notoriously difficult. Developers and researchers are often met with fragmented datasets, fragile APIs, slow load times, and anti-bot measures.

To bridge this gap, I built Village Finder a fully open-source, interactive geospatial platform covering Andhra Pradesh, Telangana, Karnataka, and Tamil Nadu.

What makes this project unique is its architecture. It manages structural data for over 50,000 villages, hosts fluid visual map layers, streams individual cadastral land parcels, handles multi-language transliteration, updates daily and operates with exactly zero server or infrastructure costs.

Here is a deep dive into how I engineered a highly scalable, serverless civic-tech application using a modern static web stack.

🏗️ The Architectural Breakdown

[ data.gov.in API ] ──(Daily GitHub Action)──> [ Python Pipeline & Pytest ]
                                                      │
                                           (Generates Static Artifacts)
                                                      │
                                                      ▼
[ Browser Client ] <──(Byte-Range Requests)── [ Cloud Storage (PMTiles) ]

1. The Data Pipeline (Python + GitHub Actions)

Instead of provisioning a heavy, always on relational database, the entire data lifecycle is orchestrated by GitHub Actions (update-data.yml).

Ingestion: The daily scheduled workflow queries the official Local Government Directory via the data.gov.in open API and cross-checks metrics against the live portal to catch stale data.
Validation: Before any updates go live, a strict automated test suite runs via pytest. This guarantees internal consistency ensuring every village maps cleanly to a valid sub-district and parent district code.
Storage: Validated datasets are compiled directly into normalized JSON and flat CSV assets, which are then automatically committed back into the repository as version-controlled data releases.

2. Serverless Vector Mapping (PMTiles & MapLibre GL)

Handling cadastral data individual survey plots and property boundaries across multiple states means dealing with millions of complex polygons. Standard geo-architectures require an expensive PostGIS database coupled with a dynamic tile server (like Tegola or Martin) to render these layers fluidly.

I bypassed this infrastructure completely using PMTiles:

The entire global state boundary and cadastral dataset is compressed into a single-file, cloud-optimized vector tile archive format.
This PMTiles archive is hosted on a cost-effective, CORS-enabled cloud storage bucket.
Using MapLibre GL on the frontend, the client browser requests only the specific byte ranges of the file required to view the user's current map viewport.

The result? Fast, pan-and-zoom vector tile maps served directly to the user with no database queries or active server computing required.

3. Native Script Translation (Offline Neural Models)

A massive usability hurdle was that the upstream API feeds lacked localized script names (Telugu, Kannada, Tamil, Hindi, Urdu). Relying on runtime machine translation APIs would introduce latency, require API keys, and spike operational expenses.

To solve this natively:

I embedded an offline compilation step using the AI4Bharat IndicXlit neural transliteration model.
During the build phase, the model maps canonical English names to localized scripts, outputting a highly compressed, static names_translit.json engine.
The frontend reads this static dictionary instantly at runtime, offering robust multilingual UI configurations and custom, morpheme-aware localized place names with zero client-side processing lag.

📈 Key Takeaways

Building impactful public utility platforms doesn't require a massive cloud infrastructure budget. By leaning heavily into static site architecture, edge-hosted assets, cloud-optimized geospatial files (PMTiles), and continuous integration pipelines, you can build remarkably fast, robust, and entirely free community applications.

Want to see it in action, audit the pipeline, or help add support for the remaining states?

🌐 Live Project: Explore India Village Finder
📦 Open Source Repository: mchittineni/india-village-finder

🗺️ Open-Source Interactive Village Finder for India

Manideep Chittineni — Fri, 26 Jun 2026 15:41:27 +0000

Finding accurate, granular information about villages in India isn’t as straightforward as it should be.

Whether you’re a developer building a logistics platform, a researcher analyzing demographic data, an organization coordinating rural outreach, or simply someone trying to locate a ancestral hometown, the workflow is often the same: navigating clunky legacy portals, bypassing captchas, downloading fragmented spreadsheets, or relying on outdated datasets.

I wanted to fix that.

So, I built India Village Finder an open-source project designed to turn raw, official government administrative registries into a fast, searchable, and interactive web application.

🌐 Live Demo: Explore the Map
📦 GitHub Repository: mchittineni/india-village-finder

🎯 Why I Built It

The Government of India’s Local Government Directory (LGD) is the absolute authoritative registry of administrative areas. However, consuming its data feeds isn't particularly developer-friendly or visually intuitive.

I wanted to build a platform that:

Transforms dense spreadsheets into clean, interactive map visualizations.
Makes over 50,000+ rural locations instantly searchable by name or postal code.
Operates entirely on a serverless, automated pipeline that keeps the data fresh.
Maintains a fully reproducible, open-source codebase that can eventually scale to cover every state in India.

The goal wasn’t just to throw together a static web page it was to build a highly maintainable data engineering pipeline backed by automated testing, continuous integration, and transparent updates.

📈 Current Coverage & Scale

The current release provides full District -> Mandal/Taluk -> Village hierarchical coverage for four major southern states: Andhra Pradesh, Telangana, Karnataka, and Tamil Nadu.

Metric	Regional Scale
Districts	130
Mandals / Taluks	1,866
Villages	~50,000+

(Note: Sub-districts are dynamically mapped to their regional names Taluks in Karnataka/Tamil Nadu and Mandals in AP/Telangana).

✨ Key Features

🗺️ Interactive Maps: A fluid UI featuring shaded district-level choropleths. Click a district to zoom into its sub-districts, list its villages, or drop precise pins.
⚡ Instant Search: Fuzzy, client-side matching across districts, mandals, villages, and PIN codes.
🗣️ Multilingual & Script-Aware UI: Full localized interface support for English, Telugu, Kannada, Tamil, Hindi, and Urdu (with native RTL layout parsing).
📛 Authoritative Local Names: Village names automatically display in their native scripts, mapped from official LGD spelling and fallback offline neural transliteration models (AI4Bharat IndicXlit).
📍 Hyper-Local Points of Interest: On-demand lookup of nearby civic infrastructure (hospitals, police stations, government offices, post offices) powered by live OpenStreetMap Overpass queries.
📦 Flat-File Downloads: Every automated build generates clean, versioned .csv and .json data dumps available directly in the GitHub Releases tab.

🛠️ The Tech Stack

To keep hosting costs at exactly $0, the entire project is built with a serverless, static architecture:

Backend & Data Pipeline: Python 3.10+ (Data processing, pytest validation suites).
Frontend Mapping & UI: Vanilla JavaScript, Leaflet.js, Fuse.js (Fuzzy search engine), and custom CSS grid structures.
Geospatial Formats: GeoJSON and localized coordinate mappings (GeoNames + OSM integrations).
Automation Infrastructure: GitHub Actions (Pipelines & CI/CD) and GitHub Pages for web hosting.

⚙️ Architecture & Automated Pipeline

Instead of manually modifying static files, the repository implements a decoupled, state-code-driven architecture. A shared Python pipeline handles the heavy lifting, generating independent, production-ready web apps for each state from a single web_template/ directory.

.
├── scraper/              # Shared data engineering pipeline
│   ├── pipeline.py       # LGD Data acquisition & normalization
│   ├── build_boundaries.py # Simplifies administrative map polygons
│   └── tests/            # Data integrity and join validation (pytest)
├── andhra_pradesh/       # Self-contained web app & localized datasets
├── telangana/            # Identical decoupled structure
├── karnataka/            
└── tamil_nadu/

The Automated Data Review Flow

Data integrity is protected by a strict CI/CD loop. Data is never pushed directly to the main branch:

Scheduled Fetch: GitHub Actions trigger a daily query to the official data.gov.in LGD API.
Validation Check: The script runs pytest to ensure internal consistency (valid codes, precise JSON-to-CSV symmetry, non-empty shapes).
Automated PRs: If changes are detected upstream, the runner automatically spins up a Pull Request featuring an auto-generated changelog detailing exactly which villages were added, removed, or reclassified.
Green-to-Deploy: Once tests pass and code owners review the structural diff, merging to main instantly triggers a GitHub Pages deployment.

🔥 What Makes This Different?

Most open government data projects end at a static cloud drive link full of stale CSVs. This project bridges the gap between data accessibility and user experience:

💡 It is an end-to-end data product. It wraps raw government registries in a beautiful, responsive interface with robust localization, provides reproducible pipelines for developers, and commits to version-controlled, auditable dataset histories via semantic releases.

🗺️ The Roadmap

We are just getting started. Future milestones include:

[ ] Expanding coverage to all remaining Indian states and union territories.
[ ] Implementing a public, read-only REST API for rapid village and PIN code lookups.
[ ] Enhancing coordinate coverage for deeper rural village centers.
[ ] Deep accessibility (a11y) pass for screen-reader and high-contrast compliance.

🤝 Looking for Contributors!

This project is completely open source (Dual-licensed: MIT for code, GODL-India for data). Whether you're a GIS developer, an open-data advocate, or someone who notices a missing village in your local district, contributions are incredibly welcome!

Here is how you can jump in:

Submit bug reports or feature ideas.
Help test localized translation engines or regional scripts.
Contribute boundary optimization tweaks to improve rendering performance.
Flag data or coordinate corrections by opening an issue.

👉 Get Involved: Open an Issue or Pull Request

If this project helps you, saves you development time, or makes open data a bit easier to interact with, please head over to the repository and drop a Star (⭐). It helps others discover the project and keeps the momentum going!

Happy coding!