DEV Community

Cover image for Indian villages with live market prices, government schemes, and soil data
Manideep Chittineni
Manideep Chittineni

Posted on

Indian villages with live market prices, government schemes, and soil data

#data #dataengineering #serverless #showdev

I mapped 78,000+ Indian villages with live market prices, government schemes, and soil data (Serverless)

If you've ever tried to query rural geographic data in India, you know the workflow: archaic government portals with deeply nested dropdowns, session timeouts, aggressive captchas, and names spelled three different ways.

The Ministry of Panchayati Raj actually maintains an authoritative registry called the Local Government Directory (LGD). It contains every administrative unit in the country, perfectly indexed with unique codes. The catch? There is no map, no global fuzzy search, and the data is locked inside fractured tabular feeds.

To solve this, I built Village Finder: a fully open-source, interactive, multilingual map tracking Andhra Pradesh, Telangana, Karnataka, and Tamil Nadu indexing over 78,000 villages across 130 districts with an automated pipeline that keeps the entire platform fresh.

⚡ What It Does

  • Visual Administrative Hierarchy: An interactive choropleth map that drills down from District -> Mandal/Taluk -> Village, backed by instantaneous client-side fuzzy search (including by PIN code).
  • Localized Neural Scripts: Fully translated into 6 languages: English, Telugu, Kannada, Tamil, Hindi, and Urdu (with native RTL layout support). Because the raw API feeds lacked local script fields, the build pipeline hooks into an offline neural transliteration model (AI4Bharat IndicXlit) to generate accurate native names.
  • Vector Land Parcels: Streams millions of raw cadastral survey plots (CC0 maps from state GIS agencies) directly on the map. You can tap any plot, copy its unique geometric identifiers, and seamlessly jump to official land portals.
  • Live Mandi Quotes: Aggregates real-time APMC market quotes (commodity variety, price ranges, modal rates) for the village's district via the national Agmarknet feed, refreshed daily.
  • Dynamic Agriculture Profiles: Delivers an hyper local experience per village, pulling 7-day agromet forecasts (Open-Meteo), organic soil profiles (pH, texture, and organic carbon from ISRIC SoilGrids), and active farmer welfare schemes matched weekly from the national myScheme platform.

🏗️ The Engineering Hook: $0 Infrastructure

The most exciting constraint of this project? Zero server costs. The entire architecture operates natively on top of the GitHub ecosystem.

[ data.gov.in API ] ──(Daily GitHub Action)──> [ Pytest Validation Suite ]
                                                      │
                                          (Static File Compilation)
                                                      │
                                                      ▼
[ Browser Client ] <──(Byte-Range Requests)── [ Git Branches / Pages CDN ]
Enter fullscreen mode Exit fullscreen mode

1. Vector Streaming via PMTiles

To render millions of highly complex land parcel polygons without spinning up an expensive PostGIS database and tile server (like Tegola or Martin), I compressed the spatial data into PMTiles a single file, cloud-optimized vector archive. Using MapLibre GL on the frontend, the client browser sends standard HTTP range requests to read only the specific bytes needed for their exact map viewport.

2. Git Branches as a Free CDN

Because public government APIs can be flaky, I designed a pipeline that runs daily via GitHub Actions. Instead of polluting the main branch commit history with volatile daily changes (like shifting mandi prices or scheme details), the workflows push isolated data snapshots to dedicated standalone data branches (e.g., data/mandi-prices). The runtime web app treats these raw file paths on GitHub as a high-availability, CORS-enabled static CDN.

3. CI/CD as a Data Audit Trail

For core LGD administrative data, the daily workflow opens an automated Pull Request. Before it can merge, a strict pytest suite running over 90 validation checks ensures that zero broken codes or invalid geometries leak into production.

Once approved, the commit history becomes a transparent, version-controlled audit log of exactly when the government updates rural boundaries such as when a routine refresh reclassified and renumbered nearly 1,600 villages overnight.

📦 The Data is Yours

A primary goal of this project was to liberate this data for other engineers. The code is under the MIT License, and the processed datasets are published under the open GODL-India framework.

You can download clean, flat data drops (.csv and .json splits mapped with pristine structural hierarchies, PIN codes, and localized scripts) directly from the repository's Releases tab.

If you are working in civic tech, agritech, logistics, or just love messing around with serverless geospatial architectures, I'd love to hear your thoughts. Issues, ideas, and Pull Requests to add new states are always welcome!

Top comments (0)