Most data engineering and geospatial projects follow a predictable infrastructure blueprint: an ingestion cron job, an enterprise database (like PostGIS), a hosted API layer, a dynamic tile server, and a frontend client. Before you write a single line of business logic, your cloud architectural diagram already carries a fixed monthly overhead.
For Village Finder an open-source interactive mapping application tracking over 78,000 Indian villages, live market rates, and land records I have decided to reject the traditional stack.
Instead, I applied core DevOps, GitOps, and DataOps principles to build an entirely serverless, self-updating data platform. The operational infrastructure bill? Exactly $0.
Here is the architectural blueprint of how I shifted left on geospatial data infrastructure.
🏗️ 1. Infrastructure as Code (IaC) is Good. Data as Code is Better.
The foundational principle of GitOps is that Git is the single source of truth. If your infrastructure states can live in a declarative YAML file, your application data can too.
In Village Finder, the core dataset of record (the structural relationship between Districts -> Mandals -> Villages) is treated exactly like code.
[Upstream API] ➡️ [Daily Ingestion Run] ➡️ [Automated Code Generation] ➡️ [PR Matrix]
When changes occur upstream in the Government of India’s Local Government Directory, a daily scheduled GitHub Actions pipeline handles the processing. But instead of updating an active runtime database, it generates normalized, flat-file static JSON and CSV tables and writes them directly back into a version-controlled Git Pull Request.
The Core Benefits:
- Auditable Log Streams: The repository’s commit history doubles as an exact, immutable audit trail of physical data mutations over time (e.g., catching exactly when local authorities reclassified or shifted thousands of village codes overnight).
-
Zero-Downtime rollbacks: If an upstream data feed introduces corrupt schema transformations, rolling back production to a prior valid data state is as simple as a
git revert.
🔀 2. Separating State: PR Pipelines vs. Historyless Data Branches
If you commit highly volatile, real-time data metrics (like daily market prices or weekly welfare shifts) directly to your master branch, your primary git tree history will explode in size within months.
To solve this, we implemented a decoupled lifecycle strategy separating Structural Data from Volatile Metrics:
Structural Data Pipeline (The PR Path)
Changes to core geographic boundaries trigger a standard code-review loop. An automated workflow creates a Pull Request accompanied by a dynamically generated markdown changelog detailing exactly which entries were modified. Consecutive nightly runs update the same open PR branch, eliminating bot spam.
Volatile Metrics Pipeline (The Flat Data Branch CDN)
Daily commodity prices change constantly. For these, our automated runners bypass the Pull Request loop entirely and push directly to standalone, isolated git tracking branches (e.g., data/mandi-prices) using an history-less force-push contract:
# Compile volatile snapshots into a completely history-less single-commit branch
work="$(mktemp -d)" && cp out/*.json "$work" && cd "$work"
git init -q -b data/mandi-prices
git add . && git commit -q -m "data: snapshot $(date -u +%FT%RZ)"
git push --force "https://x-access-token:${TOKEN}@github.com/${REPO}.git" data/mandi-prices
GitHub’s global static asset delivery network (raw.githubusercontent.com) serves public files with open CORS permissions out-of-the-box. The browser frontend handles fetches directly from the head of these flat data branches at runtime. We get a high-availability, zero-latency data delivery network with no active server management.
🧪 3. Shifting Left on Data Validation (DataOps Testing)
In a traditional DevOps loop, Continuous Integration (CI) validates code compilation and unit tests. In a DataOps loop, CI must validate data integrity.
A single missing sub-district mapping or a broken polygon geometry will crash a client-side map canvas. To prevent this, our pipeline treats raw data updates with the same rigor as code changes. Every automated Pull Request must pass an exhaustive suite of over 90 validation rules orchestrated via pytest:
def test_referential_integrity():
# Assert that every single village maps cleanly to a valid structural parent code
assert missing_parents == 0
def test_geojson_geometries():
# Enforce polygon boundary validity before deploying static vector tile coordinates
assert geometries_are_valid == True
If the upstream government registry outputs an incomplete schema variant, the automated testing matrix fails loud, blocks the integration path, and prevents the deployment runner from shipping broken maps into production.
☔ 4. Designing for Fragile Upstreams: The EX_TEMPFAIL Contract
Public data networks encounter high rates of transient downtime, network timeouts, and sudden firewall shifts. If your continuous deployment platform fires a critical failure alarm every single time a remote service faces a routine network drop, your team experiences alert fatigue. You begin to ignore notifications, and true application bugs slip past unnoticed.
To build an infrastructure that tolerates flaky external gateways, we adopted the classic Unix EX_TEMPFAIL status contract.
# Handle flaky connections inside ingestion layers
except RequestException as e:
logger.error(f"Transient upstream network drop: {e}")
sys.exit(75) # EX_TEMPFAIL
When our data acquisition routines encounter a terminal connection timeout after multiple retries, they explicitly exit with a status code of 75.
Our orchestration runner identifies code 75 not as a breaking error, but as a clean skip request. The workflow execution concludes gracefully, retains the previous day's working data cache, logs a brief summary update, and keeps the overall build pipeline completely green. Alarms only sound if something is truly broken inside our internal code logic.
📐 5. Progressive Build Aggregations (Branch Overlays)
To keep the platform highly decoupled, we use a modular architecture where separate workflows generate independent data layers (like complex cadastral survey boundaries or village point catalogs) entirely in parallel.
At deployment time, instead of running heavy multi-hour calculation steps sequentially, a highly optimized compilation worker overlays these pre-rendered data blocks straight into the target production directory using shallow checkouts:
- name: Overlay Pre-Rendered Vector Boundaries
run: |
git fetch -q --depth 1 origin "data/boundary-tiles"
git checkout -q FETCH_HEAD -- ./tiles/boundaries.pmtiles
If a developer forks the project locally to build a minor feature patch, absent data branches skip missing files quietly. This design ensures the entire local system remains lightweight, instantly reproducible, and highly modular for open-source contributors.
🎯 The Big Takeaway
DevOps is frequently treated as an enterprise enterprise cost center—a tool setup meant for managing Kubernetes nodes or provisioning massive cloud databases.
Village Finder proves that by applying core DevOps mental models—immutability, continuous data validation gates, decoupled architecture, and treating external inputs as a version-controlled pipeline—you can deliver fast public software with exceptional durability and zero maintenance costs.
- 🌐 Live App: mchittineni.github.io/india-village-finder
- 📦 Explore the Actions & Code: github.com/mchittineni/india-village-finder
Top comments (0)