DEV Community

James
James

Posted on

Why We Use Python (Not Rust) for Our Search Engine

Rust vs. Python for Web Infrastructure: What We Chose and Why

The Rust-vs-Python debate is exhausting. Both are excellent. The right choice depends on what you're building, who is building it, and how fast you need to ship.

What We Build

At Graham Miranda UG, our stack has three parts:

  1. Query dispatcher (Python/FastAPI) — Dispatches searches to 8-12 sources
  2. Result ranker (Python) — Normalizes and scores cross-source results
  3. Frontend (Next.js/TypeScript) — SSR, minimal client-side JS

Why Python Won

Developer Velocity

Our team knows Python. Moving to Rust would mean 3-6 months of learning before productive work. In a startup, that's death.

Ecosystem Maturity

Python's web ecosystem is unmatched:

  • FastAPI: async, typed, OpenAPI-native
  • Celery: battle-tested distributed tasks
  • BeautifulSoup + lxml: 20 years of HTML parsing refinement
  • Playwright: best-in-class browser automation

Async Performance

Python 3.11+ async is fast enough for our workload. 3,000 queries/day on a single VPS. We won't hit Python's ceiling for years.

Where Rust Would Help

  • Raw scraping throughput: Rust's HTTP clients are 2-3x faster
  • Memory efficiency: Rust uses 1/10th the RAM for the same workload
  • Long-term maintenance: Rust's type system catches bugs at compile time

Our Actual Hybrid Approach

We use Rust for exactly one component: a custom HTTP client for high-throughput crawling. It communicates with the Python backend via gRPC. Best of both worlds — Rust performance where it matters, Python velocity everywhere else.

The Honest Take

For most web infrastructure, Python is the right choice. You ship faster, hire easier, and debug quicker. Switch to Rust when:

  • You're processing 100K+ requests/second
  • Memory costs are your primary expense
  • You have a team experienced in systems programming

Until then, Python's "good enough" performance is genuinely good enough.


Graham Miranda builds privacy-first search and automation tools at Graham Miranda UG (Berlin, HRB 36794).

Top comments (0)