DEV Community

Umar Farook M
Umar Farook M

Posted on

How I Built a Domain - Specific LLM for Public Transit - From Data Collection to Deployment

#ai #python #opensource #machinelearning

The Problem

Ask ChatGPT how many bus routes Chicago CTA operates — you'll get an answer. But is it grounded in actual data, or is it pulling from random web pages?

General-purpose LLMs are impressive, but they struggle with structured, domain-specific data like transit schedules. They hallucinate route numbers, invent stop names, and get GTFS spec details wrong.

I wanted a model that answers from real data — verifiable, traceable, and accurate.

So I built one.

What is UmarTransit-1B?

UmarTransit-1B is a domain-specific language model that understands:

  • GTFS specification — file formats, field meanings, relationships
  • Transit routes and stops — from 15 real agencies across 10 countries
  • Schedule analysis — service patterns, frequencies, operating hours
  • Transfer connections — interchange points between routes
  • Cross-agency comparisons — comparing transit systems worldwide

It runs locally on CPU, is available in 3 formats (including GGUF for Ollama/llama.cpp), and the entire pipeline is open source.

Links:

The Pipeline

Here's how I built it, end to end.

Step 1: Data Collection

GTFS (General Transit Feed Specification) is an open standard — transit agencies worldwide publish their schedules as ZIP files containing CSV data. I used the Mobility Database to find open feeds.

I selected 15 agencies across 10 countries for geographic diversity:

Region Agencies
North America LA Metro, Chicago CTA, Boston MBTA, Portland TriMet, Austin Capital Metro, Phoenix Valley Metro, Toronto TTC
Europe Berlin VBB, Paris IDFM, Helsinki HSL, Amsterdam OVapi, Belgian Railways, Denmark Rejseplanen
Oceania Perth Transperth, Auckland Transport

Raw data: 858 MB, 77 million stop_time records, 11,000 routes, 296,000 stops.

Step 2: Data Cleaning

GTFS feeds are messy. Different agencies use different conventions, optional fields vary, and some data is just wrong. I built a cleaning pipeline that:

  • Validates against the official GTFS schema
  • Normalizes field formats (route types, stop coordinates)
  • Removes orphaned records (stops with no routes, trips with no stop times)
  • Converts everything to Parquet for efficient processing

Step 3: Synthetic Dataset Generation

Here's where it gets interesting. I couldn't just feed raw CSV data to a language model. I needed question-answer pairs that teach the model to reason about transit data.

I built 45 question templates across 8 categories:

Categories:
1. Route information    → "How many routes does [agency] operate?"
2. Stop details         → "What stops are on route [X]?"
3. Schedule analysis    → "What are the operating hours for [route]?"
4. Transfer queries     → "Where can I transfer between [route A] and [route B]?"
5. Network statistics   → "What is the total number of stops in [city]?"
6. GTFS specification   → "What does route_type 3 mean?"
7. Cross-agency compare → "Compare the transit systems of [city A] and [city B]"
8. Validation           → "Is this a valid GTFS feed?"
Enter fullscreen mode Exit fullscreen mode

Each template queries the cleaned Parquet data to generate factually grounded answers. No hallucination possible — every answer traces back to real data.

Result: 3,306 instruction/response pairs (2,971 train, 335 test).

Step 4: Training

I chose Qwen2.5-1.5B-Instruct as the base model because:

  • Small enough to train on a free Colab T4 GPU
  • Small enough to run on CPU for inference
  • Apache 2.0 license (no restrictions)
  • Strong instruction-following capability for its size

Training setup:

  • Method: QLoRA (4-bit quantized LoRA)
  • Framework: Unsloth + TRL's SFTTrainer
  • Hardware: Google Colab free tier (T4 GPU, 15 GB VRAM)
  • Time: ~30 minutes
  • Cost: $0 
# Key hyperparameters
LoRA rank: 16
LoRA alpha: 32
Learning rate: 2e-4
Epochs: 3
Batch size: 4
Max sequence length: 2048
Enter fullscreen mode Exit fullscreen mode

Step 5: Evaluation

Metric Score
ROUGE-L 0.82
Keyword Match 0.41
Best Category Transfer queries (0.90)
Test Pairs 335

The keyword match score (0.41) looks low but is actually conservative — it penalizes paraphrasing. If the model says "Bus service" instead of "Bus", it loses points even though the answer is correct.

Step 6: GGUF Export

For local deployment, I exported to GGUF format:

Format Size Use Case
Safetensors 3.09 GB Full precision, Transformers/Python
GGUF Q4_K_M 986 MB 4-bit quantized, Ollama/llama.cpp
GGUF Q8_0 1.65 GB 8-bit quantized, higher quality

The Q4 model runs at ~2 tokens/sec on a regular laptop CPU. Not fast, but usable.

Step 7: Deployment

  • Model hosting: HuggingFace Hub (all 3 formats)
  • Backend: FastAPI serving inference locally
  • Frontend: Next.js on Vercel with demo mode (pre-computed answers) + live mode (connects to local API)

What It Looks Like 

Q: What does route_type 3 mean in GTFS?
A: In the GTFS specification, route_type 3 indicates Bus service.
   This is the most common route type and covers short- and
   long-distance bus routes.

Q: How many routes does the Chicago Transit Authority operate?
A: The Chicago Transit Authority (CTA) operates 133 routes.
   8 are Subway/Metro routes and 125 are Bus routes.
Enter fullscreen mode Exit fullscreen mode

ChatGPT vs UmarTransit-1B

An interesting comparison — when I asked ChatGPT the same question about CTA routes, it said 135 (127 bus + 8 rail). My model says 133 (125 bus + 8 rail).

Both are correct, but for different points in time. ChatGPT pulls from recent web data. UmarTransit-1B answers from the specific GTFS feed it was trained on.

This is the key difference: UmarTransit-1B's answers are traceable to a specific dataset. For developers building transit apps, that traceability matters more than having the latest number.

Lessons Learned

1. Domain-specific models don't need to be huge.
1.5B parameters is tiny by today's standards, but it's enough when the domain is narrow and the training data is focused.

2. Synthetic data works if it's grounded in facts.
The trick is generating Q&A pairs from real data, not making them up. Every answer in my training set can be verified against the source GTFS feed.

3. QLoRA makes training accessible.
Free Colab, 30 minutes, $0. The barrier to building domain-specific models is lower than most people think.

4. The data pipeline is 80% of the work.
Collecting, cleaning, and generating training data took weeks. Training took 30 minutes. The model is only as good as the data behind it.

5. GGUF makes deployment practical.
Going from a 3 GB safetensors model to a 986 MB GGUF file makes the difference between "needs a GPU" and "runs on any laptop."

What's Next

  • Adding more feeds (Singapore, Bangalore, Tokyo) to expand from 10 to 13 countries
  • Scaling training data from 3,306 to 10,000+ pairs
  • Adding new categories: fare information, accessibility, real-time GTFS
  • Cloud backend deployment for live inference

Try It Yourself

Everything is open source under Apache 2.0:

If you're working with transit data or thinking about building domain-specific models, I'd love to hear your feedback.

Top comments (0)