How to Build a Data Pipeline for Your AI App with Airbyte and FastAPI

Stop writing custom Python scripts for every API. Here is how I automated my data ingestion using Airbyte.

If you are building an AI application, your biggest bottleneck isn't the LLM—it's the data.

For my project, Career Agency AI, I needed to pull job listings from multiple sources (like Remotive and Arbeitnow), clean the data, and store it locally so my Gemini-powered backend could analyze it. Initially, I wrote custom Python scripts for every single API. It was a nightmare. I had to handle pagination, rate limits, and schema changes manually.

Then, I discovered Airbyte.

In this tutorial, I will show you how I used Airbyte to build an automated ETL (Extract, Transform, Load) pipeline, syncing external job APIs directly into a PostgreSQL database, which my FastAPI backend then served to the AI.

The Architecture

Instead of writing complex integration code, I moved the heavy lifting to Airbyte. Here is the flow:

1. Source: External Job APIs (HTTP).

2. Extract & Load: Airbyte handles the syncing.

3. Destination: A local PostgreSQL database (or SQLite for mobile deployments).

4. Serve & AI: FastAPI reads the data and passes it to the Gemini API.

*Step 1: Setting up Airbyte
*
Because I was developing on constrained hardware, I used Airbyte’s Docker setup. It is incredibly straightforward.

(Pro Tip: You can also use Airbyte Cloud for a managed experience.)

git clone https://github.com/airbytehq/airbyte.git
cd airbyte
docker compose up

Once the containers spin up, you can access the Airbyte UI at http://localhost:8000.

Step 2: Configuring the Source and Destination**

This is where Airbyte shines.

1. In the UI, I connected a Source (using Airbyte's HTTP API connector or a custom Python connector if the API is highly specific).

2. I set up my Destination as PostgreSQL.

Instead of writing a 200-line Python script to paginate through the job APIs and insert records into Postgres, Airbyte did it in a few clicks.

**Step 3: Building the FastAPI Backend

Once Airbyte syncs the raw job data into my database, I use FastAPI to serve it to my frontend and pass it to the AI.

Here is a simplified endpoint showing how I query the Airbyte-synced data:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import psycopg2

app = FastAPI()

# Connection to the database Airbyte populated
def get_db_connection():
    return psycopg2.connect(
        dbname="career_agency",
        user="postgres",
        password="[YOUR_PASSWORD]",
        host="localhost",
        port="5432"
    )

@app.get("/jobs/latest")
def get_latest_jobs():
    conn = get_db_connection()
    cursor = conn.cursor()

    # Query the table that Airbyte automatically created and synced
    cursor.execute("SELECT title, company, description FROM job_listings ORDER BY created_at DESC LIMIT 10;")
    jobs = cursor.fetchall()

    conn.close()

    if not jobs:
        raise HTTPException(status_code=404, detail="No jobs found")

    return {"jobs": jobs}

Step 4: Feeding the AI

Now that FastAPI has the data, the AI integration is seamless. When a user selects a job, my backend sends the description to the Google Gemini API:

import google.generativeai as genai

genai.configure(api_key="[YOUR_GEMINI_KEY]")

@app.post("/generate-cover-letter")
def generate_letter(job_id: int):
    # 1. Fetch job description from the Airbyte-synced DB
    # 2. Pass to Gemini
    model = genai.GenerativeModel('gemini-pro')
    response = model.generate_content(f"Write a cover letter for this job: {job_description}")

    return {"cover_letter": response.text}

Conclusion

If you are building AI applications, your time is too valuable to spend writing API integration scripts.

By using Airbyte to handle the ETL pipelines, I was able to focus entirely on building the FastAPI backend and engineering the AI prompts. The data was always fresh, the schema changes were handled automatically, and I saved hours of development time.

If you are building a data-heavy AI app, spin up Airbyte. Your future self will thank you.