If you're running data workflows and don’t want to manage Kubernetes, this architecture might help.
In this project, I built a production-ready workflow system using:
- Prefect for orchestration 🔄
- Cloud Run for serverless container execution 🚀
- Cloud SQL (PostgreSQL) for metadata storage 🗄️
- IAM-based authentication between services 🔐
🏗️ Architecture Overview
Prefect Agent → Cloud Run Container → Cloud SQL (Private IP)
Why this approach works well:
- Fully serverless compute ⚡
- Auto scaling
- No VM maintenance
- Secure database connectivity
- Clean separation of orchestration and execution
🛠️ Key Implementation Steps
- Create Cloud SQL instance (Private IP)
- Build Docker image for Prefect flow
- Push image to Artifact Registry
- Deploy to Cloud Run
- Configure IAM roles properly
- Connect Cloud Run to Cloud SQL
⚠️ Common Mistakes
- Forgetting Cloud SQL Auth Proxy
- Not configuring VPC connector
- Using public IP for database
🎯 Ideal Use Cases
- ETL pipelines
- Background microservices
- Event-driven automation
- Internal workflow platforms
🧠 Architecture Check
In a Cloud Run (2nd gen) setup connecting to Cloud SQL via Private IP without a Serverless VPC Connector, what enables network access?
A) Cloud NAT
B) Direct VPC egress configuration
C) Public IP exposure
D) Load Balancer
Answer: B) Direct VPC egress configuration
💡 Interview Tip
If an interviewer asks:
“Why would you choose Cloud Run over GKE for workflows?”
Strong answer:
Cloud Run reduces operational overhead and is ideal for stateless, event-driven workloads. GKE is better when you need fine-grained networking control, sidecars, or complex orchestration.
Full step-by-step guide with commands:
https://devopspilot.com/cloud/gcp/projects/prefect-cloudrun-cloudsql/
Top comments (0)