DEV Community: Kevin

Using Async SQLAlchemy Inside Sync Celery Tasks

Kevin — Sun, 22 Mar 2026 19:50:18 +0000

Introduction

Let's be honest. You've built this beautiful, modern web application using FastAPI and Async SQLAlchemy. Everything is blazing fast and non-blocking. Then, you need to handle background jobs - sending emails, processing reports, or syncing with third-party APIs. So, you reach for Celery.

You try to run your trusty async database queries inside a Celery task, and suddenly, things get weird. You see errors about event loops, or the task just hangs. Why is this so hard?

If you’ve been there, you’re in the right place. Let’s break down why Celery and async SQLAlchemy don’t play nicely out of the box and, more importantly, how to make them work together without losing your sanity.

The Core Problem: Sync vs. Async Worlds

How Sync SQLAlchemy Works

In a synchronous environment, database operations are blocking. When you execute a query using standard SQLAlchemy:

The driver sends a request to the database server via a TCP socket.
The execution thread is blocked at the system level, waiting for data to be available on that socket.
The CPU cannot perform any other work on that thread until the database response is fully received and parsed.

In Celery, this means a worker process or thread is completely occupied during the entire I/O wait. If you have many concurrent I/O-bound tasks, you quickly saturate your worker pool, leading to significant latency and resource waste.

What is Celery?

In one sentence: Celery is a distributed task queue that processes tasks synchronously (by default) using either multiprocessing, gevent, or threading.

By default, if you give Celery an async function, it won’t know what to do with it. It expects a regular, synchronous function.

Async SQLAlchemy to the Rescue (But Wait...)

Async SQLAlchemy solves this by leveraging non-blocking I/O. Instead of blocking the thread, it yields control back to the event loop while waiting for the database response.

## This is the dream - non-blocking database call
async def get_user(user_id):
    async with AsyncSession() as session:
        result = await session.execute(select(User).where(User.id == user_id))
        return result.scalar_one()

The issue? Celery is synchronous by nature. You can’t just put an await inside a Celery task function because Celery doesn’t run an event loop.

The Naïve Attempt (And Why It Fails)

You might think, "I’ll just use asyncio.run() to run my async code inside the sync Celery task!"

import asyncio
from celery import shared_task

@shared_task
def process_user_task(user_id):
    # This seems clever, but it's a trap!
    user = asyncio.run(get_user(user_id))
    return user

This works for one task, but it will eventually crash your database connection pool. Here’s why:

asyncio.run() creates a brand new event loop to run your async function.
Your async function opens a database connection from the pool.
The function finishes, and asyncio.run() destroys the entire event loop.
The database connection was tied to that destroyed loop. When SQLAlchemy tries to return that connection to the pool, the connection is essentially "dead" or stuck in a closed loop.

After a few tasks, your connection pool becomes corrupted, and you start seeing errors like

sqlalchemy.exc.TimeoutError: QueuePool limit of size ... overflow ... reached

The Solution: No Pool + async_to_sync

To fix this, we need to change two things about how we handle database sessions for background tasks:

Don't use a connection pool. Since each task will run in its own isolated event loop (created and destroyed), we should open a fresh connection for the task and close it when the task finishes. No pooling, no cross-loop contamination.
Use async_to_sync to bridge the gap. This utility (from asgiref) allows us to run async code from a sync context without manually managing the event loop's lifecycle as poorly as asyncio.run() does.

Step 1: Configure a "No Pool" Session

In your database configuration, create a specific session maker for your Celery tasks that uses NullPool. This ensures that connections are closed when the session is closed, not kept alive.

## db.py
from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker
from sqlalchemy.pool import NullPool

## Your regular async engine (for FastAPI, etc.)
## This likely uses a pool (like AsyncAdaptedQueuePool)
main_engine = create_async_engine(
    "postgresql+asyncpg://user:pass@localhost/db",
    pool_size=20,
)

## Celery-specific engine with NO POOL
celery_engine = create_async_engine(
    "postgresql+asyncpg://user:pass@localhost/db",
    poolclass=NullPool,  # <-- This is the key!
)

## Session makers
AsyncMainSession = async_sessionmaker(main_engine, expire_on_commit=False)
AsyncCelerySession = async_sessionmaker(celery_engine, expire_on_commit=False)

Step 2: Use async_to_sync from asgiref

Instead of creating a custom decorator, we can use the async_to_sync utility from the asgiref library. This is the industry-standard way to bridge sync and async code, handling the complexities of managing a per-thread event loop.

Step 3: Call Your Async Logic Directly

Write your core database logic as a standard async function. Then, in your sync Celery task, call it directly using async_to_sync. This keeps the Celery task minimal and makes the bridge explicit.

from celery import shared_task
from asgiref.sync import async_to_sync
from db import AsyncCelerySession
from sqlalchemy import select
from models import Article


async def _run():
    async with AsyncCelerySession() as session:
        ## Perform non-blocking database queries
        result = await session.execute(select(Article).where(Article.status == "pending"))
        articles = result.scalars().all()

        return len(articles)

@celery_app.task(name="app.workers.run_article_pipeline", acks_late=True)
def run_article_pipeline_task():
    # Direct bridging without a decorator
    return async_to_sync(_run)()

How This Works Internally

Let's visualize the flow:

Celery Worker starts. It loads the run_article_pipeline_task function as a standard synchronous function.
Task Executes: The function body calls async_to_sync(_run)().
Event Loop: async_to_sync manages the lifecycle of the event loop for the current thread and executes the _run coroutine.
Database Connection: Inside _run, AsyncCelerySession() creates a new connection using NullPool. This avoids cross-loop connection contamination by opening a fresh TCP connection.
The Wait: The event loop handles the "await" points, allowing for efficient I/O management even within the sync Celery worker.
Cleanup: Once the async with block closes the session, NullPool ensures the socket is physically closed. async_to_sync then clears the loop state.

No dead connections, no pool corruption, and you get to use your beautiful async SQLAlchemy code.

A Note on Performance

Using NullPool and creating a new connection for every task adds a small overhead (TCP handshake, authentication). For background tasks that run infrequently (like once every few seconds), this is perfectly fine.

If you are running thousands of tasks per second, you might need a different architecture (like moving to a fully async task queue, or using gevent workers with a sync SQLAlchemy pool). But for 90% of use cases, this "No Pool + async_to_sync" pattern is the cleanest and most reliable way to reuse your async code in a sync Celery environment.

A Guide to Designing REST APIs Like a Pro Engineer

Kevin — Sun, 22 Mar 2026 14:16:37 +0000

How Good Engineers Design REST APIs

When engineers start building APIs for the first time, the endpoints often look like this:

POST /createBlog
GET /getBlogs
POST /deleteBlog

This works at the beginning. The API does what it needs to do, and the frontend can communicate with the backend.

But as the system grows, this style quickly becomes messy. New endpoints appear for every action, naming becomes inconsistent, and the API slowly turns into a collection of unrelated routes.

This happens because the API is designed around actions instead of resources.

Good engineers approach API design differently. Instead of thinking about what action the client wants to perform, they think about what resource the system exposes.

Think in Terms of Resources

Every application revolves around a few core entities.

If you are building a blog platform, those entities might be users, blogs, and comments.

Instead of creating endpoints for every action, the API is designed around these resources.

For example, a beginner API might look like this:

POST /createBlog
GET /getBlogById
POST /deleteBlog

This style treats the API like a collection of functions.

A resource-oriented design is much simpler:

POST /blogs
GET /blogs/1
DELETE /blogs/1

Here the URL represents the resource, while the HTTP method represents the action.

This makes the API predictable. Once someone understands one endpoint, they can guess the rest.

Let HTTP Do Its Job

HTTP already provides verbs that describe what we want to do with resources.

Instead of inventing custom actions in URLs, REST APIs rely on these standard methods.

Creating a resource uses POST, retrieving one uses GET, updating uses PUT or PATCH, and deleting uses DELETE.

For example, creating a blog might look like this:

POST /blogs

The request body contains the blog data, and the server responds with 201 Created once the resource has been stored.

Using these standard methods makes APIs predictable. Developers interacting with your system already know what to expect because the semantics come from HTTP itself.

PUT vs PATCH in REST APIs

Both PUT and PATCH update resources, but they behave slightly differently.

PUT usually replaces the entire resource. If you send a full blog object, the server overwrites the existing one.

PATCH updates only the fields provided in the request.

For example, if you only want to change the title of a blog post, PATCH is usually the better choice.

Understanding this difference helps avoid unexpected behavior when APIs evolve.

Keep URLs Predictable

A well-designed API should feel intuitive even before reading documentation.

Most APIs treat endpoints as collections, which is why resource names are usually plural.

/users
/blogs
/comments

Even when requesting a single item, we are selecting something from that collection.

/blogs/1

This structure scales naturally as the system grows.

If blogs later have comments, the relationship can be represented directly:

GET /blogs/1/comments

The URL now mirrors the data model of the system, making the API easier to understand.

REST API Naming Conventions

Naming conventions are one of the simplest ways to make an API predictable.

Most REST APIs follow a few common practices:

Use plural resource names
Use lowercase URLs
Avoid verbs in endpoints

For example:

/blogs
/blogs/42
/users/5/comments

The endpoint describes the resource hierarchy, while the HTTP method describes the action.

This separation keeps APIs consistent as they grow.

Consistency Matters More Than Perfection

One of the biggest differences between beginner APIs and well-designed ones is consistency.

Clients interacting with an API should be able to predict how responses will look.

Many teams adopt a consistent response structure so every endpoint behaves the same way.

Example:

{
  "data": {...},
  "message": null,
  "error": null
}

The exact format is less important than the consistency.

When responses follow a predictable structure, frontend developers and other services can integrate much faster.

The same principle applies to naming conventions. Whether you choose camelCase or snake_case, the important part is using the same style everywhere.

Idempotency in REST APIs

A concept that becomes important in real systems is idempotency.

An operation is idempotent if repeating it multiple times produces the same result.

For example:

DELETE /blogs/1

If the blog is deleted once, calling the same request again should not change the system further. The blog is already gone.

This property becomes important when network failures occur. Clients often retry requests, and idempotent operations ensure that retries do not create unexpected side effects.

GET, PUT, and DELETE are typically idempotent, while POST usually is not.

Designing APIs for Real Systems

In real applications, endpoints rarely return just one object. They often return collections, and those collections can grow large.

Instead of returning everything at once, APIs usually support pagination.

A typical request might look like this:

GET /blogs?page=1&limit=10

This allows the client to request manageable chunks of data, improving performance and reducing load on the server.

APIs also commonly support filtering and sorting.

For example:

GET /blogs?authorId=42&status=published&sort=-createdAt

This request asks the server to return blogs written by author 42, filter only published posts, and sort them by newest first.

Designing flexible query parameters like this allows APIs to evolve without creating dozens of new endpoints.

APIs Change Over Time

No API stays the same forever. New features appear, data models evolve, and sometimes breaking changes are unavoidable.

To avoid disrupting existing clients, APIs are often versioned.

A common approach is including the version in the URL:

/api/v1/blogs

If a future change requires a different structure, a new version can be introduced without affecting older clients.

Versioning allows systems to evolve while maintaining stability for applications already relying on the API.

REST API Design Checklist

When designing APIs, a few principles consistently lead to better systems:

Design around resources
Use HTTP methods correctly
Keep URLs predictable
Maintain consistent response formats
Support pagination and filtering
Version APIs when breaking changes occur

Following these principles keeps APIs understandable as systems grow.

Final Thoughts

Designing a good REST API is not about memorizing a long list of rules.

It is about building interfaces that are predictable, consistent, and easy for other engineers to understand.

When APIs are designed around resources, use HTTP correctly, and maintain clear URL structures, they become much easier to scale and maintain over time.

Responsive SignUp Page

Kevin — Tue, 05 Apr 2022 09:10:23 +0000

Responsive SignUp Page Design with Social Icons