DEV Community: Vishal Anand

Adding Production-Safe API Logging to Django REST Framework

Vishal Anand — Mon, 06 Jul 2026 18:05:35 +0000

When a Django REST Framework API starts getting used in production, one of the first debugging problems is visibility.

A user reports:

“The API failed.”

But from the backend side, we often need more context:

Which endpoint was called?
What was the HTTP method?
What status code was returned?
How long did the request take?
Was the request body valid?
Did the response fail because of validation, permissions, database latency, or something else?
Can we debug this without exposing passwords, tokens, or sensitive payloads?

Many teams solve this by writing custom middleware.

That works at first, but it can become risky over time.

Custom API logging often starts small:

print(request.body)
print(response.data)

Then it grows into:

storing logs in the database
masking sensitive values
handling large request/response bodies
tracking slow APIs
searching logs from Django admin
avoiding extra latency in the request path
supporting async deployments
adding request correlation or trace IDs
avoiding accidental logging of tokens, passwords, cookies, or API keys

At that point, API logging is no longer “just middleware”. It becomes part of production observability.

This is the problem I wanted to solve with DRF API Logger.

Disclosure: I maintain this open-source package. I am sharing it here because I think the problem is common in Django REST Framework projects, and I would also like feedback from other Django developers.

What DRF API Logger does

DRF API Logger is an open-source package for Django REST Framework projects that captures API request and response information.

It can log useful debugging information such as:

API URL
HTTP method
request headers
request body
response body
status code
client IP address
API execution time
optional request tracing information
slow API indicators
optional profiling information

The goal is not to replace every observability tool.

It is mainly useful when you want structured API-level logging inside a Django/DRF project without building and maintaining custom middleware from scratch.

Why not simply write custom middleware?

You can write your own middleware, and for small projects that may be enough.

But production API logging has some common traps.

1. Sensitive data exposure

This is the biggest issue.

Logging the full request body may accidentally store:

passwords
access tokens
refresh tokens
API keys
session IDs
cookies
authorization headers
personal user data

That makes logs dangerous.

Good API logging should support masking and exclusion rules from the beginning.

Example:

DRF_API_LOGGER_EXCLUDE_KEYS = [
    "password",
    "token",
    "access",
    "refresh",
    "secret",
    "api_key",
]

Even with masking, teams should still review what they log and decide what is acceptable for their own security, privacy, and compliance requirements.

2. Performance overhead

A simple custom logger may write to the database directly inside the request/response cycle.

That means every API request waits for logging work to finish.

For low traffic, this may be acceptable. For production traffic, this can add unnecessary latency.

A better pattern is to keep request-path work small and move heavier writes out of the main request flow where possible.

3. Large payloads

Some APIs return large JSON responses.

If logs store everything without limits, the logging table can grow quickly and become expensive to query.

A production logging setup should support body-size limits.

Example:

DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = 32768
DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = 65536

4. Search and debugging experience

Storing logs is only useful if developers can find the right records later.

For example:

show all 500 responses from the last hour
search by URL
filter by method
find slow APIs
inspect a specific failed request
compare status codes
check execution time

This is where Django admin visibility becomes useful for many teams.

Basic installation

Install the package:

pip install drf-api-logger

Add it to INSTALLED_APPS:

INSTALLED_APPS = [
    # ...
    "drf_api_logger",
]

Add the middleware:

MIDDLEWARE = [
    # ...
    "drf_api_logger.middleware.api_logger_middleware.APILoggerMiddleware",
]

Enable database logging:

DRF_API_LOGGER_DATABASE = True

Run migrations:

python manage.py migrate

After this, DRF API logs can be captured and viewed from Django admin.

A more production-aware configuration

For production-like environments, I would avoid enabling full logging blindly.

A safer starting point is:

DRF_API_LOGGER_DATABASE = True

DRF_API_LOGGER_EXCLUDE_KEYS = [
    "password",
    "token",
    "access",
    "refresh",
    "secret",
    "api_key",
    "authorization",
    "cookie",
    "set-cookie",
]

DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = 32768
DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = 65536

This keeps the setup practical while reducing some common risks.

You should still review:

which endpoints should be logged
which fields should always be masked
how long logs should be retained
whether request/response bodies should be stored in your environment
whether logs contain personal or regulated data
whether logs should go to a separate database or storage backend

Using it for slow API investigation

One common use case is finding slow DRF endpoints.

For example, you may want to flag APIs slower than a specific threshold:

DRF_API_LOGGER_SLOW_API_ABOVE = 200

You can also enable profiling when you need deeper investigation:

DRF_API_LOGGER_ENABLE_PROFILING = True

This can help identify whether slowness is more likely coming from:

SQL queries
too many repeated queries
middleware overhead
view/business logic
serialization cost

This is useful when an endpoint “feels slow” but the team does not yet know where the time is going.

Database logging vs signal-based logging

Not every team wants to store API logs in the default database.

Some teams may want to send API log data somewhere else:

a file
an analytics pipeline
a monitoring system
a separate database
a queue
a custom internal tool

For that, signal-based logging can be useful.

Example:

DRF_API_LOGGER_SIGNAL = True

Then a listener can handle the log event:

from drf_api_logger import API_LOGGER_SIGNAL


def send_to_internal_observability_tool(**kwargs):
    # Example only.
    # Be careful not to export sensitive request/response bodies.
    print({
        "api": kwargs.get("api"),
        "method": kwargs.get("method"),
        "status_code": kwargs.get("status_code"),
        "execution_time": kwargs.get("execution_time"),
    })


API_LOGGER_SIGNAL.listen += send_to_internal_observability_tool

This gives more flexibility if the default database is not the right place for logs.

Security and privacy notes

API logs can be extremely helpful, but they can also become a liability.

Before enabling request/response logging in production, I recommend reviewing these questions:

1. Are we logging secrets?

Avoid storing:

passwords
tokens
API keys
cookies
session IDs
authorization headers

2. Are we logging personal data?

Depending on your application, request and response bodies may contain personal information.

Examples:

names
phone numbers
email addresses
addresses
payment-related information
health-related information
user documents

Do not treat API logs as harmless debugging data.

3. Do we need full bodies?

For many APIs, logging metadata is enough:

URL
method
status code
execution time
client IP
trace ID

Full request/response bodies should be enabled carefully.

4. What is the retention policy?

Logs should not live forever by default.

Teams should decide how long API logs are useful and when they should be deleted or archived.

5. Who can access the logs?

If logs are visible in Django admin, make sure admin access is properly restricted.

API logs may reveal internal system behavior and user data.

When I would use this package

I would consider using DRF API Logger when:

the project uses Django REST Framework
the team needs request/response visibility
developers are currently debugging production issues manually
custom middleware is growing too complex
slow API investigation is needed
Django admin visibility is useful
sensitive data masking and payload limits are required
the team wants a package instead of maintaining logging middleware internally

When I would not use it

I would not enable full API body logging blindly in every project.

I would be careful if:

the application handles highly sensitive data
compliance requirements are strict
the team does not have a retention policy
admin access is not well controlled
the database is already under heavy write load
only high-level metrics are needed

In those cases, metadata-only logging, signal-based logging, OpenTelemetry, APM tools, or custom policies may be more appropriate.

Minimal example

Here is a compact setup:

INSTALLED_APPS = [
    # ...
    "drf_api_logger",
]

MIDDLEWARE = [
    # ...
    "drf_api_logger.middleware.api_logger_middleware.APILoggerMiddleware",
]

DRF_API_LOGGER_DATABASE = True

DRF_API_LOGGER_EXCLUDE_KEYS = [
    "password",
    "token",
    "access",
    "refresh",
    "secret",
    "api_key",
]

DRF_API_LOGGER_MAX_REQUEST_BODY_SIZE = 32768
DRF_API_LOGGER_MAX_RESPONSE_BODY_SIZE = 65536

DRF_API_LOGGER_SLOW_API_ABOVE = 200

Then run:

python manage.py migrate

Project links

GitHub: https://github.com/vishalanandl177/DRF-API-Logger
PyPI: https://pypi.org/project/drf-api-logger/

Feedback request

I would appreciate feedback from Django and DRF developers on a few areas:

What do you expect from API logging in production?
Do you prefer database logging, signals, OpenTelemetry, or another storage backend?
What masking rules should be enabled by default?
What would make the admin debugging experience more useful?
What concerns would stop you from using request/response logging?

I am especially interested in feedback around security, privacy, performance, and real production debugging workflows.

Thanks for reading.

MySQL Bulk Insert taking long time

Vishal Anand — Tue, 09 Jun 2020 13:51:39 +0000

I'm using python, using MySQL connector. I'm having nearly 67 Million (14GB) entries in a table. When I do a bulk insert of 2K data each time, it is taking very long to insert.

Inserted 2000 rows in 23 Seconds
Inserted 2000 rows in 25 Seconds
Inserted 2000 rows in 29 Seconds
Inserted 2000 rows in 28 Seconds
For another table (having less data), insertion speed is fine(2-4 seconds).

After using the transaction:

Inserted 2000 rows in 21 Seconds
Inserted 2000 rows in 20 Seconds
Inserted 2000 rows in 20 Seconds
Inserted 2000 rows in 18 Seconds
How can I improve the speed?

I'm using AWS RDS, Aurora MySQL version 5.7.12 (db.t3.medium) having CPU usage 4% to 8%. My objective is to insert around 50K data into a table. This table is currently having nearly 67 Million (14GB) data already. Data must need to be inserted ASAP. This almost real-time data is very important for the client. The table is having 18 columns: id(PK auto-increment), customer, serial_number, batch, data, and some others. Indexes are on (customer,serial_number) - To make the combination unique, batch - For searching, data(unique). All are by default BTREE indexed. This insertion should need to take less than 1 minute for 50K. But currently taking around 15 minutes. I've tried inserting on an empty table. It is inserting 50K data just in 5-7 seconds. As you increase the number of entries in the table, the insertion process time is increasing.

Is upgrading MySQL version is going to speed-up the insertion process anyhow? Is it the last option to split or partitioning the table? I cannot consolidate the data because each data is important, specially the last 2 years of data. Please help.

My table schema is already having some default values in 8 columns and these data are never going to update later. There are not many Read/Write operations are going on. Almost 2 or in some cases 3 selects per second as per RDS monitor shows.

How to write cron jobs on AWS EC2 instance

Vishal Anand — Thu, 16 Apr 2020 07:22:30 +0000

Cron is a time-based job scheduler run on the server in the Unix-like operating system.
To perform such action we write cron expression made of five fields, followed by a shell command to execute.

First, we have to log in to AWS EC2 instance using CLI (Command Line Interface).
For log in, type:
ssh -i <.pem file> username@ip_address
After successfully log in, type:
crontab -e
then select any editor from the list (preferred nano).
Type cron command. For example:

* * * * * php /var/www/html/file/name.php Now save the changes. If you are using nano editor type Ctrl + x and Select Y and then press Enter. To list out all the running cron type: crontab -l
is used for every minute, every hour, etc. Here ”php” is shell command followed by file location of php script, separated by white space.

If you want to run nodejs script then type:

* * * * * node /var/www/html/nodejs/file/name.js Each from five stars represents a job:

How to write CRON jobs on AWS EC2 instance

Allowed special characters are:

Comma (,) – Commas are used to separate the item list.

For example:

Cron to run at minute 1 and 10 every hour
1,10 * * * * php /var/www/html/file/name.php
Cron to run at mid-night and mid-day
0 0,12 * * * php /var/www/html/file/name.php
Hyphen (-) – Hyphen defines range.

For example:

Cron to run at every minute from 1 minute through 10 minute, every hour
0-10 * * * * php /var/www/html/file/name.php
Cron to run at minute 0 past every hour from 6 AM through 6 PM
0 6-18 * * * php /var/www/html/file/name.php
Slash (/) – Slash defines step values.

For example:

Cron to run every 5th minute
*/5 * * * * php /var/www/html/file/name.php
Cron to run at minute 0 past every 6th hour
0 */6 * * * php /var/www/html/file/name.php
Sleep cron:

For example:

Cron to run every minute after sleep of 30 seconds

* * * * * sleep 30; php /var/www/html/file/name.php

For more examples visit: https://coderssecret.com/how-to-write-cron-jobs-on-aws-ec2-instance/