DEV Community: Mukul Sharma

Blocking FastAPI Access Logs

Mukul Sharma — Sun, 14 Jul 2024 03:55:08 +0000

Have you ever found your FastAPI deployment cluttered with unnecessary access logs? These logs can create more noise than value, obscuring important information. Let's explore a neat trick to selectively block these access logs in FastAPI.

Access logs can quickly overwhelm your console, making it difficult to spot critical information.

No B.S. Solution



# main.py
import logging

block_endpoints = ["/endpoint1", "/endpoint2"]

class LogFilter(logging.Filter):
    def filter(self, record):
        if record.args and len(record.args) >= 3:
            if record.args[2] in block_endpoints:
                return False
        return True

uvicorn_logger = logging.getLogger("uvicorn.access")
uvicorn_logger.addFilter(LogFilter())

How It Works

We create a LogFilter class that inherits from logging.Filter.
The filter method checks if the log record corresponds to one of our blocked endpoints.

If the endpoint is in our block_endpoints list, we return False, preventing the log from being processed.
By default we return True to log as usual.

We apply this filter to Uvicorn's access logger, which FastAPI uses under the hood.

Going Deep

When working with Python's logging module, each log entry is represented by a LogRecord object. In our LogFilter class, the record parameter is an instance of this LogRecord class.

The args attribute of a LogRecord contains a tuple of arguments which are merged into the log message. In the context of Uvicorn's access logs, these arguments hold valuable information about each request. Here's a breakdown of what each argument typically represents:



# Typical structure of record.args for Uvicorn access logs
remote_address = record.args[0]  # IP address of the client
request_method = record.args[1]  # HTTP method (GET, POST, etc.)
query_string = record.args[2]    # Full path including query parameters
html_version = record.args[3]    # HTTP version used
status_code = record.args[4]     # HTTP status code of the response

Understanding this structure allows us to create more sophisticated filters. For example, we could filter logs based on status codes, specific IP addresses, or HTTP methods, not just endpoint paths.

Gotchas and Tips

This method works with Uvicorn's default logging setup. If you've customized your logging configuration, you might need to adjust the logger name.
At times you might have a separate file for logging config such as logger.py. You may define the class LogFilter there but make sure to apply .addFilter in main.py.

Remember that completely blocking logs for certain endpoints might make debugging more challenging. Use this technique judiciously.

As thank you for making it to the end of this post, here is a trick I use for maximum benefit.



# main.py
import logging

block_endpoints = ["/endpoint1", "/endpoint2"]

class LogFilter(logging.Filter):
    def filter(self, record):
        if record.args and len(record.args) >= 4:
            if (record.args[2] in block_endpoints 
                and record.args[4] == "200"):
                return False
        return True

uvicorn_logger = logging.getLogger("uvicorn.access")
uvicorn_logger.addFilter(LogFilter())

This will only block the request if the status code is 200. This is super helpful when blocking INFO endpoints where you only need to be informed if an unexpected status code was returned.

Conclusion

By implementing this simple logging filter, you can significantly reduce noise in your FastAPI application logs. This allows you to focus on the information that truly matters, making your debugging and monitoring processes more efficient.

What to do next?

Feel free to ask questions and share your experience in the comments below!
For further reading, check this GitHub discussion.
Check out my LinkedIn here.

Handling AWS WAF Capacity Limits with Terraform

Mukul Sharma — Sun, 09 Jun 2024 18:28:02 +0000

When managing AWS WAF resources using Terraform, one common issue is exceeding the Web ACL Capacity Units (WCUs) limit, which can lead to deployment failures. This blog post will guide you through understanding WCUs, checking capacity using AWS CLI, and integrating this process into your Terraform workflow to ensure smooth deployments.

Understanding WCUs in AWS WAF

AWS WAF uses Web ACL Capacity Units (WCUs) to measure the resources required to run rules, rule groups, and Web ACLs. Each rule type has a different capacity requirement based on its complexity and processing needs. The capacity limits help AWS manage the performance and cost of running WAF rules efficiently.

Simple Rules: Require fewer WCUs (e.g., size constraint rules).
Complex Rules: Require more WCUs (e.g., regex pattern sets).

For detailed WCU calculations, refer to the AWS WAF documentation oai_citation:1,AWS WAF web ACL capacity units (WCUs) - AWS WAF, AWS Firewall Manager, and AWS Shield Advanced.

Common Error

When you exceed the WCU limits, you may encounter the following error:

WAFInvalidParameterException: Error reason: You exceeded the capacity limit for a rule group or web ACL.

Checking WCU Requirements with AWS CLI

Terraform lacks built-in functionality to check WCUs before applying configurations. However, AWS CLI provides a check-capacity command to verify the WCU requirements. Here’s how you can use it:

Define WAF Rules in JSON: Create a JSON file (rules.json) with the rules you want to include:

   {
     "Rules": [
       {
         "Name": "Rule1",
         "Priority": 1,
         "Statement": {
           "ByteMatchStatement": {
             "SearchString": "BadBot",
             "FieldToMatch": {
               "UriPath": {}
             },
             "TextTransformations": [
               {
                 "Priority": 0,
                 "Type": "NONE"
               }
             ]
           }
         },
         "Action": {
           "Block": {}
         },
         "VisibilityConfig": {
           "SampledRequestsEnabled": true,
           "CloudWatchMetricsEnabled": true,
           "MetricName": "Rule1Metric"
         }
       }
     ]
   }

Run the check-capacity Command: Use the AWS CLI to check the capacity:

   aws wafv2 check-capacity --scope REGIONAL --rules file://rules.json

The output will look like this:

{
    "Capacity": 15
}

This command returns the total WCU requirement for your specified rules. Ensure it doesn't exceed 5,000 WCUs for a single Web ACL.

Integrating Capacity Check with Terraform

To avoid deployment failures due to WCU limits, integrate the capacity check into your Terraform workflow using a shell script:

Create a Shell Script: Write a script (check_capacity.sh) to perform the capacity check:

   #!/bin/bash

   CAPACITY=$(aws wafv2 check-capacity --scope REGIONAL --rules file://rules.json | jq '.Capacity')

   if [ "$CAPACITY" -gt 5000 ]; then
     echo "Error: Capacity units exceeded. Current capacity: $CAPACITY"
     exit 1
   else
     echo "Capacity check passed. Current capacity: $CAPACITY"
     export TERRAFORM_CAPACITY_CHECK=passed
   fi

Integrate with Terraform: Modify your Terraform workflow to include the script:

   ./check_capacity.sh

   if [ "$TERRAFORM_CAPACITY_CHECK" == "passed" ]; then
     terraform apply
   else
     echo "Terraform apply aborted due to capacity issues."
   fi

By incorporating this script, you ensure that your Terraform deployments only proceed if the WCU requirements are within limits, preventing potential failures and downtime.

Conclusion

Handling WCU limits is crucial for successful AWS WAF deployments using Terraform. By leveraging the check-capacity command in AWS CLI and integrating it into your Terraform workflow, you can proactively manage capacity and avoid deployment issues. This approach ensures a smoother, more reliable infrastructure management process especially until terraform starts to support a similar built-in functionality.

For further reading, check the official AWS WAF documentation on WCUs and the AWS CLI check-capacity command reference.