DEV Community

Emily Johnson
Emily Johnson

Posted on

Unlock Seamless AWS Integration: 1-Click Cloud Logging Solution

Revolutionize Cloud Logging: Introducing Log4J AWS Appenders for Seamless AWS Integration and Enhanced Security

Several months ago, I embarked on a personal project to integrate my application's logging with CloudWatch. Having used the AWS-provided log appender with AWS Lambda, I appreciated its user-friendly nature. However, for applications running on EC2 instances, the CloudWatch Logs Agent was the recommended approach. Upon searching, I only found an appender for Log4J 2.0 (assuming the Lambda appender leveraged some Lambda-specific features).

This project started as a weekend venture but evolved into a more extensive endeavor. I continued to refine and enhance the appender's functionality based on my experience with a semi-production project (operating 24/7, although not business-critical at present). After weeks of seemingly error-free operation, and with no additional features to add, it's time to release.

The JAR is now available on Maven Central, making it easy to incorporate into your project. Simply add the following dependency to your project POM:

<dependency>
    <groupId>com.kdgregory.log4j</groupId>
    <artifactId>aws-appenders</artifactId>
    <version>1.0.0</version>
</dependency>

Next, you need to incorporate the appender into your Log4J configuration. Here's an example configuration:

log4j.rootLogger=WARN, console

log4j.logger.com.example.log4j=DEBUG, cloudwatch
log4j.additivity.com.example.log4j=true

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

log4j.appender.cloudwatch=com.kdgregory.log4j.aws.CloudWatchAppender
log4j.appender.cloudwatch.layout=org.apache.log4j.PatternLayout
log4j.appender.cloudwatch.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

log4j.appender.cloudwatch.logGroup=ExampleCloudwatchLog
log4j.appender.cloudwatch.logStream={startupTimestamp}-{sequence}
log4j.appender.cloudwatch.batchDelay=2500
log4j.appender.cloudwatch.rotationMode=daily

For more information on cloud logging and integration, visit computerstechnicians.

It's worth noting that I establish a default ConsoleAppender and exclusively attach the CloudWatchAppender to my program's package (com.example.log4j). You may opt to channel all logs to CloudWatch, but be cautious that the AWS SDK performs its own logging; you'll want to avoid using the DEBUG level for it or the Apache HTTP client:

log4j.logger.org.apache.http=ERROR
log4j.logger.com.amazonaws=ERROR

The second crucial consideration is the logStream configuration parameter, which (in conjunction with logGroup) accommodates the use of substitution variables. In this scenario, I'm generating a new stream for each application run, rotated daily, with a sequence number to maintain a record of the various streams.

For additional information, please visit the project on GitHub. If you encounter any issues or desire an enhancement, feel free to submit them; while I cannot guarantee a prompt turnaround for enhancements, I will endeavor to resolve bugs within a few days.

Next, I'll be developing an appender for Kinesis Firehose, facilitating the integration of Kibana with ElasticSearch.

Do your career a big favor. Join DEV. (The website you're on right now)

It takes one minute, it's free, and is worth it for your career.

Get started

Community matters

Top comments (0)

Billboard image

Modern monitoring

With Checkly, you can use Playwright tests and Javascript to monitor end-to-end scenarios in your NextJS, Astro, Remix, or other application.

Get started now!

👋 Kindness is contagious

Dive into an ocean of knowledge with this thought-provoking post, revered deeply within the supportive DEV Community. Developers of all levels are welcome to join and enhance our collective intelligence.

Saying a simple "thank you" can brighten someone's day. Share your gratitude in the comments below!

On DEV, sharing ideas eases our path and fortifies our community connections. Found this helpful? Sending a quick thanks to the author can be profoundly valued.

Okay