DEV Community

Emily Johnson
Emily Johnson

Posted on

Unlock Seamless AWS Integration: 1-Click Cloud Logging Solution

Revolutionize Cloud Logging: Introducing Log4J AWS Appenders for Seamless AWS Integration and Enhanced Security

Several months ago, I embarked on a personal project to integrate my application's logging with CloudWatch. Having used the AWS-provided log appender with AWS Lambda, I appreciated its user-friendly nature. However, for applications running on EC2 instances, the CloudWatch Logs Agent was the recommended approach. Upon searching, I only found an appender for Log4J 2.0 (assuming the Lambda appender leveraged some Lambda-specific features).

This project started as a weekend venture but evolved into a more extensive endeavor. I continued to refine and enhance the appender's functionality based on my experience with a semi-production project (operating 24/7, although not business-critical at present). After weeks of seemingly error-free operation, and with no additional features to add, it's time to release.

The JAR is now available on Maven Central, making it easy to incorporate into your project. Simply add the following dependency to your project POM:

<dependency>
    <groupId>com.kdgregory.log4j</groupId>
    <artifactId>aws-appenders</artifactId>
    <version>1.0.0</version>
</dependency>

Next, you need to incorporate the appender into your Log4J configuration. Here's an example configuration:

log4j.rootLogger=WARN, console

log4j.logger.com.example.log4j=DEBUG, cloudwatch
log4j.additivity.com.example.log4j=true

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

log4j.appender.cloudwatch=com.kdgregory.log4j.aws.CloudWatchAppender
log4j.appender.cloudwatch.layout=org.apache.log4j.PatternLayout
log4j.appender.cloudwatch.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

log4j.appender.cloudwatch.logGroup=ExampleCloudwatchLog
log4j.appender.cloudwatch.logStream={startupTimestamp}-{sequence}
log4j.appender.cloudwatch.batchDelay=2500
log4j.appender.cloudwatch.rotationMode=daily

For more information on cloud logging and integration, visit computerstechnicians.

It's worth noting that I establish a default ConsoleAppender and exclusively attach the CloudWatchAppender to my program's package (com.example.log4j). You may opt to channel all logs to CloudWatch, but be cautious that the AWS SDK performs its own logging; you'll want to avoid using the DEBUG level for it or the Apache HTTP client:

log4j.logger.org.apache.http=ERROR
log4j.logger.com.amazonaws=ERROR

The second crucial consideration is the logStream configuration parameter, which (in conjunction with logGroup) accommodates the use of substitution variables. In this scenario, I'm generating a new stream for each application run, rotated daily, with a sequence number to maintain a record of the various streams.

For additional information, please visit the project on GitHub. If you encounter any issues or desire an enhancement, feel free to submit them; while I cannot guarantee a prompt turnaround for enhancements, I will endeavor to resolve bugs within a few days.

Next, I'll be developing an appender for Kinesis Firehose, facilitating the integration of Kibana with ElasticSearch.

Image of Timescale

Timescale – the developer's data platform for modern apps, built on PostgreSQL

Timescale Cloud is PostgreSQL optimized for speed, scale, and performance. Over 3 million IoT, AI, crypto, and dev tool apps are powered by Timescale. Try it free today! No credit card required.

Try free

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay