DEV Community: Joanna Wallace

How to Track DevOps Events with AWS Kinesis Data Streams

Joanna Wallace — Thu, 05 Aug 2021 14:05:23 +0000

To run a cloud platform in production, your team needs to know how things are running. There are seemingly endless metrics, measurements, and logs to analyze to ensure the platform is running as it should. Keeping clients satisfied so they continue using your platform is the goal of any cloud company.

Along with the significant amount of data you can collect from your system comes just as many tools you may use to collect them. On AWS alone, you might make use of Lambda, CloudTrail, CloudWatch, and XRay. Each of these tools also has a subset of tools that can be useful for tracking your information. However, the most interesting is not the individual data points but the analysis of that data. To properly analyze, data needs to be accessible by the same analytical tools. AWS Kinesis Data Streams can provide a method to amalgamate data quickly and efficiently for this analysis.

Features of Kinesis Data Streams

Kinesis data streams have many features that allow them to be used in a wide breadth of use cases. In this article, we will highlight features especially critical for analyzing platform health.

Real-Time Streaming Performance

Kinesis Data Stream allows data to flow through the queue at very high speeds. Each shard can consume 1MB/s input and provide 2MB/s output. AWS also limits inputs by the number of writes (1000 PUT requests per second). If you require more per second information, add new shards to increase the capacity of the stream. Adding shards will add to the capacity available directly.

With scaling speeds, streaming from a Kinesis Data Stream to a real-time analytics process can provide fast results. For DevOps security, notifications can be sent to users efficiently, so teams may address problems earlier, even while they are occurring. This speed can significantly shorten the downtime of your platform.

Easily Scale Capacity

Your platform may require different capacity settings based on predicted or spontaneous usage spikes. Kinesis Data Streams can dynamically scale with capacity ranging from the megabytes available with a single shard up to terabytes. The number of PUT requests can also scale up to millions of records per second. This scaling capacity means Kinesis can grow as your platform gains users and requires more throughput. You can stick with the same tool as your business grows and not need to rebuild infrastructure as you scale.

Resource-Linked Cost

Like many AWS services, with Kinesis Data Streams, you pay for what you use. For each shard created, AWS charges for shard hours. The actual cost is dependent on the AWS region used, ranging from $0.03/shard hour in Sao Paulo to $0.015/shard hour in North Virginia. Users are also charged per million PUT payload units, again with a cost dependent on the region and similar to the shard hour cost. AWS charges for optional features like encryption and data retention separately.

Security and Encryption

AWS encrypts data in transit by default. They also allow users to encrypt data at rest optionally. Developers can choose between managing their encryption keys or having AWS encryption applied using AWS KMS. For streaming, security data encryption at rest could be necessary. Data from AWS CloudTrail or private user information should be encrypted to limit the ability of attacks to get information of use.

Kinesis Data Streams Versus SQS

Data Streams are AWS-managed infrastructure. When setting up this service, you do not need to consider storage, provisioning, or deployment of the stream. Both Data Streams and SQS are AWS-managed queue services. Each can be useful for different requirements and flows through your cloud platform. Here, we are discussing analyzing DevOps data to detect security, scalability, and bugs in your cloud platform. Features of Kinesis Data Streams make it the better choice for this end.

Kinesis can provide ordering of records which is not available with standard SQS queues. A value called the sequence number is assigned to the kinesis value. The data is guaranteed unique per partition key per shard. Data is guaranteed to arrive at the consumer in the correct order using this value.

Kinesis also can read and re-read data in the same order by the same or new consumers. Data is stored in the queue after reading for a predetermined amount of time. This differs from SQS, which will hold data only until a consumer processes it. Both Kinesis and SQS offer retries to read data.

SQS does not give the ability to have multiple consumers listen to the same queue. SQS provides load balancing if multiple consumers are reading from a queue. Kinesis, however, will provide the same data to all consumers. Throughput is calculated for each consumer. If you need real-time speed and have a significant amount of data, consider using the available enhanced fanout setting on Kinesis Data Streams. This setting will enable each consumer to have its throughput capacity without affecting other connected consumers

Writing to Kinesis Data Streams

AWS Kinesis data streams can collect data from many sources in AWS. Kinesis can then forward data to different analytics tools like the Coralogix log analytics platform.

AWS Lambda and Kinesis Data Streams

AWS Lambda is a serverless compute system managed by AWS. These functions are commonly used in cloud computing to run the features of your system. Alternatively, developers may choose to run Fargate tasks, or EC2 compute functions. Each of these can interface to Kinesis using a similar methodology.

Compute functions can send data to Kinesis Data Streams for further analysis. This data may include interactions with APIs, data from outside sources, or results from Lambda itself. To write to your Data Stream from Lambda, use the AWS SDK. Developers can add various valuable data to the Kinesis data stream using the function laid out below.

  let kinesis = new AWS.Kinesis();
  kinesis.putRecord({
    Data: 'STRING_VALUE',
    PartitionKey: 'STRING_VALUE',
    StreamName: 'STRING_VALUE',
    ExplicitHashKey: 'STRING_VALUE',
    SequenceNumberForOrdering: 'STRING_VALUE'
  }).promise();

AWS CloudWatch to Kinesis Data Streams

CloudWatch allows users to configure subscriptions. These subscriptions will automatically send data to different AWS services, including Kinesis Data Streams. Subscriptions include filter configurations that allow developers to limit what data is sent to Kinesis.

Developers can also use these filters to send data to different Data Streams, allowing for different processing to occur based on the data’s content. For example, data needed to process DevOps logs may go to a single stream bound for an analytics engine, while user data may go to a different stream bound for long-term storage.

Use the AWS CLI to set up the subscription to a Kinesis Data Stream using the following commands. You must create the stream before assigning a subscription to it. You will also need to create an IAM role for your subscription to write to your stream. For a complete description of the steps to create a CloudWatch subscription, see the AWS documentation.

AWS CloudTrail to Kinesis Data Streams

CloudTrail can be configured to send data directly to AWS S3 or AWS CloudWatch, but not to AWS Kinesis. SInce CloudTrail can write directly to AWS CloudWatch, we can use the above configuration linking CloudWatch to Kinesis Data Streams to collect CloudTrail data.

If you are creating your CloudTrail from the console, an option to configure a CloudWatch linkage is available. Once turned on, CloudWatch pricing applies to your integration.

Consuming Kinesis Data Streams

Kinesis Data Streams can send data to different consumers for analysis or storage. Consumers available include AWS Lambda, Fargate, and EC2 compute functions. You can also configure data streams to send directly to another Kinesis product like Kinesis Analytics. Using AWS compute functions and stored data, you can calculate metrics and store information. However, doing this requires significant manual work and foreknowledge of what to look for in data. Kinesis Analytics makes computation easier by applying user-built SQL or Apache apps to process your data.

Developers can also configure Kinesis to send data to third-party tools that remove the need to set up analytics. Coralogix provides several tools that can analyze different data to produce essential metrics and notifications for your platform. The security platform can analyze AWS information streaming from Kinesis to give insights about breaches and retrospective analysis of your security weak points. Coralogix log analytics system can take CloudWatch data and notify your team when your platform is not performing optimally.

Summary

Cloud platforms use Real-time analytics to ensure your system is functioning optimally and is secure. AWS services can provide all stream data to AWS Kinesis, a real-time queue with an extensive range of capabilities that can be set to accommodate your platform’s needs. Kinesis can be set up to send data to multiple endpoints if different analysis is needed on the same data. Consumers of the Kinesis stream will perform the analytics. These may be solutions made by your platform team on AWS compute functions like Lambda or Fargate. They can be semi-manual functions made by your platform team using Kinesis Analytics tools. The most efficient way to perform analytics is to use a third-party tool designed for your needs, like Coralogix’s security or log analytics platforms.

Elastic or Open Distro: Which Distribution Should you Choose?

Joanna Wallace — Tue, 20 Jul 2021 14:51:13 +0000

What spawned Open Distro?

In January 2021, Elastic announced that Elasticsearch and Kibana would move from the OSS Apache 2.0 license to a dual license model. The dual licenses are the paid Elastic License and a free-tier Server Side Public License (SSPL). While SSPL is free for use, it is not classified as open-source by the Open Source Initiative (OSI). Instead, SSPL is a ‘source available’ license meaning that modifications you make must be made public. For more details on this license and its implications with Elastic, see the Coralogix blog’s breakdown.

While not every company is affected by this new agreement, some companies could not conform to the new restrictions. A platform that simply used the Elastic stack without modification has no new restrictions (assuming they never would change the source code). However, some businesses built functionality around the Elastic stack. Most notably, this issue affected AWS’s release of AWS Elasticsearch Service.

AWS launched Open Distro in 2019 as an Elasticsearch Distribution that would be used by AWS and could be used as an alternative to Elastic. Prior to the license change, AWS would both pull in Elastic’s source updates and contribute to the Elastic distribution. As a result of the license change, AWS decided to maintain a fork from the most recently available Elastic distribution under the Apache license. Open Distro still operates under the Apache License and is what now drives the AWS Elasticsearch service. According to the license, users are encouraged but not required to submit code changes, including features and bug fixes.

License Differences between Elastic and Open Distro

Elastic runs under a dual-license model. The free tier is the Server Side Public License which is not technically open source. The license essentially means the software is free to use, but contributors must make modifications publicly available if using the source code in any SaaS platform. There is also a paid tier under the Elastic License, allowing users to access advanced features without access to the source code.

The Open Distro Distribution is completely available under the Apache License Version 2.0. Apache is a genuinely open-source license as defined by the OSI. Modifications may be used privately or made public at the discretion of the contributor.

If you choose to use Elastic, keep in mind there are two different free tiers. The tier labeled ‘Free and Open’ can be under either SSPL or the Elastic License. The one labeled ‘Basic - Free and Open’ is the Elastic License.

Security in Elastic and Open Distro

One reason for creating Open Distro in 2019 was that security features in Elastic are mostly linked to a paid and licensed plugin service. With Elastic Security, you pay for the hardware resources you use. Open Distro uses a security plugin under the same open-source Apache 2.0 license. The two security offerings differ in their feature set as well.

Open Distro Security

Open Distro has enterprise-grade security features connected through a plugin for authentication and access control. A demo setup is available for use in testing and development environments, but a complete setup must be done to use security in production environments. Steps to set up security are clearly laid out on the Open Distro site.

Elastic Security

Elastic Stack Security has some features in the free tier, and others require Gold, Platinum, or Enterprise subscriptions. By default, security settings are disabled with the two free licenses. Security must be enabled using XPack.

Secure settings are available on every license tier of Elastic, but not all features are included in every tier. Features only included on paid tiers include Elasticsearch and Kibana audit logging, AP filtering, user authentication, and Elasticsearch token service. Platinum and Enterprise subscriptions only include single sign-on, attribute-based access control, field, and document level security, custom authentication and authorization, encryption at rest, and FIPS mode.

Security Features Available in Both Distributions

The following features are available on both distributions, with some features being paid services in Elastic Security.

Node-to-Node Encryption

Open Distro encrypts all data flowing between nodes in your cluster. Elastic also provides encrypted communications and also encryption at rest. Encrypted communications are available only in the free and open tier of Elastic. Encryption at rest is only available at the platinum or enterprise subscription level.

HTTP Basic Authentication

Both distributions use basic authentication over HTTP. This authentication uses a user name and password as part of each HTTP request. It is needed in every request since both distributions run as stateless systems.

Industry-Standard User Authentication

Both distributions use existing industry-standard authentication tools to authenticate existing users or create new users. Users are stored in an internal user database. Support includes Kerberos, OpenID Connect, and SAML. Open Distro also includes Active Directory and LDAP.

Role-Based Access

User roles determine what actions users may take in the cluster. This includes what data can be read and written, what cluster settings can be modified, and whether or not they can add users to the cluster. Roles can be reused across different users, and users may have multiple roles as well.

Audit Logging

Logs are printed from both Elasticsearch and Kibana, tracking access to your cluster. Businesses can use these logs to prove regulatory compliance or to analyze cluster exposures after an attack. Users can view and manage logs in Kibana. Events tracked include failed logins, successful authentications, missing or granted privileges, and role change attempts. Elastic only includes audit logging on its paid tiers.

Multi-Level Security

Access can be granted or restricted at multiple data levels. In Open Distro, these include at the index, document, or even field level. In Elastic, field and document level security is available only on Platinum or Enterprise tiers.

Search Features in Elastic and Open Distro

AWS forked Open Distro from Elasticsearch and Kibana 7.10 source code. Features created before this version in Elastic are the same on the two distributions since both are based on the same source code. Here we will discuss some significant diversions between the two distributions to help determine which is better for your needs.

Since Open Distro forked only minor versions back from the current Elastic version, the differences between these two distributions will only grow from this list. Open Distro has had two patch releases since the fork of Elasticsearch and Kibana. There are no significant differences added other than some bug fixes in the distribution.

Elastic has had three minor version releases since removing its open-source license. There have been many enhancements, new features, and bug fixes released in that time. In Elastic 7.11, developers released a beta version of schema on read, or schema discovery. Elastic calls these runtime fields, and they allow for the calculation of data on searching. In Elastic version 7.13.0, developers released a frozen storage tier for inexpensive data storage with a penalty of query time. For a complete list of features released on Elastic, see their release notes.

Alerting in Elastic and Open Distro

Alerting is present in Kibana and is used to send notifications to users on certain predetermined events. These alerts are critical for knowing when there has been an attack on your cluster when your cluster is not behaving optimally or on a fixed interval.

Open Distro allows users to configure monitors and send alerts using their choice of communication. Users can set up triggers for their alerts using visual graphs in Kibana, using an extraction query or the anomaly detector. In each case, the user must provide some custom setup for the trigger.

Elastic alerts run on a schedule and check that certain conditions are met before taking action. Only basic Kibana alerts are available in the free tier version using the Elastic License. All other alerting, such as anomaly detection, is behind the paywall.

Other third-party services are available with alerting capabilities as well as anomaly detection. Coralogix proves user-defined alerts that can be used with either distribution.

Summary

Earlier this year, Elastic shifted to a source-open licensing model angering the open-source community. As a result of this shift, AWS will now maintain an open-source distribution of Elasticsearch and Kibana called Open Distro. From version 7.10, the two distributions are essentially the same except for paid vs. free functionality. The divergence in functionality only started after the license change applied in Elastic, starting with version 7.11.

Only time will tell which distribution will win out in the market for search and analytics functionality. For now, it seems that Elastic will be faster at delivering new features and enhancements. Open Distro, being relatively new on the scene, will take longer to ramp up and start delivering new features. Open Distro has replicated some of the paid features in Elastic in an Open Source version.

If you are starting a new project, consider using Open Distro since the genuine open-source license means more freedom to make source code changes. As well, features are all available for use and not blocked by subscription layers. Elastic will have more features to start, but once AWS ramps up development, they will be able to push out functionality quickly. For features missing from Open Distro or behind a paywall in Elastic, consider third-party tools like Coralogix’s Cloud Security Platform or their Log Analytics Platform.