January 30th, 2023 - Instalment #143
Welcome to edition #143 of the AWS open source newsletter, and another week of great new open source projects for you to try out. This week we feature projects including "aws-cdk-in-electron", a project that lets you put AWS CDK in a graphical user interface, "lightsail-k8s-installer" that helps you deploy Kubernetes into Amazon Lightsail, "porting-advisor-for-graviton" a great project to help you migrate to Arm based AWS Graviton instance types, "aws-ebook-downloader" a browser tool to help you easily download pdf's on AWS topics, "lake-formation-permissions-sync" a useful tool to help you keep on top of your Lake Formation setups, and many more.
For those who prefer reading up on the latest open source topics, this week we have content on some of your favourite open source projects such as MySQL, PostgreSQL, MariaDB, OpenSearch, RabbitMQ, Apache Flink, Apache Kafka, AWS Distro for OpenTelemetry, Prometheus, OpenShift, DataHub, Apache Iceberg, and many more. Don't forget to check out the videos and events section at the end too, as there are some great videos this week that you do not want to miss.
Raffle prize winner
No one has claimed the winning ticket - it is out there somewhere. I will post this number again this week, and if I get no takers, I will publish another ticket.
Before Christmas we had a raffle to win some cool AWS SWAG. I can now reveal that the winning ticket is 038C123F-DA73-4584-ABE2- - if this matches your ticket, then please contact me at ricsue@amazon.com with the LAST portion of the ticket. If yours matches what I have, you are a winner!
Celebrating open source contributors
The articles and projects shared in this newsletter are only possible thanks to the many contributors in open source. I would like to shout out and thank those folks who really do power open source and enable us all to learn and build on top of what they have created.
So thank you to the following open source heroes: Rio Astamal, Yankee Maharjan, Praneeta Prakas, Mohamed ElAsmar, Julian Wood, Francisco Losada, Brandon Minnick, Anand Shah, John Preston, Sebastian Bille, Rafael Pereyra, James McIntyre, Eli Fisher, David Tippett, Pavani Baddepudi, Gary Stafford, Mark Sailes, Maciej Walkowiak, Serkan Ozal, Vadym Kazulkin, and Goran Opacic Esteh
Latest open source projects
The great thing about open source projects is that you can review the source code. If you like the look of these projects, make sure you that take a look at the code, and if it is useful to you, get in touch with the maintainer to provide feedback, suggestions or even submit a contribution.
Tools
porting-advisor-for-graviton
porting-advisor-for-graviton is a fork of Porting advisor, an open source project by the ARM High Performance Computing group. Originally, it was coded as a Python module that analysed some known incompatibilities for C and Fortran code. It is a command line tool that analyses source code for known code patterns and dependency libraries. It then generates a report with any incompatibilities with our Graviton processors. This tool provides suggestions of minimal required and/or recommended versions to run on Graviton instances for both language runtime and dependency libraries. It can run on non-ARM based machines (no Graviton processor needed). This tool does not work on binaries, just source code. The tool supports C/C++, Fortran, Go 1.11+, Java 8+, and Python 3+. Very nice indeed, and this weeks must check out project. Dive into the detailed README for more details.
iam-roles-anywhere-session
iam-roles-anywhere-session provides an easy way to create a refreshable boto3 Session with AWS Roles Anywhere. You can use AWS Roles Anywhere to obtain temporary security credentials in IAM for workloads such as servers, containers, and applications that run outside of AWS. Read more about this here, What is AWS Identity and Access Management Roles Anywhere?
aws-cdk-in-electron
aws-cdk-in-electron This project demonstrates how you can deploy CDK projects, and run SDK commands from within an Electron app. The app can contain all the dependencies the end user needs, meaning they will not have to install AWS SDK for JavaScript, AWS CDK, or use the command line. This sample project can be built as a standalone app that is able to both query and deploy changes to an account. What the standalone app is able to do is dependant on the permissions you give it when you configure it with credentials. Very cool.
lake-formation-permissions-sync
lake-formation-permissions-sync is a utility developed to create alternate backup of Glue Catalog objects and LakeFormation permissions and replicate to a target region. It supports both batch and realtime modes of operation, so check out the docs for more details on how these work.
leverage
leverage is an open source tool that helps you easily deploy ready made AWS Cloud infrastructure based on the AWS Well Architected Framework provided by Binbash. Their mission is to simplify the process of building, and deploying software in the Cloud. You can find out more about how they have laid out their open source projects by reading a post they put together, Open source Github repository pre-launch checklist
lightsail-k8s-installer
lightsail-k8s-installer is a shell script from AWS Developer Advocate Rio Astamal that simplifies the process of setting up a Kubernetes cluster on Amazon Lightsail. Not only it automates the creation of necessary resources using AWS CloudFormation, but it also automates the installation of Kubernetes packages. With just one simple command, your Kubernetes cluster should up and running in no time!
aws-ebook-downloader
aws-ebook-downloader is a handy Chrome extension from Yankee Maharjan that helps you easily download AWS documentation from the AWS Resource Hub. Check out this short video he has linked in the repo to see it in action.
Demos, Samples, Solutions and Workshops
transactional-datalake-using-apache-iceberg-on-aws-glue
transactional-datalake-using-apache-iceberg-on-aws-glue This repository provides you cdk scripts and sample code on how to implement end to end pipeline for transactional data lake by ingesting stream change data capture (CDC) from MySQL DB to Amazon S3 in Apache Iceberg format through Amazon Kinesis using Amazon Data Migration Service(DMS) and Glue Streaming.
aws-glue-streaming-etl-with-apache-iceberg
aws-glue-streaming-etl-with-apache-iceberg this project helps you to buid a streaming ETL job in AWS Glue to integrate Iceberg with a streaming use case and create an in-place updatable data lake on Amazon S3. AWS CDK scripts are provided to help simplify the installation and configuration, and after ingested to Amazon S3, you can query the data with Amazon Athena.
amazon-opensearch-batch-indexing-with-aws-lambda
amazon-opensearch-batch-indexing-with-aws-lambda This repository provides guidance on how to use the Amazon OpenSearch Python client to perform batch operations using Lambda functions and how to architect your solution using a data lake architecture. You can read the supporting blog post, Text analytics on AWS: implementing a data lake architecture with OpenSearch, where Francisco Losada provides an overview of the architecture and dives deeper into how this project works.
LambdaTriggersSample
LambdaTriggersSample if you are looking for a sample .NET app demonstrating AWS Lambda triggers, then Brandon Minnick has you covered. This is a sample app demonstrating an end-to-end mobile workflow using .NET MAUI, + Serverless AWS Lambda + AWS S3 Storage in C#.
aws-mwaa-openlineage
aws-mwaa-openlineage this repo contains all the code from the post I shared in last weeks newsletter, #142 Automate data lineage on Amazon MWAA with OpenLineage. This code will help you get started with how to integrate Marquez and OpenLineage with Apache Airflow.
AWS and Community blog posts
Apache Flink
In the post Automate deployment and version updates for Amazon Kinesis Data Analytics applications with AWS CodePipeline, Anand Shah show how you can automate deployment and version updates for Kinesis Data Analytics applications that enables better collaboration with your Platform and engineering teams. [hands on]
We also had this post, Exploring Apache Flink & AWS KDA: Realtime data streaming where engineers at Capitol One share their experience building realtime data streaming apps with Apache Flink and AWS Kinesis Data Analytics. [hands on]
Apache Kafka
AWS Community Builder and Apache Kafka wizard John Preston has a great post for you this week. Using an open source tool that John created and we have featured on the AWS open source blog, you can speed up and simplify how you deploy and integrate Apache Kafka and Conduktor in as little as three commands. Don't believe me? Well I will hand you over to John in hist post, Deploy Conduktor & a MSK Cluster in 3 commands. [hands on]
Open Source Serverless frameworks
With a number of open source serverless frameworks available for developers to use, which one should you use? AWS Community Builder Sebastian Bille has just what you need to help you explore this area in his post, Serverless Frameworks for 2023. Settle in and grab a cup of your favourite warm beverage whilst he takes you on a serverless framework journey.
AWS Distro for OpenTelemetry
Rafael Pereyra has put together, Using AWS Distro for OpenTelemetry and IAM Roles Anywhere on-premises to ingest metrics into Amazon Managed Service for Prometheus. In it he shows how to programmatically access your AWS resources running in your on-premises using IAM Roles Anywhere. IAM Roles Anywhere allows your workloads such as servers, containers, and applications to use X.509 digital certificates to obtain temporary AWS credentials and use the same IAM roles and policies that you have configured for your AWS workloads to access AWS resources. Very nice post. [hands on]
Other posts and quick reads
- Scaling container workloads with shared storage for Red Hat OpenShift Service on AWS demonstrates the use of FSx for ONTAP as a persistent storage layer for Red Hat OpenShift Service on AWS (ROSA) applications [hands on]
- 5G Core implementation on Amazon Elastic Kubernetes Service Anywhere on bare metal looks at how customers can utilise Amazon Elastic Kubernetes Service Anywhere (Amazon EKS-A) to automate the deployment of 5G Core on their customer-managed on-premises infrastructure on bare metal [hands on]
- Leveraging Slurm Accounting in AWS ParallelCluster shows how to use new features in AWS ParallelCluster 3.3.0 that enables you to configure and enable Slurm accounting for you automatically [hands on]
- Best practices for working with the Apache Velocity Template Language in Amazon API Gateway discusses best practices for using Apache Velocity Templates for direct service integration in API Gateway
- Enhancing IoT device security using Hardware Security Modules and AWS IoT Device SDK walks you through some notable examples of security challenges across IoT Industry verticals [hands on]
- Four key application protocols for the internet of things (IoT) explores the application protocols that are best suited for internet data messaging, focusing on criteria such as range, availability, maturity, and market segment applicability
Quick updates
OpenSearch
We have a few updates for OpenSearch users and developers.
First up, check out OpenSearch Project 2022 recap and what's next where Eli Fisher and David Tippett look back at the key accomplishments in 2022 and then looks ahead to what you can expect this year.
Also announced last week was the availability of OpenSearch 2.5. James McIntyre shares more info in his post, OpenSearch 2.5 is live!
Finally, for those moving from self managed to managed OpenSearch, Amazon OpenSearch Serverless is now generally available. You can check out the announcement, Amazon OpenSearch Serverless is now generally available, for more details and read the post, Amazon OpenSearch Serverless is now generally available! where Pavani Baddepudi shares the design and high-level architecture of OpenSearch Serverless.
MariaDB
Amazon Relational Database Service (Amazon RDS) for MariaDB supports encrypted SSL/TLS connections to the database instances. Starting today, you can enforce SSL/TLS client connections to your Amazon RDS for MariaDB database instance for enhanced transport layer security. To enforce SSL/TLS, enable the require_secure_transport parameter (disabled by default) through the Amazon RDS Management Console, the AWS CLI or the API. When the require_secure_transport parameter is enabled, a database client will be able to connect to the RDS for MariaDB instance only if it can establish an encrypted connection. require_secure_transport parameter is supported on RDS for MariaDB versions 10.5 and higher.
PostgreSQL
A couple of updates this week.
First, following the announcement of updates to the PostgreSQL database by the open source community, we have updated Amazon Aurora PostgreSQL-Compatible Edition to support PostgreSQL 14.6, 13.9, 12.13, and 11.18.
Also announced last week was that Amazon RDS for PostgreSQL now supports inbound replication from Amazon RDS Single-AZ database (DB) instances and Amazon RDS Multi-AZ DB instances with one standby to Amazon RDS Multi-AZ deployments with two readable standbys. You can use this inbound replication to help migrate your existing Amazon RDS PostgreSQL deployments to Amazon RDS Multi-AZ deployments with two readable standbys, which have one writer instance and two readable standby instances across three availability zones. By creating a Multi-AZ deployment with two readable standbys as a read replica of your existing RDS PostgreSQL database instance, you can promote the read replica to be your new primary, typically within minutes.
Amazon RDS Multi-AZ deployments provide enhanced availability and durability, making them a natural fit for production database workloads. Deployment of Amazon RDS Multi-AZ with two readable standbys supports up to 2x faster transaction commit latencies than a Multi-AZ deployment with one standby instance. In this configuration, automated failovers typically take under 35 seconds. In addition, the two readable standbys can also serve read traffic without needing to attach additional read replicas.
MySQL
Amazon Aurora now supports Aurora MySQL 3 (with MySQL 8.0 compatibility) as a source cluster or blue environment within Amazon RDS Blue/Green Deployments. This enables you to use Blue/Green Deployments for minor version upgrades for Aurora MySQL 3 (with MySQL 8.0 compatibility).
Read more about this in the announcement, Amazon RDS Blue/Green Deployments now supports Aurora MySQL 3 (with MySQL 8.0 compatibility) as a source cluster
RabbitMQ
Amazon MQ now provides support for RabbitMQ version 3.8.34, which includes several fixes to the previously supported version, RabbitMQ 3.8.30.
Videos of the week
DataHub
Check out the video version of Gary Stafford's post, DataHub on AWS: Data Discovery, Observability, and Governance on AWS with DataHub, the Open Source Data Catalog.
Java
A stellar cast of Java developers, Mark Sailes, Maciej Walkowiak, Serkan Ozal, Vadym Kazulkin, and Goran Opacic take a look at SnapStart and SpringCloud AWS. A must watch for all Java fans.
AWS SAM and Terraform
Praneeta Prakash, and software engineer, Mohamed ElAsmar, join Julian Wood to go through how you can now use the AWS SAM CLI together with Terraform configuration to develop and test serverless applications. See how you can speed up your local development workflow with testing and debugging. Find out how to contribute to help improve the local development process.
Build on Open Source
For those unfamiliar with this show, Build on Open Source is where we go over this newsletter and then invite special guests to dive deep into their open source project. Expect plenty of code, demos and hopefully laughs. We have put together a playlist so that you can easily access all (eight) of the episodes of the Build on Open Source show. Build on Open Source playlist
Events for your diary
If you are planning any events in 2023, either virtual, in person, or hybrid, get in touch as I would love to share details of your event with readers.
FOSSDEM
Feb 4-5th, 2023 in Brussels
FOSDEM is a free event for software developers to meet, share ideas and collaborate. Every year, thousands of developers of free and open source software from all over the world gather at the event in Brussels. 4 & 5 February 2023. A must attend event for all open source fans, check out this link. The event is free for anyone to attend.
State of Open Con 23
Feb 7-8th, 2023 in London
OpenUK will be hosting a 1000 person plus two day conference in Central London, “State of Open Con 23” in association with IEEE, the headline sponsor. Check out more info and sign up here.
PGConf India
Feb 22nd to 24th, Radisson Blu Bengaluru, India
If you are in or can get to Bengaluru, then checkout this conference for PostgreSQL developers and enthusiasts. Check out the session line up and get your tickets here.
Everything Open
March14-15th Melbourne, Australia
A new event for the fine folks in Australia. Everything Open is running for the first time, and the organisers (Linux Australia) have decided to run this event to provide a space for a cross-section of the open technologies communities to come together in person. Check out the event details here. The CFP us currently open, so why not take a look and submit something if you can.
OpenSearch
Every other Tuesday, 3pm GMT
This regular meet-up is for anyone interested in OpenSearch & Open Distro. All skill levels are welcome and they cover and welcome talks on topics including: search, logging, log analytics, and data visualisation.
Sign up to the next session, OpenSearch Community Meeting
Stay in touch with open source at AWS
I hope this summary has been useful. Remember to check out the Open Source homepage to keep up to date with all our activity in open source by following us on @AWSOpen
Top comments (2)
Thanks for adding aws-ebook-downloader. ✨
Exciting to see all these resources; really handy! ⚡️
BtW :