Nowadays, I see many devs write about "Cloud-native" software projects like Kubernetes, Prometheus, Containerd, Envoy. But there are a few other similar OSS foundation backed projects which are excellent too.
This post is about three such projects from "Apache Software Foundation (ASF)" that one should take a look at.
ASF is a very old Opensource organization which has been the home for projects like Hadoop, Zookeeper, Kafka, Lucene. If you are interested in contributing to OSS, this might be the best place to start too.
1. Apache Airflow
Apache Airflow is an open-source workflow management platform started at Airbnb. It essentially helps with many operations in a software project. It can monitor cron jobs, manage data pipelines. It generates a Directed Acyclic Graph (DAG) visualization diagram. But, it is not a streaming or ETL solution.
Cloud Platforms like GCP has Cloud Composer, AWS has Managed Workflows - which are managed SaaS versions of Airflow.
2. Apache Beam
Apache Beam is a combination of the Batch and Streaming model to design and develop data processing pipelines. There are Java, Python, Go SDKs available for Apache Beam. It also supports backends like Apache Flink, Apache Spark, and GCP Dataflow.
3. Apache Pulsar
Apache Pulsar is popular is a cloud-native, distributed messaging and streaming platform created at Yahoo. You might have heard it while researching the famous OSS message queue Kafka. Born in the cloud-native world, Pulsar can be run on Docker or Kubernetes. Pulsar has built-in connectors to MongoDB, Elasticsearch, PostgreSQL, Redis.
I believe these projects are interesting to learn and try even on a side-project. I might have missed several other interesting projects or incubating one's. If you are using or have used something, write them in the comments.
--
Stay Safe
Aravind Putrevu
Top comments (0)