The latest articles on DEV Community by DoubleCloud (@doublecloud). https://dev.to/doublecloud https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1193613%2F6dd45553-fe5c-4151-ba52-0d9fe47a4304.jpeg https://dev.to/doublecloud en DoubleCloud Wed, 25 Oct 2023 10:26:07 +0000 https://dev.to/doublecloud/were-rolling-out-managed-apache-airflow-1bp8 https://dev.to/doublecloud/were-rolling-out-managed-apache-airflow-1bp8 <p>Hey everyone, it's DoubleCloud. We've recently rolled out our <a href="https://double.cloud/services/managed-airflow/?utm-source=reddit&amp;utm-medium=referral&amp;utm-content=airflow-release">Managed Apache Airflow</a>, and I'm excited to dive into its nitty-gritty with you all.</p> <p>For us, Apache Airflow has transformed the way we handle intricate data pipelines. It's like equipping our operational processes with a turbo boost. But what makes our Managed Airflow stand out?</p> <p>Infrastructure Management: Say goodbye to manual setups. We've automated cluster creation and updates, ensuring a seamless and consistent deployment every time. This means less manual intervention.</p> <p>Auto-Scaling Worker Nodes: No more guesswork. Our nodes dynamically adjust to the workload, balancing performance and resource allocation.</p> <p>Monitoring Capabilities: The UI offers a transparent view into Airflow processes. It's not just about aesthetics; it's about delivering actionable insights through logs and notifications.</p> <p>DAG Development: We've integrated common libraries and operators, including DBT. This isn't just a convenience feature; it's about ensuring compatibility and reducing the learning curve.</p> <p>Security Measures: Our setup places Airflow Workers and Schedulers in a secured VPC. We've also implemented access controls, ensuring that only authorized personnel can make changes.</p> <p>GIT Integration: For developers, we've streamlined the process of provisioning existing DAGs. It's a direct integration with your GIT environment, ensuring continuity and version control.</p> <p>Upcoming Custom Docker Feature: This will allow to build your own deployment of Airflow. With the custom Docker image, you can have greater flexibility and control over the specific tools and libraries. </p> <p>I'm genuinely interested in any feedback or insights you might have, especially from a technical standpoint. If you've worked with Apache Airflow or have thoughts on workflow management, let's talk!</p> <h1> managed #airflow #data #doublecloud #datamanagement </h1>