The latest articles on DEV Community by Josh Liburdi (@jshlbrd). https://dev.to/jshlbrd https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1008470%2F9ee33455-5b78-45cf-a33c-08a169b204df.jpeg https://dev.to/jshlbrd en Josh Liburdi Mon, 23 Jan 2023 16:10:00 +0000 https://dev.to/jshlbrd/substation-data-pipeline-and-transformation-toolkit-written-in-go-2dh8 https://dev.to/jshlbrd/substation-data-pipeline-and-transformation-toolkit-written-in-go-2dh8 <p>Hello fellow Gophers, data enthusiasts, and security engineers! I'm excited to share a project that I've been leading for multiple years with the DEV community: <a href="https://github.com/brexhq/substation">Substation is an open-source data pipeline and transformation toolkit written in Go</a>.</p> <p>With Substation you can:</p> <ul> <li>Deploy serverless data pipelines or microservices to AWS in minutes using infrastructure as code (Terraform)</li> <li>Evaluate, process, and deliver data in real-time using configurations as code (Jsonnet)</li> <li>Create custom data processing applications written in Go</li> </ul> <p>Substation is an affordable alternative to products like <a href="https://cribl.io/">Cribl</a> (~10x cost savings) and is easier to manage than similar open-source projects such as <a href="https://www.elastic.co/logstash/">Logstash</a> and <a href="https://www.fluentd.org/">fluentd</a>. It's been used in production by the security team at Brex for 2+ years and is ready for any scale, even beyond 100,000 events per second!</p> <p>If you'd like to learn more, then please <a href="https://substation.readme.io/docs">read the documentation</a>, <a href="https://substation.readme.io/recipes">try the recipes</a>, and let me know if you have any questions!</p> go aws security opensource