The latest articles on DEV Community by CrabCanneryShip (@crabcanneryship). https://dev.to/crabcanneryship https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4151297%2F99716f6a-a7c0-4361-b093-f7908887da2a.jpeg https://dev.to/crabcanneryship en CrabCanneryShip Wed, 30 Sep 2026 05:49:05 +0000 https://dev.to/crabcanneryship/why-i-built-my-cloud-native-dfir-pipeline-and-ditched-awsopensearch-for-bigquery-ajg https://dev.to/crabcanneryship/why-i-built-my-cloud-native-dfir-pipeline-and-ditched-awsopensearch-for-bigquery-ajg <p>There are fantastic, battle-tested tools out there like KAPE and CDIR. But as a DFIR engineer, I wanted something more lightweight, highly customizable, and automated from the ground up to fit my own workflow. So, I built a fast forensics pipeline: <strong>Veloxamen</strong>.</p> <h4> The Pain Points: Why Not AWS? </h4> <p>I initially looked into building this on AWS. While Timesketch is hard to let go of, getting OpenSearch ingestion to feel <em>just right</em> on AWS always felt heavier and clunky than it should be. </p> <p>Then, I shifted my focus to GCP. The scalability and sheer convenience of <strong>BigQuery</strong> for structured log analysis completely won me over. </p> <h4> What Veloxamen Does </h4> <p>Veloxamen is designed to automatically ingest and process forensic artifacts in GCP, transforming them into a structured, queryable timeline inside BigQuery. </p> <ul> <li> <strong>Collector + Pipeline:</strong> Combined with a custom collector, it handles the heavy lifting.</li> <li> <strong>Windows First (For Now):</strong> Right now, it focuses heavily on Windows artifacts (since attackers love targeting them), but the architecture is fully extensible. Drop your logs into the staging bucket with a clean prefix, and anything supported gets automatically transformed into a unified timeline.</li> </ul> <h4> Evolutional Improvements: What's Next? </h4> <p>BigQuery is just the baseline. Because all the parsed forensic timelines live cleanly in BigQuery, the path forward opens up to some serious evolution:</p> <ul> <li> <strong>Microservices Transformation</strong>: While Log2Timeline/Plaso is undeniably powerful and sophisticated, managing and scaling its compute resources can be exhausting. It will be replaced with a highly concurrent microservices architecture for lightning-fast artifact processing.</li> <li> <strong>Looker / Looker Studio Integration:</strong> Instant visual dashboards and interactive hunting views without wrestling with heavy legacy SIEM UIs.</li> <li> <strong>Vertex AI-Powered Analysis:</strong> Leveraging LLMs and ML models directly over BigQuery data to automate anomaly detection, summarize event horizons, and speed up triage reporting.</li> </ul> <p>You can check out the source code and architecture here:<br> 🔗 <strong>GitHub:</strong> <a href="https://github.com/veloxamen" rel="noopener noreferrer">https://github.com/veloxamen</a></p> <p>I'm currently opening it up to the global community. Feedback, issues, or thoughts from fellow DFIR/cloud security folks are more than welcome!</p> dfir security go googlecloud