The latest articles on DEV Community by Sohil Shah (@bugsbunnyshah). https://dev.to/bugsbunnyshah https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1423431%2F93ec97d3-0e34-48a8-8090-d8fe976e06c9.jpeg https://dev.to/bugsbunnyshah en Sohil Shah Sun, 25 Aug 2024 17:40:11 +0000 https://dev.to/bugsbunnyshah/data-pipeline-techniques-in-action-4m9o https://dev.to/bugsbunnyshah/data-pipeline-techniques-in-action-4m9o <p>Take a deep dive into the architectural concepts of data pipelines along with a hands-on tutorial for implementation, demonstrating the concepts in action.</p> <p>The topics covered are:</p> <ul> <li><p>Data pipeline architecture</p></li> <li><p>High-scale data ingestion</p></li> <li><p>Data transformation and processing</p></li> <li><p>Data storage</p></li> <li><p>Staging data delivery</p></li> <li><p>Operational data</p></li> <li><p>Hands-on exercise</p></li> </ul> <p>Article : <a href="https://dzone.com/articles/data-pipeline-techniques-in-action" rel="noopener noreferrer">https://dzone.com/articles/data-pipeline-techniques-in-action</a></p> <p><a href="https://media.dev.to/cdn-cgi/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1fpd1jnq77jv68blzgkk.png" class="article-body-image-wrapper"><img src="https://media.dev.to/cdn-cgi/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1fpd1jnq77jv68blzgkk.png" alt="Data Pipeline Architecture" width="771" height="441"></a></p> opensource datascience softwaredevelopment dataengineering Sohil Shah Sat, 13 Apr 2024 22:41:21 +0000 https://dev.to/bugsbunnyshah/open-source-high-scale-data-pipeline-platform-for-enterprise-data-analytics-and-machine-learning-applications-ld0 https://dev.to/bugsbunnyshah/open-source-high-scale-data-pipeline-platform-for-enterprise-data-analytics-and-machine-learning-applications-ld0 <p>Braineous is designed for an optimal out-of-the-box experience for developers focused on <strong>ETL</strong>, <strong>ELT</strong>, <strong>Analytics</strong> and <strong>Machine Learning</strong>.</p> <p><strong>Documentation</strong>: <a href="https://bugsbunnyshah.github.io/braineous/guides/developer-guide">https://bugsbunnyshah.github.io/braineous/guides/developer-guide</a></p> <p><strong>Get Started</strong>: <a href="https://bugsbunnyshah.github.io/braineous/get-started/">https://bugsbunnyshah.github.io/braineous/get-started/</a></p> <p><strong>GitHub</strong>: <a href="https://github.com/bugsbunnyshah/braineous_dataplatform">https://github.com/bugsbunnyshah/braineous_dataplatform</a></p> <p><strong>License</strong>: <a href="https://github.com/bugsbunnyshah/braineous_dataplatform/blob/main/LICENSE">https://github.com/bugsbunnyshah/braineous_dataplatform/blob/main/LICENSE</a></p> <p><strong>Roadmap</strong>: <a href="https://bugsbunnyshah.github.io/braineous/about/">https://bugsbunnyshah.github.io/braineous/about/</a></p> <p><strong>Apache Kafka</strong> is the backbone for high scale data ingestion and maintenance of source of data truth and in the future for CDC and time travel for a system in the past and training AI models for predictive analytics.</p> <p><strong>More details</strong>: <a href="https://bugsbunnyshah.github.io/braineous/container-first/">https://bugsbunnyshah.github.io/braineous/container-first/</a></p> <p>The downstream engine is <strong>Apache Flink</strong>. If <strong>Apache Flink</strong> is the brain, then <strong>Apache Kafka</strong> is the spinal chord. A biological analogy.</p> <p><strong>More details</strong>: <a href="https://bugsbunnyshah.github.io/braineous/about/">https://bugsbunnyshah.github.io/braineous/about/</a></p> <p>Braineous is built on <strong>Apache Flink</strong> as its data processing engine and supports <strong>Apache Hive</strong> based data lakes.</p> <p><strong>Future releases</strong> of Braineous will include a <strong>Data Lake Connector framework</strong> that can support custom data lakes.</p> <p><strong>More details</strong>: <a href="https://bugsbunnyshah.github.io/braineous/data-lake/">https://bugsbunnyshah.github.io/braineous/data-lake/</a></p> <p>Braineous bridges the unstructured dataset to the structured dataset on the fly. Your data lake evolves with the dataset. <strong>Analytics</strong> and <strong>Machine Learning</strong> need structured queries for training the AI model.</p> <p>Braineous bridges two Worlds on the fly. Downtime is a time that is entirely unacceptable for Braineous.</p> <p><strong>More details:</strong> <a href="https://bugsbunnyshah.github.io/braineous/developer-joy/">https://bugsbunnyshah.github.io/braineous/developer-joy/</a></p> <p>We would love your feedback when it comes to developer experience, ease of use, and ability to go from 0 to 60 in 15 minutes when it comes to data processing. </p> <p>Developer input would be valuable to shape the roadmap.</p> <p><strong>Feedback</strong>: <a href="https://github.com/bugsbunnyshah/braineous_dataplatform/discussions/16">https://github.com/bugsbunnyshah/braineous_dataplatform/discussions/16</a></p> <p>Thanks</p> <p>Sohil</p> etl elt machinelearning ai