The latest articles on DEV Community by Paola Pardo (@paolapardo). https://dev.to/paolapardo https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F726504%2Ffa5829f5-eacd-476a-97fe-95624706e277.jpg https://dev.to/paolapardo en Paola Pardo Wed, 09 Mar 2022 15:23:23 +0000 https://dev.to/paolapardo/datasource-enabling-multidimensional-indexing-and-sampling-pushdown-29ca https://dev.to/paolapardo/datasource-enabling-multidimensional-indexing-and-sampling-pushdown-29ca <p>Do you wonder how a multidimensional index would look like in Spark? </p> <p>Recently we launched the Qbeast Open Source Format, a Data Lakehouse enhancement to speed up your queries!</p> <p>Based on <a href="https://delta.io/">Delta Lake</a> and available for <a href="https://spark.apache.org/">Apache Spark</a>, it allows indexing your data by different columns and read a representative sample directly from storage 🔥</p> <p>Quick example of how you can boost your query performance using Qbeast:</p> <p><strong>This is a Normal Query with Spark and Delta format.</strong><br> <a href="https://res.cloudinary.com/practicaldev/image/fetch/s--MBBDVg2c--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bdc1ngafo9jh6s740x33.gif" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--MBBDVg2c--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bdc1ngafo9jh6s740x33.gif" alt="Normal query" width="666" height="470"></a></p> <p><strong>This is the same query but with Qbeast Sampling of 1%</strong><br> <a href="https://res.cloudinary.com/practicaldev/image/fetch/s--azfeBxsx--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ln8duog3i6fzlx3moznv.gif" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--azfeBxsx--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ln8duog3i6fzlx3moznv.gif" alt="Qbeast Sample Query" width="666" height="470"></a></p> <p>The gifs are cool, right? Let's compare both executions:</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Format</th> <th>Execution Time</th> <th>Result</th> </tr> </thead> <tbody> <tr> <td>Delta</td> <td>~ 2.5 min.</td> <td>37.869383</td> </tr> <tr> <td>Qbeast</td> <td>~ 6.6 sec.</td> <td>37.856333</td> </tr> </tbody> </table></div> <p>As you can see, <strong>1% sampling</strong> provides the result <strong>x22 times faster</strong> compared to using Delta format, with an <strong>error of 0,034%</strong>.</p> <p>If you want to play with it, check out the <a href="https://github.com/Qbeast-io/qbeast-spark">Qbeast-Spark github</a></p> <p>And don't forget to give us a star!</p> <p>Your support means a lot ❤️</p> opensource datascience news github