Why I Built Parquet Data

Scott Paulin — Sat, 01 Jul 2023 08:57:23 +0000

Parquet is my favourite format for storing tabular data.

Parquet compresses well, it has a strong schema, and it's efficient to analyze. These are all things that CSV, the defacto standard for storing tabular data, lacks.

For all of the benefits of Parquet, there are some clear downsides.

Parquet is a binary format. To see inside a Parquet file you need to either write some code to parse it, or use a specialized Parquet Viewer.

Without a Parquet Viewer, the easiest way to look inside a Parquet file is to convert to CSV and then open in a regular text editor or Excel.

CSV files are interoperable, they are understood by many kinds of software, but they are harder to analyse than Parquet files. Plus many text editors struggle to load a CSV file bigger than 20MB. It would be great to keep data in Parquet format while doing analysis and debugging.

I built Parquet Data because I love the Parquet format, but had trouble working with it. I made the Parquet Viewer and converters to make it easier to debug some code that I was working on.

Data Science in your browser - introducing We Do Data Science

Scott Paulin — Mon, 17 Oct 2022 22:03:34 +0000

I love writing code, but it got painful starting a new software project every time I wanted to make some graphs or do analysis.

I tried some free online tools, but couldn't find anything I really liked.

Some even wanted me to give them my phone number to start a free account.

Protip: to make your number believable, just input your actual phone number with 1 or 2 numbers changed (also works great in bars).

One day I even tried downloading Tableau. 20 minutes later and 2 email confirmations later I was messaging support to cancel my account (they don't let you do this by yourself).

So I created We Do Data Science

We Do Data Science lets you make graphs in your browser.

My personal favourite are Nightingale graphs. I never knew about them until I started making this website.

When I make graphs sometimes I also want to do data analysis. Like performing regression or calculating correlations.

So I built regression calculators

and correlation calculators

I never knew there were different kinds of correlation until I made this site, cool huh.

When I work with data sometimes I need to convert file types. There are lots of great conversion tools online, but none of them converted parquet files.

So I made a csv to parquet converter and a parquet to csv converter.

I even got interested in statistics and made some statistics calculators

Anyway. If you do try out We Do Data Science then I hope you enjoy using it as much as I do.

If something isn't working, or you want to get in touch then you can find me on Twitter with the handle scottgpaulin.

DEV Community: Scott Paulin

Why I Built Parquet Data

Data Science in your browser - introducing We Do Data Science