VisiData for Exploratory DevOps
The folks behind VisiData recently released version 2.0.
VisiData is a tool I wish I found sooner - a tool for quickly exploring many types of files from your terminal. To celebrate its 2.0 release I want to help show some ways it's helped me. Many of VisiData's uses are well-covered by tutorials, video case studies and documentation. I'm not a data scientist though, so I'll start somewhere totally different. Let's explore S3!
Exploring S3 from the Terminal
I often want to quickly peek at data files in S3. With VisiData I can browse a bucket in a way that recalls terminal file explorers like Midnight Commander. I can open files (CSV, deeply nested JSON, Parquet...) directly as new sheets in VisiData's tabular interface.
Here's a demo that uses VisiData to explore weather data in a versioned S3 bucket:
Extensible with Python
Most of the functionality in the demo above comes out of the box with VisiData. The only work I needed to do was create a plugin that tells VisiData how to speak S3. And to be fair, that's mostly a wrapper around the excellent s3fs library! VisiData 2.0 includes a Plugin Author's Guide that can help you extend VisiData as you see fit.
VisiData as a Piece of the Puzzle
Sometimes if I'm on a screenshare, people notice VisiData and ask about it. The strange part is that as much as I love VisiData, it doesn't actually replace other tools I use. It's replaced specific uses for sure...
- jq: I still use and love it, but VisiData feels smoother for interactive exploration.
- glom: Rather than inspecting/exploring data with glom first, I explore with VisiData and potentially use glom to define repeatable transforms.
VisiData also works very well as a side "quick view" tool to consume output from other tools I use regularly:
-
Terraform: Add a few lines of Python to your
~/.visidatarc
file to automatically open .tfstate files as JSON. - Cloud Custodian: Open local policy output files or use S3 glob matching to check output for a range of dates.
-
AWS CLI: Pipe its JSON output to
vd -f json
.
Some of those use cases deserve their own post! The gist is that if you feel at home in the terminal and need a tool for ad-hoc data exploration and analysis... VisiData is probably worth a peek. It'll do as much or as little as you need, and let you get on with your life.
Top comments (0)