Creating a QuantifiedSelf Dashboard - Part 1

undersilentskies — Mon, 20 Jul 2026 18:35:22 +0000

Hello DEV.to! I've been lurking this site for years but I hadn't felt like I had anything to share until now. You see...

The problem

I've recently realized I have a problem we're all very familiar with: years of disconnected data with no central hub. What brought this to my attention was trying to visualize my health data in one place. I have years of phone step counts, months of data from a previous fitness band, lab reports, etc...

Most people don't think this is a problem but I would like it normalized.

Most of us change phones and simply throw away the data we didn't think was important. Once I noticed this problem I started grieving for all the information I lost every time I bought a new phone and only backed up pictures when I could have a very clear historical timeline of my physical activity. And it's not just health data, but music, books, screentime, coding stats, etc. This matters to me more than I was expecting.

So I came across the quantifiedself community on reddit and let's just say the rest is history. I've spent weeks trying to bring my data from all my sources into one hub, which led me to finally learning SQL properly, to learning about different databases I hadn't heard of, to trying out different visualization apps. All of this hoping to figure out whether there's a correlation between screentime, food intake, exercise, sleep and the chronic pain condition I've been trying to get diagnosed for almost 20 years. I think I'm finally onto something.

So far I've tried the following

Using Apple Health as a wearables hub, then syncing to Guava for lab reports and symptoms. this provides no integration for other types of data and ignores my previous wearables' raw data because AH can't pull it.
Parsing the GDPR exports for my Xiaomi Fit band myself and then adding to AH. Xiaomi exports are notoriously difficult to use.
Using DuckDB as a final database source. This became more promising, as DuckDB is perfect for taking data from different sources and pulling it into one final database.
Asking Gemini to parse the Fit data. Gemini isn't very smart but it did successfully help me turn it into a DuckDB table! We're getting somewhere.
Visualizing it through Grafana. This was the first time I actually felt like I was accomplishing something because I could visualize the data I had pulled with DuckDB. I don't like Grafana though and would rather code the logic and graphics myself.
Using Datasette with Dogsheep. I'm not too sure what I hoped to achieve with this, but I'm in the process of pulling my deleted Twitter account data through WaybackTweets which I found through a Dogsheep-related site.
Using Evidence.dev for visualization. This is what I've settled for. I'm an Obsidian lover, so the idea of using Markdown to design the dashboard was immediately appealing to me. Furthermore, I think my favorite part about it is how scalable the system is. I can have different connectors from different data sources, use source queries to organize my data and then graph it with ECharts which gives me full control over it. And it uses my beloved DuckDB dialect!

I also realized I've been doing everything wrong. I spent weeks trying to convert my data into the format I wanted, molding my final structure around the data I have instead of doing the opposite. I read an O'Reilly book about data engineering a few years ago and it's been in the back of my mind ever since. Isn't it better to create an ideal schema first and then convert your data to fit that? So this is what I'm doing from now on, and I'll be documenting it here to keep myself accountable!

Creating a plan

Before thinking about how I'll be pulling and transforming data, I'd like to take a data-engineering approach. First I'll be designing the system I want to store my data, which means I have to choose what data is important enough to visualize. To keep things simple, I'll only be selecting very basic information to work with for now. But the good thing about this approach is that we can always add more data later. I have my data sources, but I'll be designing the structure first, including creating the charts I'll use to visualize later. This means I'll just generate some mock data to graph, and once I have a solid schema, I'll bring in my own data and figure out how to transform each data source.

Health & Fitness

Sources I'm pulling from: Guava, Apple Health, MiFit, Honor Health, DailyNomie (for symptom tracking & sleep quality perception)