DEV Community: javi santana

[11/40 things about data] Learn SQL please

javi santana — Fri, 02 Sep 2022 12:18:21 +0000

You may not like SQL based databases but the probability of dealing with a SQL based system in your career is so high that learning it as soon as possible will compound.

I didn’t like SQL, I still don’t like it even though I work with it every single day but I have to recognize it’s a really handy tool.

[10/40 things about data] Try to use the simplest possible data structure

javi santana — Mon, 08 Aug 2022 16:57:25 +0000

A few years ago, one of the websites I was working on went on the front page of Google (yes, that small blue link). The traffic it gets is pretty high.

I had to develop search functionality. The first thing you’d think is to use the database you are currently using or maybe use a special one, like elastic.

But in this case, I needed to use the database as little as possible to be able to cope with the load.

So I decided to go the simplest way: create an index with an in-memory array where all the words would be stored. I ran a linear search, yes, a simple for loop with the search logic.

Was it the best index structure? No, if you just think about performance, but it worked, it was simple, easy to maintain and change.

There is always time to make it more advanced. With time you end up loving simple and flat arrays.

[9/40 things about data]It’s better to master just one database than be bad at two

javi santana — Wed, 03 Aug 2022 08:25:07 +0000

It’s tempting to start using another database when you run into a performance problem or the lack of a feature.

There are always ways to make it perform better or solve the problem with a workaround.

You’d be surprised how good your database can perform when you understand the internals. It’s not that bad to do that thing in two steps instead of one.

If you go after the shiny new thing just because you find a small roadblock, you’ll never understand the actual limits of your database and you may never know when there is a real reason to change.

[8/40 things about data] Analytics it’s a product, not a department

javi santana — Mon, 01 Aug 2022 18:12:54 +0000

When you have people asking for metrics and people extracting them from data. For the same metric, you’ll have as many definitions for a metric as people you have in the company.

Reporting is something that requires the same thing a digital product needs: owners, maintenance, clear definitions, improvements and you know, gives people what they want in a way that is useful for everybody in the company.

Many companies don’t consider analytics as a first-class citizen and end up spending more to have less quality.

[7/40 things about data] When I try to understand data I always end up using a histogram

javi santana — Fri, 29 Jul 2022 14:33:00 +0000

When visualizing data you have to pick the right visualization type but before that, you need to understand the data.

I start using an avg, then avg plus stddev, then min-max and finally I go with a histogram.

It captures min, max, avg and most importantly, the data distribution.

[6/40 things about data] Behind every null value there is a story

javi santana — Wed, 27 Jul 2022 06:49:16 +0000

When you join a company, just ask about it, you’ll learn a lot

[5/40 things about data] When in doubt, use Postgres as your database.

javi santana — Tue, 26 Jul 2022 15:27:00 +0000

It’s quite typical when you start a project to decide what DBMS to use. Elastic, Mongo, some key/value store like Redis, funny things like Neo4J. If you have a use case that clearly fits with a database, fine, otherwise, use Postgres or anything relational. Of course, there will be someone that says “but it does not scale”. Anyone who has worked with a system at scale knows there is no storage system that scales well (except if it’s as simple as hell and is eventually consistent, but not even then).

I love Postgres because of many things: solid, battle-tested, supports transactions (I will write about them), feature-complete, fast, it’s not owned by a VC backed company, guided by the community, calm and steady progress, great tooling, cloud services providing infra, companies with expertise…

When you pick something funny, you end up developing half of the features a solid RDBMS system provides but just worse.

I decided to use Redis as the storage for Tinybird and it’s working great but as the project evolves you miss many of the built-in features Postgres provides. Probably a mistake.

[4/40 things about data] The second most important rule of working with data: the fastest data is the one you don’t read

javi santana — Mon, 25 Jul 2022 08:44:00 +0000

As simple as it sounds, most people forget about using one of the most important database features: indexes. Well, you also need to think about what’s the data you actually need, a lot of apps are full of select * from table.

The problem is, as your system grows, so does the amount and complexity of queries. Knowing what data you need becomes harder. To avoid that you need… yes, data about how you query your data.

[3/40 things about data] Good data models make good products

javi santana — Fri, 22 Jul 2022 08:10:13 +0000

When the data model is not well designed, everything that goes after feels wrong. You feel like you are doing hacks and tweaks all the time.

When the data model is the right one everything flows, it’s easy to explain, when you make a change it just fits like a good Tetris play. Only time will tell if the data model was the right one. If after some years you still use the same data model (maybe not the same database or same code) you did it right. It’s not that different to cars, buildings, companies…

Designing a good data model takes time, prototypes and a good understanding of the reality you are modelling (see point 1 for more info).

[2/40 things about data] There is no “the best data format”

javi santana — Thu, 21 Jul 2022 07:42:56 +0000

We format the data to move it around. It could be hundreds of kilometers or a few nanometers but we always need to encode information somehow. I’ve never found the “El Dorado” of data formats.

Text formats are easy to read by a human but harder and slower to parse.

Binary formats are fast to parse but hard to debug.

XML is a good container but it’s too verbose.

JSON is easy but does not have basic data types.

Serializable formats are not good to keep in memory but specific formats for in-memory operations are not binary compatible with other languages.

The most important thing I learned is: you need to find the right balance between speed, flexibility, compatibility and the human-computer interface.

[1/40 things about data] It’s hard to capture reality with data

javi santana — Wed, 20 Jul 2022 10:08:55 +0000

Trying to recreate an accurate version of reality, no matter what that is or how simple it looks, is hard.

Another way of seeing it: modelling reality always gets complex. There are always small nuances, special conditions, things that changed, edge cases and, of course, errors (which sometimes became features).

The only models I found easy to work with and understand are the ones that reflect computer things.

4 keys of analyzing data fast

javi santana — Tue, 19 Jul 2022 16:44:52 +0000

Don't store the data you don't need. Sounds silly but a lot of the data you have to read is not useful.
Don't read the data you don't need. Discard the data using indices or any other tool your database/framework provides
Run heavy operations later. For example, filtering data is faster than aggregating it so when processing data always filter first and do other heavy things later (joins, aggregations and so on)
Sort your data before storing it. Sorting data makes compression much better and you use all the power of current hardware (sequential reads are 100x faster than random access)

Following these 3 rules I process large datasets 100-1000x faster than I usually did.

(image from craiyon.com generated with "f1 going fast")