Shrinking datasets is quite contextual. It depends on how the dataset is going to be used. Its the same concept as optimizing databases, based on the read/write patterns.
PACLEC and Traffic Patterns are the key determining factors.
Hosted on fly.io : https://materials-db.fly.dev/
The need for this was not to integrate with another external API for retrieving information on different materials and not think about rate-limiting. This was made as a part of a bigger system: Research Agent for Researchers
The materials trajectory database was around 12Gigs and is mostly used for training models like CHGNet.
But I needed it only for responding to query on finding materials and it didn't need all the variations.
The aim was to be able to serve it using my most loved database - sqlite, in under reasonable timeframe: <500ms.
A detailed write up is provided in my blog: https://ikouchiha47.github.io/2026/08/05/materialsdb-11gb-to-266mb.html
Top comments (0)