As data teams grow, finding the right dataset becomes like searching for a needle in a haystack. Organizations accumulate hundreds or thousands of datasets scattered across databases, data lakes, and warehouses, with little visibility into what exists, where it came from, or how it connects to other data assets. A well-designed data catalog solves this discovery problem by automatically indexing datasets, capturing their metadata and lineage, and making everything searchable across your organization.
Architecture Overview
A data catalog service sits at the intersection of data governance, discovery, and quality. At its core, the system needs three critical capabilities: automatically detecting new datasets as they appear in your infrastructure, extracting and organizing metadata about those datasets, and providing a queryable interface for users to find and understand what data is available.
The architecture consists of several interconnected components working in concert. Data connectors act as scouts, continuously scanning storage systems like S3, data warehouses like Snowflake, and databases like PostgreSQL for new or modified datasets. Rather than forcing engineers to manually register every table, these connectors use event-driven triggers or scheduled crawlers to detect changes automatically. When a new dataset appears, the metadata extractor kicks in, pulling information like column names, data types, row counts, and sample values. A lineage tracker then maps dependencies between datasets, showing which tables feed into which downstream processes or analytics jobs. All this information flows into a central metadata repository, typically a graph database or search index, that powers fast discovery through a web-based UI or API.
The design prioritizes loose coupling between components. Connectors, extractors, and lineage trackers operate independently and communicate through event queues or APIs, allowing you to add new data sources without redesigning the entire system. Classification engines can plug in separately, running on extracted metadata to automatically tag datasets with business domains, sensitivity levels, or data quality scores.
Handling Scale and Real-Time Updates
As organizations grow, the catalog must handle thousands of datasets and constant updates without becoming a bottleneck. Connectors typically run on a schedule (hourly or daily scans) rather than continuously, balancing freshness against resource costs. For high-velocity data sources, event-based integration with data platform tools like Apache Kafka ensures near-real-time catalog updates. The metadata repository should be horizontally scalable, often using technologies like Elasticsearch for full-text search capabilities across dataset descriptions and column names.
Design Insight: Automatic Discovery and Classification
The magic of a modern data catalog is removing the registration burden entirely. Instead of asking engineers to fill out forms, the system watches your infrastructure automatically. Metadata connectors scan storage paths and query data warehouse system catalogs, identifying new datasets without manual intervention. For classification, the catalog applies rules and machine learning models to extracted metadata. A table with columns named "ssn," "credit_card_number," or "medical_record_id" gets automatically tagged as sensitive. Dataset naming conventions can trigger domain classification, so a table prefixed with "sales_" lands in the commercial domain. Time-series analysis of table growth and query patterns can even assign quality scores, flagging datasets that are rarely accessed or growing anomalously. This automation scales to thousands of datasets while keeping the catalog current and relevant, making data discovery a self-service capability rather than a manual chore.
Watch the Full Design Process
See how this architecture comes together in real-time with AI-assisted design. Watch the complete walkthrough on your preferred platform:
Try It Yourself
Ready to design your own data catalog or another system architecture? Head over to InfraSketch and describe your system in plain English. In seconds, you'll have a professional architecture diagram, complete with a design document.
This is Day 96 of a 365-day system design challenge. What system would you design next?
Top comments (0)