Hey everyone
Just finished Week 1 of the project.
Here’s everything that got completed:
- Set up the full project structure
- Generated a synthetic multi-source dataset (used Faker and tried to make it feel realistic with Indian names, patterns, etc.)
- Built a multi-format file parser that handles CSV, JSON, and XLS/XLSX
- Added a timestamp normalizer that converts everything to ISO-8601
The individual pieces were manageable, but connecting the parser and normalizer properly took more effort than expected. I had to decide the right order and clean up the flow so they work as one pipeline instead of two separate scripts.
Final flow right now:
raw folder → parse → normalize timestamps → clean output
Basic data ingestion is in place. Next focus is storing the data and starting to link relationships between records.
If you’ve built similar data pipelines, would love any suggestions.
Building this in public for the Chandigarh Police Hackathon.
Top comments (0)