DEV Community

Cover image for DO Hackathon: Data Processing
Rohith Gilla
Rohith Gilla

Posted on

DO Hackathon: Data Processing

Data Processing

Minion gif

Welcome back to my adventures with DO hackathon.

Probably here comes the craziest part of the development process.

Excel data

Seems interesting right. I got this data from the business team.

Here is a little expectation v/s reality check

Expectation:

Each company/division is a separate sheet in the excel file. I need to do a little processing and then save em all in the database (firestore).

Reality:

As you saw in the screenshot everything is in a single file. I had to use my python skills, extracted them and saved them in the firestore.

I want my code to look really clean and beautiful. So I decided to sprinkle a tad bit of clean code architecture on the python script application.

Alt Text

Alt Text

Not trying to beautify things much, just showing how the thinking process went. Let's put in this way, handwriting and ideas scribbling don't probably go hand in hand with each other.

Python part of the github repository

This project structure uses poetry as the package manager.

If want to learn more about python package managers please do check this

Modules used for excel processing

  • pandas
  • xlrd

jsonpickle module to obtain serializability.

Other misc modules can be found in the pyproject.toml which helped to achieve code quality.

google-cloud-firestore for firebase connectivity.

Finally, the data in the firestore looks like this

Alt Text

Alt Text

In the next post, I will be describing how the web application was built.

See ya soon

Peace ✌🏻
Rohith Gilla

Do your career a big favor. Join DEV. (The website you're on right now)

It takes one minute, it's free, and is worth it for your career.

Get started

Community matters

Top comments (0)

Imagine monitoring actually built for developers

Billboard image

Join Vercel, CrowdStrike, and thousands of other teams that trust Checkly to streamline monitor creation and configuration with Monitoring as Code.

Start Monitoring

👋 Kindness is contagious

Explore a sea of insights with this enlightening post, highly esteemed within the nurturing DEV Community. Coders of all stripes are invited to participate and contribute to our shared knowledge.

Expressing gratitude with a simple "thank you" can make a big impact. Leave your thanks in the comments!

On DEV, exchanging ideas smooths our way and strengthens our community bonds. Found this useful? A quick note of thanks to the author can mean a lot.

Okay