DO Hackathon: Data Processing

Data Processing

Welcome back to my adventures with DO hackathon.

Probably here comes the craziest part of the development process.

Seems interesting right. I got this data from the business team.

Here is a little expectation v/s reality check

Expectation:

Each company/division is a separate sheet in the excel file. I need to do a little processing and then save em all in the database (firestore).

Reality:

As you saw in the screenshot everything is in a single file. I had to use my python skills, extracted them and saved them in the firestore.

I want my code to look really clean and beautiful. So I decided to sprinkle a tad bit of clean code architecture on the python script application.

Not trying to beautify things much, just showing how the thinking process went. Let's put in this way, handwriting and ideas scribbling don't probably go hand in hand with each other.

Python part of the github repository

This project structure uses poetry as the package manager.

If want to learn more about python package managers please do check this

Package Managers in python 🚀

Rohith Gilla ・ Jun 13 '20 ・ 2 min read

#python #beginners #productivity

Modules used for excel processing

• pandas
• xlrd

jsonpickle module to obtain serializability.

Other misc modules can be found in the pyproject.toml which helped to achieve code quality.

google-cloud-firestore for firebase connectivity.

Finally, the data in the firestore looks like this

In the next post, I will be describing how the web application was built.

Peace ✌🏻
Rohith Gilla