DEV Community: Ghoshan J

Hong Kong Covid-19 Data Sourcing (2020)

Ghoshan J — Thu, 14 Nov 2024 07:21:45 +0000

Hong Kong is one of the early countries to report a patient positive for covid-19 on January 23rd 2020. Hong Kong also experienced the SARS outbreak in 2003, which is still remembered as a dark chapter in the region's post colonial memory. The protests against the government and thereby in extension to the Chinese authority had reached its peak during the middle of 2019, where more than a million protesters were out on the streets up until a security law was passed to reduce the havoc and political, financial instability it was causing. As the city was repairing the damages and assuring the financial sector that Hong Kong is still the free open economy that had once attracted the foreign companies to open offices, the unexpected pandemic comes as another stress test for this "developed" asian city.

Nevertheless, I think Hong Kong has so far in 2020 done a good job at damage control and to prioritise public safety.

Github python notebook : HK-Covid-analysis

Data sourcing

The popular covid data repository Ourworldindata (owid) reports numbers from Hong Kong under China (so I didn't feel like using that). Anyways we could make use of other official sources.

Data sourcing from data.gov.hk

The government has an abundant open datasets available for download. The datasets related to covid has list of cases, list of homes under quarantine, list of transportation modes taken by infected people, daily testing statistics, occupancy of quarantine centres, etc. Seriously, huge props for the government to make all this available through an api and update it daily (I've seen this being used at my company's HR to monitor anything related to the staff's residence). I used the "Details of probable/confirmed cases of COVID-19 infection in Hong Kong"

Data sourcing from Google big data

No surprises here. Google also collects and consolidates covid-19 data that is updated regularly. This is offered through their big data platform where you could use bigquery to narrow down the dataset. What's interesting is that this dataset also includes other complementary informations such as change in mobility, weather, testings, etc.

I though there might be differences in the data as it is a point of contention between how America records a patient as positive vs the rest of the world. But, I guess Google is just using the official data submitted from each country.

High level analysis

Do restrictions work?

The government was quick to impose restrictions on social gathering and declare a state of urgency if i.e, more that 20 cases seen for 3 days in a row. In late 2020, mandatory testing for entire building or area were also imposed. Let's see if social distancing had an impact. Also, using google's mobility reports (data retrieved from location sharing) to see if the residents did in-fact follow the government guidelines to reduce going out.

Initial social distancing restrictions reduced mobility severely and also was able to contain the 1st wave. But we can see that people are visiting groceries up to 40% more than last year. This might be the reason why the 4rd wave continued for a prolonged time as there was no dip in grocery mobility (holiday season to blame).

Third wave analysis

Let's look at the third wave closely. As the number of cases increased in June and July of 2020, the government blamed it mainly on the resident not adhering to the social distancing rules, and the influx of residents from abroad. Let's have a look at the timeline of events and air traffic.The amount of air travel reduced due to the pandemic is absolutely crazy (bad times for Cathay Pacific 😞). But other than that, I wasn't able to infer much from this.

I taught a postgraduate class on image segmentation for urban planning

Ghoshan J — Thu, 14 Nov 2024 07:14:37 +0000

This was back in 2021

I have to start off by saying many thanks to my good friend Siddharth Khakhar for this opportunity and under the guidance of Associate Professor Alain J.F. Chiaradia.

The course MUDP2020 as part of the HKU Master of Urban Design students were taught how to use a range of data collection/generation and analysis. Some are well known, and some are relatively novel in the programme, like social data sentiment analysis and image segmentation analysis. They might be crude to date – yet they will get better – we got to start anywhere.

There were 4 main themes to this course, and I taught the theme "Street view images in urban research" to a group of students in this course. This course was not about teaching coding, but to introduce new tools that could be used in the field for the future of city planning.

My work

What are we trying to achieve?
Google street view images is a powerful tool for urban designers. They use it daily to analyze new prospects, carry research on existing cities, etc. My goal was to build a pipeline on analyzing an area using image segmentation. This should aid students on complementing this with other data to answer questions like "Is there enough greenery in the pathway?", "Are there any natural stressors present in this area leading to health issues?" (more on this later).

Do it all python script 🔗
Python script that does a batched analysis.

Steps:

Download the 360 panoramic images of an area. (Students were given access to SVD360 Pro to download the area in few steps)
Load the images onto Google Drive
Run the script

Actions:

Crop 360 panorama images to the different angles, so that analysis can be done on one particular view.

Cropping is required on panoramic photos. For example, left view could be used for sidewalk analysis

Run segmentation on the images on a pretrained model 'mobilenetv2_coco_cityscapes_trainfine'

Segmentation overlay as seen by the model

Produce report on excel

file_path
data_time_original
google_street_view_image_id
gps_info
altitude
segmentation categories (road, sidewalk, building, wall, fence, pole, traffic light, terrain, vegetation, sky, person, vehicles)

Challenges

Students had little/no experience with coding/python. The class has to be conducted remotely for 3 out of the 5 months of the semester due to the lockdown for COVID in HK. This was also my first time teaching and therefor I was doubtful in each step if the students understood what I was saying. Also, this was the first time the course was being taught.

Student takeaways

Google Colab and Google Drive based script was very well welcomed by the students as it didn't require them to setup the python. The course review and instructor review was above average and received generous feedback.

Out of all the student researches what caught my eye was as argument on using image segmentation based techniques to further explore on the research done by Honold et. al. on Multiple environmental burdens and neighborhood-related health of city residents. This group of students devised a research framework that uses image segmentation to get data for building quality, sky view, vegetation, complemented with other data (traffic noise, questionnaires) to identify natural stressors in a given community.

My First KubeCon Experience: KubeCon 2024 Hong Kong

Ghoshan J — Sat, 31 Aug 2024 06:34:36 +0000

As a platform engineer living in Hong Kong, attending KubeCon for the first time was an incredible opportunity to see the principles of platform engineering in action. The conference showcased a plethora of technologies that I am eager to experiment with on my home server. Here’s a recap of my experience, insights, and what I plan to do next.

Keynote Highlights

Opening Remarks by Jim Zemlin

Speaker: Jim Zemlin, Executive Director, The Linux Foundation
Key Points:
- The focus on critical open-source software and its expansion into hardware and standards.
- Discussion on AI opportunities, including model hubs and AI safety.

Accelerating Electric Vehicle Innovation

Speakers: Kevin Wang (Huawei) & Saint Jiang (NIO)
Key Points:
- The use of KubeEdge for managing electric vehicles as edge nodes, demonstrating real-world applications of cloud-native technologies.

Hong Kong's Role in Open Source and AI

Speaker: Stormy Peters, VP of Communities, GitHub
Key Points:
- Hong Kong is rapidly growing in GitHub accounts, with strong support for open-source initiatives from local companies and the government.

Technical Sessions and Insights

Multi-Cluster Management

Speakers: Jeremy Olmsted-Thompson (Google), Hongcai Ren (Huawei), Jian Qiu (Red Hat)
Key Points:
- The introduction of the Cluster Inventory API for orchestration, crucial for handling workloads that exceed current cluster capabilities.

Project Lightning Talks

Key Technologies:
- nerdctl: A Docker-compatible CLI for containerd, which I plan to use for container management.
- Karmada: An evolution of Kubernetes federation, perfect for multi-cluster management.
- Xline: A distributed storage engine designed for wide-area networks, enhancing data management across clusters.

Networking and Community Engagement

The networking opportunities were invaluable. I connected with other platform engineers and developers, sharing insights and discussing the future of platform engineering. The collaborative spirit of the conference inspired me to contribute to open-source projects as a consumer, leveraging the knowledge gained from these interactions.

Takeaways and Future Plans

Experimentation at Home: I’m excited to implement the technologies discussed at KubeCon on my home server.
Blogging about My Journey: I aim to document my experiments and findings in a series of blog posts. Sharing my learning process will not only help me solidify my understanding but also contribute to the community.
Open Source Contribution: I intend to engage with open-source projects, starting small by utilizing existing technologies and gradually contributing to their development.

Conclusion

KubeCon 2024 in Hong Kong was an eye-opening experience that deepened my understanding of platform engineering and cloud-native technologies. The insights gained and connections made will undoubtedly shape my future projects and contributions to the community. I encourage fellow developers to immerse themselves in such conferences—it's a fantastic way to learn, network, and grow in our ever-evolving field.

Stay tuned for my upcoming blog posts where I will share my experiments and insights as I dive deeper into the technologies showcased at KubeCon!