DEV Community: Edwin Kinyao

Mastering Big Data with GCP: My Capstone Journey in Cloud Data Analysis

Edwin Kinyao — Tue, 25 Mar 2025 15:06:37 +0000

Introduction

As a data enthusiast, I’ve always been fascinated by the power of cloud platforms to transform raw data into actionable insights. Recently, I completed a capstone project using Google Cloud Platform (GCP) that put my skills to the test. My task? Help a fictional fintech startup, TheLook Fintech, leverage BigQuery and Looker to tackle critical business questions about loan performance and borrower behavior. In this blog, I’ll walk you through my journey—from collecting and processing data to building a sleek dashboard—and share the lessons I learned along the way.

Whether you’re a data analyst, a cloud newbie, or just curious about BigQuery and Looker, this post will give you a front-row seat to a real-world data project.

The Scenario: A Fintech Startup’s Data Challenge

Imagine you’re a cloud data analyst hired by TheLook Fintech, a growth-stage startup revolutionizing loans for online store owners. The Treasury department, led by Trevor, needs your help to monitor cash flow, understand why customers borrow, and track loan distribution across regions. Later, they want a dashboard to keep tabs on loan health. My mission was clear: use GCP tools to collect, process, and analyze data, then visualize the results.

The project unfolded in two parts:

BigQuery Workflow: Collecting, processing, and storing loan data to answer three key questions.
Looker Dashboard: Building visualizations to monitor loan health metrics.

Here’s how I tackled it.

Part 1: Collecting, Processing, and Storing Data in BigQuery

The first leg of the project was all about getting hands-on with BigQuery, GCP’s serverless data warehouse. My goal was to answer three business questions:

How can we monitor cash flow to ensure loan funding doesn’t exceed incoming payments?
What are the top reasons customers take out loans?
Where are borrowers located geographically?

Step 1: Setting Up the BigQuery Environment

I started by creating a BigQuery dataset to house the loan data. This involved setting up tables and ensuring the schema aligned with the fintech’s needs—think columns for loan amounts, purposes, dates, and borrower locations.

Step 2: Exploring the Loan Data

With the data loaded, I ran exploratory SQL queries to get a feel for it.

For cash flow, I calculated money in (loan repayments) versus money out (loan issuances). For loan purposes, I dug into a nested field in the application data to extract reasons like “inventory purchase” or “business expansion.”

For locations, I aggregated loans by state.

Step 3: Importing Additional Data

Trevor needed a deeper geographic breakdown, so I imported a CSV file with state classifications into BigQuery. I converted this into a standard table using a CREATE TABLE AS SELECT statement—a simple satisfying tactic.

Step 4: Joining Tables

Next, I joined the loan data with the state classification table using a JOIN clause in SQL. This enriched the dataset, letting me map loans to regions and spot geographic trends.

Step 5: Cleaning Up with Deduplication

The loan purpose data had duplicates , so I used a DISTINCT query to clean it up. This ensured accurate reporting on why borrowers were seeking funds.

Step 6: Aggregating Loan Amounts by Year

Finally, I created a table with a GROUP BY query to sum loan amounts by issuance date and year. This gave Trevor a clear view of lending trends over time—crucial for cash flow monitoring.

By the end, I had a polished dataset ready for analysis, stored efficiently in BigQuery.

Part 2: Visualizing Insights with Looker Enterprise

With the data prepped, Trevor threw a new challenge my way: create a dashboard in Looker to track loan health. He wanted answers to four questions:

What’s the total outstanding loan amount?
What percentage of loans fall into each status (e.g., current, late, default)?
Which states have the most outstanding loans?
Which customers own their homes outright and have current loans?

Task 1: Getting Started with Looker

I kicked things off by connecting Looker to my BigQuery dataset. Looker’s intuitive interface made it easy to define a data model that mapped to my tables.

Task 2: Total Outstanding Loan Amount

For the first visualization, I built a single-value card showing the sum of all outstanding balances. A quick SUM measure in LookML, paired with a filter for unpaid loans, did the trick.

Task 3: Loan Status Breakdown

Next, I created a pie chart to display the percentage of loans by status. I grouped the data by categories like then used Looker’s percentage calculation to show the distribution. This was a game-changer for spotting risk areas.

Task 4: Top States with Outstanding Loans

I crafted a bar chart highlighting the top 10 states by loan count. A COUNT measure, sorted in descending order, and a limit of 10 gave Trevor a clear view of geographic concentration.

Task 5: Homeowners with Current Loans

For the final visualization, I built a table listing customers who own their homes outright and have “Current” loans. I filtered by homeownership status and loan status, then sorted by income to spotlight high earners.

Task 6: Polishing the Dashboard

To make the dashboard interactive, I enabled cross-filtering—clicking a state in the bar chart filters the other visuals. I also set a daily refresh rate to keep the data fresh. The result? A sleek, user-friendly tool Trevor’s team could rely on.

The Final Dashboard

Here’s what the dashboard looked like:

Card: Total outstanding amount ($3.08B).
Pie Chart: Loan status percentages.
Bar Chart: Top 10 states by loan count.
Table: Homeowning customers with current loans.

It was a proud moment seeing it all come together—a testament to the power of combining BigQuery’s data crunching with Looker’s visualization prowess.

Conclusion

My journey with TheLook Fintech’s data was a crash course in using GCP to tackle real-world challenges. BigQuery made it easy to handle large datasets, while Looker brought the insights to life. If you’re looking to break into cloud data analysis, I can’t recommend this kind of hands-on project enough—it’s the perfect way to build skills and confidence.

I’m excited to explore more advanced GCP features like Dataflow or AI Platform. For now, I’d love to hear your thoughts—have you worked with BigQuery or Looker? Drop a comment below!

Time Series Analysis

Edwin Kinyao — Thu, 27 Feb 2025 14:22:50 +0000

Time Series Analysis with ARMA Model and Forecasting on Amazon Stocks

Introduction

Stock market forecasting is a challenging yet crucial task for investors, traders, and financial analysts. Amazon, one of the world’s most influential companies, experiences significant stock price fluctuations influenced by various market factors. In this study, we leverage the Autoregressive Moving Average (ARMA) model to analyze and forecast Amazon’s stock prices using time series data.

Understanding the ARMA Model

The ARMA (p, q) model is a widely used statistical model for time series analysis, combining two key components:

Autoregressive (AR) Model (p): Captures the relationship between a variable and its past values.
Moving Average (MA) Model (q): Incorporates dependencies between a variable and past error terms.

This model is particularly useful when analyzing financial time series data, as it helps in understanding past price behavior and forecasting future trends.

Data Collection and Preprocessing

For this study, we obtained Amazon stock price data from a financial market dataset, covering several years of daily closing prices. The preprocessing steps included:

Handling missing values and outliers.
Converting the dataset into a time series format.
Checking stationarity using the Augmented Dickey-Fuller (ADF) test.
Differencing the data if necessary to achieve stationarity.

Model Selection and Training

Exploratory Data Analysis (EDA): We visualized the time series to detect patterns, trends, and seasonality.
Parameter Selection (p, q): We used the Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) plots to determine optimal values of p and q.
Model Fitting: The ARMA model was trained using historical stock data.
Performance Evaluation: Metrics such as Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) were used to assess model accuracy.

Forecasting Amazon’s Stock Prices

Once the ARMA model was trained and validated, we used it to forecast Amazon’s stock prices for a specified future period. The predictions were visualized alongside historical data to evaluate their accuracy.

Key Findings and Insights

The ARMA model effectively captured short-term fluctuations in Amazon’s stock prices.
The model performed well in forecasting near-future prices but had limitations in capturing long-term trends.
Stock prices exhibited seasonal and cyclical trends, indicating that an advanced model (such as ARIMA or LSTM) could further improve forecasts.

Conclusion

Time series analysis using the ARMA model provides valuable insights into stock price behavior. While effective for short-term forecasting, integrating additional factors such as macroeconomic indicators or utilizing deep learning models can enhance predictive accuracy.

This study highlights the potential of statistical models in financial forecasting and serves as a foundation for future research in stock market analytics.

Future Work

Extending the model to ARIMA for better trend analysis.
Incorporating external factors such as earnings reports and market sentiment.
Comparing ARMA with machine learning models like LSTMs for enhanced accuracy.

Comparative Analysis of Classification Techniques: Naive Bayes, Decision Trees, and Random Forests

Edwin Kinyao — Sat, 18 Jan 2025 14:34:47 +0000

Machine learning breathes life into data, uncovering patterns and making predictions that help solve real-world challenges. Imagine using these tools to explore the majestic world of dinosaurs! This article compares the performance of three popular machine learning models—Naive Bayes, Decision Trees, and Random Forests—on a unique dinosaur dataset. Follow along as we journey from data exploration to model evaluation, focusing on how each model performs and what insights they reveal.

1. Dataset Description

The dataset is a treasure trove of information about dinosaurs, covering attributes such as their diet, period, location, and size. Each row represents a unique dinosaur, offering both categorical and numerical data for analysis.

Key Features:

name: Dinosaur name (categorical).
diet: Feeding type (e.g., herbivorous, carnivorous).
period: Geological time period when the dinosaur lived.
lived_in: Geographic region of existence.
length: Approximate size (numerical).
taxonomy: Hierarchical classification.

Dataset Link: Jurassic Park - The Exhaustive Dinosaur Dataset

2. Data Preparation and Exploration

2.1 Dataset Overview

Initial inspection revealed class imbalances, with herbivores dominating the dataset. This imbalance posed challenges for the models, particularly for Naive Bayes, which assumes equal representation.

2.2 Data Cleaning

Steps to ensure data quality included:

Imputation of missing values using appropriate statistical techniques.
Identification and handling of outliers in numerical attributes like length.

2.3 Exploratory Data Analysis (EDA)

EDA uncovered fascinating trends and relationships:

Herbivorous dinosaurs were more prevalent during the Jurassic period.
Numerical features such as length showed significant variation between species.

3. Feature Engineering

Feature engineering aimed to enhance model performance by refining inputs:

Scaling and Normalization: Standardized numerical features like length for consistency.
Feature Selection: Prioritized influential attributes such as diet, taxonomy, and period to focus on relevant data.

4. Model Comparison and Training

The primary goal was to compare the effectiveness of three models on the dinosaur dataset.

4.1 Naive Bayes

Naive Bayes, a probabilistic model, assumes feature independence. Its simplicity made it computationally efficient, but it struggled with the class imbalance in the dataset, leading to suboptimal predictions for underrepresented classes.

4.2 Decision Tree

Decision Trees excel at capturing non-linear relationships through hierarchical splits. The model performed better than Naive Bayes, particularly in identifying complex patterns. However, it was susceptible to overfitting when the tree depth was not controlled.

4.3 Random Forest

Random Forest, an ensemble of Decision Trees, proved to be the most robust model. By aggregating predictions from multiple trees, it minimized overfitting and handled the dataset’s complexity effectively, achieving the highest accuracy.

5. Results and Analysis

Key Observations:

Random Forest achieved the highest accuracy and balanced performance across all metrics, highlighting its strength in managing complex data interactions.
Decision Tree delivered reasonable performance but slightly lagged behind Random Forest in predictive accuracy.
Naive Bayes struggled with imbalanced data, resulting in lower accuracy and recall.

Challenges and Recommendations:

Addressing class imbalance using SMOTE or resampling could improve the models’ performance on underrepresented dinosaur types.
Hyperparameter tuning, particularly for Decision Tree and Random Forest, could further refine model accuracy.
Experimenting with alternative ensemble methods like boosting may yield additional insights.

Conclusion

This analysis demonstrated how different machine learning models perform on a unique dinosaur dataset. From data preparation to model evaluation, the process highlighted the strengths and limitations of each model:

Naive Bayes: Fast and simple but struggled with imbalanced classes.
Decision Tree: Intuitive and interpretable but prone to overfitting.
Random Forest: The most accurate and robust model, showcasing the power of ensemble methods.

The comparative approach revealed Random Forest as the most reliable model for this dataset. Future work will delve deeper into advanced techniques like boosting and feature engineering to push the boundaries of prediction accuracy.

Happy coding! 🎉

For more on this, visit my GitHub

Interfaces in Java

Edwin Kinyao — Sun, 29 Dec 2024 16:29:13 +0000

An Interface in Java

An interface in the Java programming language is defined as an abstract type used to specify the behavior of a class. It is a blueprint of a behavior and contains static constants and abstract methods.

Syntax for Java Interfaces

interface {
    // declare constant fields
    // declare methods that are abstract 
    // by default.   
}

To declare an interface, use the `interface` keyword.

It is used to provide total abstraction. That means all the methods in an interface are declared with an empty body and are public, and all fields are public, static, and final by default. A class that implements an interface must implement all the methods declared in the interface. To implement the interface, use the implements keyword.

Uses of Interfaces in Java

It is used to achieve total abstraction.
Since Java does not support multiple inheritances in the case of a class, by using an interface, it can achieve multiple inheritances.
Any class can extend only one class but can implement multiple interfaces.
It is also used to achieve loose coupling.
Interfaces are used to implement abstraction.

Difference Between Class and Interface

Although Class and Interface may seem similar, there are certain differences between them. The major differences are outlined below:

Class	Interface
In a class, you can instantiate variables and create an object.	In an interface, you must initialize variables as they are final, but you can’t create an object.
A class can contain concrete (with implementation) methods.	An interface cannot contain concrete (with implementation) methods.
The access specifiers used with classes are `private`, `protected`, and `public`.	In an interface, only one specifier is used—`public`.

code example:

import java.io.*;

/* 
Today we are going to learn about interfaces.

>>> Interfaces are a reference type similar to class 
that can contain only constants, method signatures, 
default methods, static methods, and nested types.
*/

// Example

interface vehicle {
    void speedup(int a);
    void accelerate(int a);
    void brake(int a);
}

class automobile implements vehicle {
    int gear;
    int speed;

    @Override
    public void speedup(int newgear) {
        gear = newgear;
    }

    @Override
    public void accelerate(int increment) {
        speed = speed + increment;
    }

    @Override
    public void brake(int decrement) {
        speed = speed - decrement;
    }

    public void displayStatus() {
        System.out.println("Current Gear: " + gear);
        System.out.println("Current Speed: " + speed + " km/h");
    }
}

public class vehicle__demo {
    public static void main(String[] args) {
        automobile myAutomobile = new automobile();
        myAutomobile.speedup(4);
        myAutomobile.accelerate(80);
        myAutomobile.brake(30);

        System.out.println("Vehicle Status:");
        myAutomobile.displayStatus();
    }
}

Conclusion

Interfaces play a crucial role in Java by enabling total abstraction, loose coupling, and achieving multiple inheritances. They are powerful tools for designing robust and scalable applications. By understanding and effectively using interfaces, you can write cleaner and more maintainable code.

Happy coding! 🎉

my GitHub
java repo

Understanding Encapsulation in Object-Oriented Programming

Edwin Kinyao — Fri, 27 Dec 2024 04:24:01 +0000

Encapsulation in Object-Oriented Programming

Encapsulation is a fundamental object-oriented programming concept that involves bundling data (fields) and methods (functions) that operate on the data within a single unit, typically a class. It restricts direct access to some of the object's components, making it easier to maintain and secure the code.

Benefits of Encapsulation

Data Hiding: Internal state is hidden from the outside world, and access is controlled through methods (getters and setters).
Improved Code Maintainability: Changes to fields or methods can be made without affecting external code that uses the class.
Enhanced Security: By restricting direct access to fields, we can validate and protect data from invalid states.

Code Example: Encapsulation in Action

// Encapsulation refers to restricting access of a class from the outside world
public class Person {
    private String name;
    private String profession;
    private double height;
    private int ID;
    private int age;

    // Constructor
    public Person(String name, String profession, double height, int iD, int age) {
        this.name = name;
        this.profession = profession;
        this.height = height;
        ID = iD;
        this.age = age;
    }

    // Getters and setters for accessing private fields
    public String getName() {
        return name;
    }
    public void setName(String name) {
        this.name = name;
    }
    public String getProfession() {
        return profession;
    }
    public void setProfession(String profession) {
        this.profession = profession;
    }
    public double getHeight() {
        return height;
    }
    public void setHeight(double height) {
        this.height = height;
    }
    public int getID() {
        return ID;
    }
    public void setID(int iD) {
        ID = iD;
    }
    public int getAge() {
        return age;
    }
    public void setAge(int age) {
        this.age = age;
    }

    // Main method to demonstrate encapsulation
    public static void main(String[] args) {
        Person myPerson = new Person("Robert", "doctor", 130.4, 39, 23);

        // Accessing private fields through getter methods
        System.out.println(myPerson.getName());
        System.out.println(myPerson.getProfession());
        System.out.println(myPerson.getID());
        System.out.println(myPerson.getAge());
    }
}

Explanation of the Code

Private Fields

The fields name, profession, height, ID, and age are declared as private. This makes them inaccessible directly from outside the class.

Getters and Setters

Public methods like getName(), setName(), getProfession(), and others act as controlled access points for the private fields. These methods allow external code to retrieve and modify the private data securely.

Constructor

The constructor initializes the fields when an object of the class Person is created. This ensures that the object starts in a valid state.

Main Method

The main method demonstrates how encapsulation is used. The private fields are accessed indirectly through the getter methods.

Benefits in the Example

Data Protection:
- The private fields cannot be accessed or modified directly, preventing accidental or malicious changes.

Controlled Access:

By using setters, you can include validation logic to ensure only valid data is set. For example:

 public void setAge(int age) {
     if (age > 0) {
         this.age = age;
     } else {
         System.out.println("Age must be positive.");
     }
 }

Code Flexibility:
- If the implementation of fields changes (e.g., adding derived fields), external code using the class remains unaffected.

This example illustrates how encapsulation ensures that the Person class maintains integrity and hides its implementation details while providing a controlled interface for interaction.

Understanding Inheritance in Java Through a Practical Example

Edwin Kinyao — Thu, 26 Dec 2024 13:08:28 +0000

Understanding Inheritance in Java Through a Practical Example

Inheritance is a core concept in object-oriented programming (OOP) that allows one class to acquire the properties (attributes and methods) of another class. In Java, inheritance is implemented using the extends keyword and represents an "is-a" relationship. This article explains inheritance in Java through a practical example.

The Code Example

// Defining a class
class Animal {
    // General attributes
    protected String colour;
    protected String breed;
    protected int age;

    // General methods
    public String sleep() {
        return "Both cats and dogs sleep";
    }

    public String eat() {
        return "They also eat";
    }

    // Constructor
    public Animal(String colour, String breed, int age) {
        this.colour = colour;
        this.breed = breed;
        this.age = age;
    }

    // Getters and setters
    public String getColour() {
        return colour;
    }

    public void setColour(String colour) {
        this.colour = colour;
    }

    public String getBreed() {
        return breed;
    }

    public void setBreed(String breed) {
        this.breed = breed;
    }

    public int getAge() {
        return age;
    }

    public void setAge(int age) {
        this.age = age;
    }
}

// Cat class inheriting from Animal class
class Cat extends Animal {
    private String catName;

    public Cat(String colour, String breed, int age, String catName) {
        super(colour, breed, age); // Call the parent class constructor
        this.catName = catName;
    }

    public String getCatName() {
        return catName;
    }

    public void setCatName(String catName) {
        this.catName = catName;
    }

    public String catSound() {
        return "Cat meows!";
    }
}

// Dog class inheriting from Animal class
class Dog extends Animal {
    private String dogName;

    public Dog(String colour, String breed, int age) {
        super(colour, breed, age);
    }

    public String getDogName() {
        return dogName;
    }

    public void setDogName(String dogName) {
        this.dogName = dogName;
    }

    public String dogSound() {
        return "Dog barks!";
    }
}

public class Demo {
    public static void main(String[] args) {
        Cat myCat = new Cat("Brown", "Persian", 2, "Tom");
        Dog myDog = new Dog("Black", "Labrador", 3);

        // Display Cat details
        System.out.println("Cat's Name: " + myCat.getCatName());
        System.out.println("Cat's Colour: " + myCat.getColour());
        System.out.println("Cat's Breed: " + myCat.getBreed());
        System.out.println("Cat's Age: " + myCat.getAge());
        System.out.println("Cat Sound: " + myCat.catSound());
        System.out.println("Cat Behavior: " + myCat.eat() + " and " + myCat.sleep());

        // Display Dog details
        System.out.println("Dog's Colour: " + myDog.getColour());
        System.out.println("Dog's Breed: " + myDog.getBreed());
        System.out.println("Dog's Age: " + myDog.getAge());
        System.out.println("Dog Sound: " + myDog.dogSound());
    }
}

Key Concepts in the Code

Parent Class (Animal):

Defines common attributes (colour, breed, age) and methods (sleep, eat) that are shared among all animals.
Provides a constructor to initialize these attributes.
Includes getters and setters for encapsulation.

Child Classes (Cat and Dog):

Extend the Animal class and inherit its attributes and methods.
Add specific attributes (catName, dogName) and behaviors (catSound, dogSound).
Use the super keyword to call the parent class constructor and initialize inherited attributes.

Demo Class:

Serves as the entry point of the program.
Demonstrates how to create objects of Cat and Dog classes and access their properties and methods.

Benefits of Inheritance

Code Reusability: The Cat and Dog classes reuse the code in the Animal class.
Extensibility: New child classes (e.g., Bird, Fish) can be added easily by extending the Animal class.
Polymorphism: Shared methods like sleep and eat can be overridden in child classes to provide specific behaviors.

Output of the Program

Cat's Name: Tom
Cat's Colour: Brown
Cat's Breed: Persian
Cat's Age: 2
Cat Sound: Cat meows!
Cat Behavior: They also eat and Both cats and dogs sleep
Dog's Colour: Black
Dog's Breed: Labrador
Dog's Age: 3
Dog Sound: Dog barks!

my GitHub
java repo

Understanding Getters and Setters in Java with Examples

Edwin Kinyao — Thu, 26 Dec 2024 12:50:08 +0000

In Java, getters and setters are essential methods used to access and modify the properties of an object. They help in encapsulating the data and ensuring that the internal representation of an object is hidden from the outside. This article will provide a detailed explanation of getters and setters, along with examples to illustrate their use.

What are Setters and Getters?

Setters: These methods are used to write or update values into the object's properties.
Getters: These methods are used to read or retrieve values from the object's properties.

Example: Student Class

Below is a simple Java class named Student that demonstrates the use of getters and setters.


java
// Create a class Student
public class Student {
    private String name;
    private int ID;
    private String course;
    private double GPA;

    // Setter for name
    public void setName(String name) {
        this.name = name;
    }

    // Setter for ID
    public void setID(int ID) {
        this.ID = ID;
    }

    // Setter for course
    public void setCourse(String course) {
        this.course = course;
    }

    // Setter for GPA
    public void setGPA(double GPA) {
        this.GPA = GPA;
    }

    // Getter for name
    public String getName() {
        return name;
    }

    // Getter for ID
    public int getID() {
        return ID;
    }

    // Getter for course
    public String getCourse() {
        return course;
    }

    // Getter for GPA
    public double getGPA() {
        return GPA;
    }
}

Weather Data Collection and Analysis for Major Towns in Kenya

Edwin Kinyao — Sun, 13 Oct 2024 14:24:30 +0000

Welcome to the Weather Data Collection and Analysis for Major Towns in Kenya project! This repository demonstrates the process of collecting, storing, and analyzing weather data for five major towns in Kenya: Nairobi, Mombasa, Kisumu, Nakuru, and Eldoret. The goal is to provide valuable insights into the weather patterns of these towns over a one-year period using API data, MySQL for storage, and various Python-based tools for data analysis and visualization.

Project Overview

The project includes:

Collecting historical weather data from an API
Converting the JSON data into CSV format
Storing the data in a MySQL database
Performing exploratory data analysis (EDA) on the weather data
Visualizing weather patterns, trends, and relationships between variables
Creating geospatial visualizations using Folium
Building an interactive dashboard with Streamlit

Technologies Used
Data Collection
Data Storage
Data Analysis
Geospatial Visualization
Streamlit Dashboard
How to Run the Project
Future Enhancements

Technologies Used

Python for data processing, analysis, and visualization
MySQL for storing weather data
Streamlit for building the interactive dashboard
Folium and Plotly for geospatial and interactive visualizations
Weather API for historical weather data
SQLAlchemy for database interaction

Data Collection

Weather data was collected using the Weather API for five towns in Kenya (Nairobi, Mombasa, Kisumu, Nakuru, and Eldoret) between January 2023 and January 2024.

Data Storage

The weather data collected from the API is transformed and saved into a MySQL database for structured storage and easy retrieval. The data is stored in a table that contains key weather attributes such as temperature, humidity, wind speed, and more.

MySQL Database Schema

The database contains a table named weather, which stores the following columns:

id: Unique identifier for each record
town: The name of the town (e.g., Nairobi, Mombasa)
date: The date of the weather record
max_temp_c: Maximum temperature (in °C)
min_temp_c: Minimum temperature (in °C)
avg_temp_c: Average temperature (in °C)
humidity: Average humidity percentage
precipitation_mm: Precipitation level (in mm)
wind_kph: Wind speed (in kilometers per hour)
condition_text: A textual description of the weather condition (e.g., "Clear", "Rainy")

SQL Example

CREATE DATABASE weather_data_db;
USE weather_data_db;

CREATE TABLE weather (
    id INT AUTO_INCREMENT PRIMARY KEY,
    town VARCHAR(100),
    date DATE,
    max_temp_c FLOAT,
    min_temp_c FLOAT,
    avg_temp_c FLOAT,
    humidity FLOAT,
    precipitation_mm FLOAT,
    wind_kph FLOAT,
    condition_text VARCHAR(255)
);

Data Analysis

The data analysis phase aims to uncover insights and trends from the weather data collected across various towns in Kenya. This process involved several steps: data cleaning, exploratory data analysis (EDA), and visualization of weather attributes such as temperature, humidity, precipitation, and wind speed.

1. Data Cleaning

Before performing any analysis, the dataset underwent several cleaning steps:

Handling missing values: Any missing data was either filled using imputation methods (such as forward fill for time-series data) or removed if deemed irrelevant.
Date formatting: The date column was converted into a datetime format to facilitate proper time-series analysis.
Data type conversions: Numerical fields like temperature, humidity, and wind speed were cast to appropriate data types (e.g., float) to ensure smooth analysis.

2. Exploratory Data Analysis (EDA)

EDA was performed to better understand the structure of the weather data and the relationships between different variables.

2.1 Temperature Trends

The first step in the analysis was to examine temperature patterns across different towns. By plotting maximum, minimum, and average temperatures over time, we identified significant seasonal trends and variations between towns.

import matplotlib.pyplot as plt

# Plot maximum temperature trends for each town
plt.figure(figsize=(10,6))
for town in df['town'].unique():
    town_data = df[df['town'] == town]
    plt.plot(town_data['date'], town_data['max_temp_c'], label=town)

plt.title('Maximum Temperature Trends Across Towns')
plt.xlabel('Date')
plt.ylabel('Maximum Temperature (°C)')
plt.legend()
plt.grid(True)
plt.show()

Streamlit Dashboard

The project includes an interactive Streamlit Dashboard to provide users with an intuitive interface to explore the weather data for the five major towns in Kenya. The dashboard was designed to present data insights and visualizations in an accessible way, allowing users to interact with the data and customize their views based on their preferences.

Key Features of the Dashboard

Interactive Visualizations:
The dashboard features interactive charts and graphs, allowing users to dynamically explore weather trends across different towns. Users can select the towns of interest and view data such as temperature, humidity, wind speed, and precipitation over time. These interactive elements help users visualize how weather parameters change throughout the year.
Town Comparison:
Users can compare weather conditions across the five major towns (Nairobi, Mombasa, Kisumu, Nakuru, and Eldoret). The dashboard enables side-by-side comparisons of average temperatures, humidity levels, and wind speeds, providing insights into how different regions experience varied climatic conditions.
Date Range Selection:
The dashboard includes a feature to filter the data based on a specific date range. Users can select a start and end date, and the visualizations will update accordingly to show the weather trends during the selected period. This allows users to focus on specific timeframes, such as seasonal changes or yearly trends.

User-Friendly Interface:
The dashboard was designed with ease of use in mind, ensuring that both technical and non-technical users can navigate it without difficulty. Clear labels, intuitive controls, and informative tooltips help guide users through the process of selecting towns, filtering data, and interacting with the visualizations.
Downloadable Reports:
The dashboard includes an option for users to download the analyzed weather data in CSV format. This feature is useful for those who wish to perform further analysis on their own or store the data for offline use.

Insights and Applications

The interactive dashboard provides valuable insights into Kenya’s weather patterns, allowing users to:

Track seasonal trends: Understand how temperature, humidity, and precipitation vary throughout the year and across different regions.
Compare climatic conditions: Compare the weather conditions of multiple towns to inform decision-making, such as agricultural planning or infrastructure development.
Monitor real-time weather: Stay up-to-date with the latest weather conditions, especially during periods of extreme weather or changing seasons.(I consider this as a Future Enhancements)

The Streamlit dashboard serves as a practical tool for individuals and organizations seeking to make data-driven decisions based on weather patterns. By offering real-time data, customizable views, and easy-to-understand visualizations, the dashboard makes weather analysis accessible and actionable for a wide range of users.

How to Run This Project

To run the Weather Data Collection and Analysis for Major Towns in Kenya project, follow these steps:

Prerequisites

Ensure that you have the following installed on your machine:

Python 3.8+
MySQL Server (for storing the weather data)
Git (to clone the repository)
Virtual Environment (optional but recommended)

Step 1: Clone the Repository

First, clone the project repository from GitHub using the following command:

git clone https://github.com/your-username/weather-data-kenya.git

Find more on this project in my Github:
Github

How to Style Your Notebook for Data Analysis: A Guide with Heart Attack Prediction Example

Edwin Kinyao — Sun, 13 Oct 2024 13:55:47 +0000

Data analysis is a process that can be made much more efficient and insightful with a well-organized notebook. The way you structure your notebook not only helps with clarity but also makes it easier to track your work, replicate results, and share findings. Let’s walk through how you can style your notebook for a comprehensive data analysis project using an example project on Heart Attack Analysis and Prediction.

Start with a Clear and Informative Title
Your notebook should have a clean title that clearly reflects the purpose of your analysis. In our case:

Heart Attack Analysis and Prediction

This provides an immediate understanding of the project’s goal. Aligning the title to the center also gives it a polished, professional look.

Define the Structure of the Notebook
One of the most important aspects of notebook preparation is its structure. Defining a clear table of contents not only guides your workflow but also helps anyone reviewing your notebook to easily navigate through sections.

Project Content

Introduction
- 1.1 Examining the topic
Data Preprocessing
- 2.1 Data Cleaning
- 2.2 Feature Selection
- 2.3 Encoding
Exploratory Data Analysis
- 3.1 Summary Statistics
- 3.2 Visualizations
Feature Engineering
Model Building
- 5.1 Train-Test Split
- 5.2 Choosing the models
Model Evaluation
Model Comparison
The End
This structure offers a logical flow: from introduction and data preparation to model building and evaluation. Linking sections using markdown ensures easy navigation within your notebook, especially as the project grows larger.
Introduction: Set the Context
The Introduction should give a brief overview of the problem you're trying to solve and why it’s important. In this case, you would discuss heart disease and the goal of predicting heart attacks using machine learning.

1. Introduction

1.1 Examining the Topic

Having sub-sections under each major heading makes it easy to break down large parts into digestible pieces. When you introduce a concept, make sure it’s clear why you’re doing it and what value it brings to the analysis.

Data Preprocessing: Explain Every Step
This is where you get hands-on with your data, and it's vital that each step of your preprocessing phase is well-documented. You'll usually start with data cleaning, feature selection, and encoding:

2. Data Preprocessing

2.1 Data Cleaning

2.2 Feature Selection

2.3 Encoding

Each step in data preprocessing should explain why a specific method (like filling missing values, dropping irrelevant columns, or encoding categorical variables) was chosen. This transparency ensures that anyone reading your notebook can understand your reasoning and replicate your work.

Exploratory Data Analysis: Use Visuals to Tell a Story
Exploratory Data Analysis (EDA) is where you let the data "speak." It’s crucial to present your summary statistics and visualizations in a clean, organized manner.

3. Exploratory Data Analysis

3.1 Summary Statistics

3.2 Visualizations

In this section, show summary statistics first to provide an overview, followed by visualizations such as histograms, correlation heatmaps, and pair plots to reveal insights. Label your charts clearly, so readers can easily interpret them without having to guess.

Feature Engineering: Document Your Creative Process
Feature engineering is where you apply your domain knowledge to create new features that may enhance model performance. Any modifications you make should be documented with explanations.

4. Feature Engineering

In this section, explicitly state what new features you created and why. For example, you might create a "cholesterol-age ratio" feature because you hypothesize it has a strong relationship with heart attack risk.

Model Building: Be Clear About Your Approach
When it comes to building models, it's important to clearly state your methodology and any decisions you make.

5. Model Building

5.1 Train-Test Split

5.2 Choosing the Models

This section should include details like how you split the data into training and testing sets, which machine learning models you chose (e.g., logistic regression, random forest, etc.), and why those models were selected.

Model Evaluation: Use Metrics and Visuals
After training your models, you'll need to evaluate their performance. Always use a variety of evaluation metrics like accuracy, precision, recall, and F1-score to give a well-rounded assessment of your models.

6. Model Evaluation

You might also want to include confusion matrices and ROC curves to provide a visual evaluation of model performance.

Comparison: Summarize the Results
This is the section where you compare different models and summarize their performances based on the metrics from the evaluation stage. This helps in deciding which model to use.

7. Model Comparison

Provide a concise table or chart to visually compare model performance.

End with a Conclusion
Finally, conclude with a summary of the project, discussing the findings and any potential next steps. A well-rounded conclusion wraps up your notebook and gives it a finished feel:

<<< The End >>>

This gives the notebook a clean ending and signals that the analysis is complete.

How to Style Your Notebook for Data Analysis: A Guide with Heart Attack Prediction Example

Edwin Kinyao — Sat, 12 Oct 2024 09:32:58 +0000

Start with a Clear and Informative Title Your notebook should have a clean title that clearly reflects the purpose of your analysis. In our case:

Heart Attack Analysis and Prediction

This provides an immediate understanding of the project’s goal. Aligning the title to the center also gives it a polished, professional look.

Define the Structure of the Notebook One of the most important aspects of notebook preparation is its structure. Defining a clear table of contents not only guides your workflow but also helps anyone reviewing your notebook to easily navigate through sections. Here’s how it can be laid out:

Project Content

Introduction
- 1.1 Examining the topic
Data Preprocessing
- 2.1 Data Cleaning
- 2.2 Feature Selection
- 2.3 Encoding
Exploratory Data Analysis
- 3.1 Summary Statistics
- 3.2 Visualizations
Feature Engineering
Model Building
- 5.1 Train-Test Split
- 5.2 Choosing the models
Model Evaluation
Model Comparison
The End
This structure offers a logical flow: from introduction and data preparation to model building and evaluation. Linking sections using markdown ensures easy navigation within your notebook, especially as the project grows larger.
Introduction: Set the Context
The Introduction should give a brief overview of the problem you're trying to solve and why it’s important. In this case, you would discuss heart disease and the goal of predicting heart attacks using machine learning.

1. Introduction

1.1 Examining the Topic

Having sub-sections under each major heading makes it easy to break down large parts into digestible pieces. When you introduce a concept, make sure it’s clear why you’re doing it and what value it brings to the analysis.

Data Preprocessing: Explain Every Step This is where you get hands-on with your data, and it's vital that each step of your preprocessing phase is well-documented. You'll usually start with data cleaning, feature selection, and encoding:

2. Data Preprocessing

2.1 Data Cleaning

2.2 Feature Selection

2.3 Encoding

Each step in data preprocessing should explain why a specific method (like filling missing values, dropping irrelevant columns, or encoding categorical variables) was chosen. This transparency ensures that anyone reading your notebook can understand your reasoning and replicate your work.

Exploratory Data Analysis: Use Visuals to Tell a Story Exploratory Data Analysis (EDA) is where you let the data "speak." It’s crucial to present your summary statistics and visualizations in a clean, organized manner:

3. Exploratory Data Analysis

3.1 Summary Statistics

3.2 Visualizations

In this section, show summary statistics first to provide an overview, followed by visualizations such as histograms, correlation heatmaps, and pair plots to reveal insights. Label your charts clearly, so readers can easily interpret them without having to guess.

Feature Engineering: Document Your Creative Process Feature engineering is where you apply your domain knowledge to create new features that may enhance model performance. Any modifications you make should be documented with explanations:

4. Feature Engineering

In this section, explicitly state what new features you created and why. For example, you might create a "cholesterol-age ratio" feature because you hypothesize it has a strong relationship with heart attack risk.

Model Building: Be Clear About Your Approach When it comes to building models, it's important to clearly state your methodology and any decisions you make:

5. Model Building

5.1 Train-Test Split

5.2 Choosing the Models

This section should include details like how you split the data into training and testing sets, which machine learning models you chose (e.g., logistic regression, random forest, etc.), and why those models were selected.

Model Evaluation: Use Metrics and Visuals After training your models, you'll need to evaluate their performance. Always use a variety of evaluation metrics like accuracy, precision, recall, and F1-score to give a well-rounded assessment of your models.

6. Model Evaluation

You might also want to include confusion matrices and ROC curves to provide a visual evaluation of model performance.

Comparison: Summarize the Results This is the section where you compare different models and summarize their performances based on the metrics from the evaluation stage. This helps in deciding which model to use:

7. Model Comparison

Provide a concise table or chart to visually compare model performance.

End with a Conclusion Finally, conclude with a summary of the project, discussing the findings and any potential next steps. A well-rounded conclusion wraps up your notebook and gives it a finished feel:

<<< The End >>>

This gives the notebook a clean ending and signals that the analysis is complete.

DEV Community: Edwin Kinyao

Mastering Big Data with GCP: My Capstone Journey in Cloud Data Analysis

Introduction

The Scenario: A Fintech Startup’s Data Challenge

Part 1: Collecting, Processing, and Storing Data in BigQuery

Step 1: Setting Up the BigQuery Environment

Step 2: Exploring the Loan Data

Step 3: Importing Additional Data

Step 4: Joining Tables

Step 5: Cleaning Up with Deduplication

Step 6: Aggregating Loan Amounts by Year

Part 2: Visualizing Insights with Looker Enterprise

Task 1: Getting Started with Looker

Task 2: Total Outstanding Loan Amount

Task 3: Loan Status Breakdown

Task 4: Top States with Outstanding Loans

Task 5: Homeowners with Current Loans

Task 6: Polishing the Dashboard

The Final Dashboard

Conclusion

Time Series Analysis

Time Series Analysis with ARMA Model and Forecasting on Amazon Stocks

Introduction

Understanding the ARMA Model

Data Collection and Preprocessing

Model Selection and Training

Forecasting Amazon’s Stock Prices

Key Findings and Insights

Conclusion

Future Work

Comparative Analysis of Classification Techniques: Naive Bayes, Decision Trees, and Random Forests

1. Dataset Description

Key Features:

2. Data Preparation and Exploration

2.1 Dataset Overview

2.2 Data Cleaning

2.3 Exploratory Data Analysis (EDA)

3. Feature Engineering

4. Model Comparison and Training

4.1 Naive Bayes

4.2 Decision Tree

4.3 Random Forest

5. Results and Analysis

Key Observations:

Challenges and Recommendations:

Conclusion

Interfaces in Java

An Interface in Java

Syntax for Java Interfaces

To declare an interface, use the interface keyword.

Uses of Interfaces in Java

Difference Between Class and Interface

Conclusion

Understanding Encapsulation in Object-Oriented Programming

Encapsulation in Object-Oriented Programming

Benefits of Encapsulation

Code Example: Encapsulation in Action

Explanation of the Code

Private Fields

Getters and Setters

Constructor

Main Method

Benefits in the Example

Understanding Inheritance in Java Through a Practical Example

Understanding Inheritance in Java Through a Practical Example

The Code Example

Key Concepts in the Code

Parent Class (Animal):

Child Classes (Cat and Dog):

Demo Class:

Benefits of Inheritance

Output of the Program

Understanding Getters and Setters in Java with Examples

What are Setters and Getters?

Example: Student Class

Weather Data Collection and Analysis for Major Towns in Kenya

Project Overview

Table of Contents

Technologies Used

Data Collection

To declare an interface, use the `interface` keyword.