DEV Community: gerald mutuku

Using python in real time data weather analysis

gerald mutuku — Sat, 26 Oct 2024 15:12:53 +0000

This guide outlines the steps to set up a Python-based system for collecting, storing, and analyzing real-time weather data. We’ll pull data from a weather API, store it in a structured format, and visualize trends, making this an adaptable solution for fields impacted by weather, such as agriculture, tourism, and event planning.
Step 1: Import Libraries

We’ll begin by importing essential libraries for data collection, storage, scheduling, and visualization:

python

import requests # For API data retrieval
import pandas as pd # For data storage and manipulation
import time # For scheduling data retrieval
import matplotlib.pyplot as plt # For data visualization
from datetime import datetime # For timestamping each data entry

Step 2: Configure the Weather API

Set up the connection to the weather API. This example uses weatherapi.com, but it can be adapted to other weather APIs by updating the endpoint and request parameters.

API Key: Replace 'your_api_key' with your API key from the provider.
City: Adjust the city variable to monitor different locations.

python

api_key = 'your_api_key'
city = 'Nairobi'
url = f"http://api.weatherapi.com/v1/current.json?key={api_key}&q={city}&aqi=no"

Step 3: Function for Data Retrieval

This function queries the API, extracts the weather information, and structures it into a dictionary. Each query returns the city name, temperature, humidity, wind speed, and a timestamp.

python

def fetch_weather_data():
response = requests.get(url)
data = response.json()
weather_info = {
'city': data['location']['name'],
'temperature': data['current']['temp_c'],
'humidity': data['current']['humidity'],
'wind_speed': data['current']['wind_kph'],
'timestamp': datetime.now()
}
return weather_info

Step 4: Automate Data Collection

Using a while loop, the script repeatedly fetches data every 10 minutes (600 seconds) and appends it to a DataFrame. The script will run continuously, logging new data with each interval.

python

weather_data = pd.DataFrame(columns=['city', 'temperature', 'humidity', 'wind_speed', 'timestamp'])

try:
while True:
# Fetch and append new data to the DataFrame
new_data = fetch_weather_data()
weather_data = weather_data.append(new_data, ignore_index=True)

    # Display the latest data entries
    print(weather_data.tail())

    # Wait for the next data retrieval interval (10 minutes)
    time.sleep(600)

except KeyboardInterrupt:
print("Data collection stopped.")

Step 5: Data Visualization

The following function generates a line plot displaying temperature, humidity, and wind speed changes over time. This step provides a visual representation of weather patterns.

python

import matplotlib.dates as mdates

def plot_weather_data(df):
plt.figure(figsize=(10, 5))
plt.plot(df['timestamp'], df['temperature'], label='Temperature (°C)')
plt.plot(df['timestamp'], df['humidity'], label='Humidity (%)')
plt.plot(df['timestamp'], df['wind_speed'], label='Wind Speed (kph)')

# Format x-axis for timestamp readability

plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m-%d %H:%M'))

plt.gca().xaxis.set_major_locator(mdates.HourLocator(interval=1))

plt.gcf().autofmt_xdate()

plt.title(f'Real-time Weather Data for {city}')

plt.xlabel('Time')

plt.ylabel('Measurements')

plt.legend()

plt.show()

Call this plot function every 30 minutes for updates

while True:
plot_weather_data(weather_data)
time.sleep(1800)

Step 6: Data Storage for Historical Analysis

To store collected data, export it to a CSV file. This allows for historical weather analysis and trend evaluation.

python

Save data to a CSV file for long-term storage and analysis

weather_data.to_csv('weather_data.csv', index=False)

Optional: Setting Up Forecasting Capabilities

With an extensive dataset over days or weeks, consider implementing predictive models like ARIMA or Facebook’s Prophet to identify weather trends or create forecasts. This would involve additional libraries and configuration but can significantly enhance the project’s utility.

This setup provides a foundational framework for real-time weather monitoring and analysis, with opportunities for further customization to suit domain-specific requirements.

Python 101: Introduction to Python as a Data Analytics Tool

gerald mutuku — Sun, 06 Oct 2024 14:00:59 +0000

Python has become a go-to tool in data analytics for a good reason—it's both powerful and easy to learn. If you’re just stepping into this field, Python makes your life easier by being straightforward and versatile. Its clean, readable syntax lets you focus on the fun part—working with data—rather than getting bogged down in complicated code. Imagine you have a messy dataset from a spreadsheet, full of missing values and inconsistent formats. With Python’s Pandas library, you can clean that data in minutes, and suddenly, you’re ready to dive in and analyze it.

Python’s ecosystem is like a toolkit, packed with everything you need. NumPy helps with numbers and arrays, Matplotlib and Seaborn turn raw data into beautiful charts, and libraries like Scikit-learn even let you explore machine learning when you're ready to take it up a notch. The best part? You can start small, like generating a quick summary of sales data, and soon, you’ll be creating detailed visualizations and statistical models.

As you go through the process—collecting, cleaning, analyzing, and visualizing data—you’ll realize that Python isn’t just a tool, it’s a game-changer. It takes the tedious parts of data analysis and streamlines them, leaving you more time to uncover insights and tell stories with your data. And while learning any new skill can feel a little overwhelming at first, Python’s large community and wealth of tutorials mean that help is always just a search away. Before you know it, Python will feel like second nature, a trusty companion that makes even the toughest data challenges manageable.

SQL 101: Introduction to Structured Query Language.

gerald mutuku — Mon, 30 Sep 2024 17:14:37 +0000

SQL (Structured Query Language) is the standard language for managing relational databases. It allows you to query, update, and manage data. SQL is essential for anyone working with data, whether in analysis, development, or database administration.
SQL is a crucial skill for handling data. With it, you can query, manipulate, and manage databases effectively. Keep practicing to master more complex queries and database operations.