DEV Community: Evalyn Njagi

The Power BI and REST APIs: A Fun Guide on how to Turn Web Data into Beautiful dashboards.

Evalyn Njagi — Tue, 11 Nov 2025 11:46:50 +0000

Ever wonder how live Power BI dashboards display real-time data like weather, stock prices, or social media trends?

The solution is not complex: REST APIs - blankets that aren't seen but provide a way to establish a connection between Power BI and an infinite realm of data that can be found online.

This guide is intended for new APIs and Power BI users. You will know what are the REST APIs, how Power BI is connected to them, how to authenticate, what problems may occur in general, and how to build a real-time weather dashboard showing the current temperature in Nairobi, Mombasa, Kampala, and Kigali.

At the conclusion, you will have developed the API novice into a data explorer.

What Is a REST API?

Imagine a restaurant. You are the customer, the kitchen is the data server and the waiter is the API. One can make a request to the waiter to select something particular, say, what is the current temperature in Nairobi. The waiter is introduced to the kitchen and requests what you have ordered, goes and gets the information and comes back with it on a plate all prepared to eat.

That is precisely what an API does. It enables you to query a source of data and get them in a more structured format, typically in the form of a JavaScript Object notation (JSON) that can be easily read by Power BI and converted into visuals.

Methods of integrating Power BI with a REST API.

Power BI provides numerous alternatives to access APIs, which will depend on the complexity of the API and the type of information required.

Using "Get Data - Web"

This is the simplest and least complicated alternative.

Steps:

Home - Get Data - Web (in Power BI Desktop).

Paste your API link. For example:
https://api.openweathermap.org/data/2.5/weather?q=Nairobi&appid=YOURAPIKEY

The data will be fetched and will be displayed in the form of JSON by Power BI. Then clean, transform, and visualize it.

The technique is ideal in straightforward or open APIs that do not require intricate authentication.

Using Power Query M Code

Want to have more control, such as adding parameters, adding headers, or making a chain of API calls? The Power Query language (M) of Power BI allows creating your own script.

Example:

let
    // === Settings ===
    ApiKey = "YOUR API KEY",
    Cities = {"Nairobi,KE", "Mombasa,KE", "Kampala,UG", "Kigali,RW"},

    // === Function to call OpenWeatherMap for a single city ===
    GetWeather = (city as text) =>
        let
            url = "https://api.openweathermap.org/data/2.5/weather?q=" & city & "&appid=" & ApiKey & "&units=metric",
            Source = Json.Document(Web.Contents(url)),
            // convert record to a table so it matches your original expansion logic
            #"Converted to Table" = Table.FromRecords({Source}),
            #"Expanded coord" = Table.ExpandRecordColumn(#"Converted to Table", "coord", {"lon", "lat"}, {"coord.lon", "coord.lat"}),
            #"Expanded weather" = Table.ExpandListColumn(#"Expanded coord", "weather"),
            #"Expanded weather1" = Table.ExpandRecordColumn(#"Expanded weather", "weather", {"id", "main", "description", "icon"}, {"weather.id", "weather.main", "weather.description", "weather.icon"}),
            #"Expanded main" = Table.ExpandRecordColumn(#"Expanded weather1", "main", {"temp", "feels_like", "temp_min", "temp_max", "pressure", "humidity", "sea_level", "grnd_level"}, {"main.temp", "main.feels_like", "main.temp_min", "main.temp_max", "main.pressure", "main.humidity", "main.sea_level", "main.grnd_level"}),
            #"Expanded wind" = Table.ExpandRecordColumn(#"Expanded main", "wind", {"speed", "deg", "gust"}, {"wind.speed", "wind.deg", "wind.gust"}),
            #"Expanded clouds" = Table.ExpandRecordColumn(#"Expanded wind", "clouds", {"all"}, {"clouds.all"}),
            #"Expanded sys" = Table.ExpandRecordColumn(#"Expanded clouds", "sys", {"country", "sunrise", "sunset"}, {"sys.country", "sys.sunrise", "sys.sunset"}),
            #"Changed Type" = Table.TransformColumnTypes(#"Expanded sys",{
                {"coord.lon", type number}, {"coord.lat", type number}, {"weather.id", Int64.Type},
                {"weather.main", type text}, {"weather.description", type text}, {"weather.icon", type text},
                {"main.temp", type number}, {"main.feels_like", type number}, {"main.temp_min", type number},
                {"main.temp_max", type number}, {"main.pressure", Int64.Type}, {"main.humidity", Int64.Type},
                {"wind.speed", type number}, {"wind.deg", Int64.Type}, {"name", type text}
            })
        in
            #"Changed Type",

    // === Call function for each city and combine ===
    WeatherTables = List.Transform(Cities, each GetWeather(_)),
    Combined = Table.Combine(WeatherTables),
    #"Reordered Columns" = Table.ReorderColumns(Combined,{"sys.country", "name", "coord.lon", "coord.lat", "weather.id", "weather.main", "weather.description", "weather.icon", "base", "main.temp", "main.feels_like", "main.temp_min", "main.temp_max", "main.pressure", "main.humidity", "main.sea_level", "main.grnd_level", "visibility", "wind.speed", "wind.deg", "wind.gust", "clouds.all", "dt", "sys.sunrise", "sys.sunset", "timezone", "id", "cod"}),
    #"Renamed Columns" = Table.RenameColumns(#"Reordered Columns",{{"sys.country", "country"}}),
    #"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"country", type text}}),
    #"Removed Columns" = Table.RemoveColumns(#"Changed Type",{"id", "cod", "timezone", "weather.icon"})
in
    #"Removed Columns"

This will provide you with freedom to call automation, deal with many cities, and control more sophisticated processes.

With Custom Connector or Azure Function.

Beyond advanced configurations, businesses often use custom Power BI connectors or Azure Functions to handle authentication and data processing before sending clean data to Power BI.

This is ideal for sensitive, high-volume data requiring frequent updates.

Power BI Authentication.
To connect to APIs, Power BI must be granted data access permission via authentication, which Power BI supports in common forms.

1) Anonymous

Applied to public APIs that do not need any form of logins. You are able to stay in touch with each other without unnecessary procedures.

2) API Key

This is the most widespread technique. You append your key to an API URL or a header. It is saved safely in the Data Source Settings of Power BI.

Example:

https://api.openweathermap.org/data/2.5/weather?q=Nairobi&appid=YOURAPIKEY

3) Basic Authentication

Unlike other APIs that use a username and password for authentication, Power BI securely stores your credentials after you enter them.

4) OAuth 2.0

APIs requiring user authorization, such as Google or Microsoft Graph, utilize a secure login window for authentication. Upon signing in, Power BI receives a temporary token granting access. Always consult the API documentation to confirm the authentication method. Importing API data into Power BI can present challenges.

Integrating Power BI with an API can be exciting, but it's important to be aware of potential pitfalls.

1) Pagination

Most APIs do not provide a large number of records on a request. Power Query loops or functions will be required in order to access all the data pages.

2) Rate Limits

Other APIs limit the frequency of requests made e.g. 60 calls per minute. Going beyond this will have your key blocked temporarily.

3) Authentication Expiration

The use of tokens in APIs (such as OAuth) may have expiry times and should be automatically renewed otherwise there may be refresh errors.

4) Nested JSON

The API data is usually embedded in layers, such as boxes within boxes. The Power Query editor on Power BI allows one to expand and flatten these fields into a clean table.

5) Cloud Refresh

When publishing to Power BI Service, ensure your credentials and gateways are properly configured for scheduled refresh.

Integrating OpenWeatherMap API data with Power BI.

To be more practical, we will consider linking Power BI to the OpenWeatherMap API to get the current temperature of Nairobi, Mombasa, Kampala, and Kigali.

Step 1: Get Your API Key
Obtain your API key from the open data weather site by creating a free account.

Copy your special API key (long sequence of characters and numbers).

Step 2: Develop a Power Query Function.

In Power BI:

Home -Get Data - Blank Query - Advanced Editor.

This provides an interactive feature that recurs and gets weather information for any passing city.

Step 3: write the below code in the advanced editor

et
    // === Settings ===
    ApiKey = "Api Key",
    Cities = {"Nairobi,KE", "Mombasa,KE", "Kampala,UG", "Kigali,RW"},

    // === Function to call OpenWeatherMap for a single city ===
    GetWeather = (city as text) =>
        let
            url = "https://api.openweathermap.org/data/2.5/weather?q=" & city & "&appid=" & ApiKey & "&units=metric",
            Source = Json.Document(Web.Contents(url)),
            // convert record to a table so it matches your original expansion logic
            #"Converted to Table" = Table.FromRecords({Source}),
            #"Expanded coord" = Table.ExpandRecordColumn(#"Converted to Table", "coord", {"lon", "lat"}, {"coord.lon", "coord.lat"}),
            #"Expanded weather" = Table.ExpandListColumn(#"Expanded coord", "weather"),
            #"Expanded weather1" = Table.ExpandRecordColumn(#"Expanded weather", "weather", {"id", "main", "description", "icon"}, {"weather.id", "weather.main", "weather.description", "weather.icon"}),
            #"Expanded main" = Table.ExpandRecordColumn(#"Expanded weather1", "main", {"temp", "feels_like", "temp_min", "temp_max", "pressure", "humidity", "sea_level", "grnd_level"}, {"main.temp", "main.feels_like", "main.temp_min", "main.temp_max", "main.pressure", "main.humidity", "main.sea_level", "main.grnd_level"}),
            #"Expanded wind" = Table.ExpandRecordColumn(#"Expanded main", "wind", {"speed", "deg", "gust"}, {"wind.speed", "wind.deg", "wind.gust"}),
            #"Expanded clouds" = Table.ExpandRecordColumn(#"Expanded wind", "clouds", {"all"}, {"clouds.all"}),
            #"Expanded sys" = Table.ExpandRecordColumn(#"Expanded clouds", "sys", {"country", "sunrise", "sunset"}, {"sys.country", "sys.sunrise", "sys.sunset"}),
            #"Changed Type" = Table.TransformColumnTypes(#"Expanded sys",{
                {"coord.lon", type number}, {"coord.lat", type number}, {"weather.id", Int64.Type},
                {"weather.main", type text}, {"weather.description", type text}, {"weather.icon", type text},
                {"main.temp", type number}, {"main.feels_like", type number}, {"main.temp_min", type number},
                {"main.temp_max", type number}, {"main.pressure", Int64.Type}, {"main.humidity", Int64.Type},
                {"wind.speed", type number}, {"wind.deg", Int64.Type}, {"name", type text}
            })
        in
            #"Changed Type",

    // === Call function for each city and combine ===
    WeatherTables = List.Transform(Cities, each GetWeather(_)),
    Combined = Table.Combine(WeatherTables),
    #"Reordered Columns" = Table.ReorderColumns(Combined,{"sys.country", "name", "coord.lon", "coord.lat", "weather.id", "weather.main", "weather.description", "weather.icon", "base", "main.temp", "main.feels_like", "main.temp_min", "main.temp_max", "main.pressure", "main.humidity", "main.sea_level", "main.grnd_level", "visibility", "wind.speed", "wind.deg", "wind.gust", "clouds.all", "dt", "sys.sunrise", "sys.sunset", "timezone", "id", "cod"}),
    #"Renamed Columns" = Table.RenameColumns(#"Reordered Columns",{{"sys.country", "country"}}),
    #"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"country", type text}}),
    #"Removed Columns" = Table.RemoveColumns(#"Changed Type",{"id", "cod", "timezone", "weather.icon"})
in
    #"Removed Columns"

Step 4: Build the Dashboard

Import the data into Power BI and draw a bar chart, table or map of the temperatures of the four cities. Even refreshing it automatically can be scheduled.

Final Thoughts

REST APIs open up a world of live data for Power BI users. Whether you want to track weather, financial markets, or social media activity, APIs can feed real-time information directly into your dashboards.

You’ve learned how Power BI connects to APIs, how it handles authentication, what challenges to watch out for, and how to build a real example using OpenWeatherMap.

Once you get comfortable with APIs, you’ll realize that Power BI isn’t just about charts and tables; it’s your window into the living, breathing data of the internet.

Collecting Africa’s Energy Insights:

Evalyn Njagi — Sun, 12 Oct 2025 17:15:16 +0000

Introduction
When working with data, access to well-structured and reliable information can make all the difference. I recently developed a project called Africa Energy Coverage Data Extractor, which focuses on collecting energy data for all African countries between 2000 and 2022. This project uses Python and Selenium to automatically gather data from the Africa Energy Portal and then stores it in a format that can be analyzed or uploaded to a cloud database.

The goal was simple: create a tool that could automatically go through the Africa Energy Portal, capture yearly statistics for every country, and organize everything into one clean dataset.

What the Project Does
The Africa Energy Coverage Data Extractor uses Selenium WebDriver to browse through the website just like a human would. It navigates across multiple pages and year sliders to pull together information on different energy indicators.
Once the data is collected, it is saved into a CSV file for local use. You can also choose to upload it to MongoDB Atlas, which is a cloud database that makes it easier to store and manage large datasets.

Main Features

Here are some of the key things the extractor can do:

Collects energy data for all 54 African countries
Extracts yearly data from 2000 to 2022
Handles dynamic website elements using Selenium
Gathers a wide range of energy indicators such as electricity access, generation, and consumption
Saves data as a structured CSV file
Optionally uploads the data to MongoDB Atlas
Organizes code into simple, modular scripts

The Tools Behind the Project
The project was built with tools that are common in data work but powerful when combined:

Python 3.x for programming
Selenium WebDriver for browser automation
Pandas and NumPy for data handling
webdriver-manager for managing browser drivers
MongoDB Atlas and PyMongo for data storage and access
Google Chrome as the browser used during automation

This combination makes it easy to manage both scraping and storage without needing complicated setups.

How the Project is Organized

The code is divided into a few Python files that handle different parts of the process:

scrape.py – contains the main scraping logic
mongodb.py – manages the connection to MongoDB and data uploads
main.py – runs the full process from scraping to saving
requirements.txt – lists the Python packages you need
README.md – contains documentation

This structure keeps everything clear and easy to maintain.

Getting Started

To run the project, you’ll need to have Python 3.8 or higher, Google Chrome, and Git installed.

After cloning the project repository, you can set up your environment by creating a virtual environment and installing the dependencies:

python -m venv venv
source venv/bin/activate    # or venv\Scripts\activate on Windows
pip install -r requirements.txt

Once that’s done, you can run the main script to start scraping:

python main.py

This will collect the data and save it into a CSV file on your computer.

Connecting to MongoDB Atlas

If you’d like to upload the extracted data to MongoDB, you’ll need to create a free cluster on MongoDB Atlas.

After that, set up your environment variables with your database connection details:

MONGO_URI="your_mongodb_connection_uri"
MONGO_DATABASE="AfricaEnergyData"

Then you can upload the data by running:

python main.py

Data in MongoDB:

The data will be sent to your cloud database, where you can access it anytime.

Understanding the Data

Each record in the dataset contains information about a specific energy metric for a country and year.
Here’s what each column represents:

country – name of the country

country_serial – ISO3 country code

metric – energy metric (for example, access to electricity)

value – the numeric value of the metric

unit – the unit of measurement

sector – the main sector category

sub_sector – a more specific classification

sub_sub_sector – an even finer breakdown

source_link – link to the source on the Africa Energy Portal

source – the name of the source (default: Africa Energy Portal)

year – the year of the data (2000–2022)

This makes it easy to work with the data in analytics tools or dashboards later on.

How It Works in Practice

The process starts with Selenium opening the Africa Energy Portal. It then loops through each year from 2000 to 2022, collecting data for all available countries. Once all the information is gathered, it’s organized into a CSV file.

You can decide whether to stop there or upload the data to MongoDB Atlas for storage and sharing.

Contact
Author: Evalyn Njagi
Email: evalynnjagi02@gmail.com
LinkedIn: Evalyn Njagi

If you are interested in contributing, improving the scraper, or expanding it to new datasets, feel free to connect or open an issue on GitHub.

Final Thoughts

Working on this project taught me a lot about data automation and cloud storage. It showed how a simple Python workflow can help collect large-scale information in a structured and reusable format.
This tool can be extended for other kinds of regional or sectoral data, and it serves as a practical example of how data engineering can support research and development in different fields.

E-commerce Price Comparison: Real-Time Price Checking with Python

Evalyn Njagi — Fri, 04 Jul 2025 10:12:30 +0000

Introduction:

Have you ever wondered if the “big discount” flashing on your screen is the best deal available?
I have. And like many shoppers today, I was tired of guessing.

Shopping in 2025 is a thrilling yet confusing experience. Prices bounce up and down, discounts come and go, and no one has time to compare dozens of pages every day.
So I decided to let Python do the hard work for me.

Project Overview:

It is no secret that online shopping platforms sometimes display wildly different prices for the same item. With dynamic pricing algorithms and personalized deals, prices for the same product often vary more than most people think. The truth is that comparing prices manually is time-consuming.

What if we could:

Extract live prices from different stores
Clean and match messy product names
Compare them instantly
See where the real deal is hiding

This brief project demonstrates how Python can be used to retrieve actual data and perform a price comparison.

Project Code

https://github.com/EvalynTheAnalyst/E-Commerce-Price-Comparison.git

Price Comparison Overview

https://e-commerce-price-comparison-3v4hksca8brzaudtn2jkuc.streamlit.app/#e-commerce-price-tracker

Methodology:

The solution uses Python and a few powerful libraries. The goal is simple: collect product details from two well-known e-commerce sites, clean the data, match similar products, and display the price comparison with clear visuals.

Here's a step-by-step breakdown:

Importing necessary Libraries:

The following libraries will be used to retrieve data from e-commerce sites, clean it, and visualize the price comparison.

2. Scraping Data

Using BeautifulSoup and requests, the script visits product pages and extracts information such as name, price, discount, and reviews.

3)Cleaning Data

Pandas cleans price data and handles missing values, crucial for reliable data analysis.

4) Matching Products

The same phone or accessory may appear with slightly different names on each site. RapidFuzz helps match them smartly using fuzzy matching.

5) Comparing Prices

Once matched, prices are compared to see which site offers a better deal for the same product.

6.) Visualizing Insights

Finally, the project wraps all this into a Streamlit app. With one click, it scrapes fresh data, cleans it, matches products, and shows a clear table and charts.

Features include:

Matched products with side-by-side prices
Histogram of price differences
Average price comparison
Top 10 largest price gaps
Option to download results as CSV

Conclusion:

In today’s market, algorithms quietly adjust prices all the time. Discounts can be real or just marketing. Reviews influence buying decisions, but sometimes the lowest price does not come with the best trust signals.

By scraping data responsibly and presenting it clearly, shoppers get back control. The tool does not rely on guesswork. It checks actual listings, compares the same products, and shows where you can save money. For anyone who runs an online store, this same idea can help keep an eye on the competition.

Extracting Data from APIs Using Python — A Beginner-Friendly Guide

Evalyn Njagi — Tue, 13 May 2025 12:04:02 +0000

In today’s digital world, data is the heartbeat of modern applications and decision-making. Whether you're analyzing stock trends, tracking weather updates, or studying social media patterns, much of this data comes from external sources, often through APIs.

This article will walk you through how to extract data from an API using Python’s requests library clearly and practically.

Understanding Data Sources
Before we dive into coding, let’s first understand where data comes from. Broadly, data sources fall into two categories:

Internal Sources
These are datasets generated within an organization — such as financial reports, employee records, or sales logs. For instance, a company’s profit and loss statement or payroll data would be considered internal.

External Sources
These are obtained from outside the organization, often via the internet. Examples include weather updates, social media content, or real-time stock prices accessed using APIs.

What is an API?
API stands for Application Programming Interface. It acts like a bridge that allows two software applications to communicate. When you request information from a website via an API, the server responds with structured data, often in formats like JSON or XML, which you can then use in your application.

Think of an API as a waiter at a restaurant: you (the client) place your order (the request), the waiter delivers it to the kitchen (the server), and brings back your food (the data).

Why Use the requests Module?
Python’s requests module makes it incredibly simple to send HTTP requests and receive responses. It's widely used for calling APIs because it’s beginner-friendly and powerful.

To start, you have to install the requests module in Python

Once installed, import it into your code.

Example: Extracting Stock Ticker Data
Here’s a sample script that uses the Polygon.io API to retrieve stock ticker data:

Output

What Next? Clean and Analyze the Data
Once you’ve extracted the data, the next step is to clean it, filter out what you need, and analyze it using tools like pandas, numpy, or even visualization libraries like matplotlib or seaborn.

Final Thoughts
Getting data via APIs is a powerful skill that opens up endless possibilities for analysis, automation, and app development. With just a few lines of Python, you can tap into live, real-world data — no spreadsheets required.

If you're just starting in data science, engineering or backend development, working with APIs is one of the most valuable and practical things you can learn.