DEV Community: Afroza Nowshin

Learning Statistics with R, Part-2

Afroza Nowshin — Fri, 10 May 2024 19:13:08 +0000

This photo shows a perfectly symmetrical distribution as the mean and median are almost the same. Remember, for our distribution the mean is 23.6 and median is 24; almost equal.

The datasets that we will deal with in our projects may not have the almost equal mean and median which lead to "skewness" to the distribution.

When mean > median, this indicates that the few of the extreme values are pulling the mean upward, while most of the data points are concentrated in the lower end. This is a positively skewed distribution where the distribution is skewed to the right with a long tail extending to the right.
When mean < median, this indicates that the few of the extremely low values are pulling the mean downward while of the data points are concentrated in the higher end. This is a negatively skewed distribution where the distribution is skewed to the left with a long tail extending to the left.

To measure the skewness of a distribution, you need to use 'moments' package which provides you with the skewness function:

temperatures <- c(22, 24, 26, 27, 23, 24, 24, 25, 21, 20) 

install.packages("moments")
library(moments)
skewness <- skewness(temperatures)

In the previous post, we learned about the three quartiles that divide the entire dataset into 4 equal chunks. There are properties that will tell us the spread of data points around the mean (center) of a dataset. In the picture that is at the beginning of this post, you can see lines saying 1, -1, 2, -2. We will get into these lines shortly. Before that, we need to understand how each data points 'deviates' from the center or the mean.

In our dataset:

20, 21, 22, 23, 24, 24, 24, 25, 26, 27

If we subtract each value from the mean 23.6, we will get:

-3.6, -2.6, -1.6, 0.6, 0.4, 0.4, 0.4, 1.4, 2.4, 3.4

We need the average of these values but the negatives and positives will cancel each other out, therefore we will square each of them:

12.96, 6.76, 2.56, 0.36, 0.16, 0.16, 0.16, 1.96, 5.76, 11.56

Now we will take the sum of these values: 42.4

Since we are using a sample temperature dataset, we will divide this value with number of observations, 10 - 1.

The result is the variance of the dataset:

Variance = ∑ (x - μ)² / (n - 1)

= 42.4 / 9≈ 4.71

Where x is the individual data points and mu is the mean. Variance gives you how much each data point is varying. Since this is a squared value, this will give the unit in squares and is less preferred. The square root of variance is called Standard Deviation (SD) and is heavily used to understand the deviation of each point from the mean:

SD = √(Variance)
SD = √(4.71)
SD ≈ 2.17

In the image of the normal distribution, +1 and -1 represent one standard deviation above and below the mean, respectively. Approximately 68% of the data points are contained within this range of one standard deviation from the mean." =2 and -2 represent two standard deviation above and below the mean, respectively. Approximately 97% of the data points are contained within this range of two standard deviation from the mean.

In R, you can use the functions var() and sd() for calculating these values:

sd_temp <- sd(temperatures)

# Calculate Variance
variance_temp <- var(temperatures)

Finally,

the 'summary()' function of R will provide you all the statistical property that we discussed so far!

summary_stats <- summary(temperatures)
print(summary_stats)

The output will look like the following (I formatted to table for legibility):

Min.	1st Qu.	Median	Mean	3rd Qu.	Max.
20.00	22.25	24.00	23.60	24.75	27.00

I wanted to teach all the basics with logic so that you can understand, what the summary() function does for you 👀

To conclude this post, based on our mean=23.6 and median=24 which are less than 25°C, on May 8, you can go for silk or linen clothes if you are still wondering this question based the dataset😁 For more informed decisions, you need historical data forcast and trends to analyze.

Learning Statistics with R, Part-1

Afroza Nowshin — Thu, 09 May 2024 16:04:27 +0000

Before starting to learn machine learning and data analysis, it is crucial that one learns the basics of statistics. When I work in a data engineering pipeline, first I try to see the statistical properties of the given dataset. While there are built-in functions in Python and R, we tend to skip what does these functions do behind the scene. Today, I will explain how to do statistical data analysis with R.

Let's say, you are a statistics noob. Today is Monday and you want to know what to wear- cotton if the max temperature is above 25°C and silk or linen if the max temperature is below 25°C. Can statistics help you?

Well, weather and climate data trend relies on historical data analysis and finding pattern with various machine learning algorithm. But to understand the basics of statistics, we need to collect some data, like the following:

Day	Max Temperature (°C)
2024-04-28	22
2024-04-29	24
2024-04-30	26
2024-05-01	27
2024-05-02	23
2024-05-03	24
2024-05-04	24
2024-05-05	25
2024-05-06	21
2024-05-07	20

For making prediction and statistical analysis, you will need to collect a lot of data unlike this one. Your first order of business is make a dataframe out of this table. In R, you will do it this way:

# Create data frame
temperature_data <- data.frame(
  Day = c("2024-04-28", "2024-04-29", "2024-04-30", "2024-05-01", "2024-05-02", "2024-05-03", "2024-05-04", "2024-05-05", "2024-05-06", "2024-05-07"),
  MaxTemperature = c(22, 24, 26, 27, 23, 24, 24, 25, 21, 20)
)

With this dataframe, we are good to go for some statistics. First, we will check the 'distribution' of the data. This distribution means some 'bin's of the same width; a histogram plot. The height of each bin corresponds to the number of data points that fall inside that bin.

Temperature Range (°C)	Frequency
20-21	2
22-23	1
23-24	3
24-25	2
25-26	2
26-27	2

R will build the required data intervals and create the bins for you with its hist() function.

temperatures <- c(22, 24, 26, 27, 23, 24, 24, 25, 21, 20)
# Plotting the histogram
hist(temperatures, breaks=7, col="blue", main="Histogram of Maximum Temperatures",
     xlab="Temperature (°C)", ylab="Frequency", ylim=c(0,3))

The output will look like the following:

Notice that the number of bins are not strictly equal to the number that you set for the breaks parameters. This parameter rather suggests the number of bins that R prefers. R will divide into the optimum number of intervals that are either equal to or not equal to the value of breaks parameter.

Before going to learn the necessary R commands and functions to get an idea about the statistical property of the data, we need to understand why we need to know these properties in the first place. Let us arrange the data points from the lowest to the highest:

20, 21, 22, 23, 24, 24, 24, 25, 26, 27

The average or mean value is 23.6, and the middle value or median can be calculated by averaging 5th and 6th values, which in our case is 24. As our dataset has even number of values, we are taking the average of the middle two values. If the dataset had odd number of values, we could just take the middle value for the median. Another less used data property is mode , which indicates the highest occurring data points in a dataset. These three properties give us an overview of the data, for example, for an evenly spaced bigger dataset, the graph looks like a bellcurve. Mean is the center of the distribution for a dataset; think of it like the center of a blending machine; mean is the blades.

This is called a 'Normal Distribution'. In a normal distribution, the mean and median are equal which is denoted by 0 in the picture. The data points are mostly in the hump of the curve, about 68% of them. Following code will help you to draw the density plot:

plot(density(temperatures), type = 'l', col = "darkred", lwd = 2, 
     main = "Temperature Density Plot", xlab = "Temperature (°C)", ylab = "Density")

Since our data points are very few, and not 'ideal', the curve looks distorted, which is okay because real-life datasets are not always ideal with equal mean and median values. There are built-in functions in R to check these values:

# Calculate mean
mean_temp <- mean(temperatures)

# Calculate median
median_temp <- median(temperatures)

Median is an important property of a dataset. To find out the outliers of a dataset, you need to find out the Interquartile Range . Basically, range is the difference between the minimum and the maximum data points. What is interquartile range then?

We know that in the sorted dataset, the median is 24, which is also called the 2nd Quartile (Q2) or 50th Percentile. This Quartile splits the whole dataset in a way such that, 50% of the data points are lower than this point and 50% of the data points are higher than this point. After this split, the median of the lower chunk of the dataset will be called the 1st Quartile(Q1) or the 25th percentile. Similarly, the median of the higher chunk is the 3rd Quartile (Q3) or the 75th percentile. These 3 quartiles splits the distribution into 4 equal portions, each containing 25% of the entire dataset. Interquartile range or IQR is the difference between Q3 and the Q1. Following is the graphical representation of these properties, which is called a Boxplot or Box and Whiskers plot.

The code to draw boxplot is:

# Define the temperature data
temperatures <- c(22, 24, 26, 27, 23, 24, 24, 25, 21, 20)

# Calculate the quartiles
quartiles <- quantile(temperatures)

# Create a boxplot
boxplot(temperatures, main="Boxplot of Temperatures",
        ylab="Temperature (degrees)",
        col="lightblue", border="darkblue")

# Adjust label positions to avoid overlap
# The offset can be adjusted for better alignment
offset <- 0.7

# Add quartile labels
text(x = 1, y = quartiles[2] - offset, labels = paste("Q1 =", quartiles[2]), col="red", pos=3)
text(x = 1, y = quartiles[3], labels = paste("Q2 (Median) =", quartiles[3]), col="blue", pos=3)
text(x = 1, y = quartiles[4] + offset, labels = paste("Q3 =", quartiles[4]), col="red", pos=3)

To find out the outliers, you need to define the "bounds" beyond which you will discard the data points. The formula is like the following:

# Define lower and upper bounds to identify outliers
lower_bound <- Q1 - 1.5 * IQR
upper_bound <- Q3 + 1.5 * IQR

# Identifying outliers
outliers <- temperatures[temperatures < lower_bound | temperatures > upper_bound]

# Print outliers
print(outliers)

The output is zero since there are no outliers in our dataset. In real life, you will have datasets that will contain outliers. As part of data cleaning process, we identify the outliers and then replace the outliers with the median (Q2) value. Why?

Mean value heavily skews a dataset if there are extreme outliers, but median does not. Let's look at the following data points:

10,20,21,22,23,24,1000

The mean of this dataset is 160. This value is heavily influenced by the outlier 1000. How to reduce this effect? Just replace the outlier value with the median, in our case it is 22.

In the following post, we will explore some more concepts and R functions to finally answer the question, what to wear tomorrow?

Handling External JS Files in a React Website

Afroza Nowshin — Sun, 04 Feb 2024 11:37:32 +0000

The JS files that we integrated with our React app that we are building from a template, sometimes don't work in the current setup(blog link).

If you pay attention to the website in localhost:3000, you can see that the JS doesn't work properly but rather works if you refresh the page. Why is that?

In the image, the current year is not showing as the associated Javascript has not been loaded. If you refresh the page, the year will appear.

You are loading all the scripts when you add the scripts inside public/index.html . When you are navigating all the pages, all the Javascript codes that are written for every element of the website are removed from the virtual DOM. If you refresh the page, Javascript works. Again, refresh or change to another page; Javascript is not working.

At first when I came across this weird scenario, I was unsure what was happening. I noticed that the copyright year is not visible without a refresh or page change, so I tried to add vanilla Javascript to fix that. Then I noticed the pattern, and it hit me. Somehow, the JS files are getting removed as a result of which the JS functions are not always working.

How to solve this?

Load the plugin and main JS files when the components are inserted into the virtual DOM, that is, inside the componentDidMount(). That way, whenever you change between pages or refresh the app, the JS files are loaded into the virtual DOM of that component. Install loadjs and import into your components.


yarn add loadjs

Now, inside each page component, load the necessary JS files using this function before mounting the component:

import loadjs from "loadjs";

componentDidMount() {
    loadjs("./js/main.js");
    loadjs("./js/plugins.js");  
}

Now all the external JavaScript files will be mounted when component is mounted in the DOM without refreshing.

If you follow these posts, you have learned some tricks and tactics that you probably had to scour through Stack Overflow and Github discussion. Whether you want to build Dunder-Mifflin Infinity v20.0 or a React app from static web page materials, these techniques may come in handy. The code for scotts-totes.netlify.app can be found on my Github - https://github.com/Afroza2/ReactforReal-Turning-into-React-app.

# in an API GET Request: How to Handle?

Afroza Nowshin — Thu, 25 Jan 2024 17:59:42 +0000

I just learned a new thing that I want to share with the world.

A computer is a dumb machine. Computer programs and codes are basically teaching a dumb sh** about something.

Consider a case where you need to send data with a special character in API - @, #, $, etc. For example, I had to make an API request where id has a # before it:

http://localhost:3000/#20000

I learned the hard way that even though you can console the whole URL in any format you want. As soon as you pass the URL to an Axios or fetch API, the hash sign is auto-discarded.

What is the solution?

Pass the entire ID to the encodeURIComponent() function and use the encoded ID to your request URL.

const encodedId = encodeURIComponent(Id);

const apiUrlWithId = `http://someapi.com/${encodedId}`;

That's it. Yeah, you have to encode the hash or the special character just to make the request properly.

Step-by-step Guidelines for Integrating GPT in Your Project: Create an API for Anything Using LangChain and FastAPI

Afroza Nowshin — Sun, 14 Jan 2024 16:44:43 +0000

Generative AI has taken the world by storm. With the advent of GPT-3.5 and Bing chat powered by GPT-4, LLMs are handy. You can build your own GPT assistant by obtaining an OpenAI key and following steps from OpenAI. For creating your account, you get a 5$ credit, which is useful for the paid GPT services.

Here is a problem with most of the free GPTs.

Free GPTs lack contexts. This is critical when you are using GPTs for building your chatbot or GPT assistant. Without providing proper context, these assistants are more prone to hallucinations and out-of-context results.

Fret not; we have an open-source framework called LangChain to save the day. With LangChain, we can create GPT assistants that have contexts and get access to almost all of the LLMs for free. LangChain tools and APIs simplify the development of LLM-driven applications and virtual agents. You can develop applications with multiple LLMs using LangChain. We can even create custom chains that are suitable for our work, which I intend to cover in future posts.

The workflow at a glance is as follows:

Steps are:

Setup a Python virtual environment
Install LangChain packages and Fast API
Setup LangChain to work with OpenAI LLM or other LLM
Create a prompt for the specific task
Create your own Chain
Create an API

1. Setup a Python virtual environment

First of all, install the Python virtual environment from these links: 1 and 2. I developed my GPT-based API in Python version 3.8.18. Pick any Python versions >= 3.7.

2. Install LangChain packages and Fast API

Now go on installing the basic packages with the following command:

pip install langchain langchain-openai fastapi[uvicorn]

3. Setup LangChain to work with OpenAI LLM or other LLM

Create a python file and import the libraries.

from fastapi import FastAPI, HTTPException, Form
from langchain_openai import OpenAI as LLMOpenAI
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
from pydantic import BaseModel

For this post, we will use OpenAI's LLM, which can be set with the following code:

# Language model for LangChain
llm = LLMOpenAI(temperature=0.7, max_tokens=500, streaming=True, batch_size=50)

The parameter temperature is used to set the "creativity" level of the GPT response. The value is between 0 and 1, and the values that are nearer to 1 give less grounded results, which means the model is hallucinating and may give improper outcomes. Usually, I keep the temperature between 0.6 and 0.7. In addition, streaming and batch_size ensure concurrent chunk processing of the input.

4. Create a prompt

We need to set up a prompt, and our prompt will take one or more input variables with which we will set the template of the prompt:

prompt_template = PromptTemplate(
    input_variables=['input', 'operation'],
    template="Provide {operation} for the following code:\n\n{input}\n\nOutput:"
)

5. Create your own Chain

For input validation, we imported the Pydantic BaseModel. Then we can create our own Chain by passing our LLM and prompt to the LLMChain function.

class Input(BaseModel):
    input: str
    operation: str   

# Use dependency injection to create llm_chain within the route function
def get_llm_chain():
    return LLMChain(llm=llm, prompt=prompt_template)

6. Create an API

Finally, finish up the API for which you used the LangChain.
You need to call the function of the last step to be called by invoke() function.

@app.post('/process-input')
async def process(input: str = Form(...), operation: str = Form(...)):
    try:
        # Use langchain with dynamically included operation and essay in the prompt
        output = get_llm_chain().invoke({'input': input, 'operation': operation})

        return {'output': output}

    except HTTPException as http_error:
        raise http_error
    except Exception as e:
        raise HTTPException(status_code=500, detail=f"Error: {str(e)}")

With these steps, you can build your custom chatbots and virtual agents, summarize by integrating LangChain in the internal layers. I hope this post has helped even just one person out there. I wish to explore LangChain more and share the knowledge with you soon.

How to Predict the Temperature of Any Future Day Using Time-Series (Prophet Model and Django Backend)?-A Beginner-Friendly Guide

Afroza Nowshin — Sun, 12 Nov 2023 06:52:49 +0000

In an earlier post, we learned how to fetch data from a source (OpenMeteo) and make some decisions based on how hot or cool the source and destination are. Last week, I was assigned with another extension of the problem -

Train a simple model that forecasts the weather conditions in a given future. To simplify things, restrict predictions to the Dhaka district. After training the model, the model should be queryable via a simple API. For example, your API should be able to predict the temperature at any future date (beyond the 7 days provided by OpenMeteo).

We need to create a machine learning model with lots of historical data to predict data beyond 7 days (OpenMeteo can give up to 16 days forecast). Instead of using the default, We need the url for the historical dataset. This is what we need - https://archive-api.open-meteo.com/v1/archive .

One thing: use Jupyter Notebook instead of Colab from the get-go. This is a heavy dataset (starting from 1940), and if you train your model with this dataset, the free-tier GPU cannot keep up with it and crashes.

Here is a flow diagram of the whole process of building a prediction system using Python/Django and an ML model:

1. Fetch data from the source

First, install virtual environments. Here is a crucial part of the development: your trained model and backend should comply with the same Python version. Now, let's install some dependencies in the environment:

pip install openmeteo_requests
pip install pandas
pip install requests_cache
pip install retry_requests
pip install jupyter-notebook
pip install matplotlib

Launch the jupyter notebook and create a notebook. The OpenMeteo website has codes for fetching and storing the datasets.

import openmeteo_requests

import requests_cache
import pandas as pd
from retry_requests import retry

# Setup the Open-Meteo API client with cache and retry on error
cache_session = requests_cache.CachedSession('.cache', expire_after = 3600)
retry_session = retry(cache_session, retries = 5, backoff_factor = 0.2)
openmeteo = openmeteo_requests.Client(session = retry_session)

# Make sure all required weather variables are listed here
# The order of variables in hourly or daily is important to assign them correctly below
url = "https://api.open-meteo.com/v1/forecast"
params = {
    "latitude": 23.8103,        # Latitude for Dhaka
    "longitude": 90.4125,       #Longitude for Dhaka
        "start_date": "1940-01-01", # This is the farthest 
                        that you can go with the start date
        "end_date": "2023-11-10",   #This is the day that is 
                  the end of your range, adjust as you wish
    "hourly": "temperature_2m"
}
responses = openmeteo.weather_api(url, params=params)

# Process first location. Add a for-loop for multiple locations or weather models
response = responses[0]
print(f"Coordinates {response.Latitude()}°E {response.Longitude()}°N")
print(f"Elevation {response.Elevation()} m asl")
print(f"Timezone {response.Timezone()} {response.TimezoneAbbreviation()}")
print(f"Timezone difference to GMT+0 {response.UtcOffsetSeconds()} s")

# Process hourly data. The order of variables needs to be the same as requested.
hourly = response.Hourly()
hourly_temperature_2m = hourly.Variables(0).ValuesAsNumpy()

hourly_data = {"date": pd.date_range(
    start = pd.to_datetime(hourly.Time(), unit = "s"),
    end = pd.to_datetime(hourly.TimeEnd(), unit = "s"),
    freq = pd.Timedelta(seconds = hourly.Interval()),
    inclusive = "left"
)}
hourly_data["temperature_2m"] = hourly_temperature_2m

hourly_dataframe = pd.DataFrame(data = hourly_data)
print(hourly_dataframe)

If you want to view the data frame in a CSV file, use df.to_csv('file_name') in the end.

2. EDA

EDA or Exploratory Data Analysis is performed over the collected data to get an initial observation. Run the following pandas commands in each cell to get an overview of the dataset:

df.describe() # provides statistical insight into the dataset
df.dtypes # provides the datatype of each column
df.shapes # provide the dimension of the dataset

Check for null values in the dataset:

null_counts = df.isnull().sum()

# Print or use the result as needed
print(null_counts)

This code will return 0 for each column as there are no null values. You are lucky because if it was not, you have to spend a lot of time structuring and preprocessing the dataset.

3. Preprocess the data (If needed)

Let's say you do need to preprocess and clean up the dataset. You can check for the Interquartile Range of the datasets and replace the outliers with median values. Check for imputation methods such as KNN Imputer if your dataset needs that - .

4. Finding relationships

What model you will be required for your problem depends on what you want to answer your question. It can be discrete (yes/no) or continuous (what is the price of a house). We probe to Classification problems for discrete types of answers, whereas for continuous ones, we apply *Regression * models. Check this amazing 6-part blog series to understand the basics - Data Science for Cats

Let's plot the dataset:

import matplotlib.pyplot as plt

df.set_index('date', inplace=True)

# Plot the time series data
plt.figure(figsize=(15, 6))
plt.plot(df['temperature_2m'])
plt.title('Temperature Over Time')
plt.xlabel('Date')
plt.ylabel('Temperature (°C)')
plt.show()

# Plot a histogram of temperature values
plt.figure(figsize=(10, 6))
plt.hist(df['temperature_2m'], bins=30, edgecolor='black')
plt.title('Temperature Distribution')
plt.xlabel('Temperature (°C)')
plt.ylabel('Frequency')
plt.show()

The output looks like a rhythmic flow, is not it?

This is a special type of regression which is called a time series. In this regression, each data point is dependent on a timestamp. Following is a better view, where I sampled data from 2013 to the current year:

This graph shows the seasonality and upward trend with some residuals (Refer to 4 and 5 of the cat blog). For making the prediction, you need to do some preprocessing before fitting it to a model.

5. Applying Model to the Time-series - Prophet

For time series analysis, we usually use ARIMA and Facebook Prophet models. ARIMA is not beginner-friendly and requires a thorough theoretical understanding. FB Prophet takes care of the preprocessing (decomposing a time series and extracting seasonality, trend, and residuals) we discussed in the previous section. First, download the package:

pip install prophet

The Prophet requires just two columns, and they have to be named ds and y. ds is the date-time column, and y is what you want to predict. You need to rename your data frame, and then it will be suitable for fitting into the model. You need to create a data frame over a period in future and then pass it inside the predict dunction:

import matplotlib.pyplot as plt
from prophet import Prophet

print("Original DataFrame columns:", df.columns)

# Rename columns to match the Prophet's requirements
df = df.rename({'date': 'ds', 'temperature_2m': 'y'}, axis=1)

# Check the column names after renaming
print("DataFrame columns after renaming:", df.columns)
# Model fit
m = Prophet()
m.fit(df)

# Predict
future = m.make_future_dataframe(periods=365)
forecast = m.predict(future)

# Plot results
fig1 = m.plot(forecast)
fig2 = m.plot_components(forecast)
plt.show()
forecast

Watch something while this dataset gets trained. It took me 50 minutes to finally get the output of the model.

There is also a data frame with various parameters. Your predicted value is the yhat .

You need the model file for your API. Run the following code in a cell to get the JSON file (The prophet model can only be exported as a JSON file):


from prophet.serialize import model_to_json, model_from_json

with open('model.json', 'w') as fout:
    fout.write(model_to_json(m))

with open('model.json', 'r') as fin:
    m = model_from_json(fin.read())

To learn more about the Prophet, refer to this link - medium

6. Load the Model in Django

The AI part is done. We will now add logic for building a queryable API. First, add the notebook and the model inside your Django app. Then, load the model in the views.py:

def load_model():
    with open('./apilist/utils/model.json', 'r') as fin:
        return model_from_json(fin.read())

class WeatherPredictionAPIView(APIView):
    async def post(self, request):
        # we will implement it in the following section

7. Create Views

Let's finish the view. Essentially, your input is parsed as a ds in 'YYYY-mm-dd' format, and with the model.predict() function; you will get the yhat. This is your future prediction.

class WeatherPredictionAPIView(APIView):
    async def post(self, request):
        try:
            # Load serialized model
            m = load_model()

            # Extract input date from the request data
            input_date_str = request.data.get('date')
            input_date = datetime.strptime(input_date_str, '%Y-%m-%d')

            # Create a dataframe with the input date
            future_df = pd.DataFrame({'ds': [input_date]})

            # Make predictions
            forecast = m.predict(future_df)

            # Extract the predicted temperature for the input date
            predicted_temperature = forecast.loc[0, 'yhat']

            # Return the predicted temperature as a JSON response with status code 200 (OK)
            return Response({'predicted_temperature': predicted_temperature}, status=status.HTTP_200_OK)

        except FileNotFoundError:
            # Return an error response if the model file is not found with status code 500 (Internal Server Error)
            return Response({'error': 'Model file not found'}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)

        except Exception as e:
            # Return an error response for other exceptions with status code 500 (Internal Server Error)
            return Response({'error': str(e)}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)

8. URL for the API

You are done! Just add the URL and make a post request from your postman with any date in the form-body!

from django.urls import path
from . import views

urlpatterns = [

    path('predict-weather/', views.WeatherPredictionAPIView.as_view(), name='predict-weather')
]

9. Adding Celery and Redis to Pre-load the Model and Caching

Until now, you have applied your knowledge of time series in a practical use case. If you hit the API, you will see that the response takes over a few seconds. Well, this bothered me, therefore I further optimized the code. I introduced Celery for background loading of the model, and Redis cache for saving the model in the cache. You need to keep the Redis server running for this task.

Create a tasks.py file inside the app and add the following code with the decorator:

from django.core.cache import cache
from prophet.serialize import model_to_json, model_from_json
from rest_framework.response import Response
from rest_framework import status
from celery import shared_task
import requests


@shared_task
def load_model():
    try:
        # Load serialized model
        with open('./apilist/utils/model.json', 'r') as fin:
            m = model_from_json(fin.read())
        # Cache the model for future requests
        cache.set('serialized_model', m, timeout=None)  # Set timeout=None for indefinite caching
    except FileNotFoundError:
            # Return an error response if the model file is not found with status code 500 (Internal Server Error)
            return Response({'error': 'Model file not found'}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)

10. Loading from the Redis Cache

Modify your API view code for getting the model from the Redis cache:

class WeatherPredictionAPIView(APIView):
    def post(self, request):
        try:
            m = cache.get('serialized_model')
            if m is None:
                # Enqueue a background task to load the model
                m = load_model()

                cache.set('serialized_model', m, timeout=None)  

            # print("checking", m)

            input_date_str = request.data.get('date')
            input_date = datetime.strptime(input_date_str, '%Y-%m-%d')


            future_df = pd.DataFrame({'ds': [input_date]})


            forecast = m.predict(future_df)


            predicted_temperature = forecast.loc[0, 'yhat']


            return Response({'predicted_temperature': predicted_temperature}, status=status.HTTP_200_OK)

        except FileNotFoundError:
            return Response({'error': 'Model file not found'}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)

        except Exception as e:

            return Response({'error': str(e)}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)

At first, I tried the @sync_to_async decorator methods for asynchronously loading the Prophet model. Turns out, the Prophet does not allow asynchronous operation. That's why I implemented this optimization using Celery and Redis. Now, the API response time is a few milliseconds.

I pushed the code in the master; you can check - https://github.com/Afroza2/Strativ-AB-Travel-Management/tree/master. You need to run these commands in each terminal while keeping your Redis server running in the background. The basic setup with Celery and Redis in Django is discussed in the previous post:

python manage.py makemigrations
python manage.py migrate
python manage.py migrate django_celery_beat

I enjoyed doing this project. Let me know what you think. I hope to write about many more exciting projects ahead.

I wish you eternal sunshine!

How to Do Periodic Tasks in a Django REST API Using Celery and Redis? - A Beginner-Friendly Guide

Afroza Nowshin — Wed, 25 Oct 2023 09:49:08 +0000

Today, I want to share how to solve an interesting problem I recently encountered. I was a noob to this and had to come across some concepts I had never encountered before. Here is the problem statement:

Create an API where you take your friend's location, destination, and travel date. Compare the temperature of those two locations at 2 PM on that day and return a response deciding whether they should travel there. Hint: You might need to periodically fetch data and store it somewhere.

For fetching data periodically, we are going to use Celery, and for storing the data, we are going to use Redis.

Here is the diagram of the whole task flow:

The breakdown of the task is:

Build a background system for periodical fetching of the weather data (Step 2 - 7)
Create an API for the recommendation task (Step 8)
Extract the places from the API and match them with the stored data while parallel updating the data (Step 8,9)

Celery helps you with asynchronous task management. Celery has 'workers' that execute your asynchronous tasks in the background and 'beats' that schedule the particular task based on the time interval you set, like a cron job. You can learn more from the documentation - https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html

Let's get started.

1. Install Dependencies

Celery works best with Python 3.7+ versions; therefore, set up a virtual environment based on these versions. On the other hand, Redis needs to be installed in your OS. First, in case you are in linux or WSL, run:

sudo apt update && apt upgrade

sudo apt install redis-server

In your terminal, run redis-cli and you will get the redis shell. To check if redis is connected, type ping in the redis shell. If it gives PONG , the installation and connection is successful.

Now in your python virtual environment, run

pip install django django-celery-results celery redis

2. Edit settings.py File

First, create a Django project and an app within it.

django-admin startproject travelmanagement
cd travelmanagement
python manage.py startapp apilist

{Python}

In order for your app to work with both Celery and Redis, you need to make some changes in your Django's settings.py file. Add the following snippets in your settings.py file:

CELERY_BROKER_URL = 'redis://127.0.0.1:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_BACKEND = 'django-db'
CELERY_IMPORTS = ('apilist.tasks',) # we will implement this function in the next step, generally it is appname.file_name
CELERY_TIMEZONE = 'Asia/Dhaka'

CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'

For best practice, keep all these in a .env file and load using python-dotenv library. Let's do the setup for the redis cache:

CACHES = {
   'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': os.getenv('REDIS_LOCATION'),
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            'CONNECTION_POOL_KWARGS': {'max_connections': 100},
        },
        'KEY_PREFIX': 'my_cache_prefix',  # Add a prefix to your cache keys
        'TIMEOUT': 300,  # Cache timeout in seconds (adjust as needed)
        'VERSION': 1,  # Cache version (change this if you change your cache keys structure)
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            'IGNORE_EXCEPTIONS': True,  # Ignore cache-related exceptions
        },
    }
}

3. Write the tasks.py File

Create a file named tasks.py inside your app folder. Here, you will write the function for the asynchronous task. For this project, we are required to fetch district-wise 2 p.m. temperature data periodically and store it. I had two URLs, one is a JSON for all districts, and another is an Open Meteo API link. First, we need to find out the structure of the JSON object, traverse it, and store it in a list. Later, you need to save the list in the redis cache with a key. Just like cookies, redis caches have two methods - set and get.

cache.set("your key", your_data)

The whole code will look like the following; print statements are for debugging purposes:

from django.core.cache import cache
from celery import shared_task
import requests


@shared_task
def fetch_and_store_temperature():
    try:
        response = requests.get('https://raw.githubusercontent.com/strativ-dev/technical-screening-test/main/bd-districts.json')
        response.raise_for_status()

        districts_data = response.json().get('districts', [])

        all_temperatures = []

        for district in districts_data:
            latitude = district.get('lat')
            longitude = district.get('long')

            if latitude is not None and longitude is not None:
                api_url = f'https://api.open-meteo.com/v1/forecast?latitude={latitude}&longitude={longitude}&hourly=temperature_2m&timezone=GMT&forecast_days=7'

                weather_response = requests.get(api_url)
                weather_response.raise_for_status()

                weather_data = weather_response.json()
                hourly_data = weather_data.get('hourly', {})
                temperature_at_2pm = hourly_data.get('temperature_2m', [])

                # Cache key for each district without specifying a travel date
                cache_key = f'temperature_at_2pm_{district["name"]}'
                cache.set(cache_key, temperature_at_2pm)

                # print("checking", cache.get(cache_key))

                all_temperatures.extend(temperature_at_2pm)

        # Store all temperatures in a single cache key
        cache.set('temperature_data', all_temperatures)

    except requests.exceptions.RequestException as e:
        # Handle API request exceptions
        print(f"Error fetching weather data: {e}")
    except Exception as e:
        # Handle other exceptions
        print(f"Error: {e}")

You must use the @shared_task decorator for any function that you want to run asynchronously.

4. Write the celery.py File

Go to your project directory and open a file named celery.py. This configuration is needed for scheduling your tasks. Before that, go to the project's init.py file and add the following lines:


from celery import app as celery_app

__all__ = ("celery_app",)

We will now set the necessary configurations in the celery.py file. Let's say, we want to execute our function every 15 minutes. Here is the code for that:

from __future__ import absolute_import, unicode_literals
import os

from celery import Celery
from celery.schedules import crontab
from django.conf import settings


os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'travelmanagement.settings') # project name.settings

app = Celery('travelmanagement') # provide your project name
app.conf.enable_utc = False

app.config_from_object(settings, namespace='CELERY')

app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print(f"Request: {self.request!r}")


app.conf.CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

app.conf.beat_schedule = {
# create an object for your scheduling your task
    'fetch-and-store-temp-data-contrab': {
        'task': 'apilist.tasks.fetch_and_store_temperature', #app_name.tasks.function_name
        'schedule': crontab(minute='*/15'), #crontab() means run every minute
        # 'args' : (..., ...) In case function takes parameters, add them here
    }
}

crontab schedules the task for you. Additionally, if your asynchronous function takes arguments, comment out the last line and do the necessary changes. A list of crontab commands can be found in the official document - https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html

5. Clear pycache

Sometimes, if the code of the workers are changed too frequently, celery may not pick up the changes. For that, delete your sqlit3 database and run the following command. This command deletes all the pycache files. Do not forget to add the pycache files in the .gitignore to avoid pushing those.

`
py3clean .

6. Run Migrations

If your task is still running based on old code, clear the migration files using the following commands (for Linux):

find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc"  -delete

Now run the migrations. Because you are using celery, you need to use 3 commands, instead of the usual 2:

python manage.py makemigrations
python manage.py migrate
python manage.py migrate django_celery_beat

7. Run Celery workers and beats

Everything is configured. Now open three terminals, two for Celery and one for Django. Run the following in the first terminal:


celery -A your_project_name worker -l info

Run the following command in the other terminal to start the celery beats

celery -A your_project_name beat -l info

And finally, run the server in the last terminal.

python manage.py runserver

8. Create a view for the API

This part is about creating your API for travel suggestions. What you would do -

Extract the districts - where you are and where you want to go
Extract the temperature data from the cache using the get method
As it is a lot of data, I used the average values of temperatures of the districts and compared these. If my location has higher temperature, I suggested to pick the destination district, otherwise not.

Here is how you can do it:

class DecisionMakingAPIView(APIView):

    def post(self, request):
        try:
            data = request.data
            source_district_name = data.get('source_district_name')
            destination_district_name = data.get('destination_district_name')

            # Retrieve temperatures for source district
            source_cache_key = f'temperature_at_2pm_{source_district_name}'
            source_temperature_at_2pm = cache.get(source_cache_key)

            # Retrieve temperatures for destination district
            destination_cache_key = f'temperature_at_2pm_{destination_district_name}'
            destination_temperature_at_2pm = cache.get(destination_cache_key)

            if source_temperature_at_2pm is not None and destination_temperature_at_2pm is not None:
                # Calculate average temperature for the source district
                source_average_temperature = sum(source_temperature_at_2pm) / len(source_temperature_at_2pm)

                # Calculate average temperature for the destination district
                destination_average_temperature = sum(destination_temperature_at_2pm) / len(destination_temperature_at_2pm)

                # Compare average temperatures and make a travel recommendation
                if source_average_temperature < destination_average_temperature:
                    recommendation = f"Traveling from {source_district_name} to {destination_district_name} is Recommended as destination is cooler."
                else:
                    recommendation = f"Traveling from {source_district_name} to {destination_district_name} is NOT Recommended as destination is hotter."

                return Response({'recommendation': recommendation}, status=status.HTTP_200_OK)
            else:
                return Response({'error': 'Temperature data not available for the specified source or destination district.'}, status=status.HTTP_404_NOT_FOUND)

        except Exception as e:
            # Handle exceptions if necessary
            return Response({'error': str(e)}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)

9. Add the URL for the API

Lastly, create a urls.py file inside your app folder and add the following:

from django.urls import path
from . import views
urlpatterns = [
 path('recommendation/', views.DecisionMakingAPIView.as_view(), name='travel-recommendation')
]

10. Check the API response

In the postman, add the form data like the following and see the response. Done!

I have written this blog so that I can use this as a future reference. Find the project in the github in case you want the code - https://github.com/Afroza2/Strativ-AB-Travel-Management.git

A Definitive Guideline for Creating APIs with Django and Neo4j database, Part-2

Afroza Nowshin — Sat, 26 Aug 2023 12:29:05 +0000

In today's post, we will breakdown the middle part of the architecture (referred to this post). In a word, we need to create views for each of the API URLs. First, all you have to do is create a file named "urls.py" inside your app folder. And in your project's urls.py file, add the following line under the url for the admin page:

path('', include('yourappname.urls')),

Create a superuser for your Django project by this command :

python manage.py createsuperuser

And provide your username and password for the admin.

1. Sign Up API view and URL

Your first job is to import all the dependencies and the model in your views.py file.


from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status
from .models import NeoUser

Response returns the API response in a JSON format and status is requires for required HTTP codes while querying.

Now you need to capture the fields of your API request and check for empty field. We are using Django's class-based views (CBV). And as we are posting data to our database, the request type will be POST, hence the function.


class SignUp(APIView):
    def post(self, request):
        username = request.data.get('username')
        email = request.data.get('email')
        password = request.data.get('password')

        if not username or not email or not password:
            return Response({"error": "Please provide username, 
       email, and password."}, status=status.HTTP_400_BAD_REQUEST)

The final step is to save the data as nodes and return the response as an HTTP code:

       try:
            user = NeoUser.nodes.get(username=username)
            return Response({"error": "Username already exists."}, status=status.HTTP_409_CONFLICT)
        except User.DoesNotExist:
            user = NeoUser(username=username, email=email, password=hashed_password)
            user.save()
            return Response({"message": "User created successfully."}, status=status.HTTP_201_CREATED)

Few things to keep in mind:

For creating an object or node, return HTTP code 201
For a successful query, return HTTP code 200
For bad request or sending a request without following the rules, return HTTP code 400
For unauthorized request, return HTTP code 401
And lastly, return HTTP code 404 when not found

Now go to your app's urls.py file and add the url for your created view:

from django.urls import path
from . import views

urlpatterns = [
    path('api/signup', views.UserSignupView.as_view(), name='user-signup'),


]

2. Users API view and URL

We will retrieve data from the database, so we need to implement a GET request. The rest of the steps are like the previous section.

views.py

class UserListView(APIView):


    def get(self, request):
        users = NeoUser.nodes.all()
        if not users:
            return Response({"message": "No users found"}, status=status.HTTP_404_NOT_FOUND)
        return Response(users, status=status.HTTP_200_OK)

Now inside yourapp/urls.py, add the following line right after the sign-up url:

...

path('api/userlist/', views.UserListView.as_view(), name='user-list'),

You have made 2 APIs now. Hit them in Postman like the following way while running your Django server(my field was named fullname in the model, change it to the field you put in the model):

3. Creating Serializers

If you hit your created User list api in Curl or Postman, You will see that the response json is showing null values for each field of a user:

{
"username": null,
"phone" : null
}

This is weird, because in your cypher-shell or Neo4j Desktop app, if you run the following command, there are nodes showing there:


MATCH (n:NeoUser) RETURN n LIMIT 25

Which yields :

While we are making our queries in the database, we need to "serialize" it, which converts the saved bits of the database into respective data types of each data. Additionally, there are verification steps for our sign up API such as, not signing up with the same username or phone number. We can either do this in our views.py or inside our serializer. I will show this process inside our serializer. Create a serializers.py inside your app's folder. This is the steps for making a serializer for your user data:

Import Rest Framework's serializer class and models

from rest_framework import serializers
from .models import NeoUser

Serialize each fields of the model

class Neo4jUserSerializer(serializers.Serializer):
    username = serializers.CharField()
    email = serializers.CharField()
    phone = serializers.CharField()
    password = serializers.CharField()

Validate and check for duplicates

def validate(self, data):
        id = data.get('uid')
        username = data.get('username')
        email = data.get('email')
        phone = data.get('phone')

        if NeoUser.nodes.filter(username=username):
            raise serializers.ValidationError("Username already 
                    exists")
...
        return data

Create a queryset with the validated data

def create(self, validated_data):       
        username = validated_data.get('fullname')
        email = validated_data.get('email')
        password = validated_data.get('password')
        phone = validated_data.get('phone')

Create a node with the queryset and save the node

        neo_user = NeoUser(uid=id,fullname=username, email=email,password=password,emp_id=emp_id,phone=phone)
        neo_user.save()

       # don't forget to return the queryset

       return validated_data

The entire seializer code is here:

from rest_framework import serializers
from .models import NeoUser

class Neo4jUserSerializer(serializers.Serializer):
    username = serializers.CharField()
    email = serializers.CharField()
    phone = serializers.CharField()
    password = serializers.CharField()

def validate(self, data):
        id = data.get('uid')
        username = data.get('username')
        email = data.get('email')
        phone = data.get('phone')

        if NeoUser.nodes.filter(username=username):
            raise serializers.ValidationError("Username already 
                    exists")
        # other field's duplicate checking
        return data

def create(self, validated_data):       
        username = validated_data.get('fullname')
        email = validated_data.get('email')
        password = validated_data.get('password')
        phone = validated_data.get('phone')

    neo_user = NeoUser(uid=id,fullname=username, email=email,password=password,emp_id=emp_id,phone=phone)
        neo_user.save()

       return validated_data

4. Adding serializer in the views

We are almost done. All you have to do is add the serializers in the views that you have created before. In sign up API, you have to make a validity check before saving.

class UserSignupView(APIView):
    def post(self, request):

      if request.method == 'POST':
        serializer = Neo4jUserSerializer(data=request.data)
        if serializer.is_valid():
            serializer.save()
            return Response({"message": "User registered successfully", "status": 200}, status=status.HTTP_201_CREATED)
        else:
            return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

For the user list API, you pass your node objects in the serializer and the serializer will return the serialized nodes for you:

class UserListView(APIView):

    def get(self, request):
        users = NeoUser.nodes.all()
        if not users:
            return Response({"message": "No users found"}, status=status.HTTP_404_NOT_FOUND)
        serializer = Neo4jUserSerializer(users, many=True)
        return Response(serializer.data, status=status.HTTP_200_OK)

You will now be able to see the data as the way they were saved in the model:


{
    username: 'user 1',
    phone: '01666666666'
}

Following these steps, you can make your own login API. If you are facing any issue, let me know in the comments below. In the next post, we will learn an important topic about user authentication in the system - JWT based Authentication. Take care and bye for now.

A Definitive Guideline for Creating APIs with Django and Neo4j database, Part-1

Afroza Nowshin — Thu, 24 Aug 2023 21:44:22 +0000

In this post, we will work on our graph database and setup all the necessary connections for building our APIs. As the name suggests, we will be using Neo4j database for our project. The entire architecture for this project is like this, courtesy to this (only) link on the subject matter:

Today, we will tear down the rightmost part of this diagram - Neo4j database and the model. Basically, we are using Django framework to provide the API URLs and what each URL will serve. The steps are as follows:

Download and install Neo4j
Download cypher-shell and update credentials
Setup Remote Database connection
Setup database connection with Django
Manipulate OGM
Migrate the database

1. Download and install Neo4j

Go to the official website of Neo4j and download the desktop application - here, and sign up an account. You will be provided with an activation key while downloading, copy it paste it during installation.

2. Download cypher-shell and update credentials

This is an optional step. Cypher shell provides you with a console where you can write CQL or Cypher Query Language, just like a python shell. You can download this from here - Official Download Link.

The default graph database in free version is neo4j with the password neo4j. You can change the password with the following command:

ALTER USER neo4j SET PASSWORD '<new-password>';

Mark this new password because it is needed for setting up the connection with the database.

3. Setup Remote Database connection

Go to Projects:
Create a new project with a name :
There is a dashboard for your project that looks like this:
Click Add and then the marked menu item:
Give a DBMS name and set the url like the format shown in the picture, then click Next:
Provide your database username and password, then click save:
Click Connect to set the connection with the database:
Click Open to see in the browser:
In the Browser, click the green highlighted db icon:
Select neo4j and it will now show the default movie table.

4. Setup database connection with Django

First, create a Django project and then create an app inside the project for the APIs.

django-admin startproject mysite
cd mysite/
python manage.py startapp yourapp

Now go to your settings.py and import config from neomodel. This is required to set up our database with Django. Your console should look like this:

Go to your settings.py and the following lines in the respective places:

from neomodel import config

config.DATABASE_URL = 'bolt://username:password@localhost:7687'

# you can either use bolt or neo4j for the connection type but bolt is preferable

Replace the username and the password with yours.

5. Manipulate OGM

From Django's official documentation, "Once you've created your data models, Django automatically gives you a database-abstraction API that lets you create, retrieve, update and delete objects." - this is the ORM or object-relational mapping layer. To break it down in simpler terms, Django models are like a mirror to your database; it models your database tables with its own abstraction layer. Your API requests are passed through this abstraction layer under the hood.

As we are using a graph database, we have nodes instead of tables and edges to capture the relationship. Neo4j provides OGM or object-graph mapping library with Neo4j-OGM. To use this, first import the field types and follow this code:


from neomodel import StructuredNode, StringProperty, IntegerProperty,UniqueIdProperty, RelationshipTo, BooleanProperty, EmailProperty, DateTimeProperty, DateProperty, config

class NeoUser(StructuredNode):
    uid = UniqueIdProperty()
    username = StringProperty()
    email = EmailProperty(unique_index=True)
    password = StringProperty()

class NeoPosts(StructuredNode):
    ....

To design the relationship, You have to write like the following (refer to ERD from the previous blog post):

# Relations :
    post = RelationshipTo(NeoPosts, 'CREATES')
    followers = RelationshipTo('NeoUser','FOLLOWERS')

6. Migrate the database

For Django models, you were supposed to run the following commands.

python manage.py makemigrations
python manage.py migrate

However, since we are making nodes in this model, we need to use the following command:

python manage.py install_labels

Now you can view your created model and the nodes like the following image:

That's it for this post. In the next post, we will learn to build the APIs for the model that we have built. I will publish the next part soon. Your feedback is highly appreciated.

A Definitive Guideline for Creating APIs with Django and Neo4j database, Part-0

Afroza Nowshin — Thu, 24 Aug 2023 04:34:59 +0000

Imagine, you are asked to create the "back-end" for a product or a website. Assuming this is your first project and you are a cheem doge (May he rest in peace), you may ask:

what do you mean by bamck-emd and what does it comtaim?

Before starting with the execution, I want to provide some contexts in this blog post. Eventually we will build a project with graph database-based backend. That is why, this is the zeroth part of this backend development series.

1. What is a Backend?

What no one told while I was starting in the industry, is what exactly is backend and why does there is also a front-end. Essentially, a software project is split into these two parts for convenience. The front-end part handles the data input and visualization part. On the other hand, back-end has the data storage and from that storage, you have to fetch the data, or do queries for data and display it through the front-end.

2. Database and Graph Database

Your database will store all your data. You need to pick a database that works for your project. For this project, I have picked a unique type of database - graph database. Unlike regular databases, graph database consists of nodes and edges. Each edge emulates a relationship between nodes.

3. Draw your ERD

This step involves making an Entity-Relationship diagram for your database(s). dbdiagram.io is a great place to do so, where your written code for tables will instantly be turned into ERD.

For this project, we will create an authentication system for social media users using graph database Neo4j and Django framework. Here is the graph ERD for this project:

4. API

Imagine, you have gone to a restaurant to dine. You do not go to the kitchen of the restaurant for food, do you? You are the front-end, and the kitchen is the database of food. There has to be someone who can serve you the food that you ordered. Ergo comes the APIs! API or Application Program Interface works as a liaison between your database and your front-end application.

Oops! The APIs have given you bonk instead of data.

5. RESTful API

RESTful API means REpresentational State Transfer API. What does that mean? These REST APIs are used to establish connections to machines and web services and do various database operations with less bandwidth. REST API architecture is built upon HTTP code response format. The results of a REST APIs can be displayed in JSON, HTML, XML formats although JSON is used the most now. There are GET, POST, PUT, PATCH and DELETE operations in RESTFUL API.

We will build some GET and POST APIs in this project.

6. Environment setup

We will start to implement things from the following posts. At first, we need to have python in our machines. It is always a best practice to install and maintain virtual environments for Python which enables you to use various Python versions in the same machine. I always use the following two links for the setup; these procedures work best in Linux and MacOs:

Installing pyenv
Installing virtualenv

For this project, I used python version 3.6.15 but you can use any python 3+ versions. Now install the basic packages using pip:

pip install Django djangorestframework

pip install neomodel django-neomodel neo4j-driver

Go to your settings.py file and add these along with your app name inside the INSTALLED_APPS:


INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'yourapp.apps.MyapiConfig',
    'rest_framework',
    'django_neomodel'
]

In the next tutorial, we will setup connections between our frameworks and database. Let me know in the comments about what do you have in mind regarding this.

Adding JWT Login Authentication in a React App

Afroza Nowshin — Sun, 28 Aug 2022 14:02:00 +0000

In recent times, one of the robust ways to authenticate a login credential is JWT authentication. Today, we will discuss how we can implement JWT authentication for a login app in React. For styling, I am using Material UI for React. Originally, I implemented JWT authentication over a private login API; replace this API with your private API.

First thing first, what is a JWT? JWT is short for JSON Web Token, and the JWT authentication is a compact way that ensures the safe transmission of information between parties as a JSON object. The data is verifiable because it is digitally signed using an HMAC algorithm (Hash-based Message Authentication Code).

This is how JWT authentication works :

What is happening here is, that when you sign up to a system for the first time, your password is encrypted with an HMAC algorithm, and therefore, it is hashed. After that, if you sign in, two types of token are generated against your password, both of which will expire after some time. This time is set by the back-end and for this reason, you need to save these tokens. After that, if you try to make any API request, you will send the tokens with the header. If your header format is invalid, so is your request. If you have a valid header, both the access and refresh tokens are checked.
If both the tokens are invalid, you force a logout action on whoever is trying to send that API request.

The steps that you need to follow to implement a JWT authentication in your front-end are as follows:

Create a React App, then create a Login component with user input and a submit button and call it from App.js.
Write a handleSubmit where you will save the user input as states and also the access and refresh tokens in the localStorage.
Send an API request. Check if there is a header, if not this is an invalid request. If there is a header, check for access and refresh token and match with the values of the localStorage.
Conditionally check status codes and hit Refresh API and Logout API if Needed
If both the tokens are valid after checking all these conditions, you will receive a response.

Now that you understand how JWT works, let's code this workflow.

Step-1: Creating a React App and a Login Component

For this tutorial, I am using node version 17 and the latest version of yarn package manager. Create a react app named jwt and got to the folder jwt:

npx create-react-app jwt
cd jwt/

You will write a component named Login.js. For styling, install Material-UI with yarn:

yarn add @material-ui/core
yarn add @mui/icons-material

Inside Login component, you will be needing two text boxes for username and password, and a button to submit.


import React, { useState } from "react";
import Button from "@material-ui/core/Button";
import CssBaseline from "@material-ui/core/CssBaseline";
import TextField from "@material-ui/core/TextField";
import Grid from "@material-ui/core/Grid";
import Typography from "@material-ui/core/Typography";
import Stack from "@material-ui/core/Stack";
import { useStyles } from "../style.js";

import { MuiThemeProvider, createTheme } from "@material-ui/core/styles";

 const theme = createTheme({
    palette: {
      primary: {
        main: "#121212",
      },
    },
  });
export default function Login(props) {
  const classes = useStyles(props);

  return(
      <Grid container className={classes.root} 
       item xs={12} md={5}>
            <CssBaseline />
            <MuiThemeProvider theme={theme}>
            <Grid>
                <Typography component="h1" variant="h5">
                   Sign in
                </Typography>
            <form className={classes.form} noValidate>
                  <TextField
                   label="Username"
                   variant="outlined"
                   color="primary"
                   margin="normal"
                   required
                   fullWidth
                   id="username"
                   name="username"
                  />
                 <TextField
                   label="Password"
                   type={"password"}
                   variant="outlined"
                   margin="normal"
                   required
                   fullWidth
                   id="password"
                   name="password"
                  />
                 <Button
                   type="submit"
                   fullWidth
                   color="primary"
                   variant="contained"
                   className={classes.submit}
                  >
                    Sign In
                </Button>
            </form>
          </Grid>
        </MuiThemeProvider>
    </Grid>
  )
}

I have separated the styling codes from this component. The style.js file looks like the following:

import { makeStyles } from "@material-ui/core/styles";


const useStyles = makeStyles((theme) => ({
  root: {
    height: "100vh",
    display: "flex",
    flexDirection: "column",
    margin: theme.spacing(12, 20),
    alignItems: "center",
  },
  form: {
    width: "100%",
    marginTop: theme.spacing(1),
  },
  submit: {
    margin: theme.spacing(3, 0, 2),
    backgroundColor: "#121212",
    color: "#fff"
  },

}));
export { useStyles };

Now, call it from App.js like the following:

...
function App() {
  return (
    <div className="App">
     <Login/>
    </div>
  );
}

You can use any styling libraries to achieve this. If you follow through this, the output should look like this:

Step-2: Write a Handle function and Store Access and Refresh Tokens

In this step, we will get the data from the user and do something with it. We need to save this data with React states at first, and then we have to send in the POST API.

Declare a state which will hold the username and the password:

const [data, setData] = useState({
    username: "",
    password: "",
  });

Inside the text fields, put the states in the value property:


         <TextField
              ...
              id="username"
              name="username"
              value={data.username}
              onChange={handleChange}
            />

           <TextField

              id="password"
              name="password"
              value={data.password}
              onChange={handleChange}
            />

We need to track the change of the values with a handler function named handleChange . Write the function like the following:

const handleChange = (e) => {
    const value = e.target.value;
    setData({
      ...data,
      [e.target.name]: value,
    });
  };

Instead of individually doing a setState operation, we at first are checking the value of the target, that is a text input. After that we are spreading the initial state object.

We will do all the API operation with Axios. Install it:

yarn add axios

Put your private login API in a global variable such as login:

const login = "your login api"

Remember how all the text boxes and Sign In button were wrapped in a form tag? We need to pass a handler function to handle the submit operation. Besides, in the API body of type POST, you are required to send the username and password. In our case we can send the entire state object in the API body. The whole handleSubmit will be like the following:

const handleSubmit = (e) => {
    e.preventDefault();
    const userData = {
      username: data.username,
      password: data.password,
    };
    axios.post(login, userData)
      .then((response) => {
        if (response.status === 200) {

          console.log(response.status);
          console.log(response.data);
        }
      })
      .catch((error) => {
        if (error.response) {
          console.log(error.response);
          console.log("server responded");

        } else if (error.request) {
          console.log("network error");
        } else {
          console.log(error);
        }
      });
  };

If you check your login API in Postman, you will get to see the access tokens and refresh tokens in the output.

We have to save this in the localStorage. Do it inside the following block:

if (response.status === 200) {
           localStorage.setItem("accessToken", response.data["access"]);
          localStorage.setItem("refreshToken", response.data["refresh"]);

         window.location.href = "/home";
        }

Write a bare minimum Home page component to see the change:

import React, { useState } from "react";

export default function Home(){
    return(
        <div>this is home page</div>
    )
}

Oh no! It's in the login page even after our login. That's because, we didn't make any management for routing. Install React Router first:

yarn add react-router-dom

And now add the following code in your App.js:

import { BrowserRouter, Route, Routes } from "react-router-dom";

import Home from "./components/Home";
function App() {
  return (
    <div className="App">
    <BrowserRouter>
            <Routes>
              <Route path="/" element={<Login />} />
              <Route path="/home" element={<Home />} />
            </Routes>
          </BrowserRouter>
    </div>
  );
}

Step-3: Check the Header of the API Request

Inside your home component, console the tokens:

const access = localStorage.getItem("accessToken");
console.log("access", access)

const refresh = localStorage.getItem("refreshToken");
console.log("refresh", refresh)

Let's make an API request in the home page. Place your private API in a variable:

const details = "any of your private API"

If you do not send a header to this API, you will receive an error code 403 (or whatever the back-end sends). You won't be able to make any API requests without a header. In order to send a header, create a object header in the following format:

const header = {
  "Content-Type": "application/json",
  Authorization: `Bearer ${access}`,
};

You can check in the postman like the following way:

Say, your API is a get request. You can write it like the following, with the header you created:

const [data, setData] = useState([]);
  useEffect(() => {
    axios
      .get(details, {
        headers: headers,
      })
      .then((response) => {
        setData(response.data);
      })
      .catch((error) => {
        console.log(error);
      });
  }, [data]);

return <div>this is home page {data}</div>;

Step-4: Conditionally check status codes and hit Refresh API and Logout API if Needed

In this section, we will explore the "Access token is valid?-No" logic branch. In a nutshell, if you find both the tokens invalid, send a request to your logout API. If just the refresh token is invalid, send a request to the Refresh Token API and update the refresh token.

Here's the code:

const logout_url = "your logout url"
const refresh_url = "your refresh url"

// inside the catch block of axios
.catch((error) => {
        if (error.response.status === 401) {
          console.log(error.response);

          axios.post(refresh_url, refresh).then((request) => {
            if (request.status === 200) {
              console.log("refresh valid");
              localStorage.setItem("accessToken", request.data["access"]);
            } else if (request.status === 401) {
              console.log("invalid so logout");
              axios.post(logout_url, headers).then((response) => {
                localStorage.setItem("accessToken", "");
                localStorage.setItem("refreshToken", "");
                console.log("access", response.data["access"]);
                console.log("refresh", response.data["refresh"]);

                window.location.href = "/login";
              });
            }

Step-5: Send Response

If you have followed all these steps, you have implemented an App which is robust with a JWT authentication. Find the final code in my github - https://github.com/Afroza2/React-Apps/tree/jwt .

I think I have given a layout for implementing a JWT authentication in your React app. In some cases, only the access token is provided by the back-end, so you can omit the whole check of the refresh token. Nevertheless, once you implement it, you don't have to think about security again.

Absolute Beginner's Guide to Learn JavaScript, Part-3

Afroza Nowshin — Sun, 21 Aug 2022 20:38:00 +0000

Over the course of this series "Start with Me", I am writing about the heavily used concepts of JavaScript that as a web-developer, you are expected to know. I am giving the tools in my posts so that you can feel less overwhelmed while it comes to build the UI and UX, or adding different functionalities in the website.

If you followed till now, you know about the DOM trees of a web page and how to access the objects of the trees, how to debug your code by writing messages and types in console.log() , how to manipulate arrays and objects with various methods etc. Today, we will learn how to handle the events that needs handling, or “Event Handling” in JavaScript, what is “this” keyword, arrow functions and how to set an interval or timeout for a specific action in JavaScript. Let’s get to it.

6. Event handling

In the first post I talked about HTML elements. Think about an element, button. What do you do with a button in a website? You either use it to submit something or click it to see something. Let’s write a code to put a button in a web page:

<!DOCTYPE html>
<html>

<body>

    <h1>Event Handling</h1>
    <div>
        <button>Click me!</button>
    </div>
</body>
<script>


</script>

</html>

It will look like the following:

Nothing will happen if you click the button, because it's a static HTML element. We need to do something about it. How about if we click, we will change the text of the paragraph below, to "Hello World"?

How to do that?

Clicking the button is an event, therefore we have to attach an action to this event which will handle the event of clicking the button. We will write a function in JavaScript named clickFunc like the following:

...
<script>

function clickFunc(){
      document.getElementById("paragraph").innerHTML = "Hello World"
}

</script>

The onclick event occurs when the user clicks on the button element. In order to do this, in the button event we will write in the onclick property:

...

    <div>
           <button onclick="clickFunc()">Click me!</button>
        <p id="paragraph">meow</p>
    </div>
</body>
...

</html>

That's it. Now if we click the button, we will see that the text meow is now changed.

Event Handling means dealing with the events that the HTML elements can cause in a web page. HTML elements need JavaScript to handle their events. If you think of ordering food online when you are hungry, that is an event handling. Being hungry is your event, and to handle this event you are ordering food online.

You could write the following code in another way, with addEventListener() method. This method takes an event and a function and attaches an event handler to an element. The code will be like the following:


<!DOCTYPE html>
<html>

<body>

    <h1>Event Handling</h1>
    <div>
        <button id="btn">Click me!</button>
        <p id="paragraph">meow</p>
    </div>
</body>
<script>

const element = document.getElementById("btn");
element.addEventListener("click", function() {
  document.getElementById("paragraph").innerHTML = "Hello World";
});

</script>

</html>

Notice how there's no name of function that we wrote here. It's an anonymous function, meaning "A function has no name".

Let's handle another event; text change in the input box. Put an input element in the web page like the following:

<body>

    <h1>Event Handling</h1>
    <div>
       <input type="text 
                      onchange="changeFunc(this.value)" value=""/>

        <p id="paragraph"></p>
    </div>
</body>

The event that we will be handling now is "onchange". We will write a function that will take whatever we write in the box and print in the paragraph tag beneath. The text we will write will be captured via the value property of input tag, as a result, we are passing the value in the changeFunc function as an argument. Notice that we wrote "this.value" instead of value as the argument. We will talk about what this means in the following section of this post.

Now, let's write the changeFunc body:

function changeFunc(val){
    document.getElementById("paragraph").innerHTML = val
}

</script>

The output should look like the following if you write a text and hit enter:

These are some of the most used event handling in JavaScript. Refer to this link of W3Schools to see the list of events. If you want to learn more, you can read this blog of Eloquent JavaScript.

7. this keyword, bind() method and Getting Smart with ES6:

In the previous section, we saw "this.value" as an argument. this keyword refers to an object in JavaScript. What does that mean? Well, this attaches itself to the way it is being used. In our context, we wanted the text that we wrote in input element. Writing something in input is an event, and here this refers to the element that received the event. Input element received our text event in its value property, therefore we wrote this.value as the function argument.

Usually, this refers to the global object, and it's also true in the case of a function. In an object method, this refers to the object.

Let's write an object that has the count of your animals:


const animal = {
    "cat": 2,
    "dog": 5,

  }


}

</script>

We then write an object method that will show the count in the paragraph tag:


const animal = {
    "cat": 2,
    "dog": 5,
    count : function() {
    return this.cat + " " + this.dog;
  }


}
document.getElementById("paragraph").innerHTML = animal.count();

</script>

You can see the count 2 and 5 below:

So far, we have learned what this means. This can refer to an object or a function. In order to bind a specific object or function which has its own this, to another entity, we use bind() method in JavaScript.

I think I have come across the best explanation about bind() from this stackoverflow link - What is bind() in JS. Based on an answer which has 89 upvotes, I used bind() to bind an object with a function like the following:


var ourFunc = function(value){
    console.log(this[value])
}

var obj = {
    x : 5,
    y : 10
};

obj.log = ourFunc.bind(obj)

obj.log('x')  // Output: 5

JavaScript has a major revision in ES6 which stands for ECMAScript 6. The foundation of React JS is on ES6. In ES6 the function is written as an arrow function like the following:


hello = () => {

}


() => {

}

This syntax is concise, and if you use {} you do not have to write "return" to return something. This arrow function is not hoisted, which means it must be defined before it is used. Another interesting fact is, arrow function doe not have this attached, which makes it not suitable for object methods, otherwise, it's my go to way to define functions in React JS because I don't have to worry about binding. ES6 ups your game once you master Vanilla JS and move on to learning a JavaScript framework. This link covers the basics of ES6 if you want to learn more - W3Schools

8. setTimeout() and setInterval()

Sometimes, it may so happen that you want to run a particular piece of your code or a function to run for a certain time. That's when you use setTimeout() method. The first parameter is the code or the function, and the second parameter is the amount of time after you which the execution stops. Time is calculated in milliseconds so 1 second = 1000 miliseconds.


<script>

const timeout = setTimeout(hello, 10000);

function hello(){
    document.getElementById("paragraph").innerHTML = "Hello World"
}

</script>

Output is shown after 10 seconds:

There is another method named setInterval(), which is used to execute something after the time in milliseconds that is passed in it. If you run the following code:


<script>

const interVal = setInterval(hello, 10000);

function hello(){
    document.getElementById("paragraph").innerHTML += "Meow"
}

</script>

You will see that the browser has Meow printed every 1 second:

That's a wrap for today. Hopefully, I will write about JavaScript Promise in the upcoming post. Till then, tata!