DEV Community: Abdul Rehman

Basic Keyword Tool in Python Flask Part-1

Abdul Rehman — Tue, 10 Oct 2023 10:35:53 +0000

In this post I am trying to build some keyword research tool. Let's start by going to Keyword Suggestion like keword Sheeter write some keyword and let it sheet the keywords for you. Once you are satisfied stop the sheeter and export the file into CSV. Now we need to clean the data. Most of the time we are interested in long tail keywords.

Here is the link to GitHub Repository Consider leaving a star.

Read CSV in Python using Pandas

Read that csv file in Python Pandas library like this

import pandas as pd
input_csv_file = 'esp8266_shitter.csv'
# Read CSV file into a pandas DataFrame with custom delimiter and quoting character
df = pd.read_csv(input_csv_file , sep=',', quotechar='"')
df.head()

In my case it print out the result like this

     id                            Keyword  ...                     Domain  Words
0  724           use esp8266 without wifi  ...  suggestqueries.google.com      4
1  723      esp8266 wifi without password  ...  suggestqueries.google.com      4
2  722          flash esp8266 without usb  ...  suggestqueries.google.com      4
3  721  esp8266 https without fingerprint  ...  suggestqueries.google.com      4
4  720          power esp8266 without usb  ...  suggestqueries.google.com      4

[5 rows x 8 columns]

You can get the idea that our keywords are in keyword column which we are interested in. So let's go back and filter this column and create a new list which only have the keywords which are longest then the desired length.

If you wanted to see the names of the columns you can do that with the help of print(df.columns). When I put this command in idle python shell it returns the following column names in my csv file.


Index(['id', 'Keyword', 'Length', 'Unnamed: 3', 'CPC', 'Search', 'Domain',
       'Words'],
      dtype='object')

If I try to print the Keyword Column it returns output like this

df['Keyword']
0               use esp8266 without wifi
1          esp8266 wifi without password
2              flash esp8266 without usb
3      esp8266 https without fingerprint
4              power esp8266 without usb
                     ...                
720                esp8266 board manager
721                       esp8266 pinout
722                        esp8266wifi.h
723                  esp8266 wifi module
724            esp8266 price in pakistan
Name: Keyword, Length: 725, dtype: object

Now I can ask simply for the keywords list which are longer then specific word length by this


filtered_keywords = df[df['Keyword'].apply(lambda x: len(x.split()) >= 4)]['Keyword']
print(filtered_keywords)

Convert the Visuals to Flask Web Application

Now that we have our basic structure ready we can start building the basic flask application to give some good visual and user interface like. Let's create a backend code with flask framework and create a templates folder and inside that templates folder create index.html file.
Here is the basic app.py file for the backend code



from flask import Flask, request, render_template
import pandas as pd

app = Flask(__name__)

# Function to filter keywords with at least n words
def filter_keywords(csv_file, min_words):
    df = pd.read_csv(csv_file)
    total_keywords = len(df)
    filtered_keywords = df[df['Keyword'].apply(lambda x: len(x.split()) >= min_words)]['Keyword']
    long_tail_keywords = filtered_keywords.tolist()
    total_long_tail_keywords = len(long_tail_keywords)
    return long_tail_keywords, total_keywords, total_long_tail_keywords

@app.route('/', methods=['GET', 'POST'])
def index():
    total_keywords = 0
    total_long_tail_keywords = 0
    long_tail_keywords = []
    min_words = 1  # Default minimum number of words

    if request.method == 'POST':
        file = request.files['file']
        min_words = int(request.form['minWords'])
        if file:
            long_tail_keywords, total_keywords, total_long_tail_keywords = filter_keywords(file, min_words)

    return render_template('index.html', keywords=long_tail_keywords, total_keywords=total_keywords, total_long_tail_keywords=total_long_tail_keywords)

if __name__ == '__main__':
    app.run(debug=True)

and This is the index.html file



<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Long-tail Keyword Finder</title>
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0-alpha1/dist/css/bootstrap.min.css" rel="stylesheet">
    <style>

</style>
</head>
<body>
    <div class="container mt-5">
        <h1 class="mb-4">Upload a CSV File</h1>
        <form method="POST" enctype="multipart/form-data">
    <div class="mb-3">
        <input type="file" class="form-control" name="file" accept=".csv" required>
    </div>
    <div class="mb-3">
        <label for="minWords" class="form-label">Minimum Words in Long-tail Keywords:</label>
        <input type="number" class="form-control" name="minWords" id="minWords" min="1" required>
    </div>
    <button type="submit" class="btn btn-primary">Upload and Find Long-tail Keywords</button>
</form>
        {% if keywords %}

<h2 class="mt-5">Long-tail Keywords Found:</h2>

<p class="mt-3">
    <span class="badge bg-danger">total Keywords: {{ total_keywords }}</span>
    <span class="badge bg-success">Long-tail: {{ total_long_tail_keywords }}</span>
</p>


<table class="table">
    <thead>
        <tr>
            <th scope="col">Keyword</th>
        </tr>
    </thead>
    <tbody>
        {% for keyword in keywords %}
        <tr>
            <td>{{ keyword }}</td>
        </tr>
        {% endfor %}
    </tbody>
</table>


{% endif %}
    </div>
</body>
</html>

Getting Started with Apollo Client and GraphQL in Next.js: A Step-by-Step Guide

Abdul Rehman — Sat, 11 Mar 2023 11:16:41 +0000

Here's a step-by-step guide on how to use Apollo Client library for client-side data collection from GraphQL-based API in Next.js framework:

Step 1: Install Required Dependencies
First, you need to install the required packages for the project by running the following command in your terminal:

npm install @apollo/client graphql

Step 2: Create an Apollo Client Instance
Create an Apollo Client instance in a separate file called apollo-client.js and export it to be used in the rest of the application.


import { ApolloClient, InMemoryCache } from "@apollo/client";

const client = new ApolloClient({
  uri: "https://countries.trevorblades.com",
  cache: new InMemoryCache(),
});

export default client;

Here, we've imported the ApolloClientand InMemoryCache classes from @apollo/client. The uri property specifies the endpoint of our GraphQL server. The cache property sets up an in-memory cache to store the results of our queries.

Step 3: Use the Apollo Provider Component in _app.js
To use the Apollo Client throughout our Next.js application, we need to wrap our root component with the ApolloProvider component. This component comes from the @apollo/client package.

Create a _app.js file in the pages directory, and use the ApolloProvider component to wrap the Component prop.


import { ApolloProvider } from "@apollo/client";
import client from "../apollo-client";

function MyApp({ Component, pageProps }) {
  return (
    <ApolloProvider client={client}>
      <Component {...pageProps} />
    </ApolloProvider>
  );
}

export default MyApp;

Step 4: Write GraphQL Queries
Next, create a .graphql file to define your GraphQL queries. For this example, we'll create a file called countries.graphql in the queries directory.


query Countries {
  countries {
    code
    name
    emoji
  }
}

This query requests the code, name, and emoji fields for all the countries in the API.

Step 5: Use useQuery Hook to Fetch Data
In your page component, import the useQuery hook from @apollo/client and use it to fetch the data. For this example, we'll create a file called index.js in the pages directory.


import { useQuery } from "@apollo/client";
import { gql } from "@apollo/client";

const COUNTRIES_QUERY = gql`
  query Countries {
    countries {
      code
      name
      emoji
    }
  }
`;

export default function Home() {
  const { loading, error, data } = useQuery(COUNTRIES_QUERY);

  if (loading) return <p>Loading...</p>;
  if (error) return <p>Error: {error.message}</p>;

  return (
    <div>
      <h1>Countries</h1>
      <ul>
        {data.countries.map((country) => (
          <li key={country.code}>
            {country.emoji} {country.name}
          </li>
        ))}
      </ul>
    </div>
  );
}

Here, we've imported the useQuery hook and the gql function from @apollo/client. The COUNTRIES_QUERY constant contains our GraphQL query defined in the .graphql file.

We're using the useQuery hook to fetch the data and store it in the data variable. The loading and error variables indicate the state of the query.

Open Some React Component with TailwindCSS Modals

Abdul Rehman — Fri, 17 Feb 2023 19:18:23 +0000

If you want to open a TailwindCSS Modal with React Component inside, you may need to create a new Component with the modal class. Here Is simple example of a Modal Component.

interface ModalProps {
  isOpen: boolean;
  onClose: () => void;
  children: React.ReactElement;
}

const MyModal: React.FC<ModalProps> = ({ isOpen, onClose, children }) => {
  return (
    <>
      {isOpen && (
        <div className="fixed z-10 inset-0 overflow-y-auto p-4 ">
          <div className="flex items-center justify-center min-h-screen pt-4 px-4 pb-20 text-center sm:block sm:p-0">
            <div
              className="fixed inset-0 transition-opacity"
              aria-hidden="true"
            >
              <div className="absolute inset-0 bg-gray-500 opacity-75"></div>
            </div>

            <span
              className="hidden sm:inline-block sm:align-middle sm:h-screen"
              aria-hidden="true"
            >
              &#8203;
            </span>

            <div
              className="inline-block align-bottom bg-gray-900 p-4 rounded-lg text-left overflow-hidden shadow-xl transform transition-all sm:my-8 sm:align-middle sm:max-w-lg sm:w-full"
              role="dialog"
              aria-modal="true"
              aria-labelledby="modal-headline"
            >
              <div className="absolute top-0 right-0 pt-4 pr-4">
                <button
                  onClick={onClose}
                  className="text-gray-600 hover:text-gray-400 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-blue-500"
                >
                  <span className="sr-only">Close</span>
                  <svg
                    className="h-6 w-6"
                    xmlns="http://www.w3.org/2000/svg"
                    fill="none"
                    viewBox="0 0 24 24"
                    stroke="currentColor"
                    aria-hidden="true"
                  >
                    <path
                      strokeLinecap="round"
                      strokeLinejoin="round"
                      strokeWidth={2}
                      d="M6 18L18 6M6 6l12 12"
                    />
                  </svg>
                </button>
              </div>

              {children}
            </div>
          </div>
        </div>
      )}
    </>
  );
};

export default MyModal;

and if you are planning to use this component to open some React Component as it's child you can simply do this by

import React, { useState } from "react";
import MyForm from "../Forms/MyForm";
import MyModal from "../Modals/MyModal";


function Welcome() {
  const [isOpen, setIsOpen] = useState(false);
  const onClose = () => {
    setIsOpen(false);
  };

  const onOpen = () => {
    setIsOpen(true);
  };

  return (
    <div className="mt-9">


      <button
        className="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded"
        onClick={onOpen}
      >
        Open Modal
      </button>
      <MyModal isOpen={isOpen} onClose={onClose}>
        {<MyForm/>}
      </MyModal>
    </div>
  );
}

export default Welcome;

How to create Dynamic Tables in React js

Abdul Rehman — Tue, 14 Feb 2023 19:11:11 +0000

Let's talk about Dynamic Tables which is a very common problem. We will do this without using any third party library. Let's see the problem we are dealing with. The issue is, we normally have many kind of data available in our application and we need to render that into the table. So instead of writing all kind of CSS and HTML over and over again, why not create a generic Table component and pass a JavaScript Object array to it. The Table Component will automatically render the Object Array.

Basic DynamicTable Component

So let's create a basic dynamic table component which accept two core props, one is the actual data to display in the table, second is the schema which tells have two basic information, one is label second is key, we can optionally pass classes or width kind of data. So our DynamicTable component props interface would look something like following

interface TableProps {
  data: { [key: string]: any }[];
  schema: { label: string; key: string;width:string}[];
  onUpdate?: (id: string | number) => void;
  onDelete?: (id: string | number) => void;
}

Here you may notice we additionally passed two more functions into the props interface, one is onUpdate which is optional, second is onDelete which is also optional.
The main reason for using these props is to tell the parent component is that specific row is requested to be updated or delete. These two functions are also a basic need when we have to update or delete a specific row level data.

Rendering the Table

Now let's focus on the actual rendering process. As you may already guessed that we have to look into the schema table and render the data object. That will look basically something like following,

  <div>

      <table className="table-auto w-full text-left overflow-x-auto">
        <thead
          className="bg-gray-800 text-white"
          onClick={() => setSelectedIndex(-1)}
        >
          <tr>
            {schema.map((col, index) => (

              <th key={index} className= {`px-4 py-2 `}>
                {col.label}
              </th>

            ))}
            {(onUpdate || onDelete) && <th className="px-4 py-2 w-52">Actions</th>}
          </tr>
        </thead>
        <tbody>
          {data.map((row, index) => (
            <tr
              key={index}
              className={`${
                index === selectedIndex ? "bg-gray-500" : ""
              } hover:bg-gray-500/75`}
              onClick={() => setSelectedIndex(index)}
            >
              {schema.map((col, colIndex) => (
                <td key={colIndex} className={`border px-4 py-2 `}>
                  {row[col.key]}
                </td>
              ))}
              {(onUpdate || onDelete) && (
                <td className="border px-4 py-2">
                  {onUpdate && (
                    <button
                      className="bg-blue-500 hover:bg-blue-400 text-white font-medium py-2 px-4 rounded-lg mr-2"
                      onClick={() => {
                        onUpdate(row.id);

                      }}
                    >
                     <FaEdit/>
                    </button>
                  )}
                  {onDelete && (
                    <button
                      className="bg-red-500 hover:bg-red-400 text-white font-medium py-2 px-4 rounded-lg"
                      onClick={() => {
                        onDelete(row.id);

                      }}
                    >
                      <FaTrash/>
                    </button>
                  )}
                </td>
              )}
            </tr>
          ))}
        </tbody>
      </table>
    </div>

We had used the React Icons library and used the Fa icons for rendering the action buttons for update or delete actions.

Compete DynamicTable Component Code

Here is the complete component code for rendering the dynamic table according to above mentioned procedure.

import React, { useState } from "react";
import { FaEdit, FaTrash } from "react-icons/fa";


interface TableProps {
  data: { [key: string]: any }[];
  schema: { label: string; key: string;width:string}[];
  onUpdate?: (id: string | number) => void;
  onDelete?: (id: string | number) => void;
}

const DynamicTable: React.FC<TableProps> = ({
  data,
  schema,
  onUpdate,
  onDelete,
}) => {
  const [selectedIndex, setSelectedIndex] = useState(-1);

  return (
    <div>

      <table className="table-auto w-full text-left overflow-x-auto">
        <thead
          className="bg-gray-800 text-white"
          onClick={() => setSelectedIndex(-1)}
        >
          <tr>
            {schema.map((col, index) => (

              <th key={index} className= {`px-4 py-2 `}>
                {col.label}
              </th>

            ))}
            {(onUpdate || onDelete) && <th className="px-4 py-2 w-52">Actions</th>}
          </tr>
        </thead>
        <tbody>
          {data.map((row, index) => (
            <tr
              key={index}
              className={`${
                index === selectedIndex ? "bg-gray-500" : ""
              } hover:bg-gray-500/75`}
              onClick={() => setSelectedIndex(index)}
            >
              {schema.map((col, colIndex) => (
                <td key={colIndex} className={`border px-4 py-2 `}>
                  {row[col.key]}
                </td>
              ))}
              {(onUpdate || onDelete) && (
                <td className="border px-4 py-2">
                  {onUpdate && (
                    <button
                      className="bg-blue-500 hover:bg-blue-400 text-white font-medium py-2 px-4 rounded-lg mr-2"
                      onClick={() => {
                        onUpdate(row.id);

                      }}
                    >
                     <FaEdit/>
                    </button>
                  )}
                  {onDelete && (
                    <button
                      className="bg-red-500 hover:bg-red-400 text-white font-medium py-2 px-4 rounded-lg"
                      onClick={() => {
                        onDelete(row.id);

                      }}
                    >
                      <FaTrash/>
                    </button>
                  )}
                </td>
              )}
            </tr>
          ))}
        </tbody>
      </table>
    </div>
  );
};

export default DynamicTable;

Using this Component

Now that we have our DynamicTable Component ready, you may want to know the basic example to use that component. Here we have basic usage example for this DynamicTable component.

import React from "react";
import DynamicTable from "../Dynamic/DynamicTable";


const schema = [
  { key: "id", label: "ID",width:'32'},
  { key: "name", label: "Name",width:'32' },
  { key: "email", label: "Email",width:'64' },
  { key: "age", label: "Age" ,width: '32'},
];

const data = [
  { id: 1, name: "John Doe", email: "johndoe@example.com", age: 32 },
  { id: 2, name: "Jane Doe", email: "janedoe@example.com", age: 28 },
  { id: 3, name: "Bob Smith", email: "bobsmith@example.com", age: 35 },
];

function StoreOwnerTable() {

  const handleUpdate = (id: string | number) => {
    console.log(`Update row: ${id}`);
  };

  const handleDelete = (id: string | number) => {
    console.log(`Delete row: ${id}`);
  };





  return (
    <div className="container mx-auto ">

      {simpleUsers ? (
        <DynamicTable
          schema={schema}
          data={data}
          onUpdate={handleUpdate}
          onDelete={handleDelete}
        />
      ) : null}

    </div>
  );
}

export default StoreOwnerTable;

Detecting Change in Time Series Data

Abdul Rehman — Fri, 27 Jan 2023 16:17:53 +0000

There are several ways to detect a major change in time series data or in a 1D dataset, some of the most common methods are:

Statistical Tests:

You can use statistical tests, such as the CUSUM test or the Page-Hinkley test, to detect changes in the mean or variance of the data.

Change Point Detection:

You can use change point detection algorithms, such as the Binary Segmentation or the Bayesian Change Point Detection, to identify the point at which the data distribution changes.

Machine Learning:

You can use machine learning methods, such as anomaly detection or novelty detection, to identify patterns or behaviors that deviate from the norm.

Visual Inspection:

You can also visually inspect the data, by plotting it and looking for sudden changes in the trend or pattern.

Point to remember

It is important to note that the best method to detect major changes in 1D dataset depends on the characteristics of the data and the specific requirements of the task.

It is also important to note that the 1D data might not have any meaningful change to detect, for example, if the data is random noise or constant data.

Python Implementations

Let's check out few python implementations of the change detection in the time series data.

Statistical Methods

First of all let's check simple methods which are statistical approach

The CUSUM Test Python code

The CUSUM test, or the Cumulative Sum test, is a statistical test used to detect a change in the mean of a time series data. Here is an example of how to implement the CUSUM test in Python on a time series dataset:

import numpy as np

# Create sample time series data
data = np.random.normal(0, 1, 100)

# Insert a change in the mean at index 50
data[50:] += 3

# Define the threshold for the test
threshold = 2

# Initialize the cumulative sum and the test statistics
cusum = 0
test_statistics = []

# Iterate over the data
for i in range(len(data)):
    # Update the cumulative sum
    cusum += data[i] - 0  # assuming the change is from 0 to a positive value
    test_statistics.append(cusum)
    # Check if the cumulative sum exceeds the threshold
    if cusum > threshold:
        print("Change detected at index", i)
        break

In this example, the CUSUM test is used to detect a change in the mean of the time series data from 0 to 3 at index 50. The threshold for the test is set to 2 and the cumulative sum is initialized to 0. The test statistics are calculated by iterating over the data and updating the cumulative sum at each step. If the cumulative sum exceeds the threshold, the change is detected and the index at which it occurred is printed.

It is important to note that the CUSUM test assumes that the change is from a known value to another, in this case, the change is assumed to be from 0 to a positive value.
Also, the threshold value should be set based on the specific requirements and the characteristics of the data.

Page-Hinklay Test

The Page-Hinkley test, also known as the Page test, is a statistical test used to detect a change in the mean of a time series data. Here is an example of how to implement the Page-Hinkley test in Python on a time series dataset:


import numpy as np

# Create sample time series data
data = np.random.normal(0, 1, 100)

# Insert a change in the mean at index 50
data[50:] += 3

# Define the parameters for the test
alpha = 0.01  # significance level
lambda_ = 0.1  # decay rate
delta = 3  # change point

# Initialize the test statistics
cumulative_sum = 0
test_statistics = []

# Iterate over the data
for i in range(len(data)):
    # Update the cumulative sum
    cumulative_sum += data[i] - delta
    test_statistics.append(cumulative_sum)
    # Update the test statistics
    if cumulative_sum > 0:
        cumulative_sum = cumulative_sum * lambda_
    else:
        cumulative_sum = 0
    # Check if the test statistics exceed the threshold
    if cumulative_sum > -np.log(alpha):
        print("Change detected at index", i)
        break

In this example, the Page-Hinkley test is used to detect a change in the mean of the time series data from 0 to 3 at index 50. The significance level is set to 0.01 and the decay rate is set to 0.1. The cumulative sum is initialized to 0. The test statistics are calculated by iterating over the data, updating the cumulative sum and decay it at each step. If the test statistics exceed the threshold, the change is detected and the index at which it occurred is printed.

The Page-Hinkley test is similar to the CUSUM test, but it has the ability to adapt to the variance of the data, which means it is more robust to false alarms. The parameters of the test should be set based on the specific requirements and the characteristics of the data, especially the significance level and the decay rate.

Bayesian Change Point Detection

There are several libraries and packages available in Python for Bayesian Change Point Detection. One popular library is the pystruct library, which provides a simple and efficient implementation of Bayesian Change Point Detection. Here's an example of how you can use it to detect change points in a time series data:

from pystruct.models import ChainCRF
from pystruct.learners import OneSlackSSVM
from pystruct.datasets import load_letters

# load the data
X, y, _, _ = load_letters()

# define the model
model = ChainCRF()

# define the learner
learner = OneSlackSSVM(model=model, C=0.1, max_iter=100)

# fit the model to the data
learner.fit(X, y)

# predict change points
change_points = learner.predict(X)

Another package for Bayesian Change Point Detection is changepoint package. It has the functionality for detecting change points in a variety of different statistical models.

from changepoint import changepoint

# Define data
data = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]

# Detect changepoints
changepoints = changepoint.detect(data, method='PELT')

In both examples above, the libraries are general purpose, It's up to you to define your own data and the specific model you want to use.

Anomaly Detection

Anomaly detection in time series analysis involves identifying unusual or unexpected patterns in the data that deviate significantly from the norm. There are several approaches to anomaly detection for time series data, including:

Statistical methods: These methods involve comparing the current data point to a statistical model of the normal behavior of the time series. For example, you can use a moving average or a Gaussian distribution to represent the normal behavior and then identify data points that fall outside of this model as anomalies.

Machine learning methods: These methods involve training a model on the normal behavior of the time series and then using this model to identify data points that deviate significantly from the norm. Popular machine learning models for anomaly detection include neural networks, decision trees, and clustering algorithms.

Time series decomposition: This method involves breaking down the time series into its constituent parts, such as trend, seasonal, and residual components, and then identifying anomalies in the residual component.

Spectral analysis: This method involves analyzing the frequency components of the time series data and identifying anomalies in the frequency domain.

There are also libraries in Python that can be used for anomaly detection such as anomalydetection package, pyculiarity package, and tsfresh package.

# Using anomalydetection package
from anomalydetection import detect_anoms

#define data
data = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]

#detect anomalies
result = detect_anoms(data, k=2, alpha=0.05, direction='both')

and similarly by using pyculiarity package

from pyculiarity import detect_ts

#define data
data = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]

#detect anomalies
result = detect_ts(data)

Gaussian Distribution for Anomaly Detection

Here's an example of how you can use a Gaussian distribution to detect anomalies in time series data using Python:


import numpy as np
from scipy.stats import norm

# Define the data
data = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]

# Compute the mean and standard deviation of the data
mean = np.mean(data)
std = np.std(data)

# Define the threshold for identifying anomalies
threshold = 3

# Define the Gaussian distribution
gaussian = norm(mean, std)

# Identify the anomalies
anomalies = []
for i in range(len(data)):
    if abs(gaussian.pdf(data[i]) - gaussian.pdf(mean)) > threshold:
        anomalies.append(i)

print("Anomalies:", anomalies)

This code first computes the mean and standard deviation of the time series data. Then, it defines a Gaussian distribution using these values. After that, it uses the abs(gaussian.pdf(data[i]) - gaussian.pdf(mean)) to compare the probability density function of each data point with the mean. The threshold is set to 3, if the difference is greater than the threshold value, it's considered as an anomaly and stored in the list.

You can adjust the threshold value to increase or decrease the sensitivity of the anomaly detection. Keep in mind that this is a simple example and a more robust implementation may need more advanced techniques such as using rolling window, dynamic threshold calculation and so on.

Moving Average for Anomaly Detection

Here's an example of how you can use a moving average to detect anomalies in real-time data using Python:

import numpy as np

# Define the data
data = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]

# Define the window size for the moving average
window_size = 5

# Compute the moving average of the data
moving_average = np.convolve(data, np.ones(window_size)/window_size, mode='valid')

# Define the threshold for identifying anomalies
threshold = 0.5

# Identify the anomalies
anomalies = []
for i in range(len(data) - window_size + 1):
    if abs(data[i+window_size-1] - moving_average[i]) > threshold:
        anomalies.append(i)

print("Anomalies:", anomalies)

This code first defines the window size for the moving average, in this case, it is 5. Then, it uses np.convolve method to compute the moving average of the data. The mode='valid' is used to ignore the data points that fall outside the window range. After that, it uses the abs(data[i+window_size-1] - moving_average[i]) to compare the each data point with the moving average. The threshold is set to 0.5, if the difference is greater than the threshold value, it's considered as an anomaly and stored in the list.

You can adjust the window size and threshold value to increase or decrease the sensitivity of the anomaly detection. Keep in mind that this is a simple example and a more robust implementation may need more advanced techniques such as using dynamic threshold calculation, data normalization and so on.

Novelty Detection using OCSVM

Novelty detection in time series data involves identifying new patterns or behaviors that deviate significantly from the previously observed data. One popular method for novelty detection in time series data is the One-Class Support Vector Machine (OCSVM) algorithm. Here's an example of how you can use the OCSVM algorithm to detect novelties in time series data using Python:

import numpy as np
from sklearn.svm import OneClassSVM

# Define the data
data = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]

# Fit the OCSVM model to the data
model = OneClassSVM(nu=0.1, kernel="rbf", gamma=0.1)
model.fit(data)

# Identify the novelties
novelties = model.predict(data)

# Print the index of the novelties
for i in range(len(novelties)):
    if novelties[i] == -1:
        print("Novelty detected at index:", i)

This code first fits the OneClassSVM model to the time series data using the radial basis function (RBF) kernel. Then, it uses the predict() method to identify the novelties in the data. The nu parameter represents an upper bound on the fraction of training errors and a lower bound of the fraction of support vectors. The gamma parameter is used to define the width of the RBF kernel. A smaller value will result in a wider kernel, a larger value will result in a narrower kernel. The output will be the index of the novelty data point, if any.

The OCSVM algorithm is sensitive to the parameters, so you may need to experiment with different parameter values to achieve optimal results. Additionally, this is a simple example and a more robust implementation may need more advanced techniques such as using rolling window, dynamic threshold calculation and so on

Data Clustering Algorithms that can be used for 1D dataset

Abdul Rehman — Fri, 27 Jan 2023 15:22:22 +0000

Many time we have to deal with 1D datasets. Just like the normal dataset we can deal with 1D dataset according to their nature and the nature of the problem. In today's post we are trying to see some clustering algorithms which could be used for 1D datasets.

Here is the some Clustering Algorithms that could be used for 1D datasets.

1.DBSCAN (Density-Based Spatial Clustering of Applications with Noise):

DBSCAN is a density-based clustering algorithm that groups together data points that are close to each other in the feature space. It can handle clusters of different shapes and sizes.
Here is the example code in python

from sklearn.cluster import DBSCAN
import numpy as np

# Create sample data
data = np.random.rand(100)

# Fit the DBSCAN model to the data
dbscan = DBSCAN(eps=0.3, min_samples=5)
clusters = dbscan.fit_predict(data.reshape(-1, 1))

# Get the cluster assignments for each data point
print(clusters)

In this example, eps is the maximum distance between two points for them to be considered as in the same neighborhood. min_samples is the minimum number of data points in a neighborhood for a point to be considered as a core point.

DBSCAN works well when the data points are dense in some areas and sparse in others, which is not the case with 1D data. Also it is important to note that DBSCAN relies on the notion of density and it is difficult to define density in one dimension.

2. Hierarchical Clustering:

Hierarchical clustering creates a tree-like representation of the data, where each data point is a leaf node, and the clusters are represented by branches and nodes. There are two main types of hierarchical clustering: Agglomerative (bottom-up) and Divisive (top-down)
Here is an example of 1D data clustering using Hierarchical Clustering (Agglomerative) in Python:

from scipy.cluster.hierarchy import linkage, dendrogram, fcluster
import numpy as np
import matplotlib.pyplot as plt

# Create sample data
data = np.random.rand(100)

# Perform hierarchical clustering
Z = linkage(data, method='ward')

# Create a dendrogram
dendrogram(Z)

# Determine the clusters by cutting the dendrogram at a threshold
clusters = fcluster(Z, t=1, criterion='distance')

# Print the cluster assignments
print(clusters)

# Show the dendrogram
plt.show()

In this example, the linkage method is set to "ward" which minimizes the variance of the distances of the linkage. You can also set it to "single", "complete", or "average" linkage. The fcluster function is used to assign each data point to a cluster based on the linkage matrix Z and the threshold t which is based on the distance and criterion.

It is important to note that Hierarchical Clustering relies on the notion of proximity and similarity between the data points, which is hard to define for one dimensional data

3.Gaussian Mixture Model (GMM) :

GMM is a probabilistic algorithm that models the data as a mixture of Gaussian distributions. It is a probabilistic algorithm and can be used for density estimation.

Here is an example of 1D data clustering using Gaussian Mixture Model (GMM) in Python:

from sklearn.mixture import GaussianMixture
import numpy as np

# Create sample data
data = np.random.rand(100)

# Fit the GMM model to the data
gmm = GaussianMixture(n_components=3)
gmm.fit(data.reshape(-1, 1))

# Get the cluster assignments for each data point
clusters = gmm.predict(data.reshape(-1, 1))

# Print the cluster assignments
print(clusters)

In this example, n_components is the number of Gaussian distributions to use in the mixture model. You can also use the fit_predict method instead of fit and predict separately.

keep in mind that GMM clustering relies on the notion of probability density of the data points, which is hard to define for one dimensional data

4. Mean-Shift:

Mean-shift is a non-parametric clustering algorithm that tries to find the mode (peak) of the data distribution.
Here is the python implementation

from sklearn.cluster import MeanShift
import numpy as np

# Create sample data
data = np.random.rand(100)

# Fit the Mean Shift model to the data
ms = MeanShift().fit(data.reshape(-1, 1))

# Get the cluster assignments for each data point
clusters = ms.predict(data.reshape(-1, 1))

# Print the cluster assignments
print(clusters)

5. Affinity Propagation:

Affinity propagation is a clustering algorithm that uses a message-passing mechanism to propagate information about the similarity between data points.

In any case, here is an example of 1D data clustering using Affinity Propagation in Python:

from sklearn.cluster import AffinityPropagation
import numpy as np

# Create sample data
data = np.random.rand(100)

# Fit the Affinity Propagation model to the data
af = AffinityPropagation().fit(data.reshape(-1, 1))

# Get the cluster assignments for each data point
clusters = af.predict(data.reshape(-1, 1))

# Print the cluster assignments
print(clusters)

6. K-Mean Clustering :

Here is an example of 1D data clustering using the K-Means algorithm in Python

from sklearn.cluster import KMeans
import numpy as np

# Create sample data
data = np.random.rand(100)

# Fit the K-Means model to the data
kmeans = KMeans(n_clusters=3)
kmeans.fit(data.reshape(-1, 1))

# Get the cluster assignments for each data point
clusters = kmeans.predict(data.reshape(-1, 1))

# Print the cluster assignments
print(clusters)

Keep in mind that all the above algorithms are not suitable for one dimensional data as it would not make sense to use them for one dimensional data as the algorithms are designed for multi-dimensional data.

Applications of 1D data Clustering

Clustering 1D data can have some applications in specific domains where the data is naturally one-dimensional, such as:

Time Series Analysis:

One-dimensional time series data, such as stock prices, can be clustered to identify patterns or trends.

Signal Processing:

In signal processing, one-dimensional signals can be clustered to identify similar patterns or features.

Genomics:

In genomics, one-dimensional DNA or RNA sequences can be clustered to identify patterns or functional regions.

Speech Recognition:

In speech recognition, one-dimensional audio signals can be clustered to identify similar sounds or words.

Natural Language Processing:

In natural language processing, one-dimensional text data can be clustered to identify similar topics or themes.

Final thoughts

It is important to note that 1D clustering is not a common task, in most cases, clustering algorithms are used in multi-dimensional data where they can better define the similarity/density/distance of the data points.

Small dataset and K-fold Cross validation code in python

Abdul Rehman — Fri, 27 Jan 2023 14:47:07 +0000

Machine learning sounds interesting but as a beginner it seems really hard to dive in. There are tons of tools and libraries to get started with like, tensorflow, pytorch, or the pretty old but powerful python library Scikit-learn.

As an Embedded System developers like me, you feel all the tutorials over the internet is overcrowded and assume we have super computer and can train whatever we want. Although the Google Colab makes a life bit easier but still doing stuff in our own pace is still a really craved thing.

I want to start playing with machine learning algorithms and don't want to prepare datasets with tons of images and train them and wait like weeks for them to get trained. So the small dataset is always a first choice for rapid prototyping of proof of concept.

Despite that Small dataset is still an active research topic which is discussed in many research papers. Today we are going to talk about few Machine learning models that are best suited for such problems.

Naive Byes Classifier
Random forest or Decision trees
KNN Classifier

But due to the limited dataset, even above mentioned modals soon seems to overfitting so make sure to cross-validate them.

Cross Validation

Cross-validation is primarily used in applied machine learning to evaluate how well a machine learning model performs on untrained data. In other words, estimating the model's performance in general when used to make predictions on data that was not used during model training. It is frequently used in applied machine learning to compare and select a model for a given predictive modeling problem.

K-Fold Cross Validation

K-Fold algorithm is simple and easy to understand and implement for cross-validation. Here is the python implementation of K-fold cross validation using scikit-learn library.

from sklearn.model_selection import KFold
from sklearn.linear_model import LinearRegression

# Create a Linear Regression model
model = LinearRegression()

# Create the k-fold cross validation object
kf = KFold(n_splits=5)

# Loop through each split of the data
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

    # Fit the model on the training data
    model.fit(X_train, y_train)

    # Evaluate the model on the test data
    score = model.score(X_test, y_test)
    print("Fold score: ", score)

This code uses k-fold cross validation with k=5, meaning it will divide the data into 5 subsets. The linear regression model is trained on four subsets and tested on one, then the process is repeated with a different subset used for testing each time. The score variable returns the accuracy of the model.

You can change the value of n_splits to set the number of folds you want to use. Also, you can replace the model (LinearRegression) with any other model that you want to use.

Following is the general process:

- Sort the dataset in a random order.
- Decide on k groups for the dataset.
- For each particular group:
    -- Consider using the group as a holdout or test data set.
    -- The remaining groups can serve as a training data set.
    -- Adapt a model to the training set, then evaluate it against the test set.
    -- Delete the model and keep the evaluation result
- Summarize the model's skill using a sample of the model evaluation scores.