DEV Community: Wildan Aziz

Word Embeddings

Wildan Aziz — Thu, 12 Sep 2024 08:56:28 +0000

What are word embeddings?

Word embeddings are a type of word representation used in natural language processing (NLP) and machine learning. They involve mapping words or phrases to vectors of real numbers in a continuous vector space. The idea is that words with similar meanings will have similar embeddings, making it easier for algorithms to understand and process language.

Here’s a bit more detail on how it works:

Vector Representation: Each word is represented as a vector (a list of numbers). For example, the word "north" is represented as [0.30059, 0.55598, -0.040589, ...].
Semantic Similarity: Words that have similar meanings are mapped to nearby points in the vector space. So, "north" and "south" are close to each other, while "north" and "season" are further apart.
Dimensionality: The vectors are usually of high dimensionality (e.g., 50 to 300 dimensions). Higher dimensions can capture more subtle semantic relationships, but also require more data and computational resources.

Pre trained word embeddings

Pre-trained word embeddings are vectors that represent words in a continuous vector space, where semantically similar words are mapped to nearby points. They’re generated by training on large text corpora, capturing syntactic and semantic relationships between words. These embeddings are useful in natural language processing (NLP) because they provide a dense and informative representation of words, which can improve the performance of various NLP tasks.

What examples of pre-trained word embeddings?

Word2Vec: Developed by Google, it represents words in a vector space by training on large text corpora using either the Continuous Bag of Words (CBOW) or Skip-Gram model.
GloVe (Global Vectors for Word Representation): Developed by Stanford, it factors word co-occurrence matrices into lower-dimensional vectors, capturing global statistical information.
FastText: Developed by Facebook, it builds on Word2Vec by representing words as bags of character n-grams, which helps handle out-of-vocabulary words better.

Visualizing pre-trained word embeddings can help you understand the relationships and structure of words in the embedding space.

Import libraries.


import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.manifold import TSNE

Download pre-trained GloVe word vectors here.


# Load pre-trained GloVe embeddings
embeddings_index = {}
try:
    with open('glove.6B.50d.txt', encoding='utf-8') as f:
        for line in f:
            values = line.split()
            word = values[0]
            coefs = np.asarray(values[1:], dtype='float32')
            embeddings_index[word] = coefs
except:
    raise("Something error when opening the file.")

words = list(embeddings_index.keys())[:200]
words_vectors = np.array([embeddings_index[word] for word in words])

tsne = TSNE(n_components=2, random_state=0, perplexity=30, max_iter=300)
word_vectors_2d = tsne.fit_transform(words_vectors)

Visualizing the word embeddings.


# Convert numpy array to dataframe
df = pd.DataFrame({
    'x': word_vectors_2d[:, 0],
    'y': word_vectors_2d[:, 1],
    'word': words
})

# Set Seaborn style
sns.set(style='whitegrid')

# Create the plot
plt.figure(figsize=(5, 1.9), dpi=1000)
plt.grid(True)
ax = sns.scatterplot(data=df, x='x', y='y', hue='word', palette='tab20', legend=None, s=25, alpha=0.7)

# Customize plot appearance
ax.set(xlabel=None)
ax.set(ylabel=None)
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.set_xlim(-15, 15)  # Adjust these values as needed
ax.set_ylim(-15, 15)

# disabling xticks and yticks by Setting xticks to an empty list
plt.xticks([])
plt.yticks([])

# Show plot
plt.savefig('Word Embeddings 2D.png', bbox_inches='tight', transparent=True, dpi=1000)
plt.show()

Conclusion

Clustering of Words: Words that are semantically similar in meaning (according to their embeddings) tend to appear closer together in the plot. For instance, words like "king" and "queen" might form a cluster, while words like "dog" and "cat" would cluster separately in a different region.
Dimensionality Reduction: The word embeddings, which are typically high-dimensional vectors (e.g., 300-dimensional from Word2Vec or GloVe), have been reduced to 2D. While the original high-dimensional relationships are complex, t-SNE tries to preserve the local structures in the reduced 2D space, which may cause groups of words that were close in high dimensions to appear near each other in this plot.
Interpretation: Each point on the plot represents a word, and the distance between points reflects the similarity between the words in their original high-dimensional vector space. Clusters of words likely share semantic similarities, while isolated points might represent words that are more unique or less contextually related to others in the data set.

Multi-layer Graph Convolutional Network (GCN) Overview

Wildan Aziz — Tue, 03 Sep 2024 09:33:20 +0000

Multi-layer Graph Convolutional Network (GCN)

Multi-layer GCN is a type of neural network specifically designed for graph-structured data, where nodes represent entities and edges represent relationships between these entities. This neural network can be used to perform various tasks on graph dataset, such as node classification, link prediction, or clustering.

The Facebook Large Page-Page Network Dataset is a well-known dataset used for research in graph neural networks (GNNs) and social network analysis. This dataset represents a network of Facebook pages (nodes) where edges indicate mutual likes between pages.

Here's a step-by-step breakdown of how you can apply a multi-layer GCN to the Facebook Large Page-Page Network Dataset:

1. Data preparation

Nodes: Each node represents a Facebook page.
Edges: Edges represent mutual likes between two Facebook pages.
Features: Typically, each node would have a feature vector (e.g., one-hot encoded or continuous attributes). If the dataset doesn’t provide features, you may consider using identity matrices or applying node embeddings.

2. Graph Convolutional Network (GCN) Architecture

Input Layer: Takes node features and the adjacency matrix as inputs.
Hidden Layers: Each layer performs a graph convolution operation, aggregating information from the neighboring nodes. The depth (number of layers) depends on the complexity of the task.
Output Layer: For tasks like node classification, this layer produces a label prediction for each node. For link prediction, this layer could output the probability of an edge between pairs of nodes.

3. GCN Layers

Graph Convolution Layer: Each GCN layer can be mathematically described as.

4. Training the GCN

Loss Function: For node classification, you might use cross-entropy loss. For link prediction, a binary cross-entropy loss is common.
Optimizer: Stochastic Gradient Descent (SGD), Adam, or another optimizer to minimize the loss function.
Batching: Due to the potentially large size of the graph, batching techniques like GraphSAGE or minibatch gradient descent can be useful.

4. Evaluation

Node Classification: Measure accuracy, F1-score, or other metrics depending on the task.
Link Prediction: Use metrics like AUC-ROC or Precision-Recall.

5. Implementation

You can implement a multi-layer GCN using frameworks like PyTorch, which provide efficient operations on large-scale graphs.

Running Machine Learning Model in Azure Function

Wildan Aziz — Fri, 25 Aug 2023 14:49:30 +0000

☀️☀️☀️

Hi! How's your weekend? I hope it's great. I've been taking a long break since I moved into a new place. Anyway, I'd like to show you how I managed to utilize Azure Function to run amachine learning model. Hopefully, by the end of this post, you're able to understand how it works.

What's Azure Function?

Azure Function is just another service among other good services offered by Microsoft Azure. It's known as a serverless solution. That is, it allow us to focus on writing code instead of worrying about server infrastructure.

So what are the differences between running application on Azure Function and Virtual Machine? This is a good question. Firstly, Azure Function is more cost-effective. In fact, our application will only be charged depending on its running time. Secondly, it's also easier to manage. No need to configure extra settings whatsoever to run application.

Requirements

Python 3.7
Azure Functions Core (Not a hard requirement, but will make life easier)
Visual Studio Code + Azure Function Extension (Again, not a hard requirement)
Azure Function Service
Azure Storage Account
A machine learning model

Steps

In this article, we're making use of Graph Convolutional Network (GCN) machine learning model. To give more detail, it's a specific type of neural network model that accepts a graph data as an input. It then outputs a visualization of the graph's node embeddings. If you're interested to learn more about GCN, feel free to check out this blog first. Otherwise, go ahead clone the repository here.

Open the __init__.py and read the code carefully.

Notice there are HTTP request and response in the code. That is because our function is making use of a HTTP trigger. If you don't know what HTTP trigger is, it's simply a method to let us invoke our function using HTTP request. There are several triggers available to use in Azure Function such as HTTP, timer, and blob.

Open the local.settings.json.

Add your storage account connection string. So what's storage account? storage account is an object storage solution offered by the Azure cloud. We're using this storage to store our function's result.

Use the Azure Functions Core tool to initialize a Python function app.

$ func init

Run the following commands to create and activate a virtual environment named .venv.

$ py -3.7 -m venv .venv

$ .venv\scripts\activate

Install the dependencies by running the following command. Installation may take a few minutes.

$ pip install --no-cache-dir -r requirements.txt

Start the function by starting the local Azure Function runtime host.

$ func start

Note:

If encounter any errors, make sure the correct version of python is already installed. Otherwise, try upgrading the python pip and install the dependencies again. If the errors still persist, Google is the best place to ask questions.

Extra steps (Nice to follow!)

We may now test our function. Download graph dataset here. What is this dataset? It's a preprocessed dataset for our model.

Open the request.py and change the files location to your dataset file location.

Send a POST request to the runtime host on http://localhost:7071/api/nn by executing the code below. The process may take several minutes to complete.

$ py -3.7 request.py

In order to get the function's public endpoint, we need to deploy our function on cloud. Check out the complete documentation on how to here.

Conclusion

We've learned how to run a machine learning model with Azure Function by following the steps above. I hope you understand how Azure Function works in general by now. It's also important to note that it doesn't necessarily have to be a complex application, such as machine learning model, to run with Azure Function. We can use it for as simple as APIs that update a single row or column in database.

Deploy Docker Application to Virtual Machine Using Github Actions (CI/CD)

Wildan Aziz — Thu, 24 Aug 2023 02:52:20 +0000

Hi there! A quick question. Have you ever been unexpectedly productive during weekend? I remember spending all day doing a research on whether or not cereal was a soup 🥁.

Back to the topic, Microservice architecture has been quite popular for years. When building an application for Microservices, we probably want it to be deployed on a virtual machine let's say for scalling purposes.

In addition, a server automation can be very useful for us since it automates our application build, test, and deployment pipeline before deploying it to a virtual machine.

This post will help you carefully how to deploy your containerized application on a virtual machine using GitHub actions (CI/CD).

Prerequisites

Virtual machine - A running virtual machine instance. For this demo, I'm using a DigitalOcean droplet as I have it running already. But there shouldn't be a big difference with other virtual machine running on cloud platforms such as Amazon and Google Cloud Platform.
Docker - An open source software that consist of useful tools to containerize and manage our application. The docker software needs to be installed in the running virtual machine instance.
Git - A distributed version control system to manage our application code.
Web Application - An application server. A code example solely for demo can be viewed on github here.

What are the steps?

There are three files that are important for our deployment process.

workflow.yml tells us how our application deployed in a virtual machine.

Dockerfile is a set of instructions that run in a sequence in order to build an image of our application.

.dockerignore is a file that tells the docker what file(s) are not to include in the build process

Configure Nginx Reverse Proxy on Ubuntu VMs + SSL Encryption

Wildan Aziz — Tue, 16 May 2023 19:31:39 +0000

Hi folks! I recently just decided to host my own personal website on a cloud virtual machine. I've experimented a lot of things from doing it and now I'm more than excited to share some of my experiences. This article shows how I configured a reverse proxy and SSL encryption using Nginx.

What is a reverse proxy?

It's literally another server that sits in front of our website server. That said, whenever a request comes from a client to our website server, it will be intercepted by the reverse proxy server.

Why do we even need one?

There are, of course, advantages of using a reverse proxy.

Load balancing
Protection from attacks
SSL Encryption
Caching

Setting up project

Key technologies used

Virtual Machine - A running virtual machine instance. There are many virtual machines available for rent such as Amazon EC2, Google Compute Engine, Azure Virtual Machine, etc.
Domain Name - A registered domain name. Domain registrars like Namecheap, Google Domains, and GoDaddy provide and let us manage domain names.
Web Application - A server application. View code example on github here.
Nginx - Open source software that's often used for web serving and reverse proxying.

How it works

Configuring Nginx as a reverse proxy

Install Nginx and make sure it runs successfully.

$ sudo apt update
$ sudo apt install nginx
$ sudo service nginx status

Edit server configuration file in /etc/nginx/sites-available/.
```
$ sudo nano /etc/nginx/sites-available/wildanazz.com
```
Make sure that there are no syntax errors in the server configuration file.
```
$ sudo nginx -t
```

Enable the server configuration file by creating a link from sites-available directory to sites-enabled directory.

$ sudo rm link /etc/nginx/sites-enabled/default
$ sudo ln -s /etc/nginx/sites-available/wildanazz.com /etc/nginx/sites-enabled/wildanazz.com

Testing application

Clone a demo server application.

$ git clone https://github.com/wildanazz/demo-web-application.git

Run it.

$ sudo apt install nodejs && sudo npm install yarn -g
$ cd demo
$ yarn --frozen-lockfile && yarn start

Copy and enter the ip address of the running virtual machine in a browser.

Installing SSL/TLS certificate

Before installing SSL/TLS certificate, the registered domain name needs to be pointing to the running virtual machine.

Install Certbot to obtain a free SSL certificate.
```
$ sudo snap install --classic certbot
```
Run this code to let Certbot edit Nginx configuration automatically.
```
$ sudo certbot --nginx
```
Certbot can also renew the certificate automatically before it expires by running this code.
```
$ sudo certbot renew --dry-run
```

Summary

In this article, I discussed how I set up Nginx reverse proxy server and added a SSL certificate for my website from the ground up. If you decide to host your own website on a virtual machine, I highly recommend you to utilize a reverse proxy server. If you have found something I could improve or that you would have done differently? Let me know in the comments.