DEV Community: Huynh-Chinh

Everything you need to start working with Qdrant

Huynh-Chinh — Thu, 30 Mar 2023 04:03:38 +0000

1. Introduction
Qdrant (quadrant) is a vector similarity search engine. Nó cung cấp một dịch vụ production-ready với API thuận tiện để lưu trữ, tìm kiếm và quản lý points-vectors có additional payload.
Qdrant được điều chỉnh để hỗ trợ lọc mở rộng. It makes it useful for all sorts of neural network or semantic-based matching, faceted search, and other applications.

2. Installation

Cách dễ nhất để sử dụng Qdrant là run một pre-built image bằng docker. Download image from DockerHub:

docker pull qdrant/qdrant

And run the service inside the docker:
docker run -p 6333:6333 \ -v $(pwd)/qdrant_storage:/qdrant/storage \ qdrant/qdrant

Trong trường hợp này, Qdrant sẽ sử dụng cấu hình mặc định và lưu trữ tất cả dữ liệu trong thư mục ./qdrant_storage

Và bây giờ Qdrant có thể truy cập được tại localhost:6333

3. Collections

4. Payload

5. Points

6. Search

7. Filtering

8. Optimizer

9. Storage

10. Indexing

11. Distributed deployment

12. Snapshots

13. Quantization

14. Integraions

15. Telemetry

16. Configuration

Ref.

Conclusion
Thank you very much for taking time to read this. I would really appreciate any comment in the comment section.
Enjoy🎉

Text Processing Techniques in NLP

Huynh-Chinh — Mon, 28 Feb 2022 10:23:57 +0000

1. Introduction
Text processing is one of the most common tasks used in machine learning applications such as language translation, sentiment analysis, spam filtering and many others.

Text processing refers only the analysis, manipulation, and generation of text, while natural language processing refers of the ability of a computer to understand human language in a valuable way. Basically, natural language processing is the next step after text processing.

For example, a simple sentiment analysis would require a machine learning model to look for instances of positive or negative sentiment words, which could be provided to the model beforehand. This would be text processing, since the model isn't understanding the words, it's just looking for words that it was programmed to look for.

A natural language processing model would be translating full sentences into another language. Since syntax varies from one language to another, the computer has to understand the meaning of the sentences in order to accurately translate them. But while NLP is more advanced than text processing, it always has text processing invilved as a step in the process.

2. Sentence cleaning
Sentence cleaning or noise removal is one of the first things you should be looking into when it comes to Text Mining and NLP. There are various ways to remove noise. This includes punctuation removal, special character removal, numbers removal, html formatting removal, domain specific keyword removal (e.g 'RT' for retweet), source code removal, header removal and more. It all depends on which domain you are working in and what entails noise for your task.

 """Basic cleaning of texts."""

 # remove html markup
 text=re.sub("(<.*?>)","",text)

 #remove non-ascii and digits
 text=re.sub("(\\W|\\d)"," ",text)

 #remove whitespace
 text=text.strip()

Above are some examples of ways to clean text. There will be different ways depending on the complexity of the data and the type of language.

3. Stop Words
Stop words are a set of commonly used words in a language. Examples of stop words in English are "a", "the", "is", "are" and etc. The intuition behind using stop words is that, by removing low information words from text, we can focus on the important words instead.

For example, in the context of a search system, if your query is "what is text preprocessing?", you want the search system to focus on surfacing documents that talk about text preprocessing over documents that talk about what is. This can be done by preventing all words from your stop word list from being analyzed. Stop words are commonly applied in search systems, text classification applications, topic modeling, topic extraction and others.

stopwords=['this','that','and','a','we','it','to','is','of','up','need']
text="this is a text full of content and we need to clean it up"

words=text.split(" ")
shortlisted_words=[]

#remove stop words
for w in words:
    if w not in stopwords:
        shortlisted_words.append(w)
    else:
        shortlisted_words.append("W")

print("original sentence: ",text)    
print("sentence with stop words removed: ",' '.join(shortlisted_words))

original sentence: this is a text full of content and we need to clean it up
sentence with stop words removed:  W W W text full W content W W W W clean W W

4. Regular Expression
Read more...

5. Tokenization
Read more...

6. N-grams (Unigram, Bigram, Trigram)
Text n-grams are commonly utilized in natural language processing and text mining. It’s essentially a string of words that appear in the same window at the same time.

N-grams is a technique to tokenize a string into substrings, by equally dividing an existing string into equal substrings of length N.

Basically, N is usually from 1~3, with the corresponding names unigram (N=1), bigram(N=2), trigram(N=3).
For a simple example we have the string "good morning", parsed into bigrams:
"good morning" => {"go", "oo", "od", "d ", " m", "mo", "or", "rn", "ni", "in", "ng"}

From the above example, you can easily imagine how N-gram works. To implement N-gram, just a few lines of code as follows, like the example written in python as follows:

def split_ngram(statement, ngram):
    result = []
    if(len(statement)>=ngram):
        for i in xrange(len(statement) - ngram + 1):
            result.append(statement[i:i+ngram])
    return result

Consider the sentence "I like dancing in the rain" see the Uni-Gram, Bi-Gram, and Tri-Gram cases below.

Uni-Gram: 'I', 'like', 'dancing', 'in', 'the', 'rain'
Bi-Gram: 'I like', 'like dancing', 'dancing in', 'in the', 'the rain'
Tri-Gram: 'I like dancing', 'like dancing in', 'dancing in the', 'in the rain'

Implement N-Grams using Python NLTK

from nltk import ngrams

sentence = 'I like dancing in the rain'

ngram = ngrams(sentence.split(' '), n=2)

for x in ngram:
    print(x)

('I', 'like')
('like', 'dancing')
('dancing', 'in')
('in', 'the')
('the', 'rain')

7. Text Normalization
A highly overlooked preprocessing step is text normalization. Text normalization is the process of transforming a text into a canonical (standard) form. For example, the word "gooood" and "gud" can be transformed to "good", its canonical form. Another example is mapping of near identical words such as "stopwords", "stop-words" and "stop words" to just "stopwords".

Text normalization is important for noisy texts such as social media comments, text messages and comments to blog posts where abbreviations, misspellings and use of out-of-vocabulary words (oov) are prevalent.

Here's an example of words before and after normalization:

8. Stemming
Stemming is the process of reducing inflection in words (e.g. troubled, troubles) to their root form (e.g. trouble). The “root” in this case may not be a real root word, but just a canonical form of the original word.

Stemming uses a crude heuristic process that chops off the ends of words in the hope of correctly transforming words into its root form. So the words “trouble”, “troubled” and “troubles” might actually be converted to troublinstead of trouble because the ends were just chopped off (ughh, how crude!).

There are different algorithms for stemming. The most common algorithm, which is also known to be empirically effective for English, is Porters Algorithm. Here is an example of stemming in action with Porter Stemmer:

import nltk
import pandas as pd
from nltk.stem import PorterStemmer

# init stemmer
porter_stemmer=PorterStemmer()

# stem connect variations
words=["connect","connected","connection","connections","connects"]
stemmed_words=[porter_stemmer.stem(word=word) for word in words]
stemdf= pd.DataFrame({'original_word': words,'stemmed_word': stemmed_words})

# stem trouble variations
words=["trouble","troubled","troubles","troublemsome"]
stemmed_words=[porter_stemmer.stem(word=word) for word in words]
stemdf= pd.DataFrame({'original_word': words,'stemmed_word': stemmed_words})

9. Lemmatization
Lemmatization on the surface is very similar to stemming, where the goal is to remove inflections and map a word to its root form. The only difference is that, lemmatization tries to do it the proper way. It doesn't just chop things off, it actually transforms words to the actual root. For example, the word "better" would map to "good". It may use a dictionary such as WordNet for mappings or some special rule-based approaches. Here is an example of lemmatization in action using a WordNet-based approach:

from nltk.stem import WordNetLemmatizer
nltk.download('wordnet')
# init lemmatizer
lemmatizer = WordNetLemmatizer()

#lemmatize trouble variations
words=["trouble","troubling","troubled","troubles",]
lemmatized_words=[lemmatizer.lemmatize(word=word,pos='v') for word in words]
lemmatizeddf= pd.DataFrame({'original_word': words,'lemmatized_word': lemmatized_words})
lemmatizeddf=lemmatizeddf[['original_word','lemmatized_word']]

#lemmatize goose variations
words=["goose","geese"]
lemmatized_words=[lemmatizer.lemmatize(word=word,pos='n') for word in words]
lemmatizeddf= pd.DataFrame({'original_word': words,'lemmatized_word': lemmatized_words})
lemmatizeddf=lemmatizeddf[['original_word','lemmatized_word']]

Ref:
[1] Text processing: what, why, and how

[2] All you need to know about text preprocessing for NLP and Machine Learning

Conclusion
Thank you very much for taking time to read this. I would really appreciate any comment in the comment section.
Enjoy🎉

Server Monitoring with Prometheus and Grafana setup in Docker and Portainer

Huynh-Chinh — Fri, 25 Feb 2022 08:15:15 +0000

I-Introduction
Portainer is a free Docker Container management tool with compact size and intuitive management interface, simple to deploy and use, allowing users to easily manage Docker host or Swarm cluster. This tool works on a container deployed on Docker Engine.

Grafana is a leading time-series, an open-source platform for visualization and monitoring. It allows you to query, visualize, set alerts, and understand metrics no matter where they are stored. You can create amazing dashboards in Grafana to visualize and monitor the metrics.

Prometheus is an open-source time-series monitoring system for machine-centric and highly dynamic service-oriented architectures. It can literally monitor everything. It integrates with Grafana very smoothly as Grafana also offers Prometheus as one of its data sources.

Prometheus Architecture: At its core, Prometheus has a main component called Prometheus Server, responsible for the actual monitoring work.

The Prometheus server consists of:

Time Series Database that stores all the metric data like current CPU usage, memory usage etc.
Data Retrieval Worker is responsible for all the data pulling activities from applications, services, servers etc. and pushing them into the database.
HTTP Server API meant to accept queries for the stored data. The Server API is used to display the data in a dashboard or a Web UI.

II-Install Portainer with Docker Swarm on Linux
1. Introduction
Portainer can be deployed on top of any K8s, Docker or Swarm environment. It works seamlessly in the cloud, on prem and at the edge to give you a consolidated view of all your containers.
Portainer consists of two elements, the Portainer Server and the Portainer Agent. Both elements run as lightweight Docker containers on a Docker engine. This document will help you deploy the Portainer Server and Agent containers on your Linux environment. To add a new Linux Swarm environment to an existing Portainer Server installation, please refer to the Portainer Agent installation instructions.
To get started, you will need:

The latest version of Docker installed and working
Swarm mode enabled and working, including the overlay network for the swarm service communication
sudo access on the manager node of your swarm cluster
By default, Portainer will expose the UI over port 9443 and expose a TCP tunnel server over port 8000.
The manager and worker nodes must be able to communicate with each other over port 9001.

2. Deployment
Portainer can be directly deployed as a service in your Docker cluster. Note that this method will automatically deploy a single instance of the Portainer Server, and deploy the Portainer Agent as a global service on every node in your cluster.
First, retrieve the stack YML manifest:

curl -L https://downloads.portainer.io/portainer-agent-stack.yml \
    -o portainer-agent-stack.yml

Then use the downloaded YML manifest to deploy your stack:

docker stack deploy -c portainer-agent-stack.yml portainer

Portainer Server and the Agents have now been installed. You can check to see whether the Portainer Server and Agent containers have started by running docker ps:

root@manager01:~# docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED              STATUS              PORTS                NAMES
59ee466f6b15   portainer/agent:2.11.1          "./agent"                About a minute ago   Up About a minute                        portainer_agent.xbb8k6r7j1tk9gozjku7e43wr.5sa6b3e8cl6hyu0snlt387sgv
2db7dd4bfba0   portainer/portainer-ce:2.11.1   "/portainer -H tcp:/…"   About a minute ago   Up About a minute   8000/tcp, 9443/tcp   portainer_portainer.1.gpuvu3pqmt1m19zxfo44v7izx

3. Logging In
Now that the installation is complete, you can log into your Portainer Server instance by opening a web browser and going to:

https://localhost:9443

Accessing the Portainer dashboard page, at the first interface after successful setup, the user will see the information of the successfully connected endpoint: the number of stacks, the number of containers, the number of volumes, the number of images and one Docker and host information.
...readmore...

III-Server Monitoring with Prometheus and Grafana setup in Docker and Portainer
Ref
1. Deploy Prometheus and Grafana

Create a new stack and define or paste the content of your docker-compose file in a box web editor.
image1
We need to deploy Prometheus and Grafana so the full content of docker-compose will look like this:

version: '3'

volumes:
  prometheus-data:
    driver: local
  grafana-data:
    driver: local

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - /etc/prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped

2. Configure Prometheus
Configuring Prometheus to monitor itself

$sudo mkdir /etc/prometheus

sudo vim /etc/prometheus/prometheus.yml

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  # external_labels:
  #  monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

3. Third-Party Exporters
Node_exporter is one of the exports that help prometheus collect metrics of the machine to be monitored and installed on the target machine (the machine being monitored).
cAdvisor is to analyze usage, performance, and many other metrics from Container applications, providing users with an overview of all running containers.

At stack monitoring we need to add the content config of cadvisor and node_exporter to the docker-compose file then update the stack.

  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'

  cadvisor:
    image: google/cadvisor:latest
    container_name: cadvisor
    # ports:
    #   - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg
    restart: unless-stopped

In the Prometheus config file we need to add node-exporter and cadvisor to the scrape configuration as follows:

 # Example job for node_exporter
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node_exporter:9100']

  # Example job for cadvisor
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

4. Visualize data with Grafana
In the Grafana page we built above at localhost:3000
Here we log in with the user and password as admin.

At the Home homepage, we need to add the data source, here we will add the data source from Prometheus because what we need to do is visualize on the Grafana dashboard to receive the data poured in from Prometheus.

At the setting we need to reconfigure the HTTP tab at the URL is http://prometheus:9090

After connecting your data source, at Home we create a new dashboard => Add an empty panel.

The Metrics browser helps us query the data to visualize.

5. Import Grafana Dashboards
In the Grafana homepage https://grafana.com at the Dashboards tab we will see there are many dashboards. In this article we will learn about node exporters and cadvisor.

Get node exporter full by copying the ID and pasting it into the dashboards server we built.

Get a Cadvisor exporter by copying the ID and pasting it into the dashboards server we built.

IV-Alertmanager with Slack
1.Introduction

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.
Readmore...

Conclusion
Thank you very much for taking time to read this. I would really appreciate any comment in the comment section.
Enjoy🎉

GETTING STARTED WITH NATURAL LANGUAGE PROCESSING

Huynh-Chinh — Fri, 25 Feb 2022 02:22:37 +0000

Introduction

Natural language processing (NLP) is concerned with enabling computers to interpret, analyze, and approximate the generation of human speech. Typically, this would refer to tasks such as generating responses to questions, translating languages, identifying languages, summarizing documents, understanding the sentiment of text, spell checking, speech recognition, and many other tasks. The field is at the intersection of linguistics, AI, and computer science.

Roadmap of NLP for Machine Learning

1. Pre-processing

Sentence cleaning
Stop Words
Regular Expression
Tokenization
N-grams (Unigram, Bigram, Trigram)
Text Normalization
Stemming
Lemmatization

read more...

2. Linguistics

Part-of-Speech Tags
Constituency Parsing
Dependency Parsing
Syntactic Parsing
Semantic Analysis
Lexical Semantics
Coreference Resolution
Chunking
Entity Extraction/ Named Entity Recognition(NER)
Named Entity Disambiguation/ Entity Linking
Knowledge Graphs

3. Word Embeddings

a. Frequency-based Word Embedding

One Hot Encoding
Bag of Words or CountVectorizer()
TFIDF of TFIDFVectorizer()
Co-occurrence Matrix, Co-occurrence Vector
Hashing Vectorizer

b. Pretrained Word Embedding

Word2Vec (by Google): CBOW, Skip-Gram
GloVe (by Stanford)
fastText (by Facebook)

4. Topic Modeling

Latent Semantic Analysis (LSA)
Probabilistic Latent Semantic Analysis (pLSA)
Latent Dirichlet Allocation (LDA)
lda2Vec
Non-Negative Matrix Factorization (NMF)

5. NLP with Deep Learning

Machine Learning (Logistic Regression, SVM, Naïve Bayes)
Embedding Layer
Artificial Neural Network
Deep Neural Network
Convolution Neural Network
RNN/LSTM/GRU
Bi-RNN/Bi-LSTM/Bi-GRU
Pretrained Language Models: ELMo, ULMFiT
Sequence-to-Sequence/Encoder-Decoder
Transformers (attention mechanism)
Encoder-only Transformers: BERT
Decoder-only Transformers: GPT
Transfer Learning

6. Example Use cases

Sentiment Analysis
Question Answering
Language Translation
Text/Intent Classification
Text Summarization
Text Similarity
Text Clustering
Text Generation
Chatbots (DialogFlow, RASA, Self-made Bots)

7. Libraries

NLTK
Spacy
Gensim

Conclusion
Thank you very much for taking time to read this. I would really appreciate any comment in the comment section.
Enjoy🎉