BERT(Bidirectional Encoder Representation from Transformer)

Obieda Ananbeh — Fri, 31 Dec 2021 17:49:15 +0000

*What is BERT? *

BERT stands for (Bidirectional Encoder Representation from Transformer), was initially released in late 2018 by Google.

It has made a significant breakthrough in the field of natural language processing by achieving better outcomes in a variety of NLP tasks, including question answering, text creation, sentence categorization, and many more.

BERT's success is due in part to the fact that it is a context-based embedding model, as opposed to other prominent embedding models like word2vec, which are context-free.

Why BERT

BERT quickly became a hot topic in NLP. This is because:
It displays a high degree of linguistic sophistication, attaining human-level performance on some tasks.
It may be used for a wide range of jobs.
It has the advantages of pre-training and fine-tuning: BERT has been pre-trained on a huge text corpus by Google, and you can exploit its language comprehension by fine-tuning the pre-trained model on your own application (classification, entity recognition, question answering, etc.). With minimum design work, you may be able to acquire very precise outcomes on your assignment.

How BERT Work?

Transformer, an attention mechanism that learns contextual relationships between words (or sub-words) in a text, is used by BERT. Transformer has two different processes in its basic form: an encoder that reads the text input and a decoder that generates a job prediction. Only the encoder technique is required because BERT's purpose is to construct a language model.
The Transformer encoder reads the complete sequence of words at once, unlike directional models that read the text input sequentially (left-to-right or right-to-left). As a result, it is classified as bidirectional, however it is more correct to describe it as non-directional. This property enables the model to deduce the context of a word from its surrounds (left and right of the word)

References:

[1]S. Ravichandiran, Getting started with Google BERT : build and train state-of-the-art natural language processing models using BERT.
[2]C. Mccormick, “The Inner Workings-of-BERT,” 2020.
[3]https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270

DEV Community: Obieda Ananbeh

BERT(Bidirectional Encoder Representation from Transformer)