DEV Community

Cover image for 🧠Generative AI - 3
Abheeshta P
Abheeshta P

Posted on

🧠Generative AI - 3

How Are Generative AI Models Trained? 🏋️‍♂️

Generative AI models like GPT are trained in two main stages:

1. Unsupervised Pretraining 📚:

  • The model is fed massive amounts of text data (e.g., 45TB of text for GPT models).
  • The model learns patterns, language structures, grammar, semantics and general knowledge by predicting the next word/token in a sentence without labeled data.
  • This results in 175 billion parameters for models like GPT-3.

2. Supervised Fine-Tuning 🎯:

  • After pretraining, the model is fine-tuned on smaller, labeled datasets for specific tasks (e.g., summarization, sentiment analysis).
  • Fine-tuning ensures the model generates more accurate and task-relevant outputs. Eg : text summarization, Language translation etc

GPT-3

📝 Stay tuned in this learning journey for more on generative AI! I'd love to discuss this topic further – special thanks to Guvi for the course!

Top comments (1)

Collapse
 
abheeshta profile image
Abheeshta P

Next will be on RAG stay tuned!!