DEV Community

Cover image for 🧠Generative AI - 3
Abheeshta P
Abheeshta P

Posted on

🧠Generative AI - 3

How Are Generative AI Models Trained? 🏋️‍♂️

Generative AI models like GPT are trained in two main stages:

1. Unsupervised Pretraining 📚:

  • The model is fed massive amounts of text data (e.g., 45TB of text for GPT models).
  • The model learns patterns, language structures, grammar, semantics and general knowledge by predicting the next word/token in a sentence without labeled data.
  • This results in 175 billion parameters for models like GPT-3.

2. Supervised Fine-Tuning 🎯:

  • After pretraining, the model is fine-tuned on smaller, labeled datasets for specific tasks (e.g., summarization, sentiment analysis).
  • Fine-tuning ensures the model generates more accurate and task-relevant outputs. Eg : text summarization, Language translation etc

GPT-3

📝 Stay tuned in this learning journey for more on generative AI! I'd love to discuss this topic further – special thanks to Guvi for the course!

Do your career a favor. Join DEV. (The website you're on right now)

It takes one minute and it's free.

Get started

Top comments (1)

Collapse
 
abheeshta profile image
Abheeshta P

Next will be on RAG stay tuned!!

👋 Kindness is contagious

Immerse yourself in a wealth of knowledge with this piece, supported by the inclusive DEV Community—every developer, no matter where they are in their journey, is invited to contribute to our collective wisdom.

A simple “thank you” goes a long way—express your gratitude below in the comments!

Gathering insights enriches our journey on DEV and fortifies our community ties. Did you find this article valuable? Taking a moment to thank the author can have a significant impact.

Okay