DEV Community

Cover image for Train LLM From Scratch
Fareed Khan
Fareed Khan

Posted on

Train LLM From Scratch

I created an end-to-end LLM training project, from downloading the training dataset to generating text with the trained model. It currently supports the PILE dataset, a diverse data for LLM training. You can limit the dataset size, customize the default transformer architecture and training configuration, and more.

This is what my 13 million parameter-trained LLM output looks like, trained on a Colab T4 GPU:

In \*\*\*1978, The park was returned to the factory-plate that the public share to the lower of the electronic fence that follow from the Station's cities. The Canal of ancient Western nations were confined to the city spot. The villages were directly linked to cities in China that revolt that the US budget and in Odambinais is uncertain and fortune established in rural areas.

It's more about learning than making the absolute best AI right away.

Code, documentation, and example can all be found on GitHub:

Github Link

Image of Timescale

Timescale – the developer's data platform for modern apps, built on PostgreSQL

Timescale Cloud is PostgreSQL optimized for speed, scale, and performance. Over 3 million IoT, AI, crypto, and dev tool apps are powered by Timescale. Try it free today! No credit card required.

Try free

Top comments (0)

Image of Timescale

Timescale – the developer's data platform for modern apps, built on PostgreSQL

Timescale Cloud is PostgreSQL optimized for speed, scale, and performance. Over 3 million IoT, AI, crypto, and dev tool apps are powered by Timescale. Try it free today! No credit card required.

Try free

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay