DEV Community

Cover image for Journey towards self hosted AI code completion
Simon
Simon

Posted on

Journey towards self hosted AI code completion

So after I saw a video of Llama3 8b doing 800-1000 tokens/s I instantly thought it would be phenominal to have that on my laptop instantly giving me responses that I could select or discard with the added side benefit would be to learn some new tech.

I have once coded a small AI trained on like 100.000 items trying to predict housing prices, let me say it was not great.

But again I don't have billions upon billions of records to train on nor the compute to train a decent model but I like the control over my coding environment.

The first goal

Having llama 8b running locally autocompleting code snippets.

Yeah I know the barrier are low but is most def a useful starting point. I personally dont believe in setting unobtainable goals because failing at something hard is not as great as succeeding at small increments.

Lastly I intent to make a guide on my complete journey sub and like to get notified when new content is published

Banner video is from groq.com a great service where you can test these open source models like llama not sponsored by them just wanted to credit

API Trace View

How I Cut 22.3 Seconds Off an API Call with Sentry 🕒

Struggling with slow API calls? Dan Mindru walks through how he used Sentry's new Trace View feature to shave off 22.3 seconds from an API call.

Get a practical walkthrough of how to identify bottlenecks, split tasks into multiple parallel tasks, identify slow AI model calls, and more.

Read more →

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay