DEV Community

Cover image for Journey towards self hosted AI code completion
Simon
Simon

Posted on

Journey towards self hosted AI code completion

So after I saw a video of Llama3 8b doing 800-1000 tokens/s I instantly thought it would be phenominal to have that on my laptop instantly giving me responses that I could select or discard with the added side benefit would be to learn some new tech.

I have once coded a small AI trained on like 100.000 items trying to predict housing prices, let me say it was not great.

But again I don't have billions upon billions of records to train on nor the compute to train a decent model but I like the control over my coding environment.

The first goal

Having llama 8b running locally autocompleting code snippets.

Yeah I know the barrier are low but is most def a useful starting point. I personally dont believe in setting unobtainable goals because failing at something hard is not as great as succeeding at small increments.

Lastly I intent to make a guide on my complete journey sub and like to get notified when new content is published

Banner video is from groq.com a great service where you can test these open source models like llama not sponsored by them just wanted to credit

Sentry image

Hands-on debugging session: instrument, monitor, and fix

Join Lazar for a hands-on session where you’ll build it, break it, debug it, and fix it. You’ll set up Sentry, track errors, use Session Replay and Tracing, and leverage some good ol’ AI to find and fix issues fast.

RSVP here →

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay