DEV Community

Cover image for Journey towards self hosted AI code completion
Simon
Simon

Posted on

Journey towards self hosted AI code completion

So after I saw a video of Llama3 8b doing 800-1000 tokens/s I instantly thought it would be phenominal to have that on my laptop instantly giving me responses that I could select or discard with the added side benefit would be to learn some new tech.

I have once coded a small AI trained on like 100.000 items trying to predict housing prices, let me say it was not great.

But again I don't have billions upon billions of records to train on nor the compute to train a decent model but I like the control over my coding environment.

The first goal

Having llama 8b running locally autocompleting code snippets.

Yeah I know the barrier are low but is most def a useful starting point. I personally dont believe in setting unobtainable goals because failing at something hard is not as great as succeeding at small increments.

Lastly I intent to make a guide on my complete journey sub and like to get notified when new content is published

Banner video is from groq.com a great service where you can test these open source models like llama not sponsored by them just wanted to credit

Heroku

Simplify your DevOps and maximize your time.

Since 2007, Heroku has been the go-to platform for developers as it monitors uptime, performance, and infrastructure concerns, allowing you to focus on writing code.

Learn More

Top comments (0)

The Most Contextual AI Development Assistant

Pieces.app image

Our centralized storage agent works on-device, unifying various developer tools to proactively capture and enrich useful materials, streamline collaboration, and solve complex problems through a contextual understanding of your unique workflow.

👥 Ideal for solo developers, teams, and cross-company projects

Learn more

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay