DEV Community

Cover image for How to run LLM modal locally with Hugging-Face πŸ€—
Roshan Sanjeewa Wijesena
Roshan Sanjeewa Wijesena

Posted on

6

How to run LLM modal locally with Hugging-Face πŸ€—

Welcome Back - In this topic i would like to talk about how to download and run any LLM modal into your local machine/environment.

We again use hugging-face here πŸ€—. You would need hugging-face API key first.

Run below code to download google/flan-t5-large in to your local machine, it will take a while and you will see the progress in your jupyter notebook.

from langchain.llms import HuggingFacePipeline
import torch
from transformers import pipeline,AutoTokenizer,AutoModelForCausalLM,AutoModelForSeq2SeqLM
model_id= "google/flan-t5-large"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

pipe = pipeline("text2text-generation", model=model, tokenizer=tokenizer,max_length=128)
local_llm = HuggingFacePipeline(pipeline=pipe)

prompt = PromptTemplate(
    input_variables=["name"],
    template="Can you tell me about footballer {name}",
)
chain = LLMChain(prompt=prompt, llm=local_llm)
chain.run("messi")

Enter fullscreen mode Exit fullscreen mode

API Trace View

Struggling with slow API calls? πŸ‘€

Dan Mindru walks through how he used Sentry's new Trace View feature to shave off 22.3 seconds from an API call.

Get a practical walkthrough of how to identify bottlenecks, split tasks into multiple parallel tasks, identify slow AI model calls, and more.

Read more β†’

Top comments (2)

Collapse
 
jayanath_liyanage_3991f20 profile image
Jayanath Liyanage β€’

Great work πŸ‘

Collapse
 
ouyangzhigang profile image
Voyagergle β€’

let me try

The Most Contextual AI Development Assistant

Pieces.app image

Our centralized storage agent works on-device, unifying various developer tools to proactively capture and enrich useful materials, streamline collaboration, and solve complex problems through a contextual understanding of your unique workflow.

πŸ‘₯ Ideal for solo developers, teams, and cross-company projects

Learn more

πŸ‘‹ Kindness is contagious

Please leave a ❀️ or a friendly comment on this post if you found it helpful!

Okay