DEV Community: Kaushal Powar

How to extract JSON from LLM response

Kaushal Powar — Wed, 27 Mar 2024 07:14:37 +0000

Hey there! 👋
Ever faced the challenge of extracting JSON from Open-source Large Language model output? 🤯
Discover insights and a step-by-step guide in my latest blog on mastering this task like a pro! 💡
From navigating prompt engineering to overcoming indent errors, I've got your back with practical tips and code snippets to simplify the process. 🚀
Don't miss out on the secrets to mastering JSON extraction!

Dive into the blog now and level up your LLM game. 🌟
Ready to tackle JSON extraction head-on? This post is a must-read! 📚💬

How to extract JSON from LLM response

Your support and feedback mean the world! Let's rock the AI and language model scene together. 🤖✨

Run Mixtral 8x7B 🚀(Mixtral of Experts)in free colab

Kaushal Powar — Sat, 20 Jan 2024 13:46:49 +0000

After releasing the Mixtral 8x7B model a few weeks ago they have recently released the paper Mixtral of Experts.

What is Mixtral 8x7B ??

It is a powerful language model that's a game-changer! Imagine Mistral 7B, but better – with 8 expert blocks in each layer. Here's the scoop: for every word, Mixtral picks two smart experts to do the heavy lifting, making each token a genius with access to a whopping 47 billion parameters.
What's really cool is that Mixtral only uses 13 billion active parameters during its magic, making it super efficient. Trained with a context of 32,000 tokens, Mixtral outshines big names like Llama 2 70B and GPT-3.5 in every benchmark – especially in math, code writing, and speaking multiple languages.

This model will not run on the T4 GPU that Google Colab provides for free, but I came across this GitHub repository that solves the issue.

How will MOE work on T4?

The four contributors achieved efficient inference of Mixtral-8x7B models through a combination of techniques

Mixed quantization with HQQ. They apply separate quantization schemes for attention layers and experts to fit the model into the combined GPU and CPU memory.
MoE offloading strategy. Each expert per layer is offloaded separately and only brought back to GPU when needed. We store active experts in an LRU cache to reduce GPU-RAM communication when computing activations for adjacent tokens.

You will approximately need 16 GB of VRAM and 11 GB of RAM.

Code

Install and import libraries

#fixing numpy
!pip uninstall numpy --yes
!pip install numpy==1.24.4

# fix numpy in colab
import numpy
from IPython.display import clear_output

# fix triton in colab
!export LC_ALL="en_US.UTF-8"
!export LD_LIBRARY_PATH="/usr/lib64-nvidia"
!export LIBRARY_PATH="/usr/local/cuda/lib64/stubs"
!ldconfig /usr/lib64-nvidia

!git clone https://github.com/dvmazur/mixtral-offloading.git --quiet
!cd mixtral-offloading && pip install -q -r requirements.txt
!huggingface-cli download lavawolfiee/Mixtral-8x7B-Instruct-v0.1-offloading-demo --quiet --local-dir Mixtral-8x7B-Instruct-v0.1-offloading-demo

clear_output()

import sys

sys.path.append("mixtral-offloading")
import torch
from torch.nn import functional as F
from hqq.core.quantize import BaseQuantizeConfig
from huggingface_hub import snapshot_download
from IPython.display import clear_output
from tqdm.auto import trange
from transformers import AutoConfig, AutoTokenizer
from transformers.utils import logging as hf_logging

from src.build_model import OffloadConfig, QuantConfig, build_model
Initialize model

model_name = "mistralai/Mixtral-8x7B-Instruct-v0.1"
quantized_model_name = "lavawolfiee/Mixtral-8x7B-Instruct-v0.1-offloading-demo"
state_path = "Mixtral-8x7B-Instruct-v0.1-offloading-demo"

config = AutoConfig.from_pretrained(quantized_model_name)

device = torch.device("cuda:0")

##### Change this to 5 if you have only 12 GB of GPU VRAM #####
offload_per_layer = 4
# offload_per_layer = 5
###############################################################

num_experts = config.num_local_experts

offload_config = OffloadConfig(
    main_size=config.num_hidden_layers * (num_experts - offload_per_layer),
    offload_size=config.num_hidden_layers * offload_per_layer,
    buffer_size=4,
    offload_per_layer=offload_per_layer,
)


attn_config = BaseQuantizeConfig(
    nbits=4,
    group_size=64,
    quant_zero=True,
    quant_scale=True,
)
attn_config["scale_quant_params"]["group_size"] = 256


ffn_config = BaseQuantizeConfig(
    nbits=2,
    group_size=16,
    quant_zero=True,
    quant_scale=True,
)
quant_config = QuantConfig(ffn_config=ffn_config, attn_config=attn_config)


model = build_model(
    device=device,
    quant_config=quant_config,
    offload_config=offload_config,
    state_path=state_path,
)

Run the model

!pip install langchain torch transformers sentence-transformers accelerate

import transformers, torch
from transformers import pipeline
from langchain import HuggingFacePipeline

pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer = tokenizer,
    torch_dtype=torch.float16,
    max_new_tokens=1024,
    device=device
)

from langchain import HuggingFacePipeline

llm = HuggingFacePipeline(
    pipeline = pipeline,
    model_kwargs={"temperature": 0.5, "max_new_tokens":1024},
)

from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate

task_template = """
Write your own template.
variable1 = {input_variable1}
variable2 = {input_variable2}
"""
task_prompt_template = PromptTemplate(
    input_variables=["input_variable1","input_variable1"], template=task_template, output_key = "structured_task"
)
task_chain = LLMChain(
    llm=llm, prompt=task_prompt_template
)

question = {"input_variable1":input_variable1, "input_variable2":input_variable2}

print(task_chain.run(question))

References
https://huggingface.co/docs/transformers/model_doc/mixtral

https://arxiv.org/abs/2401.04088

https://github.com/dvmazur/mixtral-offloading

Thank you for reading 😁.

If you like my work, you can support me here: Support my work

I do welcome constructive criticism and alternative viewpoints. If you have any thoughts or feedback on our analysis, please feel free to share them in the comments section below.

For more such content make sure to subscribe to my Newsletter here

Follow me on

Twitter

GitHub

Rumi GPT

Kaushal Powar — Wed, 15 Mar 2023 06:30:02 +0000

Hello Humans 👋,

I recently created a memoryless or stateless chatbot using OpenAI's GPT 🎈.
I developed a chatbot called 'Rumi GPT'.

How does it work?

Simply enter how you are feeling right now, and it will give you a beautiful quote by Rumi to help motivate and inspire you 😇.
Whether you're looking for guidance, comfort, or just a pick-me-up, Rumi GPT has you covered. With Rumi GPT, you can find motivation and inspiration to uplift your day and help you achieve your goals.

Motivation

Why Rumi?

When I initially joined Twitter, I was unsure of whom to follow. I finally decided to follow accounts that tweet about books and motivational quotes. That was the day when I stumbled upon the work of Rumi. One of his quotes, in particular, captivated me and has held a special place in my heart ever since.

"Set your life on fire. Seek those who fan your flames."

Since then, I have delved deeper into Rumi's life and work, including his journey through Elif Shafak's masterpiece, "Forty Rules Of Love." Rumi's work has touched the lives of countless individuals over the centuries, and it is not difficult to see why. His poetry and philosophy inspire us to seek beauty in our lives, embrace love, and let go of our fears and doubts.

If you are not familiar with Rumi, here is some brief information about him:
Rumi was a 13th-century Persian poet, Islamic scholar, and Sufi mystic whose works have gained worldwide recognition and popularity. He is regarded as one of the greatest spiritual masters and poets in Islamic culture, known for his mesmerizing and insightful poetry that touches on themes of love, spirituality, and the human experience. People follow his work because of his profound insights, captivating language, and timeless relevance to the human condition, inspiring generations of readers and seekers to explore the depths of spirituality and the nature of existence.

Make sure to give it a try.
Rumi GPT

Thank you for reading 😁.
I do welcome constructive criticism and alternative viewpoints. If you have any thoughts or feedback on our analysis, please feel free to share them in the comments section below.

Follow me on

Twitter
GitHub
Linkedin

Will Google Bard Reduce Webpage Clicks?

Kaushal Powar — Fri, 10 Mar 2023 14:43:05 +0000

We rely on search engines like Google to help us find the information we need. And when we search for something, we're usually presented with a list of web pages that relate to our query.

Have you ever thought about why we click on certain web pages and not others? It's because we're looking for information, and webpages that are easy to read, have great headlines and are ranked higher in search results tend to get more clicks.

But will Google's new AI service, Bard, change all that? 🤔

What is Bard?
Google Bard is an experimental AI service that aims to provide users with a more conversational and interactive search experience. Instead of simply providing a list of web pages that relate to a query, Bard will provide a GPT-like answer right below the search bar.

This means that users will be able to get quick answers to simple queries without having to navigate to a webpage.
So, if you're asking for the weather or a definition, you won't even need to click on a webpage - Bard will just tell you what you need to know!

So, how will this affect webpage clicks? For simple, factual queries, it's likely that users will be satisfied with the answer provided by Bard and won't feel the need to click through to a webpage. However, for more complex queries or queries that require more detailed information, users may still choose to click through to a webpage to explore related topics or learn more.

So, will Bard kill webpages? Not likely. For more complex queries or topics that require deeper dives, users will still want to click on webpages to learn more. Plus, we're creatures of habit - we like clicking on things, and the top-ranked results will still catch our eye.

My personal opinion is that while Bard is an interesting development in the world of search, it's unlikely to drastically change the way we interact with webpages. Users will still need to navigate to webpages for more detailed information, and webpages that rank higher in search results and have attention-grabbing headlines will likely continue to get more clicks.

But who knows - the way we search might keep evolving, and we'll just have to wait and see!
In conclusion, the relationship between webpages and clicks is complex and multifaceted. While Google Bard has the potential to reduce webpage clicks for simple queries, it's unlikely to eliminate them altogether.

As technology continues to develop, we can expect to see more changes in the way we interact with search results, but the need for detailed and informative webpages is unlikely to go away.

Thank you for reading 😁.

If you like my work, you can support me here: Support my work

I do welcome constructive criticism and alternative viewpoints. If you have any thoughts or feedback on our analysis, please feel free to share them in the comments section below.

Follow me on

Twitter

GitHub