Summarize articles with Cloudflare Workers AI LoRAs

This tutorial will go over how to make a Streamlit web app in Python that takes in a URL and, with Workers AI LoRA adapters, summarizes the article in the tone of the user's choosing, such as gen z or humorous.

It's free to use most AI models hosted on Cloudflare!

Test it out yourself here.

The complete code can be found here on GitHub.

LoRAs

Fine-tuning is a process that takes a pre-trained model and further trains it on a specific dataset to adapt it to a particular task or domain, enhancing its performance and relevance to that task. LoRAs are a more efficient approach to fine-tuning that adds a small number of low-rank matrices to the model's parameters during fine-tuning.

This means that LoRAs offer a more efficient and scalable way to adapt LLMs to specific tasks without the heavy computational and memory demands of traditional fine-tuning.

Setup

Since this project requires Python package installations, make a new project directory and virtual environment.

If you're using a Unix or macOS system, open a terminal and enter the following commands:

mkdir lora-summarize 
cd lora-summarize
python3 -m venv venv 
source venv/bin/activate 
pip install streamlit
pip install bs4
pip install requests
pip install load_dotenv

If you're following this tutorial on Windows, enter the following commands in a command prompt window:

mkdir lora-summarize
cd lora-summarize
python -m venv venv 
venv\Scripts\activate 
pip install streamlit
pip install bs4
pip install requests
pip install load_dotenv

The last command uses pip, the Python package installer, to install the packages that you are going to use in this project, which are:

• The load_dotenv library to load environment variables from a .env file
• The streamlit library to make webapps in Python, easily connecting the frontend and backend
• The bs4 library to extract text from a webpage
• The requests library to hit the Cloudflare endpoint for a LLM hosted on Cloudflare

This project needs a Cloudflare API Token and your Cloudflare Account ID to use an LLM. After making a Cloudflare account, click Workers&Pages on the left. Then, you can find your Account ID on the right side.

Get a Cloudflare API Key by clicking AI on the left, followed by Use Rest API. Finally, click Create a Workers AI API Token. Grab it and in the root directory, make a .env file and put them in there as such:

CF_API_TOKEN=XXXX
CF_ACCOUNT_ID=XXXX

Now it's time to code the Streamlit app.

Build the Streamlit App

I love how the Streamlit app lets you build your web page from the top-down.

At the top of a Python file called app.py, add the following code to import the required libraries, load the .env secret credentials, and create the variable for the Cloudflare URL to hit in Python:

import streamlit as st
from bs4 import BeautifulSoup
import json
import os
import requests
# Load API secrets
from dotenv import load_dotenv
load_dotenv()
CLOUDFLARE_ACCOUNT_ID = os.environ.get("CF_ACCOUNT_ID")
CLOUDFLARE_API_TOKEN= os.environ.get("CF_API_TOKEN")
url = f'https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/ai/run/@cf/mistral/mistral-7b-instruct-v0.1'

Then comes the meat of the app: Make a main function, include some Streamlit modules to display markdown on the page, write text to the page, get text input asking for a news URL to summarize, and display a selectbox with options the user can select for the tone of the summary.

When the user clicks the enter button, a spinner is displayed while Beautifulsoup extracts paragraph tags from the input web page URL.

def main():
    st.markdown("""
        <style>
            .big-font {
                font-size:40px !important;
                color:green;
            }
        </style>
    """, unsafe_allow_html=True)
    st.markdown('<p class="big-font"<p>AI🤖 News🗞️ Summarizer</p>', unsafe_allow_html=True)
    st.write(":blue[This Python🐍 web🕸️ app is built👩🏻‍💻 w/ [Streamlit](https://streamlit.io/) && [Cloudflare Workers AI](https://ai.cloudflare.com/)]")

    news_link = st.text_input('Please enter a news link to summarize') # news_link = "https://www.npr.org/2024/07/08/g-s1-8731/emma-navarro-coco-gauff-wimbeldon"
    tone = st.selectbox(
        ':green[What tone do you want the news summary to take?]',
        ('humorous🤣', 'majestic🌊', 'academic📚', '✨inspirational✨', 'dramatic🎭', 'gen z👧🏻')
    )
    st.write("You selected: ", tone)
    if st.button('Enter') and tone is not None and news_link is not None:
        with st.spinner('Processing📈...'):
            resp1 = requests.get(news_link)
            soup = BeautifulSoup(resp1.text, 'html.parser')

            # Extract text data from website
            text_data = ''
            for tag in soup.find_all(['p']):
                text_data += tag.get_text()

            print('text_data' , text_data)

That text extraction of paragraphs represents the body of the input URL. It is included (with the selected tone of the summary) in the prompt passed to the LoRA model adapter that hits the LLM (Mistral) model hosted on Cloudflare via GET request. To run inference with public LoRAs, you just need to define the LoRA name in the request. The response returned is the summary of the selected tone. It is then parsed and displayed. Finally, a footer is displayed at the bottom of the webpage.

            # Define the headers
            headers = {
                'Authorization': f'Bearer {CLOUDFLARE_API_TOKEN}',
                'Content-Type': 'application/json'
            }

            # Define the data
            data = {
                "messages": [
                    {
                        "role": "user",
                        "content": f"Summarize the following content from a news article in a {tone} tone: {text_data}"
                    }
                ],
                "lora": "cf-public-cnn-summarization"
            }

            # Make the POST request
            response = requests.post(url, headers=headers, data=json.dumps(data))

            # Parse the response
            response_data = response.json()
            summary = response_data["result"]["response"]
            print("summary ", summary)
        html_str = f"""
        <p style="font-family:Comic Sans; color:Pink; font-size: 18px;">{summary}</p>
        """
        st.markdown(html_str, unsafe_allow_html=True)

    st.write("Made w/ ❤️ in SF 🌁 || ✅ out the [👩🏻‍💻GitHub repo](https://github.com/elizabethsiegle/cf-ai-lora-news-summarizer)")

The complete app.py code is below:

import streamlit as st
from bs4 import BeautifulSoup
import json
import os
import requests
# Load API secrets
from dotenv import load_dotenv
load_dotenv()
CLOUDFLARE_ACCOUNT_ID = os.environ.get("CF_ACCOUNT_ID")
CLOUDFLARE_API_TOKEN= os.environ.get("CF_API_TOKEN")
url = f'https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/ai/run/@cf/mistral/mistral-7b-instruct-v0.1'

def main():
    st.markdown("""
        <style>
            .big-font {
                font-size:40px !important;
                color:green;
            }
        </style>
    """, unsafe_allow_html=True)
    st.markdown('<p class="big-font"<p>AI🤖 News🗞️ Summarizer</p>', unsafe_allow_html=True)
    st.write(":blue[This Python🐍 web🕸️ app is built👩🏻‍💻 w/ [Streamlit](https://streamlit.io/) && [Cloudflare Workers AI](https://ai.cloudflare.com/)]")

    news_link = st.text_input('Please enter a news link to summarize') # news_link = "https://www.npr.org/2024/07/08/g-s1-8731/emma-navarro-coco-gauff-wimbeldon"
    tone = st.selectbox(
        ':green[What tone do you want the news summary to take?]',
        ('humorous🤣', 'majestic🌊', 'academic📚', '✨inspirational✨', 'dramatic🎭', 'gen z👧🏻')
    )
    st.write("You selected: ", tone)
    if st.button('Enter') and tone is not None and news_link is not None:
        with st.spinner('Processing📈...'):
            resp1 = requests.get(news_link)
            soup = BeautifulSoup(resp1.text, 'html.parser')

            # Extract text data from website
            text_data = ''
            for tag in soup.find_all(['p']):
                text_data += tag.get_text()

            print('text_data' , text_data)

            # Define the headers
            headers = {
                'Authorization': f'Bearer {CLOUDFLARE_API_TOKEN}',
                'Content-Type': 'application/json'
            }

            # Define the data
            data = {
                "messages": [
                    {
                        "role": "user",
                        "content": f"Summarize the following content from a news article in a {tone} tone: {text_data}"
                    }
                ],
                "lora": "cf-public-cnn-summarization"
            }

            # Make the POST request
            response = requests.post(url, headers=headers, data=json.dumps(data))

            # Parse the response
            response_data = response.json()
            summary = response_data["result"]["response"]
            print("summary ", summary)
        html_str = f"""
        <p style="font-family:Comic Sans; color:Pink; font-size: 18px;">{summary}</p>
        """
        st.markdown(html_str, unsafe_allow_html=True)

    st.write("Made w/ ❤️ in SF 🌁 || ✅ out the [👩🏻‍💻GitHub repo](https://github.com/elizabethsiegle/cf-ai-lora-news-summarizer)")

if __name__ == "__main__":
    main()

What's Next for LoRAs and Cloudflare Workers AI

It's free to use many open source LLMs hosted on Cloudflare. They offer so many possibilities to builders and developers of all backgrounds for building generative AI applications.

LoRAs are helpful because they provide a cost-effective, efficient, flexible, and scalable method for adapting LLMs to specific tasks, making advanced AI capabilities more accessible for a wide variety of applications.

Let me know online what you're building with AI!