Taming the Cost of Prompt Chaining with GemBatch

Benno — Tue, 12 Nov 2024 17:02:01 +0000

Prompt chaining is revolutionizing how we interact with large language models (LLMs). By linking multiple prompts together, we can create complex, dynamic conversations and tackle intricate tasks. But this power comes at a price — literally. Each API call to an LLM service like Google’s Gemini adds to your bill.

Many LLM providers offer a solution: batch processing. Send multiple prompts in a single request and enjoy significant discounts (often around 50%!). However, implementing batching within a prompt chain workflow can quickly turn into a coding nightmare.

The Challenges of Batch Prompt Chaining

Imagine you’re building a chatbot with a multi-step dialogue. With traditional prompt chaining, you’d send each user message and wait for the model’s response before formulating the next prompt. But to leverage batch discounts, you need to:

Accumulate prompts: Instead of sending immediately, gather prompts into batches.
Embrace asynchronicity: Send the batch and wait (potentially for a longer time) for the job to complete.
Map responses: Once the batch result arrives, correctly match each response to its corresponding prompt in the chain.

On top of this, you need to handle rate limits, errors, and retries. This can lead to convoluted code that’s hard to read, debug, and maintain.

Enter GemBatch: Your Prompt Chaining Hero

GemBatch is a Python framework designed to simplify batch prompt chaining with Google’s Gemini. It seamlessly integrates with Firebase, providing a familiar and scalable environment for your LLM applications.

Here’s how GemBatch makes your life easier:

Effortless batching: GemBatch handles the complexities of batching prompts behind the scenes. You define your prompt chains sequentially, just as you would with traditional methods.
Asynchronous handling: GemBatch manages asynchronous batch jobs, ensuring your prompt chains execute smoothly without blocking your application.
Simplified response mapping: GemBatch automatically routes responses back to the correct points in your prompt chains.
Cost savings: By leveraging Gemini’s batch discounts, GemBatch can significantly reduce your LLM API cost.

A Quick Example

import gembatch

# Define a simple prompt chain
def task_a_prompt1():
    gembatch.submit(
        {
            "contents": [
                {
                    "role": "user",
                    "parts": [{"text": "What is the capital of France?"}],
                }
            ],
        },  # prompt 1
        "publishers/google/models/gemini-1.5-pro-002",
        task_a_prompt2
    )

def task_a_prompt2(response: generative_models.GenerationResponse):
    gembatch.submit(
        {
            "contents": [
                {
                    "role": "model",
                    "parts": [{"text": response.text}],
                },
                {
                    "role": "user",
                    "parts": [{"text": f"And what is the population of {response.text}?"}],
                }
            ],
        },  # prompt 2
        "publishers/google/models/gemini-1.5-pro-002",
        task_a_output
    )

def task_a_output(response: generative_models.GenerationResponse):
    print(response.text)

# Start the prompt chain
task_a_prompt1()

This simple example demonstrates how Gembatch allows you to define a prompt chain with gembatch.submit(). Gembatch takes care of batching the requests to Gemini and managing the asynchronous responses.

Get Started with Gembatch

Ready to unlock the power of cost-effective prompt chaining? Check out the Gembatch repository on GitHub:

https://github.com/blueworrybear/gembatch

We welcome feedback, and suggestions!

Put Vue.js and Go together! Setup Web App in 5 mins

Benno — Fri, 09 Oct 2020 10:50:48 +0000

Hello there! I'm Benno, a blue bear falls in love with coding 💙.

The first time I met Go was about a year ago. I was looking for a self-hosted Git service for my team. I tried GitLab in the first place, but things didn't go smoothly. I had no root permission in the Linux server, and there was no Docker installed. So I could not even start the first step 😭. When I almost gave up, Gitea just went inside my eyes. It's incredible that I could start a Git service with simply:

./gitea

That's the WOW moment I decided to learn Go and how to build this kind of stunning application ✨.

So today I am going to show you how to put your frontend and backend codes into a single executable.

Let's get started!

Install Tools

It's assumed that Go, NPM and Vue CLI are already installed in your workspace. If not, you can refer to below tutorials to get it installed:

If you are ready, run below command to get go-bindata:


go get -u github.com/go-bindata/go-bindata/...

go-bindata is a command-line tool converts any file into Go source code. We are going to use it to package frontend codes.

Prepare Working Directory

Now, create a working directory:


mkdir example-goweb
cd example-goweb

Initialize the Go project with:


go mod init example-goweb

I'm going to use Gin as the web framework, but in fact you can choose any framework, e.g. Chi if you like. Run:

go get -u github.com/gin-gonic/gin

After that, create a Vue App with Vue CLI:

vue create -n web

You could just follow the default setting for quick starting. A new folder web/ will be created with Vue App source codes. You can run below commands to see the web application:

cd web
npm run serve

It should start the website at http://localhost:8080, which looks like:

Build Frontend

To build frontend, you could simply run npm run build under web/ folder. However, we could do it better with go generate.

Create a new file web/web.go with following codes:

package web

//go:generate npm run build
//go:generate go-bindata -fs -o web_gen.go -pkg web -prefix dist/ ./dist/...

//go:generate is a special comment to tell Go executing scripts when run go generate. You could get more information here.

Now let's run:

go generate ./web

It's exactly the same as running:

cd web
npm run build
go-bindata -fs -o web_gen.go -pkg web -prefix dist/ ./dist/...

go-bindata will convert all files under web/dist/ into Go source code web_gen.go, which could be used later.

Go Coding!

We are almost there, the remaining works are:

Serve Static Files
Create main() Function

To Serve static files with HTTP, create a package routers:

mkdir routers
touch routers/routers.go

Open routers.go and add codes:

package routers

import (
    "net/http"

    "example-goweb/web"

    "github.com/gin-gonic/gin"
)

// HandleIndex return HTML
func HandleIndex() gin.HandlerFunc {
    return func(c *gin.Context) {
        html := web.MustAsset("index.html")
        c.Data(200, "text/html; charset=UTF-8", html)
    }
}

// Register routes
func Register(e *gin.Engine) {
    h := gin.WrapH(http.FileServer(web.AssetFile()))
    e.GET("/favicon.ico", h)
    e.GET("/js/*filepath", h)
    e.GET("/css/*filepath", h)
    e.GET("/img/*filepath", h)
    e.GET("/fonts/*filepath", h)
    e.NoRoute(HandleIndex())
}

Let me explain what's going on. We create a gin.HandlerFunc to serve files with HTTP:

h := gin.WrapH(http.FileServer(web.AssetFile()))

web.AssetFile() is a function from web/web_gen.go, it creates a http.FileSystem and looks up static files inside web_gen.go.

Then we tell gin to handle every GET static files request:

e.GET("/favicon.ico", h)
e.GET("/js/*filepath", h)
e.GET("/css/*filepath", h)
e.GET("/img/*filepath", h)
e.GET("/fonts/*filepath", h)

When user requests for a static file, like JavaScript, gin will handle the request and http.FileSystem will return the file.
The last line in Register function tells gin to return index.html if there is no matching route.

func HandleIndex() gin.HandlerFunc {
    return func(c *gin.Context) {
        html := web.MustAsset("index.html")
        c.Data(200, "text/html; charset=UTF-8", html)
    }
}

Finally, let's create a main.go:

package main

import (
    "example-goweb/routers"

    "github.com/gin-gonic/gin"
)

func main() {
    e := gin.Default()
    routers.Register(e)
    e.Run(":8080")
}

The main() creates a gin engine to register routes and start the HTTP server with :8080 port.

Your web server is ready to go! Run below command and visit http://localhost:8080 to see the result:

go run main.go

You can build executable and start server with:

go build -o goweb ./
./goweb

Here is what your working folder should look like in the end! 🎉

.
└── example-goweb/
    ├── routers/
    │   └── routers.go
    ├── web/
    │   ├── dist/
    │   │   ├── css
    │   │   ├── favicon.ico
    │   │   ├── img
    │   │   ├── index.html
    │   │   └── js
    │   ├── README.md
    │   ├── babel.config.js
    │   ├── node_modules
    │   ├── package-lock.json
    │   ├── package.json
    │   ├── public
    │   ├── src
    │   ├── web.go
    │   └── web_gen.go
    ├── go.mod
    ├── go.sum
    └── main.go

Conclusion

Packaging your web application into a single executable makes deployment extremely easy. Many real-world applications adopt this solution, for example:

If you like this approach and would like to have more examples, check my open source project covergates:

https://github.com/covergates/covergates

It's a self-hosted coverage report alternative to Code Climate, Codecov or Coveralls.
You can have your own coverage report service simply with:

wget https://github.com/covergates/covergates/releases/download/v0.2.1/covergates-v0.2.1-linux-amd64.tar.gz
tar -zxvf covergates-v0.2.1-linux-amd64.tar.gz
./covergates-server

In fact, there is a cool trick to change base URL for Vue router on the fly. You can find it in the source code or give me a heart ❤️ to let me know you are interested in it. I'll have another tutorial next time. See Ya! 😆

DEV Community: Benno