DEV Community

Mayank Laddha
Mayank Laddha

Posted on

2

Using DSPy(COPRO) to refine prompt instructions

I recently tested DSPy with phi3, aiming to use COPRO to refine the prompt instructions and achieve the best score. I intentionally did not use assertions.

Here’s what I liked:

I used the CoLA dataset, which includes sentences from 23 linguistic publications annotated for grammaticality, I wanted outputs with "just the label" (no explanations or extra words). DSPy generated instructions that performed well, ensuring grammatical correctness:

Original Instruction: Check if the sentence is correct(1) or not(0)

Optimized Instruction: Determine if the given sentence adheres to standard English grammar rules by outputting a '1' for grammatically correct and '0' for incorrect.

Both the BootstrapFewShot and Signature Optimizer are great starting points.

What could be improved:

DSPy states, "Once the training is done, you’ll have better instructions and prefixes to edit in the signature manually." However, I believe that post-compilation, the tool should be ready to predict. The instruction optimizer needs more flexibility for easier tweaking, such as better compatibility with other languages. Alternatively, creating a program for the signature optimizer to tweak prompts as needed would be beneficial.

It is a cool concept but I guess as a framework it still needs improvement. Connect with me if you have an interesting project/common interests.

Gihub repo

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more