DEV Community: Trần Hoàng Long

High Level Judge Prompting Visualized

Trần Hoàng Long — Sun, 02 Feb 2025 14:20:27 +0000

Hi, if you’re new here, I’m Produdez, I built a simple llm judge lately to learn more about prompt engineering and the evaluation process. Allow me to share the process from a high level perspective.

Let's go through the process of making an LLM judge, with some simple examples and illustrations.

A judge is an llm bot that output include some form of binary output like (PASS / FAIL) so that automated evaluation of metric can be calculated.

1. The Brain Storm

We would start with an initial idea, eg: "judge a valid question", then, we would carve out our initial
Requirements, such as: question must have "?"
And brainstorm some data samples from that requirement.

Then what if the data sample looks ... unsure.
You likely encounter cases in this process of brain storming that require you to make a decision.

Do we ignore this case and make it FAIL
Or do we validate it and update our requirements rule set.

New "ambiguous sample" means potential new/updated requirement

This create a brainstorming loop of constantly coming up with new samples from your ideas/requirement and have your requirements refined to match the emerging data.

After that, we would end up with a list of initial requirements, call it Requirements V1, and Sample Data V1

2. Prompt Preparation / Init Prompting

So now we have a list of "rules" for our judge and the data that should represent those rules.
What I would do now is crafting a Prompt V1 from our requirements. Just a very simple, baseline prompt that adhere to some of the rules. And for the dataset, I would clean it up, format properly into Dataset V1 to later be used in the prompt iteration pipeline

An example starting prompt here for our example could be:

SysPrompt: You'll be given a sentence, is it a question? Make sure it has "?"
Message: Sentence: ...

You can see that I did not try to pack all the requirements we came up with into the prompt. Cause right now that's unnecessary, we'll be iterating on it later anyways, this is just a starting point.
The more cumbersome thing for the engineer to prepare is the Evaluation System and the Iteration

Pipeline itself, this is so that we can easily:

Try out our prompt
Run on all tests, get result
Does analysis and reasoning on fail result
Improve dataset or prompt or both (among other things)
(Repeat) Which is exactly our next step

3. Prompt Iterating

Here's a glimpse of the pipeline

At this step, it might be helpful to mention this. Some of you would wonder:

Why build this complex pipeline when we can use Anthropic Workbench?

To that the simple answer is "Scope", or "Scale".

It might be easy for you as a prompter to judge a result of a summarization bot you build and quickly iterate, but what if there are more requirements than a quick glance could capture? What if there are more test cases than you can read? What if the prompt has to evolve overtime growing with your company's interest while adhering to the same requirements?

These problems at scale are ones that a single work bench can not help capture. Hence this pipeline of automated evaluation and update.

Now, let's go through the thought process behind the pipeline using our "is question" example.

Here's the ideal progress

Using the same dataset, you keep getting better result each new prompt, aiming for the illusive 100% (or similar 98, 95)

But there's reasons this is not the full picture of the process:
Your dataset is small, they do not fully represent the requirements of the task yet
Your requirement might not be fully concrete yet, you haven't added all the rules, or haven't thought of enough
There are edge cases yet to be address, this means more examples and more rules
OR simply put, your dataset and rule is only V1. It's incomplete

There's no doubt in this process that eventually new data samples would emerge, either from your new sparks of idea, edge cases or simply contradiction in the initial setup. And with new data means modification of requirements, which could mean invalidation of many prompts that you currently have.

The example below shows

some new samples that comes up during iteration
and force a change to a requirement, making the prompt V3 very hard to be iterated on
So if you can't easily edit a prompt, what do you do?

Cause your current prompt V3, they're optimized for the current requirements, but as new ones are added/changed. You have to adapt them to the new rules.

How to iterate when rules change?

So you spent your WHOLE day prompting and prompting away then you realize the requirements need and update, and you panik? What to do

Fear not, cause here's where you prompting brain and ingenuity comes to play. Remember all those prompts you crafted? Don't just throw them away.
There's something I like to call "Core Logic" that is the main essence of your prompt. Instead of looking at the walls of text that you created, find the core and compare it to the new requirements.

If there's good logic that's not invalidated, extract that part out and keep it
If not much logic is affected, maybe you can get away with using the whole prompt again
If not, well, back to step 2 you go with a new requirement and prompt all together. But carry the core invalidated logic as something "not" to do instead. Here's how it looks below

I would argue this is also personal taste, but I find this the most organized way that one can proceed without invalidating their own work. You extract the useful parts of your best performer and continue.
But notice that sometimes the optimized version of a prompt like V3 contains a lot of FLUFF and might not be the best target for base logic extraction, maybe a more simple and purer one could be the go to, maybe your V2 or even V1.
Of course, the new prompt V4 is very pure and basic, it would not perform well, but with the core logic valid, you at least know that this is something of a baseline that you could use to iterate on.
That's how you iterate.

So to recap section 3.

When prompt iteration, when you don't have to update your dataset, you just prompt engineer.
But, when the DS update, you need to utilize the core knowledge of your prompt to reconstruct it for the new requirement.
This "core" is honestly very tricky as I have a tendency to over complicate a prompt very quick and loose sight of what are the main parts of it. But i believe that if we conduct our experiment disciplined. Keeping track of what the prompt "DO", either via note, or structure, ... it would help very much as the prompt and the complexity of the project requires us to not only patch the prompt but even rewrite it.

...

The end. Hope you had a great read. Leave a like and comment !!

Written by Produde 👨‍💻

Find me on: LinkedIn 💼 | Twitter 🐦 | YouTube 🎥 | Buy me a coffee ☕

You Should Use Git Switch 🛠️

Trần Hoàng Long — Mon, 30 Dec 2024 07:58:51 +0000

I'm trying out short blog versions for smaller topics - I hope you find it insightful! ✨

If you're working with branches in Git, you should use git switch. It's a newer, safer command specifically designed for branch operations that was introduced in Git 2.23 to reduce confusion and prevent mistakes. But wait - why am I telling you to use git checkout then? Let me explain.

Quick Syntax Overview 📝

Let's look at both commands first:

git switch:

# Switch to existing branch
git switch feature-branch

# Create and switch to new branch
git switch -c new-feature

git checkout:

# Switch to existing branch
git checkout feature-branch

# Create and switch to new branch
git checkout -b new-feature

# Restore file to last commit
git checkout -- file.txt

The Main Difference 🔍

Here's the key thing - git checkout is like a Swiss Army knife 🔪. It can switch branches AND restore files. Meanwhile, git switch only does one thing: branch operations. This specialization is exactly what makes git switch safer for branching, but also what makes git checkout more powerful overall.

When to Use Each Command ⚡️

Use git switch when: 🌿

You're working with branches (creating, switching)
You want clear, unambiguous operations
You're using Git 2.23 or newer
You want to play it safe

Use git checkout when: 🚀

You need to restore files to their last committed state
You want to check out specific commits (entering detached HEAD state)
You're working with older Git versions
You need its versatile functionality for advanced operations

Conclusion 💡

While git switch is safer for branch operations, git checkout remains essential due to its versatility. Understanding both commands and their appropriate use cases will make you a more effective Git user.

Written by Produde 👨‍💻

Find me on: LinkedIn 💼 | Twitter 🐦 | YouTube 🎥 | Buy me a coffee ☕

Welcome to My Corner of the Internet! 🏡

Trần Hoàng Long — Sun, 29 Dec 2024 10:59:17 +0000

Hey there! 👋 I'm Trần Hoàng Long, but you can call me Produde. Welcome to my little space on the internet where I share bits and pieces of my life, thoughts, and work!

Who Am I? 🤔

I'm a developer who loves building things for the web, but that's just one part of me! I'm also:

A curious mind always excited to learn new things 🧠
Someone who enjoys documenting their journey 📝
A person who believes in sharing knowledge and experiences 🌱

What I Write About 📚

This blog is my digital garden where I plant various seeds of thoughts about:

Life observations and personal growth 🌟
Technical adventures and coding challenges 💻
Book insights and learning notes 📖
Random musings and creative ideas ✨

A Bit About My Tech Stack 🛠️

When it comes to the technical side, here's what I work with:

Frontend: Building beautiful user experiences with Svelte and Vue.js 🎨. Svelte 5.0 is my new favorite framework 💪, I'm a total noob right now but learning fast 🚀.
Backend: Crafting robust systems with Laravel and its awesome ecosystem 🏗️. I did work a lot with python but not right now 😅 for my current project.
Databases: Making data work smarter, not harder 💾, with ML and LLM if possible. I'm big fan of MongoDB and PostgreSQL.

Let's Connect! 🤝

This space is meant to be a cozy corner where I share my journey - both in tech and life. Whether you're here to learn, share, or just browse around, I hope you'll find something interesting!

You can find me across the web:

🐙 GitHub: produdez
💼 LinkedIn: produdez
🐦 Twitter: produdez
✍️ Medium: @produde
📧 Substack: produde.substack.com

Feel free to explore my posts, thoughts, and projects. Maybe you'll find something that resonates with you or sparks your curiosity!

Welcome aboard this journey of continuous learning and sharing! 🚀✨

Remember: Life is more fun when we learn and grow together! 🌈