DEV Community: Gal Bashan

How ChatGPT’s Coding Skills Got Me Drunk

Gal Bashan — Mon, 12 Dec 2022 20:02:17 +0000

ChatGPT helped me scrape cocktail websites to create a universal drink index.

Lately, I've become a fan of making cocktails. After getting the best equipment and booze money can buy, I realized the leading blocker for me was cocktail recipes. I wanted a platform where I could input the ingredients I have at home, and the output would be a list of cocktails I could make. There are a few apps for that, but they are limited in their variety unless you pay for the app. And since the only thing I like more than cocktails is free cocktails, I was looking for an alternative.

It was about a week after ChatGPT was released, and it had already become my best friend. Could it also be my drinking buddy? I realized many recipes were available online, and I just had to index them correctly. Can ChatGPT build me a tool to index them on my own?

TLDR — yes, here is the GitHub repo. There are even ChatGPT-generated READMEs.

The Plan

I decided to give it a go. I wanted to use ChatGPT to build a simple system to index cocktails from the web. The solution would have two parts:

Crawler — a process that gets a domain to crawl and outputs URLs that appear to be cocktail recipes to a queue. It also follows additional URLs on the page recursively to search for other pages with recipes. This seemed like an easy enough task to let ChatGPT code on its own
Indexer — This component is meant to get a URL, determine if the page contains a cocktail recipe, and store it in a database. The problem is that blogs with cocktail recipes are highly unstructured and have a lot of unnecessary text before reaching the point. Can ChatGPT help me make sense of this mess?

With the plan in place, I set out to start building. I started by laying down the architecture and had to decide what queue to use. ChatGPT came to the rescue:

After reviewing its suggestions, I decided to go with RabbitMQ since I had some experience with it. I asked ChatGPT to lay out the foundations for me:

With the project set up, the next step was for ChatGTP to develop the crawler.

The Crawler

I asked ChatGPT to do the work for me, and boy, did it deliver:

When I ran the program, I came into two issues. First, the visited URLs were not cached, and since almost any page was linked to the home page, we ended up in an infinite loop. I asked ChatGPT to handle that, and it modified the code correctly:

The second issue was that it started crawling other domains as well — for example, for a page containing a video, the crawler started crawling youtube as well. I asked ChatGTP to fix that as well, and it obliged:

And that was it! The crawler was ready, and the next step was to code the indexer.

The Indexer

As I mentioned before, parsing the content of a web page to identify if it describes a cocktail recipe is challenging. Here is an example of a cocktail recipe web page:

It's almost impossible to handle all the different use cases to parse the page and extract the ingredients. Once again — ChatGPT to the rescue!

I came across an unofficial python implementation of the ChatGPT API and decided to use it in my indexer. The idea was simple, using a well-crafted prompt, I should be able to use ChatGPT to extract the ingredients from the cocktail page. Since ChatGPT has no internet access, it couldn't code this part, but it did help me with the generic components of it. Here is the code I used:

import pika
import time
import sys
import requests
from bs4 import BeautifulSoup
from revChatGPT.revChatGPT import Chatbot
from db import CocktailRecipe, Ingredient, session
import traceback

print(sys.argv)
config = {
    "session_token": sys.argv[1],
    "cf_clearance": sys.argv[2],
    "user_agent": sys.argv[3]
}

chatbot = Chatbot(config, conversation_id=None)

# connect to RabbitMQ
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# create a queue to consume messages from, if it does not already exist
channel.queue_declare(queue='drink_urls', durable=True)

QUESTION = """
    From the following text below, please understand if it is an article
    describing a cocktail recipe. If it is, output the following: The first
    line should be: "<cocktail name>: <ranking>". If the ranking doesn't exist
    output "-". use only lowercase letters and the generic name of the cocktail
    , without mentioning brands. the following lines contain the ingredients:
    each line should contain one ingredient and it's amount in the format
    "<ingredient>: <amount>". The ingredient part (before the semicolon)
    should contain only the ingredient name, not the amount. If it is not a
    cocktail recepie, output only the word "no". Output the ingredients in 
    their generic name, and don't include a brand. For example, instead of 
    "bacardi white rum" output "white rum". All of the output should be lower
    cased, don't capitalize any word. The text is: %s"""

# define a callback function to process incoming messages
def process_message(ch, method, properties, body):
    time.sleep(5)
    try:
        print(body)
        page = requests.get(body)
        soup = BeautifulSoup(page.content, "html.parser")

        response = chatbot.get_chat_response(QUESTION % soup.text, output="text")['message']
        if response == "no":
            print("not a cocktail")
            return
        print(response)
    except Exception:
        print(traceback.format_exc())

# consume messages from the queue, using the callback function to process them
channel.basic_consume(queue='drink_urls', on_message_callback=process_message, auto_ack=True)

# start consuming messages
channel.start_consuming()

This worked like a charm. ChatGPT did an excellent job understanding whether or not a page contained a cocktail recipe, and if it did, the output was almost always in the correct format. I used to think coding with Python was explicit as coding could be, but it was never that close.

Now I just had to store it in a database. I asked ChatGPT to create the database for me:

Then, I gave it a sample input generated by itself and asked it to write code for inserting the information into the database. The initial implementation it provided was using psycopg2. I asked it to use SQLAlchemy as I find it easier to work with an ORM:

I integrated this code into the indexer and finally had the cocktail database of my dreams!

Next, I will use ChatGPT to help me create an API and UI to browse the cocktails database and choose the one I want to make today.

Conclusion

This whole project took me roughly 3 hours from end to end. ChatGPT's ability to accelerate my development process blew my mind. The biggest win was the text parsing ChatGPT provided, which I could not do myself, and ChatGPT made simple.

However, the more complex a task got, the chances of ChatGPT performing it correctly decreased drastically. When pairing with ChatGPT, it is still the developer's job to break down the work into small enough tasks for it to swallow, at least for now.

I'm excited to see what comes next!

Stop Using Dev Metrics!

Gal Bashan — Wed, 30 Nov 2022 14:18:19 +0000

You manage the company’s best team: the fastest cycle time, the highest planning accuracy, and the least amount of rework. Your team has the lowest review time, the most weekly deploys, and the best bugs / KLOC ratio. And then someone high up shuts down your project — what just happened?

The Output Paradox

As a junior developer, I was taught that merging small changes frequently is key to delivering value fast. As a junior engineering manager, I was told that tracking & optimizing metrics were vital to the manager’s job. The first google result pointed me to a bunch of terms and acronyms, and these were my first introductions to dev metrics.

Dev metrics are essentially a set of metrics that help engineering teams track their velocity (move fast) and quality (without breaking things). Different metrics can be used to measure these attributes. Cycle time, PRs count, or closed issues track velocity. To track quality, monitor the defects count, mean-time-to-resolution, or the number of deployments that cause failure. Each of these metrics will provide different insights into the attribute you inspect. Some metrics also help with analyzing proxy attributes. Track rework to analyze efficiency, which impacts velocity. Planning accuracy is a good indicator of predictability, which enables both velocity and quality and has value on its own.

The problem with all the metrics above is that while they do an excellent job of measuring efficiency, they are usually not enough to measure success. The main reason is that they are entirely disconnected from the business KPIs (key performance indicators, yet another fancy name for metrics). This disconnection often leads to ineffective communication between organizations in the company, mainly product and engineering. After all, the product organization (and the rest of the business) doesn’t care for velocity or quality alone, as those track output. The product organization usually cares for outcome — what revenue did our new feature produce? How many new users did we attract?

As a new engineering manager, this was baffling to me. On the one hand, it was clear that carrying on measuring my team based on dev metrics over business metrics would lead to misalignment with the product team. On the other hand, it seemed unfair to measure my team based on business metrics: How can they impact them? Is it our job? Will I be stepping on the product organization’s toes?

Measure What Matters

I took a leap of faith and tried using business metrics as my north star. As time passed, it led to better results.

First, it simplified communication. it’s easier for your product counterpart to understand why it’s essential to prioritize the DB upgrade when you put it in terms of shared KPI that will be harmed. Just because it’s clear to you that if you don’t prioritize the DB upgrade, the following few user stories will take twice as long, and you will have to spend time on maintenance, which will delay the feature release to a point where you won’t have enough time to hit the usage target you set as the KPI, doesn’t mean it’s clear to your product counterpart. I mean, that’s a lot of hops; why not start with the one that matters? (no, it’s not the DB upgrade one)

Secondly, measuring my team based on business metrics made me see that we can impact them quite a bit:

We can identify features where we can build 80% of the user value for 20% of the effort. This can free up resources to help move our KPIs even faster. To do that, the team must clearly understand the business goals, user needs, and the drive to improve them.
We can use the KPIs to guide what tech debt we should accumulate: Are we trying to get our first feature users, and feature-failure is an option? Let’s release as fast as possible and worry about tech debt later. Are we trying to improve the retention of a large customer base? Let’s ensure high quality for this one.
We can suggest solutions to pain points product didn’t even know solvable, based on a new technology advancement or an innovative engineering angle.
We can (as annoying as it may be) use the KPIs to understand that there is no point in pursuing a specific cool technological solution we want to implement because the problem itself is not crucial to the business.

Lastly, I realized that measuring my team and myself on anything other than the business’ KPIs sets us up for failure. It will drive us to build & ship many durable features fast, over shipping features customers use. I understood that as someone who builds the product, I am a part of the product team. Since the product success is my success, I should be calibrated to that. It is alignment at best — when engineering and product share the same goals, they will likely leverage each other’s strengths to achieve them.

How To Measure Success?

After switching to metrics that matter to the business, My product counterpart and I had to choose a framework to measure them. At Epsagon, we used the OKR framework. I won’t go into details as there are many great resources for implementing OKRs available online. The gist is that this is an excellent framework for mid to large organizations, usually in the post-product-market-fit phase. It helps align different teams, groups, and organizations within the business toward the same outcome-driven goals.

At Epsagon, we had OKRs for every product team, and they were the goals of the engineers, product, and design people who made the product team. Once implemented, we had a better collaboration between product and engineering. Less arguing over requirements and more cooperation to solve problems. We also noticed that the increase in autonomy caused by setting goals over tasks increased innovation and ownership over outcomes. The teams set the roadmap, and solutions came from engineering as well as product.

The downside of this methodology is that it requires some effort in implementation and also requires you to be able to state your goals over periods of months. This is usually hard for small start-ups, especially before product-market-fit. For earlier stages, it’s OK to work with one primary metric and still be driven by a business objective while remaining agile enough to change your goals and focus in the granularity of weeks or days.

Whether your team is big or small, have one mission or competing priorities — as long as you measure what matters, you have a higher chance of succeeding. The “how” is just an optimization.

Communication is Key

Once we establish business KPIs, the next step is to follow through. In my experience, there are two critical aspects to communicating and implementing this change. The first one is consistency: it’s not enough to set those metrics and revisit them three months later to find out we missed our marks. As leaders, it is our job to constantly monitor these metrics and ensure their state and roadmap to improve them are always communicated to our entire team.

The second aspect is commitment: it’s tempting to fall back to old habits of measuring individuals using dev metrics to optimize them. This will send the wrong message to your team, hinting that while you say you care about business KPIs, what you really care about are dev metrics. Be sure to change your habits to reflect your priorities: celebrate business outcomes, cut non-must-have features instead of pushing for deadlines, and raise flags when business KPIs are down. You can still use dev metrics as indicators; just remember to link them to their effect on business KPIs.

Are Dev Metrics Deprecated?

Despite the click-baiting headline of this article, the answer is no — they should just be used correctly. For one, dev metrics are powerful tools for troubleshooting your business metrics. For example, a decline in user retention may be explained by an increase in defect count. In this case, you can use the dev metric as a leading indicator: Instead of waiting for the retention to drop, you should monitor the defect count to prevent it from dropping (similar to how you would monitor CPU usage to avoid 5xx API errors count from spiking and causing actual harm to you users). Another use case is dev metrics as a leading indicator for developer experience and satisfaction (which, in turn, impacts developer retention).

However, the vital thing to remember is that you should never optimize the dev metrics. You should always be optimizing business metrics and use dev metrics optimization only as a tool to get to that goal. And, of course, you should only set, track or optimize dev metrics goals after you have a clear sense of the business outcome you are trying to achieve and the business KPIs you are trying to hit. Great engineering stats won’t save your project if it has no value.

What’s Next?

If you are leading an engineering organization, switching to optimizing business outcomes over dev metrics will improve your team’s chances of creating a real impact for the business. If you are committed to becoming an outcome-driven leader, I urge you to ask yourself these questions:

Do I know what business outcomes are expected of me?
Am I doing whatever I can to optimize for these outcomes?
How am I tracking success? What are the criteria for raising a flag or changing my plans?
Is my team aware of the business goals and roadmap to achieve them? What is their input on it?
Which of my business KPIs can be improved by improving which dev metrics?

Hopefully, these will assist you with shipping features customers want to use!