Finding it Hard to Gauge Local AI Performance? Visualize the Current AI Landscape

Found it hard to gauge just how much your local AI capabilities have improved? I was facing a similar issue and found a way to visualize the current state of AI. In this post, I’ll share that process and lay it out so you can refer to it when you encounter the same challenges.

Attempts and Pitfalls

While working on improving local AI features, I spent a lot of time thinking about how to demonstrate performance enhancements to users. At first, I only considered simply informing them in text, like "Performance has been improved." However, we received feedback that it was hard to grasp *how much* it had improved, or where it stood compared to other models.

So, I tried a few things. For instance, I tried showing specific benchmark scores, but it was ambiguous to explain how closely these scores related to actual user experience.

# Benchmark Results

GPT-3.5: 95 points
Local AI (After Improvement): 98 points

# User Feedback

"So, how much faster is that exactly?"
"I can't really tell the difference with just 3 points higher than GPT-3.5."

Simply listing numbers like this didn't convey much meaning to users. If anything, it seemed to add to the confusion. I think I was stuck on this part for about 3 hours.

The Cause

Ultimately, the problem was that there wasn't a way for users to intuitively understand and relate to abstract performance improvement metrics. Simply saying "it's better" wasn't enough; we needed concrete comparisons or visual evidence.

The Solution

I added a feature to visualize the publicly available 'current AI level,' showing a reference grade (e.g., GPT-3.5 level) and improvements over time. This allowed users to see at a glance where their local AI stood and how much it had advanced.

Below is conceptual code I referred to when implementing this feature. While the actual implementation involves more complex logic, the core idea is as follows:

import matplotlib.pyplot as plt
import pandas as pd
from datetime import datetime, timedelta

# Generate dummy data (in reality, this would be fetched from a DB)
def get_ai_performance_data():
    now = datetime.now()
    data = {
        'timestamp': [now - timedelta(hours=i) for i in range(24)],
        'performance_score': [80 + i*0.5 + (i%5)*2 for i in range(24)] # Slight improvement and fluctuation over time
    }
    df = pd.DataFrame(data)
    df['timestamp'] = pd.to_datetime(df['timestamp'])
    df = df.sort_values('timestamp')
    return df

# Set reference grade (e.g., GPT-3.5 level)
REFERENCE_GRADE_SCORE = 95

def plot_ai_performance(df):
    plt.figure(figsize=(12, 6))

    # Plot of performance score trend
    plt.plot(df['timestamp'], df['performance_score'], marker='o', linestyle='-', label='Local AI Performance')

    # Reference grade line
    plt.axhline(y=REFERENCE_GRADE_SCORE, color='r', linestyle='--', label=f'Reference Grade (GPT-3.5 Level)')

    plt.title('Local AI Performance Trend')
    plt.xlabel('Time')
    plt.ylabel('Performance Score')
    plt.legend()
    plt.grid(True)
    plt.tight_layout()
    plt.show()

if __name__ == "__main__":
    performance_df = get_ai_performance_data()
    plot_ai_performance(performance_df)

This code displays a graph of AI performance scores from the past to the present and marks the reference grade, like GPT-3.5, with a horizontal line. This allows users to visually confirm how close their local AI is to the reference grade and how steadily it has been improving over time.

Results

• Users can now intuitively feel the performance improvements of the local AI.
• They can easily grasp where their AI stands compared to other models.
• Users' understanding and satisfaction with feature improvements have increased.

Summary — To Avoid the Same Pitfalls

• [ ] Recognize that when announcing new features or improvements, simply listing abstract numbers is insufficient.
• [ ] Actively consider visualization elements that users can easily understand and relate to.
• [ ] Clearly show relative performance by presenting reference grades or comparative targets.
• [ ] Remember that showing the temporal trend of a feature's performance can better convey the meaning of performance improvements.