DEV Community: Langtrace

Chat With Repos(PRs) Using Llama 3.1B

Tobi Aderounmu — Fri, 06 Sep 2024 01:39:18 +0000

Introduction:

When working with large repositories, keeping up with pull requests (PRs)-especially those containing thousands of lines of code-can be a real challenge. Whether it's understanding the impact of specific changes or navigating through massive updates, PR reviews can quickly become overwhelming. To tackle this, I set out to build a project that would allow me to quickly and efficiently understand changes within these large PRs.

Using Retrieval-Augmented Generation (RAG) combined with Langtrace's observability tools, I developed "Chat with Repo(PRs)"-a tool aimed at simplifying the process of reviewing large PRs. Additionally, I documented and compared the performance of Llama 3.1B to GPT-4o. Through this project, I explored how these models handle code explanations and summarizations, and which ones offer the best balance of speed and accuracy for this use case.

All code used in this blog can be found here

Before we dive into the details, let's outline the key tools employed in this project:
LLM Services:

OpenAI API
Groq API
Ollama (for local LLMs)

Embedding Model:

SentenceTransformers (specifically 'all-mpnet-base-v2')

Vector Database:

FAISS (Facebook AI Similarity Search)

LLM Observability:

Langtrace for end-to-end tracing and metrics

How Chat with Repo works:

The Chat with Repo(PRs) system implements a simple RAG architecture for PR analysis. It begins by ingesting PR data via GitHub's API, chunking large files to manage token limits. These chunks are vectorized using SentenceTransformers, creating dense embeddings that capture code semantics. A FAISS index enables sub-linear time similarity search over these embeddings. Queries undergo the same embedding process, facilitating semantic matching against the code index. The retrieved chunks form a dynamic context for the chosen LLM (via OpenAI, Groq, or Ollama), which then performs contextualized inference. This approach leverages both the efficiency of vector search and the generative power of LLMs, allowing for nuanced code understanding that adapts to varying PR complexities. Finally, the Langtrace integration provides granular observability into embedding and LLM operations, offering insights into performance bottlenecks and potential optimizations in the RAG pipeline. Let's dive into its key components.

Chunking Process:

The chunking process in this system is designed to break down large pull requests into manageable, context-rich pieces. The core of this process is implemented in the IngestionService class, particularly in the chunk_large_file and create_chunks_from_patch methods.
When a PR is ingested, each file's patch is processed individually. The chunk_large_file method is responsible for splitting large files:

def chunk_large_file(self, file_patch: str, chunk_size: int = config.CHUNK_SIZE) -> List[str]:
    lines = file_patch.split('\n')
    chunks = []
    current_chunk = []
    current_chunk_size = 0

    for line in lines:
        line_size = len(line)
        if current_chunk_size + line_size > chunk_size and current_chunk:
            chunks.append('\n'.join(current_chunk))
            current_chunk = []
            current_chunk_size = 0
        current_chunk.append(line)
        current_chunk_size += line_size

    if current_chunk:
        chunks.append('\n'.join(current_chunk))

    return chunks

This method splits the file based on a configured chunk size, ensuring that each chunk doesn't exceed this limit. It's a line-based approach that tries to keep logical units of code together as much as possible within the size constraint.
Once the file is split into chunks, the create_chunks_from_patch method processes each chunk. This method enriches each chunk with contextual information:

def create_chunks_from_patch(self, repo_info, pr_info, file_info, repo_explanation, pr_explanation):

    code_blocks = self.chunk_large_file(file_info['patch'])
    chunks = []

    for i, block in enumerate(code_blocks):
        chunk_explanation = self.generate_safe_explanation(f"Explain this part of the code and its changes: {block}")

        chunk = {
            "code": block,
            "explanations": {
                "repository": repo_explanation,
                "pull_request": pr_explanation,
                "file": file_explanation,
                "code": chunk_explanation
            },
            "metadata": {
                "repo": repo_info["name"],
                "pr_number": pr_info["number"],
                "file": file_info["filename"],
                "chunk_number": i + 1,
                "total_chunks": len(code_blocks),
                "timestamp": pr_info["updated_at"]
            }
        }
        chunks.append(chunk)

It generates an explanation for each code block using the LLM service.
It attaches metadata including the repository name, PR number, file name, chunk number, and timestamp.
It includes broader context like repository and pull request explanations.
This approach ensures that each chunk is not just a slice of code, but a rich, context-aware unit:

This includes:

The actual code changes
An explanation of those changes
File-level context
PR-level context
Repository-level context

Embedding and Similarity Search:

The EmbeddingService class handles the creation of embeddings and similarity search:
1. Embedding Creation:
For each chunk, we create an embedding using SentenceTransformer:

text_to_embed = self.get_full_context(chunk)
embedding = self.model.encode([text_to_embed])[0]

The embedding combines code content, code explanation, file explanation, PR explanation, and repository explanation.
2. Indexing:
We use FAISS to index these embeddings:

self.index.add(np.array([embedding]))

3. Query Processing:
When a question is asked, we create an embedding for the query and perform a similarity search:

query_vector = self.model.encode([query])

D, I = self.index.search(query_vector, k)

4. Chunk Selection:
The system selects the top k chunks (default 3) with the highest similarity scores.
This captures both code structure and semantic meaning, allowing for relevant chunk retrieval even when queries don't exactly match code syntax. FAISS enables efficient similarity computations, making it quick to find relevant chunks in large repositories.

Langtrace Integration:

To ensure comprehensive observability and performance monitoring, we've integrated Langtrace into our "Chat with Repo(PRs)" application. Langtrace provides real-time tracing, evaluations, and metrics for our LLM interactions, vector database operations, and overall application performance.

Model Performance Evaluation: Llama 3.1 70b Open-Source vs. GPT-4o Closed-Source LLMs in Large-Scale Code Review:

To explore how open-source models compare to their closed-source counterparts in handling large PRs, I conducted a comparative analysis between Llama 3.1b (open-source) and GPT-4o (closed-source). The test case involved a significant update to the Langtrace's repository, with over 2,300 additions, nearly 200 deletions, 250 commits, and changes across 47 files. My goal was to quickly understand these specific changes and assess how each model performs in code review tasks.
Methodology:
I posed a set of technical questions related to the pull request (PR), covering:

Specific code change explanations
Broader architectural impacts
Potential performance issues
Compatibility concerns

Both models were provided with the same code snippets and contextual information. Their responses were evaluated based on:

Technical accuracy
Depth of understanding
Ability to infer broader system impacts

Key Findings:

Code Understanding:

Llama 3.1b performed well in understanding individual code changes, especially with workflow updates and React component changes.
GPT-4o had a slight edge in connecting changes to the overall system architecture, such as identifying the ripple effect of modifying API routes on Prisma schema updates.

Knowledge of Frameworks:

Both models demonstrated strong understanding of frameworks like React, Next.js, and Prisma.
Llama 3.1b's versatility is impressive, particularly in web development knowledge, showing that open-source models are closing the gap on specialized domain expertise.

Architectural Insights:

GPT-4o excelled in predicting the broader implications of local changes, such as how adjustments to token-counting methods could affect the entire application.
Llama 3.1b, while precise in explaining immediate code impacts, was less adept at extrapolating these changes to system-wide consequences.

Handling Uncertainty:

Both models appropriately acknowledged uncertainty when presented with incomplete data, which is crucial for reliable code review.
Llama 3.1b's ability to express uncertainty highlights the progress open-source models have made in mimicking sophisticated reasoning.

Technical Detail vs. Broader Context:

Llama 3.1b provided highly focused and technically accurate explanations for specific code changes.
GPT-4o offered broader system context, though sometimes at the expense of missing finer technical details.

Question Comparison:

Below are examples of questions posed to both models, the expected output, and their respective answers:

Conclusion:

While GPT-4o remains stronger in broader architectural insights, Llama 3.1b's rapid progress and versatility in code comprehension make it a powerful option for code review. Open-source models are catching up quickly, and as they continue to improve, they could play a significant role in democratizing AI-assisted software development. The ability to tailor and integrate these models into specific development workflows could soon make them indispensable tools for reviewing, debugging, and managing large codebases.

We'd love to hear your thoughts! Join our community on Discord or reach out at support@langtrace.ai to share your experiences, insights, and suggestions. Together, we can continue advancing observability in LLM development and beyond.

Happy tracing!

Useful Resources
Getting started with Langtrace https://docs.langtrace.ai/introduction
Langtrace Twitter(X) https://x.com/langtrace_ai
Langtrace Linkedin https://www.linkedin.com/company/langtrace/about/
Langtrace Website https://langtrace.ai/
Langtrace Discord https://discord.langtrace.ai/
Langtrace Github https://github.com/Scale3-Labs/langtrace

How Salomatic Used Langtrace to Build a Reliable Medical Report Generation System

Tobi Aderounmu — Tue, 27 Aug 2024 16:45:07 +0000

Introduction: Transforming Healthcare Reporting with LLMs

By Tobi Aderounmu

When Anton and his co-founders launched Salomatic, their mission was clear: make medical reports understandable for everyone. Based in Tashkent, Uzbekistan, Salomatic uses large language models (LLMs) to generate detailed and easy-to-read patient consultations from just a few pages of doctor notes and lab results. In a country where medical data is often presented in a confusing and technical manner, Salomatic’s solution has quickly gained traction.

However, as Salomatic grew, so did their challenges. Generating accurate and reliable reports was becoming increasingly difficult. LLMs, though powerful, are prone to missing key medical data like lab results—a non-starter for healthcare professionals. That's when the team turned to Langtrace, an observability tool that changed the way they monitored and improved their LLM-powered system.

The Problem: Managing Complexity in Medical Data

Salomatic's solution aims to address a critical pain point in healthcare: the lack of clarity in medical reports. Doctors often scribble notes that are incomprehensible to patients, leaving them confused and unsure about their health. Salomatic’s product takes these notes and, using LLMs, generates 20-page patient-friendly consultations that explain diagnoses, treatments, and lifestyle recommendations in a way anyone can understand.

But the process wasn’t without its hurdles. LLMs frequently skipped over entire sections of lab data, leading to incomplete reports. Errors in extracting and structuring data meant Salomatic was spending hours manually correcting reports. "We were manually fixing 40% of the reports, which was unsustainable," Anton recalls.

The Solution: Langtrace’s Game-Changing LLM and DSPy Observability

Enter Langtrace. Integrated into Salomatic’s workflow, Langtrace provided the visibility they needed to diagnose and fix errors in real-time. Langtrace's ability to trace and observe the performance of LLMs allowed the team to see exactly where the system was breaking down.

One significant challenge arose when Salomatic's system struggled to handle certain data extractions, leading to inconsistencies and errors that hindered their ability to generate accurate reports. With Langtrace, the team gained the visibility needed to diagnose the root cause of these issues and quickly implement the necessary fixes. This led to a dramatic improvement in the system's reliability and overall performance.

“We learned more about how DSPy really works in a few hours with Langtrace than in months of trial and error,” Anton explains. This newfound clarity allowed Salomatic to resolve persistent issues, significantly reducing errors and enabling them to increase automation and scale their operations effectively.

The Technical Stack: Building a Reliable System

Salomatic's tech stack includes:

Python 3.12: The backbone of their codebase.
DSPy: Handles LLM interactions and structured data extraction, crucial for breaking down complex tasks.
Pydantic: Used for data modeling and validation.
Azure Cloud with Azure OpenAI API Service: Powers their LLMs.
FastAPI and SQLAlchemy & Alembic: Manage their backend services and database migrations.

The decision to use DSPy was intentional. LLMs alone couldn’t reliably extract all the necessary data from unstructured doctor notes, often missing critical lab results. DSPy allowed them to break down the extraction process into manageable tasks—first extracting lab panel names, then extracting the lab results for each individual panel. This layered approach drastically improved accuracy.

But DSPy wasn’t foolproof. That’s where Langtrace came in, giving Salomatic the insights needed to fine-tune their DSPy modules. Langtrace’s native support for DSPy made it the perfect fit for their debugging needs, ensuring that their LLMs delivered high-quality, structured reports.

Impact: Scaling with Confidence

Since integrating Langtrace, Salomatic has seen tangible improvements:

Significant reduction in report errors: Langtrace helped the team identify and resolve key issues that previously required extensive manual corrections, allowing them to significantly decrease the amount of manual work needed for each report.
Increased automation: With errors minimized, Salomatic has been able to automate more of their report generation process, saving time and resources.
Improved reliability: The system is now capable of generating 10 detailed reports per day, with plans to scale up to 500 reports daily as the technology continues to improve.

Salomatic's goal is to become the go-to solution for clinics in Uzbekistan and beyond. By fine-tuning their system with Langtrace, they are well on their way to achieving this.

Key Takeaways: Why Langtrace Stood Out

Native DSPy Support: Langtrace’s seamless integration with DSPy made it easier to diagnose and fix issues quickly.
Improved Understanding: Anton and his team learned more about their LLM system in a few hours with Langtrace than they had in months of trying to solve problems manually.
Better Product Quality: Langtrace helped Salomatic improve their report accuracy and reliability, reducing the number of complaints from their clinic partners and allowing them to scale their operations confidently.

Final Thoughts

“If LLMs are the brains of our solution, than DSPy is our hands, and Langtrace is our eyes,” Anton says. With Langtrace, Salomatic has not only been able to fix critical bugs but also gain a deeper understanding of how their system works. This has been instrumental in helping them scale their operations while maintaining the high standards required in the medical field.

If you’re a hospital looking for a better way to communicate lab results or provide clear patient consultations, don’t hesitate to reach out to Salomatic!

We’d love to hear your thoughts! Join our community on Discord or reach out at support@langtrace.ai to share your experiences, insights, and suggestions. Together, we can continue advancing observability in LLM development and beyond.

Happy tracing!