DEV Community: Juan Stoppa

What is LangGraph and How to Use It for Building AI Agents

Juan Stoppa — Fri, 21 Feb 2025 17:33:49 +0000

I keep finding myself going back to the LangChain documentation to figure out how to use LangGraph. While the documentation is comprehensive, it can be overwhelming to navigate, especially when you're trying to build advanced AI agents.

This guide is my attempt to consolidate the key concepts and practical implementations of LangGraph in one place. Whether you're building a conversational agent that needs to remember context, a multi-step reasoning agent, or a complex workflow that coordinates multiple AI components, LangGraph provides the framework to make it happen. I created this reference for myself but I hope it helps others who want a more straightforward explanation of how to use LangGraph effectively.

What is LangGraph?

LangGraph is a framework that brings state machines to Large Language Model (LLM) applications. It's particularly useful when you need to build complex, stateful applications with LLMs. The framework allows you to structure your application as a graph, where each node represents a specific task or state, and edges represent transitions between states.

Basic LangGraph Example

Let's start with a simple example to understand the core concepts: we are developing a simple agent that collects information and processes it, the actions of collecting and processing information are just fixed actions but they can be replaced with more complex actions.

from langgraph.graph import StateGraph
from typing import TypedDict, Annotated
from langgraph.graph.message import add_messages
from langchain_core.runnables.graph import MermaidDrawMethod

class State(TypedDict):
    messages: Annotated[list[str], add_messages]
    current_step: str

def collect_info(state: State) -> dict:
    print("\n--> In collect_info")
    print(f"Messages before: {state['messages']}")

    messages = state["messages"] + ["Information collected"]
    print(f"Messages after: {messages}")

    return {
        "messages": messages,
        "current_step": "process"
    }

def process_info(state: State) -> dict:
    print("\n--> In process_info")
    print(f"Messages before: {state['messages']}")

    messages = state["messages"] + ["Information processed"]
    print(f"Messages after: {messages}")

    return {
        "messages": messages,
        "current_step": "end"
    }

# Create and setup graph
workflow = StateGraph(State)

# Add nodes
workflow.add_node("collect", collect_info)
workflow.add_node("process", process_info)

# Add edges
workflow.add_edge("collect", "process")

# Set entry and finish points
workflow.set_entry_point("collect")
workflow.set_finish_point("process")

app = workflow.compile()

# Run workflow
print("\nStarting workflow...")
initial_state = State(messages=["Starting"], current_step="collect")
final_state = app.invoke(initial_state)
print(f"\nFinal messages: {final_state['messages']}")

# Save the graph visualization as PNG
png_data = app.get_graph().draw_mermaid_png(draw_method=MermaidDrawMethod.API)
with open("workflow_graph.png", "wb") as f:
    f.write(png_data)
print("\nGraph visualization saved as 'workflow_graph.png'")

This example shows the basic structure of a LangGraph application:

Define your state using TypedDict, it contains the information that the workflow will need to keep track of.
Create functions for each state, these functions are the actions that the workflow will perform.
Build a graph with nodes and edges, the nodes are the states and the edges are the transitions between them.
Compile the graph into a runnable application, this will create a callable object that can be invoked with an initial state.
I also added a simple way to save the graph visualization as a PNG file, this will work if you are running this example locally and should save a file that will show the graph structure like below.

The graph is a good way to understand the workflow, it shows the nodes and the edges between them, the entry and finish points and the state of the workflow.

This particular example is not very useful but it shows the core concepts of LangGraph, you can simply replace the fixed actions with more complex ones and build a useful agent.

Serving LangGraph as a Web Service

While LangGraph itself doesn't include built-in server capabilities, you can easily create a web service using FastAPI to serve your LangGraph workflows. Below I have modified the previous example to add a simple FastAPI server that allows you to run the workflow from a web interface.

from fastapi import FastAPI
from langgraph.graph import StateGraph
from typing import TypedDict, Annotated, List
from langgraph.graph.message import add_messages
from pydantic import BaseModel

# Initialize FastAPI app
app = FastAPI(title="LangGraph Agent API")

class State(TypedDict):
    messages: Annotated[list[str], add_messages]
    current_step: str

class AgentInput(BaseModel):
    messages: List[str]

def collect_info(state: State) -> dict:
    print("\n--> In collect_info")
    print(f"Messages before: {state['messages']}")

    messages = state["messages"] + ["Information collected"]
    print(f"Messages after: {messages}")

    return {
        "messages": messages,
        "current_step": "process"
    }

def process_info(state: State) -> dict:
    print("\n--> In process_info")
    print(f"Messages before: {state['messages']}")

    messages = state["messages"] + ["Information processed"]
    print(f"Messages after: {messages}")

    return {
        "messages": messages,
        "current_step": "end"
    }

# Create and setup graph
workflow = StateGraph(State)

# Add nodes
workflow.add_node("collect", collect_info)
workflow.add_node("process", process_info)

# Add edges
workflow.add_edge("collect", "process")

# Set entry and finish points
workflow.set_entry_point("collect")
workflow.set_finish_point("process")

# Compile the workflow
agent = workflow.compile()


@app.post("/run-agent")
async def run_agent(input_data: AgentInput):
    """
    Run the agent with the provided input messages.
    """
    initial_state = State(messages=input_data.messages, current_step="collect")
    final_state = agent.invoke(initial_state)
    return {"messages": final_state["messages"]}

@app.get("/")
async def root():
    """
    Root endpoint that returns basic API information.
    """
    return {"message": "LangGraph Agent API is running", "endpoints": ["Navigate to https://jstoppa-langgraph-basic-example-api.hf.space/docs#/default/run_agent_run_agent_post to run the example"]}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=7860)

You can run this locally or using the Hugging Face space below, this is the URL to access the swagger API for this example https://jstoppa-langgraph-basic-example-api.hf.space/docs, the API has an end point to run the agent and it returns the messages we've seen in our previous example (see below the results). In simple words, the API has an end point to run the agent and it returns the messages we've seen in our previous example.

Making it all work with a more interesting example

We are now going to create a more interesting example, an AI agent that does code reviews, this is far from a production-ready agent but it will give us a better understanding of how to use LangGraph.

The screenshot below shows the interface for the code review agent, the user can enter a code and the agent will return a report with the code analysis. This interface uses the Gradio library to create a simple web interface, this saves a lot of time compared to building a full web app.

The full code is provided after this but the most impportant part of the example is the graph, it shows the nodes and the edges between them, the entry and finish points and the state of the workflow, this is how the graph will look through the code and analyse the code with different agents that are especialised on different aspects. This is a very similar apporach we've seen in the previous example but it contains more actions and the actions do use LLMs to analyse the code.

# Create and setup graph
workflow = StateGraph(State)

# Add nodes
workflow.add_node("style", analyze_code_style)
workflow.add_node("security", analyze_security)
workflow.add_node("performance", analyze_performance)
workflow.add_node("architecture", analyze_architecture)
workflow.add_node("recommendations", generate_final_recommendations)

# Add edges
workflow.add_edge("style", "security")
workflow.add_edge("security", "performance")
workflow.add_edge("performance", "architecture")
workflow.add_edge("architecture", "recommendations")

# Set entry and finish points
workflow.set_entry_point("style")
workflow.set_finish_point("recommendations")

# Compile the workflow
agent = workflow.compile()

the full code for the agent is below and it can also be found in Hugging Face below.

import gradio as gr
from langgraph.graph import StateGraph
from typing import TypedDict, Annotated, List, Dict
from langgraph.graph.message import add_messages
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
import json
import requests
import os
from dotenv import load_dotenv
import time

# Load environment variables
load_dotenv()

# Define the state structure
class State(TypedDict):
    messages: Annotated[list[SystemMessage | HumanMessage | AIMessage], add_messages]
    current_step: str
    code: str
    style_analysis: Dict
    security_analysis: Dict
    performance_analysis: Dict
    architecture_analysis: Dict
    final_recommendations: Dict

def call_huggingface_api(prompt: str, max_retries=3) -> Dict:
    """Call Hugging Face API with retry logic and proper error handling."""
    api_key = os.getenv("HUGGINGFACE_API_KEY")
    if not api_key:
        raise ValueError("HUGGINGFACE_API_KEY not found in environment variables")

    # You can change this to any model you prefer
    API_URL = "https://api-inference.huggingface.co/models/mistralai/Mixtral-8x7B-Instruct-v0.1"
    headers = {"Authorization": f"Bearer {api_key}"}

    for attempt in range(max_retries):
        try:
            response = requests.post(
                API_URL,
                headers=headers,
                json={
                    "inputs": prompt,
                    "parameters": {
                        "max_new_tokens": 1000,
                        "temperature": 0.7,
                        "top_p": 0.95,
                        "return_full_text": False
                    }
                }
            )

            if response.status_code == 200:
                result = response.json()
                if isinstance(result, list) and len(result) > 0:
                    # Extract the generated text
                    text = result[0].get('generated_text', '')
                    # Try to parse as JSON if it contains JSON
                    try:
                        # Find JSON content between triple backticks if present
                        if "\u0060\u0060\u0060json" in text:
                            json_str = text.split("\u0060\u0060\u0060json")[1].split("\u0060\u0060\u0060")[0].strip()
                        else:
                            json_str = text.strip()
                        return json.loads(json_str)
                    except json.JSONDecodeError:
                        return {"error": "Failed to parse JSON from response", "raw_text": text}

            # If model is loading, wait and retry
            if response.status_code == 503:
                wait_time = 2 ** attempt
                time.sleep(wait_time)
                continue

        except Exception as e:
            if attempt == max_retries - 1:
                return {"error": f"API call failed: {str(e)}"}
            time.sleep(2 ** attempt)

    return {"error": "Maximum retries reached"}

def analyze_code_style(state: State) -> dict:
    """Analyze code style and best practices."""
    code = state["code"]
    prompt = f"""You are a senior code reviewer focused on code style and best practices. Analyze this code:

{code}

Focus on:
1. Code readability and clarity
2. Adherence to common style guides
3. Variable/function naming
4. Code organization
5. Documentation quality

Provide your response in JSON format with these exact keys:
{{
    "issues": ["list of identified style issues"],
    "suggestions": ["list of improvement suggestions"],
    "overall_rating": "1-10 score as a number",
    "primary_concerns": ["list of main style concerns"]
}}"""

    analysis = call_huggingface_api(prompt)
    if "error" in analysis:
        analysis = {
            "issues": ["Error analyzing code style"],
            "suggestions": ["Try again later"],
            "overall_rating": 0,
            "primary_concerns": ["Analysis failed"]
        }

    messages = state["messages"] + [AIMessage(content="Completed code style analysis")]
    return {**state, "messages": messages, "style_analysis": analysis, "current_step": "security"}

def analyze_security(state: State) -> dict:
    """Analyze security vulnerabilities."""
    code = state["code"]
    prompt = f"""You are a security expert. Analyze this code for security vulnerabilities:

{code}

Focus on:
1. Input validation
2. Authentication/Authorization
3. Data exposure
4. Common vulnerabilities
5. Security best practices

Provide your response in JSON format with these exact keys:
{{
    "vulnerabilities": ["list of potential security issues"],
    "risk_levels": {{"vulnerability": "risk level"}},
    "recommendations": ["list of security improvements"],
    "overall_security_score": "1-10 score as a number"
}}"""

    analysis = call_huggingface_api(prompt)
    if "error" in analysis:
        analysis = {
            "vulnerabilities": ["Error analyzing security"],
            "risk_levels": {"Error": "High"},
            "recommendations": ["Try again later"],
            "overall_security_score": 0
        }

    messages = state["messages"] + [AIMessage(content="Completed security analysis")]
    return {**state, "messages": messages, "security_analysis": analysis, "current_step": "performance"}

def analyze_performance(state: State) -> dict:
    """Analyze code performance."""
    code = state["code"]
    prompt = f"""You are a performance optimization expert. Analyze this code for performance issues:

{code}

Focus on:
1. Time complexity
2. Space complexity
3. Resource usage
4. Bottlenecks
5. Optimization opportunities

Provide your response in JSON format with these exact keys:
{{
    "bottlenecks": ["list of identified performance bottlenecks"],
    "complexity_analysis": {{
        "time_complexity": "Big O notation",
        "space_complexity": "Big O notation",
        "critical_sections": ["list of critical sections"]
    }},
    "optimization_suggestions": ["list of performance improvements"],
    "performance_score": "1-10 score as a number"
}}"""

    analysis = call_huggingface_api(prompt)
    if "error" in analysis:
        analysis = {
            "bottlenecks": ["Error analyzing performance"],
            "complexity_analysis": {
                "time_complexity": "Unknown",
                "space_complexity": "Unknown",
                "critical_sections": []
            },
            "optimization_suggestions": ["Try again later"],
            "performance_score": 0
        }

    messages = state["messages"] + [AIMessage(content="Completed performance analysis")]
    return {**state, "messages": messages, "performance_analysis": analysis, "current_step": "architecture"}

def analyze_architecture(state: State) -> dict:
    """Analyze code architecture patterns."""
    code = state["code"]
    prompt = f"""You are a software architect. Analyze this code's architectural patterns:

{code}

Focus on:
1. Design patterns used
2. Code modularity
3. Component relationships
4. Architectural anti-patterns
5. System design principles

Provide your response in JSON format with these exact keys:
{{
    "patterns_identified": ["list of design patterns found"],
    "architectural_issues": ["list of architectural concerns"],
    "improvement_suggestions": ["list of architectural improvements"],
    "architecture_score": "1-10 score as a number"
}}"""

    analysis = call_huggingface_api(prompt)
    if "error" in analysis:
        analysis = {
            "patterns_identified": ["Error analyzing architecture"],
            "architectural_issues": ["Analysis failed"],
            "improvement_suggestions": ["Try again later"],
            "architecture_score": 0
        }

    messages = state["messages"] + [AIMessage(content="Completed architecture analysis")]
    return {**state, "messages": messages, "architecture_analysis": analysis, "current_step": "recommendations"}

def generate_final_recommendations(state: State) -> dict:
    """Generate final recommendations based on all analyses."""
    code = state["code"]
    prompt = f"""Analyze all previous results and provide final recommendations for this code:

Style Analysis: {json.dumps(state.get('style_analysis', {}))}
Security Analysis: {json.dumps(state.get('security_analysis', {}))}
Performance Analysis: {json.dumps(state.get('performance_analysis', {}))}
Architecture Analysis: {json.dumps(state.get('architecture_analysis', {}))}

Provide your response in JSON format with these exact keys:
{{
    "critical_issues": ["list of most critical issues"],
    "priority_improvements": ["list of high-priority improvements"],
    "quick_wins": ["list of easy-to-implement improvements"],
    "long_term_suggestions": ["list of long-term improvements"],
    "overall_health_score": "1-10 score as a number"
}}"""

    recommendations = call_huggingface_api(prompt)
    if "error" in recommendations:
        recommendations = {
            "critical_issues": ["Error generating recommendations"],
            "priority_improvements": ["Try again later"],
            "quick_wins": [],
            "long_term_suggestions": [],
            "overall_health_score": 0
        }

    messages = state["messages"] + [AIMessage(content="Generated final recommendations")]
    return {**state, "messages": messages, "final_recommendations": recommendations, "current_step": "end"}

def format_output(state: State) -> str:
    """Format the analysis results into a readable output."""
    output = """🔍 Code Analysis Report

🎨 Style & Best Practices
"""
    style = state.get("style_analysis", {})
    output += f"Rating: {style.get('overall_rating', 'N/A')}/10\n"
    output += "Issues:\n" + "\n".join([f"• {issue}" for issue in style.get("issues", [])]) + "\n\n"

    output += """🔒 Security Analysis
"""
    security = state.get("security_analysis", {})
    output += f"Score: {security.get('overall_security_score', 'N/A')}/10\n"
    vulnerabilities = security.get("vulnerabilities", [])
    risk_levels = security.get("risk_levels", {})
    output += "Vulnerabilities:\n" + "\n".join([f"• {v} ({risk_levels.get(v, 'Unknown')})" for v in vulnerabilities]) + "\n\n"

    output += """⚡ Performance Analysis
"""
    perf = state.get("performance_analysis", {})
    output += f"Score: {perf.get('performance_score', 'N/A')}/10\n"
    output += "Bottlenecks:\n" + "\n".join([f"• {b}" for b in perf.get("bottlenecks", [])]) + "\n\n"

    output += """🏗️ Architecture Analysis
"""
    arch = state.get("architecture_analysis", {})
    output += f"Score: {arch.get('architecture_score', 'N/A')}/10\n"
    output += "Patterns:\n" + "\n".join([f"• {p}" for p in arch.get("patterns_identified", [])]) + "\n\n"

    output += """📋 Final Recommendations
"""
    final = state.get("final_recommendations", {})
    output += f"Overall Health Score: {final.get('overall_health_score', 'N/A')}/10\n\n"
    output += "Critical Issues:\n" + "\n".join([f"• {i}" for i in final.get("critical_issues", [])]) + "\n\n"
    output += "Priority Improvements:\n" + "\n".join([f"• {i}" for i in final.get("priority_improvements", [])])

    return output

# Create and setup graph
workflow = StateGraph(State)

# Add nodes
workflow.add_node("style", analyze_code_style)
workflow.add_node("security", analyze_security)
workflow.add_node("performance", analyze_performance)
workflow.add_node("architecture", analyze_architecture)
workflow.add_node("recommendations", generate_final_recommendations)

# Add edges
workflow.add_edge("style", "security")
workflow.add_edge("security", "performance")
workflow.add_edge("performance", "architecture")
workflow.add_edge("architecture", "recommendations")

# Set entry and finish points
workflow.set_entry_point("style")
workflow.set_finish_point("recommendations")

# Compile the workflow
agent = workflow.compile()

def analyze_code(code: str) -> str:
    """Analyze the provided code using multiple perspectives."""
    initial_state = State(
        messages=[SystemMessage(content="Starting code analysis...")],
        current_step="style",
        code=code,
        style_analysis={},
        security_analysis={},
        performance_analysis={},
        architecture_analysis={},
        final_recommendations={}
    )

    final_state = agent.invoke(initial_state)
    return format_output(final_state)

# Create Gradio interface
iface = gr.Interface(
    fn=analyze_code,
    inputs=gr.Code(
        label="Enter your code for analysis",
        language="python",
        lines=20
    ),
    outputs=gr.Textbox(
        label="Analysis Results",
        lines=25
    ),
    title="🔍 Code Architecture Critic",
    description="Paste your code to get a comprehensive analysis of style, security, performance, and architecture.",
    examples=[
        ['''def process_data(data):
    result = []
    for i in range(len(data)):
        for j in range(len(data)):
            if data[i] + data[j] == 10:
                result.append((data[i], data[j]))
    return result

def save_to_db(user_input):
    query = "INSERT INTO users VALUES ('" + user_input + "')"
    db.execute(query)

API_KEY = "sk_test_123456789"''']
    ],
    theme=gr.themes.Soft()
)

if __name__ == "__main__":
    iface.launch()

Conclusion

LangGraph makes it easier to build AI agents that need to manage complex workflows. The graph-based approach keeps things organised and flexible, especially when dealing with multi-step processes or memory.

Even though LangGraph doesn’t come with built-in server features, it works well with FastAPI and other frameworks to serve agents as APIs. Whether you’re building a chatbot, a code reviewer, or something else entirely, it gives you a solid foundation to work with.

I’m still experimenting and learning, so I’ll keep updating this post as I find better ways to use LangGraph. If you’ve built something cool with it or have any questions, let me know—happy to chat!

Resources

LangGraph Documentation
FastAPI Documentation
LangChain (LangGraph is part of the LangChain ecosystem)

I hope you like this article, if you want to hear more follow me on X at @juanstoppa where I regularly post about AI

Initial look at ChatGPT with Canvas

Juan Stoppa — Fri, 04 Oct 2024 00:06:44 +0000

AI for writing code is a hot topic these days. We've seen so many products coming out during 2024, it really caught my attention six months ago when I saw the Devin demo from Cognition (I'm still on the waiting list). Three months after Claude 3.5 Sonnet was released and it showed that it's the best model in the market for coding especially when used with Cursor. Anthropic added the artifacts to Claude a few months later bringing the ability to execute the code generated by the model directly in the browser.

OpenAI today jumped into the arena releasing Canvas and my initial reaction are very positive, I can see the team really took all the complexity that comes with a code editor and made it very simple to use with AI.

How to use Canvas?

If you head over to ChatGPT, you should be able to see the option to work with Canvas (you will need a ChatGPT Plus subscription to access it).

What makes it very simple is that you just get a regular chat window similar to what you get with the chat version of ChatGPT.
Once you prompt what you'd like to do, Canvas switches the interface into a code editor and starts writing the code. The chat functionality is still on the left hand side of the page whilst the right hand side has the code editor.

Your browser does not support the video tag.

You can still chat with the AI about your code, ask questions, or request changes, just like in the regular ChatGPT interface. But now, you can see those changes happen in real-time in the code editor. It's a neat mixed of conversation and coding that feels pretty intuitive once you start using it.

Some interesting features

The interface is very clean and simple to use and it's definitely in a beta version but it gives you an idea of where OpenAI is heading with this.

A handy feature is that you can highlight a part of your code and ask ChatGPT to explain it or change it like you would do with Cursor or GitHub Copilot. This makes it easy to work on specific bits of your code with AI help.

Your browser does not support the video tag.

The feature I like the most and gives a hint of how OpenAI is approaching coding is the little icon list on the right hand side

Your browser does not support the video tag.

Here what each action does:

Add comments - I find documenting the code very useful, AI can do part of it, I personally document the code on specific points to help the next devleoper that needs to deal with it.
Add logs - Very useful when debugging, ChatGPT adds logs in the places where you need them.
Fix Bug - It's interesting and probably aimed at people not very familiar with coding, by simply clicking the link you can fix bugs you might have in the code. I'll keep studing this functionality, it might be a great feature if AI is able to understand by itself that there are bugs in the code.
Port to a language - This is one I didn't expect but makes sense, you can port the code to another language, the options are PHP, C++, Python, Javascript, Typescript and Java. I asked ChatGPT to port it to C++ and it did pretty well.
Code Review - Developers do code review all the time, I like that it leaves the suggestions in little windows next to each line of code.

Conclusion

This is a very early beta mode but still shows how OpenAI is focusing in simplicity and making things easier to use for everyone, not only devleopers. The icons on the right hand side to comment, add logs, fix bugs, port to another language and reivew the code are very handy, they are probably the most used actions when writting code.

Once again we see Open AI focusing on making things that are intuitive and easy to use. This editor also confirms what I've seen in the OpenAI DevDay, the company seems to be shifting into focusing on making tools to write code, the market for this is huge and where most companies are going to invest in the next few years, from that perspective I think OpenAI is taking the right direction.

I hope you like this article, if you want to hear more follow me on X at @juanstoppa where I regularly post about AI.

How I used ChatGPT o1 and Claude for generating a SQL RBAC report and was surprised by the results

Juan Stoppa — Sun, 22 Sep 2024 14:17:27 +0000

One of the most comprehensive Role Based Access Control (RBAC) I ever worked with is the one that comes with Microsoft Dynamics 365, the security model is very granular and embedded deep in the SQL tables making it very hard to understand the access permissions for a given user but at the same time it's very powerful and flexible, giving the ability to model any security structure you can imagine.

Microsoft Dynamics 365 has the concept that records in the SQL table are owned by a user or a team. Access to these records is given through security roles which can be assigned to a user or team, these security roles define what actions a user can perform on a record, such as reading, creating, deleting, assigning, etc. At the same time, a user can belong to one or multiple teams inheriting all the security permissions, this makes it extremely hard to understand the access permissions for a given user. You can see below a screenshot of the interface to configure a single security role.

Dynamics 365 provides an API to check the access for a single table and access type but I've never seen anyone that knows the deep details of how the tables in the database provide this access so I was wondering if ChatGPT or Claude would be able to provide a report for such an obscure and complex area I've never seen anyone being able to crack. The whole iteration was pretty fascinating because both LLMs understood the system much more than I expected, there is very little documentation on how the tables work and still the LLMs were able to provide a lot of information that would be hard to find in any Microsoft documentation.

Generating the prompt

I started by generating a prompt that asks for the information I needed, I used the following prompt:

Generate a SQL query for Microsoft Dynamics 365 that retrieves all security privileges for each entity accessible to a specific user. 
The query should:
1. Accept a username as an input parameter
2. Include all entities the user has access to
3. Show the specific privileges (Create, Read, Update, Delete, Append, AppendTo, Assign, Share) for each entity
4. Consider privileges granted through the user's security roles and team memberships
5. Specify the level of access for each privilege using the following scale:
   * No Access
   * User Level
   * Business Unit Level
   * Parent: Child Business Unit Level
   * Organization Level
6. Display results in a format similar to: Entity | Create | Read | Update | Delete | Append | AppendTo | Assign | Share
7. For each privilege, instead of a simple 'Yes/No' or binary indicator, show the highest level of access granted to the user across all their roles and teams.
8. The query should join relevant CRM tables like SystemUserPrincipals, RolePrivileges, TeamPrivileges, and PrincipalObjectAccess. 
9. Use appropriate system views if available.
10. Ensure the query accounts for the hierarchical nature of access levels, where Organization Level access supersedes Business Unit Level, which in turn supersedes User Level access.
11. The query should be optimized for performance, as it may need to process a large number of records in a production CRM environment.
Example desired output: 
Entity          | Create        | Read          | Update        | Delete        | Append        | AppendTo      | Assign        | Share  
Account         | Organization  | User          | No Access     | Parent: Child | Business Unit | Organization  | User Contact  | Organization 
Activity        | User          | User          | User          | No Access     | Business Unit | Organization  | User Contact  | Organization 
Connection      | User          | User          | User          | No Access     | No Access     | Parent: Child | Parent: Child | Organization

I think this prompt gives a pretty good idea of what I needed, it's a bit complex what I'm asking the model to do but I was really hoping to get some super human results.

Testing the prompt on Claude and ChatGPT

Now with the prompt ready I can test it by asking the same to Claude and ChatGPT

Claude first attempt

Starting with Claude Sonnet 3.5, this was the first result:

-- Declare input parameter for username
DECLARE @Username NVARCHAR(100) = 'example.user@domain.com';

WITH UserRoles AS (
    -- Get all roles for the user, including those from team memberships
    SELECT DISTINCT sr.RoleId
    FROM SystemUserPrincipals sup
    JOIN SystemUserRoles sur ON sup.PrincipalId = sur.SystemUserId
    JOIN SecurityRole sr ON sur.RoleId = sr.RoleId
    WHERE sup.SystemUserId = (SELECT SystemUserId FROM SystemUser WHERE DomainName = @Username)
    UNION
    SELECT DISTINCT sr.RoleId
    FROM SystemUserPrincipals sup
    JOIN TeamMembership tm ON sup.PrincipalId = tm.SystemUserId
    JOIN TeamRoles tr ON tm.TeamId = tr.TeamId
    JOIN SecurityRole sr ON tr.RoleId = sr.RoleId
    WHERE sup.SystemUserId = (SELECT SystemUserId FROM SystemUser WHERE DomainName = @Username)
),
UserPrivileges AS (
    -- Aggregate privileges from all roles
    SELECT 
        rp.PrivilegeId,
        e.Name AS EntityName,
        MAX(rp.PrivilegeDepthMask) AS MaxPrivilegeDepthMask
    FROM UserRoles ur
    JOIN RolePrivileges rp ON ur.RoleId = rp.RoleId
    JOIN [Privilege] p ON rp.PrivilegeId = p.PrivilegeId
    JOIN EntityType e ON p.EntityTypeMask = e.EntityTypeMask
    GROUP BY rp.PrivilegeId, e.Name
),
AccessLevels AS (
    -- Define access levels
    SELECT 0 AS Mask, 'No Access' AS AccessLevel
    UNION SELECT 1, 'User'
    UNION SELECT 2, 'Business Unit'
    UNION SELECT 4, 'Parent: Child Business Unit'
    UNION SELECT 8, 'Organization'
)
SELECT 
    up.EntityName,
    MAX(CASE WHEN p.PrivilegeName = 'prvCreateEntity' THEN al.AccessLevel ELSE 'No Access' END) AS [Create],
    MAX(CASE WHEN p.PrivilegeName = 'prvReadEntity' THEN al.AccessLevel ELSE 'No Access' END) AS [Read],
    MAX(CASE WHEN p.PrivilegeName = 'prvUpdateEntity' THEN al.AccessLevel ELSE 'No Access' END) AS [Update],
    MAX(CASE WHEN p.PrivilegeName = 'prvDeleteEntity' THEN al.AccessLevel ELSE 'No Access' END) AS [Delete],
    MAX(CASE WHEN p.PrivilegeName = 'prvAppendEntity' THEN al.AccessLevel ELSE 'No Access' END) AS [Append],
    MAX(CASE WHEN p.PrivilegeName = 'prvAppendToEntity' THEN al.AccessLevel ELSE 'No Access' END) AS [AppendTo],
    MAX(CASE WHEN p.PrivilegeName = 'prvAssignEntity' THEN al.AccessLevel ELSE 'No Access' END) AS [Assign],
    MAX(CASE WHEN p.PrivilegeName = 'prvShareEntity' THEN al.AccessLevel ELSE 'No Access' END) AS [Share]
FROM UserPrivileges up
CROSS APPLY (
    SELECT PrivilegeId, PrivilegeName 
    FROM [Privilege] 
    WHERE PrivilegeName IN ('prvCreateEntity', 'prvReadEntity', 'prvUpdateEntity', 'prvDeleteEntity', 
                            'prvAppendEntity', 'prvAppendToEntity', 'prvAssignEntity', 'prvShareEntity')
) p
LEFT JOIN AccessLevels al ON up.MaxPrivilegeDepthMask & al.Mask = al.Mask
GROUP BY up.EntityName
ORDER BY up.EntityName;

pretty impressive with the initial results, the model definitely knows about the table structure in Microsoft Dynamics 365 but when running the query in the SQL Server you can see Claude didn't quite get the table names right.

The were a few issues:

it references a table called SecurityRole which doesn't exist, I suspect the name is TeamRoles
it tries to use a table called EntityType which again doesn't exist, it's probably trying to useEntityView
it knows about the table Privilege which does exist in the DB but the column PrivilegeName doesn't exist, the column is called Name

Once I made the changes, the query worked but it didn't return any result, it probably needs a bit more thought on how the tables are linked to each other

ChatGPT first attempt

I then tried the same prompt on ChatGPT o1-preview and got the following result:

DECLARE @username NVARCHAR(256) = 'your_username_here';
DECLARE @systemuserid UNIQUEIDENTIFIER;

-- Get the SystemUserId for the specified username
SELECT @systemuserid = su.SystemUserId
FROM SystemUser su
WHERE su.DomainName = @username;

-- CTE to get all RoleIds assigned to the user directly or via team memberships
WITH UserRoleIds AS (
    SELECT sur.RoleId
    FROM SystemUserRoles sur
    WHERE sur.SystemUserId = @systemuserid
    UNION
    SELECT tr.RoleId
    FROM TeamRoles tr
    INNER JOIN TeamMembership tm ON tr.TeamId = tm.TeamId
    WHERE tm.SystemUserId = @systemuserid
),
-- CTE to get the highest PrivilegeDepthMask for each PrivilegeId
UserPrivileges AS (
    SELECT
        rp.PrivilegeId,
        MAX(rp.PrivilegeDepthMask) AS MaxPrivilegeDepthMask
    FROM RolePrivilegesBase rp
    WHERE rp.RoleId IN (SELECT RoleId FROM UserRoleIds)
    GROUP BY rp.PrivilegeId
),
-- CTE to get Privilege details along with ObjectTypeCode
PrivilegesWithAccessRights AS (
    SELECT
        up.PrivilegeId,
        up.MaxPrivilegeDepthMask,
        p.Name,
        p.AccessRight,
        potc.ObjectTypeCode
    FROM UserPrivileges up
    INNER JOIN Privilege p ON up.PrivilegeId = p.PrivilegeId
    LEFT JOIN PrivilegeObjectTypeCodes potc ON p.PrivilegeId = potc.PrivilegeId
),
-- CTE to get Entity names
PrivilegesWithEntityName AS (
    SELECT
        pwac.PrivilegeId,
        pwac.Name,
        pwac.AccessRight,
        pwac.MaxPrivilegeDepthMask,
        pwac.ObjectTypeCode,
        ev.LogicalName AS EntityLogicalName,
        ev.OriginalLocalizedName AS EntityDisplayName
    FROM PrivilegesWithAccessRights pwac
    LEFT JOIN EntityView ev ON pwac.ObjectTypeCode = ev.ObjectTypeCode
),
-- CTE to map AccessRight and PrivilegeDepthMask to readable formats
PrivilegesWithAccessLevels AS (
    SELECT
        COALESCE(pwcn.EntityDisplayName, 'Global Privilege') AS EntityName,
        CASE pwcn.AccessRight
            WHEN 1 THEN 'Create'
            WHEN 2 THEN 'Read'
            WHEN 4 THEN 'Update'
            WHEN 16 THEN 'Delete'
            WHEN 32 THEN 'Assign'
            WHEN 64 THEN 'Share'
            WHEN 128 THEN 'Append'
            WHEN 256 THEN 'AppendTo'
            ELSE 'Unknown'
        END AS AccessRightName,
        CASE
            WHEN pwcn.MaxPrivilegeDepthMask = 8 THEN 'Organization Level'
            WHEN pwcn.MaxPrivilegeDepthMask = 4 THEN 'Parent: Child Business Unit Level'
            WHEN pwcn.MaxPrivilegeDepthMask = 2 THEN 'Business Unit Level'
            WHEN pwcn.MaxPrivilegeDepthMask = 1 THEN 'User Level'
            ELSE 'No Access'
        END AS AccessLevel
    FROM PrivilegesWithEntityName pwcn
    WHERE pwcn.AccessRight IN (1,2,4,16,32,64,128,256)
)
-- Final SELECT with PIVOT to display privileges per entity
SELECT
    EntityName,
    ISNULL([Create], 'No Access') AS [Create],
    ISNULL([Read], 'No Access') AS [Read],
    ISNULL([Update], 'No Access') AS [Update],
    ISNULL([Delete], 'No Access') AS [Delete],
    ISNULL([Append], 'No Access') AS [Append],
    ISNULL([AppendTo], 'No Access') AS [AppendTo],
    ISNULL([Assign], 'No Access') AS [Assign],
    ISNULL([Share], 'No Access') AS [Share]
FROM
    (
        SELECT
            EntityName,
            AccessRightName,
            AccessLevel
        FROM
            PrivilegesWithAccessLevels
    ) AS SourceTable
PIVOT
(
    MAX(AccessLevel)
    FOR AccessRightName IN ([Create], [Read], [Update], [Delete], [Append], [AppendTo], [Assign], [Share])
) AS PivotTable
ORDER BY EntityName;

It took 47 seconds for ChatGPT o1-review to generate a result, the query returned wasn't valid but the structure seems to be much more comprehensive than Claude's query and the table and columns names were almost correct.

The were only two issues in the query:

it referenced a table RolePrivilege which is called RolePrivilegesBase instead
it referenced a column name DisplayName in the EntiyView table which doesn't exist, it should be OriginalLocalizedName

After changing the table and column names, the query worked but it didn't return any result

Claude further attempts

I played back the modified query to Claude, it gave some reasonable comments but then it went back to generate queries referencing the table names that I previously highlighted as incorrect.

I tried a few back and forward prompts but Claude kept going into a loop without offering a valid query. I eventually asked to use Chain of Thought prompting which helped because Claude started to systematically ask me questions about the table structure to validate the query that is being generated.

This unfortunately didn't lead to a valid query either, Claude seems to get lost with so many details. I didn't try to start a completely new conversation with the best query generated but I suspect it will still struggle to generate a valid query.

ChatGPT further attempts

I played back the modified query to ChatGPT o1, it gave some reasonable comments on how we can move forward. I like because this new model seems to be implementing chain of thought prompting internally, it goes step by step and tries to understand what went wrong by asking me to gather more information.

What I also like about ChatGPT is that it gives very clear steps on what to do next, using a pattern like if Yes do this, if No do that.

I went back and forth providing as much information as possible, each response takes close to a minute to generate but their findings are very insightful and doesn't seem to go into a loop like what happens with Claude.

Conclusion

Both models are very impressive but neither manage to generate a valid report, I'd say that ChatGPT o1 did better than Claude Sonnet 3.5 but both are still very far from being able to generate a valid SQL report just by reading the documentation.

ChatGPT got very close and I think I need to spend more time answering all the questions it asks, I think the advantage is that ChatGPT doesn't seem to go into a loop like Claude does when it has too many details and is able to break down the problem in smaller steps, something that Claude doesn't do that well at least for this particular task.

You can see the full conversation for both models in

Claude - I don't seem to be able to share the entire conversation, I thought Claude was able, I'll update the blog once I find it
ChatGPT - https://chatgpt.com/share/66f01a99-e63c-8003-84f6-544c34e9d11b

I hope you like this article, if you want to hear more follow me on X at @juanstoppa where I regularly post about AI

Code Smarter, Not Harder: Developing with Cursor and Claude Sonnet

Juan Stoppa — Mon, 02 Sep 2024 23:45:24 +0000

I heard so much about Cursor on X in the last few days on how it can speed up your development process that I decided to give it a try, I have to say I'm very impressed with the product and the potential it has to change the way we develop software.

If you haven't heard or used Cursor yet, I suggest you download it from cursor.com before continuing reading this article. Once it's all setup, I'll show you all the tricks I learned so far and how it can speed up your development process.

Getting started

Cursor is a code editor that has been cloned from VSCode, it has the same layout and the same keyboard shortcuts but it has a few extra features that make it stand out, especially the AI assistant that can help you write code. Cursor is integrated with Claude Sonnet 3.5 model from Anthropic, which is a state-of-the-art model for code generation.

When opening Cursor, you will see a screen like the below, I have only created a Next.js app from scratch using the command npx create-next-app@latest

NOTE: I have never used Next.js before so this will be a fun experiment to see how Cursor can help me with that even if I don't have any prior knowledge.

On clicking on a file, you'll get the text CTRL + L to chat, CTRL + K to generate, these are the most important keyboard shortcuts to know, let's see what they do.

CTRL/CMD + L: this is to open the chat on the left hand side to talk directly to the model.
CTRL/CMD + K: this option opens a pop up where you can describe to the model what you'd like to do in the file. You can also highlight code and click on the Generate button to generate the code for the highlighted code, this will modify the existing code and give you the chance to review it and continue the conversation.

This is where it gets interesting, the pop up not only allows you to put a prompt but can also reference text from a number of sources, it can be activated by typing @
- @File: It gives a list of files in your project and you can select one to use as context for the code generation.
- @Code: It references code sections from the current file or other files in your project.
- @Web: This option constructs a search query, gets the results from the web and uses them as context for the code generation.
- @Docs: List sources to get information from, you get a list of the major frameworks and libraries but it can also add your own documentation.
- @Definitions: Adds all the nearby definitions in the source code to the context.
- @Chat: Adds messages from the current conversation to the context.
These options can be incredibly useful when you want to provide additional context or specific code snippets to the AI assistant for more accurate and relevant code generation. There are a few more options that can also be used in the chat area, see the whole list in the @-Symbols Cursor documentation.

Chat window

The chat window is where you can talk directly to the model, the @- symbol can also be used here but it has a few more options

The additional options that I found very useful are:

@Folders: It can reference a folder and ask the model to make changes to it.
@git: It finds commits or even different branches, this could be useful to make the model aware of other incoming changes that could be developed in another branch and haven't been merged yet into the main branch.
@Codebase: I haven't been able to use this option yet but according to the documentation, it gives Cursor the ability to go through steps until it finds the most important pieces of code to use.

Composer mode

One important feature still in beta is the composer mode, it gives you the ability to create a multi-step chat with the model to generate more complex code. This feature is not configured by default so you'll need to enable in File > Prefereces > Cursor Settings > Features > Enable Composer

Composer mode can be activated by using CRL/CMD + I and it will show up a window like the below, I haven't been able to show it in a bigger window size, not sure if possible at the moment, I'd personally prefer to have it on the left hand side like the chat mode

Your browser does not support the video tag.

With the composer mode, you can use the feature we discussed before to reference other files in the project, a very useful one is to create a prompt with all the steps you'd like the LLM to follow, I've even seen a cursor.directory recently where people can contribute with their own prompts.

Creating an app with Cursor composer

This is the interesting part, I've been working on a prompt that gave me the ability to create a working app on top of Next.js that was generated using the npx create-next-app@latest. You can find the the repo and more details on https://github.com/jstoppa/coding_with_cursor

The prompt below is used to generate an app that allows the user to check any stock market and display a graph timeline with the stcok value in the last 20 years

create an app what allows you to search any stock in the market and show them in a graph timeline with the value in the last 20 years. 

make sure 
- add any new library to package.json so I can easily install them
- any new file should be inside the folder /src/app
- modify /src/app files so the page loads the stock page directly
- use the free yahoo finance API and make the API call from the server side to avoid CORS issues
- the page should work if it's opened in a system where the default setting is dark mode

And here's the video demo:

Your browser does not support the video tag.

Overall I'm very impressed with Cursor and the potential it has to change the way we develop software, I'm looking forward to seeing what new features they'll add in the future and also building new prompts to create more interesting apps.

I hope you like this article, if you want to hear more follow me on X at @juanstoppa where I regularly post about AI

Query your database with AI using LangChain and Gradio

Juan Stoppa — Sun, 16 Jun 2024 11:12:40 +0000

I remember starting in the world of Machine Learning (ML) around eight years ago, doing Andrew Ng's courses on Coursera and jumping into websites like Kaggle to apply the concepts and solve Machine Learning challenges. I didn't personally imagine that in just a few years we would reach a stage where we can chat with a model that has such vast knowledge and, even more impressively, the ability to augment its knowledge with your own data and bring some level of reasoning around it, like a proper assistant.

That's what we are going to explore in this article: the ability to interact with Large Language Models (LLMs) that can use your own database to extract data, analyse it, and display it back to you, explaining what it means. This article covers the basics of how to set it up. I will covert in more details what is possible in future articles.

In this article we will use two key tools:

LangChain: a framework for building applications using Large Language Models (LLMs).
Gradio: a Python library for creating easy-to-use web interfaces for machine learning models.

If you want to go straight to the final solution, simply navigate to the Github repo that contains a working solution for using LangChain to communicate with an SQLite DB and rendering a Gradio UI interface.

Alternatively, keep reading to learn how it's built and how it can be configured for different use cases.

LangChain and SQL

With LangChain, you can pretty much make the application to reason as they describe it in their website and they have built an interface that allows you to connect to a SQL databases.

We'll use the well known sample database called Northwind created by Microsoft, it represents the sales data for a fictitious company called "Northwind Trades". I'm using the SQLLite3 version to make things simpler and avoid the need to install a SQL Server, I found the SQLLite version in this Github repo.

To start with, we need to create the llm model, LangChain support a long list of models, for this use case we need one that supports streaming since we want the results to be streamed to the Gradio UI. For this article, we'll use the OpenAI LLM (gpt-3.5-turbo API) but you can check the full list of models supported by LangChain in here.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0, api_key=os.environ.get("OPENAI_API_KEY"), streaming=True)

NOTE: You can get more details on how to setup the OPENAI_API_KEY in this article

We now need to connect to the DB and create the chain.

from langchain_community.utilities import SQLDatabase
from langchain.chains import create_sql_query_chain
from langchain_community.tools.sql_database.tool import QuerySQLDataBaseTool

db = SQLDatabase.from_uri("sqlite:///northwind.db")

execute_query = QuerySQLDataBaseTool(db=db)
write_query = create_sql_query_chain(llm, db)
chain = create_sql_query_chain(llm, db)
chain = write_query | execute_query

This is what each item above does:

execute_query: initialises the QuerySQLDataBaseTool which is designed to execute SQL queries on a database. It acts as an intermediary that takes SQL queries, runs them against the connected database and returns the results.
write_query: creates a query chain using a language model (llm) and the database connection (db). The create_sql_query_chain function sets up a pipeline to generate SQL queries.

The chain then combines the write_query chain and the execute_query tool into a single pipeline. The | operator is used to denote a sequence where the output of write_query is passed to execute_query.

The next step is to create a Prompt Template and create the final chain to combine all together.

from langchain_core.prompts import PromptTemplate
from langchain_core.output_parsers import StrOutputParser
from operator import itemgetter

answer_prompt = PromptTemplate.from_template(
    """Given the following user question, corresponding SQL query, and SQL result, answer the user question.
Question: {question}
SQL Query: {query}
SQL Result: {result}
Answer: """)

answer = answer_prompt | llm | StrOutputParser()
chain = (RunnablePassthrough.assign(query=write_query).assign(result=itemgetter("query") | execute_query) | answer )

And this is the explanation:

answer_prompt: creates a template for generating the prompt that will be fed to the language model. The template includes placeholders for the user question, the SQL query and the SQL result.
answer: sets up a pipeline where answer_prompt generates a prompt based on the template, llm processes the prompt to generate an answer and StrOutputParser parses the output from the language model.
chain: sets up a complete chain of operations using RunnablePassthrough to pass inputs unchanged, assign the query and return the results.

In summary the code above:

Generate an SQL query from a user question.
Execute the SQL query on the database.
Use the results of the query to generate a natural language answer to the user question.

Plugin the chain into Gradio

We'll now plugin the chain created above to the Gradio UI, this will allow the user to have a user interface to interact with the model which will translate into SQL queries, retrieve the information and return the details to the user.

We'll first create a stream_response function that will be used for the Gradio interface

from langchain.schema import AIMessage, HumanMessage

def stream_response(input_text, history):
    history = history or []
    history_langchain_format = []
    for human, ai in history:
        history_langchain_format.append(HumanMessage(content=human))
        history_langchain_format.append(AIMessage(content=ai))

    if input_text is not None:
        history_langchain_format.append(HumanMessage(content=input_text))
        partial_message = ""
        for response in chain.stream({"question": input_text}):
            if (response != ""):
                partial_message += response

            yield partial_message

The code above defines a function called stream_response that streams responses from a chain. The function handles both the input text and the conversation history. Worth noting the HumanMessage and AIMessage used to identify where the responses are coming from.

In summary, the code above does the following:

Converts the conversation history into the LangChain message format.
Adds the new input text to the history.
Streams responses from the AI, yielding partial responses as they are generated which allows for real time or incremental updates to the user.

And finally, we put it all together into a Gradio Chat Interface

import gradio as gr

iface = gr.ChatInterface( stream_response, textbox=gr.Textbox(placeholder="Ask a question about the database...", container=False, scale=7),)

iface.launch(share=True)

When the entire code is executed, a new web interface will be triggered (normally in http://127.0.0.1:7860) where you can start interacting with the database

Here's a demo video with a user asking questions, note that I have misspelled words like London in some prompts in purpose however the models understood the mistake and executed the queries correctly against the database.

Your browser does not support the video tag.

I hope you like this tutorial, if you want to hear more follow me on X at @juanstoppa where I regularly write about AI.

Setting up Fast API in IIS and run APIs in Python

Juan Stoppa — Mon, 27 May 2024 22:16:26 +0000

Many clients I meet struggle to implement AI within their corporate infrastructure. They often become frustrated because their companies are mostly on-premises and use Windows, while the AI industry largely operates in the cloud. Although they can call APIs like ChatGPT and Azure OpenAI, they face problems when attempting to integrate more advanced AI features into solutions primarily written in Java and .NET.

This is where I see Python playing a crucial role. The AI industry predominantly uses it to develop solutions, making it easier to get up to speed with any AI example. The real question is whether you can integrate Python into your existing tech stack if it's already based on Windows.

This article will show you how to setup an API written in Python using an amazing framework called FastAPI. This article is an introduction on how to use the framework, I blog later on more advanced use cases.

Setting up the app example in IIS

Follow the steps below to start with

NOTE: you'll need to have Python and PIP installed for this, follow these steps if you haven't installed it already

Create the website in IIS as shown in the image, note I'm using the port 8080
Copy the code example below into a main.py code (the example was taken from the Fast API website)

from typing import Union
from fastapi import FastAPI

app = FastAPI()

@app.get("/")
def read_root():
    return {"Hello": "World"}

@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
    return {"item_id": item_id, "q": q}

Install Fast API using the pip command pip install fastapi

Configuring the HttpPlatformHandler

The next step is to get the HttpPlatformHandler working in IIS, this is a handler that passes socket connections directly to the Python process (more info here).

Create a web.config file inside the website folder with the following content, make sure you change the folder to point at the location where python is installed in your local machine

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <system.webServer>
        <handlers accessPolicy="Read, Execute, Script">
            <add name="FastAPIHttpPlatformHandler" path="*" verb="*" modules="httpPlatformHandler"
                resourceType="Unspecified" />
        </handlers>
        <httpPlatform processPath="C:\Users\<username>\AppData\Local\Programs\Python\Python312\python.exe"
            arguments="-m uvicorn --port %HTTP_PLATFORM_PORT% main:app"
            stdoutLogEnabled="true" stdoutLogFile="C:\logs\python.log" startupTimeLimit="120" requestTimeout="00:05:00">
        </httpPlatform>
        <httpErrors errorMode="Detailed" />
    </system.webServer>
</configuration>

Navigate to the http://localhost:8080/ which should load the API already

Fast API also comes with swagger already installed, you can simply navigate to http://localhost:8080/docs

Other considerations

You might find some recommendations to use hypercorn rather than uvicorn, I tried that initially but I couldn't resolve an issue with socket permission, I raised it as a bug in their GitHub repo.

You might also find issues where the API loads and hangs forever. In that case, you'll most likely need to grant IIS_IUSRS read access to the folder where the app and Python are located, as mentioned in here.

DEMO - Voice to PDF - Complete PDF documents with voice commands using the Claude 3 Opus API

Juan Stoppa — Sat, 27 Apr 2024 23:30:34 +0000

I spent some time last weekend exploring the Claude 3 Opus API from Anthropic since I have heard so many comments about its potential which appears to surpass ChatGPT, especially when tasked with resolving complex problems such as writing code.

As I was looking into its capabilities, I decided to build an app that allows you to complete a PDF form with voice commands which ended up working much better than I expected.

The idea of the app was to:

Upload a PDF form.
Record voice commands to fill in the form.
Download the completed PDF form.

You can see the demo below:

Voice to PDF app using Claude 3 Opus API

You can find the app on github at https://github.com/jstoppa/voice_to_pdf

How did I build it?

I spent more time getting the functionality working than building the prompting :-), I used plain JavaScript and NodeJS as I like to keep these demos as plain as possible so I can then pick up the code and use it in other frameworks without having to rely on framework-specific nuances.

The Frontend app:

It runs using Parcel, very simple and easy to setup. The app has 3 files:

app.js: main app that brings the entire solution together.
dragDrop.js: basic primitives to handle the drag and drop file functionality.
pdfHandler.js: this file contains the logic for two fundamental actions:
- readPdf: used for reading the dropped file and displaying it on the screen, it uses PDF.js to load the file, get all fields and display it on the browser.
- writePdf: used for writing the final completed PDF after receiving the response from the Claude 3 Opus API, this uses the PDF-lib library to manipulate and modify the final file.
speechRecognition.js: this uses the not well-known Web Speech API that comes with the browser and works impressively well. The file handles the action of listening to the voice and displaying the text on the screen.

The Backend app:

It's nothing impressive, just a single NodeJS end point (server.js) to proxy the API call to Claude Opus API, it's mainly used for keeping the API key in the server side and handle the CORS constrains dictated by the browser.

The most important bit it's really the prompt which consists in 3 parts:

The task definition: this goes inside the system parameter when calling the Claude Opus API (read more here).The prompt is the following

You are tasked with assisting in the completion of a PDF questionnaire using a provided JSON dataset.

The JSON data includes the following fields for each question in the form:
- id
- question
- isValidQuestion
- answer

Your specific duties include:

1. Question validation: Form the data for Processing you need to
    a. Analyse the "question" field
    b. Determine if the question is valid based on the "isValidQuestion" field
    c. If the question is valid, incorporate the corresponding answer provided in the answer field using the data provided by the user role

2. Strict Adherence to Data: Under no circumstances should you alter, rephrase, or modify any of the the question or id field, your main task is to populate the isValidQuestion and answer fields

3. Format Requirement: Return the results strictly in JSON format. Ensure that the output contains only the required information, maintaining the integrity and structure of the original JSON, including the id fields.\n

4. Valid Questions: Only return the questions that contain a valid question based on the "isValidQuestion" field but still conserving the original id

Important Note: Do not add extraneous text or information outside of the specified JSON structure.

List of fields in the PDF: this is a JSON structure with the list of fields in the PDF form, this is generated dynamically based on the document uploaded. As the previous text describes, the task for Claude is to
- Check if the question is valid by setting the value isValidQuestion to true or false
- Answer the question using the context given on the user role.

[
    {
        "id": 0,
        "question": "First name",
        "isValidQuestion": true,
        "answer": "John"
    },
    {
        "id": 1,
        "question": "Last name",
        "isValidQuestion": true,
        "answer": "Doe"
    }
]

User Role: This describes the context which is the text generated by the speech to text api. The prompt is defined as below where contextualText variable is the text.

{
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "This is the contextual text that you need to use to complete the questionnaire\n\n ${contextualText}"
                }
            ]
        }
    ]
}

And that's pretty much it. You can see how, with a very simple app, you can perform a reasonably advanced task which was unimaginable to achieve just a few years ago.

If you like this post, you might also like Exploring the GPT-4 with Vision API using Images and Videos. And if you are completely new to working with AI, I suggest you check Getting started with OpenAI using Python in Windows and Getting started with Azure OpenAI

Getting started with Azure OpenAI

Juan Stoppa — Sun, 14 Apr 2024 02:15:39 +0000

I've been exploring ways to use AI in my day-to-day job. I personally use Azure and want to leverage as much as possible the partnership they have with OpenAI.

What you will learn in this article is how to use your own data to create a ChatGPT-like agent using Azure OpenAI API.

What I'm using in this post includes:

Azure Open AI studio: a platform where Microsoft teamed up with OpenAI to give you access to their API directly from the Azure infrastructure. It's designed for developers and businesses that want to integrate AI into their apps or services, making them smarter and more intuitive.
Azure AI Search: a service designed to retrieve information across user-generated content, both in traditional search scenarios and in applications powered by generative AI. Whether you're developing applications that require conventional search functionalities or generative AI capabilities, this service delivers that for you.
Azure APP service: a service that allows you to quickly build, deploy and scale web apps and APIs. Work with most programming languages and runs on Windows or Linux.

Signing up for Open AI Studio

First things first, you'll need to create a Microsoft account and sign up for Azure Open AI Studio, best way if you don't have one already is to navigate to https://oai.azure.com/ and create a new account

At the time of writing this post, the Azure Open AI studio was in preview mode, if this is still valid you need to request access by submitting this form in customer voice https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUNTZBNzRKNlVQSFhZMU9aV09EVzYxWFdORCQlQCN0PWcu

Using the Open AI studio

Once the account is created, you can access the studio by navigating to the URL https://oai.azure.com/, you'll be presented with some options on how to get started

In the interest of this post, we'll be using the Chat functionality in the Playground area which I define below:

Playground: this gives you the ability to test all the tools Azure Open AI Studio has, chat functionality, completions, DALL-E for generating images and assistants which are in preview mode at the time of creating this post.
Prompts: this area is where you define your prompts, you can start them from templates or create your own prompts from scratch.
Test the assistant: you can directly test your assistant in the playground by writing questions and interacting with it.

To test its capability, I will use the Microsoft Annual Report from 2023, I have a version in PDF format. For this I will use the Azure AI Search capability from Microsoft that allows us to store the document and get the agent to search information in it.

NOTE: To add your data in the steps below, you'll first need to create an Azure AI Search service, follow the steps in this Microsoft article.

NOTE 2: It's also important that you create the resource in a region where the Open AI service is available

Follow the steps below to add your data

Click on "Add your data"
Followed by "Add a data source"

This will take you to a wizard where you need to complete the following:

Select data source: this should be set to "Upload files"
Subscription where the Azure AI Search has been created
Select Azure Blob storage resource: a blob storage where the file can be uploaded
Select Azure AI Search resource: selectable from the list, it's the service mentioned in the NOTE above
Enter Index name: index to be created to search the data

Add the file in the next page in the wizard and click on Upload files

The next step is to select the type of search (Keyword) and then the chunk size which is 1024 by default

Once finished, the ingestion progress will start which should take a few minutes

Querying the data

You can now ask any question to the agent and it should come with an answer about the document, for instance I have asked to give a summary of Microsoft 2023 Annual report and returned the following

When answering the question, the chat result will show links to the sources which can be opened on the right hand side.

Another interesting feature is the ability to View the Code, this can be helpful to understand how the chat works and how the context is passed back and forth which enriches the conversation with the LLM.

Deploying the app

With the Azure AI Studio is quite easy to deploy the APP in azure, you simply need to click on "Deploy to" > "A new web app...", you can either create a new app or select an existing.

Once deployed, a link will show in the notification which will redirect you to the app

It's also possible to customise the app to your needs using environments variables in Azure, you can check the possible options in this article, this should allow you to change the logo and some of the wording in the page

And that's all, you should have the app up and running now!

If you like the article, please follow me on X at @juanstoppa where a regularly post about AI.

Exploring the GPT-4 with Vision API using Images and Videos

Juan Stoppa — Thu, 01 Feb 2024 00:33:50 +0000

I've been exploring the GPT-4 with Vision API and I have been blown away by what it is capable of. As OpenAI describes it, ChatGPT can now see, hear, and speak. But how effective is the API? In this article, I'm doing a deep dive into the GPT-4 with Vision API describing the technical details to use it.

What is GPT-4 with Vision API to start with?

GPT-4 with Vision (also called GPT-V) is an advanced large multimodal model (LMM) created by OpenAI, capable of interpreting images and offering textual answers to queries related to these images. This model blends the capabilities of visual perception with the natural language processing. There are several use cases for the Vision API, from recognising objects to translating mockups into real software or generating real time descriptions on images and videos.

The Basic - How to use the API?

The API is used similarly to the standard completion API, I recommend using Python for integration with the API, as it's the best-supported language in Open AI's documentation. If you are new to this - and perhaps using Windows like me - you might find my article Getting started with OpenAI in Python helpful where I explain all the details on how to set up Python and interact with OpenAI.

Once your Python environment is setup, the following code can connect to OpenAI and answer a simple question:
What can you see in the image?

from openai import OpenAI
import os

# create OpenAI client
client = OpenAI(
    api_key = os.environ.get("OPENAI_API_KEY"),
)

# call the API with prompt "What can you see in the image?"
response = client.chat.completions.create(
  model="gpt-4-vision-preview",
  messages=[
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What can you see in the image?"},
        {
          "type": "image_url",
          "image_url": {
            "url": "https://jstoppa.com/posts/exploring_the_GPT_with_Vision_api_image_and_video/GPTVision_example.png",
          },
        },
      ],
    }
  ],
  max_tokens=300,
)

print(response.choices[0])

here's the image used in the example:

and the result from the GPT with Vision API...

The image shows a part of a building that looks to be residential in nature. It has a slanted roof covered with dark grey slate tiles. There's a dormer with a window, which suggests an attic or loft space, and the facade includes red brickwork with stone elements that provide a decorative pattern. Below the bricks and stone there’s a ledge or overhang, possibly indicating that this section of the building is above ground level, such as on the first floor or higher. The windows are white-framed and double-glazed. Above the dormer, there are red chimney pots visible, indicating the presence of one or more fireplaces within the building. It appears to be a cloudy day, as the sky is overcast.

Very impressive results! It's able to describe the image including minimal details using very comprehensive words, it even highlights that the day is cloudy like most days in London.

Looking at more complex scenarios

I've noticed the model to start struggling when is asked more complex questions such as explaining a chart or giving the specific location of items in images. OpenAI highlights in their website that the model may not be accurate in certain scenarios, including spatial reasoning, counting, resolving CAPTCHAs or analysing medical images, last two most likely due to security concerns.
I was particularly interested in exploring how I could use the model to automate simple tasks in the computer. ChatGPT is very good at explaining tasks and meticulously outlines the steps to carry those tasks, but executing them is well beyond the model's and vision's capabilities.

My first attempt to test the above involved creating a Python script that captures a screenshot of my desktop and identifies Google Chrome on the screen. When I asked the model to provide the exact coordinate of the Google Chrome icon, it would always return very vague answers. In these cases, I realised that the model needs some hint, I tried with an overlaid grid with references

and prompt the following:

Using the alphanumeric grid overlaid on the image, identify the quadrant where the Google Chrome Icon is located. Please respond with only the quadrant reference (e.g., 'G3') if the icon is found, or 'not found' if the icon is not in the image. The output will be used for automated processing, so ensure the response is in the correct format.

This approach was still not great, GPT would correctly guess the quadrant about 50% of the time. To improve accuracy, I gave GPT a second chance: I zoomed in around the area of the first quadrant chosen and requested GPT to find the quadrant again.

this time GPT would get it right most of the times which gives a much more accurate result.

The final result was great, with that info I'd then drive the mouse to locate the icon and double click on Google Chrome. In the example below, I've run the script and GPT handled the rest, the task was to find Italian restaurants in London.

Your browser does not support the video tag.

What about Videos?

The GPT with Vision API doesn't provide the ability to upload a video but it's capable of processing image frames and understand them as a whole. A good example could involve streaming video from a computer's camera and asking GPT to explain what it can see. I've put together an example on how this can be done using Python.

We first need to switch on the camera and take some frames, for this we can use the great OpenCV library to take frames from the video and send them to Chat GPT API.

The code below will capture 200 frames at an interval of 100ms and store them in an array

import cv2, base64, time

# Initiate the video capturing object
cap = cv2.VideoCapture(0)

# Set an array to store the frames
base64Frames = []

for i in range(200):
    # Capture the frame
    ret, frame = cap.read()

    # Store the frame in the base64Frames array
    _, buffer = cv2.imencode(".jpg", frame)
    encoded_image = base64.b64encode(buffer).decode("utf-8")
    base64Frames.append(encoded_image)

    # Wait for 100ms before capturing the next frame
    time.sleep(0.01)

the array can then be sent to the GPT Vision API including a prompt to describe what it can see in the video

from openai import OpenAI
import os

# Initialise Open AI client
client = OpenAI(api_key = os.environ.get("OPENAI_API_KEY"))
# Frames from previous example 
base64Frames = []

message = [
    { "role": "user",
        "content": [ 
        "These are frames of a video. Can you describe what you can see in no more than 20 words?", 
        *map(lambda x: {"image": x, "resize": 768}, base64Frames[0::60]), ],
    },
]
parameters = { "model": "gpt-4-vision-preview", "messages": message, "max_tokens": 500 }

result = client.chat.completions.create(**parameters)
print(result.choices[0].message.content)

I took a video of a beautiful marble run I recently bought in China for my daughter, let's see what GPT Vision has to say about it...

Your browser does not support the video tag.

And the result was the following:

I can see a mechanical device made of wood and some transparent materials, possibly acrylic or glass. It has several circular elements, which look like they could be tracks or pathways for a ball to roll on. The construction also includes gear-like components, suggesting that the device has moving parts and might be operated by turning the gears, possibly as part of a kinetic installation or a marble machine. The repeated frames suggest that the device might be in motion during the video.

The model has taken the group of images and analysed them as a whole, the description gives a high level understanding of what i can see although is not entirely sure about the motions.

Combining Vision and Text to Speech

The final test I wanted to do involved testing it with the Text to speech API, rather than displaying the text on the screen it would directly say it out loud. This is also a powerful API as you could have different agents interacting with each other, that's probably an example for another day.

The Python script below simulates a conversation between "Nova" and "Onyx", two voices from the Text to speech API. The script turns on the camera and "Nova" asks "Onyx" what it can see, it iterates until the letter Q is pressed. The video is displayed in the screen and the scripts takes a frame every 100ms until it capture 200, the result is then sent to the Text to speech API to reproduce it.

from openai import OpenAI
import os, cv2, base64, requests, soundfile as sf, sounddevice as sd, io, time

def capture_and_process_frames():
    _, buffer = cv2.imencode(".jpg", frame)
    encoded_image = base64.b64encode(buffer).decode("utf-8")
    base64Frames.append(encoded_image)
    cv2.imshow('Real-time Video', frame)

def process_frames():
    print('\nGetting results from GPT Vision')
    message = [
        { "role": "user",
          "content": [ 
            "These are frames of a video. Can you describe what you can see? Start the phrase with 'I can see' followed by your answer", 
            *map(lambda x: {"image": x, "resize": 768}, base64Frames[0::60]), ],
        },
    ]
    parameters = { "model": "gpt-4-vision-preview", "messages": message, "max_tokens": 500 }

    result = client.chat.completions.create(**parameters)
    return result.choices[0].message.content

def play_audio(message: str, voice: str):
    response = requests.post(
        "https://api.openai.com/v1/audio/speech",
        headers={ "Authorization": f"Bearer { os.environ.get("OPENAI_API_KEY") }", },
        json={ "model": "tts-1-1106", "input": message, "voice": voice, "response_format": "opus" },
    )

    with io.BytesIO(response.content) as audio_io:
        data, samplerate = sf.read(audio_io, dtype='float32')
        sd.play(data, samplerate)
        sd.wait()

fourcc = cv2.VideoWriter_fourcc(*'mp4v')  # Codec for MP4
out = cv2.VideoWriter('output.mp4', fourcc, 20.0, (640, 480))  # Output file, codec, frame rate, and size

client = OpenAI(api_key = os.environ.get("OPENAI_API_KEY"))
cap = cv2.VideoCapture(0)

intro_message = """This experiment will switch on your camera, take frames, 
send them to ChatGPT Vision and produce an audio with the description.
It will continue doing it until (Q) key is pressed. Let's get started, what can you see Onyx?"""
print(intro_message)
play_audio(intro_message, "nova")
base64Frames = []
i = 0
while True:
    ret, frame = cap.read()
    out.write(frame)
    print("\r" + " " * 50, end='\r')
    print(f"Capturing frames: {i}", end='', flush=True)

    if ret:
        capture_and_process_frames()

    if i == 200: 
        out.release()
        result = process_frames()
        print(result)
        play_audio(result, "onyx")

        base64Frames = []
        i = -1
        play_audio("Let's do it again, what can you see now GPT?", "nova")
        out = cv2.VideoWriter('output.mp4', fourcc, 20.0, (640, 480))

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

    time.sleep(0.01)
    i = i + 1

cap.release()
cv2.destroyAllWindows()

I took a video to show what the script does, you can run it in your own machine to test it.

Your browser does not support the video tag.

Conclusion

This should give a good understanding of what's possible and help to dig even deeper into GPT with Vision API. With some imagination, one can bridge the gap between the model's limitations and strengths.

Some key points from this post:

The model can handle both images and videos, although a bit of image processing is required for videos.
Taking images is straightforward, it can process URLs or local images converted to Base64.
The model struggles with tasks such as counting, understanding charts or anything that involves spatial reasoning.
For spatial reasoning, it's possible to give the model hints such as highlighting areas in the image, this helps the model to produce a more accurate result.
It can process frames of a video and understand them as a whole.

If you liked this article and would like to read more, then follow me on X at @juanstoppa, where I post regularly about AI.

Open Banking - what is it and how does it work?

Juan Stoppa — Thu, 19 Nov 2020 08:25:18 +0000

Open Banking has become the latest buzzword in FinTech, it's set to revolutionise the financial industry but what is it and how does it work? This article covers all the basics and gives you some insights on how to use it.

The basics

Open Banking is a secure way to give providers access to your financial information. It all began when the European Parliament adopted the revised Payment Service Directive (PSD2) which aims at promoting innovation, competition and efficiency in payment services. This also triggered the UK to issue a ruling that required the nine-biggest UK banks to allow providers access to their data and many other banks followed suit.

Open Banking takes the same approach as OpenID took a decade or so ago when most of the major tech companies joined the initiative. This brought the ability to sign in into a service with an identity provider without having to create a new account. In practical terms, OpenID is the mechanism used for most websites nowadays where the user can sign in using an existing account such as Google, Microsoft, Auth0, etc.

This is also how Open Banking works, users can sign in into a website or mobile app by authenticating through your bank provider. But unlike OpenID, giving bank and transaction details to third party companies is something anyone would be hesitant to do, this is the reason why (at least in UK/Europe) the government regulates the providers. We are going to cover this in the next section.

The regulation

Any company or person that wants to make use of Open Banking in Europe and the UK need go through a process to become a provider.

There are three types of providers:

Third Party Provider (TPP): these are organisations or persons that can use the Open Banking APIs to access customer's accounts.
There are two subtypes of TPPs:
- Account Information Service Provider (AISP): allows the provider to access account information from different bank accounts, commonly used by budgeting apps or comparison websites.
- Payment Initiation Service Provider (PISP): this type of license allows the provider to initiate payments through a bank.
It's also possible to use Open Banking through an agent model. In this way a company or person using the a TPP service doesn't need to apply for an AISP/PISP license. This model makes sense for a company that wants to access the Open Banking APIs but doesn't have the time, funding or specialism to create an integration to every possible bank and would rather use a service that handles that complexity.
Account Servicing Payment Service Providers (ASPSP): these are banks, building societies and payment companies, they provide and maintain payments accounts for users as well as enabling third party providers to access the service through Open Banking APIs. Examples of ASPSP are the nine-biggest UK Banks and any other bank that provides Open Banking APIs.
Technical Service Providers (TSP): firms that work with regulated providers to deliver open banking products or services. These companies are not directly accessing Open Banking APIs and are not regulated by the FCA in the UK, they provide products and services to TPPs to implement Open Banking.

More information on how to become a provider in the UK can be found here. Also, the Open Banking website provides a functionality to find registered providers if you are already given your details to an existing provider and want to make sure it has a license.

How to use the Open Banking APIs?

If the previous section about regulation didn't put you off then here are good news: you don't need to complete a form application to the FCA to start exploring Open Banking APIs, there are a few options depending on what you are looking to do.

There are two options when it comes to test and use Open Banking APIs:

Use a TPP provider: this is the quickest way to start since it just needs signing up for getting access to a sandbox environment which simulates the access to bank's data. You can find providers on the Open Banking website and select the filter third party providers.
Use an ASPSP provider: this is connecting directly to a bank APIs, it's a similar process as the above but these APIs will only give access to their own customer bank information. Some banks may request to be either an AISP or be in the process of becoming one to give access to their APIs. You can find banks in the Open Banking website and select the filter account providers. There is also an ASPSP Documentation website with all providers and their respective developer portals.

Most providers will give a Postman collection to test the APIs and a client SDK to handle the authentication flow to connect to the bank.

Authentication flow and access to data

When using a TPP, the user (known as well as Payment Service User (PSU)) begins the journey by logging into an application that will contact the TPP provider to get a token. This token is then used to contact the bank selected by the user and trigger the authentication which returns an access token. This access token is what the TPP uses to exchange information with the bank on behalf of the user. The diagram below describes the authentication flow.

The application can then use the access token to retrieve the following data depending on the type of TPP:

Account and Transactions: enables an AISP to access Account information and transactions (see the API spec here).
Payment Initiation: enables a PISP to handle payments (see the API spec here).
Event Notification: enables an ASPSP to deliver event notification to a TPP (see the API spec here).

Swagger specification for all APIs can be found here.

Once a AISP application gains access to user's data, it can retain access for a period of 90 days. After this period the user will need to reconnect to the bank provider to request a new consent (find more details here). This is very important when building applications that use Open Banking, it needs to be designed in way to make this process as easy as possible.

It's also extremely important to understand that accessing this data is subject to General Data Protection Regulation (GDPR) compliance.

Conclusion

Open Banking brings a standard across the financial industry and makes the data accessible to everyone, from startups to the corporate world, this will naturally drive innovation. Security could be a major concern, however the regulated model through providers will bring transparency on who have access to the data and make it easy to identify any irregularity or fraud. End users in the other hand will benefit from having an already large number of applications to bring more transparency of their financial data and empower them to make the right decision about their finance.

Glossary

AISP - Account Information Service Provider

ASPSP - Account Servicing Payment Service Providers

GDPR - General Data Protection Regulation

PISP - Payment Initiation Service Provider

PSD2 - Revised Payment Service Directive

PSU - Payment Service User

TSP - Technical Service Providers

TPP - Third Party Provider