DEV Community: Focused

Everybody Tests

Austin Vance — Tue, 03 Feb 2026 19:43:37 +0000

Everybody Tests

Everybody tests. Every developer. Every team. The debate isn't about whether you test. It's about whether you automate it.

I hear the pushback all the time. "We don't do TDD." "We don't have time for tests." "Tests slow us down." Fine. But watch what happens when that same developer finishes a feature. They open the browser. They click around. They fill out a form. They check the button. They verify the data.

That's a test.

When QA maintains a spreadsheet of scenarios to run before each release... that's a test suite. When you demo to stakeholders and walk through the happy path... that's acceptance testing. When you "just quickly check" that your refactor didn't break anything... that's regression testing.

The tests exist. They always have. They're just manual. Trapped in someone's head.

You already think test-first

Most developers already think test-first. They just don't write it down.

Before you write a function, what do you do? You think about what it should do. You imagine calling it with some input. You picture what comes out. You might even sketch a few edge cases in your head, on a post-it, on a whiteboard. What if the list is empty? What if the user isn't authenticated?

That's the test. You've written it. It exists. It's just in your brain instead of in code.

Then you implement. Then you verify it does what you imagined.

Congratulations, you already did the hardest part of test-first development.

The cost nobody talks about

Manual testing isn't free. It's a subscription.

Every time you click through your app to verify a change, you're paying. Every time someone else on your team does the same verification, that's another payment. Before every release. After every refactor. When something breaks in production. When a new hire needs to understand what the system does.

That 30-second click-through? You'll do it a hundred times. So will everyone else. Multiply by every feature, every edge case, every browser.

The math is brutal. An automated test takes minutes to write and milliseconds to run. A manual test takes seconds to run, but you'll run it thousands of times.

The teams that say they don't have time

I've seen teams who are adamant they "don't do TDD" spend entire days before a release running through test scripts. They hire QA teams whose job, their actual full-time job, is clicking the same buttons over and over.

And then they tell me they don't have time to write tests.

What they mean is this: they don't have time to write the test once. But they have infinite time to run it manually forever.

The same satisfying satisfying, all over again

We're about to make the exact same mistake with AI.

Watch what happens when someone builds an agent. They tweak a prompt. They run it. They look at the output. They squint. They decide if it's good enough. They tweak again. They run it again. They squint harder.

That's an eval. That's literally an eval. You're evaluating the output of your system against some criteria in your head.

Everybody evals. The question is whether you automate it.

Eval Driven Development

The parallel to TDD is exact. Before you write an agent, you think about what it should do. You imagine the inputs. You picture what a good output looks like. You might even think through some edge cases, what if the context is empty? What if the user asks something adversarial?

That's the eval. You've already written it. It's in your brain. Write it down.

Define what good looks like before you start prompting. Capture the examples. Capture the edge cases. Automate the judgment, whether that's exact match, semantic similarity, or LLM-as-judge.

Then iterate. Change the prompt, run the evals, see what breaks. Change the model, run the evals, see what improves. Swap in a new retriever, run the evals, know immediately if you're moving forward or backward.

This is Eval Driven Development. It's just TDD for the age of AI.

Vibes don't ship

With traditional software, at least the output is deterministic. You click the button, you get the same result.

With agents? The output is stochastic. You run the same prompt twice, you get different results. Manual review is useful for exploration. It’s useless as a measurement system. You can't trust your own squinting because you're sampling a distribution. That one good output you saw could happen 90% of the time. Or maybe 20% of the time. You have no idea.

This is why the teams building AI systems have eval suites, they have regression sets, they run benchmarks on every commit. They know their success rates because they measure them.

The teams that don't? They're shipping on vibes. And I've seen where that ends. It ends with an agent in production that works great in the demo and fails catastrophically on real traffic. It ends with prompt changes that "seemed fine" but tanked accuracy by 15%. It ends with executives asking why the AI feature is so unreliable, and engineers shrugging because they never actually knew how reliable it was in the first place.

The choice is the same as it always was

Twenty years ago, the industry had to learn that manual QA doesn't scale. That you can't hire enough people to click through every scenario before every release. That automated tests aren't a luxury.

Some teams learned that lesson. Some are still running through spreadsheets the night before launch.

Now we get to make the same choice again. You can build agents by squinting at outputs and hoping for the best. Or you can write down what good looks like, automate the evaluation, and actually know whether your system works.

Everybody tests. Everybody evals. The only question is whether you're going to keep doing it by hand, or whether you're finally going to let the computer do its job.

If your team is building agents without evals—or still stuck in manual testing on the traditional side—drop me a note or check out focused.io. We've helped teams escape the spreadsheet and the squint.

Context Will Replace Your Design

Austin Vance — Tue, 27 Jan 2026 03:34:12 +0000

There's a particular irony in typing a Slack DM to an AI assistant that will help me write about why interfaces don't matter anymore... There’s just words in a box (or in this case,a series of manic voice notes), and a system that understands what I need.

I've been running Clawdbot, a persistent AI assistant, for a few days (yeah only days) now. It reads my email, manages my calendar, browses the web on my behalf, and remembers conversations across sessions. It does everything my current VA does. It's changed how I think about software design. Clawd has no interface, no web page for interaction, nothing... and the absence of a traditional interface hasn't diminished the experience. If anything, it's clarified something we've been dancing around since agents have started to take shape: most interface design exists to compensate for computers not understanding what users want.

The Translation Layer We Built

For fifty years, interface design has solved a specific problem: we humans think in fuzzy concepts and intentions, while computers operate on instructions and data. Going from "I want to book a flight to Chicago next Tuesday" to the seventeen form fields, dropdowns, and confirmation screens required to book a flight is what interface designers do.

There are disciplines around this translation work. Information architecture. Visual hierarchy. Affordances and signifiers. Progressive disclosure. All this design exists to help people navigate the rigid structures that computers require. Click here, not there. Fill this field before that one. Read the error message, understand what went wrong, try again.

Design makes speaking “computer” effortless, drawing on the premise that humans adapt to the system.

When the System Adapts to You

Over the weekend, I sent a voice memo to my assistant while skiing that says, "draft a 2,000-word essay about the future of design in a post-agentic world, use Clawdbot as a case study, make it sound like my writing style," and... it did a pretty mediocre job on the essay… but the interaction was perfect. I used WhatsApp and Slack, used voice notes while driving and skiing, it read my other articles on focused.io and austinbv.com. It blew me away. It's a new paradigm for design.

The agent, clawdbot, did all the input work for me. It didn't need maliciously designed input fields, instead the LLM can parse my natural language, and it doesn’t want visual hierarchy to guide attention because it just asks clarifying questions. There’s no more progressive disclosure because it can handle complexity directly.

So, what does design become when the translation layer is the system itself?

The Clawdbot Case Study

My current setup involves an AI assistant (clawdbot) that:

Lives in WhatsApp and Slack (messaging interfaces I already use)
Maintains persistent memory across conversations
Has access to my email, calendar, and files
Can browse the web and interact with services
Runs scheduled tasks and proactive check-ins

The "interface" is the chat apps, but the beauty is the tools. Clawdbot connects to other APIs and handles tasks that would require dedicated applications and logins and passwords and and 2fa and have websites with carefully designed interfaces: email clients, calendar apps, travel booking sites, task managers, note-taking tools.

It feels like noise now that I think about it... Designers in an agentic world should be thinking as the Agent, as the user, not a person. You can then frame it as:

Context structures - What information does the AI need to understand my preferences, constraints, and goals?
Memory systems - How does the assistant maintain continuity across sessions without drowning in irrelevant history?
Tool interfaces - How do we expose capabilities (email, calendar, web browsing) in ways that AI can reliably invoke?
Failure modes - What happens when the AI misunderstands? How do we make correction natural?

The design becomes decoupled from the interface and more about “How it works” rather than “how it looks.” Isn’t that what designers wanted all along?

Context Engineering: The New Discipline

There’s a new term for what's emerging: context engineering. It's the discipline of designing the information environments in which AI agents operate.

Traditional interface design asks: "How do we present information so humans can find and act on it?"

Context engineering asks: "How do we structure information so AI can understand and act on it appropriately?"

Sound similar? They are, but lead to radically different practices. Visual hierarchy doesn't matter to an AI, it can process information in any order. Color and iconography don't matter. Animation and micro-interactions don't matter.

What matters:

Information completeness - Does the AI have everything it needs to make good decisions?
Semantic clarity - Is the information structured in ways the AI can reliably parse?
Constraint specification - Are the boundaries of acceptable action clearly defined?
Context efficiency - Can we convey what's needed without wasting tokens (and money, and latency)?

That last point deserves emphasis. In the current paradigm, loading a webpage or json or whatever costs the user attention and costs the provider bandwidth, both relatively cheap. In an agentic paradigm, loading context costs tokens, which cost real money at scale, add latency that compounds, and confuse agents across multi-step tasks.

The entire economics of information presentation flips. Dense, structured, machine-readable information beats sparse, visual, human-readable information.

What Agents Need

When I watch Clawdbot browse the web on my behalf, it’s painful. It loads pages designed for humans, rich with images, navigation, whitespace, branding, and extracts the tiny fraction of information it needs. A flight search that a human might complete in five minutes of visual scanning takes the AI multiple page loads, screenshots, and navigation of UIs designed to be intuitive for people who look rather than read.

This is obviously transitional. As agents become more prevalent, services will increasingly expose information in agent-friendly formats. I don’t think this is traditional APIs (they are too noisy), but something more flexible, semantic interfaces that can adapt to varied requests while remaining token-efficient.

And a new design challenge emerges, going from "how do we make this look good to humans" to "how do we make this legible to agents while remaining interpretable by humans for oversight?" Yup, with oversight.

It’s harder than it sounds. Pure API endpoints are good for agents but opaque to human oversight. Visual interfaces are easy for humans to monitor but wasteful for agents. The emerging discipline will be the middle ground: interfaces that are simultaneously agent-efficient and human-auditable. Until the agent just gets what it needs and builds the interface on the fly.

The Skills That Transfer

If you're a designer reading this with anxiety about obsolescence. I think your core skills transfer, building APIs had users, the systems were users, the developers were users. Now the Agents are users.

Understanding user intent remains crucial. You're solving for what people want, even when the interface between the user and the system is an agent.

Information architecture evolves. Structuring information so an agent can navigate it requires the same thinking about relationships, hierarchies, and access patterns.

Systems thinking becomes more important. Agent-based systems involve more moving pieces, more failure modes, more need for coherent components.

What won’t transfer is the visual craft, the micro-interactions, the animation choreography.

Design Becomes Invisible

There's a Marshall McLuhan quality to what's happening. The interfaces that are everywhere in our visual landscape, the screens, buttons, forms, and menus, they aren’t the point. They are necessary infrastructure for the pre-AI era when computers couldn't understand intent and humans had to translate, or the interface had to force it.

As that scaffolding comes down, design becomes the structure of context and the shape of information. It’s the logic of systems that we interact with but no longer see.

So, the best interface is no interface, as the cliché goes. And we're about to find out if we meant it.

My AI assistant doesn't have a color scheme. It doesn't have a logo animation or a distinctive illustrative style. It lives in generic chat windows with access to multiple platforms. And it's the most thoughtfully designed software I’ve interacted with, because the design is in what it knows, how it reasons, and how it adapts to what I need.

That's the future of design in an agentic world.

If you like this and are thinking about design and agents check out Focused we build AI agents that actually work. We're a LangChain consulting partner and help teams move from prototypes to production... context engineering included.

3 Things I’ve Learned While Implementing My First Multi Agent Architecture

Brittany Fojtik — Fri, 01 Aug 2025 15:15:00 +0000

If you’re anywhere near the AI application development world, whether you’re an engineer, a product manager or a CEO, I’m sure that you’ve heard about multi-agent systems. But why? Why is there so much hype around multi agent systems? Are they really that good? If they are, what are they good at? And do I need it in my business or app?

In my case, it was all about scalability. Let me explain. I’ve been working for the last couple of months in a chatbot based application, and even though it’s been getting better with regular prompt engineering + evals, we still had an issue: prompts were getting bigger and bigger, and work on further improving the prompt was returning less benefit over time. Also, we wanted the chatbot to get better at certain specific things, which meant adding much more prompt to the already long existing one.

So, we decided to start looking into multi-agent architectures, with the hope that it would help us build agents that are really good at one thing, without having astronomically long prompts. After implementing our first architecture with this approach, I want to share some learnings with you.

Learning #1: names (swarm, supervisor, etc) are just names. Build whatever you want!

The two most popular multi-agent architectures you will hear about are: Supervisor and Swarm.

In the first one, you have one agent (the supervisor) that takes care of taking the user’s input, and then deciding whether to respond directly to the user, or to hand off the conversation to a subagent. Once the subagent is done (this can mean generate a response or do things like search the web), the conversation goes back to the supervisor which generates the final response.

On the other hand, swarm architectures consist of a group of independent agents that are aware of each other, and work together as a whole. When a user sends a message, it goes to the default agent, and this agent will decide if it should respond or handoff to a different agent. If it hands it off to a different agent, that agent will generate a response and send it to the user. Next time a message comes, it will go to the latest active agent, and the process begins again.

So: supervisor or swarm? Which is right for me? The answer is: why not both?

At the end of the day names don't matter. What matters is developing an application that suits your use case and adds business value. In my case, I needed output that was both consistent and checked before reaching the user, which pointed toward using a supervisor. But I also needed the application to be fast, which is where a swarm shines, since multiple agents can run in parallel with minimal latency.

To satisfy both requirements, I built a swarm of two regular agents managed by a supervisor agent. The supervisor coordinates the agents and ensures a final quality check without adding significant delay. This setup gives me fast parallel responses while still providing a reliable review step before the output reaches the user.

Learning #2: evals are key, especially in multi-agent architectures.

When you have an app with only one LLM that’s generating responses and you make changes to the prompt and want to see if the responses improved, you can simply chat with your application and see how it performs. This works, but isn’t scalable.

As your model evolves, you’ll want it to pick up new skills without forgetting the old ones. Early on, you might be able to manually track whether it’s doing the right thing, but that doesn’t scale. Over time, it becomes harder to remember what “good performance” even looked like. That’s where evals come in: they’re an automated way to measure and monitor how well your model is doing across both new and existing capabilities.

This is even more true when you start adding more than one agent to the equation. You start talking to the chatbot and all of a sudden you don’t even know which agent is generating the response you’re seeing!

Evals allow you to:

Measure the overall system performance and behavior

Test and improve each agent in an isolated way

Fine-tune the handoffs, ensuring each agent is invoked when they should be or not

Be able to iterate quickly with confidence

Learning #3: Let your use case drive the architecture—not the other way around

This is similar to what I mentioned in Learning #1, and honestly, it’s not a new challenge in software. It’s been happening forever. What’s different with AI is the sheer speed of change. Things are moving so fast that it often feels like we’re constantly scrambling to keep up with the latest development.

It’s easy to get swept up in the hype of new tools, frameworks, or architectures. We’ve all been there. I’ve been there too, which is exactly why I decided to write this post.

If I could go back just three weeks and give myself some advice, it would be this: Before implementing anything new, take the time to deeply understand what you’re actually trying to achieve. What’s the problem? Why does it matter? How will solving it improve the system?

Once you have that clarity, you can start exploring your options. But now you’ll have a clear goal to guide you, which makes it much easier to filter out distractions and focus only on what fits your needs.

Conclusion

AI development is moving pretty quickly, and it does feel like we’re trying to surf a tsunami. The whole industry is experimenting and learning about these technologies, no one has it all figured out.

In my experience, it’s best to focus on fundamentals:

Understand your use case deeply

Evals, evals, evals

Don’t get hung up on labels—build what fits your problem and adds the most business value

When you focus on the building blocks instead of the hype, you learn faster, adapt quicker, and end up with a system that actually works for your product.

Curious how we design multi-agent systems that actually hold up in production?
Learn more at https://focused.io/langchain

Customizing Memory in LangGraph Agents for Better Conversations

Austin Vance — Wed, 16 Jul 2025 17:24:47 +0000

Right now everyone is building conversational agents and having them remember past interactions is crucial for creating natural, engaging user experiences. LangChain, a powerful framework for developing LLM-based applications, has evolved its memory management. Recently, in v0.3.x they deprecated indivdual memory management classes, the recommended approach for memory in agents is to use LangGraph persistence. This tutorial dives into customizing memory using LangGraph, addressing common challenges like maintaining persistent chat history and optimizing for better conversations. Whether you're building chatbots or intelligent assistants, mastering LangGraph memory will enhance your agent's intelligence and make the UX feel more seamless across interactions.

Overview of Memory Types in LangChain

As of LangChain v0.3.1, several legacy memory types have been deprecated in favor of more robust persistence via LangGraph. Here's a quick overview of the deprecated types and the migration path:

ConversationBufferMemory: Deprecated. Previously stored entire conversation history.
ConversationBufferWindowMemory: Deprecated. Limited to recent messages.
ConversationSummaryMemory: Deprecated. Summarized interactions.
ConversationEntityMemory: Deprecated. Extracted and stored entities.

All of these have been replaced by LangGraph's checkpointing system, which provides built-in persistence, support for multiple threads, and advanced features like time travel. LangGraph uses checkpointers (e.g., InMemorySaver for in-memory, SqliteSaver for persistent storage) to manage state across conversations.

For more on the migration, refer to the official migration guide.

In this langchain memory tutorial, we'll start with simple setups using LangGraph and progress to custom persistent implementations.

Setting Up Simple Memory Buffers

Let's begin by setting up a basic conversational agent with memory using LangGraph and InMemorySaver. This provides simple, in-memory persistence across interactions within the same thread.

First, ensure you have the necessary dependencies installed. We'll use LangChain version 0.3.26 (the latest as of July 2025), LangGraph, and OpenAI for the LLM.

uv add langchain langchain-openai langgraph langchain-community --bounds lower

Now, here's a Python example to create an agent with simple memory:

from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.memory import MemorySaver

# Initialize LLM (replace with your API key)
llm = ChatOpenAI(model="gpt-4o", temperature=0)

# Define a simple tool (e.g., for math calculations)
@tool
def multiply(a: int, b: int) -> int:
   """Multiplies two numbers."""
   return a * b

tools = [multiply]

# Set up memory with MemorySaver
memory = MemorySaver()

# Create the agent
agent_executor = create_react_agent(llm, tools, checkpointer=memory)

# Interact with the agent using a thread_id for memory
config = {"configurable": {"thread_id": "chat1"}}

response1 = agent_executor.invoke({"messages": [{"role": "user", "content": "What is 3 times 4?"}]}, config)
print(response1["messages"][-1].content)  # Output: 12

response2 = agent_executor.invoke({"messages": [{"role": "user", "content": "What was the result of the previous calculation?"}]}, config)
print(response2["messages"][-1].content)  # Output: The previous calculation result

This setup uses MemorySaver to store and recall chat history within the same thread, making your langchain agents more context-aware.

Custom Memory Implementation

For more advanced scenarios, such as persistent storage across restarts, use a persistent checkpointer like SqliteSaver. First, install the required package:

uv add langgraph-checkpoint-sqlite --bounds lower

Here's how to create a persistent memory using SqliteSaver, ensuring conversations survive restarts—perfect for production chat apps.

import sqlite3
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.sqlite import SqliteSaver

# Initialize LLM (replace with your API key)
llm = ChatOpenAI(model="gpt-4o", temperature=0)

# Define a simple tool
@tool
def multiply(a: int, b: int) -> int:
   """Multiplies two numbers."""
   return a * b

tools = [multiply]

# Set up persistent memory with SqliteSaver
conn = sqlite3.connect("checkpoints.db", check_same_thread=False)
memory = SqliteSaver(conn)

# Create the agent
agent_executor = create_react_agent(llm, tools, checkpointer=memory)

# Interact with the agent using a thread_id for memory
config = {"configurable": {"thread_id": "chat1"}}

response1 = agent_executor.invoke({"messages": [{"role": "user", "content": "What is 3 times 4?"}]}, config)
print(response1["messages"][-1].content)  # Output: 12

response2 = agent_executor.invoke({"messages": [{"role": "user", "content": "What was the result of the previous calculation?"}]}, config)
print(response2["messages"][-1].content) # Output: The previous calculation result

This implementation uses SqliteSaver for file-based persistence (in "checkpoints.db"), providing true long-term conversation memory in LangGraph.

For fully custom behavior, you can subclass BaseCheckpointSaver to create your own checkpointer, tailoring persistence (e.g., to JSON files or other databases).

Example: Persistent Chat History

Building on the persistent memory above, let's apply it to a real-world example: a persistent chatbot for customer support. The agent remembers user details across sessions and restarts, improving personalization.

In the code snippet provided, after invoking with "Hi, I'm Bob.", stopping and restarting the script, then asking "Who am I?" should recall the name due to the SQLite storage. This addresses pain points in custom memory LangChain agents, ensuring seamless experiences across many sessions.

Optimization Techniques

To make your memory-efficient in LangGraph:

Limit History Length: Use message trimming functions like trim_messages to avoid token limits.
Summarization: Implement summary nodes in your graph to condense long histories.
Entity Extraction: Add tools for entity extraction and store in a separate memory store for focused recall.
Async Persistence: Use async checkpointers like AsyncSqliteSaver for high-traffic apps to prevent bottlenecks.
Monitoring: Leverage LangSmith to trace memory usage and optimize graphs.

These techniques optimize LLM agents memory, preventing issues like context overflow in extended conversations.

Conclusion: Enhancing Agent Intelligence

Customizing memory with LangGraph transforms basic chatbots into intelligent, context-aware systems. By leveraging checkpointers for persistence, you can build robust applications that remember and adapt across sessions. This not only improves user engagement but also positions your projects for scalability.

Ready to implement advanced conversation memory in LangGraph? If you need help with custom LangChain agents or optimizations, contact us at Focused.io, we're here to help you build agents that work!

Migrating Classic LangChain Agents to LangGraph a How To

Austin Vance — Wed, 16 Jul 2025 15:46:31 +0000

Takeaway: You can swap a legacy AgentExecutor for a LangGraph node in a single commit. The payoff is lower overhead, deterministic routing, and native persistence.

WHY MIGRATE NOW?

LangChain announced that with LangChain 0.2 the original agent helpers (initialize_agent, AgentExecutor) are deprecated and will only receive critical fixes. LangChain recommends moving to LangGraph’s node‑based approach for better control flow, built‑in persistence, and the ability to use multi‑actor workflows.

WHAT CHANGED IN LANGCHAIN 0.2 AND LATER?

Legacy pattern (langchain < 0.2) versus current pattern (langchain 0.2 or newer):

• Agent entry point – legacy: initialize_agent; current: graph node created with LangGraph helpers.
• Configuration – legacy: many function kwargs that are hard to extend; current: typed graph state and composable nodes.
• Persistence – legacy: DIY pickling or a custom DB; current: checkpoint helpers built into LangGraph.
• Deprecation status – legacy helpers are deprecated; the LangGraph approach is the long‑term, fully supported path.

CODE DIFF: FROM `initialize_agent` TO A LANGGRAPH NODE

Below is a minimal ReAct agent that calls a calculator tool—first the legacy way, then the LangGraph way.

Before (legacy agent)

from langchain.agents import AgentExecutor, AgentType, initialize_agent, load_tools
from langchain.chat_models import ChatOpenAI

from langchain_core.tools import tool


@tool
def capitalize(text: str) -> str:
    """Capitalize the text."""
    return text.upper()

llm = ChatOpenAI(model="gpt-4o-mini")

agent = initialize_agent(
    [capitalize],
    llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True,
)

response = agent.run("can you capitalize this text: hello world")
print(response)

After (LangGraph)

from langgraph.prebuilt import create_react_agent
from langgraph.graph import END, StateGraph
from langchain_openai import ChatOpenAI
from langchain.tools import Calculator

class State(TypedDict):
    messages: Annotated[list, add_messages]

@tool
def capitalize(text: str) -> str:
    """Capitalize the text."""
    print(f"Capitalizing text: {text}")
    return text.upper()

llm = ChatOpenAI(model="gpt-4o-mini")
tools = [capitalize]

Create the ReAct agent node

agent_node = create_react_agent(llm, tools)

Build a simple single‑node graph

graph = StateGraph(State)
graph.add_node("react_agent", agent_node)
graph.set_entry_point("react_agent")
graph.add_edge("react_agent", END)

agent_executor = graph.compile()

response = agent_executor.invoke({"messages": [{"role": "user", "content": "can you capitalize this text: hello world"}]})
print(response["messages"][-1].content)

The functional behavior is identical, but you gain an explicit state object, the ability to add router or guardrail nodes later without refactoring the agent itself, and full compatibility with LangGraph’s checkpoint and observability APIs.

UPDATING TESTS AND CALLBACKS

Unit tests with Pytest and LangSmith

rom graph import agent_executor

def test_capitalize_text():
    result = agent_executor.invoke({
        "messages": [{"role": "user", "content": "can you capitalize this text: hello world"}]
    })

    assert "HELLO WORLD" in result["messages"][-1].content

For richer coverage, use LangSmith’s pytest plugin to log run trees and score outputs with metrics instead of brittle string matches.

Callbacks
If you previously passed callbacks into initialize_agent(..., callbacks=[StdOutCallbackHandler()]), move them to the graph compile step (or to individual nodes for fine‑grained tracing):

from langchain_core.callbacks.base import BaseCallbackHandler
class PrintCallbackHandler(BaseCallbackHandler):
    def on_llm_start(self, serialized: dict[str, any], prompts: list[str], **kwargs: any) -> None:
        print(f"LLM start: {prompts}")

response = agent_executor.invoke({"messages": [{"role": "user", "content": "can you capitalize this text: hello world"}]}, {"callbacks": [PrintCallbackHandler()]})

PRODUCTION ROLLOUT CHECKLIST

Freeze versions: pin langchain>=0.2,<0.3, then move to 0.3 and the latest stable langgraph.
Refactor imports: search‑and‑replace initialize_agent( with create_react_agent(.
Compile once: cache the compiled graph (agent_executor) at application start to avoid cold‑start overhead.
State schema: define a TypedDict or Pydantic model for your graph state to catch breaking changes early.
Health probes: invoke the graph with {“message”: {role: {“user”: “ping”}}} and expect "pong" so orchestration platforms detect failures.
Checkpoint storage: configure S3, Redis, or SQLite persistence before rolling to production if your flows exceed a single request.
Observability: enable LANGCHAIN_TRACING_V2=true and send traces to LangSmith.
Canary deploy: route a slice of traffic to the new executor and compare latency and error rate against the legacy path.
Retire legacy code: delete deprecated agent imports when metrics hit parity.
Document the graph: export a GraphViz diagram and commit it so new teammates can visualize the flow.

FINAL WORD

Upgrading to LangGraph is not a risky rewrite; it is a surgical swap that positions your agent for reliable scale, granular observability, and future multi‑actor magic. Make the jump today and own the graph—before the graph owns you.

Why Big Classes Get Bigger: Understanding Preferential Attachment in Your Code

Austin Vance — Thu, 19 Dec 2024 16:15:53 +0000

The Law of Preferential attachment: Why your User class keeps growing

Software often models the natural world. One pattern I have been thinking about is why our biggest components continue to grow. In natural systems there is a phenomenon known as preferential attachment, which describes how new elements in a system tend to collect at the most connected nodes in the network. Zoologists observed this phenomenon in taxonomy where the largest class, order, family, etc collect newly named organisms.

The pull of the common class

This is exactly what happens in our codebases. Someone once said there are two hard things in computer science: “Naming things, cache-invalidation, and off by one errors.” I love the joke, but it hurts every time. Poorly named components create a gravitational pull forcing the next developer between a rock and a hard place: either comb through the component to understand what its purpose is and then refactor the name to be more specific, or take the path of least resistance and add new functionality further expanding the responsibility of the component. This decision creates a feedback loop attracting more and more “like” functionality.

Consider a component named UserHandler. The original intent could be to manage a user's authentication, but because of the vague name, the component could also logically handle anything user related. The UserHandler soon contains preferences, notifications, and social connections along with the original intention, authentication. Each addition makes new complexity even easier to add, moving it closer and closer to the “god class”.

Yes there’s math

There’s actually some math to back this up. Preferential attachment follows a power law distribution where the probability of new functionality being added to a component is proportional to the existing functionality of the component. We can express this mathematically if F(c) represents the functionality in component c, then the probability P of adding complexity to that component is:

\frac{F(c)^\alpha}{\sum_{i} F(i)^\alpha}

Where $α\alpha$ represents the strength of the preferential attachment.

Imagine a system where we have three components

* A - 100 lines of code
* B - 200 lines of code
* C - 1000 lines of code

And the strength of preferential attachment is 2 (which is a pretty common value in the power law) then we end up with

\frac{100^2}{(100+200+1000)^2}

\frac{1000^2}{(100+200+1000)^2}

The probability of adding to C is not 10x but 100x more likely. Which makes it more clear why some classes continue to grow while most remain small.

There is no strength in numbers

If it is not painfully obvious, the larger the component the more impact that that component has on the efficiency of future development. Large components become immovable objects in architectures shaping the systems evolution. This shape has a significant cost on the ability of a system to evolve.

Consider testing. As a component attracts more and more functionality its dependencies grow as well creating more and more potential interactions that must be validated every time the component changes. Changing one part of a system will eventually cascade to the “God Component” and then the changes to that component will require testing and updates to other seemingly unrelated parts of the code.

The answer to preferential attachment is more simply said than done, just name things better. It’s always easier to bring components together than it is to untangle them and refactor them apart.

Using the UserHandler example above

interface UserHandler {
    authenticate(credentials: Credentials): User;
    validateSession(token: string): boolean;
    revokeAccess(userId: string): boolean;
    authenticate(credentials: Credentials): User;
    validateSession(token: string): boolean;
    revokeAccess(userId: string): boolean;
    fetchPreferences(userId: string): UserPreferences;
    updatePreferences(userId: string, preferences: UserPreferences): UserPreferences;
    getDefaultPreferences(): UserPreferences;
    getNotificationSettings(userId: string): NotificationSettings;
    updateNotificationSettings(userId: string, settings: NotificationSettings): NotificationSettings;
    addSocial(userId: string, connectionId: string): string;
    removeSocial(userId: string, connectionId: string): string;
    listSocial(userId: string): string[];
}

This component clearly does a lot and if each method is implemented it could be hundreds of lines long with dependencies on databases, caches, and other whole components

The only constant is change

There are four clear concerns here that each could have their own class or component, creating more clear naming and a more clear place for new functionality to go

interface AuthenticationService {
  authenticate(credentials: Credentials): User;
  validateSession(token: string): boolean;
  revokeAccess(userId: string): boolean;
}

interface PreferenceService {
  fetch(userId: string): UserPreferences;
  update(userId: string, preferences: UserPreferences): UserPreferences;
  getDefaults(): UserPreferences;
}

interface NotificationService {
  getSettings(userId: string): NotificationSettings;
  updateSettings(userId: string, settings: NotificationSettings): NotificationSettings;
}

interface SocialService {
  addSocial(userId: string, connectionId: string): string;
  removeSocial(userId: string, connectionId: string): string;
  listSocial(userId: string): string[];
}

This form is more clear and gives clear indication of the behavior for each component and makes it easy to test and easy for developers to grok the context.

Build for change

Our job as developers is to build code that enables change in a system and by creating clear boundaries with naming we avoid the pull of preferential attachment making new components easier to build an new functionality easier to change old.

Just as natural systems tend toward entropy, software systems tend toward complexity, and the science supports it. Our role is to manage that complexity by introducing patterns and constraints ensuring sustainable and predictable growth of the system over time.

Shameless plug
If you've got a "god class" or "god service" and need help decomposing and untangling your largest components or systems drop me a note or check out focused.io. We love working with legacy code and helping it welcome the future!

Unlock the Power of LangChain: Deploying to Production Made Easy

Austin Vance — Wed, 14 Feb 2024 20:33:45 +0000

In this tutorial, Austin Vance, CEO and co-founder of Focused Labs, will guide you through deploying a PDF RAG with LangChain to production!

In this captivating video, we will dive deep into the process of deploying a rag to production, taking you on an informative and engaging step-by-step journey. Throughout this tutorial, we will explore the intricacies of transforming a local rag into a powerful and accessible resource on the digital ocean app platform. Don't miss out on this highly informative and exciting adventure - make sure to subscribe now to stay updated with all the latest content!
Don't forget to subscribe for more tutorials like this.

Just to remember what happened so far:

In Part One You will Learned:

Create a new app using LangChain's LangServe
ingestion of PDFs using unstructuredio
Chunking of documents via LangChain's SemanticChunker
Embedding chunks using OpenAI's embeddings API
Storing embedded chunks into a PGVector a vector database
Build a LCEL Chain for LangServe that uses PGVector as a retriever
Use the LangServe playground as a way to test our RAG
Stream output including document sources to a future front end.

In Part 2 we will focus on:

Creating a front end with Typescript, React, and Tailwind
Display sources of information along with the LLM output
Stream to the frontend with Server Sent Events

In Part 3 we will focus on:

Deploying the Backend application to @DigitalOcean
Deploying the frontend to @digitalocean_staff App Platform
Use a managed Postgres Database

In Part 4 we will focus on:

Adding Memory to the @LangChain Chain with PostgreSQL
Add Multiquery to the chain for better breadth of search
Add sessions to the Chat History

Github repo

https://github.com/focused-labs/pdf_rag

Chat With Your PDFs: Part 2 - Frontend - An End to End LangChain Tutorial. Build A RAG with OpenAI.

Austin Vance — Mon, 29 Jan 2024 23:35:58 +0000

In this video we are going to dive into part two of building and deploying a fully custom RAG with @LangChain and @OpenAI. In this tutorial, code with me, video we will take the LangServe pipeline we developed in Part 1 and build out a fully functioning React & Typescript frontend using TailwindCSS.

The video did end up getting pretty long so we will deploy the app to @digitalocean_staff and to @LangChain in Part 3!

Just to remember what happened so far:

In Part One You will Learned:

Create a new app using @LangChain's LangServe
ingestion of PDFs using @unstructuredio
Chunking of documents via @LangChain's SemanticChunker
Embedding chunks using @OpenAI's embeddings API
Storing embedded chunks into a PGVector a vector database
Build a LCEL Chain for LangServe that uses PGVector as a retriever
Use the LangServe playground as a way to test our RAG
Stream output including document sources to a future front end.

In Part 2 we will focus on:

Creating a front end with Typescript, React, and Tailwind
Display sources of information along with the LLM output
Stream to the frontend with Server Sent Events

In Part 3 we will focus on:

Deploying the Backend application to @DigitalOcean & @LangChain's LangServe hosted platform to compare
Add LangSmith Integrations
Deploying the frontend to @digitalocean_staff's App Platform
Use a managed Postgres Database

In Part 4 we will focus on:

Adding Memory to the @LangChain Chain with PostgreSQL
Add Multiquery to the chain for better breadth of search
Add sessions to the Chat History

Github repo

https://github.com/focused-labs/pdf_rag

Chat With Your PDFs: Part 1 - An End to End LangChain Tutorial For Building A Custom RAG with OpenAI.

Austin Vance — Wed, 24 Jan 2024 17:48:57 +0000

A common use case for developing AI chat bots is ingesting PDF documents and allowing users to ask questions, inspect the documents, and learn from them. In this tutorial we will start with a 100% blank project and build an end to end chat application that allows users to chat about the Epic Games vs Apple Lawsuit.

There's a lot of content packed into this one video so please ask questions in the comments and I will do my best to help you get past any hurdles.

In Part One You will Learn:

Create a new app using @LangChain 's LangServe
ingestion of PDFs using @unstructuredio
Chunking of documents via @LangChain 's SemanticChunker
Embedding chunks using @OpenAI 's embeddings API
Storing embedded chunks into a PGVector a vector database
Build a LCEL Chain for LangServe that uses PGVector as a retriever
Use the LangServe playground as a way to test our RAG
Stream output including document sources to a future front end.

In Part 2 we will focus on:

Creating a front end with Typescript, React, and Tailwind
Display sources of information along with the LLM output
Stream to the frontend with Server Sent Events
Deploying the Backend application to @DigitalOcean & @LangChain 's LangServe hosted platform to compare
Deploying the frontend to @DigitalOcean 's App Platform

In Part 3 we will focus on:

Adding Memory to the @LangChain Chain with PostgreSQL
Add Multiquery to the chain for better breadth of search
Add sessions to the Chat History

Github repo

https://github.com/focused-labs/pdf_rag

Debugging Your RAG Application: A LangChain, Python, and OpenAI Tutorial

Kaylynn — Thu, 11 Jan 2024 19:41:24 +0000

Let’s explore a real-world example of debugging a RAG-type application. I recently undertook this process while updating our company knowledge base – a resource for potential clients and employees to learn about us.

Tech Stack:

I work with Python and the LangChain framework, specifically using LangChain Expression Language (LCEL) to build chains. You can find the LangChain LCEL documentation here.

This approach services as a good alternative to LangChain’s debugging tool, LangSmith.

# Load memory
def get_session_history(session_id: str) -> ConversationBufferMemory:
    if session_id not in store:
        store[session_id] = ConversationBufferMemory(
            return_messages=True, output_key="answer", input_key="question"
        )
    return store[session_id]

def _get_loaded_memory(x):
    return get_session_history(x["session_id"]).load_memory_variables({"question": x["question"]})

def load_memory_chain():
    return RunnablePassthrough.assign(
        chat_history=RunnableLambda(_get_loaded_memory) | itemgetter("history"),
    )

# Create Question
def create_question_chain():
    return {
        "standalone_question": {
                                   "question": itemgetter("question"),
                                   "chat_history": lambda x: get_buffer_string(x["chat_history"]),
                               }
                               | CONDENSE_QUESTION_PROMPT
                               | llm
                               | StrOutputParser(),
        "role": itemgetter("role"),
    }

# Retrieve Documents
def retrieve_documents_chain(vector_store):
    retriever = vector_store.as_retriever()
    return {
        "role": itemgetter("role"),
        "docs": itemgetter("standalone_question") | retriever,
        "question": lambda x: x["standalone_question"],
    }

# Answer
def create_answer_chain():
    final_inputs = {
        "role": itemgetter("role"),
        "context": lambda x: combine_documents(x["docs"], DEFAULT_DOCUMENT_PROMPT),
        "question": itemgetter("question"),
    }
    return {
        "answer": final_inputs | ANSWER_PROMPT | llm,
        "docs": itemgetter("docs"),
    }

# Final Chain looks like this
chain = load_memory_chain() | create_question_chain() | retrieve_documents_chain() | create_answer_chain()

While debugging, I prefer using a cheaper model like gpt-3.5-turbo for its cost-effectiveness. The less advanced models are more than adequate for basic testing. For final testing and deployment to production, you might consider upgrading to gpt-4-turbo or a similar advanced model.

I also favor Jupyter notebooks for much of my debugging. This way, I can include the notebook in a .gitignore file, reducing cleanup from debugging shenanigans in my main code. I can also run very specific pieces of my code without plumbing overhead.

Initial Observations

I noticed that basic queries received correct answers, but any follow-up question would lack the appropriate context, indicating that conversational memory was no longer functioning effectively.

Here's what I observed:

Question: What are the Focused Labs core values?
> AI: The core values of Focused Labs are Love Your Craft, Listen First, and Learn Why ✅
> Sources: ...

Question: Tell me more about the first one.
> AI: Based on the given context, the first one is about the importance of the "Red" step in Test Driven Development (TDD). ❌
> Sources: ...

However, I expected responses more in line with explanations like "Love Your craft is when you are passionate about what you do."

For more context, this issue with conversational memory arose while I was implementing a new feature: allowing end users to customize responses based on their role. So, for example, a developer could receive a highly technical answer while a marketing manager would see more high-level details.

Debugging Steps

1. Ensure Role Feature Integrity

To avoid impacting the newly implemented role feature, I made it overly obvious and active in every response during this debugging session by temporarily updating my system prompt.

SYSTEM_PROMPT = """Answer the question from the perspective of a {role}."""

DEBUGGING_SYSTEM_PROMPT = """Answer the question in a {role} accent."""

Here's how the AI responded, clearly adhering to my updated prompt:

Question: What are the Focused Labs core values?
Role: pirate
> AI: Arr, the core values of Focused Labs be Love Your Craft, Listen First, and Learn Why, matey! ✅
> Sources: ...

Question: Tell me more about the first one.
> AI: Arr, the first one be talkin' about the importance of reachin' the "Red" stage in Test Driven Development... ✅
> Sources: ...

2. Creating a Visual Representation

I created a diagram of the app to visualize the process flow.

I began at the end of my flow and worked backward to identify issues. I first checked whether my LLM was answering questions based on the provided context. Upon inspecting the sources, I realized that the given context was a blog on TDD.

> Sources: [{'URL': 'https://focusedlabs.io/blog/tdd-first-step-think'}, ...]

Thus, I ruled out the answer component as the source of the bug.

3. Tracing the Bug's Origin

Next, I examined the logic for retrieving documents. I added a 'standalone question' key to every input and output chain to log runtime values, which revealed that questions were being incorrectly rephrased.

💡Adding these keys to the chains allows us to log the values seen by the components at runtime. Using breakpoints will only show the code when it’s instantiated and not populated with real-time values.

# Code Snippet with added keys
def retrieve_documents_chain(vector_store):
    retriever = vector_store.as_retriever()
    return {
          .
                .
                .
        "standalone_question": itemgetter("standalone_question") # Added
    }

def create_answer_chain():
    final_inputs = {
          .
                .
                .
        "standalone_question": itemgetter("standalone_question") # Added 
    }
    return {
          .
                .
                .
        "standalone_question": itemgetter("standalone_question") # Added
    }

I expected the standalone_question to be more specific, like “What can you tell me about the core value of Love your Craft?”

Question: What are the Focused Labs core values?
> standalone_question: What are the core values of Focused Labs? ✅

Question: Tell me more about the first one.
> standalone_question: What can you tell me about the first one? ❌

4. Identifying the Exact Source

I focused on the chat_history variable, suspecting an issue with how the chat history was being recognized.

def retrieve_documents_chain(vector_store):
    retriever = vector_store.as_retriever()
    return {
          .
                .
                .
        "standalone_question": itemgetter("standalone_question") # Added
                "chat_history": itemgetter("chat_history") # Added
    }

def create_answer_chain():
    final_inputs = {
          .
                .
                .
        "standalone_question": itemgetter("standalone_question") # Added 
                "chat_history": itemgetter("chat_history") # Added
    }
    return {
          .
                .
                .
        "standalone_question": itemgetter("standalone_question") # Added
                "chat_history": itemgetter("chat_history") # Added
    }

Question: What are the Focused Labs core values?

Question: Tell me more about the first one.
> chat_history: [] ❌

🔔 Found the issue! Since the chat_history was blank, it wasn’t being loaded as I had assumed.

5. Implementing the Solution

I resolved the issue by checking my conversation memory store. As a dict, the conversation memory store was sensitive to the type of saved messages. I saved the messages with a str converted version of session_id. But, I invoked with an Optional[UUID] version. So, while the conversation memory store itself was set up correctly, I needed to update how I invoked my chain.

result = 
chain.invoke({"question": question, "session_id": session_id, "role": role})

Therefore, I updated the session_id type to str.

result = 
chain.invoke({"question": question, "session_id": str(session_id), "role": role})

6. Confirming the Fix

I confirmed that the conversation memory now functioned correctly.

Question: What are the Focused Labs core values?

Question: Tell me more about the first one.
> chat_history: ['What are the Focused Labs core values?'] ✅
> standalone_question: Can you provide more information about the first core value: Love Your Craft? ✅
> AI: This value means that we are passionate about being the best at what we do, paying attention to every detail... ✅
> Sources: [{'URL': 'https://www.notion.so/Who-are-we-c42efb179fa64f6bb7866deb363fb7ef'}, ...] ✅

7. Final Cleanup and Future-Proofing

I reverted back from the temporary pirate accent debug feature used for easy identification of the role feature.

I decided to maintain detailed logging within the system for future debugging efforts.

Key Takeaways

Debugging AI Systems: A mix of traditional and AI-specific debugging techniques is essential.
Opting for Cost-Effective Models: Use more affordable models to reduce costs during repeated queries.
Importance of Transparency: Clear visibility into each step and component of your RAG accelerates debugging.
Type Consistency: Paying attention to small details, like variable types, can significantly impact functionality.

Thanks for reading!

Stay tuned for more insights into the world of software engineering and AI. Have questions or insights? Feel free to share them in the comments below!

Open AI Assistants: Limited, but Incredible

Kaylynn — Fri, 22 Dec 2023 17:58:47 +0000

In the ever-evolving realm of AI development, OpenAI has ventured into the domain of Retrieval Augmented Generation (RAG) with its latest offering, Assistants. Assistants with Retrieval represents OpenAI's attempt to harness the power of RAG — a technique widely embraced and refined within open-source libraries to augment an AI’s knowledge with custom information. Think ChatGPT, but with the ability to ask it about custom information. While it doesn't yet lead the pack in creating AI-driven knowledge bases, its inception marks a significant step towards more sophisticated and nuanced AI applications. As developers, our dive into this beta tool is not just about evaluating its current capabilities, but also understanding its place in the broader context of RAG's evolution and its potential to reshape our approaches to LLMs.

Incredible Accuracy for Incredible Ease of Use

OpenAI's RAG tool is an awesome prospect for developers eager to explore the nuances of retrieval-augmented models. It serves as a canvas for experimentation, offering a glimpse into the future of sophisticated AI applications. Here's why it's a valuable experimental platform:

Ease of Use: Its user-friendly nature invites developers to explore and experiment with minimal setup. This allows more people to try out more use cases to learn where LLM retrieval could improve their workflows.
Respectable Accuracy: We swapped out our gpt-3.5-turbo model in our custom chatbot with an OpenAI Assistant, and saw a slightly-lower, but similar level of accuracy. This is so cool! Being able to spin up a custom chatbot in a couple of hours with ~75% accuracy rate is incredible! For more information about our custom chatbot, visit our website: focusedlabs.io/ai.

Understanding the Limitations: A Beta Analysis

In its beta stage, this tool has its limitations. The current landscape of RAG in open-source libraries such as Langchain sets a high benchmark, and OpenAI's iteration is an ambitious stride into this territory. However, with ambition comes the teething problems of any beta technology. Here's what developers should be aware of:

Source Citation Feature: A vital feature for gaining user trust, source citation, is not yet functional, indicating future improvements but also a present gap.
Document Limitations: An assistant supports only 20 documents, each up to 512 MB, which is not scalable for most enterprise datasets. For smaller data sets, while the file size itself is not limiting, combining multiple smaller files into 1 larger file is an anti-pattern resulting in loss of structure and context.
Lack of Customization: While the simple interface facilitates quick setup times, high abstraction levels mean less control for developers to adjust and optimize the tool for specific use cases.
Polling Over Streaming: Without streaming capabilities, developers are left with polling methods, which hamper real-time efficiency.

Tips for Maximizing Your Experience

Navigating a beta tool requires a mix of patience and strategy. To make the most of your journey with OpenAI's RAG tool, keep these tips in mind:

Data Management: Efficiently handle your limited document space by converting to and concatenating data wherever possible. We recommend using *.txt formats for ease and compression.
File Type Performance: Experiment with different file types to discover which yields the best results for your specific case. For us, *.txt returned more accurate results over PDFs.
Use with Other Libraries:Langchain supports Open AI Assistants. While Assistants may not be the leader on their own, they are still powerful when combined with other techniques. I also recommend using LlamaHub’s loaders to help integrate with various data sources.

Conclusion

OpenAI's entry into the RAG space is a testament to the ongoing evolution of AI and machine learning technologies. While this particular tool may not be ready for production, it offers a valuable learning curve for developers keen on the future of retrieval-augmented models. By engaging with it critically and creatively, we can contribute to its growth and simultaneously expand our own understanding of where RAG can take us in the realm of AI development.

Enhancing AI Apps with Streaming: Practical Tips for Smoother AI Generation

Kaylynn — Thu, 14 Dec 2023 20:01:13 +0000

Introduction

Interested in building an AI-powered app using generative models? You're on the right track! The realm of generative AI is brimming with untapped potential. A popular approach is to prompt the AI to generate a list of ideas based on provided context, allowing users to select their preferred option and then refine and expand on the content.

An example of a generative AI app is one my team and I created for copywriters to seamlessly integrate storytelling into marketing emails. This user-friendly wizard elevates email creation, combining the power of generative AI with the user’s own expertise and creativity for impactful results. Our development journey led to the integration of streaming technology, significantly reducing AI response times.

In this article, I'll share 2 essential tips to enhance user experience in AI-driven apps through effective use of streaming. Let's dive in!

Tech Stack

Backend: Node.js Typescript with Express
Frontend: React Typescript
AI Integration: OpenAI’s Node SDK and GPT-4

Let’s start with the basics.

First, send requests to OpenAI leveraging their Node.js SDK. Due to our prompt, the AI response is a list with bullet numbers like “1.”. We use the bullet number format to parse the separate options.

import OpenAI from "openai";

require("dotenv").config();
const openai = new OpenAI({
    apiKey: process.env.OPEN_AI_API_KEY,
});
const chatModel = "gpt-4";

export const createIdeas = async ({ occasion }: {
    occasion: string;
}) => {
    const completion = await openai.chat.completions.create({
        messages: [
            {
                role: "user",
                content: `I want to host an event for the following occasion: ${occasion}. Write me a list of 4 separate ideas for this event`,
            },
        ],
        model: chatModel,
    });
    let choiceElementElement = completion.choices[0]["message"]["content"];
    return {ideas: parseIdeas(choiceElementElement || "")};
};

// Use the bullet number format (ex: "1.") to split the ideas into individual elelements in an array
const parseIdeas = (text: string): string[] => {
    const message = text.split(/[0-9]+\./gm);

    const messageSliced = message.slice(1, message.length);
    return [...messageSliced];
};

Then, let’s return the parsed ideas to the frontend. This is example method leverages Express.

app.post("/generate-ideas", async (req: Request, res: Response) => {
  const { occasion } = req.body;
  const generatedIdeas = await createIdeas({ occasion });
  res.send(generatedIdeas);
});

This produces a nicely formatted json response to the frontend that is very easy to pass into the appropriate UI components.

{
    "ideas": [
        " Example first idea \n\n",
        ...
    ]
}

The Challenge: Latency

The hiccup? A waiting time of up to 30 seconds before users can view the AI’s suggestions. Watching a loading icon spin for half a minute is not a good user experience.

Tip #1: Leverage Streaming

Enter OpenAI’s “streaming” feature - a savior for reducing latency. By setting OpenAI’s Node SDK input parameter **stream** to true , we display words to the user as they became available. This doesn’t expedite the complete generation process, but it cuts down the wait time for the first word. Think of it as the “typewriter” effect seen in ChatGPT.

To peek under the hood, the streaming feature uses an HTML5 capability called Server Sent Events (SSE). SSE allows servers to push real-time data to web clients over a single HTTP connection. Unlike WebSockets, which is bidirectional, SSE is unidirectional, making it perfect for sending data from the server to the client in scenarios where the client doesn't need to send data back.

So, we refactor the request to OpenAI to include the input parameter stream and we return a Stream wrapped in an API Promise to our controller method.

export const createIdeas = async ({ occasion }: {
    occasion: string;
}) => {
  return openai.chat.completions.create({
    model: chatModel,
    stream: true,
    messages: [
      {
        role: "user",
        content: `I want to host an event for the following occasion: ${occasion}. Write me a list of 4 separate ideas for this event`,
      },
    ],
  });
};

In our controller method, we clean up the data a little bit, and then send the data over the HTTP connection to the frontend as it is received from our request to OpenAI.

app.post("/generate-ideas", async (req, res) => {
const { occasion } = req.body;

for await (let chunk of await createIdeas({occasion})) {

    // If we don't receive any more content, end the connection. 
  if (chunk.choices[0]?.delta.content == undefined) {
    res.end();
    return;
  }

    // Sometimes the AI will format the response with an extra label. 
    // Remove this. 
  if (chunk.startsWith("data:")) {
    chunk = JSON.parse(chunk.replace("data: ", ""));
  }

    const text = chunk.choices[0].delta.content;
  res.write(`${text}`);
}
};

After understanding the mechanics, our next step was to synchronize the frontend client that communicates with the backend Express endpoint above with the frontend to support SSE.

export const requestIdeas = async (occasion: string) => {
  const res = await fetch(`${process.env.REACT_APP_API_BASE_URL}/generate-ideas`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({occasion})
  });
  return res.body?.pipeThrough(new TextDecoderStream()).getReader();
};

The Impact

The results are night and day. From a staggering 30-second wait, users now see the initial AI-generated content within half a second.

Tip #2: UI Components of Streaming

When users are given multiple options to choose from, the UI should split those different options into different components. For example, each option should be a radio button or a different div. But streaming text in real-time throws a wrench in the works. How can we differentiate, parse each AI-generated suggestion as distinct UI components?

The Solution: Add parsing based on a unique character.

Add a unique Bullet Identifier: Update your prompt to the AI to use an unusual character as a bullet point that is unlikely to appear in the rest of your text. We used the “¶” symbol and updated our prompt to include the following: Start each bullet point with a new line and the ¶ character. Do not include the normal bullet point character, the '-' character, or list numbers.`
Splitting the Stream: We segmented each byte array from the SSE endpoint into distinct words. This separation was pivotal, given that a single SSE byte array content was unpredictable. Sometimes it included a single word, other times it included full phrases that contained the “¶” character, like subject matter. ¶ Engage.
Append each word: Once each word is prepared, we append the value to the appropriate UI component. Tracking the “¶” occurrences helps us assign words to the correct component. For instance, a single “¶” means the content belonged to the first-option component. Repeating this process in the loop until the SSE endpoint closed.

`tsx
export const parseResponseValue = (index: number, value: string, setters: Function[]) => {
// Separate into individual words only (no phrases)
const splitValue = value.split(/(?! {2})/g);

for (let word of splitValue) {
if (word.includes('¶')) {
index++;
} else {
settersindex;
}
}
// Return the index to the calling function for use for the next byte array received from the endpoint.
return index;
};
`

`tsx

const [firstIdea, setFirstIdea] = useState("");
const [secondIdea, setSecondIdea] = useState("");
const [thirdIdea, setThirdIdea] = useState("");
const [fourthIdea, setFourthIdea] = useState("");

const getIdeas = async (newStatus: boolean) => {
const responseReader = await requestIdeas(occasion);

// The index will be incremented each time the unique bullet identifier is seen. This includes the first bullet, so offset by 1.

let index = -1;
while (true) {
const { value, done } = await responseReader.read();
if (done) {
break;
}

index = parseResponseValue(index, value, [
  setFirstIdea,
  setSecondIdea,
  setThirdIdea,
  setFourthIdea,
]);

}
};
`

Though string parsing occasionally fails due to edge cases from the AI, it facilitates an overall better user experience by stylizing real-time text streaming. On encountering an AI anomaly, equip users with the ability to retry AI generation. Generally, this fixes any parsing issue encountered the first time.

Importantly, by avoiding multiple smaller requests, we economized on tokens sent to GPT-4. This not only curtailed costs but also enriched the result quality.

Conclusion

Harnessing the power of generative AI in applications is undeniably transformative, but it doesn't come without its challenges. As we've explored, latency can be a significant hurdle, potentially hampering user experience. However, with innovative solutions like real-time streaming and strategic UI component parsing, we can overcome these challenges, making our applications not only more responsive but also user-friendly.

DEV Community: Focused

Everybody Tests

Everybody Tests

You already think test-first

The cost nobody talks about

The teams that say they don't have time

The same satisfying satisfying, all over again

Eval Driven Development

Vibes don't ship

The choice is the same as it always was

Context Will Replace Your Design

The Translation Layer We Built

When the System Adapts to You

The Clawdbot Case Study

Context Engineering: The New Discipline

What Agents Need

The Skills That Transfer

Design Becomes Invisible

3 Things I’ve Learned While Implementing My First Multi Agent Architecture

Customizing Memory in LangGraph Agents for Better Conversations

Overview of Memory Types in LangChain

Setting Up Simple Memory Buffers

Custom Memory Implementation

Example: Persistent Chat History

Optimization Techniques

Conclusion: Enhancing Agent Intelligence

Migrating Classic LangChain Agents to LangGraph a How To

WHY MIGRATE NOW?

WHAT CHANGED IN LANGCHAIN 0.2 AND LATER?

CODE DIFF: FROM initialize_agent TO A LANGGRAPH NODE

Create the ReAct agent node

Build a simple single‑node graph

UPDATING TESTS AND CALLBACKS

PRODUCTION ROLLOUT CHECKLIST

FINAL WORD

Why Big Classes Get Bigger: Understanding Preferential Attachment in Your Code

The Law of Preferential attachment: Why your User class keeps growing

The pull of the common class

Yes there’s math

There is no strength in numbers

The only constant is change

Build for change

Unlock the Power of LangChain: Deploying to Production Made Easy

Github repo

Chat With Your PDFs: Part 2 - Frontend - An End to End LangChain Tutorial. Build A RAG with OpenAI.

Github repo

Chat With Your PDFs: Part 1 - An End to End LangChain Tutorial For Building A Custom RAG with OpenAI.

Github repo

Debugging Your RAG Application: A LangChain, Python, and OpenAI Tutorial

Tech Stack:

Initial Observations

Debugging Steps

1. Ensure Role Feature Integrity

2. Creating a Visual Representation

3. Tracing the Bug's Origin

4. Identifying the Exact Source

5. Implementing the Solution

6. Confirming the Fix

7. Final Cleanup and Future-Proofing

Key Takeaways

Thanks for reading!

Open AI Assistants: Limited, but Incredible

Incredible Accuracy for Incredible Ease of Use

Understanding the Limitations: A Beta Analysis

Tips for Maximizing Your Experience

Conclusion

Enhancing AI Apps with Streaming: Practical Tips for Smoother AI Generation

Introduction

Tech Stack

The Challenge: Latency

Tip #1: Leverage Streaming

The Impact

Tip #2: UI Components of Streaming

The Solution: Add parsing based on a unique character.

Conclusion

CODE DIFF: FROM `initialize_agent` TO A LANGGRAPH NODE