We are building a design review agent that uses chain-of-thought reasoning and few-shot transfer learning to audit distributed system proposals. It walks through assumptions, spots failure modes, and delivers a verdict, all guided by curated examples that teach the model a review style without any fine-tuning. We will run it against Oxlo.ai so long contexts with many examples cost one flat request, not a mountain of tokens.
What you'll need
- Python 3.10+
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Set up the Oxlo.ai client
Initialize the OpenAI-compatible client pointing to Oxlo.ai. I use llama-3.3-70b because it handles the long few-shot contexts we will feed it.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
Step 2: Curate the transfer learning examples
These examples act as our transferred knowledge. They teach the model the exact review format and depth we want, no gradient updates required.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
EXAMPLES = [
{
"role": "user",
"content": "Design: A single PostgreSQL primary with read replicas for a high-write analytics dashboard."
},
{
"role": "assistant",
"content": (
"Step 1: Identify the workload pattern. Analytics dashboards are typically read-heavy, "
"but the prompt specifies high-write, which creates replica lag.\n"
"Step 2: Check for a single point of failure. One PostgreSQL primary means total outage on failure.\n"
"Step 3: Evaluate consistency. Read replicas may serve stale data, causing inaccurate dashboards.\n"
"Verdict: Reject. Recommend sharded writes or a column-store OLAP database instead."
)
},
{
"role": "user",
"content": "Design: An S3-backed Lambda function resizing images synchronously inside a user request."
},
{
"role": "assistant",
"content": (
"Step 1: Identify latency sensitivity. S3 Lambda cold starts plus image processing will exceed HTTP timeouts.\n"
"Step 2: Check coupling. Synchronous coupling makes the upload flow brittle.\n"
"Step 3: Evaluate cost. Lambda billed by duration; large images are expensive.\n"
"Verdict: Reject. Recommend async S3 event triggers and a pre-signed URL polling mechanism."
)
}
]
Step 3: Write the chain-of-thought system prompt
The system prompt forces the model to emit reasoning steps before any conclusion. This chain-of-thought architecture makes the agent auditable and far less likely to hallucinate a verdict.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
EXAMPLES = [
{
"role": "user",
"content": "Design: A single PostgreSQL primary with read replicas for a high-write analytics dashboard."
},
{
"role": "assistant",
"content": (
"Step 1: Identify the workload pattern. Analytics dashboards are typically read-heavy, "
"but the prompt specifies high-write, which creates replica lag.\n"
"Step 2: Check for a single point of failure. One PostgreSQL primary means total outage on failure.\n"
"Step 3: Evaluate consistency. Read replicas may serve stale data, causing inaccurate dashboards.\n"
"Verdict: Reject. Recommend sharded writes or a column-store OLAP database instead."
)
},
{
"role": "user",
"content": "Design: An S3-backed Lambda function resizing images synchronously inside a user request."
},
{
"role": "assistant",
"content": (
"Step 1: Identify latency sensitivity. S3 Lambda cold starts plus image processing will exceed HTTP timeouts.\n"
"Step 2: Check coupling. Synchronous coupling makes the upload flow brittle.\n"
"Step 3: Evaluate cost. Lambda billed by duration; large images are expensive.\n"
"Verdict: Reject. Recommend async S3 event triggers and a pre-signed URL polling mechanism."
)
}
]
SYSTEM_PROMPT = """You are a senior staff engineer who reviews distributed system designs.
You must reason step by step before giving a final verdict.
Follow this exact format:
Step 1: Identify the core workload pattern and assumptions.
Step 2: Check for single points of failure and bottleneck risks.
Step 3: Evaluate consistency, latency, and cost implications.
Verdict: Accept, Accept with modifications, or Reject. Provide one concrete recommendation.
Be concise. Use the examples to guide your reasoning style."""
Step 4: Assemble the few-shot message history
This function injects the examples into every request. The model transfers the review pattern to the new design through pure in-context learning.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
EXAMPLES = [
{
"role": "user",
"content": "Design: A single PostgreSQL primary with read replicas for a high-write analytics dashboard."
},
{
"role": "assistant",
"content": (
"Step 1: Identify the workload pattern. Analytics dashboards are typically read-heavy, "
"but the prompt specifies high-write, which creates replica lag.\n"
"Step 2: Check for a single point of failure. One PostgreSQL primary means total outage on failure.\n"
"Step 3: Evaluate consistency. Read replicas may serve stale data, causing inaccurate dashboards.\n"
"Verdict: Reject. Recommend sharded writes or a column-store OLAP database instead."
)
},
{
"role": "user",
"content": "Design: An S3-backed Lambda function resizing images synchronously inside a user request."
},
{
"role": "assistant",
"content": (
"Step 1: Identify latency sensitivity. S3 Lambda cold starts plus image processing will exceed HTTP timeouts.\n"
"Step 2: Check coupling. Synchronous coupling makes the upload flow brittle.\n"
"Step 3: Evaluate cost. Lambda billed by duration; large images are expensive.\n"
"Verdict: Reject. Recommend async S3 event triggers and a pre-signed URL polling mechanism."
)
}
]
SYSTEM_PROMPT = """You are a senior staff engineer who reviews distributed system designs.
You must reason step by step before giving a final verdict.
Follow this exact format:
Step 1: Identify the core workload pattern and assumptions.
Step 2: Check for single points of failure and bottleneck risks.
Step 3: Evaluate consistency, latency, and cost implications.
Verdict: Accept, Accept with modifications, or Reject. Provide one concrete recommendation.
Be concise. Use the examples to guide your reasoning style."""
def build_messages(user_design: str):
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
messages.extend(EXAMPLES)
messages.append({"role": "user", "content": f"Design: {user_design}"})
return messages
Step 5: Query the model and return the reasoning
We send the full context to Oxlo.ai. Because pricing is per request, adding more examples does not raise the cost per call.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
EXAMPLES = [
{
"role": "user",
"content": "Design: A single PostgreSQL primary with read replicas for a high-write analytics dashboard."
},
{
"role": "assistant",
"content": (
"Step 1: Identify the workload pattern. Analytics dashboards are typically read-heavy, "
"but the prompt specifies high-write, which creates replica lag.\n"
"Step 2: Check for a single point of failure. One PostgreSQL primary means total outage on failure.\n"
"Step 3: Evaluate consistency. Read replicas may serve stale data, causing inaccurate dashboards.\n"
"Verdict: Reject. Recommend sharded writes or a column-store OLAP database instead."
)
},
{
"role": "user",
"content": "Design: An S3-backed Lambda function resizing images synchronously inside a user request."
},
{
"role": "assistant",
"content": (
"Step 1: Identify latency sensitivity. S3 Lambda cold starts plus image processing will exceed HTTP timeouts.\n"
"Step 2: Check coupling. Synchronous coupling makes the upload flow brittle.\n"
"Step 3: Evaluate cost. Lambda billed by duration; large images are expensive.\n"
"Verdict: Reject. Recommend async S3 event triggers and a pre-signed URL polling mechanism."
)
}
]
SYSTEM_PROMPT = """You are a senior staff engineer who reviews distributed system designs.
You must reason step by step before giving a final verdict.
Follow this exact format:
Step 1: Identify the core workload pattern and assumptions.
Step 2: Check for single points of failure and bottleneck risks.
Step 3: Evaluate consistency, latency, and cost implications.
Verdict: Accept, Accept with modifications, or Reject. Provide one concrete recommendation.
Be concise. Use the examples to guide your reasoning style."""
def build_messages(user_design: str):
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
messages.extend(EXAMPLES)
messages.append({"role": "user", "content": f"Design: {user_design}"})
return messages
def review_design(user_design: str) -> str:
messages = build_messages(user_design)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
)
return response.choices[0].message.content
Run it
Here is a concrete design proposal we can audit. Running this script prints the agent's chain of thought and final verdict.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
EXAMPLES = [
{
"role": "user",
"content": "Design: A single PostgreSQL primary with read replicas for a high-write analytics dashboard."
},
{
"role": "assistant",
"content": (
"Step 1: Identify the workload pattern. Analytics dashboards are typically read-heavy, "
"but the prompt specifies high-write, which creates replica lag.\n"
"Step 2: Check for a single point of failure. One PostgreSQL primary means total outage on failure.\n"
"Step 3: Evaluate consistency. Read replicas may serve stale data, causing inaccurate dashboards.\n"
"Verdict: Reject. Recommend sharded writes or a column-store OLAP database instead."
)
},
{
"role": "user",
"content": "Design: An S3-backed Lambda function resizing images synchronously inside a user request."
},
{
"role": "assistant",
"content": (
"Step 1: Identify latency sensitivity. S3 Lambda cold starts plus image processing will exceed HTTP timeouts.\n"
"Step 2: Check coupling. Synchronous coupling makes the upload flow brittle.\n"
"Step 3: Evaluate cost. Lambda billed by duration; large images are expensive.\n"
"Verdict: Reject. Recommend async S3 event triggers and a pre-signed URL polling mechanism."
)
}
]
SYSTEM_PROMPT = """You are a senior staff engineer who reviews distributed system designs.
You must reason step by step before giving a final verdict.
Follow this exact format:
Step 1: Identify the core workload pattern and assumptions.
Step 2: Check for single points of failure and bottleneck risks.
Step 3: Evaluate consistency, latency, and cost implications.
Verdict: Accept, Accept with modifications, or Reject. Provide one concrete recommendation.
Be concise. Use the examples to guide your reasoning style."""
def build_messages(user_design: str):
messages = [{"role": "system", "content": SYSTEM_PROMPT}]
messages.extend(EXAMPLES)
messages.append({"role": "user", "content": f"Design: {user_design}"})
return messages
def review_design(user_design: str) -> str:
messages = build_messages(user_design)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=messages,
)
return response.choices[0].message.content
if __name__ == "__main__":
design = (
"Design: A Redis cache sitting in front of a MySQL database "
"handling financial ledger writes with TTL-based eviction."
)
print(review_design(design))
Example output:
Step 1: Identify the core workload pattern and assumptions. Financial ledgers require strong consistency and durability, but Redis is an in-memory store with TTL eviction, which risks data loss.
Step 2: Check for single points of failure and bottleneck risks. Redis introduces a separate failure domain. If eviction triggers before MySQL persistence, the ledger state is unrecoverable.
Step 3: Evaluate consistency, latency, and cost implications. Write-through caching adds latency and complexity without a clear read-heavy benefit for ledger writes.
Verdict: Reject. Remove the cache for writes and use MySQL with proper indexing and connection pooling, or adopt an append-only event log pattern.
Wrap-up
Swap in deepseek-r1-671b or kimi-k2.6 on Oxlo.ai if you need heavier reasoning for regulatory or safety-critical designs. You can also grow the example bank to cover domain-specific patterns, and because Oxlo.ai charges per request rather than per token, those longer few-shot contexts will not raise your inference costs. See https://oxlo.ai/pricing for plan details. A natural next step is to cache reviewed designs in a vector store and retrieve the top-k most similar examples for each new query, keeping context length tight without sacrificing transfer quality.
Top comments (0)