<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Autonomous Studio</title>
    <description>The latest articles on DEV Community by Autonomous Studio (@autonomousstudio).</description>
    <link>https://dev.to/autonomousstudio</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4248686%2F05106f1a-323e-48d5-a52d-e9fe25449a50.png</url>
      <title>DEV Community: Autonomous Studio</title>
      <link>https://dev.to/autonomousstudio</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/autonomousstudio"/>
    <language>en</language>
    <item>
      <title>How to use RAG Production Automation Kit</title>
      <dc:creator>Autonomous Studio</dc:creator>
      <pubDate>Tue, 17 Nov 2026 03:10:38 +0000</pubDate>
      <link>https://dev.to/autonomousstudio/how-to-use-rag-production-automation-kit-2p1j</link>
      <guid>https://dev.to/autonomousstudio/how-to-use-rag-production-automation-kit-2p1j</guid>
      <description>&lt;h2&gt;
  
  
  Introduction: Why Automating Retrieval‑Augmented Generation (RAG) Matters
&lt;/h2&gt;

&lt;p&gt;Building a knowledge‑base assistant often starts with a simple pipeline:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Ingest&lt;/strong&gt; raw documents (PDFs, markdown, web pages).
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chunk&lt;/strong&gt; the text and embed each piece.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Store&lt;/strong&gt; the embeddings in a vector DB.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retrieve&lt;/strong&gt; the most relevant chunks at query time.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt&lt;/strong&gt; a LLM with the retrieved context and the user question.
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;When you do this manually, each step becomes a source of friction:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Inconsistent chunking&lt;/strong&gt; leads to missed context.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Embedding drift&lt;/strong&gt; occurs when you change models but forget to re‑index.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Vector‑DB hygiene&lt;/strong&gt; (duplicate vectors, stale metadata) quickly degrades recall.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt engineering&lt;/strong&gt; is repeated across projects, making it hard to share best practices.
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The &lt;strong&gt;RAG Production Automation Kit&lt;/strong&gt; (RAG‑PAK) solves these pain points by providing a ready‑to‑run, opinionated framework that wires together ingestion, embedding, storage, retrieval, and prompting with minimal code. The kit is built on open‑source components (LangChain, FAISS/PGVector, OpenAI/Claude APIs) but abstracts the boilerplate into reusable utilities.&lt;/p&gt;

&lt;p&gt;Below is a step‑by‑step guide to get a knowledge‑base assistant up and running in under an hour.&lt;/p&gt;




&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Requirement&lt;/th&gt;
&lt;th&gt;Recommended Version&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Python&lt;/td&gt;
&lt;td&gt;3.9 – 3.12&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Virtual env&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;venv&lt;/code&gt; or &lt;code&gt;conda&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LLM API key&lt;/td&gt;
&lt;td&gt;OpenAI (&lt;code&gt;OPENAI_API_KEY&lt;/code&gt;) or Anthropic (&lt;code&gt;ANTHROPIC_API_KEY&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vector DB&lt;/td&gt;
&lt;td&gt;FAISS (local) or PostgreSQL + PGVector (remote)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Git&lt;/td&gt;
&lt;td&gt;To clone the kit repo&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip:&lt;/strong&gt; The kit works out‑of‑the‑box with FAISS for quick prototyping. Switch to PGVector when you need persistence across restarts.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Step 1: Set Up the Project
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 1️⃣ Clone the repo&lt;/span&gt;
git clone https://github.com/3km-collab/rag-production-automation-kit.git
&lt;span class="nb"&gt;cd&lt;/span&gt; rag-production-automation-kit

&lt;span class="c"&gt;# 2️⃣ Create a clean environment&lt;/span&gt;
python -m venv .venv
&lt;span class="nb"&gt;source&lt;/span&gt; .venv/bin/activate   &lt;span class="c"&gt;# Windows: .venv\Scripts\activate&lt;/span&gt;

&lt;span class="c"&gt;# 3️⃣ Install dependencies&lt;/span&gt;
pip install -r requirements.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;requirements.txt&lt;/code&gt; pins the exact versions used by the kit, ensuring reproducibility.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 2: Prepare Your Knowledge Sources
&lt;/h2&gt;

&lt;p&gt;Place every document you want the assistant to know inside the &lt;code&gt;data/&lt;/code&gt; folder. The kit supports:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Plain text&lt;/strong&gt; (&lt;code&gt;.txt&lt;/code&gt;)
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Markdown&lt;/strong&gt; (&lt;code&gt;.md&lt;/code&gt;)
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PDF&lt;/strong&gt; (&lt;code&gt;.pdf&lt;/code&gt;) – parsed with &lt;code&gt;pdfplumber&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;HTML&lt;/strong&gt; (&lt;code&gt;.html&lt;/code&gt;) – stripped of tags automatically
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mkdir -p data
cp ~/my‑docs/*.pdf data/
cp ~/my‑docs/*.md  data/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The ingestion script will walk the directory recursively, so you can organize files in subfolders if you wish.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 3: Run the Automated Ingestion Pipeline
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python scripts/run_ingest.py \
  --source-dir data \
  --embedding-model openai:text-embedding-ada-002 \
  --vector-store faiss \
  --chunk-size 500 \
  --overlap 50
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;What happens under the hood?&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;Kit Component&lt;/th&gt;
&lt;th&gt;Default Behaviour&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Chunking&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Chunker&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Splits on sentence boundaries, 500‑token windows with 50‑token overlap.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Embedding&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Embedder&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Calls the selected OpenAI embedding endpoint; caches results in &lt;code&gt;cache/&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Vector Store&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;VectorStoreFactory&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Instantiates a FAISS index (&lt;code&gt;index.faiss&lt;/code&gt;) and persists it to &lt;code&gt;store/&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Metadata&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;MetadataEnricher&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Adds source file path, page number (for PDFs), and a UUID to each vector.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The script is idempotent: running it again will only embed new or changed files, thanks to a SHA‑256 checksum stored alongside each vector.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 4: Define a Retrieval‑Augmented Prompt Template
&lt;/h2&gt;

&lt;p&gt;Create a file &lt;code&gt;prompts/assistant.txt&lt;/code&gt; with the following content (feel free to tweak):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;You are a helpful assistant with access to a corporate knowledge base.
Use ONLY the provided context to answer the user's question.
If the answer is not in the context, reply with "I don't have enough information."

Context:
{retrieved_chunks}

Question:
{question}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The kit will load this template and substitute &lt;code&gt;{retrieved_chunks}&lt;/code&gt; with the top‑k results from the vector store.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 5: Spin Up the Query Service
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python scripts/run_query.py \
  --vector-store faiss \
  --top-k 5 \
  --prompt-file prompts/assistant.txt \
  --llm openai:gpt-4o-mini
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The script launches a tiny FastAPI server on &lt;code&gt;http://localhost:8000&lt;/code&gt;. You can query it with &lt;code&gt;curl&lt;/code&gt; or any HTTP client:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -X POST http://localhost:8000/query \
  -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
  -d &lt;span class="s1"&gt;'{"question": "What is our refund policy for enterprise customers?"}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Response example&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"answer"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Enterprise customers are eligible for a 30‑day refundable period after purchase, provided they have not exceeded usage limits."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"sources"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="s2"&gt;"data/policies/refund_policy.pdf (page 3)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"data/terms/enterprise_terms.md (line 42‑48)"&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The server automatically logs the raw LLM output, the retrieved chunks, and latency metrics to &lt;code&gt;logs/&lt;/code&gt;, giving you observability without extra instrumentation.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 6: Add Continuous Updating (Optional)
&lt;/h2&gt;

&lt;p&gt;For production use you often need to ingest new documents nightly. The kit ships a &lt;strong&gt;watchdog&lt;/strong&gt; utility that monitors the &lt;code&gt;data/&lt;/code&gt; folder and triggers incremental re‑indexing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python scripts/watchdog.py \
  --source-dir data \
  --interval 300   &lt;span class="c"&gt;# check every 5 minutes&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Combine this with a CI/CD step that restarts the FastAPI service after each successful re‑index, ensuring the assistant always serves the latest knowledge.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 7: Test and Iterate
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Unit test the retrieval&lt;/strong&gt; – Use &lt;code&gt;scripts/run_test_retrieval.py&lt;/code&gt; to verify that a known query returns the expected chunk.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt tuning&lt;/strong&gt; – Adjust &lt;code&gt;top-k&lt;/code&gt;, chunk size, or the prompt template based on observed answer quality.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scale the vector store&lt;/strong&gt; – When you exceed a few hundred thousand vectors, switch to PGVector by changing &lt;code&gt;--vector-store pgvector&lt;/code&gt; and providing a Postgres connection string.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  Recap
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Goal&lt;/th&gt;
&lt;th&gt;How RAG‑PAK Helps&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Fast onboarding&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;One‑line commands for ingestion and serving.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Consistent chunking&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Centralized &lt;code&gt;Chunker&lt;/code&gt; with configurable size/overlap.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Embedding cache&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Avoids duplicate API calls, saving cost.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Observability&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Built‑in logging of retrieval, LLM response, and latency.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Extensibility&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Swap vector DB, LLM, or prompt without touching core code.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;With these steps you have a production‑ready RAG pipeline that can be dropped into any AI startup or enterprise project.&lt;/p&gt;




&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Explore the live demo&lt;/strong&gt; to see the kit in action: &lt;a href="https://3km-collab.github.io/rag-production-automation-kit-12/" rel="noopener noreferrer"&gt;https://3km-collab.github.io/rag-production-automation-kit-12/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Grab the full package&lt;/strong&gt; (including advanced monitoring hooks and Docker support) here: &lt;a href="https://autonomousstudio.gumroad.com/l/xzxbdn" rel="noopener noreferrer"&gt;https://autonomousstudio.gumroad.com/l/xzxbdn&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Happy building!&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Autonomous Studio — Self-hosted AI Micro-APIs &amp;amp; Developer Tools.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://autonomousstudio.gumroad.com/l/xzxbdn" rel="noopener noreferrer"&gt;Get the full package&lt;/a&gt; · &lt;a href="https://3km-collab.github.io/rag-production-automation-kit-12/" rel="noopener noreferrer"&gt;Live demo&lt;/a&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>datascience</category>
      <category>opensource</category>
    </item>
    <item>
      <title>How to use StageRAG Production Automation Pack</title>
      <dc:creator>Autonomous Studio</dc:creator>
      <pubDate>Tue, 17 Nov 2026 03:07:32 +0000</pubDate>
      <link>https://dev.to/autonomousstudio/how-to-use-stagerag-production-automation-pack-3lk7</link>
      <guid>https://dev.to/autonomousstudio/how-to-use-stagerag-production-automation-pack-3lk7</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;Retrieval‑augmented generation (RAG) is now the go‑to pattern for building search‑oriented chatbots and knowledge‑base assistants.  While the concept is simple—fetch relevant documents, feed them to a LLM, and let the model generate a grounded answer—getting a &lt;strong&gt;production‑ready&lt;/strong&gt; pipeline up and running can be surprisingly time‑consuming.  &lt;/p&gt;

&lt;p&gt;Typical pain points for AI startups and SaaS teams include:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Problem&lt;/th&gt;
&lt;th&gt;Why it hurts&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;Manual data ingestion&lt;/strong&gt; – Every new source (PDF, CSV, website) requires a custom scraper and indexer.&lt;/td&gt;
&lt;td&gt;Slows down onboarding of new content and introduces bugs.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;Fragmented prompting&lt;/strong&gt; – Prompt templates are scattered across notebooks, making it hard to keep them consistent.&lt;/td&gt;
&lt;td&gt;Leads to hallucinations and hard‑to‑debug responses.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;No observability&lt;/strong&gt; – No built‑in logging of retrieval scores, latency, or LLM usage.&lt;/td&gt;
&lt;td&gt;Makes troubleshooting and cost‑control a guessing game.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;Deployment friction&lt;/strong&gt; – Moving from a local notebook to a cloud‑native service often means rewriting large parts of the code.&lt;/td&gt;
&lt;td&gt;Delays time‑to‑market and forces engineers to duplicate effort.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The &lt;strong&gt;StageRAG Production Automation Pack&lt;/strong&gt; from RAGForge tackles these issues out of the box. It provides a modular, opinionated framework that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Ingests&lt;/strong&gt; and &lt;strong&gt;indexes&lt;/strong&gt; data automatically (supporting common formats).
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Standardises&lt;/strong&gt; prompt construction with reusable templates.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Collects&lt;/strong&gt; telemetry for retrieval scores, LLM token usage, and latency.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Exports&lt;/strong&gt; a Docker‑ready FastAPI service that can be deployed to any cloud provider.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Below is a step‑by‑step, code‑first tutorial that shows how to turn a raw collection of PDFs into a fully‑instrumented RAG endpoint using the pack.&lt;/p&gt;




&lt;h2&gt;
  
  
  Prerequisites
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Requirement&lt;/th&gt;
&lt;th&gt;Version&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Python&lt;/td&gt;
&lt;td&gt;3.10+&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Docker&lt;/td&gt;
&lt;td&gt;24.0+&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;pip&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;23.2+&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;An OpenAI‑compatible LLM API key (or any other provider supported by LangChain)&lt;/td&gt;
&lt;td&gt;–&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Create a fresh virtual environment to keep dependencies tidy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python -m venv .venv
&lt;span class="nb"&gt;source&lt;/span&gt; .venv/bin/activate   &lt;span class="c"&gt;# Windows: .venv\Scripts\activate&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  1. Install the Automation Pack
&lt;/h2&gt;

&lt;p&gt;The pack is distributed as a pip‑installable wheel on Gumroad. Install it directly from the URL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install https://autonomousstudio.gumroad.com/l/ecfosy/download
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This pulls in the core &lt;code&gt;stagerag&lt;/code&gt; library plus optional extras for PDF, CSV, and web‑scraping support.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. Prepare Your Data
&lt;/h2&gt;

&lt;p&gt;For this demo we’ll use a small folder of PDFs (&lt;code&gt;./data/knowledge/&lt;/code&gt;). The pack expects a &lt;strong&gt;manifest&lt;/strong&gt; file that tells it where to find the sources and how to label them.&lt;/p&gt;

&lt;p&gt;Create &lt;code&gt;manifest.yaml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pdf&lt;/span&gt;
    &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./data/knowledge/&lt;/span&gt;
    &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;category&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;product_manual&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can add more entries (CSV, HTML, plain‑text) later; the pack will automatically pick the appropriate loader.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Build the Retrieval Index
&lt;/h2&gt;

&lt;p&gt;StageRAG ships with a CLI that abstracts away the vector store setup. Run the following command to create a ChromaDB index (you can swap &lt;code&gt;faiss&lt;/code&gt;, &lt;code&gt;pinecone&lt;/code&gt;, etc., via a flag).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;stagerag index create \
  --manifest manifest.yaml \
  --store chroma \
  --embedding openai:text-embedding-ada-002 \
  --output ./index/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What happens under the hood?&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Loading&lt;/strong&gt; – Each PDF is parsed into chunks (default 500‑token windows).
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Embedding&lt;/strong&gt; – The chosen embedding model converts each chunk into a dense vector.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Storing&lt;/strong&gt; – Vectors, along with source metadata, are persisted in a local Chroma collection (&lt;code&gt;./index/&lt;/code&gt;).
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The command prints a short summary:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;✅ Indexed 12 documents → 3,842 chunks
🗄️ Stored in ./index/ (Chroma)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  4. Define a Prompt Template
&lt;/h2&gt;

&lt;p&gt;Consistent prompting is essential for reliable RAG. Create &lt;code&gt;prompt.yaml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
  &lt;span class="s"&gt;You are a helpful assistant for {category} documentation.&lt;/span&gt;
  &lt;span class="s"&gt;Use only the information provided in the retrieved passages.&lt;/span&gt;
  &lt;span class="s"&gt;Answer the user query verbatim and cite the source IDs.&lt;/span&gt;

&lt;span class="na"&gt;variables&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s"&gt;category&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;{category}&lt;/code&gt; placeholder will be filled from the metadata attached to each chunk, allowing you to tailor responses per knowledge domain.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Wire Up the FastAPI Service
&lt;/h2&gt;

&lt;p&gt;StageRAG provides a starter FastAPI app that you can customise via a small &lt;code&gt;app.py&lt;/code&gt;. Copy the scaffold:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# app.py&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;fastapi&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;HTTPException&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pydantic&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;BaseModel&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;stagerag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;core&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;RetrievalEngine&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;PromptRenderer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Telemetry&lt;/span&gt;

&lt;span class="c1"&gt;# Load the index we built earlier&lt;/span&gt;
&lt;span class="n"&gt;engine&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;RetrievalEngine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;vector_store_path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"./index/"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;embedding_model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"openai:text-embedding-ada-002"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Load the prompt template&lt;/span&gt;
&lt;span class="n"&gt;renderer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;PromptRenderer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_yaml&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"prompt.yaml"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Telemetry collector (logs to stdout; can be swapped for Prometheus, etc.)&lt;/span&gt;
&lt;span class="n"&gt;telemetry&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Telemetry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"StageRAG Demo Service"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;QueryRequest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"product_manual"&lt;/span&gt;

&lt;span class="nd"&gt;@app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/answer"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;QueryRequest&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# 1️⃣ Retrieve top‑k chunks&lt;/span&gt;
    &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;engine&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;retrieve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metadata_filter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"category"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# 2️⃣ Render the prompt with retrieved passages&lt;/span&gt;
    &lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;renderer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;render&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;passages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# 3️⃣ Call the LLM (using OpenAI API by default)&lt;/span&gt;
    &lt;span class="n"&gt;answer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;engine&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;complete&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# 4️⃣ Record telemetry&lt;/span&gt;
    &lt;span class="n"&gt;telemetry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;record&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;latency&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;engine&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last_latency&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;token_usage&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;engine&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;last_token_usage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;retrieved_ids&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"answer"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"sources"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Key points:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Metadata filtering&lt;/strong&gt; ensures the engine only pulls documents that match the requested &lt;code&gt;category&lt;/code&gt;.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Telemetry&lt;/strong&gt; captures latency and token usage automatically; you can plug in a Grafana dashboard later.
&lt;/li&gt;
&lt;li&gt;The endpoint returns both the generated answer and the IDs of the source chunks, enabling downstream UI to display citations.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  6. Containerise the Service
&lt;/h2&gt;

&lt;p&gt;Running locally is fine for development, but production demands isolation. The pack includes a Dockerfile template; create &lt;code&gt;Dockerfile&lt;/code&gt; in the project root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="k"&gt;FROM&lt;/span&gt; python:3.11-slim

&lt;span class="k"&gt;WORKDIR&lt;/span&gt; /app
&lt;span class="k"&gt;COPY&lt;/span&gt; requirements.txt .
&lt;span class="k"&gt;RUN&lt;/span&gt; pip install -r requirements.txt

&lt;span class="k"&gt;COPY&lt;/span&gt; . /app
&lt;span class="k"&gt;EXPOSE&lt;/span&gt; 8000

&lt;span class="k"&gt;CMD&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"uvicorn"&lt;/span&gt;, &lt;span class="s2"&gt;"app:app"&lt;/span&gt;, &lt;span class="s2"&gt;"--host"&lt;/span&gt;, &lt;span class="s2"&gt;"0.0.0.0"&lt;/span&gt;, &lt;span class="s2"&gt;"--port"&lt;/span&gt;, &lt;span class="s2"&gt;"8000"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Generate &lt;code&gt;requirements.txt&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip freeze &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; requirements.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Build and run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker build -t stagerag-demo .
docker run -p 8000:8000 stagerag-demo
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Your RAG service is now reachable at &lt;code&gt;http://localhost:8000/answer&lt;/code&gt;. Test it with &lt;code&gt;curl&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -X POST http://localhost:8000/answer \
  -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
  -d &lt;span class="s1"&gt;'{"query":"How do I reset my device?", "category":"product_manual"}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You should receive a JSON payload containing the answer and the source IDs.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Add Observability (Optional)
&lt;/h2&gt;

&lt;p&gt;If you already run Prometheus/Grafana, replace the &lt;code&gt;Telemetry&lt;/code&gt; instance with a Prometheus exporter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;stagerag&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;telemetry&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;PrometheusTelemetry&lt;/span&gt;
&lt;span class="n"&gt;telemetry&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;PrometheusTelemetry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;namespace&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"stagerag"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Expose the &lt;code&gt;/metrics&lt;/code&gt; endpoint (FastAPI does this automatically) and you’ll see counters for &lt;code&gt;queries_total&lt;/code&gt;, &lt;code&gt;latency_seconds&lt;/code&gt;, and &lt;code&gt;tokens_used&lt;/code&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Deploy to the Cloud
&lt;/h2&gt;

&lt;p&gt;Because the service is just a container, you can push it to any registry and spin it up on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;AWS ECS/Fargate&lt;/strong&gt; – use the provided &lt;code&gt;ecs-task-def.json&lt;/code&gt;.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Google Cloud Run&lt;/strong&gt; – &lt;code&gt;gcloud run deploy stagerag-demo --image gcr.io/…&lt;/code&gt;.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Azure Container Apps&lt;/strong&gt; – &lt;code&gt;az containerapp create …&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All you need is the same environment variable for the LLM API key (&lt;code&gt;OPENAI_API_KEY&lt;/code&gt; or the equivalent for your provider).&lt;/p&gt;




&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;The StageRAG Production Automation Pack removes the repetitive plumbing that usually surrounds a RAG pipeline. By following the steps above you can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Ingest&lt;/strong&gt; any mix of PDFs, CSVs, or web pages with a single manifest.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Index&lt;/strong&gt; them into a vector store without writing custom embedding code.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Serve&lt;/strong&gt; a consistent, observable API that respects metadata filters and returns traceable citations.
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This workflow lets AI startups focus on the &lt;em&gt;value&lt;/em&gt; of their domain knowledge rather than the mechanics of retrieval and prompting.&lt;/p&gt;




&lt;h3&gt;
  
  
  Quick Links
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Live demo&lt;/strong&gt; – see the demo UI in action: &lt;a href="https://3km-collab.github.io/stagerag-production-automation-pack-11/" rel="noopener noreferrer"&gt;https://3km-collab.github.io/stagerag-production-automation-pack-11/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Package download&lt;/strong&gt; – grab the automation pack here: &lt;a href="https://autonomousstudio.gumroad.com/l/ecfosy" rel="noopener noreferrer"&gt;https://autonomousstudio.gumroad.com/l/ecfosy&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;strong&gt;Autonomous Studio — Self-hosted AI Micro-APIs &amp;amp; Developer Tools.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://autonomousstudio.gumroad.com/l/ecfosy" rel="noopener noreferrer"&gt;Get the full package&lt;/a&gt; · &lt;a href="https://3km-collab.github.io/stagerag-production-automation-pack-11/" rel="noopener noreferrer"&gt;Live demo&lt;/a&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>datascience</category>
      <category>opensource</category>
    </item>
    <item>
      <title>How to use RAG Production Automation Kit</title>
      <dc:creator>Autonomous Studio</dc:creator>
      <pubDate>Tue, 17 Nov 2026 02:52:57 +0000</pubDate>
      <link>https://dev.to/autonomousstudio/how-to-use-rag-production-automation-kit-2fm1</link>
      <guid>https://dev.to/autonomousstudio/how-to-use-rag-production-automation-kit-2fm1</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;Retrieval‑augmented generation (RAG) has become the de‑facto pattern for building AI‑enabled SaaS platforms that need up‑to‑date, factual answers from large knowledge bases.  While the concept is simple—retrieve relevant chunks, feed them to a LLM, and let the model generate a response—the engineering effort to make it reliable, scalable, and maintainable can be daunting.  &lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;RAG Production Automation Kit&lt;/strong&gt; from VectorFlow abstracts away the repetitive plumbing and gives you a ready‑to‑run, production‑grade pipeline. In this tutorial we’ll walk through a typical use‑case, set up the kit, and programmatically turn a raw document collection into a live RAG service.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Problem
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Data ingestion&lt;/strong&gt; – Raw PDFs, markdown files, or API feeds must be chunked, embedded, and indexed.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retriever selection&lt;/strong&gt; – Choosing the right similarity search (e.g., dense vector vs. BM25) and tuning its parameters is non‑trivial.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt engineering&lt;/strong&gt; – You need a consistent prompt template that injects retrieved context without exceeding token limits.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Observability&lt;/strong&gt; – Production systems require logging, latency metrics, and fallback strategies.
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Doing all of this manually leads to duplicated code, hidden bugs, and a long time‑to‑value for your product team.&lt;/p&gt;




&lt;h2&gt;
  
  
  How the Automation Kit Solves It
&lt;/h2&gt;

&lt;p&gt;The kit bundles:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Data pipelines&lt;/strong&gt; that automatically chunk, embed (via any OpenAI‑compatible or local model), and push vectors to a configurable vector store.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retriever adapters&lt;/strong&gt; for popular stores (FAISS, Pinecone, Qdrant) with sane defaults and easy hyper‑parameter overrides.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt templates&lt;/strong&gt; that respect token budgets and expose a clean API for customisation.
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Telemetry hooks&lt;/strong&gt; (OpenTelemetry, Prometheus) baked into the request flow.
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All of this is orchestrated through a small declarative YAML file, letting you focus on business logic instead of glue code.&lt;/p&gt;




&lt;h2&gt;
  
  
  Setting Up the Environment
&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Prerequisite&lt;/strong&gt; – Python 3.9+, a virtual environment, and an API key for your embedding model (e.g., OpenAI, Cohere, or a local HuggingFace endpoint).&lt;br&gt;
&lt;/p&gt;
&lt;/blockquote&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 1️⃣ Create and activate a venv&lt;/span&gt;
python -m venv .venv
&lt;span class="nb"&gt;source&lt;/span&gt; .venv/bin/activate   &lt;span class="c"&gt;# Windows: .venv\Scripts\activate&lt;/span&gt;

&lt;span class="c"&gt;# 2️⃣ Install the kit and its dependencies&lt;/span&gt;
pip install rag-production-automation-kit
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Step 1 – Install the Kit
&lt;/h2&gt;

&lt;p&gt;The package is distributed on Gumroad, but the pip install pulls the latest public release. Verify the installation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python -c &lt;span class="s2"&gt;"import rag_kit; print(rag_kit.__version__)"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You should see a version string like &lt;code&gt;0.4.2&lt;/code&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 2 – Prepare Your Knowledge Base
&lt;/h2&gt;

&lt;p&gt;Place all source files (PDF, markdown, HTML) under a single directory, e.g., &lt;code&gt;./data/docs&lt;/code&gt;. The kit will walk the tree recursively.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# data_pipeline.yaml
&lt;/span&gt;&lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./data/docs&lt;/span&gt;
  &lt;span class="na"&gt;formats&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;pdf&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;md&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;html&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="na"&gt;chunking&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;strategy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;sliding_window&lt;/span&gt;
  &lt;span class="na"&gt;size&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;512&lt;/span&gt;          &lt;span class="c1"&gt;# tokens
&lt;/span&gt;  &lt;span class="na"&gt;overlap&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;64&lt;/span&gt;

&lt;span class="na"&gt;embedding&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;provider&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;openai&lt;/span&gt;
  &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;text-embedding-3-large&lt;/span&gt;
  &lt;span class="na"&gt;api_key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${OPENAI_API_KEY}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run the ingestion command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;rag-kit ingest data_pipeline.yaml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The command:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;extracts text,&lt;/li&gt;
&lt;li&gt;splits it into overlapping chunks,&lt;/li&gt;
&lt;li&gt;generates dense embeddings,&lt;/li&gt;
&lt;li&gt;writes vectors to the default FAISS index (&lt;code&gt;./index/faiss.index&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Step 3 – Configure the Retriever
&lt;/h2&gt;

&lt;p&gt;Create a lightweight config that tells the kit how to fetch the most relevant chunks at query time.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# retriever.yaml
&lt;/span&gt;&lt;span class="na"&gt;store&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;faiss&lt;/span&gt;
  &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./index/faiss.index&lt;/span&gt;

&lt;span class="na"&gt;search&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;top_k&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;
  &lt;span class="na"&gt;distance_metric&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;cosine&lt;/span&gt;
  &lt;span class="na"&gt;score_threshold&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.65&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can swap &lt;code&gt;faiss&lt;/code&gt; for &lt;code&gt;pinecone&lt;/code&gt; or &lt;code&gt;qdrant&lt;/code&gt; by changing the &lt;code&gt;type&lt;/code&gt; and providing the required connection credentials.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 4 – Build the Generation Pipeline
&lt;/h2&gt;

&lt;p&gt;Now wire the retriever to a language model and a prompt template. The kit supports any OpenAI‑compatible endpoint.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# rag_pipeline.yaml
&lt;/span&gt;&lt;span class="na"&gt;retriever&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;retriever.yaml&lt;/span&gt;

&lt;span class="na"&gt;generator&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;provider&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;openai&lt;/span&gt;
  &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;gpt-4o-mini&lt;/span&gt;
  &lt;span class="na"&gt;api_key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${OPENAI_API_KEY}&lt;/span&gt;
  &lt;span class="na"&gt;temperature&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.2&lt;/span&gt;
  &lt;span class="na"&gt;max_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt;

&lt;span class="na"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;
    &lt;span class="s"&gt;You are a helpful assistant for a SaaS knowledge base.&lt;/span&gt;
    &lt;span class="s"&gt;Use the following context to answer the question concisely.&lt;/span&gt;

    &lt;span class="na"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;retrieved_chunks&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="na"&gt;Question&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s"&gt;user_query&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="na"&gt;Answer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Launch the service locally:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;rag-kit serve rag_pipeline.yaml --host 0.0.0.0 --port 8000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The kit starts a FastAPI server exposing two endpoints:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;POST /query&lt;/code&gt; – body &lt;code&gt;{ "question": "…" }&lt;/code&gt; returns &lt;code&gt;{ "answer": "…", "sources": [...] }&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;GET /health&lt;/code&gt; – simple health check.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Step 5 – Deploy and Test
&lt;/h2&gt;

&lt;p&gt;For production you’ll likely containerise the service. The kit ships a Dockerfile that copies your configs and the index:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="k"&gt;FROM&lt;/span&gt; python:3.11-slim
&lt;span class="k"&gt;WORKDIR&lt;/span&gt; /app
&lt;span class="k"&gt;COPY&lt;/span&gt; . /app
&lt;span class="k"&gt;RUN&lt;/span&gt; pip install rag-production-automation-kit
&lt;span class="k"&gt;EXPOSE&lt;/span&gt; 8000
&lt;span class="k"&gt;CMD&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"rag-kit"&lt;/span&gt;, &lt;span class="s2"&gt;"serve"&lt;/span&gt;, &lt;span class="s2"&gt;"rag_pipeline.yaml"&lt;/span&gt;, &lt;span class="s2"&gt;"--host"&lt;/span&gt;, &lt;span class="s2"&gt;"0.0.0.0"&lt;/span&gt;, &lt;span class="s2"&gt;"--port"&lt;/span&gt;, &lt;span class="s2"&gt;"8000"&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Build and push:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker build -t vectorflow/rag-kit:latest .
docker push vectorflow/rag-kit:latest
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Deploy to your preferred orchestrator (Kubernetes, ECS, etc.). The kit automatically emits Prometheus metrics at &lt;code&gt;/metrics&lt;/code&gt; and logs structured JSON for easy ingestion into ELK or Loki.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick sanity test&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -X POST http://localhost:8000/query \
  -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
  -d &lt;span class="s1"&gt;'{"question":"How do I reset my API key?"}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You should receive a concise answer plus a list of source document IDs, enabling traceability for compliance.&lt;/p&gt;




&lt;h2&gt;
  
  
  Common Pitfalls &amp;amp; Tips
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;Likely Cause&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Empty &lt;code&gt;sources&lt;/code&gt; array&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;score_threshold&lt;/code&gt; too high&lt;/td&gt;
&lt;td&gt;Lower &lt;code&gt;score_threshold&lt;/code&gt; or increase &lt;code&gt;top_k&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Latency &amp;gt; 2 s&lt;/td&gt;
&lt;td&gt;Using a remote vector store without connection pooling&lt;/td&gt;
&lt;td&gt;Enable HTTP keep‑alive or switch to a local FAISS index for low‑latency workloads.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Token limit errors&lt;/td&gt;
&lt;td&gt;Prompt + retrieved chunks exceed model context&lt;/td&gt;
&lt;td&gt;Reduce &lt;code&gt;chunking.size&lt;/code&gt; or enable dynamic truncation (&lt;code&gt;rag-kit truncate&lt;/code&gt;).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Inconsistent answers across runs&lt;/td&gt;
&lt;td&gt;Non‑deterministic temperature&lt;/td&gt;
&lt;td&gt;Set &lt;code&gt;temperature: 0&lt;/code&gt; for deterministic responses in production.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;By following the steps above you’ve turned a raw document collection into a fully‑instrumented RAG service with just a few YAML files and a single command. The &lt;strong&gt;RAG Production Automation Kit&lt;/strong&gt; handles ingestion, retrieval, prompt construction, and observability, letting AI‑enabled SaaS platforms focus on product differentiation rather than infrastructure plumbing.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Next steps&lt;/strong&gt; – Explore the live demo to see the kit in action and grab the full package for deeper customisation:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://3km-collab.github.io/rag-production-automation-kit-10/" rel="noopener noreferrer"&gt;https://3km-collab.github.io/rag-production-automation-kit-10/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Package:&lt;/strong&gt; &lt;a href="https://autonomousstudio.gumroad.com/l/xzxbdn" rel="noopener noreferrer"&gt;https://autonomousstudio.gumroad.com/l/xzxbdn&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;strong&gt;Autonomous Studio — Self-hosted AI Micro-APIs &amp;amp; Developer Tools.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://autonomousstudio.gumroad.com/l/xzxbdn" rel="noopener noreferrer"&gt;Get the full package&lt;/a&gt; · &lt;a href="https://3km-collab.github.io/rag-production-automation-kit-10/" rel="noopener noreferrer"&gt;Live demo&lt;/a&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>datascience</category>
      <category>opensource</category>
    </item>
    <item>
      <title>How to use Production‑Ready RAG Automation Kit</title>
      <dc:creator>Autonomous Studio</dc:creator>
      <pubDate>Tue, 17 Nov 2026 02:48:17 +0000</pubDate>
      <link>https://dev.to/autonomousstudio/how-to-use-production-ready-rag-automation-kit-31lj</link>
      <guid>https://dev.to/autonomousstudio/how-to-use-production-ready-rag-automation-kit-31lj</guid>
      <description>&lt;h2&gt;
  
  
  Introduction: Why RAG Still Feels Like a Black‑Box
&lt;/h2&gt;

&lt;p&gt;Retrieval‑Augmented Generation (RAG) promises the best of both worlds – the factual grounding of a search engine and the fluency of a large language model. In practice, building a production‑grade RAG pipeline involves several moving parts:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Pain point&lt;/th&gt;
&lt;th&gt;Typical symptom&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Data ingestion&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Manual scripts that break when the source schema changes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Vector store management&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Stale embeddings, missing index refreshes, and costly re‑indexing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Prompt orchestration&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Hard‑coded prompts that don’t adapt to context or user feedback&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Observability&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No logs, metrics, or alerts – you only notice failures after users complain&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Scalability&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Single‑threaded retrieval that stalls under real‑world traffic&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For AI product teams, the overhead of wiring these pieces together often outweighs the value of the RAG feature itself. The &lt;strong&gt;Production‑Ready RAG Automation Kit&lt;/strong&gt; (PRRAK) from &lt;strong&gt;RAGForge&lt;/strong&gt; abstracts the plumbing, letting you focus on domain‑specific logic while keeping the stack maintainable and low‑maintenance.&lt;/p&gt;

&lt;p&gt;Below is a step‑by‑step guide to spin up a minimal yet production‑ready RAG service using the kit. The example assumes you have a Python 3.10+ environment and an OpenAI‑compatible LLM endpoint.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Set Up the Environment
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Create a fresh virtual environment&lt;/span&gt;
python -m venv .venv
&lt;span class="nb"&gt;source&lt;/span&gt; .venv/bin/activate   &lt;span class="c"&gt;# Windows: .venv\Scripts\activate&lt;/span&gt;

&lt;span class="c"&gt;# Install the automation kit and its dependencies&lt;/span&gt;
pip install ragforge-prrak&lt;span class="o"&gt;[&lt;/span&gt;all&lt;span class="o"&gt;]&lt;/span&gt;   &lt;span class="c"&gt;# pulls in vector DB, LLM client, and utils&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Tip:&lt;/strong&gt; The &lt;code&gt;[all]&lt;/code&gt; extra installs optional integrations (e.g., Pinecone, Weaviate). If you prefer a local SQLite‑based vector store, omit it and install only the core.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  2. Prepare Your Knowledge Source
&lt;/h2&gt;

&lt;p&gt;The kit works with any iterable of documents (plain text, Markdown, PDFs, etc.). For this tutorial we’ll use a small collection of Markdown files stored under &lt;code&gt;./knowledge/&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;ragforge&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prrak&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;DocumentLoader&lt;/span&gt;

&lt;span class="c1"&gt;# Load all *.md files recursively&lt;/span&gt;
&lt;span class="n"&gt;loader&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;DocumentLoader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source_path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"knowledge"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;documents&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;loader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"Loaded {len(documents)} documents"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;DocumentLoader&lt;/code&gt; normalizes line breaks, strips front‑matter, and returns a list of &lt;code&gt;Document&lt;/code&gt; objects containing &lt;code&gt;id&lt;/code&gt;, &lt;code&gt;text&lt;/code&gt;, and optional metadata.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Create the Vector Index
&lt;/h2&gt;

&lt;p&gt;PRRAK ships with a lightweight &lt;strong&gt;FAISS&lt;/strong&gt; wrapper that works out‑of‑the‑box. The following snippet creates embeddings using the configured LLM and persists the index to disk.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;ragforge&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prrak&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;EmbeddingEngine&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;VectorStore&lt;/span&gt;

&lt;span class="c1"&gt;# 1️⃣ Initialise the embedding engine (OpenAI by default)&lt;/span&gt;
&lt;span class="n"&gt;engine&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;EmbeddingEngine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"text-embedding-3-large"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# replace with your provider&lt;/span&gt;

&lt;span class="c1"&gt;# 2️⃣ Build the FAISS index&lt;/span&gt;
&lt;span class="n"&gt;store&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;VectorStore&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;embedding_engine&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;engine&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;persist_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"faiss_index"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;build&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;documents&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;          &lt;span class="c1"&gt;# this will embed and add all docs&lt;/span&gt;
&lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                    &lt;span class="c1"&gt;# writes the index to `faiss_index/`&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Vector store ready"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Why this matters:&lt;/strong&gt; The kit automatically batches embeddings, retries on transient failures, and stores the mapping between vector IDs and original document metadata – eliminating the most common source of “missing answers” bugs.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Define a Retrieval‑Augmented Prompt Template
&lt;/h2&gt;

&lt;p&gt;A good prompt separates the retrieval step from generation. PRRAK provides a &lt;code&gt;PromptBuilder&lt;/code&gt; that injects retrieved snippets into a user‑defined template.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;ragforge&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prrak&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;PromptBuilder&lt;/span&gt;

&lt;span class="n"&gt;template&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"""
You are an expert assistant for the company's internal knowledge base.
Answer the user's question using ONLY the provided excerpts. If the answer
cannot be derived, reply with "I don't have enough information."

Context:
{retrieved_chunks}

Question: {question}
Answer:
"""&lt;/span&gt;

&lt;span class="n"&gt;builder&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;PromptBuilder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;template&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;template&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The placeholders &lt;code&gt;{retrieved_chunks}&lt;/code&gt; and &lt;code&gt;{question}&lt;/code&gt; will be filled at runtime.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Wire Up the Retrieval‑Generation Loop
&lt;/h2&gt;

&lt;p&gt;Now we combine the vector store, the prompt builder, and an LLM client into a single &lt;code&gt;RagPipeline&lt;/code&gt; object.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;ragforge&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prrak&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;RagPipeline&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;LLMClient&lt;/span&gt;

&lt;span class="c1"&gt;# LLM client (OpenAI ChatCompletion by default)&lt;/span&gt;
&lt;span class="n"&gt;llm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LLMClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"gpt-4o-mini"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# swap for your own endpoint if needed&lt;/span&gt;

&lt;span class="c1"&gt;# Assemble the pipeline&lt;/span&gt;
&lt;span class="n"&gt;pipeline&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;RagPipeline&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;vector_store&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;prompt_builder&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;builder&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;llm_client&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;top_k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;                &lt;span class="c1"&gt;# number of chunks to retrieve per query&lt;/span&gt;
    &lt;span class="n"&gt;chunk_separator&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"\n---\n"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""One‑liner to get a RAG answer."""&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Quick sanity check&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"How do we onboard a new vendor?"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The pipeline performs these actions under the hood:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Embedding the query&lt;/strong&gt; → fast nearest‑neighbor search in FAISS.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chunk aggregation&lt;/strong&gt; → concatenates the top‑k results with the separator.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt rendering&lt;/strong&gt; → fills the template.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;LLM call&lt;/strong&gt; → streams the answer back (supports async if you need high throughput).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;All steps are instrumented with structured logs (JSON) and Prometheus‑compatible metrics, so you can plug them into Grafana or any observability stack without extra code.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Add Observability &amp;amp; Auto‑Refresh (Optional but Recommended)
&lt;/h2&gt;

&lt;h3&gt;
  
  
  6.1 Metrics
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;ragforge&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prrak&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;MetricsExporter&lt;/span&gt;

&lt;span class="n"&gt;exporter&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;MetricsExporter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;9464&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# Prometheus will scrape this endpoint&lt;/span&gt;
&lt;span class="n"&gt;exporter&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Metrics include &lt;code&gt;rag_query_latency_seconds&lt;/code&gt;, &lt;code&gt;rag_embeddings_total&lt;/code&gt;, and &lt;code&gt;rag_vector_store_hits&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  6.2 Periodic Index Refresh
&lt;/h3&gt;

&lt;p&gt;If your source documents change, schedule a background job:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;refresh_index&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;new_docs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;loader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;build&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;new_docs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;replace&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# rebuilds from scratch&lt;/span&gt;
    &lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Index refreshed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;every&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hours&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;do&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;refresh_index&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run_pending&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The kit’s &lt;code&gt;build(..., replace=True)&lt;/code&gt; safely swaps the on‑disk index without downtime.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Deploy as a Simple HTTP Service
&lt;/h2&gt;

&lt;p&gt;PRRAK includes a tiny FastAPI wrapper. Create &lt;code&gt;app.py&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;fastapi&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;HTTPException&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pydantic&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;BaseModel&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;pipeline&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;   &lt;span class="c1"&gt;# reuse the pipeline from step 5&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;

&lt;span class="nd"&gt;@app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/rag"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;rag_endpoint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;answer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"answer"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;HTTPException&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;uvicorn app:app --host 0.0.0.0 --port 8000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You now have a &lt;strong&gt;production‑ready&lt;/strong&gt; RAG endpoint that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Handles concurrent requests (FastAPI + Uvicorn workers)&lt;/li&gt;
&lt;li&gt;Returns structured JSON&lt;/li&gt;
&lt;li&gt;Emits logs and metrics automatically&lt;/li&gt;
&lt;li&gt;Can be containerised with a one‑line Dockerfile (the kit ships a ready‑made &lt;code&gt;Dockerfile&lt;/code&gt; in its repo)&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  8. Test Locally and Iterate
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -X POST http://localhost:8000/rag \
     -H &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; \
     -d &lt;span class="s1"&gt;'{"question":"What is the SLA for data backups?"}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the answer feels off, tweak:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;top_k&lt;/code&gt; (more context vs. token budget)&lt;/li&gt;
&lt;li&gt;Prompt wording (add “cite the source” or “keep answer under 150 words”)&lt;/li&gt;
&lt;li&gt;Retrieval chunk size (adjust &lt;code&gt;chunk_separator&lt;/code&gt; or pre‑chunk the source docs)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Because the pipeline is modular, you can swap any component (e.g., replace FAISS with Pinecone) without touching the surrounding code.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Going to Production
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Checklist Item&lt;/th&gt;
&lt;th&gt;Why It Matters&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;TLS termination&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Secure traffic to the endpoint&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Rate limiting&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Prevent LLM cost overruns&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Health checks&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Auto‑restart on failure (Kubernetes, ECS)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Secrets management&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Store API keys in Vault/Env‑vars, not code&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;CI/CD&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Run unit tests on the prompt builder and vector store logic&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All of these concerns are orthogonal to the core RAG logic and can be added around the FastAPI service.&lt;/p&gt;




&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;The &lt;strong&gt;Production‑Ready RAG Automation Kit&lt;/strong&gt; removes the repetitive glue code that usually blocks AI product teams. By following the steps above you get:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A reproducible data ingestion pipeline&lt;/li&gt;
&lt;li&gt;Automatic embedding, indexing, and refresh&lt;/li&gt;
&lt;li&gt;A clean separation of retrieval and generation via a templated prompt&lt;/li&gt;
&lt;li&gt;Built‑in observability and easy extensibility&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can now ship a reliable chatbot, knowledge‑base assistant, or enterprise search feature without reinventing the wheel.&lt;/p&gt;




&lt;h3&gt;
  
  
  Quick Links
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://3km-collab.github.io/production-ready-rag-automation-kit-9/" rel="noopener noreferrer"&gt;https://3km-collab.github.io/production-ready-rag-automation-kit-9/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Purchase the package:&lt;/strong&gt; &lt;a href="https://autonomousstudio.gumroad.com/l/gkdagd" rel="noopener noreferrer"&gt;https://autonomousstudio.gumroad.com/l/gkdagd&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Happy building!&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Autonomous Studio — Self-hosted AI Micro-APIs &amp;amp; Developer Tools.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://autonomousstudio.gumroad.com/l/gkdagd" rel="noopener noreferrer"&gt;Get the full package&lt;/a&gt; · &lt;a href="https://3km-collab.github.io/production-ready-rag-automation-kit-9/" rel="noopener noreferrer"&gt;Live demo&lt;/a&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>datascience</category>
      <category>opensource</category>
    </item>
    <item>
      <title>How to use Synthetic Credit Data API: synthetic credit data for safer ML</title>
      <dc:creator>Autonomous Studio</dc:creator>
      <pubDate>Tue, 17 Nov 2026 02:45:20 +0000</pubDate>
      <link>https://dev.to/autonomousstudio/how-to-use-synthetic-credit-data-api-synthetic-credit-data-for-safer-ml-5f4h</link>
      <guid>https://dev.to/autonomousstudio/how-to-use-synthetic-credit-data-api-synthetic-credit-data-for-safer-ml-5f4h</guid>
      <description>&lt;h2&gt;
  
  
  Introduction – Why Synthetic Credit Data Matters
&lt;/h2&gt;

&lt;p&gt;Fintech applications, credit‑scoring models, and risk‑management pipelines need realistic transaction and borrower data to train, test, and validate algorithms. Real customer data is often off‑limits because of privacy regulations (GDPR, CCPA) and internal policies. The result? Teams either work with tiny, sanitized datasets that don’t reflect production complexity, or they spend weeks fabricating mock data that still feels artificial.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;CredSynth’s Synthetic Credit Data API&lt;/strong&gt; solves this by generating on‑demand, statistically‑sound credit‑profile records that mimic real‑world distributions (age, income, credit utilization, payment history, etc.). The API is REST‑ful, returns JSON, and can be called from any language. In this tutorial we’ll:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Define a typical use‑case (building a credit‑scoring model).
&lt;/li&gt;
&lt;li&gt;Show how to request synthetic data programmatically.
&lt;/li&gt;
&lt;li&gt;Walk through data ingestion, basic validation, and a quick exploratory analysis.
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The steps are language‑agnostic, but the code examples use Python 3 and the popular &lt;code&gt;requests&lt;/code&gt; and &lt;code&gt;pandas&lt;/code&gt; libraries.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Set Up Your Environment
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Requirement&lt;/th&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Python 3.8+&lt;/td&gt;
&lt;td&gt;&lt;code&gt;python -m venv venv &amp;amp;&amp;amp; source venv/bin/activate&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;requests&lt;/code&gt; (HTTP client)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pip install requests&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;pandas&lt;/code&gt; (data handling)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pip install pandas&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Optional: Jupyter notebook for interactive exploration&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pip install notebook&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Create a new notebook or script file (e.g., &lt;code&gt;generate_credit_data.py&lt;/code&gt;). Import the libraries:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;pandas&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  2. Understand the API Contract
&lt;/h2&gt;

&lt;p&gt;CredSynth exposes a single endpoint:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="no"&gt;GET&lt;/span&gt; &lt;span class="n"&gt;https&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//api.credsynth.com/v1/records&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Query parameters&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Parameter&lt;/th&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Description&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;count&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;int&lt;/td&gt;
&lt;td&gt;Number of records to generate (max 10 000)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;count=500&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;seed&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;int&lt;/td&gt;
&lt;td&gt;Optional deterministic seed for reproducibility&lt;/td&gt;
&lt;td&gt;&lt;code&gt;seed=42&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;region&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;str&lt;/td&gt;
&lt;td&gt;Geographic region to bias demographics (&lt;code&gt;us&lt;/code&gt;, &lt;code&gt;eu&lt;/code&gt;, &lt;code&gt;asia&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;region=us&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;risk&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;str&lt;/td&gt;
&lt;td&gt;Desired risk profile (&lt;code&gt;low&lt;/code&gt;, &lt;code&gt;medium&lt;/code&gt;, &lt;code&gt;high&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;risk=medium&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Response&lt;/strong&gt; – JSON array where each element contains fields such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"customer_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"c7f3b9e2-1a4d-4f5c-9a2e-5d9c1a7b6f8d"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"age"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;34&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"annual_income"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;72000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"credit_score"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;680&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"outstanding_balance"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;12450.75&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"payment_history"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"us"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"risk_category"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"medium"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"generated_at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-17T12:34:56Z"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;payment_history&lt;/code&gt; array encodes the last five monthly payments (&lt;code&gt;1&lt;/code&gt; = on‑time, &lt;code&gt;0&lt;/code&gt; = missed).&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Fetch Synthetic Records
&lt;/h2&gt;

&lt;p&gt;Below is a reusable function that builds the request URL, handles errors, and returns a &lt;code&gt;pandas.DataFrame&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch_synthetic_credit_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;seed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"us"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;risk&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"medium"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DataFrame&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""
    Pulls synthetic credit records from CredSynth and returns a DataFrame.
    """&lt;/span&gt;
    &lt;span class="n"&gt;base_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"https://api.credsynth.com/v1/records"&lt;/span&gt;
    &lt;span class="n"&gt;params&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"count"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"region"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"risk"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;risk&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;seed&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"seed"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;seed&lt;/span&gt;

    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;RequestException&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"CredSynth request failed: {exc}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;

    &lt;span class="n"&gt;records&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# Convert list of dicts into a DataFrame&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DataFrame&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# Cast timestamps to datetime objects for easier handling&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"generated_at"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;to_datetime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"generated_at"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Example usage&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Generate 2,500 medium‑risk US profiles with a reproducible seed&lt;/span&gt;
&lt;span class="n"&gt;df_credit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch_synthetic_credit_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2500&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;seed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12345&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"us"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;risk&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"medium"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df_credit&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;head&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"Dataset shape: {df_credit.shape}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Running the snippet should print the first few rows and confirm you have 2,500 rows.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Validate the Data
&lt;/h2&gt;

&lt;p&gt;Even synthetic data can contain anomalies (e.g., negative income). Perform a quick sanity check before feeding it into downstream pipelines.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;validate_credit_dataframe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DataFrame&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Basic type checks&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"age"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;between&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Age out of realistic bounds"&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"annual_income"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Negative income detected"&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"credit_score"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;between&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;850&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Credit score out of range"&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"outstanding_balance"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Negative balance found"&lt;/span&gt;

    &lt;span class="c1"&gt;# Payment history must be a list of length 5 with 0/1 values&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;hist&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"payment_history"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hist&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hist&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Invalid payment history format"&lt;/span&gt;
        &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nb"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;hist&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Payment history contains non‑binary values"&lt;/span&gt;

    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"✅ Validation passed – data looks clean."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nf"&gt;validate_credit_dataframe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df_credit&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If any assertion fails, the function raises an informative error, allowing you to adjust the request parameters (e.g., switch &lt;code&gt;risk&lt;/code&gt; from &lt;code&gt;high&lt;/code&gt; to &lt;code&gt;medium&lt;/code&gt;).&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Quick Exploratory Analysis
&lt;/h2&gt;

&lt;p&gt;A typical first step for a data scientist is to understand the distribution of key variables.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;matplotlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pyplot&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;plt&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;seaborn&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;sns&lt;/span&gt;

&lt;span class="c1"&gt;# Set a consistent style&lt;/span&gt;
&lt;span class="n"&gt;sns&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;style&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"whitegrid"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 1️⃣ Age distribution&lt;/span&gt;
&lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;figure&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;figsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;sns&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;histplot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df_credit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"age"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bins&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kde&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;color&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"#4c72b0"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;title&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Age Distribution of Synthetic Borrowers"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;xlabel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Age"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 2️⃣ Credit score vs. income&lt;/span&gt;
&lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;figure&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;figsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;sns&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;scatterplot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;df_credit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"annual_income"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"credit_score"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;hue&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"risk_category"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;palette&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"viridis"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.7&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;title&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Credit Score vs. Annual Income"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;xlabel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Annual Income (USD)"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ylabel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Credit Score"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;legend&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Risk"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;plt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;show&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These plots give you immediate insight into whether the synthetic population matches the characteristics you expect for your model’s target market.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Integrate Into a Model Training Pipeline
&lt;/h2&gt;

&lt;p&gt;Below is a minimal example of how you could feed the synthetic data into a scikit‑learn logistic regression model that predicts whether a borrower will default in the next month (simplified to &lt;code&gt;payment_history[-1] == 0&lt;/code&gt;).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sklearn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model_selection&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;train_test_split&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sklearn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;preprocessing&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;StandardScaler&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sklearn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;linear_model&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;LogisticRegression&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sklearn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metrics&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;classification_report&lt;/span&gt;

&lt;span class="c1"&gt;# Target: missed payment in the most recent month&lt;/span&gt;
&lt;span class="n"&gt;df_credit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"default_next_month"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df_credit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"payment_history"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;apply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Features – we’ll use a subset for demonstration&lt;/span&gt;
&lt;span class="n"&gt;feature_cols&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"age"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"annual_income"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"credit_score"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"outstanding_balance"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="no"&gt;X&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df_credit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;feature_cols&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df_credit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"default_next_month"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="c1"&gt;# Train‑test split&lt;/span&gt;
&lt;span class="nc"&gt;X_train&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;X_test&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y_train&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y_test&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;train_test_split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;test_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;0.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;random_state&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stratify&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Scale numeric features&lt;/span&gt;
&lt;span class="n"&gt;scaler&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;StandardScaler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nc"&gt;X_train_scaled&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;scaler&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fit_transform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;X_train&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nc"&gt;X_test_scaled&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;scaler&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transform&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;X_test&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Fit logistic regression&lt;/span&gt;
&lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LogisticRegression&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_iter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;class_weight&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"balanced"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;X_train_scaled&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y_train&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Evaluate&lt;/span&gt;
&lt;span class="n"&gt;y_pred&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;predict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;X_test_scaled&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;classification_report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;y_test&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y_pred&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;target_names&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"No Default"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Default"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Even though the data is synthetic, the workflow mirrors a production pipeline: fetch → validate → explore → train → evaluate. Swap the &lt;code&gt;fetch_synthetic_credit_data&lt;/code&gt; call with a real data source when you’re ready to move to production.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Automate Periodic Refreshes (Optional)
&lt;/h2&gt;

&lt;p&gt;For QA teams that need fresh data on each CI run, wrap the fetch logic in a small script and expose it as a command‑line tool.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"__main__"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;argparse&lt;/span&gt;

    &lt;span class="n"&gt;parser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;argparse&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ArgumentParser&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Download synthetic credit data"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"-c"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"--count"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;help&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Number of records"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"-s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"--seed"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;help&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Random seed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"-r"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"--region"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"us"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;help&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Geographic region"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_argument&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"-k"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"--risk"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"medium"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;help&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"Risk profile"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;parser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse_args&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch_synthetic_credit_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;seed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;seed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;region&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;risk&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;risk&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;out_path&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"synthetic_credit_{datetime.now():%Y%m%d_%H%M%S}.csv"&lt;/span&gt;
    &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;to_csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out_path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"✅ Saved {len(df)} records to {out_path}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Add this script to your repository, and your CI pipeline can invoke it before running integration tests that depend on realistic credit data.&lt;/p&gt;




&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Synthetic credit data removes the legal and logistical hurdles of using real customer records while still giving fintech developers, data scientists, and QA engineers a high‑fidelity sandbox. By programmatically pulling data from CredSynth’s API, validating it, and feeding it into standard analytics or machine‑learning pipelines, you can iterate faster and maintain compliance.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://3km-collab.github.io/synthetic-credit-data-api-1/" rel="noopener noreferrer"&gt;https://3km-collab.github.io/synthetic-credit-data-api-1/&lt;/a&gt;&lt;br&gt;&lt;br&gt;
&lt;strong&gt;Package (downloadable SDK &amp;amp; extended docs):&lt;/strong&gt; &lt;a href="https://autonomousstudio.gumroad.com/l/icmdfe" rel="noopener noreferrer"&gt;https://autonomousstudio.gumroad.com/l/icmdfe&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;Autonomous Studio — Self-hosted AI Micro-APIs &amp;amp; Developer Tools.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://autonomousstudio.gumroad.com/l/icmdfe" rel="noopener noreferrer"&gt;Get the full package&lt;/a&gt; · &lt;a href="https://3km-collab.github.io/synthetic-credit-data-api-1/" rel="noopener noreferrer"&gt;Live demo&lt;/a&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>datascience</category>
      <category>fintech</category>
    </item>
  </channel>
</rss>
