DEV Community

Choosing the Right RAG Strategy A Complete Decision Guide to Chunking, Agentic RAG, and GraphRAG

Seenivasa Ramadurai — Wed, 20 May 2026 21:37:54 +0000

Introduction

Here is a scenario many RAG builders know well, you wire up a pipeline, load your documents, ask a question and the answer is wrong, vague, or confidently hallucinated. The information was right there in your knowledge base. So what went wrong?

In most cases the problem is not your embedding model. It is not your LLM. It is how you cut up your documents before storing them the under appreciated craft called chunking and whether the retrieval architecture you chose actually matches the complexity of your queries.

This blog walks you through every major chunking strategy, explains how retrieval and augmentation work on top of those chunks, covers two advanced architectures Agentic RAG and GraphRAG and most importantly gives you a complete decision framework so you can walk away knowing exactly which combination fits your use case.

🐘 The Elephant & The LEGO Pieces

Your document is an elephant.

A 200+ pages of legal contract, a dense research paper, a massive product manual, or years of enterprise knowledge large, complex, interconnected, and full of valuable information.

A Large Language Model cannot effectively consume the entire elephant at once because of:

Context window limitations
Retrieval precision constraints
Latency considerations
Token cost optimization
Context dilution and retrieval noise

So the elephant must be divided into smaller pieces.

But this is where most RAG systems fail.

If you cut the elephant randomly, you destroy meaning.
Sentences lose context. Ideas become fragmented. Relationships disappear. Retrieval quality collapses.

Good chunking is not about making text smaller.
It is about preserving meaning while making retrieval efficient.

That is why chunking is better understood as turning the elephant into LEGO pieces.

LEGO pieces are:

Modular — each piece can stand on its own
Structured — pieces connect cleanly to related pieces
Consistent — standardized enough for reliable retrieval
Meaningful — each piece preserves semantic value
Composable — you assemble only the pieces needed for the task

Good chunking works the same way.

A well designed chunk should preserve structure, semantics, relationships, and surrounding context while remaining small enough for efficient retrieval and generation.

The real goal of chunking in RAG systems is not simply splitting documents.

Chunking is not simply about making documents smaller.

The actual goals are:

Preserve semantic meaning
Improve retrieval precision
Reduce hallucinations
Optimize context windows
Improve grounding quality
Balance latency and cost

In practice:

Better chunks lead to better retrieval, better prompts, and better answers.

The goal is to retrieve:

the right piece,
with the right context,
from the right section,
at the right time.

That is the foundation of effective Retrieval Augmented Generation (RAG).

The RAG Pipeline:End to End

Every RAG system regardless of complexity follows the same four stage flow. Understanding each stage makes chunking and architecture decisions obvious rather than arbitrary.

Stage 1: Document

Your raw source material: PDFs, Word files, web pages, transcripts, database exports. Too large to pass directly to an LLM. Needs to be broken into chunks before it can be indexed or searched.

Stage 2: Chunking and Embedding

Documents are cut into units and each unit is converted into a vector embedding a numerical representation of its meaning. These embeddings are stored in a vector database and form your searchable index. Your chunking strategy here determines everything that follows.

Stage 3: Retrieval

When a user asks a question, the query is also embedded. The vector database returns the chunks whose embeddings are closest in meaning to the query. These are your retrieved LEGO pieces.

Stage 4: Augmentation and Generation

The retrieved chunks along with surrounding parent context are assembled into a prompt and sent to the LLM. The model generates an accurate, grounded answer from the material it receives.

Core insight: The quality of your answer is bounded by retrieval quality, which is bounded by chunk quality. Better chunks → better retrieval → better answers. Every architectural decision downstream is built on this foundation.

1. Fixed-Size Chunking

The simplest and most widely used strategy. Documents are split into equal sized blocks by token count, character count, or word count without regard for meaning, sentence boundaries, or document structure.

LangChain Methods
CharacterTextSplitter: splits on a single separator (default \n\n), then enforces chunk_size by character count.

TokenTextSplitter: splits by token count using a tokenizer (e.g. tiktoken for OpenAI models); more accurate for LLM context budgets than character based splitting.

from langchain.text_splitter import CharacterTextSplitter, TokenTextSplitter


# Character-based
splitter = CharacterTextSplitter(
    chunk_size=1000,    # max characters per chunk
    chunk_overlap=200,  # characters repeated at chunk boundaries
    separator="\n\n"
)

# Token-based
splitter = TokenTextSplitter(
    chunk_size=512,  # max tokens per chunk
    chunk_overlap=50 # tokens repeated at chunk boundaries
)

Overlap guidance: A 10–20% overlap is typical. For chunk_size=1000, set chunk_overlap between 100–200. Overlap reduces the risk of a relevant answer being split across two chunks, at the cost of minor redundancy.

Strengths: Simple to implement, fast, predictable, easy to scale.
Weaknesses: Frequently breaks sentences mid-way, degrading semantic continuity and retrieval quality on complex documents.
Best for: Logs, telemetry, JSON, CSV, and other uniform structured content.

2. Recursive Chunking

Rather than splitting blindly, recursive chunking respects natural document structure. It works down a priority list of separators — \n\n, then \n, then . / ! / ?, then spaces — only moving to a finer separator when a chunk still exceeds the size limit.
This is the recommended default strategy in LangChain for most document types.

LangChain Methods
RecursiveCharacterTextSplitter: The primary implementation; tries each separator in the list before falling back to the next.

RecursiveCharacterTextSplitter.from_language(): pre-configured separator lists for specific programming languages (Python, JS, Markdown, HTML, etc.).

from langchain.text_splitter import RecursiveCharacterTextSplitter, Language

# General prose
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150,
    separators=["\n\n", "\n", ".", "!", "?", " ", ""]
)

# Language-aware (e.g. Python source code)
splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON,
    chunk_size=1000,
    chunk_overlap=100
)

Overlap guidance: 10–15% overlap works well for most prose. For code, keep overlap low (50–100 tokens) to avoid duplicating function signatures across chunks.

Strengths: Better semantic retention than fixed size chunking; good general-purpose strategy; improves retrieval coherence.

Weaknesses: Structure aware rather than meaning aware; performance depends on document formatting quality.

Best for: Documentation, PDFs, articles, knowledge bases, and web pages.

3. Semantic Chunking

Instead of asking how large should the chunk be, semantic chunking asks which sentences belong together.
Sentences are converted into vector embeddings, similarity is measured between adjacent sentences, and chunk boundaries are drawn where similarity drops below a threshold — indicating a topic transition.

LangChain Methods

SemanticChunker (from langchain_experimental) — supports three breakpoint detection strategies: percentile, standard_deviation, and interquartile.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

splitter = SemanticChunker(
    embeddings=OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",  # or "standard_deviation", "interquartile"
    breakpoint_threshold_amount=95           # top 5% of similarity drops become boundaries
)

Overlap guidance: Semantic chunking does not use a fixed chunk_overlap boundaries are drawn on meaning, so overlapping would undermine the approach. If continuity is needed at boundaries, consider appending the last sentence of the previous chunk manually.

Strengths: High retrieval relevance; strong semantic continuity; well-suited to precision-sensitive systems.

Weaknesses: Computationally expensive; requires an embedding model at chunking time; similarity thresholds need tuning per dataset.

Best for: Enterprise knowledge systems, research platforms, policy documents, and AI assistants requiring contextual precision.

4. Hierarchical Chunking

Creates two levels of chunks: large parent chunks for context, and smaller child chunks for precision.

Retrieval targets the child level to find relevant passages, then expands to the parent level to return surrounding context. This directly addresses the core RAG trade off: small chunks improve precision, large chunks preserve context.

LangChain Methods

ParentDocumentRetriever: stores parent chunks in a document store and child chunks in a vector store, then links them at retrieval time.

from langchain.retrievers import ParentDocumentRetriever
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.storage import InMemoryStore
from langchain_community.vectorstores import Chroma

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)  # large context chunks
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)    # precise retrieval chunks

retriever = ParentDocumentRetriever(
    vectorstore=Chroma(embedding_function=embeddings),
    docstore=InMemoryStore(),
    child_splitter=child_splitter,
    parent_splitter=parent_splitter
)

Overlap guidance: Apply overlap only on the child splitter (typically 10–15%). Parent chunks are retrieved wholesale for context, so overlap there adds noise rather than value.

Strengths: Strong retrieval precision without sacrificing context; effective for long documents.

Weaknesses: More complex to index and retrieve; requires additional storage and orchestration.

Best for: Legal documents, technical manuals, books, enterprise documentation, and compliance systems.

5. Structure and Metadata Aware Chunking

Uses the document's own structure titles, headers, sections, tables, and page layout as natural chunk boundaries rather than treating the document as plain text.
Especially important for enterprise PDFs and structured reports, where layout carries semantic meaning that arbitrary splits would destroy.

LangChain Methods

MarkdownHeaderTextSplitter: splits on Markdown heading levels and attaches header text as metadata to each chunk.

HTMLHeaderTextSplitter: same pattern for HTML documents, splitting on '<h1>-<h4>' tags.

from langchain.text_splitter import MarkdownHeaderTextSplitter, HTMLHeaderTextSplitter

# Markdown
md_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[
        ("#",   "h1"),
        ("##",  "h2"),
        ("###", "h3"),
    ]
)
chunks = md_splitter.split_text(markdown_text)
# Each chunk carries metadata: {"h1": "Section Title", "h2": "Subsection"}

# HTML
html_splitter = HTMLHeaderTextSplitter(
    headers_to_split_on=[("h1", "h1"), ("h2", "h2")]
)

Overlap guidance: These splitters produce structurally bounded chunks rather than size bounded ones. If downstream chunks are still too large, pipe the output into a RecursiveCharacterTextSplitter with a modest overlap (100–150 characters) as a second pass.

Strengths: Preserves layout semantics; keeps tables intact; improves retrieval quality for structured enterprise documents.

Weaknesses: Requires a capable document parser; parser quality directly limits performance.

Best for: Financial reports, compliance documents, technical PDFs, medical documentation, and enterprise records.

6. Hybrid Chunking

Applies different chunking strategies based on content type within the same corpus fixed-size for logs, recursive for documentation, semantic for research papers, structure aware for Markdown or HTML.
LangChain does not have a dedicated hybrid splitter. Hybrid pipelines are composed manually using the building blocks above.

from langchain.text_splitter import (
    TokenTextSplitter,
    RecursiveCharacterTextSplitter,
    MarkdownHeaderTextSplitter,
)
from langchain_experimental.text_splitter import SemanticChunker

def hybrid_chunk(doc):
    content_type = doc.metadata.get("type")

    if content_type == "log":
        return TokenTextSplitter(
            chunk_size=512, chunk_overlap=0
        ).split_documents([doc])

    elif content_type == "markdown":
        return MarkdownHeaderTextSplitter(
            headers_to_split_on=[("#", "h1"), ("##", "h2")]
        ).split_text(doc.page_content)

    elif content_type == "research":
        return SemanticChunker(
            embeddings=embeddings,
            breakpoint_threshold_type="percentile"
        ).split_documents([doc])

    else:
        return RecursiveCharacterTextSplitter(
            chunk_size=1000, chunk_overlap=150
        ).split_documents([doc])

Overlap guidance: Set overlap per strategy based on content type. Logs and structured data: zero or minimal overlap. Prose and documentation: 10–15%. Code: 5–10%.

Strengths: Flexible and adaptable; better performance across mixed-content corpora.

Weaknesses: Higher engineering complexity; harder to evaluate and tune consistently.

Best for: Enterprise AI platforms, large mixed content corpora, knowledge management systems, and multi source RAG pipelines.

7. Agentic Chunking

An emerging approach where an LLM dynamically determines what information belongs together, how chunks should be formed, and how retrieval should adapt to user intent. This transforms chunking from static preprocessing into query aware reasoning at inference time.
LangChain supports this through its agent and chain abstractions rather than a dedicated splitter class.

from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain_openai import ChatOpenAI
import json

llm = ChatOpenAI(model="gpt-4o", temperature=0)

prompt = PromptTemplate.from_template("""
You are a document analyst. Split the following text into coherent topical sections.
Return ONLY a JSON list of objects, each with a "title" and "content" key.

Text:
{text}
""")

chain = LLMChain(llm=llm, prompt=prompt)

def agentic_chunk(text):
    result = chain.run(text=text)
    return json.loads(result)

Overlap guidance: Not applicable in the traditional sense the LLM determines boundaries based on meaning. To preserve continuity between sections, include a brief summary of the prior section in the prompt context.

Strengths: Highly adaptive; strong semantic preservation; query aware retrieval.

Weaknesses: Higher compute cost and latency; requires orchestration and guardrails; not yet widely proven in production at scale.

Best for: AI copilots, multi-agent systems, research assistants, and enterprise reasoning workflows.

8. Agentic RAG

Not to be confused with Agentic Chunking (#7).

Agentic Chunking is about how documents are split at index time. Agentic RAG is about how an LLM decides what to retrieve at query time and whether what it found is good enough to answer with.

Standard RAG pipelines are static: a query comes in, a fixed retrieval step runs, the top-k chunks are passed to the LLM, and an answer comes out. Agentic RAG breaks that linearity. An LLM agent decides when to retrieve, what to search for, whether the results are sufficient, and whether to re-query with a refined question before generating an answer.

Common patterns built on this idea include Corrective RAG (CRAG) which scores retrieved documents for relevance and falls back to a web search if they are poor and Self-RAG, where the LLM reflects on its own output and decides whether it needs to retrieve again.

LangChain Methods
create_retriever_tool wraps any retriever as a tool an agent can call on demand.

AgentExecutor the classic LangChain agent loop; the agent decides which tools to call and when.

LangGraph — the recommended approach for production Agentic RAG; models retrieval as a stateful graph of nodes (retrieve → grade → rewrite → retrieve again) with explicit conditional edges.

from langchain.tools.retriever import create_retriever_tool
from langchain_openai import ChatOpenAI
from langgraph.graph import StateGraph, END
from typing import TypedDict, List
from langchain_core.messages import BaseMessage

llm = ChatOpenAI(model="gpt-4o", temperature=0)

# Wrap retriever as a tool
retriever_tool = create_retriever_tool(
    retriever=vector_store.as_retriever(search_kwargs={"k": 5}),
    name="search_documents",
    description="Search the knowledge base for relevant information."
)

# --- LangGraph: Corrective RAG pattern ---

class AgentState(TypedDict):
    question: str
    documents: List[str]
    generation: str
    rewrite_count: int

def retrieve(state: AgentState):
    docs = vector_store.similarity_search(state["question"], k=5)
    return {"documents": docs}

def grade_documents(state: AgentState):
    # LLM scores each doc for relevance; filters out poor ones
    prompt = f"Is this document relevant to the question '{state['question']}'? Answer yes or no.\n\n{{doc}}"
    relevant = [
        doc for doc in state["documents"]
        if "yes" in llm.invoke(prompt.format(doc=doc.page_content)).content.lower()
    ]
    return {"documents": relevant}

def rewrite_query(state: AgentState):
    # If docs were poor, rewrite the question before re-retrieving
    rewritten = llm.invoke(
        f"Rewrite this question to improve retrieval: {state['question']}"
    ).content
    return {"question": rewritten, "rewrite_count": state["rewrite_count"] + 1}

def generate(state: AgentState):
    context = "\n\n".join(d.page_content for d in state["documents"])
    answer = llm.invoke(f"Answer using this context:\n{context}\n\nQuestion: {state['question']}").content
    return {"generation": answer}

def should_rewrite(state: AgentState):
    if len(state["documents"]) == 0 and state["rewrite_count"] < 2:
        return "rewrite"
    return "generate"

# Build the graph
workflow = StateGraph(AgentState)
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade", grade_documents)
workflow.add_node("rewrite", rewrite_query)
workflow.add_node("generate", generate)

workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade")
workflow.add_conditional_edges("grade", should_rewrite, {"rewrite": "rewrite", "generate": "generate"})
workflow.add_edge("rewrite", "retrieve")
workflow.add_edge("generate", END)

app = workflow.compile()
result = app.invoke({"question": "What are the risks of GraphRAG?", "rewrite_count": 0})

Overlap guidance: Overlap is set on the underlying retriever's chunking strategy — not on the agent itself. The agent layer operates above chunking. Use whatever overlap matches the chunking strategy feeding the vector store (typically 10–15% for recursive or fixed-size chunks).

Strengths: Handles multi-step and ambiguous queries that single-pass retrieval fails on; self-corrects when initial retrieval is poor; can combine multiple retrieval sources (vector DB, web search, SQL) in one query cycle.

Weaknesses: Higher latency per query due to multiple LLM calls; harder to debug than a linear pipeline; requires careful graph design to avoid infinite retrieval loops.

Best for: Complex Q&A systems, enterprise copilots where queries are open-ended, research assistants, and any pipeline where retrieval quality is highly variable.

9. GraphRAG

GraphRAG, originally developed by Microsoft Research, moves beyond treating documents as flat text sequences. Instead of chunking text into linear passages, it extracts entities and relationships from documents and stores them as a knowledge graph. Retrieval then traverses the graph to answer questions that require connecting information across multiple sources or document sections — something vector search alone handles poorly.

There are two primary retrieval modes: local search, which answers specific entity-level questions by traversing nearby graph nodes, and global search, which synthesizes themes across the entire corpus using community summaries generated at indexing time.

LangChain Methods

LangChain integrates with graph databases (Neo4j, Amazon Neptune, ArangoDB) and provides tooling to build graph-based RAG pipelines.
LLMGraphTransformer uses an LLM to extract entities and relationships from text and convert them into graph documents.

Neo4jGraph + GraphCypherQAChain store the graph in Neo4j and query it in natural language via generated Cypher queries.
Neo4jVector — hybrid approach that combines vector similarity search with graph traversal on a Neo4j backend.

from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_community.graphs import Neo4jGraph
from langchain.chains import GraphCypherQAChain
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o", temperature=0)

# Step 1: Extract entities and relationships from chunks
transformer = LLMGraphTransformer(llm=llm)
graph_docs = transformer.convert_to_graph_documents(documents)

# Step 2: Store in Neo4j
graph = Neo4jGraph(
    url="bolt://localhost:7687",
    username="neo4j",
    password="password"
)
graph.add_graph_documents(graph_docs)

# Step 3: Query the graph in natural language
chain = GraphCypherQAChain.from_llm(
    llm=llm,
    graph=graph,
    verbose=True,
    return_intermediate_steps=True
)
response = chain.invoke({"query": "Which authors collaborated with researchers at MIT?"})
For hybrid vector + graph retrieval:
pythonfrom langchain_community.vectorstores import Neo4jVector
from langchain_openai import OpenAIEmbeddings

# Store chunks as vectors alongside the graph
vector_store = Neo4jVector.from_documents(
    documents,
    embedding=OpenAIEmbeddings(),
    url="bolt://localhost:7687",
    username="neo4j",
    password="password",
    index_name="document_chunks",
    node_label="Chunk",
    embedding_node_property="embedding"
)

retriever = vector_store.as_retriever(search_kwargs={"k": 5})

Overlap guidance: GraphRAG does not rely on chunk overlap for continuity — relationships between entities bridge that gap structurally. When pre-chunking documents before graph extraction, use a RecursiveCharacterTextSplitter with modest overlap (100–150 characters) to ensure entity mentions near chunk boundaries are captured in at least one chunk before the LLM extracts them.

Strengths: Excels at multi-hop reasoning (e.g. "find all projects involving X that also relate to Y"); surfaces cross-document relationships invisible to vector search; global search enables corpus-wide thematic synthesis.

Weaknesses: Significantly higher indexing cost and complexity; graph quality depends on LLM extraction accuracy; Cypher query generation can be brittle on complex schemas; not well-suited to simple factual lookups where vector search is faster and cheaper.

Best for: Knowledge graphs, research corpora, compliance and regulatory systems, enterprise wikis with dense cross-references, and any domain where answering questions requires connecting facts across multiple documents.

The Core Trade-Off

A common misconception is that smaller chunks always improve retrieval. In practice, chunks that are too small lose context, fragment meaning, and can increase hallucinations.

Chunking is a balancing act across four competing factors:

There is no universally optimal strategy. The right choice depends on your data characteristics, query patterns, retrieval architecture, and business requirements.

Quick Reference

Final Thoughts

The strongest production RAG systems rarely rely on a single chunking strategy. A robust architecture typically combines:

Recursive chunking for general prose
Semantic chunking for precision-sensitive content
Hierarchical retrieval for long or dense documents
Structure-aware parsing for enterprise PDFs
Hybrid orchestration where content types vary

As enterprise AI matures, retrieval architecture is becoming just as important as model selection. And intelligent retrieval begins with intelligent chunking.

The Egregious Cost of Compliance: One Platform's Overly Broad Restrictions

Faith Sithole — Wed, 20 May 2026 21:37:36 +0000

The Problem We Were Actually Solving

We were actually trying to solve the classic problem of onboarding new creators. We believed that by supporting PayPal and Stripe, we'd open the door for more users. In our minds, the added security and flexibility these payment processors offered would be a huge win for our platform. But what we failed to consider was the fine print – or rather, the broad brushstrokes those payment processors use to restrict access.

What We Tried First (And Why It Failed)

We initially tried to get around these restrictions by manually adding creators' countries as exceptions. This approach didn't work for several reasons. Firstly, it was a manual, error-prone process that didn't scale. Secondly, the payment processors have an ever-changing list of restricted countries, making it difficult to stay up-to-date. Lastly, this approach compromised our platform's core principle of automating onboarding to make it easier for creators to start selling.

The Architecture Decision

In hindsight, our architecture decision was flawed. We prioritized integrating with well-known payment processors over addressing the underlying issue – the restrictions themselves. By not scrutinizing these restrictions, we inadvertently created a bottleneck in our onboarding process. This decision not only frustrated our creators but also exposed our platform to supply chain risks we'd rather not have.

What The Numbers Said After

We analyzed our onboarding data for a three-month period and found a stark contrast between the countries we restricted (and subsequently allowed) vs. those we didn't. Specifically, for a country like Nigeria, which has a significant number of digital creators, we saw a 40% reduction in successful onboarding due to PayPal's restrictions. When we finally switched to a more Africa-friendly payment processor, our success rate shot up by 65%.

What I Would Do Differently

In retrospect, we should have taken a different approach from the very beginning. We should have scrutinized the payment processors we chose, weighing their benefits against the restrictions they imposed. We should have also considered open-source alternatives that focus on inclusivity, such as Stripe's own competitor, Mollie, which has a more relaxed approach to country restrictions. Had we done so, we might have avoided the costly delays, reputational damage, and missed opportunities that ultimately came with trying to work around these restrictions.

GitHub Breach via VSCode Extension, ZTE Router CVE-2026-34472, & Public Repo Secrets Leaks

soy — Wed, 20 May 2026 21:36:39 +0000

GitHub Breach via VSCode Extension, ZTE Router CVE-2026-34472, & Public Repo Secrets Leaks

Today's Highlights

Today's security news highlights a significant GitHub internal breach traced to a compromised VSCode extension, underscoring supply chain risks. Additionally, a new CVE affects ZTE routers with pre-auth credential exposure, alongside a prominent example of critical secrets exposed in public GitHub repositories.

GitHub hit by a compromised VSCode extension (r/netsec)

Source: https://reddit.com/r/netsec/comments/1tiiyxq/github_hit_by_a_compromised_vscode_extension/

The tech community is abuzz with news of a significant internal breach at GitHub, reportedly stemming from a compromised VSCode extension. This incident allowed unauthorized access to GitHub's internal repositories, showcasing a critical vulnerability in the software supply chain. Attackers exploited this vector to potentially exfiltrate sensitive data, including an unspecified number of private repositories, although GitHub has stated there's no current evidence of impact to customer information. This marks a concerning development as it targets the very tools developers rely on daily.

This breach underscores the escalating risks associated with developer tooling, where malicious extensions or compromised components can act as sophisticated entry points into corporate networks. The incident highlights the challenge of securing complex development environments and the need for rigorous vetting of all third-party dependencies, from development libraries to IDE plugins. Such supply chain attacks are increasingly common and bypass traditional perimeter defenses by targeting trusted components. Organizations must adopt a zero-trust mindset towards their development ecosystems to mitigate these evolving threats.

Comment: Developers must scrutinize VSCode extensions and other third-party tools for potential supply chain risks. This incident reinforces the need for robust endpoint security, strict access controls, and regular audits of integrated development environment (IDE) components to prevent similar breaches.

CVE-2026-34472: Pre-auth credential exposure and auth bypass in ZTE H188A V6 routers (r/netsec)

Source: https://reddit.com/r/netsec/comments/1timb3b/cve202634472_pre2auth_credential_exposure_and_auth/

A newly disclosed vulnerability, identified as CVE-2026-34472, has been found to impact ZTE H188A V6 routers, presenting a significant security risk through pre-authentication credential exposure and an authentication bypass. Detailed technical analysis indicates that a routing flaw is the core issue, allowing unauthenticated attackers to gain access to sensitive administrative interfaces and functions without requiring prior authentication. This critical flaw, if exploited, could grant full control over the affected router, enabling malicious actors to manipulate its configuration and compromise the network it serves.

Such a compromise on a network edge device like a home or small business router has severe implications. Attackers could manipulate network settings, redirect internet traffic to malicious sites, launch further attacks on internal networks, or even monitor sensitive communications passing through the device. This vulnerability emphasizes the critical importance of keeping networking hardware patched and regularly monitored, as these devices often represent the first line of defense against external threats and are frequently overlooked in regular security audits, despite their critical role in network security.

Comment: Users of ZTE H188A V6 routers should immediately check for and apply any available firmware updates to patch CVE-2026-34472. This vulnerability underscores the importance of regularly reviewing and updating network edge devices, as they often serve as critical entry points for attackers.

America's top cyber-defense agency left a GitHub repo open with with passwords, keys, tokens – and incredibly obvious filenames (r/cybersecurity)

Source: https://reddit.com/r/cybersecurity/comments/1ticxu3/americas_top_cyberdefense_agency_left_a_github/

In a concerning disclosure, a leading US cyber-defense agency was found to have inadvertently exposed critical secrets within a publicly accessible GitHub repository. The leaked data reportedly included passwords, API keys, and authentication tokens, further compounded by the use of 'incredibly obvious filenames' that clearly indicated their sensitive nature. This incident serves as a prominent real-world example of how even highly security-conscious organizations with robust cyber defenses can fall prey to fundamental misconfigurations and lapses in secrets management practices, impacting national security.

The implications of such an exposure are profound, potentially leading to widespread unauthorized access, compromise of internal systems, and severe data breaches for the affected agency and its partners. This event powerfully highlights that technical prowess in cybersecurity does not negate the need for stringent adherence to basic security hygiene, particularly in managing developer secrets across the software development lifecycle. It underscores the ongoing challenge of enforcing secure coding and deployment practices within large, complex organizations, emphasizing the human element in operational security.

Comment: This is a critical lesson for all developers and organizations: never store sensitive credentials directly in source code repositories, even private ones, and especially not in public ones. Implement robust secrets management solutions (like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) and automate credential rotation to mitigate this common and highly dangerous vulnerability.

Applied AI: From Agent Orchestration to Workflow Automation & Code Generation

soy — Wed, 20 May 2026 21:36:08 +0000

Applied AI: From Agent Orchestration to Workflow Automation & Code Generation

Today's Highlights

This week, we explore the practical applications of AI, from leveraging LLMs for business workflow automation to integrating AI agents into open-source projects. We also discuss the evolving role of AI in professional code generation, highlighting the need for structured approaches in development.

Open Source Contribution: Seeking AI Agent & FastAPI Projects (r/Python)

Source: https://reddit.com/r/Python/comments/1ti9dbd/open_source_contribution/

A developer is actively seeking open-source projects to contribute to, specifically mentioning AI agents, Python, and FastAPI backends. This highlights a growing demand for and active development in the realm of AI agent orchestration and robust Python tooling for applied AI systems. Projects in this space typically involve orchestrating Large Language Models (LLMs) to perform complex tasks, often requiring integration with external APIs, data sources (potentially RAG frameworks), and user interfaces.

Such AI agent systems built with Python and FastAPI serve as powerful backends for production-grade AI applications. Developers contribute to components like task planning modules, memory management, tool integration (e.g., for search, data analysis, or code execution), and API endpoints for seamless interaction. This focus on practical, deployable systems using popular Python frameworks is crucial for moving AI research into real-world workflows and production environments. Contributing to these projects offers hands-on experience in building the next generation of intelligent automation tools that can be easily containerized and scaled.

Comment: This signals a vibrant community building practical AI agent solutions. Diving into such a repo would be a fantastic way to learn about agent design and FastAPI deployment patterns firsthand.

Streamlining RV Rental Business with Claude AI for Workflow Automation (r/ClaudeAI)

Source: https://reddit.com/r/ClaudeAI/comments/1ti9xo5/claude_is_improving_my_rv_rental_business_but/

An RV rental business owner shares an experience of using Claude AI to significantly improve their operations, despite the owner feeling overwhelmed by the resulting workload. This anecdotal account demonstrates a clear applied use case for AI in workflow automation, where an LLM is directly impacting a real-world business process. The AI is likely handling tasks such as customer communication, answering frequently asked questions, managing inquiries, or even assisting with scheduling and booking logistics.

While the original post doesn't detail specific AI frameworks, this scenario is ripe for structured application using tools from our category focus. For instance, a RAG framework could be employed to allow Claude to query specific rental agreements, maintenance schedules, or localized rules, ensuring accurate and consistent responses. AI agents could be orchestrated to automate follow-up emails, update booking calendars, or flag urgent customer requests. The challenge of feeling 'worked to death' indicates the current ad-hoc integration of AI, underscoring the need for more sophisticated workflow automation and RPA patterns to truly scale and optimize such operations.

Comment: This is a great example of an LLM enabling tangible business benefits, but it also highlights where proper workflow automation frameworks could alleviate manual overhead and scale the solution.

Managing AI-Assisted Code Generation in Professional Developer Workflows (r/ClaudeAI)

Source: https://reddit.com/r/ClaudeAI/comments/1tis9s6/how_to_address_vibe_coding_at_the_professional/

The discussion revolves around the practice of 'vibe coding'—a term used to describe rapid, often improvisational, code generation with the assistance of AI in a professional setting. The user observed a colleague generating substantial code quickly (30 minutes, one shot) using AI without extensive prior planning. This highlights the transformative impact of AI on developer workflows, specifically in the realm of code generation.

Integrating AI-generated code effectively into production deployment patterns presents both opportunities and challenges. While AI accelerates development, questions arise regarding code quality, maintainability, testing, and compliance with architectural standards. Frameworks and best practices for integrating AI into the software development lifecycle (SDLC) are becoming critical. This includes implementing automated testing for AI-generated code, establishing clear review processes, and potentially leveraging specialized tools that can validate, refactor, or even explain the AI's output to ensure it aligns with project requirements and company standards. Addressing 'vibe coding' professionally means developing structured approaches to harness AI's power while mitigating potential risks to project integrity.

Comment: AI's impact on code generation is undeniable. The real work now is in building frameworks and processes to ensure this rapid output is robust, testable, and maintainable in professional environments.

SQLite Journaling on SMB, TypeGraph for SQL Graphs, Cross-Engine Migrations

soy — Wed, 20 May 2026 21:35:37 +0000

SQLite Journaling on SMB, TypeGraph for SQL Graphs, Cross-Engine Migrations

Today's Highlights

This week's highlights include advanced SQL techniques for graph queries in PostgreSQL and SQLite, deep dives into SQLite's journaling behavior on network shares, and a strategic framework for cross-database synchronization and migration.

TypeGraph: graph queries that compile to a single recursive CTE on Postgres/SQLite (r/PostgreSQL)

Source: https://reddit.com/r/PostgreSQL/comments/1tiszu2/typegraph_graph_queries_that_compile_to_a_single/

This insightful discussion highlights TypeGraph, a method for performing complex graph queries directly within traditional relational databases like PostgreSQL and SQLite. Instead of resorting to specialized graph databases, this approach leverages the power of SQL's recursive Common Table Expressions (CTEs) to model and query relationships. This allows developers to handle intricate data connections, such as inheritance in permissions systems, content relationships, or entities for Retrieval-Augmented Generation (RAG), using their existing SQL database infrastructure.

The core idea is to translate graph traversals into a single, efficient recursive CTE. This method simplifies data modeling by preserving optionality when the schema is uncertain, and it avoids the overhead and complexity of managing an additional graph database system. For those already deeply invested in the SQLite or PostgreSQL ecosystem, TypeGraph presents a powerful technique to extend the capabilities of their databases without introducing new dependencies, making the existing stack more versatile for various data relationship challenges.

Comment: This technique is powerful for avoiding specialized graph databases when your data is already in SQL. Mastering recursive CTEs unlocks new ways to model complex relationships directly within SQLite or Postgres, making your existing stack more versatile and efficient.

SQLite journal files on read-only SMB share — can they be redirected to a local path? (SQLite Forum)

Source: https://sqlite.org/forum/info/05b54df8420cdb25ebcbe507dcace4dc66dddf017db2594e13c2e1fcae6752c2

This forum discussion addresses a critical operational challenge for SQLite deployments: managing journal files, especially when the main database file resides on a read-only network share like SMB. SQLite's default transaction model relies on journal files (or WAL files) to ensure atomicity and durability, typically created alongside the database file. When the main directory is read-only, SQLite cannot create these temporary files, leading to operational failures or unexpected behavior.

The discussion explores solutions and workarounds for this embedded database pattern, focusing on how to redirect SQLite's temporary files and journal data to a writable local path. Techniques such as using the PRAGMA temp_store_directory or SQLITE_TEMP_STORE compile-time options are implicitly or explicitly considered. Understanding how SQLite handles journaling and temporary files is crucial for maintaining data integrity and achieving optimal performance in constrained or shared network environments, offering valuable insights for developers deploying SQLite in complex system architectures.

Comment: Anyone deploying SQLite in less-than-ideal network environments, particularly with shared storage, will appreciate this discussion. Understanding SQLite's journaling and temp file handling is crucial for robust and performant embedded applications, especially in read-only scenarios.

Architecting a 3-stage framework for cross-engine DB synchronization and migration. (r/database)

Source: https://reddit.com/r/Database/comments/1tizn6x/architecting_a_3stage_framework_for_crossengine/

This post delves into the complexities of modernizing legacy systems, specifically tackling the persistent headaches associated with database schema evolution and cross-engine synchronization. The author proposes a 3-stage framework designed to streamline this often-fraught process, aiming to reduce friction and improve reliability during significant database changes across disparate systems. The framework likely outlines distinct phases such as discovery, transformation, and application, providing a structured approach to what can otherwise be an chaotic undertaking.

Such a framework is invaluable for teams dealing with migration strategies and the development of robust data pipeline tools. It offers a blueprint for how to systematically manage changes when moving data between different database engines – for instance, migrating from an older system to PostgreSQL, or synchronizing data with DuckDB or SQLite in a microservices architecture. By formalizing the process, this architectural pattern helps ensure data consistency, minimize downtime, and ultimately achieve a smoother transition when modernizing database infrastructure.

Comment: This framework provides a methodical approach to tackle the often painful process of schema migration and data synchronization across disparate databases. It’s a great read for data engineers looking to standardize their migration pipelines and reduce operational headaches, especially when dealing with legacy systems.

How to Deploy a Virtual Machine in Azure

Noble — Wed, 20 May 2026 21:35:31 +0000

A virtual machine is a mini computer inside another computer in the cloud. Now we are deploying this virtual machine in an operating system called Linux. Linux is a free, open sourced operating system family used on computers, servers, smartphones and even supercomputers. Let me give you a hint, the very popular Android is built on the Linux Kernel.
Okay that's enough intro for now, let's get started.

Step 1: Login in to the[(https://portal.azure.com/#home)]

Step 2: Look up for virtual machine on the search bar just atop and start creating your Linux virtual machine.

select the +create icon which enables you to create a new VM

Step 3:I am assuming this is your first time creating a Linux virtual machine, it is best to create a new "resource group" as seen in the image below , one might wonder what a resource group is?. A resource group is the container that will be holding the virtual machine, so in layman terms the resource group is just like the "pot" and the Linux virtual machine is what we are cooking inside this pot.
so you create a new resource group, in my case I create one called "mylinuxvirtualmachinegroup" and I selected it, so create yours and select it.
Step 4: Configuring instance details
a: create a name for your VM that suits you
b: select a location that befits you
c: I chose the first option for the zone options, you can use the drop down and choose 'no infrastructure redundancy" required if you are on a free trial account. _
_ d: for security type; I used the Trusted launch virtual machine as this is basic security for Linux VM.
e: for Image; Ubuntu is encouraged as that is the operating system Linux works on so everything rhymes.

Step 5: Login method and Network access configuration

Secure Safe Shell SSH Key (More Secure)
Password (Simpler for first timers) The choice is yours, personally I would use password as its easier and not so demanding. Create a username (like azureadmin) Create a strong password (write it down!) Public inbound ports: Select Allow selected ports Select inbound ports: Choose HTTP (80) and SSH (22) scroll back to the monitoring tab to disable diagnostics after that click review and create

go to Resource to see, learn and understand everything about the Linux VM you just created

Step 6:Open the public IP address as shown

then click connect and click check access

Step 7: Now open Powershell on Windows or Terminal on Mac to connect to Linux with your IP address you just copied in image@9 above

click enter after pasting your IP address
after that a question pops up, are you sure you want to continue connecting? type in YES,
then the next prompt is to type in the password you used for your azure admin access, Linux wont show the normal asterisk ***** for password, don't be restless its working in the background, this is a security measure to protect you, if your password is type correctly, hit enter and you would see the your IP address and port 22

type in sudo su to run command as root. Becoming a root user makes our commands run smoothly.
next step is to run apt update, (Advanced Package Tool) is a command-line utility for managing software on Debian-based Linux distributions, such as Ubuntu and Linux Mint, Kali Mint, etc. It handles installation, removal, upgrading, and dependency management of .deb packages by automating retrieval from online repositories. It also tells you the status of your Linux Virtual Machine and if everything is running very well.

next install nginx which is a high performance web server software used to host websites, serve web applications, act as reverse proxy, load balance traffic, and improve overall performance and security, nginx is one of the most widely used servers in the world.

when prompted, confirm with a Y
Although optional, you can also install Vim which is a text editor used in Linux and Unix systems which makes your layout more colorful, developers and system administrators use it to edit configuration, write code, manage servers and also modify scripts directly from terminal.
when prompted, confirm with a Y

Finally copy your IP address in image number 9 above.
load it on your browser and paste.

Voila, if you see the image above, you did everything correctly.

This is a beautiful introduction to Linux, the operating system that hosts most tech products you are using everyday like open AI, LinkedIN, YouTube etcetera.

Mr LINUX says welcome!

Stop Putting dd() Everywhere Debug the Database From the Source Instead

Tahsin Abrar — Wed, 20 May 2026 21:33:32 +0000

Every backend developer has done this at some point.

A request fails.

Something is not inserting into the database.

And suddenly the codebase turns into this:

dd($request->all());

dd($user);

dd($campaign);

Log::info($query);

You add logs in controllers.

Then services.

Then repositories.

Then queue workers.

Then event listeners.

And after two hours, you are still confused.

Because the real problem is not where you are looking.

The Day I Realized Controller Debugging Wasn’t Enough

A few months ago, a developer on a team I worked with faced a strange issue.

A simple form submission was supposed to:

create a user
assign a campaign
generate a lead
push a welcome email into queue
write an activity log

Pretty normal workflow.

But randomly, some users were being created without campaign data.

Sometimes the queue job worked.

Sometimes it didn’t.

Sometimes the transaction rolled back silently.

The first reaction?

Start adding logs everywhere.

Inside controllers:

Log::info('Controller hit');

Inside services:

Log::info('Campaign assigned');

Inside queue workers:

Log::info('Email queued');

Inside transactions:

DB::beginTransaction();

The code became a forest of debugging statements.

Still no clear answer.

The Real Problem

Here’s what most developers forget:

No matter how beautiful your architecture is…

No matter whether you use:

Laravel
Raw PHP
Node.js
Python
Java

At the end of the day, every query goes to one place:

MySQL.

Always.

Your application is just the messenger.

The database sees everything.

That realization changes how you debug systems.

Instead of Guessing, Watch the Database Directly

The moment we enabled MySQL General Query Log, everything became obvious.

We literally watched the queries execute live.

One terminal command showed the full story.

SET GLOBAL general_log = 'ON';

Then:

tail -f /var/lib/mysql/hostname.log

And suddenly we could see:

INSERT INTO users ...
UPDATE campaigns ...
ROLLBACK;

That last line explained everything.

The transaction was rolling back because another process locked a table for a few seconds.

No controller log would have shown the full picture that clearly.

This Is Why Framework-Level Debugging Often Fails

Modern applications are no longer simple.

A single request may involve:

queues
events
transactions
scheduled jobs
caching
background workers
third-party APIs

By the time you debug from inside the controller, the actual issue may already be happening somewhere else entirely.

This is why developers sometimes spend hours debugging “application logic” when the real problem is:

a deadlock
a lock wait
a slow query
missing index
rollback
transaction conflict

And MySQL already knows all of this.

The Most Useful MySQL Debugging Tools

Honestly, there are four MySQL features every backend developer should know.

Not just DBAs.

Actual application developers.

Because these tools solve real-world production headaches.

1. General Query Log See Everything

This is the closest thing to “watching the database think.”

Enable it:

SET GLOBAL general_log = 'ON';

Now every query gets logged.

You can literally submit a form and watch the database flow in real time.

This becomes incredibly useful when:

one request touches multiple tables
transactions fail
duplicate queries happen
hidden queries execute behind ORM magic

Especially in large Laravel applications where Eloquent sometimes hides too much.

2. Slow Query Log Find Hidden Performance Problems

This one saves applications.

Seriously.

Sometimes developers optimize controllers endlessly while the real problem is one terrible query.

Enable it:

SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 1;

Now MySQL logs anything slower than one second.

This helps identify:

missing indexes
N+1 problems
huge scans
inefficient joins

I once saw a dashboard drop from 12 seconds to under 800ms just because a slow query log exposed a missing index.

The application code itself was completely fine.

3. InnoDB Status Understand Deadlocks & Locks

This command feels like magic the first time you use it.

SHOW ENGINE INNODB STATUS;

If your application ever experiences:

hanging requests
frozen queues
stuck inserts
transaction conflicts

This command can immediately reveal:

LATEST DETECTED DEADLOCK

or:

LOCK WAIT

And suddenly the mystery disappears.

4. Audit Tables & Triggers Permanent Visibility

This is where things become enterprise-level.

Sometimes you need permanent tracking.

For example:

who changed data
when it changed
which table changed
what action happened

Instead of depending on controller logs forever, you let MySQL automatically track important events.

Using triggers.

Example:

CREATE TRIGGER users_after_insert
AFTER INSERT ON users
FOR EACH ROW
INSERT INTO audit_logs(table_name, action_type)
VALUES ('users', 'INSERT');

Now the database tracks itself.

Clean.

Reliable.

Framework-independent.

One Important Thing Most Developers Ignore

Do not leave general_log permanently enabled in production.

That is a bad idea.

Because:

logs become massive
storage grows quickly
performance can suffer

The smarter workflow is:

Temporary Debugging

SET GLOBAL general_log = 'ON';

Investigate the issue.

Then disable it:

SET GLOBAL general_log = 'OFF';

Africa's Digital Ecosystem is Not Dead

mary moloyi — Wed, 20 May 2026 21:32:42 +0000

The Problem We Were Actually Solving

We were trying to build a platform that made it easy for African digital creators to sell their courses, ebooks, and music online. But our platform's payment gateways were failing, resulting in frustrated users and lost sales. We thought our problem was with our users, but it turned out that the issue was with the payment providers.

What We Tried First (And Why It Failed)

We tried using PayPal's "Borderless Payments" feature, thinking that it would solve our problem. However, we quickly realized that it was not enabled for our users' countries. We then switched to Stripe, hoping that their more robust infrastructure would work. But Stripe's "Risk and Compliance" team flagged our users' transactions as "high-risk," effectively blocking them. We tried to work around this by implementing additional verification checks, but it only made things worse.

The Architecture Decision

After weeks of debugging and trying different approaches, we finally decided to abandon the popular digital service providers and build our own payment processing system using Adyen and Paystack. Adyen's global reach and Paystack's expertise in African payment systems made it possible for us to process transactions from users in over 20 African countries. We also implemented a fallback system that used local payment methods like mobile money and bank transfers, ensuring that our users had a seamless payment experience.

What The Numbers Said After

After implementing our custom payment processing system, our transaction success rate improved by 80%. Our users were no longer facing payment failures, and our sales increased by 25%. We also saw a significant decrease in support tickets related to payment issues. The numbers were clear: our homegrown solution was working, and it was time to let go of the limitations of the popular digital service providers.

What I Would Do Differently

In hindsight, I would have invested more time upfront in researching the limitations of popular digital service providers. I would have also considered using local payment providers from the start, rather than trying to work around their limitations. But I learned a valuable lesson: when it comes to building a platform that serves a specific region or community, you can't rely on generic solutions. You need to build a solution that is tailored to the specific needs of your users, and that requires a deep understanding of the underlying technology and infrastructure.

Digital Payments in Africa: A System Designer's Lament

pretty ncube — Wed, 20 May 2026 21:32:14 +0000

The Problem We Were Actually Solving

Our client pool was largely made up of African creatives trying to monetize their work online. But when we tried to integrate the usual payment processors, we hit a brick wall: no matter what combination of payment flows and API keys we tried, we just couldn't get it to work. The errors were always the same: "payment declined due to regional restrictions" or simply "your account has been flagged for suspicious activity." It was as if we were trying to solve a problem that didn't exist, but clearly did.

What We Tried First (And Why It Failed)

We started by trying to massage the existing payment flows to work around the region restrictions. We tweaked the API requests, adjusted the currencies, and even tried using different payment gateways (e.g., Paystack for Nigeria, M-Pesa for Kenya). But no matter what we did, we just couldn't get it to work. It wasn't until we started digging deeper into the problem that we realized the issue wasn't with our code or the payment processors – it was with the underlying architecture of the system itself.

The Architecture Decision

We decided to take a step back and rethink the entire payment processing system from the ground up. We opted to use Mollie, a Dutch-based payment processor that specialized in multi-currency transactions and regional-specific payment flows. The decision was based largely on their experience with African markets and their ability to bypass the usual regional restrictions. We also implemented a custom payment flow using the Interchangeable Payment Interface (IPI), a set of APIs that allowed us to swap out different payment processors as needed.

What The Numbers Said After

After implementing the new payment processing system, we saw a significant decrease in payment errors (from 25% to 5%) and an increase in successful transactions (from 40% to 75%). We also saw a corresponding decrease in support requests related to payment issues (from 30% to 10%). The numbers spoke for themselves: our new system was not only more reliable but also more user-friendly.

What I Would Do Differently

Looking back, I wish we'd taken a more aggressive approach to identifying and mitigating regional restrictions from the outset. We could have saved a significant amount of time and resources by focusing on finding a payment processor that specialized in African markets from the get-go. That being said, the experience taught us a valuable lesson about the importance of system architecture and the need to think beyond the constraints of existing systems. As engineers, we need to be willing to challenge assumptions and explore new solutions – even if it means starting from scratch.

# How to Validate UK VAT Numbers, NINO, Company Numbers and UTR in Any Language (2026)

ValidatorAPI — Wed, 20 May 2026 21:31:10 +0000

How to Validate UK VAT Numbers, NINO, Company Numbers and UTR in Any Language (2026)

If you're building software for the UK market, you'll eventually need to validate fiscal and identification documents. This guide covers the algorithms, code examples, and the fastest way to add validation to any project.

What are these documents?

VAT number — Tax ID for UK businesses registered for VAT. Format: GB + 9 digits (e.g. GB123456789)
NINO — National Insurance Number, used for tax and social security. Format: 2 letters + 6 digits + letter (e.g. AB123456C)
Company Number — Companies House registration number. Format: 8 digits or prefix + 6 digits (e.g. SC123456 for Scottish companies)
UTR — Unique Taxpayer Reference, used for Self Assessment. Format: 10 digits (e.g. 1234567890)

Option 1: Implement it yourself

VAT number validation (Python)

UK VAT numbers use HMRC's mod-97 checksum on the first 7 digits:

def validate_uk_vat(vat: str) -> bool:
    vat = vat.strip().upper().replace(" ", "")
    if vat.startswith("GB"):
        vat = vat[2:]
    if len(vat) not in (9, 12) or not vat.isdigit():
        return False

    digits = [int(d) for d in vat[:7]]
    weights = [8, 7, 6, 5, 4, 3, 2]
    total = sum(d * w for d, w in zip(digits, weights))
    check = int(vat[7:9])

    return check == (97 - total % 97) or check == (97 - (total + 55) % 97)

NINO validation (JavaScript)

function validateNINO(nino) {
  nino = nino.replace(/\s/g, '').toUpperCase();
  const invalidFirst = /[DFIQUV]/;
  const invalidSecond = /[DFIOQUV]/;
  const invalidPrefixes = ['BG','GB','NK','KN','NT','TN','ZZ'];

  if (!/^[A-Z]{2}\d{6}[ABCD]$/.test(nino)) return false;
  if (invalidFirst.test(nino[0])) return false;
  if (invalidSecond.test(nino[1])) return false;
  if (invalidPrefixes.includes(nino.slice(0,2))) return false;
  return true;
}

Company number validation (Python)

PREFIXES = {'SC', 'NI', 'OC', 'SO', 'FC', 'BR', 'CE', 'CS'}

def validate_company_number(number: str) -> bool:
    number = number.strip().upper()
    if number[:2] in PREFIXES:
        return len(number) == 8 and number[2:].isdigit()
    return number.isdigit() and len(number) <= 8

Option 2: Use an API (3 lines of code)

For VAT the algorithm is manageable, but NINO has 15+ edge cases and Company Numbers have 15+ prefix types. If you don't want to maintain all of this, the UK Document Validator API handles everything:

Python

import requests

r = requests.get(
    'https://uk-document-validator1.p.rapidapi.com/validate/vat',
    params={'value': 'GB123456789'},
    headers={
        'X-RapidAPI-Key': 'YOUR_API_KEY',
        'X-RapidAPI-Host': 'uk-document-validator1.p.rapidapi.com'
    }
)
print(r.json())
# {'input': 'GB123456789', 'valid': True, 'type': 'VAT', 'country': 'GB', 'formatted': 'GB123 456 789'}

JavaScript

const r = await fetch(
  'https://uk-document-validator1.p.rapidapi.com/validate/nino?value=AB123456C',
  {
    headers: {
      'X-RapidAPI-Key': 'YOUR_API_KEY',
      'X-RapidAPI-Host': 'uk-document-validator1.p.rapidapi.com'
    }
  }
);
const data = await r.json();
// { valid: true, type: 'NINO', formatted: 'AB 12 34 56 C' }

PHP

$curl = curl_init();
curl_setopt_array($curl, [
    CURLOPT_URL => 'https://uk-document-validator1.p.rapidapi.com/validate/company?value=SC123456',
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => [
        'X-RapidAPI-Key: YOUR_API_KEY',
        'X-RapidAPI-Host: uk-document-validator1.p.rapidapi.com'
    ],
]);
$r = json_decode(curl_exec($curl));
// $r->company_type = "Scottish company"

API response examples

VAT:

{ "input": "GB123456789", "valid": true, "type": "VAT", "country": "GB", "formatted": "GB123 456 789" }

NINO:

{ "input": "AB123456C", "valid": true, "type": "NINO", "formatted": "AB 12 34 56 C" }

Company Number:

{ "input": "SC123456", "valid": true, "type": "COMPANY_NUMBER", "company_type": "Scottish company", "formatted": "SC123456" }

Auto-detect (any document type):

{ "input": "1234567890", "valid": true, "type": "UTR", "detected_as": "UTR", "formatted": "12345 67890" }

When to use the API vs DIY

	DIY	API
VAT validation	~15 lines	3 lines
NINO edge cases	15+ rules to implement	Included
Company prefix types	15+ prefixes	Included
UTR checksum	Complex	Included
Auto-detect document type	Extra logic	Included
Maintenance	Your problem	Automatic

Free tier: 500 requests/month. Subscribe here.

Full docs and interactive playground: uk-validation-api.vercel.app/docs

Chat with your database in plain English — locally, for free

retrovirusretro — Wed, 20 May 2026 21:28:03 +0000

"What were our top 10 customers last quarter by revenue, as a bar chart?"

DB-GPT translates that to SQL, runs it against your database, and renders the chart. No SQL knowledge required. Fully local. MIT licensed. 17K GitHub stars — almost no English content.

What is DB-GPT?

DB-GPT is an open-source framework that puts a natural language interface on top of your databases. You connect PostgreSQL, MySQL, SQLite, or others — then ask questions in plain English. It generates the SQL, executes it, and can visualize results automatically.

Think Metabase meets AI, but fully self-hosted and free.

Supported Databases

PostgreSQL · MySQL · MariaDB · SQLite · ClickHouse · DuckDB · Spark SQL

Setup

Clone the repo, copy .env.example to .env, add your database connection string, then docker compose up -d. Open localhost:5670 → admin / admin → Settings → Database → Add → paste connection string.

Full docker-compose: chinese-ai-tools-english-guide/tools/db-gpt

Example Queries

Once your database is connected:

"Show total sales by month for the last 12 months as a bar chart"
"Which products have inventory below 10 units?"
"Top 5 customers by order value with their email addresses"
"Average order fulfillment time grouped by warehouse"

DB-GPT generates SQL for each, runs it, returns results. Charts render automatically in the UI.

Connecting Ollama

Settings → LLM Provider → Ollama → Base URL: http://ollama:11434/v1 → Model: llama3

For best SQL accuracy use sqlcoder — fine-tuned specifically for SQL generation. Pull it with docker exec -it ollama ollama pull sqlcoder.

DB-GPT vs Vanna.ai

Both let you query databases with natural language:

	DB-GPT	Vanna.ai
License	MIT	MIT
Built-in UI	✅ Full app	Minimal
Charts	✅ Built-in	❌ External
Visual pipeline	✅ AWEL	❌
Self-hosted	✅	✅

DB-GPT if you want a complete self-contained app. Vanna.ai if you're embedding the capability programmatically into your own product.

n8n Automation

A ready-to-import workflow JSON is in the repo (integration/n8n-workflows/db-gpt-query.json). POST {question, db_name} → returns {answer, sql, data}.

Full Guide

→ chinese-ai-tools-english-guide

Previous articles in this series:

Your data never leaves your machine. No API keys, no cloud, no SQL knowledge needed.

The simplest self-hosted RAG you'll ever set up (Apache 2.0, 20K stars)

retrovirusretro — Wed, 20 May 2026 21:24:39 +0000

Most RAG tools make you choose between simplicity and power. MaxKB doesn't try to be powerful — it tries to be simple, and it nails it.

20K+ GitHub stars. Apache 2.0. Almost no English content. Here's the guide.

What is MaxKB?

MaxKB (Max Knowledge Base) is a knowledge base Q&A system by the 1Panel team. It connects to any OpenAI-compatible API — including Ollama — and lets you upload documents, ask questions, and embed a chat widget into any website.

That last part is the killer feature: MaxKB generates a JavaScript snippet that drops a chat widget into any HTML page. One script tag. No iframe, no backend changes needed. Apache 2.0 means you can embed this in commercial products with no restrictions.

Setup

3 commands, under 5 minutes.

Clone the repo, copy .env.example to .env, then docker compose up -d. Open localhost:8081 → admin / admin123 → Settings → Model Provider → Ollama → http://ollama:11434.

Create a knowledge base, upload a PDF, start asking questions. That's the entire setup.

Full docker-compose: maxkb-english-guide

Embed Widget

MaxKB generates a JavaScript snippet — drop it into any HTML page and a chat widget appears bottom-right. No iframe, no backend changes. This is what makes MaxKB unique among RAG tools: it's designed to be embedded.

Go to: Application → your app → Embed → copy the script tag → paste into any HTML page.

API

Works from Python, JavaScript, curl, n8n — anything that speaks HTTP. POST your question to /api/application/{app_id}/chat/completions with a Bearer token. Returns {"content": "the answer"}.

Full Python + JavaScript examples in the repo.

MaxKB vs the Alternatives

	MaxKB	WeKnora	FastGPT	RAGFlow
Setup time	⚡ 3 min	5 min	10 min	15 min
License	Apache 2.0	MIT	Custom*	Apache 2.0
Embed widget	✅	❌	❌	❌
Autonomous agent	❌	✅	✅	❌
PDF table parsing	Basic	Basic	Good	Excellent
Commercial embed	✅	✅	❌	✅

*FastGPT prohibits SaaS resale.

Pick MaxKB when: working knowledge base in 5 minutes, embed widget needed, Apache 2.0 matters.

Pick something else when: complex PDFs → RAGFlow · pipeline builder → FastGPT · multi-hop reasoning → WeKnora

Supported LLM Providers

MaxKB works with any OpenAI-compatible API: Ollama (local, free), OpenAI, Groq, Together, or Anthropic via LiteLLM proxy.

Full Guide

→ github.com/retrovirusretro/maxkb-english-guide

Part of a broader series:
→ chinese-ai-tools-english-guide

MaxKB is the tool I recommend to people who just want RAG working quickly without reading 40 pages of docs.