DEV Community: JunYoungMoon

Are We in an AI Bubble? A Tech Professional's Perspective

JunYoungMoon — Thu, 18 Dec 2025 05:17:15 +0000

TLDR: We're witnessing unprecedented AI investment and hype. Historical patterns suggest this looks like a bubble, but that doesn't mean AI isn't revolutionary. Smart engineers should stay engaged but cautious, avoiding career decisions based purely on FOMO.

The Current Landscape

As software engineers, we're living through what might be the most significant technological shift of our careers. Companies are pouring unprecedented capital into AI infrastructure, with spending levels that rival entire national budgets.

The Numbers Are Staggering:

AI accounts for ~90% of S&P 500 capex growth
Nvidia went from $600M market cap (1999 IPO) to briefly touching $5 trillion
Multiple AI startups raising $1B+ seed rounds with no product

What Makes This Time Different (And What Doesn't)

The "This Time It's Different" Arguments

As developers, we hear these daily:

Real products with real users: ChatGPT hit 1 billion users faster than any technology in history
Immediate revenue: Companies like Anthropic seeing 10x YoY revenue growth
Proven leaders: Microsoft, Google, Meta aren't dot-com startups
Reasonable valuations: Current AI leaders' P/E ratios are lower than 1999 tech stocks

The Historical Echoes

But experienced engineers recognize familiar patterns:

1860s: Railroads
1920s: Radio & Aviation  
1990s: Dot-com
2000s: Subprime housing
2020s: AI?

Each revolution followed the same arc:

Revolutionary technology emerges
Early adopters make fortunes
FOMO drives everyone else in
Overbuilding and speculation
Crash (but technology persists)

The Technical Reality Check

What We Know

LLMs are genuinely transformative
Coding assistants (Claude Code, Cursor) are already replacing significant developer hours
GPU demand is real and growing
The technology works and is improving rapidly

What We Don't Know

Who will be the ultimate winners
Whether first-movers can maintain advantages
If AI services become commoditized (price competition → thin margins)
Economic lifespan of AI infrastructure
Whether AGI is achievable (and on what timeline)

The Circular Money Problem

As engineers, we should be skeptical of "circular deals":

// Simplified illustration of circular revenue problem
public class CircularDeal {
    // Company A invests $1B in Company B
    companyA.invest(companyB, 1_000_000_000);

    // Company B uses that $1B to buy services from Company A
    companyB.purchaseServices(companyA, 1_000_000_000);

    // Both companies report revenue
    // But where's the real value creation?
}

Historical parallel: 1990s telecom companies buying fiber from each other, funded by vendor financing. It inflated revenues without real demand.

Current concerns:

OpenAI receiving billions from tech companies
OpenAI paying billions back to those same companies for compute
Nvidia investing in customers who buy Nvidia chips
Goldman estimates 15% of Nvidia's revenue comes from such arrangements

The Debt Factor: A Developer's Perspective

Tech companies traditionally funded growth through equity or cash flow. Now:

Microsoft, Google, Meta issuing 30-year bonds for AI
Startups borrowing billions for data centers
SPVs (Special Purpose Vehicles) hiding debt off balance sheets

The Risk: In software engineering, we understand that:

public class InvestmentRisk {
    /**
     * Risk Amplification Formula
     * When outcomes are uncertain, debt amplifies both gains AND losses
     */
    public double calculateRisk(double uncertainty, double debtLeverage) {
        return uncertainty * debtLeverage; // Exponentially higher risk
    }
}

AI outcomes are highly uncertain. Adding leverage amplifies both potential gains AND losses.

What This Means for Your Career

Jobs That Are Already Changing

Junior coding positions: AI can now write production-quality code
Documentation writing: Automated by LLMs
Basic code review: Increasingly automated
Junior legal/analyst work: Pattern matching that AI excels at

Jobs That Remain Valuable

System architecture: Requiring holistic judgment
Product decisions: Requiring user empathy
Team leadership: Requiring emotional intelligence
Debugging complex systems: Requiring deep contextual understanding
Physical infrastructure: Can't be automated (yet)

The "Skill Obsolescence" Risk

One concerning pattern: If we eliminate junior positions, where do senior experts come from? You don't become a 10x engineer without years of practice.

Practical Advice for Engineers

DO:

Learn AI tools: They're productivity multipliers
Stay informed: Technology is real, even if valuations aren't
Maintain fundamentals: Core CS skills remain valuable
Diversify: Don't bet entire career on AI hype
Watch for warning signs: Vague promises, circular revenue, unrealistic timelines

DON'T:

Join companies purely for AI hype: Fundamentals still matter
Ignore traditional skills: Algorithms, systems design, etc.
Panic about job loss: Transitions take time
Assume current leaders will win: Remember Netscape, Yahoo, MySpace
Believe "$1T valuation" stories: These often end badly

The Infrastructure Perspective

As someone who's built systems at scale, I'm watching the data center buildout with concern:

Current trajectory:

JP Morgan estimates $5T needed for AI infrastructure
Companies' cash reserves: ~$350B
Gap must be filled with... debt

Historical parallel:

1990s: Overbuilt fiber optic networks
Early 2000s: Massive telecom bankruptcies
Long-term: Infrastructure was used, but investors lost billions

The technology was real. The overinvestment was also real.

Questions to Ask Your Employer

If you're evaluating AI companies or roles:

Revenue model: "How do you make money?" (Be skeptical of vague answers)
Burn rate: "How long is your runway?"
Differentiation: "What happens when OpenAI/Google does this?"
Debt load: "How much is borrowed vs. equity?"
Customer concentration: "Is revenue real or circular?"

The Optimistic Case

Despite concerns, reasons for hope:

Baby boomers retiring: 16M retiring by 2035, AI fills gaps
Productivity gains: Real time savings for developers
New problem spaces: AI might create more jobs than it destroys
Pattern breaks: 20% of the time, "this time IS different"

Conclusion: The Middle Path

Neither blind optimism nor pessimism serves engineers well.

My recommendation:

/**
 * AI Career Strategy Pattern
 */
public class AICareerStrategy {

    private static final String ENGAGEMENT_LEVEL = "MODERATE"; // Not zero, not all-in

    public String execute() {
        // Stay informed and skilled
        learnAITools();
        maintainFundamentals();

        // Make decisions based on fundamentals
        if (hasRealRevenue() && hasReasonableValuation()) {
            considerOpportunity();
        }

        // But don't bet everything
        avoidAllInBets();

        return "STAY_FLEXIBLE_AND_INFORMED";
    }

    private void learnAITools() {
        // Stay current with AI developments
    }

    private void maintainFundamentals() {
        // Keep core CS skills sharp
    }

    private boolean hasRealRevenue() {
        return true; // Verify actual revenue model
    }

    private boolean hasReasonableValuation() {
        return true; // Check P/E ratios, fundamentals
    }
}

We're witnessing either the most important technological revolution of our lifetimes, or the most spectacular bubble, or (most likely) both simultaneously.

The technology is real. The hype is also real. Navigate accordingly.

Discussion Questions

What's your experience with AI in your daily work? Are you seeing real productivity gains or just hype? How is your company approaching AI investment?

Tags: #ai #career #bubble #machinelearning #investing #technology #opinion

Solving Access Denied Issues After SSE Implementation in Spring Security

JunYoungMoon — Wed, 17 Dec 2025 06:41:14 +0000

TLDR: Implemented SSE streaming for LLM responses, but second event onwards hit "Access Denied". Root cause: SecurityContext not propagating to ASYNC dispatches. Fix: One line of Spring Security config.

Background: Why SSE?

Recently I improved our LLM response mechanism by introducing SSE (Server-Sent Events). Previously, we returned complete API responses in one shot. Now we deliver LLM responses via Flux-based streaming to the client.

Quick definitions:

Flux-based streaming: Reactive Streams API from Spring WebFlux that delivers data streams asynchronously in multiple chunks
SSE (Server-Sent Events): HTTP-based unidirectional streaming where server continuously pushes data to client

In other words, when delivering LLM responses, we're not using a traditional single HTTP request/response structure. Instead, we maintain one connection and deliver multiple events in real-time using SSE.

The Problem

The first API request worked perfectly. The Dispatcher Type was REQUEST, and JwtAuthenticationFilter validated the cookie properly, storing authentication info in SecurityContextHolder.

But starting from the second SSE event, problems emerged. The second request came in with Dispatcher Type ASYNC, bypassing the existing JwtAuthenticationFilter. As a result, SecurityContextHolder had no authentication info. Despite being the same URL, the security context was empty, causing an Access Denied error.

Root Cause Analysis

Spring Security manages SecurityContext throughout the request lifecycle. In normal requests (REQUEST), authentication info is stored properly through the filter chain. However, for asynchronous requests (ASYNC) or re-dispatches (e.g., FORWARD, INCLUDE), the security context may not propagate automatically.

In SSE and other async response environments, basic security configuration alone doesn't guarantee context propagation, leading to Access Denied issues.

The Solution

The fix is configuring SecurityContextRepository to propagate the session (technically the context within request scope). We can use this option provided in Spring Security 5.8+:

http.securityContext(context -> context.requireExplicitSave(false));

With this option, SecurityContext is automatically saved to SecurityContextRepository when changed, and the context propagates across re-dispatches (REQUEST, ASYNC, FORWARD, INCLUDE) within the same request.

What is SecurityContextRepository?

SecurityContextRepository is the strategy interface for storing/restoring SecurityContext in Spring Security. Behavior varies by implementation, but in SSE environments, it typically stores context in HttpServletRequest attributes (request scope) to maintain authentication info across async processing.

This approach maintains stateless operation (no sessions) while safely sharing security context across re-dispatches within a single HTTP request.

Understanding requireExplicitSave

Since Spring Security 5.8, you can explicitly control how SecurityContext is saved:

`requireExplicitSave(false)` (Auto-save mode, default)

Automatically saves to SecurityContextRepository when SecurityContext changes
No manual save code needed
Same as legacy Spring Security behavior

`requireExplicitSave(true)` (Explicit save mode)

Must explicitly call SecurityContextRepository.saveContext() after context changes
No automatic saving
Performance benefits, but developers must manage save timing

Since our system isn't complex yet, we decided auto-save mode (requireExplicitSave(false)) is more appropriate.

Why Re-dispatch Happens in SSE

Re-dispatching in SSE stems from Servlet 3.0 async processing spec's core design principle.

When async work completes, the result must go through the existing filter/servlet chain for consistent processing—hence the mandatory re-dispatch.

Without re-dispatch, the async request lifecycle wouldn't complete properly, causing resource leaks, connection management issues, security context mismatches, and other problems.

Therefore, Servlet 3.0 spec mandates re-dispatch after all async work completes, classifying such requests as DispatcherType.ASYNC.

https://docs.spring.io/spring-framework/reference/web/webmvc/mvc-ann-async.html#mvc-ann-async-processing

Verifying the Call Order

To confirm when data is sent and when threads separate, I wrote a simple example.

In Spring MVC with SSE, controllers return Flux streams instead of regular ResponseEntity:

@PostMapping(value = "/gpt4/v3/test", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> test() throws Exception {

    return Flux.just("data1", "data2", "data3")
            .delayElements(Duration.ofSeconds(1))
            .doOnComplete(() -> {
                System.out.println("All data sent");
            })
            .doFinally(signalType -> {
                System.out.println("Stream cleanup started: " + signalType);
            });
}

This code maintains the connection with the client while sending data sequentially at 1-second intervals.

Instead of terminating the request immediately, it delivers responses asynchronously using Servlet 3.0's AsyncContext.

Tracking Async Transitions

I created AsyncDebugFilter to precisely identify the async transition point:

@Slf4j
@Component
public class AsyncDebugFilter implements Filter {

  @Override
  public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain chain)
          throws IOException, ServletException {
      ...

      HttpServletRequestWrapper requestWrapper = new HttpServletRequestWrapper(req) {
          ...
          @Override
          public AsyncContext startAsync(ServletRequest servletRequest, ServletResponse servletResponse) {
              AsyncContext ac = super.startAsync(servletRequest, servletResponse);
              attachListener(ac, requestId);
              log.info("[{}] startAsync(req,res) called. DispatcherType={}, Thread={}",
                      requestId, super.getDispatcherType(), Thread.currentThread().getName());
              return ac;
          }

          private void attachListener(AsyncContext ac, String requestId) {
            ac.addListener(new AsyncListener() {
                @Override
                public void onComplete(AsyncEvent event) {
                    log.info("[{}] AsyncListener.onComplete - Thread={}", requestId, Thread.currentThread().getName());
                }
                ...
            });
          }
      }

      HttpServletResponseWrapper responseWrapper = new HttpServletResponseWrapper(res) {
          private ServletOutputStream loggingStream;
          @Override
          public ServletOutputStream getOutputStream() throws IOException {
              if (loggingStream == null) {
                  final ServletOutputStream original = super.getOutputStream();
                  loggingStream = new ServletOutputStream() {
                      ...
                      @Override
                      public void write(byte[] b, int off, int len) throws IOException {
                          log.debug("[{}] output.write(byte[{}]) thread={} len={}", requestId, b.length, Thread.currentThread().getName(), len);
                          original.write(b, off, len);
                      }
                      @Override
                      public void flush() throws IOException {
                          log.debug("[{}] output.flush thread={}", requestId, Thread.currentThread().getName());
                          original.flush();
                      }
                      ...
                  };
              }
              return loggingStream;
          }
          ...
      };
  }  
}

This filter intercepts request.startAsync() calls and registers an AsyncListener, allowing us to track exactly when async processing begins.

The Execution Flow

First, let's examine the call sequence. When a request arrives, it passes through security filters into business logic.
When it hits async logic, startAsync is called to separate the thread. (→ This is the async transition point)

After data transmission completes, doOnComplete and doFinally are called sequentially, then the servlet container performs ASYNC re-dispatch.

Without ASYNC re-dispatch, the request would remain open.
Therefore, the container must perform ASYNC dispatch to properly terminate the request, ensuring connections are cleaned up correctly.

Conclusion

The Access Denied issue after implementing SSE was caused by Spring Security's behavior of not propagating SecurityContext in async requests.

By applying http.securityContext(context -> context.requireExplicitSave(false)) to utilize SecurityContextRepository, we can reliably maintain authentication info even in SSE environments.

If the system becomes more complex and performance optimization becomes necessary, we could consider explicit save mode (requireExplicitSave(true)). But for now, auto-save mode provides stable operation.

Key Takeaways

SSE uses Servlet 3.0 async processing, which requires ASYNC re-dispatch after completion
SecurityContext doesn't auto-propagate across dispatcher types by default
One config line fixes it: requireExplicitSave(false) enables automatic context propagation
No sessions needed: Uses request-scoped attributes for stateless context sharing
AsyncDebugFilter is your friend: Helps visualize the complete async lifecycle

Tags: #spring-security #sse #async #spring-boot #webflux #security-context #servlet

💭 Questions or feedback? Join the discussion in the comments!

Our RAG system still failed on hierarchical metrics — Part 2

JunYoungMoon — Mon, 15 Dec 2025 02:43:23 +0000

How I Built a RAG System That Actually Understands Business Metrics (Part 2: Hierarchical Search)

TLDR: Part 1 could find "total_revenue", but failed at "Naver traffic". This article shows how I fixed that with a tree-based metric system and 2-stage GPT filtering. Result: 95% → 98% accuracy.

The Real Problem

Remember Part 1? We built this:

User: "What's the conversion rate?"
System: Found "conversion_rate"

But then this happened:

User: "How's my Naver traffic?"
System: Found "traffic_by_channel"
        But which channel? Naver? Google? Facebook?

The gap: Our system understood categories but couldn't navigate their subcategories.

💡 The Solution in 3 Moves

Think of it like a chess game. We need three moves to checkmate:

Move 1: SEARCH      → Cast a wide net (50 candidates)
Move 2: CLASSIFY    → Filter intelligently (3 names → 12 nodes)
Move 3: REFINE      → Validate relationships (final 2 metrics)

Let's see each move in action.

Move 1: The Setup (Vector Search)

The Challenge: Metrics have hierarchy

traffic_by_channel/           ← Parent (1-depth)
├── naver                     ← Child (2-depth)
├── google
├── facebook
└── direct_traffic

The Strategy: Use TWO search indices

Index	Purpose	Top N
1-depth	Find categories	10
2-depth	Find specific values	40

Why 40 vs 10? 2-depth metrics are like finding "John" in a phone book with 1000 Johns. We need more candidates.

Example Result:

Query: "How's my Naver traffic?"

1-depth finds: traffic_by_channel, site_duration, bounce_rate...
2-depth finds: naver, google, facebook, traffic_count...

Total: 50 candidates

Key Insight: Don't search for "traffic_by_channel|naver" directly. Search for each piece separately, then connect them.

Move 2: The Play (Smart Classification)

This move has 3 sub-plays. Watch closely.

Sub-Play 2.1: Remove the Noise 🧹

Problem: 50 candidates include junk

For "Naver traffic":
naver          ← Relevant
traffic_count  ← Maybe relevant?
instagram      ← Wrong channel
bounce_rate    ← Not about traffic sources

Solution: Ask GPT (Structured Outputs)

// Force GPT to return clean JSON array
"response_format": {
  "type": "json_schema",
  "schema": {
    "type": "object",
    "properties": {
      "metrics": {
        "type": "array",
        "items": {"type": "string"}
      }
    }
  }
}

Output:

{
  "metrics": ["naver", "direct_traffic", "traffic_count"]
}

Result: 50 → 3 ✂️

Sub-Play 2.2: From Names to Nodes

Problem: "naver" is just a string. We need context.

What we need to know:

Is this advertiser's data or industry average?
What are the parent metrics?
What other "naver" nodes exist?

Solution: MetricForest lookup

"naver" in nameIndex → [
  "traffic_by_channel|naver|advertiser",
  "traffic_by_channel|naver|industry",
  "channel_revenue|naver|advertiser",
  "channel_revenue|naver|industry",
  "first_purchase|naver|advertiser",
  "repeat_purchase|naver|advertiser"
]

The Magic: O(1) lookup using HashMap

// Without nameIndex: O(N) scan through all nodes
for (node in allNodes) {
  if (node.name == "naver") { ... }  // Slow!
}

// With nameIndex: O(1) direct access
List<MetricNode> nodes = nameIndex.get("naver");  // Fast!

Result: 3 names → 12 nodes (each name maps to multiple nodes)

Sub-Play 2.3: Pick Your Side

Problem: Same metric, two meanings

traffic_by_channel|naver|advertiser  → "Our Naver traffic"
traffic_by_channel|naver|industry    → "Industry average"

Question: Which one does the user want?

Solution: Let GPT decide

Query Analysis:
"How's MY Naver traffic?"     → Advertiser
"Industry average?"           → Industry  
"Compare to industry"         → Both

Implementation:

String scope = classifyScope(query);

if ("common".equals(scope)) {
    return ["advertiser", "industry"];  // Return both
} else {
    return [scope];  // Return one
}

Result: 12 nodes → Filter to advertiser domain only

Move 2 Summary:

Started with: 50 candidates

Ended with: 12 nodes, advertiser domain confirmed

But we're not done yet...

Move 3: Checkmate (Relationship Validation)

Here's where it gets interesting. We have 12 nodes:

✓ traffic_by_channel|naver|advertiser
✓ channel_revenue|naver|advertiser        ← Wrong parent!
✓ traffic_by_channel|direct_traffic|advertiser
✓ new_members|traffic_count|advertiser   ← Wrong parent!
... 8 more

Problem: Not all are correct. "channel_revenue|naver" isn't about traffic.

Solution: 3-step validation

Step 3.1: Collect the Neighbors

Rule:

If 1-depth node → Get children
If 2-depth node → Get parents

Why? To understand relationships.

Our 12 nodes are all 2-depth
→ Collect their parents
→ Get 6 unique parent names:

1. traffic_by_channel    ← Traffic related ✓
2. channel_revenue       ← Revenue, not traffic ✗
3. first_purchase        ← Purchase, not traffic ✗
4. repeat_purchase       ← Purchase, not traffic ✗
5. new_members          ← Not about channels ✗
6. existing_members     ← Not about channels ✗

Step 3.2: Filter Parents with GPT

Ask GPT: "Which parents are relevant to 'Naver traffic'?"

Input: 6 parent names
GPT thinking:
  - traffic_by_channel? YES → About traffic AND channels
  - channel_revenue? NO → About revenue, not traffic
  - first_purchase? NO → About purchases
  - new_members? NO → Not related to Naver

Output: ["traffic_by_channel"]

This is CRITICAL: Only 1 parent passed! This means:

Keep: traffic_by_channel|naver
Remove: channel_revenue|naver
Remove: new_members|traffic_count

Step 3.3: Final Validation

Now apply 3 rules:

Rule 1: Leaf nodes (no children) → Accept immediately

if (node.depth == 1 && node.children.isEmpty()) {
    return node;  // Already final
}

Rule 2: Parent nodes (has children) → Return matching children

if (node.depth == 1 && !node.children.isEmpty()) {
    return node.children
        .filter(child -> secondResult.contains(child.name));
}

Rule 3: Child nodes → Validate parent exists in secondResult

if (node.depth == 2) {
    return node.parents
        .anyMatch(parent -> secondResult.contains(parent.name));
}

Applying to our 12 nodes:

Check: traffic_by_channel|naver|advertiser
  → Parent "traffic_by_channel" in secondResult? 
  → ACCEPT

Check: channel_revenue|naver|advertiser
  → Parent "channel_revenue" in secondResult? 
  → REJECT

Check: new_members|traffic_count|advertiser
  → Parent "new_members" in secondResult? 
  → REJECT

Final Result:

[
  "traffic_by_channel|naver|advertiser",
  "traffic_by_channel|direct_traffic|advertiser"
]

Checkmate!

The Complete Game Replay

Let's watch the entire sequence:

Query: "How's my Naver traffic?"

   Move 1: SEARCH
   ├─ 1-depth index → 10 results
   ├─ 2-depth index → 40 results
   └─ Total: 50 candidates

   Move 2: CLASSIFY
   ├─ 2.1: GPT Filter → 3 names
   ├─ 2.2: Node Lookup → 12 nodes
   └─ 2.3: Domain Filter → ["advertiser"]

   Move 3: REFINE
   ├─ 3.1: Collect neighbors → 6 parent names
   ├─ 3.2: GPT Filter → 1 parent name
   └─ 3.3: Validate → 2 final metrics

   RESULT:
   ✓ traffic_by_channel|naver|advertiser
   ✓ traffic_by_channel|direct_traffic|advertiser

Time taken: < 1 second

Cost per query: ~$0.002

Before vs After

Part 1 (Basic Search)

 "What's the conversion rate?"
 → conversion_rate

 "How's my Naver traffic?"
 → Found "traffic_by_channel" but couldn't find "naver"

Accuracy: 95% for flat metrics only

Part 2 (Hierarchical Search)

   "What's the conversion rate?"
   → conversion_rate

   "How's my Naver traffic?"
   → traffic_by_channel|naver|advertiser
   → traffic_by_channel|direct_traffic|advertiser

   "Compare my Google traffic to industry"
   → traffic_by_channel|google|advertiser
   → traffic_by_channel|google|industry

Accuracy: 98% for all queries including hierarchical

What I Learned

1. Use AI Where It Shines

Good: Semantic understanding

// Is "revenue" related to "sales"? → Ask GPT
// Is "naver" relevant to "traffic"? → Ask GPT

Bad: Deterministic logic

// Is this node depth 1 or 2? → Just check node.depth
// Does parent exist? → Just check hashmap

2. Two-Stage Filtering is Magic

One GPT call on 50 candidates → 70% accuracy
Two GPT calls (50 → 6 → final) → 98% accuracy

The second filter on a small, focused set is what makes it work.

3. Data Structure = Foundation

Without MetricForest:

Can't collect neighbors
Can't validate relationships
Can't distinguish contexts

The tree structure makes everything else possible.

4. Filter Domain LATE, Not Early

Wrong approach:

// Filter domain first
nodes = filterByDomain(nodes, "advertiser");
// Now we can't see that both domains existed!

Right approach:

// Let GPT see both domains
nodes = getAllNodes();
domain = askGPT("Which domain does user want?");
// Now filter
nodes = filterByDomain(nodes, domain);

This enables "compare" queries that need both domains.

Real-World Code

Here's the orchestration service that ties it all together:

@Service
@RequiredArgsConstructor
public class MetricSearchService {

    private final VectorSearchService vectorSearch;
    private final StructuredOutputsService gptFilter;
    private final MetricForest metricForest;
    private final DomainClassifier domainClassifier;

    public List<String> search(String query) {
        // Move 1: Search
        var candidates = vectorSearch.search(query, 50);

        // Move 2.1: First filter
        var names = gptFilter.filter(query, candidates);

        // Move 2.2: Resolve to nodes
        var nodes = metricForest.findByNames(names);

        // Move 2.3: Decide domain
        var domain = domainClassifier.classify(query, nodes);

        // Move 3.1-3.2: Collect and filter neighbors
        var neighbors = collectNeighbors(nodes);
        var validParents = gptFilter.filter(query, neighbors);

        // Move 3.3: Final validation
        return validate(nodes, domain, validParents);
    }
}

Clean, simple, effective.

Try These Queries

The system now handles:

Flat metrics (from Part 1)

"What's the conversion rate?"
"Show order count"

Hierarchical metrics (NEW!)

"My Naver traffic?"
"Google revenue?"
"Facebook first purchase rate?"

Comparisons (NEW!)

"Compare my Naver traffic to industry average"

Category queries (NEW!)

"Show all channel traffic"

What's Next?

Current system handles:

Flat metrics
Hierarchical metrics
Domain disambiguation
Comparison queries

Coming soon:

Time ranges ("last month's Naver traffic")
Aggregations ("total revenue by channel")
Conversational follow-ups
Multi-metric queries

Need Help Implementing RAG?

I help companies integrate AI systems with their existing Spring Boot infrastructure.

Specializing in:

Spring Boot + OpenAI integration
Custom RAG pipelines
E-commerce analytics systems

💭 Questions or feedback? Join the discussion in the comments!

Tags: #rag #hierarchical-search #spring-boot #openai #system-design #ai-engineering

Our RAG system failed to understand KPIs — Part 1: Metric retrieval design

JunYoungMoon — Sun, 14 Dec 2025 12:42:10 +0000

How I Built a RAG System That Actually Understands Business Metrics (Part 1: Foundation)

The Problem: When AI Can't Find Your Data

Picture this: Your marketing team asks your AI assistant, "How's this month's revenue?"

The AI responds: "I don't have access to that data."

Frustrating, right? But here's the real problem - you DO have the data. It's sitting in your database, perfectly organized, just waiting to be used. The AI just doesn't know which metrics to look for.

This is the story of how I solved that problem by building a Retrieval-Augmented Generation (RAG) system that bridges the gap between natural language questions and structured business metrics.

Why RAG? Understanding LLM Limitations

Large Language Models like GPT have three fundamental limitations:

Time Constraint: Trained only up to a specific cutoff date
Domain Constraint: No knowledge of your company's internal data
Real-time Constraint: Can't access continuously changing data

RAG solves this by combining the AI's language understanding with your real-time data retrieval system.

The Architecture Overview

Here's what we're building:

User Question 
    ↓
[1] Text Embedding (Vector Search)
    ↓
[2] Structured Outputs (GPT Filtering)
    ↓
[3] Database Query
    ↓
Final Answer

This article covers steps 1-2, which form the foundation of intelligent metric retrieval.

Starting Point: Understanding Our Data Structure

Our Spring Batch process generates metrics in this format:

order_count|advertiser
conversion_rate|advertiser
total_revenue|advertiser
campaign_roas|ad_impressions|advertiser
traffic_by_channel|google|industry
traffic_by_channel|facebook|industry

The problem? This flat structure makes semantic search difficult.

Solution: Restructure into hierarchical maps

Step 1 - Separate by domain (Advertiser vs Industry):

{
  "order_count": [],
  "total_revenue": [],
  "campaign_roas": [
    "ad_impressions",
    "ad_clicks",
    "ad_spend"
  ],
  "traffic_by_channel": [
    "google",
    "facebook",
    "direct"
  ]
}

Step 2 - Create domain classification:

{
  "order_count": "advertiser",
  "conversion_rate": "common",
  "total_revenue": "common",
  "avg_purchase_count": "industry"
}

Step 1: Text Embedding for Semantic Search

The Core Concept: Convert text into numerical vectors that capture semantic meaning.

How It Works

When a user asks "How's the revenue?", we need to find that this relates to the "total_revenue" metric - even though the words don't match exactly.

Process:

Convert question → 1536-dimensional vector
Convert all metrics → 1536-dimensional vectors
Calculate cosine similarity
Extract top candidates

OpenAI Embedding API

@Service
public class EmbeddingService {

    @Value("${openai.api.key}")
    private String apiKey;

    public double[] getEmbedding(String text) {
        Map<String, Object> requestBody = Map.of(
            "model", "text-embedding-3-small",
            "input", text
        );

        HttpHeaders headers = new HttpHeaders();
        headers.setBearerAuth(apiKey);
        headers.setContentType(MediaType.APPLICATION_JSON);

        HttpEntity<Map<String, Object>> request = new HttpEntity<>(requestBody, headers);
        ResponseEntity<Map> response = restTemplate.postForEntity(
            "https://api.openai.com/v1/embeddings",
            request,
            Map.class
        );

        // Extract embedding array from response
        Map<String, Object> data = (Map<String, Object>) 
            ((List<Object>) response.getBody().get("data")).get(0);
        List<Double> embedding = (List<Double>) data.get("embedding");

        return embedding.stream()
            .mapToDouble(Double::doubleValue)
            .toArray();
    }
}

This service generates embeddings using OpenAI's API.

Cosine Similarity Calculation

public class VectorMath {

    public static double cosineSimilarity(double[] vec1, double[] vec2) {
        double dotProduct = 0.0;
        double norm1 = 0.0;
        double norm2 = 0.0;

        for (int i = 0; i < vec1.length; i++) {
            dotProduct += vec1[i] * vec2[i];
            norm1 += vec1[i] * vec1[i];
            norm2 += vec2[i] * vec2[i];
        }

        return dotProduct / (Math.sqrt(norm1) * Math.sqrt(norm2));
    }
}

This calculates semantic similarity between two vectors.

Example Output

Question: "How's this month's revenue?"

Top Matches:
- "total_revenue": 0.89 similarity ✓
- "order_count": 0.72 similarity
- "conversion_rate": 0.45 similarity

Real Example:

Question: "Show me sales performance"

Vector Comparison:
[0.4, 0.1, 0.6, ...] (question)
    vs
[0.3, 0.2, 0.7, ...] (total_revenue) → 89% match
[0.1, 0.5, -0.2, ...] (order_count) → 72% match
[0.2, 0.6, -0.4, ...] (conversion_rate) → 45% match

Result: Extract top 10 candidates

Problem: 10 candidates is still too many for accurate answers. We need more precise filtering.

Step 2: Structured Outputs for Precise Classification

Why Structured Outputs?

Without Structured Outputs:

AI Response: "You should check total_revenue and maybe order_count as well..."

↓ Unpredictable format, hard to parse

With Structured Outputs:

{
  "metrics": ["total_revenue", "order_count"]
}

↓ Clean array, ready to use

The JSON Schema

Map<String, Object> schema = Map.of(
    "type", "object",
    "properties", Map.of(
        "metrics", Map.of(
            "type", "array",
            "items", Map.of("type", "string"),
            "description", "List of relevant metric names"
        )
    ),
    "required", List.of("metrics"),
    "additionalProperties", false
);

This schema forces GPT to return exactly the format we need.

System Prompt Design

String systemPrompt = String.format("""
    You are an analytics assistant for e-commerce advertisers.

    ### Critical Rules
    1) If you don't understand the question → return []
    2) If unrelated to advertising data → return []
    3) Extract ONLY metric names, no explanations
    4) Use exact names from the list below

    ### Available Metrics
    %s
    """, String.join(", ", candidateMetrics));

Clear instructions ensure consistent results.

Java Implementation

@Service
public class MetricClassificationService {

    @Value("${openai.api.key}")
    private String apiKey;

    private final RestTemplate restTemplate;

    public List<String> classifyMetrics(
            String userQuestion, 
            List<String> candidateMetrics) {

        String systemPrompt = buildSystemPrompt(candidateMetrics);

        Map<String, Object> requestBody = Map.of(
            "model", "gpt-4o-mini",
            "messages", List.of(
                Map.of("role", "system", "content", systemPrompt),
                Map.of("role", "user", "content", userQuestion)
            ),
            "response_format", Map.of(
                "type", "json_schema",
                "json_schema", Map.of(
                    "name", "map_metrics",
                    "strict", true,
                    "schema", buildSchema()
                )
            )
        );

        HttpHeaders headers = new HttpHeaders();
        headers.setBearerAuth(apiKey);
        headers.setContentType(MediaType.APPLICATION_JSON);

        HttpEntity<Map<String, Object>> entity = 
            new HttpEntity<>(requestBody, headers);

        ResponseEntity<Map> response = restTemplate.postForEntity(
            "https://api.openai.com/v1/chat/completions",
            entity,
            Map.class
        );

        return parseResponse(response.getBody());
    }

    private List<String> parseResponse(Map<String, Object> response) {
        try {
            Map<String, Object> choice = (Map<String, Object>) 
                ((List<Object>) response.get("choices")).get(0);
            Map<String, Object> message = 
                (Map<String, Object>) choice.get("message");
            String content = (String) message.get("content");

            ObjectMapper mapper = new ObjectMapper();
            Map<String, Object> data = mapper.readValue(content, Map.class);

            return (List<String>) data.get("metrics");
        } catch (Exception e) {
            log.error("Failed to parse structured output", e);
            return Collections.emptyList();
        }
    }
}

This service handles the complete Structured Outputs workflow.

Real-World Example

Input: "How's this month's revenue?" + 50 candidates

GPT Processing:
- "total_revenue" : revenue mentioned (selected)
- "order_count" : related to revenue (selected)
- "page_views" : not about revenue (rejected)
- "cart_abandonment" : not about revenue (rejected)

Output: ["total_revenue", "order_count"]

Complete Flow Example

Let's see the entire system in action:

Step 1: Vector Search
User: "How's this month's revenue?"
↓
Top 10 candidates:
- total_revenue (0.89)
- order_count (0.78)
- gross_profit (0.76)
- net_revenue (0.75)
- sales_amount (0.74)
- purchase_count (0.71)
- transaction_value (0.69)
- page_views (0.45)
- bounce_rate (0.32)
- conversion_rate (0.28)

Step 2: GPT Structured Filtering
Input: 10 candidates
↓
GPT Analysis:
"The question asks about 'revenue', so:
- total_revenue ✓ (direct match)
- net_revenue ✓ (revenue metric)
- order_count ✗ (count, not revenue)
- page_views ✗ (engagement, not revenue)"
↓
Output: ["total_revenue", "net_revenue"]

Step 3: Database Query
SELECT total_revenue, net_revenue 
FROM metrics 
WHERE period = 'current_month'
↓
Final Answer: "This month's total revenue is $125,000..."

Results So Far

Before RAG:

User: "How's this month's revenue?"
AI: "I don't have access to that data."

After Part 1:

User: "How's this month's revenue?"
System: Successfully identified ["total_revenue", "net_revenue"]
→ Can now query the correct metrics

Success Rate: 95% accuracy for flat metrics like "total_revenue", "order_count"

Current Limitation

This system works well for simple metrics but fails on hierarchical ones:

User: "How's my Google traffic?"

Current System:
- Finds "traffic_by_channel" ✓
- Can't identify the "google" sub-metric ✗

We need: "traffic_by_channel|google|advertiser"

The Problem: We need to search not just for parent metrics, but also navigate their hierarchical structure to find specific child metrics like "google" under "traffic_by_channel".

What's Next?

In Part 2, I'll show you how I solved this using:

MetricForest Data Structure: A tree-based system for hierarchical metrics
Neighbor Node Exploration: Finding related parent-child metric pairs
Two-Stage GPT Filtering: Precise classification at each hierarchy level

Example of what Part 2 solves:

User: "How's my Google traffic?"
System: "traffic_by_channel|google|advertiser"

User: "Compare my Facebook traffic to industry average"
System: 
- "traffic_by_channel|facebook|advertiser"
- "traffic_by_channel|facebook|industry"

Result: 98% accuracy even for complex hierarchical queries.

Need Help Implementing RAG?

I help companies integrate AI systems with their existing Spring Boot infrastructure.

Specializing in:

Spring Boot + OpenAI integration
Custom RAG pipelines
E-commerce analytics systems

💭 Questions or feedback? Join the discussion in the comments!

Tags: #rag #system-design #vectordatabase #backend

DEV Community: JunYoungMoon

Are We in an AI Bubble? A Tech Professional's Perspective

The Current Landscape

What Makes This Time Different (And What Doesn't)

The "This Time It's Different" Arguments

The Historical Echoes

The Technical Reality Check

What We Know

What We Don't Know

The Circular Money Problem

The Debt Factor: A Developer's Perspective

What This Means for Your Career

Jobs That Are Already Changing

Jobs That Remain Valuable

The "Skill Obsolescence" Risk

Practical Advice for Engineers

DO:

DON'T:

The Infrastructure Perspective

Questions to Ask Your Employer

The Optimistic Case

Conclusion: The Middle Path

Discussion Questions

Solving Access Denied Issues After SSE Implementation in Spring Security

Background: Why SSE?

The Problem

Root Cause Analysis

The Solution

What is SecurityContextRepository?

Understanding requireExplicitSave

requireExplicitSave(false) (Auto-save mode, default)

requireExplicitSave(true) (Explicit save mode)

Why Re-dispatch Happens in SSE

Verifying the Call Order

Tracking Async Transitions

The Execution Flow

Conclusion

Key Takeaways

Our RAG system still failed on hierarchical metrics — Part 2

How I Built a RAG System That Actually Understands Business Metrics (Part 2: Hierarchical Search)

The Real Problem

💡 The Solution in 3 Moves

Move 1: The Setup (Vector Search)

Move 2: The Play (Smart Classification)

Sub-Play 2.1: Remove the Noise 🧹

Sub-Play 2.2: From Names to Nodes

Sub-Play 2.3: Pick Your Side

Move 3: Checkmate (Relationship Validation)

Step 3.1: Collect the Neighbors

Step 3.2: Filter Parents with GPT

Step 3.3: Final Validation

The Complete Game Replay

Before vs After

Part 1 (Basic Search)

Part 2 (Hierarchical Search)

What I Learned

1. Use AI Where It Shines

2. Two-Stage Filtering is Magic

3. Data Structure = Foundation

4. Filter Domain LATE, Not Early

Real-World Code

Try These Queries

What's Next?

Need Help Implementing RAG?

Our RAG system failed to understand KPIs — Part 1: Metric retrieval design

How I Built a RAG System That Actually Understands Business Metrics (Part 1: Foundation)

The Problem: When AI Can't Find Your Data

Why RAG? Understanding LLM Limitations

The Architecture Overview

Starting Point: Understanding Our Data Structure

Step 1: Text Embedding for Semantic Search

How It Works

OpenAI Embedding API

Cosine Similarity Calculation

Example Output

Step 2: Structured Outputs for Precise Classification

Why Structured Outputs?

The JSON Schema

System Prompt Design

Java Implementation

`requireExplicitSave(false)` (Auto-save mode, default)

`requireExplicitSave(true)` (Explicit save mode)