Natural language analytics looks simple from the outside:
User asks a question.
The system generates SQL.
The database returns an answer.
In real enterprise environments, this breaks quickly.
The reason is not just SQL generation. The reason is semantic ambiguity.
When a user asks:
Show sales by product code for last month.
The system needs to resolve several questions before it can safely generate a query:
- What does "sales" mean?
- Is it gross sales, net sales, recognized revenue, bookings, or paid order amount?
- What does "product code" mean?
- Is it internal product code, SKU, marketplace product code, barcode, or ERP material number?
- Which date field should represent "last month"?
- Which tables contain the right facts and dimensions?
- Which join path is trusted?
- Is the current user allowed to access the required data?
This is why a semantic layer is becoming a critical part of natural language analytics.
But not all semantic layers are designed for the same job.
Databricks is building semantic capabilities natively inside its lakehouse platform. Semora is building an independent semantic layer for complex, cross-platform enterprise data. From an engineering perspective, the interesting question is not "Which product has a better chat interface?"
The better question is:
What system architecture makes natural language analytics both fluent and accurate?
## The Core Problem: Language Models Should Not Guess the Data Model
A large language model can usually understand that these phrases are related:
- product code
- product ID
- item number
- SKU
- material number
But in a real company, those phrases may not be interchangeable.
One table may contain:
product_code
product_id
sku_id
barcode
platform_item_id
material_number
If the system maps "product ID" to product_code just because the words sound close, the answer may be fluent but wrong.
On the other hand, if the system asks the user to clarify every similar term, the experience becomes painful.
The engineering challenge is to build a runtime that can decide:
- When can we answer directly?
- When should we show the assumption?
- When should we ask a lightweight clarification?
- When must we require explicit confirmation?
- How do we save the user's answer so we do not ask again?
This is where Semora can differentiate.
## Databricks: Platform-Native Semantic Context
Databricks has been expanding its semantic layer and natural language analytics stack through Unity Catalog business semantics, metric views, Genie, and Genie Agents.
The Databricks approach is powerful because semantic assets live close to the governed data platform.
A Databricks-style flow can look like this:
User question
-> Genie Agent scope
-> Unity Catalog metadata
-> Metric views
-> Table and column descriptions
-> Trusted queries and examples
-> SQL generation
-> Query inspection
-> Governed answer
This works especially well when:
- The company already uses Databricks as the main analytical platform.
- Tables, permissions, lineage, and metrics are managed in Unity Catalog.
- Business domains are curated into focused Genie Agents.
- Data teams maintain metric views, examples, and trusted assets.
The key engineering lesson is that Databricks does not treat the language model as the only source of intelligence. It surrounds the model with governed context.
That context reduces the search space.
If a user is asking inside a Sales Genie Agent, "revenue" can be interpreted using sales-domain semantics. If they ask inside a Finance context, the default meaning may be different.
This domain scoping is one of the most important reasons a natural language analytics experience feels smooth.
## Semora: Independent Semantic Runtime
Semora has a different opportunity.
Many enterprises do not have one clean analytical platform. They have multiple warehouses, operational databases, business intelligence tools, spreadsheets, and legacy systems. Field names are inconsistent. Foreign keys are missing. Business terms vary by department.
In this environment, an independent semantic layer is valuable because it can sit above the data platforms instead of being tied to only one.
A Semora-style flow can look like this:
User question
-> Intent classification
-> Role and permission scope
-> Business domain routing
-> Synonym and semantic retrieval
-> Business concept mapping
-> Certified field selection
-> Relationship discovery
-> Trusted question template matching
-> Query generation
-> Query validation
-> Answer with lineage
-> Clarification memory update
The important word here is runtime.
It is not enough to have pages for synonyms, business semantics, data definitions, question templates, and knowledge bases. Those assets must be actively retrieved, ranked, and applied while the user is asking a question.
Otherwise, the product can have many semantic assets and still feel rough.
## Runtime Component 1: Intent Classification
The first step should not be SQL generation.
The first step should be intent classification.
For example, the user may ask:
What questions can you answer for me?
This is not a data query. It is a capability discovery question.
A poor system routes it into the query pipeline, fails to find a metric, and asks for clarification.
A better system recognizes the intent and answers based on the user's role, permissions, business domains, and available question templates.
Useful intent classes include:
DATA_QUERY
CAPABILITY_DISCOVERY
METRIC_EXPLANATION
FIELD_EXPLANATION
DOMAIN_EXPLORATION
TEMPLATE_RECOMMENDATION
PERMISSION_SCOPE_EXPLANATION
PRODUCT_HELP
For CAPABILITY_DISCOVERY, the response should be generated from available semantic assets:
- Accessible business domains
- Certified metrics
- Question templates
- Frequently used questions
- Available dimensions
- Permission boundaries
Example response:
Based on your current access, I can answer questions about sales, products, inventory, and channels. You can ask about sales trends, product rankings, inventory shortages, channel comparisons, and customer repeat purchase behavior.
This small routing layer can dramatically improve first-time user experience.
## Runtime Component 2: Business Domain Routing
Even if Semora keeps one universal question box, the backend should not treat all data as one global space.
The system should route each question into one or more business domains.
For example:
"sales by product code"
-> Sales domain
-> Product domain
"inventory risk by SKU"
-> Inventory domain
-> Product domain
"recognized revenue by customer"
-> Finance domain
-> Customer domain
Domain routing should use:
- User role
- Permission scope
- Question text
- Matched metrics
- Matched templates
- Historical user behavior
- Business glossary terms
The user can still see one clean interface. Internally, the system should load a scoped semantic context.
This reduces ambiguity before the language model starts planning the query.
## Runtime Component 3: Multi-Path Semantic Retrieval
A common failure mode is exact or weak lexical matching.
If the physical field is called product_code, and the user says "product number," the system may fail unless that synonym was manually configured.
This does not scale.
Semora should use multi-path retrieval:
User phrase
-> Exact match
-> Token match
-> Synonym match
-> Embedding similarity
-> Field description match
-> Sample value match
-> Historical clarification match
-> Question template match
-> Domain default match
For the phrase "product code," candidates might include:
Business concept: Product Code
Fields:
- dim_product.product_code
- order_items.product_code
- inventory_sku.sku_code
- marketplace_items.platform_item_id
- product_barcodes.barcode
The retrieval layer should not immediately decide. It should return candidates with evidence.
Example evidence:
{
"candidate": "dim_product.product_code",
"business_concept": "Product Code",
"evidence": {
"synonym_match": ["product code", "item code"],
"domain": "Product",
"field_description": "Internal product code used across sales and inventory",
"historical_confirmations": 17,
"certification_status": "certified"
},
"confidence": 0.93
}
This gives the language model structured context instead of asking it to guess from raw schema names.
## Runtime Component 4: Confidence-Based Clarification
Clarification should be a decision, not a default.
A useful policy could be:
confidence >= 0.85
-> answer directly and show the assumption
0.60 <= confidence < 0.85
-> answer with a lightweight change option, or ask a short clarification
confidence < 0.60
-> require clarification
high-risk metric or high-risk join
-> require confirmation even if confidence is moderate
For a high-confidence case:
I interpreted "product code" as the certified Product Code concept and used the internal product code field.
For an ambiguous case:
Which product identifier do you mean?
- Internal product code
- SKU number
- Marketplace product code
- Barcode
- ERP material number
This is better than showing raw column names to a business user.
Technical users can still inspect the underlying fields, joins, and SQL.
## Runtime Component 5: Trusted Question Templates
Question templates should not be simple prompt examples.
They should be trusted analytical paths.
For example, a template for:
Top-selling products in a time period
Should include:
template_id: top_selling_products
intent: ranking
metric:
name: Sales Quantity
certification: certified
dimensions:
- Product Code
- Product Name
time:
default_field: Paid Time
filters:
- Paid Orders Only
relationships:
- order_items -> dim_product by product_code
parameters:
- time_range
- channel
- category
validation:
- sales_quantity >= 0
- product_code is not null
owner: Sales Analytics Team
status: certified
When the user asks:
Show last month's product code sales ranking.
The system should match the template first, extract parameters, and reuse the certified query path.
The language model should not rebuild the business logic from scratch.
## Runtime Component 6: Relationship Discovery and Join Safety
Natural language analytics systems often fail silently when joins are wrong.
A query can run successfully and still be incorrect because:
- It joined at the wrong grain.
- It created duplicate rows.
- It used a weak relationship.
- It joined through the wrong entity.
- It mixed internal identifiers with external identifiers.
For Semora, relationship discovery should be a major differentiator.
A relationship engine should evaluate:
- Field name similarity
- Data type compatibility
- Value overlap
- Inclusion ratio
- Cardinality
- Null rate
- Uniqueness
- Historical usage
- Human approval
- Domain relevance
The output should be explainable:
{
"join_path": [
"order_items.product_code",
"dim_product.product_code"
],
"confidence": 0.91,
"evidence": {
"value_overlap": "high",
"right_side_uniqueness": "strong",
"approved_by": "data_owner",
"fanout_risk": "low"
}
}
This is where independent semantic layers can create real value. Platform-native semantic layers work well when relationships are already modeled. Independent semantic layers can help discover and govern relationships where metadata is incomplete.
## Runtime Component 7: Query Validation
SQL generation is not the end of the pipeline.
The system should validate the generated query before presenting the answer.
Validation checks may include:
- Does the user have permission to access every field?
- Is the selected metric certified?
- Is the date field appropriate for the metric?
- Are filter values valid?
- Is the join path approved?
- Is there fanout risk?
- Is the aggregation grain correct?
- Is the result within a reasonable historical range?
- Did the query return no rows because of a value mapping issue?
This validation layer is one of the most important differences between a demo and a production-ready product.
## Runtime Component 8: Clarification Memory
Every clarification should become a reusable asset.
If a user confirms:
By "product code," I mean the internal product code.
The system should save:
user_phrase: product code
standard_concept: Product Code
field_mapping: dim_product.product_code
business_domain:
- Product
- Sales
confirmed_by: user_or_owner
scope: user | team | organization
status: draft | reviewed | certified
usage_count: 1
Over time, repeated confirmations should create candidate semantic rules. Data owners can review and promote them.
This makes the system smoother with use.
The goal is not to ask better clarification questions forever. The goal is to ask fewer clarification questions over time because the semantic layer is learning.
## A Practical Architecture
Here is a practical architecture for a natural language analytics runtime:
+----------------------+
| User Question |
+----------+-----------+
|
v
+----------------------+
| Intent Classifier |
+----------+-----------+
|
+---------------------+---------------------+
| |
v v
+------------------------+ +------------------------+
| Non-query Intents | | Data Query Intent |
| Help, scope, examples | | Metrics, filters, SQL |
+------------------------+ +-----------+------------+
|
v
+---------------------------+
| Domain Router |
+------------+--------------+
|
v
+---------------------------+
| Semantic Retrieval |
+------------+--------------+
|
v
+---------------------------+
| Template Matching |
+------------+--------------+
|
v
+---------------------------+
| Confidence Decision |
+------------+--------------+
|
+--------------------------+--------------------------+
| |
v v
+------------------------+ +------------------------+
| Clarification | | Query Planning |
+-----------+------------+ +-----------+------------+
| |
v v
+------------------------+ +------------------------+
| Semantic Memory | | SQL Generation |
+------------------------+ +-----------+------------+
|
v
+------------------------+
| Query Validation |
+-----------+------------+
|
v
+------------------------+
| Answer + Lineage |
+------------------------+
The language model appears in multiple places:
- Intent classification
- Query planning
- Clarification generation
- Parameter extraction
- Result explanation
- Follow-up question handling
But the model is not operating alone. It is grounded by semantic assets, trusted templates, permissions, relationship evidence, and validation rules.
That is the difference between natural language analytics as a demo and natural language analytics as enterprise infrastructure.
## Where Semora Can Stand Apart
Databricks has a strong platform-native story. If a company has standardized its data and governance inside Databricks, that integration is hard to beat.
Semora should not try to be a smaller version of Databricks.
It should focus on the problems that appear when the enterprise data world is messy:
- Business terms vary across teams.
- Field names do not match user language.
- Relationships are not fully documented.
- Metrics conflict across systems.
- Users need answers across multiple platforms.
- Clarification should become governed semantic memory.
- Natural language queries need explainable lineage.
This is a valuable and distinct engineering problem.
The best semantic layer is not the one that lets the model guess more freely. It is the one that gives the model enough context, evidence, and constraints to answer correctly.
## Conclusion
Natural language analytics is not solved by connecting a language model to a database.
It requires a semantic runtime.
That runtime must classify intent, route by business domain, retrieve semantic assets, match trusted templates, evaluate confidence, clarify only when necessary, generate governed queries, validate results, and save human feedback as reusable semantic knowledge.
Databricks approaches this through a platform-native lakehouse stack.
Semora's opportunity is to build an independent semantic intelligence layer for enterprises whose data, terminology, and relationships span many systems.
For developers and data engineers, the lesson is clear:
Do not make the model guess the enterprise data model. Build the semantic system that lets it reason with evidence.
Top comments (0)