This article introduces Alibaba Cloud SLS's field index auto-discovery, which automatically indexes new log fields for immediate search and analysis without manual configuration.
You've probably been here. A production incident hits, you follow the trail, and you finally pin it to a log field that a recent release added — a field carrying exactly the information you need. Then you run the query, and the console tells you: this field has no index configured and cannot be analyzed.
Semi-structured logs are increasingly the norm. The write path lets you add fields freely, but the analysis path still depends on indexes configured in advance. Every release that introduces new fields forces you to patch the index by hand, and covering historical data may mean waiting for a reindex. So the flexibility you get at write time never fully carries over to analysis.
AI Agents make that problem worse. An Agent generates new log fields as it works through a task, and across multiple rounds of inference and retrieval it keeps adjusting its query and analysis path based on intermediate results. Both the fields and the analysis requirements appear at runtime. You can't define them up front, and you can't expect anyone to update the index configuration in real time on every iteration.
Alibaba Cloud Simple Log Service (SLS) now offers field index auto-discovery, which lets indexes keep pace with changing data: SLS creates an index for each newly written field and enables statistics on it automatically, with no manual steps. Developers, operations engineers, and Agents can all analyze new fields without configuring field indexes one by one, so the data is usable out of the box.
1. Manual Index Maintenance Is Struggling to Keep Up with Change
The Field Set Is Rarely Complete at Onboarding
To run log queries and SQL analysis through field indexes, you normally pin down the fields first and then configure the indexes. That works well when the log structure is stable and the query requirements are clear.
In real workloads, however, fields tend to accumulate over time. Application upgrades bring new telemetry, new data sources bring new log structures, and some fields appear only under specific conditions. The log samples you see at onboarding rarely cover everything that arrives later.
Index maintenance therefore becomes an ongoing chore. Each new field raises the same questions: will anyone query it, will anyone analyze it, and is the matching index already in place? The faster the business changes, the harder it is to cover every field.
You Discover the Missing Index Only When You Need the Field
You usually find out whether a field has an index only when you try to use it. A new query needs one more filter condition, or an aggregation needs a different grouping dimension — and the field behind it turns out to have no index.
Adding the index then means waiting before you can analyze. Covering data that has already been written may require a reindex, and the larger the historical volume, the longer the wait.
The data is already there, yet you can't analyze it right away. An investigation that could have gone deeper stalls on index preparation.
Agents Make Log Fields and Query Requirements Even Harder to Predict
When an Agent generates and writes the logs, the log content shifts with the task, the tool calls, and the execution results. Different tasks may record different fields, and execution may add new status values and intermediate results along the way, so the log structure can rarely be pinned down in full at onboarding.
When an Agent analyzes the logs, the analysis path is just as hard to define in advance. The Agent drives the task forward in rounds: it reasons, calls tools, and observes the results. It narrows the scope with field-level search, then uses SQL for aggregation, sorting, and comparison. Each round's results determine which fields to query next and which hypotheses to test, so query requirements emerge dynamically as the reasoning proceeds.
If the Agent hits a field without an index mid-analysis, the query may have to pause while the index is added or rebuilt — or even wait for someone to step in — which breaks the continuous flow of reasoning and tool calling.
2. SLS Field Index Auto-Discovery: New Fields Get Indexes Automatically, No Per-Field Setup
Once field index auto-discovery is enabled, SLS picks up newly written fields that have no index yet, builds indexes for them, and updates the index configuration.
Continuous Discovery: Indexes Built and Statistics Enabled Automatically
Auto-discovery keeps scanning newly written logs. When SLS finds a top-level field with no field index configured, it creates one and enables statistics, so the field can take part in field-specific search and SQL analysis.
Auto-discovery runs continuously alongside log writes. Fields your application adds later, fields introduced by other data sources, and fields that did not exist at onboarding are all discovered when they are actually written, so you no longer patch the configuration after every addition.
Application developers no longer have an extra maintenance step when they add a log field. Data consumers can use the new fields in analysis without extra setup and adjust query conditions and aggregation dimensions as they go.
Indexed on Write: Data Written Before the Configuration Becomes Visible Is Covered Too
With field index auto-discovery enabled, SLS starts building the index for a new field the moment the log is written. Index building and configuration visibility are two separate processes: building starts as soon as the data is written, but the configuration takes some time to sync before it becomes visible.
Data written during that window is still indexed. Once the configuration takes effect, you can query and analyze that data directly, with no need to add or rebuild indexes.
Indexed Uniformly as Text: Handles Fields of Uncertain Type
When a field is discovered automatically, its first observed value is a poor guide to its real business type. A new field might arrive with the value 123 in its first log, which looks numeric, yet later writes could just as easily be abc — a random ID, for example. Committing the index type to numeric on first detection would create a type conflict as soon as a non-numeric character showed up.
SLS therefore indexes every auto-discovered field as text. Text accommodates values in any format, so new fields are written and indexed successfully no matter what they contain, avoiding the type conflicts and missed indexes that inaccurate type inference would cause.
Adaptive SQL Type Conversion: No Type Declaration Needed
Storing every such field as text does not make numeric calculations any harder. During SQL analysis, when the SLS engine encounters a numeric expression on an auto-discovered field, it converts the value to a numeric type at runtime and skips values that cannot be converted. The effect is the same as rewriting sum(x) as sum(try_cast(x as double)).
As a result, in most cases you can write SQL against auto-discovered fields as if they were numeric, with no need to change their type first.
3. How to Enable and Use Field Index Auto-Discovery
Enable Field Index Auto-Discovery in the Console
Go to the index configuration page of the target LogStore, confirm that full-text indexing is enabled, select "Auto-discover unspecified fields (full-text index required)", and save. SLS then starts discovering fields in newly written logs on an ongoing basis.
The console entry point is being rolled out to accounts in batches. If it isn't available to your account yet, submit a ticket to request whitelist access.
View the Auto-Discovered Fields
After you enable the feature, keep writing logs to the LogStore. New fields are indexed automatically as they appear, and you can then view them in the auto-discovered fields section of the index configuration page.
These field indexes all use the text type. They use the same delimiters, case-sensitivity setting, and Chinese-character handling as the full-text index, and statistics are enabled by default.
To change the properties of an auto-discovered field, click the + icon above it to promote it to a manually configured field, then edit its properties.
Use Auto-Discovered Fields for Query and Analysis
Once the field configuration is visible, you can use these auto-discovered fields in LogSearch for field-specific search, quick analysis, and regular SQL analysis, exactly as you would with a manually created field index.
Statistics are enabled by default on auto-discovered fields, so SQL analysis works right away. At the index layer these fields are text, but in SQL you can treat them as numeric.
Enable and Manage Auto-Discovered Fields via SDK
The SDK route works in all regions, with no whitelist needed.
In the index configuration JSON, "auto_key_detect": true in the line field enables field index auto-discovery. auto_text_keys lists the auto-discovered fields.
{
"line": {
"auto_key_detect": true,
"auto_text_keys": ["status", "code", "latency"]
},
"keys": {
}
}
The SDK also exposes APIs for these operations. The following example uses the Python SDK (0.9.52 or later):
from aliyun.log import IndexConfig, IndexLineConfig
#Enable automatic field index discovery
client.create_index(
project, logstore,
IndexConfig(line_config=IndexLineConfig(
token_list=[" ", ",", ";", "\n", "\t"],
auto_key_detect=True,
))
)
#Get the configuration related to automatic field index discovery
index = client.get_index_config(project, logstore).get_index_config()
print("Automatic discovery enabled:", index.line_config.auto_key_detect)
print("Discovered fields:", index.line_config.auto_text_keys)
For more configuration details, limits, and FAQs, see the official documentation.
4. Summary
As AI Agents take on task planning and problem analysis, the role of the underlying data system is changing: query requirements can no longer be defined in advance, because they emerge at runtime.
To support this high-frequency, open-ended exploration, a modern data system needs elastic capacity and efficient retrieval for data volumes an LLM cannot consume directly. Just as important, it has to automate the groundwork — field detection and index building — so that analysis isn't held up by wait times or manual intervention.
Field index auto-discovery is one step SLS has taken toward data that adapts on its own. Elastic capacity handles changes in scale, a flexible schema lets the structure evolve, and auto-discovery keeps new fields usable immediately.
When the underlying configuration no longer limits exploration, developers and AI Agents can both focus on the business problem itself. That's what it means for data to be genuinely ready out of the box and open to whatever analysis you need.
Top comments (0)