This article is based on a user-submitted Tencent Cloud Log Service (CLS) best-practice post. The source case focuses on two cost drivers: log storage and traffic. In the production example from the source, storage and traffic accounted for 99.73% of the monthly log-topic cost.
The practical lesson is that CLS cost control is mostly about four choices:
| Lever | What to do |
|---|---|
| Reduce volume | Remove meaningless logs and print only useful logs. |
| Reduce retention | Keep logs for the shortest period that still fits the business scenario. |
| Use low-frequency storage | Move or write logs to low-frequency storage when fast query response, SQL analysis, dashboards, or alerts are not required. |
| Be careful with full-text indexing | Prefer key-value indexes, and avoid rebuilding indexes casually. |
How CLS costs are discussed in the source
The source groups CLS billing into four areas:
- Storage.
- Traffic.
- Calls.
- Partitions.
In the user's production case, the author says storage and traffic were the dominant cost items, so the optimization discussion focuses on those two.
Understand log storage vs index storage
The source separates two concepts:
| Concept | Meaning in the source article |
|---|---|
| Log storage | Raw logs stored in CLS. When collected by LogListener, the source says the logs are compressed by default, commonly with a 4x to 5x compression ratio. Pricing is calculated on the compressed storage size. |
| Index storage | Storage occupied after indexing is enabled for the log topic. The source says this storage measurement is based on uncompressed logs. |
The source also notes that index storage has multiple cases, and the recommendation is to use key-value indexes instead of directly enabling full-text indexes.
The article's core index guidance is:
Be careful with full-text indexing, use key-value indexing well, and do not rebuild indexes casually.
Cost example: standard storage plus full-text indexing
The source uses a real production topic as the example. It was using standard storage, retained logs for 30 days, and enabled full-text indexing.
The source gives these values:
| Item | Value |
|---|---|
| Standard log storage | 7867 GB |
| Standard index storage | 44785 GB |
| Standard storage price used in the calculation | 0.0115 yuan/GB/day |
| Retention period | 30 days |
The monthly storage calculation in the source is:
Standard log storage cost = 7867 * 0.0115 * 30 = 2714 yuan
Standard index storage cost = 44785 * 0.0115 * 30 = 15450 yuan
Total monthly storage cost = 15450 + 2714 = 18164 yuan
The point is not just that storage costs money. The sharper point is that full-text indexing can make index storage much larger than log storage. In this example, index storage was more than five times the log storage size.
When to use standard storage vs low-frequency storage
The source gives this selection logic:
| Storage type | Use when |
|---|---|
| Standard storage | Logs need alerts, dashboards, fast query response, or SQL analysis. The source mentions a high response expectation such as 0.5 seconds for hundreds of millions of records. |
| Low-frequency storage | Logs do not require high query response, SQL analysis, dashboards, or alerts. The source positions it for search and historical backtracking. It mentions a lower response expectation such as 2 seconds for hundreds of millions of records. |
| Standard plus low-frequency | Keep recent data in standard storage and move older data to low-frequency storage. The source notes that this is currently available for standard data older than 7 days. |
The source also says directly created low-frequency topics require log retention of at least 7 days, but a 7-day low-frequency topic can still be cheaper than a 1-day standard topic.
Cost example: 7 days standard, 23 days low-frequency
The source keeps the same example and changes the storage strategy. It uses low-frequency storage at 0.0025 yuan/GB/day and moves logs older than 7 days to low-frequency storage.
Original monthly storage cost = 18164 yuan
Log storage:
(7867 * 0.0115 * 7) + (7867 * 0.0025 * 23) = 1085 yuan
Index storage:
(44785 * 0.0115 * 7) + (44785 * 0.0025 * 23) = 6180 yuan
New monthly storage cost = 1085 + 6180 = 7265 yuan
Monthly difference = 18164 - 7265 = 10899 yuan
The source conclusion is that standard plus low-frequency storage can reduce cost, and some business logs can be written directly to low-frequency storage. The article states this can save about 60% to 70% in the discussed cases.
Traffic optimization: reduce invalid logs and avoid full-text indexing
The source calls traffic another cost dimension that is easy to miss.
There are two directions:
| Direction | Why it matters |
|---|---|
| Reduce invalid raw logs | Controlling log volume at the source is described as the most fundamental way to solve cost issues. |
| Avoid full-text indexing when possible | The source says full-text indexing turns raw logs into index traffic, and that this traffic cost can be much higher than log write traffic. |
The source also warns against casually clicking the "rebuild index" button. Rebuilding an index reconstructs the index for logs in the selected time range and produces index traffic cost. The article says this can be slow and expensive when the data volume is large.
Design log topics by application and scenario
The source compares a CLS log topic to an Elasticsearch index and recommends one topic per application instead of dumping many applications into one broad topic.
For each log collection request, the operations side should evaluate:
| Decision | Source recommendation |
|---|---|
| Retention period | Default to 3 days, up to 7 days. For more than 7 days, configure low-frequency tiering or directly choose low-frequency storage. For more than 15 days, require special explanation. For more than 30 days, consider archiving to COS. |
| Storage type | Default to low-frequency storage. Choose standard storage when alerts, dashboards, or SQL analysis are required, and combine that with the retention rule. |
| Index configuration | Turn off full-text indexing by default. Configure key-value indexes for specific fields. Enable full-text indexing only when it is truly required. The source says full-text indexing can cost 3x to 4x more than key-value indexing. |
This gives each application a topic design that matches its query, alerting, and retention needs.
Split topics by log level when the application has a logging standard
The source uses this example file convention:
/data/logs/user-api-info-2023-05-20.log
/data/logs/user-api-warn-2023-05-20.log
/data/logs/user-api-error-2023-05-20.log
In the example, info logs may be around 10 GB per day, while warn and error logs may be around 100 MB. info logs may be useful for context during troubleshooting, while warn and error logs may need alerts.
The source proposes this topic split:
| Topic | Collection rule | Storage and use |
|---|---|---|
user-api-base-log |
/data/logs/user-api-warn-*.log and /data/logs/user-api-error-*.log
|
Standard storage, 7-day retention, alert rules. |
user-api-low-log |
/data/logs/user-api-info-*.log |
Low-frequency storage, for search and historical backtracking. |
The source then compares cost with a 10-node assumption.
The result in the source is:
Original monthly cost = 5979 yuan
Optimized monthly cost = 3886 yuan
Monthly savings = 5979 - 3886 = 2090 yuan
If there is no logging standard yet
If the application does not already classify logs by level, the source suggests at least agreeing on two categories with developers:
/data/logs/user-api-base-2023-05-20.log
/data/logs/user-api-low-2023-05-20.log
Operations can then create separate base and low topics and configure different collection rules.
Practical checklist
Use this checklist when reviewing a CLS logging cost problem:
- Check whether storage and traffic are the main cost drivers.
- Compare log storage size with index storage size.
- Check whether full-text indexing is enabled.
- Replace full-text indexing with key-value indexing where the query scenario allows it.
- Avoid rebuilding indexes unless the cost and time impact are understood.
- Reduce invalid or excessive raw log output with developers.
- Shorten retention where possible.
- Use low-frequency storage for logs that are mainly used for retrospective search.
- Split topics by application, and then by log level or scenario when useful.
- Use standard storage only where alerts, dashboards, SQL analysis, or fast response are required.
FAQ
Why can index storage become more expensive than log storage?
The source says log storage collected by LogListener is compressed, while index storage is measured based on uncompressed logs. In the example, index storage was more than five times the log storage size.
What is the default index strategy recommended by the source?
Turn off full-text indexing by default and configure key-value indexes for specific fields. Enable full-text indexing only when it is clearly required.
When should logs go directly to low-frequency storage?
When the logs do not need alerts, dashboards, SQL analysis, or high query response, and are mainly used for search and historical backtracking.
What is the most fundamental cost-control method?
The source says reducing invalid raw logs at the source is the most fundamental way to control cost.
Top comments (0)