This article introduces a low-cost cross cloud real-time observability data collection solution using LoongCollector combined with CDN to dramatically cut outbound traffic.
Background
Today, as multicloud strategies become increasingly popular, enterprises often need to deploy operational systems on different cloud platforms. At the same time, enterprises want to uniformly collect observability data to a single platform for analysis and management. However, the high cost of cross-cloud data transmission has become a major obstacle for enterprises when the enterprises implement a unified observability strategy.
By using a CDN as a "stepping stone" for data transmission, you can significantly reduce cross-cloud transmission costs.
Based on this discovery, we designed a LoongCollector + CDN cross-cloud low-cost collection solution:
● As a new generation observability data collector, LoongCollector provides a throughput performance that is 10 times that of similar open-source solutions. In addition, LoongCollector reduces resource usage by more than 50%, which ensures the efficiency and stability of the data link.
● As a traffic exit, CDN utilizes its price advantage and global acceleration capabilities to significantly reduce costs while the transmission quality is ensured.
This solution can significantly reduce cross-cloud data transmission costs, allowing enterprises to realize the vision of a unified observability platform at a lower cost.
Existing Solutions and Pain Points
Scenario 1: Pure Internet
Simple Log Service (SLS) provides a public domain name. Users can directly send data to SLS over the Internet. In addition, SLS does not charge inbound traffic fees.
Pain points
● Cost issue: Although SLS does not charge inbound traffic fees, cross-cloud collection faces outbound traffic fees from the source cloud platform. Taking third-party cloud vendor as an example, the fee for data transmission to the Internet is approximately $0.09/GB. For large-scale data collection scenarios, the cost cannot be ignored.
● Network quality issue: Cross-cloud public network access is significantly affected by network fluctuations. Issues such as packet loss and increased latency may occur, which affects the stability and real-time performance of data collection.
Solution 2: Pure Internet + SLS Accelerated Domain Name
SLS utilizes globally distributed cloud data centers for transfer acceleration. This feature resolves access requests from users worldwide to SLS to the nearest access point via smart routing. This feature uses optimized networks and protocols to greatly improve access speed.
Pain points
● Double costs: In addition to the outbound traffic fees of the source cloud platform, you must also pay the acceleration fees of DCDN. Consequently, the overall cost further increases.
Solution 3: Cross-cloud Leased Line Connection
You can establish cross-cloud private network connectivity through the leased line services of cloud service providers, such as Alibaba Cloud Express Connect.
Pain points
● High construction cost: Leased line construction requires a large one-time investment, including port fees and leased line rental fees.
● Complex maintenance: A professional team is required to maintain the leased line connection, resulting in high O&M costs.
● Poor flexibility: The leased line bandwidth is fixed, which makes it difficult to meet burst traffic requirements.
● Long construction cycle: The process from request to activation usually takes weeks or even months.
Cross-cloud Low-cost Collection Solution
CDN products usually provide tiered pricing and batch discounts. As usage increases, the unit cost further decreases. By using the acceleration link of the CDN and configuring the SLS as the origin, you can reuse the forwarding link of the CDN to achieve the following advantages:
● Cost optimization: You can utilize the price advantage of the CDN to reduce data transmission costs.
● Easy implementation: You do not need to build a leased line. The configuration is simple, and the service can be quickly published.
● Scalability: You can use resources on demand without reserving bandwidth. This allows you to flexibly handle traffic fluctuations.
Prices for CloudFront regional data transmission to the origin:
Overall Solution
This solution takes CloudFront as an example. The overall collection solution is shown in the graph:
Architecture
Third-party cloud vendor (LoongCollector)
● Collection/Forwarding program deployed on third-party cloud vendor
● Main responsibilities:
Collect logs or data from local sources or applications.
Package data according to the SLS write protocol (HTTP POST).
Send data to the target SLS project.
CloudFront
● Serving as the transit entrance of the data link, it provides a unified domain name access point and point of presence (POP) access capabilities.
● Main responsibilities:
Receive Requests (HTTP/HTTPS) from LoongCollector.
Forward to the origin (the origin here is the writing endpoint of Alibaba Cloud SLA) based on behavior rules.
SLS
● Serves as a log/data receiving and storage analysis platform.
● Exposes HTTP/HTTPS write APIs externally.
● After writing, Data falls into the specified project (in the figure)/Logstore.
SLS ConfigServer (Management Endpoint)
● Used to distribute "control plane" capabilities such as collection configuration, heartbeat, metadata management, and authentication information refresh.
● Low requirements for data volume, and relatively controllable requirements for real-time performance.
Link Layering: Control Link & Data Link
A. Control Link (Control Plane) - Direct Connection to the Internet
Features: small request volume, small data volume, and insensitive to bandwidth.
● LoongCollector accesses the SLS ConfigServer directly via the Internet.
● Typical actions include:
Pull collection configuration/rules.
● Reasons for choosing direct connection to the Internet:
Control traffic is small, and requirements for cost and link quality are not high.
The architecture is simpler (reduces transit layers).
B. Data Link (Data Plane) - Forward to SLS via CloudFront
Features: continuous writing, sensitive to stability/connectivity, and potential cross-border nNetwork fluctuations.
● LoongCollector sends log data to the CloudFront domain name via HTTP POST.
● CloudFront then forwards the request (origin fetch) to the SLS write endpoint.
● SLS writes the data to the specified project/Logstore after receiving the data.
CloudFront Configurations
This example collects data to a project in the SLS China (Shanghai) region.
Source configurations
Note:
Do not include the project prefix in the SLS domain name.
When CloudFront accesses the SLS domain name, you can use either HTTP or HTTPS.
Behavior Configurations
Note:
The CDN caches response content by default, but LoongCollector sends data via POST requests, so you need to configure it to not cache.
For requests from CloudFront to the SLS domain name, you need to forward all headers except HOST.
CloudFront Domain Validation
Directly curl the CloudFront domain name. If the following response is returned, the configuration has succeeded.
LoongCollector Configurations
Use HTTP Protocol to Send Data
# /usr/local/ilogtail/ilogtail_config.json
{
"primary_region" : "cn-shanghai",
"config_servers" :
[
"https://logtail.cn-shanghai.log.aliyuncs.com"
],
"data_servers" :
[
{
"region" : "cn-shanghai",
"disable_subdomain" : true,
"endpoint_list": [
"http://xxx.cloudfront.net"
]
}
],
...
}
Key configuration description:
● In config_servers, configure the SLS Internet domain name. The standard format is logtail.${region}.log.aliyuncs.com.
● In data_servers:
You only need to configure the primary region. Set endpoint_list to the HTTP CloudFront domain name.
disable_subdomain: true (disable subdomain forwarding).
Use HTTPS protocol to send Data
# /usr/local/ilogtail/ilogtail_config.json
{
"primary_region" : "cn-shanghai",
"config_servers" :
[
"https://logtail.cn-shanghai.log.aliyuncs.com"
],
"data_servers" :
[
{
"region" : "cn-shanghai",
"disable_subdomain" : true,
"endpoint_list": [
"https://xxx.cloudfront.net"
]
}
],
"enable_host_ip_replace": false,
...
}
Key configuration description:
● In config_servers, configure the SLS Internet domain name. The standard format is logtail.${region}.log.aliyuncs.com.
● In data_servers:
You only need to configure the primary region. Set endpoint_list to the HTTPS CloudFront domain name.
disable_subdomain: true (disable subdomain forwarding).
enable_host_ip_replace: false (Disable internal DNS resolution of LoongCollector).
Configure Resource Parameters
LoongCollector is deployed on EC2 or nodes. Therefore, you need to estimate the raw data volume of logs collected by a single machine and adjust resource parameters. For more information, you can refer to the help document.
Note: When LoongCollector sends data, it uses LZ4 for compression by default. For log data, it can achieve a compression ratio of 5 to 10 times.
Network Quality Test Results
Test scenario:
● Collect data from EC2 in the South Korea region to SLS in the South Korea region
● About 800 KB per packet after compression
You can see that in the same region scenario, the access quality of CloudFront is basically on par with direct public network access, and the HTTP access latency is slightly lower.
Limitations:
● LoongCollector ≥ V3.3.0
● Currently, only log collection is supported. Data such as time series and host monitoring is not supported to be sent via CDN links yet.
● This feature is being gradually released in grayscale by region. To use it, contact SLS tech support helpdesk.
Summary and Outlook
Scenario Summary
The real-time collection solution of cross-cloud low-cost observability data introduced in this topic achieves the following by using the combination of CDN and LoongCollector:
Cost reduction: Compared with the pure Internet solution, it significantly reduces cross-cloud data transmission costs.
Performance improvement: By leveraging the global nodes of CDN and the high performance of LoongCollector, it improves the speed and stability of data collection.
Easy implementation: Simple configuration, no need to build leased lines, and it can be quickly published.
Flexible extension: Pay-as-you-go, automatic scalability, adapting to traffic fluctuations.
As a new generation of unified observability agent, LoongCollector will continue to be dedicated to providing users with high-performance, low-cost, and easy-to-use cross-cloud data collection solutions, helping enterprises build a unified observability platform.
References
● LoongCollector official documentation
● CloudFront official documentation
● Alibaba Cloud SLS official documentation
Top comments (0)