DEV Community

ObservabilityGuy
ObservabilityGuy

Posted on • Edited on

Cracking Five Challenges in Heterogeneous Log Cleaning: Fully Boosting O&M Data Observability

#observability #programming #webdev #javascript

In the new energy vehicle (NEV) industry, the operation and maintenance (O&M) system spanning vehicle, mobile, and cloud ends generates massive amounts of log data every day. These logs cover a wide range and come in diverse formats. Most of them are weakly structured or even unstructured data, with inconsistent fields and variable structures, posing great challenges to subsequent data processing and analysis. Efficiently performing standardized cleaning and structured conversion of these complex and heterogeneous log data is a key foundation for achieving automated O&M management, intelligent alerting, and business insights.

Leveraging the one-stop data cleaning capabilities of Log Service (SLS), complex raw logs are efficiently converted into a unified and analyzable data format. Data aggregation and analysis are implemented by using SQL, with monitoring alerts that can be flexibly configured. This comprehensively enhances the observability and responsiveness of O&M data, enabling in-depth insights into the operational status of NEV systems and their intelligent management.

Five Challenges in Log Data Analysis of NEVs

  1. Cleaning Challenges of Mixed Log Data To ensure full-link observability, logs from vehicle, mobile, and backend cloud services are usually unified into the same log platform (such as Logstore). Log collection methods are varied, such as SDK, Logtail, OSS Bucket, and open-source agents. The collected logs come in vastly different formats (such as syslog, Kubernetes logs, KV, and JSON), resulting in inconsistent log field structures. What's more, these logs are even stored mixedly in the same Logstore, greatly complicating log parsing, field standardization, and query analysis. In addition, sensitive fields in the data (such as VIN codes, location information, phone numbers, and ID numbers) need to be desensitized during the collection phase to ensure the security of user privacy.

  1. Challenge of Data Aggregation The business and O&M of NEVs are often scattered across different regions and cloud platforms, requiring data from various regions to be efficiently and securely aggregated into specific centers for analysis. Such data synchronization and integration across regions and cloud environments are highly prone to issues such as network latency, transmission security, and data consistency.

  1. Pressure of Achieving a Second-level Response In O&M scenarios, high speed is required for log query and analysis, especially when urgent customer complaints or security incidents occur. It is imperative to ensure that logs can be collected in real-time and queried instantly, enabling frontline O&M personnel to locate and resolve issues within seconds.

  1. Challenge of Cost Control Large-scale data circulation and temporary storage across multiple stages will significantly increase storage costs. Therefore, enterprises aim to reduce unnecessary intermediate storage in all stages such as log circulation and processing, with logs occupying less temporary Logstore space, thereby lowering overall operating costs.

  1. Challenge of Transparent O&M Processes Complex data circulation and processing typically involve a multitude of Logstores, as well as various tasks such as processing, scheduled SQL, and importing, with intricate correlations between them. In such cases, the flow of data between Logstores is often difficult to sort out clearly, which not only significantly increases the difficulty of overall O&M but also greatly reduces the manageability of tasks.

SLS Data Cleaning Solution
To address the complexity of logs in the system O&M of NEVs, SLS provides the following solutions to address the preceding five data challenges, including: SLS's full-link log access and processing capabilities, cross-regional data processing capabilities, high-performance data cleaning, optimization of data circulation and storage costs, and visualization of SLS Logstore lineage relationships.

Cleaning challenges of mixed log data: SLS full-link log access and processing capabilities
Unified access to multi-source data: SLS allows you to write logs from vehicles, mobile phones, and the cloud to SLS Logstores by using SDKs, Logtail, OSS buckets, and other open protocols, such as Kafka and Syslog, achieving query logs from all sources and ensuring full-link coverage.

Intelligent extraction of complex formats: SLS provides powerful built-in field extraction capabilities for a variety of weakly structured logs, such as JSON, KV, syslog, and Kubernetes (k8s) logs. It supports log format identification and processing driven by custom regular expressions and conditional statements (let syntax), enabling shunting and parsing even when there is no clear field differentiation.

Syslog protocol log parsing

K8 container logs: mixed parsing of KV and JSON

Privacy-compliant desensitization: SLS provides a dynamic desensitization mechanism based on regular expressions. This mechanism removes sensitive information such as VIN, location tracks, and mobile phone numbers during data collection or data warehousing, thus meeting data security and privacy compliance requirements.

Data aggregation challenges: SLS cross-region data processing capabilities

SLS supports cross-region data processing, enabling business data from different regions and cloud environments to be securely and efficiently aggregated into the same Logstore for analysis following specified rules and permissions. This eliminates regional data silos and supports centralized management and analysis.

Pressure of Achieving a Second-level Response: SLS High-performance Data Cleaning

SLS provides powerful real-time processing and query capabilities. Data processing supports second-level latency, with the processing speed of a single job being 2Mil/s (millions of records per second, equivalent to 2 GB/s). In addition, the processing speed can be linearly increased based on the number of shards. This enables quick problem localization in scenarios such as emergencies, customer complaints, and business peak periods, significantly improving O&M efficiency and customer response speed.

Cost Control Challenges: SLS Data Flow and Storage Optimization

● SLS Ingest Processor has the same data processing capabilities as the data processing processor, enabling direct data cleaning, desensitization, and denoising during log writing.
● Cost optimization by using the Ingest Processor: SLS uses the "zero temporary storage" Ingest Processor to implement data processing and field standardization during data writing without the need for temporary Logstores transit. This reduces storage costs and system complexity. When there is no need to retain raw logs, the storage overhead of temporary Logstores can be saved.

Difficulty in O&M Process Transparency: SLS Logstore Lineage Visualization
Alibaba Cloud Observable MCP Server provides tools at the Project, Logstore, and task levels. Through MCP tools, you can list your project lists and the task lists corresponding to each project (including data import, data processing, data delivery, and scheduled SQL). Through LLM's parsing of task configurations, the source Logstore and target Logstore can be identified. In addition, you can interact with agents to generate text compatible with plotting tools like Mermaid. Pasting this text into the tool will visualize the data flow between Logstores.
Note: The SLS task list tool is in beta testing. For access, please contact SLS technical support.

graph TD
    %% Source: OSS -> Target: Logstore
    A[oss: gyhangzhou] --> B[hcltestcd101.test1]
    C[oss: gyhangzhou] --> D[hcltestcd101.test2]

    %% Logstore -> ETL
    B -->|test1| E[etl: etl-1748412289-337270]
    D -->|test2| F[etl: etl-1748412289-337270]

    %% ETL -> Targets
    E --> G[hcltestcd103.test1]
    E --> H[hcltestcd1102.test1]
    E --> I[hcltestcd103.test2]
    E --> J[hcltestcd1102.test2]

    F --> K[hcltestcd103.test1]
    F --> L[hcltestcd101.test1]
Enter fullscreen mode Exit fullscreen mode

Display effect through the tool website:

Next, taking NEV charging as an example, this article explains how to analyze the charging scenario by accessing various heterogeneous data sources, cleaning the data with Ingest Processor, and ultimately completing the analysis.

Practical Analysis of Log Cleaning in NEV Charging Scenarios

When the user drives the vehicle to the charging station, the system automatically records the positioning track. After the user parks and gets off the vehicle, they plug in the charging gun. At this time, the vehicle or charging pile detects the gun insertion event and automatically initiates cloud authentication. The user starts charging through QR code scanning, in-vehicle screens, or Apps, and the operation information interacts with the charging pile, the vehicle, and the cloud, generating logs while the backend simultaneously processes logic such as authentication and billing. During the charging process, the vehicle and the charging pile continuously report charging status and alarms, while the App or cloud synchronously displays the charging progress and costs. After charging is completed, the relevant devices report the "charging completion" log.

Introduction to Ingest Processor Solution

Throughout the entire interaction process of NEVs, multiple terminals and services are involved, including charging piles, vehicles, mobile phones, and backend services. Among them: Charging piles can upload logs to SLS Logstore through methods such as SDK, MQTT, Logtail, and periodic OSS uploads. Mobiles use SDK to directly transmit logs to SLS Logstore. Vehicles upload logs by periodically uploading files to OSS, then using SLS to import OSS files in real-time to SLS Logstore. Logs from in-vehicle backend services deployed in K8s clusters are directly collected to SLS Logstore through SLS's Logtail-ds.

By aggregating logs from multiple terminals into a single Logstore, we can achieve comprehensive charging analysis and fault diagnosis. However, due to significant differences in log formats across various terminals, data cleaning and standardization are required. Here, we use SLS Ingest Processor with SPL syntax to structure weakly structured data, facilitating subsequent data monitoring, O&M analysis, and alerting.

Log Cleaning of Ingest Processor

  1. Logs of vehicle-side inbound and outbound trajectories (JSON format) 
{
  "ts": "2024-06-17T11:26:00Z",
  "vin": "LVSHF6196L1234567",
  "event": "arrive_station",
  "station_id": "SH_CHG_001",
  "location": {
    "lat": 31.2362,
    "lon": 121.4798
  }
}
Enter fullscreen mode Exit fullscreen mode

Ingest Processor statement:

* | parse-json content 
  | parse-json location 
  | project-away content,location
  | extend __tidme__ = cast(to_unixtime(date_parse(ts, '%Y-%m-%dT%H:%i:%sZ')) as bigint)
Enter fullscreen mode Exit fullscreen mode

Processing result:

__time__: 1718623560
ts: 2024-06-17T11:26:00Z
vin: LVSHF6196L1234567
event: arrive_station
station_id: SH_CHG_001
lat: 31.2362
lon: 121.4798
Enter fullscreen mode Exit fullscreen mode
  1. Charging gun insertion (vehicle-side/pile-side event) Charging pile-side API logs: 
2024-06-17T11:27:11Z station=SH_CHG_001 event=plug_in vin=LVSHF6196L1234567 result=success connector=DC
Enter fullscreen mode Exit fullscreen mode

Ingest Processor statement:

* | parse-regexp content, '(\d+-\d+-\d+T\d{2}:\d{2}:\d{2}Z) (.*)' as time, content
  | parse-kv content, ' ', '=' 
  | project-away content
  | extend __time__ = cast(to_unixtime(date_parse(time, '%Y-%m-%dT%H:%i:%sZ')) as bigint)
Enter fullscreen mode Exit fullscreen mode

Processing result:

__time__: 1718623631
connector: DC
event: plug_in
result: success
station: SH_CHG_001
time: 2024-06-17T11:27:11Z
vin: LVSHF6196L1234567
Enter fullscreen mode Exit fullscreen mode
  1. Charging initiation (mobile phone QR code scanning/in-vehicle screen operation) Mobile App logs (in KV format): 
2024-06-17 11:27:12 user_id=87234 action=charge_start vin=LVSHF6196L1234567 station_id=SH_CHG_001 os=android13
Enter fullscreen mode Exit fullscreen mode

Ingest Processor statement:

* | parse-regexp content, '(\d+-\d+-\d+ \d{2}:\d{2}:\d{2}) (.*)' as time, content
  | parse-kv content, ' ', '=' 
  | project-away content
  | extend __time__ = cast(to_unixtime(date_parse(time, '%Y-%m-%d %H:%i:%s')) as bigint)
Enter fullscreen mode Exit fullscreen mode

Processing result:

__time__: 1718623632
action: charge_start
os: android13
station_id: SH_CHG_001
time: 2024-06-17 11:27:12
user_id: 87234
vin: LVSHF6196L1234567
Enter fullscreen mode Exit fullscreen mode

Cloud authentication/billing service logs (with embedded JSON):

2024-06-17T11:27:13Z pod=charge-gw ns=prod log={"user_id":"87234","vin":"LVSHF6196L1234567","api":"/api/charge/start","station_id":"SH_CHG_001","result":"started","ts":"2024-06-17T11:27:13Z"}
Enter fullscreen mode Exit fullscreen mode

Ingest Processor statement:

* | parse-regexp content, '(\d+-\d+-\d+T\d{2}:\d{2}:\d{2}Z) (.*)' as time, content
  | parse-kv content, ' ', '=' 
  | project-away content
  | parse-json log
  | project-away log
  | extend __time__ = cast(to_unixtime(date_parse(time, '%Y-%m-%dT%H:%i:%sZ')) as bigint)
Enter fullscreen mode Exit fullscreen mode

Result:

__time__: 1718623633
api: /api/charge/start
ns: prod
pod: charge-gw
result: started
station_id: SH_CHG_001
ts: 2024-06-17T11:27:13Z
user_id: 87234
vin:  LVSHF6196L1234567
Enter fullscreen mode Exit fullscreen mode

Vehicle-side charging initiation logs:

{
  "ts": "2024-06-17T11:27:14Z",
  "vin": "LVSHF6196L1234567",
  "event": "charging_start",
  "station_id": "SH_CHG_001",
  "battery": {
    "voltage": 381.0,
    "current": 25.1,
    "soc": 45
  }
}
Enter fullscreen mode Exit fullscreen mode

cessor statement:

* | parse-json content
  | project-away content
  | parse-json battery
  | project-away battery
  | extend __time__ = cast(to_unixtime(date_parse(ts, '%Y-%m-%dT%H:%i:%sZ')) as bigint)
Enter fullscreen mode Exit fullscreen mode

Result:

__time__: 1718623634
current: 25.100000
event: charging_start
soc: 45
station_id: SH_CHG_001
ts: 2024-06-17T11:27:14Z
vin: LVSHF6196L1234567
voltage: 381.000000
Enter fullscreen mode Exit fullscreen mode
  1. Charging in progress (regular reporting by the vehicle of charging process data) 
{
  "ts": "2024-06-17T11:47:14Z",
  "vin": "LVSHF6196L1234567",
  "event": "charging_status",
  "station_id": "SH_CHG_001",
  "battery": {
    "voltage": 410.6,
    "current": 3.2,
    "soc": 88
  }
}
Enter fullscreen mode Exit fullscreen mode

Ingest Processor statement:

* | parse-json content
  | project-away content
  | parse-json battery
  | project-away battery
  | extend __time__ = cast(to_unixtime(date_parse(ts, '%Y-%m-%dT%H:%i:%sZ')) as bigint)
Enter fullscreen mode Exit fullscreen mode

Result:

__time__: 1718624834
current: 3.200000
event: charging_status
soc: 88
station_id: SH_CHG_001
ts: 2024-06-17T11:47:14Z
vin: LVSHF6196L1234567
voltage: 410.600000
Enter fullscreen mode Exit fullscreen mode
  1. Charging completion (automatic completion/manual stop) 
{
  "ts": "2024-06-17T12:05:56Z",
  "vin": "LVSHF6196L1234567",
  "event": "charging_stop",
  "station_id": "SH_CHG_001",
  "battery": {
    "voltage": 415.0,
    "current": 0.0,
    "soc": 92
  },
  "result": "success"
}
Enter fullscreen mode Exit fullscreen mode

Ingest Processor statement:

* | parse-json content
  | project-away content
  | parse-json battery
  | project-away battery
  | extend __time__ = cast(to_unixtime(date_parse(ts, '%Y-%m-%dT%H:%i:%sZ')) as bigint)
Enter fullscreen mode Exit fullscreen mode

Result:

__time__: 1718625956
current: 0.000000
event: charging_stop
result: success
soc: 92
station_id: SH_CHG_001
ts: 2024-06-17T12:05:56Z
vin: LVSHF6196L1234567
voltage: 415.000000
Enter fullscreen mode Exit fullscreen mode

Pile-side log:

2024-06-17T12:05:57Z station=SH_CHG_001 event=charge_end vin=LVSHF6196L1234567 
result=success duration=2256s total_kwh=28.7
Enter fullscreen mode Exit fullscreen mode

Ingest Processor statement:

* | parse-regexp content, '(\d+-\d+-\d+T\d{2}:\d{2}:\d{2}Z) (.*)' as time, content
  | parse-kv content, ' ', '=' 
  | project-away content
  | extend __timde__ = cast(to_unixtime(date_parse(time, '%Y-%m-%dT%H:%i:%sZ')) as bigint)
Enter fullscreen mode Exit fullscreen mode

Result:

__time__: 1718625957
duration: 2256s
event: charge_end
result: success
station: SH_CHG_001
time: 2024-06-17T12:05:57Z
total_kwh: 28.7
vin: LVSHF6196L1234567
Enter fullscreen mode Exit fullscreen mode

Cloud billing settlement logs (JSON Format):

{
  "ts": "2024-06-17T12:06:01Z",
  "vin": "LVSHF6196L1234567",
  "user_id": "87234",
  "station_id": "SH_CHG_001",
  "event": "charge_bill",
  "total_kwh": 28.7,
  "total_fee": 88.23,
  "pay_method": "wechat"
}
Enter fullscreen mode Exit fullscreen mode

Ingest Processor statement:

* | parse-json content
  | project-away content
  | extend __time__ = cast(to_unixtime(date_parse(ts, '%Y-%m-%dT%H:%i:%sZ')) as bigint)
Enter fullscreen mode Exit fullscreen mode

Result:

__time__: 1718625961
event: charge_bill
pay_method: wechat
station_id: SH_CHG_001
total_fee: 88.230000
total_kwh: 28.700000
ts: 2024-06-17T12:06:01Z
user_id: 87234
vin: LVSHF6196L1234567
Enter fullscreen mode Exit fullscreen mode

O&M Analytics
After obtaining the cleaned and standardized data, we can perform O&M analysis on it. For charging scenarios, SQL can be used for such O&M analysis:
● Total daily charging times/total power/peak charging times by time segments
● Main cause of charging failure (gun insertion abnormality/certification abnormality/equipment malfunction)
● One-stop tracing of the full-link charging behaviors (arrival, gun insertion, authentication, start, completion, and payment)
● State of charge (SOC) boost curve and charging speed monitoring
● Automatic alarm for abnormal and continuous low-speed charging or frequent charging interruption
Note: The chart data below is simulated data to illustrate the SQL analysis results.

  1. Total daily charging times/total power/peak charging times by time segments Total daily charging times and total power: 
* | SELECT
  station_id,
  date_trunc('day', __time__) AS day,
  COUNT(*) AS charge_count,
  SUM(total_kwh) AS total_kwh
FROM
  log
WHERE
  event = 'charging_stop'
GROUP BY
  station_id, day
ORDER BY
  charge_count DESC
Enter fullscreen mode Exit fullscreen mode

Peak charging times by time segments (such as hourly peak statistics):

* | SELECT
  date_trunc('hour', __time__)  as hour,
  station_id,
  COUNT(*) AS charge_count
FROM
  log
WHERE
  event = 'charging_start'
GROUP BY
  hour, station_id
ORDER BY
  hour,station_id 
limit all
Enter fullscreen mode Exit fullscreen mode
  1. Main cause of charging failure (gun insertion abnormality/certification abnormality/equipment malfunction) Count the occurrence times of each abnormal type: 
* | SELECT
  reason,           -- Assume there is a failure cause classification field reason after cleaning
  COUNT(*) AS fail_count
FROM
  log
WHERE
  event IN ('charging_start', 'plug_in', 'charging_auth')
  AND result = 'fail'
GROUP BY
  reason
ORDER BY
  fail_count DESC
Enter fullscreen mode Exit fullscreen mode
  1. One-stop tracing of the full-link charging behaviors (arrival, gun insertion, authentication, start, completion, and payment) Use the query mode to trace the entire process logs of a user/vehicle by VIN and chronological order: 
vin: LVSHF6196L5181426
Enter fullscreen mode Exit fullscreen mode
  1. SOC boost curve and charging speed monitoring Extract SOC change with time during a certain charging process 
vin:LVSHF6196L0034 and station_id: SH_CHG_008 | SELECT
  date_trunc('minute', __time__) as dt,
  battery_soc
FROM
  log
ORDER BY
  dt
Enter fullscreen mode Exit fullscreen mode

Calculate the average charging speed (SOC boost rate):

event: charging_status
| SELECT date_trunc('hour', __time__) as dt,
  vin,
  station_id,
  (MAX(battery_soc) - MIN(battery_soc)) AS avg_soc_per_hour
FROM
  log
GROUP BY
  dt,vin, station_id
Enter fullscreen mode Exit fullscreen mode
  1. Automatic alarm for abnormal continuous low-speed charging or frequent charge interruption Detect an abnormally low average speed during the charging process (for example, the SOC increases too slowly): 
event: charging_status
| SELECT date_trunc('hour', __time__) as dt,
  vin,
  station_id,
  (MAX(battery_soc) - MIN(battery_soc)) AS avg_soc_per_hour
FROM
  log
GROUP BY
  dt,vin, station_id
HAVING avg_soc_per_hour < 0.2    -- Trigger alert when below threshold
Enter fullscreen mode Exit fullscreen mode

Detect multiple charging starts and completions within a short time (frequent interruptions):
Select a relative 1-hour time range in the upper-right corner of the query analysis

* | SELECT
  vin,
  station_id,
  COUNT(*) AS interrupt_count
FROM
  log
WHERE
  event IN ('charging_start', 'charging_stop')

GROUP BY
  vin, station_id
HAVING interrupt_count > 3         -- Threshold can be set as needed
Enter fullscreen mode Exit fullscreen mode

Summary
In the NEV industry, log data faces challenges such as difficult cleaning, cross-regional aggregation, cost control, and operational transparency due to its diverse sources, mixed formats, and the need for real-time processing. SLS addresses these issues through full-link access to multi-source logs, intelligent format parsing and dynamic desensitization, cross-regional data processing, million-level real-time processing capacities, cost reduction through Ingest Processor to achieve zero temporary storage, and clear presentation of data flow through lineage visualization tools. Finally, take the charging scenario as an example. After the logs of charging piles, in-vehicle terminals, and mobile phones are uniformly cleaned by SLS, troubleshooting and analysis are supported.
Reference
● SPL syntax
● Ingest Processor
● Comparison between Ingest Processor and Logtail in terms of configuration settings and data processing
● Alibaba Cloud Observable MCP Server

Top comments (0)