Issue
Commerce Cloud offers customers to view and search logs in dedicated instance of AWS OpenSearch. Disk quota there is limited, so after some time logs are gone there, but you can find them still available dumped in your dedicated Azure Blob Storage.
You can find that on your Cloud Portal: Environments > [given environment] : Cloud Storage > logs
Unfortunately these files are hard to use as blob entries are impossible to filter by creation date and also by content.
Solution
In order to filter files I have used java code to connect to Azure Blob Storage and then filter roughly by date.
Download Azure Blob Files
In below example I wanted to filter our files which has been created from 11-Aug-2023, 5 AM UTC to 12-Aug-2023 11:59 PM UTC:
package pl.com.like.zip;
import com.azure.identity.*;
import com.azure.storage.blob.*;
import com.azure.storage.blob.models.*;
import java.io.*;
public class BlobLogFetch {
public static void main(String[] args) throws IOException {
var connectionString =
"AccountName=XXXXXXXXXXX;AccountKey=YYYYYYYYYYYYY;EndpointSuffix=core.windows.net;DefaultEndpointsProtocol=https;";
var container = "commerce-logs-separated";
var containerClient = new BlobContainerClientBuilder().connectionString(connectionString)
.containerName(container).buildClient();
System.out.println(containerClient.exists());
var from = OffsetDateTime.of(2023, 8, 11, 5, 0, 0, 0, ZoneOffset.UTC);
var to = OffsetDateTime.of(2023, 8, 12, 23, 59, 0, 0, ZoneOffset.UTC);
containerClient.listBlobs().stream().filter(i -> i.getProperties().getCreationTime().compareTo(from) > 0)
.filter(item -> item.getProperties().getCreationTime().compareTo(to) < 0).forEach(i -> {
var blob = containerClient.getBlobClient(i.getName());
var path = Paths.get(i.getName()).normalize().toAbsolutePath();
// create dirs
path.toFile().getParentFile().mkdirs();
blob.downloadToFile(path.toString(), Boolean.TRUE);
Above code will list all blob files, filter by file properties if meets date range requirements. If creation date is fine, file is downloaded.
Install OpenSearch
Then, we would like to be able to store file content where data are easily searchable.
There are few options on table, but if we have access to SAP Commerce Cloud, we naturally have used Amazon OpenSearch, so we will follow same tooling locally.
It is possible to find OpenSearch on Docker Hub searching for opensearchproject/opensearch.
I have used linked docker-compose.yml as it contains additionally Dashboard.
version: '3'
services:
opensearch-node1:
image: opensearchproject/opensearch:latest
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
opensearch-node2:
image: opensearchproject/opensearch:latest
container_name: opensearch-node2
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node2
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data
networks:
- opensearch-net
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- "5601"
environment:
OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
networks:
- opensearch-net
volumes:
opensearch-data1:
opensearch-data2:
networks:
opensearch-net:
To run it locally just download file and run:
docker-compose up -d
OpenSearch Configuration
When docker compose is started, you should be able to find OpenSearch Dashboard available on http://localhost:5601/ and logging with credentials admin:admin.
Initially instance will not have any data. We need to create an index first. To do that, click on hamburger menu: OpenSearch Plugins > Index Management, there go to: Index Management > Indices and create index. On my instance I have created index called commerce.
During index creation you can set up data mapping. Below you can find mapping which I have used:
{
"properties": {
"@timestamp": {
"type": "date"
},
"_p": {
"type": "text"
},
"kubernetes": {
"type": "object",
"properties": {
"annotations": {
"type": "object",
"properties": {
"ccv2_cx_sap_com_build-code": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_deployment-id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cni_projectcalico_org_containerID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cni_projectcalico_org_podIP": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cni_projectcalico_org_podIPs": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"data-ingest_dynatrace_com_injected": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"dynakube_dynatrace_com_injected": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"fluentbit_io_parser": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"oneagent_dynatrace_com_injected": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"platform-security-configmap-hash": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"container_hash": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"container_image": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"container_name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"docker_id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"host": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"labels": {
"type": "object",
"properties": {
"app_kubernetes_io_component": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"app_kubernetes_io_managed-by": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"app_kubernetes_io_name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"app_kubernetes_io_part-of": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_build-code": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_deployment-id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_platform-aspect": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_service-name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_service-version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"controller-revision-hash": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"controller-uid": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"job-name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"pod-template-hash": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"statefulset_kubernetes_io_pod-name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"update_ccv2_cx_sap_com_mode": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"namespace_name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"pod_id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"pod_name": {
"type": "text"
}
}
},
"log": {
"type": "text"
},
"logs": {
"type": "object",
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"buildDate": {
"type": "date"
},
"container": {
"type": "object",
"properties": {
"image": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"content": {
"type": "object",
"properties": {
"apiToken": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"apiUrl": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"clusterID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"hasHost": {
"type": "boolean"
},
"hostGroup": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"initialConnectRetry": {
"type": "long"
},
"monitoringNodes": {
"type": "object",
"properties": {
"aks-ey4c7ymiac-68098534-vmss000012": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"aks-ey4c7ymiac-68098534-vmss000013": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"aks-ey4c7ymiac-68098534-vmss000016": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"aks-ey4c7ymiac-68098534-vmss000017": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"aks-ey4c7ymiac-68098534-vmss000019": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"aks-ey4c7ymiac-68098534-vmss00001a": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"networkZone": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"paasToken": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"proxy": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"skipCertCheck": {
"type": "boolean"
},
"tenantUUID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"tlsCert": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"trustedCAs": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"contextMap": {
"type": "object",
"properties": {
"CronJob": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"RemoteAddr": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"Tenant": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"audit": {
"type": "object",
"properties": {
"actionId": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"actionName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"data": {
"type": "object",
"properties": {
"PK": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"changeType": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cronJob": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cronJobResult": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cronJobStatus": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"gyIsActive": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"gyUID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"idVerified": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"idVerifiedAt": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"job": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"newUser": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"prevUser": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"residenceVerified": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"residenceVerifiedAt": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"typePK": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"user": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"sessionId": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"user": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"bytesSent": {
"type": "long"
},
"commitMillis": {
"type": "long"
},
"connectionStatus": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"myid": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"processMillis": {
"type": "long"
},
"remoteHost": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"requestLine": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"sourceClassName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"sourceMethodName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"statusCode": {
"type": "long"
}
}
},
"endOfBatch": {
"type": "boolean"
},
"env": {
"type": "object",
"properties": {
"containers": {
"type": "object",
"properties": {
"image": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"dataIngestInjected": {
"type": "boolean"
},
"failurePolicy": {
"type": "boolean"
},
"installPath": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"installVersion": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"installerFlavor": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"installerTech": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"installerUrl": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8BasePodName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8BasePodUID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8ClusterID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8Namespace": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8NodeName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8PodName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"mode": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"oneAgentInjected": {
"type": "boolean"
},
"workloadKind": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"workloadName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"gitCommit": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"goVersion": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"hostTenant": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"instant": {
"type": "object",
"properties": {
"epochSecond": {
"type": "text"
},
"nanoOfSecond": {
"type": "text"
}
}
},
"level": {
"type": "text"
},
"level_value": {
"type": "long"
},
"logger": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"loggerFqcn": {
"type": "text"
},
"loggerName": {
"type": "text"
},
"logger_name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"marker": {
"type": "object",
"properties": {
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"message": {
"type": "text"
},
"msg": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"origin": {
"type": "text"
},
"platform": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"thread": {
"type": "text"
},
"threadId": {
"type": "integer"
},
"threadPriority": {
"type": "integer"
},
"thread_name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"thrown": {
"type": "object",
"properties": {
"cause": {
"type": "object",
"properties": {
"cause": {
"type": "object",
"properties": {
"cause": {
"type": "object",
"properties": {
"commonElementCount": {
"type": "long"
},
"extendedStackTrace": {
"type": "object",
"properties": {
"class": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"exact": {
"type": "boolean"
},
"file": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"line": {
"type": "long"
},
"location": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"method": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"localizedMessage": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"message": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"commonElementCount": {
"type": "long"
},
"extendedStackTrace": {
"type": "object",
"properties": {
"class": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"exact": {
"type": "boolean"
},
"file": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"line": {
"type": "long"
},
"location": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"method": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"localizedMessage": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"message": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"commonElementCount": {
"type": "long"
},
"extendedStackTrace": {
"type": "object",
"properties": {
"class": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"exact": {
"type": "boolean"
},
"file": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"line": {
"type": "long"
},
"location": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"method": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"localizedMessage": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"message": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"commonElementCount": {
"type": "long"
},
"extendedStackTrace": {
"type": "object",
"properties": {
"class": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"exact": {
"type": "boolean"
},
"file": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"line": {
"type": "long"
},
"location": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"method": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"localizedMessage": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"message": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"timeMillis": {
"type": "long"
},
"ts": {
"type": "date"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"record_date": {
"type": "long"
},
"stream": {
"type": "text"
},
"time": {
"type": "date"
}
}
}
Feeding Search Index
Now is the time to upload data from local files to index. Data in blob items is already nicely prepared and stored in JSON format as 1-liner.
Additionally I have added code, which ignores default unsigned SSL certificate, which by default is used locally:
import groovy.json.JsonOutput
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLSession;
import javax.net.ssl.HttpsURLConnection
import javax.net.ssl.SSLContext
import javax.net.ssl.SSLEngine
import javax.net.ssl.SSLSocketFactory
import javax.net.ssl.TrustManager
import javax.net.ssl.X509ExtendedTrustManager
import java.security.cert.CertificateException
import java.security.cert.X509Certificate
import java.util.Base64;
class UnsafeTrustManager extends X509ExtendedTrustManager {
@Override
void checkClientTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {}
@Override
void checkServerTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {}
@Override
void checkClientTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {}
@Override
void checkServerTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {}
@Override
void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}
@Override
void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}
@Override
X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[0]
}
}
def handleError(httpResponseException) {
println "Request failed with status code: ${httpResponseException.statusCode}"
println "Error message: ${httpResponseException.message}"
}
def dir = new File('./')
// List all files in the directory
def postmanPost = new URL('https://localhost:9200/commerce/_doc')
def v = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
def sslContext = SSLContext.getInstance("TLS")
sslContext.init(null, new TrustManager[]{new UnsafeTrustManager()}, null)
def sslSocketFactory = sslContext.getSocketFactory()
def userCredentials = "admin:admin"
String basicAuth = Base64.getEncoder().encodeToString(userCredentials.getBytes());
HttpsURLConnection.setDefaultHostnameVerifier(v);
dir.eachFile { file ->
if (file.name.endsWith(".log")) {
println "Sending file ${file.name}"
def inputFile = new File(dir, file.name);
// Read file line by line
inputFile.eachLine { line ->
try {
HttpsURLConnection conn = (HttpsURLConnection) postmanPost.openConnection();
HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
});
conn.setSSLSocketFactory(sslSocketFactory);
conn.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
conn.setRequestProperty ("Authorization", "Basic ${basicAuth}");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestMethod("POST");
conn.setHostnameVerifier(v);
def os = conn.getOutputStream();
os.write(line.trim().getBytes("UTF-8"));
os.close();
if (conn.getResponseCode() >= 300) {
println("Failed : HTTP error code : ${conn.getResponseCode()} for ${line}");
}
} catch(Exception e) {
println("Error: ${e}, for line ${line}");
}
}
}
}
Run script by calling:
groovy [script-name].groovy
After that, you will be able to see data in you Dashboard.
Top comments (0)