DEV Community: Pawel Wolanski

SAP Commerce Cloud Framework Update JDK 21 - issue with ANT gradle target

Pawel Wolanski — Thu, 19 Mar 2026 12:50:11 +0000

Recently I have been involved in Framework Update engagement to bump Commerce Cloud solution from JDK 17 to JDK 21.

Upgrade to Java 21 It has been announced some time ago by SAP as a Framework Update
which bumps Java, Spring and long list of dependencies to latest Java LTS.
More information about service you can find on:

One of nice things which has been added by SAP is upgrade automation based on OpenRewrite. More information about whole activity you can find as SAP KBA 3618495 - Using OpenRewrite recipes to support SAP Commerce Cloud JDK21 Framework Update adoption

Issue

At the beginning when you run ant command ant gradle you may get similar error like:

/opt/hybris/hybris/bin/platform/build.xml:204: Cannot invoke "String.startsWith(String)" because the return value of "org.apache.maven.model.Dependency.getVersion()" is null

That error is related to scanning process for Maven dependencies in external-dependencies.xml.

Root cause

In my case one of external-dependencies.xml for custom extension got dependency defined:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>de.hybris.platform</groupId>
    <artifactId>knackbravaocc</artifactId>
    <version>2211.1</version>
    <packaging>jar</packaging>

    <dependencies>
...
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
...
    </dependencies>
</project>

In order to fix it, I needed to add version explicitly for that dependency.

Solution

Check if your all dependencies have version xml tag defined.

BTP CAP App + HANA deployment

Pawel Wolanski — Tue, 18 Feb 2025 00:53:30 +0000

Recently I had issue with deploying to my just created SAP HANA DB sample data from my SAP CAP application.

I got an error:

@sap/hdi-deploy, version 5.3.2 (mode default), server version unknown (0.0.0.0), cloud version , node version 23.7.0, HDI version -1, container API version -1

Deployment started at 2025-02-18 01:27:52
Using hdb@undefined for connection
Detection of server version failed; root cause: Could not connect to any host: [ <id>.hana.prod-us10.hanacloud.ondemand.com:443 - Client network socket disconnected before secure TLS connection was established ]
Detection of cloud version failed; root cause: Could not connect to any host: [ <id>.hana.prod-us10.hanacloud.ondemand.com:443 - Client network socket disconnected before secure TLS connection was established ]
Detection of container API version failed; root cause: Could not connect to any host: [ <id>.hana.prod-us10.hanacloud.ondemand.com:443 - Client network socket disconnected before secure TLS connection was established ]
Detection of HDI version failed; root cause: Could not connect to any host: [ <id>.hana.prod-us10.hanacloud.ondemand.com:443 - Client network socket disconnected before secure TLS connection was established ]
The server does not support live updating of make messages. The messages will be displayed, when the make is done.


Collecting files...

Collecting files... ok (0s 2ms)
3 directories collected
10 files collected
0 reusable modules collected
Target service: preferences-db

Could not check if session variable APPLICATION is set correctly: Could not connect to any host: [ <id>.hana.prod-us10.hanacloud.ondemand.com:443 - Client network socket disconnected before secure TLS connection was established ]

Could not determine status of last build: Could not connect to any host: [ <id>.hana.prod-us10.hanacloud.ondemand.com:443 - Client network socket disconnected before secure TLS connection was established ]

I have been googling for a while, but couldn't find the root cause.

I finally found a solution in SAP BTP cockpit. I have clicked on HANA instance on Sub-Account page, then in Instance Details page I have clicked on Manage Configuration and found Connections section. I have selected option:

Allow all IP addresses

then, clicked on Review and Save

Voilà!

Creating SAP BTP Solution Diagrams Right Way

Pawel Wolanski — Thu, 23 Jan 2025 21:16:36 +0000

Recently we have built simple solution on SAP BTP platform. Now is the time to generate a bit of noise around that. We wanted to have content and diagrams built the same way as ones presented at SAP Discovery Center, to have consistent look and feel.

As you can see, UI is nice, clean and... I just wanted to make it the same way.

After some searching, I found out that SAP did a decent job documenting the whole process of creating it and... shared draw.io collection of elements ready to use.

References

SAP Fiori Generator gimmicks

Pawel Wolanski — Tue, 02 Jul 2024 12:09:03 +0000

Working with SAP CAP framework sometimes leaves bitter-sweet taste.

Cannot Generate SAP Fiori Application in VS Code

SAP Fiori CAP applications do not have clear or documented application configuration file, which is used for cds or SAP's Yeoman (dead?) Fiori app generator @sap/generator-fiori.
Recently, I have spent significant time to figure out why I am unable to add another application using that generator. Apparently there is an if condition based on package.json dependency check. When I added @sap/cds to package.json, it started to detect it as valid Node.js project.

...
    "@sap/cds": ">=7"
...

From the other side, cds CLI does not check it at all, BUT... it checks if in project root folder is able to find pom.xml it detects it as a Java CAP project. Otherwise, it treats that as Node.js.

The critical issue with figuring it out is:

application behavior it is not documented,
both of these utility apps are NOT open-source
@sap/generator-fiori it transpiled from TypeScript and not possible to review,
- (cds CLI fortunately is pure JS tho)

Note

I get it. If I generated new project from the scratch using cds and copy-paste all CAP model/service configuration, I would avoid that situation. I did at the end, but we are not living in perfect greenfield world and even when CAP is pretty new, you still have to have someone to take care of brownfield.

Frustrating Docker Desktop on Mac Containerd issue

Pawel Wolanski — Sat, 13 Apr 2024 11:23:56 +0000

Recently, I have been having issue with my Docker Desktop on Mac. With latest docker version, I have been having weird issues with docker buildx build with or without --platform command parameter.


<-- Building application... with file xxx.jar
[+] Building 4.4s (9/9) FINISHED                                                                                docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                            0.0s
 => => transferring dockerfile: 250B                                                                                            0.0s
 => [internal] load metadata for docker.io/library/sapmachine:17-jre-ubuntu                                                     1.2s
 => [auth] library/sapmachine:pull token for registry-1.docker.io                                                               0.0s
 => [internal] load .dockerignore                                                                                               0.0s
 => => transferring context: 2B                                                                                                 0.0s
 => [1/3] FROM docker.io/library/sapmachine:17-jre-ubuntu@sha256:5e62bb11c7b261b456ed92333244971b1147606f1de19308a475f40fb87d3  0.0s
 => => resolve docker.io/library/sapmachine:17-jre-ubuntu@sha256:5e62bb11c7b261b456ed92333244971b1147606f1de19308a475f40fb87d3  0.0s
 => [internal] load build context                                                                                               0.5s
 => => transferring context: 82.01MB                                                                                            0.5s
 => CACHED [2/3] WORKDIR /opt/app                                                                                               0.0s
 => [3/3] COPY ../../../target/product-syndication-services-1.0.0-SNAPSHOT.jar ./app.jar                                        0.4s
 => exporting to image                                                                                                          2.2s
 => => exporting layers                                                                                                         2.1s
 => => exporting manifest sha256:7651ad6af65e3b1766371fd77aa2e2d735916fc6b1ad1c09d6073b309d35860d                               0.0s
 => => exporting config sha256:48c636cab66cd0d3e0af535d3df2597bd49df7ecb443e3836ad6e2d6809af57a                                 0.0s
 => => exporting attestation manifest sha256:15913c47d0839153b098d0a03a021ce540f850b4ad7ff0cfb01d84eb161db7ee                   0.0s
 => => exporting manifest list sha256:5c93ea385bb027dfa952924698fb4505898d5f95c7d1911bb2d6a8f4618c7bd0                          0.0s
 => => naming to docker.io/library/xxx-services:latest                                                          0.0s
 => => naming to docker.io/library/xxx-services:20240412183028                                                  0.0s
 => => naming to xxxx.common.repositories.cloud.sap/xxx-services:20240412183028                          0.0s
 => => naming to xxxx.common.repositories.cloud.sap/xxx-services:latest                                  0.0s

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/i8ehocxx8km11ulpgs8xuggtr
<-- Building application...
The push refers to repository [xxxx.common.repositories.cloud.sap/xxx-services]
236773733796: Layer already exists
92a775d24bc3: Pushed
06b5b4631fbb: Pushed
23828d760c7b: Layer already exists
33136b601654: Layer already exists
missing content: content digest sha256:7651ad6af65e3b1766371fd77aa2e2d735916fc6b1ad1c09d6073b309d35860d: not found
Note: You're trying to push a manifest list/index which references multiple platform specific manifests, but not all of them are available locally or available to the remote repository.
Make sure you have all the referenced content and try again.

I have been looking for the solution and found few suggestions like setting command parameter --provenance=false, it worked for couple times, but then stopped working. Another one was checking docker buildx imagetools inspect, but image was never pushed and found, so I failed with that.

Finally, I was able to find raised GitHub issue #4959 related to containerd.

I was genuinely surprised, as it have been working so long and with one of recent releases stopped.

I disabled one option in Docker settings and finally started to work again.

Here is my docker version:

⋊> ~/b/b/btp on feature/XXX-XXX ⨯ docker version                                             18:44:39
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.21.1
 Git commit:        ed223bc820
 Built:             Thu Aug 31 17:24:32 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.28.0 (139021)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:22 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

BTP Kyma ORY Hydra OAuth2 Client migration… ver. extended, revised

Pawel Wolanski — Mon, 06 Nov 2023 10:56:42 +0000

If you are an active SAP BTP Kyma user, under certain conditions you may see a notification in your dashboard.

Note says:

You are using Ory Hydra Auth2, which has been deprecated since May 19, 2022 and is scheduled to be removed on October 14, 2023. During the Kyma upgrade on October 14, 2023, workloads that still rely on deprecated Ory Hydra Auth2 may experience minor outages. To prevent this, follow the procedure outlined in the blog post below before the specified date. If you are unable to follow the procedure, Ory Hydra Auth2 will be migrated to the hydra-deprecated Namespace to prevent outages caused by API call failures.
However, you will then be fully responsible for managing ORY Hydra Auth2.
This includes troubleshooting, maintenance, security updates, vulnerability fixes, and adherence to product standards. If you want to continue using the Auth2 functionality provided as part of the SAP BTP, Kyma runtime offering, delete the hydra-deprecated Namespace or follow the migration steps outlined in the blog post below.

There is a link to SAP Blog, (nope, not a part of official SAP documentation)
SAP BTP, Kyma runtime ORY Hydra OAuth2 Client migration, which partially covers topic.

NOTE
There are no blog posts, documentation, or SAP Support KBA how to install or "troubleshooting, maintenance, security updates, vulnerability fixes, and adherence to product standards." for your custom ORY installation.

I am not going to cover that too, as there is too much of an effort in doing that. Sorry.
What I will add here are just steps to make your migration less painful.

Partially, as that one covers only setting up authentication rule in APIRule.

The blog post discusses adding a new rule to secured endpoint by APIRule, which checks that standard OAuth2 Kyma client (that part is that mysterious ORY thingy from the API).

There is a lot of gray area here, to be frank.

Firstly, placeholders used in scripts, where do I get this information to fill values?

OAuth2 Credentials

The first code snippet describes a sample OAuth2 client, which should be replaced.

cat <<EOF | kubectl apply -f -
apiVersion: hydra.ory.sh/v1alpha1              
kind: OAuth2Client       
metadata:
  name: test_app                   
  namespace: $NAMESPACE
spec:
  grantTypes:      
    - "client_credentials"
  scope: "test"                 
  secretName: test_app            
EOF

$NAMESPACE – that is the easiest one, you can get it from top right corner in dashboard. That should be the namespace where your OAuth2 configuration is located.

In my case that is services.

$ export NAMESPACE=services

These namespaces are part of Kubernetes environment.

API Rule

Second and third code snippet contains another variables:

curl -ik -X POST "https://oauth2.$KYMA_DOMAIN/oauth2/token" -H "Authorization: Basic $ENCODED_CREDENTIALS" -F "grant_type=client_credentials" -F "scope=test"

$KYMA_DOMAIN – is the URL, which you can take directly from dashboard. Just open main page for your Kyma cluster.

You can find a URL to your API service. $KYMA_DOMAIN is that one without api. subdomain.

$ export KYMA_DOMAIN=c-9XXXXXX9.kyma.ondemand.com

The second placeholder is $ENCODED_CREDENTIALS. It is strictly related to secrets to given OAuth2 client.
When you open: Configuration > OAuth2 Clients on the left menu and view one of entry, you can find on the bottom client_id and client_secret. Firstly Decode them, and then copy for later. These data have to be Base64 encoded

export CLIENT_ID="<client_id>"
export CLIENT_SECRET="<client_secret>"
export ENCODED_CREDENTIALS=$(echo -n "$CLIENT_ID:$CLIENT_SECRET" | base64)

Adding SAP Cloud Identity Services

It is explicitly mentioned to use SAP Cloud Identity Services, so be it.

Let's go to your BTP cockpit and search in Services > Service Marketplace that particular service:

I have opened it and clicked Create and followed the 2-step wizard to complete the process.

Error

In my case, instance creation failed with a note to SAP Help: Establish Trust and Federation Between SAP Authorization and Trust Management Service and Identity Authentication

Tip
There is an official SAP BTP Learning video about that step available on YT: Cloud Identity Services Identity Authentication | SAP Business Technology Platform.

I have followed steps from help page and added as Sub-Account Administrator new entry to Trust Configuration.

NOTE
Read through restrictions on help page, as that operation may not be possible in some cases.

Gray Area

The part related to IAS provisioning and making it available as a tenant in Trust area on BTP is beyond my knowledge, as that operation is done... via IT ticket. I wasn't able to do that so had to ask my colleagues to request it.

Configuration

Tip
Pieter Janssens wrote nice blog about technical users configuration. I recommend to go through that later on SAP Blog: Managing technical users for BTP platform access

When you log in to IAS, by default you are redirected to profile page: https://<instance-id>.accounts<optional-id>.ondemand.com/ui/protected/profilemanagement

In order to access admin cockpit, you need to visit sub-page:
https://<instance-id>.accounts<optional-id>.ondemand.com/admin

You need to create new application:

After clicking on create, I have configured it as a OpenID Connect:

SAP Cloud Identity Services Configuratoin

In order to get IAS environment variables, we need to open auto-discovery link: https://<instance-id>.accounts<optional-id>.ondemand.com/.well-known/openid-configuration.

In my case that is: https://sandbox.accounts.ondemand.com/.well-known/openid-configuration

export IAS_ENC_CREDS={APPLICATION_ENCODED_CREDENTIALS}
export IAS_TOKEN_URL={TOKEN_URL}
export IAS_INTRO_URL={INTROSPECTION_URL}
export IAS_INTRO_ENC_CREDS={INTROSPECTION_ENCODED_CEDENTIALS}

Credentials Migration

The next step is to move login data to the new service.

Fetching data from Kyma

In order to get data from Kyma we need to extract data stored in Kubernetes Secret resources.

First, it is required to set up your KUBECONFIG value pointing to kubeconfig.yaml for your environment. File is available from your BTP and Kyma cockpits.

export KUBECONFIG=<path-to-kubeconfig.yaml>

Then we can filter out our resources:

kubectl get secrets

Extracting the right data is up to you. Depending on your data, you can filter out by namespace and name of the secret by using --field-selector. In my case, filtering out was done after fetching data from Kubernetes.

Data are encoded with base64. In order to import user data to IAS, you need to decode them first.

Importing User Profile to IAS/CIS

In IAS admin cockpit you can find possibility to upload CSV data. Then read about Full User Profile and this: Import or Update Users for a Specific Application

https://me.sap.com/notes/0002607696

NOTE
CSV file format is tricky here. Documentation says that:

there are 3 mandatory columns: userName (or loginName), emails[0].value (or mail), name.familyName (or lastName),

columns are separated by commas,

multi-values for one column are separated by semicolons,

column names are case-sensitive,

mapping commas to columns is strict, meaning there should not be any orphan commas, otherwise will end up with error

there is no clear information what is wrong with file-content

import will fail if file contains additional spaces

I never thought that I will get so many import errors with simple CSV file… At the end, I was not able to find what was the import error and created users manually.

CSV Import process does not allow you to set up same password as it used to be. It is possible to set it manually.

SAP BTP Kyma startup

Pawel Wolanski — Mon, 06 Nov 2023 10:55:09 +0000

I have created that blog post to provide some guidelines for beginners from a beginner, how to start from right foot there.

Set up your local environment

First steps, first. Install essentials required for building application and deploying to Kubernetes.

Prerequisites @ github.com/SAP-samples/kyma-runtime-extension-samples.

SAP Custom utility binaries

Cloud MTA Build Tool - MBT

SAP build tool, which based on configuration in your project mta.yaml file will build multiple parts of your project. That includes Java, Node.js and SAP BTP specific implementations e.g. approuter. More details on MBT page.

Yeah. Spent 1 day setting it up, and it was worthless, as it builds MTAR possible to deploy on Cloud Foundry only.
Why I did that? Because I got information from one BTP Kyma lead dev, that is supported (2023). Well… not yet, and it doesn't look like it will work with Kyma at all. Just looking at their github.com/cloud-mta-build-tool since 2022 tool development has stopped, and they're just bumping versions of dependencies.

A pity, as it could be really nice glue for all microservices which you will end up having in your project.

🔴 Skip that tool.

SAP BTP CLI

BTP CLI which that may help setting up environment and make that step automatic if you do that quite frequently. Otherwise it is just little help for a dev and can be described and done manually.

You can find out more about tool on Get Started with the SAP BTP CLI tutorial.

If you are going to build an application fully on Kyma, you may use that tool just for spinning up Kyma runtime. Steps are listed on SAP Blog: Creating SAP BTP, Kyma runtime via the SAP BTP CLI.

🟡 You may skip that tool.

SAP CDS CLI

SAP CLI tool to initialize, add new features to your dev project. If you follow CAP way fully, it will simplify working with your project a lot, otherwise it just adds a little help, as your Node.js project will be initialized other way, e.g. by React framework, or other. Then you will build it as a docker container and push to your Kyma runtime.

🟢 You can install it for CAP projects.

Set Up BTP Environment

In order to do that and understand what are the steps, it is worth going through 2021 TechEd session available on github and link to recording.

Do you want to use SAP BTP Services?

Well, I would say yes, if you are not to build just a Hello World application. SAP Discovery Center contains a lot of services, which you can consume on your BTP environment.

Problem: How to connect to a SAP BTP service from Kyma environment
Solution: SAP BTP Manager
SAP Blog Post: Kyma's Transition to Modular Architecture

SAP BTP Kyma Metrics and Log export

Pawel Wolanski — Thu, 02 Nov 2023 15:25:41 +0000

Need

Everyone of you would love to see logs from your application deployed on Kubernetes nice way. We all used to luxury having Kibana, Splunk, you name it. Unfortunately with SAP Kyma it is not available anymore. You need set it up by yourself to see them collected in one place.

In the past you could have loki exposed and then look at it using your Grafana instance locally. Unfortunately that part also gone.

https://blogs.sap.com/2022/11/02/deprecation-of-loki-based-application-logs-in-sap-btp-kyma-runtime/

https://blogs.sap.com/2023/06/02/removal-of-loki-based-application-logs-in-sap-btp-kyma-runtime/

You can still navigate to Pod's logs on Kyma Dashboard or use kubectl from command line, but... that is not what we expect use in the long run.

Kyma offers Log Pipeline which I have tried to use, but for some reason I got an error with Grafana... If you would like to go through their docs, you can find that on Kyma: Application Logs page.

Solution

I have used free Grafana account to set it up their way. It was blazing fast. Start from user account page:

Alternatively you can find on the bottom of that page a shortcut:

find on home page "+ Connect Data" (https://<user-account>.grafana.net/connections/add-new-connection).
select "Kubernetes monitoring" and create token
follow their helm installation

Code snippet, which has been generated for my account looked like this:

helm repo add grafana https://grafana.github.io/helm-charts &&
helm repo update &&
helm upgrade --install grafana-k8s-monitoring grafana/k8s-monitoring --namespace "default" --create-namespace --values - <<EOF
cluster:
  name: services
externalServices:
  prometheus:
    host: https://prometheus-prod-13-prod-us-east-0.grafana.net
    basicAuth:
      username: "1244838"
      password: PASTE_AUTH_TOKEN
  loki:
    host: https://logs-prod-006.grafana.net
    basicAuth:
      username: "721487"
      password: PASTE_AUTH_TOKEN
opencost:
  opencost:
    exporter:
      defaultClusterId: services
    prometheus:
      external:
        url: https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom
EOF

When I run it on Kyma I got one error due to existing pods installed by default. Helm printed out solution for it in the console. I needed to add at the end one configuration entry:

prometheus-operator-crds:
  enabled: false

After that I got a notification in console:

Release "grafana-k8s-monitoring" does not exist. Installing it now.
NAME: grafana-k8s-monitoring
LAST DEPLOYED: Thu Nov  2 15:23:43 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1

and finally I could browse logs.

SAP BTP on Kyma Runtime: PAIN

Pawel Wolanski — Tue, 19 Sep 2023 11:03:44 +0000

Recently I have been playing around with SAP BTP on Kyma. I have to confess, that is provided with a hell of chaotic documentation.
One reason is that is caused by being hosted on Kyma. I have nothing to Kyma itself as it is runtime based on rock solid Kubernetes, so what could go wrong… In fact, most fuss is caused by technical debt and lack of documentation update to include Kyma.
Initially, SAP BTP was designed to be run on Cloud Foundry, but due to well known CF downsides. If you're looking for more details about that, I do recommend reading two articles:

So, what is the issue... Let me list you just a few.

Kyma Deprecations

Kyma's technical stack changes a lot:

Minikube deprecation on local environment (2023 - there is gardener and k3d only)
Jaeger deprecation
Loki deprecation
Grafana and Prometheus removal
removal of pre-defined roles

There is more of that on What's New 2021 and What's New 2023

Looking at diagram from December 2021 that is updated version for 2023:

I do not want to judge all these changes (although missing out Grafana and all logging date is huge bummer, especially, that with e.g. AWS K8s you get basic monitoring and logging for free, just for development purposes)

Second thing is… that documentation doesn't provide sample solution which just works. I have tried one how to migrate from ORY Hydra and failed. I also asked a skilled colleague in BTP and also failed.

That way of documentation delivery isn't working.

Commerce Cloud logs in OpenSearch locally

Pawel Wolanski — Tue, 05 Sep 2023 23:49:03 +0000

Issue

Commerce Cloud offers customers to view and search logs in dedicated instance of AWS OpenSearch. Disk quota there is limited, so after some time logs are gone there, but you can find them still available dumped in your dedicated Azure Blob Storage.

You can find that on your Cloud Portal: Environments > [given environment] : Cloud Storage > logs

Unfortunately these files are hard to use as blob entries are impossible to filter by creation date and also by content.

Solution

In order to filter files I have used java code to connect to Azure Blob Storage and then filter roughly by date.

Download Azure Blob Files

In below example I wanted to filter our files which has been created from 11-Aug-2023, 5 AM UTC to 12-Aug-2023 11:59 PM UTC:


package pl.com.like.zip;

import com.azure.identity.*;
import com.azure.storage.blob.*;
import com.azure.storage.blob.models.*;
import java.io.*;


public class BlobLogFetch {

    public static void main(String[] args) throws IOException {

        var connectionString =
                "AccountName=XXXXXXXXXXX;AccountKey=YYYYYYYYYYYYY;EndpointSuffix=core.windows.net;DefaultEndpointsProtocol=https;";
        var container = "commerce-logs-separated";


        var containerClient = new BlobContainerClientBuilder().connectionString(connectionString)
                .containerName(container).buildClient();
        System.out.println(containerClient.exists());
        var from = OffsetDateTime.of(2023, 8, 11, 5, 0, 0, 0, ZoneOffset.UTC);
        var to = OffsetDateTime.of(2023, 8, 12, 23, 59, 0, 0, ZoneOffset.UTC);

        containerClient.listBlobs().stream().filter(i -> i.getProperties().getCreationTime().compareTo(from) > 0)
                .filter(item -> item.getProperties().getCreationTime().compareTo(to) < 0).forEach(i -> {
                    var blob = containerClient.getBlobClient(i.getName());
                    var path = Paths.get(i.getName()).normalize().toAbsolutePath();
// create dirs
                    path.toFile().getParentFile().mkdirs();
                    blob.downloadToFile(path.toString(), Boolean.TRUE);

Above code will list all blob files, filter by file properties if meets date range requirements. If creation date is fine, file is downloaded.

Install OpenSearch

Then, we would like to be able to store file content where data are easily searchable.

There are few options on table, but if we have access to SAP Commerce Cloud, we naturally have used Amazon OpenSearch, so we will follow same tooling locally.

It is possible to find OpenSearch on Docker Hub searching for opensearchproject/opensearch.

I have used linked docker-compose.yml as it contains additionally Dashboard.

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

To run it locally just download file and run:

docker-compose up -d

OpenSearch Configuration

When docker compose is started, you should be able to find OpenSearch Dashboard available on http://localhost:5601/ and logging with credentials admin:admin.

Initially instance will not have any data. We need to create an index first. To do that, click on hamburger menu: OpenSearch Plugins > Index Management, there go to: Index Management > Indices and create index. On my instance I have created index called commerce.

During index creation you can set up data mapping. Below you can find mapping which I have used:

{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "_p": {
      "type": "text"
    },
    "kubernetes": {
      "type": "object",
      "properties": {
        "annotations": {
          "type": "object",
          "properties": {
            "ccv2_cx_sap_com_build-code": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_deployment-id": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "cni_projectcalico_org_containerID": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "cni_projectcalico_org_podIP": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "cni_projectcalico_org_podIPs": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "data-ingest_dynatrace_com_injected": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "dynakube_dynatrace_com_injected": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "fluentbit_io_parser": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "oneagent_dynatrace_com_injected": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "platform-security-configmap-hash": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "container_hash": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "container_image": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "container_name": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "docker_id": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "host": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "labels": {
          "type": "object",
          "properties": {
            "app_kubernetes_io_component": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "app_kubernetes_io_managed-by": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "app_kubernetes_io_name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "app_kubernetes_io_part-of": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_build-code": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_deployment-id": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_platform-aspect": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_service-name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "ccv2_cx_sap_com_service-version": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "controller-revision-hash": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "controller-uid": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "job-name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "pod-template-hash": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "statefulset_kubernetes_io_pod-name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "update_ccv2_cx_sap_com_mode": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "namespace_name": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "pod_id": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "pod_name": {
          "type": "text"
        }
      }
    },
    "log": {
      "type": "text"
    },
    "logs": {
      "type": "object",
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "buildDate": {
          "type": "date"
        },
        "container": {
          "type": "object",
          "properties": {
            "image": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "content": {
          "type": "object",
          "properties": {
            "apiToken": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "apiUrl": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "clusterID": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "hasHost": {
              "type": "boolean"
            },
            "hostGroup": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "initialConnectRetry": {
              "type": "long"
            },
            "monitoringNodes": {
              "type": "object",
              "properties": {
                "aks-ey4c7ymiac-68098534-vmss000012": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "aks-ey4c7ymiac-68098534-vmss000013": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "aks-ey4c7ymiac-68098534-vmss000016": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "aks-ey4c7ymiac-68098534-vmss000017": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "aks-ey4c7ymiac-68098534-vmss000019": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "aks-ey4c7ymiac-68098534-vmss00001a": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                }
              }
            },
            "networkZone": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "paasToken": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "proxy": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "skipCertCheck": {
              "type": "boolean"
            },
            "tenantUUID": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "tlsCert": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "trustedCAs": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "contextMap": {
          "type": "object",
          "properties": {
            "CronJob": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "RemoteAddr": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "Tenant": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "audit": {
              "type": "object",
              "properties": {
                "actionId": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "actionName": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "data": {
                  "type": "object",
                  "properties": {
                    "PK": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "changeType": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "cronJob": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "cronJobResult": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "cronJobStatus": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "gyIsActive": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "gyUID": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "idVerified": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "idVerifiedAt": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "job": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "newUser": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "prevUser": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "residenceVerified": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "residenceVerifiedAt": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "typePK": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "user": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    }
                  }
                },
                "sessionId": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "user": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                }
              }
            },
            "bytesSent": {
              "type": "long"
            },
            "commitMillis": {
              "type": "long"
            },
            "connectionStatus": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "myid": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "processMillis": {
              "type": "long"
            },
            "remoteHost": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "requestLine": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "sourceClassName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "sourceMethodName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "statusCode": {
              "type": "long"
            }
          }
        },
        "endOfBatch": {
          "type": "boolean"
        },
        "env": {
          "type": "object",
          "properties": {
            "containers": {
              "type": "object",
              "properties": {
                "image": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "name": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                }
              }
            },
            "dataIngestInjected": {
              "type": "boolean"
            },
            "failurePolicy": {
              "type": "boolean"
            },
            "installPath": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "installVersion": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "installerFlavor": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "installerTech": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "installerUrl": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8BasePodName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8BasePodUID": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8ClusterID": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8Namespace": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8NodeName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "k8PodName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "mode": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "oneAgentInjected": {
              "type": "boolean"
            },
            "workloadKind": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "workloadName": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "gitCommit": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "goVersion": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "hostTenant": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "instant": {
          "type": "object",
          "properties": {
            "epochSecond": {
              "type": "text"
            },
            "nanoOfSecond": {
              "type": "text"
            }
          }
        },
        "level": {
          "type": "text"
        },
        "level_value": {
          "type": "long"
        },
        "logger": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "loggerFqcn": {
          "type": "text"
        },
        "loggerName": {
          "type": "text"
        },
        "logger_name": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "marker": {
          "type": "object",
          "properties": {
            "name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "message": {
          "type": "text"
        },
        "msg": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "origin": {
          "type": "text"
        },
        "platform": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "thread": {
          "type": "text"
        },
        "threadId": {
          "type": "integer"
        },
        "threadPriority": {
          "type": "integer"
        },
        "thread_name": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        },
        "thrown": {
          "type": "object",
          "properties": {
            "cause": {
              "type": "object",
              "properties": {
                "cause": {
                  "type": "object",
                  "properties": {
                    "cause": {
                      "type": "object",
                      "properties": {
                        "commonElementCount": {
                          "type": "long"
                        },
                        "extendedStackTrace": {
                          "type": "object",
                          "properties": {
                            "class": {
                              "fields": {
                                "keyword": {
                                  "type": "keyword",
                                  "ignore_above": 256
                                }
                              },
                              "type": "text"
                            },
                            "exact": {
                              "type": "boolean"
                            },
                            "file": {
                              "fields": {
                                "keyword": {
                                  "type": "keyword",
                                  "ignore_above": 256
                                }
                              },
                              "type": "text"
                            },
                            "line": {
                              "type": "long"
                            },
                            "location": {
                              "fields": {
                                "keyword": {
                                  "type": "keyword",
                                  "ignore_above": 256
                                }
                              },
                              "type": "text"
                            },
                            "method": {
                              "fields": {
                                "keyword": {
                                  "type": "keyword",
                                  "ignore_above": 256
                                }
                              },
                              "type": "text"
                            },
                            "version": {
                              "fields": {
                                "keyword": {
                                  "type": "keyword",
                                  "ignore_above": 256
                                }
                              },
                              "type": "text"
                            }
                          }
                        },
                        "localizedMessage": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "message": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "name": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        }
                      }
                    },
                    "commonElementCount": {
                      "type": "long"
                    },
                    "extendedStackTrace": {
                      "type": "object",
                      "properties": {
                        "class": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "exact": {
                          "type": "boolean"
                        },
                        "file": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "line": {
                          "type": "long"
                        },
                        "location": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "method": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        },
                        "version": {
                          "fields": {
                            "keyword": {
                              "type": "keyword",
                              "ignore_above": 256
                            }
                          },
                          "type": "text"
                        }
                      }
                    },
                    "localizedMessage": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "message": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "name": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    }
                  }
                },
                "commonElementCount": {
                  "type": "long"
                },
                "extendedStackTrace": {
                  "type": "object",
                  "properties": {
                    "class": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "exact": {
                      "type": "boolean"
                    },
                    "file": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "line": {
                      "type": "long"
                    },
                    "location": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "method": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    },
                    "version": {
                      "fields": {
                        "keyword": {
                          "type": "keyword",
                          "ignore_above": 256
                        }
                      },
                      "type": "text"
                    }
                  }
                },
                "localizedMessage": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "message": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "name": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                }
              }
            },
            "commonElementCount": {
              "type": "long"
            },
            "extendedStackTrace": {
              "type": "object",
              "properties": {
                "class": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "exact": {
                  "type": "boolean"
                },
                "file": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "line": {
                  "type": "long"
                },
                "location": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "method": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                },
                "version": {
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  },
                  "type": "text"
                }
              }
            },
            "localizedMessage": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "message": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            },
            "name": {
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              },
              "type": "text"
            }
          }
        },
        "timeMillis": {
          "type": "long"
        },
        "ts": {
          "type": "date"
        },
        "version": {
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "type": "text"
        }
      }
    },
    "record_date": {
      "type": "long"
    },
    "stream": {
      "type": "text"
    },
    "time": {
      "type": "date"
    }
  }
}

Feeding Search Index

Now is the time to upload data from local files to index. Data in blob items is already nicely prepared and stored in JSON format as 1-liner.

Additionally I have added code, which ignores default unsigned SSL certificate, which by default is used locally:


import groovy.json.JsonOutput
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLSession;
import javax.net.ssl.HttpsURLConnection
import javax.net.ssl.SSLContext
import javax.net.ssl.SSLEngine
import javax.net.ssl.SSLSocketFactory
import javax.net.ssl.TrustManager
import javax.net.ssl.X509ExtendedTrustManager
import java.security.cert.CertificateException
import java.security.cert.X509Certificate
import java.util.Base64;

class UnsafeTrustManager extends X509ExtendedTrustManager {

    @Override
    void checkClientTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {}

    @Override
    void checkServerTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {}

    @Override
    void checkClientTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {}

    @Override
    void checkServerTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {}

    @Override
    void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}

    @Override
    void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}

    @Override
    X509Certificate[] getAcceptedIssuers() {
        return new X509Certificate[0]
    }

}


def handleError(httpResponseException) {
    println "Request failed with status code: ${httpResponseException.statusCode}"
    println "Error message: ${httpResponseException.message}"
}

def dir = new File('./')

// List all files in the directory
def postmanPost = new URL('https://localhost:9200/commerce/_doc')

def v = new HostnameVerifier() {
    public boolean verify(String hostname, SSLSession session) {
        return true;
    }
};

def sslContext = SSLContext.getInstance("TLS")
sslContext.init(null, new TrustManager[]{new UnsafeTrustManager()}, null)
def sslSocketFactory = sslContext.getSocketFactory()
def userCredentials = "admin:admin"
String basicAuth = Base64.getEncoder().encodeToString(userCredentials.getBytes());

HttpsURLConnection.setDefaultHostnameVerifier(v);

dir.eachFile { file ->

    if (file.name.endsWith(".log")) {
        println "Sending file ${file.name}"
        def inputFile = new File(dir, file.name);
        // Read file line by line
        inputFile.eachLine { line ->
            try {
                HttpsURLConnection conn = (HttpsURLConnection) postmanPost.openConnection();
                HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
                    public boolean verify(String hostname, SSLSession session) {
                        return true;
                    }
                });
                conn.setSSLSocketFactory(sslSocketFactory);
                conn.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
                conn.setRequestProperty ("Authorization", "Basic ${basicAuth}");
                conn.setDoOutput(true);
                conn.setDoInput(true);
                conn.setRequestMethod("POST");
                conn.setHostnameVerifier(v);
                def os = conn.getOutputStream();
                os.write(line.trim().getBytes("UTF-8"));
                os.close();

                if (conn.getResponseCode() >= 300) {
                    println("Failed : HTTP error code : ${conn.getResponseCode()} for ${line}");
                }
            } catch(Exception e) {
                println("Error: ${e}, for line ${line}");
            }
        }
    }
}

Run script by calling:

groovy [script-name].groovy

After that, you will be able to see data in you Dashboard.

Panasonic Aquarea Smart Cloud without WPS

Pawel Wolanski — Mon, 10 Jul 2023 14:12:38 +0000

Panasonic Aquarea Heat-Pumps

Panasonic produces one of the best heat pumps on the market. On top of built-in configuration panel, it is possible to upgrade it to smart device and control via Internet with additional Panasonic's adapter: Panasonic CZ-TAW1

Smart Cloud Adapter CZ-TAW1

Panasonic heat-pump can be controlled remotely via their own Comfort Cloud. That gives you the same possibilities as from built-in control panel.

Addition is easy to configure and can be connected via Ethernet wire or via Wi-Fi.

One downside is wireless connection can be established via WPS, which isn't safe to have it available in your network.

You can have somewhere stashed a cheap WPS-enabled router to set it up your heat-pump, but there is another way to do that.

CZ-TAW1 is equipped with USB port, which initially I thought is for service only. It can be used for setting up your wireless connection. It is possible to create configuration for USB stick, when you're registering an adapter, but then that option on Panasonic cloud is not available any more.

I did some searching and found on Ubiquity forum a post how to create that configuration manually.

you need a USB stick formatted with FAT partition (I have checked 64 GB stick and was read by adapter)
create settings.txt file with Windows (CRLF) new line codes
enter there 2 properties: SSID and KEY with plain text values
place settings.txt file in root folder on USB

SSID=homewifi
KEY=12345678

The author of that post mentioned that adapter may have issues with complex passwords, with characters other than ones from ranges: a-z, A-Z, 0-9.

SAP Commerce Cloud and Broken Smart Edit

Pawel Wolanski — Sun, 04 Jun 2023 13:20:21 +0000

Issue

Recently I have been involved in website go-live. Testers have been complaining that they were not able to see website in Smartedit built-in iFrame.

Looking at the console we realised that recently jsapps endpoints started to send one HTTP Header:



X-Frame-Options: deny

That is probably a consequence of SAP internal security audit OWASP Secure Headers X-Frame-Options.

At SAP Help you can find an article Adding HTTP CSP Frame-Ancestors. You will NOT find explanation how to do that.

Solution

Fortunately there is possibility to add in-the-runtime HTTP Response Headers in Cloud Portal in sub-page Security -> HTTP Response Header Sets.

SAP Help has one section about it here: HTTP Response Header Sets.

Unfortunately X-Frame-Options: deny is a default value and it is not possible to remove from system... but fortunately you can unset it in Cloud Portal.

My configuration for Smartedit contains two entries:

setting Content-Security-Policy with wildcard to allow any request from Commerce Cloud.
unsetting X-Frame-Options to make it finally working, as it is replaced by CSP (more info on MDN XFO