DEV Community: Mirek Sedzinski

@Autowired magic in SpringBoot

Mirek Sedzinski — Sun, 16 Jun 2024 10:01:26 +0000

Disclosure: I'm quite an experienced developer (currently go/python/bit of rust, scala/c# in the past), but a newbie when it comes to Spring Boot.

Recently, I've had to use it in one of my projects and somewhere in the code, I've come across this interesting annotation:
@Autowired

When I've read about it, I couldn't believe my eyes:

In Spring Boot, the @Autowired annotation is used for automatic dependency injection.
This means that Spring will automatically resolve and inject any beans that your object depends on.

I'm familiar with Dependency Injection, but I've never thought that anyone could use, or even propose to use, such an approach.

It's pure magic. I strongly believe that DI should be explicit, and the developer should be aware of what dependencies are injected into his/her class.

Here is, in my opinion, a much better approach to the problem:
https://github.com/google/wire

Wire is a compile-time dependency injection tool. Dependencies are defined in a separate file, and the developer is always aware of what is injected into his/her class.

I'm curious about your opinion on this topic. Do you think that @Autowired is a good approach to DI? Or maybe you prefer a more explicit approach like Wire? Or maybe you have your own way of doing DI? I'm looking forward to your answers.

Shamir's secret sharing

Mirek Sedzinski — Sat, 03 Feb 2024 20:20:11 +0000

Shamir's secret sharing (SSS) is an efficient secret sharing algorithm for distributing private information (the "secret") among a group. The secret cannot be revealed unless a quorum of the group acts together to pool their knowledge.

Behind the concept sits quite simple yet powerful and beautiful math.

In my proof-of-concept I expose SSS implemented in Rust as a WebAssembly. It is then used in a simple web application where interaction with WebAssembly happens via JavaScript.
Please have a look:
https://github.com/msedzins/shamir_rust

Learning Rust - Merkle Tree

Mirek Sedzinski — Sun, 18 Sep 2022 23:08:36 +0000

Recently, in my spare time, I've started to learn Rust and as an exercise I've decided to implement a Merkle Tree (Wiki).

Sounds easy but it's turned out to be a real adventure :) In this post I'm sharing some of my experiences.

Step 1

I've started with preparing the data structures. Here is something really basic, that would work with many languages:

struct MerkelTree {
    leaves: Vec<MerkelNode>
}

struct MerkelNode {
    parent: Option<MerkelNode>,
    hash: String
}

Merkel Tree consists of list of nodes labelled as "leaves". Each leaf contains hash of the corresponding data block and link to the parent node. Parent node groups two leaves (binary tree), and contains hash of their hashes. It also contains link to its parent node which groups two nodes at higher level. That rule is repeated upwards, until we end up with a single node at the top - root node. It contains so called "root hash" and it has no parent.

Needless to say - the structures I've designed didn't work.
The error said:

struct MerkelNode {
   | ^^^^^^^^^^^^^^^^^ recursive type has infinite size

It's turned out to be quite well known error in Rust. At compile time it needs to know how much space a type takes up. Which is not possible with the approach I've taken.

There is an easy solution to that problem described here:
Enabling-recursive-types-with-boxes.
So, let's try it out.

Step 2

I've used "box" smart pointer which allows storing data on the heap. In that way my structure has a known, fixed size at compile time.

struct MerkelNode {
    parent: Option<Box<MerkelNode>>,
    hash: String
}

The code above compiles, so I was happy to start implementation. However, even after a few hours I was still not able to make my code compile...
The problem was not with the logic. The problem was that I was not able to express what I wanted in the way that was acceptable by Rust.

Then it came to me - it just can't work and the problem is with the structures again.
Two core rules in Rust say:

Each value in Rust has an owner.
There can only be one owner at a time.

But in my case I have, for example, two leaves that point to the same parent. Which one of them is owner of the parent?

Step 3

Ok, I can't have two owners, so it means I have to borrow values and use references:

struct MerkelNode {
    parent: Option<Box<&MerkelNode>>,
    hash: String
}

However, above code will not compile because references in structures require using lifetimes:

struct MerkelNode<'a> {
    parent: Option<Box<&'a MerkelNode<'a>>>,
    hash: String
}

The code looks a bit cluttered but at least it compiles. Let's continue with implementation then.

After another few hours, once again, I came to conclusion - it can't work this way.
I can borrow the value multiple times. That's fine. But what about this rule:

Each value in Rust has an owner.

Looks like I don't have an owner now. I mean - I have it somewhere along the line, but I don't have any dedicated place to store it for the entire duration of the program.

Step 4

At that point of time I've come to conclusion that I was doing something fundamentally wrong. There must be a pattern in Rust that will allow me to do what I want...

After some googling, I finally found this:
The Reference Counted Smart Pointer

It provides the multiple ownership capability, whoa!

Definition of the structure looks like this:

struct MerkelNode {
    parent:Option<Rc<MerkelNode>>,
    hash: String
}

Now, finally, I can have two owners for the same parent node.

There is only one final catch here - Rc<T> allows sharing the data for reading only. It is not sufficient in my case because once I created the node, I need to modify it with the pointer to the parent which is not known at the time of creation.

After an hour or so....

Step 5

It turns out there is an interior mutability pattern in Rust that will allow me to do exactly what I need.

The final definition of the structure looks like this:

struct MerkelNode {
     parent: Option<Rc<RefCell<MerkelNode>>>,
     hash: String
}

With this structure I was able to implement working solution. I'm sure there is a lot ot things still to be optimised in my code but today I call it a day and celebrate :)

Getting details of blockchain transaction with GetTransactionByID

Mirek Sedzinski — Sun, 15 May 2022 17:20:04 +0000

In HyperLedger Fabric, to get details of a given transaction, one can use QSCC system chaincode call:

GetTransactionByID

The problem I came across, was with parsing the response. I couldn't find any good explanation how to do this.

Finally, with a bit of experimentation, I've managed to figure it out. Sample code and description are available here:
https://github.com/msedzins/GetTransactionByID

DevOps in Oracle Blockchain Platform

Mirek Sedzinski — Mon, 24 Jan 2022 00:51:09 +0000

Intro

Oracle Blockchain Platform is built on top of the HyperLedger Fabric.
It provides additional capabilities like web administration console, Oracle Berkley DB, with SQL syntax support, as World State, improved transaction validation and, what we will focus in this post on, set of REST APIs in front of native fabric GRPC APIs.

REST APIs cover things like:

chaincode installation, approval, committing
chaincode invocation
updating configuration of nodes (peer, ca, orderer)
checking node health, etc. (full list of APIs is available here: API documentation)

Goal

Let's use said REST APIs to rapidly automate daily devops tasks. To that end we need a CLI that we can use easily in our CI/CD pipelines or to perform ad-hoc tasks.

Implementation

Step 1

First, we need to go to OBP web console and download swagger file with all REST APIs definitions.

Step 2

Now, we need to generate REST client from a swagger spec file. It will be "golang" client in my case so I will use go-swagger tool.

./swagger generate client -f OBP_swagger.yml -A obp-admin

where:

"OBP_swagger.yml" - name of a swagger file
"obp-admin" - arbitrary client name

Generated code is ready to be used straight away. It contains "golang" structures that define requests/responses for respective APIs and "client" package that orchestrates all HTTP communication.

Step 3

We can start to implement calls to REST APIs. Let's make a call to Get Installed Chaincode List endpoint as an example.

Note: Below code uses "cobra" as a library to create CLI application, but of course it't not a requirement.

var ListChaincodesCmd = &cobra.Command{
    Use:   "list",
    Short: "Get a list of installed chaincodes, optionally for a given peer.",
    Run: func(cmd *cobra.Command, args []string) {

        //set basic authentication header (user + password)
        auth := runtimeClient.BasicAuth(cmd.Flag("user").Value.String(), cmd.Flag("passwd").Value.String())

        //set URL to OBP
        config := client.DefaultTransportConfig().WithHost(cmd.Flag("obpHost").Value.String())

        //create client
        cl := client.NewHTTPClientWithConfig(strfmt.Default, config)

        //optionally take id of a peer
        peerId := cmd.Flag("peerId").Value.String()

        //make a call to REST API, parse the response
        resp, err := cl.Chaincode.GetInstalledChaincodes(chaincode.NewGetInstalledChaincodesParams().WithPeerID(&peerId), auth)
        if err != nil {
            logger.Fatalf("Unexpected error:%v", err)
        }

        //display the response (list of chaincodes)
        aJSON, _ := json.MarshalIndent(resp.GetPayload(), "", "\t")
        fmt.Printf("\n%v \n", string(aJSON))
    },
}

Summary

Using swagger spec we can implement devops CLI tool for OBP rapidly.

A few practical things to keep in mind:

in case of failure (for example: due to bad input parameters) the response from the server can be very vague - HTTP code without any further information. It's good then to check the server logs, usually they are very descriptive.
sometimes, changes to autogenerated code are needed. One example is, when "bool" parameter in API request is required, but is marked as "omitempty" in autogenerated code - we need to remove "omitempty" tag then (otherwise, if parameter is set to "false", it will be omitted from the request payload which will result in an error).

The misconceptions about Polyglot programming

Mirek Sedzinski — Sun, 18 Apr 2021 21:08:55 +0000

Back in the day, a life was easy. N-tier architecture reigned the world and each tier was implemented roughly in one technology, for example:

data layer - relational database
business layer - Java EE/.Net
user interface - HTML + CSS + Java Script

Then microservices and functions came along, together with the approach called Polyglot programming (and related term PolyglotPersistence which pertains specifically to the data layer).

So, don't get me wrong. I completely accept the fact that there are many languages out there and different languages can be a better fit for different things.

If I want to provision infrastructure - I use Terraform. If I want to do some scripting - I use Bash/Python. If I want to analyse data I would go with Python or R.

What I strongly disagree with, is introducing new technologies to projects just "because I can" or "because I want to play with something new" or "because technology X is a bit better/fancy/popular than technology Y".

Different languages on the project

As architects and developers we design and implement systems so that they meet all customer requirements and at the same time we strive to make them as simple as possible.
Simplicity is a key for maintainability and extensibility. Every new requirement, every new test we create, every new library we use - makes it more complex.
And introducing a new language is one of the most complex things I can imagine....

This complexity is not about the syntax. Syntax is usually easy. And it's not about writing one-off code. Writing a code that you plan to throw away tomorrow is super easy.

But when we write enterprise grade systems the bar is set much higher. Just a few things off the top of my head:

Idiomatic ways of approaching common problems (logging, error handling, input parameters etc.)
New libraries (when approaching new language, usually we have to learn completely new libraries that do exactly the same as libraries we already know but in completely different ways)
Packages/modules/dependencies management (again: completely new way of doing the same things)
New toolset (compilation, testing, debugging)
Configuration of dev. environment
Integration with CI/CD framework

And of course, when one person in the team, starts to use a new technology, all other members, sooner or later, have to learn it (to do code reviews or to stand in when needed)

Different databases on the project

Let me tell you my story:
One day I was asked to provision MongoDB and ElasticSearch for one of the projects.
I had no experience with those databases, but spinning up containers and exposing the endpoints was super easy. Development team was happy.

Then, a few "small" activities came along:

High availability - Turned out, that I had to set up two 3-node clusters. And of course - setting up cluster for MongoDB was very different than for ElasticSearch. After provisioning - I had to test if both clusters work.
Backups - For MongoDB it was quite easy, but for ElasticSearch it turned out that shared volume was required (so NFS server had to be configured). Again - backup and recovery procedure had to be tested.
Security - users/roles/which ports must be open, which ports can be closed etc. Completely different ways of doing the same things in each database.
Patching, monitoring, performance tweaking and troubleshooting.....

The conclusion is the following:
Of course one can learn how to operate any database in the world. But it requires significant effort and time. Each new database brings new tools, new concepts and tons of documentation.

My conclusion - whenever possible stay with one generic database to handle most of the traffic. Add new databases only to handle very specific workloads and only when really necessary.
Some examples of specific workloads are: caching, graphs, big data, time series. However, we should also bear in mind, that current databases usually support more than one type of workload.

And what is your take on this? I would love to hear your comments and experiences!

Why do we need the Abstract Factory pattern?

Mirek Sedzinski — Thu, 08 Apr 2021 22:50:39 +0000

I had a really hard time understanding what are real benefits of using AbstractFactory pattern. Now, when I've finally got it, let me explain it in 6 steps :)

Step 1

Let's imagine a quite common situation in which we've been working on a custom code and we've just realised that we need to reuse some logic from utility class created by the friend of ours, working on the same team.

It can look like this:

Step 2

We have a working code but there is at least one problem with it - it violates Dependency Inversion principle. Let's try to fix it then:

Step 3

Ok, much better. But the next question arises: how to create Utility class before using it...? Of course, we have a Factory pattern for this:

Our custom code is using Factory class that creates instance of Utility class and returns it as an interface for us to use.

Step 4

One thing immediately to notice is that we again violate Dependency Inversion principle. For some people it can be enough reason for putting AbstractFactory in between custom code and Factory class.
But we don't have to stick to the rule, just for the sake of doing so :)
Let's ignore Dependency Inversion principle for a moment and see what will happen.

Well, on many occasions nothing will happen.....difference can be seen for more complex scenarios though.

Let's imagine that we have more than one Utility class that implements the same interface. It can look like this:

Step 5

What is the problem with our design? To notice it, let's look at source code dependencies:

Custom code is placed in File 1. To make it work we need to import File 2, File 3 and apparently all files that contain all possible implementations of *Utility class - even if we use only one implementation.
It is clearly not an optimal solution.

Step 6

We would like to apply Common Reuse principle. In short - we don't want to depend on things that we don't need.
If we use Utility class, version X - we wan't to depend only on its source code. Nothing more.

So, here is the final design with a use of AbstractFactory:

Assuming that each Factory class is a different module, we depend physically only on File 1, File 2, File 3, File 4, File 5.
For complex and big systems this kind of flexibility can bring many benefits.

Super lightweight approach to unit testing in Bash

Mirek Sedzinski — Sat, 20 Feb 2021 12:54:08 +0000

There is already a good, recognised testing framework for Bash.

But if someone is looking for a minimalistic approach, please check out my repo:
lightweight unit testing

Very simple rule of thumb on when to write tests

Mirek Sedzinski — Fri, 01 Jan 2021 22:41:30 +0000

Testing is a critical part of software development process. And there is a ton of literature on when and how to test.

I'm not a big fun of sticking to any particular approach just because it's currently popular.

Over the course of time I found a simple way to identify potential candidate for tests in my code:

As I write a piece of the code, very often, I start to feel uncertain whether it will work or not. This feeling is accompanied by the urge to execute the code and validate its behaviour at runtime.
Well, this for me is a strong indicator that I should write tests.

Having said that, please mind that I'm talking about potential candidate. Sometimes (rarely), on second thought, I come to conclusion that no tests are needed.

A few practical Terraform tips

Mirek Sedzinski — Fri, 20 Nov 2020 17:08:28 +0000

Terraform is a popular, open-source infrastructure as a code software tool.
This article aims to present a few tips on how to use it, based on hands-on experience. Readers are assumed to have at least some level of Terraform working knowledge.

Teamwork - the State

Let's say we created a bunch of Terraform scripts. Most probably we keep them in the repository of our choice. By doing so, we can easily share them between team members.
The question arises: what about the State?

By default, it's stored in a file in a current working directory where Terraform was run. Should it be pushed to the repository together with Terraform scripts?
Actually, it's not a best idea. State file is machine-generated and there is a significant probability of frequent merge conflicts between different revisions. Those conflicts will have to be resolved by hand and it won't be easy.

There are two options to handle this:

Local state - state kept as a file in a shared location. Sharing can be achieved with the use of a network-attached storage. Or there can be one dedicated "builder" machine reused by the whole team.
Remote state - state kept on a remote storage. This is a feature of Backends, and there are several of them to choose from. What's good to check and be aware of is whether given Backend supports locking mechanism (for example Oracle Object Storage currently doesn't). Locking mechanism is a measure to avoid two or more different users accidentally running Terraform at the same time, and thus ensure that each Terraform run begins with the most recent updated State.

Teamwork - running the scripts

Whatever Backend we use, and regardless of whether it supports the locking mechanism or not, if two users run the same set of terraform scripts, which are out of sync, we are in a trouble.

Let's imagine a situation where two developers pull the same scripts from the repository. Developer A modify scripts by adding an additional Compute instance. She runs the scripts and the instance is provisioned. Shared state is updated.
A few minutes after that, Developer B runs his version of the scripts (which he didn't modify). Terraform compares the content of shared state with the content of the scripts and finds out that:

Instance was provisioned on the infrastructure [information from the State]
There is no instance in the current scripts

Based on the above, Terraform comes to conclusion that the Compute instance has to be decomissioned. Obviously, this is not what we expected.

To prevent such situations, one must make sure that Terraform is run always using up-to-date scripts. It can be done by defining a manual process or with a tool.
CI/CD pipeline or a job can be created for that purpose. Or specific service can be used like Resource Manager, that is part of OCI offering.

Organising the scripts

Two things should be taken into consideration here: avoiding redundancy and planning for efficient use.

For redundancy part, one should consider:

Moving common elements to modules to promote reusability
Using variables to parametrise the scripts

Side note
All sensitive data should be removed from the scripts and loaded from external variables. In this post I'm talking about one possible approach to do that in a safe way: Link

When it comes to efficiency, we should first reflect on how we are going to provision and decommission our infrastructure. Things that we want to provision/deprovision together should obviously go together in the scripts.
However, at the same time, we should keep in mind that in Terraform we usually use "everything or nothing" approach. In other words - either we provision everything or nothing. The same holds true for decommissioning. Of course, there are ways to narrow down the scope (the "-target" option can be used to focus Terraform's attention on only a subset of resources), but it should be treated more like and exception than a rule.
So, it's better to have a few independent set of scripts which we can run separately and orchestrate as needed, even if they are tightly coupled and pertain to the same piece of the software and infrastructure.

For example, let's say we want to provision Kubernetes cluster. Instead of putting everything into one big set of scripts, we can divide it into following components:

Scripts to provision identity provider
Load Balancer
Image registry
Control + data plane
Extensions like storage, cert manager, etc.

Each component, from the list above, is a complex thing. It's good to have a possibility to approach them separately or together, depending on a need.

Terraform vs Ansible

Frequent question is: which tool should be used, Ansible or Terraform?

To answer it let's first make a differentiation between management of:

Infrastructure - vm, storage, networking etc.
Configuration - software installed on top of the infrastructure

So, we can definitely use either tool to cover both areas. It especially makes sense for easy use cases. For example, we can go with Terraform only and use cloud-init/provisioners for configuration management.

However, in more complex situations, in my opinion it's good to use the tools for what they were originally designed for, which means: Terraform for infrastructure and Ansible for configuration management. It just makes things easier and more natural.

Idempotency

And last important advice: regardless of the tool used, scripts should be idempotent. It increases a bit effort to implement them (especially in case of Ansible) but pays off greatly later on.

Using Oracle Streaming service as a managed Apache Kafka

Mirek Sedzinski — Tue, 22 Sep 2020 15:59:30 +0000

The Oracle Cloud Infrastructure Streaming is a fully managed and scalable service for processing high-volume streams of data in real time.
We can think of it as an equivalent of Apache Kafka and actually - they are API-compatibile, which means we can use applications written for Kafka with Streaming without having to rewrite a code.

Problem statement

Why we would even consider using Streaming service? Well, for the same reasons we would consider using Kafka.

The additional bonus is that because it's a PaaS service, it's extremely easy to set up and there's almost no effort related to maintenance (forget about patching, scaling, configuring HA or running out of disc space).

How to use it

Configuration can be done in a standard way - manually in the cloud console or automatically with REST API/SDKs/Terraform/CLI.

Things are getting more interesting when it comes to publishing/consuming messages. There are two approaches:

OCI REST API
Kafka-compliant API

Usually, I tend to go with the latter, for one reason: long-lived TCP connections used by Kafka protocol. Using this mechanism, makes the process of reading the data much more interactive - one can listen certain amount of time on a topic and when a message arrives it is consumed at once. In case of OCI REST API, the only way is to call GetMessages operation in a loop and it exits immediately when there is nothing to consume.

In the next section I will show how easy is to configure Streaming and connect to it using Kafka client.

Setup

Stream Pool

First, let's create stream pool which groups streams (and the easiest way to think about a stream is to treat it as a Kafka topic):

The only required parameter that has to be provided is actually a Stream Pool Name. For other settings we can use default values.

However, there are two interesting configuration parameters to mention:

Endpoint type - we can choose whether our streams are public (ingress traffic allowed from the internet) or private (accessible only within Oracle Cloud region).
Encryption settings - all messages are encrypted by default (at rest and in-transit). We can specify details of the configuration.

Stream

Now we can create a Stream (aka topic):

First we need to assign a Stream to a Stream Pool (the one we created in the previous section). Then we need to provide the name of the Stream.

We can control read/write throughput using mechanism of partitions (works the same as in Kafka).

Kafka configuration

When we open a page with details of any given Stream Pool, at the top of the screen we will find the following button:

After clicking it, the page with connection details pops up:

Having all that data we are almost ready to connect to Streaming service using Kafka client. But first we need to touch on one very important topic - security.

A few important words about security

Encryption

Connection between client and server is encrypted in transit using TSL (successor of SSL). And there is a 1-way authentication enabled by default, which means that client authenticates the server certificate.

To make that authentication work, client has to trust the certificate presented by the server. Usually it requires additional configuration.

There is a setting that can be used on client side: "ssl.ca.location". It can be used to point to the file with proper certificate chain of trust. The file itself should look like this:

# Server certificate
-----BEGIN CERTIFICATE-----
MIIFaDCCBFCgAwIBAgISESHkvZFwK9Qz0KsXD3x8p44aMA0GCSqGSIb3DQEBCwUA
VQQDDBcqLmF3cy10ZXN0LnByb2dyZXNzLmNvbTCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBAMGPTyynn77hqcYnjWsMwOZDzdhVFY93s2OJntMbuKTHn39B
...
bml6YXRpb252YWxzaGEyZzIuY3JsMIGgBggrBgEFBQcBAQSBkzCBkDBNBggrBgEF
BQcwAoZBaHR0cDovL3NlY3VyZS5nbG9iYWxzaWduLmNvbS9jYWNlcnQvZ3Nvcmdh
bml6YXRpb252YWxzaGEyZzJyMS5jcnQwPwYIKwYBBQUHMAGGM2h0dHA6Ly9vY3Nw
lffygD5IymCSuuDim4qB/9bh7oi37heJ4ObpBIzroPUOthbG4gv/5blW3Dc=
-----END CERTIFICATE-----

# Trust chain intermediate certificate
-----BEGIN CERTIFICATE-----
MIIEaTCCA1GgAwIBAgILBAAAAAABRE7wQkcwDQYJKoZIhvcNAQELBQAwVzELMAkG
C33JiJ1Pi/D4nGyMVTXbv/Kz6vvjVudKRtkTIso21ZvBqOOWQ5PyDLzm+ebomchj
SHh/VzZpGhkdWtHUfcKc1H/hgBKueuqI6lfYygoKOhJJomIZeg0k9zfrtHOSewUj
...
dHBzOi8vd3d3Lmdsb2JhbHNpZ24uY29tL3JlcG9zaXRvcnkvMDMGA1UdHwQsMCow
KKAmoCSGImh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5uZXQvcm9vdC5jcmwwPQYIKwYB
K1pp74P1S8SqtCr4fKGxhZSM9AyHDPSsQPhZSZg=
-----END CERTIFICATE-----

# Trust chain root certificate
-----BEGIN CERTIFICATE-----
MIIDdTCCAl2gAwIBAgILBAAAAAABFUtaw5QwDQYJKoZIhvcNAQEFBQAwVzELMAkG
YWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT
aWduIFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDaDuaZ
...
jc6j40+Kfvvxi4Mla+pIH/EqsLmVEQS98GPR4mdmzxzdzxtIK+6NiY6arymAZavp
38NflNUVyRRBnMRddWQVDf9VMOyGj/8N7yy5Y0b2qvzfvGn9LhJIZJrglfCm7ymP
HMUfpIBvFSDJ3gyICh3WZlXi/EjJKSZp4A==
-----END CERTIFICATE-----

How to build the chain of trust? We start at the bottom of the chain ("Server certificate" in the example above, but please mind that direction in the file is reversed).

To get the server certificate we can run the following command:

echo -n | openssl s_client -connect <endpoint taken from Stream details page>

It will return two very important pieces of information:

"Certificate chain" section which lists certificates. All of them should be placed in our file. In my case it looks like this (there are 3 certificates in the chain):

Certificate chain
 0 s:/C=US/ST=California/L=Redwood City/O=Oracle Corporation/OU=Oracle OCI-PROD FRANKFURT/CN=streaming.eu-frankfurt-1.oci.oraclecloud.com
   i:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
 1 s:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
 2 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA

"Server certificate" - section between BEGIN/END CERTIFICATE lines. This is a certificate number "0" in a "Certificate chain" and should be placed at the top of our file.

The last step is to collect remaining certificates ("1" and "2") and place them in the file. The root certificate goes at the bottom.

BTW:
For testing purposes we can use quick and dirty workaround. Setting "enable.ssl.certificate.verification" to "false" will disable server authentication altogether.

Authentication and authorisation

It's highly recommended to create a dedicated user for each Stream Pool (or set of Stream Pools, depending on the requirements).
After creating such a user, we have to:

generate for him Auth Token, which will be used as a password.
grant him appropriate level of access using OCI Policies

Kafka client

Finally, let's try to connect to our Streaming service using Kafka Client. In my case I will use Go and Confluent package.
Example of producer and consumer code that we can use is available here:

Consumer configuration

consumer, err := kafka.NewConsumer(&kafka.ConfigMap{
        //taken from Stream Pool details page
        "bootstrap.servers":                   "cell-1.streaming.eu-frankfurt-1.oci.oraclecloud.com:9092",
        //arbitrary value, when using consumer groups
        "group.id":                            "foo",
        //taken from Stream Pool details page
        "sasl.mechanisms":                     "PLAIN",
        //user authorized to read from the  stream(s) 
        "sasl.username":                       "[tenancy]/[user name]/[stream pool id]",
        //Auth Token for the user
        "sasl.password":                       "[token]",
        "enable.ssl.certificate.verification": "true",
        //Full path to the file with certificates that make up chain of trust
        "ssl.ca.location":                     "/dir/ca.pem",
        //taken from Stream Pool details page
        "security.protocol":                   "SASL_SSL",
        "auto.offset.reset":                   "earliest"})  

//subscribe to given Stream (aka topic)
err = consumer.SubscribeTopics([]string{"testStream"}, nil)

Producer configuration

p, err := kafka.NewProducer(&kafka.ConfigMap{
        //taken from Stream Pool details page
        "bootstrap.servers":                   "cell-1.streaming.eu-frankfurt-1.oci.oraclecloud.com:9092",
        //taken from Stream Pool details page
        "sasl.mechanisms":                     "PLAIN",
        //user authorized to write to the  stream(s) 
        "sasl.username":                       "[tenancy]/[user name]/[stream pool id]",
        //Auth Token for the user
        "sasl.password":                       "[token]",
        //example of how to disable server certificate validation
        //don't do it in production!
        "enable.ssl.certificate.verification": "false",
        //taken from Stream Pool details page
        "security.protocol":                   "SASL_SSL"})

Sensitive data in bash scripts

Mirek Sedzinski — Sat, 18 Jul 2020 07:47:37 +0000

Problem

Sensitive data (passwords etc.) should not be placed directly in the configuration files.
Quite common approach is to use environment variables to do the trick: we assign sensitive data to the environment variables and use these variables later on in our configuration/scripts/code.
But this raises another question: how we should make assignment itself? Should we use for that just another script in which sensitive data is stored in plain text?

Below is the approach that I've been talking.

Solution

There is a really nice script out there - encpass.sh

What it allows to do is to store a secret in a reasonably safe way and retrieve it easily in a bash script.

But why only in a reasonably safe way?
Well, secrets are encrypted using symmetric keys and both values (secret + password) are stored in a hidden directory that can only be accessed by the user who run the script. It obviously doesn't protect from situations in which attacker has an access to the user's hidden directory (for example if an attacker has a root access).
This risk can be mitigated by securing access to the keys with an additional password. In such a case, before using the secret, user is asked to provide the password. Although it works good for manual use cases, can become cumbersome in case when automation is used (we get back to the original problem - how to store password in a secure way?).

Quick demo

After setting up secrets using encpass.sh we can have a configuration file named configuration.sh, that looks like this:

#!/bin/sh

configuration_param1=data1 # not sensitive data

source encpass.sh
configuration_param2=$(get_secret bucket_name secret_name)

Then, in our script we use it like this:

#!/bin/sh

source configuration.sh

echo configuration_param1
echo configuration_param2

BTW: This approach can also be used with other languages, for example recently I've used it with Go - there is a script in bash which uses encpass.sh to decrypt the secrets and then exports them. Afterwards I can access environment variables from my Go code.