DEV Community: Kooin-Shin

Log4J JNDI Remove Utility

Kooin-Shin — Tue, 14 Dec 2021 08:35:39 +0000

Some days ago, There was big shocking security issue disclosed about Log4J JNDI exploit vulnerability.

It was almost veiled over 8 years!

Some kind of worried thing blows in my head because it's very most used Logger API ever.

But we have to get a chance to recover it.

Not fully complete, but complementary thing is here. It's a way to remove JndiLookup.class in Log4J jar files.

So I introduce a utility for removing JNDI class in jars by batch processing from top level directory.

Download and execute jar like 'java -jar log4j-jndi-remover.jar', then you can fix your applications and services.

How to find value using path array on structural document in Java

Kooin-Shin — Thu, 02 Sep 2021 05:46:46 +0000

Sometimes we need to find a value on structural document like JSON, YAML. Also there is sometime to need to find a value by path of document.

Structural document can be easily converted to Map-List object with parser libraries. Because of it, we can get a value of the document using key and index array.

To do that,

First, we need to get a parser library that can parse JSON or YAML to Java Map-List structure. That library can be found easily in internet library repository like Maven Central.

For this post, we assume that we target JSON document.
It's down below

{
  "id": "0001",
  "type": "donut",
  "name": "Cake",
  "ppu": 0.55,
  "batters": {
    "aaa": {
      "bbb": {
        "ccc": {
          "value": 123.0
        }
      }
    },
    "batter": [
      {
        "id": "1001",
        "type": "Regular"
      },
      {
        "id": "1002",
        "type": "Chocolate"
      },
      {
        "id": "1003",
        "type": "Blueberry"
      },
      {
        "id": "1004",
        "type": "Devil\u0027s Food"
      }
    ]
  },

We use recursive algorithm to find a value and if path can't find value or path isn't exist, It will return null.

Completed code is down below

    /**
     * To find value with path array on structural data
     * 
     * @param obj
     * @param keys
     * @return
     */
    public static Object findValue(Object obj, Object[] keys) {
        if (obj instanceof List) {
            List list = (List) obj;
            if (keys.length == 1) {
                if (keys[0] instanceof Integer && list.size() > (int) keys[0]) {
                    return list.get((int) keys[0]);
                }
            } else if (keys.length > 1 && list.size() > 0) {
                int idx = (int)keys[0];
                if(idx < list.size()) {
                    return findValue(list.get((int) keys[0]), Arrays.copyOfRange(keys, 1, keys.length));
                }
            }
        } else if (obj instanceof Map) {
            Map map = (Map) obj;
            if (keys.length == 1 && map.containsKey(keys[0])) {
                return map.get(keys[0]);
            } else if (keys.length > 1 && map.containsKey(keys[0])) {
                if(keys.length > 0) {
                    return findValue(map.get(keys[0]), Arrays.copyOfRange(keys, 1, keys.length));
                }
            }
        }
        return null;
    }

Well, let's start test now

Above JSON document save to file named 'sample.json'
Below code is read 'sample.json' and result is image more below.

Gson gson = new Gson();
Map<String, Object> map = gson.fromJson(new FileReader("simple.json"), Map.class);
Object[] keys = new Object[] {"batters", "batter", 3, "type"};

Object obj = findValue(map, keys);
System.out.println(obj);

Console Application Input Library in Java.

Kooin-Shin — Fri, 23 Apr 2021 02:51:31 +0000

Hi all,
I will introduce a library to build console application with more easier, and with this library, you can make fine interactive methodology on your console application.

You can define queries for user input using this library on your application and later, you able to use the input data in your code.

All queries are managed with YAML file and you can redefine message content of it.
You can manage trigger process by implementing ConsoleTrigger interface and treat user input data on this object.

More detail thing of this library to get, you can have it to visit below GitHub project.

Dependencies

Maven dependency

<!-- https://mvnrepository.com/artifact/io.github.9ins/console-input-manager -->
<dependency>
    <groupId>io.github.9ins</groupId>
    <artifactId>console-input-manager</artifactId>
    <version>1.0.1</version>
</dependency>

Gradle dependency

// https://mvnrepository.com/artifact/io.github.9ins/console-input-manager
implementation group: 'io.github.9ins', name: 'console-input-manager', version: '1.0.1'

GitHub link: [console-input][https://github.com/9ins/console-input-manager]

How to load JDBC MySQL driver using instantiated driver on runtime.

Kooin-Shin — Tue, 16 Feb 2021 01:38:24 +0000

Lately, I have worked to develop Java application that should need to be loaded JDBC driver on it's runtime.

By default, Java is providing solution of class loading method with called ClassLoader class.
But JDBC driver of third-party vender is very sensitive and fussy to use.
If you would approach it with general java class loading method, You might be failed about your goal. Especially MySQL driver don't work and do output message "No suitable driver found". As my experience until now, MySQL Driver only worked to do wrong like that, Maybe other drivers could be working wrong like that.

Nevertheless, I will show you about general usage of Java class loading. Generally, JDBC class loading code on runtime is down below.

    public static Connection getConnection(File jarFile) throws MalformedURLException, ClassNotFoundException, InstantiationException, IllegalAccessException, SQLException {
        URL jarUrl = jarFile.toURI().toURL();
        URLClassLoader loader = new URLClassLoader(new URL[]{jarUrl}, ClassLoader.getSystemClassLoader());
        Class clazz = Class.forName("com.mysql.jdbc.Driver", true, loader);
        Properties prop = new Properties();
        prop.put("user", "admin");
        prop.put("password", "1234");
        Connection conn = (Connection)DriverManager.getConnection("jdbc:mysql://192.168.1.157:3306/mysql", prop);
        return conn;
    }

Upper code snippet will fail with error message on console that is down below.

D:\Projects\chaos-commons> cmd /C "c:\Users\chaos\.vscode\extensions\vscjava.vscode-java-debug-0.31.0\scripts\launcher.bat C:\jdk-11\bin\java.exe -Dfile.encoding=UTF-8 @C:\Users\chaos\AppData\Local\Temp\cp_f58mz4nw79996julgmkyp9ycf.argfile org.chaostocosmos.commons.loader.JobClassLoader "
Exception in thread "main" java.sql.SQLException: No suitable driver found for jdbc:mysql://192.168.1.157:3306/mysql
        at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:702)
        at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:189)
        at org.chaostocosmos.commons.loader.JobClassLoader.getConnection(JobClassLoader.java:75)
        at org.chaostocosmos.commons.loader.JobClassLoader.main(JobClassLoader.java:92)

In above code, I create URLClassLoader class from path of jar file, load JDBC driver by Class.forName method with the URLClassLoader object and then have gotten Connection object by DriverManager object.

Now we are going to see the code sample that is working well. It has to create Driver object by URLClassLoader's loadClass method and then getting Connection object with connect method of Driver object.

The code of that be completed is down below.

    public static Connection getConnectionByDriver(File jarFile) throws MalformedURLException, InstantiationException, IllegalAccessException, ClassNotFoundException, SQLException {
        URL jarUrl = jarFile.toURI().toURL();
        URLClassLoader loader = new URLClassLoader(new URL[]{jarUrl}, ClassLoader.getSystemClassLoader());
        Driver driver = (Driver) loader.loadClass("com.mysql.jdbc.Driver").newInstance();
        Properties prop = new Properties();
        prop.put("user", "admin");
        prop.put("password", "1234");
        Connection conn = driver.connect("jdbc:mysql://192.168.1.157:3306/mysql", prop);
        return conn;
    }

As my experience, Oracle driver was working well with general Java class loading method but MySQL was not. Any other DB driver have not been tested yet by me.
If you have been found something about it of other drivers, Let all of DEV know :) Thanks.

Introduce Open-Source project TCPProxy.

Kooin-Shin — Wed, 06 Jan 2021 01:59:29 +0000

Hi all, Now I gonna introduce a Open-Source project.

It's name is TCPProxy. It's small size project, but having quite performance.

TCPProxy could be used a proxy for secure purpose and a Gateway server for entry point of server pool and even a HA server or Load-Balancer.

Link is: TCPProxy

TCPProxy can manage multiple sessions and configure each of that.

I will post how to build services with TCPProxy soon.
If you wanna know right now, please visit link above to get some information.

This project just beginning and be needed of many people's sounds.

Your cheer and interest of that, would became strength to the project.

Thanks,

How to handling internal Yaml in Jar.

Kooin-Shin — Mon, 04 Jan 2021 06:27:03 +0000

Intro.

Some far ago, I've been introducing 'How to manage your configuration file with YAML in Java programmatically' in previous post.
if you wanna know it, follow link: How to manage your configuration file with YAML in Java programmatically

It was about handing Yaml file contents to create Java Bean object. That was including information of a configuration or serialized Valued-Object(VO). Simply to say, It was about load or dump information between Yaml file and Java object.

Meanwhile, Someone who has keen thinking asked how to access a Yaml file in internal resources?
So I realized it's need,
Here I would show how to load and dump Yaml file in Jar.

Setting up environments.

As same as previous post, To handle Yaml file, We need SnakeYAML library.
For our works, Any IDE Tools is able to use, But I gonna to use Visual Studio Code(VSCODE) on fever in these days.
And we have to set SnakeYAML MAVEN dependency to POM.xml. In previous post I used Gradle but now, Would use MAVEN for diversity :)
If you are prepared with above things, Readied to start coding.

Designing Java Class.

Before we gonna see implemented code, I would show you entire design of our works.

We will define java class named "YamlInternalHandler" to handle Yaml file in Jar.
Will define two method named "loadInternalJar" and "dumpInternalJar". First one is to load Yaml's contents to Java object with jar entry path and a Class object to be loaded the contents. Second one is to dump a object values to Yaml file in Jar with jar entry path and instance of Java object including a values.
We could load and dump a specified object mapping with a Yaml file in jar. Surely, It's good to make sure syntax of Yaml file in Jar.

Let's dive into code.

Contents of Yaml file in Jar is down below.

firstName: Kooin
lastName: Shin
age: 48
profile:
  - character: bright
    hobby: claiming
    address: Seoul
    postalCode: 123
  - character: delight
    hobby: ski
    address: Seoul
    postalCode: 123

Java Bean to be loaded contents is 'Customer' and 'Profile' representing Yaml's values.

package org.chaostocosmos.doc.yml;

import java.util.List;

public class Customer {

    private String firstName;
    private String lastName;
    private int age;
    private List<Profile> profile;

    public Customer() {}

    public String getFirstName() {
        return this.firstName;
    }

    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    public String getLastName() {
        return this.lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    public int getAge() {
        return this.age;
    }

    public void setAge(int age) {
        this.age = age;
    }

    public List<Profile> getProfile() {
        return this.profile;
    }

    public void setProfile(List<Profile> profile) {
        this.profile = profile;
    }

    @Override
    public String toString() {
        return "{" +
            " firstName='" + firstName + "'" +
            ", lastName='" + lastName + "'" +
            ", age='" + age + "'" +
            ", profile='" + profile + "'" +
            "}";
    }
}

package org.chaostocosmos.doc.yml;

public class Profile {

    String character;
    String hobby;
    String address;
    int postalCode;    

    public Profile() {}

    public String getCharacter() {
        return this.character;
    }

    public void setCharacter(String character) {
        this.character = character;
    }

    public String getHobby() {
        return this.hobby;
    }

    public void setHobby(String hobby) {
        this.hobby = hobby;
    }

    public String getAddress() {
        return this.address;
    }

    public void setAddress(String address) {
        this.address = address;
    }

    public int getPostalCode() {
        return this.postalCode;
    }

    public void setPostalCode(int postalCode) {
        this.postalCode = postalCode;
    }

    @Override
    public String toString() {
        return "{" +
            " character='" + character + "'" +
            ", hobby='" + hobby + "'" +
            ", address='" + address + "'" +
            ", postalCode='" + postalCode + "'" +
            "}";
    }    
}

And Now we gonna show core code of 'YamlInternalHandler' class.

package org.chaostocosmos.doc.yml;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStreamWriter;
import java.util.jar.JarEntry;
import java.util.jar.JarFile;
import java.util.jar.JarOutputStream;
import java.util.zip.ZipEntry;

import org.yaml.snakeyaml.DumperOptions;
import org.yaml.snakeyaml.Yaml;
import org.yaml.snakeyaml.DumperOptions.FlowStyle;
import org.yaml.snakeyaml.constructor.Constructor;

/**
 * Yaml Internal Handler 
 * 
 * @author Kooin-Shin
 * @since 2021.01.04
 */
public class YamlInternalHandler {

    /**
     * Jar file having yaml file
     */
    File file;

    /**
     * Constructor
     * @param file
     * @throws IOException
     */
    public YamlInternalHandler(File file) throws IOException {
        this.file = file;
    }

    /**
     * Load object from internal yaml file in jar
     * @param jarEntry
     * @param clazz
     * @return
     * @throws IOException
     */
    public Object loadInternalJar(String jarEntry, Class<? extends Object> clazz) throws IOException {
        if (!jarEntry.endsWith(".yml")) {
            throw new IllegalArgumentException("This component only support a yaml file in jar!!!");
        }
        JarFile jarFile = new JarFile(this.file);   //Create JarFile object
        JarEntry entry = jarFile.getJarEntry(jarEntry);  //Get JarEntry specified by first parameter
        InputStream is = jarFile.getInputStream(entry);  //Get InputStream from JarFile  by JarEntry
        Constructor constructor = new Constructor(clazz); //Create Constructor object specified by second parameter
        Yaml yaml = new Yaml(constructor);  //Create Yaml object with Constructor object
        Object obj = yaml.load(is);   //Load contents to object instance
        is.close();  //Close InputStream
        jarFile.close();  //Close JarFile
        return obj;
    }

    /**
     * Dump object to yaml file in jar
     * @param jarEntry
     * @param obj
     * @throws FileNotFoundException
     * @throws IOException
     */
    public void dumpInternalJar(String jarEntry, Object obj) throws FileNotFoundException, IOException {
        JarOutputStream jos = new JarOutputStream(new FileOutputStream(this.file));  //Create JarOutputStream object 
        jos.putNextEntry(new ZipEntry(jarEntry));  //Put jar entry by first parameter
        DumperOptions options = new DumperOptions();  //Set dump options
        options.setDefaultFlowStyle(FlowStyle.BLOCK); 
        options.setPrettyFlow(true);
        Yaml yaml = new Yaml(options);  //Create Yaml object with DumperOptions object
        yaml.dump(obj, new OutputStreamWriter(jos));  //Dump contents of second parameter
        jos.close(); // Close JarOutputStream
    }

    public static void main(String[] args) throws IOException {
        File file = new File("D:\\Projects\\chaos-commons\\yaml.jar");
        YamlInternalHandler ymlHandler = new YamlInternalHandler(file);
        Customer customer = (Customer) ymlHandler.loadInternalJar("tmp/yaml/Customer.yml", Customer.class);
        System.out.println(customer.toString());
        customer.setFirstName("Jim ");
        customer.getProfile().get(0).setAddress("Busan");
        ymlHandler.dumpInternalJar("tmp/yaml/Customer.yml", customer);
    }
}

Conclusion

Above code is assumed very simple cases of accessing of a Yaml in Jar. As a situation, It can be transformed, modified and improved.
Always elastic thinking is yours!!!
Good Luck!!!

How to implement to select specified ratio using Random function in Real-Time

Kooin-Shin — Thu, 10 Dec 2020 06:54:38 +0000

Intro

What is efficient way of Load-Balanced selection algorithm in Real-Time.
The best way of this is that it's always calculate separate ratio of previous and pick up a less selected part and then let a new request to be assigned into that.
But it's not easy and likely to be complicated.

Here, There's quite simple and reliable way to do that.
Let's dive in.

Logical consideration

It would express to specify each ratio separated with semi-colon. e.g. "10:30:40:20"
It would use Random class provided by Java.
And return value should be a index among the specified ratios.

Sequential flow

Create Random object.
Make sum of all of ratios.
Generate random integer value ranging 0 ~ sum.
Compare the random value where it's placed at.
Return the index in range among ratios

Code review

package org.chaostocosmos.dev.io;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import java.util.stream.Collectors;

public class RandomRatioIndexSelection {

    List<Integer> ratioList;

    public RandomRatioIndexSelection(String ratioExpr) {
        this(Arrays.asList(ratioExpr.split(":")).stream().map(e -> Integer.parseInt(e)).collect(Collectors.toList()));
    }

    public RandomRatioIndexSelection(List<Integer> ratioList) {
        this.ratioList = ratioList;
    }

    public int getSelectIndex() {
        Random random = new Random();
        int sum = ratioList.stream().reduce((a, b) -> Integer.sum(a, b)).get().intValue();
        int start = 0;
        int bound = 0;
        int ran = random.nextInt(sum);
        for (int i = 0; i < ratioList.size(); i++) {
            int ratio = ratioList.get(i);
            bound += ratio;
            if (ran >= start && ran < bound) {
                return i;
            }
            start += ratio;
        }
        return -1;
    }

    public static void main(String[] args) {
        RandomRatioIndexSelection rris = new RandomRatioIndexSelection("10:30:40:20");
        List<Integer> ratioList = new ArrayList<Integer>();
        for(int i=0; i<10000; i++) {
            int index = rris.getSelectIndex();
            ratioList.add(index);
        }
        long r0 = ratioList.stream().filter(i -> i == 0).count();
        long r1 = ratioList.stream().filter(i -> i == 1).count();
        long r2 = ratioList.stream().filter(i -> i == 2).count();
        long r3 = ratioList.stream().filter(i -> i == 3).count();
        System.out.println("Toal count: "+r0+"  "+r1+"  "+r2+"  "+r3);
    }
}

Test dive

If you would run above code, the result is like below image.

It's show what the index count be selected as it's each part of ratios. but there's some strange here. the index counts isn't exactly match with ratios.
It's because that is based on random function and it's not static calculation but also it's Real-Time counting to process continuously. Therefore some counts may have some difference from specified ratio.

Conclusion

We have been to talk about how to implement random selected index given ratios. I guess this could contributing that a service is needed Load-Balancing, For that, may have to be customized.
Always elastic thinking is yours!

Thanks.

How to manage your configuration file with YAML in Java programmatically

Kooin-Shin — Wed, 09 Dec 2020 02:53:01 +0000

Introduction

YAML is most useful data format to describe structural hierarchy in these days. It's more simple and collision resistant.
It is useful to describe your configuration attributes in your developments.
We are going to learn to manage configuration YAML in Java programmatically.
For our goal, we will use SnakeYAML library.

OK, Let's dive in.

Set up environment

At first We need to choose development tool. Even if any development IDE for Java could be used, but I would use Visual Studio Code that have been hotted on these days.
Setting dependency to build.gradle file. if you do like below, It's would be configured classpath automatically.

dependencies {
    // Use JUnit test framework
    testImplementation 'junit:junit:4.12'

    // https://mvnrepository.com/artifact/org.yaml/snakeyaml
    compile group: 'org.yaml', name: 'snakeyaml', version: '1.27'
}

All of environment setting is over doing this.

Let's go to code

First, we need to define configuration on config.yml file formatted with YAML.

adminPassword: 1234
adminPort: 9292
adminUser: admin
forbiddenRemote: []
host: localhost
sessions:
  Oracle:
    allowedHosts: [127.0.0.1]
    connectionTimeout: 3
    remoteHosts: [192.168.1.152:1521]
    retry: 3
  Kafka:
    allowedHosts: [127.0.0.1]
    connectionTimeout: 3
    remoteHosts: [192.168.1.153:2181]
    retry: 3

Second, we have to write Java Bean code mapping with config.yml fields. Top object of configuration is implemented to Config.java.

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class Config {
    private String adminPassword;
    private String adminUser;
    private String host;
    private int adminPort;
    private List<String> forbiddenRemote = new ArrayList<>();
    private Map<String, Sessions> sessions = new HashMap<>();

    Config() {}

    public String getAdminUser() {
        return adminUser;
    }
    public void setAdminUser(String adminUser) {
        this.adminUser = adminUser;
    }
    public String getAdminPassword() {
        return adminPassword;
    }
    public void setAdminPassword(String adminPassword) {
        this.adminPassword = adminPassword;
    }
    public int getAdminPort() {
        return adminPort;
    }
    public void setAdminPort(int adminPort) {
        this.adminPort = adminPort;
    }
    public String getHost() {
        return this.host;
    }
    public void setHost(String host) {
        this.host = host;
    }
    public List<String> getForbiddenRemote() {
        return forbiddenRemote;
    }
    public void setForbiddenRemote(List<String> forbiddenRemote) {
        this.forbiddenRemote = forbiddenRemote;
    }
    public  Map<String, Sessions> getSessions() {
        return sessions;
    }
    public void setSessions(Map<String, Sessions> sessions) {
        this.sessions = sessions;
    }
    public Sessions getSessions(String sessionName) {
        return this.sessions.get(sessionName);
    }

    @Override
    public String toString() {
        return "Config [adminUser=" + adminUser + ", adminPassword=" + adminPassword + ", host=" + host
                + ", adminPort=" + adminPort + ", forbiddenRemote=" + forbiddenRemote + ", sessions="
                + sessions + "]";
    }
}

Sub classification of Config object is Sessions object. It will be placed it's own to Config object's sessions field as Map element.

import java.util.List;

public class Sessions {
    List<String> allowedHosts;
    int connectionTimeout;
    List<String> remoteHosts;
    int retry;

    public Sessions() {
    }
        public List<String> getAllowedHosts() {
        return allowedHosts;
    }

    public void setAllowedHosts(List<String> allowedHosts) {
        this.allowedHosts = allowedHosts;
    }

    public int getConnectionTimeout() {
        return connectionTimeout;
    }

    public void setConnectionTimeout(int connectionTimeout) {
        this.connectionTimeout = connectionTimeout;
    }

    public List<String> getRemoteHosts() {
        return remoteHosts;
    }

    public void setRemoteHosts(List<String> remoteHosts) {
        this.remoteHosts = remoteHosts;
    }

    public int getRetry() {
        return retry;
    }

    public void setRetry(int retry) {
        this.retry = retry;
    }

    @Override
    public String toString() {
        return "Sessions [allowedHosts=" + allowedHosts + ", connectionTimeout=" + connectionTimeout + ", remoteHosts="
                + remoteHosts + ", retry=" + retry + "]";
    }
}

Now, we need to write Handler source file to load config.yml's contents to Java Beans objects we wrote before. To do this, we have to use SnakeYAML library and it's example down below.

Path configPath = Paths.get("./config.yml");
//At first, construct Constructor object using Config.class root object of contents.
Constructor constructor = new Constructor(Config.class);
//Construct Yaml object with constructor object.
Yaml yaml = new Yaml(constructor);
//And then load by given Stream object specified of config.yml file.
yaml.load(new FileInputStream(configPath.toFile()));

Also, If we want to dump contents of Config object in memory to file, we should write dumping code like below.

Path configPath = Paths.get("./config.yml");
DumperOptions options = new DumperOptions();
options.setDefaultFlowStyle(FlowStyle.BLOCK);
options.setPrettyFlow(true);        
Yaml yml = new Yaml(options);
yml.dump(this.config, new FileWriter(configPath.toFile()));

As setting detail options of DumperOptions object, we can handle YAML content for what we want.

We could write Handler to improve for our purpose, then will go to complete Handler code to ConfigHandler.java.
ConfigHandler object will be Singleton style for calling wherever we want in our application. and will have functionality of loading, dumping and referring Config object.
Complete ConfigHandler.java code is down below.

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;

import org.yaml.snakeyaml.DumperOptions;
import org.yaml.snakeyaml.DumperOptions.FlowStyle;
import org.yaml.snakeyaml.Yaml;
import org.yaml.snakeyaml.constructor.Constructor;

public class ConfigHandler {

    public static final Path configPath = Paths.get("./config.yml");

    private static ConfigHandler configHandler;

    Config config;

    /**
     * Get instance of ConfigHandler
     * @return
     * @throws FileNotFoundException 
     */
    public static ConfigHandler getInstance() throws FileNotFoundException {
        return getInstance(configPath);
    }

    /**
     * Get instance of ConfigHandler
     * @param configPath
     * @return
     * @throws FileNotFoundException
     */
    public static ConfigHandler getInstance(Path configPath) throws FileNotFoundException {
        if(configHandler == null) {
            configHandler = new ConfigHandler(configPath);
        }
        return configHandler;
    }

    /**
     * Constructor
     * @param configPath
     * @throws FileNotFoundException 
     */
    private ConfigHandler(Path configPath) throws FileNotFoundException {
        this.config = loadConfig(configPath);       
    }

    /**
     * Load config.yml
     * @param configPath
     * @throws FileNotFoundException
     */
    public Config loadConfig(Path configPath) throws FileNotFoundException {
        Constructor constructor = new Constructor(Config.class);
        Yaml yaml = new Yaml(constructor);
        return yaml.load(new FileInputStream(configPath.toFile()));
    }

    /**
     * Dump config to config.yml
     * @throws IllegalArgumentException
     * @throws IllegalAccessException
     * @throws IOException
     */
    public void dumpConfig() throws IllegalArgumentException, IllegalAccessException, IOException {
        dumpConfig(this.config, this.configPath);
    }

    /**
     * Dump config to config.yml
     * @param configPath
     * @throws IllegalArgumentException
     * @throws IllegalAccessException
     * @throws IOException
     */
    public void dumpConfig(Config config, Path configPath) throws IllegalArgumentException, IllegalAccessException, IOException {
        DumperOptions options = new DumperOptions();
        options.setDefaultFlowStyle(FlowStyle.BLOCK);
        options.setPrettyFlow(true);        
        Yaml yml = new Yaml(options);
        yml.dump(config, new FileWriter(configPath.toFile()));
    }

    /**
     * Get config object
     * @return
     */
    public Config getConfig() {
        return this.config;
    }   

    /**
     * Get session mapping object by session name
     * @param sessionName
     * @return
     */
    public Sessions getSessions(String sessionName) {
        return this.config.getSessions().get(sessionName);
    }

    public static void main(String[] args) throws IllegalArgumentException, IllegalAccessException, IOException, NoSuchFieldException, SecurityException {
        ConfigHandler handler = ConfigHandler.getInstance();
        Config config = handler.getConfig();
        System.out.println("ADMIN: "+config.getAdminUser());
        System.out.println("PASSWD: "+config.getAdminPassword());
        System.out.println("ORACLE: "+config.getSessions("Oracle").toString());     
        config.setAdminPassword("123456789");
        handler.dumpConfig();
    }
}

Conclusion

YAML is most simple and very fine format to manage contents.
Easy readability and Structural expression is proving why it's famous in these days.
For the more, we could use this format to generate huge data files of Java Bean data for Business or Analysis on Big-Data Store.
All of elastic thinking is yours.

Thanks.

Wildcard style search in Java Pattern

Kooin-Shin — Mon, 02 Nov 2020 08:51:35 +0000

Liquid syntax error: Unknown tag 'endraw'

How to create dynamic chart image on AWS S3 with ChaosGraph Library

Kooin-Shin — Tue, 27 Oct 2020 06:16:39 +0000

There is times when we are needed chart on a part of Web page or needed to save business analytic image to storage in daily or monthly. Also we wanna manage that images on AWS S3 for historical purpose.
At this point, ChaosGraph library is very useful tool for it.

We are going to review a use-case for these purpose using with AWS infrastructure e.g. API-Gateway, Lambda, S3...

A use-case what we do is down below.

Scenario

We need to create chart image to AWS S3 bucket requested from AWS API Gateway. To do this, API Gateway is received JSON as chart data values by REST request and AWS Lambda be triggered to build a chart image and create a chart image file at temporary directory, and than it will be putting the chart image file to S3 bucket.

Now let's dive in.

We are going to make JSON descriptor file that describes configuration and numerical data for target chart. See down below.

{
  "GRAPH" : "LINE",
  "INTERPOLATE" : "SPLINE",
  "WIDTH" : 1600,
  "HEIGHT" : 900,
  "XINDEX" : ["0", "", "2", "", "3", "", "4", "", "5", "", "6", "", "7", "", "8", "", "9", "", "10"],
  "YINDEX" : [50, 80, 500],
  "ELEMENTS" : [
      {
        "ELEMENT" : "Kafka",
        "LABEL" : "Kafka usage",
        "COLOR" : [180,130,130],
        "VALUES" : [44, 35 , 0, 32, 0, 33, 29, 43, 25, 22, 32, 43, 23]
      },
      {
        "ELEMENT" : "elastic search",
        "LABEL" : "elastic search usage",
        "COLOR" : [180,130,130],
        "VALUES" : [43,25,10,32,0,23,52,32,32,23,54,23,48, 20, 60, 140, 500, 10]
      },
      {
        "ELEMENT" : "Oracle",
        "LABEL" : "Oracle usage",
        "COLOR" : [180,180,140],
        "VALUES" : [500,93,0,49,0,24,93,63,92,84,69,46,28]
      },
      {
        "ELEMENT" : "S3",
        "LABEL" : "S3 usage",
        "COLOR" : [150,200,158],
        "VALUES" : [20,36,0,24,22,37,33,54,23,48,53,150,22]
      }
    ]
}

As you can see, this JSON descriptor defined with type of chart, elements attributes, chart's size and interpolation type.

We define this and complete Lambda function. we will deploy Lambda function and configure it's Role and permission. Also API Gateway, S3 as well. And then we request to AWS API Gateway with this JSON via REST PUT method.

Step 1 - Download library

Download ChaosGraph libray at Github. Visit to https://github.com/9ins/chaosgraph
Add the library into your project classpath

Or Set dependency to your build tool configuration file.

Maven dependency

<!-- https://mvnrepository.com/artifact/io.github.9ins/chaosgraph -->
<dependency>
    <groupId>io.github.9ins</groupId>
    <artifactId>chaosgraph</artifactId>
    <version>2.0.0</version>
</dependency>

Gradle dependency

// https://mvnrepository.com/artifact/io.github.9ins/chaosgraph
implementation group: 'io.github.9ins', name: 'chaosgraph', version: '2.0.0'

Step 2 - How to configure IDE tool

To write Lambda function, You need IDE like Eclipse or Visual Studio Code and those be to support build tool like Maven, Gradle etc. For experienced tool of mine, Will use Eclipse and Gradle.

Dependency setting

apply plugin: 'java'
apply plugin: 'eclipse'

sourceCompatibility = 1.8
version = '1.0.0'
jar {
    manifest {
        attributes 'Implementation-Title': 'aws-lambda-chart',
                   'Implementation-Version': version
    }
}

compileJava.options.encoding = 'utf-8'

tasks.withType(JavaCompile) {
    options.encoding = 'utf-8'
}

repositories {
    jcenter()
    mavenLocal()  
    mavenCentral()
    flatDir {
        dirs 'libs'
    }
}

dependencies {  
    // Define AWS SDK dependencies
    compile group: 'com.amazonaws', name: 'aws-java-sdk-s3', version: '1.11.883'
    compile group: 'com.amazonaws', name: 'aws-lambda-java-core', version: '1.2.1'
    compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.11'

    // Define ChaosGraph dependencies, downloaded before
    compile files('libs/chaosgraph-2.0.0.jar')
}

jar {
    doFirst {
        from {
            configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
        }
    }
    exclude 'META-INF/*.RSA', 'META-INF/*.SF','META-INF/*.DSA'
}

As you can see, Upper Gradle build file defines dependencies of AWS SDK and ChaosGraph.

Step 3 - How to code Lambda function

Open your editor at IDE, Create Java class implemented Lambda 'RequestHandler'. It's named 'ChartGeneratorHandler'. Completed code like down below.

package org.chaostocosmos.aws.lambda;

import java.awt.image.BufferedImage;
import java.io.File;
import java.util.Date;
import java.util.Map;

import org.apache.commons.lang3.exception.ExceptionUtils;
import org.chaostocosmos.chaosgraph.Graph;
import org.chaostocosmos.chaosgraph.GraphUtility;
import org.chaostocosmos.chaosgraph.GraphUtility.CODEC;
import org.joda.time.LocalDateTime;

import com.amazonaws.regions.Regions;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;

public class ChartGeneratorHandler implements RequestHandler<Map<String, Object>, String> {

    Regions clientRegion = Regions.AP_NORTHEAST_2;
    AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(clientRegion).build();
    String bucketName = "line.chart.image";

    @Override
    public String handleRequest(Map<String, Object> event, Context context) {
        String response = null;
        LambdaLogger logger = context.getLogger();

        try {
            //Extract body from event
            String requestJson = event.get("body")+"";

            //Create graph object from requested JSON to make chart image.
            Graph graph = GraphUtility.createGraphWithJson(requestJson);
            //Get buffered image from chart.
            BufferedImage bi = graph.getBufferedImage();

            //Save chart image to file in temp directory.
            String fileObjectKey = "line-chart-"+LocalDateTime.now().toString()+".png";
            File imageFile = new File("/tmp/"+fileObjectKey);
            GraphUtility.saveBufferedImage(bi, imageFile, CODEC.PNG);

            // Upload a png image as a new object with ContentType and title specified.
            PutObjectRequest request = new PutObjectRequest(bucketName, fileObjectKey, imageFile);
            ObjectMetadata metadata = new ObjectMetadata();
            metadata.setContentType("image/png");
            metadata.addUserMetadata("title", "line chart");
            request.setMetadata(metadata);
            s3Client.putObject(request);
            response  = new String("{\r\n" + 
                    "  \"statusCode\": 200,\r\n" + 
                    "  \"headers\": {\r\n" + 
                    "    \"Content-Type\": \"application/json\"\r\n" + 
                    "  },\r\n" + 
                    "  \"body\": {\r\n" + 
                    "    \"message\" : \"success\"\r\n" + 
                    "  },\r\n" + 
                    "  \"isBase64Encoded\": false\r\n" + 
                    "}") ;

        } catch (Exception ex) {
            response =  new String("Error:"+ex.getMessage());
            response += ExceptionUtils.getStackTrace(ex);
            logger.log(response);
        }
        return response;
    }
}

And you should export that class to fat jar(included all library dependency) by running Gradle build.

Step 4 - IAM Role creating for the chart image service

Because of handling of API Gateway/Lambda/S3, you have to create a Role to manage Lambda/API Gateway/S3 in IAM service. To do that, you should sign in AWS console and create Role for API Gateway, Lambda and S3.
Created Role like down below.

For the convenience, setting permissions of API Gateway/Lambda/S3 to FullAccess.
Don't forget to ensure that 'Trust Relationship' include above three service.

For detail setting to Lambda and API Gateway with Role, See AWS Documentation.

Step 5 - Create Lambda function, Upload jar and Configurations

To deploy your artifact on Lambda, First, you have to create Lambda function and set Role that was created previous step.

Second, Upload your fat jar file to Lambda code section as choosing 'Action' under 'Function code' group on Lambda IDE.

Third, Attach API Gateway on Lambda trigger and then set proper API Gateway configuration and Role.

Util this we done, we can test by configuring Lambda test event menu that be right-upper side of Lambda IDE.
How to try that and Detail configurations of AWS Service, See AWS Documentation.

step 6 - Test with API Gateway

And Now we are going to API Gateway and we can request REST method. Just click 'Test' button.

If all things are fine, You can see like below.

And make sure that line chart image be created on S3.

See created line chart image on browser.

You can try to change JSON descriptor attributes for the more good visual.

To show more values, added more x axis indexes.

For more the technical challenge, You can add or modify JSON descriptor to include more attribute for dynamic management of service. e.g. Adding S3 bucket name, Adding S3 key prefix...

...
      },
      {
        "ELEMENT" : "S3",
        "LABEL" : "S3 usage",
        "COLOR" : [150,200,158],
        "VALUES" : [20,36,0,24,22,37,33,54,23,48,53,150,22, 300, 400, 600, 800]
      }
    ],
    {"S3-BUCKET" : "line.chart.image"},
    {"OBJECT-KEY_PREFIX" : "line_chart_image"}
}

Conclusion

AWS provide flexible and dynamic services based on it's durable and scalable infrastructures. e.g. Lambda, API Gateway, S3 etc.
ChaosGraph is effective and powerful library to express a chart in Java 2D Graphics.
Thus we were learned how to create dynamic chart image generation for user experience.

This, example could help you to make Web UI/UX or analytic image archive storage... Anything else you wanna place it where you are needed at.

If you are more curious about ChaosGraph, visit Github wiki page : https://github.com/9ins/chaosgraph/wiki or notice to me.

How to develop a chart using ChaosGraph on AWT/SWT based application.

Kooin-Shin — Mon, 19 Oct 2020 03:07:08 +0000

Instroduction

You could need to make chart based application on pure java GUI, Use this library. Also when you need to build dashboard on non-web-based java application, Just use Chaosgraph library. Maybe when if you are working with Eclipse RCP to make analysis app or CRM, MIS, This library could successful choice for your project's goal as UX to users. This library's core is java. Nevertheless, if you use Eclipse RAP, you can make a chart on Web Browser.

Github: https://github.com/9ins/chaosgraph/

Download jar: https://github.com/9ins/chaosgraph/tree/master/build/libs/chaosgraph-2.0.0.jar

Maven dependency

<!-- https://mvnrepository.com/artifact/io.github.9ins/chaosgraph -->
<dependency>
    <groupId>io.github.9ins</groupId>
    <artifactId>chaosgraph</artifactId>
    <version>2.0.0</version>
</dependency>

Gradle dependency

// https://mvnrepository.com/artifact/io.github.9ins/chaosgraph
implementation group: 'io.github.9ins', name: 'chaosgraph', version: '2.0.0'

Examples AWT/SWT

How to code

To make a chart on your App, follow below steps.
If you visit the Github page, Can get more information. e.g. Examples of a chart.

You need JRE over 1.8 or more.
Download Jar file on above link.
Add [GRAPH_HOME]/build/libs/chaosgraph-2.0.0.jar to classpath on your project.
Create code defining indexes of x, y axis.

    List<Object> xIndex = new ArrayList<Object>();  
    List<Double> yIndex = new ArrayList<Double>();  
    xIndex.add("Jan");  
    xIndex.add("Feb");  
    xIndex.add("Mar");  
    xIndex.add("Apr");  
    xIndex.add("May");  
    xIndex.add("Jun");  
    xIndex.add("Jul");  
    xIndex.add("Aug");  
    xIndex.add("Seb");  
    xIndex.add("Oct");  
    xIndex.add("Nov");  
    xIndex.add("Dec");  
    yIndex.add(1024d*1024d*100d);  
    yIndex.add(1024d*1024d*500d);  
    yIndex.add(1024d*1024d*1000d);  
    yIndex.add(1024d*1024d*5000d);

Create GraphElements object with x, y indexes.

   GraphElements graphElements = new GraphElements(GRAPH.AREA, xIndex, yIndex);

Add GraphElement objects being created to GraphElements object created above.

    double[] values = {1024d*1024d*45d, 1024d*1024d*55d, 1024d*1024d*3d, 1024d*1024d*66d, 1024d*1024d*33d, 1024d*1024d*6d, 1024d*1024d*9d, 1024d*1024d*600d, 1024d*1024d* 500d, 1024d*1024d*980d, 1024d*1024d*103d, 1024d*1024d*666d}  
    GraphElement ge = new GraphElement("Kafka", Color.GRAY, values);  
    graphElements.add(ge);  
    .... (could be added more elements)

Create GraphPanel object with pre-created GraphElements object.

    //Area graph type with width 600 pixel, height 400 pixel  
    GraphPanel gpArea = new GraphPanel(GRAPH.AREA, graphElements, 600, 400);

To setting proper or wanted option at Graph object retrived from GraphPanel object.

    AbstractGraph graph = (AreaGraph)gpArea.getGraph();
    graph.setTitle("This is simple area graph.");
    graph.setShowShadow(false);
    graph.setGridStyle(GRID.DOT);
    graph.setLimit(1000);
    graph.setPopupStyle(POPUP_STYLE.ROUND);
    graph.setSelectionEnable(true);
    graph.setSelectionBorder(SELECTION_BORDER.DOT);
    graph.setShowGraphXY(false);

Add GraphPanel object to be required target UI component.

    getContentPane().add(gpArea , BorderLayout.CENTER);
    getContentPane().validate();
    gpArea .repaint();

For more information, visit Github wiki : https://github.com/9ins/chaosgraph/wiki

Sample images

Area chart
Line chart
Line chart applied with interpolation
Bar chart
Bar Ratio chart
Circle chart