DEV Community: Dviejo

Character set walkthrough

Dviejo — Sat, 23 Nov 2019 15:11:03 +0000

How do we write to a file a character like Ñ and the word привет? (hello in Russian: privet). We have the idea that to represent a character we need a byte, but this has not been so for many years.

At the programming level, this currently represents one of the biggest problems when we have to make an application that contemplates several languages, with different character sets.

This forces us to when we create a file we have to express the character set with which it is made. Whoever receives the file should know which character set it's been written to be able to read it properly.

And in the databases, in which character set are the data recorded? When the database is created, database managers should usually express what character set they want to work with. If our database must store information in Russian and Spanish, we must use a character set that allows both.
If we open the notepad ++, we write a Ñ and record as test.txt and do type test.txt we see

but if we do it with the Powershell

We have written a Ñ and we have two different representations. This already poses a problem. One thing is what is in the file, which is a Ñ and another thing is how the programs interpret it, in this case, the cmd and PowerShell do it differently.

If we read it with a program, for example in python,

what is happening, in python, is that when you open the file we are not telling you that the character set is utf-8

But why have we had to say utf-8 and not something else? In what character set has the notepad ++ recorded the file? The notepad, if we say nothing else, records the file in utf-8. When we tell the python that the file is in utf-8 we see the Ñ.

What programmers do when we see "weird characters", we say, "that is a problem of the character set," but we don't really know very well what has happened. "

One lesson to learn is that when we open a file or record a file we should specify the character set with which it is recorded.

But why does it come out
as a representation of the Ñ in the cmd? Well, when we open a cmd it starts with a character set that is not the utf-8. If we run chcp ithit does notmy machine:

Active code page: 850. If we see what code 850 is like in the following url

We will see that the first character is from row C column 3 and the second is from row 9 col 1. Therefore, refer to the C391 characters.

If we change the character set to utf8 with the command

chcp 65001

and type type.txt then it appears to us

The CMD program has correctly interpreted the character set of the file.

We have to keep in mind that sometimes the problem will not be in the file, but in the program that interprets the file.

If we open the file with the notepad.exe, it interprets it properly and the Ñ appears

What bytes have actually been recorded in the file? If we look at the content in hexadecimal with the powershell

Format-Hex test.txt

And we have:

The Ñ is saved using two bytes C3 91

If we see the utf-8 specification, we see that the Ñ is represented with two bytes (https://www.utf8-chartable.de/)

The UTF-8 IS a multibyte character encoding, which allows the exchange between different systems in a secure way.

The first 128 characters are serialized in the same way, from 128 they are serialized with 2 bytes. There are characters that are serialized with 3 4 5 and 6 bytes.

This article is the first in a series dedicated to character sets.

Loading data into Postgresql using Nodejs and Minio

Dviejo — Tue, 12 Mar 2019 22:31:51 +0000

I was recently loading a lot of files into PostgreSQL, and i had the need of doing it without many resources and to be fast.

The requirements I had were the following:

Low footprint, not reading all lines of a file at once
Easy to load data
Write data efficiently into Postgres using "COPY TO" sentence

With these requirements, I came up with the idea of doing with streams in NodeJS because:

It's efficient
It's simple

Let's say I have the following file

first_name, last_name, country
Karie,Zorn,Ukraine
Thebault,Flicker,France
Danya,Atcock,Canada
Lorne,Westmorland,Russia
Page,Greeve,Canada
Rene,Riccardini,Serbia
Adair,Mullin,Philippines
Della,Gumb,China
Charlie,Swadlinge,Cambodia
Celka,Karlowicz,Canada

And the following table in PostgreSQL:

CREATE EXTENSION "uuid-ossp";   
CREATE TABLE PERSON(
    id uuid default uuid_generate_v4(),
    first_name varchar(30) NOT NULL,
    last_name varchar(30) NOT NULL,
    country varchar(30) NOT NULL
);

As we're going to use curl in the PostgreSQL sentence in order to get the file, we're going to extend the Postgresql Docker image with the following:

FROM postgres:10.4
RUN apt-get update && apt-get install -y curl

I named this file postgresql.Dockerfile, when you have that file, we're going to get all running with the following commands:

docker build -f postgresql.Dockerfile -t postgres-with-curl .
docker run --restart always --name postgres --network host -d postgres-with-curl
docker run --network host -e MINIO_ACCESS_KEY=user -e MINIO_SECRET_KEY=password --restart  always --name minio -d minio/minio:RELEASE.2018-06-22T23-48-46Z server /data

So let's do it, first we need to create the connections for PostgreSQL and Minio:

const pg = require("pg")
const minio = require("minio")

const minioClient = new minio.Client({
    accessKey: "user",
    secretKey: "password",
    secure: false,
    endPoint: "localhost",
    port: 9000,
    region: "us-east-1",
})
const pgPool = new pg.Pool({
    user: "postgres",
    password: "postgres",
    host: "localhost",
    port: 5432,
    database: "postgres",
})

Then we're going to do basically 3 things:

Read the file
Upload to minio
Execute SQL in PostgreSQL instance

const fs = require("fs")
const path = require("path")

const TABLE_NAME = "public.people"

async function main() {
    const input = fs.createReadStream(path.join(__dirname, "people.csv"), {
        encoding: "utf-8",
    })
    const bucketName = "tmp"
    const objectName = "new.csv"
    await minioClient.putObject(bucketName, objectName, input)
    const csvUrl = await minioClient.presignedGetObject(
        bucketName,
        objectName,
        10000, // duration in seconds of the url
    )
    const copyDataSql = `
  COPY ${TABLE_NAME} (FIRST_NAME, LAST_NAME, COUNTRY)
    FROM PROGRAM 'curl "${csvUrl}"'
    WITH (FORMAT CSV , HEADER, QUOTE '"', DELIMITER ',' );`
    const client = await pgPool.connect()
    await client.query(copyDataSql)
    await client.release()
    await pgPool.end()
}

main()

If we execute the following code in nodejs we should see the rows in the database with the following query:

SELECT * FROM PUBLIC.PEOPLE;

The script can fail because of the following reasons:

Minio is not up
Postgresql is not up
Postgresql table has not been created
Minio bucket "tmp" has not been created

This technique can be useful when we have no control over the disk in postgres and we have to perform a batch operation.

Thanks for reading.

Building SDKs for your SAAS

Dviejo — Tue, 13 Nov 2018 02:25:42 +0000

Building SDKs for your SAAS

I recently build a service, docxmerge, which does a merge from a Microsoft Word and JSON document and converts it to PDF.

The API is very simple, but in order to make easy for developers to integrate it in their applications, I chose to create a library for at least four languages, NodeJS, Dotnet, Java, and Python.

Yeah, it looks like we got a bit of work. This task can be either too easy or too embarrassing. But I had a benefit, which was to use Swagger for the API definition, so basically, I could generate all the boilerplate code and just do a class in every language and publish it to the corresponding registry.

With Swagger, all is based on the JSON generated in the backend in this case the next one:

https://api.docxmerge.com/api/swagger/v1/swagger.json

Having that JSON, the swagger codegen is able to generate the code to interact with your api in a typed way and with no effort at all. I generate that code with docker which is much cleaner:

docker run --rm --network=host -v ${PWD}/src/swagger:/local swaggerapi/swagger-codegen-cli:v2.3.1 generate -i ${SWAGGER_URL} -l typescript-node -o /local --additional-properties supportsES6=true

As a said, Swagger was the biggest advantage, as it was effort free and I didn't have to worry about paths, query strings, multipart requests, etc. And lucky me, I was able to generate it from C# annotations using this Swashbucle library , so didn't have to write that mega JSON ;)

I will do one post per language explaining how I built each library from the swagger-codegen to the publish.

This is the repository where I'm working:
https://github.com/Docxmerge/docxmerge-sdk

It's still in progress, and I'm still missing the Java one to be deployed because of the verification, but it works.

Haproxy multiple certificates over single IP using SNI.

Dviejo — Wed, 04 Jul 2018 19:06:24 +0000

Haproxy multiple certificates over single IP using SNI

Hello!, I'm a fullstack/devops developer who is going to start sharing solutions to problems around. Over the last two years i have specialized on Kubernetes/Docker, NodeJS, Java and Angular/React. I will be writing about software solutions in the next posts.

I had a problem with routing in haproxy and i didn't know why, i had the haproxy doing routing with SNI(Server name indication), which is an extension of the TLS protocol which is useful in order to get the hostname at the start of the handshake, so i could do load balancing over multiple backends without doing handshake. But i found that if you do routing with SNI and present the same certificate in multiple hostname you won't always get the same result.

To see the things clearly let's present an example.

A configuration of haproxy with SNI routing could be like this:

global
  log /dev/log    local0
  log /dev/log    local1 notice
  stats socket /haproxy-admin.sock mode 660 level admin
  stats timeout 30s
  daemon
defaults
  maxconn 1000
  mode http        
  log global
  option dontlognull # bind *:443 ssl crt .
  timeout http-request 5s
  timeout connect 5000
  timeout client 2000000 # ddos protection
  timeout server 2000000 # stick-table type ip size 100k expire 30s store conn_cur

frontend env_ssl_frontend
  bind *:443
  mode tcp
  option tcplog
  tcp-request inspect-delay 10s
  tcp-request content accept if { req_ssl_hello_type 1 }
  use_backend bk_app1 if { req.ssl_sni -m end app1.domain.com }
  use_backend bk_app2 if { req.ssl_sni -m end app2.domain.com }
  use_backend bk_app3 if { req.ssl_sni -m end app3.domain.com }

backend bk_app1
  balance source
  mode tcp
  option ssl-hello-chk
  server main 127.0.0.1:3000
backend bk_app2
  balance source
  mode tcp
  option ssl-hello-chk
  server main 127.0.0.1:3001
backend bk_app3
  balance source
  mode tcp
  option ssl-hello-chk
  server main 127.0.0.1:3002

The frontend looks pretty clear, we have it listening on 443 on TCP mode (L4) and routing to bk_app1, app2 or app3, depending on the server_name coming from the Client Hello, and then, sends the traffic to the corresponding App.

In the backend server you will have to terminate the handshake, if not, it will not work.

You can test this easily without having to install haproxy by having Docker installed having a file named haproxy.cfg with the previous config:

docker run -p 443:443 --name haproxy -v "${PWD}":/usr/local/etc/haproxy:ro haproxy:1.8

If you want to see which server_name is being sended, you can use Wireshark

With this configuration, if you present the same certificate in bk_app1 and bk_app2, you will be redirected based on the current handshake. So, you access app1.domain.com, then access app2.domain.com, then app2.domain.com will routed to bk_app1.

So always that you do routing over SNI, you have to do redirecto to the same backend per certificate!

I hope it helped!

I'm planning on writing about microservices, CI/CD and Kubernetes. So if you want me to talk about any themes feel free to ask!