DEV Community: koba-yu

Convert legacy Japanese encoding by Red

koba-yu — Sat, 06 Aug 2022 07:42:18 +0000

In this post, I talk about coding with Red programming language. The repo is here. Red has great feature to parse and convert string/binary data. I hope this post helps you understand Red's ability!

Encoding is a curse in Japan

This paragraph does not explain codings but the Japanese cultural context. Skip if you are not interested.

Japanese is probably one of the most suffering people from encoding trouble in the world. We have 3 traditional character types - hiragana, katakana, and kanji - and even unique symbols like ♨(hot spring) and emojis! Especially kanji has too many characters. For example, 辺, 邉, and 邊 have the semantically same meaning but we need to select the character correctly when it is used as surname kanji, especially in a formal context like weddings. It depends on which character was chosen when his/her ancestor registered his/her surname to the government in the Meiji era(around 1900). Anyway, we needed custom Japanese encoding until Unicode was established. In those days, Japanese people mainly used encoding called "ShiftJIS" and many legacy systems still output their files by this encoding. To make matters worse, Microsoft Excel in Japan can open the file as UTF-8 only when it has BOM. Without BOM, its default choice is "ShiftJIS" (and there are fewer chances the file has BOM...). If you open a UTF-8 file without BOM by Japanese Excel, it cannot show characters correctly. This has been continuing to spread confusion for people unfamiliar with IT, and many people still think "System should be able to output a ShiftJIS file". These are why even nowadays Japanese programmers (like me) need to deal with "ShiftJIS" encoding files.

Let's convert ShiftJIS to UTF-8 by Red

Motivation

I often use Red programming language for my data processing. Though Red is a fantastic language, it is young and does not support easy encoding transformation at least now. Of course, most popular languages can read/convert a ShiftJIS file to UTF-8 so I can handle encoding by using those languages. However, it is more useful for me if I can convert only by Red, without other language. Therefore I tried to implement the code. This post explains how to code it.

About codes

The code I made is here. I explain what I did and some points of the code in this post.

1st Step, make a map for byte mapping (by C#)

Unfortunately, there is no logical codepoint conversion rule between ShiftJIS and UTF-8. Therefore I had to get a byte map, from 1 Shift-JIS character bytes to corresponding UTF-8 character bytes. However, I could not find any reliable comparison table on the internet. So I wrote C# code to make it. You can run this code by LINQPad app, a very famous C# scratch pad. It is free for basic features. This code loops through all of the possible int values from the beginning of ShiftJIS codepoint to the end, gets a char instance from the int value and then gets a byte array from the char. You think it is easy, don't you? But there are a lot of "pitfalls" - ShiftJIS has "undefined" codepoints everywhere! So I have to skip them but there is no logical way to judge whether a certain codepoint is "defined" or not. I have to hard-code specific codepoints to skip...here is a list of the value ranges where I had to skip. It was too many to type😂. shift-jis-utf8-bytes.txt is the resulting output of the code. In this file, a pair of the byte expressions are in each line, separated by |. The left part is a hexadecimal ShiftJIS byte expression and the right is the corresponding UTF-8 one. Some of the lines are listed below as an example.

80|C280
8150|EFBFA3
8151|EFBCBF
8152|E383BD
8153|E383BE
  ・
  ・
  ・
EAA0|E6A787
EAA1|E98199
EAA2|E791A4
EAA3|E5879C
EAA4|E78699

2nd Step, make Red map! value as a conversion table.

Now I have got the byte table so I can process the table to convert it to Red’s map! value. This is very easy. The code is here. The step is just;

Split each line by |
Convert hexadecimal string to binary! value by dabase/base xxxx 16

The resulting file is bytemap.red.

Final Step, parse and replace binary value for converting.

Finally, I can write actual code to convert. shiftjis-to-utf8.red is the final file. The code is really short and neat thanks to Red's parse feature.

In line 11, I include the bytemap.red file that I created in the previous step. Then I make a rule block special-bytes to judge whether the binaries are ShiftJIS-defined bytes or not. If not, the bytes must be inside the ASCII code range. ShiftJIS has the same codepoints up to ASCII's code end as UTF-8 does. Special-bytes block looks like below;

[
    #{80} | 
    #{8150} | 
    #{8151} | 
    #{8152} | 
    #{8153} | 
    #{8154} |
    　・
    　・
    　・
    #{EAA0} | 
    #{EAA1} | 
    #{EAA2} | 
    #{EAA3} | 
    #{EAA4}]
]

This line is the part processing ShiftJIS bytes. If current bytes match any of the special-bytes, the code (select bytemap sb) is executed and ShiftJIS bytes are replaced with UTF-8 ones as a result. If the bytes do not match as ShiftJIS bytes, this line is executed and the code (make binary! reduce [ascii]) just keep the byte without changing since they are ASCII bytes.

In the last line, I append binary! value #{EFBBBF} to the head of the converted binary. This is BOM. Because I am poor Japanese, I need to add this to let Excel open the file correctly...😭

How is the code? It is very clear!

I love the code style of Red. Binary processing code tends to be complicated in many languages but I could write it as very declarative and readable code in Red. If you get interested in Red or have any questions about the code, feel free to ask!

My Red Story #2 Red to interact with WEB API 2/2

koba-yu — Sat, 02 Jan 2021 06:46:03 +0000

Introduction

In my previous post, I describe Red's good points, mainly focused on non-functional aspects. In this post, I am talking about actual coding and some practices about using WEB API by Red.

Let's use JSON Placeholder's WEB APIs as example codes, that can be accessible for everyone.

Calling Get API

The 1st example is just to get the "posts" data.

read https://jsonplaceholder.typicode.com/posts

Ultimately easy! You have to do no package-installing, import or anything like that before running the code above. To look at the result, put print or probe before the code and execute it GUI console.

Calling with request headers

If you need to set a request header, you can use write function.

write https://jsonplaceholder.typicode.com/posts [
    get [Content-type: "application/json; charset=UTF-8"]
]

The 2nd argument is a block that contains a http method name(like get, post) and an inner block that contains the header's key/value.

Calling Post API with request body

When you have to set a request body value, you can add it as a JSON value after the header.

json: to-json #(
    title: "foo"
    body: "bar"
    userId: 1
)

write https://jsonplaceholder.typicode.com/posts compose [
    post [Content-type: "application/json; charset=UTF-8"]
    (json)
]

Here is what the code does. to-json converts Red value to JSON string and #( ) is a Red's map! type literal value to be converted. Then you can pass the JSON next to the request header. compose function is one of the Red's elegances. This function evaluates only inner words of (). Therefore after compose executed, the code line of calling write becomes below;

write https://jsonplaceholder.typicode.com/posts [
    post [Content-type: "application/json; charset=UTF-8"]
    {{"title":"foo","body":"bar","userId":1}}
]

If you want the code shorter, you can do like below;

write https://jsonplaceholder.typicode.com/posts compose [
    post [Content-type: "application/json; charset=UTF-8"]
    (
        to-json #(
            title: "foo"
            body: "bar"
            userId: 1
        )
    )
]

Save response at any time

When you are making a program to interact with the WEB API of the other company, you might feel like to save the API response, because the response is not always stable. For example, WEB API often limits the number of calls during certain minutes and returns a 429 response when your API calls exceed it. In JSON Placeholder, the remaining count to call is set X-Ratelimit-Remaining header and if you exceed it, you have to wait until the time indicated by X-Ratelimit-Reset header. While you are writing code to handle this restriction properly, it is helpful to save API responses once you get it, instead of repeatable calling API until it gets over the limit. To get response header, you can use read/info or write/info. Here is a 'read' example;

r: read/info https://jsonplaceholder.typicode.com/posts/1

save %response.red r

This code saves API response as a "response.red" file at the folder Red binary exists. The saved data is like this;

I also put the saved data on my GitHub Gist

This is a really understandable format. The 1st is the HTTP response code, the 2nd is the header represented by map! data type and 3rd is response body as a string. By the way, if you do not prefer for the 3rd value to be JSON, you can convert it to a Red value.

r: read/info https://jsonplaceholder.typicode.com/posts/1
r/3: load-json r/3
save %response-2.red r

And here is the value to be saved: GitHub Gist

So, you have saved response as a file, and you can restore it simply by doing this;

r: load %response.red

This set the same value of the API response to a word r (in Red there is no variables but words). That's why you can resume your coding immediately even after you restart GUI console, without calling API again. When you want to test your code of handling a special condition, like when the limit number is 1 or 0 and you have to wait until the limitation is reset, just rewrite the value X-Ratelimit-Reset in "response.red" and load it. Then you can run the handling code and check the code behaves as you expect. As another choice, you can save 429 response after intentionally occurring it once.(Unfortunately, JSON placeholder's API seems not to return 429 response so I can not write an example data here). In any case, you are freed from calling API so many times.

Seems great? Try Red!

I hope I could introduce the Red's attractiveness to you. If you want to check a more practical code with interacting WEB API, here is my GitHub repo. This code saves data from ChatWork(a popular business chat service in Japan) to local disk. I am still not an expert Red coder so any advice to the repo is welcome. Feel free to put a message on Issues, Discussions of the repo if you have something to talk with me.

My Red Story #1 Red to interact with WEB API 1/2

koba-yu — Sat, 02 Jan 2021 06:40:02 +0000

Make a tool with Red-lang, it's fun!

This year I have released a simple tool coded by Red-lang. It saves ChatWork's messages and files to local. ChatWork is a chat service for business, very popular in Japan. Because my company stopped using ChatWork, I wanted to save my data in ChatWork. That was the motivation.

Repository

Here is my repository for the tool, ChatWorkKeeper. Sorry the description in the repo is in Japanese only, but most people who need this tool are Japanese. 😅

Why using Red?

This post mainly explains non-functional aspects. If you are interested in only codes, take a look at my next post!

Red is an alpha stage programming language, not so many production use cases yet(maybe). However it already has amazing capabilities and is comfortable to code. Following points are benefits with coding Red, I think.

REPL included, official VS Code plugin is awesome
Simple and immersive coding, not requiring powerful PC
Typing comfortability - less times to press Shift key and symbol keys
Compiled to single binary - easy to share and no worries about runtime installed
Interacting with WEB API is very easy (written in next post)

1. REPL included, official VS Code plugin is awesome

Red has a REPL called GUI console. It is very comfortable especially to code a program interacting with WEB APIs of other companies. Since I am not sure how the API returns response, GUI console gives an efficient process to write a code, run it and see the results. Also Red has an official VS Code pulgin. It's very handy.

2. Simple and immersive coding, not requiring powerful PC

Red is a very simple language and does not force you to consider much other than coding. I have used C# in my business for 10 years. Though I like and thank C#, I need to consider many things before I start coding when use it. For example, what version of .NET I use (and is the version available for all of the users I expected?). Which types of project I use?(Console app or WPF app? The project files' structure decided by the type). In Red it is a really easy choice whether my app has GUI or not. All I need is to use view function. It does not shake project level structure at all. With Red, I can write codes that directly match with what I am thinking, at any time. It helps me motivate myself a lot to keep coding at my leisure time. Also, Red and its editor do not require quite high resources, I can develop my app on my modest spec laptop.

3. Typing comfortability - less times to press Shift key and symbol keys

If you took a glance at my repo above, you might find there are fewer symbols in the source code. We are too used to seeing many symbols in traditional programming languages to have doubts about that. But is it really readable using ? and ?? to conditional operators? Isn't it uncomfortable to type () every place method called? (it means, most lines of the codebase in practice!) After meeting Red, I have gotten too lazy to type () because it needs to press the Shift key (and ( is a little far from Shift!).

Let's look at an example, JavaScript's code below;

var result = 'Hello, 2020!'.replace('2020', '2021');
console.log(result);
// => Hello, 2021!

would be in Red,

result: replace "Hello, 2020!" "2020" "2021"
print result
; => Hello, 2021!

Red is much neater and more readable, isn't it?

Or if you want to write in one line,

console.log('Hello, 2020!'.replace('2020', '2021'));

would be

print replace "Hello, 2020!" "2020" "2021"

Probably, it seems possible even for a non-programmer person to read and guess what Red's code means. Great readability!

4. Compiled to single binary - easy to share and no worries about runtime installed

While I can use the GUI console when stacking codes, I can compile the Red code after coding done. It makes a single machine code binary file so really easy to share it. Because it's machine code binary, UPX can be used to compress the file. Final compressed size of my tool is only 369KB!

5. Calling WEB API and handling response are very easy

This post has been a little long, so I have written one more post about this topic. Let's focus on the actual code next!