DEV Community: Yasuhiro Matsumoto

Contributed to some OSSs with pull-requests in this year too.

Yasuhiro Matsumoto — Thu, 07 Oct 2021 16:13:10 +0000

Hacktoberfest 2021

In this year too, I contributed to few OSSs. As far as I remember, I joined this event from 2019. 🎉

Contributions

souvikinator/gofuzz

https://github.com/souvikinator/gofuzz/pull/5

This pull-request add go-colorable (This is my package) to enable colors on Windows.

ArkScript-lang/Ark

https://github.com/ArkScript-lang/Ark/pull/318

This pull-request fixes possibly memory overflow.

https://github.com/ArkScript-lang/Ark/pull/319

This pull-request fixes compiler warnings.

fsouza/fake-gcs-server

https://github.com/fsouza/fake-gcs-server/pull/595

Fix to use path/filepath package instead of path package.

Conclusion

Shut the fuck up and write some code. 🤘

I made 4 pull-requests

Yasuhiro Matsumoto — Fri, 02 Oct 2020 05:13:35 +0000

elsaland/elsa

https://github.com/elsaland/elsa/pull/54

Fixed colors with escape sequences on Windows. 💪

mattn/longcat

https://github.com/mattn/longcat/pull/36

This is my project. Added -v to show version. 💪

federico-terzi/espanso

https://github.com/federico-terzi/espanso/pull/476

Fixed build.rs to be possible to build espanso on Windows Rust gcc toolchain. 💪

gorgonia/gorgonia.github.io

https://github.com/gorgonia/gorgonia.github.io/pull/98

Translate documentation to Japanese. 💪

Please stop hack "chcp 65001"

Yasuhiro Matsumoto — Thu, 15 Nov 2018 02:04:53 +0000

On GitHub, there are many codes using chcp 65001. This is hack to display UTF-8 on Windows command prompt. Probably, this hack may works well on non multi-byte locales. But this doesn't work correctly on multi-byte locales. For example, jq used this hack. https://github.com/stedolan/jq/pull/824

This hack break Windows command prompt on multi-byte users.

You can see the font is changed and not restored.

On East Asian locale, figure of backslash is not same on non East Asian locale. On non East Asian locale, backslash is displayed like below.

But on East Asian locale, backslash are displayed like following.

FYI, Most of Windows users normally uses non UTF-8 codepage. For example, japanese use codepage 932.

I wrote a patch to fixed this. https://github.com/stedolan/jq/issues/1121

Furthermore, curl used this hack. https://github.com/curl/curl/issues/3008

This also break cmd.exe.

I wrote a patch similar above again. https://github.com/curl/curl/pull/3212

As I wrote on the description of this PR, when changing codepage on Windows console modify font to suitable possibly. And restoring codepage also set suitable font. So original font is not restored correctly. To write UTF-8 without changing console codepage, you should use Wide String APIs.

Please stop to use hack chcp 65001.

P.S.

If you want to run batch file written in UTF-8, you should run with below to avoid breaking command prompt:

start /wait /min foo.bat

My stupid mistake about goroutine/defer

Yasuhiro Matsumoto — Wed, 24 Jan 2018 10:16:13 +0000

Today, I found small program that can be show colors like green or red from output of go test (ex: PASS/FAIL).

https://github.com/rakyll/gotest

This is good to see easily pass or fail of the test. After playing with this for a while, I met a strange behavior of gotest. The part of output often was not displayed. But I could figure out cause in soon.

func main() {
    setPalette()
    enableOnCI()
    gotest(os.Args[1:])
}

func gotest(args []string) {
    r, w := io.Pipe()
    defer w.Close()

    args = append([]string{"test"}, args...)
    cmd := exec.Command("go", args...)
    cmd.Stderr = w
    cmd.Stdout = w
    cmd.Env = os.Environ()

    go consume(r)

    if err := cmd.Run(); err != nil {
        if ws, ok := cmd.ProcessState.Sys().(syscall.WaitStatus); ok {
            os.Exit(ws.ExitStatus())
        }
        os.Exit(1)
    }
}

func consume(r io.Reader) {
    reader := bufio.NewReader(r)
    for {
        l, _, err := reader.ReadLine()
        if err == io.EOF {
            return
        }
        if err != nil {
            log.Fatal(err)
        }
        parse(string(l))
    }
}

When external command is exited, and finish to call cmd.Run(), goroutine consume must be finished to read until calling os.Exit(). So I patched like below.

func main() {
    setPalette()
    enableOnCI()
    os.Exit(gotest(os.Args[1:]))
}

func gotest(args []string) int {
    r, w := io.Pipe()
    defer w.Close()

    args = append([]string{"test"}, args...)
    cmd := exec.Command("go", args...)
    cmd.Stderr = w
    cmd.Stdout = w
    cmd.Env = os.Environ()

    var wg sync.WaitGroup
    wg.Add(1)
    defer wg.Wait()

    go consume(&wg, r)

    if err := cmd.Run(); err != nil {
        if ws, ok := cmd.ProcessState.Sys().(syscall.WaitStatus); ok {
            return ws.ExitStatus()
        }
        return 1
    }
    return 0
}

func consume(wg *sync.WaitGroup, r io.Reader) {
    defer wg.Done()
    reader := bufio.NewReader(r)
    for {
        l, _, err := reader.ReadLine()
        if err == io.EOF {
            return
        }
        if err != nil {
            log.Print(err)
            return
        }
        parse(string(l))
    }
}

Added sync.WaitGroup to wait groutine consume, and return with exit code. I expected this should work as intended. But this goes panic.

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc0420083fc)
        c:/go/src/runtime/sema.go:56 +0x40
sync.(*WaitGroup).Wait(0xc0420083f0)
        c:/go/src/sync/waitgroup.go:129 +0x79
main.gotest(0xc04203e260, 0x0, 0x0, 0x1)
        c:/dev/go/src/github.com/rakyll/gotest/main.go:54 +0x3bf
main.main()
        c:/dev/go/src/github.com/rakyll/gotest/main.go:33 +0x74

At the first, I thought this is a bug of Go. So I was going to file this issue on GitHub. But this was simply my fault. wg.Wait() is called at exiting function gotest. And w which is opened by io.Pipe() is closed at the next. But reader.ReadLine() is blocking until w.Close() is called. So this is cleanly deadlock (as the message says above). Correct fixes must be below.

func main() {
    setPalette()
    enableOnCI()
    os.Exit(gotest(os.Args[1:]))
}

func gotest(args []string) int {
    var wg sync.WaitGroup
    wg.Add(1)
    defer wg.Wait()

    r, w := io.Pipe()
    defer w.Close()

    args = append([]string{"test"}, args...)
    cmd := exec.Command("go", args...)
    cmd.Stderr = w
    cmd.Stdout = w
    cmd.Env = os.Environ()

    go consume(&wg, r)

    if err := cmd.Run(); err != nil {
        if ws, ok := cmd.ProcessState.Sys().(syscall.WaitStatus); ok {
            return ws.ExitStatus()
        }
        return 1
    }
    return 0
}

func consume(wg *sync.WaitGroup, r io.Reader) {
    defer wg.Done()
    reader := bufio.NewReader(r)
    for {
        l, _, err := reader.ReadLine()
        if err == io.EOF {
            return
        }
        if err != nil {
            log.Print(err)
            return
        }
        parse(string(l))
    }
}

The teachings of Gopher.

Read the error message well.

Go's method is curried funtion

Yasuhiro Matsumoto — Tue, 21 Nov 2017 00:42:24 +0000

Go's method is not same in other languages. In C++, this should not be null.

class Foo {
  public:
    void doSomething() {}
};

int
main() {
  Foo *foo = nullptr;
  foo->doSomething();
}

Go's method does not require that the receiver is instanciated.

package main

import (
    "fmt"
)

type Foo struct {
}

func (f *Foo) doSomething() {
    fmt.Println(f)
}

func main() {
    var foo *Foo
    foo.doSomething()
}

And the method hold receiver value. This is called Method Values.

package main

import (
    "fmt"
)

type Foo struct {
    Value string
}

func (f *Foo) WhatsUp() {
    fmt.Println(f.Value)
}

func doSomething(f func()) {
    f()
}

func main() {
    foo := &Foo{Value: "Hello, World"}
    doSomething(foo.WhatsUp)
}

The WhatsUp is evaulated with the receiver.

package main

import (
    "fmt"
)

type Foo int

func (f Foo) doSomething() {
    fmt.Println(f)
}

func main() {
    Foo.doSomething(123)
}

So this code is valid. Foo.doSomething is the function object. It can be called with let the function object to a variable. The function type should take an first argument which is type of the receiver.

package main

import (
    "fmt"
)

type Foo int

func (f Foo) doSomething() {
    fmt.Println(f)
}

func main() {
    var fn func(Foo)

    fn = Foo.doSomething
    fn(123)
}

This idiom is useful to call runtime.SetFinalizer to Close something resource must be closed.

package main

import (
    "log"
    "os"
    "runtime"
    "myresource"
)

func main() {
    f := myresource.OpenSomethingMustBeClosed()
    if err != nil {
        log.Fatal(err)
    }
    runtime.SetFinalizer(f, (*myresource.Writer).Close)
}

Call Go function from C function

Yasuhiro Matsumoto — Fri, 17 Nov 2017 10:19:51 +0000

Cgo is useful to embed C function into Go.

package main

/*
void doSomething(int *p) {
  *p = 123;
}
*/
import "C"

func main() {
    var n C.int
    C.doSomething(&n)
    println(n)
}

This print 123 which is set from C. And works as intended with no problem. So in your first look, you might think you will not met any issue in future.

Please imagine the case when you want to call APIs which take a callback function and pointer of user_data.

APIs

typedef void (*callback)(void *);
void register_callback(callback cb, void *user_data);
void wait_event();

As you guess, register_callback registers a callback function, wait_event waits for some event, and calls the callback function when an event occured.

Go

func my_callback(v unsafe.Pointer) {
}

func main() {
    user_data := "my userdata"
    C.register_event(/* ? */, /* ? */)
    C.wait_event()
}

Do you wonder how to pass my_callback or user_data into register_event? Unfortunately, type of my_callback in Go is not same as C.callback. In short, you can't write below.

C.register_event(my_callback, &user_data) // compilation errors

To make a function which is possible to be called from C, hack is required.

export Go function to be called from C
pass valid pointer from Go

To export Go function, put comment line //export FuncName above function.

//export hello
func hello() {
    // do something
}

Wow this is simplify. Thus, it is possible make callback proxy.

package main

/*
#include <myapi.h>

void cb_proxy(void *v);

static void _register_callback(void *user_data) {
  register_callback(cb_proxy, user_data);
}
*/
import "C"
import (
    "fmt"
)

func my_callback(v string) {
    fmt.Println("hello", v)
}

func main() {
    C._register_callback(...)
    C.wait_event()
}

//export cb_proxy
func cb_proxy(v unsafe.Pointer) {
    // call my_callback
}

_register_callback is wapper of register_callback to call cb_proxy. wait_event will invoke callback internally with taking argument user_data. user_data should be passed from main. However, one another issue you meet.

Rules for passing pointers between Go and C

Unfortunately again, Go can't pass pointer which is allocated in Go into C function.

But that's possible. See https://github.com/mattn/go-pointer. This package exchange pointers between Go and C.

var s string
C.pass_pointer(pointer.Save(&s))
v := *(pointer.Restore(C.get_from_pointer()).(*string))

As the trick, go-pointer allocate 1-byte dummy memory for passing to C. And it is pointed to the value. i.e. this is unique key of map which is related on the real Go pointer. pointer.Save(&v) store Go pointer into the map and return C pointer which is allocated as dummy.

package main

/*
#include <myapi.h>

void cb_proxy(void *v);

static void _register_callback(void *user_data) {
  register_callback(cb_proxy, user_data);
}
*/
import "C"
import (
    "fmt"
    "unsafe"

    "github.com/mattn/go-pointer"
)

type Callback struct {
    Func     func(string)
    UserData string
}

func my_callback(v string) {
    fmt.Println("hello", v)
}

func main() {
    C._register_callback(pointer.Save(&Callback{
        Func:     my_callback,
        UserData: "my-callback",
    }))
    C.wait_event()
}

//export cb_proxy
func cb_proxy(v unsafe.Pointer) {
    cb := pointer.Restore(v).(*Callback)
    cb.Func(cb.UserData)
}

_register_callbackpass cb_proxy. And cb_proxy restore the poinetr as *Callback. And it call original Callback.Func with Callback.UserData. You can pass callback function to C from Go.

You can try this behavior on this code.

package main

/*
typedef void (*callback)(void *);

static callback _cb;
static void *_user_data;
static void register_callback(callback cb, void *user_data) {
    _cb = cb;
    _user_data = user_data;
}
static void wait_event() {
    _cb(_user_data);
}

void cb_proxy(void *v);

static void _register_callback(void *user_data) {
  register_callback(cb_proxy, user_data);
}
*/
import "C"
import (
    "fmt"
    "unsafe"

    "github.com/mattn/go-pointer"
)

type Callback struct {
    Func     func(string)
    UserData string
}

func my_callback(v string) {
    fmt.Println("hello", v)
}

func main() {
    C._register_callback(pointer.Save(&Callback{
        Func:     my_callback,
        UserData: "my-callback",
    }))
    C.wait_event()
}

//export cb_proxy
func cb_proxy(v unsafe.Pointer) {
    cb := pointer.Restore(v).(*Callback)
    cb.Func(cb.UserData)
}

Why Vim does not fade out?

Yasuhiro Matsumoto — Thu, 16 Nov 2017 05:03:17 +0000

Have you ever thought about why Vim is still used by many people for more than 20 years and did not fade out?

Reputations

Some people says:

I do not understand why there are people continue to use a fossil-like text editor.

Certainly, Vim is character-based text editor. if Vim was same as it was 20 years ago, your opinion would be correct. But you have to know that Vim is changing still now.

In the fact

In the fact, Vim improved many features without stopping.

Version5

Add some GUIs including Win32 GUI
Python/Perl infterface
And many useful features

Version6

Support UTF-8
Vertical split
Netrw
Server-Client
Quickfix
And many useful features

Version7

Spell checking
Support MzScheme
omni-completion
Tab pages
vimgrep
And many useful features

Version8

Asynchronous I/O support, channels and jobs, timers
Partial function, Lambda/Closure function
Packaging
DirectX drawing
GTK+ 3 support

Recently, Vim add new feature:terminal. It is possible to open terminal window in Vim. Run the shell, watching log files, sending e-mail, start Emacs on the :terminal. This feature gave us a lot of possibilities.

This feature was noted as not-good-design for Vim in :help design-not. But the text is written in 20 years ago. Can you say that the design written in 20 years ago is always correct even in nowadays? Bram Moolenaar who is author of Vim decided to change the doc.

Vim is not a shell or an Operating System. It does provide a terminal window, in which you can run a shell or debugger. E.g. to be able to do this over an ssh connection. But if you don't need a text editor with that it is out of scope (use something like screen or tmux instead).

Interestingly, support for :terminal includes Windows. (FYI: I implemented windows :terminal, yay!)

You can use :terminal on Windows GUI/CUI both to run Windows command-prompt. This doesn't depend on Cygwin/msys2. Just require winpty.dll and winpty-agent.exe, for the description, see the bottom of :help Terminal-mode. You can use one window in Vim as if it is command prompt. To know more features of terminal, see :help terminal.

Contributors

And you've better to know how may contributors send patches to vim-dev, we are discussing to fix bugs and add new features. For example, do you know how many commits in this october 2017?

This include adding features and fixing bugs. Can you say that development status of Vim is not active? Vim will continue to change and probably will not become a fossil.

Small Tips to merge GitHub pull-request into your local repository

Yasuhiro Matsumoto — Wed, 15 Nov 2017 08:57:24 +0000

PRO tip to merge GitHub pull-request into local repository easily.

How are you doing checkout GitHub pull-request on local repository?

$ git fetch origin pull/ID/head:BRANCHNAME

ID is pull-request number, BRANCHNAME is name of local branch you will create

Like this? Would you remember this format in tomorrow? Maybe I don't.

Thankfully GitHub serve some extra contents with appending file extension into URLs. See Diff and patch media types. For example, put .diff to the end of pull-request URL:

"https://github.com/vim/vim/pull/2070" + ".diff"

https://github.com/vim/vim/pull/2070.diff

This request will be redirected to the page of unified diff. If put .patch to the URL of pull-request.

"https://github.com/vim/vim/pull/2070" + ".patch"

https://github.com/vim/vim/pull/2070.patch

This will be redirected to the patch file. So you can try to apply the diff like below.

$ curl -sL https://github.com/vim/vim/pull/2070.diff | patch -p1

Also if you want to merge pull-request into your local repository,

$ git checkout -b pr-2070
$ curl -sL https://github.com/vim/vim/pull/2070.patch | git am

Do tests and push into master branch.

$ git checkout master
$ git merge pr-2070
$ git push origin master

Can you see that? It's not difficult at all.

Recent changes in go-sqlite3

Yasuhiro Matsumoto — Wed, 15 Nov 2017 06:39:27 +0000

In brief, I added some important features in go-sqlite3.

Hook events

It was introduced before, but it was not the default. Now you can use this in default. To use this, you need to register hook.

To be honest, I don't think this way is not smart, but no way to do in Go master.

sql.Register("sqlite3_with_hook_example",
    &sqlite3.SQLiteDriver{
        ConnectHook: func(conn *sqlite3.SQLiteConn) error {
            conn.RegisterUpdateHook(func(op int, db string, table string, rowid int64) {
                switch op {
                case sqlite3.SQLITE_INSERT:
                    log.Println("Notified of insert on db", db, "table", table, "rowid", rowid)
                }
            })
            return nil
        },
    })

Then you can see when the record is inserted/deleted/updated. Similar to triggers but easier to handle events, I think. For example, this may be helpful to make replication of the database.

Get limits

This also need to register hook.

var sqlite3conn *sqlite3.SQLiteConn
sql.Register("sqlite3_with_limit", &sqlite3.SQLiteDriver{
    ConnectHook: func(conn *sqlite3.SQLiteConn) error {
        sqlite3conn = conn
        return nil
    },
})
db, err := sql.Open("sqlite3_with_limit", "./foo.db")
if err != nil {
    log.Fatal(err)
}
defer db.Close()

After connect to your database, you can use SQLite3Conn#GetLimit().

varnum := sqlite3conn.GetLimit(sqlite3.SQLITE_LIMIT_VARIABLE_NUMBER)

Export Go func into your SQL

RegisterFunc export your Go function into your SQL.

sql.Register("sqlite3_custom", &sqlite.SQLiteDriver{
    ConnectHook: func(conn *sqlite.SQLiteConn) error {
        if err := conn.RegisterFunc("pow", pow, true); err != nil {
            return err
        }
        return nil
    },
})

pow should be like below. Just call Math.Pow. Note that you must use int64 instead of int for the arguments/return types of the func.

func pow(x, y int64) int64 {
    return int64(math.Pow(float64(x), float64(y)))
}

That's all to be done for enable this feature. So now you can call pow in your query.

err = db.QueryRow("SELECT pow(2,3)").Scan(&i)
if err != nil {
    log.Fatal("POW query error:", err)
}

If you want to use regular expression in the query:

sql.Register("sqlite3_with_go_func", &sqlite3.SQLiteDriver{
    ConnectHook: func(conn *sqlite3.SQLiteConn) error {
        return conn.RegisterFunc("regexp", func(re, s string) (bool, error) {
                return regexp.MatchString(re, s)
        }, true)
    },
})

How fast dev.to is!

Yasuhiro Matsumoto — Wed, 15 Nov 2017 05:47:48 +0000

Today I looked up the techniques how dev.to is doing for speed. Seems to be that use ServiceWorker for fetching contents and store cache. Also use webp images smaller size. pre-fetching contents when mouse-over the link. Embeding CSS-inline to avoid blocking rendering. also external resources ex font.

Great! There are many things to be helpful.

https://dev.to/ben/making-devto-insanely-fast