DEV Community: Kang-min Liu

My new workflow of collecting news clips

Kang-min Liu — Sat, 01 Mar 2025 02:41:00 +0000

Markdownload: https://addons.mozilla.org/en-GB/firefox/addon/markdownload/

I have recently started using Markdownload as a way to collect news articles locally.

Open a new browser window or tab group or workspace.
Read news in my RSS reader, skim through the news and open the desired news website in the browser.
Repeat step 2 until it's enough.
Right-click on the browser's context menu and select "Download all tabs as Markdown."

Then, new Markdown text files would appear the downloaded folder. The content is the core portions of those webpages. Most of the time, it works quite accurately. There might be some irrelevant information at the beginning or the en and would require manual trimming. Perhaps an extra refinement with local LLM would be interesting.

The main reason I'm doing this is that sometimes when I see something that might have an impact in the future, I find it difficult to locate the original news article later on. By saving news articles as Markdown files, I can then easily use grep to search for certain keywords and since there will be no irrelevant information in the search space, the result would be very accurate.

I've been doing this for about two weeks now and have collected 60 news articles. Perhaps in a few months I'll discover something new.

Safe-navigation monad

Kang-min Liu — Mon, 19 Jun 2023 16:01:49 +0000

Recently on reddit.com/r/perl, pmz posted a question like this:

XML::Twig, get value (or not) without dying

Currently I do:
if (defined $elt->first_child('addr')->first_child('postalCode')) { $patient{patient_postal_code} = $elt->first_child('addr')->first_child('postalCode')->text ; }
because if I don't check for "defined" and the resulting value is null , it dies.

Link to the original post: https://www.reddit.com/r/perl/comments/1492sc1/xmltwig_get_value_or_not_without_dying/

While on one hand the question is about how to use XML::Twig, on the other hand the obvious inconvenience here is when first_child('addr') returns undef, which means there are no <addr> element underneath, the following call of first_child('postalCode') would make the programm die. Generally speaking: in a chain of calls we expect objects to be present in all positions, but sometimes there are undef. Given that, is there a way to avoid the program from dying and let the entire call chain return undef if undef is encountered in the calling chain ?

To formalize the question a bit more generically: assume a class with instance methods a(), b(), and c(). These methods may return an instance of same class, or ocassionally, undef. Consider the following chain of calls originally from $o:

$res = $o->a()->b()->c();

In case any of a(), b(), or c() returns undef, the program dies with messages like this:

Can't call method "c" on an undefined value

Which suggests b() returns undef and since undef is not an object, we cannot call methods on it.

Now, could we rewrite the same program to prevent the abovementioned error from happening, while making $res be undef if any of a(), b(), c() returns undef, or otherwise, the return value of c() ?

In some other programming languages, such purpose could be satisfied by using the safe-navigation operator, such as ?. in javascript or kotlin:

res = o.a()?.b()?.c();

Or in raku, .?

$res = $o.a().?b().?c();

However, we haven't seen anything similar up until perl 5.38 just yet.

A rather intuitive way to rewrite would be something like this:

$res_a = $o->a();
$res_b = $res_a && $res_a->b();
$res   = $res_b && $res_b->c();

However, besides making the program much longer and less easier to grasp, the rewrite is not generic. It'll be different for similar statements with different method names. Not a good strategy.

Meanwhile, here's a super simple and generic way:

$res = eval { $o->a()->b()->c() };

However, with the powerful side-effect of eval, all exceptions would be ignored while we are only interested in ignoring undefined values. That is a lot more than what we want. Even though it looks simple, it is probably not applicable.

Here is a solution with Monad design pattern.

The rewritten version looks like this:

$res = SafeNav->wrap($o) ->a()->b()->c() ->unwrap();

The SafeNav is defined as the folowing.

use v5.36;
package SafeNav {
    sub wrap ($class, $o) { bless \$o, $class }
    sub unwrap ($self)    { $$self            }

    sub AUTOLOAD {
        our $AUTOLOAD;
        my $method = substr $AUTOLOAD, 2 + rindex($AUTOLOAD, '::');

        my ($self, @args) = @_;

        # [a]
        (defined $$self) ?
            __PACKAGE__->wrap( $$self -> $method(@args) ) :     # [a.1]
            $self;                                              # [a.2]
    }

    sub DESTROY {}
};

SafeNav is a class that wraps all scalar values and equips with AUTOLOAD for responding to all method calls.

Inside AUTOLOAD there is the core part of our logic in [a]: If we are not wrapping an undef value, we call the original method on it, then re-wrap the return value ([a.1]). Or if we are wrapping an undef, we ignore the method call and just lay down and keep being ourselves ([a.2]).

Thanks to the mechanism of AUTOLOAD, the original form of ->a()->b()->c() is kept exactly the same after the rewrite. Let's put both versions side-by-side for a quick comparison:

$res = $o ->a()->b()->c();
$res = SafeNav->wrap($o) ->a()->b()->c() ->unwrap();

The wrap() at the front, together with unwrap() at the back, form a clear boundary in which SafeNav is effective. Method calls after unwrap() are not not guarded by SafeNav.

With that, we properly ignore undef values, nothing more. If other kinds of exceptions are thrown from method a, b, c, the program would correctly abort. In the 3 proposed ways to rewrite the program in this article, the SafeNav monad is both generic and not adding too much verbosity to the original program.

Original post: https://gugod.org/2023/06/perl-safe-navigation-monad-en/

Making Single-binary Release with pp

Kang-min Liu — Thu, 21 Oct 2021 15:01:08 +0000

pp comes with PAR::Packer, which is a tool for "compiling" a bunch of modules and codes and makes a single binary.

perldoc pp already contains a good amount of documentation one can refer to.

While it works with system perl, I found it even easier to first prepare a directory of local::lib, then just package that entire directory.

pp tries its best to determine the list of dependencies of the given program in various ways but none of those are guaranteed to be 100% accurate. As matter of fact that guarantee is impossible to make. This is partially due to the fact that a running program can load modules in ways that cannot be easily determined by reading the program source code, or in a relative hidden code path that cannot be easily captured.

That is a good amount of flexibility, but definitely a pricey one. Although, arguably it is also an issue brought by the tool (pp, perlcc, perl2exe, etc.). I guess that is because the dynamic nature is so convenient as long as all the installation are done right. Having a tool that is able to automically complie all dependencies together was not needed that much. It has definidently needed, that is why we have those tools now, but in the last mile of their completion, lies an undecidable problem.

So we will need to manually add those missing depedencies to pp command, which is fine only when the list is small. Since we wish to just pack all the declared dependencies together, we don't care that much even if that's going to make the result a little bigger than it has to be. If we can have the same build script that works everywhere, it is as happy as in Christmas. (An pure metaphoric experssion. Pesonally I feel nothing spceial in Dec 25.)

Anyway...... it turns out to be much easier to decide the dependency at installation time, since that's all well-declared and tools like cpm, or cpanm already does this perfectly. If we install dependencies in a self-contained directory, we could just archive the entire directory together with the program we are packing, and that should be good to go.

Let's say we cd into the source code of foo and we are trying to compile the program foo as a binary. The executable is at bin/foo, while its own moulders such as Foo.pm, Foo/Bar.pm are put under the conventional directory lib.

Given that, this script should produce foo as a single binary that as if all dependencies are "statically-linked" inside:

#!/bin/bash

# Prepare local/ 
cpanm -L local -q --installdeps .
# or: cpm install

perlversion=$(perl -MConfig -e 'print $Config{version}')
pp -B \
    -I ./local/lib/perl5 \
    -a "./local/lib/perl5/;$perlversion/" \
    -a lib \
    -o foo \
    bin/foo

Alas, this is almost perfect -- except modules in corelist might still be missing. They won't be inside local/ and if they are somehow not discovered by pp then they'll be missing in the end result. We won't know this until we manually test the result foo thoroughly. Basically we should always add a bunch of -M flags in the build script instead of assuming pp would do the right thing.

For example, like so, when all of Getopt::Long, JSON::PP, and Sys::Hostname are required.

pp -B \
    -M Getopt::Long:: \
    -M JSON::PP:: \
    -M Sys::Hostname:: \
    -I local/lib/perl5 \
    -a "./local/lib/perl5/;$perlversion/" \
    -a lib \
    -o foo \
    bin/foo

A rather tedious modification as the list of dependent modules now exists in two places in the repo. Surely there is some way to refactor this.

I've verified the following script build-minicpan.sh that can download the tarball of CPAN::Mini and build a working minicpan out of it:

#!/bin/bash
set -e

curl --silent -O https://cpan.metacpan.org/authors/id/R/RJ/RJBS/CPAN-Mini-1.111016.tar.gz

tar -xzf CPAN-Mini-1.111016.tar.gz

cd CPAN-Mini-1.111016

cpanm -n -q -L local --installdeps .

perlversion=$(perl -MConfig -e 'print $Config{version}')

pp -B \
   -M Getopt::Long:: \
   -I ./local/lib/perl5 \
   -a "./local/lib/perl5/;$perlversion/" \
   -a lib \
   -o ../minicpan \
   bin/minicpan

echo "DONE: minicpan"

To me this seems to be an area worth exploring... I've been experimenting the automation of this in a side-project: pau, which is a collection of shell functions that can install app to their own self-contained directory and expose just the program itself. Very similar to what pipx does. pp support was added not long ago but still there is no good way to figure out all the missing modules and automatically add them as -M arguments.

Maybe as a lazy solution, we should just always produce a heavy-pack that includes the whole core lib directory (usually $INC[-1]) regardless whether any of them are used.

Maybe.

Originally posted at: gugod's blog -- Making single binary release with pp

Email for everything ?

Kang-min Liu — Sun, 17 Oct 2021 13:06:44 +0000

Owning an IMAP-capable email account can be very convenient since that
be used as a centralized storage for information around us. After all,
the easiest way to synchronize information across multiple devices is by
sending emails to ourselves.

Since we already setup our emails on multiple devices, there are no more
accounts to sign-up, no more passwords to be managed.

Several purposes can be easily satisfied by briefly re-purposing emails
in various ways.

Note-taking -- quick notes taken on-the-road with mobile devices can be
saved as drafts -- usually in folder named like "Drafts". The saved
draft can be edited in a full-fledged UI. Since those are draft mails
they can just be re-saved over and over. Basically just like normal
files.

TODO list -- by conventionally adding TODO in Subject line, we can make
a personal TODO lists. Each TODO item would be one mail, with subject
line being the gist and body being details. Marking things DONE is just
replying the mail with modified title.

A "Read it Latter" queue -- similar how TODO lists can be
implemented. For each reading we wish to postpone, we mail to ourselves
with the link and perhaps with a gist of what it is about.

Perhaps with some extra tooling, we can easily send a full copy of the
whole web page so we have the whole thing ready to be consumed when we
are ready. For the moment it seems on macOS, this can be easily done
with Safari + Mail -- we can "Share" the page under "Reader" mode and
Safari will just clip the content out. Very nice.

These can be all be implement with one IMAP-capable email account. We
would probably have different folders for different type of usages. In
addition, we could also use different alias or +-suffixes to further
distinguish different contexts too.

These ideas are about exchanging ideas from our present selves with
our future selves. We write something and let future us to read,
follow-up, react.

If we self-host IMAP servers and configure it to share certain folders
to multiple users. Besides emailing to each other, users can discover
new information by browsing those shared folders. Should user be able
to directly post new messages to those folders, that is a basic BBS.
In a sense this setup is conceptually the same having a single NNTP
server for that group of user.

If each users can own one dedicated folder that is specially
configured so that new posts can be made only by its owner, while
replies can be posted by everyone -- that is a multi-blog service,
or a SNS.

Suppose such IMAP-based SNS exists... since users are required to use
email client to access the SNS and since advanced email client all
have search/filtering feature, users can fine-tune the so called
"algorithm" at the client side to boost posts however they like.

Since this requires users to do a whole lot more to fine tune "the
algorithm" on the client side, it also means to give users more
freedom and less there can be less chance of having some centralized
recommendation system in between content and eyeballs.

Wouldn't that be nice ?

--
Best,
Kang-min Liu

Originall posted at gugod's blog -- Email for everything ?

CPAN Release of TooMuchCode 0.18

Kang-min Liu — Wed, 29 Sep 2021 00:32:55 +0000

Perl::Critic::TooMuchCode is a set of policy addons that generally checks for dead code or redundant code.

It's just about a week after the release of 0.17 and after asynchronously exchanged some thoughts with @oalders and @ferki (Issue #25), I decide to make a follow-up release shortly.

In version 0.17 we updated the policy ProhibitDuplicateLiteral to include a new config parameter whitelist to replace the old parameter whitelist_number. In version 0.18 they are both removed. The replacement for them is the parameter allowlist.

So now if we really need to write number 42 and "forty two" literally many times, we list them in .perlcriticrc:

[TooMuchCode::ProhibitDuplicateLiteral]
allowlist = "forty two" 42

The name "allowlist" was previously suggested by @ferki already and I amended to match existing convention of using "whitelist". Now I decide that such convention should be changed (See PR #28). It's a better naming choice in the sense that it direct and explicit, perhaps until the day the word "allow" lost its current meaning.

The other improvement is to address Issue 18 (thanks to @oalders again.), in which assignment of imported symbols to @EXPORT or @EXPORT_OK as correctly counted as using those symbol by the policy ProhibitUnusedImport.

There have not counted because it is also a correct thing to do. Considering we have three symbols in place: foo, $bar, and @baz, and an assignment like this:

my @stuff = qw(foo $bar @baz);

Although those symbols appears in the statement literally, evaluating that statement does not depends on the evaluation of those symbols because they are all quoted inside of qw(). @stuff is evaluated to 3 strings: 'foo', '$bar', and '@baz', which has nothing to do with foo (bareword, perhaps a subroutine name), $bar, and '@baz'.

However, if the LHS array variable is either @EXPORT or @EXPORT_OK, the assignment has different meaning:

our @EXPORT = qw(foo $bar @baz);
our @EXPORT_OK = qw(@baz);

@EXPORT itself is still assigned with just 3 strings, but eventually when the current module is imported (use-ed or require-ed) by a foreign piece of code, those 3 strings refer some content in the symbol table and they might be used remotely.

Well, whether foo is really used remotely is probably unknown-able. It should be sufficient to just consider the case of assignments involving @EXPORT and @EXPORT_OK.

I'm guessing that there are probably more special cases yet to be discovered regarding those special variables in perlvar, or however famous module X defined its own sub-language for us to cope with.

Originally posted at gugod's blog -- CPAN Release of TooMuchCode 0.18

CPAN installation as a test, with GitHub workflow

Kang-min Liu — Wed, 22 Sep 2021 00:48:05 +0000

If I made a distribution Foo and uploaded it to CPAN, I'd expect that it is install-able via various CPAN clients such as cpan, cpanm, cpm, that is, from a fresh perl installation, these commands should be successful:

cpan Foo
cpanm Foo
cpm install -g Foo

What's a bit non-obvious is that this requires a chain of conventions to be satisfied in order to happen smoothly... distribution Foo needs to properly ship with build procedure / installer and META.json, in which all dependencies are correctly declared.

But that is not enough, even if I correctly authored all the meta-data in distribution Foo, if there is even a single miss in any other dependencies in Foo's dependency tree, the installation would fail.

For that reason, I thought it might be worth it to test the installation process during development, at least make a CI workflow that tries to install everything. The successful of such test would just be whether the installation is successfully finished or not. Conventionally the exit status ($?) reflects that.

Here's something I came up with. It is a GitHub workflow that first build a distribution, a .tar.gz file, then try to install that file with cpanm:

name: Installation Test

on:
  push:
    branches:
      - fun

jobs:
  build:
    runs-on: 'ubuntu-latest'
    steps:
      - uses: actions/checkout@v2
      - name: Setup Perl environment
        uses: shogo82148/actions-setup-perl@v1
      - name: Install Authoring Dependencies
        run: cpanm -n App::ModuleBuildTiny && cpanm -q -n --no-man-pages --installdeps .
      - name: Generate dist
        run: mbtiny dist
      - name: Display what is generated
        run: |-
          echo *.tar.gz
          tar tvzf *.tar.gz
      - uses: actions/upload-artifact@v2
        with:
          name: dist-for-installation-test
          path: '*.tar.gz'
          retention-days: 1
  install:
    runs-on: 'ubuntu-latest'
    needs: build
    container:
      image: perl:5.34
    steps:
      - uses: actions/download-artifact@v2
        with:
          name: dist-for-installation-test
      - name: Display the downloaded files
        run: ls -R
      - name: Install in a perl:5.34 container (--notest)
        run: cpanm --notest *.tar.gz

There are a lot of instructions but what matters is the final one cpanm --notest *.tar.gz. This would install whatever was built and all its dependencies, without running module tests`, and if even that failed, that means some meta data is missing and the distribution would not be install-able even if it is uploaded to CPAN.

Of course that is just cpanm, I could also add cpan and cpm there to verify whether there is a difference between all these CPAN clients. Also, I could setup a matrix run so it runs on all perl versions.

This workflow is currently used in Perl-Critic-TooMuchCode, with a few runs already finished here.

The workflow contains two jobs, build and install. install needs build so it is always executed after successful build.

The build job runs on the default ubuntu-latest machine, but the install job runs in a container instance with presumably a different version of perl, and also without a clone of current repo. With this setup I can simulate the installation process on a fresh machine.

To share the distribution .tar.gz file across different jobs, the only way I can fin is by uploading the file to the "artifact" storage -- basically an external storage. This probably cost me something if a lot of files are accumulated. I changed the retention period to 1 day because I don't plan to download this anyway.

On the other hand, it could be part of doing the actual CPAN release. Maybe with some modification the workflow would build a new version with new version number, and I would just have to download the artifact then re-upload to CPAN. This would save the setup of preparing authoring tools. It could be a useful scenario for teaching new developers to upload something to CPAN.

The installation process also download a lot of stuffs from cpan.org and cost some bandwidth on the way. It's best not to do so on every commits or on a branch with frequent pushes. Definitely suitable pre-release though.

Most likely, this idea of "Installation as a test" isn't new, and it is a bit convenient to have it checked in CI.

Originally posted on gugod's blog -- CPAN installation as a test, with GitHub workflow

CPAN Release of TooMuchCode 0.17

Kang-min Liu — Mon, 20 Sep 2021 23:05:15 +0000

Perl::Critic::TooMuchCode is a set of policy addons that generally checks for dead code or redundant code.

I feel grateful that this small project starts to draw some attention and endorsement and now it is receiving pull-requests from the Perl/CPAN community.

In version 0.17 we improved the policy ProhibitDuplicateLiteral and now it we can whitelist strings and numbers in configurations. If you somehow really need to use number 42 and "forty two" literally in the code many times, you list them in .perlcriticrc:

[TooMuchCode::ProhibitDuplicateLiteral]
whitelist = "forty two" 42

Thanks to @ferki from project Rex!

Rex is an automation framework, or remote-execution framework. In a sense, similar to Ansible. It's a simple way of telling machine what to do, with some code named Rexfile -- a task manifest. which is also perl code. Within which you'd repeat some strings literally such as:

service 'apache2',  ensure => 'started';
service 'mysql',    ensure => 'started';
service 'memcached, ensure => 'started';

... and that is OK. Because those manifests should be direct instead of indirect/abstract with the use of variables / constants to just hold the string 'started'. Doing such is an unnecessary level of indirection since the purpose of task manifests is to tell machine what to do and we do not want to make human readers read twice just to understand what those statement means. After all, repeatition by itself isn't necessarily the problem.

Well, apparently in the context of coding Rexfile, some repetition are allowed. In general, perhaps repetition are OK in any DSL code. DSL itself already reduce some repetition by hiding the details and DSL keywords tends to be high-level constructs that are designed to be easily understandable for human readers. Repetition of DSL keywords almost never leads to be unreadable or "bad small".

I might have repeated the word "repetition" too many times.

Originally posted at: gugod's blog -- CPAN Release of TooMuchCode 0.17

CPAN Release of Time::Verbal module

Kang-min Liu — Wed, 08 Sep 2021 15:09:37 +0000

Time::Verbal 1.1.1 made its way to CPAN.

This module does one little thing: converting time durations to verbal formats, rounded to some "sensible" unit. Just as how you would say such thing as a human.

my o = Time::Verbal->new();
say $o->distance( time(), time() - 4567 );
#=> about 1 hour

Besides the default English format, other 67 languages identified by these ISO language codes are supported:

ar bg bn-IN bs ca cy da de-AT de-CH de dsb el en-AU en-GB en-US eo es-AR es-CL es-CO es-MX es-PE es et eu fa fi fr-CA fr-CH fr fur gl-ES gsw-CH he hi-IN hi hr hsb hu id is it ja ko lo lt lv mk mn nb nl nn pl pt-BR pt-PT rm ro ru sk sl sr-Latn sr sv-SE sw tr uk vi zh-CN zh-TW

I don't know all these languages (nor their language codes), but Locale::Wolowiz does. Or, at least it is the library that does the translation, given a bunch of JSON files that contain the body of translations.

So here's how you say "about 1 hour" in Vietnamese:

my o = Time::Verbal->new( "locale" => "vi" );
say $o->distance( time(), time() - 4567 );
#=> khoảng 1 giờ

A default version of translations of all the above-mentioned 68 languages are provided with Time::Verbal itself and is ready to use once installed properly -- a usual cpan or cpanm or cpm command would suffice. However, in case some customize version is need because, say, you wish to provide better UX to whom speak the Vulcan variant of Klingon, you need to create a file named after the language code -- which can be arbitrary such as tlh-Vulcan. You would create tlh-Vulcan.json and put it under /app/i18n, or really any directory of your choice, then do:

my o = Time::Verbal->new(
    "locale" => "tlh-Vulcan",
    "i18n_dir" => "/app/i18n"
);
say $o->distance( time(), time() - 4567 );
#=> Chaq wa' rep

Disclaimer -- I don't know if that's how Vulcanized Klingon speaks. Don't take my example as if it is authentic.

To prepare the translation, take a look of the json files under the default i18n_dir, which can be printed by running this oneliner:

perl -MTime::Verbal -E 'say Time::Verbal->new()->i18n_dir'

It shouldn't be difficult to grasp the structure of those translation files and produce a new one. It is however, very difficult to start describing the schema of thoes json file as an indolent developer.

The translations are accquired from rails-i18n project -- which is one of the first modules that does things like this.

Originally posted at https://gugod.org/2021/09/cpan-release-of-time-verbal/

A simple thought about key-value pairs

Kang-min Liu — Fri, 03 Sep 2021 23:54:17 +0000

Very often, we wrote Web API that produce JSON output and also very often, we want to ensure that certain values in the output are produced as numerical literals (42 instead of "42") or as boolean literals (true or false). The only reliable way, imho, is to cast them when printing the JSON output, or when preparing the structure for that output -- that is basically as "late" as possible.

Since we also often produce JSON with JSON module, we may have a TO_JSON around that looks like this, with many inline type-coercion expression:

sub TO_JSON ($self) {
    return {
        # string
        "align" => "". self->align,

        # number
        "height" => 0+ $self->height,
        "width" => 0+ $self->width,

        # boolean
        "closed" => $self->is_closed ? \1 : \0
        "is_landscape" => ( $self->height < $self->width)  ? \1 : \0
        "fits" => ( ($self->height * $self->width) < LIMIT ) ? \1 : \0
    }
}

When the unavoidable fate comes and we include more and more attributes in the TO_JSON, such inline expressions becomes bulky and a bit tedious to look at... just imagine the situation of having a few dozens of inline ternary expressions not perfectly aligned and one of them being wrong. It would be nicer to make them look good and make those inline expressions look like some sort of annotation:

sub TO_JSON ($self) {
    return {
        "align" => json_str(self->align),

        "height" => json_num( $self->height ),
        "width" => json_num( $self->width ),

        "closed" => json_bool( $self->is_closed ),
        "is_landscape" => json_bool( $self->height < $self->width ),
        "fits" => json_bool( ($self->height * $self->width) < LIMIT ),
    }
}

These helper methods that does type coercion by themselfs are trivial:

sub json_str ($x) { "". $x }
sub json_num ($x) { 0 + $x }
sub json_bool ($x) { $x ? \1 : \0 }

Or, perhaps even better, we could annotate the attributes in groups, arranged by their expected type:

sub TO_JSON ($self) {
    return {
        kvpairs_json_str(
            "align"  => self->align,
        ),

        kvpairs_json_num(
            "height" => $self->height,
            "width"  => $self->width,
        ),

        kvpairs_json_bool(
            "closed" => $self->is_closed,
            "is_landscape" => ( $self->height < $self->width ),
            "fits" => ( ($self->height * $self->width) < LIMIT ),
        )
    }
}

Using the pairmap function from List::Util, the new helper methods looks also trivial:

use List::Util ('pairmap');

sub kvpairs_json_str (@kvlist) {
    pairmap { ($a, json_str($b)) } pairs @kvlist
}
sub kvpairs_json_num (@kvlist) {
    pairmap { ($a, json_num($b)) } pairs @kvlist
}
sub kvpairs_json_bool (@kvlist) {
    pairmap { ($a, json_bool($b)) } pairs @kvlist
}

Since for our purpose of printing object as JSON, keys can be ignored because they are always string literals directly written as part of our code. We only have to deal all the values which are every 2nd elements in @kvlist. Also, because the overall result of TO_JSON is a HashRef, the order of keys as they are written in the code is irrelevant and that allows us to arrange those key-value pairs in groups so overall it looks nicer and organized.

Meanwhile, we also often hide partial kvpairs given some condition because some attributes are optional and perhaps convey special meaning when they are part of the output. A typical example would be the the conventional "error" attribute in API responses.

my $response = {
    defined($o->error) ? (
        "error" => $o->error,
    ):(),

    ...
};

That's a ternary op with an ever weirder look. I wonder if this is better:

my $response = {
    kvpairs_without_undef(
        "error" => $o->error,
        ...
    ),
    ...
};

That is like marking a section of kvpairs as optional and exclude all pairs with undef value. The helper function kvpairs_without_undef can be done with the help of pairgrep:

use List::Util ('pairgrep');

sub kvpairs_without_undef (@kvlist) {
    pairgrep { defined($b) } @kvlist
}

This is probably doing a lot of copying and perhaps there are some performance enhancements to be done. But well, these are just thoughts, optimizing thoughts is even before the so called premature optimization.

(Perhaps that's fun though.)

Originally published at: https://gugod.org/2021/07/perl-simple-thoughts-kvpairs/

Visualization of perl development history

Kang-min Liu — Thu, 10 Jun 2021 06:32:06 +0000

About 10 years ago I tried gource and made a visualization of the commit history of perl5.git. It was when grouce was new, shiny and cool:

10 years passed like a blink and it has been a remarkable decade for perl5.git. As we see the growth of commit volume, there is also stable pace of releasees thanks to the standardization of release procedure. With relocation of communication platform, participation and small fixes seems to be much easier. While it has not be a perfectly stable decade, it is a good one.

So as a 10-yearly review, here's the updated version of that gource visualization on perl5.git, all the way to mid 2021:

These are the commands used to produce that video:

cd ~/src/perl5
gource -s 0.01 --auto-skip-seconds 0.1 -r 30 -f --user-scale 3 --file-idle-time 0 --hide bloom,filenames,dirnames --date-format '%Y' -o ~/var/gource-perl.ppm
ffmpeg  -an -y -f image2pipe -vcodec ppm -i ~/var/gource-perl.ppm -r 30.000 ~/var/gource-out.webm

本篇文章為 perl 開發歷史紀錄之視覺化之英文版

Solving Perl Weekly Challenge 096 -- Reverse Words and Edit distance.

Kang-min Liu — Thu, 21 Jan 2021 12:25:05 +0000

This week, Perl Weekly Challenge 096 gives us two tasks about string manipulation. Besides Raku, let's try janet and rust.

The algorithm for computing edit distance is of pratcial uses, such as in git: git/levenshtein, for recomending commands when user runs an unknown command:

# git lag
git: 'lag' is not a git command. See 'git --help'.

The most similar commands are
    log
    tag

Same as in perlbrew: perlbrew/editdist. Meanwhile in rakudo, the compiler of Raku language: rakudo/levenshtein, for recommending subroutines when seeing unknown subroutine name in the code. Like this:

# raku -e 'pay "hello"'
===SORRY!=== Error while compiling -e
Undeclared routine:
    pay used at line 1. Did you mean 'say'?

There is a difference though. In git, the algorithm is 'Damerau-Levenshtein distance'. While in perlbrew and raku, the algorithm is ' levenshtein distance'. The former sees character-swaping as one operation, the latter does not.

TASK #1 › Reverse Words

Submitted by: Mohammad S Anwar

You are given a string $S.

Write a script to reverse the order of words in the given string. The string may contain leading/trailing spaces. The string may have more than one space between words in the string. Print the result without leading/trailing spaces and there should be only one space between words.

Example 1:

Input: $S = "The Weekly Challenge"
Output: "Challenge Weekly The"

Example 2:

Input: $S = "    Perl and   Raku are  part of the same family  "
Output: "family same the of part are Raku and Perl"

Solution #1 > Revrse Words

For this task, we first split the string by whitespacse to get a collection words, and re-join them in reverse order.

Raku:

sub reverse-words (Str $S --> Str) {
    $S.words.reverse.join(" ");
}

Janet:

(defn reverse-words
  "Reverse the string word by word"
  [S]
  (string/join (reverse (filter (fn [x] (not (= x "")))
                        (string/split " " S)))
               " "))

Rust:

fn reverse_words(x: String) -> String {
    return x.split_whitespace().rev()
            .collect::<Vec<&str>>()
            .join(" ");
}

Turns out it looks more or less the same in these 3 languages. The string/split in janet producess one empty string when encountering two consecutive whitespaces, therefore an extra filter is required to remove those extra empty strings.

TASK #2 › Edit Distance

Submitted by: Mohammad S Anwar

You are given two strings $S1 and $S2.

Write a script to find out the minimum operations required to convert $S1 into $S2. The operations can be insert, remove or replace a character. Please check out Wikipedia page for more information.

Example 1:

Input: $S1 = "kitten"; $S2 = "sitting"
Output: 3

Operation 1: replace 'k' with 's'
Operation 2: replace 'e' with 'i'
Operation 3: insert 'g' at the end

Example 2:

Input: $S1 = "sunday"; $S2 = "monday"
Output: 2

Operation 1: replace 's' with 'm'
Operation 2: replace 'u' with 'o'

Solution #2 > Edit Distance

In the following, a navie recursive implementation of Levenshtein distance is provided, basically follow the definition from Wikipedia: Levenshtein-distance#Definition.

It is pretty straight forward this way. However, the implementations from the abovementioned projects are all done with Dynamic Programming. When the inputs (S1 or S2) are lengthy, the about of computation should be greatly reduced.

Raku:

sub edit-distance (Str $S1, Str $S2 --> Int) {
    my sub lev ($a, $b) {
        return $a.chars if $b.chars == 0;
        return $b.chars if $a.chars == 0;
        return lev( $a.substr(1), $b.substr(1) ) if $a.substr(0,1) eq $b.substr(0,1);
        return 1 + (
            lev($a, $b.substr(1)),
            lev($a.substr(1), $b),
            lev($a.substr(1), $b.substr(1)),
        ).min;
    };

    return lev($S1, $S2);
}

Janet:

(defn lev
  "Compute the Levenshtein distance between string a and b"
  [a b]
    (cond (= 0 (length a)) (length b)
          (= 0 (length b)) (length a)
          (let [
                a_head (string/slice a 0 1)
                a_tail (string/slice a 1)
                b_head (string/slice b 0 1)
                b_tail (string/slice b 1)
                levtail (lev a_tail b_tail)
                ]
            (if (= a_head b_head) levtail
                (+ 1 (min
                      (lev a b_tail)
                      (lev a_tail b)
                      levtail))))))

Rust:

fn edit_distance(a: &str, b: &str) -> u32 {
    let chars_a = a.chars().collect();
    let chars_b = b.chars().collect();
    return lev( &chars_a, 0, &chars_b, 0);
}

fn lev (a: &Vec<char>, offset_a: usize, b: &Vec<char>, offset_b: usize) -> u32 {
    if offset_a == a.len() {
        return (b.len() - offset_b) as u32;
    }
    if offset_b == b.len() {
        return (a.len() - offset_a) as u32;
    }

    let n3 = lev( a, offset_a+1, b, offset_b+1 );
    if a[offset_a] == b[offset_b] {
        return n3;
    } else {
        let n1 = lev( a, offset_a+1, b, offset_b );
        let n2 = lev( a, offset_a,   b, offset_b+1 );
        return [n1, n2, n3].iter().min().unwrap() + 1_u32;
    }
}

本文為《解 Perl Weekly Challenge 096 -- 單字次序倒轉與兩字串編輯距離》之英文版。

Solving Perl Weekly Challenge 095 -- Palindrome Number and Stack

Kang-min Liu — Mon, 11 Jan 2021 13:04:08 +0000

Two basic tasks from Perl Weekly Challenge 095. Palindrome number is also on leetcode: leetcode/palindrome-number. For languages that coerce values to Int / Str automatically, these two tasks does not require too much code.

TASK #1 › Palindrome Number

Submitted by: Mohammad S Anwar

You are given a number $N.

Write a script to figure out if the given number is Palindrome. Print 1 if true otherwise 0.

Example 1:

Input: 1221
Output: 1

Example 2:

Input: -101
Output: 0, since -101 and 101- are not the same.

Example 3:

Input: 90
Output: 0

Solution #1 > Palindrome Number

It is pretty straightforward: simply convert the input number to string, then check whether that string is palindromic. Based on the example, all negative numbers are non-palindromic. I wonder numbers with decimal points are considered.

Let's say we are only dealing with integers. Here's the implementation in Raku.

sub is-palindrome-number (Int $n --> Bool) {
    return "$n" eq "$n".flip;
}

The operation of flipping a string corresponds to the flip subroutine from Str in Raku. To convert a number $n to string, either $n.Str or "$n" would do. The part where it says (Int $n --> Bool) is the signature of this subroutine. It means that the subroutine takes one Int argument, and returns a Bool (Boolean values. True or False.)

TASK #2 › Demo Stack

Submitted by: Mohammad S Anwar

Write a script to demonstrate Stack operations like below:
push($n) - add $n to the stack
pop() - remove the top element
top() - get the top element
min() - return the minimum element

Example:

my $stack = Stack->new;
$stack->push(2);
$stack->push(-1);
$stack->push(0);
$stack->pop;       # removes 0
print $stack->top; # prints -1
$stack->push(0);
print $stack->min; # prints -1

Solution #2 > Demo Stack

If we are just defining a class with the abovementioned methods, they can all be delegated to one subroutine in Raku. Let's use this IntStack as an example, which is stack that can hold a bunch of integers.

class IntStack {
    has Int @!store;

    method push(Int $n) {
        @!store.push($n);
    }
    method pop(--> Int) {
        @!store.pop;
    }
    method top(--> Int) {
        @!store.tail;
    }
    method min(--> Int) {
        @!store.min;
    }
}

Which can be used like this:

    my $stack = IntStack.new;
    $stack.push(2);
    $stack.push(-1);
    $stack.push(0);

    $stack.pop;
    say $stack.top;  #=> -1

    $stack.push(0);

    say $stack.min;  #=> -1

本文為《解 Perl Weekly Challenge 095 -- 回文數與堆疊》之英文版