DEV Community: Lance Wicks

Building a tool to integrate Readwise.io highlights into my Zettelkasten via Perl

Lance Wicks — Mon, 06 Nov 2023 00:00:00 +0000

Recently I started using the Readwise.io as a replacement both for my RSS reader and my bookmarking app. I also use a Zettelkasten system of thinks I want to record; so decided to integrate via the API so that highlights I make in Readwise end up as markdown files on my local machine (and git repo).

The editor I use mostly for Zelltelkasten is the lovely Zettlr or vim; so I wanted to craft simple markdown files. Being me; I reached for Perl and have written a small CPAN module along the way called WebService::Readwise which includes a example script of the basics of what I cover in this tale.

TDD for Good... strings

Lance Wicks — Sun, 18 Jun 2023 00:00:00 +0000

One of the things about Test Driven Development (TDD) is that it's something that takes practice. A great way to do that is via the amazing Weekly Challenge which is led by the amazing Mohammad Anwar. Mohammad is a force for good in the Perl community and this week I wanted to take the time to thank him and express how much I appreciate him and wish him well always... but especially at the moment when he faces some challenges! Kia Kaha my friend!

The first challenge this week (https://theweeklychallenge.org/blog/perl-weekly-challenge-221/) was to develop some software to calculate a sum of the lengths of "Good Strings" where a good string is described as:

A string is good if it can be formed by characters from $chars, each character can be used only once.

For me this starts with creating a .t test file. Yes, a test file first. I use Test2::V0 and target a module to test, so something like this:


use Test2::V0 -target => 'Good::Strings';
ok 1;
done_testing;

This fails as I have not created the module file yet.

So my next step is to create a package/module in a lib directory, with this as content:


package Good::Strings;

1;

Then I run my tests again. I make lots of typo mistakes; so running my tests regularly/constantly and making small changes helps prevent those small mistakes causing frustration. To help me, I like to run the tests automatically. My current tooling (when working purely from terminal session) is to run my tests in a separate window like this:

git ls-files -mo | entr -s 'yath'

What the above does is use entr to watch a selection of files, which it gets from git. So each time I save a file, yath gets fired and it runs my tests.

Once I have confirmed that the test file is able to open the package, no syntax errors etc. It's time to write first test, against the first example:


use Test2::V0 -target => 'Good::Strings';

subtest "Example 1" => sub {
    my @words = ( "cat", "bt", "hat", "tree" );
    my $chars = "atach";
    my $expected = 6;

    is $CLASS->sum_lengths(
        words => \@words,
        chars => $chars,
    ), $expected;

};

done_testing;

This fails as expected as there is no sum_lengths method/sub in the package. So I add that sub sum_lengths{}; this fails in a different way. This time the test calls the method successfully; but it the method itself is not returning the right answer... so I change it to sub sum_legths { return 6; }. At this point I am "green" and the code does exactly what is required for it.

It does not do any real calculations, and that is the right thing to do in a TDD exercise. Each steps you take wants to be tiny. You want to make the smallest step you can each time. Do the smallest thing that makes the tests pass; including hard-coding the return value as we don't need to do anything else to make the test pass. It seems silly; but it does a couple of good things. First teaches you not to try and solve the whole problem at once. As a professional software developer... we do that all too often. Rather than solving the small easy problem immediately in front of us; we will try and solve the bigger problems whilst solving small problems. The cognitive load is hard to manage. We constantly run the risk of over complicating things and is a recipe for disaster. So intentionally practising doing the smallest change that meets the immediate requirement is good for discouraging that trap. The other thing it helps with is catching the human errors; typos, spelling mistakes, syntax errors and the ever threatening missing semi-colon. As a developer you need to plan for human error; not pretend we can be perfect.

Having reached this point, I'd normally commit the change; grab your favourite beverage, take a moment and look at what the next requirement (in this case, example 2) and whilst hydrating it's a chance to think about how the code needs to change. Again, this is a good habit that working in a TDD manner encourages; stopping, thinking, moving forward.

So as you'd expect, I start by adding a new test:


subtest "Example 2" => sub {
    my @words = ( "hello", "world", "challenge" );
    my $chars = "welldonehopper";
    my $expected = 10;

    is $CLASS->sum_lengths(
        words => \@words,
        chars => $chars,
    ), $expected;
};

This test fails because the code always returns 6 so, now is the right time to change the function to make this test pass.

I'm going to skip over the repeated cycles I took to get to a working solution... why, because it was probably 50 small steps... and that would be dull. It included things like alternating between passing named parameters to using sub signatures (still a Perl feature I don't use as often so decided to use it for educational purposes). I decided not to explore using a CPAN module like List::MoreUtils and wrote a really ugly solution... but one that worked. Here it is:


sub sum_lengths ( $self, $words, $chars ) {
    my @chars_array = sort split '', $chars;
    my %chars_hash = map { $_ => $_ } @chars_array;

    my $char_count = 0;
    for my $word (@$words) {
        my @word_array = split '', $word;

        my $built_word;
        for my $char (@word_array) {
            $built_word .= $chars_hash{$char} if $chars_hash{$char};
        }
        $char_count += @word_array if $word eq $built_word;

    }

    return $char_count;
}

At this point I stopped and had another coffee, coming back to the screen I realised I had stopped doing good TDD. I'd written a lot of code without actually having tests supporting that beyond the larger view of if it worked. I'd used Data::Dumper and warn to check what the code was doing rather than a test. This happens a lot in "the real word", as a developer you cut corners. Mainly when you think you know what you want to do. Your intuition leads you to skipping some steps. Coming back to this code after a break, I was able to see I had "cheated".

This is part of the reason to practice TDD on non-work code. I get that chance to see the behaviour on a exercise; not on my employers production code. It's good to remind myself that I am fallible; even when intention is to do TDD... I skip to intuitive coding. No harm done and it works... but the steps went from small to large. If someone was looking at this as a merge request... they have more code to understand, there is no story of what I approached, why, etc.

This is a micro demonstration; it's good for me to see it here; much better writing this poorly here than doing it at work and making my colleagues suffer a big change that is ugly. What was/is interesting also; is that I felt tired mentally after this change (hence the coffee break). That was in part because of duration; but also because without the TDD it was one big block of concentration without the micro-breaks that come from TDD test, change, test, change, test, change cycles.

Having "solved" the problem, both the examples are passing at this stage. I have met the "business requirements". Now is the time to refactor and improve on what I know is a "sub-optimal" solution. Time to TDD this, TDD in fact makes this easier I think.

So how might we refactor this?

Well the code that build the word looks like a likely candidate, in part because a loop inside a loop is never a good look. So we can try refactoring that out into a sub, something like this:


subtest "_build_word" => sub {
    is $CLASS->_build_word(
        word => 'cat',
        chars => {
            c => 'c',
            a => 'a',
            t => 't',
        },
        ),
        'cat';
};

Then we can make the sum_lengths sub look like this:


sub sum_lengths ( $self, $words, $chars ) {
    my @chars_array = sort split '', $chars;
    my %chars_hash = map { $_ => $_ } @chars_array;

    my $char_count = 0;

    for my $word (@$words) {
        my @word_array = split '', $word;

        my $built_word = $self->_build_word(
            word => $word,
            chars => \%chars_hash,
        );

        $char_count += @word_array if $word eq $built_word;
    }

    return $char_count;
}

So that is a little cleaner; it's perhaps easier to follow that we loop around the words. Looking at this, we can see that the only reason why we split the word into an array is to tell us how many characters to add to the count. This is a idiomatic Perl trick where an array in scalar context returns the number of elements of the array. It's a bit magical... and not necessary. We can use Perl length function to achieve the same thing. Which is both shorter and probably easier for others to understand (especially non Perl natives).


sub _build_word {
    my ( $self, %params ) = @_;

    my @word_array = split '', $params{word};

    my $built_word;
    for my $char (@word_array) {
        $built_word .= $params{chars}{$char} if $params{chars}{$char};
    }

    return $built_word;
}

What is important here is that because I have tests, I know immediately that I've not broken the functionality. So I can make my refactorings with confidence. Because I have confidence in not breaking the functionality, I can experiment with the code and see if I can hone it.

This freedom also lets me test a bit more thoroughly, doing a little exploration of edge cases. Such as "what happens if we can't build any words", so I can add another test:


subtest "Edge case: unable to build any words" => sub {
    my @words = ( "hello", "world", "challenge" );
    my $chars = "xxx";
    my $expected = 0;

    is $CLASS->sum_lengths( \@words, $chars, ), $expected;
};

This test passes... BUT; we get a warning as we Perl does not like us using eq on $build_word when it is undef. Knowing this we can change our test as follows:


subtest "Edge case: unable to build any words" => sub {
    my @words = ( "hello", "world", "challenge" );
    my $chars = "xxx";
    my $expected = 0;

    my $got;
    ok no_warnings { $got = $CLASS->sum_lengths( \@words, $chars, ); };
    is $got, $expected;
};

This test fails as we would hope, so I can fix it with a simple skip in the loop if it's falsey; something like next unless $built_word; just above where we check if the built word and the target word are equal. You could of course do it other ways. :-)

So there you have it, my contribution to the Weekly Challenge. I wanted to make the effort to participate this week as I wanted to support Mohammad as he put the effort in to administer the challenge this week despite all that is going on with him at the moment. Hang in there Mohammad!!

The right tool for the job, building a tool for the IJF in Go-lang

Lance Wicks — Mon, 05 Dec 2022 12:42:33 +0000

Originally published on: The right tool for the job, building a tool for the IJF in Go-lang

Recently I started building a new tool. It sends UDP data packets that the International Judo Federation (IJF) TV graphics software(s) listen for and present on screen. Normally these packets are sent by the four scoreboard systems at the event.

The reason for the tool, is that to test the graphics software you need packets of data.
That means running four computers with the scoreboard software running. This is inconvenient and also requires manual operation of the scoreboards. It’s the right way to test at an event where you care that those specific laptops can communicate with the wider systems… but for bug investigation or even regression testing after making changes it is not very convenient.

I originally started in Perl, which is my preferred language for most things.
But to be frank… it did not feel comfortable. The other feeling I had was that I wanted to share this tool with people; and Perl can be difficult to make portable for non-techies. I thought of NodeJS (which the main live stream graphics is written in), but again it is difficult to make portable at times.

So I was looking for a language with good network libraries that is portable… and I felt I could make a working prototype in.

Go has networking in the standard libraries, so no need to find a third party module. Go also compiles to a single executable; and building a Windows exe from my Linux workstation was/is easy to do. Lastly, I’ve written Go and have friends who know it well, so I felt I had a good tool.

So one Saturday morning, coffee in hand; I started.

Thankfully, people have walked this path before me. I was able to find a blog from Mihalis Tsoukalos that got me started.
With this as my guide I was quickly able to build a server and client pair, and using go run server.go was able to quickly capture the contents of the packets coming from the scoreboard software.

Using these packets as my template, I was able to alter the “client” code to send a packet out; and after minimal adjustments, saw the packet data display on the graphics system we use to overlay on the video people watch online.

Having proven this much, I tested building a Windows (32-bit) executable and ran that on a windows machine off a USB stick with no problems at all.

Missions accomplished !

So then I made some adjustments, such as sending multiple mat information and tweaking the data packets. I refactored some of the IJF specifics into a module so that the main code is pretty small:

package main

import (
    "fmt"
    "ijf/ijf"
    "net"
)

func main() {
    s, _ := net.ResolveUDPAddr("udp4", "255.255.255.255:5000")
    c, _ := net.DialUDP("udp4", nil, s)


    fmt.Printf("Broadcasting UDP to %s\n", c.RemoteAddr().String())
    defer c.Close()

    for {
        for i := 1; i < 5; i++ {
            packet := ijf.Mat(i)
            data := []byte(packet)
            _, _ = c.Write(data)
        }
    }
}

As you see, there is simple infinite loop, within which it loops around creating and sending packets for each of the contest mats.

(And yes, I have cut out all the error checking of err to make it even smaller for this post)

The windows EXE file is now with my colleague who will test it in anger next weekend in Tbilisi, Georgia.

Next?

I have started on a change to read a config file and send a specific scoreboard situation; and after Tbilisi I’ll be adding the ability to read the data file we capture of a scoreboard over an event and play that back. This will be helpful as a debugging tool when people tell us something rendered strangely. We should be able to use the exact data from the scoreboard(s) and replay it so that the TV graphics software can be observed, altered, and tested.

The weekend project was really rewarding and reminded me of how important it is to choose the right tool for the job (Go-lang). Not just the tool I know best (Perl)… or even the tool that has been used before (Javascript/Node).

Writing a Dist::Zilla test plugin

Lance Wicks — Wed, 02 Nov 2022 00:00:00 +0000

Recently I wrote a small Dist::Zilla plugin to help with the problem of dependencies being old in modules I author.

I use update-cpanfile to check dependencies in projects I write using a cpanfile (and carton); I wanted a tool to do the same thing with libraries I write.

It was pretty simple to do over a couple of evenings; and worked really well; a trial version is on Cpan now at Dist::Zilla::Plugin::Test::Prereqs::Latest.

Using it is really simple you just add [Test::Prereqs::Latest] in your dist.ini and then when you run dzil test it will add and run a test file (xt/author/prereqs-latest.t) which checks the version of each module in the [Prereqs] secion of the dist.ini and fail if the version you have specified is lower than the latest CPAN.

The code is pretty simple, using existing work by HITODE whose update-cpanfile I use regularly to help keep dependencies up to date in projects I have that use cpanfile and carton.


use strict;
use warnings;

use App::UpdateCPANfile::PackageDetails;
use Dist::Zilla::Util::ParsePrereqsFromDistIni qw(parse_prereqs_from_dist_ini);
use Test::More;

my $prereqs = parse_prereqs_from_dist_ini(path => 'dist.ini');
my $checker = App::UpdateCPANfile::PackageDetails->new;

for my $key (sort keys %$prereqs) {
    for my $req (sort keys %{$prereqs->{$key}->{requires}}) {
        my $current_version = $prereqs->{$key}->{requires}->{$req};
        $current_version =~ s/v//g;
        my $latest_version = $checker->latest_version_for_package($req) || '0';
        my $out_of_date = ($latest_version <= $current_version);

        ok( $out_of_date,"$req: Current:$current_version, Latest:$latest_version");
    }
}

It's simplistic, but works well. If the version in the dist.ini is lower than the version on CPAN it fails the test and tells you as much.

Once written, I was easily able to test it against a couple of modules I maintain by installing the package with dzil install then in the other module I add [Test::Prereqs::Latest] to the dist.ini and ran dzil test and it worked.

Once I had done basic testing locally; was able to create a trial release and upload to CPAN with dizil release --trial which built and uploaded the distribution.

Of course CPAN is amazing, so shortly afterwards the cpan testers started discovering the module and testing that it built on a variety of versions of Perl and a variety of platforms. People love GitHub actions but cpan testeers was first and I did literally nothing to get all this amazing testing without any configuration work, nothing... it just happens. It's amazing, seriously amazing.

The module is not ready for a proper release, but it's been nice to "scratch my own itch" so to speak.

Tags: perl cpam distzilla

Wrapping a JSON API to access your personal data

Lance Wicks — Tue, 15 Mar 2022 00:00:00 +0000

In my last tale I outlined how I used Dist::Zilla to do much of the heavy lifting for packaging and releasing a Perl module to CPAN. In this tale I want to outline how quick and easy it was to write that module; wrapping a proprietary undocumented API so that I could easily use my personal data in ways the company did/does not provide.

The module in question is WebService::SmartRow. SmartRow is a physical device and app that provides power and other performance data from a indoor rowing machine (specifically the WaterRower) machines.

It is a bluetooth device, that sends data to a web service via a mobile application. This data is then accessed via the app, or the website. The data includes things like the power in watts, stokes per minute, heart rate, etc.

The specific data I wanted to see, was/is how my average power changes over time. Specifically, across a series of 5000 meter rows, am I generating more power over time. Also I wanted to see my personal best times, over time. I.e. how much better was this 5000m over my previous best.

The API

SmartRow do not provide a developer API (at the time of writing, and as far as I was able to see). What they do have is a website that is mainly driven by JavaScript and a JSON API. As logged in user my browser makes calls to the API and then creates pretty HTML from the JSON returned.

The API is really quite tidy and easy to understand. Which is nice! I have previous experience wrapping JSON APIs and it can be really ugly; thankfully the SmartRow team have made a clean, easy to understand API.

The API relies on you being logged in, but quick exploration showed that basic auth is possible on the API endpoint. Meaning I did not need to worry about OAuth or keys and so forth.

The web requests

Knowing that I could get away with basic auth, that meant that the code I needed could work simply by including the username and password in the URL I make a HTTPS request to.

I turned to HTTP::Tiny and was quickly able to craft a get request that looked a little like this:

    my $response = HTTP::Tiny->new->request( 'GET',
              'https://'
            . $user . ':'
            . $pass . '@'
            . $api_url
            . '/public-game'
    );

Then (after some basic tests to make sure I got a 200), I could parse the $response->{content} from JSON to a Perl data structure using Cpanel::JSON::XS.

This gave me a data structure looking like this:

  [
    {
    "accessory_mac" => E,
    "account" => E,
    "ave_bpm" => E,
    "ave_power" => E,
    "calc_ave_split" => E,
    "calc_avg_stroke_rate" => E,
    "calc_avg_stroke_work" => E,
    "calories" => E,
    "confirmed" => E,
    "created" => E,
    "curve" => E,
    "device_mac" => E,
    "distance" => E,
    "elapsed_seconds" => E,
    "extra_millies" => E,
    "id" => E,
    "mod" => E,
    "option" => E,
    "option_distance" => E,
    "option_time" => E,
    "p_ave" => E,
    "protocol_version" => E,
    "public_id" => E,
    "race" => E,
    "strava_id" => E,
    "stroke_count" => E,
    "time" => E,
    "user_age" => E,
    "user_max_hr" => E,
    "user_weight" => E,
    "watt_kg" => E,
    "watt_per_beat" => E,
    },
    ...
  ]

(The above code is actually taken from the Test2::V0 .t file I wrote to confirm the structure.)

So you can see it's pretty easy to understand, the keys are all mainly understandable. In my case I wanted the p_ave and distance, so I could filter on 5000 meters and build an array of all the average power values.

Module creation

At first this could have been a simple script, but I wanted to make this something portable and usable by anyone wanting to work with their personal data.

So after I proved the approach would work, I started a module (using dzil new WebService::SmartRow). This is currently a single file with little refinement.

I used Moo for some simple OOPness and structure. This allowed me to specify the attributes I want:

has username => ( is => 'ro', required => 1 );
has password => ( is => 'ro', required => 1 );

There are pretty self-explanatory, to use the API you need those, so add them as required attributes.

Next I added an http attribute:

has http => (
    is => 'ro',
    default => sub {
        return HTTP::Tiny->new();
    },
);

The default here creates a HTTP::Tiny object which I can later use in methods via $self, which meant my earlier get request changes to look like this:

    my $response = $self->http->request( 'GET',
              'https://'
            . $user . ':'
            . $pass . '@'
            . $api_url
            . '/public-game'
    );

You can set your own http attribute when creating via WebService::SmartRow->new() so if you need to do something like change the user agent, or have a preferred module, you can inject it easily (assuming the methods match HTTP::Tiny).

Testing

Currently the module is pretty simple, three attributes and 4 public methods. The module has little smarts so the t directory is pretty spartan as the object is pretty simple.

I am using the xt directory to hold tests that talk to the API and as such require an internet connection and credentials.

Not wanting to include my personal credentials in the repo, I have a private sub in the class that gets the username and password from environment variables. Which is good as it means I can commit my tests, and if someone using this module does not need to commit their credentials in code either.

Perl makes the environment variables easy to work with, so the small sub that handles it looks like this:

sub _credentials_via_env {
    my $self = shift;

    my $user = $self->username || $ENV{SMARTROW_USERNAME};

    my $pass = $self->password || $ENV{SMARTROW_PASSWORD};

    return ( $user, $pass );
}

So if you have instantiated the module with username or password it will use those. If they are not present it will use SMARTROW_USERNAME or SMARTROW_PASSWORD.

Then (and I know I can make this a bit smarter) my get_workouts() method, has a call to my ( $user, $pass ) = $self->_credentials_via_env; prior to calling the URL.

This means I can run my tests like this:

SMARTROW_USERNAME=yyyyyyy SMARTROW_PASSWORD=xxxxxxx prove -lvr xt/

And it will go and connect to the API and execute tests for me by getting real data from the API.

Earlier I mentioned I am using Test2::V0 for tests, so in xt I have a growing collection of files that primarily confirm the structure of the data returned from the API.

Mainly they use E() to prove hash element exists, some could/should/do test the content. For example in one test I just wanted to confirm that the data and distribution fields are arrays before later testing the content of those arrays. So I have a test like this:

is $leaderboard, {
    distribution => {
        ageMode => E,
        data => array {
            etc(),
        },
        max => E,
        mean => E,
        min => E,
        range => E,
        userPercentile => E,
    },
    id => E,
    mod => E,
    records => array {
        etc(),
    },
    },
    'Leaderboard structure is as expected';

It is mainly E() checking that is exists, then array to confirm it contains an array. etc() tell the testing code to assume some rows, but not to check the content of them. They just have to exist. As etc() is the only thing in there, as long at data and records contain arrays with some rows, the tests pass.

Having tests like this is really helpful when wrapping someone elses API previous pain has taught me. If the structure of the return data changes, I can easily confirm what has changed.

When you are wrapping an API this way it is an inevitability that the API will change so having the tests is one of the only ways to ensure you can maintain it over time. My Webservice::Judobase has had this happen a multitude of times in the past 5 or so years.

Summary

As you can see from the brevity of this tale, wrapping an API is pretty easy to to. I know from previous experience that doing it helps others build things with Perl. So it's the sort of project that helps both the service and our language.

Perl is a famous "glue code" language and this is a great example of why. Two CPAN modules do almost all the heavy lifting and our fantastic test tooling means I was easily able to write tests that will make it easy to maintain.

Dist::Zilla (as per previous post) I use to automate most of the publishing work. I will write something up about that another day. It has lots of plugins that make sure your module is up to scratch.

The JSON API and JavaScript front-end trend has meant that lots of websites have a JSON endpoint that you could wrap. This means you could as I have create a tool that uses your data in a way that the website may never build for you.

It also gives me confidence that I could pull my personal data out of the service and store it in my backups (I am doing that next), so that if the company ever goes bust, I have my data there and have it backed up securely and independently. I can do with it what I please as it is just data on my disk(s), either as plain files or maybe pushed into another database.

If you use a website and there is no module for it on CPAN, maybe it has an API you can wrap too?

Tags: perl projects cpan

Using Dist::Zilla to create a new CPAN module

Lance Wicks — Sat, 05 Mar 2022 00:00:00 +0000

Recently, I posted about some tools that are perhaps not as well known as they should be. I asked on Twitter (from @perlkiwi) for suggetsions and one was distzilla.

I didn't include it at the time in part because I knew I wanted to mint a new CPAN module and planned on using distzilla to do it, so this tale is going to cover how it works and how it helped me put a module together.

What is Dist::Zilla

Dist::Zilla (aka distzilla aka dzil) is a command line tool to help you create and perhaps more importantly maintain packages you intend to share via CPAN.

It really does help ensure you create a good package and make it easy to maintain that module over time. Perl people know the importance of maintaining legacy code; so dzil is really valuable.

Getting started

If you have never tried distzilla, then do pop over to the main website dzil.org which has a fabulous "Choose your own adventure" way of introducing you to the tool. I really like the site and LOVE a different approach to doing documentation.

In short... you start by typing something like dzil new WebService::SmartRow, which will create a new directory for you with a dist.ini a base lib/WebService/SmartRow.pm file and a few other bits and pieces.

From here, I started a new repo on github and added the "remote" to the created directory (git init then git remote add origin git@github.com:lancew/WebService-SmartRow.git) after which I could happily git add . then commit and push the changes.

Writing the code

The module I was writing is a small wrapper around a JSON API endpoint from https://smartrow.fit/. I just wanted to be able to access the workout data so I could munge the data in a few different ways and create some charts for myself.

The module itself is pretty simple. I used Moo out of habit and HTTP::Tiny for the network part and Cpanel::JSON::XS for the JSON handling.

These are specified in the dist.ini file as prerequisites:

[Prereqs]
perl = v5.06.0 ;
Cpanel::JSON::XS = 4.27 ;
HTTP::Tiny = 0.080 ;
Moo = 2.005004 ;
namespace::clean = 0.27 ;

[Prereqs / TestRequires]
Test2::V0 = 0.000145 ;

You can see I have two sets of dependencies, one for the app itself and one for testing dependencies for installation. I use Test2::V0 so I pop it in there.

Uploading to CPAN

After having written the code and tests I wanted to upload to CPAN, this I can do with distzilla via the helpful dzil release --trial command; which as I have already used dzil to upload modules before worked forst time. :-)

All you need to know is you pause credentials.

Dist::Zilla plugins for the win!

Having uploaded the package as trial, it was a good time to use distzilla to fine tune my module. This is super easy via the large array of plugins (and plugin bundles). The biggest problem is choosing them and dealing with the odd conflict.

A good one to start with is @TestingMania bundle includes a lot of really helpful tetsing tools. PerlTidy, PerlCritic, PodCoverage and more intricate ones to test your META info. WHich is important if the package is going to CPAN.

Add the bundle is easy, just add [@TestingMania] into your dist.ini and run dzil test. If the dependencies you need are not there distzilla will prompt you with the command you need which is dzil authordeps --missing | cpanm.

Git and GitHub integration

I am using GitHub to host the code for this module, so I added a few plugins:

GitHub::Meta

This plugin includes things like the repository url and issue tracker in the META for the package.

By inlcuding the plugin (and META infor), metacpan will know that the repo is on GitHub and present it in the UI. It also tells metacpan that I am using the issue tracker on GitHub and this prevents it from defaulting to RT.

@git

The @Git bundle is helpful for ensuring a few Git related things are done correctly. I.e. that the repo is "clean" before you release. It also does a git tag automatically on release and pushes it to GitHub for me automatically.

This bundle did push me to update my .gitignore file to ignore the WebService-SmartRow-* files and directory that Dist::Zilla builds.

Git::NextVersion

The last of the Git plugins is my favourite, thos one automatically bumps the version when I do a release which meant I could remove the version = 0.001 from the dist.ini as the plugin increments based on the tags in the repo. It's cool, I like it.

Changes

I am terrible at updating teh Changes file in repositories, so I added these two plugins:

Test::ChangesHasContent

This plugin creates a .t file that confirms you have content in the Changes file matching the next release.

CheckChangesHasContent

This plugin is similar, in that it actually prevents you from releasing if you have not got content for the new release in the changes file.

I use both, as I like the warning early via dzil test that the first plugin gives me and the second is there mainly in case I ever removethe first I suppose. Maybe I could should delete it?

The Bundles gotcha

The plugin bundles are awesome, with one exception for new dzil users. They can cause conflicts where the same plugin is referenced twice. This is not too scary once you have seen it a couple of times; but can be a stress when it hits you the first time. Just read the message and try and understand which two bundles are conflicting. You can (at least I could in @TestingMania) disable a plugin and that can solve the issue for you (I just added disable = Test::Version ; for example immediately after the [@TestingMania] in dist.ini; YMMV).

dzil as tool.

dzil is really helpful. dzil test and dzil test --release ensures my module is in a pretty good state.

dzil build (and dzil clean) are handy for building the module so I can look at what will end up on CPAN.

dzil install installs the module like a normal cpan module; so I could use the module in a script elsewhere on my machine as if I had downloaded it from CPAN. Given this module is mainly about accessing a JSON API; this is a helpful feature to save uploading to cpan and then installing from cpan more than I need to.

Summary

Dist::Zilla can be a little overwhelming, but I find it pretty intuitive if you take small steps. I added each plugin one at a time as that helps keep the confusion to a minimum. Bundles are great; but can have conflicts. It is by it's own definition "maximum overkill"; but it's easy overkill so give it a try.

Let me know how you get on.

Lance (perl.kiwi).

Tags: distzilla cpan cpants kwalitee

Perl tooling - less well known but powerful

Lance Wicks — Thu, 24 Feb 2022 00:00:00 +0000

Perl is a mature language, businesses have been relying on it for decades, a common refrain is that Perl is battle tested. One of the strengths of a mature language is that the community builds tooling around the language. Perl has some well known tools and some less well known ones. In this tale I want to talk about some of the tooling I like and perhaps are less well known.

Having a good set of tools helps developers build software better and more easily.

Well known tools

Here are some of the netter known tools that many Perl developers use everyday.

Perl::Tidy

Consistent formatting is surprisingly important and time consuming. Especially in a team environment running Perl::Tidy is helpful for visually formatting the code so that no matter who writes the changes they are formatted consistently. Be that tabs vs. spaces or aligning the "=" signs in variable assignments.

Perltidy is similar to gofmt (GoLang), prettifier (Node, etc) or elm-format (Elm).

Perl::Critic

perlcritic is a static analysis tool, it is a very flexible and configurable tool that allows you to define and detect rules on things you don't want in your code base.

For example, a common rule is that all sub routines need an explicit return; perlctitic can enforce this across your code base. There are over 100 "policies" on CPAN for perlcritic. Another one Perl::Critic::Policy::logicLAB::RequireParamsValidate insists that the Params::Validate module is used to validate all parameters to your subroutines.

Perl::Critic::Policy::Variables::ProhibitUnusedVarsStricter prevents you defining a variable that is never used. This is a pretty common thing to find in a large code base, either because the variable is no longer used or there is a typo somewhere.

prove

Perl has a strong and well developed testing culture and tooling. prove is the test runner most developers are used to using.

There are a wide selection of Test:: modules that you can use for mocking, unit testing, BDD, even enforcing Perl::Critic policies.

perlbrew

Perlbrew is relatively well known, though perhaps less often used than it might deserve to be.

Perlbrew allows an easy way to install various versions of Perl and switch between them. It is often used on your development machine especially if you need to work on specific versions of Perl so you can support legacy applications. Increasingly we are seeing it used in the building of Docker container.

Less well known tools

carton

If you have written any node you will understand carton. With carton you describe your module dependencies in a file called cpanfile and then install them with carton install This installs all the modules in a ./local directory and you can then run the perl application with carton exec and it runs the code with all the dependencies in the correct path.

This is particularly helpful for when you have multiple projects and they use differing versions of the same dependencies.

Because I use carton with almost all my projects now, I have the following two aliases setup:

perl='carton exec perl'

prove='carton exec prove'

These mean that if I forget to type carton exec perl some_script.pl and type perl some_script.pl it works as expected using the local dependencies. The prove alias is handy as I almost never remember to type carti exec prove.

update-cpanfile

This command-line tool is really helpful for maintenance of your dependencies. When you run it with update-cpanfile pin it will pin your dependencies in the cpanfile. carton install will then install the specific versions in the file. This keeps you dependencies consistent; but you could do that by hand in the cpanfile.

Where cpanfile-update is really helpful is when you run it with update-cpanfile update, this will make the tool check cpan.org for the latest version of modules and will update the cpanfile for you with the new version.

If you maintain a variety of projects or have lots of dependencies update-cpanfile is a real time saver.

yath

The simplest way to describe yath is to say that yath is the new prove... but better.

The venerable prove has been used for a long time, yath is relatively new (2016) and can be used pretty much to do all the things prove does.

However, it's able to do some really interesting things such as running in a daemon mode saving startup times... give it a try.

perlvars and perlimports

These two tools are super handy for analyzing your code base to ensure that some undesirable code smells.

perlvars identifies unused variables in your code.

perlimports identifies unused modules you may have included in your code but are not using.

Removing unused variables and modules helps keep your code "tidy" and improve memory consumption and protect against such things as methods from an unused module causing issues when another module has a method with same name for example.

What am I missing?

This is just a short list of tools I wanted to mention, in part to invite people to let me know what they use.

So if you have some tools you use all the time, drop me an email or a tweet.

Tags: perl tools

Building a simple RSS aggregator with Perl and Mojolicious to replace a PHP app

Lance Wicks — Tue, 23 Nov 2021 00:00:00 +0000

In the last tale I shared a little of my experience of using the Digital Ocean app services for build, deploy and hosting of a simple web application.

In this tale, I'd like to summarise the small application I built.

Introduction

The application I am replacing is a old (early 2000s) PHP application. It aggregates RSS feeds from a variety of Judo blogs and produces both a web presentation and a new RSS feed of the combined RSS items. The old site has limped along with little modification since early 2010s. It did well untill the hosting platform it was on (cPanel) upgraded PHP and broke the site.

There were two specific hacks that both helped and hindered. The web pages were being writen to static HTML files by a cron job that ran the PHP. To address multi-ligual need, the site had three installations of the same code running against different OPML files of sites. This actually menat the site stayed "up" when the PHP upgrade broke the app.

New Version

I decided to build the site using Mojolicious. I've done a lot of work with Dancer2 in the past, so this was a good opportunity to build something real with Mojo.

Making the site properly multi-lingual was high on my agenda. Fast also an important factor for me.

Starting

I started by using the Mojolicious command line tool to scaffold a working application:

mojo generate app

This gave me the shape of the app; Mojo/Mojolicious gives a pretty standard MVC web application structure to work with. As someone familiar with Dancer2 it's not difficult to adjust. All your regular tools for handling GET and POST requests are there already. Mojo also comes with Morbo.

Morbo

Morbo is a development server that handles HTTP and has an almost mandatory feature for me... hot reloading. When you change some code, Morbo identifies that and reloads. This is pretty common in other languages and it's nice to have it baked into Mojo.

Test::Mojo

Mojo comes with Test::Mojo which is a really nice library to help you test the WebUI elements of your application. This is really nice and works well. I like to try and build things in a Test Driven Development style. Both in terms of unit tests AND integration style tests. Test::Mojo makes it easy to describe the web page I want to build and then confirm I have done it as I build it.

It does not replace unit testing for business logic (I still use Test2::V0) for that. Test::Mojo provides a ovely out of the box, pre-wired web page testing tool. The ease of use as a developer is important to me and frankly makes doing a more TDD style of development more of a default option than if you have to wire things together yourself.

XML

This was actually the hardest part of the puzzle. There are many options and in the end I have been using XML::OPML to parse the lists of sites in each language. XML::Feed handles the reading and and writing of the RSS feeds. Beyond this, I use HTML::Strip to clean the content from the external feeds along with Text::Truncate to limit the text shown for each item shown on screen. Lastly, I am using Moo for forming the object oriented "business logic" part of the code.

SInce starting (and in part from writing this tale) I see that Mojolicious has "JSON and HTML/XML parser with CSS selector support." so maybe I don't need all this? I shallneed to explore.

Code structure

The site is pretty simple at the moment, I have broken it into two parts. Feed fetching, parsing, aggregating is one part. Starting with reading an OPML file and culminating in writing an RSS file.

The second part is the website that reads the RSS file and presents it on screen. Currently, the website does not fetch the feeds from the external sites. This is being done locally on my machine as a stand alone script; this will probably change; though it does mean that the site is very simple currently.

Routes

The code is simple easy:


 $r->get('/')->to('Home#welcome');
 $r->get('/:lang')->to('Main#index');

The whole site is basically one route that is the language we are serving i.e. /english, /french or /spanish. The :lang is used within the code to decide which RSS file to read and display. So /english reads rss_english.xml and displays it. Currently the controller looks like this:


sub index ($self) {
    my $xml
        = XML::Feed->parse( 'public/rss_' . $self->param('lang') . '.xml' );
    my $opml_parser = XML::OPML->new;
    my $opml
        = $opml_parser->parse( 'public/' . $self->param('lang') . '.opml' );

    # Rendel template "example/welcome.html.ep" with message
    $self->render(
        lang => ucfirst $self->param('lang'),
        rss_xml => $xml,
        opml => $opml,
    );
}

As you can see, we read the OPML file (to show a list of sites down one side of the page) and the aggregated RSS feed for each item from the sites on the list.

Hidden complexity in the template

The conroller is simple, which is in part becuase I have hidden a large about of code in the template itself. The template engine is complex enough to allow me to generate loops, strip HTML, truncate etc. This is convenient for me, but feels very wrong.

Bootstrap

There is a little Bootstrap used to apply some design.

Summary

So for me Mojolisious has been enjoyable; well worth a try if you are looking for a developer focussed web framework; no matter what language you are coming from. The tooling (Morbo and Test::Mojo) make for an easy and reliable development process. The app structure is familiar to someone coming from most any MVC framework (be that Dancer2, Ruby on Rails, Django or Express.

Give it a try.

Using Digital Ocean App service to run a Perl Mojolicious docker app

Lance Wicks — Sun, 21 Nov 2021 00:00:00 +0000

Recently I needed to renovate an old web project; RSS aggregator site written in PHP in early 2000s which after not being maintained for about a decade had finally died.

It is now a new Perl Mojolicious website, moved from the cPanel hosting onto Digital Ocean's docker "App" service. In this post I'd like to share some of the learnings and experiences of moving the app to Digital Ocean's service.

I am not going to cover in this tale/post the code side of the Mojo app, just the infrastructure side. Mainly because I have been really impressed with the ease and usefulness of the Digital Ocean service.

Please note, I am NOT sponsored, or paid by Digital Ocean. This is just a service I have used and enjoyed.

Overview

I write changes in git | -> Push to GitHub | -> Digital Ocean picks up the git change | -> Digital Ocean builds the docker image | -> Digital Ocean runs the image (including HTTP).

More detailed explanation:

I used the basic Docker file that Mojo generates, with a couple of small modifications:

FROM perl
WORKDIR /opt/mojo-planetjudo
COPY . .
# XML::LibXML needs to be there before we
# try to install SimpleObject, Parser.
# TODO: See if parser needed here
# TODO: See why cpanm does not sort this out
RUN cpanm install -n XML::LibXML
RUN cpanm install -n XML::Parser
RUN cpanm install -n XML::SimpleObject
RUN cpanm --installdeps -n .
EXPOSE 3000
CMD ./script/planetjudo prefork

You can see that it's pretty basic. The only oddity is that I install a couple of modules before installing the rest via a cpanfile. That's just because they were not installing nicely for some reason. ;-)

The other thing I do is have a .dockerignore file, it only has two entries:


.git
local

This prevents Docker copying in the .git and local directories. .git is pretty self explanatory you don't need it. local is where cpanm has installed my Perl modules and that can cause an issue I struck when it tried to build modules and failed.

And that's about it!

Developer Workflow

The really nice thing with this setup is that my entire workflow consists of git add, commit, push.

After that Digital Ocean takes care of everything, the build/deploy is all done for me. It is currently a bit slower than I'd like (mainly cpanm, so cpm might be faster). But I love that I don't need to think about anything.

Niceties

Digital Ocean's app service takes care of not only the build, deploy and hosting of the app. It also provides console logs, graphs of things like memory and CPU usage and even a terminal I can use to interact with the app.

Cost

The smallest instance you can run is $5usd per month, which is equivalent to their smallest "droplet" (virtual machine). If, like me it's a low traffic, resources light, site; then it's a great price to pay for a simple, reliable solution.

Summary

Digital Ocean's app service is great for a Perl web developer. With virtually no effort we can get a Perl Mojolicious dockerised application up and running. No need to worry about pipelines or even SSL, it's all just taken care of for you. It does put you firmly in the "vendor lock-in" situation; but it's a pleasant trap to be in. We as Perl developers get to host our applications on a new service.

As a developer who works in Perl (as well as other languages), it's great to have tried this and felt it work really smoothly. It's sometimes not the case; sometimes you try a new approach or tool and Perl is not practical on it.

The only build and deploy step I need to make is git push which is phenomenal. I didn't have to worry about LetsEncrypt or anything like that. Everything "just worked".

Give it a try fellow Perl devs, it's nice.

Renovating a CGI app talk

Lance Wicks — Fri, 09 Jul 2021 00:00:00 +0000

Recently I was invited to give a tech talk at the Southampton Perl Mongers group online event. It was great to spend some time with local and not so local (Hello to our new friends in Texas!) Perl users.

The talk was an abbreviated version of the talk I had planned to give at the German Perl Workshop 2021 but was unable to do. In this post I want to share the content and some bonus content not included that I should have included probably.

Here is the video of the talk I made after the event:

Overview

The talk was some experiences I've had mainly renovating an old Perl CGI app, I wrote in the late 1990s and early 2000's which was parked gathering dust till Christmas time 2020.

The 3 steps I describes were: * Get it working * Tidy up * Modernise

Followed by some learnings and some advice.

Get it working

This seems obvious, but made a big difference for me. Mixing making it work and improvements is a mistake I think. It's tempting but just getting it to work will more than likely force changes in the code. Couple those changes with tidying up and you are more likely to break things.

One of the big challenges I found was the (re)learning of how the code works. What ideas and approaches were in use at the time. Just "getting it working" provides opportunity to learn the general shape of the code and what it depends on to work. Be that old CPAN modules, environment variable, files on disk or databases (and specific database versions).

You will have to change code even if you are just trying to get a working local version.

Getting it working TWICE is super valuable too. So both a local dev setup and a fresh server somewhere. This doing it twice helps find things that are not obvious. For me the differences from a local ArchLinux machine and a Ubuntu Server helped me identify differences.

In the talk and afterwards in the discussion, the idea of writing down what you learn as you go along was covered. This is skill and habit well worth developing if you are working on legacy code... or new code in fact. I kept notes on blog articles I read and followed. I scribbled notes on how to start/stop things. I absolutely had moments where I did something on Friday and come Monday had forgotten what I had done, why or how I came to the information I used to influence what I tried. WRITE IT DOWN... it's worth it!

Plack::App::CGIBin

Specifically for me, I was working with an old set of .cgi files. There were previously running on a server under Apache. Locally, I did not have; nor want that overhead. So I needed a tool called Plack::App::CGIBin which allows you to simply point it at a directory of .cgi files and serve it as a plack app via plackup. This was essential for my local development setup so I could see the working app once again.

At this stage the home page loaded nicely but not a lot else as CPAN modules were not on my system.

Carton and cpanfile

Managing the dependencies the app had was important (especially with an older application, where breaking changes can often appear). My choice here is to use the tool carton which reads a cpanfile and installs the CPAN modules locally (into a directory called local in the working directory). Then you can run with these specific CPAN modules using carton exec.

When I was trying to get the application up and running, I had some issues and it was really helpful to have two copies of the source code in different directories and be able to use carton to run different versions of the same CPAN modules. This helped me identify breaking changes. Not having to rely (or mess with) system wide CPAN modules was/is really valuable.

Hard coding and deleting

As I got to understand the code better when getting it working; it became clear that some things were not worth retaining. So the delete key was a really effective method to get the application working. The other trick that I used was to simplify the problem by hard coding some variable that were originally designed to more flexible but generated complexity.

Deleting code and hard coding things helped get the app to the state that it "worked" again. It was not 100% functionality restored; that in itself was a great learning experience. It's easy to think that all the features the legacy code had in place matter and/or worked.

Tidy up

Once the application was working, the next phase was to tidy the code in advance of planned modernisation.

I feel it's valuable to separate the tidy up from the modernisation. I find and found that the act of tidying up involves some modernisation anyway. What I mean here is that knew at this stage that I wanted to replace the data handling part of the code; but decided that I wanted to tidy up first as I knew that it would involve some changes to the code AND some collateral modernisation. If I tried to modernise during or while cleaning up I think it makes the task more difficult.

Perltidy

I suspect most Perl developers have used Perltidy and are familiar with the way it formats the source code in a consistent manner. When picking up a legacy code base it's really helpful to find a moment to perltidy everything. This will make the existing code look more familiar (especially for those of use using Perltidy a lot and are used to the style choices). I tend not to customise the perltidy setting too much leaving it pretty much on defaults.

Perlcritic

Another well used tool, perlcritic allows you to automate some stylistic decisions that are regarded and "Best Practices". Specifically, "Perl Best Practices" the book standards. I am not saying that all the PBP standards are ones I follow but I appreciate the standardisation it offers. It is a wonderful tool to help shape legacy codebase. It did help me identify some common things to improve (double to triple arg file opens for example).

WebService::SQLFormat

SQL is a language in itself, so like the Perl I think it's valuable to have some consistent formatting of the SQL in the app. I didn't use a ORM when the app was first written and broadly find the benefits of writing SQL outweigh the benefits of an ORM... your mileage may vary.

I started out using a website and copy and pasting SQL back and forward. Later I identified the WebService::SQLFormat module and was able to write a small script that would format my SQL in a more automated fashion.

As with Perltidy I don't necessarily agree/like all the formatting choices it makes. But I value the consistency and ease to automate more than my aesthetic preferences.

Human Eye

A trick I applied to this code is the simple trick of zooming out my editor so that the code is tiny and I can see the "shape" of the code.

It's a remarkably effective way of "seeing" code smells. The easiest to describe is complexity added by layers of loops or if statements. You can easily see multiple levels of indentation and see that something there is complex. This is helped by having previously having run Perltidy; so do that first.

You also see large/long subs and are able to see dense code.

Zooming in on the areas that from "30,000 feet" look wrong, you can make quick gains by tackling these areas, then zooming back out to find what else looks problematic.

I did use some other tools to help with this, like trying to see cyclomatic complexity or file size. Frankly though helpful, the human eye and mind is exceptionally good at seeing patterns and I got more benefit from this trick than the tools.

Modernise

Having tidied up the code I was in a position to modernise. Starting with introducing Dancer2 as my web framework (Mojo/Catalyst might have been your choice... I went with Dancer2 as it's a tool I know well).

Having created the basics, the next stage was a cut and paste exercise of moving code out of CGIs and into routes. I was fortunate that I had used HTML templates originally so was saved the pain of breaking the HTML out of the code... many of use have been there and it's not fun.

Database change

The original code used a module called DBD::Anydata which was (it's deprecated now) handy at the time. I used it to read and write from CSV files using SQL statements. Yes really. It was a mad decision to use CSV files as the data store for the app; but in terms of modernisation it was fortunate as it meant that I'd not had written code to read/write data to files. I had written SQL inserts and selects, which meant that it was comparatively easy to migrate the app to Postgres.

I did need to write the schema creation etc.

Database Migrations

The schema was "OK" but not perfect and as I tidied more and had to make more changes I became annoyed with destroying and recreating the database each time. I explored using migration tools like sqitch XXX, but ended up quickly writing a migration tool in the admin area of the app that applied SQL statements in numerical order from a directory of .sql files. (With a migration level being stored in the database to prevent re-applying the same changes. Not sophisticated... but it works and was simplest solution at the time.

Docker

Initially I had a local installation of Postgres, but I work from multiple machines and quickly moved to using a dockerised installation of Postgres to simplify my development cycles.

Following this I added a Perl container to run the app itself.

Learnings

In preparing and giving the talk I was struck by how much of what mattered to me was not "technical" but "human" parts.

Write it down This ended up being more important pretty much anything else. Having notes on what I was doing and why proved important when I took breaks from the code and came back to it. When I did poorly I would come back and not recall what specific things I was trying to do and why. This was really prevalent in the Postgres changes. I am not a docker guru and followed several guides; at least once I came back the next day and got lost as I did not take notes on what article I had read and what it taught me.
Automate it Perltidy, Perlcritic, etc. The more I was able to automate the easier it became to use the tools and to remain consistent. This is true also of deploying the code. A "pipeline" be it Github actions or bash script makes life so much easier and the easier the process the better. My mind could stay of the code and the problems not on how did I get this deployed etc. So take those moments to automate things.
Do the simple things first It's tempting to get stuck in and tackle hard problems at the start. I think this is a mistake. By starting with small simple things you gain familiarity with the code and the "domain". You discover the hidden complexity before you get deep into hard problems. This way when you get into the hard problems, your familiarity is high and you've discovered many of the complexities already. So do those simple things; it's why we often give the new member to a development team the simple update the template typo ticket right. Simple tasks mean you run through all the steps sooner and resolve the things that are not obvious.
Legacy code is a great place There is enjoyment to be had in an older code base. Unlike a writing from new, a legacy code base is nuanced. You often have multiple ideas spanning the code base. As a developer touching legacy code you have the opportunity to discover how it was built, how it was changed, and why. This is satisfying work; learning how to code in the "old" style can be fun. Also moving a code from old to new can be satisfying. Finding the multiple styles of a code base and bringing them into alignment is a skill in itself and one that deserves more highlighting.

Legacy code, has made many decisions already. So often that "paralysis by analysis" problem is avoided as you are stepping into a situation where decisions have already been made. Another skill is understanding the constraints and mindsets that ended up in the code looking as it does. The old adage that the people who wrote were doing the best they could given the situation is valuable to keep in mind.

Legacy code is important, it exists and did something. New code is a gamble in a way. It's not proven, it's not been used. So Legacy code and maintaining it is a vital part of our industry and we need more people to value it.

Perl's Legacy Working in multiple languages, it's interesting to see the influence it has had on newer languages.

For example, Go has great testing tools out of the box and formatting. JavaScript's NPM is CPAN. PHP is getting better and has better tooling than it once did. Raku oddly given it's lineage does not have great tooling. There is no Rakucritic or Rakutidy. Go in many ways shows more influence of legacy Perl. Go is arguably built with Legacy in mind. It's intentionally constrained and comes with tools to help the legacy code age well.

Summary

I'll try and record myself giving the talk and share it on the site till then, thanks for reading along.

Tags: legacy_code perl_projects

Perl flexibility for the win

Lance Wicks — Sun, 04 Jul 2021 00:00:00 +0000

This week I spent some time on the Weekly Challenge 119 which is always a good opportunity to get some deliberate practice in.

This week I completed task one, "Swap Nibbles" in both Perl and GoLang.

Now I know I am biased and my main language is Perl, but this was definitely easier/faster than doing it in GoLang.

The point of the exercise is to take a decimal integer, convert it to the binary representation, flip the two "nibbles" and return the resulting decimal integer.

Even with my verbose style of coding these things my solution was this:


sub swap {
    my ( $self, $n ) = @_;

    my $oct = sprintf( "%08b", $n );

    $oct =~ /^(.{4})(.{4})$/;

    return oct "0b$2$1";
}

I streamed myself coding this up on the Perl.Kiwi Twitch Channel which is also on YouTube:

Perl's ability to allow me to take a variable $n and treat it like an integer, then the awesome power of Perl's regex engine makes it simple to swap the nibbles. Then in the return statement auto-magically takes the strings taken in the regex and combines them into another string and oct treats it like a number and converts it to decimal.

It's helpful in a way that non-Perl people can find disconcerting. It's hugely flexible and useful. It's also one of the reasons people hate Perl; it is super easy to write code that is buggy as things are "fluid".

My second attempt was in GoLang:


func NSwap(n int) int {
    bin := fmt.Sprintf("%08b", n)

    a := bin[0:4]
    b := bin[4:8]

    c := b + a

    d, _ := strconv.ParseInt(c, 2, 64)
    return int(d)
}

It's not too dissimilar to be honest, probably influenced by my Perl implementation (and certainly not idiomatic Go). It's not much more complex and follows a similar style of approach. However the conversion steps are more distinct as GoLang will not allow different types to interact in the way Perl does.

The second task was another example of how Perl's flexibility can make it very easy to solve problems quickly and easily. In this task, you need to create a sequence of numbers using just 1,2,3 and that don't have the number 1 next to itself (so 11 excluded, 211 excluded).

My implementation looks like this:


sub no_one_on_one {
    my ( $self, $n ) = @_;

    my @seq;

    my $x = 0;
    while (1) {
        $x++;
        next unless $x =~ /^[123]/;
        next if $x =~ /[4567890]/g;
        next if $x =~ /11/g;
        push @seq, $x;
        last if @seq > $n - 1;
    }
    return $seq[-1];
}

Again, Perl's regex engine for the win. And unless is super helpful here, it helps me read easily.

Rather than trying to create a sequence that obeys the rules; I just loop around incrementing $x by one each time. Then I skip to the next iteration if the $x if the number does not start with one of our numbers (1,2,3). We skip if any number other than 1,2,3 appears. Finally we skip if "11" appears.

If we get past the "next" tests, we push the number onto an array. If that array is long enough to match the limit the user inputs $n we exit the loop with last and return the final element in the array $seq[-1];

I should probably point out here that I am not criticising Go, it's a great language. But Perl is a great language too. And much of the time I prefer Perl over Go, Elm, Raku, PHP, etc. Why? Mainly familiarity if I am being honest with you. It's a really mature, battle tested language. Perl is a general purpose language designed for flexibility and expressiveness. It is not the fastest (or slowest); it has some weaknesses and strengths... as do all programming languages.

Perl is designed to be the "swiss-army chainsaw" a developer reach for when they have a problem to solve. Go on the other hand is design to create "simple, reliable and efficient software"

The next post might be on last weeks challenge, I did take a crack at task #2 and it was really interesting to resolve and to do in a specific way.

Tags: perlweeklychallenge go

TDD a perlscript with modulinos (PWC-118)

Lance Wicks — Mon, 21 Jun 2021 00:00:00 +0000

Perl is a language that has a strong testing culture, the CPAN Testing Service (CPANTS) for example is amazing. When you release any module it gets tested on a huge variety of systems and operating systems and Perl versions automatically and for free.

Perl also has great testing tools, Test:: family of modules are hard to beat.

Writing a new module in a Test Driven Development (TDD) style is very much possible and is my preferred way of tackling the Perl Weekly Challenge when I put time aside to take them on. What you can also do is TDD the script itself via a "modulino".

This week's challenge is a script to tell you if the number you provide is a binary palindrome, you'll probably want that context for the code examples that follow. :-)

To make this possible you can wrap all the code in your script in a sub (for example run()). then call that sub only when the code is being called from Perl, rather than from the test suite. But first we want to start with a test.

use Test::More;

require_ok('./ch-1.pl');

done_testing;

This uses the standard Test::More module for "require_ok" so we can load the script itself. This test will fail till you create the script itself.

As I am creating a script that outputs to the screen (stdout) I want to test the output of the command so I use the Test::Output module.

use Test::More;
use Test::Output;

require_ok('./ch-1.pl');

stdout_is { &run() }
    'foo',
    'Dumb test to see if the test works';

done_testing;

So then we can expand the script to look like this:

__PACKAGE__ ->run() unless caller;

sub run {
    print "foo";
}

1;

From here we can take small steps towards out goal whilst having the safety of tests. So the test might become more sensible like this:

use Test::More;
use Test::Output;

require_ok('./ch-1.pl');

stdout_is { &run(5) }
    'xxxxx',
    '5 is a palindrome';

done_testing;

This is a small test that confirms that the script gives the desired answer in the format we want. In this case I have already coded up the module in a TDD style; so this is really about testing the wiring up of the code. Version one might look like this:

__PACKAGE__ ->run() unless caller;

use lib './lib';
use Binary::Palindrome;

sub run {
    my $n = $ARGV[0] || shift;

    my $bp = Binary::Palindrome->new;

    print $bp->is_palindrome($n);
}

1;

So in this still more than I actually did in step one. I actually just added the use lines and ran the tests to see if everything stayed the same. I use tests that way a lot; confirming I have not added a typo or simple syntax error. I take small steps so spotting the simple mistakes is... well simple.

The my $n = $ARGV[0] || shift; line is a way of making the sub handle input from the command line (the first arg, aka $ARGV[0]) or a scalar from the tests via shift.

In the above situation the test should fail as I am looking for "xxxxx", but I know that the code "should" return 1. So the test fails and tells me I am ok. SO then I can change the test to:

stdout_is { &run(5) }
    '1',
    '5 is a palindrome';

And this should pass, next we want to confirm the real behaviour, so test becomes:

stdout_is { &run(5) }
'1 as binary representation of 5 is 101 which is Palindrome.',
    '5 is a palindrome';

This will fail of course, so lets change the script to pass this test (and forgive me I am skipping steps):

__PACKAGE__ ->run() unless caller;

use lib './lib';
use Binary::Palindrome;

sub run {
    my $n = $ARGV[0] || shift;

    my $bp = Binary::Palindrome->new;
    if ( my $res = $bp->is_palindrome($n) ) {
        print "$res as binary representation of $n is ",
            $bp->represent_as_binary($n), " which is Palindrome.";
    }
}

1;

This should pass, so lets add the next test:

stdout_is { &run(4) }
'0 as binary representation of 4 is 100 which is NOT Palindrome.',
    '4 is NOT a palindrome';

Which fails, so then we add the code:

    if ( my $res = $bp->is_palindrome($n) ) {
        print "$res as binary representation of $n is ",
            $bp->represent_as_binary($n), " which is Palindrome.";
    }
    else {
        print "$res as binary representation of $n is ",
            $bp->represent_as_binary($n), " which is NOT Palindrome.";
    }

Adding this else makes our test passes. So we are in a good place.

This is a slightly contrived example, and there are many improvements to make. However, I now have a solid base to work from. This is a style of working I quite like. Think of it perhaps as Behaviour Driven Development (BDD) as well as TDD. o Or perhaps this could be described as an "Integration test" or even an "End to End test" given what we are building. I am not pedantic about the terms of purity of my TDD/BDD.

Personally I am aiming for tools that make my development easier and more reliable and this sort of testing does that for me.

When I coded this up I did TDD the module itself, so the two methods (is_palindrome and represent_as_binary) have tests.

Not doing the script and it's tests first, has affected the design of the module. The script is more complicated and the code less efficient as I a using both methods in the script AND I use represent_as_binary in the is_palindrome method. If it had side effects (i.e. wrote to a file or DB) then this would be bad; or if it was slow.

The next step might be to create a third method, that returns a data structure with the result and the binary representation. This would be more efficient; I would probably leave the construction of the text in the script as it's the "presentation layer" and leaves the module a little cleaner; though your mileage may vary. A method that returned the full answer would be very tidy in the script (thin controller).

Please do checkout the full code in the Perl Weekly Challenge git repo (https://github.com/manwar/perlweeklychallenge-club/tree/master/challenge-118/lance-wicks/perl).

And if you have any questions please drop me a message.

Tags: perlweeklychallenge tdd