DEV Community: Dave Jacoby

I Met A Var That Wasn’t There: Return of the Spectre of Unicode in Perl

Dave Jacoby — Thu, 11 Nov 2021 22:10:20 +0000

So, let’s make an exploit!

#!/usr/bin/env perl

use strict;
use warnings;
use experimental qw{ say signatures state };

use utf8;

if (1) {
    my $timeout;
    ( $timeout, ㅤ() ) = poorly_trusted_code_from_elsewhere();
    my @array = ( 'curl -s http://perl.com/', 'ping -c 1 purdue.pl', ㅤ() );

    for my $cmd (@array) {
        say $cmd;
        sleep $timeout;
    }
    say join "\n", @array;
    exit;
}

sub ㅤ : lvalue {
    state @array;
    @array = 1 .. 10 unless scalar @array;
    if ( scalar @_ ) {
        @array = @_;

        # say instead of a try {exec()} catch {};
        # because I'm just trying to make a point
        say join ' - ', @array;
    }
    else { return @array }
}

sub poorly_trusted_code_from_elsewhere () {
    return ( 4, 'cat /etc/passwd' );
}

(if (1){ ,,, ; exit} is a convention I use in my test files, where I work to see if I can do the code thing I’m thinking about, not actually solving the problem, and because there can be dozens of things I’m poking at, I put them in this simple if statement and exit out, occasionally turning it to if (0) and certainly stopping things at the end of the block.)

So, rather than actually stand up an untrustworthy server, I’m mockig it with poorly_trusted_code_from_elsewhere, and having it return a command that … well, we use shadow passwords, so passing /etc/passwd isn’t the horror it was in the 1990s, but still, don’t.

Then we have a for loop that handles all the commands, in this case saying them rather than running them, because I don’t really want to get the source code to Perl.com.

If I run this with use utf8:

$ ./exploit.pl
HOLY SIGNIFICANT WHITESPACE, BATMAN!
HOLY SIGNIFICANT WHITESPACE, BATMAN!
curl -s http://perl.com/
ping -c 1 purdue.pl
cat /etc/passwd
curl -s http://perl.com/
ping -c 1 purdue.pl
cat /etc/passwd

If I comment it out:

$ ./exploit.pl
Unrecognized character \xE3; marked by <-- HERE after $timeout, <-- HERE near column 17 at ./exploit.pl line 11.

And, if I run perlcritic:

Problem while critiquing "exploit.pl": Can't parse code: Encountered unexpected character '227'

But then, you have sub ㅤ : lvalue { ... } right in your program. “Hey Joe” , you ask. “What’s the deal with this nameless subroutine you put in my code?”

So, instead, Joe writes BadMod, a bad module that does bad things.

package BadMod;

use strict;
use warnings;
use experimental qw{ say signatures state };
use utf8;

use Exporter qw{import};
our @EXPORT = qw{
    poorly_trusted_code_from_elsewhere
    ㅤ
};

sub ㅤ : lvalue {
    say 'HOLY SIGNIFICANT WHITESPACE, BATMAN!';
    state @array;
    @array = 1 .. 10 unless scalar @array;
    if ( scalar @_ ) {
        @array = @_;

        # say instead of a try {exec()} catch {};
        # because I'm just trying to make a point
        say join ' - ', @array;
    }
    else { return @array }
}

sub poorly_trusted_code_from_elsewhere () {
    return ( 4, 'cat /etc/passwd' );
}

1

And there’s nothing too noticable in the main code, beyond () in places that kinda smell, and a warning on perlcritic.

#!/usr/bin/env perl

use strict;
use warnings;
use experimental qw{ say signatures state };
use utf8;

use lib '.';
use BadMod;

if (1) {
    my $timeout;
    ( $timeout, ㅤ() ) = poorly_trusted_code_from_elsewhere();
    my @array = ( 'curl -s https://perl.com/', 'ping -c 1 purdue.pl', ㅤ() );

    for my $cmd (@array) {
        say $cmd;
        sleep $timeout;
    }
    say join "\n", @array;
    exit;
}


$ ./exploit.pl
HOLY SIGNIFICANT WHITESPACE, BATMAN!
HOLY SIGNIFICANT WHITESPACE, BATMAN!
curl -s https://perl.com/
ping -c 1 purdue.pl
cat /etc/passwd
curl -s https://perl.com/
ping -c 1 purdue.pl
cat /etc/passwd

And exploit.pl still fails perlcritic has weird code-smelly braces in there.

So, if you write BadMod a bit like this:

package BadMod;

# pseudocode, this won't actually run
my @bad_commands;

sub poorly_trusted_code_from_elsewhere () {
    push @bad_commands, 'cat /etc/passwd' ;
    return 4;
}

sub execute_commands ( @good_commands ) {
    push @good_commands, @bad_commands;
    for my $cmd (@good_commands) {
        say $cmd;
        sleep $timeout;
    }
}

1

Just rename @bad_commands to something more innouous and fill the module with dead code, obfuscated code and a thousand globals and you could never tell there’s a problem, and it will pass perlcritic.

So, could I get exploit.pl to ‘execute’ things that it shouldn’t? Definitely.

Are there better ways to get exploit code into a code base that don’t use a character set that can be easily turned off on a whim and don’t trigger any of the default Perl::Critic policies? Absolutely.

Is this something you should be aware of? Sure.

Should anyone be afraid of using a character set that allows the use of letters foreigners use (like لؤلؤة) or images the kids use while texting (like 🍆)? So afraid they categorize it over real threats or problems? No. That doesn’t make a lick of sense.

If you have any questions or comments, I would be glad to hear it. Ask me on Twitter or make an issue on my blog repo.

“Won’t SOMEBODY Think Of The Children?!?!?”: The Spectre of Unicode in Perl

Dave Jacoby — Wed, 10 Nov 2021 19:33:17 +0000

Having lived through the 1980s, I’m not a big fan of Moral Panics, where there’s a whole lot of “What if…?” and “We must protect ourselves!” and not a whole lot of really understanding the issue. I say “issue” instead of “problem” because I find there isn’t one.

So, I look on r/perl and avoiding other things, and I see “Scary, hard to detect code hiding”.

It points to a blog post where they demonstrate how to use space-like unicode characters in Javascript. Specifically, they use ㅤ. That’s Hangul Filler, kids! You can read the whole thing, but in essence, they do an AJAX call where they can send another command that does what it wants, hides it in ㅤ, then execs ㅤ after a ping and curl, and because the dev never sees ㅤ (unless they have space hightlighting turned on in their editor), they never even think that the non-JS “ending” commas are not actually “ending”.

I mean, Holy Significant Whitespace, Batman!

To quote the person sending this:

Perl must be vulnerable to some if not all of these. What tools do we have/should we have in the perl ecosystem to help detect and warn or block these code smells?

To my mind, Perl comes with the solution to this.

Sigils. And no utf-8 by default, but mostly sigils.

Perl has some nice Unicode support, but lots of it is not what you think it is or want it to be. I would ♥ it if use utf-8 did the work of telling all the filehandles, but you have to do binmode STDOUT, ':utf8' instead. Mostly what you get from utf-8 is the ability to use Unicode in your code. That’s a fun thing, but I’m not sure it’s useful.

For example:

#!/usr/bin/env perl

use strict;
use warnings;
use experimental qw{ say };

use utf8;

if (1) {
    my $π = 3.14159;
    my $ㅤ = 'blank';
    say join '|', 1, $π, $ㅤ, 2;
    exit;
}

__DATA__

1|3.14159|blank|2

Here I’m using $π instead of $pi to hold the first handful of digits of Pi. I used to work in academic research, and there, PI means Primary Investigator, so having the Unicode character tells me that I’m definitely dealing with a mathematical constant and not a professor’s name. Because I don’t want to go over Unicode tables or search pi unicode whenever I want to use a variable name, I will likely only use this rarely.

Also, in a decade of coding for the lab, I never had to identify an important client and find the circumference of a circle in the same program. 😀

This is the important part of the example code.

app.get('/network_health', async (req, res) => {
    const { timeout,ㅤ} = req.query;
    const checkCommands = [
        'ping -c 1 google.com',
        'curl -s http://example.com/',ㅤ
    ];
    ...
});

If you had this and formatted it, most JS formatters will remove the trailing commas, but here they survive it, because it’s really …

app.get('/network_health', async (req, res) => {
    const { timeout, HANGUL_FILLER} = req.query;
    const checkCommands = [
        'ping -c 1 google.com',
        'curl -s http://example.com/', HANGUL_FILLER
    ];
    ...
});

And whatever /network_health sends long with the timeout will be run as you in the try/catch promise block I cut out for space.

But, while trailing commas are not liked in the JS world, they’re incredibly common in Perl. (Or maybe they’re possible in Perl and I just really like them, because I want to be able to easily reorder the arrays I build by hand. YMMV.)

So, I get why, in non-Perl languages, you would have problems, and you might want to add a pre-commit hook that searches for HANGUL_FILLER in your code, but in Perl?

my ($timeout, $ㅤ) = suspect_function() and my @commands = ( 'ping -c 1 google.com', 'curl -s http://example.com/', $ㅤ) will look funny, because when in Perl do you just see $ just hanging there? Same thing with @ and % and &, mostly.

I mean, look at this:

use utf8;
if (1) {
    my $π = 3.14159;
    my $ㅤ = 'blank';
    say join '|', 1, $π, $ㅤ, 2, &ㅤ, ㅤ(), 3;
    exit;
}

sub ㅤ () {
    return 'HANGUL_FILLER';
}
__DATA__
1|3.14159|blank|2|HANGUL_FILLER|HANGUL_FILLER|3

With masterful symbol table manipulation, you can stick &ㅤ or ㅤ() (the function answers to both) into main from a library, but that’s it’s own issue.

I suppose I could try to do 104 %ㅤ 20, but I can’t think of where digit %hash digit won’t immediatly be a syntax error. Same thing with digit @array digit. But &…

use experimental qw{ say signatures state };
use utf8;

if (1) {
    my $x = 0;
    my $π = 3.14159;
    my $ㅤ = 'blank';
    say join '|', 1, $π, $ㅤ, 2, &ㅤ, ㅤ(), 3;
    ( $x, ㅤ($π) ) = ( 'lvalue', 'test' );
    exit;
}

sub ㅤ : lvalue ( $a='a', $b='b' ) {
    say join '|', $a, $b;
    return 'HANGUL_FILLER';
}
__DATA__

a|b
a|b
1|3.14159|blank|2|HANGUL_FILLER|HANGUL_FILLER|3
3.14159|b
Can't return a readonly value from lvalue subroutine at ./test.pl line 25.

Here, we have the HANGUL_FILLER subroutine, now defined as an lvalue, meaning it can sit on either side of the assignment. Here there’s an error because I have forgotten (if I ever knew) how to tell if the function’s being used in an lvalue context, but if I had it at hand, I would certainly if it. But there, yes.

And really, if someone is mucking around with your symbol table,

($x,()) = 1..20 is valid, with $x getting 1 and the rest of the numbers dropping on the ground. But looking at that, I would likely shorten it to ($x)=1..20 anyway, so ( $x, ㅤ() ) would just look like a code smell.

So, personally, I’m not worried. There’s enough you can do if you own BadModule and I use BadModule that throwing ㅤ() into this is the least of my worries, especially knowing that before, after and around can redefine so much of the functionality I expect. I see this as a non-issue, but one that could be caught with git hooks and regular expressions. Maybe I should write a Perl::Critic module for this?

If you have any questions or comments, I would be glad to hear it. Ask me on Twitter or make an issue on my blog repo.

What Do Front-End Devs Think About Safari?

Dave Jacoby — Fri, 22 May 2020 00:19:25 +0000

I'm mostly a back-end guy, and have by and large worked with Firefox and Chrome being the browsers of choice, and have only had two issues with Safari (see below). As of today, I may have a third, but I may just have a configuration issue. So, please, what do you think about Safari? Is it the new IE 6? Is it considered decent? What do you think?

Look at this URI: http:/~foo/ It's wrong. You stick it in the browser and it doesn't work, but put it in a page and it will just assume the current server. It's just one step away from /~foo/, right? At least that's the logic my coworker had, but it broke the page for some users. When I had this issue in 2012, it worked in Firefox and Chrome and Opera and IE, but, I figured out by the logs, not Safari. The issue is that every other browser supported it — I don't know if they still do — and Safari didn't.
HTML5 added an Audio API. To learn it, I created a page where, when you go there, you got an old-school pre-VOIP dial tone, which is actually two tones. But the Secret Masters of HTML decided that pages making noise without us saying "you can make noise" is a bad thing, and so I rewrote it so that, and I asked my wife to check on the iPhone and it wouldn't play. I never fully figured out the issue between iOS Safari and my HTML5 stuff, but I installed a WebKit-based browser on my Linux box to try to work it out, and I don't believe I ever fully did.

So, that's my bits. What do you think about Safari? Do you have any Safari stories?

Transitioning from Ubuntu/Windows+WSL to Mac?

Dave Jacoby — Thu, 24 Oct 2019 14:53:10 +0000

I have accepted a new position, starting next month, and will be issued a MacBook. I have worked from Windows, Solaris or Linux from all of my development career, and have a bag of tricks that is based on that.

I mean, I've touched Macs, but last time one was close to my daily driver, it was System 7.

Yes, I'm old.

What suggestions do you have for someone moving from Linux on the Desktop (and also Win/WSL) to Mac?

MAC and Me: How Many MAC Addresses Are On Me Right Now?

Dave Jacoby — Tue, 08 Oct 2019 14:21:44 +0000

This is an idle thought that I didn’t finish, but accidentally posted instead of another post yesterday, so I’m gonna fix.

There are thoughts that computer professionals ask that other people don’t, and as long as we only ask these things of ourselves and other computer professionals, we won’t be thought of as too weird.

In this case, the thought that came to me as I packed up to go to my local coworking space, was “How many MAC addresses are on me right now?”

What’s A MAC Address?

MAC stands for Media Access Control is is how computers identify each other over the network. Instead of diving into the magic of ARP…

Metaphor Time!

You go to your favorite coffee shop and order a latte. They ask for your name, write it on the cup, and start to prepare your caffeinated drink.

Eventually, after the brewing and the pouring and all, they call out the name you gave, and you come to collect the beverage.

MAC addresses serve as your device’s name in the networking context. It cannot receive the packets (the “coffee” in the metaphor) unless it answers to the name it gave.

“…the name it gave.”

I emphasize this because it doesn’t have to be your name, or any name you have ever had. At a conference last year, I organized a “Dave BOF” after realizing that there were at least five other Daves in the conference, my name is so common. So I often give the name “Clem” at coffee shops, because that name’s rare.

And let’s dive down on it a bit. These things are unique to the networking device, which means if you switch your cards to another computer (which, in the days of desktop computers, I used to do), that MAC address went with it. In our coffee shop metaphor, I might generally say “Clem”, but go by “Larry” to the barista who is a bit too much of a Buffy the Vampire Slayer fanboy.

In a less metaphorical sense, I’ve heard stories of high-availability servers have their fail-over machines replicate their MAC addresses, so that the traffic never stops. It strikes me that, with Kubernetes and load-balancing and everything we’ve developed in the last 20 years, this is not something we do with new systems, but I could be wrong.

The Count

Headphones: Bluetooth. 1 MAC.

FitBit: Bluetooth. 1 MAC.

Android Tablet: Bluetooth and WiFi. 2 MACs.

Bluetooth keyboard: Bluetooth. Duh. 1 MAC.

Laptop: Bluetooth and WiFi. 2 MACs.

USB-to-Wired-Ethernet dongle. 1 MAC.

Logitech Unifyng dongle: Unifying is like Bluetooth but different. We’ll say 1 MAC.

Logitech presentation remote: I know that these things have unique dongles, use non-IR protocols so line-of-sight isn’t required, and for this to be true, it must use some sort of pre-Unifying networking protocol, so I have to assume 2 MACs , for the dongle and device, although I cannot prove it.

Raspberry Pi Zero: I put it together as a conference nametag. It’s a full computer, more powerful than many I used professionally in my life, and I used it to run an e-ink display. Anyway, it is in my backpack, and has onboard WiFi and Bluetooth. 2 MACs.

Android Phone: Here we get a new one. It has Bluetooth (I’m listening to Purple Rain on the above-mentioned headphones over Bluetooth), and WiFi, but it can also talk to my cellular provider’s data network. I have to say 3, but there’s a digression from what I’ve seen.

Should I count MEID/IMEI?

So, 16 uniquely-identifying numbers that my devices throw around.

Digression

1X/3G/4G interfaces on cellular devices do have a MAC address, but those MACs are dynamically assigned and change on every reboot of the device… this is because MAC addresses only apply to IEEE 802 technologies, of which cellular networks are not.

So yes, cellular networks are dynamically assigned a MAC address on a smartphone when that device is powered on or rebooted, however, these dynamically assigned MACs cannot be used in a firewall (it would literally be pointless to do so).

– ServerFault

This is answering a question that isn’t ours, but it seems that, for some of our cellular networking protocols, your device is not permanently unique.

This can be good: Your phone checks for cells, and those are logged and can be used to track you. iOS creates a random MAC for this. (I deleted the link to the source on that one; sorry!)

So, I might get to dock one.

And Why Should I Care?

Maybe you shouldn’t. As I pointed out when I started this, this is just geeky weirdness that will make your eyes glaze over.

TCP/IP. We can skip the Bluetooth and Logitech MACs for a moment to point out that there are five MACs – tablet WiFi, phone WiFi, laptop WiFi, wired networking dongle and phone cellular – that also need an IP address. More than one, as they’re all mobile and moved between home, office, coffee shop and co-working space. With IPv4 networking, there are 4,294,967,296 available addresses. We wallpaper over this a lot with NAT, but the big solution is to move to IPv6, which we, by and large, haven’t done. This is computing’s current impending-disaster-we’re-ignoring-for-now. Or at least one of them.

Tracking. This is the most important one, to the right people. If I’m sniffing for Bluetooth devices and looking for my boss’ fitness tracker, so I know that he’s right behind me while I’m coding or blogging, that’s one thing, but if instead, it’s a retail location identifying repeat shoppers by sniffing for BT and WiFi, it’s perhaps a little bit more suspect, and if it’s an angry ex, putting a sniffer along an exercise trail to determine the other’s schedule in order to plan something dire, we’ve gone into dangerous territory.

As Leslie Carhart wrote, “Knowing your personal threat model is important!”, and your health and connectivity might trump these issues, but I really think this is something you should consider.

If you have any questions or comments, I would be glad to hear it. Ask me on Twitter or make an issue on my blog repo.

Scripting the Dropbox API

Dave Jacoby — Thu, 19 Sep 2019 16:17:51 +0000

I am a big fan of Dropbox and have been for some time. One of my things is to have bin and lib directories so a lot of the tools I write for myself are available on all my hosts.

The issue is that Dropbox’s client for Linux wants to exist in context of a desktop, and there are places, such as a research cluster or a hosted machine, where there’s no desktop running and the Dropbox client won’t work.

But don’t worry. Dropbox has an API. And Perl has a library. Well, libraries, but I like and use WebService::Dropbox. Your first step is to use the App Console and get all the keys and you need.

I put all that into YAML so that I can share the code at will, without worrying about sharing keys, but maybe you like JSON or INI files or something. Whatever you do, make sure that you chmod your stuff so that only you can access it, especially if you’re running this on a shared resource.

---
key: gec7xnarw6yy3w8
secret: ioq7dyn412v833m
token: u1viTaKGoFNtAi6aekV4JMMd81HTSeFOn3LN-6m6pUvI1Vd867utfHmfzMVwrO2I
note: "Not my keys, but random numbers I generated. Don't share yours either."

And here is my program, dropbox_copy.pl. If you wanted to download a bin directory from Dropbox, you’d run dropbox_copy.pl -d bin.

#!/home/djacoby/webserver/perl/bin/perl

# This program is used to download a Dropbox directory onto a machine
# without Dropbox tools installed

use strict;
use warnings;
use utf8;
use feature qw{ postderef say signatures state };
no warnings qw{ experimental::postderef experimental::signatures };

use Carp;
use IO::File;
use JSON;
use Getopt::Long;
use Pod::Usage;
use WebService::Dropbox;
use YAML qw{LoadFile};
use File::Path qw{make_path};

my $json = JSON->new->canonical->pretty;
my $config = config();
my $dropbox = WebService::Dropbox->new( { key => $config->{key}, } );

# Authorization
if ( $config->{token} ) {
    $dropbox->access_token( $config->{token} );
}
else {
    my $url = $dropbox->authorize;

    print "Please Access URL and press Enter: $url\n";
    print "Please Input Code: ";
    chomp( my $code = <STDIN> );
    unless ( $dropbox->token($code) ) {
        die $dropbox->error;
    }
    print "Successfully authorized.\nYour AccessToken: ",
        $dropbox->access_token, "\n";
}

if ( $config->{directory} ) {
    my $remote = '/' . $config->{directory};
    get_dir( $remote, $dropbox );
}

exit;

# get_dir() takes $remote, the directory to be copied,
# and a WebService::Dropbox object. I am on an anti-globals
# kick but otherwise would've kept that and just passed $remote

sub get_dir ( $remote, $dropbox ) {
    my $local = join '', $ENV{HOME}, '/Dropbox', $remote;
    if ( !-d $local ) { make_path($local) }
    my $result = $dropbox->list_folder($remote);
    for my $e ( $result->{entries}->@* ) {

        # if it's a folder/directory, recurse
        if ( $e->{'.tag'} eq 'folder' ) {
            my $next = $e->{path_display};
            get_dir( $next, $dropbox );
        }

        # if it's a file, we download it
        if ( $e->{'.tag'} eq 'file' ) {
            my $file = $e->{path_display};
            get_file( $file, $dropbox );
        }
    }
}

# get_file() takes $remote, the file to be copied, and a
# WebService::Dropbox object

sub get_file ( $remote, $dropbox ) {
    my $local = join '', $ENV{HOME}, '/Dropbox', $remote;
    my $fh = IO::File->new( $local, '>' );
    my $response = $dropbox->download( $remote, $fh );

    # say $json->encode($response);
}

# one-stop shop to get the Dropbox configuration and the flags.

sub config () {
    my $config_file = $ENV{HOME} . '/.dropbox.yml';
    croak 'No Config' unless -f $config_file;
    my $config = LoadFile($config_file);
    $config->{download} = 0;
    $config->{upload} = 0;
    GetOptions(
        'help' => \$config->{help},
        'man' => \$config->{man},
        'directory=s' => \$config->{directory},
    );

    pod2usage( -verbose => 2, -exitval => 1 ) if $config->{man};
    pod2usage( -verbose => 1, -exitval => 1 ) if $config->{help};
    pod2usage( -verbose => 1, -exitval => 1 )
        unless $config->{directory} =~ /\w/;
    delete $config->{help};
    delete $config->{man};
    return $config;
}

exit;

=head1 NAME

dropbox_copy.pl - Download a directory from Dropbox

=head1 SYNOPSIS

    dropbox_copy.pl -d 'Images'

=head1 DESCRIPTION

This program copies whole directories from Dropbox

=head1 OPTIONS

=over 4

=item B<-d>, B<--directory>

The name of the directory to be copied.

=item B<-h>, B<--help>

Short-form user documentation

=item B<-m>, B<--manual>

Long-form user documentation

=back

=head1 LICENSE

This is released under the Artistic
License. See L<perlartistic>.

=head1 AUTHOR

Dave Jacoby L<jacoby.david@gmail.com>

=cut

As a GitHub Gist

I should make better tools like this, with listing directories and downloading files and uploading. I should make a repo with all this stuff. This would allow others to help make it better. And because Dropbox owns the name “Dropbox”, I should come up with another name.

But, in the meantime, please enjoy.

If you have any questions or comments, I would be glad to hear it. Ask me on Twitter or make an issue on my blog repo.

Scaled Vector Graphics on the Web for Fun and ?

Dave Jacoby — Fri, 13 Sep 2019 13:56:21 +0000

This is an image.

I enjoy playing with clocks, and this is from that tool.

This is done with Scaled Vector Graphics , or SVGs. SVGs are a text-based XML format, and they look like this.

<svg xmlns="http://www.w3.org/2000/svg" id="neck" viewBox="0 0 900 900">
    <style>
        .loop {
            stroke: #666;
            stroke-width: 20;
            fill-opacity: 0;
            }
        #cen, #day, #hor, #min, #sec {
            stroke: #666;
            stroke-width: 10;
            }
        #cen { fill: lightblue; }
        #day { fill: yellowgreen; }
        #hor { fill: yellow; }
        #min { fill: orange; }
        #sec { fill: red; }
    </style>

    <circle id="d_loop" cx="450" cy="450" r="100" class="loop"/>
    <circle id="h_loop" cx="450" cy="450" r="200" class="loop"/>
    <circle id="m_loop" cx="450" cy="450" r="300" class="loop"/>
    <circle id="s_loop" cx="450" cy="450" r="400" class="loop"/>

    <circle id="sec" cx="450" cy="050" r="30" class="dot"/>
    <circle id="min" cx="450" cy="150" r="30" class="dot"/>
    <circle id="hor" cx="450" cy="250" r="30" class="dot"/>
    <circle id="day" cx="450" cy="350" r="30" class="dot"/>
    <circle id="cen" cx="450" cy="450" r="30" class="dot"/>
</svg>

What do we see? We see things with ID tags, so they can be uniquely identified and with CLASS tags so they can be grouped.

This means, in a web context, you can uniquely address those things and change them.

function draw_second(seconds) {
  let deg = seconds * 6;
  let deg2 = (deg - 90) % 360;
  let sec = document.getElementById("sec");
  let xy = circleCoords(400, deg2);
  let x = 450 + xy.x;
  let y = 450 + xy.y;
  sec.setAttribute("cx", x);
  sec.setAttribute("cy", y);
}

function circleCoords(radius, degFromTop) {
  const rads = degToRad(degFromTop);
  return {
    x: radius * Math.cos(rads),
    y: radius * Math.sin(rads)
  };
}

function degToRad(deg) {
  return (deg / 360) * (Math.PI * 2);
}

There is more, of course, but each circle has a center, which is defined by the x and y coordinates, and so the cx is the x coordinate of that circle and cy is the y coordinate. The base image is 900x900, so this is placed in relation to 450,450 and the radius is 400, so we set that with setAttribute, just like we’re messing with the DOM in other contexts.

There are two other uses I know for SVGs. Logos are great in SVG, because they scale, still looking sharp when blown up to poster and side-of-truck sizes. They are also useful for laser cutters, where different colors are used for different laser intensity, so that a thick black section is made darker, but a thin red line will be cut through.

But remember that, in a web context, the parts of the image are as addressable and modifiable as anything else on a web page.

If you have any questions or comments, I would be glad to hear it. Ask me on Twitter or make an issue on my blog repo.

Playing with Windows Terminal

Dave Jacoby — Wed, 24 Jul 2019 14:59:00 +0000

I have been a WSL user for some time, and also someone keen on hacking my Bash prompt, which has lead me to find that Windows Terminal has lagged for some time on Gnome Terminal for Unicode support, which means that, for example, if you want icons to tell you that you’ve had several cups of coffee, your laptop battery is low, or your Git repo’s status is that of smiling poo, they just wouldn’t show up well on Windows.

There were places that were okay. They worked fine with the Cygwin terminal and the Git Bash that was derived from it, and I recall that PuTTY handled things well (but now that you can SSH from PowerShell and CMD prompts, I never open it anymore), but the built-in terminals were basically unchanged from NT and maybe before.

Rich Turner would know, because he wrote the … well, blog series … on the history of terminals in Windows, which lead to the release of the new Windows Terminal. You can download it from the MS Store now, and unlike some other favorite toys, you don’t have to be Fast Track Windows Insider. So let me show you around!

Pretty

I’m starting with the WSL prompt, because, as a Linux guy, I’ve done much more work with my Bash prompt customization. So far, that is.

I point out a few things here. The coffee cup emoji (☕) indicating the cups I’ve consumed that day. Also, the computer emoji (💻) indicating that I don’t have packages to add, and the battery (🔋), which doesn’t mean much because the term is on my desktop, but would indicate battery charge level. Also, I try to use Taskwarrior and the check (✔) signifies that I have no tasks outstanding. And all of it shows up in the new Windows Terminal, which was not always true.

I’ll also mention that I’m using Fira Code, a font that supports ligatures, and it shows <= as one character, not two.

You can’t see it from the screen capture, but the cyberpunk image is an animated GIF, that is animated behind the text.

I consider this a big thing, because I wrote a tool to set your Windows background image, and that only accepts JPEG.

This configuration is specific to the kind of term you’re opinging, so if you want a PowerShell terminal, it looks different.

Configuration

Shortened, the configuration file in C:\Users\you\AppData\Local\Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\RoamingState\profiles.json looks like this:

{
  "globals": {},
  "profiles": [],
  "schemes": []
}

And is available with ctrl-, or under Settings.

(Color) Schemes

Back to front, schemes are color schemes, and look like this:

{
  "background": "#0C0C0C",
  "black": "#0C0C0C",
  "blue": "#0037DA",
  "brightBlack": "#767676",
  "brightBlue": "#3B78FF",
  "brightCyan": "#61D6D6",
  "brightGreen": "#16C60C",
  "brightPurple": "#B4009E",
  "brightRed": "#E74856",
  "brightWhite": "#F2F2F2",
  "brightYellow": "#F9F1A5",
  "cyan": "#3A96DD",
  "foreground": "#CCCCCC",
  "green": "#13A10E",
  "name": "Campbell",
  "purple": "#881798",
  "red": "#C50F1F",
  "white": "#CCCCCC",
  "yellow": "#C19C00"
}

As you can see, it’s taking the names of default colors, plus foreground, background, etc., and putting a name to them.

Campbell is the out-of-the-box scheme, but you can change that and you can change these, in…

Profiles

{
  "acrylicOpacity": 0.5,
  "background": "#012456",
  "backgroundImage": "ms-appdata:///roaming/dotty-me.jpg",
  "backgroundImageOpacity": 0.25,
  "closeOnExit": true,
  "colorScheme": "Campbell",
  "commandline": "powershell.exe",
  "cursorColor": "#00FF00",
  "cursorShape": "bar",
  "fontFace": "Consolas",
  "fontSize": 10,
  "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
  "historySize": 9001,
  "icon": "ms-appx:///ProfileIcons/{61c54bbd-c2c6-5271-96e7-009a87ff44bf}.png",
  "name": "Windows PowerShell",
  "padding": "0, 0, 0, 0",
  "snapOnInput": true,
  "startingDirectory": "%USERPROFILE%",
  "useAcrylic": false
}

This is the profile for PowerShell, and I have customized it some. background is background color, and you jump to backgroundImage to set an image.

But, of course, you want to see the text over your image, so backgroundImageOpacity is a float between 0 and 1 setting how much is there, which allows you to read text over it.

Valid cursorShape values are bar, emptyBox, filledBox, underScore, and vintage. Thanks, Donovan Brown!.

Evidently, Acrylic is a way to make a terminal translucent; the background is viewable through, but not in focus. This is where useAcrylic and acrylicOpacity fit in, but I have yet to make them work.

There’s a lot of Windows short-hand stuff here. ms-appdata:///roaming/ means C:\Users\you\AppData\Local\Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\RoamingState. %USERPROFILE% is a long and complex way of writing ~ and the startingDirectory field it sets doesn’t do much for starting wsl, so the hack is to make your commandLine ask for "wsl.exe ~ -d Ubuntu-18.04". And then there’s the unique identifier, or guid. This is used primarily to indicate which choice comes up when you open WT, in …

Globals

    "globals": {
      "alwaysShowTabs": true,
      "defaultProfile": "{7c51c86f-d763-44fc-b952-e04d5922f631}",
      "initialCols": 120,
      "initialRows": 30,
      "keybindings": [{
        "command": "closeTab",
        "keys": [
          "ctrl+w"
        ]
      }, {
        "command": "OTHERS",
        "keys": [
          "plenty"
        ]
      }],
      "requestedTheme": "system",
      "showTabsInTitlebar": true,
      "showTerminalTitleInTitlebar": true
    }

Most of these seem to not do much, especially everything with Tab in the key. defaultProfile is what decides which of your choices come up when you open a tab or create a window, and that is set by that profile’s guid, and these GUIDs seem unique to the project, so your Ubuntu-18.04 GUID just might be mine.

Issues

Personally, I would love for the big array of keybindings to be at the top level, not stuck in the middle of the globals, but I’m not in position to change that.

On Linux, I use a transparent terminal set to about 30%, so you can see the Desktop (including other windows) through the terminal. It is mostly a “this is so cool” thing, but on occasion it has proved helpful. I would love the transparent/translucent backgrounds that Acrylic promises, but I have yet to make them work. I’m not generally the big fan of terminal-specific backgrounds, but I am enjoying them while I wait.

I would love to have some way to, when opening a term, be able to choose WSL Ubuntu or WSL Fedora or PowerShell ISE (As Admin), rather than creating a tab for it once it’s open, and the Admin powers part is not started to be solved yet, as far as I can tell.

You’ll notice the LONG tab which shows the location of the PowerShell executable, and that, instead, my Ubuntu tab says jacoby@Lion: ~, this is because I have tamed my $PS1 so it contains the string \[\033]0;\u@\h: \w\007\], which tells the computer that every time the prompt is written to set \u@\h: \w, or username@hostname: current_working_directory, as the terminal title, which, in this case, is tab title. Such magic must exist with PowerShell, but I have yet to dive deeply and understand such things. The default is so ugly, I’m going to have to.

The lack of emoji and Unicode characters has been my big annoyance with Windows for a while, and this release really fixes that. I’ll have to spend some time changing settings and my workflow to make this really fit my use, but I’m very happy with the new terminal and can hardly wait for improvements.

If you have any questions or comments, I would be glad to hear it. Ask me on Twitter or make an issue on my blog repo.

My Shame

Dave Jacoby — Thu, 18 Jul 2019 14:47:45 +0000

I tweeted this, saying “my shame”.

In the beginning there was format_all_that_data. It was an internal function that took a big data structure and turned it into HTML.

Using Perl’s CGI.pm to create the HTML. I knew this was a bad idea a decade before, but it was the lab style, and it took a while before I understood Template Toolkit (and other templating engines) as the way of the future.

I needed to make changes, but I had to work in production (a situation I still have yet to escape) and wanted to keep the old one working, so I called it format_all_that_data_new , and when it was done, I changed the places I knew used format_all_that_data to format_all_that_data_new and made it so that format_all_that_data returned an empty string. That way, if output was expected but not forthcoming, someone would tell me.

And then I was asked to do a few small changes. One change would be fine; do it, test it, release it, forget it. But there were more changes, which is enough to dive onto templating and fixing a lot of other stuff in that program. And again, I am developing in production, so I create format_all_that_data_tt , which uses Template Toolkit to do the HTML correctly.

And, once I have it, I want to switch over, but I want to minimize the changes, so I move format_all_that_data_new to format_all_that_data_old , so that it and all the knowledge I stuck into it and fear to remove are there but not gone.

So I rewrote format_all_that_data_new to, in it’s entirety:

sub format_all_that_data_new($data) { format_all_that_data_tt($data) }

I keep dead code because I fear, and add cruft to route around it.

This is my shame.

If you have any questions or comments, I would be glad to hear it. Ask me on Twitter or make an issue on my blog repo.

I did an xargs!

Dave Jacoby — Thu, 27 Jun 2019 15:52:45 +0000

A fundamental thing you must know is that there is not a canonical set of knowledge for ... most anything, really, but here, for Linux/Unix expertise. There have been studies. We all have gaps in our knowledge. I am sure there is someone with substantial knowledge of the linked-lists underlying file system representation but is unaware that ls -lh put file sizes in a human-readable form. (If that's you, you're welcome.)

For me, one of my long-standing knowledge gaps has been xargs. I've seen it for decades, all find . | xargs <something> and I've copied and pasted those commands, but the knowledge never stayed.

segue

For the last month and a half, an API we use has not been working, meaning we couldn't update data into it. But that bug has been fixed, and now I have hundreds of updates that have been requested of me.

I have an API tool that allows me to get a whole month's worth of data out of the API and into columns, and this time I remembered awk is the one I wanted (sed and awk being two other gaps in my Unix knowledge, but that's another column), getting me:

1sf701vdlahb
3lm63b58riej
3o4rhtaxe8lx
b5hqvs6tuqnm
eoirmijclnl1
uihc1q3yzkvm
vq2uskrrlb5h
w6ulq3sdvyir
wxk3bbdpwdx3
y2npmn6yjkcx
zepz3n2e2to7
zk08o7dysokt
...

(And no, that's just random strings, but bear with me.)

So, I needed to run api_tool -status done -r for all of those, and thought through the pain of all the other options. List of IDs into a file, and then ... bash for loop? add api_tool ... to every line and sh id_list.txt?

And then I remembered xargs, and a half-dozen tabs were opened.

xargs has a lot of flags, so there's a LOT more to it than what I'm using here, but what I used was api_tool -q not_done -f 2019-06-01 | awk "{print $1}" | xargs -n 1 api_tool -status done -r.

This awk command was right the first time (yay me!) but I ran the xargs against echo a few times before I got it right. -n <number> determines the number of entries it pulls from the list of entries at once (we just want the one) and appends it to the end of the list, so <whatever> | xargs -n 1 <command> becomes <command> <first entry> and so on.

This saved me a whole lot of pointless typing, and I'm so glad.

moral

The point is that you should recognize when you have gaps, exploit opportunities to learn things, and record these things so that you (and others) can use them later.

And, xargs is cool, yo.

"Perl Out Loud", or another way to handle RSI

Dave Jacoby — Sun, 23 Jun 2019 18:05:13 +0000

The Perl Conference was in Pittsburgh this year, and one of the talks, and by far a favorite, was given by Emily Shae, a software engineer at Fastly.

She had been developing repetitive stress injury, which is "is what black lung is to miners", as Neil Stephenson wrote, and nothing really worked, until ...

I'll let her tell it.

(All you anti-Perl gripers, her workplace is Perl, the conference is Perl and the examples for the audience are Perl, but there is not much Perl in the toolchain and it can be easily be changed into your language of choice. That isn't the point. Troll better.)

I already use Google Assistant for many things, so I'm primed, but the Dragon Dictation and Talon Voice are right there on the machine (ATM Mac-only, but with Linux and Windows on the feature roadmap), there shouldn't be the lag that I get between "turn off the lights" and darkness.

The ergonomic benefits are cool, but for me, the idea that we might get further away from the Keyboard, Monitor, Mouse standard for developer workstations is exciting. Yeah, for open office plans, it'd start to get very bad, but open office plans are already bad.

So, if you are beginning to feel the pain, or simply want to try something new, give this a try.

Stable Sort and Ways Around It

Dave Jacoby — Mon, 15 Apr 2019 18:20:11 +0000

There’s a service called Last.fm which allows you to store what tracks you listen to. I have started to export that data from Last.fm into a local database, where I have all the songs I have scrobbled (listened to and reported to their system) since Jan 1, 2008. Here we have the first four, sorted on id.

[
  {
    "artist": "Planes Mistaken For Stars",
    "id": "++2oW+aGn4m9JoPRd5tgXMRuZY4",
    "plays": 1,
    "song": "Police Story / Wasted"
  },
  {
    "artist": "Chairlift",
    "id": "++9JAle7S9huuAw0bUPH2Nx15ds",
    "plays": 1,
    "song": "Crying in Public - Alternate Version"
  },
  {
    "artist": "Ali Farka Touré",
    "id": "++D58rizJf3r8hBYEsI9iChDM0k",
    "plays": 3,
    "song": "Erdi"
  },
  {
    "artist": "fIREHOSE",
    "id": "++ehP6ysoH1N0JHUJB0GhNAx7I4",
    "plays": 1,
    "song": "From One Cums One"
  }
]

But this isn’t such a helpful sort, is it? I can imagine two preferable sorts: by number of plays and by artist/song. Perhaps artist/release year/song, but I don’t currently have all that information.

Because rigged demo, I know that I have listened to Chris Thile songs 86 times. (Does not count Punch Brothers or Nickel Creek or duet projects.) And because of albums, I know I’ve heard seveal songs the same number of times.

[
  {
    "artist": "Chris Thile",
    "id": "XgLBRBSQ4jKrrDn2GZ0C7W+rFbY",
    "plays": 1,
    "song": "Trail's End"
  },
  {
    "artist": "Chris Thile",
    "id": "xsoV1/NPsRcbiL47F6JNFLbpBPw",
    "plays": 1,
    "song": "Shadow Ridge"
  },
  {
    "artist": "Chris Thile",
    "id": "z8f9Kr1ZDO+NwweoYp474ebzBwU",
    "plays": 6,
    "song": "Wayside (Back in Time)"
  },
  {
    "artist": "Chris Thile",
    "id": "zvDn5OFuvO99CyfCeBizrlxeHuE",
    "plays": 6,
    "song": "I'm Yours If You Want Me"
  },
  {
    "artist": "Chris Thile",
    "id": "ZWvpfxcBiOKDaLCMCQ+2PyvICeU",
    "plays": 1,
    "song": "Ready for Anything"
  }
]

So, what we want is a list of his songs sorted first by number of plays and then alphabetically by song name.

#!/usr/bin/env perl

use strict;
use warnings;
use utf8;
use feature qw{ postderef say signatures state switch };
no warnings qw{ experimental::postderef
    experimental::smartmatch
    experimental::signatures };

use JSON;
use List::Util qw{uniq};

my $json = JSON->new->pretty->canonical;
my $file = 'lastfm.json';
if ( -f $file && open my $fh, '<', $file ) {
    my $text = join '', <$fh>;
    close $fh;

    my $obj = $json->decode($text);
    my $bigger->@* =
        grep { $_->{plays} > 19 } $obj->@*; # at least 10 plays per song

    my $thile->@* =
        grep { defined $_->{artist} && $_->{artist} eq 'Chris Thile' }
        $obj->@*;

    # reminder that you read this line back to front, so we are
    # sorting on song name first, than play count
    say join "\n", map { join "\t", $_->{plays}, $_->{song} }
        sort { $b->{plays} <=> $a->{plays} }
        sort { $a->{song} cmp $b->{song} }
        $thile->@*;
}

# 30    Heart in a Cage
# 6 Brakeman's Blues
# 6 How to Grow a Woman From the Ground
# 6 I'm Yours If You Want Me
# 6 Stay Away
# 6 Wayside (Back in Time)
# 5 Cazadero
# 5 If The Sea Was Whiskey
# 5 O Santo De Polvora
# 5 The Beekeeper
# 5 The Eleventh Reel
# 5 You're An Angel and I'm Gonna Cry
# 4 Dead Leaves and the Dirty Ground
# 3 Alderaanian Melody
# ...
# There are 86 songs in total, so we're cutting this down

Perl offers stable sort, where we can stack sorts like this, knowing that order gets preserved if not otherwise sorted.

So, what about JavaScript?

#!/usr/bin/env node

const fs = require("fs");
var text = fs.readFileSync("lastfm.json", "utf8");
var obj = JSON.parse(text);

var thile = obj
  .filter(t => t.artist === "Chris Thile")
  .sort((a, b) => {
    if (a.song > b.song) {
      return 1;
    }
    if (a.song < b.song) {
      return -1;
    }
    return 0;
  })
  .sort((a, b) => b.plays - a.plays);

console.log(thile.map(t => [t.plays, t.song].join("\t")).join("\n"));

// 30   Heart in a Cage
// 6    Brakeman's Blues
// 6    I'm Yours If You Want Me
// 6    Wayside (Back in Time)
// 6    Stay Away
// 6    How to Grow a Woman From the Ground
// 5    You're An Angel and I'm Gonna Cry
// 5    O Santo De Polvora
// 5    If The Sea Was Whiskey
// 5    Cazadero
// 5    The Eleventh Reel
// 5    The Beekeeper
// 4    Dead Leaves and the Dirty Ground
// 3    Watch 'at Breakdown
// 3    Alderaanian Melody

// W before A? S before H?

We did sort by song title before we sorted by plays. We can see that. So, what happened?

“I wish JS offered stable sort but committee could not agree on it even in 1996, and you can stabilize at some cost above the API.”

– Brendan Eich (@BrendanEich)

(I was discussing this with Gizmo and threw in an aside to Brendan, and he answered. We live in a world where we can ask “Hey, language creator? How come your language is like this?” and get an answer. How cool is that?)

So, the goal here here is to stabilize it myself, and I know how.

#!/usr/bin/env node

const fs = require("fs");
var text = fs.readFileSync("lastfm.json", "utf8");
var obj = JSON.parse(text);

var thile = obj
  .filter(t => t.artist === "Chris Thile")
  .sort((a, b) => {
    /*
      the sort for plays comes first, but instead the short form
      we just handle the greater or less then cases, letting the 
      equals case drop through...  
     */
    if (a.plays < b.plays) { return 1; }
    if (a.plays > b.plays) { return -1; }
    /*
      ... to here, where we can do greater than and less than on
      strings. I don't GET or APPROVE of that, but I am given no
      other choice, and so we sort on song title.
     */
    if (a.song > b.song) { return 1; }
    if (a.song < b.song) { return -1; }
    /*
      And we return 0 if everything else is even.
     */
    return 0;
  });
console.log(thile.map(t => [t.plays, t.song].join("\t")).join("\n"));

// 30   Heart in a Cage
// 6    Brakeman's Blues
// 6    How to Grow a Woman From the Ground
// 6    I'm Yours If You Want Me
// 6    Stay Away
// 6    Wayside (Back in Time)
// 5    Cazadero
// 5    If The Sea Was Whiskey
// 5    O Santo De Polvora
// 5    The Beekeeper
// 5    The Eleventh Reel
// 5    You're An Angel and I'm Gonna Cry
// 4    Dead Leaves and the Dirty Ground
// 3    Alderaanian Melody

And there you go. Have I given the subtle clue that I’m a frustrated mandolin player?

If you have any questions or comments, I would be glad to hear it. Ask me on Twitter or make an issue on my blog repo.