DEV Community: John Colagioia (he/him)

Frameworks of the Future?

John Colagioia (he/him) — Wed, 21 Jul 2021 12:29:06 +0000

Something has been nagging me in the back of my mind for a while: Where are the "modern" web frameworks?

What I mean is that---apologies for the history lesson---once Ruby on Rails got its footing, it did enough that it changed how web development happen. Web applications in the 1990s was a nightmare, hacking everything together through CGI and session variables, which didn't substantially change until Rails.

Today, most of that mess is just how you threaten children to behave, because everybody else now has their equivalent to Rails. C#/VB.NET's ASP.NET, Elixir's Phoenix, Go's Beego, PHP's Laravel, Haskell's IHP, Java's Jakarta EE, JavaScript's Express, and literally dozens of others now have effective parity with Rails.

I was also struck by a coincidence: I picked up Rails about fourteen years ago, and wrote my first (awful) web application about fourteen years before that, and yes, I'm really that old...But that triggered the question. If web development in 1993 looked like it would in 2007 (except for still-obscure Rails), and Rails looked like web development today, is there a framework out there that has all the features that we'll demand as necessary in 2035?

I'm not looking for the overthrow of CRUD-through-MVC. Rather, Hobo and Hoodie seemed like advances---Hobo was sort of Rails for Rails, and Hoodie was an offline-first framework for something like what we now call Progressive Web Apps---when I tried them early in their life-cycle, but both seem to have withered away. And nobody else (that I can find) seems interested in improving graphical design (as in, "just use Material Design, or Carbon, or whatever"), cleaner parent/child relationships, automatically updating views and controllers to match changes to the models, and probably features that I don't know that I need.

I asked a similar question in a different community, and the closest they came up with was the niche Apache Beam and the obligatory vague hand-waving about no-code systems. So, maybe DEV seeming to skew younger and more deliberately technical might get a better view of things? Is anybody using a "Framework of the Future" that we should all know about?

How Long Will That Project Take?

John Colagioia (he/him) — Thu, 28 Jan 2021 08:51:23 +0000

As a quick note, I released this post (well, a version with slightly different editing) on my blog, Wednesday morning, so it can get to be (as I tend to be) a bit rambling in places. However, the original text is on GitHub (licensed CC-BY-SA), so if anything seems muddy, by all means:

Leave a comment here,
Leave a comment on the blog,
File an issue on GitHub, or
Add a pull request!

jcolag / entropy-arbitrage

John Colagioia's Blog Posts

Every once in a while, I’m reminded that one of the most important tasks in software development that most developers never learn to do is to estimate projects. Interestingly, rather than drawing from other disciplines to teach people how to produce better estimates, the industry has largely shifted to management techniques that assume that estimates are going to be embarrassingly wrong.

They don't need to be.

So, since the topic has come up a couple of times in the past few weeks, I wanted to centralize my thoughts, in case they’re useful to someone else.

Estimating or Guessing

To start off, let’s be clear that a project estimate isn’t hearing a quick description of the finished project and picking a number that “sounds about right.” That’s a guess. There are certainly cases where guesses are acceptable, and your guess might sometimes be right, but estimates are something different.

Let’s start with the dumb “the dictionary defines…” trope, here, borrowing from Wiktionary.

estimate (third-person singular present estimates , present participle estimating , simple past and past participle estimated )

1

To calculate roughly, often from imperfect data.

2

To judge and form an opinion of the value of, from imperfect data.

Those are basically the same thing in different contexts, and I hope it shows the distinction between estimating and guessing. Your estimate would be the correct answer, if you had better information. Your guess would probably not change, however, if more information was available.

Fudge Factors

While, again, guessing is fine in certain environments, it has a huge weakness: To try not to over-promise, workers (especially software developers, in my experience) tend to try to find some magic multiplier that will turn their fantasy numbers into a valid estimates. It doesn’t work, being a perfect example of "garbage in, garbage out," but that doesn’t stop people from hoping to make the proposed time-frame long enough to sound credible, but short enough for the investment to sound like a good bargain.

In my experience, these factors come in a small handful of varieties, some more legitimate than others.

After reviewing the plan, you can point to the places where the project has a lot of uncertainty that could go horribly wrong, and don’t think that there’s any way to measure that impact without doing the work. For example:
- If you need to integrate with a third-party service that’s still under development, you’re going to spend an unknown amount of time testing with your counterparts, discussing options in conference calls, and rewriting code as the target moves.
- If you need to optimize an algorithm to trim requests below a certain threshold, you can (and should) calculate a lower-bound for the run-time, but time spent actually trimming time can easily hit a point of diminishing returns, dragging out the time required.
You’re including overhead of work that you don’t personally do, but is an unavoidable part of the project. I’ll talk about this more, later.
You haven’t the foggiest idea of what’s involved in this project, so you picked a number out of a hat and feel that you need a fudge factor, just in case you’re wrong; you’re probably also planning to work late when the deadline approaches to meet the deadline. This is the situation that I described earlier, where developers tend to just find a compromise between sounding appealing and sounding possible.
You don’t trust the other party to uphold their end of your agreement, so you’re just flat-out assuming that you’re going to need to do everything multiple times.

Uncertainty in the plan (the first item) is almost always a legitimate reason to “pad” a schedule. The padding is localized on the schedule and each instance has specific justifications, so it’s possible to isolate that part of the project.

Overhead—design, management, testing, debugging—is also generally going to be legitimate, and probably even good to consider, as long as you understand the different roles involved in the project and which need to be part of your estimate.

You see the “fudge factor” in cases where the developer doesn’t understand how to build a real estimate. You’ll see people talk about “rules” of multiplying all initial guesses by two or three, and we all start our careers thinking that's how you do the job. But it’s basically an admission of being far too optimistic, as well as showing ignorance of scheduling, so it’s probably best to steer clear of doing this, unless you can be specific about it, as I'll get to later. Basically, this multiplier is a step away from “it’ll get done when it’s done.”

Finally, trust is, as they say, a two-way street. In one direction, you have developers padding schedules because they believe that management or the customer is going to change the scope or even the nature of the project, meaning that someone will need to scrap or rework a fair amount of existing work. In the other direction, management or customers might push to pad the schedule (or might treat the budget for the schedule as if it was several times larger), because they don’t trust the developer(s) to do the right work. If you’re adjusting the schedule because of a trust problem, you need to build trust, rather than tweak the schedule, or your schedules are always going to leave someone looking foolish.

Note, by the way, that Agile development processes have tried to take trust out of the process, by shifting the focus of planning to “sprints,” when developers just try to get through as many tickets as possible, and then use that data point to predict how many "points" the team will complete during the next sprint. In doing that, however, it also mostly eliminates the idea of rigorous estimates in favor of gut reactions.

As I said about guessing above, there are contexts where guessing is fine, and Agile development processes are (usually) one of them. Just be mindful about what you’re doing and the risk that’s introduced, compared with how that risk is contained.

Process

Now that we better understand what not to do and why we might care about getting accurate estimates, let’s talk about how to put a credible estimate together. Improved estimates are important, because reliably getting work done on time helps to fix the trust issue discussed above.

Divide and Conquer

That all said, the way I build my own estimates to increase the chances of it being correct is to recursively break tasks down with best-case and worst-case estimates, until the worst-case estimate is no more than a few hours long. I pick that size, since just about anybody is going to get that scale right, even if it’s just a guess.

That maximum-size time (of a few hours) is based on projects that are likely to take an on-one-hand number of worker-months, and so should probably be adjusted if that’s not the likely scale. Otherwise, you’re mapping out individual hours on a twenty worker-year project or half-days on a weekend hack, which is either a waste of time or too abstract to help. To adjust, if we figure that three months has around five hundred work hours in them, you’re talking about your smallest unit being a bit less than 1% of the guessed-at size of the whole project. If you get that guess wrong, no harm done if your minimum size gives you three times as many sub-tasks.

By contrast, looking for the worst case for each sub-task keeps you honest and reminds you about all the little tasks and pitfalls that you were going to just assume happened magically. Don’t skip that part.

A plan like this might look something like the following excerpt. Let’s say that the project is a traditional blog. Ignore the fact that the estimates assume a multi-month project, when the reality is that you'll probably be done in about a week.

Task Name	Best Case	Worst
Index	👇2 h	👇5 h
👆 List Posts	30 m	1 h
👆 Paginate	1.5 h	4 h
Create Post	👇21 h	👇26 h
👆 Implement Default Editor (Scaffolding)	0 h	0 h
👆 Evaluate Rich-Text Editors	16 h	16 h
👆 Replace Default Editor with Rich Text	2 h	3 h
👆 Validate Input	2 h	4 h
👆 Preview Post	1 h	1 h
👆 Save Post	0 h	2 h
Edit Post	👇1 h	👇6 h
👆 Load Existing Post to Create Page	1 h	4 h
👆 Overwrite Existing Post on Save	0 h	2 h
Display Post	👇2 h	👇7 h
👆 Convert Internal Representation to HTML	0 h	1 h
👆 Insert CSS Classes into Post	1 h	4 h
👆 Show Post in Context	1 h	2 h
Add Comments	👇3.5 h	👇6 h
👆 Show Existing Comments for Post (No Threading)	1 h	1 h
👆 Create Comment Form on Post Page	2 h	4 h
👆 Save Comment Associated with Post	30 m	1 h

I didn’t bother to break up Evaluate Rich-Text Editors, because (a) this isn’t a real project, (b) breaking it down would probably just mean listing the candidates, and (c) it’s not out of the question to say “after I’ve spent two days on this, I’m just going to pick whatever I’ve seen that looks best, and forget the rest.”

Anyway, this project outline gives us a fifty-hour prediction. As tasks are completed, fill out the actual time spent in the fourth column, as honestly as possible.

Account for Overhead

Next, take your development-time estimate and put together a 2:1:3 schedule of design/architecture time (2x), development time (your actual estimate), and quality assurance/debug time (3x). I forget where I got that guideline from, honestly, but it’s basically fixed overhead that you don’t want to think about up-front.

I’ve never seen anybody successfully cheat that ratio, by the way. I've seen many attempts, but they invariably slow the project down, rather than speeding it up. You can push the work around—like having the development team use test-driven development—and many projects can often afford to ship incomplete. But as a sell-able product, you need that design and test time, even though it might feel wasteful.

That is, for the blog plan described above that’s planned to take fifty hours, the design time (all choices, coordination, and prototyping) is going to take approximately one hundred hours, and testing/debugging time is going to come in at around one hundred fifty hours, for a total of three hundred worker-hours to make this a polished product.

Well, that’s not actually entirely true. That sixteen-hour step to evaluate rich-text editors isn’t a development task, even though a developer is going to do that work. Instead, it’s a design task that almost certainly incurs no QA time. So as you get better at this, you might say that the real development schedule is thirty-four hours, with sixty-eight plus sixteen hours of design time, and about a hundred hours of testing and debugging.

That's still a lot, but...well, I'll get to that later.

Feedback

As you work, you then track real time elapsed against your low-level estimates, in order to forecast how you should adjust estimates in the future. Divide the worst-case estimates by the actual time spent. Take the average of those ratios—that’s not quite mathematically valid, but it’s close enough for our purposes—and use that final ratio to adjust the sub-task estimates on the next project.

Note that this isn’t a “fudge factor” like the above. This is a recognition that you’re still learning to get the small estimates right, so this is deliberately correcting for your bias. As you go through more projects, you’ll notice that the ratio gets close enough to one-to-one that you won’t need to correct for bias.

Don’t stop tracking the bias, though. If it starts drifting, again, you’re going to want to know that and start accounting for it. In some extreme cases, you might want to make sure that you're not burning out or otherwise unhappy, too.

Viability

Yes, you’re thinking that this is much more work for a proposal than “eh, sounds like a nine-month job to me,” but it’s work that makes it easy to get estimates pretty close to the actual work that’ll be done, and also quickly see how any scope changes are going to impact the schedule. That builds everybody’s confidence, which is helpful.

Because this gives you a good view on the cost of features, a process like this also helps argue for and against work based on the return on investment, which is more useful than just arguing that a project will be “good” for the product. For example, our estimate for that blog came to hundreds of hours of work, which would easily cost the organization tens of thousands of dollars. Is that money going to bring the organization more value than installing WordPress?

However, like I hinted a couple of times, this process doesn’t mesh particularly well with Agile-style project planning, because Agile is built around the idea that long-term estimates are going to be wrong, so take all of this with a grain of salt for a modern office. I suspect that there’s probably a way to fit the two worldviews together, but I haven’t gotten it to work, yet, beyond mapping out individual tickets.

Credits : The header image is untitled by an unknown PxHere photographer, released under the terms of the Creative Commons CC0 1.0 Universal Public Domain Dedication.

Impostor Syndrome

John Colagioia (he/him) — Sun, 23 Aug 2020 11:07:55 +0000

As a quick note, I released this post (well, a version with slightly different editing) on my blog, Sunday morning, so it can get to be (as I tend to be) a bit rambling in places. However, the original text is on GitHub (licensed CC-BY-SA), so if anything seems muddy, by all means:

Leave a comment here,
Leave a comment on the blog,
File an issue on GitHub, or
Add a pull request!

jcolag / entropy-arbitrage

John Colagioia's Blog Posts

I've had several conversations, recently, about dealing with impostor syndrome, so I decided to collect my thoughts over the years, in case they're useful to someone down the line. I didn't write this with the software industry specifically in mind, but it's obviously a central part of the conversation for this community.

No Impostor Left Behind

The thing I’ve found myself talking about recently—as you can probably tell by the Quora question titles above—to the point that it made sense to round up my thoughts on the matter, is impostor syndrome, the feeling that we’re going to be revealed as frauds in our work.

Most of the talk about this—studies, online articles, and so forth—focus on a subset of people who are sometimes considered by the dominant group, such as women and racial/sexual minorities. There’s a good reason for that focus, of course. People who look like me rarely have our backgrounds questioned and there’s no history of shunting us into poorer education.

However, doing so treats this as a problem for women and minorities—something that separates them from everybody else at work—instead of what it really is: Something that everybody deals with, pretty much all the time. The reason you feel that way has nothing to do with the relative lengths of your chromosomes or your ethnicity.

We all deal with it, because…

Work Isn’t...Real

Well, that title’s an exaggeration. What I mean is that jobs are a social construct, which are real things. However, being artificial, none of us is born to do any job naturally. Likewise, no education or experience truly prepares you for the next emergency.

So, we all feel like we’re impostors, because…well, we actually are all impostors. We all pretend that we have the answers, knowing full well that we probably won’t actually have any solid answers until the job is done. We continue on after every failure, and we fail a lot—unavoidably—because we’re only human. And we scramble to figure things out as we go.

So, yeah, we’re all impostors. Perfection isn’t worth the price. It’s not a bad thing, except to the extent that it bothers you.

Keep in mind that this isn’t limited to work. Nobody actually fits completely into their social circle, either. Our relationships are never going to be perfect and there’s always something more we should probably be doing or getting, that we can’t even see. It’s the nature of the world, and a lot of growing up seems to involve coming to terms with that.

Exceptions

Well, sorry. Yes, it’s not all of us. There’s a handful of completely terrible people in every field who are terrible at their jobs, of course, yet somehow spend their lives 100% sure they can (and should) do everything. Learn to find them, and if you can find a way to get away with it, nobody’s going to think less of you for murdering them, because they somehow are never able to figure out that their confidence in themselves is grossly misplaced. ( Note/PSA : That’s a joke. Please don’t actually murder people. It’s all sorts of bad. Definitely avoid them, though.)

Some people will also have obscenely boring jobs, where they get to be confident that nothing new is going to come up. If that person is you, leave that job as soon as you can manage. No cutesy note, here. Seriously, quit as soon as you realize that you have that job and can afford it. That’s a career-ender.

And some of us don't really think about it.

But apart from those sorts of people, pretty much anybody who tells you that they think they fit into a new group on their first day is lying. Anybody who doesn’t have a learning curve when they start a new job is either lying or taking what have to be the most boring dead-end jobs available.

My point is that it’s both laughable and hurtful to talk about—for an example I’m familiar with, because so much has been written on the topic—women in the software industry suffering with impostor syndrome. It makes them seem like the very outsiders they worry they are, rather than bringing them into the fold, where some of us just come to terms with the worry more than others.

Possible Solutions

So, yes, we’re all worse at our jobs than we would like to be and there’s a part of each of us worried that someone is going to notice at the worst possible time. And the problem isn’t so much that this is a possible problem than this is your problem. That is, the effect the feeling has on you is what’s causing damage.

Can it be overcome? I don’t know. I’ve read quite a few articles over the years, and they’re all pretty silly, with “solutions” ranging from telling people that you believe in them—in hopes of normalizing it enough to sound sincere, I suppose—to a victim-blaming system that talks about the worker assessing where they really can improve.

As you might guess, I have a different outlook on the topic.

Instead, I would encourage you to come to terms with that anxiety and use it to your advantage.

First, try to understand that others aren’t necessarily better than you, and certainly not to the extent that you think you see. I have this conversation with colleagues all the time, because people tend to think that I’m particularly good at my job, even though I know my limits. The biggest difference, as I explain, is that they only see my work when it’s complete , whereas we can all see the flaws in our own work as we go. Anybody who sees the blind alleys I run down and obvious mistakes I consistently would realize that I’m not doing anything different except maybe trying more dumb things than my colleagues so that I keep moving. All the most successful people I’ve worked with function pretty much the same way, in any job, not being afraid to work outside our limits.

Second, realize that the ubiquity of impostor syndrome means that you have a support group of seven billion fellow people in need. Find people you trust and talk it out with them. Listen to their fears and validate the good that you’ve seen of their work. If they don’t volunteer it, ask them to assess where you are, in comparison to other people in similar jobs with backgrounds like yours. If you’re relatively privileged in society—like a White or Asian man in programming tends to be—take this a step further and be open about your own mistakes and things you needed to learn or re-learn. Normalize being imperfect, because there’s nothing more normal than that.

To build on that idea, also normalize being excited when someone tells you they’re not familiar with something, rather than merely being surprised. Many of us have become jaded by having information constantly thrown at us that we need to celebrate the process of seeking out what we missed more often. We need to normalize imperfection, self-education, and not feeling like you fit in, because it’s so normal that most of us ignore the feeling, like we ignore the air we breathe until it becomes a problem for us.

And finally, if you can, embrace that worry. The best part of a lot of jobs can be finding your footing, taking on opportunities to grow quickly. That nagging feeling inside you that something isn’t quite right can be a terrific compass to find those nooks and crannies. That fear is telling you where the interesting stuff is going on. If Sheryl Sandberg didn’t pirate the term for women giving up their lives to the corporate cause, I’d say it’s the sign to “lean in.” So, I won’t say that. Instead, I’ll just say that you have a tiny voice pointing you at interesting things to learn and opportunities to make things better.

Above all, don’t make it a contest. Given enough time and education, you can be better than anyone else at anything and anybody can be better than you. Acting like those temporary differences in skill are indicative of something deeper is just a waste of everybody’s energy.

Credits : The header image is Pug Imposter ‘Pug Love’ by DaPuglet on Wunderstock, released under the terms of the Creative Commons Attribution Share-Alike 2.0 Generic license.

Changing the Primary Git Branch

John Colagioia (he/him) — Wed, 29 Jul 2020 10:55:02 +0000

As a quick note, I released this post on my blog, this morning, so it can get to be (as I tend to be) a bit rambling. Oh, and the original text is on GitHub (licensed CC-BY-SA), so if anything seems muddy, by all means:

Leave a comment here,
Leave a comment on the blog,
File an issue on GitHub, or
Add a pull request!

jcolag / entropy-arbitrage

John Colagioia's Blog Posts

Recently, I decided to spend some time updating the repositories I use regularly to use a main branch instead of master, since the latter has unpleasant an alienating connotations.

Before you tell me that the term “obviously” derives from some different meaning, keep in mind that git takes a lot of its ideas from BitKeeper, which includes the concept of a master branch with slave branches. Likewise, if you’re going to argue that it’s from “master copy,” well, I have some awful news for you…

Similarly, if you’re going to ask why it’s suddenly important…that’s actually a good question, because this sort of terminology should never have been acceptable, and it’s embarrassing that we need to be reminded that our choice of words has consequences. But since everybody is thinking about it now, it’s a good time to do it, rather than sheepishly operating the minority of repositories that didn’t bother.

In other words, the best time to fix a mistake is always now , because you can’t fix anything before now and waiting until later just leaves the mistake where it is.

Altering the Branch

Plus, regardless of the intent, it’s still an uncomfortable term to have around, and it can easily give the impression that the project (and the industry as a whole) isn’t ready to welcome certain kinds of people. So, I’m going to change it where I can. Because I have dozens of repositories floating around, I decided to get it right once and then move the information to a script to make it reproducible when I stumble on another forgotten repository. Here’s the almost-final version of the script for GitHub, in case anybody wants to play along.

#!/bin/sh
branchpath=/settings/branches
url=$(grep 'url = ' .git/config | \
  cut -f2- -d'=' | \
  sed 's/^ *//g' | \
  sed 's/\.git *$//g' )
if case $url in git@*) true;; *) false;; esac
then
  url=$(echo $url | \
    sed 's/:/\//g' | \
    sed 's/^git@/https:\/\//g')
fi
git branch -m master main
git push -u origin main
echo Go update the default "${url}${branchpath}"
echo Then run:
echo git push origin --delete master

It doesn’t have any error checking, so don’t run it outside your repository’s local root folder, where you have administrative control over the remote/upstream repository.

The url variable digs into the git configuration file to find the remote repository. If the url we find starts with git@, modify it to a real GitHub URL instead of GitHub’s git@github.com:name/project.git format. All the pieces are there, of course. The case syntax looks like a mess, but it works without any special exceptions to compare the names and should be completely portable across shell scripts.

Then, we have two git commands. The first moves (-m) or renames the master branch to main. Next, we push the new branch and set the upstream (-u) to point to it. At this point, the local repository only has a main branch and is set to work exclusively with the remote main branch.

Why main and not trunk, primary, production, or something else entirely? When I change branches (not that I have many public-facing branches), my muscle-memory is still for typing git branch ma and then letting the shell handle auto-completion. You can call yours what you want, obviously, and even update the script to default to a name if you don’t provide it with one without too much trouble.

However, there are still two manual steps remaining. Over on the GitHub page, the default branch still reads master, so someone needs to change it by hand. So, that url work the script did earlier pays off here, producing a link to https://github.com/NAME/PROJECT/settings/branches, where the user can update the default. Obviously, I make no claims about the URL being correct for any host other than GitHub. (I really should start looking at GitLab; they seem like a nice company that people should pay some attention to.)

Once that’s done, you can delete the master branch entirely and never worry about it again…except for the rest of this post, because there are still some edge cases and improvements.

I’m not currently running any CI/CD, but if you choose to work with this script and do, you should take a minute and make sure that the tool never explicitly references the master branch in any of its work, at this point.

Advanced

It eventually dawned on me that the big manual step (changing the default branch) should probably be handled through GitHub’s API, so that the entire experience is just running a script. I initially didn’t feel like digging into it for this post, because learning how to make and handle API calls takes this project from a “quick script” to a full application that needs to handle remote authentication and all the possible error states. So, I wasn’t going down that path…until I remembered that GitHub has a CLI tool.

Installing that takes care of the “application” part of this. And with some research, I was able to work out how change the default branch entirely through the tool.

gh api repos/the-user/the-repository -X PATCH
  -F name="the-repository" \
  -F default_branch="main"

Obviously, change the-user and the-repository (both instances) to yours. But now that we have this piece of information, we can replace the echo lines above with the following.

user=$(echo "${url}" | cut -f4 -d'/')
repo=$(echo "${url}" | cut -f5 -d'/')
gh api repos/${user}/${repo} -X PATCH -F name="${repo}" -F default_branch="main"
git push origin --delete master

We know that a repository URL is going to look like https://github.com/the-user/the-repository/, so we pull off the fourth and fifth items, as delimited by slashes. The first is https:, the second is empty, and so forth.

Then, we call the GitHub CLI tool. The first time you successfully call it, it will ask you to hit Enter to open GitHub in a browser. There, you can authorize the tool to operate on your behalf. Then, once you’re authenticated, the script continues on to delete the branch.

Except for the one-time authentication step, there are no manual steps. I call that a success.

Local Clones

The above is all well and good, but it won’t work for an ordinary clone, because the work there is primarily administrative. If you have a clone of a project that has been making changes like this (for example, those of my projects that have migrated over), you’ll see an error something like the following.

From url-to-repository
 * [new branch] main -> origin/main
Your configuration specifies to merge with the ref 'refs/heads/master'
from the remote, but no such ref was fetched.

The above script won’t work, because it wants to affect the origin, which has already been affected and you might not have access to. Instead, you need something more straightforward, like the following.

#!/bin/sh
git checkout master
git branch -m master main
git fetch
git branch --unset-upstream
git branch -u origin/main
git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main

This makes sure that you start out on the master branch (to make sure that you don’t break a different branch), moves/renames the branch as above, gets the latest commits from upstream, disconnects the upstream repository, links the new branch upstream (almost the same as above, though not quite), and finally updates the clone’s local default branch to point to the right place.

Like above, run it in the clone’s root folder and don’t expect the script to recover easily if there are any errors.

Never Again…

It would, of course, be nice if we never needed to see a master branch again. Most of us don’t yet have the ability to just tell git that we never want a repository with one, but we can create a decent alias to use instead of git init until we do.

git config --global alias.new '!git init && git symbolic-ref HEAD refs/heads/main'

Now, we can call git new to initialize our repositories and the alias will create it and set the default branch name.

That’s not an ideal situation, since muscle memory will inevitably type the default command that we can’t override. But as a temporary measure, it should work out for most cases until the latest version of git shows up. To prepare for that, you might as well configure things now.

git config --global init.defaultBranch main

When Git 2.28 comes to your platform (Ubuntu is still on v2.25, for example, and if you're on a machine that doesn't auto-update git, you're probably always a few versions behind, too), it will pick up that option and all of your new local repositories will have default branches with your custom name.

Credits : The header image is tree, nature, branch, winter, black and white, plant, trunk, bark, high, monochrome, twig, trees, twigs, woods, branches, tall, monochrome photography, atmospheric phenomenon, woody plant by an anonymous photographer, released under the terms of CC0 1.0 Universal Public Domain Dedication.

Configuring a Visual Studio Code Extension

John Colagioia (he/him) — Wed, 15 Jul 2020 11:51:02 +0000

Leave a comment here,
Leave a comment on the blog,
File an issue on GitHub, or
Add a pull request!

jcolag / entropy-arbitrage

John Colagioia's Blog Posts

Continuing the discussion on creating an extension for Visual Studio Code, and now that VSCode Rat is—seemingly against all odds—working far better than it has any right to work, it’s time to add a configuration page. After all, it doesn’t make a lot of sense to ask people to blindly trust the URL that I provided or expect them to edit the code to insert their custom URL.

Writing Visual Studio Code Extensions

John Colagioia (he/him) ・ Jul 8 '20 ・ 5 min read

#techtips #programming #vscode

jcolag / vscode-rat

An extension (that nobody should use) to Visual Studio Code to "rat out" what the user is working on

Like last time, this turns out to be a project where it’s harder to find the information needed than to actually perform the tasks required. After a shocking amount of digging, I finally found the poorly named Contribution Points reference, which gives an overview of…a lot.

Adding Configuration Options

In the process of creating the extension, you might recall the contributes property in the package.json file. We used it to add a command to explicitly call our new feature.

That command is now obsolete, since all the VSCode Rat code runs autonomously, but the property is still useful, because it’s where our configuration options go. In the case of this extension, the property looks something like the following.

"contributes": {
  "commands": [{
    "command": "vscode-rat.helloWorld",
    "title": "Hello World"
  }],
  "configuration": {
    "title": "VSCode Rat",
    "properties": {
      "vscodeRat.targetUrl": {
        "type": "string",
        "default": "http://localhost:8080",
        "description": "Server waiting for VSCode Rat to rat you out!"
      }
    }
  }
},

We only have one option: vscodeRat.targetUrl, which takes a string (the URL), and has a default and description. And we can see that instantly, the next time we run, by opening File/Preferences/Settings in the menu, opening the Extensions item, and scrolling to the bottom.

That, then, gives us our option. If we need more, we can add more.

Not Quite

Now that I think about it, we probably want that URL to actually be a URL, so we need some validation. Visual Studio Code doesn’t give us great validation, but it’s decent. We can set a pattern property on each configuration option, which is a regular expression. So, we’ll throw that in.

"vscodeRat.targetUrl": {
  "type": "string",
  "default": "http://localhost:8080",
  "description": "Server waiting for VSCode Rat to rat you out!",
  "pattern": "https?:\\/\\/(www\\.)?[-a-zA-Z0-9@:%._\\+~#=]{1,256}\\.[a-zA-Z0-9()]{1,6}\\b([-a-zA-Z0-9()@:%_\\+.~#?&//=]*)"
}

Since JSON doesn’t know or care about regular expressions, all the back-slashes in the regular expression need to be escaped…with back-slashes, which is why they’re doubled.

Using Configuration Options

Now that we have a URL for any server willing to accept our POSTs, we can also use it. Since the configuration options can change at any time, we need to check it on every use, modifying our handler code to something like this.

vscode.workspace.onDidOpenTextDocument((document) => {
  const file = document.fileName;
  const post_data = `Opened ${file}`;
  const config = vscode.workspace.getConfiguration('vscodeRat');
  const urlParts = url.parse(config.targetUrl);

  httpPost({
    body: post_data,
    headers: {
      'Content-Type': 'text/plain',
    },
    hostname: urlParts.host,
    path: urlParts.path
  });
});

You’ll notice that we now call vscode.workspace.getConfiguration() with the prefix of our configuration options specified above, to get an object with the extension’s options. It returns something like this.

{
  targetUrl: "http://localhost:8080"
}

In this case, I’m using Node’s built-in url library to parse the single URL to provide the host name and path required by the httpPost() function I snagged for last week.

And that’s it. I can now configure VSCode Rat to send all of its updates to a mailbox on Henry’s Post Test Server, and we all lived happily ever after. Well, almost.

Oops…

When creating the extension, in order to get things moving as quickly as possible, I prototyped with Node’s built-in http library, which shouldn’t make HTTPS calls. That’s fine for the sake of this experiment, but an extension ready for production will need to handle any kind of URL, which probably means choosing a different networking library—I’ve heard good things about axios, for example—and rewriting the POST calls to work with it.

That, however, is far less important. At this point, we now have an extension that we can point to any server, even if the code itself isn't prepared to carry out that promise.

Credits : The header image is the sample image for VSCodium and made available (like the application and their entire site) under the terms of the MIT License.

Writing Visual Studio Code Extensions

John Colagioia (he/him) — Wed, 08 Jul 2020 11:15:02 +0000

Leave a comment here,
Leave a comment on the blog,
File an issue on GitHub, or
Add a pull request!

jcolag / entropy-arbitrage

John Colagioia's Blog Posts

Similar to the work on creating a browser extension such as URL Rat—part 1 and part 2 for what it took to make those work...

jcolag / url-rat

A browser extension to send visited URLs to a server, written as a test

Writing Browser Extensions

John Colagioia (he/him) ・ Jun 18 '20 ・ 5 min read

#javascript #browser #extension

Writing Browser Extensions with Configuration

John Colagioia (he/him) ・ Jun 26 '20 ・ 5 min read

#techtips #programming #javascript #browser

I’ve been looking at what it takes to develop an extension for Visual Studio Code, VSCode Rat.

jcolag / vscode-rat

An extension (that nobody should use) to Visual Studio Code to "rat out" what the user is working on

Setup

The easy way to get moving is to use the Yeoman scaffolding tool to generate the base code for you.

$ npm install --global yo generator-code
$ yo code

The yeoman (that’s the ASCII art fellow) will then ask for some basic information:

The kind of extension: A normal extension, color theme, language, snippet, key map, or pack of extensions;
The extension’s name;
The extension’s ID;
The extension’s description;
Whether you want type-checking in the configuration file;
Whether you need a git repository; and
Your preferred package manager.

This list of questions might change by the time you read this, but it should give you an idea of what to expect, regardless.

The file that matters most, of course, is extension.js. There’s also jsconfig.json, which you can look at and—at least for now—confirm that it’s mostly just the answers you gave to the questions.

Packaging

If you’re not familiar with Node.js projects, you’ll also want to get familiar with package.json. You rarely need to update it manually, but it’s good to know what’s lurking in there. Whether you know this file or not, the new aspects are:

activationEvents: The term is not subtle, referring to the list of events that activate the extension’s code. Visual Studio Code won’t load the extension until one of those events occurs. The default is a custom “Hello, World” event.
commands: Here, you’ll find a list of commands the user can run to operate the extension. The default—probably no surprise, given the custom event’s name—is a “Hello, World!” command.

You can probably guess that those are going to be important.

Try It

The easy way to test the extension is to open the folder in Visual Studio Code for editing and use the existing scripts to launch a new instance of Code with your extension loaded. To do that, click the “Run” button on the left (#1 on the picture below)—you can also type Ctrl+Shift+D—to get to the run/debug commands, then click the green Start Debugging arrow, marked #2 in the image.

This will start a second Visual Studio Code window. From here, you can open the Command Palette (View/Command Palette… or Ctrl+Shift+P) and, as you start typing Hello, you’ll see the new “Hello World” command listed. Select it and you’ll get a little toast notification, fulfilling the extension’s registered action.

So, that’s what we get without doing any work.

Monitoring Files

So, now we need to fill in real code.

The first step is going to be to give our extension a reason to run other than a manual command. Microsoft has provided a list of activation events, but as mentioned, that tells Visual Studio Code when to load the extension, not when to run the extension.

In our case, since we want to monitor what our user is doing—just like with URL Rat, please don’t try to use this in production, because it’s a security nightmare—we want to load the extension when the editor starts, so that just looks like this:

"activationEvents": [
  "*"
],

As in, change the activationEvents list in package.json to that.

Then, take a look at extension.js. The only code that looks important is activate(), which registers and subscribes to the command. So, we need to make our changes, there, subscribing to events that are relevant. Try something like the following in that activate() function:

vscode.workspace.onDidSaveTextDocument((document) => {
  const file = document.fileName;
  vscode.window.showInformationMessage(`Saved ${file}`);
});

vscode.workspace.onDidOpenTextDocument((document) => {
  const file = document.fileName;
  vscode.window.showInformationMessage(`Opened ${file}`);
});

vscode.workspace.onDidChangeTextDocument((changes) => {
  const file = changes.document.fileName;
  vscode.window.showInformationMessage(`Modified ${file}`);
});

This is extremely primitive, but it points us in the right direction. Rather than pop up the static message when the user asks for it, we now pop up messages when files are opened, saved, and modified.

Sending to a Server

Unfortunately, the version of JavaScript supported by Visual Studio Code doesn’t appear to support fetch(), so we can’t pull off the same trick we did with URL Rat. Instead, we need a function that looks something like this one, conveniently licensed CC-BY-SA 4.0, like the rest of my blog.

answer re: How is an HTTP POST request made in node.js?

Jun 16 '18

Simple and dependency-free. Uses a Promise so that you can await the result. It returns the response body and does not check the response status code.

const https = require('https')
function httpsPost({body, ...options}) {
    return new Promise((resolve,reject) => {
        const req = https.request({
            method: 'POST',
            ...options,
        }, res => {

…

Open Full Answer

function httpPost({body, ...options}) {
  return new Promise((resolve,reject) => {
      const req = http.request({
          method: 'POST',
          ...options,
      }, res => {
          const chunks = [];
          res.on('data', data => chunks.push(data))
          res.on('end', () => {
              let body = Buffer.concat(chunks);
              switch(res.headers['content-type']) {
                  case 'application/json':
                      body = JSON.parse(body.toString());
                      break;
              }
              resolve(body)
          })
      })
      req.on('error',reject);
      if(body) {
          req.write(body);
      }
      req.end();
  })
}

Then, we can replace our vscode.window.showInformationMessage() function calls with something like the following:

httpPost({
  body: `Opened ${document.fileName}`,
  headers: {
    'Content-Type': 'text/plain',
  },
  hostname: 'localhost',
  path: '/'
});

The verb used in the string depends on which action we’re talking about, of course. In addition, we might want to do something more with the change actions. When the files change, the object contains the same filename information as the other actions, but also includes enough information to describe the change, including the replacement text, the file offset where the change occurs, and the start and end line and column of the change.

With that information, we can collect the incremental changes made in Visual Studio Code over time, which would make it easy to reconstruct ideas that were either deleted and forgotten or lost in a crash.

What’s Left?

Like with the “first draft” of URL Rat , you probably noticed that VSCode Rat uses a hard-coded URL, whereas we would obviously want to be able to configure that information, just like we would with any other extension. And just like with the browser extension posts, this post is getting long enough that it’s probably a better idea to save that configuration discussion for next week.

Otherwise, it seems surprisingly difficult to know what events we can subscribe to and what they do (it took a long time to track down vscode.workspace.onDidSaveTextDocument(), probably longer than it should have), but once you know them, the rest of the development is remarkably fast.

Credits : The header image is the sample image for VSCodium and made available (like the application and their entire site) under the terms of the MIT License.

Writing Browser Extensions with Configuration

John Colagioia (he/him) — Fri, 26 Jun 2020 11:33:02 +0000

As a quick note, I released this post on my blog yesterday, so it can get to be (as I tend to be) a bit rambling. Oh, and the original text is on GitHub (licensed CC-BY-SA), so if anything seems muddy, by all means:

Leave a comment here,
Leave a comment on the blog,
File an issue on GitHub, or
Add a pull request!

This follows on to my post from last week.

Writing Browser Extensions

John Colagioia (he/him) ・ Jun 18 '20

#javascript #browser #extension

To recap it briefly, I have recently been investigating some possible projects that would benefit from having a simple browser extension to pass along real-time data on the user’s actions. It’s simple enough, but has enough detail to make a viable post…or two.

In this case, our extension is going to report every visited URL to a configurable remote address. The URL Rat extension, by last week’s post, was able to record every visited URL and send that information to a hard-coded URL. So far, so good. I’ll assume that you read that, since it’s short.

But now, we need to make that user-configurable, so we need to work with the browser storage.

Setting Configuration

First, we need to set up the extension to allow for some configuration work, which means adding some elements to manifest.json.

"options_ui": {
  "page": "popup/configure.html",
  "browser_style": true
},
"browser_action": {
  "default_icon": "icons/urlrat32.png",
  "default_title": "URL Rat"
},
"background": {
  "scripts": [
    "background.js"
  ]
}

Each of the three does its small part.

Browser Action

The browser_action element creates a toolbar button, in this case with an additional image that fits the button profile. All things considered, it’s pretty boring. Name and image.

The handlers go in…

Background

In the background.scripts element, we list the files containing the toolbar button’s event handlers. In this case, we just have one file, since we’re only handling one click. But if we had multiple buttons and/or had multiple features, we might consider separating that code into multiple files, listing them there.

I’ll talk about our click-handler in a bit.

Configuration Interface

Finally, the options_ui element tells the browser where to find the page with the configuration controls. In my case, I created a popup folder—even though it doesn’t actually pop up, but I initially considered that approach and never changed the name—where I’ve dumped all the code related to that options page.

These three items guide most of the work from here.

Event Handlers

As mentioned, we keep the toolbar handlers in a script that the browser runs in the background. In our case, we don’t have much to do, so it’s just this.

function handleClick() {
  browser.runtime.openOptionsPage();
}
browser.browserAction.onClicked.addListener(handleClick);

Just…open the options page when we hear that the button has been clicked.

Honestly, we don’t even need this, since the options page is going to be accessible from the list of browser extensions, but we might want to add features later and I wanted to make the extension visible, since it’s such a terrible idea to run it.

Configuration

I’ll spare you the boring HTML and CSS for URL Rat ’s options page. It’s a form with controls. If you’re not familiar with how those work, I’m not being dismissive when I say that you can find a better explanation than I could write here in just about any HTML tutorial.

Instead, we’ll just focus on the JavaScript code, since that’s the part that interacts with the browser storage. There are a few pieces.

function saveOptions(e) {
  browser.storage.sync.set({
    dest: document.querySelector('#dest').value,
    isActive: document.querySelector('#on').checked.toString()
  });
  e.preventDefault();
}
document.querySelector('form').addEventListener('submit', saveOptions);

Save options will take our two options (a URL and an on/off setting) and push them to browser.storage.sync, where we can pick them up later. I slightly reorganized the file for clarity, but the call happens when our options form is submitted.

In other words, the user clicks “Save” and the settings are stored in the browser.

function restoreOptions() {
  var storageItem = browser.storage.managed.get();
  storageItem.then((res) => {
    setOptions(res);
  });

  var gettingItem = browser.storage.sync.get();
  gettingItem.then((res) => {
    setOptions(res);
  });
}
document.addEventListener('DOMContentLoaded', restoreOptions);

When we open the options page, we want the controls to reflect the saved data. So, restoreOptions() takes two steps, instead of just the one involved in saving.

The first step is to pull the default options out of the browser.storage.managed area. After pushing that information to the options page, we check browser.storage.sync to see if the user has saved anything and, if so, set those options and overwrite the managed version.

The setOptions() function isn’t worth showing here, just a utility function to avoid duplicating some updates to the controls. You can find it in the repository along with the HTML and CSS, if you need it.

Wait, What Default Values?

You noticed that there’s no way to populate the manage storage, too, right? Getting this to work was probably the least-appealing part of the process for me.

This may only be for Firefox, but we need a JSON file to get information there. You might remember that the manifest.json file included a browser_specific_settings.gecko.id element. Implausible as it sounds, we use that ID to identify a new JSON file that holds our default values. In this case, it’s literally named urlrat@example.com.json, because our identifier is that imaginary e-mail address. And it looks like the following.

{
  "name": "urlrat@example.com",
  "description": "ignored",
  "type": "storage",
  "data":
  {
    "dest": "http://localhost:8080/",
    "isActive": "true"
  }
}

Copy, link, or move that file—depending on your preferences and how often you expect to update and keep the file around—to your ~/.mozilla/managed-storage/ folder, and you have now initialized your browser’s Managed Storage.

I did warn you that it wasn’t appealing.

Use the Configuration

From here, we should already know what to do. The main extension code, url-rat.js for this example, now needs to replace its hard-coded values with values from the browser’s storage. I took the following approach.

browser.storage.managed
  .get()
  .then((managed) => {
    browser.storage.sync
      .get()
      .then((local) => {
        var store = Object.assign(managed, local);

        // Assign the values in the "store" variable to individual
        // variables already used by the extension, and then put the
        // original extension code here.
      });
  });

I nested the then() handlers to make sure we have the data, even though that’s probably unnecessary, and then used Object.assign() to merge the two configuration objects into one.

Like the comment says, I should now have store.dest with the saved URL and store.isActive to decide whether to send the current URL to the server.

Overall

All things considered, if we ignore the managed storage shenanigans, the development process seems smooth. The only real snag I hit was in not realizing that the sync and managed storage areas were different, breaking the configuration page.

Also, the configuration page is ugly.

Other than minor issues like that, though, the fact that it only took a couple of hours in total to hack out a fully functional browser extension that performs a defined task starts to make this look appealing for certain kinds of projects.

Credits : The header image is Untitled by an anonymous PxHere user, made available under the terms of the CC0 1.0 Universal Public Domain Dedication.

Writing Browser Extensions

John Colagioia (he/him) — Wed, 17 Jun 2020 11:11:02 +0000

Leave a comment here,
Leave a comment on the blog,
File an issue on GitHub, or
Add a pull request!

I have recently been investigating some possible projects that would benefit from having a simple browser extension to pass along real-time data on the user’s actions. It’s simple enough, but has enough detail to make a viable post.

In this case, our extension is going to report every visited URL to a configurable remote address.

The Short Version

A browser extension for both Firefox and Chrome-based web browsers is some JavaScript code with a Manifest file. If you’re not packaging them up for the official download sites and are familiar with JavaScript, you can look up the manifest and work from there.

It’s a little bit more complicated than that, but not a lot.

Project Layout

A simple browser extension project has four parts.

manifest.json, which is (unsurprisingly) the project’s manifest file,
Some JavaScript code that does what the extension needs,
A folder for any assets that might be used, and
Icons to represent the project.

In the case of URL Rat, it looks something like this.

├── icons
│   ├── border-48.png
│   └── border-96.png
├── LICENSE
├── manifest.json
├── README.md
└── url-rat.js

LICENSE and README.md were created when I started the repository, and I created the images with ImageMagick, based on the suggestions on Mozilla’s tutorial.

convert -size 48x48 xc:#6187db border-48.png
convert -size 96x96 xc:#6187db border-96.png

Or whatever color I actually used. It’s not in my command history, for some reason. You can create a real icon, if that interests you for the purpose of the project.

Manifest Destiny

Because my plug-in needs to actually do something, I made some changes to the sample suggested by the Mozilla tutorial linked above.

{
  "manifest_version": 2,
  "name": "URL Rat",
  "version": "1.0",
  "description": "Sends each visited URL to a local server.",
  "permissions": [
    "<all_urls>"
  ],
  "icons": {
    "48": "icons/border-48.png",
    "96": "icons/border-96.png"
  },
  "browser_specific_settings": {
    "gecko": {
      "id": "urlrat@example.com"
    }
  },
  "content_scripts": [
    {
      "matches": ["*://*/*"],
      "js": ["url-rat.js"]
    }
  ]
}

Obviously, I changed the name, description, and script name. If this ever becomes a real project, the ID will need to change. But the two important items to talk about are the following.

matches provides a list of patterns that a visited URL needs to match. In the case of the Mozilla example, it’s only for Mozilla pages, whereas mine will be active on all pages, hence *://*/*, all protocols (HTTP, HTTPS, FTP, FTPS, and whatever else modern browsers support), all hosts, and all files on that host.
permissions is the list of resources the extension needs to access. I hate that this needs to be <all urls>, allowing it to send data to and receive data from any page on the Internet, since that’s a potential security problem that a bad actor or clumsy developer could exploit. However, since we’ll eventually want to configure the destination URL to point to any server (not in this post), it does make sense to request that flexibility, regardless.

I tried to narrow permission to a specific URL, like the one actually used in the HTTP requests, but couldn’t get that to work unless I specifically visited my own server, which is…somewhat less than useful.

Code

The code to capture and send every visited URL is simple enough, if a bit obnoxious to deal with the asynchronous code.

The first line is just configuration. You’ll need your own server listening on a port, somewhere, and url should point there.

var url = 'http://localhost:8080/';

This gets us our current URL.

var currentUrl = document.location.href;

The aforementioned obnoxiousness. We create an asynchronous, anonymous function to call fetch, so that we can call it immediately and not get yelled at by the interpreter for using await inside something other than an asynchronous function.

(async () => {

Now, we make the request. Note that it’s an HTTP POST request, so that it can usefully carry a message body (with the URL as that body), but the server I threw together wasn’t recognizing the bodies, so I also stuffed it into the header as X-This-Is-The-Url. The HTTP specification has no problem with you adding headers, as long as they all start with X- to avoid confusing any parsing code.

  const rawResponse = await fetch(url, {
    method: 'POST',
    headers: {
      'Accept': 'text/plain',
      'Content-Type': 'text/html',
      'X-This-Is-The-Url': currentUrl
    },
    body: currentUrl
  });

Now, we just need to wait for the response to come back and (if desired) do something with it.

  const content = await rawResponse;
  console.log(content);
})();

Once debugging is complete, we can scrap the logging statement entirely, since it only clutters the console window.

Testing the Extension

For Firefox, the Mozilla tutorial is right on. But to summarize...

Navigate to about:debugging,
Click This Firefox on the left-hand panel,
Click Load Temporary Add-on ,
Navigate to your extension’s folder,
Select any file in that folder, such as manifest.json, and
Click Open.

Assuming there were no errors, it should run until you reload or unload it, or until you shut Firefox down.

On Chrome (or Chromium, and probably most browsers built on Chromium, but I’m not testing them…), it’s similar.

Navigate to chrome://extensions/,
Switch to Developer Mode to the upper-right,
Click Load Unpacked to the upper-left,
Navigate to your extension’s folder,
Click Open.

Chromium will complain about the gecko.id field in the manifest, but that won’t affect your testing.

Where Next?

This is already getting too long for a “tip,” so I’ll save it for next week, but the obvious next big step to make this usable would be to add a configuration pop-up to replace the current destination URL with something other than http://localhost:8080. If you want to get to it before I do, the Mozilla tutorial links to “a more complex extension” that includes a toolbar button and a pop-up window. The Favourite Coluor is closer to the idea of a configuration page, too.

This would basically be that, but with a single place to fill in a URL (maybe validate it) and, optionally, a switch to turn the feature on and off to allow people to step out of the Panopticon when necessary.

Check back in next week, for that.

Packaging

Browser extensions are ZIP files of the folder contents (not the folder itself), renamed to *.xpi for Firefox. Here’s Mozilla’s example. It can then be sent to whoever needs to sign it, and you have yourself a browser extension.

Credits : The header image is Fotowerkstatt by Mattes, released into the public domain.

Experimenting with Worker Threads

John Colagioia (he/him) — Fri, 17 Apr 2020 11:17:21 +0000

As a quick note, I released this on my blog the other day and so it can get to be (as I tend to be) a bit rambling. One big change is that the blog version has an additional section at the end with a bunch of non-color design resources that I recommend. Oh, and the original text is on GitHub (licensed CC-BY-SA), so if anything seems muddy, by all means:

Leave a comment here,
Leave a comment on the blog,
File an issue on GitHub, or
Add a pull request!

As I have started working on a prototype desktop client for the twtxt social network, one of the key technical aspects is making a large number of web requests. Since I’m prototyping this using Proton Native and JavaScript is traditionally single-threaded, this presents a small problem: Since web requests can take a while to complete, traditional programming techniques would lock up the user interface, and that isn’t really viable.

Fortunately, as of Node.js v10.5.0, JavaScript on the desktop (such as Proton Native) has what they call worker threads, an approach to forcing JavaScript to perform multiple tasks at (approximately) the same time.

So, these are some quick notes on getting worker threads…well, working. It was easy enough to make it work, but there are some points where it’s unclear what is supposed to happen, with “minimal example” code all having strange and unnecessary features.

Threads, in General

Originally, Sun Microsystems created what they called “light-weight processes,” a system where multiple code-paths can run in parallel within the same program or processes. As other languages implemented similar approaches, the term evolved into “threads.”

If multiple threads are run under the same process, this typically gives benefits over a multi-process approach with interprocess communication, since most of the system state can be shared, saving overhead on context switches and thread creation. If you haven’t taken an operating systems course and don’t recognize those terms, they basically boil down to not needing to keep pausing and restarting programs, since everything should be running from the same package.

Generally speaking, threads have a handful of common operations:

Create sets up the new thread and assigns it a workload and initial data to work with.
Exit ends the thread from the inside, leaving the data to be harvested by the main program.
Join takes the data from the ended thread to make it available to the main program.

That’s not the entire model, of course. There are a lot of utility features allowing the programmer to set different parameters and retrieve information, but the core process is create-exit-join.

Worker Threads

Node’s worker threads…aren’t that.

In some ways, it makes sense. The standard approach to threading goes back to the early 1990s, and it’s now almost thirty years later, so maybe we’ve learned some things that make life easier. And then again…well, we’ll see.

Thread Creation

We launch a thread almost normally, but with a twist that makes me extremely suspicious about how this all works under the covers.

const { Worker } = require('worker_threads');
const worker = new Worker(
  './workercode.js',
  {
    workerData: someObjectWithInitialData,
  }
);

Typically, threads are given functions to run. Worker threads are different, though, taking a file. This is where suspicion starts to come in, as sending execution to a separate file implies that the thread is a separate program, rather than a single program sharing state.

Thread Handlers

The worker thread has three events we can chose to handle.

worker.on('message', this.acceptUpdate);
worker.on('error', this.reportUpdateError);
worker.on('exit', this.reportExit);

Each handler function takes a single parameter. The message can be an arbitrary object. The error is a JavaScript Error object. The exit code is an integer.

There is also an online handler, announcing when the thread has started execution, taking no parameters, if that’s useful to you.

Returning Data

Worker threads don’t really exit and join, though I suppose an exit value could be used to simulate that. Instead, the thread takes its initial state from a default workerData variable (imported from the worker_threads library) and sends messages back to the main thread.

const {
  parentPort,
  workerData,
} = require('worker_threads');
parentPort.postMessage(someObjectWithResults);

The message handler (acceptUpdate(), in the example above) then receives a copy of someObjectWithResults.

This also works in the opposite direction, with the main thread sending messages to the worker.

worker.postMessage(updateForTheThread);

These are surprising improvements over traditional threading libraries, because it allows the thread to easily send and receive updates whenever it gets them instead of waiting until it’s out of work to return everything it has collected or messing around in shared memory. However, this still smells of running in a separate process, basically treating the thread as a peer to coordinate with across a network connection or a special kind of shared file called a “pipe” that I won’t bother to discuss, here.

Join

All that said, we still get a traditional join operation, where the main thread can harvest data from the worker.

worker.getHeapSnapshot();

This call fails unless the thread has exited, meaning that it’s best run in the exit handler (reportExit(), in the example above), and makes the worker threads feel less like a separate process.

Going Further

So, after all that, I’m still not 100% convinced that worker threads are actually threads, but they seem to mostly do the job and that’s mostly what matters.

There’s actually a lot more available, here, too. The threads can communicate through console I/O. A thread can set up additional communications channels, which can be passed to the parent for another thread, allowing two worker threads to communicate directly. Ports (endpoints to a communications channel) can be manipulated to prevent the thread from exiting, and so forth.

Like I said, though, we have our basic create-exit-join model plus communication back and forth, which is fairly useful for a lot of kinds of work. If they’re not “really” threads, it doesn’t matter much, as long as the code doesn’t block and they basically act like threads.

Credits : The header image is Threads by Dave Gingrich and made available under the terms of the Creative Commons Attribution Share-Alike 2.0 Generic license.

Recutils, the Plain-Text Database

John Colagioia (he/him) — Sat, 28 Mar 2020 13:48:46 +0000

I originally posted a version of this to my blog as the first of my "tech tips" that are 90% notes to myself so that I don't need to keep researching. To my surprise, it has become one of the most popular posts, there, so maybe it'll be useful to someone here, too.

Background on Databases

Before I jump into recutils itself, if you're not already conversant in databases, I'll try to give a capsule version. For example, If you've taken the undergraduate database class or have written more than a few lines of SQL, you can probably safely skip down to Recutils, in General.

For the rest of you, here are some definitions, and then I'll go over some of the basics.

A database is a collection of related data, regardless of the form.

A relational database is a database (surprise!) where we store data elements based on their relationships (surprise again!) to other elements. Sometimes those relationships are described as records and sometimes they're described by metadata that connects separate records. The full definition centers on Codd's Rules.

A database management system (DBMS) is software that mediates access to a database to minimize the chances of a user or program shooting itself in the foot. For example, it might (depending on the system) moderate concurrent access or prevent you from deleting a record that another record points to.

Every database has a universe of discourse or mini-world that explains how to interpret the data, whether anybody bothered to write it down. We all somehow agree to never talk about this, even though "what does this data actually mean?" is one of the most important questions we can be asking, so I'm bringing it up here...

Instead of a universe of discourse, the thing we really get is a schema, the layout of what we store in each kind of record and how they relate to each other through foreign keys. I'll explain foreign keys in a bit, rather than try to assemble a coherent definition.

With that out of the way, record sets are called "relations" by theoreticians and implemented as tables, with records implemented as rows and individual elements as columns. The terms are mostly interchangeable, but you'll find table/row/column used by just about everybody in industry.

The last thing you probably need to know (unless you want to implement a DBMS) is that finding data ("querying") has what amounts to three steps.

First, we join the relations/tables we need, by which I mean taking records/rows that are connected through a common element/field/column. That commonality is (and should be specified as) a "foreign key," a kind of pointer into that other table's space. Even if we only care about one table, we still need to specify it.

Second, we filter the available records down to those whose elements are interesting to us, very similar to writing an if statement in general programming. We might want all the records, of course, which makes this step trivial. Or this might be extremely complicated, depending on what we want.

Third, we project the remaining records to just the fields that interest us. Again, we might want every field/column, making this part easier to specify.

If you're familiar with SQL, you'll probably recognize the SELECT column-or-columns FROM table-or-tables WHERE conditions-are-true format, where the FROM might list tables arbitrarily or might explicitly use JOIN to connect the tables. If you're not, recutils doesn't use SQL, so don't worry about it.

Now that we're all up to speed on databases, I can rant for a bit about why I care about recutils...

Recutils, in General

When I store information, I'm a big fan of light-weight, text-based formats for a variety of reasons.

They store and compress well, when that's important.
It's possible to quickly inspect or edit whatever I'm working on.
I can quickly search anything I'm working on with a tool like grep.
I'm not tied to specific applications.
It's easier to convert to another format, making it closer to future-proof.
Version control understands text better than anything else, so I can easily analyze changes over time.

For example, my blog, my novel, and my personal notes are generally all written in Markdown files. I store most of my spreadsheets as sets of CSV files, unless the formatting or calculated cells are important. In the rare cases I need something like a flow chart, it's usually in pic format.

Databases don't usually do any of this well, however. For the sake of efficiency, you typically dump all of your data into a binary file that the database management system has optimized for its own access and analytical purposes.

So, I got a little bit excited when I discovered recutils, a system to manage plain-text relational databases. It's not perfect (unless I'm missing something important), but it's definitely close to the sort of thing I would generally reach for, closer than using a free-form JSON file, for example.

File Organization

Unlike a traditional relational database management system, recutils needs you to make decisions about where to store records. Normally, the files are obscured from your vision, if only so that you're not tempted to monkey around with the content in an editor. Here, however, you need to decide what files you want and what you want to store in each.

And in fact, one of the clunkier aspects to recutils that I found is that it's difficult (impossible?) to work with an arbitrary subset of files, such as asking it to operate on two (and only two) files at once. So, there seem to be two options:

Put all the record types into one file, or
Separate the record types into different files, while specifying all of them with a wildcard (such as "*.rec") whenever working in situations that require more than one record type (like a Join.

Neither of these is ideal, of course, but the single file is probably acceptable for small databases and the latter is probably the most intuitive. I'll use the one-record-type-per-file approach, here.

Obviously, recutils doesn't actually care either way, and it's possible to combine the two, so that record types that are often used together can share a file, while the other clusters of types or individual types can be in their own files. You don't even really need to separate the files by type, and instead could assign records to files by any criteria you like, like your favorite color at the moment you're adding the records. You shouldn't do that, because it will make your life harder, but recutils isn't going to care, is my point.

Record Schema

While it's not strictly necessary to have a schema for records to conform to, it makes life significantly easier, if for no other reason than having some documentation. So, the header of a file of records, optionally, shows the types of each field, keys, and so forth.

It looks to me like unspecified fields are all arbitrary strings, multi-line with a plus sign (+) at the beginning of every line after the first. Other than that, if we are looking to store something like application users, we might have something like...

# The name of the "table" and a human-readable
# description
%rec: user
%doc: User information

# The non-multi-line string fields
%type: id int
%type: login line
%type: email email
%type: password line
%type: name line
%type: created_at: date
%type: updated_at: date
%type: role enum Guest User Moderator Administrator
%type: deleted bool

# Roles for fields; note that we sneak in "profile,"
# which is our multi-line string.
%key: id
%mandatory: login email password
%allowed: name profile created_at updated_at role deleted
# We could also %prohibit fields we want to ensure don't
# exist until we're ready to set them.

# Special restrictions
%auto: id created_at updated_at
%confidential: password
%unique: login email

That's probably more extensive than it needs to be for an example, but we see the record type name (%rec), decent coverage of the available types (int, line, email, date, enum, and bool, the different kinds of fields (other than prohibited), and constraints (%auto, %confidential, and %unique). I expect all of them are self-explanatory, except maybe line for a single line of text and confidential encrypting the field.

There's one useful type not shown here, that I'll get to in a bit.

Records

Each user record, now, is going to need to have our specified fields and the right types, all in name: value format. For example, I created the following user via the Fake Name Generator.

id: 7760
login: Brouch
email: VanessaJStolz@dayrep.com
password: $2a$11$WBkGS/r5/0tn2muqPXreVO40wy6dxdEzKj/lY5.y7BQ5YoOHKerNW
name: Vanessa J. Stolz
profile: A 78-year-old Libra from Atlanta, Georgia, who works
+ for De Pinna as a machine feeder and drives a blue 2000
+ Toyota Pod.
created_at: 2020-02-05 06:48:55-0500
updated_at: 2020-02-05 06:48:55-0500
role: Moderator
deleted: False

As a brief aside, recutils doesn't care about extra fields. So, if you want to add a birthday, significant other's name, and hat size to Vanessa's record above, but not anybody else's, that's probably going to be somewhat difficult to work with, but not destructive.

Beyond just editing our users.rec file manually, we have a few options to populate our file with data.

Conversion

If you have a pre-existing data source, recutils comes with two utilities to convert from common database types:

csv2rec turns each line of a CSV file into a record, with the header line providing the field names.
mdb2rec converts a Microsoft Access database (or a single table) into records.

I can confirm that the CSV program works well, but couldn't find an Access database to test.

Insertion

The recins utility, probably unsurprisingly, inserts records. Continuing with our users, it can be used something like...

recins --type=user --field=login --value=Gode1962 \
  --field=email --value=PhillipSCollins@teleworm.us \
  --field=password --value=password \
  --field=name --value="Phillip S. Collins" \
  --field=profile --value="Suburban driver of Dodge Rams" \
  --field=role --value=User \
  --field=deleted --value=False \
  --verbose users.rec

Assuming that none of the constraints have been violated, here, this will add the record to the users.rec file. If it fails, the --verbose argument will produce an error message explaining what went wrong.

If you have one record definition/schema in each file, recins will figure out where to store it, thankfully.

Exploring

We can get some basic information, now. The following gives us a count of each record type in the file.

recinf users.rec

The difference between that and retrieving the schema is a matter of an argument.

recinf --descriptor users.rec

And then we can check into more general queries of the data, though the syntax isn't SQL. We can pick a random set of records.

recsel --random 5 users.rec

We can get a "projection" of all data by specifying columns. The columns can also be renamed for convenience, which will be important, later.

recsel --print-values=login,email users.rec
recsel --print-values=login:Username,email:EMail users.rec

Replacing --print-values with just --print produces output that looks like a record, with name: value pairs. Also, if there are multiple types of records, we would need to specify --type=user.

We can also run arbitrary queries.

recsel --expression="id = 23573" users.rec
recsel --expression="email ~ gmail.com" users.rec

The expressions can get fairly sophisticated and this doesn't support SQL's syntax.

Deleting Records

Now that we understand insertion and query expressions, we should be able to figure out how to delete a record.

recdel --type=user --expression="login = Selits" --comment users.rec

The --comment argument deletes the record by commenting it out in the file, so that it can be found or restored later.

Multiple Tables

Now that we've settled our users, we can give the users something to work with, like favorite websites. We'll keep this website.rec schema simpler than our users.

%rec: website
%doc: Favorite websites submitted to the database.

%type: id int
%type: url line
%type: owner rec user
%type: created_at: date
%type: updated_at: date

%key: id
%mandatory: url owner

%auto: id created_at updated_at

The important field, here, sets up a foreign key:

%type: owner rec user

That is, owner is a reference to a record of type user. Because the user record type's key is an integer, the owner field is an integer, as well.

So, we have an example record.

id: 2031
url: https://www.gnu.org/software/recutils/manual/Foreign-Keys.html#Foreign-Keys
owner: 7760
created_at: 2020-02-05 06:48:55-0500
updated_at: 2020-02-05 06:48:55-0500

The 7760, in this case, maps this URL to the user table through its key, the id field.

Joining Tables

Now, we get to the important stuff. We can now get a list of URLs and the name of the person who submitted each one. Remember, we can't explicitly specify multiple files, so we either have a single file or a wildcard.

recsel --type=website --field owner \
  --print-values=url:URL,owner_name:Owner *.rec

And, of course, we could add a query expression to the command, if we wanted, or we could involve additional tables.

Note, incidentally, that the join is an inner join. One of the problems with recutils seems to be that other kinds of joins are not permitted.

Programming Support

In the grand scheme, a database that only allows ad hoc queries isn't particularly useful. Instead, we need some kind of API that can be used by software. This is only an API if you're writing shell scripts, obviously.

For programming in C, there's the librec-dev library package, and a lot of other languages can (indirectly) use that. But I only see bindings available for Python and PHP, with neither library updated in several years.

Also, the library documentation is "go read the source code," specifically the main header file. Apparently, the important functions in the library approximately map to...

librec Function	recutils Utility
`rec_db_query`	`recsel`
`rec_db_insert`	`recins`
`rec_db_delete`	`recdel`
`rec_db_set`	`recset`
`rec_int_check_db`	`recfix --check`

That's not spectacular, but still has some potential to improve, at least. Maybe at some point, when other things settle down, I'll revisit recutils specifically to document the library.

Other Capabilities

This just scratches the surface, honestly. Queries can be grouped. Aggregate functions are available. Subtypes are available, such as strings that conform to a regular expression or integers in a limited range, and can be named with %typedef. Records can have compound fields or be sorted. There are more encryption options than just %confidential. Output can be formatted with templates. Records can be updated. The recfix program performs diagnostics and repairs some issues.

In other words, it's a sophisticated system with a lot going for it, despite the handful of problems I've called out in the preceding discussion. It won't replace CSV files for my simple data sets, but for anything more complicated than that (other than calculation, which still needs a lightweight solution that doesn't have the overhead of a typical programming language), it looks pretty good!

As a good example of where I'll start using it, for the Silver Bat stories (more in that universe are definitely coming), I currently just have a Markdown file that lists all the characters. It includes their profiles from my profile generator, plus a brief biography and possible plot threads. When the time comes to share that information with other people, it would be nice to manage that data better and also turn it into a kind of concordance to be able to see where all the characters have been used.

For much more information, you can read through the documentation. It's not particularly friendly to newcomers who don't already understand how things work---which is hopefully nobody who got to the end of this post---but it's extensive enough to answer most questions that aren't related to the library.

All things considered, this seems like a decent foundation for projects where transparency is important.

Bonus: SQLite Export

Since I wanted to get up and running quickly with data with recutils, I decided to use a database I was already working with to populate my records. SQLite, which is where the data is, has a mildly clunky way of exporting tables to CSV files.

First, get into the SQLite shell.

sqlite3 database.sqlite3

Then, decide what you want to export and go for it.

.tables # <- This lists all tables
# Now that we know what tables are available,
# we can export them.
.headers on
.mode csv
.output users.csv
SELECT * FROM users;
# The table and file names don't need to match.
.output websites.csv
SELECT * FROM websites;
# If you don't issue a new .output directive,
# all your data will end up in the same file,
# which is going to make everything else more
# complicated than it needs to be.
.quit

Now, we have files that can be converted to record sets with csv2rec as described above.

Credits: The header image is work, technology, vintage, wheel, retro, clock, by an anonymous PxHere photographer, is made available under the terms of the CC0 1.0 Universal Public Domain Dedication.

Metal Umlauts, Searching, and Other Unicode Fun

John Colagioia (he/him) — Wed, 25 Mar 2020 13:38:28 +0000

(You can find the original version of this article on my blog, where I talk about this and a variety of other topics.)

Unicode—the computer “alphabet” that includes all the characters you see on this page, plus most modern writing systems in common use (∂), plus punctuation and currency (௹), plus arrows and mathematical notation (↛), plus drawing symbols (✵), plus emoji (🐣), and more—has a lot going on in it beyond the obvious complexity of multiple formats (UTF-8, UTF-16, GB18030, UTF-32, BOCU, SCSU, UTF-7, and probably others) and byte orderings. The part that grabbed my interest, recently, is the idea of Normal Forms, of which Unicode has four.

NFD: Canonical Decomposition
NFC: Canonical Composition
NFKD: Compatibility Decomposition
NFKC: Compatibility Composition

Specifically, Normalization Form Canonical Decomposition interests me, because it represents each accented letter in a string as the base letter followed by any accents.

Better yet, in JavaScript (and more languages; see below), it’s easy to change normalization forms. Specifically, for these purposes, we want:

str.normalize('NFD');

These decomposed letters have some nice uses.

Sorting

At least in English, diacritical marks are usually a marker for either history (fiancée, über, soupçon, Māori, piñata) or pronunciation (naïve, coöperate), rather than as an element of spelling; some of us are sticklers for getting the accents right, but most English-speakers ignore them completely. This is especially true of names, where we generally want a person’s name to be represented properly out of respect (Karel Čapek, Charlotte Brontë, Beyoncé Knowles), when that name can come from anywhere in the world, but English treats it more as an affectation than a critical element of the name.

Of particular importance, here, is that we generally wish to sort a name with accented letters as if the accents don’t exist. So, we want piñata to sort as if it was spelled “pinata” and Čapek to sort like “Capek.”

The decomposed form allows us to do this by stripping the diacritical marks out of the string when we sort it.

var sortedStrings = strings.sort((a,b) => {
  var aNorm = a
    .normalize('NFD')
    .replace(/[\u0300-\u036f]/g, '')
    .toLowerCase();
  var bNorm = b
    .normalize('NFD')
    .replace(/[\u0300-\u036f]/g, '')
    .toLowerCase();
  return aNorm < bNorm ? -1 : 1;
});

That admittedly looks a bit complicated, given the regular expression, but the entire process boils down to decomposing each string, and stripping off the diacritical marks (Unicode code-points 0x0300 to 0x036f), and converting the remaining letters to lower-case. Then, we just compare the resulting strings.

In other words, by normalizing the name, the computer represents “Čapek” something like

[C] [caron] [a] [p] [e] [k]

Then, we remove any diacritical marks (the caron or ˇ in this case) by replacing it with nothing, leaving us with only the unaccented Latin letters.

Or…

I can't think of a use for this idea, but it occurs to me that it's also possible to keep the diacritical marks and throw out or replace the letters.

Searching

More so than with sorting, it’s also a better experience to search without regard for diacritical marks. For example, an increasing number of laws (with political motivations that we don't need to discuss, here) are posed as “exact match” measures, which require that voter registration documents transcribed from handwritten forms be identical to personal identification documents, meaning that the exactness of accents and diacritical marks relies primarily on the comprehension and interest of an underpaid, overworked data entry clerk using a keyboard that doesn't have accents on it.

By the same token, even something with much lower stakes like searching an employee directory shouldn’t rely on the person searching for Beyoncé realizing that she has an acute accent in her name and that Human Resources input her name properly.

And that just barely touches on the problem that a standard keyboard for English doesn’t have a way to type accented characters, with operating systems often adding ways that aren't exactly trivial. So, even if a user has cleared the above hurdles, it’s still a waste of the user’s time to make them hunt down the exact spelling with diacritical marks.

We can solve this problem using an approach similar to what we saw in sorting, normalizing and stripping both the target string and the corpus being searched.

Metal Umlauts (or M͇ͭeţal Um͆l̼a͍u̓t̨s)

It’s a bit before my time, but one of my favorite television shows growing up (via re-runs and now streaming) is Mission: Impossible, in no small part because of the signage in their fictional foreign countries. Especially in earlier episodes, to make foreign countries seem both exotic and approachable to American audiences, show creator Bruce Geller had the idea of creating signs written mostly in English, but a version of English with clever misspellings representative of stereotypes of certain parts of the world, often including bogus diacritical marks.

For example, if you pay careful attention, you’ll easily spot both Zöna Restrik (for Restricted Area) or Prıziion Mılıtık (for Military Prison) in certain episodes.

And, of course, if you’re a heavy metal music fan, you’re undoubtedly familiar with the similar but distinct Metal Umlaut, though its use seems surprisingly limited to the diaeresis (¨) mark.

If we wanted to do something like transforming English text to "Gellerese"…well, you’re on your own figuring out how to change the base spelling in a reasonable way. But adding bogus diacritical marks? That, we can definitely do.

let output = '';
str = str.normalize('NFD');
for (let i = 0; i < str.length; i++) {
  const c = str[i];
  output += c;
  if (c.match(/[a-z]/i)) {
    // The math on the next line isn't necessary to the example;
    // I'll explain what it's for in the paragraph below.
    const rLen = Math.floor(Math.log2(Math.random() * 3));
    for (j = 0; j < rLen; j++) {
      const rCh = 0x0300 + Math.floor(Math.random() * 0x006f);
      output += String.fromCharCode(rCh);
    }
  }
}

Again, we normalize the input string. But instead of removing diacritical marks as we’ve been doing, here we visit each character and, if it’s a letter, we pick a random-but-small number of diacritical marks to add (using log2() pushes the numbers lower and biases the distribution towards the lower end, so we're more likely to get zero or one mark, but can potentially get more), and then selects the necessary diacritical marks from that same 0x0300 to 0x036f range we previously needed to remove.

If desired, this can easily be made more “intelligent” with lists of diacritical marks that are more appropriate to that letter, so that you don’t end up with implausible combinations like what you see in the above section heading.

While this sounds like just a joke or a tool for fiction, I now sometimes use techniques like this to make sure that diacritical marks display properly after processing text. By generating them randomly, in bulk, and in ways not generally found in real text, I get a better sense of how bad a display might look.

In any case, it might be a decent idea to call output.normalize('NFC') at the end, to set the characters back to their “composed” forms. And when I say “decent idea,” I mean “probably not necessary, but nice for the sake of consistency.”

Exception

One place where normalization has no effect is the Polish L-with-stroke (Ł or ł). It turns out that those are letters unto themselves, rather than letters with a diacritical mark. So, if you’re planning on using any of these techniques, you will want to take that into account, probably by replacing the character separately.

Other (Programming) Languages

The above sample code snippets are all in JavaScript, but the Windows API supports NormalizeString() and .NET has supported String.Normalize() for quite some time. Ruby, similarly, supports string.unicode_normalize(). It shouldn’t be hard to find the equivalent for other languages, now that we know the key words to search for are “unicode normalize,” maybe throwing in “nfd” or “decomposed” to make the context clearer.

Happy…err, umlauting? Sure. Let’s go with that!

Credits: Untitled header photograph from PxHere, made available under the CC0 1.0 Universal Public Domain Dedication.

Small-D date Night

John Colagioia (he/him) — Wed, 18 Mar 2020 12:24:23 +0000

This is the software kind of date—the kind at a UNIX/Linux command line—not the kind that requires you to break your social-distancing protocols…

Recently, I decided that I needed to pre-generate skeletal posts for an ongoing project on my personal blog. On the one hand, I’m interested in the overall result of seeing how many weeks I've committed to staring me in the face. But on the other—more relevant to an audience of fellow developers—the process of generating a list of dates in a shell script turned out to be interesting and satisfying.

Unsurprisingly, this centers on the GNU date command.

Another Day, Another Time

The date documentation makes it clear enough that there is a --date (or -d) option that allows specifying a date, but what’s not made clear is how flexible this specifier can be. Any of the following work. Deep breath!

A text-based date string, such as “18 Mar 2020” or “18 March 2020”
A numerical date string, such “01/02/03” (Day/Month/Year) or “01-02-03” (Year-Month-Day) or even “20200314 1440” (YYYYMMDD HHMM); years from 00–68 are in the 21^st century (2000–2068) and 69–99 are in the 19^th century, but 1–9 (no leading zero) are literal years CE 1–9
A time, whether on a twelve-hour (“10:35 p.m.”) or twenty-four-hour (“22:35”) clock, optionally including a time-zone specifier by name (“GMT” or “PDT”)
Complete timestamps (“2020-03-14T17:23:53.000000+0700”)
Upcoming days of the week (“Thursday”)
Space-separated relative specifiers (“last Thursday” or “2 years ago” or “last Monday +3 years +7 hours -21 minutes” or “now”)
Seconds since the UNIX Epoch
Any of the above prefixed with a long-form timezone like “TZ=’Europe/Geneva’”
A lot of these can be mixed, such as a numerical date string followed by relative specifiers.

Most of those are useful, but obvious and uninteresting. However, those relative specifiers? That’s the stuff of programming! With relative specifiers, we can do something like this somewhat-contrived example.

count=0
for filename in *
do
  date --date "thursday +${count} days"
  count=$(($count + 7))
done

This takes each file in the current directory (not for any useful reason, just because I wanted a finite set) and prints the date of every Thursday going forward by counting up in increments of seven days.

Harvesting Information

As far as it goes, the loop will serve us well…except that, if you’re familiar with Jekyll (which is what I use to run my blog), you probably know that I need a couple of different kinds of dates. The post filenames are in a “YYYY-MM-DD-title.md” format (the file for my original post on this is named 2020-03-18-date.md, for example), and the front-matter of the post includes a date and time, where the date can be a wide variety of formats.

This brings us to formatting. The format needed for the file name’s date is easy. In fact, there’s an in-built formatter for dash-delimited values in descending order of significance.

date +%F
# outputs "2020-03-14"

Another interesting (if not relevant to where this conversation is going) format gives us UNIX time in seconds.

date +%s
# outputs "1584534083"

You can prepend an @ to that output and use it to set the time using --date, as above. I also have some scripts that subtract two times in this format and feed the difference into date to get minutes and seconds.

What I decided to do with the pre-generated posts was to give them random release times. I actually publish the Star Trek posts whenever I have a few free minutes on Thursday evenings, so the actual time doesn’t really matter and I didn’t want to convince myself that there was some “official” time. So, I generate random numbers for the minutes and seconds after 5 P.M.; I then assemble the time in the post’s front-matter.

…Except that doing this isn’t aware of daylight saving time. So, if I generate 2020-03-19 17:06:15-0400 for my post tomorrow, that’s fine, because I’m currently on Eastern Daylight Time and will be tomorrow, too. But 2020-12-19 17:06:15-0400 is supposed to be on Eastern Standard Time, so the displayed time is “04:06:15 PM EST.” And if I leave that specifier off, Jekyll assumes that I must mean UTC.

I know, I know, daylight saving is stupid. Time zones are stupid. We should all probably just use UNIX time or Swatch Internet Time or something…

Anyway, one way that we can solve this new problem is by requesting the time zone’s offset on the relevant day, instead of filling it in manually.

date +%z --date "thursday +343 days"
# outputs "-0500" for me
date +%z
# outputs "-0400"
date +%F,%z
# outputs "2020-03-18,-0400"

I can then parse that time offset and append to my generated time to get most of what I want.

…Except that I didn’t actually use Thursday +343 days in my script, since I only learned that combining mixed kinds of specifiers was an option later in the process. Rather than specifying “Thursday,” I stupidly specified the date of the first Thursday that I wanted using the faketime utility, which came with an explicit time of midnight. So as I crossed daylight saving thresholds, the output times would bump back and forth by an hour, which meant being one day off (standard time shows as an hour behind, so midnight becomes 11 P.M. the previous night) for part of the year.

I could have solved this by doing better research on the aforementioned --date option, but I didn’t, and so the solution I came up with was to add another eight hours and request that date (%F) and time offset (%z). That's close enough for a quick hack.

Have a Good Time…

There’s obviously a little bit more to the script, like using a here document as a template for each file and creating the abbreviated titles for the filenames, but that’s all superfluous for this discussion, which is probably more than anybody could realistically want to know about generating a list of times.

Credits : The header image is untitled calendar by an unknown photographer from PxHere, made available under the CC0 1.0 Universal Public Domain Dedication.