Right tool for the job, but chances are the right tool is Perl.

The title is intentionally provocative. This is not really a post about Perl--though I do think Perl usually is the right tool (at least for me). It is about the general philosophy of picking a language for a project and learning new languages.

Chances are the right tool is [insert language here]

I sometimes spend days before starting a new project agonizing over what language I should use. I often create elaborate text files comparing the features, libraries, and ecosystems of several different languages as they relate to the project. I want the right tool for the job.

After all that, I usually end up going with Perl. Perl is my favorite programming language. It's what I'm most familiar with. In most cases, Perl is the right tool.

You may have a different language that you're most experienced with. In that case, chances are the right tool is [insert language here]. Insert your preferred programming language.

Like many hackers, I began with HTML and CSS. I started learning HTML and CSS when I was around 12 years old. I quickly progressed to JavaScript. I spent hours on the computers at school trying to build games and utilities that ran in the browser. I built an on-screen keyboard for...I don't really know why. I started working on a JavaScript Settlers of Catan that I never finished.

In high school, I got an after-school job and had actual money. I graduated from GeoCities to my own domain name and cheap web host. It wasn't long before I wanted to do more than just countdowns and animations in the browser. I wanted to learn the black magic of server-side programming. I remember choosing between learning Perl and PHP. I don't remember why I chose Perl. Maybe it was because I found a book about Perl on clearance at a mall bookstore. Maybe history would be different if I had picked up a PHP book instead. Had Node.js existed at that time, maybe I would have never been motivated to move beyond JavaScript.

Perl was the first programming language I fell in love with. I didn't dislike JavaScript, but I only ever used it as a means to an end. I wanted to create my own programs, and JavaScript and a browser allowed me to do that. With Perl it was different. I enjoyed Perl. It was fun to hack.

I wrote my own CMS for my website...from scratch. I'm wise enough now not to try that again, but it was a rite of passage (plus, before that, my website had been completely hand-coded HTML).

Perl provided me my first paying developer position as a student worker my freshman year in college (albeit, as a student worker, it was minimum wage and limited to ten hours per week). Students scanned their ID cards as they entered and left the weekly chapel service. A certain number of chapels were required each semester. At that time, the card scans were saved to a text file on a network drive. I wrote a Perl script that processed the files, stored the data in a SQL Server database, and created a web interface that allowed the dean to run reports on how many chapels students had attended and enabled students to view how many more chapels they needed to attend to meet the requirement for the semester. After coding the CMS for my personal website with nothing but CGI.pm, I was wise enough to pick CGI::Application for this project. (Today I usually use Mojolicious whenever I need to create a web app or web interface.)

Today I work at that same university as a DBA. Recently a co-worker dug up a copy of that web app I wrote as a freshman (thankfully the university has since moved off that homegrown solution to an off-the-shelf solution). I was appalled at the code. Both Perl and I have changed a lot in fifteen years. Even as a DBA, Perl is still the tool in my tool chest I most often turn to. Most of the tasks I need to solve involve getting data out of (or into) a database, formatting it (usually as CSV or XML), and sometimes using an API or SFTP to integrate it with a third party. Those are the kind of tasks Perl excels at. CPAN has DBI, Text::CSV, and Net::LDAP.

...except when it isn't.

Right tool for the job will usually mean using more than one language on a project. Even if you're writing the back-end of your web app in Perl (or Ruby or Python or Groovy), you're going to need HTML to build what your user will see in their browser, CSS to style it, and JavaScript if you want any client-side interaction. I think all of these are so obvious most developers don't even give it a second thought. When I talk about choosing a language for my next web app, I'm talking about whether I should go with Perl that I'm familiar with or try out something new like Erlang for the back-end. HTML, CSS, and JavaScript are a given.

If my app needs to query a relational database, I'm going to use SQL. I've tried a couple different ORMs, and maybe I'm biased because I'm a DBA, but I always find working with SQL directly to be a better experience. I think what a lot of developers are trying to avoid with ORMs is mixing SQL and Perl. I agree that's ugly. I keep my SQL in separate files and load it with Template::Toolkit. Treating my SQL as templates, I'm able to do a lot more than I could with just bind variables.

While SQL is the right tool for querying databases, most database vendors have extended SQL to turn it into a Turing complete, general-purpose programming language. In the case of Oracle (the database at work), that is PL/SQL. I know I'm a DBA, but I find PL/SQL particularly unpleasant. It is very verbose. What I could do in a few lines of Perl takes me tens of lines of PL/SQL. We have an emergency notification service we use to send texts to all students and employees in the case of an emergency. On a regular basis, we need to export mobile phone numbers from our database, format them in a particular format, and transfer them using WebDAV to the service. My predecessor had implemented this as a stored procedure written in PL/SQL. We had need to make some changes to the output format. I decided to rewrite it in Perl, and I haven't regretted my decision for a second.

I also know a co-worker who did the opposite. We recently hired a new analyst. The analyst he replaced had a lot of Perl scripts to automate formatting files and such. The new analyst is rewriting most of these in PowerShell. Even though I think Perl excels at automation and formatting files, PowerShell is probably the right tool for him, because he is more familiar with PowerShell. I tried to convince the co-worker of the benefits of learning Perl, but ultimately I wasn't successful.

I like to tinker with the Arduino. I can use other languages on the Arduino, but because of the ease of use of the Arduino IDE, chances are the right tool is the Arduino language.

I haven't done any mobile app development yet, but I would like to develop a mobile app someday (even if just for fun). In that case, the right tool is often dictated by what is supported by the mobile platform you're targeting. If I want to write an Android app, I could choose Kotlin. For an iOS app, Swift is probably the right tool. I could also choose a cross-platform framework such as LambdaNative, Flutter, or React Native.

Most of the time performance is not really that big of a factor in choosing the right tool. Developer productivity often outweighs shaving off a few milliseconds of processing time. For most applications, whatever language you're most familiar with is probably fast enough. There are exceptions though.

I know Electron is everyone's favorite punching bag right now, so I hate to heap on more criticism, but it's mostly deserved. I've tried using VS Code. It is attractive and has some cool features, but it's also slow as molasses. If I have a few tabs open (say a Perl script, HTML document, and CSS style sheet), it takes several seconds when switching between tabs. My workstation is a Core 2 Duo with 8 GB of RAM. I know, I know, Moore's law and all that, but I shouldn't be forced to upgrade my computer just to use a text editor. It's not like I'm trying to play a realistic 3D game. Maybe you have an app you wrote with Electron and it is simple enough that it performs well even on older systems. Good for you. Then Electron probably is the right tool for you. But I would say that Electron was the wrong tool for VS Code (not that me saying that really matters).

I'm working on a project where I'm writing a FUSE file system to mount a cloud service. My first thought was that something like a FUSE file system might be one job where my beloved Perl might not be the right tool. For performance reasons, I thought I would need to write it in a compiled language like C or Rust. Then I came across an article about writing a network file system in Python, and that any latency from the network outweighed choosing Python. Sure enough there was a CPAN module with FUSE bindings. I did some tests, and Perl was more than fast enough. I should have never doubted.

I also write scripts in Bash and AWK. For simple system administration tasks, they are the right tools.

So Many Tools to Choose From

I have an ever growing list of programming languages I would like to learn someday. Lately dependent types have caught my interest, and I would like to learn Agda (or maybe I should learn Idris instead). Rust seems to be the latest hotness, and I would like to learn a compiled language. I've wanted to learn Ruby almost since I first learned to program, but I've never got around to it.

I've read articles that recommend learning a new programming language every year. I like the idea, but I average more like a new programming language every seven years. There's a book titled Seven Languages in Seven Weeks. Madness, pure madness, I say. I'm somewhat ashamed to admit the list of languages I know is short (in the order I learned them): JavaScript, Perl, and Scheme.

Well, I guess it kind of depends on what you mean by "know a language." I consider myself to have mastered JavaScript, Perl, and Scheme. I know the idioms. I'm familiar enough with the ecosystems to know what libraries and frameworks best fit a situation. I can write simple programs without consulting the documentation. I have created at least one non-trivial project with each.

After I had known Perl for several years, probably inspired by one of those many learn a new programming language blog posts, I decided to expand my horizons. I had heard all kinds of wonderful things about functional programming, so I wanted to learn a functional language. I considered Haskell, but I had a friend who had actually majored in computer science (I majored in religion and have never taken a computer science or programming class) that a class on Haskell had been one of his most difficult classes, so I sought out a more gentle introduction to the world of functional programming. I was drawn to Lisp and finally decided upon (for reasons I no longer remember) the Scheme dialect.

I'm also comfortable with HTML, CSS, SQL, Bash, AWK, YAML, JSON, and the list probably goes on. I take these as a given though. They are necessary tools that go along with being a developer. I guess if you're an embedded software engineer who only ever writes C for embedded systems, you may not know HTML and CSS. If you're a Windows developer, maybe you don't need Bash (but I assume you do know some equivalent like PowerShell).

Programming languages share a lot in common. If you know JavaScript, even if you've never used Python or C, you could probably understand a lot of source code written in either. In fact, I consider it an essential skill of a developer to be able to read source code in a language you don't know and at least be able to identify what certain sections of the code do.

If two programming languages use two different paradigms (say, object-oriented and functional), they may be more different than two languages of the same paradigm. That's why it's useful to learn at least one language from each paradigm, but even a C++ hacker shouldn't have too much trouble figuring out Haskell or OCaml source.

I've fixed bugs in or made minor modifications to programs written in C, COBOL (yes, parts of our ERP are still written in COBOL), Java, PHP, Python, and Ruby (and maybe a few more I've forgotten about). I don't consider any of these languages that I know. I was able to use my knowledge of programming in general to browse the source code, identify the relevant area of the code, look up any unfamiliar keywords or syntax in that area, and then make the needed change.

The university I work for recently changed the LMS we use. The analyst who supports the LMS could export grades to a CSV from the previous LMS that he could then import into our ERP. The new LMS doesn't have a feature to export grades manually. Instead you can configure an HTTPS endpoint that it publishes grades in CSV format to. The documentation gave an example PHP script that would take grades published to the endpoint and save them to a text file. We could, of course, have written our own endpoint in Perl, Ruby, Python, etc., but to save time we used the example from the documentation. I don't know PHP, but I was able to look up just enough info about the keywords in the example to make a few small modifications to it. (Now the analyst has requested a few more modifications, and I am seriously considering rewriting it in Perl to make it easier for me to maintain.)

We also recently switched from a proprietary COBOL compiler to GnuCOBOL to save money. Our ERP vendor only officially supports the proprietary compiler. We have an analyst who is old enough that he actually started out programming COBOL. His help (and reading the Wikipedia article on COBOL) was enough to make the few changes I needed to get the COBOL to compile with GnuCOBOL. I actually modified the Makefile (another language itself) to use sed to modify the COBOL right before compiling it. This way upgrades to our ERP won't overwrite our customizations to the COBOL.

I've also read the introductory tutorial on the websites of several languages I'm interested in and sometimes went on to solve a few Project Euler problems with the language. I don't consider those languages I know. I don't consider myself knowing the language until I've actually built something with it (and, no, a few Project Euler problems don't count).

I completed the interactive tour on the Go website. I didn't like the Go language (I felt like it was telling me what to do), but I think that tour should be the gold standard for language tutorials. Every language should have an interactive tutorial on their website like that one.

The Joy of Perl

I took the Go tour when I was thinking I would need to write the FUSE file system I mentioned above in a compiled language. It was disheartening. For me, programming is something I enjoy. Yes, I program for work, but I also program for fun on my own time. I enjoy programming in Perl. Go just didn't seem fun. If I used Go for the project, I feared I might not enjoy it.

I think that is one thing holding me back from learning more languages. What if I invest all this time into learning a new language, but then I don't enjoy the language?

I may also love the language and never find out if I don't give it a try. I learned Scheme after I had several years of Perl under my belt. I like Scheme a lot. I don't use it as much as I use Perl. The main reason is libraries. There is a CPAN module for almost anything I need. Even Racket with its batteries included philosophy and a package repository doesn't come close to what's available on CPAN.

I learned Scheme in 2011 (at least that's the date of my first commit of Scheme code on GitHub). If I don't want my average of a new language every seven years to slip, I better get to learning my next language. I want to create a ray tracer using Ray Tracing in One Weekend. Maybe I'll write it in Rust. Or OCaml. Oh no, I feel days of research and detailed text files coming on. Maybe I'll end up doing it in Perl.

Cover photo by Todd Quackenbush on Unsplash. Title inspired by this article.

Comments

[Dave Jacoby]

Fellow Perl guy here, but...

I agree that's ugly. I keep my SQL in separate files and load it with Template::Toolkit. Treating my SQL as templates, I'm able to do a lot more than I could with just bind variables.

Template::Toolkit++, but I'm not quite sure how it works with SQL. With bind values, I get protection that I have to cover myself if I add the strings in with Template. My friend Petdance has a site on how to protect your database.

So I guess I'm wondering how you use TT and how you keep yourself and your DB safe.

[Matthew D. Miller]

I still use bind variables in combination with TT, but I'm able to do "more than I could with just bind variables." For example, query.sql might contain:

select [% column_name %] from some_table where some_column = ?

and this is how I would process it with Perl (using Template::Toolkit::Simple):

my $sql = tt->render( 'query.sql', {
  column_name => $dbh->quote_identifier($column_name),
});
my $sth = $dbh->prepare($sql);
$sth->execute($some_value);

Of course, you could also do the above with sprintf, but splitting it into a separate file with TT I'm not mixing SQL and Perl. I can open my SQL up in an editor and get syntax highlighting for SQL.

I can use TT to include SQL snippets in a larger query instead of copy and pasting the same snippet over and over. In my context of higher ed, I need to limit many of the queries I run to the current term. I can throw SQL querying the term table for terms where the start date is less than sysdate and the end date is greater than sysdate into a file and then include it in other queries with where term in ([% INCLUDE active-terms.sql %]).

[Aurelia Peters]

Thank you for this. I'm a Perl guy, and in my current position, I and my junior colleague are the only Perl people in the company. Mostly we're a .NET shop, and word from On High is that one of the big applications we are responsible for is to be rewritten in .NET.

Every year, it seems to me, Perl jobs are getting fewer and farther between, and the language that I love is becoming less and less relevant. So I'm trying to learn new languages to keep my skills relevant. It's nice to know that there's still some love for Perl in the world.

[Matthew D. Miller]

Although Perl is in the title, this post was more about programming in general. I too am sad Perl seems to be becoming less and less relevant. I may follow this post up with a post where I advocate for Perl and give reasons why someone should consider Perl in 2019.

I'm fortunate that since I work as a DBA instead of as a developer, most of the coding I do are scripts and tools used just by me to automate database and system administration tasks. My work doesn't really care what language I write them in. I could probably write them all in Smalltalk if I wanted. I manage databases on both Solaris and Linux servers, and Perl can be found on both, which makes Perl a natural choice if you ask me.

[Jesse Phillips]

I'm going to roll back to the perl to powershell rewrite. The right tool here was probably the one that already exists. There is a time and place for rewrite, but I don't suggest not knowing the language as being on.

[Matthew D. Miller]

I agree with you in general. I considered adding a point that when contributing to an existing project, the right tool for the job is the language the project is already written in. In the example I gave of the co-worker rewriting Perl scripts in PowerShell, these are all little scripts that were written by his predecessor to make certain tasks easier (what Perl excels at). They are only used by that position.

I'm known as the Perl guy around the office, so early on (the new co-worker has been here a little over a year now) the co-worker did ask for my help when he needed to make a few adjustments to one of the scripts. The script was only a few dozen lines (and definitely not what I'd consider Modern Perl--no use strict or use warnings). The script I helped with takes a spreadsheet provided by the payroll office (saved as a CSV) of retirement contributions, and sums up contributions per employee. The spreadsheet from payroll has multiple lines per employee: employer contribution, employee contribution, employer match, etc., but when the file is submitted to the company that manages our 403(b)s, they want a single line per employee. The script does have to be maintained. There are occasional changes to contribution codes and/or the format the third party expects.

It may have been best to leave them in Perl, but since the scripts are short, self-contained, perform simple tasks, only used by that position, and must be maintained by that position, I don't think rewriting them was a terrible decision.

[Simon Hawkin]

If you like scheme, check out clojure, it is a joy to work with.

[Eljay-Adobe]

Scheme and Lisp are programmers programming languages. Clojure combines that with functional programming.

I'd expect that to be a happy union.