DEV Community: Zak B. Elep

A few tips for Perl on Docker and Kubernetes

Zak B. Elep — Sun, 29 Nov 2020 11:49:11 +0000

@davorg started a thread on the FB Perl Community and Perl Programmers about docker-perl, and as its maintainer, I'm overdue sharing some tips about it so might as well write them here now:

Use a custom name/tag for Perl images you use

As documented in the Docker Hub, one can use the perl image with something like

docker pull perl
docker run -it perl perl -E 'say "hi there from Docker!"'

While doing this is enough for simple or exploratory cases, it is inadequate for development or deployment scenarios, as perl alone will pull the latest Perl image build of the latest supported Perl version (which is 5.32.0 now but could be different later on.) Fortunately, docker-perl provides tags to indicate specific versions (as well as options like :threaded) or size variants (like :slim) so one can also do

docker pull perl:5.32-buster
docker run -it perl:5.32-buster perl -E 'say qq{hi again!}'

There is another concern though: while Docker images have tags, these tags are floating in the sense that they don't always point to the same underlying image layer at creation time, as tags can be updated to point to another layer. This is best seen in the :latest tag which Docker uses by default when calling images with their bare names, but this behavior is also present in any other tags like :5 which is pointing currently to :5.32.0, but might later be updated to point to :5.32.1 or even :5.34.0, when the perl manifest on docker-library/official-images is updated.

These tags also get updated indirectly by way of updates on their base images; as docker-perl uses the buildpack-deps or debian:slim base images, when these get updated for security patches, these patches will eventually make their way into the perl images as well through the official-images build tooling.

For cases that require finer control of images, it might be helpful then to use a custom image name, tag, or both, when using docker-perl:

docker pull perl:5.32-slim-threaded-buster
docker tag perl:5.32-slim-threaded-buster myorg/perl:5.32
docker run -it myorg/perl:5.32 perl -e 'printf "hello from myorg/perl v%vd!\n", $^V'

This allows for some independence for updates and development between the perl published on Docker Hub and myorg's custom perl image:

# docker build -t myorg/myapp:dev .
FROM myorg/perl:5.32
ADD . /app
WORKDIR /app
RUN cpanm --installdeps .
CMD ["perl", "myapp.pl"]
...

Consider keeping a local/private registry to host Perl images

Using the perl images (especially like in the previous section) means that these will be copied into the local Docker host image storage, usually in /var/lib/docker/images but could vary depending on what storage driver your host is using. While this is usually enough in simple cases, consider that the base perl image is usually big (around 700-800MB unpacked,) so pulling this image afresh over multiple container hosts on a network will probably be wasteful.

Hence it is usually recommended to have some kind of local or private registry in the host or on a network if frequently working with containers (and more so if internet connectivity is slower/expensive.) There are several approaches to do this, and I've written about these before in my old blog:

Nowadays though, there are more options for using local/private registries: for example, if you have a Kubernetes cluster running in your network (or even as a small setup like microk8s, or an internal Docker container cluster using KinD or k3d,) there's usually an addon
that enables a local registry on the cluster. Here's an example for microk8s, which I extended for my own local network to be served under a TLS ingress:

microk8s enable registry
microk8s kubectl apply -f - <<REGISTRY_SVC
apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: default
spec:
  # microk8s deploys registry in its own namespace, so reach out
  externalName: registry.container-registry.svc.cluster.local
  ports:
  - name: registry
    port: 5000
    protocol: TCP
    targetPort: 5000
  type: ExternalName
REGISTRY_SVC
microk8s kubectl apply -f - <<REGISTRY_INGRESS
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: registry
  namespace: default
spec:
  rules:
  # I don't own home.example btw, just for illustration;
  # I use globally-accessible subdomains pointing to private IP
  - host: registry.home.example
    http:
      paths:
      - backend:
          serviceName: registry
          servicePort: 5000
  tls:
  - hosts:
    - registry.home.example
    secretName: wildcard-home-example
REGISTRY_INGRESS

I have a LetsEncrypt wildcard SSL cert installed and maintained on my microk8s via cert-manager, so this lets me push to this registry anywhere on my local network as

docker push myorg/perl:5.32 registry.home.example/myorg/perl:5.32
docker push myorg/myapp:dev registry.home.example/myorg/myapp:dev

and on somewhere else which might be running Podman instead:

podman run -it registry.home.example/myorg/perl:5.32 perl -V
podman run -d registry.home.example/myorg/myapp:dev

It also works on authoring Dockerfiles, especially useful when implementing GitOps workflows:

# docker build -t myorg/mojo:8.65
# docker push myorg/mojo:8.65 registry.home.example/myorg/mojo:8.65
FROM registry.home.example/myorg/perl:5.32
RUN cpanm Mojolicious@8.65
EXPOSE 3000
CMD ["/usr/local/bin/mojo", "daemon"]

For more sophisticated/production deployments, consider using Harbor or vendors such as Amazon ECR, Google Container Registry or Red Hat Quay.

Use container entrypoint for proper signals handling in Perl

This is already documented in the official-images docs for perl, but bears repeating here: containers don't provide a parent init process by default, which means that when starting a new container, any ENTRYPOINT or CMD set in the image the container will boot from will usually become the parent process within this container. This is important to note here for Perl (and also for other languages) as these tend to fall back to their parent init process for handling signals, so without this, it can appear to become unresponsive:

docker run perl:5.32 perl -E 'sleep 300'
^C
[refuses to die, even if sent SIGINT like above]

This is particularly important for Kubernetes, as its container pod lifecycle can use signals to probe for liveness or readiness of containers, and a running pod/container can seem to "hang" around indefinitely if these signals aren't handled correctly.

For perl, one needs to install a %SIG handler if they want/need to run perl directly as PID 1:

docker run perl:5.32 perl -E '$SIG{TERM} = sub { $sig++; say "recv TERM" }; sleep 300; say "waking up" if $sig'
^C
waking up
recv TERM

For most normal deployments however, it is easier to use a tool like tini, catatonit, or dumb-init for the container's ENTRYPOINT, alongside your perl script/executable in CMD, for example this small Mojolicious demo:

# docker build -t mojo-with-tini:dev .
FROM perl:5.32-buster
RUN apt-get update && \
    apt-get install -y --no-install-recommends tini && \
    cpanm Mojolicious
ENTRYPOINT ["/usr/bin/tini", "--"]
EXPOSE 3000
CMD ["/usr/local/bin/mojo", "daemon"]

For Kubernetes, one can also set the command and args for a pod:

kubectl apply -f - <<MOJO_POD
apiVersion: v1
kind: Pod
metadata:
  name: mojo-with-tini-demo
  labels:
    purpose: demonstrate-perl-signal-handling
spec:
  containers:
  - name: mojo-with-tini-demo-container
    image: mojo-with-tini:dev
    # redundant as we already set these in the Dockerfile above,
    # but showing here for illustration
    command: ["/usr/bin/tini", "--"]
    args: ["/usr/local/bin/mojo", "daemon"]
MOJO_POD

Outro

There's a few more tips on the queue that I'd write more here but this post is getting rather long now, so maybe later.

I'd love to hear more about how people use docker-perl, and especially more about how make developing Perl projects on container environments easier!

Scheduling cron tasks for Slackware Linux laptops

Zak B. Elep — Tue, 24 Nov 2020 05:39:28 +0000

Here's a small tip for Slackware Linux users on laptops, especially those that have periodic tasks run via the standard cron tool.

A default full install of Slackware will include the dcron implementation of cron that runs some scheduled system maintenance tasks (like log rotation or building locate databases) for root with a config suitable for a 24/7 machine:

root@darkstar:~# crontab -l
# If you don't want the output of a cron job mailed to you, you have to direct
# any output to /dev/null.  We'll do this here since these jobs should run
# properly on a newly installed system.  If a script fails, run-parts will
# mail a notice to root.
#
# Run the hourly, daily, weekly, and monthly cron jobs.
# Jobs that need different timing may be entered into the crontab as before,
# but most really don't need greater granularity than this.  If the exact
# times of the hourly, daily, weekly, and monthly cron jobs do not suit your
# needs, feel free to adjust them.
#
# Run hourly cron jobs at 47 minutes after the hour:
47 * * * * /usr/bin/run-parts /etc/cron.hourly 1> /dev/null
#
# Run daily cron jobs at 4:40 every day:
40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null
#
# Run weekly cron jobs at 4:30 on the first day of the week:
30 4 * * 0 /usr/bin/run-parts /etc/cron.weekly 1> /dev/null
#
# Run monthly cron jobs at 4:20 on the first day of the month:
20 4 1 * * /usr/bin/run-parts /etc/cron.monthly 1> /dev/null

On something like a laptop, this config will almost certainly not run adequately, especially if that laptop will be in sleep mode for the above scheduled times. Fortunately though, the dcron that ships with Slackware (as of 14.2 and -current/15.0) has some anacron-like features, so one can now replace this crontab with this:

@hourly ID=hourly /usr/bin/run-parts /etc/cron.hourly 1> /dev/null

@daily ID=daily /usr/bin/run-parts /etc/cron.daily 1> /dev/null

@weekly ID=weekly /usr/bin/run-parts /etc/cron.weekly 1> /dev/null

@monthly ID=monthly /usr/bin/run-parts /etc/cron.monthly 1> /dev/null

The @hourly ID=hourly syntax looks strange, but that's more a consequence of dcron's implementation of these features (see crontab:)

The formats @hourly, @daily, @weekly, @monthly,  and  @yearly  need  to
update  timestamp  files  when their jobs have been run.  The timestamp
files are saved as /var/spool/cron/cronstamps/user.jobname.  So for all
of  these formats, the cron command needs a jobname, given by prefixing
the command with ID=jobname.  (This syntax was chosen to  maximize  the
chance that our crontab files will be readable by other cron daemons as
well.  They might just interpret the ID=jobname as a command-line envi‐
ronment variable assignment.)

Finally, as indicated above, just check /var/spool/cron/cronstamps/root.{hourly,daily,weekly,monthly} to see the last periodic run for a given root cron job. Of course, this tip isn't limited to just root/system tasks; normal users can also use these anacron-like features for their own crontabs as well.

Enjoy!

Hacktoberfest 2020 report

Zak B. Elep — Fri, 13 Nov 2020 09:41:19 +0000

FPBP!

So, I'm rather new here at dev.to and since Hacktoberfest 2020 has come and gone, I might as well get started doing a new blog here. I've been doing Hacktoberfests for maybe a couple of years now, and while this year has seen quite the kerfuffle, that alone should not be enough reason to stop doing it.

Background

I work primarily with #perl stuff with bits of #docker and other tools in between, so most of my previous Hacktoberfests have focused on contributing to these. For this year though, I wanted to learn and use #nix some more, especially as I found the idea of reproducible developer environments across different machines to be interesting, and I've tried doing similar things (to mixed results) with Docker.

To help both Nix and Perl efforts, I thought learning enough of Nixpkgs to port missing CPAN modules I want or need would be a good idea to start...

Contributions

The Hacktoberfest game approved these PRs:

Add --dry-run completion to nix-build and nix-shell #37

zakame posted on Oct 02, 2020

nix-build explicitly documents --dry-run in its --help, while nix-shell passes it onto nix-store --realise.

View on GitHub

This got me started as I'm using zsh as my default shell, but the Nix tooling uses bash. Having become an avid user of nix-shell, I used the --dry-run flag (which it passes down to nix-store --realise) for estimating download sizes of stuff I want in the shell. Since this particular flag was missing in the completion, it seemed easy enough to add it in with this PR, despite some other note about changing the implementation entirely for the new nix command interface.

perlPackages.FutureAsyncAwait: init at 0.44 #100652

zakame posted on Oct 15, 2020

Motivation for this change

This adds async/await keywords to Perl to enhance the use of Future-using code.

This also adds XS::Parse::Sublike as a dependency of this package.

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)
- [x] NixOS
- [ ] macOS
- [x] other Linux distributions
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

View on GitHub

At my work, we're using Future::AsyncAwait heavily, so I missed this somewhat in Nixpkgs enough to get me started learning how to port it from CPAN to Nixpkgs. Porting is somewhat easy to do using nix-generate-from-cpan, but the other half of work is testing if it actually builds using nix-build to make additional changes if needed, and testing builds of other packages depending on it with nixpkgs-review preferably on both Nix-enhanced Linux installs as well as on native NixOS and macOS.

Thanks to stigtsp for showing me the ropes!

perlPackages.IOAsyncSSL: init at 0.22 #100820

zakame posted on Oct 17, 2020

Motivation for this change

This pure-Perl addon for IO::Async extends it to allow creating TLS/SSL servers or clients via IO::Socket::SSL.

This is a somewhat required module for other IO::Async-based code that connect/serve TLS/SSL resources, such as those in the Net::Async namespace.

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)
- [x] NixOS
- [ ] macOS
- [X] other Linux distributions
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

View on GitHub

Pretty much anything external that one can substantially reach with IO::Async needs TLS/SSL, so missing this in Nixpkgs seems odd. Adding it here seems a no-brainer, and would help in upcoming packages like Net::Async::HTTP and Net::Async::WebSocket (with the latter having pretty much a hard dependency for IO::Async::SSL anyway for working on anything non-trivial.)

Convert t/mojolicious/exception_lite_app.t to use subtests #1579

zakame posted on Oct 18, 2020

Summary

As the title says.

Motivation

I was already in the area for #1578 so might as well convert the test!

References

For #1520.

View on GitHub

This was a PR derived from an earlier more substantial work which was merged later, beyond the Hacktoberfest cutoff:

Replace prettify.js with highlight.js #1578

zakame posted on Oct 17, 2020

Summary

This replaces the old prettify.js highlighter with highlight.js.

Motivation

prettify.js has long been abandoned by Google so we need to replace it.

That said, highlight.js is also somewhat dormant (https://github.com/highlightjs/highlight.js/issues/1678) but still sees enough activity to get around.

References

This is an update of #737, fixes #1544.

Most of the work was already done by @dotandimet, please give credit where due. I just updated it for latest Mojo and highlight.js 💪

View on GitHub

I also made a few followup PRs to Nixpkgs that Hacktoberfest counted as bonus:

perlPackages.Future: 0.45 -> 0.46 #101118

zakame posted on Oct 20, 2020

Motivation for this change

https://metacpan.org/release/PEVANS/Future-0.46

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)
- [x] NixOS
- [ ] macOS
- [X] other Linux distributions
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

View on GitHub

perlPackages.LinuxInotify2: add description/license to meta #101182

zakame posted on Oct 20, 2020

Motivation for this change

Applying my suggestion from https://github.com/NixOS/nixpkgs/pull/101122#discussion_r508457565 to update package meta/documentation. No functional changes.

Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)
- [X] NixOS
- [ ] macOS
- [ ] other Linux distributions
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

View on GitHub

perlPackages.FutureAsyncAwait: 0.44 -> 0.45 #101402

zakame posted on Oct 22, 2020

Motivation for this change

https://metacpan.org/release/PEVANS/Future-AsyncAwait-0.45

Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)
- [ ] NixOS
- [ ] macOS
- [X] other Linux distributions
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

View on GitHub

There's also a few other PRs that didn't make the count, like the one for Mojo above, and these:

Replace prettify.js with highlight.js #22

zakame posted on Oct 24, 2020

Summary

This replaces the old prettify.js highlighter with highlight.js.

Motivation

prettify.js has long been abandoned by Google so we need to replace it.

References

https://github.com/mojolicious/mojo/issues/1544 https://github.com/mojolicious/mojo/pull/1578

View on GitHub

This change is now live at both https://mojolicious.org and https://docs.mojolicious.org (though still keeping an eye on the highlightjs stuff, should they need more updating.)

nixGL.nix: Add libGLX_indirect.so.0 symlink for nixGLIntel #64

zakame posted on Oct 28, 2020

This aids discovery of GLX capabilities on Intel-based setups for e.g. glxinfo, alacritty, kitty.

Not sure if using runCommand here is the right way, let me know if this needs revising, but it seems to work well enough for me...

Ref #24

View on GitHub

This one came about as I was looking to replace KDE4 Konsole on my Slackware 14.2 with a more modern terminal, and tried both Alacritty and Kitty. I eventually went with Kitty, but since this requires GPU acceleration I needed nixGL and Nixpkgs' own mesa_drivers as well.

perl: Add note about coexisting with Debian's /usr/bin/perl #1822

zakame posted on Oct 28, 2020

A few users of docker-perl have stumbled upon (and getting confused with) the default /usr/bin/perl shipped by Debian, so put a short note here explaining why it remains and also better describe how docker-perl and its tooling is installed/configured.

References:

View on GitHub

This was a somewhat longstanding documentation issue on docker-perl (which I maintain for sometime now,) and since it was flaring up on and off again, might as well make the time to document this issue so people using the docker-perl image know what to expect.

Progress

I was able to do a good many more PRs this year than in previous years, but at the same time many of them weren't eligible for this time due to the changes in the Hacktoberfest game that supposedly ruined it for everyone. OTOH, I never really actively sought for the reward(s), but rather the experience of this yearly adventure to do a bit more for Free and Open Source Software than what I usually do regularly.

Another thing that tempers my expectation for rewards is that I'd have to go waiting a few more months to receive it, and go through post office procedures which take a good hour or two to complete, which in this pandemic clown world would be a risk I'd rather not take...

Reflections

Learning Nix and Nixpkgs through porting Perl modules to it was a great experience, and I hope to continue doing so beyond Hacktoberfest. I'm particularly interested in making a reproducible Perl development environment more easier to do in Nix, maybe with some insight with sister languages like Ruby Bundix or Python pynixify.

I'll also be continuing some docker-perl work, especially as frameworks like Mojolicious are keen to get onto the cloud-native bandwagon (if not so already) in order to do easy, reproducible app deployments to Docker or Kubernetes; Nix may help yet again with that too, as it already has the tooling for exporting to Docker or OCI image formats.