DEV Community: Joel Hayhurst

WSL for Rails developers

Joel Hayhurst — Wed, 19 May 2021 17:43:58 +0000

It is now possible, thankfully, to develop in Rails on Windows. For many years, it seemed one needed a Mac for this purpose; and I have nothing against Macs, and enjoy the MacBook Pro, which I have used throughout my career. But I also have a Windows machine. I like to play games sometimes, and now that I work from home, along with an increasing number of people, I have access to it. I like the idea that developers should be able to enjoy their preference in environment, where reasonable. Some prefer to boot directly into Linux, which is also a perfectly fine choice. On a whim, I decided to try Rails development on Windows, and it has worked out nicely.

With WSL, the Windows Subsystem for Linux, we can now run a Linux distribution within Windows 10. You can use it similarly to how you might use Homebrew on a Mac, and the whole thing is surprisingly seamless, especially after some recent improvements. All of the Linux development and server tools you're used to are present, and work the same as you are familiar with.

Installation

It's relatively easy to install; first you have to enable WSL as a "Windows feature", and then you go to the Microsoft Store and install your preferred distro. I know they have Debian and Ubuntu available, as well as some others. I chose Debian.

Terminals

As for terminals, there's a few options. The default terminal that comes with WSL is named after the distribution you installed, so for me it is simply called "Debian". It has the Debian swirl logo, which I like, and the default color scheme and font it comes with I find pleasant. On the downside, I've found it will always scroll down if new content comes up (say you are tailing a test log), so I got in the habit of waiting for the tests to finish before examining logs. It also doesn't have tabbing, but I found I don't need tabbing as much as I thought; I'm not on a laptop screen, so I have enough space to keep separate terminals open. It is possible to use PowerShell to access WSL, by running the wsl command, but I find the color scheme is a bit off. And there's other options, like the new Windows Terminal, which does have tabbing and lots of customization options, but requires editing a text file to change settings (you can do it, you're a developer!), and still feels kind of in beta. In general though, I'm fine with the default WSL terminal.

VSCode

WSL also integrates perfectly with VSCode, which I use for most of my code editing. You can edit files directly within WSL and have them immediately affect the app. I mention this because there used to be an issue with this in WSL 1, but it has been resolved since WSL 2.

Docker

Docker Desktop also integrates very nicely with WSL since version 2. Before WSL 2, Docker had to use Hyper-V to emulate Linux, but now we don't even need Hyper-V enabled. Docker now uses the WSL 2 backend, and it seems quite a bit faster to start as well, in my experience.

Filesystem

When you use WSL, you will want to store all of your relevant code within the WSL filesystem, not on the Windows filesystem. In WSL 1 there was more crossover, but not in WSL 2; everything goes on the Linux virtual drive. It is still possible to browse these files using Windows Explorer, if you are so inclined, but in general probably best to stick with Linux tools when using this drive. Still I do open one of my WSL directories in Explorer regularly for the purpose of easily opening screenshots produced by Capybara.

Copy-and-paste

One other note about transitioning from Mac, you'll probably want to enable the Ctrl-Shift-C and Ctrl-Shift-V copy-paste functionality in the terminal. On Mac, we have that handy Command key, but on Windows we are stuck with just Ctrl, which is used plenty by Linux itself; so Ctrl-Shift is a necessary compromise. It's not that hard to get accustomed to, and besides that, we can also use right-click to copy and paste text.

Caveats

I've been developing with this system for over a year now and haven't run into any serious problems. Probably the biggest issue I had is that WSL, combined with the code repositories of some large apps I'm working on, used up a lot of disk space, and my C: drive only has 256 GB. I did however find a way to move WSL to a separate drive, a nice 1 TB SSD, and that solved my space issues. This can be accomplished using the wsl --export and wsl --import commands without much fuss.

Another issue I've faced is that when working on a larger application, a lot of memory is used up by WSL. Running two operating systems simultaneously uses more resources. Even after quitting the applications, and quitting Docker, this memory would still be reserved by WSL. The solution is that when I'm done with using WSL for the day, I run wsl --shutdown in PowerShell and that frees up all the memory again. I went ahead and upgraded to 32 GB of RAM to avoid these paging annoyances. However, this upgrade is only necessary because of the size of this application. Every other application I've worked in was comfortable with 16 GB.

Conclusion

These are all of the thoughts I have on WSL at this time. I hope you find it useful to be able to program in Rails on Windows, if that is your desire. If you give it a shot, you might find it's more comfortable than you expected. Thanks for reading.

Ruby on AWS Lambda: Layer Dependencies

Joel Hayhurst — Tue, 11 Aug 2020 20:00:41 +0000

This article is part of our Ruby on AWS Lambda blog series. A recent project had us migrating an existing PDF document processing system from Rails Sidekiq to AWS Lambda. The processing includes OCR, creating preview images, splicing the PDF, and more. Moving to Lambda reduced processing time by 300 times in some cases.

This series of articles will serve less as a step-by-step process to get OCR serverless infrastructure up and running and more of a highlight reel of our "Aha!" moments. In part one, we talk about creating a AWS Lambda Layer with Docker. Check out the other posts in the series:

Building dependencies for Lambda can be confusing. They need to be compiled, zipped up, and then made into a layer. We also have to keep the file size for the dependencies under Lambda's limit. Here is how we did that using a Dockerfile.

You can check out the full Dockerfile here.

Dockerfile details

One question when building the dependencies was which Docker image to use. We first tried using the amazonlinux image, but this actually resulted in some build problems with one of our dependencies. We later found the LambCI images and ended up using lambci/lambda:build-ruby2.7, because we are using Ruby. This worked perfectly for us, and it has the benefit of already having build tools installed, making for faster Docker builds.

# Use AWS Lambda ruby2.7 build environment
FROM lambci/lambda:build-ruby2.7

AWS Lambda currently has a limit of 250 MB for its dependencies. If you are building significantly larger dependencies, as we were, then you are likely to reach this limit. It is therefore very important to compile your dependencies in a way that reduces file size. By using the -Os compile time option, we were able to reduce the size of our binaries by over 90%.

# Optimize compilation for size to try and stay below Lambda's 250 MB limit
# This reduces filesize by over 90% (!) compared to the default -O2
ENV CFLAGS "-Os"
ENV CXXFLAGS $CFLAGS

While slightly less performant than the default -O2, the massively reduced file size is worth it in this situation.

Next up is building Leptonica.

WORKDIR /root

# Leptonica image-reading dependencies
RUN yum install -y libjpeg-devel libpng-devel libtiff-devel

RUN curl -O http://www.leptonica.org/source/leptonica-1.79.0.tar.gz
RUN tar zxvf leptonica-1.79.0.tar.gz

WORKDIR leptonica-1.79.0
RUN ./configure --prefix=/opt
RUN make install

You will need to install the -devel packages when compiling, but won't need the -devel variant when providing the dependencies for runtime.

Then Tesseract:

WORKDIR /root

# Optional Tesseract foreign language training dependencies
# libicu-devel on Yum is of insufficient version (50, 52 is required)
# These are also not really necessary for our usage.
#RUN yum install -y libicu-devel pango-devel cairo-devel

RUN curl -Lo tesseract-4.1.1.tar.gz \
  https://github.com/tesseract-ocr/tesseract/archive/4.1.1.tar.gz
RUN tar zxvf tesseract-4.1.1.tar.gz

WORKDIR tesseract-4.1.1
RUN ./autogen.sh --prefix=/opt
# These ENV vars have to be set or it will not build
ENV LEPTONICA_CFLAGS -I/opt/include/leptonica
ENV LEPTONICA_LIBS -L/opt/lib -lleptonica
RUN ./configure --prefix=/opt
RUN make install

# English training data
WORKDIR /opt/share/tessdata
RUN curl -LO https://github.com/tesseract-ocr/tessdata_best/raw/master/eng.traineddata

GhostScript was technically installable via RPM, but the amount of dependencies was too great. More on that later. We decided to just compile it with a minimal dependency set.

WORKDIR /root

RUN curl -LO \
  https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs952/ghostscript-9.52.tar.gz
RUN tar zxvf ghostscript-9.52.tar.gz

WORKDIR ghostscript-9.52
RUN ./configure --prefix=/opt
RUN make install

Ironically, we end up installing ghostscript-devel so that ImageMagick can be built. It might be possible to use the prior GhostScript installation here, but this was simple enough for build purposes.

WORKDIR /root

RUN yum install -y ghostscript-devel

RUN curl -Lo ImageMagick-7.0.10-6.tar.gz \
  https://github.com/ImageMagick/ImageMagick/archive/7.0.10-6.tar.gz
RUN tar zxvf ImageMagick-7.0.10-6.tar.gz

WORKDIR ImageMagick-7.0.10-6
RUN ./configure --prefix=/opt
RUN make install

We considered using libvips instead of ImageMagick, but it would have added scope to the project. Nonetheless I have some commented out code in the Dockerfile for building libvips in case we decide to switch to it in the future.

For Ruby gems, a Gemfile in the same directory that contains all of the gems is all we need.

source 'https://rubygems.org'

# PDF processing gems
gem 'simhash2'
gem 'phashion'
gem 'rtesseract'
gem 'mini_magick'
gem 'pdf-reader'
gem 'hexapdf'

# Other gems used in these files
gem 'procto'
gem 'adamantium'
gem 'concord'
gem 'activesupport', '~> 6.0.2'
gem 'chronic'
gem 'activestorage'

Note that a little bit of trickery is necessary to modify the gem paths for loading in Lambda.

WORKDIR /root

# Phashion dependencies
# Can skip this step because they are already installed above for Leptonica
#RUN yum install -y libjpeg-devel libpng-devel

# Copy Gemfile from host into container's current directory
COPY Gemfile .

RUN bundle config set path vendor/bundle
RUN bundle

# Modify directory structure for Lambda load path
WORKDIR vendor/bundle
RUN mkdir ruby/gems
RUN mv ruby/2.* ruby/gems
RUN mv ruby /opt
WORKDIR /root

Now the RPM packages. I left in the installation for GhostScript and its dependencies from RPM as a comment, mainly so you can see how many packages I had to specify manually.

Note that you will have to specify the x86_64 variants of these packages when using these tools.

WORKDIR /root

# Install yumdownloader and rpmdev-extract
RUN yum install -y yum-utils rpmdevtools

RUN mkdir rpms
WORKDIR rpms

# Download dependency RPMs
RUN yumdownloader libjpeg-turbo.x86_64 libpng.x86_64 libtiff.x86_64 \
  libgomp.x86_64 libwebp.x86_64 jbigkit-libs.x86_64
# GhostScript and dependencies
# To reduce dependencies, we are compiling GhostScript from source instead
# RUN yumdownloader ghostscript.x86_64 cups-libs.x86_64 fontconfig.x86_64 \
#   fontpackages-filesystem freetype.x86_64 ghostscript-fonts jasper-libs.x86_64 \
#   lcms2.x86_64 libICE.x86_64 libSM.x86_64 libX11.x86_64 libX11-common \
#   libXau.x86_64 libXext.x86_64 libXt.x86_64 libfontenc.x86_64 libxcb.x86_64 \
#   poppler-data stix-fonts urw-fonts xorg-x11-font-utils.x86_64 avahi-libs.x86_64 \
#   acl.x86_64 audit-libs.x86_64 cracklib.x86_64 cracklib-dicts.x86_64 cryptsetup-libs.x86_64 \
#   dbus.x86_64 dbus-libs.x86_64 device-mapper.x86_64 device-mapper-libs.x86_64 \
#   elfutils-default-yama-scope elfutils-libs.x86_64 gzip.x86_64 kmod.x86_64 kmod-libs.x86_64 \
#   libcap-ng.x86_64 libfdisk.x86_64 libpwquality.x86_64 libsemanage.x86_64 \
#   libsmartcols.x86_64 libutempter.x86_64 lz4.x86_64 pam.x86_64 qrencode-libs.x86_64 \
#   shadow-utils.x86_64 systemd.x86_64 systemd-libs.x86_64 ustr.x86_64 util-linux.x86_64 \
#   expat.x86_64 xz-libs.x86_64 libgcrypt.x86_64 libgpg-error.x86_64 elfutils-libelf.x86_64 \
#   bzip2-libs.x86_64

# Extract RPMs
RUN rpmdev-extract *.rpm
RUN rm *.rpm

# Copy all package files into /opt/rpms
RUN cp -vR */usr/* /opt

# The x86_64 packages extract as lib64, we need to move these files to lib
RUN yum install -y rsync
RUN rsync -av /opt/lib64/ /opt/lib/
RUN rm -r /opt/lib64

Notice some more path management. We used rsync for copying because cp gave us some problems.

Now we just need to zip up the dependencies.

WORKDIR /opt
RUN zip -r /root/ProcessDocumentLayer.zip *

And lastly, the entrypoint for the Dockerfile, which copies the zip file to an output directory.

ENTRYPOINT ["/bin/cp", "/root/ProcessDocumentLayer.zip", "/output"]

Now we just need the Docker commands to build this. I put them at the very top of the file under a "Usage" section.

# Usage:
# docker build -t lambda .
# docker run -v $(pwd):/output lambda
# ./publish_layer.sh.

The publish_layer.sh script is a small one we wrote that uses awscli to upload and publish the layer. You'll have to authenticate with AWS for it to work. I used aws configure for this purpose but you can check out this article for more info.

#!/bin/sh
aws s3 cp ProcessDocumentLayer.zip s3://process-document-layers
aws lambda publish-layer-version --layer-name ProcessDocumentLayer --description "Process Document dependencies" \
  --content S3Bucket=process-document-layers,S3Key=ProcessDocumentLayer.zip --compatible-runtimes ruby2.7

And that's it. With this Dockerfile, we are able to easily build and publish a dependency layer for our OCR system on Lambda. We hope this was useful for you!