<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Zac Siegel</title>
    <description>The latest articles on DEV Community by Zac Siegel (@zac_siegel).</description>
    <link>https://dev.to/zac_siegel</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F121442%2F1d1d93fb-acdb-4db5-96f0-843592684a53.jpg</url>
      <title>DEV Community: Zac Siegel</title>
      <link>https://dev.to/zac_siegel</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/zac_siegel"/>
    <language>en</language>
    <item>
      <title>Using Docker for more on Apple Silicon</title>
      <dc:creator>Zac Siegel</dc:creator>
      <pubDate>Thu, 07 Jan 2021 16:28:37 +0000</pubDate>
      <link>https://dev.to/zac_siegel/using-docker-for-more-on-apple-silicon-1gdd</link>
      <guid>https://dev.to/zac_siegel/using-docker-for-more-on-apple-silicon-1gdd</guid>
      <description>&lt;p&gt;This article originated at &lt;a href="https://www.zsiegel.com/2021/01/04/using-docker-for-more-on-apple-silicon"&gt;zsiegel.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Recently I have been using my new M1 Macbook Air as my primary development machine. This is in addition to my primary Intel based desktop machine and so when I started the ritual of installing my standard development tools I hit a bit of a snag.&lt;/p&gt;

&lt;p&gt;Like many others I store all of my &lt;a href="https://github.com/zsiegel/dotfiles"&gt;dotfiles in github&lt;/a&gt;. This helps me get up and running on a server or new computer quickly. I can then execute a simple script to install my tools via scripts - except Homebrew is not fully ready for the M1... and many of the tools I need are not ready either. &lt;/p&gt;

&lt;p&gt;I read through Sam Soffes excellent post titled &lt;a href="https://soffes.blog/homebrew-on-apple-silicon"&gt;Homebrew on Apple Silicon&lt;/a&gt; and installed &lt;code&gt;brew&lt;/code&gt; in a separate Rosetta terminal but I then took a step back to think about if this setup was really suiting my needs at this point. I was on a laptop with a small hard drive and I honestly did not want to have to install a ton of different development tools and all of their different versions to support my projects.&lt;/p&gt;

&lt;p&gt;I stopped and asked myself - &lt;strong&gt;How could I setup my system such that I didnt need to install all these tools in the first place?&lt;/strong&gt; &lt;/p&gt;

&lt;p&gt;I knew Docker was the current standard for packaging and distributing tools and I used it daily for deploying code but could I leverage it more? Could I develop a system that would let me edit on my local machine but then run, debug, and test in containers?&lt;/p&gt;

&lt;h2&gt;
  
  
  Expert Docker Fu
&lt;/h2&gt;

&lt;p&gt;In the past I had read about the crazy Docker fu from the likes of &lt;a href="https://blog.jessfraz.com/post/docker-containers-on-the-desktop/"&gt;Jessie Frazelle&lt;/a&gt; where she runs chrome, spotify and skype in Docker with GUIs - but the commands to spin those up at the time when I read them felt a bit out of reach for me. &lt;/p&gt;

&lt;p&gt;I had a feeling this might really be an ideal way to go about things and over my winter holiday break I ran across a great blog post from &lt;a href="https://twitter.com/jbergknoff"&gt;Jonathan Bergknoff&lt;/a&gt; titled &lt;a href="https://jonathan.bergknoff.com/journal/run-more-stuff-in-docker/"&gt;Run More Stuff in Docker&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;After reading Jonathans post and spending a little more time with the docker documentation I felt more confident in being able to achieve what I wanted. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Here is what I learned along the way as I attempted to set up all of my development projects to run with tools that only execute in Docker. Freeing me from installing any sofware via homebrew, letting me edit in my native app of choice, and even letting me easily opt-in to using arm64 compatible tools when available.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Mounting your current directory into a container
&lt;/h2&gt;

&lt;p&gt;One of the most interesting bits from Jonathans blog post was how to mount your existing directory into the container and to set its working directory to that very same directory inside the container. The example below will take your current directory, mount it in your container of choice (in this case alpine), and set the working directory. This feels like magic. Its like taking your existing directory and dropping it right into the container with no changes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker run &lt;span class="nt"&gt;-it&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;:&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-w&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; alpine:latest
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Side note: On the M1 this launches almost instantly which is truly amazing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Isolating tools and credentials
&lt;/h2&gt;

&lt;p&gt;I am constantly switching between AWS, GCP and other cloud tools that have CLI clients with persistent configuration and credentials on my machine. I am always paranoid about running a command against the wrong environment and hate the thought of all my credentials being in some random directory managed by the tool itself. &lt;/p&gt;

&lt;p&gt;Because of this I decided to create some shell aliases for each environments. They each have their own configuration directory that I specify for credentials. This means each container has only the tools and the specific credentials it needs.&lt;/p&gt;

&lt;p&gt;An example of this utilizes a simple alpine container that has both Terraform and the Google Cloud CLI tools installed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env dockerfile-shebang&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="s"&gt; alpine:3.8&lt;/span&gt;
&lt;span class="k"&gt;ARG&lt;/span&gt;&lt;span class="s"&gt; CLOUD_SDK_VERSION=321.0.0&lt;/span&gt;
&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="s"&gt; CLOUD_SDK_VERSION=$CLOUD_SDK_VERSION&lt;/span&gt;
&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="s"&gt; PATH /google-cloud-sdk/bin:$PATH&lt;/span&gt;

&lt;span class="k"&gt;RUN &lt;/span&gt;apk add &lt;span class="nv"&gt;terraform&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0.14.3-r0 &lt;span class="nt"&gt;--repository&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;http://dl-cdn.alpinelinux.org/alpine/edge/community

&lt;span class="k"&gt;RUN &lt;/span&gt;apk &lt;span class="nt"&gt;--no-cache&lt;/span&gt; add &lt;span class="se"&gt;\
&lt;/span&gt;        wget &lt;span class="se"&gt;\
&lt;/span&gt;        python &lt;span class="se"&gt;\
&lt;/span&gt;    &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;CLOUD_SDK_VERSION&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="nt"&gt;-linux-x86_64&lt;/span&gt;.tar.gz &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="se"&gt;\
&lt;/span&gt;    &lt;span class="nb"&gt;tar &lt;/span&gt;xzf google-cloud-sdk-&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;CLOUD_SDK_VERSION&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="nt"&gt;-linux-x86_64&lt;/span&gt;.tar.gz &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="se"&gt;\
&lt;/span&gt;    &lt;span class="nb"&gt;rm &lt;/span&gt;google-cloud-sdk-&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;CLOUD_SDK_VERSION&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="nt"&gt;-linux-x86_64&lt;/span&gt;.tar.gz &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="se"&gt;\
&lt;/span&gt;    &lt;span class="nb"&gt;ln&lt;/span&gt; &lt;span class="nt"&gt;-s&lt;/span&gt; /lib /lib64 &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="se"&gt;\
&lt;/span&gt;    gcloud config &lt;span class="nb"&gt;set &lt;/span&gt;core/disable_usage_reporting &lt;span class="nb"&gt;true&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="se"&gt;\
&lt;/span&gt;    gcloud &lt;span class="nt"&gt;--version&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I can then launch this container with a unique local config directory to isolate the credentials and mount them for use by the CLI tool. In this case im taking credentails from a local directory at &lt;strong&gt;/opt/local/gcloud/{PROJECT_ID}/{ENVIRONMENT}&lt;/strong&gt; and mounting them at &lt;strong&gt;root/.config&lt;/strong&gt; which is where the gcloud CLI tools expect the config to be. &lt;/p&gt;

&lt;p&gt;Note that when I first start this container that might be empty! In this case because its a volume mapped to my local machine as soon as I run &lt;code&gt;gcloud auth login&lt;/code&gt; those credentials will be saved from the container right into my local directory.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker run &lt;span class="nt"&gt;-it&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="s2"&gt;"/opt/local/gcloud/{PROJECT_ID}/{ENVIRONMENT}"&lt;/span&gt;:&lt;span class="s2"&gt;"root/.config"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
               &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;:&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
               &lt;span class="nt"&gt;-w&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
               zsiegel/devops:latest
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Using Dockerfile-Shebang
&lt;/h2&gt;

&lt;p&gt;If you read the Dockerfile above carefully above you will notice a &lt;code&gt;shebang&lt;/code&gt; at the top of the Dockerfile. This is a bit odd at first but it allows you to make your dockerfiles executable so you can skip the build step which would be required if you need a custom image like in the example above. &lt;/p&gt;

&lt;p&gt;This also makes it incredibly easy to setup a Dockerfile for each project and have it run fully isolated. No more pulling down a project and installing its dependecies on your local machine. It can all be isolated in a container! &lt;/p&gt;

&lt;p&gt;&lt;a href="https://twitter.com/JakeWharton"&gt;Jake Wharton&lt;/a&gt; has created a &lt;a href="https://github.com/JakeWharton/dockerfile-shebang"&gt;script in homebrew&lt;/a&gt; to support this if thats your thing or you can just drop it right into your &lt;strong&gt;PATH&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;With &lt;strong&gt;dockerfile-shebang&lt;/strong&gt; installed I can now isolate any tools I need inside a dockerfile that is purpose  built for that project. &lt;/p&gt;

&lt;p&gt;For example I have a number of node projects for static websites. In each of my projects I have a filed called &lt;strong&gt;{PROJECT_NAME}.dockerfile&lt;/strong&gt; at the root. The contents of the Dockerfile might look like the one below.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env dockerfile-shebang&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="s"&gt; amd64/node:lts-slim&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With this file in the root of my project I can then run the following to have a fully isolated project environment that can be edited on my local machine in my editor of choice.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;
&lt;span class="c"&gt;# Make the dockerfile executable first - do this only once&lt;/span&gt;
&lt;span class="nb"&gt;chmod&lt;/span&gt; +x zsiegel.com.dockerfile

&lt;span class="c"&gt;# Run the environment&lt;/span&gt;
./zsiegel.com.dockerfile &lt;span class="nt"&gt;-it&lt;/span&gt; &lt;span class="nt"&gt;-p&lt;/span&gt; 8000:8000 &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;:&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-w&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--&lt;/span&gt; /bin/sh

&lt;span class="c"&gt;# In the container&lt;/span&gt;
yarn &lt;span class="nb"&gt;install
&lt;/span&gt;yarn dev
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The command above is straight forward and builds upon what I learned in the earlier sections. I run my Dockerfile which is an executable thanks to the shebang package &lt;strong&gt;(note the dockerfile will be built automatically if needed)&lt;/strong&gt;, expose some ports so I can access it locally, then mount my current directory, set my work directory, and finally ask for a shell prompt.&lt;/p&gt;

&lt;h2&gt;
  
  
  Using AMD64 and ARM64
&lt;/h2&gt;

&lt;p&gt;Before I mentioned that some tools are running on Apple Silicon and others are not. Another benefit I have found is the ability to opt-in to arm64 compiled tools where available. If you look below you will see examples of three Dockerfiles that specify different architectures.&lt;/p&gt;

&lt;h3&gt;
  
  
  Default Architecture - arm64 on Apple Silicon
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env dockerfile-shebang&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="s"&gt; alpine:latest&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt; &lt;span class="nb"&gt;uname&lt;/span&gt; &lt;span class="nt"&gt;-sm&lt;/span&gt;
 Linux aarch64
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This dockerfile without the architecture designation will detect your current machines architecture and try to use an appropriate container - in my case on Apple Silicon this will be an &lt;code&gt;arm64&lt;/code&gt; architecture.&lt;/p&gt;

&lt;h3&gt;
  
  
  x64 Architecture
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env dockerfile-shebang&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="s"&gt; amd64/node:lts-slim&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;uname&lt;/span&gt; &lt;span class="nt"&gt;-sm&lt;/span&gt;
Linux x86_64
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This dockerfile has the architecture designation so it will have the x64 architecture specified and run under emulation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tool Versioning
&lt;/h2&gt;

&lt;p&gt;If you are a developer I am sure you know the pain of having to juggle multiple different versions of java, node, python, ruby and other tools. With the above tricks in hand I can now properly ensure all of my projects are isolated against the versions of software that they need.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;If project X needs an older version of Go? I will setup a Dockerfile with the right version. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;My new project Y can compile against the latest version of Java? Perfect its just another Dockerfile away. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;I want to try out that new library but it requires some strange old version of something? No problem, just run it in a container!&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  2021 Docker and Me
&lt;/h2&gt;

&lt;p&gt;The newfound knowledge above has allowed me to really streamline my workflow and minimize the tools I have to install and manage on this new M1 Macbook Air. I hope to continue to learn more about Docker and leverage it for running my development tools on my local machine.&lt;/p&gt;

</description>
      <category>docker</category>
      <category>homebrew</category>
      <category>containers</category>
      <category>macos</category>
    </item>
    <item>
      <title>A Developers Journey into Machine Learning: Installing Python, Jupyter, TensorFlow, and Keras</title>
      <dc:creator>Zac Siegel</dc:creator>
      <pubDate>Fri, 31 May 2019 22:03:18 +0000</pubDate>
      <link>https://dev.to/zac_siegel/a-developers-journey-into-machine-learning-installing-python-jupyter-tensorflow-and-keras-4epp</link>
      <guid>https://dev.to/zac_siegel/a-developers-journey-into-machine-learning-installing-python-jupyter-tensorflow-and-keras-4epp</guid>
      <description>&lt;p&gt;This article originated on &lt;a href="https://www.zsiegel.com/2019/05/30/developers-journey-into-machine-learning-part1"&gt;zsiegel.com&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Machine Learning as a developer
&lt;/h2&gt;

&lt;p&gt;Machine learning is without a doubt one of the most interesting topics in computer science today. The breakneck pace and WOW factor associated with many machine learning demos has given it some mystique that still has me scratching my head wondering how it is even possible. &lt;/p&gt;

&lt;p&gt;When I finally decided to jump in to machine learning I started with a list of questions that will undoubtedly take a very long time to eventuall answer but it at least gives me an initial direction.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Is a strong math and statistics background necessary? &lt;/li&gt;
&lt;li&gt;Is there a set of basic terminology and concepts that will serve as a good foundation? &lt;/li&gt;
&lt;li&gt;Is a high end GPU really necessary to do machine learning?&lt;/li&gt;
&lt;li&gt;Are their types of machine learning that do not require a GPU?&lt;/li&gt;
&lt;li&gt;Given the incredible pace of change are any of the books, tutorials and examples online from 1-2 years ago still relevant?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In order to explore these questions I set out to get up and running quickly so I could spend my time learning and experimenting. The following is what helped me get going rather quickly with minimal fuss.&lt;/p&gt;

&lt;h2&gt;
  
  
  Linux, Python, CUDA, TensorFlow, and Keras
&lt;/h2&gt;

&lt;p&gt;I mentioned before that the breakneck pace of change in Machine Learning is staggering. I spoke to a number of colleagues that had dipped their toes into the space and one of the core challenges they faced was around the tooling. &lt;/p&gt;

&lt;p&gt;&lt;code&gt;Many of them had challenges keeping their toolchains and dependencies in simple working order&lt;/code&gt;. They were often experiencing minor version updates of various frameworks causing compatibility issues. &lt;/p&gt;

&lt;p&gt;During my initial setup I experienced similar troubles so I slowly and systematically inched back to what appears to be a stable setup and has been reproduced across multiple machines.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;PopOS 19.04&lt;/li&gt;
&lt;li&gt;Python 3.7&lt;/li&gt;
&lt;li&gt;TensorFlow 1.13.1&lt;/li&gt;
&lt;li&gt;Keras 2.2.4&lt;/li&gt;
&lt;li&gt;CUDA 10.0&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Being a developer familiar with &lt;a href="https://www.docker.com/"&gt;Docker&lt;/a&gt; I had to resist the urge to add yet another dependency to the tool chain so I instead looked through various Dockerfiles and found a combination that has worked across a number of projects and been rock solid on &lt;a href="https://system76.com/pop"&gt;PopOS&lt;/a&gt; which is a linux distribution based on Ubuntu.&lt;/p&gt;

&lt;p&gt;It all starts with some basic tools that can be isntalled via &lt;code&gt;apt&lt;/code&gt;.&lt;/p&gt;



&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;apt &lt;span class="nb"&gt;install &lt;/span&gt;python3 python3-venv system76-cuda-10.0 system76-cudnn-10.0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;It is important to note I chose &lt;a href="https://system76.com/pop"&gt;PopOS&lt;/a&gt; for its developer repositories that made installing CUDA very simple. &lt;/p&gt;

&lt;p&gt;If you are running a Debian based operating system and have NVIDIA display drivers installed you can get the System76 developer packages by doing the following.&lt;/p&gt;



&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo echo&lt;/span&gt; &lt;span class="s2"&gt;"deb http://apt.pop-os.org/proprietary bionic main"&lt;/span&gt; | &lt;span class="nb"&gt;sudo tee&lt;/span&gt; &lt;span class="nt"&gt;-a&lt;/span&gt; /etc/apt/sources.list.d/pop-proprietary.list
&lt;span class="nb"&gt;sudo &lt;/span&gt;apt-key adv &lt;span class="nt"&gt;--keyserver&lt;/span&gt; keyserver.ubuntu.com &lt;span class="nt"&gt;--recv-key&lt;/span&gt; 204DD8AEC33A7AFF
&lt;span class="nb"&gt;sudo &lt;/span&gt;apt update
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h2&gt;
  
  
  Setting up Python and Jupyter Notebooks
&lt;/h2&gt;

&lt;p&gt;With the basic frameworks installed it was time to setup my development environment. As someone who develops primarily in strongly typed languages with a compiler I was not sure how I would fare living primarily in a python repl but I have found Jupyter to be incredibly flexible and enjoyable. &lt;/p&gt;

&lt;p&gt;Jupyter notebooks are a wonderful way to explore and run examples both locally on your machine but also in the cloud. During my initial experiments I spent a ton of time browsing &lt;a href="https://jupyter.org/"&gt;Jupyter&lt;/a&gt; notebooks on Github and playing around in &lt;a href="https://colab.research.google.com/"&gt;Google Colab&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;&lt;code&gt;You can think of Jupyter notebooks as a combination of prose, documentation, runnable code, and console output.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;At the moment I have decided to store all of my work in a single directory on my computer but I could see making dedicated environments as needed in the future. &lt;/p&gt;

&lt;p&gt;I chose a single directory for everything because I can leave my jupyter server running and remotely access my &lt;a href="https://jupyter.org/"&gt;Jupyter&lt;/a&gt; server from my iPad Pro using an app called &lt;a href="https://juno.sh/"&gt;Juno&lt;/a&gt;. This is something I will write about later in the future!&lt;/p&gt;

&lt;p&gt;Now on to setting up Python and Jupyter.&lt;/p&gt;



&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Create the directory where we will be storing everything&lt;/span&gt;
&lt;span class="nb"&gt;mkdir&lt;/span&gt; ~/src/jupyter
&lt;span class="nb"&gt;cd&lt;/span&gt; ~/src/jupyter

&lt;span class="c"&gt;# Create a fully isolated python environment - this is similar to gradlew or other wrapper scripts you may have encountered in other languages&lt;/span&gt;
python3 &lt;span class="nt"&gt;-m&lt;/span&gt; venv jupyter-env

&lt;span class="c"&gt;# This activates the virtual environment - when activated you will see your prompt change to indicate you are in `jupyter-env`. Any commands run after this will be scoped strictly to this environment&lt;/span&gt;
&lt;span class="nb"&gt;source &lt;/span&gt;jupyter-env/bin/activate

&lt;span class="c"&gt;# Install the dependencies. Again its important to understand these dependencies are not globally available on your system. They are only available to the virtual environment we activated just before this&lt;/span&gt;
pip &lt;span class="nb"&gt;install &lt;/span&gt;pandas numpy tensorflow-gpu keras jupyter matplotlib pillow scikit-learn

&lt;span class="c"&gt;# Store all of the dependencies into a text file. Make sure you run this command if you add any new dependencies using pip. You can use this file to re-install the dependencies on a new machine using `pip install -r requirements.txt`&lt;/span&gt;
pip freeze &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; requirements.txt

&lt;span class="c"&gt;# Launch the notebook - this will start a jupyter server and open your web browser&lt;/span&gt;
jupyter notebook
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;With this setup I have been able to work on a number of learning exercises and the toolchain has not gotten in my way... at least not yet :).&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>python</category>
      <category>keras</category>
      <category>tensorflow</category>
    </item>
    <item>
      <title>Using Docker Machine to provision a remote docker host</title>
      <dc:creator>Zac Siegel</dc:creator>
      <pubDate>Sat, 06 Apr 2019 22:43:18 +0000</pubDate>
      <link>https://dev.to/zac_siegel/using-docker-machine-to-provision-a-remote-docker-host-1267</link>
      <guid>https://dev.to/zac_siegel/using-docker-machine-to-provision-a-remote-docker-host-1267</guid>
      <description>&lt;p&gt;This article originated on &lt;a href="https://zsiegel.com/2019/04/06/using-docker-machine-to-provision-a-remote-docker-host"&gt;zsiegel.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Like many developers my primary machine is a laptop and when developing applications I typically use Docker containers to provide local instances of  databases, caches, message queues, and other supporting services. &lt;/p&gt;

&lt;p&gt;When running these services via Docker this can often wreak havoc on my machine and cause it to slow down to a crawl even with lots of ram and CPU power. I began exploring options on how to get the ease and flexibility of Docker but use remote computing resources instead. I found that Docker has a great tool for enabling this type of workflow where the containers run on a different host but you can still access them just like they were on your machine. &lt;/p&gt;

&lt;h2&gt;
  
  
  Docker Machine
&lt;/h2&gt;

&lt;p&gt;Docker has a built in tool called &lt;code&gt;docker-machine&lt;/code&gt; that allows you to provision many different types of remote machines as docker hosts. In order for this to run all you need is a remote system that you have SSH access to. Docker can then provision that remote system and your local &lt;code&gt;docker&lt;/code&gt; tools can interface with it instead of your local machine.&lt;/p&gt;

&lt;p&gt;This allows you to run all sorts of heavier services on the remote machine instead of your local machine. Let’s take a look at how you might provision a simple docker host on a home server.&lt;/p&gt;

&lt;h2&gt;
  
  
  Docker Machine drivers
&lt;/h2&gt;

&lt;p&gt;Take a look through the &lt;a href="https://docs.docker.com/machine/drivers/"&gt;documentation&lt;/a&gt; and find what driver is appropriate for you. They provide many cloud drivers for running docker instances remotely. In this case we are going to provision a &lt;a href="https://docs.docker.com/machine/drivers/generic/"&gt;generic&lt;/a&gt; host on a ubuntu server running I have running in my home office.&lt;/p&gt;

&lt;p&gt;I went ahead and installed Ubuntu 18.04 LTS and I have a basic root account setup. The first thing we need to do is enable SSH key authentication.&lt;/p&gt;

&lt;h2&gt;
  
  
  SSH Key Authentication
&lt;/h2&gt;

&lt;p&gt;We are going to start by generating a set of keys to use for authentication with  &lt;a href="https://www.ssh.com/ssh/keygen/"&gt;ssh-keygen&lt;/a&gt;. I named my keys &lt;code&gt;zdocker&lt;/code&gt; since that is the hostname of the server I will be connecting to in my office.&lt;/p&gt;

&lt;p&gt;Once these keys are generated you can utilize a tool called &lt;a href="https://www.ssh.com/ssh/copy-id"&gt;ssh-copy-id&lt;/a&gt; to automatically transfer the public key information to your server. In my case the command looked something like the following.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh-copy-id &lt;span class="nt"&gt;-i&lt;/span&gt; ~/.ssh/&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;PUBLIC_KEY&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;USERNAME&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;@&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;IP_ADDRESS&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; 
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;You will want to replace the placeholders above with the &lt;code&gt;public_key, username, and ip_address&lt;/code&gt; of your server in the command above. Once transferred you can now log into your server via public key authentication. I then went ahead and disabled password login via SSH since the only way we will allowing logins going forward is with the keys.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;nano /etc/ssh/sshd_config
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;





&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Turn off PasswordAuthentication by setting the value to no 
# and making sure it is not commented out like below
ChallengeResponseAuthentication no
PasswordAuthentication no
UsePAM no
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Now that we have SSH key authentication we need to enable password-less sudo privileges for the user that we intent to login with. This is required by the docker-machine installer. &lt;/p&gt;

&lt;h3&gt;
  
  
  Edit the sudoers file
&lt;/h3&gt;



&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;visudo
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;





&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Insert a new entry at the bottom that enables passwordless sudo
${USERNAME} ALL=(ALL) NOPASSWD: ALL
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;You will want to replace the &lt;code&gt;username&lt;/code&gt; with the user that you have created on the machine.&lt;/p&gt;

&lt;h2&gt;
  
  
  Provision a Docker host
&lt;/h2&gt;

&lt;p&gt;Now that we have SSH public key authentication setup with root access its time to let &lt;code&gt;docker-machine&lt;/code&gt; provision our server.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;docker-machine create &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--driver&lt;/span&gt; generic &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--generic-ip-address&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$IP_ADDRESS&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--generic-ssh-user&lt;/span&gt; &lt;span class="nv"&gt;$USERNAME&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--generic-ssh-key&lt;/span&gt; ~/.ssh/&lt;span class="nv"&gt;$PRIVATE_KEY&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    zdocker

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;You will want to replace the placeholders above with the &lt;code&gt;ip_address, username, private_key, and machine_name&lt;/code&gt; which in my example is &lt;code&gt;zdocker&lt;/code&gt;. &lt;/p&gt;

&lt;p&gt;Note that the &lt;code&gt;machine_name&lt;/code&gt; will be what shows up in your CLI tool when you run &lt;code&gt;docker-machine ls&lt;/code&gt; . It is how you will reference this remote docker instance when choosing which docker host to use for running your services.&lt;/p&gt;

&lt;p&gt;If the provisioning was a success you will be greeted with some output similar to what is below.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;DOCKER_TLS_VERIFY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"1"&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;DOCKER_HOST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"tcp://192.168.1.10:2376"&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;DOCKER_CERT_PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"/path/to/a/cert"&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;DOCKER_MACHINE_NAME&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"zdocker"&lt;/span&gt;
&lt;span class="c"&gt;# Run this command to configure your shell: &lt;/span&gt;
&lt;span class="c"&gt;# eval $(docker-machine env zdocker)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Next run the eval command in your current shell to point your &lt;code&gt;docker&lt;/code&gt; CLI at the remote host.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;eval&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;docker-machine &lt;span class="nb"&gt;env &lt;/span&gt;zdocker&lt;span class="si"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;You now have  &lt;code&gt;docker&lt;/code&gt;  pointed at a remote docker host.  Within this shell all docker commands will utilize the remote system to do its work and your laptops hardware can breathe a little bit easier!&lt;/p&gt;

</description>
      <category>docker</category>
      <category>containers</category>
    </item>
    <item>
      <title>Java and Docker - JVM Runtime Basics</title>
      <dc:creator>Zac Siegel</dc:creator>
      <pubDate>Thu, 13 Dec 2018 01:57:24 +0000</pubDate>
      <link>https://dev.to/zac_siegel/java-and-docker---jvm-runtime-basics-2l20</link>
      <guid>https://dev.to/zac_siegel/java-and-docker---jvm-runtime-basics-2l20</guid>
      <description>&lt;p&gt;This article originated on &lt;a href="https://zsiegel.com/2018/11/25/java-and-docker-runtime-basics"&gt;zsiegel.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;As container deployments become increasingly common, it is important to understand how your application and runtime cooperate with various Linux container technologies.&lt;/p&gt;

&lt;p&gt;Depending on your JVM target version, you may need to do a little extra work to ensure your Java runtime is aware of the processor and memory usage limits. If you are not careful, your application may have incorrect optimization or heuristics applied, potentially leading to performance issues and even crashes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Namespaces and Cgroups
&lt;/h2&gt;

&lt;p&gt;First, if you need to brush up on the underlying Linux technologies that make Docker possible, you might want to read &lt;a href="https://mobile.twitter.com/b0rk"&gt;Julia Evans&lt;/a&gt; excellent blog post &lt;a href="https://jvns.ca/blog/2016/10/10/what-even-is-a-container/"&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;With a basic understanding of &lt;a href="https://en.wikipedia.org/wiki/Cgroups"&gt;cgroups&lt;/a&gt; and &lt;a href="https://en.wikipedia.org/wiki/Linux_namespaces"&gt;namespaces&lt;/a&gt; we can now turn our attention to ensuring our Java runtime is aware of these technologies. Here is what you need to know in order to ensure your Java runtime is aware of the memory and processor limits set on your containers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Java 7
&lt;/h2&gt;

&lt;p&gt;Sadly there are no flags or features available in JDK 7 to provide container awareness. If you are running on JDK 7, you can tune your applications thread counts to reduce potential processor resource issues.&lt;/p&gt;

&lt;h2&gt;
  
  
  Java 8
&lt;/h2&gt;

&lt;p&gt;Java 8 has a few flags that can help the runtime operate in a more container aware manner. These updates were added in the Java 8 Update 131 release which has been available since April 18, 2017.&lt;/p&gt;

&lt;p&gt;The update contains improved processor and memory container awareness, and allows for use of experimental VM flags to enable automatic heap adjustment. To unlock this capability simply add the following flags to your &lt;code&gt;java&lt;/code&gt; command line options.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;java &lt;span class="nt"&gt;-XX&lt;/span&gt;:+UnlockExperimentalVMOptions &lt;span class="nt"&gt;-XX&lt;/span&gt;:+UseCGroupMemoryLimitForHeap &lt;span class="nt"&gt;-jar&lt;/span&gt; app.jar
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;These experimental flags set the heap size based on the value specified in the container’s &lt;code&gt;sys/fs/cgroup/memory/memory.limit_in_bytes&lt;/code&gt; file. You can check out the bug report and work related to this &lt;a href="https://bugs.openjdk.java.net/browse/JDK-8170888"&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Java 9 &amp;amp; 10
&lt;/h2&gt;

&lt;p&gt;Oracle has signaled that JDK 9 and JDK 10 will be automatically container aware respecting both processor and memory limits. They are also providing manual overrides to allow for disabling or setting these limits manually. There are a number of tickets and work on these features can be tracked on the &lt;a href="https://bugs.openjdk.java.net/projects/JDK/issues/JDK-8192936?filter=allopenissues%20%22OpenJDK%20Bug%20Tracker"&gt;OpenJDK Bug tracker&lt;/a&gt; which you can check out below.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://bugs.openjdk.java.net/browse/JDK-8196595"&gt;JDK 10 Release Note: Improvements for Docker Containers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bugs.openjdk.java.net/browse/JDK-8170888"&gt;Docker memory limits&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bugs.openjdk.java.net/browse/JDK-8140793"&gt;Docker processor limits&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;There is still work to be done in order for the JVM to be fully aware of containerization, but so far excellent progress has been made and, more importantly, backported into JDK 8 in order to garner more feedback from the community. &lt;/p&gt;

&lt;p&gt;If you find yourself running your Java application in containers and need help check back here frequently for more updates as I start a new series of articles on running java apps with &lt;a href="https://www.docker.com/"&gt;Docker&lt;/a&gt; and &lt;a href="https://kubernetes.io/"&gt;Kubernetes&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>java</category>
      <category>kotlin</category>
      <category>docker</category>
      <category>containers</category>
    </item>
    <item>
      <title>Java and Docker - Memory and CPU Limits</title>
      <dc:creator>Zac Siegel</dc:creator>
      <pubDate>Thu, 13 Dec 2018 01:54:17 +0000</pubDate>
      <link>https://dev.to/zac_siegel/java-and-docker---memory-and-cpu-limits-3h77</link>
      <guid>https://dev.to/zac_siegel/java-and-docker---memory-and-cpu-limits-3h77</guid>
      <description>&lt;p&gt;This article originated on &lt;a href="https://zsiegel.com/2018/11/26/java-and-docker-memory-and-cpu-limits"&gt;zsiegel.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For many years java developers have gotten used to tweaking their code and the JVM flags to gain both performance and stability. &lt;/p&gt;

&lt;p&gt;Now that we are deploying more often to containers its important to also understand how the underlying container technologies effect our application. &lt;/p&gt;

&lt;p&gt;Let’s take a look at a real world example to understand how our application reacts when we adjust container flags that control memory and CPU allocation. &lt;/p&gt;

&lt;p&gt;This example is applicable to Docker, Kubernetes, Mesos and other orchestrated container environments.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;All of the following commands can be run on Docker CE for Linux and Mac v18 and up. We use JDK8 in these examples to demonstrate the lowest container aware JVM.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  A Simple Executable
&lt;/h2&gt;

&lt;p&gt;Let’s start off with a simple Java executable that logs to the console the processors detected and the max memory available to the runtime.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Docker&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

  &lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="o"&gt;[]&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;

    &lt;span class="nc"&gt;Runtime&lt;/span&gt; &lt;span class="n"&gt;runtime&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Runtime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getRuntime&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;

    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;processors&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;runtime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;availableProcessors&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
    &lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="n"&gt;maxMemory&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;runtime&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;maxMemory&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;

    &lt;span class="nc"&gt;System&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;format&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Number of processors: %d\n"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;processors&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="nc"&gt;System&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;format&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Max memory: %d bytes\n"&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;maxMemory&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
  &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;We then create a simple Dockerfile that contains the JAR with this main function.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;FROM openjdk:8-jre
COPY /build/libs/java-and-docker-1.0.jar java-and-docker.jar

CMD ["java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-jar", "java-and-docker.jar"]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;In this example, we are going to run JDK 8, which is the lowest container-aware JDK. For more information about the differences in container awareness capabilities between JDK versions, you can check out the previous article &lt;a href="https://zsiegel.com/2018/11/25/java-and-docker-runtime-basics"&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Now we need to build the container.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;docker build &lt;span class="nt"&gt;-t&lt;/span&gt; zsiegel:java-and-docker &lt;span class="nb"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;With the container built, let’s run this on the current machine and see what we get.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;docker run zsiegel:java-and-docker
Number of processors: 4
Max memory: 466092032 bytes
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;You can compare this result to the CPU and memory settings in the Docker app preferences on the Mac or look at how it compares to your machine specs on Linux. The number of cores should match and the ram displayed should be slightly lower.&lt;/p&gt;

&lt;h2&gt;
  
  
  Memory Limits
&lt;/h2&gt;

&lt;p&gt;Let’s take a look at limiting the memory the container can use and see how the runtime adjusts.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;docker run &lt;span class="nt"&gt;-it&lt;/span&gt; &lt;span class="nt"&gt;--memory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;512m zsiegel:java-and-docker
Number of processors: 4
Max memory: 119537664 bytes
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Now that we have adjusted the amount of memory available to the container itself the heap size is adjusted by the runtime. How did the runtime decide on this value of &lt;code&gt;119537664 bytes&lt;/code&gt; which equates to roughly &lt;code&gt;120 megabytes&lt;/code&gt;? If you dig into the &lt;a href="https://docs.oracle.com/javase/9/gctuning/parallel-collector1.htm#JSGCT-GUID-74BE3BC9-C7ED-4AF8-A202-793255C864C4"&gt;JVM Tuning guide&lt;/a&gt; you will see the following.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Unless the initial and maximum heap sizes are specified on the command line, they're calculated based on the amount of memory on the machine. The default maximum heap size is one-fourth of the physical memory while the initial heap size is 1/64th of physical memory. The maximum amount of space allocated to the young generation is one third of the total heap size."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The runtime by default will use &lt;code&gt;1/4th&lt;/code&gt; of the available memory. In our case this means &lt;code&gt;512/4 = 128 megabytes&lt;/code&gt; which is roughly the number we see.&lt;/p&gt;

&lt;h2&gt;
  
  
  Processor Limits
&lt;/h2&gt;

&lt;p&gt;Let’s take a look at limiting the cpu that is available to the container and see what happens. As of JDK 8 Update 131, the runtime should be aware of the number of cpus available and will tune thread counts accordingly. In the JVM's case one cpu is equal to one core.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;docker run &lt;span class="nt"&gt;-it&lt;/span&gt; &lt;span class="nt"&gt;--cpus&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 zsiegel:java-and-docker
Number of processors: 4
Max memory: 468189184 bytes
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;This is &lt;code&gt;not&lt;/code&gt; what I expected the first time I ran it so let’s dig in further to understand what is really going on.&lt;/p&gt;

&lt;p&gt;The Docker &lt;code&gt;--cpus&lt;/code&gt; flag specifies the percentage of available CPU resources a container can use. Specifically it refers to the &lt;code&gt;cpu-shares&lt;/code&gt;. It does &lt;code&gt;not&lt;/code&gt; adjust the number of physical processors the container has access to, which is what the jdk8 runtime currently uses when setting the number of processors.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;There is future work to correct this shortcoming in JDK 10. You can follow the progress and discussion &lt;a href="https://bugs.openjdk.java.net/browse/JDK-8146115"&gt;here&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Let’s try using a different flag called &lt;code&gt;--cpuset-cpus&lt;/code&gt;. This flag is used to limit the cores a container can use. On a 4 core machine we can specify 0-3. We can specify a single core or even multiple cores by comma separating the index of the cores.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;docker run &lt;span class="nt"&gt;-it&lt;/span&gt; &lt;span class="nt"&gt;--cpuset-cpus&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0 zsiegel:java-and-docker
Number of processors: 1
Max memory: 508887040 bytes 
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;





&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;docker run &lt;span class="nt"&gt;-it&lt;/span&gt; &lt;span class="nt"&gt;--cpuset-cpus&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0,1 zsiegel:java-and-docker
Number of processors: 2
Max memory: 508887040 bytes 
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;This result is more in line with what we expected. We now have the runtime properly seeing 2 cores available instead of 4. When we do this the runtime will then tune the number of compiler and garbage collection threads accordingly. &lt;/p&gt;

&lt;p&gt;It is important to note that many libraries and frameworks that rely on thread pools will tune the number of threads based on these numbers. &lt;code&gt;You would think that this would address our problems but there is a catch!&lt;/code&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Container Orchestration
&lt;/h2&gt;

&lt;p&gt;There is a major problem with the above example. The vast majority of container orchestration tools like Mesos and Kubernetes set the &lt;code&gt;cpu-shares&lt;/code&gt; and not the &lt;code&gt;cpuset-cpus&lt;/code&gt;. This means that until the work in JDK10 &lt;a href="https://bugs.openjdk.java.net/browse/JDK-8146115"&gt;mentioned earlier&lt;/a&gt; is completed the runtime and frameworks that rely on &lt;code&gt;runtime.availableProcessors()&lt;/code&gt; will be unable to tune their thread count properly. My hope is that this work will be backported to at least JDK9 if possible.  &lt;/p&gt;

&lt;p&gt;If you want more info on the flags available in Docker for adjusting resource limits you can check out the documentation &lt;a href="https://docs.docker.com/config/containers/resource_constraints/"&gt;here&lt;/a&gt;&lt;/p&gt;

</description>
      <category>java</category>
      <category>kotlin</category>
      <category>docker</category>
      <category>containers</category>
    </item>
    <item>
      <title>How to deploy Java apps in Docker </title>
      <dc:creator>Zac Siegel</dc:creator>
      <pubDate>Thu, 13 Dec 2018 01:38:02 +0000</pubDate>
      <link>https://dev.to/zac_siegel/how-to-deploy-java-apps-in-docker--15h</link>
      <guid>https://dev.to/zac_siegel/how-to-deploy-java-apps-in-docker--15h</guid>
      <description>&lt;p&gt;This article originated on &lt;a href="https://zsiegel.com/2018/12/12/how-to-deploy-java-apps-in-docker"&gt;zsiegel.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Many java developers are familiar with deploying java applications via jar files and application containers such as &lt;a href="http://tomcat.apache.org/"&gt;Tomcat&lt;/a&gt; however many organizations are now deploying applications with &lt;a href="https://www.docker.com/"&gt;Docker&lt;/a&gt; containers. Lets take a look at the bare minimum steps to get a java application running in a Docker container. &lt;/p&gt;

&lt;h2&gt;
  
  
  Install Docker
&lt;/h2&gt;

&lt;p&gt;In order to run your java app in a container your are going to need &lt;a href="https://www.docker.com/"&gt;Docker&lt;/a&gt;. Once installed &lt;a href="https://www.docker.com/"&gt;Docker&lt;/a&gt; will allow you to build and run your containers on your local machine. It runs on many platforms including Linux, MacOS, and Windows. Head over to the Docker website and look for information on the &lt;a href="https://docs.docker.com/install/#supported-platforms"&gt;supported platforms&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Once installed you can test out your installation by building and running the &lt;code&gt;hello-world&lt;/code&gt; container. The output below indicates your installation was a success.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;▲ :zsiegel &lt;span class="o"&gt;(&lt;/span&gt;master&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;docker run hello-world
Unable to find image &lt;span class="s1"&gt;'hello-world:latest'&lt;/span&gt; locally
latest: Pulling from library/hello-world
d1725b59e92d: Pull &lt;span class="nb"&gt;complete 
&lt;/span&gt;Digest: sha256:0add3ace90ecb4adbf7777e9aacf18357296e799f81cabc9fde470971e499788
Status: Downloaded newer image &lt;span class="k"&gt;for &lt;/span&gt;hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the &lt;span class="s2"&gt;"hello-world"&lt;/span&gt; image from the Docker Hub.
    &lt;span class="o"&gt;(&lt;/span&gt;amd64&lt;span class="o"&gt;)&lt;/span&gt;
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 &lt;span class="nv"&gt;$ &lt;/span&gt;docker run &lt;span class="nt"&gt;-it&lt;/span&gt; ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h2&gt;
  
  
  Create a Dockerfile for your Java app
&lt;/h2&gt;

&lt;p&gt;Now that you have Docker installed lets take a simple Java app built by Gradle and create a Dockerfile. We are going to both build and run the jar file within the container itself which gives us a more consistent environment. You can checkout my sample project on &lt;a href="https://github.com/zsiegel/java-and-docker"&gt;Github&lt;/a&gt; and work from this project if you would like.&lt;/p&gt;

&lt;p&gt;Now that we have our basic Java app lets take a look at what a Dockerfile might look like.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="c"&gt;# NOTE: This is not a production ready Dockerfile. &lt;/span&gt;
&lt;span class="c"&gt;# Utilize this only for development purposes&lt;/span&gt;

&lt;span class="c"&gt;# Use a container image that has both Gradle and the JDK&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="s"&gt; gradle:5.0.0-jdk8-alpine&lt;/span&gt;

&lt;span class="c"&gt;# Switch to the `gradle` user defined by our container image&lt;/span&gt;
&lt;span class="k"&gt;USER&lt;/span&gt;&lt;span class="s"&gt; gradle&lt;/span&gt;

&lt;span class="c"&gt;# Copy over the project directory into the container&lt;/span&gt;
&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="s"&gt; --chown=gradle:gradle . /java-and-docker&lt;/span&gt;

&lt;span class="c"&gt;# Set our working directory to our project directory that we set above&lt;/span&gt;
&lt;span class="k"&gt;WORKDIR&lt;/span&gt;&lt;span class="s"&gt; /java-and-docker&lt;/span&gt;

&lt;span class="c"&gt;# Run the build&lt;/span&gt;
&lt;span class="k"&gt;RUN &lt;/span&gt;gradle build

&lt;span class="c"&gt;# Run the jar file&lt;/span&gt;
&lt;span class="c"&gt;# Since we are using JDK8 we set some additional flags to be more container aware&lt;/span&gt;
&lt;span class="k"&gt;CMD&lt;/span&gt;&lt;span class="s"&gt; ["java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-jar", "build/libs/java-and-docker-1.0.jar"]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h2&gt;
  
  
  Build and Run your Java app in the Docker container
&lt;/h2&gt;

&lt;p&gt;Now that we have our Dockerfile created we can ask docker to first build and then run our container with the following commands.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight shell"&gt;&lt;code&gt;docker build &lt;span class="nt"&gt;-t&lt;/span&gt; zsiegel:java-and-docker &lt;span class="nb"&gt;.&lt;/span&gt;
docker run zsiegel:java-and-docker 
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h2&gt;
  
  
  Success
&lt;/h2&gt;

&lt;p&gt;Now that you have successfully built and run a java app within a Docker container you can start exploring the related articles. These will help you understand how the java runtime and your apps will behave depending on the JVM version and the container flags you set. This is important because running java in a container is not as straightforward as it may seem.&lt;/p&gt;

</description>
      <category>java</category>
      <category>kotlin</category>
      <category>docker</category>
      <category>containers</category>
    </item>
  </channel>
</rss>
