<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: John</title>
    <description>The latest articles on DEV Community by John (@jborak).</description>
    <link>https://dev.to/jborak</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F209108%2Fedfcbb48-adee-4e82-b1db-0d4ecf9c90c5.jpg</url>
      <title>DEV Community: John</title>
      <link>https://dev.to/jborak</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jborak"/>
    <language>en</language>
    <item>
      <title>Host a Private Docker Registry</title>
      <dc:creator>John</dc:creator>
      <pubDate>Mon, 30 Mar 2020 13:54:37 +0000</pubDate>
      <link>https://dev.to/jborak/host-a-private-docker-registry-146e</link>
      <guid>https://dev.to/jborak/host-a-private-docker-registry-146e</guid>
      <description>&lt;p&gt;Hosting a private Docker Registry can help the productivity of teams that are building containers to deploy software and services.  Websites like &lt;a href="https://hub.docker.com"&gt;Docker Hub&lt;/a&gt; provide free public repos but not all teams want their containers to be public.  Private repos require a &lt;a href="https://hub.docker.com/pricing"&gt;paid&lt;/a&gt; plan that begins at $7/month.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--2zNQi16j--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/5z1b8vht7cx8pxy70j1x.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--2zNQi16j--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/5z1b8vht7cx8pxy70j1x.png" alt="Alt Text"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;AWS, Google, Azure and other public clouds also provide Docker Registry services as well.  These are all priced around $0.10/Gb.  For example if you were using 50 Gb of storage across any number of repos you're hosting them with, it could be around $5/month.&lt;/p&gt;

&lt;p&gt;If you maintain a lot of Docker images this could potentially add up.  If you have experience building containers you know how difficult it can be to trim the size of a container.  It's a process to optimize size. &lt;/p&gt;

&lt;p&gt;In this tutorial I'm going to show you how to setup a private Docker Registry, make it publicly available, password-protected and use TLS.  It'll be secure and only available to you and your team, wherever they're located.&lt;/p&gt;

&lt;h1&gt;
  
  
  Requirements
&lt;/h1&gt;

&lt;p&gt;This is what you need to run through this tutorial.  We need Docker installed, in addition we'll use docker-compose to make setting this up and maintaining (software updates) more convenient.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Packetriot &lt;a href="https://packetriot.com/signup"&gt;account&lt;/a&gt; &lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.docker.com/install/"&gt;Docker CE&lt;/a&gt; installed (Linux, Mac or Windows)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.docker.com/compose/install/"&gt;Docker-compose&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h1&gt;
  
  
  Setup the Containers
&lt;/h1&gt;

&lt;p&gt;I use a convention for deploying a set of containers using docker-compose.  I create a directory for storing the compose file and any other small configuration items that I want available my host system.  I use Docker volumes for storing large amounts of data and making it accesseable and recoverable in-case the containers are accidentally destroyed.&lt;/p&gt;

&lt;p&gt;This convention makes upgrades easy to perform and we'll review that later in this tutorial.&lt;/p&gt;

&lt;p&gt;To start, let's create a directory called &lt;code&gt;private-registry&lt;/code&gt; and place our compose file &lt;code&gt;docker-compose.yml&lt;/code&gt; there.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/] mkdir private-registry

# change directories
[user@host:~/] cd private-registry

[user@host:~/private-registry ]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Below is the compose file we'll use.  It creates two containers, one for the Packetriot client and another for the Docker Registry.  It's good practice to segment containers that don't need to communicate.  To do this we create a separate network called &lt;code&gt;registry-net&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Finally, our registry will store &lt;em&gt;a lot&lt;/em&gt; of data.  Container images can consume space quickly and we don't want to lose any of that data accidently.  We also want to spare the hosts that are pushing these images time.  For that we want to reuse existing layers whenever possible throughout the operational life of this registry.  So we define a volume called &lt;code&gt;registry-data&lt;/code&gt; to store image data.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;version: '2'

networks:
  registry-net:
    external: false

volumes:
  registry-data:

services:
  pktriot:
    container_name: tunnel
    image: packetriot/pktriot:latest
    networks:
      - registry-net
    restart: unless-stopped
    volumes:
      - ./pktriot:/data

  registry:
    container_name: registry
    image: registry:2
    networks:
      - registry-net
    restart: unless-stopped
    environment:
      REGISTRY_AUTH: htpasswd
      REGISTRY_AUTH_HTPASSWD_REALM: "Registry Realm"
      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
    volumes:
      - registry-data:/var/lib/registry
      - ./auth:/auth

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;There are some other important items to highlight in this compose file.  Under the &lt;code&gt;pktriot&lt;/code&gt; service we have a local directory mapped to &lt;code&gt;/data&lt;/code&gt; in that container.  &lt;code&gt;/data&lt;/code&gt; is the default Packetriot client configuration directory for containers.  It's useful to maintain this outside the container so we can upgrade our container.  You want to avoid reconfiguring your tunnel just to upgrade the client.&lt;/p&gt;

&lt;p&gt;We also added a volume mapping for the &lt;code&gt;registry&lt;/code&gt; container.  In addition, we mapped a local directory called &lt;code&gt;auth&lt;/code&gt; to &lt;code&gt;/auth&lt;/code&gt; in the container.  This is where we will store the hashed authentication credentials.  Generating them is our next step.&lt;/p&gt;

&lt;p&gt;Create a sub-directory called &lt;code&gt;auth&lt;/code&gt; inside our &lt;code&gt;private-registry&lt;/code&gt; directory.  We are going to create and hash credentials for Docker Registry using a program called &lt;code&gt;htpasswd&lt;/code&gt;.  If you don't have &lt;code&gt;htpasswd&lt;/code&gt; already installed, here are some commands to install on different Linux systems.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# debian or ubuntu
[user@host:~/private-registry ] apt-get install apache2-utils

# latest fedora releases
[user@host:~/private-registry ] dnf install httpd-tools

# centos 7.x
[user@host:~/private-registry ] yum install httpd-tools
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;On a Linux system you can use the following command to create a hash for registry credentials.  Replace &lt;code&gt;username&lt;/code&gt; and &lt;code&gt;secretpassword&lt;/code&gt; with your own values.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] mkdir auth

[user@host:~/private-registry ] htpasswd -Bbn username secretpassword &amp;gt; auth/htpasswd
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Not on a Linux system?  Well we can use the &lt;code&gt;registry:2&lt;/code&gt; container image to do this as well.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] mkdir auth

[user@host:~/private-registry ] docker run --entrypoint htpasswd registry:2 -Bbn username secretpassword &amp;gt; auth/htpasswd
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Check out the official &lt;a href="https://docs.docker.com/registry/deploying/"&gt;documentation&lt;/a&gt; for more options on setting up authentication for a registry.&lt;/p&gt;

&lt;p&gt;We're ready at this point to create our containers and start them up.  Just run the following command &lt;code&gt;docker-compose up -d&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] docker-compose up -d

Creating network "services_registry-net" with the default driver
Creating volume "services_registry-data" with default driver
Pulling pktriot (packetriot/pktriot:latest)...
latest: Pulling from packetriot/pktriot
e7c96db7181b: Pull complete
bf73012dc58e: Pull complete
144a721ceff9: Pull complete
94370349f88c: Pull complete
1ca8e6a39ad4: Pull complete
Digest: sha256:c0008414d7c0dd40d29d4150bdc67b0d32ae7454372629aafcff6a354f250343
Status: Downloaded newer image for packetriot/pktriot:latest
Pulling registry (registry:2)...
2: Pulling from library/registry
486039affc0a: Pull complete
ba51a3b098e6: Pull complete
8bb4c43d6c8e: Pull complete
6f5f453e5f2d: Pull complete
42bc10b72f42: Pull complete
Digest: sha256:7d081088e4bfd632a88e3f3bcd9e007ef44a796fddfe3261407a3f9f04abe1e7
Status: Downloaded newer image for registry:2
Creating tunnel   ... done
Creating registry ... done

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Docker-compose will create our the &lt;code&gt;registry-net&lt;/code&gt; network for our two containers and the volume for our registry.  It will pull down the latest versions of both of our images for the tags we're using in our compose file.  Run the command &lt;code&gt;docker ps&lt;/code&gt; and check that our containers are running.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] docker-compose up -d
CONTAINER ID        IMAGE                       COMMAND                  CREATED              STATUS              PORTS               NAMES
74545f6b7c1b        packetriot/pktriot:latest   "/usr/bin/pktriot --…"   About a minute ago   Up About a minute                       tunnel
000d2a809b65        registry:2                  "/entrypoint.sh /etc…"   About a minute ago   Up About a minute   5000/tcp            registry

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Our containers are created, the registry is running but our Packetriot client is not yet configured.&lt;/p&gt;

&lt;h1&gt;
  
  
  Configure Packetriot Client
&lt;/h1&gt;

&lt;p&gt;The Packetriot client container works just like the client when it's locally available on your computer.  However, we need to use the command &lt;code&gt;docker exec -it tunnel&lt;/code&gt; to execute commands inside the container.&lt;/p&gt;

&lt;p&gt;Before we continue, print out the contents of the local &lt;code&gt;pktriot&lt;/code&gt; directory.  We'll see a config.json file there.  It's empty, as expected.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] ls pktriot/
config.json

[user@host:~/private-registry ] cat pktriot/config.json
{}

[user@host:~/private-registry ] 
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;We're going to configure our client like normal.  If your new to Packetriot check out the &lt;a href="https://docs.packetriot.com/quickstart"&gt;quickstart&lt;/a&gt; for a quick run-through of configuring tunnels.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] docker exec -it tunnel pktriot configure
Choose a path to the configuration file:
[1] /etc/pktriot/config.json
[2] /data/config.json
[3] /root/.pktriot/config.json

Input selection [#]: 2

Authenticate client with login credentials:

Email: user@example.com
Password: 
Authenticated!

Choose the region for the edge server to connect to:
+------------------------+
| #   | Region           |
+------------------------+
| 1   | us-east          |
+------------------------+
| 2   | us-west          |
+------------------------+
| 3   | eu-central       |
+------------------------+
| 4   | asia-southeast   |
+------------------------+

Input selection [#]: 1

Tunnel configuration:
  Hostname: epic-butterfly-62904.pktriot.net
  Server: us-east-65319.packetriot.net
  IP: 159.203.126.35

Start the tunnel and visit URL to check its working:
  pktriot --config /data/config.json start
  https://epic-butterfly-62904.pktriot.net

Detailed help and step-by-step tutorials:
  https://packetriot.com/docs
  https://packetriot.com/tutorials.

Need more support?
  Email: packetriot@gmail
  Twitter: @packetriot (please follow us, we like new friends :)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;After we configure the tunnel the client will print out all the information we need to setup our HTTP/S traffic rules.  Note the hostname assigned to this tunnel &lt;code&gt;epic-butterfly-62904.pktriot.net&lt;/code&gt;.  Yours will obviously be different.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Tunnel configuration:
  Hostname: epic-butterfly-62904.pktriot.net
  Server: us-east-65319.packetriot.net
  IP: 159.203.126.35
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;I don't be setting up a custom domain in this tutorial.  That is possible and you can use Let's Encrypt in the client to manage your certificates for you.  Briefly, using a custom domain would require:&lt;/p&gt;

&lt;p&gt;1) Verifying ownership of the domain&lt;br&gt;
2) Create an DNS record (A or CNAME) that will point to the server we connect to, &lt;code&gt;us-east-65319.packetriot.net&lt;/code&gt; or &lt;code&gt;159.203.126.&lt;/code&gt; &lt;br&gt;
3) Substitute the assigned hostname in the commands below with your custom domain.&lt;br&gt;
4) Enable Let's Encrypt with the --letsencrypt flag (if you plan to use LE)&lt;/p&gt;

&lt;p&gt;Back to the tutorial... &lt;/p&gt;

&lt;p&gt;We can quickly verify that our tunnel is running by just restarting the container.  When there are no rules a client will just provide the default "Welcome" page when you visit the hostname assigned to your tunnel.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] docker restart tunnel
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;In our example, we would visit &lt;a href="https://epic-butterfly-62904.pktriot.net"&gt;https://epic-butterfly-62904.pktriot.net&lt;/a&gt; in our browser to verify the tunnel is working as expected.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--eXhNL8M_--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/ldsvqseska0hft6iyt47.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--eXhNL8M_--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/ldsvqseska0hft6iyt47.png" alt="Alt Text"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Let's configure our HTTP/S traffic rule.  The hostname assigned to our tunnel will make use the wildcard TLS certificates &lt;code&gt;*.pktriot.net&lt;/code&gt; on the Packetriot server we connect to, so that takes cares of HTTPS support.&lt;/p&gt;

&lt;p&gt;When you run a set of containers in a private Docker network, it will provide name resolution for the container.  E.g. &lt;code&gt;registry&lt;/code&gt; will be resolved to the containers private IP.  Since these IPs can change, we will use the container name, this will eliminate any need to maintain our traffic rule.&lt;/p&gt;

&lt;p&gt;Our rule will forward to the &lt;code&gt;registry&lt;/code&gt; container using the &lt;code&gt;--destination&lt;/code&gt; flag and since the registry listens on port 5000 so we will pass that to use that as the argument with &lt;code&gt;--http&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] pktriot tunnel http add --domain epic-butterfly-62904.pktriot.net \
    --destination registry --http 5000

[user@host:~/private-registry ] docker restart tunnel
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Restarting the tunnel is necessary.  Our new rule will be setup once the container restarts and now &lt;a href="https://epic-butterfly-62904.pktriot.net"&gt;https://epic-butterfly-62904.pktriot.net&lt;/a&gt; will make our Docker Registry accessible over TLS.  Since we added authentication, it'll be private.&lt;/p&gt;

&lt;p&gt;Verify the tunnel has connected and our new rule will be printed out as well.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] docker logs tunnel
Connecting...

Running HTTP services:
+----------------------------------------------------------------------------------------------------------+
| Domain                             | Destination   | HTTP   | TLS   | Secure   | Protect   | Site Root   |
+----------------------------------------------------------------------------------------------------------+
| epic-butterfly-62904.pktriot.net   | registry      | 5000   | 0     | true     |           | --          |
+----------------------------------------------------------------------------------------------------------+
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h1&gt;
  
  
  Test
&lt;/h1&gt;

&lt;p&gt;From the computer you're working on, or a remote one, connect to the Docker Registry using the command &lt;code&gt;docker login&lt;/code&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] docker login epic-butterfly-62904.pktriot.net
Username: username
Password: 
WARNING! Your password will be stored unencrypted in /home/user/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Congrats!  Our private Docker Registry is working.  Let's do a quick test to make sure it works.  We'll pull down the &lt;code&gt;hello-world&lt;/code&gt; image, tag it and then push to our registry.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] docker pull hello-world

[user@host:~/private-registry ] docker tag hello-world epic-butterfly-62904.pktriot.net/hello-world

[user@host:~/private-registry ] docker push epic-butterfly-62904.pktriot.net/hello-world
The push refers to repository [epic-butterfly-62904.pktriot.net/hello-world]
af0b15c8625b: Pushed 
latest: digest: sha256:92c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a size: 524

[user@host:~/private-registry ] 
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h1&gt;
  
  
  Updating Our Registry
&lt;/h1&gt;

&lt;p&gt;Updating our set of containers is simple.  We've configured them such that the our authentication credentials, client configuration and image data are all hosted and accessible outside of the containers.  The containers can be destroyed and recreated and to update the underlying software in the containers.&lt;/p&gt;

&lt;p&gt;We'll perform some update steps and then verify we can pull down that &lt;code&gt;hello-world&lt;/code&gt; from our registry when we bring them back up.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# stop the containers
[user@host:~/private-registry ] docker-compose stop
Stopping tunnel   ... done
Stopping registry ... done

# destroy the containers
[user@host:~/private-registry ] docker-compose rm 
Going to remove tunnel, registry
Are you sure? [yN] y
Removing tunnel   ... done
Removing registry ... done

# recreate the containers, note, when updates exist the typical
# "image pulling layers" information will be printed
[user@host:~/private-registry ] docker-compose up -d
Creating tunnel   ... done
Creating registry ... done
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Verify we can still connect to our registry after this "updating" operation.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[user@host:~/private-registry ] docker pull epic-butterfly-62904.pktriot.net/hello-world:latest
latest: Pulling from hello-world
Digest: sha256:92c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a
Status: Image is up to date for epic-butterfly-62904.pktriot.net/hello-world:latest
epic-butterfly-62904.pktriot.net/hello-world:latest

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;Awesome, it works!  Now you're ready to host and &lt;em&gt;maintain&lt;/em&gt; your own private, secure and Internet accessible Docker Registry.&lt;/p&gt;

&lt;h1&gt;
  
  
  Conclusion
&lt;/h1&gt;

&lt;p&gt;The process for setting up a private Docker Registry is pretty straight-forward, especially for those already familiar with Docker and the Packetriot client.  &lt;/p&gt;

&lt;p&gt;What else could we do next?  I would setup the registry using a custom domain.  You just need to follow the typical steps for exposing a service with a custom domain.  And you can use Let's Encrypt for your TLS certs as well.&lt;/p&gt;

&lt;p&gt;It's important to note that container images can be quite big.  Pushing and pulling from your private registry will use bandwidth.  You can definitely exercise this tutorial on your free tier but for practical use, you will want to sign up for at least a Basic ($5/month) plan.&lt;/p&gt;

&lt;p&gt;Have any questions or issues reproducing this setup?  Please leave comments below I'm glad to help.  &lt;/p&gt;

&lt;p&gt;If you really found this helpful I invite you to share with friends.  Cheers!&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>productivity</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Blocking Disposable Emails</title>
      <dc:creator>John</dc:creator>
      <pubDate>Mon, 23 Mar 2020 17:27:47 +0000</pubDate>
      <link>https://dev.to/jborak/blocking-use-of-disposable-emails-1a3i</link>
      <guid>https://dev.to/jborak/blocking-use-of-disposable-emails-1a3i</guid>
      <description>&lt;p&gt;There are legitimate reasons why some users decide to use a disposable or temporary email to sign up for a service.  They just want to try it out and don't want more email (or spam) in their inbox, I get it.&lt;/p&gt;

&lt;p&gt;However, there's other users, and some bots, where it's a means to stuff accounts on your website or abuse your service.  &lt;/p&gt;

&lt;p&gt;This can also mess with metrics your tracking.  Are you building an email list for a newsletter?  Trying to gauge interest and collect leads for your new idea through a landing page?  &lt;/p&gt;

&lt;p&gt;Disposable emails can degrade your efforts here.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--FrMc4Wvb--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/cj52h8gr5nd6rxpfj8ej.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--FrMc4Wvb--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/i/cj52h8gr5nd6rxpfj8ej.jpg" alt="Alt Text"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Even though a disposable email can serve legitimate scenarios, you may want to consider how you deal with them and if it's worth the effort to block those services or at least distinguish those users.&lt;/p&gt;

&lt;h1&gt;
  
  
  My Projects Concerns
&lt;/h1&gt;

&lt;p&gt;The SaaS I build, &lt;a href="https://packetriot.com"&gt;Packetriot&lt;/a&gt;, provides an application tunnelling service and it's used primarily by developers that are building an application and want to host it instantly.  This could be an MVP or a iteration you want to show a client or a colleague.&lt;/p&gt;

&lt;p&gt;There's many people that use Packetriot to just self-host applications.  Some want more control over their data and others because storage and cloud computing are just expensive.  If you know how to configure a Linux instance on AWS then you can probably self-host many services yourself for much cheaper.&lt;/p&gt;

&lt;p&gt;Unfortunately, it can also be used to host a spam link or phishing website and I don't want Packetriot to be used a delivery system for bad actors.  Similar services like &lt;a href="http://serveo.net"&gt;Serveo&lt;/a&gt; were shutdown because of abuse.  I want to protect my users, my business and other Internet users.&lt;/p&gt;

&lt;p&gt;I wanted to share the strategy I took to stop spammy users and bots stuffing accounts on my website.  I also have some &lt;a href="https://github.com/packetriot/abuse"&gt;code&lt;/a&gt; to share that I wrote to help me add this in my registration and other workflows.  I'll also discuss how effective this was as well, TLDR; it worked well!&lt;/p&gt;

&lt;h1&gt;
  
  
  Goals
&lt;/h1&gt;

&lt;p&gt;In the case of Packetriot there's a big opportunity for abuse by spammers and phishers.  Both these groups want to send you a link to click on... never click the link!!  My goal is to prevent abuse the best I can but also avoid creating more onboarding obstacles for potential users.&lt;/p&gt;

&lt;p&gt;Many Packetriot alternatives have different levels of onboarding.  All of the registration-less services eventually fail due to abuse.  In addition, I want to prevent the resource utilization by abusive users and maintain the highest quality of service.&lt;/p&gt;

&lt;p&gt;Packetriot has a free-tier that provides with enough features for casual users or those who want to try the service, integrate our client into their workflow and evaluate how well we can host their website or app.&lt;/p&gt;

&lt;p&gt;We use link confirmations to begin onboarding a new user.  You'd be surprised how well bots are programmed to automatically traverse this workflow.  To combat this, I use &lt;a href="https://google.com/recaptcha"&gt;reCAPTCHA V3&lt;/a&gt;.  However, they're lots of people manually doing this, and that's where disposable emails broke my captcha.&lt;/p&gt;

&lt;h1&gt;
  
  
  Some Code
&lt;/h1&gt;

&lt;p&gt;Packetriot is written in Go and our package &lt;a href="https://github.com/packetriot/abuse"&gt;abuse&lt;/a&gt; is a small library I wrote to perform checks on email address or domains during registration and email confirmation.  It's opensource and free to use and contribute to.&lt;/p&gt;

&lt;p&gt;To start using, import into your Go app.  Calling the &lt;code&gt;Init()&lt;/code&gt; function isn't necessary, but if you want to be able to update the list of abusive domains you'll need to do this.  Later on in this article I share a behavior by most disposable email providers that enable us to add their new domains automatically :)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;import (
    "github.com/packetriot/abuse"
)

func main() {
    abuse.Init("/path/to/app-data/abusive-domains.txt")
    defer abuse.Close()
}

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;During registration I check if the domain in the email address is already in my list abusive domains.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;func registerUser(w http.ResponseWriter, r *http.Request) {

    // parse email from HTML form submission
    email := "..."

    if abuse.IsTempEmail(email) {
        serveNoTempEmail(w, r)
        return
    }
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;p&gt;We created an update mechanism into our package that leverage how some disposable email services work.  For example, the domain names of the email addresses generated change over time, but the primary sitename, and thus &lt;code&gt;Referrer&lt;/code&gt;, does not.  So when a user clicks on a confirmation link we send them, the provider's sitename is set as the &lt;code&gt;Referrer&lt;/code&gt; value in the HTTP header.&lt;/p&gt;

&lt;p&gt;This simple check helps us keep up with providers automatically.  We can add the domain from this new email address to our list and then the next time we see an email address with this domain, they won't be able to get passed the registration screen.&lt;/p&gt;

&lt;p&gt;Here is an example of how to implement this.  The important function is the &lt;code&gt;abuse.Add(domain string)&lt;/code&gt; function.  Once we discover the &lt;code&gt;Referrer&lt;/code&gt; is a domain we already have in our list then we can add this new domain and automatically keep up with the new domains that are generated almost weekly from some providers.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight"&gt;&lt;pre class="highlight plaintext"&gt;&lt;code&gt;func confirmEmail(w http.ResponseWriter, r *http.Request) {
    user := getUserFromSession(r)
    if referrer :=  r.Header.Get("Referrer"); len(referrer) &amp;gt; 0 {
        u, _ := url.Parse(referrer)
        if abuse.IsAbusiveDomain(u.Host) {
            // Must be a new domain for emails but coming from a provider we know
            abuse.Add(u.Host)

            serveNoTempEmail(s, w, r)
            return
        }
    }

    // ...
}

func getUserFromSession(r *http.Request) *User {
    // ...
    return user
}

type User struct {
    Email string
    // ...
}

&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;



&lt;h1&gt;
  
  
  Conclusion
&lt;/h1&gt;

&lt;p&gt;Incorporating this in my site has eliminated a vast majority of accounts that I eventually find were created with a disposable email.  In your scenario, you may want to just flag a user that is using a temporary email and then deal with them in some other way.&lt;/p&gt;

&lt;p&gt;Many of my competitors are outright blocked on corporate networks because they're abused by bad actors and Infosec professionals don't trust them.  I want to build trust with them and my users. Ideally, I don't want my network to ever carry malicious traffic.&lt;/p&gt;

&lt;p&gt;For Packetriot it's the right call because we don't want malicious actors using our service to perpetuate their bad stuff on the Internet.&lt;/p&gt;

&lt;p&gt;I played cat-and-mouse with disposable email services for months, and I routinely found that the people using them (on my service) were never up to any good.  If this is an issue that impacts your project then I hope this article was helpful.&lt;/p&gt;

&lt;p&gt;I routinely update our &lt;a href="https://github.com/packetriot/abuse"&gt;package&lt;/a&gt; when I discover new services.  If you'd like to use it and want any help with integrating it please feel free to reach out.  Also, let me know if you add it to your website.  Cheers!&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>go</category>
      <category>security</category>
      <category>programming</category>
    </item>
    <item>
      <title>Local Hosting Tools</title>
      <dc:creator>John</dc:creator>
      <pubDate>Mon, 25 Nov 2019 12:47:55 +0000</pubDate>
      <link>https://dev.to/jborak/local-hosting-tools-446o</link>
      <guid>https://dev.to/jborak/local-hosting-tools-446o</guid>
      <description>&lt;p&gt;Local Hosting tools are useful to developers in many scenarios.  There's many tools and providers to choose from and they are all built around the basic purpose of providing access to a locally running service that would otherwise be inaccessible to the Internet.  &lt;/p&gt;

&lt;p&gt;This solves a lot of problems in web development.  The most basic is testing your code or sharing it with your team or a client.  In other scenarios they make it possible to develop and test with an API that expect an HTTP/S callback to be publicly available.  Examples of these APIs come from companies like Stripe, Intercom, Slack, GitHub, etc...&lt;/p&gt;

&lt;p&gt;You may want to self-host applications or services that are part of your CI/CD workflow.  &lt;/p&gt;

&lt;p&gt;Let's quickly dive into how they work, how to roll your own, and some options for providers and tools that can save you time.&lt;/p&gt;

&lt;h1&gt;
  
  
  How They Work
&lt;/h1&gt;

&lt;p&gt;Typically, they work by creating a reverse tunnel to a server from your workstation.  The server uses HTTP virtual hosts and TLS-SNI parsing to route traffic to the reverse tunnel and eventually your system.  Below is a diagram illustrating the basic architecture of most of these local hosting services and tools.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fthepracticaldev.s3.amazonaws.com%2Fi%2Fb03euc72hwcwvsp3a7jp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fthepracticaldev.s3.amazonaws.com%2Fi%2Fb03euc72hwcwvsp3a7jp.png" alt="Alt Text"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h1&gt;
  
  
  Roll Your Own
&lt;/h1&gt;

&lt;p&gt;There's many ways to expose a local service.  You can always setup your own server on a cloud provider and push code or fresh builds to that system.&lt;/p&gt;

&lt;p&gt;Another option is to setup similar infrastructure to what local hosting services provide to you, a reverse tunnel that routes traffic to your workstation.  There's a few ways to do this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Run a VM in the cloud, set it up as a VPN server, connect to it from your workstation, setup IP masquerading rules so that all HTTP/S or other traffic is NAT'd across the VPN connection to your server. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Run a VM in the cloud, use SSH to create some reverse tunnels.  Forward all HTTP/S and other traffic to your local server. &lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Implementing either isn't difficult if you have experience with VPNs or networking.  Maybe you're juggling multiple projects, builds, or workstations, the effort can begin to accumulate and its not worth your time.  &lt;/p&gt;

&lt;p&gt;You'll also need to purchase a domain and setup DNS records.  If your need is temporary or intermittent it may be too much effort.&lt;/p&gt;

&lt;h1&gt;
  
  
  Choosing a Provider
&lt;/h1&gt;

&lt;p&gt;There's tools and services and that make this easy and automatic.  I'll provide a list below, some opensource alternatives, and some honorable mentions at the end.&lt;/p&gt;

&lt;h2&gt;
  
  
  PageKite
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://pagekite.net" rel="noopener noreferrer"&gt;PageKite&lt;/a&gt; has been around for while.  It requires a Python interpreter and is easy to setup as long as Python is already installed.  For most systems, this isn't an obstacle.  There's paid tiers for more bandwidth, more simulatenous clients, custom subdomais from  .pagekite.me and the ability to use create CNAME records using the assigned subdomains.   &lt;/p&gt;

&lt;p&gt;The PageKite client can serve to upstream servers or static content, but not both for the same site.  If your upstream server has static content, it must serve it, or you need a web server like nginx in conjunction to help serve both.  &lt;/p&gt;

&lt;p&gt;With PageKite you can also IP whitelist and password protect subdomain and these features are available on their free tier.&lt;/p&gt;

&lt;p&gt;Getting started with Pagekite is quick and easy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ curl -O https://pagekite.net/pk/pagekite.py
$ python pagekite.py 80 yourname.pagekite.me 
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Registration and configuration is performed through a series of prompts in the client program.  Email confirmation is required any real use of the service.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ngrok
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://ngrok.com" rel="noopener noreferrer"&gt;Ngrok&lt;/a&gt; is a another option that has been around for a few years and although it supports HTTP/S and TCP relaying, it was built to make developing and testing webhooks and API callbacks more efficiently.  It does this by introducing a a web dashboard that enables you to inspect requests and replay them.  &lt;/p&gt;

&lt;p&gt;Ngrok comes with paid tiers as well that enable more custom domains, reserved TCP ports, IP whitelists, more bandwidth and more simultaneously running processes.&lt;/p&gt;

&lt;p&gt;Getting started with Ngrok is straight-forward.  You setup an account, download the right client for your platform and use the provided auth token to setup your client.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ ./ngrok authtoken ABC123...
$ ./mgrok http 80
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Packetriot
&lt;/h2&gt;

&lt;p&gt;Let me first state that I'm the author of &lt;a href="https://packetriot.com" rel="noopener noreferrer"&gt;Packetriot&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The project has been around for almost a year and I've had a big interest in this space of tool development for some time. I started it to resolve gaps I experienced with existing tools.  I wanted more integrated features and long-lived tunnels to enable more reliable self-hosting.&lt;/p&gt;

&lt;p&gt;Packetriot provides the same features as tools mentioned earlier and fill the gaps they leave behind.  Like PageKite it can serve static content, but it can also forward to upstream servers as well, on the same domain.  Eliminating the need for setting up a separate web server like nginx or apache.&lt;/p&gt;

&lt;p&gt;You probably own lots of domains (that you may never use...)  You can setup CNAME and A records and they will work just fine.&lt;/p&gt;

&lt;p&gt;Let's Encrypt is supported by the Packetriot client, along with automatic management of those certificates.  The client will automatically renew certs before their expiration.  No need to install certbot, setup a cron job to renew, or debug why the certbot integration isn't working.&lt;/p&gt;

&lt;p&gt;Unlike Ngrok, the subdomain, e.g. abc-1234.pktriot.net, that is assigned to your tunnel is static.  So for any given project you can create configurations with that hostname for testing and do not have to worry about editing it constantly.&lt;/p&gt;

&lt;p&gt;All websites can be protected with a password.  Paid plans include IP whitelists and blacklists, access logs and metrics per connection and grouped by destination website or TCP port.  &lt;/p&gt;

&lt;p&gt;There is even a upstream service health monitor that will email you when those services die.  No need to integrate or pay for another monitoring tool.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://packetriot.com/downloads" rel="noopener noreferrer"&gt;Packaging&lt;/a&gt; and diverse platforms are well supported, along with a commitment to long-lived tunnels.  Support for systemd services is included with each Debian and RPM package.  All archives for Linux, Windows and Mac includes example configurations so that constant run-time can be setup.  Containers are built for each new release as well.&lt;/p&gt;

&lt;p&gt;You can serve any number of websites behind a Packetriot tunnel.  The primary restriction is bandwidth.  Paid plans provide you more tunnels, bandwidth, and more advanced features.&lt;/p&gt;

&lt;p&gt;Getting started with Packetriot is easy.  Once you create an account and confirm your email, find the right client and then setup traffic in one command.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ pktriot http 8080

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This will kick off several prompts to configure your client and then begin forwarding traffic to your local service.  &lt;/p&gt;

&lt;p&gt;You can perform more advance routing options with traffic rules that will be persistent.  Below are some examples:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ pktriot configure --login

# setup my custom domain, serve static and upstream, use automatic
# Let's Encrypt and redirect all HTTP requests to HTTPS.

$ pktriot route http add --domain example.com --webroot /path/to/static --http 8080 --letsencrypt --redirect

# transparently forward HTTP/S traffic to a host
$ pktriot route http add --domain example.com --http 80 --tls 443 --destination 192.168.0.10

# setup a password protected, static-file sharing site using 
$ Let's Encrypt and secure redirecting all requests.
$ pktriot route http add --domain example.com --webroot /path/to/static --password --letsencrypt --redirect


# allocate a TCP port that is assigned your tunnel until you delete
# the tunnel or release the port.
$ pktriot route tcp allocate
Port 22000 was allocated!

$ pktriot route tcp forward --port 22000 --destination 192.168.0.10 --dstport 22
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Opensource Alternatives
&lt;/h2&gt;

&lt;p&gt;There are a few opensource alternatives you can choose as well.  &lt;a href="https://github.com/antoniomika/sish" rel="noopener noreferrer"&gt;sish&lt;/a&gt; uses SSH for setting up tunnels to a VM you create and setup.  &lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/fatedier/frp" rel="noopener noreferrer"&gt;FRP&lt;/a&gt; is a highly configurable and capable opensource alternative. &lt;/p&gt;

&lt;h2&gt;
  
  
  Honorable Mention
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://serveo.net" rel="noopener noreferrer"&gt;Serveo&lt;/a&gt; was an extremely popular local hosting tool.  It was free, required no account creation and was cleverly built to leverage the existing ssh client to build it's tunnels.  However, a system with no user creation or confirmation, was riped to be targeted by spammers and phishers, to obscure their systems and abuse users.  &lt;/p&gt;

&lt;p&gt;This led to constantly interruptions in service and its temporary shutdown.  Who knows if it'll come back in its original form.&lt;/p&gt;

&lt;h1&gt;
  
  
  Conclusion
&lt;/h1&gt;

&lt;p&gt;There are many local hosting tools to choose and although they try to serve the same core problem of making a local services available for sharing or testing, they all come with pros and cons.&lt;/p&gt;

&lt;p&gt;If local hosting tools are new to you then I'm glad if I've been able to introduce them and provide a survey of what's available.  Note, this is not a comprehensive list.&lt;/p&gt;

&lt;p&gt;If you've been using these types of tools for some time, I hope there's  tools or services I mentioned that are new to you.  Please share your experience and let us know how you integrate these tools in your workflow.&lt;/p&gt;

&lt;p&gt;Interested in more use-cases or how to configure these tools with your work environment?  Let me know know in the comments.&lt;/p&gt;

</description>
      <category>devtools</category>
      <category>webdev</category>
      <category>testing</category>
      <category>productivity</category>
    </item>
  </channel>
</rss>
