Private Docker Repository

James Linden — Fri, 09 Apr 2021 01:17:56 +0000

The normal way to setup a private Docker repository is to simply use the container which has it all setup. This works really well for a lot of cases. I recently deployed a private repo where I need an implementation that was a bit easier to manage with existing tooling and infrastructure. Basically, this meant not running docker for the repo and integration with an existing authentication framework.

Instructions are CentOS 7 specific, but shouldn't be too hard to map to your distribution of choice.

nginx is used as a proxy to terminate TLS and handle the federated authentication, leaving the docker repository service running on localhost only.

Install

Repos: CentOS Extras, Docker-CE, (possibly EPEL)
Packages: docker-distribution, docker-ce, nginx

Configure

docker-distribution

I only customized the rootdirectory and http addr. See Docker's documentation.
/etc/docker-distribution/registry/config.yml (select parts)

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    layerinfo: inmemory
  filesystem:
     rootdirectory: /path/to/storage/repo
http:
   addr: localhost:5000

nginx

(Not including SSL and federated authentication)

/etc/nginx/conf.d/docker.conf

server {
   server_name myrepo.mydomain;
   listen 443 ssl;
   # omit SSL directives here
   client_max_body_size 0;
   # This just has a simple HTML info page
   root /path/to/storage/docroot;
   index index.html;
   # proxy docker API
   location /v2 {
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header Host myrepo.mydomain;
      proxy_pass http://localhost:5000;
   }
}

docker

(This is optional. I use it to do local admin and some maintenance scripts.

/etc/docker/daemon.json

{
   "data-root": "/path/to/storage/root"
}

Run and Test

docker-distribution

systemctl start docker-distribution
systemctl enable docker-distribution

Verify

curl http://localhost:5000/v2/_catalog

Should return an empty repository list

{
   "repositories": []
}

nginx

systemctl start nginx
systemctl enable nginx

Verify

curl https://myrepo.mydomain/

Should return whatever you put at /path/to/storage/docroot/index.html

curl https://myrepo.mydomain/v2/_catalog

Should return an empty repository list

{
   "repositories": []
}

docker

systemctl start docker
systemctl enable docker

Verify

docker info

Should return some information about your docker installation (including the customized root path).

You have now setup a local Docker repository which will work just like the docker container based one.

HUD: The incomplete project...

James Linden — Wed, 07 Apr 2021 00:38:36 +0000

A recent post by @tobychui about his web OS UI made me nostalgic for one of my own related projects. I have never gotten as far as him to a (human) usable system, but I figured I might find a new hope for my project if I talk about it.

My project (hud.im) was always intended to be open source, once I got it to some basic operational state that anyone could start with. That hasn't happened yet, but maybe we can get it to that point...

The basics

While the terms didn't exist when the project started, HUD is essentially an IoT backend for microservices. The latest semi-functional code is based around RabbitMQ and MongoDB. Any application can connect to it and declare three queues - app_cmd for input to the application, app_notify for information coming from the application, and app_health for basic available/unavailable state.

Shared services

Each application can be configured to store data in the MongoDB. This includes data-retention settings. If a message in any queue is tagged as 'store', a script will inject it into the database, using the queue name as the collection name. Thus, any message in gps_cmd tagged 'store' will end up in a gps_cmd collection in MongoDB.

Other functionality for doing some filesystem operations, debugging, etc are similar.

Example application - GPS device

The GPS data comes from a Garmin GPS USB "puck". A script uses gpsd to continually read data from the USB device. gpsd publishes JSON, so with only a little cleanup, the data is basically usable. The script publishes to gps_notify for anything that wants the data. There is some logic to set the 'store' tag for a percentage of the messages based on interval and/or distance delta. Another script subscribes to gps_cmd and will act on a small command set. The GPS device doesn't have any commands, so in this case, commands are basically to restart gpsd or "bounce" the USB port (sometimes the device just "goes away").

Excerpt from hud-gps.rb

Note: This code (6 yrs old) does not 100% match the explanation, and various generations have php, node.js, ruby, bash, and/or python for the backends.

IO.popen('/usr/bin/gpspipe -p -w /dev/ttyUSB0') { | data |
   mq = Bunny.new(:host => cfg['mq']['host'], :user => cfg['mq']['user'], :pass => cfg['mq']['pass'], :port => cfg['mq']['port'], :vhost => cfg['mq']['vhost'], :heartbeat => 30)
   mq.start
   push = Bunny::Exchange.new(mq.create_channel, :fanout, cfg['mq']['exchange'])
   loop do
      j = JSON.parse(data.gets.chomp).to_h
      if j and j.has_key?('mode') and j.has_key?('class') and j['class'] == 'TPV' and j['mode'] != 1
         j.delete('tag')
         j.delete('class')
         j.delete('device')
         push.publish(JSON.generate({'type' => 'gps', 'data' => j}))
      end
   end
}

Interface

To date, all the various interface (attempts) have been web-based, and communicate with the backend using socket.js to the RabbitMQ service. The UI can subscribe to any of the *_data and *_health queues, and publish to any of the *_cmd queues. Simple JS event handling can update the map overlay with the GPS coordinates from gps_notify, and a couple of buttons can fire off commands for 'reset', etc to gps_cmd.

Basic Map with GPS "dot"

This basic system can be used for anything -- in a vehicle, smart home devices, media centers, etc. I've experimented with many types of devices: GPS puck, OBD2, media players, text-to-speech, voice notes, cameras, Arduino + shields for lights and locks and sensors, various web APIs, messaging (discord, slack, etc) and more.

The questions on everyone's mind are probably, "Does it work?", or "Where is it?".

It's an on again / off again type project. I love the backend and nothing beats the thrill of connecting another device/application into the system. I'm able to make UI that looks like this (Bootstrap) -

But this is what I see in my head (conceptual renderings done for me) -

Anyone else interested in playing with something like this?

DEV Community: James Linden

Private Docker Repository

Install

Configure

docker-distribution

nginx

docker

Run and Test

docker-distribution

Verify

nginx

Verify

docker

Verify

HUD: The incomplete project...

The basics

Shared services

Example application - GPS device

Excerpt from hud-gps.rb

Interface

Basic Map with GPS "dot"