Photon OS & Octopus Tentacles: Installation and Setup

What is it?

Photon OS is a purpose built Linux flavor by VMware specially tuned for running in a VM and hosting container products such as Docker containers.
Octopus is a Continuous Delivery platform that is great for managing deployments and run-books via it's agents called Tentacles - Octopus has a strong commitment to their naming metaphors 🐙.

But Why?

Octopus Deploy simplifies and allows repeatable, stable self documenting, institutionalized deployments and release management. Marrying the Strengths of Octopus with Docker deployments seems a natural Win-Win. Also I wanted to create, for myself, an easy to deploy-to, stable Docker environment to experiment and play with & learn Docker.

Introduction

Both at work and at home I use VMware's ESXi vSphere for hosting and managing virtual machines. At work we've started using Docker containers for some projects and have Photon OS running these containers. We also use Octopus Deploy to manage our CD infrastructure. Our at-work setup uses the SSH model for communication to the Photon OS, this is because Octopus Tentacles (Octopus's name for their software agents) are not supported on Photon OS at this time. However, I'm a little determined and doggedly set out to find out if we could run an Octopus Tentacle as a Listening Tentacle on Photon OS.

Assumptions

First let's establish some assumptions:

• We have a functioning VMware ESXi Host
  • There is a $0 licensing tier, vSphere Hypervisor ESXi
• We've installed a base Photon OS OVA downloaded from VMware and it starts as expected. Photon OS
• We've done the tdnf distro-sync to bring that Photon OS Host up-to-date
• We have an Octopus Deploy on-premises or cloud instance running and we have the required permissions to add machines in the infrastructure config.
  • If you've not used Octopus Deploy, they offer a $0 pricing tier for their Cloud Instance. I use this to manage my personal servers and laugh a little each time they send me a $0 invoice. Octopus Deploy Pricing
• You have SSH or other terminal access into the Photon OS host.
• I did a bad thingh and performed all these actions as root, being too lazy at this time to set up a proper user account added to the sudo group. For a production environment, and even non-production, security best practices should be followed! Don't do exactly as I have done! 😉 I googled it for you

Optional: Naive attempt: Following the instructions

Skip to things that work
I joke about this being the naive way to do it, but in most cases following this listed instructions is the best way to get a handled on how it should work... and where your setup doesn't match expectations.

According to the documentation we should be able to use the RPM install method (Photon OS's package manager tdnf uses RPM packages):

$ wget https://rpm.octopus.com/tentacle.repo -O /etc/yum.repos.d/tentacle.repo
$ yum install tentacle

results in

-bash: wget: command not found

So we can see that Photon OS doesn't come canned with some of the common tools we expect in a base Linux install, but we can get them.

$ tdnf install wget

Installing:
wget                     x86_64       1.20.3-1.ph3     photon-updates   2.80M 2936976

Total installed size:   2.80M 2936976
Is this ok [y/N]: y

Downloading:
wget                                   1128116   100%
Testing transaction
Running transaction
Installing/Updating: wget-1.20.3-1.ph3.x86_64

Complete!

Ok, Let's try that original again... one line at a time

$ wget https://rpm.octopus.com/tentacle.repo -O /etc/yum.repos.d/tentacle.repo
--2021-01-27 15:39:45--  https://rpm.octopus.com/tentacle.repo
Resolving rpm.octopus.com... 104.20.11.14, 104.20.10.14, 2606:4700:10::6814:a0e, ...
Connecting to rpm.octopus.com|104.20.11.14|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 156 [application/octet-stream]
Saving to: ‘/etc/yum.repos.d/tentacle.repo’

/etc/yum.repos.d/te 100%[===================>]     156  --.-KB/s    in 0s

2021-01-27 15:39:46 (34.0 MB/s) - ‘/etc/yum.repos.d/tentacle.repo’ saved [156/156]

That worked and now we have the octopus tentacle repo added. We should be able to install next.

$ yum install tentacle
Refreshing metadata for: 'Octopus Tentacle'
Error: 403 when downloading https://rpm.octopus.com/x86_64//repodata/repomd.xml
. Please check repo url.
Error(1622) : Invalid argument
Error: Failed to synchronize cache for repo 'Octopus Tentacle' from 'https://rpm.octopus.com/x86_64/'
Disabling Repo: 'Octopus Tentacle'
Error(1011) : No matching packages

Oops, the tentacle repo seems to not work right. Looks like the URL is malformed (note the double / in there.) We will need to modify that repository entry.

$ vi /etc/yum.repos.d/tentacle.repo

and change the line:

baseurl=https://rpm.octopus.com/$basearch/

to

baseurl=https://rpm.octopus.com/$basearch

Removing the trailing slash should allow it to work right.

$ yum install tentacle
Refreshing metadata for: 'Octopus Tentacle'
1. nothing provides openssl-libs needed by tentacle-6.0.390-1.x86_64
Found 1 problem(s) while resolving
Error(1301) : Solv general runtime error

Well this one is a problem. OpenSSL is installed in the base build of Photon OS, but nothing allows OpenSSL-Libs to be installed.

I can tell you from my experience, as best as I know, installing the Octopus Tentacle via RPM is a dead end for now. I've added an issue to the Photon OS' GitHub issue list requesting the OpenSSL-Libs package be included.

What we learned

We learned:

• Photon OS comes out of the box, very, very lean and thus needs the following packages installed from the Photon Repositories.
  • wget - to get things from the web
• I'm not clear on which party is at fault, but adding the Tentacle Repository resulted in the malformed URL being tried. We needed to modify the repo URL manually to remove the extra slash.
• Photon OS's package repositories don't include a OpenSSL-Libs package to satisfy the Tentacle's RPM Dependencies.

Next Steps

Lets skip some of the rest of the trial and error...

• Install OpenSSL from source This was a false result!
  • I had originally believed this was the case, but in the process of writing this post, I tried the steps and couldn't replicate the results. Somewhere along the line I did something I didn't document that was the real cause of success. I started over and, with some more diligent tracking, discovered what really works.
• Setup remaining dependencies
• Use the Archive method from the documentation

Non-Optional: Things that actually work

OK, if you didn't skip the optional stuff, you know that Photon OS is very lean and doesn't come with some quality of life things like wget preinstalled. We also learned there is some packages we would expect to be able to install but just aren't available on the provided repositories. OpenSSL-libs for example.

Actual Steps that allow the Octopus Tentacle to function on Photon OS

• install libstdc++ tar sudo zlib-devel and icu packages
• create/copy a lsb_release script

lsb_release

The Octopus Tentacle calls lsb_release and uses the output during its Health Checks. Thus if it is missing the Health Check will result in a warning. Photon OS doesn't provide lsb_release. I looked into lsb_release on Arch Linux (my current go-to flavor) and found it to be a script. I copied it from an Arch Linux install and it seems to satisfy the Octopus Tentacles.
I'll leave this step as an exercise to the reader. I couldn't find a good way to provide the lsb_release script that I used. I literally copied the text of the script into a new /usr/bin/lsb_release file, then set its executable flag.
I've not looked into better alternatives. I'm excited to see the recommendations of people with more knowledge in this area!

Installing & Configuring the Tentacle

The Linux Octopus Tentacle requires a few things to get started so let's get those installed.

$ RUN tdnf -y install libstdc++ tar sudo zlib-devel icu

Note: the libstdc++ was needed when I was testing this in with the photon Docker image, but appears to be already installed in VMware's downloadable VMs.

Next let's get the Tentacle archive installer (again from the documentation, also modified to be more succinct):

$ cd ~
$ wget https://octopus.com/downloads/latest/Linux_x64TarGz/OctopusTentacle -O tentacle-linux_x64.tar.gz

$ mkdir -p /opt/octopus
$ tar xvzf tentacle-linux_x64.tar.gz -C /opt/octopus

At this point, the instructions at Octopus tell you to execute configure-tentacle.sh but that doesn't work right away as it doesn't have the executable flag set.

$ chmod +x /opt/octopus/tentacle/configure-tentacle.sh

Great, now we can configure the tentacle service

$ /opt/octopus/tentacle/configure-tentacle.sh

This will ask you several questions related to configuring the Octopus tentacle. If you've not already, now is the time to get into your Octopus Instance and start getting ready there...

Octopus and Tentacles

Octopus Deploy acts through agents called Tentacles. The Octopus server instance, either on-premises or their Cloud offering can do Polling Tentacles that reach out periodically to the server to see if there are any instructions, Listening Tentacles that wait for a communication from the server as needed and SSH targets that communicated over SSH on demand. We'll be configuring our tentacle as a Listening Tentacle. If your network has a firewall and/or router between the Octopus Server and the Tentacle, you will likely need to add some port forwarding or other firewall rules to allow communication to work as expected.

Octopus Tentacles by default use the TCP port 10933 to communicate but it configurable and often required. I use those ports plus others to communicate between Octopus Cloud and my tentacles though my firewall, each tentacle has an assigned port number and paired port forwarding at the router.

Let's now start configuring the Tentacle in Octopus,
In your Octopus Dashboard, under the Infrastructure Deployment Targets area, there is a big, handy button for Add Development Target.

This launches a short wizard that walks you through registering the Tentacle.
Select the Linux and Listening Tentacle options:

The next screen has many options that instruct Octopus how to connect to and communicate with your Tentacle:

• The Octopus Server Thumbprint, used as part of the secure communication between Server and Tentacle. You will use this code when configuring the Tentacle.
• The DNS Hostname or IP Address of the server. If you are behind a router, this will be your router IP/Hostname or the machine hostname if on the same network as the server. I use a dynamic DNS setup.
• The Port number to be used for communication. This will need to be the port as available to the Octopus Server if you've configured a port change during port forwarding setup.
• Proxy configuration. Hopefully you will know if you need to configure the proxy and be able to get that information from your network admin.
• The Next button will initiate Discovery (trying to talk to the tentacle, so just hold tight)

Configure the tentacle

If you've not yet started the config utility for your tentacle, run

$ /opt/octopus/tentacle/configure-tentacle.sh

Your first prompt will be

Name of Tentacle instance (default Tentacle):

This will give you a chance to give the Tentacle instance a specific name if you choose. I'm using the default.
Next prompt:

What kind of Tentacle would you like to configure: 1) Listening or 2) Polling (default 1):

In this case, we are using a Listening Tentacle so the default is fine.
The next two options relate to log files and artifact deployment locations. Again I used the defaults.

Where would you like Tentacle to store log files? (/etc/octopus):
Where would you like Tentacle to install applications to? (/home/Octopus/Applications):

The next prompt is the Port number to listen on. Be sure this matches up with the Port number as assigned in Octopus OR the port number on the private side of your port forwarding configuration. Today I'm using port 10939:

Enter the port that this Tentacle will listen on (10933):10939

Next is the very important Octopus Thumbprint mentioned earlier. If possible copy-paste this to prevent errors:

Enter the thumbprint of the Octopus Server: 25------------------------------------EC

Then a note will be displayed informing you of the configurations that will be run. As instructed, press enter to continue!

Hopefully when all is done, you will see this message:

Tentacle instance 'Tentacle' is now installed

One thing we have yet to do is allow port access at the OS level. This will be done with an addition to iptables

$ iptables -A INPUT -p tcp --dport 10939 -j ACCEPT
$ iptables-save

Conclusion

At first glance, running an Octopus Tentacle on Photon OS is not easily accomplished. However after a few false starts, based on some faulty assumptions, I did find a fairly simple path to success. They key missing pieces are the zlib-devel and icu packages along with a functioning lsb_release script.
A properly functioning Octopus Deploy Target provides may benefits such as Machine Health alerts and monitoring. Being able to orchestrate your Docker container activities with Octopus has smoothed out several of my own personal project's infrastructure considerations.

Recent Updates (as of 2022-01-06)

Recently, it seems Octopus's method of tentacle self update has changed to try to use the host's package managers. Unfortunately this isn't method still doesn't work well on Photon OS. I had to re-perform the download and tar extraction steps and restart the service with systemctl. I don't yet know if this was a one-time or every-time change to behaviour but I will add an update here after I find out.