DEV Community: Ryan Brown

A Mermaid to the rescue - visualizing complex problems

Ryan Brown — Fri, 24 Mar 2023 16:55:02 +0000

The Setup

As a developer there are fewer things more frightening than an ask or requirement phrased similar to the following:

I have this excel spreadsheet, but we need it as an application.

Invariably the spreadsheet has a convoluted sequence of functions and back-references, look-ups and unexplainable inconsistencies in format and display.

Ideally sometime before a developer is handed a spreadsheet and the the above “requirement” there could be a detailed analysis of business requirements including data sources, and all transformations expected of that data including test cases and example data. However in a fast paced modern working environment that is a lofty luxury indeed.

So what is our hard-working, indefatigable developer to do? Trying to suss out the hierarchy of column formulas extending up to and possibly column AX, some referencing sums of other formula derived columns, is a head scratchier indeed.

The Situation

In a recent exercise the (nearly) exact situation described above was encountered. The resulting artifact was to be in the form of a Database View. The data was (or would be) available in tables in the target database so that was at least covered. Still, I needed to map out the column inter-relationships and devise which structure of nested select statements would be appropriate to capture the same functionality of the Excel spreadsheet.

I first considered mapping out the column relationships on paper, but quickly realized that doing so would need a substantial amount of erasing as I moved on to the next column and had to re-draw a portion of my map. Clearly a more automatic solution was needed and being a developer... I had an idea for a tool to use.

The Search

A few months ago I stumbled across a diagramming tool that could take diagram descriptions in a markup language and produce decent quality graphical outputs. This was an amazing discovery as I’ve always struggled with the visual documentation tools such as draw.io or Visio. I much prefer to construct things like this in a declarative way rather than using visual tools. For me the benefits of declarative ways include being able to copy-paste easier, check into source control and even programmatically create the declarative statements.

The tooling I found is called Mermaid and it has a really nifty live editor for quick work and learning: Since discovering Mermaid, I’ve used it for some light documentation of processes, data-flows and... some unorthodox things.

The Solution

Back to our problem of trying to get a handle on the rat’s nest of excel formulas. Let’s look at one...

Doesn’t look too bad... bet let’s look again.

$AA$1 is a “constant” to be provided from fairly static table. Not doing too bad so far.

P7 is another formula=SUM(M7:O7) OK so now referencing another formula. Now we're getting more complicated.

$P$19 is =SUM(P6:P18) Now things are interesting... a sum of the set of data.

The whole sheet is littered with things like this. Very often the results of one formula are used in later or earlier formulas.

I figured I could use a type diagram to help me understand these relationships.flowchart
So to start with....

flowchart TD
    AA7 --> $AA$1 & P7 & $P$19

Line 1 Flowchart tells Mermaid that this is a type diagram.

Line 2 is indented for readability. Let’s take it piece by piece
AA7 This is the first node in the diagram. This is the simplest way to define a node, just an identifier.

--> This is a line, in this case an arrow line. It indicates a directional relationship. In this case, that the identifier (column) to the left is related (references) the identifiers (columns) the right.

$AA$1 The next identifier (column) that is the referenced item of the relationship.

& P7 We can indicate that more than 1 thing is related, we could have put the relationship on a separate line as AA7 -> P7 if we wanted to, but I like this one line per referencing column format for clarity.

& $P$19 The next column referenced.

So far we’ve defined a flowchart that has a node AA7 that references $AA$1, P7 and $P$19

Let’s see what it looks like:

We can clearly see that AA7 references the other three columns.

let’s expand on that...

P7 is =SUM(M7:O7) and becomes a line in our diagram P7 --> M7 & N7 & O7 Mermaid doesn’t understand Excel range notation, so I manually unrolled this.

$AA$1 is a value so we don’t need to expand it anymore.

$P$19 is =SUM(P6:P18) I’m really interested at a line-by-line model of this so I can just say that it references back to P7 This becomes a line in the diagram $P$19 --> P7

With our full (so far) diagram markup of

flowchart
    AA7 --> $AA$1 & P7 & $P$19
    P7 --> M7 & N7 & O7
    $P$19 --> P7

The resulting diagram is

Now we can more clearly see the hierarchy of our Excel formulas.

Using this method I created a flowchart of all the formulas for one row of the “spreadsheet that is a requirement”. Here is what the full markup looks like:

flowchart BT
    H6 --> F6 & G6
    I6 --> AT6
    J6 --> H6 & I6
    KG --> AU6
    P6 --> M6 & N6 & O6
    T6 --> R6 & S6
    AA6 --> P6
    AA6 --> P19_SUM_P6
        P19_SUM_P6 --> P6
    AB6 --> Y6 & Y19_SUM_Y6 & W6 & W19_SUM_W6 & T6 & T19_SUM_T6 & V6 & V19_SUM_V6
        Y19_SUM_Y6 --> Y19
        W19_SUM_W6 --> W6
        T19_SUM_T6 --> T6
        V19_SUM_V6 --> V6
    AC6 --> P6 & T6 & V6 & W6 & Y6 & AA6
    AD6 --> P6 & T6 & V6 & W6 & Y6 & AB6
    AE6 --> AL6
    AF7 --> AL6
    AJ6 --> AG6
    AK6 --> AH6
    AL6 --> AJ6 & AK6
    AM6 --> P6
    AN6 --> AD6 & AM6
    AO6 --> AN6 & AL6
    AP6 --> AO6 & AO19_SUM_AO6
        AO19_SUM_AO6 --> AO6
    AQ6 --> AO6 & AP6 & AN6
    AR6 --> AN6 & H6
    AS6 --> AQ6 & AQ19_SUM_AQ19
    AT6 --> AS6 & AQ6
        AQ19_SUM_AQ19 --> AQ6
    AU6 --> F6 & J6
    AV6 --> J6 & H6
    AW6 --> AU6
    AX6 --> AV6 & AW6

It’s big.

I did change the direction of the flowchart by adding BT after the flowchart definition. This stands for Bottom-to-Top. Other directions supported are LR and RL in addition to TD the default for Top-Down.

Here is the resulting diagram:

Is your mind blown yet?

This diagram clearly shows me the conceptual layers of the SQL View I’ll be creating. I could now see which columns I needed to have ready for
each following layer.

At initial count I see 18 layers. I count by horizontal groupings. This a naive count of layers as many of the smaller functions can be in-lined into a sibling layer. In practice the very large functions that include checks for divide by zero were not rolled together with other layers and... the summing columns could not be rolled in without some correlated-sub-query mess. It was much cleaner to have these as separate layers.

The Summary

Had I not used Mermaid to help me diagram the requirements, it would have taken me multiple times longer than it did with using the diagram to assist in understanding the chain of data formulas.

That I don’t need to manually adjust the positions of nodes or connect and draw lines myself, saved me a ton of time. Being able to save the
markup to a text file for later editing or adding to this very same blog-post... priceless (insert appropriate credit-card commercial here💳).

I plan on using Mermaid diagrams in this manner in the future to better capture the true nature of requirements.

I strongly encourage you to check out the Live Editor and Documentation to learn more about how you can use Mermaid diagrams.

Of note, in this task we used just ONE type of Mermaid diagram out of many... and more are being added. Mermaid is open-source and being constantly improved on.

Here are the diagram types that Mermaid has as of this writing:

Flowchart
Sequence Diagram
Class Diagram
State Diagram
Entity Relationship Diagram
User Journey
Gantt
Pie Chart
Gitgraph (Git) Diagram
C4C Diagram (Context) Diagram
Mindmaps
Timeline

shift+Insert, when did this become my goto for paste

Ryan Brown — Wed, 19 Jan 2022 16:10:53 +0000

My computing history (where a clipboard existed) began on old school Macintosh's. For past from clipboard ⌘+v was how it was, ctl+v on Windows when that system became my usual OS.
Dabbling in Unix and Linux necessitated the learning of shift+insert (and middle click in most window managers.)

I still use Windows the most due to work and gaming, but usually have a terminal open to a Linux system for various uses. I'm glad that many windows interfaces support the shift+insert variant for pasting copied or cut content.

What do you use most now? Did you change your default paste command? What triggered that change?

Photon OS & Octopus Tentacles: Installation and Setup

Ryan Brown — Sun, 07 Feb 2021 16:12:50 +0000

I've updated some things with recent developments:

Switch off VMWare
Switch off Octopus

What is it?

Photon OS is a purpose built Linux flavor by VMware specially tuned for running in a VM and hosting container products such as Docker containers.
Octopus is a Continuous Delivery platform that is great for managing deployments and run-books via it's agents called Tentacles - Octopus has a strong commitment to their naming metaphors 🐙.

But Why?

Octopus Deploy simplifies and allows repeatable, stable self documenting, institutionalized deployments and release management. Marrying the Strengths of Octopus with Docker deployments seems a natural Win-Win. Also I wanted to create, for myself, an easy to deploy-to, stable Docker environment to experiment and play with & learn Docker.

Introduction

Both at work and at home I use VMware's ESXi vSphere for hosting and managing virtual machines. At work we've started using Docker containers for some projects and have Photon OS running these containers. We also use Octopus Deploy to manage our CD infrastructure. Our at-work setup uses the SSH model for communication to the Photon OS, this is because Octopus Tentacles (Octopus's name for their software agents) are not supported on Photon OS at this time. However, I'm a little determined and doggedly set out to find out if we could run an Octopus Tentacle as a Listening Tentacle on Photon OS.

Assumptions

First let's establish some assumptions:

We have a functioning VMware ESXi Host
- ~~There is a $0 licensing tier, vSphere Hypervisor ESXi~~Use Proxmox instead
We've installed a base Photon OS OVA downloaded from VMware and it starts as expected. Photon OS
We've done the tdnf distro-sync to bring that Photon OS Host up-to-date
We have an Octopus Deploy on-premises or cloud instance running and we have the required permissions to add machines in the infrastructure config.
- If you've not used Octopus Deploy, they offer a $0 pricing tier for their Cloud Instance. I use this to manage my personal servers and laugh a little each time they send me a $0 invoice. Octopus Deploy Pricing *Octopus Deploy has recently (edit date is Aug 8 2024) let me know that they have discontinued their free tier pricing entirely due to costs and bla, bla, bla, even on their non-cloud plans; further they have altered their pricing to be focused by project and effectively priced themselves out of my consideration for personal and professional use except within a very small subset of deployment scenarios that I'm unlikely to encounter in the wild. I No longer recommend octopus, and am currently investigating alternatives, like shell scripts and so on. *
You have SSH or other terminal access into the Photon OS host.
I did a bad thing and performed all these actions as root, being too lazy at this time to set up a proper user account added to the sudo group. For a production environment, and even non-production, security best practices should be followed! Don't do exactly as I have done! 😉 I googled it for you

Optional: Naive attempt: Following the instructions

Skip to things that work
I joke about this being the naive way to do it, but in most cases following this listed instructions is the best way to get a handled on how it should work... and where your setup doesn't match expectations.

According to the documentation we should be able to use the RPM install method (Photon OS's package manager tdnf uses RPM packages):

$ wget https://rpm.octopus.com/tentacle.repo -O /etc/yum.repos.d/tentacle.repo
$ yum install tentacle

results in

-bash: wget: command not found

So we can see that Photon OS doesn't come canned with some of the common tools we expect in a base Linux install, but we can get them.

$ tdnf install wget

Installing:
wget                     x86_64       1.20.3-1.ph3     photon-updates   2.80M 2936976

Total installed size:   2.80M 2936976
Is this ok [y/N]: y

Downloading:
wget                                   1128116   100%
Testing transaction
Running transaction
Installing/Updating: wget-1.20.3-1.ph3.x86_64

Complete!

Ok, Let's try that original again... one line at a time

$ wget https://rpm.octopus.com/tentacle.repo -O /etc/yum.repos.d/tentacle.repo
--2021-01-27 15:39:45--  https://rpm.octopus.com/tentacle.repo
Resolving rpm.octopus.com... 104.20.11.14, 104.20.10.14, 2606:4700:10::6814:a0e, ...
Connecting to rpm.octopus.com|104.20.11.14|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 156 [application/octet-stream]
Saving to: ‘/etc/yum.repos.d/tentacle.repo’

/etc/yum.repos.d/te 100%[===================>]     156  --.-KB/s    in 0s

2021-01-27 15:39:46 (34.0 MB/s) - ‘/etc/yum.repos.d/tentacle.repo’ saved [156/156]

That worked and now we have the octopus tentacle repo added. We should be able to install next.

$ yum install tentacle
Refreshing metadata for: 'Octopus Tentacle'
Error: 403 when downloading https://rpm.octopus.com/x86_64//repodata/repomd.xml
. Please check repo url.
Error(1622) : Invalid argument
Error: Failed to synchronize cache for repo 'Octopus Tentacle' from 'https://rpm.octopus.com/x86_64/'
Disabling Repo: 'Octopus Tentacle'
Error(1011) : No matching packages

Oops, the tentacle repo seems to not work right. Looks like the URL is malformed (note the double / in there.) We will need to modify that repository entry.

$ vi /etc/yum.repos.d/tentacle.repo

and change the line:

baseurl=https://rpm.octopus.com/$basearch/

baseurl=https://rpm.octopus.com/$basearch

Removing the trailing slash should allow it to work right.

$ yum install tentacle
Refreshing metadata for: 'Octopus Tentacle'
1. nothing provides openssl-libs needed by tentacle-6.0.390-1.x86_64
Found 1 problem(s) while resolving
Error(1301) : Solv general runtime error

Well this one is a problem. OpenSSL is installed in the base build of Photon OS, but nothing allows OpenSSL-Libs to be installed.

I can tell you from my experience, as best as I know, installing the Octopus Tentacle via RPM is a dead end for now. I've added an issue to the Photon OS' GitHub issue list requesting the OpenSSL-Libs package be included.

What we learned

We learned:

Photon OS comes out of the box, very, very lean and thus needs the following packages installed from the Photon Repositories.
- wget - to get things from the web
I'm not clear on which party is at fault, but adding the Tentacle Repository resulted in the malformed URL being tried. We needed to modify the repo URL manually to remove the extra slash.
Photon OS's package repositories don't include a OpenSSL-Libs package to satisfy the Tentacle's RPM Dependencies.

Next Steps

Lets skip some of the rest of the trial and error...

~~Install OpenSSL from source~~ This was a false result!
- I had originally believed this was the case, but in the process of writing this post, I tried the steps and couldn't replicate the results. Somewhere along the line I did something I didn't document that was the real cause of success. I started over and, with some more diligent tracking, discovered what really works.
Setup remaining dependencies
Use the Archive method from the documentation

Non-Optional: Things that actually work

OK, if you didn't skip the optional stuff, you know that Photon OS is very lean and doesn't come with some quality of life things like wget preinstalled. We also learned there is some packages we would expect to be able to install but just aren't available on the provided repositories. OpenSSL-libs for example.

Actual Steps that allow the Octopus Tentacle to function on Photon OS

install libstdc++ tar sudo zlib-devel and icu packages
create/copy a lsb_release script

lsb_release

The Octopus Tentacle calls lsb_release and uses the output during its Health Checks. Thus if it is missing the Health Check will result in a warning. Photon OS doesn't provide lsb_release. I looked into lsb_release on Arch Linux (my current go-to flavor) and found it to be a script. I copied it from an Arch Linux install and it seems to satisfy the Octopus Tentacles.
I'll leave this step as an exercise to the reader. I couldn't find a good way to provide the lsb_release script that I used. I literally copied the text of the script into a new /usr/bin/lsb_release file, then set its executable flag.
I've not looked into better alternatives. I'm excited to see the recommendations of people with more knowledge in this area!

Installing & Configuring the Tentacle

The Linux Octopus Tentacle requires a few things to get started so let's get those installed.

$ RUN tdnf -y install libstdc++ tar sudo zlib-devel icu

Note: the libstdc++ was needed when I was testing this in with the photon Docker image, but appears to be already installed in VMware's downloadable VMs.

Next let's get the Tentacle archive installer (again from the documentation, also modified to be more succinct):

$ cd ~
$ wget https://octopus.com/downloads/latest/Linux_x64TarGz/OctopusTentacle -O tentacle-linux_x64.tar.gz

$ mkdir -p /opt/octopus
$ tar xvzf tentacle-linux_x64.tar.gz -C /opt/octopus

At this point, the instructions at Octopus tell you to execute configure-tentacle.sh but that doesn't work right away as it doesn't have the executable flag set.

$ chmod +x /opt/octopus/tentacle/configure-tentacle.sh

Great, now we can configure the tentacle service

$ /opt/octopus/tentacle/configure-tentacle.sh

This will ask you several questions related to configuring the Octopus tentacle. If you've not already, now is the time to get into your Octopus Instance and start getting ready there...

Octopus and Tentacles

Octopus Deploy acts through agents called Tentacles. The Octopus server instance, either on-premises or their Cloud offering can do Polling Tentacles that reach out periodically to the server to see if there are any instructions, Listening Tentacles that wait for a communication from the server as needed and SSH targets that communicated over SSH on demand. We'll be configuring our tentacle as a Listening Tentacle. If your network has a firewall and/or router between the Octopus Server and the Tentacle, you will likely need to add some port forwarding or other firewall rules to allow communication to work as expected.

Octopus Tentacles by default use the TCP port 10933 to communicate but it configurable and often required. I use those ports plus others to communicate between Octopus Cloud and my tentacles though my firewall, each tentacle has an assigned port number and paired port forwarding at the router.

Let's now start configuring the Tentacle in Octopus,
In your Octopus Dashboard, under the Infrastructure Deployment Targets area, there is a big, handy button for Add Development Target.

This launches a short wizard that walks you through registering the Tentacle.
Select the Linux and Listening Tentacle options:

The next screen has many options that instruct Octopus how to connect to and communicate with your Tentacle:

The Octopus Server Thumbprint, used as part of the secure communication between Server and Tentacle. You will use this code when configuring the Tentacle.
The DNS Hostname or IP Address of the server. If you are behind a router, this will be your router IP/Hostname or the machine hostname if on the same network as the server. I use a dynamic DNS setup.
The Port number to be used for communication. This will need to be the port as available to the Octopus Server if you've configured a port change during port forwarding setup.
Proxy configuration. Hopefully you will know if you need to configure the proxy and be able to get that information from your network admin.
The Next button will initiate Discovery (trying to talk to the tentacle, so just hold tight)

Configure the tentacle

If you've not yet started the config utility for your tentacle, run

$ /opt/octopus/tentacle/configure-tentacle.sh

Your first prompt will be

Name of Tentacle instance (default Tentacle):

This will give you a chance to give the Tentacle instance a specific name if you choose. I'm using the default.
Next prompt:

What kind of Tentacle would you like to configure: 1) Listening or 2) Polling (default 1):

In this case, we are using a Listening Tentacle so the default is fine.
The next two options relate to log files and artifact deployment locations. Again I used the defaults.

Where would you like Tentacle to store log files? (/etc/octopus):
Where would you like Tentacle to install applications to? (/home/Octopus/Applications):

The next prompt is the Port number to listen on. Be sure this matches up with the Port number as assigned in Octopus OR the port number on the private side of your port forwarding configuration. Today I'm using port 10939:

Enter the port that this Tentacle will listen on (10933):10939

Next is the very important Octopus Thumbprint mentioned earlier. If possible copy-paste this to prevent errors:

Enter the thumbprint of the Octopus Server: 25------------------------------------EC

Then a note will be displayed informing you of the configurations that will be run. As instructed, press enter to continue!

Hopefully when all is done, you will see this message:

Tentacle instance 'Tentacle' is now installed

One thing we have yet to do is allow port access at the OS level. This will be done with an addition to iptables

$ iptables -A INPUT -p tcp --dport 10939 -j ACCEPT
$ iptables-save

Conclusion

At first glance, running an Octopus Tentacle on Photon OS is not easily accomplished. However after a few false starts, based on some faulty assumptions, I did find a fairly simple path to success. They key missing pieces are the zlib-devel and icu packages along with a functioning lsb_release script.
A properly functioning Octopus Deploy Target provides may benefits such as Machine Health alerts and monitoring. Being able to orchestrate your Docker container activities with Octopus has smoothed out several of my own personal project's infrastructure considerations.

Recent Updates (as of 2022-01-06)

Recently, it seems Octopus's method of tentacle self update has changed to try to use the host's package managers. Unfortunately this isn't method still doesn't work well on Photon OS. I had to re-perform the download and tar extraction steps and restart the service with systemctl. I don't yet know if this was a one-time or every-time change to behaviour but I will add an update here after I find out.

DISCUSS: What scale is your CI/CD and Which tools/platforms do you use?

Ryan Brown — Thu, 04 Feb 2021 18:18:27 +0000

The company I work for is looking to migrate CI/CD tool and platforms. We have a large number of individually deployable assets built from (300+) of various kinds. We are NOT a software company and most of the projects are tiny.

I'm curious about other people's setups:

What kind of organization is it (Mostly is your core business making software or technology products or support the core business)
About how many projects/pipelines etc does your org maintain
Which platforms you use, why and does it work well at the scale you operate

Thanks, and I'll start!

I work for a resource extraction company and we use technology to support the core business.
We have about 350 "active" projects that are deployed to production, most common types are web apps, database projects (scripts etc), reporting and Business Intelligence assets along with a smattering of think-apps and other more exotic items.
Currently we use TeamCity and Octopus Deploy to great effect and have great tie-ins with our tracking system of Atlassian JIRA

Looking forward to hear from you!