DEV Community: Rob Pearson

Lessons learned porting Octopus Server to .NET Core 3.1

Rob Pearson — Fri, 20 Mar 2020 00:04:18 +0000

With the release of Octopus 2020.1, Octopus Server now runs on .NET Core 3.1, which means it can be installed on Linux, Docker Containers, and Kubernetes. This was a significant effort, we already shared our reflections on the launch of Octopus Cloud 1.0 and why we chose Kubernetes, Linux, and .NET Core for Octopus Cloud 2.0, in this post, we share the benefits of the change and our top three lessons learned.

Benefits

Before we share the lessons learned, here are the benefits of porting Octopus Server to .NET Core:

Modern development environment, framework, and tooling.
Cross-platform support for Windows and Linux.
Access to the mature ecosystem for running Linux containers in Kubernetes.
Choice and flexibility: Our customers can choose to run Octopus on Windows or Linux, or they can use Octopus Cloud.
Octopus Cloud has gained improved performance and reduced operating costs. See the links above for concrete numbers.

This also makes our development environment more productive as our engineering teams now have the choice to develop on Windows or Linux.

Top three lessons learned

We learned a lot going through this process; however, there are three major lessons we learned.

1. Know and plan for differences between Windows and Linux

There are platform-specific differences in the implementations of .NET Core on Windows and Linux. Most of the issues were small and had easy workarounds, but we did find a few significant problems that are worth sharing.

Configuration settings and the Windows registry

To support both Windows and Linux platforms, we had to remove any Windows-specific code. Octopus Server started as a Windows product, and it followed the platform conventions and stored some configuration settings in the Windows registry, which is a problem for Linux.

Solution:

We shifted everything stored in the registry to the file system or the Octopus database. This was a simple task, but it took time and testing to get it right.

Database performance problems

The most significant problem we encountered was abysmal database performance due to the different way of handling database queries on Windows and Linux. Octopus uses Microsoft SQL Server as its data store, and we discovered a significant problem in the .NET Core SQL client library on Linux. If we have the MultipleActiveResultSets setting set to True we get exceptions and database timeouts. The GitHub issue linked above shares the full details and a simple code sample to reproduce the problem.

Solution:

Our short-term solution was to disable the MultipleActiveResultSets setting and use it very sparingly. In general, we open two connections to our database, one with this setting enabled and one with it disabled. We primarily use the disabled connection and only use the enabled one when it is required.

We have been working with Microsoft to help provide information to resolve the issue, and we hope to see a proper fix in the future.

Authentication providers

We also encountered the need to host the Octopus Server web host differently on each platform. We use HttpSys on Windows and Kestrel on Linux, and this made our authentication challenging. Octopus needs to support multiple authentication schemes, including cookies-based authentication, and the ability for users to log in and out and have multiple authentication providers enabled at once.

The core issue we hit was that HttpSys supports integrated authentication (i.e., Windows authentication), but its a binary on/off setting for every endpoint in the host. This is inflexible, and it's a change from our non-.NET Core code-base. Users could log in automatically, but they could never log out.

Note: We don't use Kestrel on Windows because it doesn't support virtual directories and we have customers that share the same port with other services. So to ensure we maintain backwards compatibility, we decided to use HttpSys for Windows only.

Solution:

We considered several options, but after going through this ASP.NET Core issue, we decided to follow the advice there and use two hosts. One standard web host and a second one to look/behave like a virtual directory off the main API site's root, i.e., /integrate-challenge, and is therefore consistent with the location in earlier versions of Octopus Server. The host only has that one route, and it initiates the login challenge using a 401 response when the user isn't already authenticated.

2. Sharpen your debugging skills for Linux and Docker

As we progressed through the .NET Core port, we also learned how to code, test, and debug problems with Windows, Windows Subsystem for Linux (WSL), Linux, and Docker. Historically, our team all developed on Windows, but this has evolved into individuals coding on Windows, Linux, and macOS, and as a result, we've learned several lessons:

Running as non-root or non-admin

We found it much easier to run all builds and test Octopus Server on Linux as non-root or non-sudo to limit permission-based errors and problems. That said, we sometimes need to run commands as root using sudo and then afterwards fix up the ownership of files that were created during those commands with chown, like sudo chown -R $user:$user ~/Octopus/MyInstance. This is a bit quick and dirty, but it does the job.

We are planning to change this in future, such that octopus runs as a user that is a member of a group, and during installation we configure the group owner of /etc/octopus to be that group.

Certificate management

Our end-to-end (E2E) test suite runs a range of tests that run against the Octopus Server listening over HTTPS (i.e., port 443). This requires us to convert and import some self-signed certificates into the local certificate store in /etc/ on a Linux box. To solve this problem, we wrote the following script:

#!/bin/bash
echo "Setup certificates"
CERTS_PATH_DEST="/usr/local/share/ca-certificates"
if [ ! -d "$CERTS_PATH_DEST" ]
then
    echo "Creating $CERTS_PATH_DEST"
    mkdir ${CERTS_PATH_DEST}
fi
MY_PATH="`dirname \"$0\"`"
CERT_PFX_FILES=(${PWD}/${MY_PATH}/../Octopus.E2ETests/Universe/*.pfx)
for CERT_PFX in "${CERT_PFX_FILES[@]}"
do
    FILE_NAME=$(basename "$CERT_PFX")
    CERT_CRT="${CERTS_PATH_DEST}/${FILE_NAME%.*}.crt"
    if [ ! -e "${CERT_CRT}" ]
    then
        echo "Converting '${CERT_PFX}' to '${CERT_CRT}'"
        openssl pkcs12 -in "${CERT_PFX}" -clcerts -nokeys -out "${CERT_CRT}" -passin pass:password
    fi
done
update-ca-certificates

Connecting to the SQL Server database

Octopus uses Microsoft SQL Server as its database, and teams generally connect to it via integrated Windows authentication on Windows Servers. This no longer works. Our solution here was to switch to user name and password-based authentication.

Further, we found our suite of end-to-end (E2E) tests runs much faster and more reliable with database connection pooling turned off. We haven't gotten to the bottom of this yet, but it's likely a platform-specific problem related to the database performance issue mentioned above.

Debug Octopus Server on Linux with Visual Studio Code

Our team writes code with a variety of tools, including the following:

Currently, the most popular is Visual Studio Code with the Remote Development extension. This extension is still in preview, but we find it works very well.

With Visual Studio Code and the Remote Development extension, we can run applications, test, and debug them as well as edit code in Linux (or a Docker container). Simply point VS Code at the folder that contains the Octopus Server code, and then it's pretty much just F5 from there. It's that simple!

3. Simplify with self-contained packages

Porting Octopus to .NET Core has allowed us to ship self-contained packages which brings multiple benefits.

Fewer dependencies: Shipping a single self-contained executable means we no longer require .NET Core to be installed on the Octopus server. The result is reduced installation requirements that make Octopus easier to install. This is a big win.
Improved supportability: In a nutshell, fewer dependencies make Octopus easier to install and support. There are fewer components and fewer things that can be accidentally changed. Shipping Docker container images for Windows and Linux (coming soon) eliminates further dependencies as even more of the dependencies are built into the containers.
Modern software and tooling: Using modern tools and frameworks enables our team to continue to innovate and ship software quickly with useful features for our customers.

Unfortunately, this also has some tradeoffs as moving to .NET Core 3.1 required us to drop support for older operating systems, including Windows Server 2008-2012 and some Linux distros. Supporting older servers and browsers drains our time and attention, making it harder for us to innovate and move the Octopus ecosystem forward.

Conclusion

We ported Octopus Server to .NET Core 3.1 so that the server can run on Linux, Docker Containers, and Kubernetes. We made this change to reduce costs and increase the performance of our Octopus Cloud SaaS product, and it has been a great success.

It wasn't a simple journey, but we learned a lot on the way.

Know and plan for differences between Windows and Linux
Sharpen your debugging skills for Linux and Docker
Simplify with self-contained packages

MVPs and $100k AWS Bills: Reflections on the launch of Octopus Cloud 1.0

Rob Pearson — Mon, 25 Nov 2019 11:45:34 +0000

We (Octopus Deploy) are publishing a series about our engineering journey with Octopus Cloud. It’s the story of our v1 launch of Octopus Cloud on AWS, our $100K/month AWS bills, MVP’s and testing customer demand, spending 6 months of engineering effort and then running the service at a loss, spending another 9 months rebuilding it from the ground-up, and of all the considerations we made when rebuilding Octopus Cloud v2, including switching from AWS to Azure, going all-in on Kubernetes, and more.

In this first post in the series, we’ll look at the design choices we made in v1, why it cost so much, and why we decided to start over.

About a year ago we launched Octopus Cloud, a SaaS version of Octopus, as an experiment to see whether it would deliver significant value to our customers and simplify their deployments. We wanted to empower developers to focus on their deployment needs and leave managing the infrastructure to us, but we had no idea how difficult it would be to implement or how much time and money it would cost to get up and running.

Overall, we think it has been a huge success; enough for us to invest the last year rebuilding almost the entire platform from the ground up. I was a part of the team that shipped the current version of Octopus Cloud, and I wanted to take some time to celebrate a few of our wins and reflect on the lessons we learned that have shaped our redesign.

Will anybody use it?

Going into this experiment, we knew there was interest in a cloud solution, but there were a lot of things we didn’t know:

How many customers will actually use it?
What is it all going to cost?
What should we charge for it? Is it going to cover the infrastructure costs?

We could guess most of these, but knowing the answers was impossible. Engineering time is very expensive too, and it’s time we could spend adding features to our self-hosted product. We didn’t want to spend years designing a perfectly-optimized cloud-native product if there was no demand for it.

We decided to build an MVP based on our best estimates and test the market that way. The goal was to launch something in 6 months and test if the demand was there; and if it wasn’t, we’d only wasted 6 months. We chose to optimize for getting to market quickly rather than worrying about how much it would cost.

Well, the demand was there. In the first few days, we had over 500 new cloud trials spin up. As customers came to our website and decided between whether to trial Octopus self-hosted or Octopus Cloud, roughly half chose to trial Octopus in the cloud.

V1 architecture

To bring Octopus Cloud to market quickly, we did the simplest thing possible; we took our self-hosted Octopus Server product and bundled it into an EC2 instance for each customer that signed up. We had to make changes to the product, but mostly around permissions.

We actually had an internal alpha of Octopus Cloud v1 ready within a month or two; I remember the team doing bug bashes to test our security. What took longer, was all the steps required to make something we were happy for customers to use: hardening security, pentesting, planning for recovery, etc.

To ensure there was no way for one user’s data to mingle with another, each cloud instance had its own dedicated VM, database, and a large number of security configurations to prevent any funny business. Here’s a diagram of what it all looked like. Note that we actually use Octopus Deploy to provision and deploy each Octopus Cloud v1 customer:

Service limits

You know that feeling when you're driving, and you seem to hit every red traffic light? That’s kinda what launching Octopus Cloud felt like; only the red lights were AWS service limits! We quickly learned that everything in AWS has some kind of service limit, and we hit all of them. Customers would sign up, we’d hit a limit, we’d ask Amazon to increase it, we’d onboard more customers, and we’d hit another limit. Every time we thought we were in the clear, we’d hit another service limit we didn’t know about.

This caused a few issues as we scaled, and at one point, we had to pause new signups while we tried to provision more headroom.

// Detect dark theme var iframe = document.getElementById('tweet-1015048915605831680-609'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1015048915605831680&theme=dark" }

Cloud stuff can be really expensive

An EC2 instance for every customer adds up, and as our databases were backed by Amazon RDS, we were limited to 30 databases per RDS instance. Add storage, network, etc. and we were spending over $100 per month to keep a single Octopus Cloud instance online.

Octopus Cloud customers could start a free 30-day trial, which meant that those hundreds of trial signups per month, each of which cost us $100 to host, quickly added up.

We also didn’t have our pricing quite right. We initially launched Octopus Cloud with a $10/month starting tier, with a different pricing model from the one we currently use. Unfortunately, this was one of the most painful lessons we learned because the deficit between what we were charging and spending was magnified by the sheer number of people using Octopus Cloud; continued growth would further amplify the problem.

Two months in, we started having serious conversations about our $100k USD monthly AWS spend, and the fact that we had very little revenue to offset that expense:

We should explain that Octopus Deploy as a company is not publicly listed or VC funded. We’ve been proudly bootstrapped and profitable since Octopus 1.0, and we’ve always run the business conservatively and sustainably. We suddenly found ourselves with a new business that was costing us money at an alarming rate.

We decided to take the lessons we were learning and start a huge body of work we called Cloud v2; a reimagining of Octopus Cloud, built to scale sustainably.

Even when we were building v1, the team knew it wasn’t the ideal architecture. Before v1 even launched, there were plenty of conversations in our Slack about whether we should port Octopus to Linux and run it on Kubernetes, or see if we could run it on Windows within Kubernetes or use Nomad by Hashicorp? And all of this was back in early 2018 when everything was churning and not as mature as it is today. So the unit cost wasn’t really a surprise.

What was surprising was the demand. If only a few customers had signed up each month, we could have easily worn the costs (and truth be told, we still can, Octopus self-hosted has great margins!). But with so many customers signing up, it became much more urgent. $100K+ per month is $1.2M+ over the year, which is plenty to justify spending engineering effort to bringing it down.

Starting over

We explored all kinds of options to bring down our costs, and the initial plan was to iterate on the v1 architecture and look for cost savings, and we did make some progress there, but eventually, we concluded the best way to dramatically reduce our costs (without sacrificing customer performance) involved some substantial architecture changes, and essentially, starting from scratch.

In the rest of this series, we’ll go into each of the decisions we made when re-engineering Octopus Cloud for v2. These include:

Switching from AWS to Azure.
Porting Octopus Server to Linux.
Running Octopus in containers and using Kubernetes.

With Octopus Cloud v2, we’re also backing ourselves to learn and use a lot of technologies at the leading edge of hosting and orchestration. Terraform, containers, Kubernetes, and Azure Functions are just a few of the spaces we are currently working in. This approach brings its own challenges, but it will also feed into the next generation of Octopus tooling we can build with the expertise we acquire along the way. Drinking our own champagne has already improved a ton of the functionality within Octopus as we’ve become one of our own biggest customers.

Conclusion

Deciding to rebuild something from scratch is brutal, it can feel like your earlier attempt was wasted time and effort, and that work is being thrown out. It’s also very easy to go into an “MVP” project knowingly, but then become attached to it, assuming it will be the final architecture. We needed to recognize that a second step was only possible because the first step was taken, and every lesson learned along the way was essential input.

Building Octopus Cloud v2 has been very different from v1, where before we were guessing and flying blind, this time we have real data we can analyze to answer some of the questions we have: we know what users will spend, we know what things will cost, we know what sort of resource consumption to expect, and between those things we know how to make a platform that will truly scale.

This post was originally published at octopus.com.