Managing Server Access, SSH Keys and our solution!

After seeing the complexities of managing SSH keys within a growing team and all the new legislation with GDPR, myself and good friend Rick set about seeing if we could solve the issue. Both our places of work suffer from the pain of managing team members SSH keys, both with slightly different reasons from a smaller team but 100% remote and no actual central storage of SSH keys to a larger office based team but with lots of servers which rely on server deployment scripts being individually updated and run every time you want to add or remove someone's access.

While the SSH Keys stored in the deployment scripts is a step up than nothing at all, there is a script required for each server so to modify the keys you need to add or remove the keys on each server and then wait while the rebuild scripts run, which is not ideal - we all have better things to be doing!

We decided early in the planning process that the central API, which would store the ssh keys, should not need access to the server nor should it know anything about the server other than a ‘server name’ label which is to be used within the GUI when granting / revoking access. We also wanted the option to choose which accounts on the server would be managed by this central ssh key api. This narrowed down our options for executing this idea to an agent that would run on the server and call to the api with the unique key for that account, allowing us to return the correct SSH keys.

We started testing our concept by simply placing a text file on a test domain to replicate an api endpoint, we knew we could built up a CRUD based GUI to manage the servers, accounts and SSH Keys no problem but our main concern was being able to find the right combination of cron jobs, server permissions and distribution methods for the agent. Given our knowledge of PHP and the Laravel framework we decided to give Laravel Zero a go (Laravel Zero is a console framework, built using Laravel components), we could easily build out the console commands needed to setup the agents config on the server, contact the api and write to the correct authorized_keys file to set the allowed SSH Keys on the server.

After some trial and error we got everything working, however we ran into one snag, mainly from us not reading documentation correctly, and not fully understanding what people use their servers for. We'd used Zero's built in compiler (which uses box under the hood), which creates a PHAR archive. This can be executed as a binary on Linux systems. However as we quickly discovered, would require a compatible php version to be installed. This wasn't ideal as we’d need to require, or install a specific php version. This was a bit of a dead end for us. Whilst we're both PHP developers clearly not everyone out there is using, or would want php installed on their servers.

Given what we wanted to accomplish and the requirements around it, we realised that PHP wasn’t going to cut it and that we’d need to actually look at a different language. After further research we settled on Go as the language to use for the agent as it would allow us to both provide pre-made binaries for all major Linux distributions (along with the source code for anyone wanting to manually compile it).

So, with that we drew straws, which resulted in Rick taking on the task of learning Go and creating the agent, leaving myself to build out the api and application gui.

Building the Agent

With the division of work decided, Rick set about following the Go docs and decided to make use of Cobra, an awesome package for building CLI based applications. This sped up development time significantly, and allows for building out the required functionality with minimal fuss.

The agent ended up being pretty simple. It supports adding a new system account, removing system accounts and then syncing the ssh keys for all accounts. A config file is created and contains the api keys needed to run each of the sync commands for the servers system accounts. The sync command can then be set up to run as a cron task at frequent intervals.

This probably sounds insanely basic, but this was the intention. We don't want to know the ip addresses or location of the servers connected to ServerAuth, so at no point are any server details required, we just ask for a generic name (although this is more for your own reference than ours). The agent sits on your server, and connects to us instead of the other way around. More on this later.

Building the API & Web App

I obviously drew the task of creating the GUI / API. As we both knew Laravel, it was the obvious choice to use this as in the future we would both be able to make updates if necessary. Before starting the agent, Rick also put together the ui design. Not only did this make my life easier, but was super quick to do as it was built using TailwindCSS!

Early on I decided to avoid doing anything fancy with Javascript and decided to keep it simple and functional. We can always roll out more bells and whistles at a later date.

For the GUI, we settled on having a user register and create a team, this person can then invite other people to create an account and join. Each user would then be able to manage their SSH Keys. If a user had the correct permissions, they would then be able to “create a server”, this simply creates a row in our database with a unique key and will then provide the user with the code to run on their server to setup and configure the agent.

Once the server was configured, the user would then be able to manage that server within the GUI and assign other users from their team to individual accounts on the server. This would allow you to give team leaders access to the root user for example, but only give standard team members access to the users on the server that they need access to!

Then obviously came the actual API endpoint, this required nothing complicated other than receiving unique keys for the team, server, server account and then returning all the SSH Keys for the users assigned to that server account. This would require multiple calls to the API per server for each account managed by the API but it would make things easier to handle and secure rather than a single API call to retrieve SSH Keys for every account on the server.

From proof of concept to working product was pretty rapid. By making use of open source packages, and modern development techniques we've been able to rapidly take a proof of concept to workable product in just a few months.

With our built in SSH Key Syncing, Scheduled SSH Access and an array of other features, we've been able to make SSH Access Management painless and easy to manage!

We're now in the advanced testing phase, using ServerAuth in production setting and everything has been working great. We’re now wanting to open things up and get more people involved to iron out all the last few kinks in the system, and get some important feedback from others, especially if this is something that may make life easier for them too. We're keeping ServerAuth as a free service for now but will likely look at offering a couple of paid plans for larger organisations at some point down the line.

If you work somewhere that you feel would benefit from ServerAuth we'd love to work with you! We've put together some details over at https://serverauth.com but we're also on twitter if you’d like to chat (@MikeBarlow, @RickMillsUK and @ServerAuth).

Comments

[Thomas H Jones II]

I kinda lost the ball, somewhere: are you making use of SSH's AuthorizedKeysCommand capability in this to interface with your centralized key-repository?

[Mike Barlow]

Nope, the GO agent that sits on the server runs on crontab and periodically retrieves the keys from the centralized repo and writes to the authorized_keys file.