How we Converted our Products into a Platform

Organisations which design systems […] are constrained to produce designs which are copies of the communication structures of these organisations.
— M. Conway

JFrog’s products cover a broad range of DevOps processes: from Artifactory which helps development teams manage binaries, to Mission Control which helps manage Artifactory instances, Xray which helps avoid security vulnerabilities and Distribution which helps distribute binaries in a large organisation.

Over time, Conway’s law has applied to our products. Despite our best efforts, our products had begun to diverge in their user experience, architecture, interfaces and design choices.

A chief goal of the JFrog Platform was to fix this divergence and provide a unified experience to our users. A platform comes with several benefits - this blog would be twice as long if I listed them all. Instead, you can read more about them here.

Instead, I will focus on answering the question: How do you go about converting several products into a single platform?

This was a mammoth exercise involving every team in JFrog. This blog will explain how we in JFrog RnD broke down the task.

When we considered what areas end-users interact with most in our products, the following stood out.

• User Experience
• Installation
• Configuration
• Logging
• APIs

User Experience

We’ve always had a common base library of components and styles which every UI team consumed, but now we went all the way and created a single pane of glass for all our products: A Unified UI.

To create a unified UI, we restructured our codebases.

• Every product’s UI code was extracted and merged into a single one. A thin backend layer was introduced to ensure requests were still routed to the original product. Developers still remained part of product teams: ensuring deeper domain knowledge and expertise.

• Pull Requests would now be available to be reviewed by the entire Front End Developers (FEDs) team. This process initially introduced some delays but made up for it by improving our quality and standardisation.

• Common modules (such as global search, permissions, users, groups, roles, etc.) were dropped from individual products and rewritten to be generic across products.

Installers

Our products can be installed on several different Operating Systems. Some, like Artifactory, appear monolithic and others create several microservices. Artifactory, can be configured to work with PostgreSQL, Oracle, MySQL, MS SQL or MariaDB while others need explicit databases to function (ex: Mission Control needs Elasticsearch, Xray: RabbitMQ, Distribution: Redis).

Moreover, not all users would want every product pre-bundled in the platform. They would want to be able to choose exactly what they install. This necessitated an installation process that allowed all the products to work together seamlessly while still being discrete entities.

• A dedicated team was created to develop and maintain installers.
  This team collected feedback from Support engineers, Architects and product RnD groups. Its goals were to standardise the user experience of our installers while maintaining backward compatibility.

• A single repository was created for shared code and routines.
  This allowed the team to abstract common logic and put them in a single location

• This repository is referenced in each product’s build to create the final installer. This allowed product-level customisation of the installation process where necessary.

An important outcome of this restructuring is that all our installers feel similar. For instance, every product installation variant now also contains an interactive wrapper script which asks similar questions. Another is that all our products are now installed with a standard folder structure. You can read more about this structure in Eldad’s blog

Configuration

Our products are written in several different programming languages. Java is the most common, but Go is fast becoming popular. This has meant that how we read configuration is also very different between products. Some read property files, others: XML, etc.

For the unified platform, we decided to standardise this to a single configuration file: the system.yaml.

To implement this, we needed process and code changes.

• We developed a common configuration manager library in every language we needed it in: bash, go, node, java. We did briefly consider creating a shared “configuration microservice” but dismissed it - since reading configuration is such a basic task which shouldn’t need API calls.

• We then mandated that every microservice should read configuration using only this library.

Logging

Unifying Logging was critical. Doing so ensured a consistent troubleshooting experience, easier support, better automation and easier integration with third-party log aggregators.

JFrog’s software architects came up with a proposed structure. Each team only had to change the configuration of the logger library. Or in case of languages like bash, which do not support Log4j, implement the new structure.

One of the outcomes of this exercise is that all our logs now follow a standard structure. The platform also produces a new console.log which aggregates logs from all our products’ microservices and shell scripts into a single file.

APIs

We needed to implement a consistent, and futuristic approach for APIs while also ensuring backward compatibility. We did this by retaining our individual product APIs, but adding a layer around them which would allow incremental standardisation.

If you inspect the microservices which each product now starts up, you’ll notice that one is repeated in every product: a router.

The router forms the backbone of all api-based communication between microservices within a product, between products, and between a product and the outside world. It takes care of network-level details, allowing product developers to focus solely on business logic.

• To implement this, we formed a dedicated team whose mandate was to manage, standardise, and optimise communication.
• Artefacts produced by this team are bundled in every product’s build and used as microservices.

Conclusion

Converting our products to the unified platform required a combination of organisational restructuring, architectural, design & process changes. This last year, our teams have evolved along with the product itself. The experience has been an enriching one, and one we hope will show in the product we are proud to have created.