DEV Community: IcePanel

Microservice catalogs and the best tools for the job

IcePanel — Thu, 05 Oct 2023 14:00:00 +0000

All about microservice catalogs and the top tools for an up-to-date list of your services

⚡ Tl;dr

A microservice catalog is a central repository that is the source of truth for all entities in your software architecture and the relationships between them.
Microservice catalogs give teams visibility into services so they can reuse existing functionality and ensure changes are compatible with the wider ecosystem.
Microservice catalogs store information such as name, description, owner, on-call, dependencies, runtime, packages, API specs, environment and links.

🚀 Let’s kick-off

In recent years, many engineering teams have transitioned to microservice architectures, and there's no doubt it has provided many benefits, such as stability and re-usability. As architectures reach hundreds and even thousands of microservices distributed across many teams, keeping track of what exists and who the point of contact is becomes a challenge.

This is where microservice catalogs can help; let’s jump right into it.

❓ What is a microservice catalog?

A microservice or service catalog is a central, up-to-date repository that lists your software architecture's microservices. It is the source of truth for all entities in your software system and the relationships between them.

The goal is to give teams and individuals visibility into the services available so they can properly reuse existing functionality and ensure their changes are compatible with the wider ecosystem. A simple user interface makes it easy for developers to discover the information they need to introduce new services, change existing ones and deprecate old ones.

Microservice catalogs help distribute previously tribal knowledge across teams and departments to ensure everyone is on the same page working in the ecosystem.

📦 What information should catalogs store?

Name and description to identify the service and provide a high-level summary for anyone who is not familiar with the purpose and responsibility of the service.

Owning team or individual for anyone who needs to contact the person responsible for a specific microservice so they can ask a question or request changes.

On-call team members in case an incident occurs that is caused by or affects a given microservice in which only specific team members have domain expertise.

Dependencies for visibility into other services needed to function or services that may rely on this. Seeing dependencies makes it possible to see how much impact making a change may have on other parts of the architecture.

Runtime and packages to give an overview of what technologies are being used in this microservice and can provide insights into issues caused by specific technologies.

Tags or labels allow the service to be discovered when filtering the list to a smaller subset. For example, each service could be tagged with its version numberer or risk level.

API specifications such as OpenAPI or Protocol Buffers give insight into how to integrate with the service and allow the generation of client-side libraries for communication.

Deployment status for visibility on which environments the service has been deployed into, such as the testing, staging or production environment. This helps developers know if a service is ready to be depended upon.

External links to source control, CI/CD pipelines, documentation, cloud platforms or live websites. A great resource to allow developers to quickly jump to other places where the microservice exists or is referenced.

🏢 Service catalogs for enterprise architectures

While a microservice catalog may be great for most organizations looking to keep track of their services, it may not be broad enough for larger enterprises with a complex landscape.

A typical enterprise architecture often contains legacy services, monoliths, deprecated technologies and other components that don’t fit into the bracket of a microservice. For these situations, a more flexible tool that can represent a wide range of IT assets and service types would be more suitable.

🧰 Top tools for microservice catalogs

Microservice catalog tooling is still relatively new but is becoming more popular for teams adopting microservice architectures. We’ve collated three tools to try out in your team.

1️⃣ Backstage.io

Backstage is an open-source platform from Spotify for maintaining your software catalog through source control and exposing it through a unified developer portal. Backstage supports various projects, such as microservices, libraries, data pipelines, websites, and ML models. It provides templates for quickly creating new services with standardized tooling and frameworks.

2️⃣ Cortex.io

Cortex is a service catalog for helping teams track their microservice ecosystem. Catalogs can be created through the user interface or source control using a YAML entity description. The cortex software catalog tightly integrates with its ecosystem to provide project scaffolding, team tracking and integrations with 3rd party tools.

3️⃣ OpsLevel.com

OpsLevel detects services from various 3rd party integrations and lists them in a centralized and up-to-date view. It surfaces markdown documentation in the repository to give your team instant context and details about each service. OpsLevel integrates with other services, such as incident management tools, to provide deeper insight into what is happening right now in your software architecture.

🏁 To wrap up

As organizations move to microservice architectures, catalog tools will become essential for tracking hundreds of services and other entities in the software architecture. The data queried from these tools gives teams insights into their architecture and helps them make changes safely, aligning them on common goals across services and teams.

Let us know if you’ve found other tooling for microservice catalogs that we should include on this list!

Stay chill 🧊

How to create interactive zoomable software architecture diagrams

IcePanel — Tue, 05 Sep 2023 16:23:03 +0000

Designing layered software architecture diagrams with IcePanel

⚡ Tl;dr

When using abstractions to communicate software architectures, it’s important to choose the right terminology and level of detail based on the audience.
The C4 model provides common abstractions with a hierarchical structure that allows diagrams to be tailored to the audience's technical ability.
IcePanel supports the C4 model's hierarchical diagram types, providing interactive diagrams that can be zoomed in and out of different levels of technical detail.
Simple diagrams created at a high-level view can be a superpower to help bridge communication gaps between technical and non-technical team members.

🚀 Let’s kick-off

We often use abstractions in software engineering to communicate complex architectures and software systems. In this article, we’ll discuss how abstractions are inherently hierarchical and how the C4 model provides a nested structure for defining your software architecture. We’ll then cover how IcePanel allows you to create interactive and zoomable diagrams for your audience to zoom in and out of different levels of technical detail.

🖼️ Communicate using abstractions

When discussing software, engineers almost always use abstractions to describe concepts, ideas and services at higher levels. The software itself consists of layers of patterns, services, code, and protocols, down to low-level machine code. It’s important to choose the right level of detail when talking about software architectures, which comes naturally to most software engineers.

Visual diagrams are one of the primary forms of communicating architectures in most teams, and surprisingly, most don’t consider or separate the different levels of abstraction in their diagrams. For example, a diagram with mixed abstractions might show labels containing high-level business concerns and low-level technical details about protocols. This can be confusing to non-technical audiences, so making sure the right level abstraction is being used helps answer the right questions.

Business people usually gain more value from a higher level of detail, while engineers are more concerned about low-level technical details. As engineers are usually the ones who know the systems, diagrams we often see are created at lower levels of abstractions and concepts.

Simple diagrams created at a high-level view can be a superpower to help bridge communication gaps between technical and non-technical team members.

🧅 Standardize abstractions with the C4 model

Simon Brown created the C4 model to formalize a set of common abstractions he used when drawing software architecture diagrams. Formalizing the abstractions and diagram types with the C4 model means everyone on the team uses the same language to communicate. This is a powerful tool for cross-team collaboration as it ensures everyone produces consistent and meaningful diagrams across the organization.

The C4 model essentially consists of two things:

1️⃣ A set of hierarchical abstractions:

Software systems
Containers
Components
Code

2️⃣ A set of hierarchical diagrams:

System context
Containers
Components
Code

📦 C4 model abstractions

The abstractions in the C4 model are designed to be hierarchical, making them perfect for telling a story from a high level, increasing in technical detail as you zoom further into the abstractions.

There are four abstraction types:

🧍 Person
The end user who uses the systems.

Examples: Customer, admin user or employee.

🏢 System
The highest level of abstraction delivers value to end users, whether they are human or not. Systems are often owned by a development team.

Examples: SendGrid is an external system.

📦 Container
Applications and data stores that make up a system. These are independently deployable and runnable. (Not to be confused with Docker containers)

Example: API service, front-end application or mobile app.

🧩 Component
Building blocks and modules that make up each container.

Example: Authentication module or payment card validator.

🖼️ C4 model diagrams

Each diagram in the C4 model prescribes which abstractions can be drawn at each level to ensure the high-level diagrams remain high-level.

There are four diagram levels:

1️⃣ Level 1 - System context diagram
A high-level overview of how your users interact with the internal and external systems to get value.

Audience: Architects, developers, product and business. For technical and non-technical audiences.

Scope: Persons and systems.

2️⃣ Level 2 - Container diagram
A zoomed-in view of one system that shows the running containers inside it.

Audience: Architects, developers and product. For high-level technical audiences.

Scope: Persons, systems and containers.

3️⃣ Level 3 - Component diagram
A zoomed-in view of one container that shows the components making it up.

Audience: Architects and developers. For low-level technical audiences.

Scope: Persons, systems, containers and components.

4️⃣ Level 4 - Code diagram
Usually in the form of UML class diagrams that are ideally autogenerated from the actual code. This level is rarely used as it goes into too much technical detail for most teams.

Audience: Architects and developers. For low-level technical audiences.

Scope: Persons, systems, containers and components.

🧊 Create zoomable diagrams in IcePanel

When we discovered the C4 model at IcePanel, we fell in love with how simple and effective it was at communicating software architectures to both technical and non-technical people in the team.

The hierarchical nature of the C4 model feels like it wants to be consumed using an interactive and zoomable interface so anyone can zoom in for more technical detail. So IcePanel supports hierarchical diagrams out of the box using the C4 model abstractions to navigate through different levels of detail.

You can easily create interactive and zoomable diagrams in IcePanel using the following steps:

1️⃣ Create a system abstraction in the top-level diagram:

2️⃣ Zoom into the system using the magnifying glass button in the top left corner:

3️⃣ Create an app or store (container) abstraction inside the system and continue zooming in:

The model tree shows you’ve created diagrams at two levels of abstraction, allowing your audience to zoom in and out depending on how much technical detail they need to see.

🏁 To wrap up

It’s important to consider how we use and communicate in abstractions daily. Improving the consistency of communication across teams can be a great way to get everyone on the same page and increase productivity.

Let us know if there’s anything we’ve missed!

Stay chill 🧊

Top 6 tools for text-based UML sequence diagrams

IcePanel — Wed, 16 Aug 2023 14:00:00 +0000

Comparison and features for tools that create UML sequence diagrams with code

🚀 Let’s kick-off

UML (Unified Modelling Language) consists of 14 diagram types for visualizing and communicating software architectures. Of those, sequence diagrams are one of the most commonly used in modern software engineering.

The tools listed below allow the quick and easy creation of UML sequence diagrams using declarative languages. Text-based allows them to be easily edited, checked into source control and manipulated in existing IDEs.

1️⃣ D2

D2 is a modern declarative language for designing diagrams with text. It supports a variety of different diagram types as well as traditional UML sequence diagrams. D2 is free and open-source under the Mozilla Public License 2.0 license.

Features:

Simple and clean declarative language.
Styling and theming customizations support sketch mode.
Produce animated diagrams with each step of the sequence.
Supports spans, groups, notes and self-messages.
Supports icons and images.
Export to SVG, PNG, PDF, PPTX and GIF.

2️⃣ SequenceDiagram.org

SequenceDiagrams.org is a web-based tool for easily designing sequence diagrams. The tool is free, not open source and maintained by donations from their community.

Features:

Save sequence diagrams to a file, local storage, Google Drive or OneDrive.
Supports a wide variety of arrow types and styles.
Supports icons and images.
Presentation mode for hiding the interface.
Export to SVG, PNG and URL.

3️⃣ MermaidJS

Mermaid is a Javascript-based diagramming tool for rendering many diagram types from Markdown-inspired text definitions and supports sequence diagrams. Mermaid is free and open-source under the MIT license.

Features:

GitHub will render Mermaid diagrams in your Markdown files.
Well maintained and supported by the open-source community.
Can be rendered or embedded into a Javascript page.
Supports messages, notes, alts, loops, parallel, breaks and background highlighting.
Customization and styling using CSS stylesheets.

4️⃣ WebSequenceDiagrams

WebSequenceDiagrams is a freemium web-based tool for easily writing and rendering sequence diagrams. The tool is free for the basic designing of sequence diagrams and paid for features such as revision history, branding, parallel flows and collaboration.

Features:

Variety of preset styling options to choose from.
Auto-save and revision history.
Include and reuse text from other diagrams.
REST API for plugins and extensions.
Export to PNG, PDF, SVG, text and URL.

5️⃣ StarUML

StarUML is a desktop application that runs on Windows, macOS and Linux for designing UML, SysML, sequence diagrams and various other diagram types. It requires a paid license and is not open-source.

Features:

Offline desktop app for security-conscious teams.
Supports high-resolution displays with light and dark themes.
Model-driven diagrams for quicker creation and reduced maintenance.
Export to PNG, JPEG, SVG, PDF and HTML.

6️⃣ PlantUML

PlantUML is a tool for rendering diagrams from its text-based declarative language. It supports some commonly used UML diagrams, including sequence diagrams and many other non-UML diagram types. PlantUML is free and open-source under the GNU General Public License.

Features:

Generate ASCII art for sequence diagrams.
Custom styling using CSS stylesheets.
Extensive customization of arrowheads, swimlanes, actors and icons.
Export to PNG, SVG, EPS, PDF, VDX, XMI, TXT, Latex and HTML

🏁 To wrap up

These were six commonly used tools for creating UML sequence diagrams using text. Let us know if we didn’t include any of your favourite sequence diagram tools!

Stay chill 🧊

Relational (SQL) vs non-relational (NoSQL) databases

IcePanel — Wed, 12 Jul 2023 13:00:00 +0000

A comparison of relational databases and the different types of non-relational databases

⚡ Tl;dr

Relational databases, sometimes called SQL databases, store data in tables with rows and are good at maintaining reference integrity and data consistency.
NoSQL databases are designed to have flexible schemas, making them great for modern applications where data structures evolve quickly. There are a few types of NoSQL databases, as listed below.
- Document store
- In–memory cache
- Graph database
- Search engine database
- Time series database
- Key-value database
- Wide-column database

🚀 Let’s kick-off

Relational and non-relational are the two popular types of databases that are most widely used. They are very different in their features and trade-offs and appropriate for different use cases.

In this article, we’ll discuss the differences between relational and non-relational databases and then dive into the various non-relational database types you can choose.

Relational databases

Relational databases, sometimes called SQL databases or Relational Database Management Systems (RDBMS), store data in tables with columns and rows.

The main advantage of relational databases is the integrity of references which means the data is always accurate and consistent. Using primary/foreign keys ensures that when a row is added or deleted, any references to other tables must also be valid and exist. This prevents orphaned records where the reference points to a row that does not exist, which can cause errors in your application.

Querying relational databases is done by using Structured Query Language (SQL), which is a robust language designed to create, retrieve, update and delete rows of a table.

Example relational databases:

Non-relational databases

NoSQL databases are designed to have flexible schemas, making them great for modern applications where data structures might evolve quickly. They typically offer an easy development experience due to the JSON-like format and lack of structure imposed by tables, rows and columns.

NoSQL databases scale horizontally, allowing them to operate at a much larger scale and performance than relational databases. Not worrying about the accuracy of relations makes it much easier to shard non-relational data sets.

There are various types of NoSQL databases which we’ll cover next.

Document store

Document databases store and query documents in a key-value JSON-like structure, making them easy to work with. The flexible and hierarchical nature of these documents provides straightforward mutation of the data structure as it evolves over time. Document databases typically scale horizontally, allowing them to scale to large datasets.

Examples of document stores:

In–memory cache

In-memory caches are key-value stores designed to minimize latency by storing the data in the main memory, removing the need for slower disks or SSDs. They are typically used as a caching mechanism and can persist data on disks by periodically taking memory snapshots.

Examples of in-memory cache:

Graph database

Graph databases are often the most complex type of database as they’re designed to store and query relations between objects. They store data in a graph-like structure allowing efficient querying of related and inter-connected data. Relationships between nodes are persisted in the database, making it very fast to traverse the graph.

Examples of graph databases:

Search engine database

Search engine databases are specialized databases for indexing and searching large amounts of data for similar characteristics. They are optimized for operations such as full-text search, ranking based on similarity and complex search expressions. They are often used as secondary databases, indexing data from a primary database and making it searchable.

Examples of search engine databases:

Time series database

Time series databases (TSBD) are optimized for storing sequential data collected over time, usually associated with a timestamp. They are efficient at collecting, storing and querying large and high-velocity data sets and are great for analyzing how data changes over time. Time series databases are usually append-only.

Examples of time series databases:

Key-value database

Key-value databases store a collection of key-value pairs that can hold simple or complex object types such as strings, numbers or objects. Due to its simple and optimized key-based structure, it’s highly performant and can reach scales that other database types cannot.

Examples of key-value databases:

Wide-column database

Wide-column stores are conceptually similar to relational databases but organize data by columns instead of rows. This allows them to store tables with many columns and handle massive data sets.

Examples of wide-column databases:

🏁 To wrap up

There are many factors and trade-offs involved when making a decision to choose a specific type of database. It’s essential to research and experiment with different technologies to find the right solution for your use case.

Let us know if we missed your favourite database!

Stay chill 🧊

Top 9 NoSQL databases in the cloud

IcePanel — Thu, 22 Jun 2023 16:00:00 +0000

Document, key-value and in-memory databases from the 3 major cloud providers

⚡ Tl;dr

Document databases store and query documents in a key-value JSON-like structure. Some examples:
Key-value databases store a collection of key-value pairs that can store simple or complex object types. Some examples:
In-memory caches are key-value stores designed to minimize latency by storing the data in the main memory. Some examples:

🚀 Let’s kick-off

NoSQL databases are becoming increasingly popular for modern software applications as a data storage solution for fast-evolving systems. In this article, we’ll cover the top document, key-value and in-memory database solutions from the three major cloud providers, AWS, GCP and Azure.

❓ What are NoSQL databases?

NoSQL databases are designed to have flexible semi-structured schemas, making them great for modern applications where the data structures may be evolving quickly. They typically offer an easy development experience and high performance at scale. There are various types of NoSQL databases, such as document, key-value, and in-memory.

📄 Document stores

A document database stores and queries documents in a key-value, JSON-like structure, which makes it easy to work with. The flexible and hierarchical nature of these documents provides straightforward migration of the data structure as it evolves. Document databases typically scale horizontally, making them suitable for large data sets.

1️⃣ Amazon DocumentDB

Amazon DocumentDB is a fully managed native JSON document database that is highly available and scalable.

Features:

Compatible with MongoDB APIs, drivers and tools.
Scales to millions of read/writes per second and petabytes of storage.
Storage, querying, indexing and aggregation of data.
Automatic continuous backups for a 1-35 day period.
High availability, durability and low-latency global reads.
Uses MongoDB client libraries to connect to the database.

Subscribe to the latest updates for DocumentDB on StackShare.io.

2️⃣ Google Cloud Firestore

Google Cloud Firestore is a fully managed serverless document database with high availability and scalability.

Features:

Scales to meet any demand with no maintenance.
99.999% availability guarantees with multi-region replication.
Direct connections from mobile and web apps with a client SDK.
Built-in live synchronization and offline mode with client SDK.
Built-in security and data validation rules.
Official Web, iOS, Android, Flutter, C++ and Unity client-side SDKs.
Official NodeJS, Java, Go, Ruby and PHP server-side SDKs.

Subscribe to the latest updates for Firestore on StackShare.io.

3️⃣ Azure CosmosDB

Azure CosmosDB is a distributed NoSQL or relationship database for scalable and high-performance applications.

Features:

Instant and limitless scalability with fast reads and multi-region writes.
99.999% availability for mission-critical applications.
Automatic continuous backups for either 7 or 30-day periods.
SDKs for .NET, Java, NodeJS, Python and Go.
Support for PostgreSQL, MongoDB and Apache Cassandra.

Subscribe to the latest updates for CosmosDB on StackShare.io.

🔑 Key-value databases

A key-value database stores a collection of key-value pairs that can store simple or complex object types such as strings, numbers or objects. Due to its optimizations, simplicity and key-based structure, it’s highly performant and can reach scales that other database types cannot.

4️⃣ Amazon DynamoDB

Amazon DynamoDB is a fast key-value database for high-performance applications at any scale.

Features:

Consistent single-digit millisecond performance.
Near-unlimited throughout and storage.
Multi-region replication with 99.999% availability.
Automated and continuous backups of each table for the last 35 days.
Use PartiQL for SQL-compatible query, insert, update and delete.
Import and export data directly from Amazon S3.
Compatible with NoSQL workbench.

Subscribe to the latest updates for DynamoDB on StackShare.io.

5️⃣ Google Cloud Bigtable

Google Cloud Bigtable is an enterprise-grade key-value and wide-column store for fast access to large amounts of data.

Features:

Single-digit millisecond latency with high read and write throughput.
Supports structured, semi-structured or unstructured data structures.
Compatible with Apache HBase and Cassandra with no downtime/live migrations.
99.999% availability with multi-primary replication in up to 8 regions.
Cluster scale and storage reconfiguration with no downtime.
Write data once and automatically replicate with eventual consistency.

Subscribe to the latest updates for Bigtable on StackShare.io.

6️⃣ Azure Table Storage

Azure Table Storage is a key-value store for fast development with massive semi-structured data structures and high availability.

Features:

Store petabytes of unstructured data.
Geo-redundant storage and three replicas per region.
Client libraries for .NET, Java, Android, C++, PHP, Ruby and Python.
Perform OData-based queries.

Subscribe to the latest updates for Table storage on StackShare.io.

🧠 In-memory caches

An in-memory cache is a key-value store designed to minimize latency by storing the data on the main memory, removing the need for slower disks or SSDs. They are typically used as a caching mechanism and can persist data on disks by periodically taking memory snapshots.

7️⃣ Amazon MemoryDB

Amazon MemoryDB is a Redis-compatible in-memory database for ultra-fast performance.

Features:

Extremely low latency with 160 million requests per second.
Compatible with Redis data types, parameters and commands.
Scale to hundreds of millions of requests per second.
Storage space of over one hundred terabytes per cluster.
Native JSON support and partial JSON document updates.
99.999% availability with near-instant recovery without data loss.

Subscribe to the latest updates for MemoryDB on StackShare.io.

8️⃣ Google Cloud Memorystore

Google Cloud Memorystore is a low latency, scalable and highly available in-memory database for Redis and Memcached.

Features:

Sub-millisecond data access with more than 1 million queries per second.
Fully compatible with Redis or Memcached protocols.
Supports cluster sizes up to 5 terabytes.
99.9% availability supporting up to five read replicas and automatic failover.
Supports Redis 6, 5.0, 4.0 and 3.2.

Subscribe to the latest updates for Memorystore on StackShare.io.

9️⃣ Azure Cache

Azure Cache is a distributed in-memory database for Redis with fast and scalable data access.

Features:

Handles millions of requests per second with sub-millisecond latency.
Compatible with Redis data types, parameters and commands.
Integrates with RedisBloom, RediSearch, RedisJSON and RedisTimeSeries modules.
99.999% availability with clustering, Redis on Flash and geo-replication.

Subscribe to the latest updates for Cache on StackShare.io.

🏁 To wrap up

This list only includes managed databases provided by the three major cloud platforms. There are many other clouds, managed databases and open-source alternatives that aren’t on this list that you should investigate and compare.

Stay chill 🧊

Risk storming

IcePanel — Tue, 20 Jun 2023 17:44:26 +0000

Risk storming is a technique for collaboratively identifying risks in your software architecture

⚡ Tl;dr

Risk storming is a technique for collaborating with your team to identify risks in your software architecture visually.
Risk storming is recommended as a collaborative workshop as diverse opinions and perspectives help identify the non-obvious risks.

🚀 Let’s kick-off

It’s important to realize that no software architecture, process or team is perfect, and there are always scenarios that can pose risks to your company. Considering these risks and actioning the important ones is an essential part of modern software engineering.

Risk storming is a technique developed by Simon Brown for collaborating with your team and visually identifying risks in your software architecture. Collaboration is key to risk-storming as diverse expertise and perspectives from your team help uncover the risks that may not be obvious to everyone.

🤔 Types of risks to consider

The types of risks you should consider can vary wildly, from human, to technical, so it’s important to consider a wide range of possible scenarios. Let’s run through some examples of areas where you can identify risks in your architecture.

People

What happens if a person with tribal knowledge leaves the team?
Do we have the skills in the team to maintain this moving forward?
Could we need consultants in this area, and how much would that cost?

Process

Are there important processes that people in the team aren’t trained on?
Can the support team troubleshoot and resolve questions about this area?
Are there sensitive processes or artifacts that could be compromised?

Operational

Is there a budget for this service, and could it cause runaway cloud costs?
Are there legal or compliance risks with storing data in this way?
Is this impacted by things happening in other areas of the business?

Technology

Do we depend on old technologies or libraries which could be deprecated?
Is this stable and secure enough to satisfy our commitment to our customers?
Is this technology the right tool for the job?

❗ Evaluating the priority of risks

To decide which risks are acceptable and which need to be addressed quickly, you should quantify each risk by evaluating their impact and probability.

The following matrix allows you to rank risks on a 1-9 scale and classify them into low, medium and high-priority groups. Simply multiply the probability and impact to get the final priority score.

You’ll likely want to focus on actioning the high-priority risks first and then the medium ones.

🖼️ Collaborative risk storming

Risk storming is recommended as a collaborative workshop with your team. Including a diverse set of opinions and perspectives helps identify and discuss the non-obvious risks which could go unnoticed otherwise.

You can run an in-person or remote workshop with your team with the following four steps.

Step 1. Draw a diagram of your software architecture

The first step is to gather your team members and draw diagrams of your software architecture. It’s recommended to draw multiple diagrams at different levels of detail as each diagram could have different risks associated with it.

The diagrams can include potential future state architectures that have not yet been built. Doing risk-storming upfront helps prevent these risks from being built and reduces possible re-work later.

We recommend using the C4 model as a simple structure to diagram your system and C4 model stickies as an easy way to use the C4 model with a whiteboard.

Tip: Add extra space to your diagram to allow risk sticky notes to be added afterwards.

Step 2. Individually write coloured risks on sticky notes

Each person should identify their risks and quantify them using the priority matrix above. Note the impact and probability scores on the sticky notes so others can comment. Then write the result on a green/orange/red coloured sticky note.

Each team member will have a different opinion which is why this is important to be an individual task. You don’t want to bias any opinions of the risks created.

Step 3. Add sticky notes to the diagram and discuss the risks

Ask everyone to add sticky notes to the diagram(s) wherever it makes the most sense based on the services or connections involved. Try to group sticky notes together if people create similar or matching risks, as this helps show the commonly identified risks.

Everyone should now evaluate the board and discuss the risks if needed. It’s helpful to discuss risks only reported by one person or where there is disagreement from the team about the impact or probability. You can prune or move risks on the diagrams after you have discussed them.

Step 4. Write up actions to address the high-importance risks

Review the final risks on the diagram and decide which are high priority and need resolution or mitigation. Create a plan with the team about how to mitigate these risks and assign tasks to people so they are scheduled to be worked on.

Methods to mitigate risks can be things such as running training sessions, doing implementation or refactoring work, or introducing new processes.

🏁 To wrap up

Risk storming is a powerful tool to identify and mitigate risks in your software architecture and collate different perspectives from your team. Doing it in advance can be great for building solutions that are low-risk and secure from the get-go.

Check out the official website for risk storming for more information.

Stay chill 🧊

Top 6 message queues for distributed architectures

IcePanel — Thu, 18 May 2023 14:31:02 +0000

Message queues for decoupled distributed architectures and their advantages and disadvantages

⚡ Tl;dr

Message queues can be used for asynchronous communication between services and for processing batched workloads.
As applications become decoupled, they often need mechanisms to share state, mutate data and handle events in different areas of the system.

🚀 Let’s kick-off

As architectures move toward the cloud and become increasingly distributed, the need for communication and messaging is becoming a critical part of modern software systems. As applications are decoupled into independent building blocks, they often need mechanisms to share state, mutate data and handle events in different areas of the system.

❔ What is a message queue?

Message queues can be used for asynchronous communication between two or more services. They’re great for processing large or batched workloads and help smooth out their processing. Messages are usually pushed into the queue and processed sequentially by the receiver.

Let’s explore both open-source and cloud-hosted message queues.

1️⃣ RabbitMQ

RabbitMQ is an open-source message broker for asynchronous messaging, queueing and delivery acknowledgement. It can be deployed in the cloud or on-premise with Kubernetes, Chef, Docker or Puppet.

👍 Advantages

Supports a wide range of messaging patterns and easily scales horizontally by adding more nodes to the cluster.
RabbitMQ has many official and unofficial client libraries and developer tools linked on its website.
Provides durability for its message by storing backups on disk through other nodes in the cluster.

👎 Disadvantages

Can be slow at processing larger datasets, typically handling tens of thousands of events per second.
Some integration services, such as intra-cluster compression and warm standby replication, are only available as part of their paid commercial offering.
RabbitMQ is somewhat complex and will require technical expertise to set up and maintain over time.

2️⃣ Amazon Simple Queue System (SQS)

Amazon Simple Queue System or SQS is a fully managed message queue built for distributed and serverless platforms. Out of the box, it allows you to send, store and receive any volume of messages between services. It has a free and paid tier.

👍 Advantages

Simple to set up and requires minimal maintenance over time as SQS is a managed service by Amazon.
Automatically handles high availability and redundancy by storing messages across multiple availability zones in the region.
Highly scalable in a way that’s automatically handled based on the number of messages being processed.

👎 Disadvantages

It may be difficult to estimate the cost of using SQS as you’re charged for each message based on its size and network bandwidth used.
Latency is variable based on the number of messages being sent and received but is generally less than that of self-hosted solutions.
The maximum message size is only 256KB.
SQS doesn’t support real-time data streaming or parallel processing of messages.

3️⃣ Apache ActiveMQ

Apache ActiveMQ is an open-source Java-based message queue that can be accessed by clients written in Javascript, C, C++, Python and .NET. There are two versions of ActiveMQ, the existing “classic” version and the next generation “Artemis” version, which is currently being worked on.

👍 Advantages

Lots of flexibility for customizing options such as prefetch size, threading, notification to producers and failover.
You can configure ActiveMQ to guarantee message delivery to ensure they are delivered at least once.

👎 Disadvantages

It requires manual installation, configuration and maintenance, meaning you’ll likely need someone technical to manage it.
The ActiveMQ documentation is difficult to read, understand and find the information you’re looking for.

4️⃣ Apache Kafka

Apache Kafka is an open-source, distributed event streaming platform with message communication and storage capabilities. Although Kafka is not technically a message queue, it has the functionality of a message queue using a topic partition.

👍 Advantages

Easily scales horizontally and span multiple data center and cloud environments.
Kafka is reliable, with high availability and fault tolerance through replication and partitioning.
Kafka now has libraries for most languages, including Java, C++, Python, Go, .NET, Rub and NodeJS.

👎 Disadvantages

Kafka has a steep learning curve due to its technical complexity, likely requiring in-house or external experts.
Kafka doesn’t have an official interface for management and monitoring, and third-party tools may be required.

5️⃣ Google Cloud Pub/Sub

Google Cloud Pub/Sub is a fully-managed, globally scalable and secure queue provided by Google Cloud for asynchronous processing messages. Cloud Pub/Sub has many of the same advantages and disadvantages as SQS due to also being cloud hosted. It has a free and paid tier.

👍 Advantages

High scalability and availability that requires no setup and is managed for you automatically by Google.
The maximum message payload size is 10MB, considerably higher than most other solutions.
Supports push and pull delivery types.
The easy-to-use interface on the Google Cloud console for managing and debugging topics and subscriptions.

👎 Disadvantages

It can be difficult to calculate how much you will be billed for Pub/Sub, as they charge for throughput, storage and egress as described on their length and complex pricing page.
Low latency for message delivery is not guaranteed as Google prioritizes scalability over speed for Pub/Sub.

6️⃣ Microsoft Azure Service Bus

Microsoft Azure Service Bus is a reliable, fully managed Cloud service for delivering messages via queues or topics. It has a free and paid tier.

👍 Advantages

High reliability and scalability are automatically provided as this service is fully managed by Microsoft Azure.
Azure Service Bus integrates well with other products in the Azure ecosystem.

👎 Disadvantages

The learning curve is high for new users getting started with Azure Service Bus as it can be complex to understand due to poor documentation.
Azure Service Bus has a maximum message size of 256KB.

🏁 To wrap up

Let us know if you’ve had your own experiences with any of these message queues and comment with your own experiences, pros and cons.

Stay chill 🧊

Document service events with EventCatalog

IcePanel — Wed, 03 May 2023 14:58:45 +0000

Visualize and communicate event definitions, payload schemas and service dependencies

⚡ Tl;dr

EventCatalog is an open-source project created by David Boyne for easily documenting and discovering services and events in an event-driven architecture.
EventCatalog includes a simple web interface that can be statically generated for anyone on your team to search, filter and visualize services, events and dependencies.

🚀 Let’s kick-off

Event-driven architectures are becoming increasingly popular, especially for complex service-based architectures. When working on a larger software system with potentially hundreds of services, it’s important to collaboratively discuss and design the events. Tools like event storming and event modelling are helping to facilitate these discussions.

EventCatalog aims to help further by creating a shared understanding of your events across one or more teams. It’s an open-source project created by David Boyne for easily documenting and discovering your events in event-driven architectures. It provides a markdown-style syntax that can be checked into source control and a web interface to visualize your events and dependencies.

💡 Key concepts

EventCatalog has a few key concepts to understand first.

1️⃣ Event: Messages that flow through your system. Events can have owners, schemas, producers, consumers and code examples. You can version these event properties over time as changes are made.

2️⃣ Service: Applications that produce or consume events. Services can be assigned an owner and optionally have OpenAPI or AsyncAPI specs attached.

3️⃣ Domain: Groups of events and/or services. Domains allow you to define high-level boundaries across your architecture to stay organized. Domains can be assigned an owner.

📄 Creating an event catalog

Here’s a high-level overview of the steps to get up and running.

1️⃣ Installation: Follow the installation and configuration instructions on the EventCatalog website. The docs suggest using the command line tool to quickly generate a skeleton project.

2️⃣ Create Owners: Add the names, roles and avatars for each of your owners in the user property of eventcatalog.config.js. This way, you can start assigning owners when creating your domains and services.

3️⃣ Create Markdown Files: Add files in your source control repository to describe your domains, services and events as described in the EventCatalog docs. Use the consumers and producers array to create references between your services and events.

4️⃣ Optional Event Schemas: Add a JSON schema file to each event to be more explicit with their payload structure. The schema communicates the structure with your team rather than performing validation. Read more about managing your event schemas.

5️⃣ Publish Your Catalog: EventCatalog makes it easy to build and deploy a static version of your catalog that can be hosted in a place easily accessible to your team.

🖼️ Visualize your events

EventCatalog includes a simple web interface for anyone on your team to easily discover and visualize events using search and filters. Learn about specific services to see their dependencies with other services, and quickly contact the owner of the service or domain to answer questions.

You can see a high-level overview of all the producers, consumers and events in your system architecture. Then filter down the view to a domain to visualize a smaller subset.

🏁 To wrap up

EventCatalog is a great way to help people understand, document and discover events in your software architecture. Use it to facilitate collaborative team discussions and improve the design of your services and events.

Give EventCatalog a try if you haven’t already, and let us know how it goes.

Stay chill 🧊

Subscribe to your cloud updates with StackFeed

IcePanel — Mon, 01 May 2023 16:04:29 +0000

Create custom RSS feeds with release notes for your cloud stack

⚡ Tl;dr

StackFeed.io saves AWS, GCP and Azure services in your cloud stack and provides a custom-tailored news feed of release notes and important updates.
Receive real-time messages for your service updates directly to your Slack or Teams channel with a tailored RSS feed.

🚀 Let’s kick-off

We’re excited to tell you about the motivation behind StackFeed.io, a simple website we created to stay updated with the ever-evolving world of cloud services.

As part of adding technology choices into IcePanel, we cataloged over 600 cloud services with descriptions, icons and release notes. But rather than sit on the data, we wanted to help everyone stay updated with the rapid pace of cloud for free with StackFeed. In this article, we’ll cover the problem it solves and dive into some core features.

🤯 Cloud providers release lots!

The three major cloud providers have hundreds of services for all kinds of use cases and audiences. If you regularly check their update pages it can be like drinking from a firehose. You have to scan the never-ending list of release notes and filter through them to find the services you actually care about.

There are usually anywhere from 100 to 150 updates published every week by Amazon Web Services, Google Cloud Platform and Microsoft Azure combined. This includes updates from popular services like AWS S3 and obscure services you’ve likely never heard of, such as AWS ground station. Although it’s difficult to know exactly how many services there are, we estimate there are likely around 600-700 across the three major cloud providers.

We thought there must be a better way to consume relevant updates from cloud providers.

😍 Tailored feeds with your cloud release notes

When creating StackFeed.io, our goal was to collect the overwhelming amount of release notes from the major cloud providers and allow you to create a custom-filtered news feed that’s always relevant to you.

StackFeed allows you to search through our large catalog of cloud services and save them to your stack if they’re important to your team. Once you’ve created a stack we’ll show you a tailored news feed containing only the services you care about.

StackFeed provides a tailored RSS feed for your stack that allows you to receive service updates directly into your Slack channel, teams chat or other RSS reader. Let’s go over how to set this up.

📰 Cloud release notes in your Slack or Teams channel

Subscribing to StackFeed updates in your company Slack or Teams channel is a great way to keep you and your team up to date with news about your cloud services.

First, install the official Slack RSS app into your workspace from the Slack App Directory. Choose your stack on StackFeed.io if you haven’t already, click the “Add to Slack” button and copy the feed URL provided in the text field. Then paste the feed URL into the RSS app and select a channel to post updates.

To test this without creating a stack you can use https://stackfeed.io/feed to subscribe to all cloud updates from AWS, GCP and Azure.

You may need to wait about 24 hours for a new cloud update to appear in the channel. If everything is working you should be able to see Slack updates like in the screenshot below.

The process is very similar if you use Microsoft Teams or an RSS reader app, just select the “Add to Teams” or “RSS feed” buttons on the StackFeed website.

🏁 To wrap up

We’ll soon be making this data available to users in IcePanel for selecting technology choices for apps and stores inside their architecture diagrams. We hope the data is valuable to you, and we’ll continue to maintain it over time.

If this looks interesting then give StackFeed.io a try and let us know how it went for you on our feedback form.

There’s scope to extend StackFeed further than just the three major cloud providers it supports today, so we’d love to hear any ideas you have in the comments.

Stay chill 🧊

AsyncAPI for event-driven architectures

IcePanel — Thu, 13 Apr 2023 14:32:25 +0000

An industry standard for defining asynchronous APIs for event-driven architecture

⚡ Tl;dr

AsyncAPI is a standard for describing the communication between services and applications for asynchronous event-driven architectures (EDA).
AsyncAPI specs allow you to easily generate data models, client libraries, server code and documentation in various languages to speed up the development process.

🚀 Let’s kick-off

You’ve likely heard of the Swagger OpenAPI specification, as it’s widely used in software engineering to describe how systems communicate over REST APIs. However, one of the biggest challenges with Swagger has been describing event-driven systems, as this wasn’t what it was designed to do.

AsyncAPI aims to solve this problem by making it easy to describe communication for bi-directional event-based architectures. This includes asynchronous technologies such as WebSockets, Kafka, RabbitMQ, etc.

💡 Key concepts

AsyncAPI has seven core concepts that are used when writing the definition. They align closely with the terminology and concepts you hear about when discussing event-driven architectures.

1️⃣ Server: Servers represent message brokers that transmit messages between producers and consumers. Their job is to transmit and deliver asynchronous messages between one or more clients.

2️⃣ Producer: Producers are applications that publish messages, usually based on a state change or user action. When messages are published they’re delivered to other entities that subscribe to them.

3️⃣ Consumer: A consumer is an application that subscribes to specific events from a server and reacts to them. They aren’t aware of other producers and consumers involved, only the messages they receive through subscriptions.

4️⃣ Channel: Channels are created by the server and help organize the transmission of messages. They ensure the right messages are routed to the right consumers. You can define channels as a topic, queue, routing, key, path or subject.

5️⃣ Application: An application is one or more services, devices, processes, etc. They should either be a procedure of events, a consumer of events, or both.

6️⃣ Protocol: A protocol is a set of rules for communicating between many applications and/or servers. Examples of protocols that can be used are WebSockets, gRPC, HTTP, Kafka and MQTT.

7️⃣ Message: Messages transmit data from one sender to one or more receivers through a channel and can be defined as an event, query or command. Query and commands can be thought of as instructional messages carrying information, while an event provides details of something that’s already occurred. A payload can be included in messages and is formatted as JSON, XML, binary, etc.

📄 Specification

AsyncAPI started as an adaption of OpenAPI, making most parts of the specification similar. It was created with the intention that users could interchangeably reuse elements in either specification.

The spec can be written as YAML or JSON and is usually checked into source control, either globally or per service/application exposing the interface. Below is a short example spec showing how to define an application that allows consumers to subscribe to a signed-up user.

asyncapi: 2.6.0
info:
  title: Example
  version: 0.1.0
channels:
  user/signedup:
    subscribe:
      message:
        description: An event describing that a user just signed up.
        payload:
          type: object
          additionalProperties: false
          properties:
            fullName:
              type: string
            email:
              type: string
              format: email
            age:
              type: integer
              minimum: 18

The following image from the AsyncAPI website gives a high-level overview of the structure.

🔨 Tools

Studio: Studio allows you to design, validate and visualize your AsyncAPI specs in a web browser. It includes a YAML-based editor and HTML documentation renderer.

Generator: This generates documentation, server and client code from an AsyncAPI specification. There are many official and community-supported templates, including NodeJS, Java, Python, HTML, Markdown, Typescript, Golang, .NET and PHP.

CLI: The CLI tool lets you create and validate AsyncAPI files from your command line. It also supports generating typed models for Typescript, C#, Golang, Java, Javascript, Dart, Rust and Kotlin.

GitHub Action: An official GitHub action that makes it easy to generate HTML documentation from your AsyncAPI spec. You can then deploy it to GitHub pages or similar.

Parser: Easily parse and validate AsyncAPI specs from YAML or JSON. The library also includes Typescript types for manipulating and accessing the properties of your spec.

Modelina: Automatically generate data model types from your AsyncAPI, OpenAPI or JSON schema documents into languages like Java, Typescript, C#, Go, Javascript, Dart, Rust, Python and Kotlin.

🏁 To wrap up

AsyncAPI has been listed in the early adoption phase of the InfoQ Architecture Trends Report for the last 3 years, with adoption from companies like Slack, Salesforce and SAP. They are also part of the Linux Foundation, which is a non-profit that supports leading open-source software projects and standards.

Check out the AsyncAPI site for more info.

Stay chill 🧊

Architecture decision records (ADRs)

IcePanel — Wed, 29 Mar 2023 21:21:05 +0000

ADRs document architecturally significant decisions made over time

⚡ Tl;dr

Architecture decision records (ADRs) are short text descriptions documenting architecturally significant decisions during the software development lifecycle. They capture high-level decisions that may not always be obvious from code or infrastructure. ADRs include important context for each decision, such as reasoning, consequences and tradeoffs.

ADRs can be thought of as an immutable list of decisions over time. Each time a new decision is made and accepted by the team it gets appended to the end of the timeline with a timestamp. Newly created ADRs supersede previous ones and can be deprecated when they’re no longer relevant.

If you want ADRs to be easily accessible to everyone on your team (including non-technical people) they can be stored in wiki tools such as Confluence or Notion. Checking ADRs into source control is also very powerful and provides all the benefits of Git, such as pull requests, version history and blame.

😮 The benefits of ADRs

Align teams on important decisions: ADRs can be published in a place accessible to everyone on the team, making it easy to communicate and align on important decisions.

Persist important decisions long-term: Details about decisions are easily lost when people move between teams or companies, and ADRs ensure they are persisted long-term.

Context of decisions for new team members: ADRs give new team members valuable learning about the decisions and tradeoffs that were made before they joined the team.

Document decisions that aren’t implemented: Findings from investigations explored but not implemented are just as important to document as ADRs in case they’re revisited later.

📝 How to write an ADR

ADRs can be customized to include the information you need, but there are standard templates, such as one created by Michael Nygard.

Title: A descriptive title that’s usually included in the file or document name.

Status: Current status of the ADR, for example: proposed, accepted, rejected, deprecated, superseded.

Context: The motivation or problem that lead to decisions being made and any context needed to understand it.

Decision: Details and specifics about the change being proposed.

Consequences: Any positive or negative effects this decision has on other parts of the architecture.

ADRs can also be formatted similarly to user stories, as written out below.

In the context of <use case/user story u>, facing <concern c> we decided for <option o> to achieve <quality q>, accepting <downside d>.

🔨 Tooling for ADRs

There are a few tools available for creating, viewing and managing your architecture decision records.

adr-manager - Web-based application for creating and managing ADRs using Markdown stored in your GitHub repository.

adr-viewer - Show off your ADRs on a web page, either as a local web server or generated static content.

ADR templates - Collection of example ADR templates maintained by Joel Henderson.

Loqbooq - Web and Slack-based decision logs inspired by ADRs.

Log4Brains - Log ADRs from your IDE and publish them automatically as a static website.

🏁 To wrap up

Most teams make architecture decisions verbally in meetings that are easily lost or forgotten about. Documenting these important decisions as ADRs is something your future self and teammates will be grateful for!

Check out adr.github.io for information on ADRs.

Stay chill 🧊

Reimagining UML sequence diagrams in IcePanel

IcePanel — Tue, 21 Mar 2023 15:18:28 +0000

Sequence diagrams that are understandable to everyone in your team

⚡ Tl;dr

Even though sequence diagrams are suited to visualizing low-level technical details, they are often used to communicate high-level flows with product and business stakeholders.
At IcePanel we’re focused on making sequence diagrams easily understandable to your technical and non-technical team members. This article goes through some of the design decisions we made when creating the flows feature.

🚀 Let’s kick-off

UML sequence diagrams are a popular tool for software engineers to describe the flow of messages in a software system. They’re great for showing interactions between different services or classes and the order in which they are executed. It paints a great picture of the dependencies involved in a particular process flow.

Even though sequence diagrams are better suited for visualizing complex low-level technical details, they are also used to communicate flows to higher-level people such as product and business stakeholders. As you can imagine, this can overwhelm anyone who does not deal with low-level code daily. Even the simple UML sequence diagram below could cause non-technical people to glaze over and not fully understand what they read.

IcePanel helps bridge the communication gap between product and engineering, so we felt there must be a friendly way to visualize sequence diagrams in a way that appeals to everyone on your team.

Let’s dive into how we approached this.

🖼️ Overlaying sequences on existing diagrams

Our initial consideration was the process of presenting diagrams and sequences to your team to tell a specific story. The cognitive load caused by completely switching layout and visual styling mid-presentation between regular and sequence diagrams can be distracting, especially for those unfamiliar with the UML sequence diagram notation. It requires your audience to re-read the layout of your objects and connections, which takes attention away from your ability to tell your story effectively.

We chose to overlay sequences of messages onto your existing diagrams and let you seamlessly transition into a flow during a presentation or conversation without any distractions. This also reduces maintenance as you don’t need to create separate diagrams for each sequence you need to show.

🔎 Automatic zooming that keeps the focus

Sequence diagrams can quickly become very high or wide, requiring scrolling to consume at a readable size. This makes it difficult to navigate your audience through the elements of your flow, especially if you keep having to scroll up and down to show different areas. For example, the sequence diagram below is unreadable when embedded into a word document.

Being an interactive tool, we have luxuries that are impossible for static printed or image-based sequence diagrams exports. IcePanel shows you a list of steps so you can quickly flip through them one by one, automatically re-centring and zooming the camera to bring the important elements into focus for easy readability. It even applies a slight opacity to objects that are not used in the currently focused step and a more aggressive opacity to objects and connections that are not part of the flow at all. This helps break down the complexity and make it more digestible for your audience.

👉 Showing the direction of travel

It can also be difficult to see the direction of travel for connections in traditional sequence diagrams, especially if the arrowheads are small or the lines are drawn very wide. To help here, we added an animated pulsating line to communicate the direction of travel and bring focus to the current connection. Product and business people love this as it adds a cool visual element which helps them understand the flow.

🔄 Alternate and parallel flow paths

Showing concurrent and alternate paths can be a challenging concept to show non-technical audiences and requires some basic understanding of the UML notation. It can also be difficult to see where the second or third concurrent request happens with many other steps listed in between.

Pulsating multiple connections at once easily communicates the concurrent nature of these requests in a way that’s much more obvious to everyone. A simple grouping of steps with a tab for switching between alternate flow paths allows you to easily communicate how thee paths run independently and focuses the conversation on the currently selected path.

💨 Rapid creation of flow sequences

We believe that creating and editing sequence diagrams should be insanely fast and easy, especially to help keep up with the rate of change. Product and business people often have great insights into high-level non-technical flows but may not feel comfortable creating or editing sequence diagrams, especially if generated from a markup language. We want to make it easy and quick for everyone to make changes when needed.

IcePanel already has context from the diagram about the objects being drawn and the direction of connections, so it can make helpful assumptions for you. Selecting a connection on the diagram and clicking “add step” will automatically infer the origin, receiver and connection for the flow step, leaving you to just fill in the description.

🏁 To wrap up

We understand there are cases where a UML sequence diagram is the best tool for the job, especially when the audience is only engineers or it includes low-level implementation details. There may also be advanced UML notations you need to use that aren’t supported in IcePanel, where another tool would be better suited.

IcePanel does allow flow export as code for rendering in tools like PlantUML or SequenceDiagrams.net for those that prefer visualizing in a traditional UML sequence diagrams format.

We’ve spent a lot of time making it seamless to tell both high-level and technical stories in IcePanel in a way that’s digestible and engaging to all your team. I hope you’ve enjoyed the insight into our design decisions for flows.

Stay chill 🧊