DEV Community: Mia-Platform

Advancing Developer Experience in Software Engineering: A Technical Deep Dive

Paolo — Tue, 16 Jan 2024 13:47:53 +0000

The best development tools are the ones that disappear. They make the programmer so productive that they’re unaware of the tool itself.
Kent Beck, Creator of Extreme Programming.

The developer’s toolkit has ballooned, but the developer experience (DevX) is shrinking. Fragmented platforms, tool sprawl, and disjointed workflows are squeezing the joy out of coding. While new technologies and languages promise innovative solutions, they often contribute to this complexity, leading to inefficient development practices and hindered collaboration.

This fragmentation is beyond choice overload. Developer marketplaces lack flexible application composition, private assets remain siloed within organizations, and access control remains fragmented, hindering security posture. Additionally, game-changing innovations like GitOps, despite their potential for declarative and automated infrastructure management, face adoption hurdles due to implementation complexities. CNCF reports that although 70% of survey respondents are familiar with GitOps, only 32% actively use it.

The consequence? Inefficient tools and platforms create bottlenecks in workflow integration, hinder seamless collaboration, and ultimately impede the timely delivery of innovative solutions. Developers grapple with constant context switching between disparate platforms and programming languages, extending development cycles and reducing productivity.

The good news, however, is your organization does not have to tackle this challenge alone. This article explores strategies and solutions, spotlighting how technological transformation can redefine DevX.

Deconstructing DevX: Technical Framework and Fundamentals

Building a seamless and empowering DevX requires a solid foundation of several key technical elements working in harmony. To create a truly conducive environment for developers, we must consider the following elements:

Toolchains

Toolchains are comprehensive programming tools used to perform complex software development tasks and deliver software products. Seamless interaction between these tools streamlines and automates the software development process. Toolchains like the GNU toolchain often include essential components like assemblers, linkers, debuggers, compilers, and runtime libraries.

Workflows

Workflows define the sequence of steps involved in software development, from ideation to deployment. These software development workflows are adaptable frameworks designed to be flexible and responsive to change.

Workflows readily embrace iterative development cycles, allowing for continuous feedback and integration. Understanding that no two projects are identical, workflows are tailored to specific needs, enabling teams to optimize their workflows for each unique project. We can think of workflows like living organisms constantly evolving alongside the software they create.

Environment setups

Environment setups establish a consistent and reliable development environment, minimizing setup time and ensuring compatibility across development machines, inherently promoting DevX.

The importance of environment setups has been immortalized in anecdotes over the years because of phrases like “it’s not working? It is working on my end” due to problems arising from differing development environments.

As an illustration, a software development team where developer A on a Windows machine uses Library v2.0 for a web application feature. At the same time, Developer B on a macOS system relies on Library v1.5 due to compatibility constraints. Developer A’s code leveraging feature exclusive to Library 2.0 may fail on Developer B’s environment, causing runtime errors, functionality gaps, or even complete feature breakdowns.

Environment setups solve these problems by ensuring everybody is on the same page concerning development environments. These setups may involve provisioning virtual machines, configuring cloud environments, and managing dependencies.

Architectural Paradigms

Architectural paradigms provide a foundation for designing and building software applications. They encompass patterns, practices, and principles that guide software components’ structure, organization, and communication. Architectural paradigms include Java EE, Monolith, Microservice, Serverless, Functional & Reactive, and Lambda.

These technical frameworks are pieces of a much larger DevX puzzle and, as a result, do not work or exist in isolation.

You can think of toolchains as the building blocks, providing the essential tools and technologies for building software while workflows guide the development process. Environment setups ensure everyone works with the same tools and settings, and architectural paradigms define the software’s structure and communication.

Harmony between these technical elements does form the bedrock of a conducive DevX environment. Still, to elevate the developer experience, it’s essential to incorporate a structured blend of methodologies and strategies to harness the potential of these tools and frameworks.

Enriching DevX with Methodologies and Strategies

In 2001, seventeen software practitioners came together to create the Agile Manifesto in response to the limitations of traditional development methods like Waterfall. This manifesto, which promoted iterative development, continuous communication, and delivery, birthed practices, and strategies like the DevOps movement and Platform engineering, revolutionizing how developers approach their work.

DevOps, which emphasizes collaboration and automation, bridges the gap between development and operations, empowering developers to release software faster and improve reliability.

On the other hand, Platform Engineering introduces standardized infrastructure, enabling developers to build and deploy applications seamlessly across diverse environments. Together, these approaches streamline workflows and automate repetitive and mundane tasks, empowering developers to focus on innovation and value creation.

In addition to practices like DevOps and Platform Engineering that have transformed software development and delivery, it is important to note other technological innovations that have also played significant roles in shaping the developer experience during software creation.

Technological Innovations Shaping DevX

With the emergence of DevOps and Platform engineering, the importance of developer experience in streamlining workflows and retaining talent has been highlighted. This recognition has sparked numerous technological innovations aimed at addressing various developer pain points, significantly influencing the platforms we use today. Mia-Platform, particularly its latest release, Mia‑Platform v12, stands out as a prime example of these platforms embodying these advancements.

Some of the technological innovations to take note of are:

Low-Code or No-Code Development: Low-code or no-code development platforms have transformed how developers build, making it easier to create fully functional applications with minimal code. This recent shift democratizes application development, empowering more people to participate in creating solutions. Mia-Platform v12 embraces the future of software development as it introduces its Microfrontend Composer, formerly the BackOffice. This feature leverages its newfound Low-Code functionality to enable your organization to easily compose and customize its internal applications using ready-to-use and custom libraries and templates.
GitOps deployment strategy: This approach uses Git repositories as a single source of truth for managing both application code and deployment configurations. GitOps facilitates the automated deployment and management of infrastructure, making the deployment process more reliable, reproducible, and efficient. GitOps integration offers an array of benefits to organizations, which is why contemporary platforms are expected to offer some GitOps support with its pull-based approach. Thankfully, Mia-Platform v12 with the Enhanced Project Workflow feature allows developers to choose the deployment experience during project creation. Devs can use the Pull or Push deployment strategy, based on their needs.
Artificial intelligence (AI) and Machine Learning (ML): AI and ML dominate discussions in software development circles, and rightly so. With their ability to automate tasks like debugging, predictive analysis, and even code generation, these technologies streamline workflows, offer insights, and optimize performance, leading to faster development cycles and more innovative solutions.
User-centric design: Adopting user-centric design is a universal concept beyond the walls of software engineering. This push for user-centric design in technology teaches us to prioritize users’ needs, preferences, and experiences. By placing users at the core of the design process, you ensure that solutions are intuitive, functional, and tailored to address specific user pain points. User-centric design can take various forms, such as Mia-Platform v12’s intuitive console and its capacity to recognize and deliver on organizations’ need for specialized private assets. By focusing on the user, we create products and a culture that values empathy, fueling innovation that resonates deeply with its audience.

In line with the evolving landscape, Mia-Platform v12 strongly advocates prioritizing DevX, asserting that it should never be an afterthought. It places developers at the forefront, introducing innovative features and redefining application development and deployment. Explore these exciting features in the detailed Mia-Platform v12 documentation.

While this conversation predominantly revolves around DevX in the context of Platform engineering and Internal Developer Platforms, its fundamental principles—such as prioritizing developers, automation, and standardization—transcend beyond internal platforms.

Adapting DevX Beyond Platform-Specific Implementations

DevX principles, rooted in enhancing developer productivity and satisfaction, are universally beneficial throughout any existing developer ecosystem. With transferrable techniques like continuous integration/continuous deployment (CI/CD), test-driven deployment (TDD), and more, DevX can seamlessly integrate into existing development processes, regardless of the platform or language.

Additionally, organizations can foster a developer-centric approach and use developer feedback to tailor DevX principles to their unique needs. This adaptability enables organizations to cultivate an environment where developers can thrive, increasing their productivity and software quality. Here are some recommended strategies for aligning DevX principles to your team’s needs:

Identify pain points and prioritize improvements Conduct surveys and user interviews to gain insights into your developers’ key challenges and frustrations in their day-to-day work. Prioritize working on the most impactful pain points hindering productivity and satisfaction.
Align Culture with DevX and vice versa The adoption of DevX is a moment that puts the developer at the center and involves a huge effort to evolve the corporate culture. To be successful, the process involves aligning the values, working styles, and culture of the developers to the DevX principles.
Pilot and iterate Actively involve developers throughout the DevX implementation process. Seek their feedback on proposed changes, gather their input on tool preferences, and incorporate their suggestions into the design. Continuous improvement ensures that DevX efforts align with the team’s evolving needs.

In summary

DevX is a holistic approach to enhancing the developer experience, encompassing toolchains, workflows, environment setups, and architectural paradigms. By understanding the interplay between these technical elements, methodologies, and strategies like Agile development, DevOps, and Platform engineering, organizations can create a conducive environment that empowers developers to experiment, innovate, and continuously deliver high-quality software.

Although DevX is built on universal concepts of optimization and efficiency, it is not a one-size-fits-all approach. Organizations are required to tailor their DevX principles to their specific needs and culture.

This article also explores how Mia-Platform, a leading cloud-native platform builder, addresses various developer pain points in introducing platform engineering in their ecosystem through its latest release, Mia-Platform v12. With its intuitive user interface console designs, enhanced project workflows with GitOps support, and a host of other new features, Mia-Platform v12 is revolutionizing the way developers work.

Intrigued to learn more about Mia-Platform v12 and how it can empower your organization to create and deliver high-quality software? Check out the Mia‑Platfrom v12 documentation.

Zero Trust Architecture: Principles and Practical Use Cases

Paolo — Thu, 14 Dec 2023 08:47:33 +0000

The constant progression of technology brings a drawback in the form of increased cyber threats. In recent years, there has been a surge in the rate of cybercrimes and threats to various organizations, as in the cases of LastPass and Twilio.

These attacks come in various forms: phishing attacks, malware, email attacks, IoT, DDoS, and other attack vectors. In this landscape, enterprises need solutions that enhance security and provide robust protection against the constantly evolving internet threats.

The zero trust architecture (ZTA) is a security approach based on the principle that “trust is good, control is better”. It assumes the need to treat every access request as potentially dangerous and conduct a thorough check before granting access, regardless of the requester’s identity or location. It’s a concept that emphasizes proactive, granular, and dynamic approaches to safeguarding data and resources.

This article dives deep into the following:

The three pillars of ZTA and its principles;
The advantages and disadvantages of migrating to a ZTA;
Some practical use cases of ZTAs and how organizations can leverage them to fortify their defenses.

What is a ZTA?

The term “zero trust” refers to the cybersecurity concept that guides the development of defense strategies that deviate from the static, network-based perimeters and instead focus more on users, assets, services, and resources. ZTA assumes an organization should never grant implicit trust and should prioritize securing resource access irrespective of network location, subject, or asset.

The principle behind zero trust is that “every device, user, and application is untrusted”. It is an end-to-end approach to securing enterprise networks, resources, and data—it encompasses identity, credentials, operations, endpoints, access management, hosting environments, and interconnecting infrastructure.

This security concept safeguards modern digital businesses, including DevOps, robotic process automation (RPA), public and private clouds, and SaaS apps. Companies like Microsoft, AWS, and Google have created ZTA frameworks and solutions, which is a testament to its popularity.

What is zero trust network access (ZTNA)?

Zero trust network access (ZTNA) is a critical component of the zero trust model, emphasizing a context-aware and proactive approach to remote access management. It offers centralized management and flexibility for IT and security teams to achieve an effective zero trust model based on defined access control policies.

ZTNA is often implemented through various technologies and solutions, including software-defined perimeter (SDP) solutions and secure access service edge (SASE). It utilizes identity-based authentication to establish trust and distributes access to internal applications while concealing physical network data like IP addresses.

Limitations of traditional security models

The traditional security model, often known as the perimeter-based approach, is no match for the threats within our dynamic and ever-evolving environments for the following reasons:

It relies heavily on the idea that everything within the network perimeter is trustworthy and uses perimeter defenses, like firewalls and virtual private networks (VPNs), to keep threats out.
Authorization is checked only at the perimeter, and an attacker can avoid the authorization perimeter through software bugs.
Since this authorization applies only to one layer, once an attacker enters a network, they can move laterally through it until they come across a target.

The limits of traditional perimeter security are more evident as cloud-based assets and remote work become prevalent, increasing the proliferation of security blind spots. As a result, attackers can easily infiltrate a business network using traditional security models.

According to IBM, there was a 15% rise in the worldwide cost of data breaches in 2023. Meanwhile, the Identity Security Threat Landscape (ISTL) report states that 71% of organizations have suffered from confidential data loss through employees, ex-employees, and third-party vendors.

Why is a zero trust strategy important?

Organizations increasingly align their security policies with business goals rather than leaving cybersecurity as an afterthought. Adopting the zero trust approach can boost business flexibility and enhance overall security, thus preventing reputational harm, financial losses, and regulatory penalties.

Moreover, ZTAs employ rich intelligence and analytics technologies like security information management, advanced security analytics platforms, and user behavior analytics. Thus, security experts leverage it to observe real-time network activities for more intelligent defense strategies.

How does zero trust work?

The ZTA concept handles security tasks by combining various security controls and processes. These security processes include identity and access management (IAM), risk-based multi-factor authentication (MFA), next-generation firewall (NGFW), next-generation endpoint security, end-to-end encryption, and robust cloud workload technology. Together, they authenticate and verify the health of assets and endpoints before authorizing access requests for resources or a more comprehensive network.

The ZTA divides security into multiple layers. It eliminates pre-authorized access and enforces specific user access controls at a highly granular level. Thus, authentication and authorization (subject and device) are performed before a connection on a corporate network is established.

ZTA requires organizations to enforce risk-based access controls, continuously inspecting, monitoring, and logging interactions to identify and verify user and systems access. Effectively implementing ZTA necessitates comprehensive monitoring and verification of users, traffic, and application identity attributes, including those encrypted across different environment segments.

Foundations of the zero trust architecture

The ZTA is built upon foundational pillars and principles designed to fortify security in modern technologies.

Pillars of zero trust

Zero trust operates on the principle of “never trust, always verify”. According to the NIST, organizations should develop a zero-trust strategy based on these guiding principles applied across the six core pillars.

Applications;
Data;
Endpoints;
Identities;
Infrastructure;
Network.

Principles of zero trust

Every organization has its needs in terms of security depending on the size, industry, existing infrastructure, and risk profile. As a result, there is no one-size-fits-all approach to implementing ZTA in an organization. The following core principles create the foundation for a ZTA.

Multi-Factor Authentication (MFA): MFA is a principle of zero trust that enhances security by requiring users to provide multiple forms of verification before gaining access to a system or application. It often involves presenting two or more authentication factors like PIN, security questions, email verification, text messages, biometric ID checks, etc. Before authorization is given, each verification step must be confirmed.
Micro-segmentation: Micro-segmentation is breaking up security perimeters into small logical units and applying policies to control access to data and applications within these segments. This limits lateral movement for attackers even if they gain access to a part of the network without separate authorization. Security teams use this to determine how applications share data, regulate data transfer limits between servers and applications, and implement authentication processes for specific interactions.
Least privilege access controls: ZTAs enforce least-privileged access controls, establishing trust based on context. Using role-based access control (RBAC), companies can authorize and restrict system access to users, devices, and applications based on their roles. So, all network traffic is prohibited by default, permitting only authorized connections. Strict authentication is mandatory before access is granted, even from familiar individuals.
Device Access Control: Zero trust requires strict rules on device access, as it does for users. Identity-centric controls are extended to the device endpoints to continuously verify devices, which means organizations can ensure that all endpoints accessing corporate resources are initially enrolled according to system requirements.

Pros and cons of zero trust

If you are considering implementing ZTA, remember that, like any other technological approach, it has pros and cons.

Pros	Cons
ZTAs provide robust protection against security breaches	ZTAs aren’t entirely immune to insider attacks
Decrease attack surface	Requires more time and effort to set up
Reduce susceptibility to insider attacks	Complex implementation
Limit the impact of a breach	Increased cost considerations
Provide increased visibility	More applications, devices, and users to monitor and manage
Offer improved data protection	Introduces more data to protect

Best practices for implementing zero trust

ZTAs involve securing your endpoints, adopting the principle of least privilege, and harnessing the power of artificial intelligence, machine learning, and automation. This undoubtedly demands meticulous planning and adherence to best practices to guarantee its effectiveness. Here are four core best practices for implementing a Zero Trust strategy.

Re-evaluate legacy investments: When implementing a zero trust strategy, assess legacy systems, processes, and tools to ensure that they align with the principles of this security approach by reviewing and enhancing data protection measures. To centralize and streamline user identity controls, ensure existing systems seamlessly integrate with solid identity management solutions, such as identity and access management (IAM) platforms.
Always assume breach: Consider designing a network with a zero trust mindset, assuming that threats may already exist within the network. This will help detect anomalies and automatically revoke access once malicious activity is identified.
Continuous monitoring: While implementing a zero trust strategy, consider proactive approaches to preventing data loss, like continuous monitoring and authentication. Even if a breach occurs, real-time monitoring can help identify and limit the ‘blast radius’ without sacrificing user experience.
Strong authentication mechanisms: Enforce robust authentication mechanisms, including implementing MFA. This will give a system an additional layer of protection, ensuring that only authorized users with verified identities gain access to sensitive resources.

Practical use cases of the ZTA

As more employees work remotely and organizations embrace cloud solutions, more attack vectors are inevitably exposed. Organizations can leverage the zero trust security model to apply consistent security policies across all environments, maintain a unified security strategy, and protect against potential threats.

Historically, businesses seeking remote access have often relied on traditional VPN solutions. Alternatively, the ZTA leverages secure web gateways and SASE to securely provide access for any user, from any device to any corporate resource, regardless of where it’s hosted, in the cloud or on-premises.

Google has pioneered the implementation of zero trust security through its framework “BeyondCorp“. Essentially, BeyondCorp establishes detailed access control rules for Google Cloud Platform and Google G Suite, considering variables such as IP address, device security status, and user identification.
Financial institutions handle sensitive customer data and are attractive targets for cyberattacks. For instance, Wells Fargo has proactively invested in a zero-trust security model to protect its network and customer data. “One of the key tenets of a ‘zero trust’ framework is to assess the risk exposure of the organization, which starts with privileged access”, said Sridhar Sidhu, senior vice president and head of the enterprise security services group at Wells Fargo.

How to Implement ZTA with Mia‑Platform

Mia-Platform can be instrumental in simplifying the implementation of a ZTA. It can help your organization in the following ways.

Connecting the product to your identity provider of choice to enable authentication and role-based authorization.
Allows you to configure your infrastructure by providing robust and reliable authentication and authorization mechanisms that control access to your microservices and APIs.
You can also standardize how your microservices are created, with security rules and access checks already in place.
In cases of many users, you can create groups to manage access to Mia-Platform Console and create Service Accounts to enable M2M authentication to the microservices and APIs.
Through Mia-Platform’s open-source Rönd project, you can create access policies and distribute security policy enforcement throughout your application. It also allows you to build an RBAC or attribute-based access control (ABAC) solution by defining roles, permissions, and users.

If you want to learn more about ZTAs and how RBAC operates, this white paper about RBAC is a great place to start.

Summing up

Unlike the traditional security model, zero trust suggests that organizations should not automatically trust any user, device, or system, even if they are inside the corporate perimeter. Though the zero trust security model is a complex and continuous interactive process, it can seamlessly be integrated into existing architectures; organizations need not remove their existing infrastructure. Implementing a zero-trust strategy goes beyond combining technological solutions and robust policies to eliminate potential threats and breaches. It involves a cultural shift toward a security-first mindset to meet the realities of today’s networks, workforces, and threats.

Shift Down to the Platform, not Left to the Dev

Paolo — Mon, 23 Oct 2023 06:31:00 +0000

In the ever-evolving world of software development, a new approach to developing and delivering software, called shift down, has emerged to challenge conventional methods.

The concept of shifting down to the platform instead of shifting left to the developer brings about a paradigm shift that reimagines how we optimize our software delivery process. Richard Seroter's "The Modernization Imperative: Shifting left is for suckers. Shift down instead" is a great article that emphasizes the relevance and urgency of the shift down philosophy.

Platform engineering is a key discipline in this new era of DevOps teams shifting down to the platform. It involves designing and creating streamlined workflows and tools that make it easier for software teams to build and deploy their applications.

By embracing this new change, organizations can gift their development teams the tools to work to ensure that software is delivered with better quality and speed.

This article explores the shift left and right approaches to building and testing software, their limitations, and how shifting down helps bridge the gaps and provide a more comprehensive strategy for modern software development.

What is shift left?

The "shift left" concept coined by Larry Smith in 2001 changed how we approached software development. It involves moving the testing phase to the beginning of the software development journey. This shift represents a proactive and strategic approach where the primary focus is to test the software at its early stages and repeat the process frequently.

The significance of this approach lies in its ability to detect and address defects before they can permeate the entire application.

Although shift left's emphasis on early testing works, it does not fully capture real-world usage scenarios and production environments. This problem highlights the importance of incorporating shift-right testing.

Understanding the traditional shift left V Model
The shift-model is a software development lifecycle (SDLC) model where the execution of processes happens sequentially in a V-shape. It is also known as Verification and Validation model.

It is a shift left approach based on the association of a testing phase to each corresponding development phase. The V-Model is an extension of the Waterfall model, as the next phase only begins after the completion of the previous phase.

Using the V-Model has its advantages. It is a highly disciplined approach, as one phase starts when the previous one has ended. It is also easy to use and understand. However, its resemblance to the Waterfall Model also brings forth some of the limitations associated with the model:

For larger projects, the V-Model's structured and interconnected phases might become cumbersome for testing teams to manage, leading to coordination challenges and increased complexity.
Customer involvement is typically more concentrated toward the V-Model's later stages of the development process resulting in reduced opportunities for early feedback and potential misalignment with customer expectations.
The model does not inherently support rapid changes or iterative development.

The need for more advanced shift left testing methodologies, such as incremental and model-based approaches
With modern applications becoming more complex, there is a need for more advanced types of shift left testing approaches, such as incremental testing and model-based testing approaches.

Incremental testing approach
The incremental shift left testing approach is a software development process that promotes early and frequent testing of individual components or units during development. This approach works best with complex systems and helps mitigate the risks of issues snowballing into more complex problems during later phases.

Model-based testing approach
The model-based shift left approach testing helps to mitigate defects in requirements definition, architecture, and design phases. This approach tests executable requirements, architecture, and design models to eliminate 45-65 percent of errors introduced in these early phases.

The model-based approach is the newest method in shift left testing, and its major advantage is that it works in the earlier stages of the project's development cycle.

What is the shift right model?

The "shift right" concept introduces a dynamic shift in how we approach software testing and quality assurance. Unlike traditional methods like shift left focusing solely on pre-production testing, "shift right" advocates extending the testing phase into the live production environment. This approach captures real-world user experience by conducting rigorous assessments under genuine conditions.

Shift right involves deploying our application into the real world and observing how our application responds to users, data, and the environment. By subjecting our application to authentic user loads and actual usage scenarios, shift right allows us to gain valuable insights into how our application behaves, how responsive it is, and how well it stands up to the demands of the users.

Here are some of the salient benefits of shift right testing:

Identifying the actual workload of customer traffic and ensuring that the software can handle precise user demands and requirements in a live production environment.
Identifying user preferences through A/B testing or Blue-Green deployments.
Detecting and resolving potential problems in the production environment before they escalate and impact a more extensive user base.

Balancing Shift Left and Shift Right: A Holistic Approach to Application Security

Although the shift left approach served its purpose over the years, overemphasizing shift-left methodologies can present certain drawbacks. It does not encompass the complexities of real-world scenarios and production environments. This gap in testing could strain the development team and shift their attention away from essential tasks to address these issues during later stages.

This section discusses the potential security issues that may arise from a heavy focus on shift left practices and the importance of shift right testing coupled with shift left testing in minimizing risks and ensuring robust application security.

Security concerns from a heavy focus on shift left testing
Shift left security testing does not deliver the full context to secure infrastructure but rather a tiny piece of the puzzle to deepen security across modern applications.

Shift left works really well for developmental testing. However, it loses value rather quickly when applied to applications and technologies already in use within an environment, so the shift right testing approach will always be important. Another example of the inefficiency with shift left testing is the case of APIs. The shift left approach might need to thoroughly assess the potential vulnerabilities in API authentication and authorization mechanisms, leaving the API vulnerable to unauthorized access, data breaches, and malicious attacks.

A comprehensive operational testing strategy that includes shift right practices is crucial to mitigate these security risks.

Importance of balancing shift left and shift right to address security concerns effectively
The over-reliance on just the shift left testing makes us miss certain security vulnerabilities that only become apparent in real-world usage; shift right, on the other hand, extends testing into the production environment.

In essence, shift left testing brings development and testing together by including development testing in the software development cycle. On the other hand, shift-right testing encompasses operational testing. Security teams can establish a strong security framework by combining these testing approaches.

The Application of Shift Left and Shift Right in Microservices Architecture

Microservices' modularity and interoperability enable frequent delivery of large and complex applications, which challenges the shift left testing approach for many reasons:

Microservices often communicate through APIs, making their interactions intricate. Shift left testing might struggle to simulate the dynamic and interconnected nature of these services accurately.
Isolating individual microservices for testing can be challenging. Shift left practices might need to effectively address issues that arise when multiple services interact, resulting in undetected problems in the overall system.
Microservices architecture can involve many services, making comprehensive testing resource-intensive.
Shift left might miss security vulnerabilities specific to microservices, such as inadequate data validation between services, token propagation issues, or improper access controls.

Importance of incorporating shift right tests to ensure comprehensive monitoring and feedback
In microservices, shift right, and shift left testing approaches are vital tools for comprehensive monitoring and feedback. Shift right allows us to evaluate the entire ecosystem of different microservices under genuine conditions, providing valuable insights into how these services interact with each other, scale, and respond to user loads.

The shift right approach enables continuous improvement through iterative feedback loops. This iterative approach allows us to improve the customer experience with our application based on user feedback and real-world data usage patterns.

What is the shifting down approach?

The shift down approach leverages existing platforms and empowers less technical experienced people to solve more issues earlier in the process. With the shift down approach, we reduce the cognitive load on the software developers by taking full advantage of the available technology., and pushing more workloads down the platforms they are already using.

The shift down approach in development aims to improve efficiency and reduce cost by freeing senior developers to focus on more complex and innovative tasks. We can observe the different examples of shift down approaches in different scenarios, including:

Empowering Tier 1 support engineers to troubleshoot and resolve issues without escalating to more senior engineers.
Self-service tools and documentation allow users to resolve their problems without contacting support.

Advantages of compressing the tech stack and reducing cognitive load on developers
Compressing the tech stack involves strategically choosing and using a smaller set of technologies, frameworks, and tools in development. This approach aims to reduce the complexity of managing multiple dependencies and interactions, thereby streamlining development for software developers.

Compressing the tech stack and reducing the cognitive load on developers can offer several advantages, some of which are:

Reducing the number of technologies and frameworks that a developer needs to work with
With a smaller set of technologies and tools, developers can quickly get up to speed and start working on the project.
Reduced cognitive load means that developers have enough time and resources to work on new projects and achieve business goals instead of struggling to understand new complex tools.

Shifting Down in the Software Delivery Cycle

With the shifting down approach moving responsibility to lower teams and platforms, it helps to promote smoother collaboration, quicker issue resolution, and improved software delivery efficiency.

Platform engineering is a discipline that shares shift down's goal of improving efficiency in software delivery. Platform engineering is designing and building toolchains and workflows that enable self-service capabilities for software engineering in the cloud-native era.

Alongside enabling self-service capabilities, there are other benefits of embracing platform engineering across different stages of the software delivery cycle:

Platform engineering enhances the Developer Experience by streamlining processes and offering a user-friendly environment.
Platform engineering helps improve efficiency and accelerate development in the software delivery pipeline by automating tasks and providing standardized tools and processes;
The standardized processes, also known as golden paths, ensure uniformity across stages, therefore, reducing errors and enhancing quality;
Platform engineering can lead to lower costs and more efficient use of resources.

Importance of collaboration and communication in implementing platform engineering practices
To effectively implement platform engineering, we need a cross-functional effort that requires collaboration and communication between different teams. This collaboration ensures that everyone understands and works towards shared goals.

Communication bridges gaps between teams and encourages constant feedback, shared insights, and best practices, enhancing continuous improvement, learning, and problem-solving.

Challenges and Mitigation Strategies for Shifting Down

This article discussed in detail why shifting down to platform or less experienced team members is the right move for our organization; however, shifting down can be challenging for many reasons.

Firstly, shifting down requires a significant cultural shift in the organization, as teams must be willing to give up a little more control and work collaboratively. Secondly, there needs to be a significant investment in tooling, infrastructure, and training for less experienced team members to support the shifting down approach.

Tips for overcoming the challenges posed by the shifting down approach
To navigate the challenges in shifting down to the platform, organizations can adopt a number of practical strategies, some of which include:

Start with a small project and team, then gradually expand the approach. Shifting down the entire project at once is exhausting.
Invest in proper tooling and infrastructure that can support the shift down approach. Embrace automation and easy-to-use self-service tools that can empower developers to adapt and thrive in platform engineering
Build a strong team of platform engineers that have a deep knowledge of shifting down to platforms

Benefits of shifting down to the platform instead of shifting left
Shifting down to the platform offers significant long-lasting benefits, and these benefits include:

Increased efficiency in the software delivery process: Shifting down to the platform helps to free up resources and experienced engineers to focus on more complex tasks to streamline development, testing, and operations processes. Shifting down to platform leads to faster issue resolution and faster time to market
Improved agility: Shifting down to the platform makes it easier to deploy and debug new components and features, thereby improving the overall code quality
Reduced costs: Shifting down can reduce costs by centralizing infrastructure and resources, leading to lower IT costs and more efficient use of resources

Conclusion

Shifting down to the platform offers a transformative approach that optimizes resource utilization to fast-track and enhances organizations' software delivery pipeline process. Organizations traditionally employed the familiar shift-left strategy in software development, but its limitations have spurred the shift down approach.

In the world of platform engineering, organizations can leverage existing platforms and tools to design, build, and deploy robust toolchains that offer standardized processes, automation, and simplified deployments transcending intricate technical complexities.

Embracing the shift down philosophy, hand in hand with platform engineering, ensures not only efficient utilization of resources and expertise but also an ecosystem where continuous learning, collaboration, and innovation flourish.

Leading platform builders like Mia-Platform empower organizations to seamlessly integrate platform engineering into their software development and deployment cycles.

Download this white paper to understand more about Why and how to evolve into a Platform Company and the advantages of shifting down.

This article was written by Michel Murabito, Developer Advocate at Mia‑Platform.

Building and Reusing Packaged Business Capabilities (PBC)

Paolo — Fri, 22 Sep 2023 16:08:03 +0000

One of the main goals of leading IT figures is to try to improve the productivity of the IT department they manage. One way to achieve this goal, and one that is becoming increasingly popular among organizations of all industries and sizes, is to leverage composable and reusable elements. In addition to optimizing productivity and speeding up the development process, this solution also ensures better governance.

These reusable components, known as packaged business capabilities (PBCs), belong to a specific type of packaged application that encapsulates distinct and self-contained business functionalities. By embracing this modular approach, organizations can rapidly assemble and deploy digital capabilities without the burden of time-consuming development from the ground up.

When implemented effectively, PBCs hold immense potential for organizations worldwide. By the end of 2023, around 30% of new applications are projected to be built using PBCs, a remarkable increase compared to the mere 5% recorded in 2020. PBCs' modular components enable organizations to streamline development processes, reducing time and costs.

This article discusses PBCs and their remarkable capacity to revolutionize building and deploying software. Taking a close look at its characteristics, applications, and value, we discuss how PBCs address the limitations posed by traditional architectures.

What are PBCs?

Packaged business capabilities are standardized, pre-packaged software components representing specific business capabilities and functions.

PBCs are self-contained modules consisting of a data schema and a collection of services, APIs, and event channels. PBCs are functionally complete and usually have all the necessary resources to operate independently without relying on external components. This autonomy is achieved by minimizing critical external dependencies and eliminating the need for direct access to its data.

Are microservices the same as PBCs?

Microservices (or microservices architecture) are a cloud-native architectural approach in which a single application comprises many loosely coupled and smaller deployable independent components or services. Although microservices and PBCs promote composability and reusability in software development, they are not the same.

The key difference in principle between microservices and PBCs is that microservices are an architectural style that defines how we break down applications into services. These services can communicate through APIs; each can be developed, deployed, and scaled independently. However, PBCs are custom combinations of certain microservices that work together to carry out a specific business function.

To better grasp the distinction, consider this example: In the microservice approach, a microservice might be responsible for handling user registration, another for login, and a third one for managing user profiles. However, a PBC might take our entire user authentication flow.

Microservices, although valuable, can be difficult to handle for the average business user, as they are complex and challenging to maintain.

Redefining Limitations: Monolithic Architectures vs. PBCs

For years, monolithic architectures have been the bedrock of organizations. Although they serve their purpose, when these architectures are confronted with the demands of the digital age, they reveal their vulnerabilities and crumble.

Some of the challenges that plague monolithic applications are listed below.

Rigidity: Unlike PBCs that utilize modular interchangeable components, traditional monolithic architectures are effectively a single unit, making them resistant to change and innovation, especially as size increases.
Scalability challenges: Scaling individual components within a monolithic application can be challenging. It is inefficient and prone to resource waste, hindering organizations from effectively meeting fluctuating demands.
Inhibiting innovation: The tightly coupled and intertwined nature of components in monolithic architectures limits experimentation and innovation.

Applications of PBCs

Practical applications of PBCs are found across different industries and business teams. Some of the key applications include the following cases.

Supply chain management (SCM) systems
The SCM system is a company's software system used to manage the entire flow of goods and services, incorporating all processes in transforming raw materials into finished products. The integration of PBCs in this process helps to optimize this whole flow, as it may include capabilities like demand forecasting, inventory optimization, logistics management, supplier collaboration, etc.

E-commerce and online marketplaces
Online retailers and marketplaces leverage PBCs to handle payment processing, inventory management, order fulfillment, and customer support. PBCs allow them to focus on their core business without investing heavily in developing these functionalities from scratch.

Financial services
Financial industries can harness the power of PBCs for risk management, fraud detection, trading platforms, and more. These capabilities allow organizations to build robust and secure applications without the need to build everything from the ground up.

Approaches to building and reusing PBCs

Organizations can select from various approaches when constructing PBCs tailored to their unique goals and needs. These approaches involve strategies for integrating and leveraging third-party solutions and resources. In this section, we explore these approaches, discuss their considerations, and highlight the benefits they provide.

Standalone implementation approach
The standalone implementation approach involves deploying PBCs as separate, independent systems within an organization's infrastructure, and some of the benefits of this approach include the following.

Isolation and flexibility: With PBCs operating separately and independently, the risk of widespread failures is reduced. Isolation can reduce or eliminate the need for external dependencies and thus increase system flexibility.
Customization: Organizations can tailor each PBCs to their requirements to care for specific needs.
Scalability: Organizations can scale these components independently to optimize performance and resource allocation.

Legacy modernization approach
Legacy modernization involves replacing outdated systems and traditional workflows with modern technologies. This approach can leverage existing PBCs to replace and upgrade functionality within a more extensive legacy system.

System enhancement approach
Organizations use this approach to enhance existing systems by incorporating PBCs. This approach promotes reusability and efficiency in the following ways.

Maximizing investments: With PBCs, organizations can extend the value of their existing systems to fulfill new requirements, minimizing the need for extensive redevelopment.
Reducing redundancy: PBCs streamline development efforts by utilizing pre-built functionalities. This reduces redundancy and ensures consistency.
Accelerating time-to-market: By leveraging existing PBC solutions, organizations can accelerate development cycles, allowing them to create new features and capabilities to market quickly.

The value of PBCs

Using a unifying API, PBCs stitch various services together to streamline workflows; this gives organizations one central interface to work with.

Unlike microservices, PBCs allow businesses to deal with fewer building blocks, thereby streamlining operations. They enable companies to swiftly adopt digital platforms, paving the way for efficient processes and seamless customer experiences.

The versatility of PBCs grants businesses the ability to pivot and adapt rapidly to changing circumstances. With PBCs, organizations can easily customize and expand their offerings, exploring new avenues for innovation and growth.

Composability and PBCs

Composable thinking revolves around building systems and solutions using modular and interchangeable components. This philosophy aligns perfectly with the teachings of PBCs.

Fueled by PBCs, composable business architectures capitalize on pre-built capabilities that organizations can quickly assemble to form solutions that cater to specific business needs. This approach minimizes development time, reduces complexity, and enhances adaptability, enabling businesses to embrace technological advancements with agility.

Advantages of composable architecture
The Composable Architecture proposes a set of principles, patterns, and associated technologies that support composable enterprise where businesses adopt in-house and third-party components that are pluggable, replaceable, scalable, and independently deployed and improved.

An API-first approach within a composable architecture allows individual components of a system to interact and communicate effectively using well-defined user interfaces. The composable business architecture leverages the power of APIs, giving businesses the ability to seamlessly integrate, modify, or replace components without disrupting the entire ecosystem. This combination of modularity and interoperability fosters a responsive, scalable, and efficient business environment perfectly aligned with the dynamic demands of today's market.

Essentially, composable thinking with PBCs creates a synergy that gifts businesses with digital transformation and propels them into a future of boundless innovation and change.

PBCs and innovation

PBCs allow businesses to offload more noncore functionalities to specialized vendors or partners, freeing up resources and energy organizations can use to enhance their core competencies, differentiating factors, and creating new products and services. PBCs also provide a sandbox for experimentation. Organizations can easily integrate PBCs to test new ideas and features, moving forward with ideas they deem fit and discarding the rest without taking apart the entire system.

PBCs in response to changing business demands
Volatility, uncertainty, complexity, and ambiguity (VUCA), a term coined in 1987, has been used repeatedly to characterize the modern business landscape. Organizations need the flexibility and adaptability that PBCs offer to navigate these challenges.

PBCs are designed to be modular and scalable; organizations can scale up or down specific capabilities as the market demands without significant development cycles, reducing both time and cost. Organizations can leverage PBC's modular superpower to minimize the risk of widespread failures, thereby gifting systems resilience.

The added value of application composition and the role of PBCs in driving innovation
Application composition is the process of assembling applications from individual PBCs, also known as Composable Applications. It is pivotal in driving innovation as organizations can compose unique applications tailored to specific customer needs by combining various PBCs.

PBCs facilitate rapid prototyping and experimentation, which allows organizations to quickly assemble and test new ideas, features, and business models, enabling them to validate concepts and innovate more efficiently.

Conclusion

Modern organizations are contending with business changes in a world characterized by dynamic market shifts and ever-evolving customer expectations. In the business domain, this series of innovations is driven by the evolution from monolithic architectures to composable architectures or composable enterprises in general.

At the heart of composable applications are PBCs. As discussed, PBCs are more than composable components — they’re building blocks for reimagined ecosystems. PBCs’ modular architecture empowers enterprises to redefine agility and accelerate innovation, propelling organizations to fast-track their time to market and seamlessly adapt to the evolving business landscape.

From standalone implementations that embrace flexibility to legacy modernization that breathes new life into aging systems, the versatility of PBC approaches is a testament to their transformative potential. By embracing PBCs, organizations maximize the value of their investments and foster an environment of operational efficiency.

Download the free white paper "Composable Enterprise. Why composability is the future" to better understand PBC's role in a composable future.

Why your DevOps Toolchain Needs a Governance Platform

Paolo — Thu, 31 Aug 2023 08:29:00 +0000

Implementing an efficient DevOps toolchain has become indispensable for organizations striving to achieve agility, scalability, and accelerated time-to-market. However, as DevOps environments grow in complexity, maintaining control, compliance, and security across the entire software delivery lifecycle becomes increasingly challenging. Hence the need for a governance platform.

This article discusses the evolving landscape of DevOps toolchains and how governance can help manage the chaos they can cause. We will also answer the popular question, “Does a governance platform come at the expense of developer velocity?” After reading this article, you’ll come away with the four key features you need in a governance platform.

The evolving landscape of DevOps toolchains

In the past, a typical DevOps toolchain consisted of a relatively small set of tools focused on specific DevOps tasks such as version control, build automation, and deployment. However, with the adoption of cloud computing, microservices, containerization, and other technological advancements like platform engineering, the complexity of toolchains significantly increased.

Today's toolchains encompass many tools for application delivery, including continuous integration (CI) servers, configuration management tools, infrastructure provisioning tools, container orchestration platforms, monitoring and logging systems, etc. Although the expanded toolchain ecosystem brings immense potential, it also introduces new challenges for organizations.

Challenges when scaling your DevOps toolchain

One of the key challenges in a growing DevOps toolchain is managing the growing number of tools and their interdependencies. With multiple DevOps teams using different tools, ensuring compatibility and seamless integration becomes complex. Also, maintaining visibility into the entire toolchain and managing access controls across different tools become challenging.

Another challenge is the lack of standardized practices and configurations across the toolchain. Inconsistencies in tool configurations can lead to issues in deployment — as well as increased cognitive load, security vulnerabilities, and performance bottlenecks. Ensuring adherence to best practices and compliance standards becomes critical but difficult in a decentralized toolchain environment. Organizations are recognizing the need for a governance platform to address these challenges.

A governance platform is a centralized solution that provides visibility, control, and standardization across the DevOps toolchain ecosystem. It is a single source of truth, offering a comprehensive view of all tools, configurations, and integrations within the toolchain. It enables organizations to track tools' usage, health, and performance, facilitating effective management and troubleshooting.

Does a governance platform come at the expense of developer velocity?

Seeing the “centralized” nature of a governance platform, developers ideally would start to think about its effects on their application development velocity. The relationship between governance and developer velocity is not inherently negative. Still, it can vary depending on how an organization implements and enforces governance.

Strict and rigid governance can sometimes create bottlenecks and decelerate the development process. If you burden developers with excessive bureaucracy, lengthy approval DevOps processes, or a lack of autonomy, their velocity and efficiency can indeed be impeded. However, an effective governance platform can also facilitate developer velocity in several ways, for example:

Shift-left testing: Shift-left testing is an approach that involves testing early and continuously throughout the development process rather than waiting until the end. A governance platform can incorporate automated testing tools and practices, encouraging developers to perform testing during development. This tactic catches bugs and issues early on, reducing the time spent on bug-fixing in later stages of development and improving overall velocity.
Reducing technical debt: Technical debt can significantly hinder a development team's velocity. A governance platform can enforce guidelines and code quality standards that encourage developers to address technical debt as they go. By investing time and effort in code refactoring and maintenance enforced by the governance platform, technical debt is reduced, making the codebase more maintainable and allowing for faster development in the future.
Compliance and risk mitigation: Adhering to regulatory requirements is crucial for certain industries. Governance ensures that developers follow compliance guidelines, reducing the risk of legal issues and costly penalties. Organizations should adopt a governance platform that supports developers and encourages productivity while ensuring compliance, security, and quality to strike the right balance.

Four key features you need in a DevOps governance platform

By adopting the right governance platform, businesses can strike the perfect balance between speed, control, and security — thus achieving DevOps excellence while safeguarding their software development processes.

To help you make an informed decision, the four essential features below form the foundation of a successful DevOps governance platform.

Platform engineering

The best feature you need in a governance platform is platform engineering. Platform engineering is the process of designing and building toolchains based on best practices, industry standards, and the experiences of software engineers in the organization. You deliver these toolchains through an integrated product called an “internal developer platform”.

An internal developer platform enables developer self-service. Developer self-service is an approach that empowers individual developers to access and utilize tools and services without depending on external assistance or approvals. This increases developer productivity.

Creating your DevOps toolchain through platform engineering paves golden paths that reduce cognitive load and standardize your infrastructure operations and software delivery process.

Policy management

Effective policy management is crucial for enforcing the desired standards and best practices across a DevOps workflow. Your ideal governance platform should allow you to define, customize, and enforce policies for security, compliance, code quality, and deployment processes.

The first State of Cloud Native Policy Management report by Nirmata, the creators of the Cloud Native Computing Foundation (CNCF) project Kyverno, revealed that nearly 50% of cloud native production environments had adopted some level of policy management. Though adoption has come with its challenges, the benefits greatly outweigh them.

Typically, you write policies in a human-readable language like YAML (Yet Another Markup Language) or JSON (JavaScript Object Notation). However, your preferred governance platform should also allow you to write policy-as-code: write policies with high-level programming languages like Java, Python, etc. Policy-as-code helps with version control and improves visibility, as stakeholders can easily see policy changes.

Aside from the ability to write policy-as-code, your ideal governance platform should enable you to test those policies before pushing them to production. The image below shows how Mia-platform’s policy editor helps developers write and test policies side by side.

Seamless integration with tools across the DevOps ecosystem

The ability to integrate with any tool is a crucial and powerful feature that a DevOps governance platform should possess. DevOps production environments typically involve diverse tools catering to various aspects of the development and its entire lifecycle. A governance platform with strong integration capabilities empowers DevOps teams to choose best-in-class solutions for version control, CI/CD, testing, monitoring, and more, fostering toolchain flexibility.

By centralizing visibility and promoting seamless collaboration, product teams can exchange information and make data-driven decisions, ensuring consistency and standardization across the entire development lifecycle.

Identity and access management (IAM)

IAM is more than just security. It plays a vital role in enhancing efficiency and ensuring compliance. IAM is a fundamental and critical feature that should be integrated into a DevOps governance platform.

IAM ensures that the right individuals have the appropriate level of access to resources, DevOps tools, and sensitive data within the DevOps environment. This feature is key to maintaining security, compliance, and accountability throughout the development and deployment process.

Your governance platform should enable assigning specific roles and permissions to different identity types when defining responsibilities and privileges within your organization and projects. RBAC is one of the techniques commonly used within IAM systems to define and enforce access policies. To learn more about it, download this free white paper on RBAC implementation in Platform Engineering.

Conclusion

Without proper governance, a DevOps toolchain can quickly run wild, leading to compatibility issues, security vulnerabilities, and inefficient workflows. Software engineering organizations must recognize the importance of taming their toolchains and implementing governance with platform engineering tools like Mia-Platform Console to bring order and efficiency to DevOps practices.

Mia-Platform Console can help you to govern all projects in one place as well as industrialize and automate DevOps to avoid organizational bottlenecks. It also offers self-service capabilities to reduce cloud infrastructure complexity and enhance accountability and reliability engineering. To see Mia-Platform in action, request a free demo now.

Pave Golden Paths with Platform Engineering

Paolo — Thu, 03 Aug 2023 07:28:49 +0000

Every organization constantly seeks ways to streamline its operations, enhance developer productivity, and deliver exceptional developer experiences. One crucial factor that can make or break these endeavors is the efficiency of their technological infrastructure. To make their technology stack efficient, many organizations have adopted Platform Engineering.

This article will start by discussing golden paths, what they are, their advantages and challenges, and how you can design golden paths with platform engineering. Ultimately, this article will share best practices for implementing golden paths.

What are Golden Paths?

The term golden paths has been referenced in software development as far back as 2009; it describes an opinionated and well-supported path to ‘build something’. Now, let's unpack the statement:

Opinionated: Golden paths are not meant to be one-size-fits-all solutions. Generally, they are designed to reflect your organization's specific needs and goals.
Supported: Golden paths should be supported by documentation, an array of tools, and training. This will help developers to understand and follow the golden path.
Path to ‘build something’: these paths are not just about the technical aspects of software development. They also include the business requirements and the overall goals of the project.

Golden paths are often established based on best practices, industry standards, and the accumulated knowledge and experience of software engineers in an organization. They are designed to minimize risks, ensure consistent results, and optimize the software development process.

Advantages of Golden Paths

Following a golden path generally improves quality, maintainability, software performance, and better developer experience. The advantages of golden paths in software development include the following:

Consistency across the codebase: Golden paths provide a standardized approach to development, ensuring that all team members follow the same guidelines and practices. This promotes consistency in coding styles, architecture, and implementation, making the codebase easier to understand, maintain, and collaborate on.
Knowledge sharing and onboarding process: Golden paths serve as a knowledge-sharing mechanism within an organization. Establishing standardized practices facilitates the transfer of knowledge and expertise among team members. New developers joining the team can quickly familiarize themselves with the established supported path, reducing the learning curve and enabling them to contribute effectively from the beginning. The shared understanding of the golden path promotes effective collaboration and enables smoother onboarding processes.
Improved quality and reduced risk over time: Paved roads are established based on best practices and accumulated knowledge, considering lessons learned from previous projects. Following these paved paths allows developers to produce higher-quality, more robust, reliable, and maintainable code. This leads to fewer bugs, easier debugging, and a more stable software product.
Better Developer Experience: “Developer Productivity + Developer Impact + Developer Satisfaction = Developer Experience.” — GitHub Blog. With consistency across the codebase and proper knowledge-sharing, developer productivity will increase, impacting the existing software positively, leading to developer satisfaction and an overall improvement in the developer experience.
Reduced costs: The expense associated with hiring can surpass $40,000 per employee. While some of this cost can be attributed to training and decreased productivity during the onboarding phase, a significant portion is incurred during recruitment. When developers of an organization are happy and see the impact of what they build, they will stay long-term and grow, avoiding hiring costs. Aside from retaining developers, the reduced time spent dealing with bugs and security also contributes to lower costs.

Challenges of Golden Paths

While golden paths in software development offer several advantages, they also come with some challenges:

Ever-Changing Best Practices: The software development landscape constantly evolves, and what constitutes best practices can change over time. The golden path established at one point may become outdated or less effective as new technologies and approaches emerge. Regular evaluation and updates are required to ensure that the golden path remains relevant and aligned with the current industry standards.
Balancing Innovation and Standardization: Golden paths aim to provide consistency and reliability, but they can sometimes hinder innovation. Developers may feel limited by predefined guidelines, preventing them from exploring new ideas or adopting cutting-edge technologies. Finding the right balance between standardization and allowing for innovation can be challenging, requiring ongoing discussions and adjustments.
Compatibility and Integration: Golden paths often recommend specific technologies, frameworks, or libraries that work well together. However, integrating these components with existing systems or third-party tools may present compatibility challenges. Careful consideration and testing are necessary to ensure smooth integration and minimize potential conflicts or issues.

Designing Golden Paths with Platform Engineering

Seeing the challenges associated with golden paths, one may ask, “How can I design a golden path without it becoming a bottleneck in the near future?”. Well, the answer is to design golden paths with platform engineering.

Platform engineering is an aspect of software engineering that majored in designing and building toolchains of workflows that enable developer self-service functions for software engineering organizations in the cloud native era. Platform engineers build and deliver an integrated product, often called an “Internal Developer Platform”, which covers the operational necessities of the entire lifecycle of an application.

An Internal Developer Platform (IDP) encompasses a comprehensive set of tools, services, and infrastructure integrated under a single platform in a manner that reduces the cognitive load on developers while retaining essential context and underlying technologies.

The IDP reduces the cognitive load on developers by abstracting away infrastructure complexities, providing standardized workflows, and automating repetitive tasks. This allows developers to focus more on writing code, delivering features, and improving software quality rather than dealing with tools and infrastructure-related challenges.

Aside from reducing the cognitive load on developers, platform engineering, when done right, provides golden paths that match the preferred abstraction level of the individual developer, who interacts with the IDP layer. Also, improving collaboration and accelerating the software development process along the way.

Platform engineering, when done right, means the IDP should be the “one-stop shop” where developers can find all the tools, templates, and documentation they need with a high emphasis on the following:

Streamlined developer onboarding experience: The IDP should simplify the developer onboarding experience by providing them with the right delivery setup or linking them directly to an external one.
Collaborative work environment: The IDP should support composability, reuse, sharing, and collaboration among teams and assets. All these features improve cooperation, visibility, and ownership using innersource practices.
Self-service workflow: The IDP should enable the developers to self-serve assets and resources. This includes access to preapproved software packages, CI/CD tools, test and development environments, and end-to-end visibility.

Many companies are now adopting platform engineering to pave golden paths in their organizations. Among the first who started this journey, it is worth mentioning Spotify. Prior to 2016, as Spotify grew into a large startup, its agile engineering culture with autonomous teams became a bottleneck which caused fragmentation in their ecosystem developer tooling: this led to the creation of Backstage — An open platform for building developer portals.

Also in 2016, Mia-Platform was founded and become one of the very first product companies dedicated to platform engineering. Mia-Platform has now evolved into an international business that has at its core the creation of enterprise-ready and personalized Internal Developer Platforms.

Best Practices for Implementing Golden Paths

Implementing golden paths with platform engineering and evolving into a Platform Company requires a deliberate and strategic approach. Also, it is a gradual process that requires reorganization on multiple levels.

From an architectural point of view, it is essential to go through a decoupling process that makes independent the fundamental elements that characterize the infrastructure serving a Platform Company. In addition to this, the following best practices ensure successful adoption and utilization. Here are some recommended practices for implementing golden paths:

Gradual Implementation
Implement golden paths with platform engineering gradually rather than attempting a complete overhaul all at once. This approach allows for better understanding, adaptation, and gradual integration into existing workflows. By taking incremental steps, you can assess the impact and make adjustments as needed.

It's important to note that gradual implementation does not imply a slow or drawn-out process. The pace of implementation will depend on your organization's unique circumstances and objectives. While some organizations may choose to introduce changes incrementally over a longer period, others may opt for a more accelerated timeline. The key is to strike a balance that allows for effective implementation while considering your organization's capacity for change.

Educate and Communicate
Educate your development team about the rationale, benefits, and best practices associated with the golden paths. Clear communication and documentation are essential to ensure that everyone understands the purpose and guidelines. Providing training, workshops, and documentation that explains the principles and techniques involved helps foster a shared understanding and encourages adherence to the golden paths.

Documentation is key
Create clear, concise, and step-by-step guidelines, tutorials, or knowledge bases that outline the best practices and procedures associated with the path. Make this documentation easily accessible to the development team, ensuring that it is regularly updated as new insights or improvements emerge.

Feedback and Testing
Encourage feedback from the developer team and stakeholders throughout the implementation process. Regularly solicit input and listen to the experiences and suggestions of those following the golden paths. Collecting feedback helps identify pain points, areas for improvement, and potential roadblocks. Additionally, perform thorough testing and quality assurance to validate the effectiveness and suitability of the golden paths in real-world scenarios.

Continuous Improvement
Embrace a mindset of continuous improvement and evolution of the golden paths. Actively seek opportunities to refine and enhance the established practices based on feedback, industry advancements, and emerging trends. Regularly review and update the guidelines to ensure their relevance and effectiveness over time. Encourage team members to contribute their ideas and insights for refining and improving the golden paths.

User-Centric Approach
Ensure that the golden paths do not compromise the customer/user experience. While focusing on standardization and efficiency, it is crucial to prioritize the end users and their needs. Evaluate the impact of the golden paths on user experience and make necessary adjustments to avoid negatively affecting usability, accessibility, or overall satisfaction.

By following the above best practices, you can successfully implement golden paths while ensuring effective adoption, continuous improvement, and alignment with user needs. This approach promotes a collaborative and iterative development process that benefits both the development team and the end-users of the software.

Conclusion

This article discussed golden paths in software development, what they are, their advantages and challenges, and how you can design golden paths with platform engineering. Towards the end, this article shared with you the best practices to consider when implementing golden paths.

To truly harness the potential of platform engineering, organizations should invest in robust architecture, cutting-edge technologies, and cross-functional collaboration. From an architectural point of view, it is essential to go through a decoupling process that makes independent the fundamental elements that characterize the infrastructure serving a Platform Company.

To learn more, check out this white paper on Why and how to transform your company into a Platform Company.

This article was written by Michel Murabito, Developer Advocate at Mia‑Platform.

How a frictionless Developer Experience improves software development

Paolo — Tue, 11 Jul 2023 13:01:14 +0000

In many IT departments, you can frequently experience that when an issue is reported, Dev and Ops teams start solving the problem by adopting a fire‑fighting mode. In fact, the Ops team can both give support to the Dev team for some tasks (e.g., activation of new services) and also needs the Dev team for some assignments on the infrastructure, and it often happens that these tasks block each other’s job. Besides being very inefficient, this working methodology also contributes to generating friction between the two teams, resulting in mutual unpleasant situations.

Some companies have attempted to mitigate this situation by adopting DevOps practices, an Agile approach, and scrambling members of former Dev and Ops teams within Feature Teams. While achieving a significant improvement over the previous case, these solutions are beginning to show their limitations, especially in more structured contexts in which Feature Teams are very numerous and well‑segmented. Ops staff on these teams, in fact, often risk having little work to do. This leads to unmotivated and uninterested team members who will eventually be more likely to leave the team.

The solution to this problem is to move away from the infrastructure‑centric view, and instead put the Developer Experience at the center of the development process.

What is Developer Experience?

Developer Experience (also known as DevX) is a transposition of the much more widely known concept of User Experience, referring, though, to those who develop software artifacts. Developers are considered as end users of the software they are creating. Therefore, it becomes important to ensure that the developers have a positive and frictionless experience while creating the software. This is a radical shift in perspective, in which developers are valued and placed at the center of business projects and decisions.

The Developer Experience covers every stage of the Software Development Life Cycle (SDLC). Starting from architecture design, through code writing, testing, deployment, and ending with monitoring in production, all moments of the SDLC are involved. Each of these stages should be improved in order to provide the smoothest experience possible. As we will further explain below, tools, both software and hardware, and the general work environment also affect DevX for better or worse.

Why it is important to value the Developer Experience

Investing in the Developer Experience primarily allows you to develop better software, and to do it faster. Indeed, if it is brought to the center, empowered, and valued, the development team is more pleased and more likely to deliver high‑quality software. A positive Developer Experience contributes to the realization of part of the principles behind the Agile Manifesto, such as:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Agile processes harness change for the customer's competitive advantage.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
Working software is the primary measure of progress.
The best architectures, requirements, and designs emerge from self‑organizing teams.

Also, the Developer Experience is now a significant trend for talent attraction and retention. In a market where developers are in high demand and receive a large number of job offers, they are free to choose the best fit for them, and DevX becomes critical for selection. Companies that can provide the most inviting work environment, the most interesting projects, and the best‑performing tools, have a big competitive advantage compared to the ones that do not consider this issue. In this perspective, if you do not take into consideration the Developer Experience, you might fall behind and lose attractiveness and competitiveness.

In summary, the main benefits provided by a positive Developer Experience are:

Removed frictions during daily work;
Improved quality of the delivered software;
Reduced time to market;
Improved confidence in the development team;
Created a positive work environment;
Avoided waste of time and incidents;
Valued and rewarded work;
Fostered new ideas.

How to improve Developer Experience

First of all, it is worth stating that since DevX affects all aspects of software development, it is good to take a holistic approach, and create a well-structured Developer Experience Strategy before implementing any solution. An holistic approach should address both the individual Developer Experience and the Team's Developer Experience considered as a single unit.

In this section, we will confine our remarks to just a few areas where action can be taken to improve the Developer Experience, particularly with regard to the environment and working tools. Both of these aspects, in fact, can significantly affect the experience of the development team.

As for the work environment and the company culture in general, investing in the continuous learning of each developer is definitely a great way to enhance team members. Similarly, encouraging the development of personal and open‑source projects, for example, by offering some of the company's resources and infrastructure, on the one hand, enhances the creative and inventive aspect of developers, and on the other hand, can be an opportunity to develop new corporate assets. The adoption of practices such as Agile, Platform Engineering, and Docs as Code methodology, and organizing the company according to the Composable Enterprise paradigm helps improve the overall Developer Experience.

Regarding hardware tools, a high‑performance laptop can significantly reduce the technical time required for some routine operations, improving and speeding up the release of new developments. The ability to customize the workstation, such as adding a second screen, equipping it with a standing desk, or providing headphones to reduce surrounding noise, is also a great way to optimize the work of the development team.

However, it is the equipment of software tools that provides the best results in terms of DevX. In particular, the most effective tool is the Internal Developer Portal (IDP), a single portal that brings together all the tools and technologies available and used within the company. With the Internal Developer Portal, Devs become the end users of the Ops team: the IDP provides Devs what they used to ask the Ops team in a self-service mode. The evolution of the Internal Developer Portal should be a goal and responsibility of both teams, and it becomes an important asset for the company as it provides the following benefits:

Fostered communication between teams;
Speeded up time to market;
Ensured distributed governance;
Improved software quality;
Provided visibility, traceability, auditability and observability throughout the DevOps cycle.

Conclusions

In summary, to solve the inefficiencies due to the different situations that arise between Dev teams and Ops teams, it is necessary to start putting the Developer Experience at the center of the development process. Treating developers as end users of the software that they develop, and who must therefore enjoy the best, smooth and frictionless experience while they work, is a radical shift in perspective.

Companies need to invest in improving the Developer Experience. Thanks to it, the development team is more pleased and gratified by its work, and, most importantly, the quality of the delivered software is improved. Working environment and tools, both hardware and software, are the key factors that can be improved to enhance the Developer Experience. The most effective tool is the Internal Developer Portal, a single portal that collects all available services and technologies in one place.

Creating an IDP from scratch can be time‑consuming and complex, both in terms of time and resources, especially in the initial phase. Adopting a ready‑made Internal Developer Portal such as the one provided by Mia‑Platfom allows you to reduce costs, accelerate its implementation, and focus on key assets to be exposed within it.

7 core components of an Internal Developer Platform

Paolo — Wed, 05 Jul 2023 08:02:19 +0000

The world of software development is constantly and rapidly changing. New technologies emerge every day to solve problems that slow down the daily work of IT departments. So while innovation is a positive factor in that it helps simplify work, there is also a proliferation of new tools and services that need to be managed by developers, thus increasing the cognitive load needed to develop and release software. Internal Developer Platforms (IDPs) aim to solve this problem and simplify the use of tools and services by providing a single point of access and interaction. Those familiar with the topic of Platform Engineering will have heard this statement before but not always is clear which tools are orchestrated by the Internal Developer Platforms.

With this article we want to answer this question, using Mia‑Platform's experience over the years. In this way, we want to provide a term of comparison for those already involved in Platform Engineering. In this article, we will outline 7 components that we believe are indispensable in an Internal Developer Platform. For each one we will show the real underlying problem, and how the component responds to it.

7 components of Internal Developer Platforms

Before we begin, however, a few brief premises should be made. First, these 7 components do not cover all the capabilities that an IDP can offer, and the order in which we will mention them does not necessarily represent their importance. In addition, we have chosen to mention only 7 components for the sake of article length, not because these are necessarily the most important in every IDP. The very choice of these 7 components may not apply in all cases, where perhaps others are more relevant, e.g., to the target business or the context of the organization.

However, we have experienced that these 7 components have proven to be crucial in most cases.

Pipelines

Since we are talking about software development, attention should initially be directed toward the code. Taking for granted the practice of versioning code, which is now a de facto standard, the next step concerns pipeline management. Once the code has been written and versioned, developers need to automate the processes of validating the code itself, and possibly even its release. Pipelines first and foremost meet this need.

There are several tools for pipeline management. To mention just a few of the most widely used, we find Azure Pipeline, GitHub Actions, GitLab CI Runner, Jenkins.

Artifact Repository

Another important issue involves artifact management. Having versioned images of already compiled, containerized, ready‑to‑use code available is very important for governance and security reasons, as well as providing a quick fallback in case of downtime.

Examples of tools for versioning and saving containerized artifacts are containerd, CRI‑O, Docker Engine, Dragonfly, Harbor, Nexus.

Runtime

Taking further steps in the software lifecycle, without dwelling on very important topics such as Continuous Deployment, we come to its heart, namely the runtime phase. For the purpose of this article, we consider the runtime phase in its entirety, thus including security, permission management, networking, etc.

Considering its broadness, for the runtime phase we find a large number of tools, each dedicated to a particular aspect. For the orchestration of containerized applications, one cannot fail to mention Kubernetes, which can now be considered the de facto standard. In particular, its autoscaling, self‑healing, and load-balancing capabilities (not to mention the many others it guarantees) make Kubernetes one of the best tools for scaling the runtime.

To ensure security, a noteworthy tool is Rönd, an open‑source project developed and maintained by Mia-Platform. Rönd is a container sidecar designed to protect APIs through simple security policies. It also allows roles and permissions to be created and managed using an RBAC/ABAC approach.

In addition to these two tools, we also mention Cilium for networking and Falco for security.

API Gateway / Service Proxy

Another essential component that ensures the products and services can communicate with the outside world is the API Gateway. Depending on the type of product and its use, we also include in this section Service Proxies, which can be implemented for a similar purpose.

To further explore this topic, please refer to this video. Some of the most widely used tools in the Service Proxy and API Gateway category are Gloo Edge, Emissary Ingress, Envoy, NGINX, and Traefik.

Monitoring

One of the most important aspects of the software life cycle is related to monitoring and log tracking. Indeed, monitoring is crucial to know the status of systems so that proactive action can be taken to resolve problems and ensure service continuity. By collecting metrics, displaying them in dedicated dashboards, and triggering alerts, monitoring tools help development teams identify and resolve critical issues. In the context of microservices and distributed applications, collecting logs can be complex but is still a crucial part of successful projects.

Among the most widely used tools for ensuring software observability are Elasticsearch, Fluentd, Grafana, Kibana, and Prometheus.

FinOps and Sustainability

Going beyond the realm of development in the narrow sense, two very important and closely related topics are FinOps and sustainability. The main need from which the FinOps discipline arose is to manage cloud-related operations from a financial perspective. Controlling and optimizing cloud costs also brings sustainability benefits.

Sustainability, primarily environmental but not limited to this aspect, should not only be a benefit that is gained reflexively from other optimization activities but should be seen as a goal to be achieved. The issue is increasingly relevant, so much so that energy efficiency is becoming a real architectural requirement, according to Thoughtworks. Platform Engineering can contribute to making software more sustainable, as Michel Murabito, Developer Advocate at Mia-Platform, explains in this article.

One of the most relevant tools in this category is [kube‑green], an open‑source Kubernetes operator developed by Davide Bianchi, Senior Technical Leader at Mia‑Platform. kube‑green significantly reduces energy consumption by automatically shutting down some unused resources according to a predetermined schedule, also bringing big cost savings.

Data Management

Last, but certainly not least, there is the issue of data management, as applications need it to run. Databases are needed for data storage: depending on the type of application, databases can be relational or documental databases (SQL and NoSQL). Examples of relational databases are MySQL, Oracle, PostgreSQL, while one of the most popular NoSQL databases is MongoDB.

But data management is not limited to databases alone. Data can initially be collected from the various systems in which they reside and be made available in real‑time: this task is performed by data stream systems, such as Confluent or Kafka.

Finally, it is very important to handle data differently depending on its temperature, that is, how often it is used. In this way, it is possible to optimize costs by leveraging the storage capabilities of storage buckets such as Amazon S3 and Oracle Object Storage.

An Internal Developer Platform for a better Developer Experience

The components illustrated above are fundamental to application software development, and as noted above, each requires knowledge and management of many different services. Performing the various steps independently and manually connecting all the various components is time‑consuming, difficult, and error‑prone.

For this reason, Internal Developer Platforms have emerged and are spreading. Thanks to them, developers have a single point of access, namely the platform, which is responsible for managing the entire software lifecycle, simplifying and speeding up their work. The Developer Experience improves significantly by using an Internal Developer Platform: the delivered software will be of higher quality and productivity will increase significantly as developers can focus on code and new features. Secondary activities, namely those related to the management of development tools, will in fact be automated and simplified by the platform, easing the cognitive load of developers.

Conclusion

In this article, we have outlined some of the components for application software development. Based on our experience in Mia‑Platform, these components are critical for most Internal Developer Platforms, but they certainly do not cover all the capabilities that platforms can provide. In addition, considering that new tools emerge almost daily to simplify development or extend functionality, it is important that platforms are built with a modular perspective so that new services and capabilities can be easily integrated.

Download our free white paper to find out how to evolve your company into a Platform Company!

This article was written by Michel Murabito, Developer Advocate at Mia‑Platform.

How Machine Learning Can Improve DevX: Insights from a CTO

Paolo — Tue, 27 Jun 2023 08:23:56 +0000

With customer needs evolving rapidly, businesses and companies are constantly searching for innovative ideas to stay ahead of the competition, drive growth, and increase customer retention. Machine learning (ML) has emerged as a go‑to strategy. It offers a way to craft profitable product roadmaps and gain valuable insights into customer behaviour and preferences.

During the software development life cycle, developers lean heavily on tools like code editors, project management software, programming languages, deployment pipelines, version control, etc. In fact, there are always debates among developers about which editor is the best, which tools can boost their productivity, and which frameworks to use.

Beyond individuals advocating for their favourite tools, companies have adopted DevOps practices and the Agile methodology to improve developers' experience when building and maintaining software. Although these methods have brought about significant progress and improved experience, they have presented certain limitations, especially in companies with multiple teams or individuals working alongside other developers.

In this article, we will explore what ML is, discuss how it can improve the Developer Experience (DevX), and provide insights from a CTO, Giulio Roggero.

What is developer experience?

Before we dive into what ML is and how it can improve DevX, let’s discuss what exactly DevX entails. Developer Experience is the overall experience of a developer using a product. A product is said to have good DevX if the associated tools, processes, and working environments are properly set up to ensure maximum developer productivity.

What is machine learning?

ML is a technique that uses mathematical data models to help computers learn without direct instruction. It uses data and algorithms to emulate human learning and gradually improves accuracy. Systems or applications using ML are categorized as intelligent computers, as they think like humans and perform tasks on their own.

How Machine Learning can improve DevX

In recent years, ML has revolutionized how products are built. It has given companies of diverse backgrounds innovative ideas to unlock new possibilities and satisfy their customer's needs. ChatGPT, GitHub Copilot, and DALL.E, to name a few, have made ML accessible to companies and individuals seeking to build intelligence into applications and improve the overall experience.

Let’s look at some use cases and how they improve the developer experience.

Quality Control

Let’s take the case of pair programming, an agile software development technique involving two developers working together to achieve a common goal simultaneously. This approach involves one programmer writing the code while the other reviews and provides guidance. It has improved the software development life cycle (SDLC) by offering the following benefits:

Fewer bugs and mistakes during development;
Improved teamwork;
Faster training;
Better knowledge sharing.

Despite being a proven solution in the SDLC, it also has drawbacks. Pair programming is resource‑intensive, difficult to sustain, and sometimes leads to burnout.

Recently, developers are far more empowered and can seamlessly pair programme with ML‑enabled tools like ChatGPT, GitHub Copilot, CaptainStack, Tabnine, etc. Unlike humans, these tools are relatively easy to integrate, require minimal resources, and increase productivity.

Even before ChatGPT, we were introduced to Github Copilot, another AI‑based tool to assist developers with coding. Both of these applications can provide a lot of help in writing simple snippets of code to accelerate task completion. A popular approach to ensure that mistakes in the code are avoided is to use a technique called pair programming — where one developer writes code, and another sits alongside them and helps to write the code according to the strategic vision and swap roles every 30’. These new applications allow this pairing to now include an AI rather than a human.

Giulio Roggero - CTO & Co‑Founder - Mia‑Platform

Improved workflow

In the modern‑day SLDC, developers’ roles are constantly changing: they are not only required to write just code but also to manage code changes, test use cases, debug, create workflows, and review pull requests. These additional responsibilities often shift developers’ focus from their primary assignment, and the associated cognitive load and workload burdens them.

Crafting good source code means crafting testable applications. If you can give expressiveness in your test, you can declare your intentions in the code, which is the business problem that you want to solve. This also helps debugging because ChatGPT has more context when you ask a question about troubleshooting problems providing you with some insights and advice on why you have the problem. In that way, you may speed up the time for bug resolution.

Giulio Roggero - CTO & Co‑Founder - Mia‑Platform

Improved software development process

Companies often build entirely new applications or integrate features into existing applications. The development process requires a traditional approach of estimation, strategic decision‑making, rapid prototyping, and code review. These tasks can be repetitive and require a lot of heavy lifting from the developers tasked to develop them.

ML can help the development team prioritise features to build, provide precise estimates, shorten the time spent on prototyping products, automatically review code, and optimize performance.

Drawbacks to implementing Machine Learning

Like everything else, ML is imperfect. It has some serious limitations that companies and individuals using or intending to use it need to consider.

Ethics and data acquisition

The concept of ML is rooted in identifying useful data. The results will be incorrect if credible data is not provided. Companies ranging from small businesses to large enterprise need to protect the data associated with their innovations. Since ML heavily depends on tailored data, building solutions to tackle DevX for such companies or individuals can be difficult.

Accessibility

In a recent survey by Accenture, ML and related technologies are still not used to their full potential. Integration and compatibility are still major barriers as some companies' technologies, programming languages, human resources, and frameworks are limited.

ChatGPT doesn’t necessarily make software development more accessible to people that are not already trained. You still need to understand the code ChatGPT is writing for you; otherwise, you could create a large tech debt when the incorrect code has to be refactored. ChatGPT can support citizen developers – or those who are not Dev professionals but can use applications for simple IT projects – in configuring and scripting already coded components. This will be a great help for moving projects forward. However, for now, machine learning cannot substitute professional coders in writing new code, but it can be a great accelerator when coders need to work with common patterns. It will be exciting to see where it can go.

Giulio Roggero - CTO & Co‑Founder - Mia‑Platform

Doing the heavy lifting with Mia‑Platform

Improving DevX through ML can be a daunting task. It comes with a lot of overhead around implementation and maintenance. Before we dive into what Mia-Platform is and how it does the heavy lifting, it is paramount that we understand what platform engineering is.

Platform engineering is a process organisations use to enhance developer productivity by reducing the complexity and uncertainty of modern software delivery. Its core focus is on continuously improving the DevX by eliminating obstacles between developers and production.

As mentioned, platform engineering provides many benefits and significantly improves DevX. However, there are cases where it is not possible or cost‑effective to build and own a platform. The time, resources, and manpower required are usually high.

What is Mia‑Platform?

Mia‑Platform is a digital platform builder that allows companies from small to large enterprises to build efficiently their platform and ship software faster. With Mia‑Platform, companies can fully focus on developing their software and better dedicate themselves to customers’ needs while the platform does the heavy lifting.

You can learn more about Mia‑Platform, exploring all the benefits and services it offers.

Conclusion

In summary, DevX is an essential part of the SDLC; it ensures that tools, processes, and working environments are properly set up to ensure maximum productivity.

Companies and individuals looking to stay ahead of the curve and have a competitive advantage need to invest in improving the DevX. With Mia‑Platform, the development team is fully focused on what matters, and the quality of delivered software is greatly improved.

From Data Warehouse to Data Fabric: the Evolution of Data Architecture

Paolo — Tue, 20 Jun 2023 07:08:00 +0000

In the last century, data has become the lifeblood of every organization, ranging from e‑commerce giants to healthcare providers and government agencies. Collecting and managing this data effectively can give organizations valuable insights to aid decision‑making. However, this has been proven to be a daunting task.

As important as data is, CIOinsight reports that only 10% of organizations feel that their company excels in data analytics management. Recognizing this significant data utilization gap, organizations have embraced modern data architectures to bridge the divide.

Data architectures are the structured frameworks and systems that define how data is organized, integrated, and accessed within an organization. The architecture sets the blueprint and establishes guidelines for data and how it flows through data storage systems.

This article discusses the evolution of data architecture, its underlying principles, and the advantages of adopting modern data architectures for effectively managing organizational data.

The evolution of data architecture

Over the years, data architecture has evolved to adapt to the growing needs of businesses. A notable transformation discussed in this section is the shift in data architecture from logical warehouses to data fabrics.

The Logical Warehouse

Logical warehouses, also known as data warehouses, have been the foundation of data management for decades. These data warehouses are central repositories designed to store data from different sources like transactional systems, application log files, relational databases, etc., providing a unified view of the information.

In general, logical warehouses use Extract, Transform, Load (ETL) processes to extract data from source systems, transform it to ensure consistency, and load it into the warehouse. Logical warehouses are solely intended to perform queries and analysis and often contain large amounts of historical data.

Challenges of Logical Warehouses

While logical warehouses served their purpose, they faced several challenges as data volumes increased. Some of the key limitations included:

Data Silos: Logical warehouses often resulted in data silos, where different departments or teams would maintain their own isolated data sets, leading to inconsistencies and duplication.
Performance: Since the data had to go through multiple processes and stages before it was available for analysis, this greatly impacted the performance of data warehouses.
Scalability: Implementing data warehouses is complex and costly due to hardware limitations. It also required expertise in data modeling, ETL processes, and database management making it harder to handle exponential data growth.

Data Lakes

In response to the data warehouse challenges, data lake architecture was introduced in 2010. Although the data lake architecture is very similar to data warehouses, they differ because data lakes also cater to semi‑structured and unstructured data.

Data lakes' ability to store large amounts of data in its natural or raw format helps us to:

Use data from multiple sources regardless of data types (unstructured, semi‑structured, structured);
Create focused portions of data specifically tailored to meet the requirements of a particular use case;
Discover new insights by leveraging advanced analytics and machine learning techniques without being limited by pre‑defined schemas.

Challenges of Data Lakes

This open format feature of the data lakes made it more popular than data warehouses. However, data lakes came with challenges as data ingested without strict standardization led to inconsistencies in the database. Additionally, the data stored in data lakes require significant transformation and integration efforts before it can be used for analysis which can be complex and time‑consuming.

Data Mesh

Data Mesh architecture is an emerging approach that provides an alternative to the central data lake architecture. Data Mesh, coined by Zhamak Dehghani in 2019, is a decentralized data architecture that organizes data by a specific business domain.

By introducing domain‑oriented data ownership, domain teams become accountable for their data and products, improving data quality and governance.

Traditional data lakes often encounter challenges related to scalability and performance when handling large volumes of data. However, data mesh architecture solves these scalability issues through its decentralized and self‑serve data infrastructure.

With each domain having the autonomy to choose the technologies and tools that best suits their needs, data mesh allows teams to scale their data storage and processing systems independently.

Data Fabrics

Data Fabric is an integrated data architecture that is adaptive, flexible, and secure. It is an architectural approach and technology framework that addresses data lake challenges by providing a unified and integrated view of data across various sources.

Data Fabric allows faster and more efficient access to data by extracting the technological complexities involved in data integration, transformation, and movement so that anybody can use it.

To understand the difference between the data mesh and data fabric architecture approach to achieve data quality and what data paradigm to adopt, check out this Mia‑Platform article Data Mesh vs Data Fabric: a tale of two new data paradigms.

Modern data architecture principles

According to Dataversity, data architecture principles refer to a collection of policies regulating the enterprise data framework and the operational rules for gathering, integrating, and managing data assets. These principles help us to create a consistent, reliable, and efficient data architecture that aligns with our organization's goals and objectives.

To effectively leverage data as a competitive asset, here are some of the common modern data architecture principles to follow:

Data Quality (DQ): Data quality is essential to any data architecture; establishing data quality standards and processes ensures data accuracy, completeness, and reliability. The data quality principle teaches us to implement data profiling, cleaning, and validation techniques to identify and rectify data issues immediately to avoid bad data quality. Effectively managed and reliable data is crucial in developing precise models and robust schemas for extracting valuable insights.
Data Governance (DG): The Experian data quality report tells us that 78% of organizations globally are plagued with poor data governance, which leads to mistrust in the data and the resulting insights drawn from the data. Data governance teaches that at any point in the data lifecycle, the data consumers should know the location, format, use relationship, and any other relevant information relating to the data to avoid data debt. Data governance is closely related to data quality, enabling data architectures to ensure data integrity and increased data quality.
Strive for consistency: This data principle emphasizes maintaining uniformity across all data structures, formats, and processes throughout an organization. Using standard terms and vocabulary ensures that data is consistently defined across different systems, making it easier for developers and non‑developers to collaborate on the same projects. Consistency in data representation across organizations presents teams with a "single version of the truth", making data and data analysis easily understood by a wide range of users within the company.
Make data a Shareable asset: Making data a Shareable asset emphasizes that we treat data as a valuable resource to be shared and accessed across different systems. This principle aims to eliminate data silos and encourages the design of data architectures that facilitate efficient data sharing. Doing so ensures that all stakeholders understand the company and its data comprehensively, fostering collaboration and informed decision‑making.
Data Security and Privacy: Modern data architecture must ensure the confidentiality and integrity of data while protecting sensitive information from unauthorized access, breaches, or misuse.

Characteristics of modern data architecture

This section discusses some of the characteristics of modern data architecture:

Automation: Modern data architectures use automated processes, tools, and technologies to optimize data‑related tasks and operations. Unlike traditional systems, modern data architecture can build complex processes in hours or days using cloud‑based tools.
Adaptable: A modern data architecture must be flexible to respond to our evolving business needs. It must support multiple kinds of users, query operations and deployments, data processing engines, pipelines, etc.
Scalability: Scalability is a crucial characteristic of modern data architecture that allows us to scale up or down quickly and affordably when our business demands change.
Cost Efficient: Modern data architectures leveraging scalable infrastructures allow us to easily accommodate increasing demands in the future without overbuying hardware at the start. Also, modern data architecture sometimes utilizes cloud‑computing platforms that use a "pay as you go model" so that we only pay for the resources we use.

Benefits of modern data architecture

After discussing modern data architectures, and their characteristics, this section discusses the benefits of modern data architectures to businesses and organizations.

Provides a comprehensive view of a company: With data integration enabling seamless data integration from various data sources and systems within an organization, data is collected and stored consistently and organized. This integration ensures that the data at any point offers a comprehensive view of the company, "a single source of truth."
Reducing redundancy: Data integration discourages overlap in data fields across an organization by collecting and harmonizing different data and data sources.
Improved data quality: Modern data architectures incorporate techniques like data cleansing and validation, data standardization, data quality monitoring and remediation, etc., that ensure data reliability.

Conclusion

The evolution of data architecture has been driven by the growing importance of data in organizations. From traditional data warehouses to modern data mesh and data fabric approaches, these architectures addressed specific challenges and brought new opportunities.

By embracing modern data architectures, organizations can benefit from improved data quality and comprehensive data insights to unlock the full potential of their data and stay competitive in today's world.

An amazing cohabitation of the two prominent modern data architectures, data mesh and data fabric is Mia‑Platform Fast Data. To understand more about Mia‑Platform Fast Data, read this article and download the free infographic.

Achieving Better Governance through Composability and Reuse

Paolo — Tue, 13 Jun 2023 06:00:00 +0000

Most software projects fail due to non-technical issues. According to a survey by Gartner, 59% of developers said that poor governance and management practices are the primary cause of project failures. In addition, poor governance can lead to security breaches, data loss, and regulatory violations, resulting in significant financial and reputational damage to organizations.

To address these challenges, software development teams need to establish effective governance practices that ensure that software is developed and deployed in a consistent, reliable, and secure manner. One way to achieve this is through a composable and reusable architecture.

Composability and reuse are powerful techniques that can help teams achieve better governance by providing the flexibility and speed needed to add, reuse, and scale new services with reduced time‑to‑market. By breaking down software into several independent modules and using standardized processes, teams can improve consistency, reduce risk, and improve quality.

This article will discuss the challenge of governance in software development, explore the concept of composability and reuse, and discuss how they can achieve better governance in software development. In the end, you will learn how you can easily implement composability.

The Challenge of Governance in Software Development

Governance in software development refers to the set of policies, procedures, and practices that organizations use to ensure that software is developed and deployed consistently, reliably, and securely. Despite its importance, governance remains a significant challenge for many organizations due to a range of factors, including:

Lack of Clarity*: There is often a lack of clarity around the roles, responsibilities, and processes involved in software development governance, which can lead to confusion, delays, and errors.
Complexity: Software development projects can have many interdependencies, hidden bugs, and unforeseen interactions between different components, making it difficult to establish effective governance processes that work for everyone.
Resistance to Change: Many organizations resist change and may be reluctant to adopt new processes or technologies to improve governance, even when they would benefit from doing so.
Limited Resources: Organizations may lack the necessary resources, such as staff, budget, or technology (most common), to implement and maintain governance processes effectively.

Understanding Composability and Reuse in Software

Composability is the ability to create complex systems or applications by combining several independent software modules. These individual modules were called Packaged Business Capability (PBC) by Gartner. A PBC combines data schemas, APIs, and event channels to perform a well-defined business capability. PBCs can be developed internally within an organization, purchased from third‑party vendors, or can be open‑source components.

The Composable Architecture paradigm does not specify in detail how PBCs should be developed, nor its final architecture. However, the architecture should ensure that each PBC can be scaled, replaced, or removed individually with little impact on the rest of the architecture.

Reuse, on the other hand, refers to the practice of using existing code or modules to build new applications or systems. By reusing modules, developers can save time and effort, reduce costs, and ensure greater consistency and quality across different projects.

Together, composability and reuse provide a powerful framework for software development that can help teams build complex systems quickly and efficiently. By breaking down software into independent, reusable modules, developers can create systems that are easier to manage and maintain over time.

How Composability and Reuse Can Help Achieve Better Governance

Composability and reuse are powerful techniques that can help software development teams achieve better governance by providing a framework for building and maintaining software. Here are some ways in which composability and reuse can help achieve better governance:

Standardization: Standardization is a key aspect of enforcing software governance. With composability and reuse, development teams can break down software into reusable, standardized, documented, and tested modules. This ensures that software is developed and deployed consistently and reliably, reducing the risk of errors and improving quality while also meeting security and compliance requirements.
Consistency: Composability and reuse help ensure consistency across different software development projects by providing a common set of standards and practices that can be applied across the organization. This improves governance by making managing and monitoring software development activities easier.
Flexibility: Composability and reuse allow software development teams to adapt to changing requirements and business needs. This helps ensure software development projects align with organizational objectives and regulatory requirements.
Scalability: With Composability, each module can be independently scaled to quickly respond to growing traffic, either by allocating more resources (vertical scaling) or adding more of the same module (horizontal scaling). This ensures that the organization maintains existing governance practices while still meeting the needs of its users and stakeholders.

By leveraging the benefits of composability and reuse, software development teams can establish effective governance policies and procedures that ensure the software is developed and deployed consistently, reliably, and securely. This helps organizations reduce the risk of project failures, security breaches, and other costly and damaging problems while improving quality and reducing costs.

Implementing Composability and Reuse with Mia‑Platform

Adapting quickly to market changes is the only way to remain competitive. To adapt to changes quickly, organizations should be flexible, agile, and able to implement new strategies and technologies quickly.

Mia‑Platform is a Platform Builder that supports organizations in creating their Internal Developer Platform, a tool that accelerates the work of developers by self‑serving them with all the technology they need. Mia‑Platform Console simplifies the complexity of microservice architectures by facilitating a composable‑by‑design approach. Mia‑Platform Marketplace provides a service catalog of ready‑to‑use components, facilitating the reuse of modules in different projects or in different sections of the same project.

With Mia‑Platform's technology, organizations will be able to easily adopt composability and reuse in their software projects and can begin the transition to the Composable Enterprise paradigm. To learn more, check out this free white paper on Composable Enterprise: Why composability is the future.

Service Account 101: the Power of M2M with Security in Mind

Paolo — Tue, 06 Jun 2023 08:46:49 +0000

Service accounts are an essential component of many IT environments, allowing applications and services to access various resources without requiring user intervention. However, if not managed properly, service accounts can pose significant security risks to an organization.

In this article, we will explore the benefits of using service accounts, as well as the limitations and security risks associated with them. We will also discuss some best practices for effectively managing service accounts, in regard to a security and governance point of view.

Service account vs user account: what’s the difference?

In computing systems, user accounts and service accounts are both used for authentication and authorization. However, they have distinct purposes and characteristics that are important to understand.

A user account is created for individual users and holds the personal information associated with their identity: such information typically only makes sense when it is attributed to a real human being, who is identified by the account inside a digital environment.

Human-to-machine communication is typically facilitated by user accounts, which require human interaction to authenticate, such as entering a username and password. Users are typically granted access to computing resources based on their role and level of authorization.

On the other hand, a service account is an account used by an application or service to access computing resources: therefore, it lacks all those characteristics that could be associated by nature with a human user’s identity.

Service accounts are typically used for automated processes or to run specific services on behalf of multiple users. They are commonly employed in machine‑to‑machine (M2M) communication and follow specific authentication and authorization standards.

Service accounts are particularly handy in cloud computing environments, where applications and services need to interact with resources across different servers and networks. The registration and login process of a service account can be managed by dedicated services, such as the Client Credentials service we talked about in a previous article.

How can you use a service account? Some examples

One of the possible uses of a service account is to access resources, such as files, databases, or APIs that are not available to regular user accounts. In a cloud environment, a service account can manage resources such as virtual machines, containers, or infrastructure components. For instance, it could be used to monitor and log Kubernetes resources, such as Pods, Deployments, and Services. Once authorized, the service account could use specific tools to collect and analyze data for auditing and troubleshooting purposes.

Another common scenario is having a service account tasked with executing processes or scripts that automate activities or interact with other systems. For instance, you can grant a service account permission to make automatic code deploys, improving the speed of the software development process.

A service account can also be useful to securely store and manage the credentials used to authenticate with other systems or services.

Advantages of service accounts

Before we delve into the advantages of service accounts, it is important to note that many of the benefits we will talk about in this section can easily become issues if we ignore the recommended security and governance best practices that will be covered later in this article.

One of the greatest advantages of service accounts is automation: they are designed to access resources and perform operations without the need for human intervention. This is useful for applications that require continuous monitoring, reporting, or data processing.

Service accounts are also potentially great for security. Using service accounts promotes the separation of roles and responsibilities, which reduces the risk of credential misuse or theft. In fact, service accounts are - or should be - typically granted only those permissions necessary to achieve a specific purpose. Limiting permissions and only granting access to the resources explicitly needed by the application or service can reduce the risk of privilege escalation.
In contrast, user accounts typically have higher privileges than necessary to perform their assigned tasks, which increases the risk of unwanted actions or data breaches. Moreover, service accounts can also help eliminate human error during login, password reset, and password change, ensuring that every process is carried out consistently.

Another great perk of service accounts is that they provide an opportunity for efficient governance: firstly, they can be created and deleted as needed, making them highly scalable.

To achieve this, however, it is fundamental to establish a rigorous management system for these accounts, in order to keep track of their existence, capabilities, and purpose from the moment they are created until the moment they are no longer useful and can be deleted.
If managed properly, this is a great advantage over traditional user accounts, which can be time‑consuming and challenging to manage at scale.

Moreover, service accounts are useful when differentiating access to multiple environments. Multiple accounts can be created for different environments, such as development, testing, or production, making it easier to manage access control between environments.

Finally, service accounts enable more straightforward auditing, as they are not tied to individual users who may come and go. Instead, they are linked to specific applications or systems, making it easier to track and review activity logged by those accounts.
This is another situation in which it is crucial to have a strong governance structure for service accounts, without which it would be impossible to trace back to their specific function inside an organization’s digital environment.

How can a service account become a security issue?

Although service accounts are a useful tool, there are several ways in which a service account can be abused that must be taken into account in order to prevent them.

One of the main risks of using service accounts is privilege escalation, meaning that a bad actor might gain access to resources they otherwise wouldn't have access to by impersonating the service account. Impersonation happens when an identity, which can be a user or another service account, manages to authenticate as the service account and is therefore granted elevated access to certain resources by temporarily gaining the privileges of the impersonated service account.

Impersonation can also be used as a form of spoofing, to obscure the identity of the bad actor, and if a monitoring system has not been properly set up to check for service account activity, it might not be possible to trace the malicious actions back to their source.

In fact, service accounts can create blind spots in audit logs or security analytics tools, making it difficult to identify who has used the account. Therefore, they can create ambiguity and a lack of accountability around who is responsible for the activities performed by them.

Securing your service accounts: best practices

Service accounts play a critical role in the secure management of IT environments, and it is essential to safeguard them against potential security threats.

The dual nature of service accounts

When it comes to secure management of service accounts, it is essential to acknowledge their two‑fold nature.

First of all, a service account is an identity that can be authorized to access resources. Therefore it is crucial to restrict its privileges, to minimize the potential harm that can be inflicted by it in the case it becomes compromised.

For instance, if a bad actor has already gained a foothold in an environment, it might not have sufficient permissions to directly access certain resources in that environment. However, if the malicious identity finds a way to impersonate an existing service account with higher privileges, it might gain the capability to further extend its own privileges and do more damage.

From the previous example, we can notice that a service account is also a resource, so it must be safeguarded against any unauthorized access to avoid any potential compromises, just like any other critical resource.

Managing service account credentials

The first way to protect service accounts is to properly generate and store their credentials. In the OAuth 2.0 standard, there are two methods available for authenticating a service account:

Client Secret Basic: it requires an identifier for the service account, called the client id, and a client secret, so it is very similar to the concept of authenticating with username and password. This method is simple to implement and provides a basic level of security.
Private Key JWT: it uses asymmetric encryption keys for authentication. This method is to be preferred since it provides a higher level of security: the private key is never transmitted over the network nor shared with the server, so it is better protected from any attacks.

Knowing the private key of a service account’s key pair is similar to knowing a user’s password, therefore it is important to store it in a safe place where it can not be subject to leakage. A poorly protected service account key could be used by a malicious identity to perform undesired operations in disguise.

One good security measure is to avoid uploading service account keys to cloud storage systems or source code repositories, as well as trying not to share them between users through e-mails or chats where they could be easily accessed. It is also recommended to limit the proliferation of service account keys, while also monitoring the presence of unused ones and using expiry times to let them expire automatically.

More generally, you could ensure that access to service accounts is restricted only to authorized users or systems by implementing IP restrictions or role-based access controls.

Limiting service account privileges and range of action

In general, when you decide to set up a service account, remember to implement least privilege access control by granting it the least privileges required to perform its tasks and limiting its access to critical resources. Also, try to follow the principle of separation of duties: create separate service accounts for different applications in order to limit their range of action to the specific purpose of the service.

Once you have some service accounts in your environment, you should set up a proper monitoring system to check for any signs of suspicious activity regularly.

It is also a good practice to keep an inventory of all service accounts and their purposes, in order to avoid leaving any unused service accounts around. For example, legacy accounts that have been replaced by newer systems, or accounts that were created for temporary activities and have not been removed after fulfilling their purpose.

Another useful thing to do is regularly review service account permissions, in order to determine if they are still appropriate or should be redefined to stay in line with current business requirements and changing security risks.

Last but not least, to minimize security risks and maintain a secure environment, always avoid using a service account for human login purposes. When human users are granted access using service accounts, it becomes harder to track and audit the actions performed, since it is difficult to distinguish between automated and manual activities. This could make it difficult to investigate security incidents and identify the root cause of security issues. Moreover, using service accounts for human login may result in credentials being stored in insecure ways, which could be compromised by attackers.

Tips for efficient identity management

There are some useful factors to take into account when determining the appropriate use of service accounts for the benefit of an organization.

First of all, consider the purpose of the account. If the account will be used by an application or system process, it's best to use a service account. If the account is for a human user, a normal user account is appropriate. Identify the services or applications that might need to automate tasks and integrate systems without human intervention: in those cases, creating a dedicated service account might be the appropriate solution.

Then, think about the scope of the account. Service accounts are often used for specific processes, while normal user accounts are used for general computing tasks.

Also, consider the level of security needed for the account. Service accounts typically have more access and privileges than normal user accounts, so it's important to ensure the account is secure and not easily compromised.

With regards to governance, consider the ease of user account management. Service accounts can often be managed centrally, while normal user accounts may require more individual attention.

Finally, be aware of any compliance requirements that may dictate the use of a specific type of account. Companies may create service accounts dedicated to handling critical operations involving personal data, like credit cards or healthcare information, to comply with regulations such as the GDPR, which sets rules for how organizations should process and protect the personal data of EU citizens.

The Bottom Line

In conclusion, service accounts can provide a convenient and secure way to manage access to cloud resources, but they also come with their own set of limitations and risks.

By following best practices you can minimize the potential for security breaches and ensure that your service accounts are being used in a responsible and secure manner. ]]With the right approach, service accounts can be a valuable tool for any organization looking to manage access to their cloud resources more efficiently]].

As always, it's important to stay vigilant and up‑to‑date on the latest security threats and best practices. By staying informed and taking a proactive approach to managing your service accounts, you can help ensure the long‑term security and success of your cloud‑based infrastructure.

Starting from the latest major release, Mia‑Platform Console features Service accounts management. Please refer to the documentation for further details on the feature. You can also learn more about Mia‑Platform v11 and all its new functionality on this page.

This article was written by Giovanna Monti, Full Stack Developer Specialist at Mia‑Platform.