DEV Community: Harry

Best Practices for Fostering Collaboration with the Open-Source Community in 2024

Harry — Fri, 31 May 2024 08:00:00 +0000

This post originally appeared on my blog

Open-source software (OSS) has become an integral part of modern software development, with many organisations relying on open-source components to build their products.

However, using open-source software comes with its own set of challenges, such as security risks, licensing issues, and the need to contribute back to the community.

In this post, we delve into effective practices for engaging with and contributing to the open-source community, enhancing both your projects and the broader ecosystem.

Open Source Program Office

An Open Source Program Office (OSPO) is a corporate entity that is responsible for managing an organisation’s open-source efforts. The OSPO is a “central nervous system” for open source within an organisation and provides governance, oversight, and support for all things related to open source (Haddad, 2022). It trains staff, promotes efficient software development with open-source parts, guides open-sourcing projects, ensures legal compliance, and builds community engagement.

An OSPO helps bridge the gap between an organisation’s internal development and the external open-source community, helping to ensure that the organisation is a good steward of open-source software and can reap the benefits of open-source adoption, all the while minimising risks (TODO Group, 2023).

Six common characteristics of successful open source programs (Walker, 2016) are:

Value marketing and branding highly.
Choose open-source communities that match your tech goals.
Get good legal advice to balance risk and innovation.
Your open-source efforts should support your product goals.
Explain your support plans clearly to everyone involved.
Hire someone passionate about open source to lead.

Uber, for example, was motivated to create its own OSPO because of the need to streamline support for various open-source initiatives. For instance, there was a significant demand for advice on incorporating open-source technology into external products from engineers. Key concerns included navigating compliance, licensing, and related legal matters. Additionally, there was a crucial need to offer engineers direction on how to release Uber’s own software as open source, whether by initiating new projects or contributing to existing ones (Uber Case Study n.d.).

“It was natural and organic for Uber to create an open source program office since it allowed us to build our platform and scale the technology at unprecedented speed […] in essence, Uber loves open source because it’s essential to our success.” (Hsieh, n.d.).

Since the OSPO is a relatively new concept, with Google opening one of the first OSPOs in 2004 (Google, 2022), the literature references mostly large corporate entities with the resources to create an OSPO. The Linux Foundation recommends a minimum of five employees to start a successful OSPO (Haddad, 2022). Therefore, an OSPO is not best suited for smaller organisations that cannot afford to dedicate the resources to an OSPO.

Open Sourcing Previously Proprietary Software

Open-sourcing previously proprietary software is the process of releasing software to the open-source community for free use and modification. This process can be beneficial for organisations; it allows for increased innovation, collaboration, and community engagement and can lead to the development of features that are beneficial to the organisation, as well as the resolution of issues that impact the organisation’s operations.

There are a series of examples of organisations open-sourcing previously proprietary software. Some of the most notable examples include:

Microsoft Open-sourcing .NET By open-sourcing .NET, Microsoft allowed for a single, collaborative codebase across platforms rather than separate implementations. Ultimately, open development allows more community engagement to provide feedback and contributions, making the stack better for all (Landwerth, 2014).

Microsoft Open-sourcing Powershell Microsoft open-sourced PowerShell to make it more widely available and helpful for system administrators and developers who work with multiple operating systems. By making it open source and available cross-platform, more people can use PowerShell to automate tasks and manage their diverse computing environments that include different OSes (Foley, 2016).

NVIDIA Open-sourcing PhysX NVIDIA open-sourced PhysX because physics simulation has “dovetails” with AI, robotics, and computer vision. NVIDIA considered physics simulation “foundational for so many things” and open-sourcing it would allow it to be developed and applied more widely than NVIDIA could do alone (Lebaredian, 2018).

Microsoft Open-sourcing Live Writer Microsoft open-sourced Live Writer to allow the community to continue to develop and improve it, as Microsoft no longer had the resources to maintain it (ARS STAFF, 2015).

The primary motivation for open-sourcing previously proprietary software is to allow for increased community engagement, feedback, and contributions, making the software better for all.

However, open-sourcing previously proprietary software does have some technical considerations. For example, the software must be properly documented, the code must be clean and well-structured, and the software must be properly licensed. We saw from the interviews that these technical considerations are usually a deal-breaker and the reason why organisations do not open-source their software.

Hiring

To reduce the risk of using OSS, organisations should hire internal staff to manage OSS. Develop in-house proficiency and aim to reduce reliance on external service providers by cultivating organisational expertise to oversee projects. This approach will enable quicker deployment of upgrades and enhancements (Team Srijan, 2010).

Smaller organisations might not necessarily hire new personnel but rather delegate the responsibility to an existing staff member who is recognised as a subject-matter expert within the specific area (Interviewee 8, 2024). This approach is viable because, as noted, “In general, open source developers are experienced users of the software they write. They are intimately familiar with the features they need, and what the correct and desirable behavior is” (Mockus et al., 2000). This innate understanding significantly mitigates one of the primary challenges in large software projects: the lack of domain knowledge. Consequently, smaller organisations can effectively rely on their in-house experts, capitalising on their comprehensive knowledge and experience.

Hiring is a significant investment, and it is not always feasible for smaller organisations to hire new personnel to manage OSS.

Culture

Open source has always been deeply rooted in culture, originating as a grassroots movement advocating for software freedom. Culture encompasses a broad spectrum, and individuals and organisations get involved in open source for various reasons.

When inquiring about the cultural drivers behind companies’ engagement in open source, innovation emerged as the leading motivation, with 84% of respondents to Fintech Open Source Foundation State of Open Source Survey affirming it as a critical factor (Ellison et al., 2021).

Uber instituted internal standards for governing and incentivising contributing upstream and back to the community to encourage ongoing, regular involvement in open-source projects. Contributing back to the open-source community is one of the best ways to help ensure open-source project sustainability (Uber Case Study n.d.).

A big part of culture is fostering an environment where developers feel comfortable taking an unconventional route to solve a problem (Autodesk Case Study n.d.), and where they are encouraged to share code and collaborate with others (Abernathy, n.d.).

An organisation’s culture is a significant factor in the adoption of open-source software, one that is quite hard to quantify and measure. There was very little mention of culture in the best practices we reviewed, and from our interviews, we found that there isn’t an “open-source” culture but rather a more collaborative and innovative culture that is conducive to using open-source software.

Contributions

Active engagement in the open-source community offers substantial rewards. When you contribute to the projects your organisation utilises, you’re boosting its reputation and playing a significant role in steering the software’s development path. Such proactive participation can lead to the development of features that are beneficial to your organisation, as well as the resolution of issues that impact your operations.

Contributing to the open-source ecosystem extends beyond just coding. Offering documentation, identifying bugs, and aiding in translations represent other crucial ways to contribute. Through these contributions, you help cultivate a mutually beneficial relationship with the community (Yborra, 2024).

Contributing also helps keep the open-source component active and maintained. In recent years, a notable challenge has been the shortage of contributions or ongoing maintenance for projects, affecting even highly utilised packages like gorilla/mux, which have struggled to secure maintainers and consequently, had to archive their projects (Norblin, 2023).

Another way to contribute is through financial support. A recent survey by DigitalOcean found that only 20% of respondents had been paid for their contributions to open source, while 53% agreed or strongly agreed that individuals should be compensated for their work (DigitalOcean, 2022). This suggests that there is room for improvement in financial support for open-source contributors (Tandochain, 2023).

More work is being done to support financial contributions to open source; notable examples include GitHub Sponsors and Open Source Collective, both offering platforms for financial support of open-source projects (Open Source Collective 2024; Explore GitHub Sponsors 2024).

Contributing to the open-source community is a best practice for managing the security risks associated with OSS. Most of the literature we reviewed mentioned contributing to the open-source community as a best practice. However, from our interviews, very few organisations contribute to the open-source community, and those that do do so in a limited capacity.

Hosting Events

Hosting events is a great way to engage with the open-source community. Events can be an excellent way to engage with the open-source community, and they can take many forms, from hackathons to conferences to meet-ups.

Schumacher, 2022, went as far as to say that corporate organisations looking to lead in open source should host events, as they are a great way to engage with the community and build relationships.

However, our interviews found that hosting events is not widely used in practice and that the best practices landscape does not reflect the real-world use of hosting events. Companies do send their employees to events, but larger organisations typically host events, as smaller organisations do not have the resources to host events.

Conclusion

In conclusion, fostering collaboration with the open-source community is essential for organisations that rely on open-source software. By engaging with the community, contributing back to the community, and hosting events, organisations can enhance their projects and the broader ecosystem.

However, there are challenges to fostering collaboration with the open-source community, such as the need to hire internal staff to manage OSS, the technical considerations of open-sourcing previously proprietary software, and the cultural challenges of adopting open-source software.

Despite these challenges, the benefits of fostering collaboration with the open-source community are significant. By following best practices for engaging with and contributing to the open-source community, organisations can enhance their projects and the broader ecosystem, ultimately leading to better software for all.

Best Practices for Managing the Security Risks Associated with Open-Source Software in 2024

Harry — Fri, 24 May 2024 08:00:00 +0000

This post originally appeared on my blog

Welcome to this four-part mini-series on open-source best practices. In this series, we will discuss the best practices for adopting open-source software (OSS) into your organisation, managing OSS dependencies daily, fostering collaboration with the OSS community, and finally, my opinion on which best practices are best adopted for organisations of all sizes and industries.

This article will discuss the best practices for managing open-source software, including the internal considerations organisations must make, the metrics to judge the quality of OSS components, the approval process, and the licensing of OSS components.

Open source software (OSS) is a critical component of modern software development. It allows developers to leverage existing code, reduce development time, and focus on building new features.

However, OSS also introduces security risks. Vulnerabilities in OSS components can expose organisations to cyber threats, data breaches, and financial losses. To mitigate these risks, organisations must adopt best practices for managing the security risks associated with OSS.

In this blog post, we explore best practices for managing the security risks associated with OSS.

We review the literature, conduct interviews with industry experts, and analyse case studies to identify key strategies for securing OSS components. Our research highlights the importance of regular vulnerability assessments, tooling, continuous monitoring, binary repositories, risk management and mitigation, static application security testing, and forking as essential practices for managing the security risks associated with OSS.

Regular vulnerability assessments

Regular vulnerability assessments identify, prioritise, and apply patches to systems and software to address vulnerabilities. As the OSS landscape constantly evolves, organisations encounter thousands of new vulnerabilities. Yet, many businesses do not have a robust patch management strategy in place and fail to implement critical patches promptly, leaving them vulnerable to security breaches (Hackerone, 2024).

With an estimated 84% of open source components having at least one vulnerability (Sawers, 2021), regular vulnerability assessments are essential for managing the security risks associated with OSS.

A rough overview of the process is as follows (Hackerone, 2024):

Initial Preparation – Defining the scope and goals of the vulnerability testing to ensure a targeted and practical assessment.
Vulnerability Testing – Running automated tests to identify vulnerabilities within the systems included in the predefined scope.
Prioritise Vulnerabilities – Assessing which vulnerabilities are most critical, requiring immediate attention, and evaluating their potential business impact.
Create Vulnerability Assessment Report – Produce a detailed report outlining medium and high-priority vulnerabilities found and recommended remediation strategies.
Continuous Vulnerability Assessment – Conducting scans for vulnerabilities on a continuous basis to verify if previous vulnerabilities have been effectively remediated and to discover new ones.

Regular vulnerability assessments are frequently mentioned in the best practices for managing the security risks associated with OSS. Without automated vulnerability assessments, it is difficult to keep up with the large number of OSS components used in modern software systems and the many vulnerabilities discovered each year. Therefore, with automation, regular vulnerability assessments are a best practice for managing the security risks associated with OSS.

Tooling

Tooling is the functionality that allows for automating the management of OSS components. It can be very useful as a way to manage the large number of OSS components used in modern software systems without introducing unnecessary manual work. Any software project that actively works to reduce security vulnerabilities is less risky. Therefore, tools that aid in identifying security vulnerabilities within software projects are critical (Department of Defense, 2022).

There are three tools found from our literature review that are considered “leaders”; Snyk, Sonatype, and Synopsys (Black Duck) (Worthington et al., 2023). These tools are designed to help organisations manage the security risks associated with OSS. They can be used to automate vulnerability assessments, manage licensing, enforce an OSS policy, and generate SBOMs. With this in mind, due to the diversity of OSS components, there is no one-size-fits-all solution (Wen et al., 2019).

We consider tooling to be a best practice for managing the security risks associated with OSS. It enables organisations of all sizes to manage many OSS components used in modern software systems without introducing unnecessary manual work. There is a lot of research into tooling, and a wide range of tools are available. We found that many tool-promoting papers and articles are written by the companies that produce the tools, so it is down to the organisation to evaluate the tools and decide which one is best for them.

Continuous monitoring

Modern software applications are built using a complex web of dependencies, including open-source libraries and frameworks. This process is typically recursive, with each dependency having its dependencies, and so on (Dietrich et al., 2023). Infamously, this web of dependencies can lead to security vulnerabilities, as seen in the equifax data breach (Fruhlinger, 2020) and the log4j vulnerability (Gallo, 2022).

Continuous monitoring is the process of monitoring OSS components for vulnerabilities and other security risks on an ongoing basis. These can be manual but are often automated and part of an organisation’s CI/CD pipeline. It is crucial for application developers to detect dependencies on vulnerable libraries as soon as possible, to assess their impact precisely, and to mitigate any potential risk (Ponta et al., 2020).

Now more than ever, developers are using automation to secure their dependencies; in 2023, developers merged 60% more pull requests from GitHub’s Dependabot, a tool that automates dependency updates, than in 2022 (Daigle, 2023).

Software composition analysis (SCA) can be used to conduct regular vulnerability assessments, can be implemented as part of the continuous integration/continuous deployment (CI/CD) pipeline, and can be used to enforce an OSS policy (Alvarenga, 2023a). SCAs will provide an inventory of all open-source components used in a project, including their versions and licenses, and will also identify any known vulnerabilities in these components; effectively, producing an SBOM (Alvarenga, 2023a).

Due to the dynamic nature of the open-source ecosystem, continuous monitoring is essential for managing the security risks associated with OSS. An SCA is a great way to monitor OSS components continuously and can be implemented as part of the CI/CD pipeline and can be used to enforce an OSS policy. It is a best practice for managing the security risks associated with OSS.

Binary repositories

A binary repository manager is a tool designed to store and manage binary files (the compiled version of your source code) and their metadata. Some popular binary repository managers include JFrog Artifactory, Sonatype Nexus Repository, and GitHub Packages.

A binary repository manager is indispensable for managing open-source components effectively. It facilitates the caching of local copies of these elements, ensuring that frequently used packages remain accessible even during downtimes of external repositories. This capability is essential for maintaining continuous project development and operations without interruptions.

Moreover, such a tool distinguishes between approved third-party artefacts and those pending approval, significantly improving the management and visibility of open-source components within projects. It also allows access to specific libraries to be excluded and limited, thereby safeguarding projects from incorporating non-compliant artefacts (Wainstein, 2018).

By caching versions of all open-source software components, you also safeguard against incidents like the left-pad debacle, where the withdrawal of a minor package from the npm package manager (a package manager for the JavaScript programming language) resulted in widespread failure in numerous projects (Williams, 2016).

There is a licensing concern, specifically with GPL-licensed open-source software. The GPL license requires that any software distributed that includes or is derived from GPL-licensed code must itself be available under the GPL. If a binary repository contains GPL software and is shared externally, the entire repository may need to be open-sourced. For example, Microsoft released Hyper-V drivers with GPL-licensed binaries, leading to Microsoft having to open-source the drivers in question (Linux Network Plumber, 2009).

Binary repositories appear to be a polarising issue in the literature. The Department of Defense, 2022 recommended binary repositories as a best practice for managing the security risks associated with OSS, whereas Anand, 2023 (Salesforce) mentioned securing a repository as a best practice, but didn’t mention binary repositories specifically.

Risk management and mitigation

Risk management and mitigation is the process of identifying, assessing, and prioritising risks, and then taking steps to reduce or eliminate them.

According to the Open Worldwide Application Security Project (OWASP), the most significant risk associated with open source software is “Vulnerable and Outdated Components” and has moved from 9th to the 6th most critical risk to web applications (OWASP, 2020).

OSS vulnerabilities grew by 50% year-on-year — from just over 4,000 in 2018 to over 6,000 in 2019 (Zorz, 2020).

One reason for the increase in vulnerabilities is the growing number of open source components being used in software projects from developers utilising the speed and convenience of open source, especially under time pressure, without considering the security implications. This highlights the need for formal processes to manage the risks. Typically, these correlated to implementing some continuous monitoring and regular vulnerability assessments, as outlined in Section 4.2.1 and 4.2.3 (Anand, 2023, Chandler et al., 2022, Mathpati, 2023). Such efforts should avoid being overly laborious and manual to ensure they do not frustrate developers and slow down the development process (Contrast Security, 2020).

Like those outlined in this report, the most effective way to mitigate OSS risks is to follow best practices (Chandler et al., 2022).

Interestingly, our interviews and case studies found that the approach to risk and risk mitigation varied from organisation to organisation. Larger organisations seemed more proactive in mitigating risk, with dedicated teams and processes in place. Whereas smaller organisations were more reactive, only taking action when a risk materialised and looking to improve their processes after the event. This suggests there is a gap in the best practices landscape, as it doesn’t consider the different approaches to risk management and mitigation based on the size of the organisation.

Static application security testing

Static application security testing (SAST) is a method of testing the security of an application by examining its source code, byte code, or binary code for security vulnerabilities.

Frequent scanning and security assessment of your repository aids in detecting security concerns at early stages, minimising the chance of significant security problems developing later on. Static code analysis tools can pinpoint typical security weaknesses, and dependency vulnerability scanners can highlight any recognised problems in your project’s dependencies (Anand, 2023). SAST goes hand in hand with the continuous monitoring of OSS components, as it allows for detecting vulnerabilities in OSS components as soon as possible, allowing for their impact to be assessed and mitigated, see Section 4.2.3.

From our research, SAST is only suggested by Salesforce as a best practice. However, current tooling is comprehensive and can be used to automate the continuous monitoring of OSS components, as outlined in Section 4.2.2. Therefore, SAST can be considered a technical implementation of continuous monitoring.

Forking

Forking is the process of creating a new project based on an existing project. It allows you to monitor alterations in open-source components, as there will always be a connection to the original repository. One of the key advantages mentioned in the definition of open-source is the explicit permission granted to duplicate and independently alter the source code of an open-source project as desired (The Open Source Definition 2024).

However, it’s important to be mindful that opting to create a private fork entails the responsibility of integrating any updates from the upstream version of the component. This responsibility grows as the differences between the forked components and the original increase. An ideal situation for forking is when you take an open-source component from a project that is unlikely to receive many updates in the future (Wainstein, 2018).

The process often includes creating a distinct OSS developer community (OSS-DC) for the forked version, allowing the organisation to focus on modifying or enhancing specific components of the software—usually those deemed critical for their purposes (Méndez-Tapia et al., 2021).

Forking does get mentioned in varying mediums as a best practice. However, the added responsibility of integrating updates from the upstream version of the component is often overlooked, and from our interviews, we found that forking is not widely used in practice.

Conclusion

In this blog post, we explored best practices for managing the security risks associated with open-source software. We reviewed the literature, conducted interviews with industry experts, and analysed case studies to identify key strategies for securing OSS components.

Our research highlights the importance of regular vulnerability assessments, tooling, continuous monitoring, binary repositories, risk management and mitigation, static application security testing, and forking as essential practices for managing the security risks associated with OSS.

By adopting these best practices, organisations can reduce the likelihood of security breaches, data loss, and financial losses associated with OSS vulnerabilities. We recommend that organisations implement these practices to enhance the security of their software systems and protect their data and assets.

Best Practices for Adopting Open-Source Software in 2024

Harry — Fri, 17 May 2024 08:00:00 +0000

This post originally appeared on my blog

The second and third parts are already readily available on my website here.

This article will discuss the best practices for adopting open-source software, including the internal considerations organisations must make, the metrics to judge the quality of OSS components, the approval process, and the licensing of OSS components.

Introduction

Open-source adoption includes the strategic and tactical considerations of introducing open-source software to your organisation. A robust adoption process is critical to ensuring new open-source components are correctly evaluated before being integrated into the organisation’s systems.

This article lists all the general best practices for adopting OSS. It is by no means an opinion piece.

OSS Policy

An OSS policy is a formal document that outlines the rules and guidelines for using open-source software within an organisation.

Our literature review found an OSS policy to be a recurring theme in the best practices for adopting open-source software. Best practices recommend that organisations have a formal OSS policy in place.

An OSS policy should (Hammond, 2009);

Be concise
Developer consumable
Involve general counsel early and often
Classify OSS license types
Take control of the process
Measure internalisation
Be revisited and revised on a regular basis

It should include the goals for OSS adoption, the acquisition process, the rules for the business case, and guidelines for the appropriate use of OSS (Hammond, 2009).

Lacking a formal policy for OSS usage and adoption, which typically includes licencing guidelines, can lead to legal and security risks (Schofield, 2008).

However, a policy alone is not enough. It is essential to ensure that all employees understand and follow the policy. Most importantly, the policy steps should apply to the organisational context (Osborne et al., 2023).

Open-source policies are a formal way for an organisation to set guidance and rules for using open-source software. As we will see later, most developers use their own discretion rather than a formal organisational process, which can lead to inconsistencies and licensing risks.

A policy is an excellent way for an organisation to ensure that all developers are following the same process, the correct stakeholders are included in the adoption process, and that the process is tailored to the organisation’s needs.

Maturity Models

Maturity models are a way to evaluate the quality of open-source software components. They can play a key role in the adoption process, as they allow for evaluating OSS components before they are integrated into the organisation’s systems. We found that there are several maturity models for evaluating OSS components; however, there wasn’t a consensus on which model was the best.

Fundamentally, the adoption of OSS components differs from the adoption of proprietary software due to the fact OSS is stored in public repositories, accessible by anyone, and is often developed by a community of developers (Umm-e-Laila et al., 2016). Due to this, the quality of open-source software can be very different from one product to another; therefore, evaluation methods have been developed to evaluate OSS components properly.

The most popular OSS evaluation methods are the Capgemini-Open Source Maturity Model (C-OSSM), Navicasoft-Open Source Maturity Model (N-OSSM), Qualification and Selection of Open Source (QSOS), Open Business Readiness Rating (Open BRR) and Easiest Open Source (E-OSS) (Umm-e-Laila et al., 2016).

None of those interviewed were familiar with these models, let alone using them in their organisation. This suggests that these models are not widely used in practice. We also found, from our literature review, that there is no consensus on which model is the best, with Umm-e-Laila et al., 2016 going as far as to say that “in order to take advantage of OSS properly, it is recommended to propose a new framework/model that will eliminate the weakness of all models.”.

We can conclude that while maturity models are an excellent way to evaluate OSS components, there is no consensus on which model is the best, and they are not widely used in practice. From our interviews, we found that most developers judge the maturity of OSS components based on their own discretion, usually stemming from their own experiences and the experiences of their colleagues.

Trust in Open-Source Software

Trust in OSS is a critical concept when adopting OSS components. How does one come to trust an OSS component? More often than not, “there is no sound basis for trust in the Software Ecosystems (SECO) hubs”, with trust being considered “founded or unfounded” (Hou et al., 2022).

Trust is subjective and based on the end-developer’s perception and intrinsic and extrinsic motivations. When adopting a new OSS component, the developer will consider intrinsic factors such as:

Source code
Reliability
Security

Extrinsic factors such as:

Documentation
Structural assistance
Cost

Quality is the most discussed factor in trustworthiness, with most developers considering the quality of the OSS component as the most critical factor in trustworthiness (Hou et al., 2022).

From our literature review and the interviews, we found that each developer uses their own trust model and that there is no documented process for evaluating the trustworthiness of OSS components within an organisation.

Outside of academic papers, trustworthiness wasn’t mentioned in any of the best practices we reviewed.

This is a significant gap in the best practices landscape, as trust plays a vital role in adopting OSS components.

Approval process

The approval process is a formal process for evaluating and approving OSS components before they are integrated into the organisation’s systems. It’s an opportunity for the organisation to evaluate the OSS component and ensure that it meets the organisation’s needs and standards, and engage relevant stakeholders in the approval process.

When adopting OSS, there are several “deal-breakers” that can prevent the adoption of an OSS component. Outlined by Spinellis, 2019, these include:

Lacking functionality
Incompatible license
Nonfunctional properties

Businesses should approve OSS based on technical needs, licensing, longevity, and business factors like cost and benefits. Yet, businesses must balance risk with commercial advantages and be willing to adapt their methods quickly. The approval process should be formal, with a clear set of guidelines and rules, and should be managed by a central team and engage relevant stakeholders (Butler et al., 2022).

When selecting an OSS component, ensure it demonstrates the following security and maintenance best practices (Department of Defense, 2022):

Detection Tool Usage: Actively employs detection tools within its integration pipeline and externally.
Vulnerability Reporting Transparency: Has a clear process for the open reporting and documentation of security vulnerabilities.
Security Review History: Possesses a record of security reviews.
Cybersecurity Testing: Undergoes regular cybersecurity assessments, notably third-party audits, to verify security measures.
Issue Resolution: Demonstrates a commitment to promptly addressing and resolving reported issues.
Timely Vulnerability Remediation: Shows a track record of quickly remedying vulnerabilities.

We found that an approval process was a recurring theme in the best practices for adopting open-source software. Every interviewee demonstrated some approval process for the adoption of OSS components. Smaller companies tended to build a minimal viable product (MVP) with the OSS in question; larger teams had a multistep process that involved multiple stakeholders.

The US Department of Defense, 2022 recommendations primarily emphasise the quantifiable attributes of OSS dependencies, including security features and measurable performance metrics. In contrast, Butler et al., 2022 analysis portrays the approval process as somewhat ambiguous, describing it as more reliant on assessment and discretion than on strict, predefined criteria.

To conclude, the approval process for adopting open-source software is dynamic and customised to each organisation’s specific requirements, yet it remains a universally acknowledged standard that all entities conform to.

Software Bill of Materials

Software Bill of Materials (SBOM) is a formal list of components used in a software system. It allows for tracking OSS components and their dependencies, which is essential for managing the security and compliance risks associated with OSS.

Since 2018, SBOMs have surged in popularity, driven in part by a collaborative effort headed up by the American National Telecommunications and Information Administration’s (NTIA) multi-stakeholder process (National Telecommunications and Information Administration, n.d.). This work culminated in the US government’s issuance of an executive order mandating the use of SBOMs in all federal software (The White House, n.d.) in direct response to the SolarWinds supply chain attack (Schwartz, 2021).

The two primary standards for SBOM are SPDX and CycloneDX.

SPDX17 is an open standard developed by the Linux Foundation to communicate details of a SBOM, including components, licenses, copyrights, and security references, recognised internationally as ISO/IEC 5962:202118 (System Package Data Exchange (SPDX®) 2024). CycloneDX19, originating from the Open Web Application Security Project (OWASP) community, is an SBOM standard crafted for application security and supply chain component analysis, now extended to encompass a broader array of applications such as software-as-a-service BOM (SaaSBOM) (CycloneDX 2024).

Both formats are popular, yet organisations are advised to select the format that most aptly meets their specific requirements (Alvarenga, 2023a).

At a minimum, an SBOM must contain three categories (United States Department of Commerce, 2021);

Data fields Data fields must be standardised and contain essential information on each component for effective tracking and management. The aim is to ensure these components can be easily identified throughout the software supply chain, linking them to other valuable data sources (United States Department of Commerce, 2021). Naming conventions should try and follow the guidance outlined by NTIA Multistakeholder Process on Software Component Transparency Framing Working Group, 2021.

Automation Support Automation support is essential for the effective use of SBOMs, as it allows for the automatic generation and updating of SBOMs, which is essential for managing the large number of OSS components used in modern software systems.

The data formats that are being used to generate and consume SBOMs are:

Software Package Data eXchange (SPDX)
CycloneDX13
Software Identification (SWID) tags

Practices and Processes Practices and processes are essential for the effective use of SBOMs, as they allow for the effective management of OSS components and their dependencies, necessary for managing the security and compliance risks associated with OSS.

There are two elements that should be explicitly stated in the SBOM:

Frequency – When a software component is updated or new information about its components is discovered, the supplier must issue an updated SBOM to reflect these changes (United States Department of Commerce, 2021).

Depth – An SBOM must include all primary components and their transitive dependencies, ensuring top-level dependencies are detailed enough to identify all subsequent dependencies recursively (United States Department of Commerce, 2021).

Licencing

The licensing of OSS components is a crucial consideration when adopting OSS as it can significantly impact the organisation’s ability to use the OSS component and considerably affect the organisation’s ability to distribute the software that uses the OSS component.

As of 2022, 78% of open source components are under permissive licenses, which allow users wide freedom to use, modify, and distribute the software, with minimal requirements on how it can be redistributed. Meanwhile, copyleft licenses, which require that any modified versions of the software also be distributed under the same license terms, make up the remaining 22% of open source components (Murray, 2022).

The Apache License is the most popular OSS license, with 30% of all OSS projects using it (Murray, 2022).

Larger organisations (>5000 employees) are more likely to have an internal legal team familiar with OSS licensing when compared to small organisations (100 employees) 31.46% vs 22.30%, respectively (OpenLogic, 2023).

To help reduce the risk of legal problems arising with the use of open-source software (Mathpati, 2023), organisations should watch over all open-source code being brought into the organisation, ensuring that licence requirements are met. To achieve this, there are several tools available, such as (The Linux Foundation, 2024):

Black Duck Protex – A fee-based tool for license compliance and open source management.
Copyright review tools – Command line utilities for copyright file management.
FOSSA – Automates code dependency tracking and license compliance.
FOSSology – An open-source toolkit from the Linux Foundation for license compliance featuring a web UI for compliance workflows.

Managing the licensing of OSS components is a key consideration when adopting OSS. Large organisations often have the advantage of an internal legal team to navigate open-source software (OSS) licensing, a resource that smaller organisations lack. Therefore, relying on internal legal teams for OSS license management cannot be deemed a universally applicable best practice, as it disadvantages smaller entities. The easiest way to manage your OSS licensing is to use a tool that can automatically track and manage the licensing of OSS components, as outlined by The Linux Foundation, 2024.

Conclusion

In this article, we have discussed the best practices for adopting open-source software, including the internal considerations organisations must make, the metrics by which to judge the quality of OSS components, the approval process, and the licensing of OSS components.

The adoption of open-source software is a complex process that requires careful consideration of the organisation’s needs and the OSS components being adopted. Having a robust adoption process is critical to ensuring that new open-source components are correctly evaluated before being integrated into the organisation’s systems.

We can see that the current literature is broad and extensive, with many different opinions on the best practices for adopting open-source software. However, there are some recurring themes, such as the need for an OSS policy, the importance of an approval process, and the need to manage the licensing of OSS components.

We hope that this article has provided you with some valuable insights into the best practices for adopting open-source software and that you can use this information to improve your organisation’s OSS adoption process.

Implementing Sign in with Apple with Expo

Harry — Fri, 18 Jun 2021 14:17:40 +0000

This post originally appeared on Forward Digital's blog

Making your whole authentication process as seamless a possible is vital to ensuring that any potential users don't get put off by filling out a long-winded create account form.

Back in WWDC 2019 Apple introduced Sign in with Apple, this allows users to authenticate with their Apple Id with forced 2FA. A seamless solution to getting users into your app.

Today I will be running through the steps to integrate this into your managed Expo project using a Nest.js backend to authenticate that user and handle the user accounts.

Before we begin I will make the assumption the reader is familiar with:

Expo
Nest.js
JavaScript
TypeScript

Although the backend code will be using Nest, a lot of the code will be transferrable to any other Node backend framework you may be using.

What do we need?

Expo

For our managed Expo project, we will need to add:

"expo-apple-authentication" by running: expo install expo-apple-authentication which will install the version compatible with your SDK version. This tutorial is written for SDK41.

There are extra configuration steps you need to follow that can be found here: https://docs.expo.io/versions/latest/sdk/apple-authentication/#configuration

Backend

For our Nest.js backend we need to add:

https://github.com/auth0/node-jsonwebtoken (npm install jsonwebtoken)
- We use this to verify the JWT token was signed by Apple's public key
https://github.com/auth0/node-jwks-rsa (npm install jwks-rsa)
- We use this to retrieve the signing keys from Apple's keys API endpoint and to retrieve Apple's public key
https://github.com/auth0/jwt-decode (npm install jwt-decode)
- We use this to decode the JWT generated by the Sign in with Apple button

A brief overview of the auth flow

A user presses the Sign in with Apple button and logins in through the native Apple modal
Once authorised through 2FA on the device, will return to us a JWT token with a nullable name value. (plus other values, we don't need to worry about these)
Determine if the user is attempting to log in or create an account
We send the JWT token off to our backend with the user's name (if they are creating an account)
We then decode the JWT token, get Apple's public key and verify the JWT token was signed by Apple
Validate the token against our bundle id
At this point, we can safely assume they are authenticated with Apple and we can either login them in through our backend auth or create an account on our database using the values we have from step 2.

An overview of the Expo package

Expo's expo-apple-authenticator is in effect a one-stop-shop with everything you need for your app.

The Sign-in with Apple button will only appear on iOS devices, so make sure to consider this when building the UI.

The package contains the functions for displaying the modal and even comes bundled with the button itself, although this isn't required if you want to use a custom button.



<AppleAuthentication.AppleAuthenticationButton
    buttonType={AppleAuthentication.AppleAuthenticationButtonType.CONTINUE}
    buttonStyle={AppleAuthentication.AppleAuthenticationButtonStyle.BLACK}
    cornerRadius={BUTTON_BORDER_RADIUS}
    style={{
        width: BUTTON_WIDTH,
        height: BUTTON_HEIGHT,
    }}
    onPress={async () => {
        //
    }}
/>

There's also plenty of style options so the button fits in line with other buttons on the app. BUTTON_BORDER_RADIUS, BUTTON_WIDTH and BUTTON_HEIGHT are custom values I declared for all buttons on the app.

When a user presses the Sign in with Apple button they will be presented with a native modal that sits on top of your app, with module fields dependent on the scope you requested.

You have access to two scopes:



const credential = await AppleAuthentication.signInAsync({
    requestedScopes: [
        AppleAuthentication.AppleAuthenticationScope.FULL_NAME,
        AppleAuthentication.AppleAuthenticationScope.EMAIL,
    ],
});

If the user doesn't cancel the modal, the response will look something like:



export type AppleAuthenticationCredential = {
  user: string;
  state: string | null;
  fullName: AppleAuthenticationFullName | null;
  email: string | null;
  realUserStatus: AppleAuthenticationUserDetectionStatus;
  identityToken: string | null;
  authorizationCode: string | null;
};

We only really care about 4 fields that get returned:

user - this is a stable unique identifier that will persist between app versions/device changes for the app.
fullName - The user's name object, includes family name, given name, nickname etc.
email - The user's email, used for the backend to create a user account
identityToken - A JWT token that stores information about the audience, issuer and user.

Email and name access

Be warned!

The email and fullName will only be populated ONCE. The first time they press the button, this applies even if they change their device or update the app. A user's email is still available in the JWT token, but once fullName gets pulled down the first time, you can never request it again and subsequent requests will return null.

Is the user creating an account or logging in?

As this button will act as both a create account button and a login button, how do we determine which the user is trying to do?

As mentioned above we know the first time they press the button, email and name values will be populated. If the name and email values are undefined, it is a fair assumption this is the users first time on the app and they are trying to create an account.

What is the problem with this? If the user signs in through the button, pulls down the name and email values but, say, the device crashes/the subsequent API call fails or they kill the app; they can't create an account through the Sign up with Apple button.

One mitigation of this is to cache the name value to the phone's storage if it is populated and so if any edge-case happens that causes the device to lose the name then we can fall back to the cache.

Save the name value to the cache with the key value of the credential.user as we know this is a stable unique identifier.



// Expo Apple button
onPress={async () => {
    try {
        const credential: AppleAuthenticationCredential = await AppleAuthentication.signInAsync({
            requestedScopes: [
                AppleAuthentication.AppleAuthenticationScope.FULL_NAME,
                AppleAuthentication.AppleAuthenticationScope.EMAIL,
            ],
        });
        const cachedName: string = await Cache.getAppleLoginName(credential.user);
        const detailsArePopulated: boolean = (!!credential.fullName.givenName && !!credential.email);
        if (!detailsArePopulated && !cachedName) {
            await login(credential.identityToken);
        } else if (!detailsArePopulated && cachedName) {
            await createAccount(cachedName, credential.user, credential.identityToken);
        } else {
            await createAccount(
                credential.fullName.givenName, credential.user, credential.identityToken,
            );
        }
    } catch (error) {
        if (error.code === 'ERR_CANCELED') {
            onError('Continue was cancelled.');
        } else {
            onError(error.message);
        }
    }
}}

You can see in the above code snippet we are:

Requesting the user's email and name
Retrieving the name value stored in the cache (if any)
If both the cache and credential values are undefined, we are assuming they have already used the button successfully to create an account, so start the login flow
If the cached name is declared but the credential details are not, we are assuming something happened between requesting the values from Apple and the server creating them an account, so create an account using the cached name
Otherwise, create an account with the requested values that are populated.

This is not bulletproof! If the user requests the details but the cache gets cleared/they change devices before creating an account, they will not be able to create an account using this flow. We hedged a bet that this was edge case enough to warrant the risk.

Decoding and validating the JWT token

If the user is creating an account or logging in, we need to handle decoding and validating the JWT token sent in the payload to our Nest.js API.

Using Nest.js we need to create two new endpoints to handle creating an account and logging in. Both these endpoints I will add a respective guard to. It's in this guard that we call the Apple auth strategy code.

From experience, I couldn't use the existing login and create account endpoints as these require the information we receive after decoding and validating the token, not once the request is made on the device.



public async canActivate(context: ExecutionContext): Promise<boolean> {
    const request = context.switchToHttp().getRequest();
    const response = context.switchToHttp().getResponse();
    const token: string = <string>request.body.identityToken;
    const jwt: JwtTokenSchema = await this.apple.ValidateTokenAndDecode(token);

    try {
        const [sessionId, user]: [string, IUser] = await this.login.AttemptLogin(jwt.email);
        if (sessionId && user) {
            request.user = user;
            response.set('Auth-Token', sessionId);
            response.set('Access-Control-Expose-Headers', 'Auth-Token');
            return true;
        }
    } catch (error) {
        throw new HttpException(`Validation failed for login. ${error.message ?? ''}`, HttpStatus.UNAUTHORIZED);
    }

    return false;
}

The most important line is:



const jwt: JwtTokenSchema = await this.apple.ValidateTokenAndDecode(token);

As we are injecting calling the Apple strategy to do most of the logic. The logic is as follows.

Getting Apple's public key

Before we use the JWT token, we need to make sure that it was signed by Apple's private key. To do that, we need Apple's public key to verify the signature.

Firstly we need to decode the JWT token sent by the client and extract the kid value found in the token's header.



import jwtDecode, { JwtHeader } from 'jwt-decode';

// rest of file 

const tokenDecodedHeader: JwtHeader & { kid: string } = jwtDecode<JwtHeader & { kid: string }>(token, {
    header: true,
});

The "kid" (key ID) Header Parameter is a hint indicating which key was used to secure the JWS. This parameter allows originators to explicitly signal a change of key to recipients. The structure of the "kid" value is unspecified. Its value MUST be a case-sensitive string. Use of this Header Parameter is OPTIONAL.
When used with a JWK, the "kid" value is used to match a JWK "kid" parameter value.

Then we need to perform an HTTP request to Apple's auth/keys public endpoint to return an object which contains an array of keys. Read more about this here.



const applePublicKey: { keys: Array<{ [key: string]: string }> } = await this.api.Get(
    'https://appleid.apple.com/auth/keys',
);

(this.api.Get is a HTTP client we built for Nest, find out more here)

This endpoint will return a JSON Web Key Set which will look like:



{
  "keys": [
    {
      "kty": "RSA",
      "kid": "AIDOPK1",
      "use": "sig",
      "alg": "RS256",
      "n": "lxrwmuYSAsTfn....",
      "e": "AQAB"
    },
        { 
            //
        } 
  ]
}

As the keys array typically has more than one element, we need to filter the keys array to the one that has a matching kid value from the decoded header of the JWT token.



const kid: string = tokenDecodedHeader.kid;
const sharedKid: string = applePublicKey.keys.filter(x => x['kid'] === kid)[0]?.['kid'];

Now we need to get Apple's public key using the kid value. To do this we use the package mentioned above: jwks-rsa.



const client: jwksClient.JwksClient = jwksClient({
    jwksUri: 'https://appleid.apple.com/auth/keys',
});
const key: jwksClient.CertSigningKey | jwksClient.RsaSigningKey = await client.getSigningKey(sharedKid);
const signingKey: string = key.getPublicKey();

If all is dandy, we should now have Apple's public key that we can use to verify our JWT token was signed with the same key. Thanks to jsonwebtoken:



try {
    const res: JwtTokenSchema = <JwtTokenSchema>jwt.verify(token, signingKey);
} catch (error) {
    // token is invalid
}

I wrote my own JwtTokenSchema for type safety:



export type JwtTokenSchema = {
    iss: string;
    aud: string;
    exp: number;
    iat: number;
    sub: string;
    nonce: string;
    c_hash: string;
    email: string;
    email_verified: string;
    is_private_email: string;
    auth_time: number;
};

Now we have the decoded JWT token we just need to perform some basic validation before we proceed with the rest of the auth process.



private ValidateToken(token: JwtTokenSchema): void {
    if (token.iss !== 'https://appleid.apple.com') {
        throw { message: 'Issuers do not match!' };
    }
    if (token.aud !== this.audience) {
        throw { message: 'Audiences do not match!' };
    }
}

this.audience is your app's bundle ID. This will change depending on if you are on Expo or in production. I do a ternary in the class' constructor to set this:



this.audience = this.isInProd ? 'co.repetitio.repetitio' : 'host.exp.Exponent' // this will be your bundle id, found in app.json

Now we should have the decoded token that we have validated against Apple's public key and can safely assume that the user is whom they say they are and are trying to login into the correct app. The full class is linked at the bottom.

From here you are free to proceed to the log in process or create them an account.

I have uploaded the code to a public repo so you can take a deeper look here.

Thanks for reading!

When to use TailwindCSS over CSS

Harry — Tue, 15 Jun 2021 15:04:00 +0000

This post originally was shared @ Forward Digital

I find the premise of TailwindCSS exciting, an un-opinionated CSS framework that has an extensive list of utility classes by which to build UIs.

Coming from a background of using Bootstrap 3 quite heavily before we collectively decided as an industry that Bootstrap is past it (we did do that right? I can't keep up), there was plenty to complain about; their use of !important would keep me up at night, jQuery was great - in 2008, among others.

In their defence, they also did a lot right. The Bootstrap grid system is still fantastic, the modal is equally great and ultimately they made building consistent and functional UIs a breeze. For that alone, I'll always be grateful.

There has been an appetite for something more in recent memory. Something a bit, simpler? I think Tailwind (or rather the creators) realised people were looking for something just like Tailwind, so they made it.

There are whole swathes of developers out there who believe CSS is black magic and anyone that is remotely fluent in it is this all-powerful style-sheet, colour, whitespace, Shoreditch coffee drinking genius. When someone comes to me with a CSS issue they have spent "the last few days trying to solve" and I did it in one line (position: relative folks) from then on I was considered a sorcerer of black magic.

These whole swathes of developers are pre-dominantly backend developers and trust me guys CSS is a bizarre bit of kit, the main difference is exposure (as with most things), so I'm not gunning at ya. I sympathies with never really knowing how CSS works, I still don't think I do.

What I'm ultimately trying to say is, Tailwind came along and said: "don't worry about that nonsense, do it all in HTML and never write another line of CSS again". Immediately interest was piqued and Tailwind became very popular.

I finally found a use-case to try out Tailwind (this website uses it, as well) and thought I write my thoughts about it and how I think it compares to using good ol' CSS (or SCSS/LESS whatever takes your fancy).

So, how is Tailwind?

Tailwind has shined when I've been using it to get something up and running quickly, be it a prototype or a proof-of-concept. You can just focus on building your component (if we were using React, for example), there's no worry with importing stylesheets and ensuring they get parsed correctly.

Tailwind is extendable to be able to interject your styles and colour schemes, add your classes and modify existing ones. Tailwind also allows you to target pseudo-classes right in the class name, which is probably my favourite feature.

I think Tailwind's use of utility classes bodes well for component-based frameworks as it allows you to encapsulate styles to minimise class name bleed. As one component can encapsulate the template and the corresponding style, rather than having them in two separate places. This applies to small components (which they all should be, of course 😝) to not have complicated components made more complicated by hundreds of class names.

Tailwind also comes with the ability to "apply" Tailwind classes in your style sheets. So your button goes from:

<button class="text-blue bg-blue-500 hover:bg-blue-800 hover: text-white" />

to:

<button class="btn-blue" />

.btn-blue {
   @apply text-blue;
   @apply bg-blue-500;

   &:hover {
      @apply bg-blue-800;
      @apply text-white;
   }
}

Which helps in minimising the class name spam. However, this begs my most important question...

Does it scale?

Increasingly, I am re-writing my applications that are using TailwindCSS and moving all the HTML classes to the SCSS style-sheets. Why?

<div className="flex md:w-2/3 w-full p-2 md:p-8">
   <div className="flex flex-col items-start justify-center text-center md:text-left">
      <h1 className="font-black header-text-xl md:text-6xl">{title}</h1>
      <h2 className="text-2xl md:text-3xl mt-2 opacity-75">{description}</h2>
      <div className="flex w-full mt-6 justify-center md:justify-start">
         <Link to={findOutMoreLink}>
            <button type="button" className="bg-red-500 hover:bg-red-700 text-white font-bold py-2 px-4 rounded text-2xl ml-auto">
               {findOutMoreLinkText}
            </button>
         </Link>
      </div>
   </div>
</div>

That is not readable (and that is a very tame example). At the very least, you need to study that HTML block to start to figure out how it is styled or what it is trying to convey.

With a bit of clever SCSS you can transform the block above to something more akin to:

<div className="wrapper two-thirds">
   <div className="content">
      <h1>{title}</h1>
      <h2>{description}</h2>
      <div className="center">
         <Link to={findOutMoreLink}>
            <button 
               type="button" 
               className="btn btn-red"
            >
               {findOutMoreLinkText}
            </button>
         </Link>
      </div>
   </div>
</div>

Sure, you still need to go to the SCSS file and read that understand how it styled, but that is the purpose of SCSS. Are we not crossing streams, muddying the waters, by merging the styling and markup into the same file?

When I view HTML I want to see the semantics of the file. When I view the stylesheet I want to see how the content is styled.

If I've decided to move all my Tailwind classes to custom SCSS style-sheets, then why wouldn't I just use SCSS? Spoiler: this is exactly what I've done.

Now I get complete control over my styling, I can use variables and all the other lovely functionality bundled with SCSS and can apply proven architectural structures to my styling that makes it easier to read and understand.

It is far easier to organise your code when you can have a distinction between your stylesheets and your templates. It also allows you to structure your styles and take full advantage of all the utility that comes with SCSS.

Ultimately I had projects where I was struggling to understand how to tweak the UI as I found following all the class names so difficult.

If I, the author of the code, was struggling to follow it - how would anyone else?

Tailwind fills a niche

It is a great tool, but I can only see myself using it if I need to prototype something fast, or a backend heavy side-project with a minimal front-end. That is where Tailwind shines.

If your project has scalability concerns or it is a style heavy project then I recommend to run your own SCSS architecture and follow something like the 7-1 pattern (which is by far my favourite way of organising my SCSS). This allows you to reap all the benefits of using SCSS and keeps a clear concern between your style-sheets (here be styles) and your HTML (here be HTML).

Creating a framework-agnostic API module in TypeScript

Harry — Thu, 30 Jul 2020 18:29:45 +0000

This post originally appeared on my personal blog

Hello, today I'm going to be writing about how to write a framework-agnostic API module in TypeScript.

Due to the day job demands, I've recently transitioned from writing Angular UIs to writing primarily React UIs. I'm not going to get into the finer points of Angular vs React, framework vs library, as that horse is long dead!

However, one thing I do miss from the Angular days is the HttpClient provided by Angular. It 'provides simplified client HTTP API for Angular applications'. That it does.

Today I'm going to show how I write something similar using TypeScript, which is more explicit and more functional than our Angular counterpart.

Please note as this is written entirely in TypeScript I will make no assumptions about what framework you are using. I will also tailor this tutorial to be generic that either fetch or Axios can be used. However, I will touch on some Axios specific features further down the post.

Another assumption; if you are reading this, you are familiar with TypeScript. So let's jump straight into it.

If you just wanna see the code, no danger! I uploaded it here. Enjoy.

Why?

Good question. By writing an API module is allows us to encapsulate all our API calls into a single module/folder within our codebase. By splitting up each facet of an API call (creating the URL, passing the data, handling the correct Http verb) we can easily test, maintain and understand our codebase or someone else's. Code readability to me is of the utmost importance and should be valued above much else.

The API module will:

Handle the sending and receiving of all server calls
Create our server compliant URLs, handle all parsing of parameters required and set global headers for all requests
Encapsulate a generic Client Factory layer that will handle our POST,PUT,GET and DELETE request, whilst also giving us type safety and IntelliSense
Serve as a logical layer to declare and reference any data transfer objects (DTO) your API sends or receives
Allow the developer to be as verbose as required for each server call, as to minimise confusion and emphasis readability of the codebase
Be easy to test as each step on the process is small, simple and functional

To give you a brief example, we can see the differences:

// oh look at me I'm not using an API module 
function updateUserDetails (user: User): Promise<any> {
  return fetch(
    method: 'PUT',
    headers: {
      "Content-type": "application/x-www-form-urlencoded; charset=UTF-8"
    },
    body: JSON.stringify(user)
  )
}

import * as API from "api";
// ohohhh yeah look at that API module
function updateUserDetails (user: User): Promise<API.UserDTO> {
    try {
        return API.getUserDetails(user);
    } catch (error) {
        // handle the error
    }
}

DRY!

Write an API module once and never worry about URLs, headers, return types ever again. Cover your API module in tests (easy using a functional approach) and never fret about your server calls again!

Let us begin

Right, if you've come this far your either sold or intrigued.

I will be using the getUserDetails and other user-centric calls in these examples.
Folder structure is simple:

/src
    /API
        /Client
            Client.ts
            Headers.ts
            HttpInstance.ts // axios only
            ResponseCode.ts
            index.ts
        /URL
            CreateQueryString.ts
            CreateUrl.ts
            JoinPaths.ts
            index.ts
        /User
            GetUserDetails.ts
            DeleteUser.ts
            // etc
        index.ts
    / //rest of app

The Client folder is where the actual server calls happen through the use of factories that handle each respective Http verb.

The URL folder will parse our base url, any params and remove all unwanted characters.

Here, the User is an example but you can start to see how I am very explicit and verbose. Each file within /User performs one simple action, which is immediately identifiable at a glance.

How many times have you been burned by having say: user.service.ts which contains all server calls for your user domain, soon it is 600 LOC and completely unreadable. We mitigate that entirely by spitting each action our user can perform into its own file. This results in more files, however, this is an improvement over a monolithic service (IMO).

URL

Our URL layer needs to ultimately create the correct URL for the API. We need to:

Sort any parameters into a key-value pair query string
Join our base URL with the params
Make our URL error-proof

export type CreateUrl = (baseUrl: string, template: string, param?: object | undefined) => string;

I may call createUrl like so:

createUrl('https://exmaple.com/', '/api/user/', { userId: '123', token: 'abc5' }),

Which produces:

https://exmaple.com/api/user?userId=123&token=abc5

Creating the API URL

Our createUrl needs to first and foremost join the baseUrl and the template together. This is where JoinPaths.ts comes into the ring, as seen in the folder structure overview above.

We need to ensure we don't simply join two strings together, as shown in the code above, this could result in a bad API URL, for example:

https://exmaple.com//api/user/?userId=123

We need to take both strings in as an array to be able to trim the trailing and leading slashes and handle the slashes from with the JoinPaths file.

const trimLeadingSlash = (path: string): string => (
    // if first character is slash, return string minus first char
    path.charAt(0) === '/' ? path.substr(1) : path
);

const trimTrailingSlash = (path: string): string => (
    // if last character is slash, return string minus last char
    path.substr(-1) === '/' ? path.substr(0, path.length - 1) : path
);

export const joinPaths = (...segments: Array<string>): string => (
    segments.reduce(
        (path: string, currentSegment: string) => (`${trimTrailingSlash(path)}/${trimLeadingSlash(currentSegment)}`),
    )
);

This allows anyone working on this codebase to not have to worry about trivial problems like leading and trailing slashes. We move the responsibility into our API module and if you write some unit tests you can sleep safely knowing JoinPaths will look after your URLs.

Sort the parameters

So now we have a URL that has been parsed and formatted, we can append our API parameters to it. Ultimately, we need to iterate through the object we passed into createUrl().

Typically there are two ways to pass parameters to a server:

https://example.com/api/:token

https://example.com/api?token=token

Let's not discriminate and create a parameter pattern that will allow us to decide on the fly which way we please.

let url: string = joinPaths(baseUrl, template);
const queryParams: object = {};
Object
    .keys(params)
    .filter(key => params[key] !== undefined && params[key] !== null)
    .forEach(key => {
        const paramPlaceHolder: string = `${key}`;

        if (template.indexOf(paramPlaceHolder) > -1) {
            url = url.replace(paramPlaceHolder, params[key]);
        } else {
            queryParams[key] = params[key];
        }
    });

Here we're iterating and filtering over each key in the parameter object, then we either do one of two things:

If the parameters being passed in match the template then we assume the token is being passed out of parameters like :token and we replace the template match with the parameter value.

Otherwise, we add the key-value pair to our new queryParams object defined above.

Now that all remains is to put it all together and sort the query parameters:

export const createKeyValuePair = (key: string, value: any): string => {
    if (Array.isArray(value)) {
        return value.map(x => createKeyValuePair(key, x)).join('&');
    }

    return `${encodeURIComponent(key)}=${encodeURIComponent(value)}`;
};

export const createQueryString = (params: object): string => (
    params && Object
        .keys(params)
        .filter(key => params[key] !== undefined && params[key] !== null)
        .map(key => createKeyValuePair(key, params[key]))
        .join('&')
);

The above code will iterate over all the parameters in the object we created in the previous step, each parameter we call a recursive function that first checks to if the passed in value is an array and if so then work through each element in the array and create the parameters.

Otherwise, return the value in the format the server is expecting. By joining each element with an ampersand we will create:

?key=value

Again, array or object both use cases get covered.

Now all is todo is add the createQueryString reference and return the URI:

url = url.replace(/\/{.*}/, '');
const queryString: string = createQueryString(queryParams);
return `${encodeURI(url)}${queryString && `?${queryString}`}`;

This code appears below the object parameter code we wrote above.

Now the URL is sorted we can turn our attention to the generic factories.

Factories

What are the factories? They will be the layer that is responsible for actually making the request to the server. This layer acts as a wrapper around either the Axios client or the native fetch client. We also need to handle each HTTP verb and leverage TypeScript to allow IntelliSense. This is the /Client folder from the folder structure overview above.

import { AxiosRequestConfig, AxiosResponse, AxiosError } from 'axios';
import { axios } from './HttpInstance';

const sendRequest = <T>(config: AxiosRequestConfig): Promise<T> => axios.request<T, AxiosResponse<T>>(config)
    .then(res => Promise.resolve(res.data))
    .catch((error: AxiosError) => Promise.reject(error?.response?.data ?? error));

export const getFactory = <T>(url: string) => sendRequest<T>(
    { url, method: 'GET' as const },
);

export const postFactory = <T, K>(url: string, body: T) => sendRequest<K>(
    {
        url,
        method: 'POST' as const,
        data: body,
    },
);

export const putFactory = <T, K>(url: string, body: T) => sendRequest<K>(
    {
        url,
        method: 'PUT' as const,
        data: body,
    },
);

export const deleteFactory = (url: string) => sendRequest<void>(
    { url, method: 'GET' as const },
);

The above code is for the Axios client, however, for fetch it's almost identical but you will need to handle the stringification yourself and make sure to reject the request promise if res.ok is false, as with fetch a failed request does not get caught in the catch block of a promise.

We declare each verb as it's own factory function, and through the use of generics and declare the response and request types we will declare when calling these functions.

Actions

Now we've sorted the request URL, parameters and created a layer to communicate with the API, now we need to introduce our actions layer. This is where I like to declare DTOs and will act as a buffer between the application and our API module.

// ../API/User/createAccount.ts
import { createUrl } from '../URL/CreateUrl';
import { environmentVariables } from '../../../environment';
import { postFactory } from '../Client';

const route: string = 'user';
export type CreateDTO = {
    password: string;
    confirmPassword: string;
    email: string;
    name: string;
}
export const createAccount = async (payload: CreateDTO): Promise<{ 
    token: string, 
    userId: string 
}> => postFactory(
    createUrl(environmentVariables().apiUrl, route),
    payload,
);

From the example above you can see all the pieces coming together and can see the TypeScript generics in play. It's clear from a glance exactly what this code is doing, what the data looks like and how the data will look on a successful request.

Please note the environmentVariables() function just returns the base URL of the server.

Now all you need to do is call and await createAccount and make sure to wrap it in a try/catch block. That's it! You now have an API module that handles all the things you need to make a request to your server without you ever having to repeat yourself or worry about incorrect syntax/data structures.

Axios specifics

Headers

I like to create an Axios instance on application load that will take a headers object as an argument, this ensures that you don't have to keep creating/passing in a headers object. Fire and forget!

export const defaultHeaders: Record<string, string> = {
    'Content-Type': 'application/json',
    Accept: 'application/json',
    credentials: 'include',
};
export const axios: AxiosInstance = Axios.create({
    timeout: 10000,
    headers: {
        ...defaultHeaders,
    },
});

Interceptors

I also like to register interceptors for the client (axios) we initialise above (so all requests and responses get accounted for)

export function RegisterInterceptor(setState: (value: Action) => void) {
    axios.interceptors.response.use(response => response, async (error: AxiosError) => {
        if (!error.response) {
            // no response from the server (is the user offline?)
            throw error;
        }
        if (error.response.status === 403) {
            await clearCache();
            setState({
                type: StateActions.LOGGED_OUT,
            });

            throw error;
        }

        throw error;
    });
}

You can see my interceptor is used to handle unauthorised messages and log out the user. You can also use this layer to handle any response type or no response at all for your client.

Lastly, register this interceptor on application load, something like:

// app.tsx
import * as API from './src/API';

// app component code:
React.useEffect(() => {
    API.RegisterInterceptor(setState);
}, []);

That is it all! I hope you learnt something today. All the code I've uploaded to a GitHub repository here. Enjoy. :)

Applying Domain-Driven Design principles to a Nest.js project

Harry — Sat, 25 Jul 2020 08:48:52 +0000

This post originally appeared on my personal blog.

Hey, today I'm going to be writing about how you can apply domain-driven design (DDD) principles to a Nest.js project.

I've created a quick repository showing the architecture explained in this blog. Find it here.

Disclaimer

I am by no means an expert in DDD, we merely decided to adopt it for Repetitio when we re-wrote the server.
I will not cover what DDD is in this blog, please refer to this dev.to blog post or the DDD bible.
Nest.js - A progressive Node.js framework for building efficient, reliable and scalable server-side applications. It mirrors Angular's architecture style and is built with TypeScript from the ground up.

Why Nest?

Nest is opinionated on how to structure your code, this works well in a DDD case as you need to be able to put strict boundaries around your code as to keep your code maintainable and readable. If you are looking for a powerful and scalable framework for your Node.js application, I highly recommend Nest.js.

So why would we do this?

Having used Nest.js in production for about 4 months we ended up, due to the project being unrestricted without clear boundaries, with a mess of spaghetti code for our server and API. We're talking about services that are 600+ LOC long.

This is by no means a reflection of Nest.js, but rather a reflection of what happens to codebases without strict rules and boundaries implemented.

We decided to jump on the bandwagon and start to look into DDD as a way forward after we had decided to rewrite the server.

So ultimately what we wanted to get out of this exercise is a codebase that by nature of design stays tameable.

How do you do this?

We found that DDD outlines a clear structure to your code that not only makes sense but can minimise the amount of code/methods/classes that resides in aspects of your domain.

To use our old server as an example; we had a service called user.service.ts that contained ALL logic for the user. Now, as with most applications, the user tends to take a center stage and encapsulates a lot of domain logic. Therefore, our service became huge, covering the entire user logic. It became hard to read and understand what each method was trying to achieve and what actions it was performing.

So let us use this example and apply some DDD logic to it, firstly we decided to be very strict with our domain layer, therefore in the user domain we only care about our user domain model, all other relationships are abstracted out into a new domain layer or a subdomain.

We kept the usage simple and made a class per action, rather than lumping them all into the same omnipotent, omniscient service. By the fact of simplifying, our domain layer becomes very functional with each class performing typically one action, with the methods within further reflecting this.

Our architecture

This is our architecture: (using a very generic user as an example for the different files and services)

src/
    /API
        /User
            UserController.ts
            CreateUserDTO.ts
        APIModule.ts
    /Auth
        AuthModule.ts
    /Database
        DatabaseModule.ts
    /Domain
        /User
            CreateUser.ts
            IUserRepository.ts
            User.ts
            UserModule.ts
    /Persistence
        /User
            UserRepository.ts
            UserRepositoryModule.ts
            UserEntity.ts
    /Utils
        /Mappers
            /User
                CreateUserDTOToUser.ts
        /Services
            /Email
                EmailSenderService.ts
    AppModule.ts
    Environment.ts
    main.ts

The Module in this case is a Nest Module. We decided that each Domain would have its module, as would each persistence entity. However, we decided to lump all API endpoints into one module, as the only thing importing the APIModule was the AppModule on application bootstrap.

API

This layer contains all our endpoints and controllers. We tried to make a Controller class mirror a domain entity. So, if we had a user domain, we'd have a user controller. This pattern extends to the persistence layer too.

This is also an obvious space to declare any DTOs - sending or receiving.

Auth

This is where all our Guards, Strategies and general Auth configuration lies.

Database

You guessed it, this layer is responsible for connecting to any kind of data store, or multiple data stores for that matter.

Domain

The most important part of our codebase. The domain is a reflection of the problem we are trying to solve. This is broken up into our domain models (each warranting their folder). Our domain model, in our case, is a TypeScript types file.

We want to keep our domain layer pure from third party artefacts, so within our domain we should only reference our code. We don't want to see MongoDB schemas, third-party packages, reference to any database-specific logic or anything related to our API layer.

We define an interface that mocks our repository layer. This is defined in our domain layer as it is directly related to our domain. We are outlining how we want to be able to mutate our domain models, we are not bothered with the implementation (that is the persistence layer's job).

A few rules we adhere to with our domain layer:

The domain actions should only accept the domain model as a parameter or an Id in string form. DTOs should be mapped before calling our domain layer.
Leave all third-party libraries, packages etc outside the domain layer. It should be third-party dependency-free.
It should only reference code that exists in the domain layer
In theory, you should be able to cut and paste your domain layer it into any project (language-dependent) and it should work.

Dependency-free domain

One of the biggest mental challenges is organising your code in such a way that your domain layer is only dependent on other classes and files within your domain layer.

For example, to communicate with the persistence layer we can introduce the UserRepositoryModule as a dependency to our UserModule but would go against a key component of DDD - a dependency free domain. It's also why we have a User.ts (domain) and UserEntity.ts (persistence). One is our domain model, in its purest form. The other, our domain model but with the added attributes/functionality for whatever data store.

One way (and thanks to SeWaS for showing me how) we can use interface injection, rather than typical module injection to communicate with the persistence layer.

DomainAction.ts is just a generic name which represents the many actions our domain will perform.

// Domain/User/DomainAction.ts
import { Injectable } from '@nestjs/common';
import { Injectable, Inject } from '@nestjs/common';
import { UserRepository } from '../../Persistence/User/Repository';
import { IUserRepository } from './IUserRepository';

const UserRepo = () => Inject('UserRepo');

@Injectable()
export class DomainAction {
    constructor(
        @UserRepo() private readonly userRepository: IUserRepository,
    ) {}
}

// Persistence/User/UserPersistenceProvider.ts
import { Provider } from "@nestjs/common";
import { UserRepository } from "./Repository";

export const UserRepoProvider: Provider = {
    provide: 'UserRepo',
    useClass: UserRepository
}

// Persistence/User/UserRepositoryModule.ts 
import { UserRepoProvider } from './UserPersistenceProvider';

@Module({
    providers: [UserRepoProvider],
    exports: [UserRepoProvider],
})
export class UserRepositoryModule {}

Domain structure

Each domain action that gets performed on our domain model should constitute its own file and class. The name of this class should be explicit and leave no-one guessing as to its purpose.

For example:

/Domain
    /User
        Create.ts
        Update.ts
        Delete.ts
        GetEmail.ts
        RemoveToken.ts
        IUserRepository.ts
        UserEntity.ts
        UserModule.ts

We can even drop the User from the name of the class as it is implied due to it residing within the user domain.

Persistence

Our persistence layer is where all our database queries are performed. This will contain the entity's Module and its Repository. The repository in this sense is a class that contains all database operations. Again, this should mirror our domain entities 1:1 and should typically only contain around 4/5 methods - predominately CRUD operations. Unlike our domain layer, we couple multiple actions within the same class.

Utils

These typically share functionality required across our domain.

This is where we store all our mappers that map data-transfer-objects to domain models and vice versa.

It is a good place to store singleton services, like the example, an email sender service.

Typical flow through our server

As I feel the need to show some code, I'll show some snippets of how this all looks if we were to create a very basic user in a purely fictional server. Obviously, I'm excluding import statements here.

// the user DTO we receive from the client
export class CreateUserDTO {
    @IsString()
    @IsNotEmpty()
    public name: string;

    @IsString()
    @IsNotEmpty()
    public password: string;
}

Class validator is great for validation, couple that with an AuthGuard (in the Auth layer) and you can handle all exceptions for your code in one place and handle the response object.

// UserController.ts
@Controller('user')
export class UserController {
    constructor(
        private readonly user: Create,
    ) {}

    @Post()
    public async Register(@Body() createUser: CreateUserDTO): Promise<HttpStatus> {
        // all our mappings get done in static classes
        const domainModel: User = UserMap.mapCreateDTOToUserModel(createUser);
        await this.user.Create(domainModel);
        return HttpStatus.OK;
    }

As our domain is only concerned with our domain we need to make sure that all API layer related objects get mapped to the domain appropriate model, hence our mapping happening in the API layer.

// /Domain/User/Create.ts
const UserRepo = () => Inject('UserRepo');
const EmailService = () => Inject('EmailService');

@Injectable()
export class Create {
    constructor(
        @UserRepo() private readonly userRepository: IUserRepository,
        @EmailService() private readonly email: IEmailSenderService,
    ) {}

    public async Register(user: User): Promise<void> {
        const registeredUser: User = await this.repository.Create(user);
        await this.email.SendEmail(registeredUser.email, EmailOptions.AccountCreationEmailOptions, {
            userId: registeredUser._id,
        });
    }
}

The EmailSenderService is an example of shared logic across our domain that would exist in the Utils layer mentioned above.

// /Persistence/User/UserRepository.ts

@Injectable()
export class UserRepository implements IUserRepository {
    constructor(@InjectModel('User') private readonly user: Model<UserEntity>) {}

    public async Create(newUser: User): Promise<UserEntity> {
        return new Promise<UserEntity>((resolve, reject) => {
            const createdUser: UserEntity = new this.user(newUser);
            this.user.create(createdUser, (err: GenericError, addedEntity: UserEntity) => {
                if (err) {
                    reject(err);
                }

                resolve(addedEntity);
            });
        });
    }
}

We can also see the need to have two types of User, one for our domain which is the purest form of our domain model and one that exists in our persistence layer that would be very similar to our domain model but would contain third-party (in this case, database-related) attributes or logic specific to our persistence layer.

This further re-enforces the need for our domain layer to be pure of all third-party artefacts.

Summary

This is a brief overview of the architecture we've employed at Repetitio. By holding ourselves to such rigid rules we've found our codebase has been pleasantly manageable, nothing like it was before! With clear logical layers to our server, it is easy to navigate and understand, allowing us to easily dive in and iterate with an ever-evolving set of requirements.

I've created a quick repository showing the architecture explained in this blog. Find it here.