DEV Community: Anders Tornblad

Building business processes with Azure Durable Functions

Anders Tornblad — Tue, 14 May 2024 14:17:13 +0000

In cloud-native development, leveraging Azure Durable Functions and Durable Entities provide an new and interesting way to build robust, scalable workflows. Moving away from traditional development complexities, this "code-first" approach enhances productivity, streamlines onboarding, and improves code quality. In this article, we'll explore how these tools can transform your business operations, making development faster and more efficient.

Traditional approaches vs. modern cloud-native development

Traditionally, building business processes involves a highly database-centric approach. Development typically begins with a comprehensive database design, setting up schemas and tables before moving on to create classes and properties. These classes would then map to the database schema using Object-Relational Mappers (ORMs) like Entity Framework, NHibernate or Dapper. This method, while effective, often leads to a tightly coupled system where the database structure heavily influenced the application design.

Developers have to manually implement dehydration and hydration logic to save and restore object states. Even when using an ORM, this means writing a lot of boilerplate code to ensure that the application could persist and reload states, adding complexity and increasing the likelihood of errors. The state has to be carefully managed to ensure data consistency, especially in long-running processes involving multiple steps and decision points.

Another tried and tested way of implementing business processes would be to use some integration and orchestration engine, like Microsoft BizTalk Server or Mule ESB, where you would design your business process using a "point and click" approach. For each step or transition in the process, you would then hook up the necessary code or use pre-made plugins to perform the actual work.

In contrast, the modern approach with Azure Durable Functions and Durable Entities represents a shift in mindset. Developers can focus on writing business logic using a code-first approach. This method abstracts away the underlying infrastructure, allowing developers to concentrate on the functionality rather than the setup. The serverless nature of Azure Functions means that there is no need to manage servers – the platform handles automatic scaling and infrastructure concerns, adapting seamlessly to varying loads and demands.

Automatic state management is one of the standout features of Durable Entities. Unlike the manual state management of the past, Durable Entities persist state automatically. They use event sourcing to ensure that the state can be reconstructed from its history without requiring manual intervention. This not only simplifies development but also reduces the potential for errors and makes state management more reliable and efficient.

Example implementation

Here is an example of a business process orchestration using Azure Durable Functions and Durable Entities. The example involves a time report submitted by an employee, which waits for approval from a manager. If denied, the employee can fix errors and resubmit until it gets approved and sent to a payout service.

Step-by-Step Explanation

Orchestration Function: Manages the workflow of the time report process.
Durable Entity: Represents the state of the time report.
Approval Process: Simulates manager approval (external call).
Payout Process: Simulates sending the approved report to a payout service (external call).

Durable Entity for Time Report

First, let's create a durable entity to hold the state of the time report:

using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Newtonsoft.Json;

public class TimeReportEntity
{
    [JsonProperty("status")]
    public string Status { get; set; }

    [JsonProperty("reportDetails")]
    public string ReportDetails { get; set; }

    public void SubmitReport(string reportDetails)
    {
        ReportDetails = reportDetails;
        Status = "Submitted";
    }

    public void ApproveReport()
    {
        Status = "Approved";
    }

    public void DenyReport()
    {
        Status = "Denied";
    }

    [FunctionName(nameof(TimeReportEntity))]
    public static Task Run([EntityTrigger] IDurableEntityContext ctx)
        => ctx.DispatchAsync<TimeReportEntity>();
}

Orchestration Function

Next, we create the orchestration function to manage the process:

using System;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Extensions.Logging;

public static class TimeReportOrchestration
{
    [FunctionName("TimeReportOrchestration")]
    public static async Task RunOrchestrator(
        [OrchestrationTrigger] IDurableOrchestrationContext context, ILogger log)
    {
        string reportId = context.GetInput<string>();
        EntityId entityId = new (nameof(TimeReportEntity), reportId);
        bool approved = false;

        while (!approved)
        {
            // Submit the report
            string reportDetails = await context.CallActivityAsync<string>("GetTimeReportDetails", null);
            await context.CallEntityAsync(entityId, "SubmitReport", reportDetails);

            // Simulate manager approval process
            string approvalStatus = await context.CallActivityAsync<string>("RequestApproval", reportId);

            if (approvalStatus == "Approved")
            {
                await context.CallEntityAsync(entityId, "ApproveReport");
                // Exit the loop
                approved = true;
            }
            else
            {
                await context.CallEntityAsync(entityId, "DenyReport");
                // The employee will correct and resubmit the report
                // Wait for a period of time before checking for resubmission
                var retryInterval = TimeSpan.FromHours(1);
                await context.CreateTimer(context.CurrentUtcDateTime.Add(retryInterval), CancellationToken.None);
            }
        }

        // Simulate sending the report to payout service
        await context.CallActivityAsync("SendToPayoutService", reportId);
    }
}

Activity Functions

Finally, we implement the activities for getting time report details, requesting approval, and sending to the payout service:

using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Extensions.Logging;

public static class TimeReportActivities
{
    [FunctionName("GetTimeReportDetails")]
    public static Task<string> GetTimeReportDetails([ActivityTrigger] string name, ILogger log)
    {
        // Simulate getting time report details
        return Task.FromResult("Time report details for employee.");
    }

    [FunctionName("RequestApproval")]
    public static Task<string> RequestApproval([ActivityTrigger] string reportId, ILogger log)
    {
        // Simulate manager approval process
        // This could involve an external API call or manual process
        string approvalStatus = new Random().Next(0, 2) == 0 ? "Approved" : "Denied";
        return Task.FromResult(approvalStatus);
    }

    [FunctionName("SendToPayoutService")]
    public static Task SendToPayoutService([ActivityTrigger] string reportId, ILogger log)
    {
        // Simulate sending the approved report to the payout service
        log.LogInformation($"Report {reportId} sent to payout service.");
        return Task.CompletedTask;
    }
}

Example Usage

When we have deployed the above functions and entities to our Azure Function App, to start the orchestration, we can trigger it using an HTTP-triggered function or any other trigger mechanism, for example from a queue or an incoming file from a time reporting system:

using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.DurableTask;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;

public static class HttpStart
{
    [FunctionName("HttpStart")]
    public static async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequest req,
        [DurableClient] IDurableOrchestrationClient starter)
    {
        // Generate a unique ID for the time report
        string reportId = Guid.NewGuid().ToString();

        // Start the orchestration
        await starter.StartNewAsync("TimeReportOrchestration", reportId);

        return new OkObjectResult($"Orchestration started with ID = '{reportId}'.");
    }
}

This example demonstrates how to use Azure Durable Functions and Durable Entities to manage a business process involving a time report submission, manager approval, and subsequent payout. The orchestration function handles the workflow, leveraging durable entities for state management and activity functions for individual tasks. By adopting this approach, we can streamline complex business processes, ensuring they are both maintainable and scalable.

Enhancing Productivity and Onboarding

Developing with Azure Durable Functions and Durable Entities can simplify the implementation process, starting with the use of the async/await syntax in C#. This allows developers to write code that is straightforward to read and maintain, significantly reducing the complexity typically associated with implementing business process orchestration.

Durable Functions take this simplicity further by enabling declarative definitions of workflows and state machines. This means that business logic can be expressed in a clear, concise manner, making it easier for teams to understand and modify the workflow as needed. The declarative approach aligns well with how business processes are typically conceptualized, bridging the gap between technical implementation and business requirements.

Improved code quality is another major benefit of using Durable Functions. The structured nature of these functions makes it easier to write unit and integration tests, ensuring that the code is robust and reliable. Higher test coverage leads to fewer bugs and more confidence in the system's stability. Also, following Azure's best practices promotes a consistent approach across the team, resulting in code that is not only easier to maintain but also more reliable in the long run.

Onboarding new developers also becomes easier, thanks to the readability of the process orchestration code. New team members can quickly get up to speed with the code base, assuming they have experience with modern C# and dotnet. By leveraging these modern tools and approaches, teams can significantly boost productivity, streamline onboarding, and ensure that their code is of high quality, all while simplifying the development process.

Middleware for Security, Monitoring, and Error Handling

Middleware is a fundamental concept for both ASP.NET web server code and Azure Functions, providing a structured way to handle cross-cutting concerns such as security, monitoring, and error handling.

Security

For security, middleware can centralize authentication and authorization processes. This ensures all functions comply with security policies without requiring individual developers to implement these checks manually. One of the most common approaches is using OAuth for secure authentication and authorization. Azure Functions can integrate with Azure Entra Id (previously Azure Active Directory) to provide centralized identity management and role-based access control (RBAC).

For example, Azure Functions can use the Microsoft.Identity.Web library to facilitate authentication and token acquisition, enabling secure API calls on behalf of the authenticated user. Additionally, you can configure the functions to use the JWT Bearer token middleware, which validates tokens and enforces authorization policies before the function execute.

Development teams often write their own middleware for custom scenarios, such as logging user activity or enforcing additional security checks beyond what's provided out-of-the-box.

Monitoring

Monitoring is another critical aspect where middleware is useful. Azure Monitor collects and analyzes telemetry data from Azure Functions, providing insights into application performance and reliability. Application Insights, which integrates seamlessly with Azure Functions, offers detailed monitoring including request rates, response times, failure rates, and dependency tracking. This integration can be enabled with basically just a configuration setting, ensuring that all telemetry data is automatically collected and available for analysis.

For custom scenarios, developers might implement middleware to log specific business process details or to integrate with other monitoring tools. For instance, if Application Insights doesn't automatically track certain dependencies, developers can use the TrackDependency API to manually log these dependencies, ensuring comprehensive monitoring coverage.

Error Handling

Middleware also plays a crucial role in error handling. Global exception handling middleware ensures that all exceptions are logged and managed consistently across functions. This simplifies the error-handling process and ensures uniformity. Built-in support for retries and circuit breaker patterns in Azure Functions helps manage transient faults and prevent cascading failures, enhancing the resilience of your application.

For example, a custom middleware could be written to catch all unhandled exceptions, log them to a central logging service, and return a standardized error response to the client. This approach ensures that all functions have consistent error-handling behavior without requiring repetitive error-handling code in each function.

Using middleware in these ways not only streamlines development but also ensures that essential aspects like security, monitoring, and error handling are robust and managed efficiently. By leveraging both built-in solutions from Microsoft and custom middleware, developers can focus more on business logic, confident that the foundational concerns are well taken care of.

Example implementations

Centralized Error Handling:

public class ErrorHandlerMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<ErrorHandlerMiddleware> _logger;

    public ErrorHandlerMiddleware(RequestDelegate next, ILogger<ErrorHandlerMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task Invoke(HttpContext context)
    {
        try
        {
            await _next(context);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "An unhandled exception occurred.");
            context.Response.StatusCode = 500;
            await context.Response.WriteAsync("An internal server error occurred.");
        }
    }
}

// In Startup.cs or Program.cs
app.UseMiddleware<ErrorHandlerMiddleware>();

Monitoring with Application Insights:

public class FunctionApp
{
    private readonly TelemetryClient _telemetryClient;

    public FunctionApp(TelemetryClient telemetryClient)
    {
        _telemetryClient = telemetryClient;
    }

    [FunctionName("MonitoredFunction")]
    public async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
        ILogger log)
    {
        _telemetryClient.TrackEvent("MonitoredFunctionInvoked");

        // Business logic here

        _telemetryClient.TrackEvent("MonitoredFunctionCompleted");
        return new OkResult();
    }
}

Conclusion

Building with Azure Durable Functions and Durable Entities enables a modern, cloud-native approach to implementing business processes. Compared to traditional development methods, this method offers enhanced productivity, faster onboarding, and improved code quality. By adding middleware for security, monitoring, and error handling, developers can focus on business logic while ensuring robust, secure, and maintainable applications. This shift not only simplifies development but also aligns with the dynamic needs of today's business environments, allowing for rapid adaptation and growth.

Leveraging cloud-native infrastructure also provides advantages. Azure's cloud infrastructure offers built-in redundancy, high availability, and automated scaling. This reduces the operational burden on developers or operations teams, who no longer need to worry about hardware failures or capacity planning. The cloud environment ensures that applications are resilient and can handle varying loads with ease, providing a more robust and reliable foundation for business processes.

Teach the present first, history later

Anders Tornblad — Thu, 11 Jan 2024 10:40:46 +0000

Programming languages, frameworks and best practices evolve over time. New features are added, new versions are released, and what we thought were the "right way" of doing things in the past, is considered obsolete and even harmful today.

The approach to teaching technical subjects is an important factor in preparing students for real-world application of their skills. One disturbing trend that I see, particularly when browsing through LinkedIn in the era of ChatGPT, is that programming tutors and grinding tech content creators often teach features in the order of their historical development, rather than their current state. While understanding the historical evolution of technology is important, this approach might not be the most effective in equipping learners with the skills they need today.

The issue with historical sequencing

Consider JavaScript, a language that has undergone significant evolution over the years. Traditional teaching methods often introduce concepts like var for variable declaration before let and const, concatenation using the + plus sign for building strings before template literals, and function-based constructors before class definitions. This historical sequencing can be seen in other languages like Java and C# as well.

The problem here is two-fold:

Relevance and efficiency: Students first learn outdated or less efficient methods, only to later relearn the modern practices that are more relevant in today's tech landscape. This not only confuses learners but also makes the learning process less efficient.
Perception of modern features: Newer features are often viewed as more difficult, and are considered mere "add-ons" or "syntactic sugar", undervaluing their importance in contemporary programming practices.

Teaching the present

Immediate relevance: Teaching the current state of technology first ensures that learners are immediately equipped with the most relevant and efficient tools and practices. This approach aligns better with industry standards and prepares students for real-world scenarios more effectively.
Building a strong foundation: By starting with the latest features and best practices, students build a strong, up-to-date foundation, making it easier for them to adapt to future advancements.
Simplifying learning: Modern features in programming languages are often designed to make life easier for developers, and software more robust. For instance, JavaScript's let and const provide a clearer scope management than var, and template literals offer a more intuitive and efficient way of handling strings. Teaching these features first can make learning more accessible and enjoyable.

Integrating historical context

While the focus should be on current technologies, historical context is still valuable:

Appreciating evolution: Understanding the history of technology helps in appreciating its evolution and the reasons behind certain changes and improvements.
Dealing with legacy systems: In many industries, older technologies are still in use. A historical perspective equips students with the knowledge to work on or transition from these legacy systems.
Curriculum structure: Introducing the history of technology as an extra curriculum or a supplementary module can provide depth without compromising the immediacy of current knowledge.

Updating teaching practices

For educators, staying updated with the latest developments in technology is imperative. Continuous professional development and adapting teaching materials to reflect the current state of technology are key.

Professional development: Educators should engage in ongoing learning to stay up-to-date with new developments.
Curriculum overhaul: Academic institutions should regularly review and update their curricula to ensure that they're teaching the most relevant and current technologies.
Industry collaboration: Collaboration with industry professionals can help in aligning academic teachings with real-world requirements.

Computer engineering studies

I my experience, the academic realm of computer engineering studies, is where this problem is the most clear. The most common teaching method often starts with foundational elements like transistors and logic gates, then progresses through machine code, assembly, lower-level languages like C, C++, before finally approaching modern languages like Java and Python. While academically rigorous, this bottom-up approach may inadvertently hinder learning, particularly for those aspiring to a career in software development. Such a method can be intimidating for beginners, potentially scaring away some students due to its complexity and distance from practical, real-world applications.

To foster more engagement and accessibility, there's a need to reevaluate this approach. Introducing students first to more contemporary and widely-used programming languages, followed by a gradual exploration of underlying principles, could strike a better balance between foundational understanding and practical skills, making the field more approachable and less daunting for newcomers.

Conclusion

Education should prioritize the present state of knowledge while treating its history as a valuable context. By focusing on what's most relevant and efficient today, we prepare learners not just to understand technology, but to excel in using it. As educators and institutions, it’s our responsibility to ensure that our teaching methodologies evolve alongside the technologies we teach.

Scale is a feature

Anders Tornblad — Thu, 02 Nov 2023 07:40:29 +0000

Scale in software is a feature in itself. Consider a simple game where a ping-pong ball navigates an obstacle course. With just a few objects, you'd model basic physics, forces, collisions and interactions on the Euclidean level. Add a hundred balls, ten thousand balls, and the physics still work. But introduce billions of objects, an entire ocean of things to emulate, and you might have to switch your model to fluid dynamics. The same transition happens in business systems, logistics, databases, and social media when scaling up.

At petabyte-scale, downtime isn't just inconvenient, it's potentially catastrophic. Redundancy becomes essential, not a luxury. This immense scale demands a complete re-evaluation of architecture, data distribution, and fault tolerance. As requests grow to millions per second, milliseconds of latency matters. Databases need fine-tuning, queries require optimization, and you start looking into edge computing to spread load.

You change the way you think about data integrity. The ACID approach might fail, so you need to embrace eventual consistency and the CAP theorem. With such vast amounts of data, caching isn't just for optimization; it's vital. Truly distributed solutions like Redis, Couchbase or Elasticsearch can help save your business.

Petabyte-scale solutions is also where microservices really shine, but remember to keep your services "micro". If you're implementing your microservices as full-fledged dotnet MVC applications with lots of layers, you're probably doing it wrong!

The sheer volume of logs and metrics becomes a challenge in itself. Monitoring and alerting systems transition from being supportive tools to critical lifelines. In such scenarios, even the slightest inefficiency can bottleneck an entire system. Performance testing and profiling are no longer periodic checks; they're continuous essentials, while cybersecurity evolves from mere breach prevention to real-time threat detection, isolation, and recovery.

Curiously, at this point, you might need to circle back to the basics: complexity theory and Big-O notation. Anything with a complexity larger than O(n) will be a problem. And sometimes, the component struggling to scale might not even be within our control. It could be an OS thread scheduler, a hardware driver, or even a network protocol.

When scaling to world-wide proportions, every layer, every component, and every line of code must come under scrutiny. It's demanding, but an exciting challenge!

LGBTQIA+ ally flag in pure CSS

Anders Tornblad — Fri, 30 Jun 2023 08:07:10 +0000

I was experimenting with linear-gradient, clip-path and aspect-ratio in CSS, and was able to create an LGBTQIA+ ally flag using nothing but CSS. Since the html, head, and body elements are always implicitly present in the browser DOM, you don't even have to include them in an HTML file. The full index.html file looks like this:

<style>
html {
  display: flex;
  flex-direction: row;
  inset: 0 0 0 0;
  position: absolute;
  margin: 0;
  background: linear-gradient(180deg,
    black 0%, black 16.7%,
    white 16.7%, white 33.3%,
    black 33.3%, black 50%,
    white 50%, white 66.7%,
    black 66.7%, black 83.3%,
    white 83.3%, white 100%);
}

body {
  clip-path: polygon(0% 100%, 50% 0%, 100% 100%, 66.7% 100%, 50% 66.7%, 33.3% 100%);
  background: linear-gradient(180deg,
    red 0%, red 16.7%,
    orange 16.7%, orange 33.3%,
    yellow 33.3%, yellow 50%,
    green 50%, green 66.7%,
    blue 66.7%, blue 83.3%,
    purple 83.3%, purple 100%);
  aspect-ratio: 7/4;
  margin: 0 auto;
}
</style>

An ally flag is a flag to show allyship with the LGBTQIA+ community. The colors in the middle are the normal rainbow flag colors, representing the LGBTQIA+ community itself. The black and white stripes on the outside represents me, as the normative, cis-gendered, heterosexual, embracing the rainbow flag in a hug, shaped like an "A" for "ally". In this implementation, the black and white stripes are the background of the html element, and the rainbow colors are the background or the body element, which has a clip-path to provide its shape.

As this year's LGBTQIA+ Pride Month is coming to an end, it provides an opportunity for introspection. As a cis-gendered, heterosexual, white man of privilege, it's important to me to reflect on my position in respect to the LGBTQIA+ community, particularly given the rising challenges faced by trans people.

I try to be an ally. I voice my support in LGBTQIA+ issues on social media, I go to Pride events, and I have a little "Ally Flag" sticker on my computer. It may not be much, and despite best intentions, I might stumble, but aiming to be an LGBTQIA+ ally is a journey of learning and engagement.

This month, the past year (and years) have been transformative. Listening to narratives different from my own, unlearning subconscious biases, and engaging in conversations have all been part of my journey. This isn't for applause, but to help dismantle oppressive systems that benefit me, simply because of who I am.

Allyship isn't confined to a month; it's a call to action that requires empathy, patience, and humility. It's about trying to make the world a more inclusive space, one conversation, one unlearned bias at a time. As Pride Month ends, my commitment as an ally continues, promising to learn, grow, and stand firm with the LGBTQIA+ community.

The role of software architects in Agile teams

Anders Tornblad — Thu, 22 Jun 2023 05:17:00 +0000

As the world of software development evolves, the role of the software architect is becoming increasingly critical, and yet it is often thought of as unnecessary or outdated. In Agile teams, the architect becomes a key player, taking on additional roles and responsibilities that drive the success of projects and the growth of the organization as a whole.

In Agile environments, architects provide technical expertise and direction, and ensure that projects are completed with high quality, efficiency, and appropriate tooling. They manage technical risks, foster collaboration within and outside the team, and play a pivotal role in driving innovation across the organization. Architects blend technical know-how, leadership, and strategic thinking, and can be indispensable members of any Agile team.

This article will explore the role of the software architect in Agile teams, highlighting the responsibilities they shoulder, the benefits of their inclusion in the team, and the modern trends influencing their approach. As we journey through this topic, we'll uncover how the role of the software architect not only shapes the software development process but also influences the very success and efficiency of Agile teams.

Understanding Agile teams

The Agile methodology is a distinctive approach to managing software development, that emphasizes continuous collaboration and improvement. This methodology breaks down work into iterations, creating a cycle of planning, executing, and evaluating. In contrast to older "waterfall" approaches, where one discipline completes their contribution and then "hands it over" to the next, Agile promotes the concept of collaborative, cross-functional teams. The essence of Agile lies in open communication, collaboration, adaptation, and trust amongst team members. The project lead or product owner typically prioritizes the work to be delivered, but the team takes the initiative in deciding how the work will be executed, self-organizing around detailed tasks and assignments.

Modern software development teams choose Agile for its ability to respond to change in the marketplace or feedback from customers swiftly without disrupting long-term plans. The methodology encourages "just enough" planning and shipping in small, frequent increments, enabling teams to gather feedback on each change and integrate it into future plans at a minimal cost. The Agile Manifesto emphasizes that authentic human interactions are more valuable than rigid processes, and collaboration with customers and teammates is prioritized over predefined arrangements. Agile teams unite under a shared vision and bring it to life in the best way they see fit, setting their own standards for quality, usability, and completeness.

The composition of Agile teams

Each specific Agile framework, like Scrum, can mandate a specific set of roles to fill in a team, like Scrum Master and Product Owner, but in general, the roles can be diverse, ranging from product managers and marketers to developers, testers, and software architects, each bringing their unique skill sets to the table. The structure of these teams is often fluid, adapting to changing business realities.

The process of building an effective Agile team starts with forming a group of individuals who are willing to collaborate, learn, and grow together. As the team matures, the members learn to trust each other and understand each other's strengths, optimizing how they work together to deliver high-quality software. Two fundamental pillars of great Agile teams are continuous mentoring and shared skill sets. This environment allows each member to contribute effectively, making the team's overall impact greater than the sum of its individual members. Meanwhile, the shared skill sets mean the team can handle a diversity of work and support each other better.

The software architect

The software architect role, with its blend of technical acumen, strategic oversight, and collaborative spirit, can be a driving force behind successful delivery of high-quality software solutions. Let's take a look at the dynamics of this role, the responsibilities and contributions to the Agile methodology.

Who is a software architect?

Essentially, software architects are the designers of the software world. They employ their knowledge of software technologies, design principles, and patterns to architect systems that are scalable, reliable, and maintainable. But their role extends beyond just technical design. Architects also play a crucial part in defining a system's overall architecture on a higher level, based on an understanding of the business domain, the needs of various stakeholders, and future scalability considerations.

However, it is important that the role of a software architect is not confined to an ivory tower, detached from the day-to-day realities of a software project. In contrast, architects should be deeply involved in the project lifecycle, working closely with other team members, including developers, testers, project managers, and stakeholders. They help translate business requirements into technical solutions, guide the technological choices of the team, and oversee the quality of the implemented solutions. They also foresee and manage technical risks that could impact the project's progress or outcome.

The effectiveness of a software architect is reflected in their ability to balance multiple aspects of the software development process. They need to align the architecture with the business strategy and goals, ensure that it accommodates future growth and changes, and make sure it's implementable within the project's constraints. The architects' skills and influence are integral to these balancing acts, making them an indispensable member of any software team, especially in an Agile context.

The importance of the software architect

One of the primary contributions of a software architect is the establishment of a technical vision for the project. This vision is informed by the requirements of the stakeholders and is aligned with the overall strategic goals of the organization. The architect translates this vision into a set of design principles and architectural guidelines that steer the development efforts. This guidance ensures that the software solutions developed are scalable, maintainable, and robust, addressing both the current and anticipated needs of the organization.

In addition to setting the technical vision, a software architect also plays a key role in managing technical risks. They accomplish this by identifying potential pitfalls and obstacles early in the project, allowing the team to address these challenges proactively. They bring their experience and expertise to the team, enabling them to anticipate and avoid potential issues that could derail the project. This risk management role is crucial in Agile environments, where the emphasis on iterative development and rapid delivery could otherwise lead to the accumulation of technical debt and the emergence of architectural issues later in the project.

Furthermore, software architects foster collaboration within the team and drive innovation. They facilitate communication between the stakeholders and the development team, ensuring that the developers understand the strategic goals of the project and that the stakeholders are kept informed about technical considerations and trade-offs. The architect's understanding of the domain also allow them to introduce new ideas and technologies to the team, promoting a culture of continuous improvement and innovation.

The software architect in Agile teams

In an Agile environment, the software architect is a crucial bridge between the technical team and various stakeholders. Stakeholders, in this context, broadly include everyone from project managers and product owners to customers and end-users who have a vested interest in the project's outcome.

Collaboration with stakeholders

The architect's collaboration with stakeholders begins with understanding the system's needs. This understanding isn't limited to just the immediate functional requirements but extends to the non-functional requirements and constraints, the business context, the strategic direction of the organization, and even the challenges faced by the users.

The architect's goal is to translate the diverse needs of stakeholders into a cohesive architectural vision that aligns with the Agile principles of incremental and iterative development. This involves engaging in constant communication with stakeholders, seeking their input, and explaining the architectural decisions in a language they can understand. This active collaboration helps in building a shared understanding of the project, reducing ambiguities, and fostering a sense of ownership among stakeholders.

Moreover, software architects also play a pivotal role in managing stakeholder expectations. Given the iterative nature of Agile, not all features can be delivered at once, and some decisions might need to be made about the trade-offs between various aspects like speed, quality, cost, and scope. Here, the architect's ability to communicate effectively with stakeholders about these trade-offs, while still maintaining the integrity of the system design, becomes critical.

Working with Developers

The relationship between software architects and developers is pivotal. This collaboration is not merely about sharing responsibilities, but it involves a continuous exchange of ideas, knowledge, and skills aimed at designing scalable and maintainable solutions.

A software architect, in this context, acts as a guide and mentor for the developers. They help the team in interpreting and implementing the architectural vision into code. While doing so, they respect the developers' autonomy and encourage self-organization, a core Agile principle. They provide insights into design patterns, best practices, and coding standards, ensuring the technical quality of the product. They also play a crucial role in identifying potential technical debts and risks early, enabling the team to address them proactively.

In the Agile world, the architect's collaboration with developers extends beyond the boundaries of a single team. Architects often work closely with developers from multiple teams to ensure that the architectural principles and practices are consistently applied across the entire organization. This includes facilitating knowledge sharing sessions, conducting code reviews, and providing feedback. This cross-team interaction fosters a culture of learning and continuous improvement, which is fundamental to Agile's success.

However, an effective architect-developer collaboration goes beyond just the technical aspects. A software architect must also understand the developers' perspectives, their challenges, and their needs. They should foster a culture of open communication and mutual respect, where developers feel comfortable discussing their ideas and concerns. This empathetic approach not only improves team morale but also helps in creating solutions that are truly aligned with the developers' capabilities and the project's needs.

Modern trends

As Agile practices mature, so too does the approach to software architecture. Here is an overview of two key modern trends that are important to be aware of.

Evolutionary architecture

The idea of evolutionary architecture represents a paradigm shift in the approach to software design. Traditionally, architects have attempted to predict and design for the system's needs years into the future. In an ever-changing technological landscape, this approach has become increasingly untenable. Enter the concept of evolutionary architecture, which enables the continuous evolution and adaptation of system architecture to meet emerging requirements and environmental changes.

Evolutionary architecture centers around making architecture components flexible and adaptable, with a focus on incrementally modifying and improving the system. The design principles include deferring architectural decisions until they are necessary, making changes incrementally to minimize disruption, and implementing automated testing to ensure system robustness against changes. This dynamic approach to architecture provides a mechanism to react to changing requirements without significant rework or redesign. It aligns well with Agile methodologies, where adaptability and incremental development is valued.

There are many sides to the role of a software architect in an evolutionary setting. As a strategic thinker, the architect tries to anticipate areas where the system may need to evolve and designs flexible solutions that can accommodate these changes. At the same time, they must ensure that the architecture remains cohesive and maintainable as it evolves. This involves close collaboration with the development team to understand the impact of proposed changes, and to guide the team in implementing these changes effectively and efficiently. The architect also plays a critical role in risk management, identifying and mitigating potential issues that could arise from architectural changes.

In an Agile context, the software architect contributes to evolutionary architecture by fostering an environment that encourages continuous learning and adaptation. They guide the team in adopting practices such as refactoring and technical debt management that support evolutionary changes. Above all, the architect ensures that the evolving architecture continues to align with the organization's strategic goals and provides business value. This constant dance between foresight, strategy, and adaptability is what makes the software architect's role in evolutionary architecture so challenging, but also fascinating.

Emergent architecture

Emergent architecture refers to an approach where the system's design evolves over time, often as a response to changing requirements or new insights gathered during the software development process. This concept aligns well with Agile methodologies, as Agile encourages iterative development and adaptation to changes.

In an Agile context, the software architect plays a significant role in supporting and facilitating emergent architecture. As each iteration or sprint progresses, the architect collaborates closely with the development team to build a common understanding of emerging patterns and structures. This collaboration often involves the architect in hands-on coding, design sessions, and code reviews, ensuring that the evolving design adheres to the system's overall vision and quality standards.

However, the adoption of emergent architecture does not imply a lack of planning or disregard for system architecture. On the contrary, the software architect ensures that each increment contributes meaningfully to the evolving system's architecture. They guide the team in making informed decisions that balance immediate needs with long-term sustainability. The architect also helps manage technical risks, mitigating potential pitfalls that might hinder the architecture's organic evolution. This approach allows Agile teams to remain flexible and responsive, adapt to changes, and deliver value continuously and effectively.

In conclusion, emergent architecture is a dynamic, responsive approach that complements the Agile mindset. The software architect plays a critical role in this process, guiding the system's evolution and ensuring it grows in a way that is sustainable, maintainable, and in line with the project's strategic objectives.

Please note that while I have attempted to give a comprehensive overview of emergent architecture within an Agile context, it's important to recognize that the understanding and application of such concepts can vary depending on specific project circumstances and organizational contexts.

The architect's role in Scaled Agile Framework

The Scaled Agile Framework (SAFe) is a popular framework that organizations use to scale Agile and Lean practices across an enterprise. SAFe is designed to address the complexities that come with larger software development projects involving multiple teams and stakeholders. It provides a structured approach to scaling Agile that emphasizes collaboration, alignment, and delivery across large numbers of Agile teams.

SAFe is based on three primary bodies of knowledge: Agile software development, Lean product development, and systems thinking. The principles of Agile are at the core of SAFe and guide all aspects of the framework. These principles include prioritizing individuals and interactions, delivering working software frequently, collaborating with customers, and being responsive to change.

In addition to these Agile principles, SAFe incorporates Lean thinking, which aims to optimize efficiency, eliminate waste, and deliver value quickly. Systems thinking, the third pillar, involves viewing the system as a whole rather than focusing solely on its parts. It encourages teams to understand how their work fits into the larger system and impacts other teams and the final product.

Like Agile, SAFe values flexibility and responsiveness over rigid plans and schedules. However, SAFe incorporates additional layers of planning and coordination to manage the interdependencies between different teams and align them towards the common organizational goals. This creates a balance between the flexibility of Agile and the structure needed to manage large-scale projects.

The software architect in SAFe

The role of the software architect takes on additional layers of complexity within a Scaled Agile Framework environment. The primary responsibility still lies in creating and maintaining the software architecture, but the software architect also has a significant part in guiding system development across multiple Agile teams. This involves coordinating with other architects, fostering alignment between teams, and ensuring that system-level considerations are appropriately addressed.

In a SAFe environment, the software architect operates at the level of the Agile Release Train (ART), a team of Agile teams that aligns around common technological and business objectives. In large scale settings like this, there are also multiple architect roles, operating on multiple level of organization, like Enterprise Architect, Business Architect, and Solution Architect. All architect roles work closely together to ensure the architectural integrity of the solutions being developed. This collaboration is essential to the realization of the Architectural Runway, an ongoing vision of the architecture that supports the implementation of upcoming business features. They help establish and communicate this runway, providing the necessary technical context and guidance to the teams.

Moreover, the software architect's role in a SAFe environment extends beyond the technical domain. They're instrumental in fostering a culture of shared responsibility for architecture, encouraging teams to make architectural decisions at the last responsible moment, in line with SAFe principles. They also contribute to a broader understanding of the system's architecture among all members of the Agile teams, thereby promoting an environment where everyone is aware of and respects the architectural boundaries and decisions. In summary, within SAFe, the software architect's role is a careful balancing act that blends technical leadership with effective collaboration, ensuring both the quality of the software architecture and the smooth execution of the Agile Release Train.

Conclusion

Software architects play an instrumental role in the success of Agile teams. They are the ones who bridge the gap between stakeholders and developers, translating business requirements into technical designs that are both scalable and maintainable. They are the ones who ensure quality, working hand in hand with the rest of the team throughout the development and testing processes. Their role is not limited to the confines of the development team, but extends to collaborating with various stakeholders, ensuring the alignment of software solutions with the organization's business goals.

In the Agile world, the role of the software architect is more dynamic than ever. The concepts of evolutionary and emergent architecture have reshaped the way architects approach their work. They now focus on guiding the system's evolution over time, allowing for architecture to emerge through iterative and incremental development. They balance the need for upfront design with the need for flexibility and adaptability, ensuring that the architecture can effectively respond to changing requirements.

Furthermore, the scaled Agile framework (SAFe) presents yet another layer of complexity and opportunity for software architects. Architects in a SAFe environment have additional responsibilities, and their role becomes even more critical as they help navigate larger organizational structures and processes, ensuring alignment across multiple Agile teams.

In conclusion, the role of the software architect in Agile teams is multifaceted and indispensable. It is a role that requires a blend of technical expertise, strategic thinking, and leadership. As Agile methodologies continue to evolve and adapt to the needs of modern software development, so too does the role of the software architect. By appreciating the crucial role of the software architect, we can foster better collaboration, improve software quality, and accelerate delivery, ensuring that we build software solutions that truly deliver value to users and stakeholders.

Making the MOD player available

Anders Tornblad — Tue, 20 Jun 2023 05:45:00 +0000

The MOD player is now available publicly for anyone to use, without having to clone a repository and include any JavaScript files manually. To include the MOD player in your own web site, you can now simply add the following line to your HTML:

<script type="module">
    import { ModPlayer } from 'https://atornblad.se/files/js-mod-player/player.js';
    const player = new ModPlayer(new AudioContext());
    await player.load(URL_OF_YOUR_MOD_FILE);
    window.addEventListener('click', () => player.play());
</script>

Content Security Policy and the browser security model

If you encounter any problems with including the MOD player in your web site, it might be because of Content Security Policy (CSP). Some web sites use a strict CSP that prevents pages from loading JavaScript files from other domains. If you are the owner of the web site, you can add the following HTTP header to allow the MOD player to load:

Content-Security-Policy: script-src 'self' https://atornblad.se

If you cannot add HTTP headers, you can instead add the following line to the head of your HTML:

<meta http-equiv="Content-Security-Policy" content="script-src 'self' https://atornblad.se">

Another thing I discovered when making the MOD player available was that the browser security model prevents external scripts from creating an AudioContext. To work around this, the MOD player now accepts an AudioContext as a parameter to the constructor. This is the reason why the example above passes new AudioContext as an argument to the ModPlayer constructor.

I highly recommend that you use the player by importing it from atornblad.se/files/js-mod-player/player.js instead of copying the JavaScript files to your own web site. This way, you will always get the latest version of the MOD player, and you will not have to worry about the browser security model or the Content Security Policy of your web site. The example above is a good starting point, and the full documentation of the ModPlayer class can be found on atornblad.se/js-mod-player.

The player should work from any web site, on any modern browser. As a small proof of concept, I have also created a pen for you.

You can try this solution at atornblad.github.io/js-mod-player.

The latest version of the code is always available in the GitHub repository.

Evolving Legacy Software Architecture

Anders Tornblad — Mon, 19 Jun 2023 14:32:51 +0000

So you've got a legacy system. It's served your organization for years, maybe even decades. It's been patched, updated, extended, and reconfigured more times than you can count. And while it's still serving your business in some capacity, you're starting to feel the strain.

The performance may be lagging, it might not be compatible with the newer technologies you want to adopt, or maintaining it could be becoming a nightmare. Whatever the reason, it's clear: your legacy system needs to evolve, and it's up to you and your team to guide it.

In this article, we'll delve into how we can evolve legacy systems through software architecture strategies. We'll explore approaches like refactoring, modularization, and migration to cloud-native architectures. In particular, we'll examine a refactoring pattern known as the Strangler Fig pattern.

Encapsulation

If we're to see software architecture as a growing city, encapsulation would be the building codes and zoning laws that keep everything running smoothly. It's a vital principle that ensures the city's structures — or in our case, software components — can interact cohesively and effectively.

In essence, encapsulation is about bundling related data and procedures into cohesive units — components or modules — and exposing a clear interface for interaction. This approach is like packing your belongings into clearly labeled boxes during a house move. Each box holds items that belong together, and you can tell what's inside without having to look through the contents.

So, how does encapsulation fit into the evolution of legacy systems?

Encapsulating Legacy Systems

One way to use encapsulation when dealing with legacy systems is to 'wrap' older code with a new interface. This interface serves as a sort of translation layer, allowing the old code to interact with other system components in a more modern and efficient manner.

This technique might involve creating APIs or wrappers that 'hide' the legacy system behind a more modern façade. For example, you might create a RESTful API that translates incoming HTTP requests into calls that the legacy system can understand, then translates the system's responses back into HTTP.

The beauty of this approach is that the rest of your software ecosystem doesn't need to know anything about the underlying legacy system. It simply interacts with the modern, intuitive API, thereby reducing the risk of dependencies and integration issues. This can be a first step towards building more modern solutions, if the wrapper API you create is designed in a future-proof way, and not just blindly recreate the legacy interface with new technology.

Encapsulating Business Logic

Another approach to encapsulation in legacy systems focuses on the business logic — the rules and procedures that define how your system behaves.

In many legacy systems, business logic can become deeply entwined with the system's infrastructure code, making it difficult to change one without affecting the other. By abstracting this business logic away from the older components and encapsulating it within its own module or service, you can provide a fresh interface that's easier to maintain and adapt to evolving business needs.

Agile and Encapsulation

Encapsulation aligns perfectly with the Agile methodology of software development. Agile emphasizes iterative, incremental updates rather than huge, monolithic changes, and encapsulation allows us to do just that — to isolate and evolve pieces of our system independently, one at a time.

Working on encapsulation in Agile sprints allows you to frequently assess and reassess the system's architecture and make necessary changes in small, manageable chunks. Each sprint can focus on encapsulating a specific piece of the legacy system, which then becomes a self-contained unit that can be updated, tested, and deployed independently from the rest of the system.

Leading the Encapsulation Process

Leading an encapsulation effort requires a fine balance of technical skill and leadership prowess. Here are a few practical tips to help you navigate the process:

Understand your system: Before you can encapsulate, you need a deep understanding of your system's architecture and business logic. Spend time getting to know the intricacies of your legacy system and identify which components would benefit most from encapsulation.
Prioritize: Not all parts of the legacy system need to be encapsulated at once. Prioritize based on factors like business value, complexity, and risk.
Collaborate and communicate: Encapsulation isn't a one-person job. Encourage collaboration and open communication among your team members. Make sure everyone understands what encapsulation is, why you're doing it, and what the benefits will be.
Embrace testing: Testing is critical to ensuring your encapsulated components work as expected. Invest in robust testing frameworks and practices to catch any issues early.
Celebrate progress: Every piece of the legacy system successfully encapsulated is a step forward. Celebrate these milestones to boost team morale and keep the momentum going.

Remember, evolving a legacy system is a marathon, not a sprint. Encapsulation is just one tool in your architectural toolkit, but it's a powerful one. By thoughtfully applying the principle of encapsulation, you can increase your legacy system's maintainability, reduce dependencies, and pave the way for a more modern, agile software architecture. Happy encapsulating!

Replatforming

Replatforming, in essence, is like moving home. It involves picking up your software application from its old, perhaps somewhat antiquated surroundings, and relocating it to a swanky, new environment. In the digital realm, this could often mean moving from a traditional on-premise data center to the cloud.

Why Replatform?

Just like a growing family might need a larger home, a legacy system might also outgrow its initial environment. And with growth comes the need for scalability, reliability, and a performance boost — qualities that a cloud environment is designed to provide.

Migrating to the cloud not only accommodates growth, but it also offers a platform compatible with modern technologies. This means that your system can interact more smoothly with new tools, frameworks, and services. Plus, it prepares your application for further modernization down the line, such as adopting microservices or serverless architectures.

While replatforming may sound like a radical transformation, it's crucial to note that it doesn't necessarily involve changing your application's codebase. The application's functionality remains the same; it's just living somewhere new.

Agile Advantage in Replatforming

Replatforming fits neatly into an Agile framework. Rather than a massive one-time migration, which can be disruptive and risky, an Agile approach would favor moving components of the system incrementally, maintaining business continuity, and learning from each move.

This 'step-by-step' method allows teams to assess and resolve potential issues at each stage, instead of discovering a mountain of problems after a full-scale migration. It's a safer, more manageable approach that aligns with Agile's core principles of iterative development and continuous improvement.

Leading Replatforming Efforts: Key Considerations

Define Your Goals: Before you start any migration process, be clear about what you want to achieve. Is it scalability? Lower costs? Better performance? Having a clear understanding of your objectives will guide your decision-making throughout the process.
Know Your Application: Thoroughly assess your application before moving it. Identify potential issues that could occur during the move and plan how to address them. Knowing your application inside and out will make the move smoother and less stressful.
Choose the Right Cloud Platform: There are multiple cloud platforms out there, and they're not all created equal. Research and choose a platform that suits your needs, budget, and future plans.
Develop a Migration Plan: Moving an entire application is a big task. Break it down into manageable parts, prioritizing based on complexity, risk, and business value. Then, tackle each part one at a time.
Test, Test, Test: Testing is vital during replatforming. Ensure every component functions correctly in the new environment before moving on to the next. This approach helps maintain business continuity and reduces the risk of post-migration issues.
Communicate: Keep all stakeholders in the loop. Regularly update them on the progress, any issues encountered, and how those issues are being resolved. Transparency builds trust and makes the entire process smoother.

Replatforming is a significant step in the evolution of your legacy system. It's an opportunity to boost performance, enhance scalability, and prepare your application for a future of continued growth. Remember, just like moving homes, it may come with a fair bit of heavy lifting and unexpected hitches, but the benefits of a new, improved environment make it a journey worth taking. So, lace up your digital moving boots, and let's get replatforming!

Refactoring and the Strangler Fig Pattern

Just as nature has its ways of renewing and replacing the old, software development has its methods of breathing new life into legacy systems. Two such approaches are refactoring and the Strangler Fig pattern. Let's delve into these methods and see how they help modernizing legacy systems.

Refactoring

Refactoring is essentially the act of improving your code without changing its behavior. It's like reorganizing your kitchen - you're not getting new ingredients or equipment; you're just optimizing their placement for more efficient use.

In the software world, this might involve simplifying complex code structures, renaming variables for clarity, or breaking down large functions into smaller, more manageable ones. The idea is to make your code cleaner, more understandable, and easier to maintain, without altering what it does.

And like an Agile chef continually refining their cooking process, refactoring should be a constant activity in Agile software development. As you add new features or fix bugs, take the opportunity to clean up and streamline your code. Remember, small, regular improvements often lead to significant long-term benefits.

If you want to improve your refactoring skills, I recommend you take a look at The Gilded Rose Kata, a structured approach to refactoring training, initially created by Terry Hughes, and further refined by Emily Bache. It emphasizes test-driven development, proven refactoring patterns and an evolutionary approach.

The Strangler Fig Pattern

Nature often provides us with good metaphors, and software architecture is no exception. The Strangler Fig pattern is a refactoring strategy inspired by the growth of the strangler fig tree.

The strangler fig starts as a seed high up in the branches of a host tree. Over time, it grows and extends its roots around the host tree, eventually replacing it. In software terms, you start by building new components alongside your legacy system, reimplementing its functionality. These new components gradually take over responsibilities from the old system until, eventually, they replace it entirely.

The beauty of this approach is that the system continues to operate as normal during this process. If a new component doesn't work as expected, you can simply switch back to the old one, reducing the risk and potential disruption.

Leading the Refactoring and Strangler Fig Pattern Process

Identify Refactoring Opportunities: The first step in refactoring is to understand where improvements can be made. Tools like static analyzers and linters can help identify areas of your code that could benefit from refactoring.
Start Small: Start by refactoring small parts of your codebase. This is where the Agile principle of iterative, incremental changes shines. Small changes are easier to manage and less likely to introduce bugs.
Automate Testing: Automated testing is crucial during refactoring. Tests give you the confidence to refactor without fear of breaking existing functionality.
Strangler Fig Approach: Identify parts of your system that could benefit from the Strangler Fig pattern. These might be parts that are particularly difficult to maintain or are due for significant changes.
Incrementally Replace Components: Replace components one by one, testing each new component thoroughly before moving on to the next.
Communicate: Keep your team and stakeholders informed about your refactoring and Strangler Fig efforts. Explaining what you're doing and why can help gain their support and understanding.
Continually Learn and Adapt: Like everything in Agile, refactoring and the Strangler Fig pattern are about continual learning and adaptation. Reflect on what's working and what isn't, and don't be afraid to adjust your approach accordingly.

Refactoring and the Strangler Fig pattern offer powerful ways to improve and modernize your legacy systems. They reflect the Agile principles of continual improvement and adaptive planning, helping you transform your systems increment

ally and manageably. Happy refactoring and may the growth of your software be as steady and reliable as that of the Strangler Fig!

Re-architecting

Re-architecting might sound overwhelming, but in essence, it's similar to home renovation. Imagine you're in a house where the rooms no longer serve their intended purpose — the kitchen's too small, the living room can't accommodate your guests, or maybe, the building lacks the necessary insulation. You don't want to move, but it's clear that mere reorganization won't suffice. So, you decide to make structural changes, knocking down walls, adding extensions - fundamentally changing your home's architecture.

In software terms, re-architecting involves a similar approach. Sometimes, legacy systems reach a point where the architecture doesn't support business needs, or it's incompatible with modern technologies. In such situations, refactoring or minor adjustments might not be enough. You need to change the system's architecture, its underlying blueprint.

Often, this transition might involve moving from a monolithic architecture to a more modular architecture. A monolithic system is like a big old house where everything is interconnected. Changes in one area could impact others, making the system difficult to scale and maintain. On the other hand, a modular architecture is like a modern apartment complex, with independent units (services) that can be scaled, updated, or replaced individually. This architecture enhances scalability, resilience, and can accelerate development by enabling teams to work on different services simultaneously.

The Agile Intersection with Re-architecting

Agile practices align well with re-architecting. Instead of a complete, one-time overhaul (which can disrupt business operations and introduce a significant risk), Agile encourages incremental changes, one piece at a time. You don't renovate the entire house in one go; you work on one aspect at a time. You move one wall, you rewire the electricity, you add insulation where it is needed. The same principle applies to re-architecting software systems.

Agile practices also promote learning and adaptation. By implementing changes incrementally, teams can learn from each iteration, adapt the architecture based on these learnings, and gradually shape a system that's closely aligned with business needs.

Leading Re-architecting Efforts

Assess the Current State: Understanding your existing architecture is the first step. Identify the bottlenecks, areas of complexity, and limitations that are driving the need for re-architecting.
Define Your Goals: Clearly articulate what you want to achieve through re-architecting. Is it to improve scalability, enhance resilience, or increase development speed? Clarity in objectives will guide your architectural decisions.
Design the Target Architecture: Draw a blueprint of your target architecture, bearing in mind your goals. This might involve exploring architectural styles like microservices, event-driven architecture, or serverless.
Develop an Incremental Transition Plan: Plan your transition in a way that each increment delivers value and doesn't disrupt your operations. This could involve starting with less critical or standalone services, and gradually moving to more complex ones.
Test Continuously: Re-architecting involves significant changes, so rigorous testing is crucial. Each component or service should be thoroughly tested before being integrated.
Communicate and Collaborate: Re-architecting is a team effort that affects various parts of the organization. Regular communication, collaboration, and feedback loops with all stakeholders are key.
Learn and Adapt: Learn from each iteration and be ready to adapt your plan and design based on these learnings.

Remember, re-architecting is a journey, not a one-time event. It's a process of continual evolution, where you learn and adapt as you incrementally transform your system's architecture. Embrace the challenge, enjoy the journey, and watch your legacy system transform into a more modern, agile, and resilient structure that's ready to meet the ever-evolving business needs. So, don your architect's hat and start laying the blueprint for your new digital edifice!

Rebuilding and Replacing

At times, the most pragmatic approach to dealing with legacy systems is either rebuilding them from scratch or replacing them entirely with commercial solutions. It's like having a dilapidated house; you might decide to demolish and rebuild, or you might choose to move into a new, fully-furnished home. Each approach comes with its unique considerations and implications. Let's unravel these two strategies and see how they intersect with Agile methodologies.

Rebuilding

Rebuilding means recreating your legacy system from scratch using modern technologies and techniques. It's a bit like building a new house; you keep the essence, the memories of the old house, but create it anew with contemporary designs and materials.

Rebuilding gives you a fresh start, unencumbered by the constraints and complexities of the old system. It offers an opportunity to rethink your architecture, choose modern, more efficient technologies, and redesign your system to be more aligned with your current and future business needs.

However, it's crucial to remember that rebuilding can be time-consuming and expensive. It's a significant undertaking that requires a considerable investment of resources. You're starting from scratch, which means designing, coding, testing, and deploying an entirely new system. There is a real risk of missing some important aspects, as legacy design decisions are often not documented. Also, there can be implicit and tacit knowledge about the business domain that exists only as implementations in the legacy code base. Recapturing enough of the legacy knowledge to be successful in rebuilding a system can prove a challenge.

In Agile terms, rebuilding doesn't mean creating the new system in one big bang. Agile promotes an incremental, iterative approach. You start by building the most critical features in the form of a Minimum Viable Product, you test, get feedback, and gradually build the rest of the system over multiple iterations. This approach reduces risk, provides early value, and allows for continual learning and adaptation.

Replacing

Replacing, on the other hand, involves substituting your legacy system with a commercial off-the-shelf (COTS) software or Software as a Service (SaaS) solution. It's like moving to a new, fully-furnished house that's ready for you to live in.

This strategy can be a practical option if there's an available solution in the market that fits your business needs and processes. The advantages include quicker implementation, lower development cost, and often, a solution that's been tested and improved upon over time.

However, the flip side is that these solutions might not be a perfect fit for your unique business needs. They may require you to adjust your processes to fit the software, which could lead to inefficiencies or reduced productivity. It's crucial to carefully evaluate potential solutions, considering their fit, flexibility, cost, and the vendor's reputation and support.

Navigating the Rebuilding and Replacing Process

Evaluate Your Needs: Understand your current system and business needs. Identify the gaps in your legacy system and the requirements that a new system or commercial solution should meet.
Consider Your Resources: Assess your budget, timelines, and technical capabilities. Rebuilding requires significant technical resources and time, while replacing involves a financial investment and possible recurring costs.
Explore the Market: If you're considering replacing, explore the market. Look for solutions that are close fits to your needs, have good reputations, and come with solid vendor support.
Agile for Rebuilding: If you're rebuilding, adopt an Agile approach. Prioritize your features, build incrementally, and continually test and gather feedback.
Integration Considerations: For both rebuilding and replacing, consider how the new system will integrate with your existing systems. Integration challenges can cause significant disruption and inefficiency.
User Training: Whichever path you choose, user training will be crucial. Ensure your users are adequately trained and supported during the transition.

Whether it's rebuilding or replacing, both strategies come with their unique challenges and benefits. It's like choosing between building a house or moving into a new one. It's a significant decision, and one that should be made thoughtfully, considering your unique needs, constraints, and future aspirations. So, carefully consider your options, and choose the path that best paves the way for your software evolution.

Final Thoughts

Modernizing legacy systems is no easy task. It requires a deep understanding of the system, its role within your business, and the technologies that can support its evolution. It's a journey of change, and like all journeys, it's best taken one step at a time.

By applying principles like encapsulation, and strategies such as refactoring, re-architecting, and replatforming, we can gradually transform our legacy systems, enhancing their performance, compatibility, and maintainability.

As software architects, we're guides to evolution, helping our organizations adapt and grow. And with the right approaches in our toolkit, we can ensure that our legacy systems don't get left behind, but instead, transform into platforms that support the future of our businesses. So, let's embrace the challenge and start shaping the evolution today!

Using music as a timing source for demos

Anders Tornblad — Mon, 19 Jun 2023 05:45:00 +0000

When it comes to the demoscene productions, the real-time effects realized by a programmer, the graphics and the music are all very impressive separately, but the real magic happens when they are combined together. The music is often the glue that holds the whole production together, and a demo with good timing is always a joy to watch. Often, the music is used as a timing source for the demo, and events in the music are used to trigger transitions and effects.

Subscribing to every row

The first subscription I will implement is to be able to subscribe to every row. For some demos that do a lot of music-triggered effects, this will be very useful. To implement this, the worklet thread will notify the main UI thread for each row, and the main UI thread will be responsible for actually calling the callback functions. Communication between the two threads is done using the MessagePort, which I already use for starting the music. I will take this opportunity to first refactor the code, and create a new class called ModPlayer, which implements the UI thread side of the MOD player.

import { loadMod } from './loader.js';

export class ModPlayer {
    constructor() {
        this.mod = null;
        this.audio = null;
        this.worklet = null;
        this.playing = false;
    }

    async load(url) {
        if (this.worklet) this.unload();
        if (this.playing) this.stop();

        this.mod = await loadMod(url);
        this.audio = new AudioContext();
        await this.audio.audioWorklet.addModule('player-worklet.js');
        this.worklet = new AudioWorkletNode(this.audio, 'player-worklet');
        this.worklet.connect(this.audio.destination);
    }

    unload() {
        if (this.playing) this.stop();

        this.worklet.disconnect();
        this.audio.close();

        this.mod = null;
        this.audio = null;
        this.worklet = null;
    }

    async play() {
        if (this.playing) return;
        if (!this.worklet) return;

        this.audio.resume();
        this.worklet.port.postMessage({
            type: 'play',
            mod: this.mod,
            sampleRate: this.audio.sampleRate
        });

        this.playing = true;
    }

    async stop() {
        if (!this.playing) return;

        this.worklet.port.postMessage({
            type: 'stop'
        });

        this.playing = false;
    }
}

import { ModPlayer } from './player.js';

const modPlayer = new ModPlayer();

const loadButton = document.getElementById('load');
const playButton = document.getElementById('play');
const stopButton = document.getElementById('stop');
const unloadButton = document.getElementById('unload');

loadButton.addEventListener('click', async () => {
    // Load Elekfunk from api.modarchive.org
    const url = 'https://api.modarchive.org/downloads.php?moduleid=41529';
    await modPlayer.load(url);
});

playButton.addEventListener('click', async () => {
    await modPlayer.play();
});

stopButton.addEventListener('click', async () => {
    await modPlayer.stop();
});

unloadButton.addEventListener('click', () => {
    modPlayer.unload();
});

<!DOCTYPE html>
<html>
    <head>
        <title>JS Mod Player by Anders Marzi Tornblad</title>
    </head>
    <body>
        <button id="load">Load</button>
        <button id="play">Play</button>
        <button id="stop">Stop</button>
        <button id="unload">Unload</button>

        <script type="module" src="main.js"></script>
    </body>
</html>

After this refactoring, it's time to add the event subscription mechanism. The worklet thread will send a message to the main UI thread for each row, and the main UI thread will call the registered callback functions.

class PlayerWorklet extends AudioWorkletProcessor {
    constructor() {
        super();
        // ...

        this.publishRow = false;
    }

    onmessage(e) {
        switch (e.data.type) {
            // ...
            case 'enableRowSubscription':
                this.publishRow = true;
                break;
            case 'disableRowSubscription':
                this.publishRow = false;
                break;
        }
    }

    // ...

    nextRow() {
        // ...

        if (this.publishRow) {
            this.port.postMessage({
                type: 'row',
                position: this.position,
                rowIndex: this.rowIndex,
            });
        }
    }

export class ModPlayer {
    constructor() {
        // ...

        this.rowCallbacks = [];
    }

    async load(url) {
        // ...

        this.worklet.port.onmessage = this.onmessage.bind(this);
    }

    onmessage(event) {
        const { data } = event;
        switch (data.type) {
            case 'row':
                for (let callback of this.rowCallbacks) {
                    callback(data.position, data.rowIndex);
                }
                break;
        }
    }

    watchRows(callback) {
        this.worklet.port.postMessage({
            type: 'enableRowSubscription'
        });
        this.rowCallbacks.push(callback);
    }

To use this feature, the user can call watchRows(callback) and pass in a callback function that will be called for each row. I have updated the index.html and main.js files to demonstrate this feature.

<!DOCTYPE html>
<html>
    <head>
        <title>JS Mod Player by Anders Marzi Tornblad</title>
    </head>
    <body>
        <button id="load">Load</button>
        <button id="play">Play</button>
        <button id="stop">Stop</button>
        <button id="unload">Unload</button>
        <br>
        <button id="watchRows">Watch rows</button>
        <br>
        <input type="text" id="output" size="40">

        <script type="module" src="main.js"></script>
    </body>
</html>

// ...
// Store the new button and the output element
const watchRowsButton = document.getElementById('watchRows');
const output = document.getElementById('output');

// ...
// Add a click handler to the new button
watchRowsButton.addEventListener('click', () => {
    // Call the watchRows method and pass in a callback function
    modPlayer.watchRows((position, rowIndex) => {
        // Update the output element with the current position and row index
        output.value = 'Position: ' + position + ', row: ' + rowIndex;
    });
});

Subscribing to known points in time

The next step is to be able to subscribe to known points in time. With this feature, a demo coder can subscribe to a combination of position and row, and a callback will be called when the music reaches that point. For this, I will need to store the callbacks in a dictionary, where the key is a string that represents the position and row. Because there can be multiple callbacks for the same position and row, the values of the dictionary will be an array of callbacks.

export class ModPlayer {
    constructor() {
        // ...
        this.singleCallbacks = { };
    }

    onmessage(event) {
        const { data } = event;
        switch (data.type) {
            case 'row':
                // Call all the general row callbacks
                for (let callback of this.rowCallbacks) {
                    callback(data.position, data.rowIndex);
                }

                // Call all the single row callbacks
                const key = data.position + ':' + data.rowIndex;
                if (this.singleCallbacks[key]) {
                    for (let callback of this.singleCallbacks[key]) {
                        callback(data.position, data.rowIndex);
                    }
                }
                break;
        }
    }

    watch(position, row, callback) {
        this.worklet.port.postMessage({
            type: 'enableRowSubscription'
        });

        // Store the callback in a dictionary
        const key = position + ':' + row;

        // There can be multiple callbacks for the same position and row
        // so we store them in an array
        if (!this.singleCallbacks[key]) {
            this.singleCallbacks[key] = [];
        }

        // Add the callback to the array
        this.singleCallbacks[key].push(callback);
    }

Conclusion

Now I can use the event system to synchronize demos to the music. This will come in handy when I move forward with the Artsy refactoring. I will keep adding to the js-mod-player project as I move forward, and I will push the changes to GitHub. If you like my work, feel free to use it. The player is licenced under a Creative Commons Attribution-NonCommercial 4.0 International License, which means you are free to copy, redistribute, remix, transform and build upon my work, as long as you give appropriate credit, provide a link to the licence, and indicate if changes were made. You may not use my work for commercial purposes or apply legal terms or technological measures that legally restrict others from doing anything the licencse permits.

You can try this solution at atornblad.github.io/js-mod-player.

The latest version of the code is always available in the GitHub repository.

Syncopation and a human touch

Anders Tornblad — Fri, 16 Jun 2023 05:45:00 +0000

The last two effects that I want to implement have to do with note timing. The first is called Retrigger note, and can be used to restart the note from the beginning after a certain number of ticks, and keep repeating it during one row. The second is called Delay note, and is used to delay the start of a note by a certain number of ticks. Both of these effects are used to add some more feeling to the music, and are often added to drum sounds or bass notes.

~~Effect 0 – Arpeggio~~ (done)
~~Effect 1 – Slide up~~ (done)
~~Effect 2 – Slide down~~ (done)
~~Effect 3 – Tone portamento~~ (done)
~~Effect 4 – Vibrato~~ (done)
~~Effect 5 – Tone portamento + volume slide~~ (done)
~~Effect 6 – Vibrato + volume slide~~
~~Effect 9 – Sample offset~~ (done)
~~Effect A – Volume slide~~ (done)
~~Effect C – Set volume~~ (done)
~~Effect D – Pattern break~~ (done)
Effect E9 – Retrigger note
Effect ED – Delay note
~~Effect F – Set speed~~ (done)

Effect E9 – Retrigger note

To retrigger a note, I need to check each tick if it is divisible by the number of ticks specified in the effect parameter. When it is, I can start the sample from the beginning again by setting the sample position to 0:

    performTick() {
        if (this.volumeSlide && this.worklet.tick > 0) {
            // Unchanged
        }

        if (this.vibrato) {
            // Unchanged
        }
        else if (this.periodDelta) {
            // Unchanged
        }
        else if (this.arpeggio) {
            // Unchanged
        }
        else if (this.retrigger && (this.worklet.tick % this.retrigger) == 0) {
            this.sampleIndex = 0;
        }
        // Rest of performTick unchanged
    }

    effect(raw) {
        // Add one line of initialization:
        this.retrigger = false;

        // Everything unchanged apart from one new case in the switch:
        switch (id) {
            // ...
            case RETRIGGER_NOTE:
                this.retrigger = data;
                break;

Effect ED – Delay note

When the Delay note effect is used, the currently playing note must not be interrupted until the specified number of ticks have passed. To implement this, I need to first prepare by storing the changes of instrument, volume and period in separate variables, and only apply them when the delay has passed. This makes the play(note) method a bit more complicated, but it is still quite readable:

    play(note) {
        // Store the changes for later:
        this.setInstrument = false;
        this.setVolume = false;
        this.setPeriod = false;

        if (note.instrument) {
            this.setInstrument = this.worklet.mod.instruments[note.instrument - 1];
            this.setVolume = this.setInstrument.volume;
        }

        this.setSampleIndex = false;
        this.setCurrentPeriod = false;

        if (note.period) {
            this.setPeriod = note.period - (this.setInstrument || this.instrument).finetune;
            this.setCurrentPeriod = true;
            this.setSampleIndex = 0;
        }

        this.effect(note.effect);

        if (this.setInstrument) {
            this.instrument = this.setInstrument;
        }
        if (this.setVolume !== false) {
            this.volume = this.setVolume;
            this.currentVolume = this.volume;
        }

        if (this.setPeriod) {
            this.period = this.setPeriod;
        }

        if (this.setCurrentPeriod) {
            this.currentPeriod = this.period;
        }

        if (this.setSampleIndex !== false) {
            this.sampleIndex = this.setSampleIndex;
        }
    }

After this change, the volume control is a bit broken, but just a small change in the effect method is needed to fix it:

    effect(raw) {
        // ...
        case SET_VOLUME:
            this.setVolume = data;
            break;
        // ...
    }

To finally implement the delay, I need to do some small changes to the performTick, play and effect methods:

    performTick() {
        if (this.volumeSlide && this.worklet.tick > 0) {
            // Unchanged
        }

        if (this.vibrato) {
            // Unchanged
        }
        else if (this.periodDelta) {
            // Unchanged
        }
        else if (this.arpeggio) {
            // Unchanged
        }
        else if (this.retrigger && (this.worklet.tick % this.retrigger) == 0) {
            // Unchanged
        }
        else if (this.delayNote === this.worklet.tick) {
            this.instrument = this.setInstrument;
            this.volume = this.setVolume;
            this.currentVolume = this.volume;
            this.period = this.setPeriod;
            this.currentPeriod = this.period;
            this.sampleIndex = 0;
        }

        // Rest of method unchanged
    }

    play(note) {
        this.setInstrument = false;
        this.setVolume = false;
        this.setPeriod = false;
        this.delayNote = false;

        if (note.instrument) {
            // Unchanged
        }

        this.setSampleIndex = false;
        this.setCurrentPeriod = false;

        if (note.period) {
            // Unchanged
        }

        this.effect(note.effect);

        // If note is delayed, do nothing right now
        if (this.delayNote) return;

        // Rest of method unchanged
    }

    effect(raw) {
        // Add one more line of initialization:
        this.delayNote = false;

        // Everything unchanged apart from one new case in the switch, and
        // a small but important change to SAMPLE_OFFSET:
        switch (id) {
            // ...
            case DELAY_NOTE:
                this.delayNote = data;
                break;
            // ...
            case SAMPLE_OFFSET:
                this.setSampleIndex = data * 256;
                break;
            // ...
        }
    }

Fixing an end-of-song bug

When listening to a few songs from start to end, I noticed that some songs would repeat a single pattern over and over again at the end. This is because I haven't stored the length of the song anywhere, so the player doesn't know when to stop. To fix this, I need to add a length member variable to the Mod class, and check it in the nextRow method of the PlayerWorklet class:

export class Mod {
    constructor(modfile) {
        // Store the song length
        this.length = new Uint8Array(modfile, 950, 1)[0];

        // The rest of the class is unchanged

class PlayerWorklet extends AudioWorkletProcessor {
    // ...

    nextRow() {
        ++this.rowIndex;
        if (this.patternBreak !== false) {
            this.rowIndex = this.patternBreak;
            ++this.position;
            this.patternBreak = false;
        }
        else if (this.rowIndex == 64) {
            this.rowIndex = 0;
            ++this.position;
        }
        if (this.position >= this.mod.length) {
            this.position = 0;
        }

Conclusion

With this, the player is now able to play all the songs that I set out to support. I know there are lot of effects that I haven't implemented yet, but I'm happy with the progress so far. I've learned a lot about the inner workings of the MOD format, and I've also learned a lot about the Web Audio API. I'm looking forward to implementing more effects, but I'll take a break from that for now.

The main reason for making this player is to use it in Artsy, my JavaScript remake of Arte by Sanity. I will use the player for two things: First, I will use it to play the music in the demo. Second, I will use it as a timing source. Because of this, I will need to implement a way to make the player trigger events at certain points in time, driven by the music. That will be the focus of the next article, which will be the final one in this series. I hope to publish it in about a week. Until then, happy modding!

You can try this solution at atornblad.github.io/js-mod-player.

The latest version of the code is always available in the GitHub repository.

Adding pitch-related effects

Anders Tornblad — Thu, 15 Jun 2023 05:45:00 +0000

The vanilla JavaScript MOD player is progressing nicely. Now it's time for some pitch-related effects – vibrato, portamento and arpeggio. I'll also add variants of these effects that let you slide the volume up and down, at the same time as the pitch is changing. These are the effects that I want to implement:

Effect 0 – Arpeggio
Effect 1 – Slide up
Effect 2 – Slide down
Effect 3 – Tone portamento
Effect 4 – Vibrato
Effect 5 – Tone portamento + volume slide
Effect 6 – Vibrato + volume slide
~~Effect 9 – Sample offset~~ (done)
~~Effect A – Volume slide~~ (done)
~~Effect C – Set volume~~ (done)
~~Effect D – Pattern break~~ (done)
Effect E9 – Retrigger note (in next article)
Effect ED – Delay note (in next article)
~~Effect F – Set speed~~ (done)

Effect 1 – Slide up

This effect slides the pitch up by decreasing the period of the note. Each tick, the period is reduced by the value specified in the effect parameter. To manage this, I need to add a currentPeriod variable, and a periodDelta variable to the Channel class, and update the period in the performTick method. The code for calculating the sample speed is moved from the play method to the performTick method, so that the speed is updated every tick. The period can never be less than 113 (B-3, the highest note in the MOD format)_or more than 856 _(C-1, the lowest note), so I clamp the period to that range:

// Showing only the changed methods of the Channel class
    play(note) {
        if (note.instrument) {
            this.instrument = this.worklet.mod.instruments[note.instrument - 1];
            this.sampleIndex = 0;
            this.volume = this.instrument.volume;
            this.currentVolume = this.volume;
        }

        if (note.period) {
            this.period = note.period - this.instrument.finetune;
            this.currentPeriod = this.period;
        }

        this.effect(note.effect);
    }

    performTick() {
        if (this.volumeSlide) {
            this.currentVolume += this.volumeSlide;
            if (this.currentVolume < 0) this.currentVolume = 0;
            if (this.currentVolume > 64) this.currentVolume = 64;
        }

        if (this.periodDelta) {
            this.currentPeriod += this.periodDelta;
            if (this.currentPeriod < 113) this.currentPeriod = 113;
            if (this.currentPeriod > 856) this.currentPeriod = 856;
        }

        const sampleRate = PAULA_FREQUENCY / this.currentPeriod;
        this.sampleSpeed = sampleRate / this.worklet.sampleRate;
    }

    effect(raw) {
        this.volumeSlide = 0;
        this.periodDelta = 0;

        if (!raw) return;

        let id = raw >> 8;
        let data = raw & 0xff;

        if (id == EXTENDED) {
            id = (id << 4) | (data >> 4);
            data = data & 0x0f;
        }

        switch (id) {
            case SET_VOLUME:
                this.volume = data;
                this.currentVolume = this.volume;
                break;
            case SLIDE_UP:
                this.periodDelta = -data;
                break;

        // The rest of the effect method is unchanged

Effect 2 – Slide down

Now that the slide up effect is working, the slide down effect is easy to implement. It comes down to just changing the sign of the periodDelta variable, and the rest is already done:

            case SLIDE_DOWN:
                this.periodDelta = data;
                break;

Effect 3 – Tone portamento

The Tone portamento effect is a combination of the slide up and slide down effects, and it uses the period of the note that is played to determine the direction of the slide. When the pitch has reached the target pitch, the effect is stopped. Tone portamento is also the first effect to implement that "remembers" its parameter between lines. This means that you can play a note and set the a slow portamento effect, using for example 301, and then any consecutive portamento effects with a value of zero will use the same speed. To implement this, I need to make sure that a note playing with the portamento effect will not set the currentPeriod immediately. Instead, the period is only set by the performTick method. One thing to note is that a note playing with this command will not start the sample from the beginning, but instead continue from where it was when the portamento effect was started. To implement this, I need to delay setting the sampleIndex variable until the effect method is called.

    constructor(worklet) {
        // Skipping unchanged lines
        this.portamentoSpeed = 0;
    }

    performTick() {
        // Skipping unchanged volume handling

        if (this.periodDelta) {
            if (this.portamento) {
                if (this.currentPeriod != this.period) {
                    // Which direction to slide?
                    const sign = Math.sign(this.period - this.currentPeriod);
                    // Don't slide past the target period
                    const distance = Math.abs(this.currentPeriod - this.period);
                    const diff = Math.min(distance, this.periodDelta);
                    this.currentPeriod += sign * diff;
                }
            }
            else {
                this.currentPeriod += this.periodDelta;
                if (this.currentPeriod < 113) this.currentPeriod = 113;
                if (this.currentPeriod > 856) this.currentPeriod = 856;
            }
        }

        const sampleRate = PAULA_FREQUENCY / this.currentPeriod;
        this.sampleSpeed = sampleRate / this.worklet.sampleRate;
    }

    play(note) {
        // Keep track of whether the sample should start from the beginning
        this.setSampleIndex = false;

        if (note.instrument) {
            this.instrument = this.worklet.mod.instruments[note.instrument - 1];
            this.setSampleIndex = 0;
            this.volume = this.instrument.volume;
            this.currentVolume = this.volume;
        }

        // Keep track of whether period should change immediately
        this.setCurrentPeriod = false;

        if (note.period) {
            this.period = note.period - this.instrument.finetune;
            this.setCurrentPeriod = true;
        }

        // setSampleIndex and setCurrentPeriod could change back to false in the effect method
        this.effect(note.effect);

        if (this.setCurrentPeriod) {
            this.currentPeriod = this.period;
        }

        if (this.setSampleIndex !== false) {
            this.sampleIndex = this.setSampleIndex;
        }
    }

    effect(raw) {
        this.volumeSlide = 0;
        this.periodDelta = 0;
        this.portamento = false;

        if (!raw) return;

        let id = raw >> 8;
        let data = raw & 0xff;

        if (id == EXTENDED) {
            id = (id << 4) | (data >> 4);
            data = data & 0x0f;
        }

        switch (id) {
            case SET_VOLUME:
                this.volume = data;
                this.currentVolume = this.volume;
                break;
            case SLIDE_UP:
                this.periodDelta = -data;
                break;
            case SLIDE_DOWN:
                this.periodDelta = data;
                break;
            case TONE_PORTAMENTO:
                this.portamento = true;
                if (data) this.portamentoSpeed = data;
                this.periodDelta = this.portamentoSpeed;
                this.setCurrentPeriod = false;
                this.setSampleIndex = false;
                break;

Effect 4 – Vibrato

Vibrato affects the period of the note at each tick. The effective period is calculated by adding a vibrato value to the current period. The upper 4 bits of the parameter is the vibrato speed, and the lower 4 bits is the vibrato depth. It is actually quite tricky to find any good documentation on the correct implementation of this effect, but I found an explanation to get me started in a very old Multimedia Wiki. It appears to use a 64-step sine wave to change the period, and the speed is the number of steps forward in the sine wave per tick. Depth and speed of the vibrato effect are remembered individually between notes for each channel.

    constructor(worklet) {
        // Added three variables
        this.vibratoDepth = 0;
        this.vibratoSpeed = 0;
        this.vibratoIndex = 0;
    }

    performTick() {
        if (this.volumeSlide) {
            this.currentVolume += this.volumeSlide;
            if (this.currentVolume < 0) this.currentVolume = 0;
            if (this.currentVolume > 64) this.currentVolume = 64;
        }

        if (this.vibrato) {
            this.vibratoIndex = (this.vibratoIndex + this.vibratoSpeed) % 64;
            this.currentPeriod = this.period + Math.sin(this.vibratoIndex / 64 * Math.PI * 2) * this.vibratoDepth;
        }
        else if (this.periodDelta) {
            if (this.portamento) {
                if (this.currentPeriod != this.period) {
                    const sign = Math.sign(this.period - this.currentPeriod);
                    const distance = Math.abs(this.currentPeriod - this.period);
                    const diff = Math.min(distance, this.periodDelta);
                    this.currentPeriod += sign * diff;
                }
            }
            else {
                this.currentPeriod += this.periodDelta;
            }
        }

        if (this.currentPeriod < 113) this.currentPeriod = 113;
        if (this.currentPeriod > 856) this.currentPeriod = 856;

        const sampleRate = PAULA_FREQUENCY / this.currentPeriod;
        this.sampleSpeed = sampleRate / this.worklet.sampleRate;
    }

    play(note) {
        this.setSampleIndex = false;

        if (note.instrument) {
            this.instrument = this.worklet.mod.instruments[note.instrument - 1];
            this.setSampleIndex = 0;
            this.volume = this.instrument.volume;
            this.currentVolume = this.volume;
        }

        this.setCurrentPeriod = false;

        if (note.period) {
            this.period = note.period - this.instrument.finetune;
            this.setCurrentPeriod = true;
        }

        this.effect(note.effect);

        if (this.setCurrentPeriod) {
            this.currentPeriod = this.period;
        }

        if (this.setSampleIndex !== false) {
            this.sampleIndex = this.setSampleIndex;
        }
    }

    effect(raw) {
        this.volumeSlide = 0;
        this.periodDelta = 0;
        this.portamento = false;
        this.vibrato = false;

        if (!raw) return;

        let id = raw >> 8;
        let data = raw & 0xff;

        if (id == EXTENDED) {
            id = (id << 4) | (data >> 4);
            data = data & 0x0f;
        }

        switch (id) {
            // Added one case
            case VIBRATO:
                const speed = data >> 4;
                const depth = data & 0x0f;
                if (speed) this.vibratoSpeed = speed;
                if (depth) this.vibratoDepth = depth;
                this.vibrato = true;
                break;

Effect 5 – Tone portamento + volume slide

This effect is a combination of effects 3 and A, which are already implemented. The only changes needed to the effect method is to add a new case to the switch statement, and combine the two existing effects:

            case TONE_PORTAMENTO_WITH_VOLUME_SLIDE:
                this.portamento = true;
                this.setCurrentPeriod = false;
                this.setSampleIndex = false;
                this.periodDelta = this.portamentoSpeed;
                if (data & 0xf0) {
                    this.volumeSlide = data >> 4;
                }
                else if (data & 0x0f) {
                    this.volumeSlide = -(data & 0x0f);
                }
                break;

Effect 6 – Vibrato + volume slide

Just like effect 5 is a combination of effects 3 and A, effect 6 is a combination of effects 4 and A.This means that everything is already in place to implement this effect. The only thing that needs to be done is to add a case for it in the effect method of the Channel class.

            case VIBRATO_WITH_VOLUME_SLIDE:
                this.vibrato = true;
                if (data & 0xf0) {
                    this.volumeSlide = data >> 4;
                }
                else if (data & 0x0f) {
                    this.volumeSlide = -(data & 0x0f);
                }
                break;

Effect 0 – Arpeggio

This effect is used to play a chord by playing three notes in quick succession. The first note is the note that is played, and the second and third notes are played by adding the arpeggio values to the note. The arpeggio values are set by the upper 4 bits and the lower 4 bits of the parameter. Every tick, the period is changed to the next note in the chord.

    constructor(worklet) {
        // Adding one line:
        this.arpeggio = false;
    }

    performTick() {
        if (this.volumeSlide && this.worklet.tick > 0) {
            // Unchanged
        }

        if (this.vibrato) {
            // Unchanged
        }
        else if (this.periodDelta) {
            // Unchanged
        }
        else if (this.arpeggio) {
            const index = this.worklet.tick % this.arpeggio.length;
            const halfNotes = this.arpeggio[index];
            this.currentPeriod = this.period / Math.pow(2, halfNotes / 12);
        }

        if (this.currentPeriod < 113) this.currentPeriod = 113;
        if (this.currentPeriod > 856) this.currentPeriod = 856;

        const sampleRate = PAULA_FREQUENCY / this.currentPeriod;
        this.sampleSpeed = sampleRate / this.worklet.sampleRate;
    }

    effect(raw) {
        // Add one line at the beginning
        this.arpeggio = false;

        // In the switch statement:
        switch (id) {
            case ARPEGGIO:
                this.arpeggio = [0, data >> 4, data & 0x0f];
                break;

Some bug fixes

Now, the three songs I have chosen should play almost perfectly. However, when listening to them, I realize that there are some bugs that need to be fixed. The first one is that notes should restart from the beginning of the sample when there is a new note, even if the instrument is set to zero. I discovered this by listening to the song from Phenomena Enigma. In pattern 30 (song position 45) channel 3 sounds almost silent. The fix is to move the code that sets the sample index to another position:

    play(note) {
        if (note.instrument) {
            this.instrument = this.worklet.mod.instruments[note.instrument - 1];
            this.volume = this.instrument.volume;
            this.currentVolume = this.volume;
        }

        this.setSampleIndex = false;
        this.setCurrentPeriod = false;

        if (note.period) {
            this.period = note.period - this.instrument.finetune;
            this.setCurrentPeriod = true;
            this.setSampleIndex = 0;
        }

        this.effect(note.effect);

        if (this.setCurrentPeriod) {
            this.currentPeriod = this.period;
        }

        if (this.setSampleIndex !== false) {
            this.sampleIndex = this.setSampleIndex;
        }
    }

The other bug is that the volume slide effect runs a bit too fast. When listening to the opening of Livin' Insanity , in patterns 2 and 4 (song positions 0 and 1) the volume goes down to zero too quickly. After some googling, I found that the volume slide effect should not be run on the first tick of each row. So for a normal speed setting of 6 ticks per row, the volume slide effect should not run on tick 0, but should run on ticks 1, 2, 3, 4 and 5.

    performTick() {
        if (this.volumeSlide && this.worklet.tick > 0) {
            this.currentVolume += this.volumeSlide;
            if (this.currentVolume < 0) this.currentVolume = 0;
            if (this.currentVolume > 64) this.currentVolume = 64;
        }

Conclusion

The only two effects that are missing right now for reaching my goal are Retrigger note and Delay note, and I'll implement those in the next article. Those effects are not used in the song from Enigma, so that one actually plays correctly right now. So far, this has been a fun challenge, and I have learned a lot about the inner workings of the MOD format.

You can try this solution at atornblad.github.io/js-mod-player.

The latest version of the code is always available in the GitHub repository.

Implementing looping and the first effects

Anders Tornblad — Wed, 14 Jun 2023 05:45:00 +0000

Looping samples

Currently, the player can only play "one-off" samples, which means that they are played once and then stopped. This means that looping samples don't work properly. In the instrument data, there are two fields called repeatOffset and repeatLength. These are used to define the loop points of the sample. For samples without looping, repeatOffset is set to 0 and repeatLength is set to 0 or 2, depending on which version of the MOD format is used. I add a new member variable to the Instrument class to check if the sample is looped: isLooped, and use that in the Channel class to determine if the sample should be looped or not.

class Instrument {
    constructor(modfile, index, sampleStart) {
        const data = new Uint8Array(modfile, 20 + index * 30, 30);
        const nameBytes = data.slice(0, 21).filter(a => !!a);
        this.name = String.fromCodePoint(...nameBytes).trim();
        this.length = 2 * (data[22] * 256 + data[23]);
        this.finetune = data[24]; // Signed 4 bit integer
        if (this.finetune > 7) this.finetune -= 16;
        this.volume = data[25];
        this.repeatOffset = 2 * (data[26] * 256 + data[27]);
        this.repeatLength = 2 * (data[28] * 256 + data[29]);
        this.bytes = new Int8Array(modfile, sampleStart, this.length);
        this.isLooped = this.repeatOffset != 0 || this.repeatLength > 2;
    }
}

    nextOutput() {
        if (!this.instrument || !this.period) return 0.0;
        const sample = this.instrument.bytes[this.sampleIndex | 0];

        this.sampleIndex += this.sampleSpeed;

        if (this.instrument.isLooped) {
            if (this.sampleIndex >= this.instrument.repeatOffset + this.instrument.repeatLength) {
                this.sampleIndex = this.instrument.repeatOffset;
            }
        }
        else if (this.sampleIndex >= this.instrument.length) {
            return 0.0;
        }

        return sample / 256.0;
    }

Finding the most important effects

The MOD format defines a lot of effects, but some of them are not used very often in demo songs. My goal for this article series is to be able to play three songs:

Musiklinjen: The song from Enigma by Phenomena
Livin' Instanity: The first song from Arte by Sanity
Elekfunk: The second song from Arte by Sanity

To find out which effects are used by those three songs, I refactor the effect method of the Channel class to log the effect id of each unimplemented effect. After reading some more about the different effect IDs, I also find out that the 0x0E effect is used as a prefix for extended effects, so that is also added.

// Declared at the top of the file, outside of the class definition
const unimplementedEffects = new Set();
const EXTENDED = 0x0e;
const SET_SPEED = 0x0f;

// Updated effect method of the Channel class
    effect(raw) {
        let id = raw >> 8;
        let data = raw & 0xff;

        if (id == EXTENDED) {
            // Effect ID can be E0..EF
            id = (id << 4) | (data >> 4);
            data = data & 0x0f;
        }

        if (id == SET_SPEED) {
            if (data >= 1 && data <= 31) {
                this.worklet.setTicksPerRow(data);
            }
            else {
                this.worklet.setBpm(data);
            }
        }
        else {
            if (!unimplementedEffects.has(id)) {
                unimplementedEffects.add(id);
                console.log('Unimplemented effect ' + id.toString(16));
            }
        }
    }

After running the player with the three songs, I get the following list of effects to implement, and I can add definitions for them to the player-worklet.js file:

Effect 0 – Arpeggio
Effect 1 – Slide up
Effect 2 – Slide down
Effect 3 – Tone portamento
Effect 4 – Vibrato
Effect 5 – Tone portamento + volume slide
Effect 6 – Vibrato + volume slide
Effect 9 – Sample offset
Effect A – Volume slide
Effect C – Set volume
Effect D – Pattern break
Effect E9 – Retrigger note
Effect ED – Delay note
~~Effect F – Set speed~~ (done)

const ARPEGGIO = 0x00;
const SLIDE_UP = 0x01;
const SLIDE_DOWN = 0x02;
const TONE_PORTAMENTO = 0x03;
const VIBRATO = 0x04;
const TONE_PORTAMENTO_WITH_VOLUME_SLIDE = 0x05;
const VIBRATO_WITH_VOLUME_SLIDE = 0x06;
const SAMPLE_OFFSET = 0x09;
const VOLUME_SLIDE = 0x0A;
const SET_VOLUME = 0x0C;
const PATTERN_BREAK = 0x0D;
const EXTENDED = 0x0e;
const SET_SPEED = 0x0f;
const RETRIGGER_NOTE = 0xe9;
const DELAY_NOTE = 0xed;

In the documentation I have found, the effects are described in a very technical way, but with a lot of details missing. Also, it is impossible to know exactly what the effects are supposed to sound like without hearing them in action. Fortunately, I have found a great source of information in the form of a YouTube playlist by demoscene musician Alex Löfgren (Wasp) where a lot of the effects are explained in detail, and shown in action. Throughout the implementation of the effects, I'm' listening to the examples in the playlist to make sure that I get them right, and also making minimal test songs in ProTracker to try out the effects.

In this article, I implement four of the effects, and I leave the rest for future articles:

Effect C – Set volume
Effect 9 – Sample offset
Effect D – Pattern break
Effect A – Volume slide

Effect C – Set volume

So far, all samples are playing at full volume, which is not correct. First of all, each instrument has a volume property, which is a value between 0 and 64 (the maximum volume for audio in the Amiga Paula chip). Second, the Set volume effect can be used to override the volume of the instrument. It can be set when a note is triggered, or changed at any time during the note. To implement this, I add a volume property to the Channel class, update the nextOutput method to use the channel volume instead of the instrument volume, and update the effect method to set the volume when the effect is encountered.

    constructor(worklet) {
        this.worklet = worklet;
        this.instrument = null;
        this.playing = false;
        this.period = 0;
        this.sampleSpeed = 0.0;
        this.sampleIndex = 0;
        this.volume = 64;
    }

    nextOutput() {
        if (!this.instrument || !this.period) return 0.0;
        const sample = this.instrument.bytes[this.sampleIndex | 0];

        this.sampleIndex += this.sampleSpeed;

        if (this.instrument.isLooped) {
            if (this.sampleIndex >= this.instrument.repeatOffset + this.instrument.repeatLength) {
                this.sampleIndex = this.instrument.repeatOffset;
            }
        }
        else if (this.sampleIndex >= this.instrument.length) {
            return 0.0;
        }

        return sample / 256.0 * this.volume / 64;
    }

        switch (id) {
            case SET_VOLUME:
                this.volume = data;
                break;
            case SET_SPEED:
                if (data >= 1 && data <= 31) {
                    this.worklet.setTicksPerRow(data);
                }
                else {
                    this.worklet.setBpm(data);
                }
                break;
            default:
                if (!unimplementedEffects.has(id)) {
                    unimplementedEffects.add(id);
                    console.log('Unimplemented effect ' + id.toString(16));
                }
                break;
        }

Effect 9 – Sample offset

This effect makes the player set the position in the sample to a specific value. The value is the upper eight bits of the sample offset, so it must be multiplied by 256 to get the actual offset. The effect can be used to jump forward and backward in a sample while it is playing, or to start from a selected position when triggering a note. Implementing this is very simple, I just need to update the effect method to set the sample index when the effect is encountered.

            case SAMPLE_OFFSET:
                this.sampleIndex = data * 256;
                break;

Effect D – Pattern break

This effect makes the player jump to a specific row in the next pattern. The jump takes place after the current row has finished playing, so the effect is not triggered immediately. The value of the effect is the row number to start playing in the next pattern, in binary coded decimal notation. The first hexadecimal digit is the "tens" of the row number, and the second hexadecimal digit is the "ones". To make this work, I add a patternBreak member variable of the PlayerWorklet class that I initialize to false in the constructor, and I add a setPatternBreak method , that I call from the effect method when the effect is encountered.

    constructor() {
        super();
        this.port.onmessage = this.onmessage.bind(this);
        this.mod = null;
        this.channels = [ new Channel(this), new Channel(this), new Channel(this), new Channel(this) ];
        this.patternBreak = false;
    }

    setPatternBreak(row) {
        this.patternBreak = row;
    }

    nextRow() {
        ++this.rowIndex;
        if (this.patternBreak !== false) {
            this.rowIndex = this.patternBreak;
            ++this.position;
            this.patternBreak = false;
        }
        else if (this.rowIndex == 64) {
            this.rowIndex = 0;
            ++this.position;
        }

        // The rest of the nextRow method is unchanged

            case PATTERN_BREAK:
                const row = (data >> 4) * 10 + (data & 0x0f);
                this.worklet.setPatternBreak(row);
                break;

Effect A – Volume slide

The Volume slide effect works by repeatedly changing the volume of the channel, a number of times per row. Since the number of times is the same as the ticksPerRow value set by the Set speed effect, there is already an idea of "ticks" in the playback code. Unfortunately, the player doesn't keep track of the current tick, so I need to add a tick member variable and a nextTick method to the PlayerWorklet class. Instead of keeping track of when to move to the next row, I will keep track of when to move to the next tick, and call nextRow when the tick reaches the ticksPerRow value. When all this is done, I can implement the Volume slide effect, and other effects that use the tick count.

// Only the changed methods are shown here
    onmessage(e) {
        if (e.data.type === 'play') {
            this.mod = e.data.mod;
            this.sampleRate = e.data.sampleRate;

            this.setBpm(125);
            this.setTicksPerRow(6);

            // Start at the last tick of the pattern "before the first pattern"
            this.position = -1;
            this.rowIndex = 63;
            this.tick = 5;
            this.ticksPerRow = 6;

            // Immediately move to the first row of the first pattern
            this.outputsUntilNextTick = 0;
        }
    }

    setTicksPerRow(ticksPerRow) {
        this.ticksPerRow = ticksPerRow;
    }

    setBpm(bpm) {
        this.bpm = bpm;
        this.outputsPerTick = this.sampleRate * 60 / this.bpm / 4 / 6;
    }

    nextTick() {
        ++this.tick;
        if (this.tick == this.ticksPerRow) {
            this.tick = 0;
            this.nextRow();
        }

        for (let i = 0; i < 4; ++i) {
            this.channels[i].performTick();
        }
    }

    nextOutput() {
        if (!this.mod) return 0.0;

        if (this.outputsUntilNextTick <= 0) {
            this.nextTick();
            this.outputsUntilNextTick += this.outputsPerTick;
        }

        this.outputsUntilNextTick--;

        const rawOutput = this.channels.reduce((acc, channel) => acc + channel.nextOutput(), 0.0);
        return Math.tanh(rawOutput);
    }

// Only the changed methods are shown here
    nextOutput() {
        if (!this.instrument || !this.period) return 0.0;
        const sample = this.instrument.bytes[this.sampleIndex | 0];

        this.sampleIndex += this.sampleSpeed;

        if (this.instrument.isLooped) {
            if (this.sampleIndex >= this.instrument.repeatOffset + this.instrument.repeatLength) {
                this.sampleIndex = this.instrument.repeatOffset;
            }
        }
        else if (this.sampleIndex >= this.instrument.length) {
            return 0.0;
        }

        return sample / 256.0 * this.currentVolume / 64;
    }

    performTick() {
        if (this.volumeSlide) {
            this.currentVolume += this.volumeSlide;
            if (this.currentVolume < 0) this.currentVolume = 0;
            if (this.currentVolume > 64) this.currentVolume = 64;
        }
    }

    effect(raw) {
        this.volumeSlide = 0;

        if (!raw) return;

        let id = raw >> 8;
        let data = raw & 0xff;

        if (id == EXTENDED) {
            id = (id << 4) | (data >> 4);
            data = data & 0x0f;
        }

        switch (id) {
            case SET_VOLUME:
                this.volume = data;
                this.currentVolume = this.volume;
                break;
            case VOLUME_SLIDE:
                if (data & 0xf0) {
                    this.volumeSlide = data >> 4;
                }
                else if (data & 0x0f) {
                    this.volumeSlide = -(data & 0x0f);
                }
                break;
        // The rest of the effect method is unchanged

Conclusion

Effects are a very important part of the musical experience of MOD files, and after just these four effects, the reference MODs are starting to sound much better. In the next couple of article, I will implement the remaining effects, and then I will move on to implementing more things that will help me use MOD files in a demo.

You can try this solution at atornblad.github.io/js-mod-player.

The latest version of the code is always available in the GitHub repository.

Playing a full song, almost

Anders Tornblad — Tue, 13 Jun 2023 05:45:00 +0000

In the previous article, I described how to extract the raw sample data from a MOD file and how to play it. The next step is to get the song data out from the MOD file and into a more useful data structure.

According to the sources I found (see the previous article), the song data is stored using 1024 bytes per "pattern". A pattern is a sequence of 64 notes in 4 channels, each note being a 1/16 of a bar. Original MOD files contain four channels, which means that a total of four samples, or instruments, can be played at the same time. Each note is 4 bytes, each row contains 4 notes, and each pattern contains 64 rows, which makes 1024 bytes per pattern.

For each note, the instrument index, the period (explained later), and the effect (also explained later) are stored. The instrument index is stored in the first 4 bits of byte 0 and the first 4 bits of byte 2. The period is stored in the last 4 bits of byte 0 and all 8 bits of byte 1. The effect is stored in the last 4 bits of byte 2 and all 8 bits of byte 3. Using this information, I can write three new classes to store the song data, Pattern, Row, and Note.

class Note {
    constructor(noteData) {
        this.instrument = (noteData[0] & 0xf0) | (noteData[2] >> 4);
        this.period = (noteData[0] & 0x0f) * 256 + noteData[1];
        this.effect = (noteData[2] & 0x0f) * 256 + noteData[3];
    }
}

class Row {
    constructor(rowData) {
        this.notes = [];

        // Each note is 4 bytes
        for (let i = 0; i < 16; i += 4) {
            const noteData = rowData.slice(i, i + 4);
            this.notes.push(new Note(noteData));
        }
    }
}

class Pattern {
    constructor(modfile, index) {
        // Each pattern is 1024 bytes long
        const data = new Uint8Array(modfile, 1084 + index * 1024, 1024);
        this.rows = [];

        // Each pattern is made up of 64 rows
        for (let i = 0; i < 64; ++i) {
            const rowData = data.slice(i * 16, i * 16 + 16);
            this.rows.push(new Row(rowData));
        }
    }
}

The constructor of the Mod class must also be updated to read the song data, by calling the Pattern constructor for each pattern.

export class Mod {
    constructor(modfile) {
        // Store the pattern table
        this.patternTable = new Uint8Array(modfile, 952, 128);

        // Find the highest pattern number
        const maxPatternIndex = Math.max(...this.patternTable);

        // Extract all instruments
        this.instruments = [];
        let sampleStart = 1084 + (maxPatternIndex + 1) * 1024;
        for (let i = 0; i < 31; ++i) {
            const instr = new Instrument(modfile, i, sampleStart);
            this.instruments.push(instr);
            sampleStart += instr.length;
        }

        // Extract the pattern data
        this.patterns = [];
        for (let i = 0; i <= maxPatternIndex; ++i) {
            const pattern = new Pattern(modfile, i);
            this.patterns.push(pattern);
        }
    }
}

Playing one channel of one pattern

Now that the song data is stored in a more useful format, it is time to play it. The first step is to play one channel of one pattern. The playback is driven by the process method of the worklet processor class. For each sample that is played, time moves forward by 1/48 000 or 1/44 100 of a second, depending on the sample rate of the audio context. Because of this, we can use the samples as a time source, and base all time calculations on the progression of samples.

The period of a note is the number of clock pulses of the Amiga Paula co-processor, which run at half the speed of the main CPU clock. On a PAL Amiga, the main CPU clock frequency is 7 093 789.2 Hz, so the co-processor frequency is 3 546 894.6 Hz. A sample playing with period 214 (the period of the note C-3) will play at 3 546 894.6/214 = 16 574.3 Hz. For an audio context with a sample rate of 48 000 Hz, this means that each sample will be output 48 000/16 574.3 = 2.896 times before the next sample is output. These calculations must be done each time the period changes.

To step forward in the song, i also need to keep track of the current position in the song, the current row to play, and how many samples are left to play before moving to the next row. For now, the number of samples per row is calculated using the BPM of 125, and the sample rate of the audio context.

const PAULA_FREQUENCY = 3546894.6;

class PlayerWorklet extends AudioWorkletProcessor {
    constructor() {
        super();
        this.port.onmessage = this.onmessage.bind(this);
        this.mod = null;
    }

    onmessage(e) {
        if (e.data.type === 'play') {
            this.mod = e.data.mod;
            this.sampleRate = e.data.sampleRate;

            this.bpm = 125;
            this.outputsPerRow = this.sampleRate * 60 / this.bpm / 4;

            // Start at the last row of the pattern "before the first pattern"
            this.position = -1;
            this.rowIndex = 63;

            // Immediately move to the first row of the first pattern
            this.outputsUntilNextRow = 0;

            this.instrument = null;
            this.period = null;
        }
    }

    nextRow() {
        ++this.rowIndex;
        if (this.rowIndex == 64) {
            this.rowIndex = 0;
            ++this.position;
        }

        const patternIndex = this.mod.patternTable[this.position];
        const pattern = this.mod.patterns[patternIndex];
        const row = pattern.rows[this.rowIndex];
        const note = row.notes[0];

        if (note.instrument) {
            this.instrument = this.mod.instruments[note.instrument - 1];
            this.sampleIndex = 0;
        }

        if (note.period) {
            this.period = note.period - this.instrument.finetune;
        }
    }

    nextOutput() {
        if (!this.mod) return 0.0;

        if (this.outputsUntilNextRow <= 0) {
            this.nextRow();
            this.outputsUntilNextRow += this.outputsPerRow;
        }

        this.outputsUntilNextRow--;

        if (!this.instrument || !this.period) return 0.0;
        if (!this.period) return 0.0;

        if (this.sampleIndex >= this.instrument.bytes.length) {
            return 0.0;
        }

        const sample = this.instrument.bytes[this.sampleIndex | 0];
        const sampleRate = PAULA_FREQUENCY / this.period;
        this.sampleIndex += sampleRate / this.sampleRate;

        return sample / 256.0;
    }

    process(inputs, outputs) {
        const output = outputs[0];
        const channel = output[0];

        for (let i = 0; i < channel.length; ++i) {
            const value = this.nextOutput();
            channel[i] = value;
        }

        return true;
    }
}

registerProcessor('player-worklet', PlayerWorklet);

To finally play the first channel of the song, I need to send the complete mod data to the worklet, and not just one instrument. I also need to send the sample rate of the audio context, so that the worklet can calculate all timings correctly. Here are the changes to the main thread code:

// Load Elekfunk from api.modarchive.org
const url = 'https://api.modarchive.org/downloads.php?moduleid=41529';
const mod = await loadMod(url);

// Play a sample when the user clicks
window.addEventListener('click', () => {
    audio.resume();
    player.port.postMessage({
        type: 'play',
        mod: mod,
        sampleRate: audio.sampleRate
    });
});

After these changes, when you click the browser window, the first channel of the song starts playing. However, the timing is a bit wrong, and some of the notes are not playing correctly. This is because no effects have been implemented yet. The Elekfunk song uses the currently unimplemented "Set Speed" effect to speed up the playback, and that's why the timing is wrong. Also, to really enjoy the song, all four channels need to be mixed together. The last changes I make in this post are to create a four-channel mixer, and to implement the "Set Speed" effect.

Mixing multiple channels

To mix the four channels of the song together, I need to keep track of the instrument and period for each channel. Each channel needs to have its own sample index, and move forward in the sample independently of the other channels. I create a Channel class to keep track of this information, and I create four instances of this class in the worklet. The nextOutput function now loops over all four channels, and adds the output of each channel together. To limit the output to the range of -1.0 to 1.0, I use the Math.tanh function, which is a hyperbolic tangent function. It has the nice property that the output is always between -1.0 and 1.0, and the output is also scaled to the input.

class Channel {
    constructor(worklet) {
        this.worklet = worklet;
        this.instrument = null;
        this.period = 0;
        this.sampleSpeed = 0.0;
        this.sampleIndex = 0;
    }

    nextOutput() {
        if (!this.instrument || !this.period) return 0.0;
        const sample = this.instrument.bytes[this.sampleIndex | 0];

        this.sampleIndex += this.sampleSpeed;
        if (this.sampleIndex >= this.instrument.length) {
            return 0.0;
        }

        return sample / 256.0;
    }

    play(note) {
        if (note.instrument) {
            this.instrument = this.worklet.mod.instruments[note.instrument - 1];
            this.sampleIndex = 0;
        }

        if (note.period) {
            this.period = note.period - this.instrument.finetune;
            const sampleRate = PAULA_FREQUENCY / this.period;
            this.sampleSpeed = sampleRate / this.worklet.sampleRate;
        }
    }
}

class PlayerWorklet extends AudioWorkletProcessor {
    constructor() {
        super();
        this.port.onmessage = this.onmessage.bind(this);
        this.mod = null;
        this.channels = [ new Channel(this), new Channel(this), new Channel(this), new Channel(this) ];
    }

    onmessage(e) {
        if (e.data.type === 'play') {
            this.mod = e.data.mod;
            this.sampleRate = e.data.sampleRate;

            this.bpm = 125;
            this.outputsPerRow = this.sampleRate * 60 / this.bpm / 4;

            // Start at the last row of the pattern "before the first pattern"
            this.position = -1;
            this.rowIndex = 63;

            // Immediately move to the first row of the first pattern
            this.outputsUntilNextRow = 0;
        }
    }

    nextRow() {
        ++this.rowIndex;
        if (this.rowIndex == 64) {
            this.rowIndex = 0;
            ++this.position;
        }

        const patternIndex = this.mod.patternTable[this.position];
        const pattern = this.mod.patterns[patternIndex];
        const row = pattern.rows[this.rowIndex];

        for (let i = 0; i < 4; ++i) {
            this.channels[i].play(row.notes[i]);
        }
    }

    nextOutput() {
        if (!this.mod) return 0.0;

        if (this.outputsUntilNextRow <= 0) {
        if (this.outputsUntilNextRow <= 0) {
            this.nextRow();
        }

        this.outputsUntilNextRow--;

        const rawOutput = this.channels.reduce((acc, channel) => acc + channel.nextOutput(), 0.0);
        return Math.tanh(rawOutput);
    }

    process(inputs, outputs) {
        const output = outputs[0];
        const channel = output[0];

        for (let i = 0; i < channel.length; ++i) {
            const value = this.nextOutput();
            channel[i] = value;
        }

        return true;
    }
}

Adding the first effect

The first effect I want to implement is the "Set Speed" effect. This effect can be used in two different ways. If the value is less than 32, it sets the number of "ticks" per row. If the value is greater than or equal to 32, it sets the number of beats per minute. A "tick" is a concept of the MOD format, and is normally a 1/6th of a row. Most older MOD files never change the BPM, but instead changes the number of ticks per row to speed up or slow down the song. A lot of effects perform calculations based on the number of ticks per row, so it's important to implement this effect. I'll start by adding an effect method to the Channel class, which I can use to implement all the effects. I'll also add a setTicksPerRow method and a setBpm method to the PlayerWorklet class, which I can use to update the number of ticks per row.

    play(note) {
        if (note.instrument) {
            this.instrument = this.worklet.mod.instruments[note.instrument - 1];
            this.sampleIndex = 0;
        }

        if (note.period) {
            this.period = note.period - this.instrument.finetune;
            const sampleRate = PAULA_FREQUENCY / this.period;
            this.sampleSpeed = sampleRate / this.worklet.sampleRate;
        }

        if (note.effect) {
            this.effect(note.effect);
        }
    }

    effect(raw) {
        const id = raw >> 8;
        const data = raw & 0xff;
        if (id == 0x0f) {
            if (data >= 1 && data <= 31) {
                this.worklet.setTicksPerRow(data)
            }
            else {
                this.worklet.setBpm(data);
            }
        }
    }

    onmessage(e) {
        if (e.data.type === 'play') {
            this.mod = e.data.mod;
            this.sampleRate = e.data.sampleRate;

            this.setBpm(125);
            this.setTicksPerRow(6);

            // Start at the last row of the pattern "before the first pattern"
            this.position = -1;
            this.rowIndex = 63;

            // Immediately move to the first row of the first pattern
            this.outputsUntilNextRow = 0;
        }
    }

    setTicksPerRow(ticksPerRow) {
        this.ticksPerRow = ticksPerRow;
        this.outputsPerRow = this.sampleRate * 60 / this.bpm / 4 * this.ticksPerRow / 6;
    }

    setBpm(bpm) {
        this.bpm = bpm;
        this.outputsPerRow = this.sampleRate * 60 / this.bpm / 4 * this.ticksPerRow / 6;
    }

Conclusion

With these changes, the player can now render a somewhat decently sounding version of Elekfunk by Moby, from the Arte demo. The instruments play correctly and the song is played at the correct speed, but there are still a lot of effects missing. Those will be the subject of the next couple of parts of this series.

You can try this solution at atornblad.github.io/js-mod-player.

The latest version of the code is always available in the GitHub repository.

DEV Community: Anders Tornblad

Building business processes with Azure Durable Functions

Traditional approaches vs. modern cloud-native development

Example implementation

Step-by-Step Explanation

Durable Entity for Time Report

Orchestration Function

Activity Functions

Example Usage

Enhancing Productivity and Onboarding

Middleware for Security, Monitoring, and Error Handling

Security

Monitoring

Error Handling

Example implementations

Conclusion

Teach the present first, history later

The issue with historical sequencing

Teaching the present

Integrating historical context

Updating teaching practices

Computer engineering studies

Conclusion

Scale is a feature

LGBTQIA+ ally flag in pure CSS

The role of software architects in Agile teams

Understanding Agile teams

The composition of Agile teams

The software architect

Who is a software architect?

The importance of the software architect

The software architect in Agile teams

Collaboration with stakeholders

Working with Developers

Modern trends

Evolutionary architecture

Emergent architecture

The architect's role in Scaled Agile Framework

The software architect in SAFe

Conclusion

Further reading

Making the MOD player available

Content Security Policy and the browser security model

Evolving Legacy Software Architecture

Encapsulation

Encapsulating Legacy Systems

Encapsulating Business Logic

Agile and Encapsulation

Leading the Encapsulation Process

Replatforming

Why Replatform?

Agile Advantage in Replatforming

Leading Replatforming Efforts: Key Considerations

Refactoring and the Strangler Fig Pattern

Refactoring

The Strangler Fig Pattern

Leading the Refactoring and Strangler Fig Pattern Process

Re-architecting

The Agile Intersection with Re-architecting

Leading Re-architecting Efforts

Rebuilding and Replacing

Rebuilding

Replacing

Navigating the Rebuilding and Replacing Process

Final Thoughts

Further reading

Using music as a timing source for demos

Subscribing to every row

Subscribing to known points in time

Conclusion

Syncopation and a human touch

Effect E9 – Retrigger note

Effect ED – Delay note

Fixing an end-of-song bug

Conclusion

Adding pitch-related effects

Effect 1 – Slide up

Effect 2 – Slide down

Effect 3 – Tone portamento

Effect 4 – Vibrato