DEV Community: Márk Sági-Kazár

How to become an Open Source Software contributor?

Márk Sági-Kazár — Mon, 24 Jul 2023 23:19:10 +0000

Originally published at https://sagikazarmark.hu.

A question I often get from people—both from beginners stepping into the world of programming and seasoned developers alike—is about how they can start contributing to open source software projects. In this post, my aim is to provide practical advice that will guide you to become not only an active participant in the open source community but also a significant contributor to it.

There's a certain allure to the idea of open source, not only because of its collaborative nature that allows for diverse minds to come together, but also the ability to actively make a difference in the software tools that many of us use daily. No wonder so many people aspire to be a part of this innovative community.

Unfortunately, a perceived high bar often deters people from becoming contributors. This perception is misguided, as many fail to realize the significant value they can bring to a project simply by participating or providing feedback. The common belief that meaningful contributions must consist of hundreds of lines of code could not be further from the truth.

My journey in the world of open source software

I've spent most of my career working on open source software projects. I consider myself fortunate, as not only did working on open source projects shape my career in IT, but it also evolved into a hobby over the years. I've invested countless hours in numerous side-projects, some of which have gained significant popularity over time, and I've made contributions to every project that crossed my path.

While it may seem as though I'm boasting, this journey has in fact been quite a humbling experience. Beyond learning that I'm rarely the smartest person in the room, it provided perspective on what truly matters in the lifespan of an open source software project. Of course, coding and discussions play a crucial role, but as time progresses, things such as high-quality bug reports, timely feature requests, up-to-date documentation, responding to issues, and building a community become just as important as anything else.

As a maintainer, I've come to appreciate all kinds of contributions. As a contributor, I've learned various ways to give back to the community.

Over the years, I've helped many people become contributors and then maintainers in open source projects. I've witnessed their struggles and triumphs, and I've supported them in overcoming their fears and achieving remarkable feats. Drawing on this experience, allow me to share a few tips that could set you on the path to becoming a valuable contributor to this great community.

You can check out some of the projects I'm involved in on GitHub.

Who is this guide for?

This guide was crafted with a particular audience in mind, but it holds beneficial insights for a broad range of individuals. If you've ever felt a pull towards contributing to open source projects, this guide is for you. It can serve as a resource for those who wish to dip their toes into the open source waters occasionally, opening an issue or submitting a pull request from time to time. However, the primary focus is on those individuals who wish to become active contributors within a project.

What sets this guide apart is its focus on interactions and activities that can bring you closer to becoming an integral part of an open source community. Rather than getting lost in the technical weeds, such as how to use Git or GitHub, I've emphasized the human element, which is often the most challenging yet rewarding aspect of the journey.

While a certain degree of technical knowledge is undoubtedly beneficial, open source communities are inherently diverse and encompass a vast array of skill levels, from beginners to experts. Whether you're an experienced coder or a newcomer to the programming world, your unique perspective and input can contribute to the growth and success of the project.

This guide can provide you with the roadmap you need to navigate the open source landscape, helping you build connections, share your skills, and become an active participant in a project that resonates with you. It is a guide for the enthusiastic, the curious, and the committed who are ready to embark on their open source journey.

Find the right project

Many people hesitate to contribute to open source projects simply because they're unsure of where to begin. This uncertainty often forms a mental barrier that keeps them from becoming valuable contributors. Although taking the first step can seem daunting, the task itself is fairly straightforward: finding the right project to contribute to.

Start with a project you already use

One of the best ways to start contributing to open source is to begin with a project or software that you already use. You might wonder why this is advised, and the answer lies in familiarity and vested interest.

By virtue of being a user, you are already familiar with the software. You know how it works, you know its strengths, and, more importantly, you are aware of its weaknesses. This familiarity allows you to identify potential areas of improvement and therefore contributions that you could make.

Furthermore, when you're a regular user of a project, you have a vested interest in its progress and development. This means you'll likely feel more invested and motivated to improve it because it directly affects your own work or hobbies. This personal connection can serve as a strong motivator, particularly when you're navigating the challenges and learning curve associated with your first open source contributions.

Contributing to a project you already use allows you to become an active participant in the development of the tools that help you in your daily tasks, giving you a sense of accomplishment and belonging in the open source community.

Find a project that falls into your area of expertise

The tools you use daily may not necessarily align with your interests or fall within your area of expertise. That's completely okay, and it's where this next piece of advice comes in handy.

While familiarity with a project is a great advantage, another crucial aspect to consider is your expertise and passion. By choosing a project related to your field of knowledge or an area you're passionate about, your contributions will not only feel more rewarding, but they'll also be of greater value to the project.

For instance, if you're an expert in machine learning, there are many open source projects where your insights could be incredibly beneficial. Likewise, if you're a cybersecurity specialist, there are numerous initiatives focusing on developing secure, open source software where your expertise would be highly valuable.

Perhaps you're a web developer who has mastered the art of creating accessible digital content. There are countless projects looking to improve their accessibility that could use your skills. Or maybe you're a tech writer who has a knack for creating clear, concise, and helpful documentation—skills that are always in high demand in open source projects.

Even if the project you choose isn't directly related to your day job, that's perfectly fine. What matters is that you are interested and motivated to contribute. Remember, open source is not only about coding; it's about collaboration, sharing, and learning. By aligning your project with your passions and areas of expertise, you can ensure that your contributions are fulfilling and impactful.

Ask a friend or colleague for tips

Knowing the first step to becoming a contributor—finding the right project—is one thing, but actually choosing or discovering the right one isn't always straightforward. The vast expanse of the open source universe can be both exhilarating and overwhelming, making the selection process somewhat daunting for novices.

Over the years, I've had countless people approach me asking which projects are good candidates for aspiring contributors. They seek my advice because they know I'm familiar with various open source projects and have insights into which ones are welcoming to newcomers, which align with specific areas of expertise, and which are actively seeking more contributors.

In fact, leveraging your network can be an excellent way to identify potential projects. While there's a certain sense of satisfaction that comes from finding a project on your own, don't hesitate to ask for suggestions from friends, colleagues, or even online communities dedicated to open source software. This simple step could save you hours of searching and potentially point you towards projects you might not have discovered on your own.

Your acquaintances might be privy to information that can guide you in the right direction, including the current needs of certain projects, their community's friendliness towards newcomers, or whether the project's goals align with your interests. It's also a great opportunity to learn about their personal experiences, challenges they've faced, and how they've navigated their own open source journeys.

Remember, open source is all about collaboration and community. Don't be shy about reaching out for help—doing so might just lead you to the perfect project that will kickstart your open source journey.

Get involved

At this point, you should already have a few strong candidates that you want to contribute to. The next step is introducing yourself to the project.

Become active in the community

In the open source universe, it's essential to recognize that the code is just one part of the project. A large, and arguably equally important, component is the community that forms around it. Most open source projects have various channels where individuals involved in the project interact with each other. Nowadays, these are often Slack or Discord channels (though some of the old school projects may still be on IRC). Additionally, they have issue trackers, forums, and various other communication channels.

The participants in these channels range from core maintainers and occasional contributors to end users. Each person, regardless of their role, has a unique perspective that can provide invaluable insights.

Engage with the community. Observe how people interact, participate in discussions, answer questions from those seeking help, and don't be afraid to ask your own. If there's a space for introductions, take a moment to share a bit about yourself and why you're there. This not only helps the community get to know you, but also demonstrates your interest in and commitment to the project.

Most communities welcome active members with open arms. Your involvement may start as answering questions or participating in conversations, but this initial engagement can quickly evolve into more substantial contributions. Remember, open source communities are a rich source of knowledge, mentorship, and support, and they play a critical role in the project's evolution.

Active participation in the community paves the way to more meaningful contributions and can significantly enrich your open source journey.

Join community meetings

While active participation in community channels is a critical first step in getting involved in open source projects, attending community meetings holds its unique significance and deserves special mention.

A common misconception is that community meetings are solely for maintainers or serious contributors. However, I invite people holding this belief to consider the name "community meeting". These meetings are public for a reason - to give the broader community a chance to follow the project more closely, irrespective of their contribution levels.

Community meetings are an excellent opportunity to introduce yourself to the maintainers and the community via video call, adding a personal touch that text-based interaction might lack. This setting allows you to detail your use cases or elaborate on the issues you're facing, overcoming the constraints of written communication.

Additionally, attending these meetings can provide a greater sense of belonging, making you feel more connected to the project and its progress. This experience can spark further motivation to participate actively, further solidifying your role within the community.

In sum, community meetings are a valuable avenue for engagement in open source projects. They offer unique opportunities for personal interaction, deeper understanding, and a stronger sense of involvement. So next time you see an invitation to a community meeting, don't hesitate to attend. Your participation could make a significant difference, both to your journey and to the project's success.

Report bugs or request features

A common misconception is that contributions to an open source project must always involve code. In reality, many first contributions are bug reports or feature requests. While these might seem less significant than submitting code, they play a crucial role in improving the project, and it's often where maintainers form their initial impressions about new contributors.

People frequently open issues when they encounter a problem or need a specific feature, focusing on their personal needs. While this is understandable, it's essential to remember that maintainers are not your free support. They may cooperate with you to resolve a bug or implement a feature, but the ultimate goal is to benefit the entire project, not just individual needs.

When opening an issue, whether you're reporting a bug or requesting a feature, always check if someone else has already opened a similar issue. This includes taking the time to look through closed issues. If an issue already exists, feel free to comment on it with more context or additional use cases, but avoid sending +1s or messages urging for a quicker resolution. It's perfectly acceptable to ask for an update on a stale issue, but only do so if the conversation seems genuinely stalled to avoid triggering any unwanted reactions.

If you find a rejected feature request but believe you have a compelling argument for a slightly different use case, don't hesitate to open a new issue. Ensure to link back to the old one and clearly explain how your use case differs. Always be respectful, and be prepared that your suggestion might also be rejected.

If you can't find an existing issue that addresses your particular bug or feature request, go ahead and open a new one. Provide as many details as possible. Someone on the other side will have to read your issue and figure out how to handle it. Merely stating that something isn't working or that you need something without sufficient details may not elicit the response you're hoping for.

Overall, approach each interaction with respect and empathy, and remember to provide as many details as possible. The maintainers likely have to navigate through dozens of issues each week, so your consideration will go a long way in fostering a positive relationship with them. This could be the stepping stone to your journey as a valued contributor in the world of open source software.

Familiarize yourself with project guidelines

At the risk of seeming tedious, there comes a time in your open source journey where you need to become intimately familiar with the project's rules. As your involvement in the project deepens, understanding these guidelines becomes crucial to your ongoing participation.

Whether it's the project's Code of Conduct (a regrettable necessity in today's world), contribution guidelines (including specific information required for bug reports and feature requests), or development protocols (how to run the project, write tests, etc.), these rules form an essential part of the project's operations.

Typically, these guidelines are documented in the project's README, contribution guidelines, or dedicated documentation. Take the time to familiarize yourself with these documents; they provide the framework for successful participation in the project.

While following these rules may initially seem daunting, remember to empathize with the maintainers. These rules are in place to streamline their work and maintain the project's consistency and quality, tasks that are no small feat given the open source nature of the project.

Becoming conversant with these guidelines will not only ease your future contributions but also establish and reinforce the maintainers' trust and respect in you. It marks you as a committed member of the community, willing to adhere to the shared principles that underpin the project's success. So don't overlook this vital step – the benefits are manifold and rewarding.

Triage issues and review Pull Requests

Each week, I receive hundreds of emails from GitHub. The majority of these emails are about new issues or pull requests that someone has submitted to a project I maintain. Assessing these issues, figuring out what they entail, and deciding whether they are valid or not (you'd be surprised how many issues I have to close immediately) all take a significant amount of time.

Let's do some quick math. Suppose it takes 10-20 minutes to triage an issue. That involves deciding whether it's a duplicate or not (which means I either recall having seen a similar issue or manually search for it, rather than the submitter doing so), determining if the feature request or bug report is valid, and deciding on the best course of action (finding a workaround, scheduling it for fixing or implementation, directing the user to the documentation, etc). This is just the triage; we're not even talking about fixing the bug or implementing the feature. By the above calculation, I can triage about 4 issues every hour, on average.

Remember when I mentioned earlier that I receive hundreds of emails from GitHub each week? Let's say 40 of those are new issues. That would require 10 hours of my time just to triage them. And that's before we even get to fixing the bugs or implementing the features.

Ask yourself, when was the last time you spent 10 hours a week responding to other people's problems or requests (outside of your day job)?

I might have exaggerated the above slightly (although I'm not entirely sure about that), but it's clear that maintaining an open source project demands a lot of time. And maintainers often spend that time simply managing issues. Wouldn't it be better if maintainers could focus on more significant issues and features?

Well, here's a surprise: you can help make that happen.

Do you see a duplicate issue? Link to the original and ask the submitter to close the duplicate if possible.
Do you see an incomplete bug report or feature request? Ask the submitter to provide the missing details.
Do you suspect a bug report is inaccurate? Try to reproduce the problem and share your findings.
Do you notice a violation of the contributing guide? Ask the submitter to correct their mistake.
Do you see a new pull request without tests? Ask the submitter to write tests.
Even better, go ahead and test the changes yourself to ensure they work.

There are countless ways you can help out with issues or pull requests. Many people think this is something only maintainers can do, but as a maintainer, there's nothing more satisfying than seeing the community collaborate on issues and pull requests.

Step on the gas

Congratulations: you became an Open Source Contributor. And I bet what comes next is not going to be nearly as scary as you thought before.

Open your first Pull Request

Determining the right time to open your first pull request varies from person to person and project to project. If you're an absolute beginner, I recommend going through the previously mentioned steps before opening pull requests, especially if you're aiming to be a long-term contributor.

However, sometimes opening a pull request without immersing yourself fully in the community is the right move. For instance, I often find myself opening PRs to fix typos in projects on a weekly basis. Simply submitting a PR with a quick fix is occasionally the best course of action.

Generally, though, you'll want to complete at least a few of the aforementioned steps before submitting your first pull request. By this time, you should have built some reputation with the project maintainers who, ideally, trust your judgement and commitment to the project.

I usually advise starting with a smaller task, such as writing a documentation page for a feature, crafting a test to cover an edge case (or to prove a bug), or implementing a minor change to an existing feature. Small changes are easier to review and merge, providing you with a sense of achievement that can fuel your motivation to contribute further.

By this stage, you should be familiar with the project's contribution guidelines (and the degree to which they are enforced). Try to adhere to these guidelines as closely as possible to make it easier for the maintainers to review and approve your changes.

When you ask for a review and receive feedback, remain humble, patient, and respectful. Maintainers often have a unique perspective on the project that you may lack, leading to decisions you may not agree with. You're free to respectfully challenge their views, provide additional input, or request further information if you don't understand or agree with a particular decision. However, always remember that maintainers prioritize the needs of the project, and they likely have a better understanding of what those needs are.

If all goes well (and why wouldn't it?), your first pull request will be straightforward to review and should be merged relatively quickly.

Once again, congratulations! Although you were already a contributor, you now have your own commits in the repository.

Keep up the good work

Congratulations on having your first pull request merged! However, your journey as a contributor to an open source project doesn't end here. It is just the beginning. The real magic lies in consistency and continuous effort.

Maintaining the momentum is crucial. Keep engaging with the community, asking and answering questions, participating in discussions, and contributing to the codebase. Remember that open source contribution is not just about code; it's about making the project better. That could mean improving documentation, fixing bugs, implementing new features, or helping others to use the project more effectively.

Remember to continue following the project's guidelines and respect the decisions of the project maintainers. Disagreements may arise from time to time, but it's essential to navigate these moments with grace and respect for all parties involved. Open source is a space for collaboration and shared growth, so strive to learn from each experience and interaction.

Every contribution you make, no matter how small, adds up and makes a difference. Over time, your continued involvement can lead to more substantial contributions and a deeper understanding of the project. You may find yourself guiding new contributors, becoming a trusted reviewer, or maybe even a project maintainer someday.

As you gain experience, you might want to challenge yourself with more complex tasks or contribute to other projects that align with your interests or professional goals. Remember, every contribution counts, and your growth is the true testament to your dedication to the open source community.

So, keep up the good work! The open source community values your participation, and your ongoing efforts will only serve to strengthen the project and the community surrounding it. You're on an exciting journey, and the road ahead is filled with endless opportunities to learn, grow, and make a lasting impact.

Conclusion

I began writing this post following a conversation with a colleague who sought advice on how to get involved with open source projects. While my initial response was much more brief, I soon realized the potential to provide a more comprehensive guide that could benefit anyone interested in the open source world. And so, this detailed guide was born.

Participating in open source is about more than just contributing code. It's about becoming part of a community, sharing knowledge, learning from others, and working together to create something that benefits everyone. It's a rewarding journey, filled with opportunities for personal and professional growth, and it all starts with that first step: the decision to get involved.

Whether you're a seasoned developer or a newcomer to the world of programming, there's a place for you in the open source community. The journey may seem daunting at first, but with patience, persistence, and a willingness to learn, you can become a valuable contributor to any project you choose.

Keep in mind that this is not a strict blueprint that you must adhere to, but a set of guidelines and recommendations based on my own experiences and observations. Every open source project is unique, with its own set of rules, community, and culture, and your journey may look different based on these factors.

I hope that you found this guide helpful, and that it serves as a useful resource as you embark on your open source journey. Remember, every contribution, no matter how small, makes a difference. The world of open source is vast and varied, and it's waiting for you to dive in and make your mark.

Happy contributing!

Go adds structured logging to its standard library…but should it?

Márk Sági-Kazár — Fri, 31 Mar 2023 11:46:35 +0000

Originally published at https://sagikazarmark.hu.

The Go programming language has recently accepted a proposal to add a structured logging package to its standard library. This long-awaited addition is expected to land in a future version of the language, sparking excitement and debate among developers.

Before diving into the pros and cons of this decision, let's first take a look at the history of structured logging in Go.

Structured logging in Go so far

Structured logging has been a topic of discussion in the Go community for quite some time. As the language gained popularity and more developers started using it to build increasingly complex applications, the need for a standardized, built-in structured logging solution became clear. However, previous attempts to address this issue within the Go standard library have not been successful.

The absence of a standardized structured logging solution led to the emergence of a plethora of third-party libraries, each aiming to fill the void. While these libraries have provided developers with viable options for implementing structured logging in their applications, their coexistence has also contributed to fragmentation within the Go ecosystem.

This fragmentation has led to several problems that hinder the development and maintenance of Go applications.

First and foremost, the lack of a standardized approach to structured logging has resulted in difficulties in interoperability between different Go projects. For instance, when two or more projects that use different logging libraries need to be integrated, developers often face the challenge of bridging the gap between the disparate logging solutions.

Moreover, this fragmentation has created a lack of consensus on best practices for logging within Go applications. With multiple libraries offering various features and approaches, developers are left to navigate a maze of options, often leading to inconsistencies in how logging is implemented across different projects. This lack of consistency can make it difficult for newcomers to the language to learn and adopt best practices.

Another issue arising from the fragmentation is the increased complexity of dependency management. When using multiple third-party logging libraries, developers may encounter compatibility issues, especially as these libraries evolve and introduce breaking changes. This can result in additional work to keep dependencies up-to-date and ensure that the logging functionality remains stable.

Finally, the abundance of third-party logging libraries has led to a duplication of effort among the Go community. Instead of focusing on a unified, standardized solution, developers have spent time and resources creating and maintaining various libraries that essentially attempt to solve the same problem. This can be seen as an inefficient use of the community's collective energy and expertise.

All hail the new `slog` package

Integrating structured logging into the Go standard library offers numerous advantages and has the potential to address many of the challenges faced by the Go community thus far. The existing fragmentation within the ecosystem has led to difficulties in interoperability, a lack of consensus on best practices, increased complexity in dependency management, and a duplication of effort. By adding a structured logging package to the Go standard library, the language can take a significant step towards overcoming these issues.

One of the main benefits of having structured logging built into the standard library is the potential for improved interoperability between different Go projects. With a standardized logging solution, developers no longer need to bridge the gap between disparate third-party libraries when integrating projects. This streamlined approach can reduce the friction associated with project integration and promote more seamless collaboration within the Go community.

Additionally, incorporating structured logging into the Go standard library can foster a consensus on best practices for logging within Go applications. With a unified solution, developers will have clearer guidance on how to implement logging effectively and consistently across projects. This can make it easier for newcomers to the language to learn and adopt best practices, ultimately enhancing the overall quality of Go applications.

The inclusion of structured logging in the Go standard library also paves the way for future improvements, such as the introduction of "structured" errors. With the foundation of a standardized logging solution in place, the language can further enhance error handling by providing more structured and actionable information when logging errors. This can improve the debugging process and overall robustness of Go libraries and applications.

Is it too good to be true? Maybe...

While incorporating structured logging into the Go standard library offers numerous benefits, it's important to consider the challenges and downsides it may pose.

One reason that previous attempts to introduce structured logging in the Go standard library have failed is the difficulty of creating a solution that caters to a wide variety of use cases. Tools included in a standard library should be as universal as possible. There have already been voices raised against the current proposal, including from prominent communities like Kubernetes. It's crucial to acknowledge that it may be impossible to create a solution that satisfies every developer's needs, which could lead some to continue relying on third-party libraries and also raise questions about whether this feature should be included in the standard library.

Another concern with adding such a solution to the Go standard library is that it is bound by the Go backward compatibility promise, which means that any feature added to the library must be maintained and supported throughout future versions of the language. This constraint makes it difficult to address design flaws or make iterative improvements to the structured logging package once it has been added to the standard library. In contrast, community-driven efforts like OpenTelemetry can undergo multiple iterations before being marked as generally available and can even introduce breaking changes in major versions if necessary.

Lastly, introducing a new structured logging package in the Go standard library does not guarantee immediate adoption within the broader ecosystem. Reusable components and libraries often support multiple versions of Go for an extended period,
which means it could take considerable time before the new package becomes widely used. This delayed adoption can hinder the realization of the benefits that a standardized structured logging solution offers. Once again, a community-driven effort does not have this problem.

Conclusion

The addition of structured logging to the Go standard library is undoubtedly a significant step towards enhancing the language and addressing existing challenges within the ecosystem. A standardized solution has the potential to improve interoperability, promote best practices, and simplify dependency management. However, it's essential to acknowledge the hurdles that must be overcome before this feature can be successfully implemented and widely adopted.

As discussed, the process of designing a universally applicable solution is challenging, and immediate adoption within the broader ecosystem is not guaranteed. It's also important to consider the constraints imposed by the Go backward compatibility promise, which can limit the ability to address design flaws.

Given these concerns, community-driven projects like OpenTelemetry may provide a more suitable home for a standard structured logging solution in Go. Community-driven projects can be more flexible and adaptable, allowing for multiple iterations and even breaking changes when necessary.
However, it's also worth recognizing that the process of adding a feature to the Go standard library often begins with experimentation and exploration within the community itself.

Building a CI pipeline for a Go CLI application with Dagger

Márk Sági-Kazár — Thu, 07 Jul 2022 08:37:48 +0000

Originally published at https://sagikazarmark.hu.

In my previous post I explained the basic steps of building a CI pipeline for a Go library with Dagger. Let's move onto the next level: Go CLI applications.

TL;DR: Check out this repository for a complete example.

Important: As in the previous post, I'm going to focus on Go CLI specific details and will not explain Dagger basic concepts. Please check out the documentation for an introduction into Dagger and my previous post for Go related examples.

Recap

In my previous post I defined four common categories of CI steps:

Running various builds and tests in a build matrix (eg. for multiple Go versions)
Static analysis and linters
Publishing analysis results (eg. code coverage) to a third-party service
Publishing artifacts

One could argue that the last two categories are actually the same, but they usually serve different audiences, so I prefer listing them separately.

Most steps of the Go library use case fit into the first three categories. While they might still be present in the pipeline for a CLI tool, the most important step for this use case is building and publishing artifacts.

Prior art: GoReleaser

GoReleaser is the standard tool in the Go ecosystem for building and publishing Go CLI applications. It's capable of building, packaging and publishing CLI applications in several package formats to several artifact stores.

GoReleaser itself is a CLI application (written in Go) which makes it a good candidate for being integrated into a Dagger pipeline.

To do that, create a new CUE file for the action definition (in ci/goreleaser/release.cue):

// Release Go binaries using GoReleaser
#Release: {
    // Source code
    source: dagger.#FS

    // GoReleaser version
    version: *"1.10.1" | string

    _image: docker.#Pull & {
        source: "index.docker.io/goreleaser/goreleaser:v\(version)"
    }

    go.#Container & {
        name: "goreleaser"
        entrypoint: []
        "source": source
        input:    _image.output
        command: {
            name: "goreleaser"
        }
    }
}

Similarly to other Go related actions, we use the go.#Container definition to make use of all the caches and mounts set in it.

Next, create a new action in your Dagger plan:

release: goreleaser.#Release & {
    source: client.filesystem["."].read.contents

    env: {
        if client.env.GITHUB_TOKEN != _|_ {
            GITHUB_TOKEN: client.env.GITHUB_TOKEN
        }
    }
}

Finally, add and commit all pending changes, tag a new version and run GITHUB_TOKEN=<TOKEN> dagger do release to publish a new release to GitHub.

Custom release pipeline

Looking closer at what GoReleaser does under the hood, it's possible to do with a "pure" Dagger pipeline:

Build the application for multiple targets/platforms (build matrix)
Package the binaries and additional files (eg. README.md) into archives/distro packages
Push the resulting packages to artifact stores (eg. GitHub Releases)

Build the application

Let's start with the build step. Similarly to running tests for a library, you can use Cue templating to setup a build matrix to support multiple OSes and architectures:

build: {
    "linux/amd64":   _
    "darwin/amd64":  _
    "windows/amd64": _

    [platform=string]: go.#Build & {
        source: client.filesystem["."].read.contents

        package: "."

        os:   strings.Split(platform, "/")[0]
        arch: strings.Split(platform, "/")[1]

        ldflags: "-s -w"

        env: {
            CGO_ENABLED: "0"
        }
    }
}

You can build the application (all variants) with dagger do build.

Packaging

The next step is packaging binaries and other files (eg. README.md) into archives. Since there is no builtin archive action in Dagger Universe, we have to build our own.

First we need to create an image with the necessary tools (in ci/archive/image.cue):

// Build an archive base image
#Image: {
    alpine.#Build & {
        packages: {
            bash:      _
            coreutils: _
            p7zip:     _
        }
    }
}

Next, we need a definition for an action that creates an archive from a list of files. The action has to be able to create .tar.gz files for Linux and Darwin, .zip files for Windows:

// Create a new archive
#Create: {
    // Source files for the archive
    source: dagger.#FS

    // Archive name
    name: string

    _image: #Image

    _sourcePath: "/src"

    bash.#Run & {
        input: _image.output

        script: contents: """
case "\(name)" in
    *.tar.gz | *.tgz)
        tar -czvf \(name) *
        ;;

    *.zip)
        7z a \(name) *
        ;;

    *)
        echo "Unsupported archive type"
        exit 1
        ;;
esac

mkdir -p /result
mv \(name) /result
"""

        workdir: _sourcePath
        mounts: {
            "source": {
                dest:     _sourcePath
                contents: source
            }
        }

        export: directories: "/result": _
    }
}

As a next step, let's add package building to our Dagger plan!

Binary archives often contain additional files, like README.md, so prepare those files first:

package: {
    _files: core.#Copy & {
        input:    dagger.#Scratch
        contents: client.filesystem["."].read.contents
        include: [
            "README.md",
            "LICENSE",
        ]
    }

    // ...
}

Create the actual archives using the archive.#Create we defined earlier. The input for each archive is the binary from the build action and the additional files we prepared earlier:

package: {
    // ...

    _archives: {
        "linux/amd64":   _
        "darwin/amd64":  _
        "windows/amd64": _

        [platform=string]: archive.#Create & {
            _source: core.#Merge & {
                inputs: [
                    _files.output,
                    if platform == "darwin/amd64" {
                        build."darwin/amd64".output
                    },
                    if platform == "linux/amd64" {
                        build."linux/amd64".output
                    },
                    if platform == "windows/amd64" {
                        build."windows/amd64".output
                    },
                ]
            }

            source: _source.output

            _os:   strings.Split(platform, "/")[0]
            _arch: strings.Split(platform, "/")[1]
            _type: string | *"tar.gz"
            if _os == "windows" {
                _type: "zip"
            }

            name: "dagger-go-cli_\(_os)_\(_arch).\(_type)"
        }
    }

    // ...
}

Finally, define the output of the package action:

package: {
    // ...

    _packages: core.#Merge & {
        inputs: [
            package._archives."linux/amd64".export.directories."/result",
            package._archives."darwin/amd64".export.directories."/result",
            package._archives."windows/amd64".export.directories."/result",
        ]
    }

    output: _packages.output
}

package.output will contain all three archives for the three supported platforms. You can build them running dagger do package.

In case you want to check the content of the resulting archives, you can write them to the filesystem by adding the following to the Dagger plan:

client: filesystem: "./_build": write: contents: actions.package.output

Publishing the release on GitHub

The final step is creating a release on GitHub and uploading the archives. The easiest way to do that is using the GitHub CLI.

First (as usual), we need to create an image definition (in ci/github/image.cue):

#Image: {
    version: string | *"2.13.0"

    docker.#Build & {
        steps: [
            docker.#Pull & {
                source: "index.docker.io/alpine:3.15.0@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300"
            },
            docker.#Run & {
                command: {
                    name: "apk"
                    args: ["add", "bash", "curl", "git"]
                    flags: {
                        "-U":         true
                        "--no-cache": true
                    }
                }
            },
            bash.#Run & {
                script: contents: """
apk add curl tar
curl -L https://github.com/cli/cli/releases/download/v\(version)/gh_\(version)_linux_amd64.tar.gz | tar -zOxf - gh_\(version)_linux_amd64/bin/gh > /usr/local/bin/gh
chmod +x /usr/local/bin/gh
"""
            },
        ]
    }
}

Then we can create the definition for the release action (in ci/github/release.cue):

#Release: {
    // Source files
    source: dagger.#FS

    // Artifact files
    artifacts: dagger.#FS

    // Create release from this specific tag
    tag: string

    _image: #Image

    _sourcePath:   "/src"
    _artifactPath: "/artifacts"

    bash.#Run & {
        input: *_image.output | docker.#Image
        script: contents: "gh release create --title '\(tag)' \(tag) /artifacts/*"
        workdir: _sourcePath
        mounts: {
            "source": {
                dest:     _sourcePath
                contents: source
            }
            "artifacts": {
                dest:     _artifactPath
                contents: artifacts
            }
        }
    }
}

Finally, we can integrate the new action into your Dagger plan:

release: github.#Release & {
    source:    client.filesystem["."].read.contents
    artifacts: package.output

    tag: client.env.GITHUB_REF_NAME

    env: {
        if client.env.GITHUB_TOKEN != _|_ {
            GITHUB_TOKEN: client.env.GITHUB_TOKEN
        }
    }
}

You can run GITHUB_REF_NAME=SOME_TAG GITHUB_TOKEN=<TOKEN> dagger do release locally to create a new release or run it on GitHub Actions.

Conclusion

GoReleaser is the standard tool for publishing CLI applications and it can be easily integrated into a Dagger pipeline. Based on the complexity of the pipeline, it might not worth the effort compared to just running GoReleaser on GitHub Actions, but Dagger can still be useful when it comes to debugging pipelines.

Alternatively, you can choose to build your pipeline without GoReleaser which gives you more control, but requires (a lot) more work that may not be worth the investment for small projects.

Personally, I'm looking forward to see which pattern becomes more popular in the Dagger community: using ready-made tools (like GoReleaser, MegaLinter, etc) integrated into Dagger or writing "pure" Dagger pipelines using lower-level tools.

Make sure to check out the complete example for more details.

Functional options on steroids

Márk Sági-Kazár — Sun, 26 Jun 2022 16:30:58 +0000

Originally published at https://sagikazarmark.hu.

Functional options is a paradigm in Go for clean and extensible APIs popularized by Dave Cheney and Rob Pike. This post is about the practices that appeared around the pattern since it was first introduced.

Update (2022-06-26): Improved Option type

Functional options came to life as a way to create nice and clean APIs with configuration, specifically involving optional settings. There are many obvious ways to do it (constructor variants, config struct, setter methods, etc), but they fall short when a package has a dozen options and don't produce nearly as nice APIs as functional options do.

Recap - What are functional options?

Normally, when you construct an "object", you do that by calling a constructor and passing the necessary arguments to it:

obj := New(arg1, arg2)

(Let's ignore the fact that there are no traditional constructors in Go for a moment.)

Functional options allow extending the API with optional parameters, turning the above line into this:

// I can still do this...
obj := New(arg1, arg2)

// ...but this works too
obj := New(arg1, arg2, myOption1, myOption2)

Functional options are basically variadic function type arguments that accept the constructed (or an intermediary config) type as a parameter. Thanks to the variadic nature, it's perfectly valid to call a constructor with no options at all, keeping it clean even when you want to fall back to defaults.

To better demonstrate the pattern, let's take a look at a realistic example (without functional options first):

type Server struct {
    addr string
}

// NewServer initializes a new Server listening on addr.
func NewServer(addr string) *Server {
    return &Server {
        addr: addr,
    }
}

After adding a timeout option the code looks like this:

type Server struct {
    addr string

    // default: no timeout
    timeout time.Duration
}

// Timeout configures a maximum length of idle connection in Server.
func Timeout(timeout time.Duration) func(*Server) {
    return func(s *Server) {
        s.timeout = timeout
    }
}

// NewServer initializes a new Server listening on addr with optional configuration.
func NewServer(addr string, opts ...func(*Server)) *Server {
    server := &Server {
        addr: addr,
    }

    // apply the list of options to Server
    for _, opt := range opts {
        opt(server)
    }

    return server
}

The resulting API is easy to use and read:

// no optional paramters, use defaults
server := NewServer(":8080")

// configure a timeout in addition to the address
server := NewServer(":8080", Timeout(10 * time.Second))

// configure a timeout and TLS in addition to the address
server := NewServer(":8080", Timeout(10 * time.Second), TLS(&TLSConfig{}))

In comparison, here is how constructor variants and a config struct version look like:

// constructor variants
server := NewServer(":8080")
server := NewServerWithTimeout(":8080", 10 * time.Second)
server := NewServerWithTimeoutAndTLS(":8080", 10 * time.Second, &TLSConfig{})


// config struct
server := NewServer(":8080", Config{})
server := NewServer(":8080", Config{ Timeout: 10 * time.Second })
server := NewServer(":8080", Config{ Timeout: 10 * time.Second, TLS: &TLSConfig{} })

The advantage of using functional options over constructor variants is probably obvious: they are easier to maintain and read/write. Functional options also beat config struct when there are no options passed to the constructor (empty struct),
but in the following sections I will show more examples where a config struct may fall short.

Read the full story of functional options by following the links in the post introduction.

Functional option practices

Functional options themselves are nothing more than functions passed to a constructor. The simplicity of just using plain functions gives flexibility and a lot of potential. Because of that, it's no surprise that quite a few practices emerged around the pattern over the years. Here is a list of what I consider the most popular and useful practices.

Feel free to leave a comment if you think something is missing.

Option type

The first thing you might want to do when applying the functional options pattern is defining a type for the option function:

// Option configures a Server.
type Option func(s *Server)

While this may not seem like a huge improvement, it actually makes the code more readable by using a type name instead of a function definition:

func Timeout(timeout time.Duration) func(*Server) { /*...*/ }

// reads: a new server accepts an address
//      and a set of functions that accepts the server itself
func NewServer(addr string, opts ...func(s *Server)) *Server

// VS

func Timeout(timeout time.Duration) Option { /*...*/ }

// reads: a new server accepts an address and a set of options
func NewServer(addr string, opts ...Option) *Server

Another advantage of having an option type is that Godoc organizes the option functions under the type:

A better Option type

Unfortunately, the Option type above introduces a serious flaw into the API: it allows changing the configuration after initialization, thanks to the exported function:

server := NewServer(":8080", Timeout(10 * time.Second))
Timeout(999 * time.Hour)(server) // potential race condition

There are several ways to solve this problem without losing any of the advantages of the Option type.

The easiest one is declaring the option function as a non-exported type first:

// option configures a Server...
type option func(s *Server)

// ...but the outside world doesn't know that.
type Option option

Another solution is using a non-exported type for the receiver:

type serverOptions struct { /* ... */ }

// Option configures a Server.
type Option func(o *serverOptions)

This is obviously a bit more work, but it also lets you deal with defaults more elegantly:

type serverOptions struct {
    timeout time.Duration
}

// call this function when initializing Server.
func (o serverOptions) getTimeout() {
    if o.timeout == 0 {
        return 10 * time.Second // default timeout is 10 seconds
    }

    return o.timeout
}

The third (and most advanced) solution is making the Option type an interface:

// Option configures a Server.
type Option interface {
    apply(s *Server)
}

Read on to learn more about this solution.

Option list type

Usually, functional options are used for creating a single instance of something, but that's not always the case. Reusing a list of default options when creating multiple instances is not uncommon either:

defaultOptions := []Option{Timeout(5 * time.Second)}

server1 := NewServer(":8080", append(defaultOptions, MaxConnections(10))...)

server2 := NewServer(":8080", append(defaultOptions, RateLimit(10, time.Minute))...)

server3 := NewServer(":8080", append(defaultOptions, Timeout(10 * time.Second))...)

That's not quite readable code though and the point of using functional options is having friendly APIs. Luckily, there is a way to simplify it. We just have to make the []Option slice an Option itself:

// Options turns a list of Option instances into an Option.
func Options(opts ...Option) Option {
    return func(s *Server) {
        for _, opt := range opts {
            opt(s)
        }
    }
}

After replacing the slice with the Options function the above code becomes:

defaultOptions := Options(Timeout(5 * time.Second))

server1 := NewServer(":8080", defaultOptions, MaxConnections(10))

server2 := NewServer(":8080", defaultOptions, RateLimit(10, time.Minute))

server3 := NewServer(":8080", defaultOptions, Timeout(10 * time.Second))

`With` / `Set` option name prefix

Options are often complex types, unlike a timeout or a maximum number of connections. For example, the server package might define a Logger interface as an option (and fall back to a noop logger by default):

type Logger interface {
    Info(msg string)
    Error(msg string)
}

Logger as a name obviously cannot be used for the option as it's already taken by the interface.
LoggerOption could work, but it's not really a friendly name. When you look at the constructor as a sentence though, the word with comes to mind, in our case: WithLogger.

func WithLogger(logger Logger) Option {
    return func(s *Server) {
        s.logger = logger
    }
}

// reads: create a new server that listens on :8080 with a logger
NewServer(":8080", WithLogger(logger))

Another common example of a complex type option is a list (slice) of values:

type Server struct {
    // ...

    whitelistIPs []string
}

func WithWhitelistedIP(ip string) Option {
    return func(s *Server) {
        s.whitelistIPs = append(s.whitelistIPs, ip)
    }
}

NewServer(":8080", WithWhitelistedIP("10.0.0.0/8"), WithWhitelistedIP("172.16.0.0/12"))

In this case, the default behavior is usually append instead of set which aligns with the fact that with rather suggests addition to a list than overwriting it. If you need to overwrite the existing set of values, you can use the set word in the option name:

func SetWhitelistedIP(ip string) Option {
    return func(s *Server) {
        s.whitelistIPs = []string{ip}
    }
}

NewServer(
    ":8080",
    WithWhitelistedIP("10.0.0.0/8"),
    WithWhitelistedIP("172.16.0.0/12"),
    SetWhitelistedIP("192.168.0.0/16"), // overwrites any previous values
)

Similarly, an option for prepending can easily be created if necessary.

The preset pattern

Specific use cases are often generic enough for supporting them in a library. In case of configuration, this could mean a set of options being grouped together and used as a preset for a use case. In our example, Server might have a public and an internal use case that configures timeouts, rate limits, number of connections, etc differently:

// PublicPreset configures a Server for public usage.
func PublicPreset() Option {
    return Options(
        WithTimeout(10 * time.Second),
        MaxConnections(10),
    )
}

// InternalPreset configures a Server for internal usage.
func InternalPreset() Option {
    return Options(
        WithTimeout(20 * time.Second),
        WithWhitelistedIP("10.0.0.0/8"),
    )
}

While presets can be useful in a few cases, they probably have more value in internal libraries and less value in public, generic libraries.

Default values vs default preset

In Go, empty values always have a default. For numbers it's generally zero, for boolean values it's false, and so on. It's considered to be a good practice to rely on default values in optional configuration. For instance, zero value should mean unlimited timeout instead of "no timeout" (which is usually pointless).

In some cases though the zero value is not a good default. For example, the default value of a Logger is nil which would lead to panics (unless you guard log calls with conditional checks).

In those cases setting a value in a constructor (before applying options) is a good way to define a fallback:

func NewServer(addr string, opts ...func(*Server)) *Server {
    server := &Server {
        addr:   addr,
        logger: noopLogger{},
    }

    // apply the list of options to Server
    for _, opt := range opts {
        opt(server)
    }

    return server
}

I've seen examples for having a default preset (using the pattern explained in the previous section). However, I don't consider that a good practice. It's much less expressive than just simply setting default values in the constructor:

func NewServer(addr string, opts ...func(*Server)) *Server {
    server := &Server {
        addr:   addr,
    }

    // what are the defaults?
    opts = append([]Option{DefaultPreset()}, opts...)

    // apply the list of options to Server
    for _, opt := range opts {
        opt(server)
    }

    return server
}

Config struct option

Having a Config struct as a functional option is probably not so common, but it's not without precedent either. The idea is that functional options reference a config struct instead of referencing the actual object being created:

type Config struct {
    Timeout time.Duration
}

type Option func(c *Config)

type Server struct {
    // ...

    config Config
}

This pattern is useful when you have tons of options and creating a config struct seems cleaner than listing all of them in a function call:

config := Config {
    Timeout: 10 * time.Second
    // ...
    // lots of other options
}

NewServer(":8080", WithConfig(config))

Another use case for this pattern is setting defaults:

config := Config {
    Timeout: 10 * time.Second
    // ...
    // lots of other options
}

NewServer(":8080", WithConfig(config), WithTimeout(20 * time.Second))

Advanced patterns

After writing dozens of functional options, you might start to wonder if there is a better way to do it. Not from a consumer point of view, but from the maintainer's perspective.

For example, what if we could define types and use those as options:

type Timeout time.Duration

NewServer(":8080", Timeout(time.Minute))

(Notice how the API from the consumer's point of view remains the same)

It turns out that by changing the Option type we can easily do that:

// Option configures a Server.
type Option interface {
    // apply is unexported,
    // so only the current package can implement this interface.
    apply(s *Server)
}

Redefining the option function as an interface opens the door to a number of new ways to implement functional options:

A variety of builtin types can be used as options without a function wrapper:

// Timeout configures a maximum length of idle connection in Server.
type Timeout time.Duration

func (t Timeout) apply(s *Server) {
    s.timeout = time.Duration(t)
}

Option lists and config structs (seen in the previous sections) can also be redefined like this:

// Options turns a list of Option instances into an Option.
type Options []Option

func (o Options) apply(s *Server) {
    for _, opt := range o {
        o.apply(s)
    }
}

type Config struct {
    Timeout time.Duration
}

func (c Config) apply(s *Server) {
    s.config = c
}

My personal favorite though is the possibility to reuse an option in multiple constructors:

// ServerOption configures a Server.
type ServerOption interface {
    applyServer(s *Server)
}

// ClientOption configures a Client.
type ClientOption interface {
    applyClient(c *Client)
}

// Option configures a Server or a Client.
type Option interface {
    ServerOption
    ClientOption
}


func WithLogger(logger Logger) Option {
    return withLogger{logger}
}

type withLogger struct {
    logger Logger
}

func (o withLogger) applyServer(s *Server) {
    s.logger = o.logger
}

func (o withLogger) applyClient(c *Client) {
    c.logger = o.logger
}

NewServer(":8080", WithLogger(logger))
NewClient("http://localhost:8080", WithLogger(logger))

Summary

Functional options is a powerful pattern for creating clean (and extensible) APIs with dozens of options. Although it's a bit more work than maintaining a simple config struct, it provides a lot more flexibility and produces much cleaner APIs than the alternatives.

Building a CI pipeline for a Go library with Dagger

Márk Sági-Kazár — Tue, 21 Jun 2022 23:05:16 +0000

Originally published at https://sagikazarmark.hu.

I've been playing with Dagger for months now using it in various projects. In this post, I'll share my experience with using Dagger to build CI pipelines for Go libraries.

TL;DR: Check out this repository for a complete example.

Important: I'm going to focus on Go library specific details and will not explain Dagger basic concepts. Please check out the documentation for an introduction into Dagger.

Go library CI

The easiest way to evaluate Dagger for the Go library use case is to compare it to an existing solution. It's probably fair to say that GitHub Actions dominates the CI market for Open Source Software these days, so it makes sense to compare building a pipeline with Dagger to an existing GitHub Actions workflow.

Here is a simple one for a Go library:

name: CI

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    strategy:
      matrix:
        go: ['1.16', '1.17', '1.18']

    steps:
      - name: Set up Go
        uses: actions/setup-go@v3
        with:
          go-version: ${{ matrix.go }}

      - name: Checkout code
        uses: actions/checkout@v3

      - name: Test
        run: go test -race -coverprofile=coverage.txt -covermode=atomic ./...

      - name: Upload coverage
        uses: codecov/codecov-action@v3
        if: always()
        with:
          files: coverage.txt

  lint:
    name: Lint
    runs-on: ubuntu-latest

    steps:
      - name: Set up Go
        uses: actions/setup-go@v3
        with:
          go-version: 1.18

      - name: Checkout code
        uses: actions/checkout@v3

      - name: Lint
        uses: golangci/golangci-lint-action@v3

In general terms it consists of the following steps:

Build matrix to run tests (in parallel) for different environments and settings (eg. multiple Go versions)
Static analysis and linters
Publishing analysis results (eg. code coverage) to a third-party service

There are way more complex pipelines out there that run various static analysis tools and integrate with many more services, but most of them fall into one of the three categories above. (Actually, there is a fourth one, but that's hardly ever relevant for a Go library: artifact publishing)

Let's see how we can build a pipeline with Dagger!

Build matrix

Go libraries generally support at least two (or more) Go versions which means CI has to execute tests on all supported versions.

Dagger allows nesting actions, so a naive implementation of a build matrix just repeats the same steps with different Go versions:

test: {
    "1.16": go.#Test & {
        source:  client.filesystem["."].read.contents
        package: "./..."

        _image: go.#Image & {
            version: "1.16"
        }

        input: _image.output
    }

    "1.17": // go.#Test ...
    "1.18": // go.#Test ...
}

The different actions then can be executed either per version (in parallel CI jobs) or at the same time (eg. locally):

dagger do test 1.18 # Run tests for a single Go version
dagger do test # Run tests for all Go versions

Naturally, this isn't the most optimal solution: it doesn't work well with multiple dimensions and it requires a lot of duplication even for a single dimension.

An alternative solution is using templating in CUE. It reduces the amount of code and works well with multiple dimensions:

test: {
    "1.16": _
    "1.17": _
    "1.18": _

    [v=string]: go.#Test & {
        source:  client.filesystem["."].read.contents
        package: "./..."

        _image: go.#Image & {
            version: v
        }

        input: _image.output
    }
}

Admittedly, this is not a real build matrix either as all variations have to be listed manually, but for most of the use cases it's close enough.

Static analysis

The Go ecosystem has an exceptionally large number of static analysis tools and chances are not all of them are going to be available in Dagger Universe (Dagger's central repository of actions).

Writing a custom action allows integrating any arbitrary tool into a Dagger plan.

First, make sure that cue.mod/module.cue contains a module name:

module: "github.com/sagikazarmark/dagger-go-library"

Then create a new CUE file for the tool definition (eg. ci/golangci/lint.cue):

// Lint using golangci-lint
#Lint: {
    // Source code
    source: dagger.#FS

    // golangci-lint version
    version: *"1.46" | string

    // Timeout
    timeout: *"5m" | string

    _image: docker.#Pull & {
        source: "index.docker.io/golangci/golangci-lint:v\(version)"
    }

    go.#Container & {
        name:     "golangci_lint"
        "source": source
        input:    _image.output
        command: {
            name: "golangci-lint"
            flags: {
                run:         true
                "-v":        true
                "--timeout": timeout
            }
        }
    }
}

Generally speaking, an action needs a source code input and some tool specific parameters as inputs.

The action can then be added to the plan:

lint: {
    "go": golangci.#Lint & {
        source:  client.filesystem["."].read.contents
        version: "1.46"
    }
}

Note: GolangCI has been recently added to the alpha channel of Dagger Universe, so you can use that one instead.

Publishing analysis results

It's common for libraries to publish (static) analysis results after test runs to analyze and visualize various code metrics.

One of those metrics is code coverage that tells us how much of the source code was actually executed (covered) by test runs. Code coverage information is generated by the go test tool, so the output of the test step needs to be passed to another step.

Dagger handles these kind of dependencies well, all we need to do is reference the output of one action in another:

test: {
    "1.16": _
    "1.17": _
    "1.18": _

    [v=string]: {
        _test: go.#Test & {
            // ...
            command: flags: {
                "-covermode":    "atomic"
                "-coverprofile": "/coverage.out"
            }

            export: files: "/coverage.out": _
        }
        _coverage: codecov.#Upload & {
            // Merge the coverage file with the source code (for VCS information)
            _write: core.#WriteFile & {
                input:    client.filesystem["."].read.contents
                path:     "/coverage.out"
                contents: _test.export.files."/coverage.out"
            }

            source: _write.output
            file:   "coverage.out"
        }
    }
}

Note: You can find the definition for the Upload action here.

Running it on GitHub Actions

Although we have a plan, it still needs to be executed somewhere. GitHub Actions is a perfect candidate, especially for Open Source Go libraries.

Rewriting the above workflow, we can easily run Dagger actions:

name: Dagger

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    strategy:
      matrix:
        go: ['1.16', '1.17', '1.18']

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Dagger
        uses: dagger/dagger-for-github@v3
        with:
          cmds: |
            project update
            do check test go ${{ matrix.go }}

  lint:
    name: Lint
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Dagger
        uses: dagger/dagger-for-github@v3
        with:
          cmds: |
            project update
            do check lint

There is one more change we have to make to our plan: Some actions (like coverage upload) might need detailed VCS information. One option is to pass them manually to each action or let the tools collect them from the environment (eg. environment variables):

client: env: {
    CI:                string | *""
    GITHUB_ACTIONS:    string | *""
    GITHUB_ACTION:     string | *""
    GITHUB_HEAD_REF:   string | *""
    GITHUB_REF:        string | *""
    GITHUB_REPOSITORY: string | *""
    GITHUB_RUN_ID:     string | *""
    GITHUB_SERVER_URL: string | *""
    GITHUB_SHA:        string | *""
    GITHUB_WORKFLOW:   string | *""
}
actions: {
    test: {
        "1.16": _
        "1.17": _
        "1.18": _

        [v=string]: {
            _test: go.#Test & {
                // ...
            }
            _coverage: codecov.#Upload & {
                // ...

                // No need to run coverage upload unless running in CI
                dryRun: client.env.CI != "true"

                env: {
                    GITHUB_ACTIONS:    client.env.GITHUB_ACTIONS
                    GITHUB_ACTION:     client.env.GITHUB_ACTION
                    GITHUB_HEAD_REF:   client.env.GITHUB_HEAD_REF
                    GITHUB_REF:        client.env.GITHUB_REF
                    GITHUB_REPOSITORY: client.env.GITHUB_REPOSITORY
                    GITHUB_RUN_ID:     client.env.GITHUB_RUN_ID
                    GITHUB_SERVER_URL: client.env.GITHUB_SERVER_URL
                    GITHUB_SHA:        client.env.GITHUB_SHA
                    GITHUB_WORKFLOW:   client.env.GITHUB_WORKFLOW
                }
            }
        }
    }
}

Check out the example repository for more details and examples (like caching on GitHub Actions).

Conclusion

The first question that needs to be answered: Is it really worth it?

Well, it depends.

On one hand, building pipelines locally is certainly easier
than doing so on a CI server. It's also comforting to know that if it runs locally, it'll run on the CI server.

On the other hand, CI pipelines for Go libraries tend to be simple and with GitHub Actions the building blocks are as reusable as Dagger actions are. Native CI runs also tend to be a bit faster due to the lack of extra runtime initialization, but that performance hit can be minimized with caching and is less noticeable in large projects.

So it really comes down to personal preference.

My personal favorite feature in Dagger is its portability: I like that I can run and build it on my own machine, but portability really becomes important for complex pipelines and large projects. For a simple Go library, I'll probably stick to Makefiles and GitHub Actions.

Make sure to check out the complete example for more details.

The Perfect Development Environment 2022

Márk Sági-Kazár — Sun, 12 Jun 2022 23:04:07 +0000

Originally published at https://sagikazarmark.hu.

I'm a huge fan of Developer Experience and I continuously chase The Perfect Development Environment every day. Not just for my sake, but to make the development process as painless as possible for my peers. But after pursuing the dream environment for years, I had to realize it's a moving target: technology evolves every day and development processes have to evolve as well.

Previously on The Perfect Development Environment

I praised Makefiles above all else before. Then modern build systems (eg. Please) became my new religion. I'm pretty sure I believed (and preached) that PHP and NodeJS project had to use Composer and NPM (and nothing else) as task runners at some point in my career. I even remember rewriting build scripts from Grunt to Gulp, then from Gulp to the next fancy JavaScript task runner over and over again.

I've used various tools and methods in various projects over the years, but one thing was always common: whatever tools I used for development, I always aimed to use the same tools in CI to build and deploy the projects I worked on.

It wasn't always easy, but it seemed to be a good practice: it reduced the maintenance burden of maintaining two sets of tools (for dev and CI) and it made building CI pipelines easier (by making them thin layers of integration around dev tools).

It never occurred to me to do things the other way around, until Dagger came along.

The root of all evil: portability across platforms

If everyone in the world used Linux on x86 I wouldn't be writing about developer experience, because there really would be only one perfect development environment for everyone to use and the world would be a much happier place.

Unfortunately (or not?), this is not the case. Developers use and target various platforms during development and that makes development processes and the tooling infinitely more complex: different architectures, different package managers, different available software versions and tons of other factors contribute to making portability across environments a pain. Even on different versions of the same OS reproducing identical environments proved to be a challenge.

Around 2013 Docker came to life and it quickly revolutionized development environments. It wasn't true portability of course, since the technology under Docker required Linux, but it introduced an abstraction layer that made packaging and running applications across various (Linux) platforms much easier.

Nowadays, using Docker for development is trivial, even on Windows and MacOS and it essentially resolved the portability issue for a large number of use cases.

Dev vs CI

As I mentioned earlier, using the same tools for development and running them in CI pipelines is not always easy, because they don't always serve the same purposes.

For example: you don't necessarily want to run all test suites during development, just the ones you are working on. So in your local development environment, you need tools that can selectively run tests, but in CI you want to run them all, thus if you want to reuse the same tools in CI, you need something that supports both.

But you also want to be able to run all tests locally, because that's what the CI does and that's what decides whether your change will be accepted or not at the end of the day.

In other words: you want to be able to do and run all things the CI does in your local environment.

New kid on the block: Dagger

Dagger markets itself as a "new way to create CI/CD pipelines" and a "portable devkit for CI/CD pipelines". The first closed beta was released earlier this year and was quickly followed by a public release in March.

The idea behind Dagger is that instead of running CI pipelines on a central server (often written in a platform specific DSL), you can run them anywhere (where Docker is available). It doesn't mean you should stop running/using a CI server, but it puts building and running those pipelines in a new perspective:

What if we could build, debug and run those pipelines locally?

Dev + CI

So here is the novel idea (at least compared to my earlier believes): instead of integrating dev tools into CI pipelines, let's just build a pipeline that you can run both locally and on a CI server.

Whenever you want to confirm that your changes adhere to the current rules enforced by the CI (aka. everything is green), you can just run the pipeline on your machine.

With that taken care of, you can create a set of development tools that you only use for development. You don't have to worry about running them on the CI.

To give you an example how this looks like in practice: I have a Go project where I use Dagger. I can run tests and the static analysis tools locally and on a CI service (eg. GitHub Actions). At the same time, I have a Makefile (or lately, a Taskfile) for dev tasks, like formatting code (often using the same static analysis tools) or running code generators.

If you want to see more examples, feel free to check out my repositories on GitHub. I use Dagger combined with a task runner in more and more projects.

This is one of the first projects I used Dagger in.

Conclusion

I can't say this is going to be The Perfect Development Environment, because I don't believe it exists anymore. Even if it does, it's only perfect for a moment or until the next, better thing comes along.

Building and running CI pipelines on their own instead of integrating dev tools into them is an exciting new perspective and I'm looking forward to trying it in large, established projects. Not having to worry about breaking builds that already passed once on my machine feels extremely comforting so far.

On the other hand, Dagger is extremely young so I wouldn't start rewriting every single CI pipeline just yet.

DEV Community: Márk Sági-Kazár

How to become an Open Source Software contributor?

My journey in the world of open source software

Who is this guide for?

Find the right project

Start with a project you already use

Find a project that falls into your area of expertise

Ask a friend or colleague for tips

Get involved

Become active in the community

Join community meetings

Report bugs or request features

Familiarize yourself with project guidelines

Triage issues and review Pull Requests

Step on the gas

Open your first Pull Request

Keep up the good work

Conclusion

Go adds structured logging to its standard library…but should it?

Structured logging in Go so far

All hail the new slog package

Is it too good to be true? Maybe...

Conclusion

Building a CI pipeline for a Go CLI application with Dagger

Recap

Prior art: GoReleaser

Custom release pipeline

Build the application

Packaging

Publishing the release on GitHub

Conclusion

Functional options on steroids

Recap - What are functional options?

Functional option practices

Option type

A better Option type

Option list type

With / Set option name prefix

The preset pattern

Default values vs default preset

Config struct option

Advanced patterns

Summary

Further reading

Building a CI pipeline for a Go library with Dagger

Go library CI

Build matrix

Static analysis

Publishing analysis results

Running it on GitHub Actions

Conclusion

The Perfect Development Environment 2022

Previously on The Perfect Development Environment

The root of all evil: portability across platforms

Dev vs CI

New kid on the block: Dagger

Dev + CI

Conclusion

All hail the new `slog` package

`With` / `Set` option name prefix