Learn about real-life CI/CD with TDD, at scale and in public, all via contributing to Forem

The best way to get involved in modern application development right now is through continuous integration and continuous delivery (CI/CD).

And the best way to experience it yourself is through an open-source project on Github and I am going to show you how by actually writing some code, submitting a PR and getting it through a CI/CD pipeline and into production to over 500,000 users and millions of daily consumers within a matter of days (yes, at that scale, within days). Welcome to the world of modern app dev.

At the same time, I am going to show you how dev.to do CI/CD at scale (over 500k members, over 5 million monthly views) with test-driven deployment (TDD), in public.

The best way to learn is to get involved, so I am going to go through the following stages of CI/CD so that you'll get a real feel for what it is and why dev teams use it. This is what I am going to cover.

• Forking the forem repository.
• Checking out the forked source code locally.
• Creating a new branch.
• Implementing a feature.
• Committing and testing the feature locally.
• Pushing the branch back to my fork.
• Creating a pull request.
• Watching the CI/CD pipeline process and code reviews.
• Merged into the main branch and rolled out to over 500k members and millions of daily consumers.

By the time I have finished, you should have a solid conceptual understanding of CI/CD as there is no better place than to do it in public.

First of all, what is Forem? (clue, you are on it)

Forem is open source software for building communities. Communities for your peers, customers, fanbases, families, friends, and any other time and space where people need to come together to be part of a collective.

dev.to (or just DEV) is hosted by Forem. It is a community of software developers who write articles, take part in discussions, and build their professional profiles.

Forem is also built on Ruby on Rails, the only programming language (Ruby) and full-stack web framework (Rails) that works with my brain.

This is the feature that I developed that ultimately passed the public CI/CD process and was merged into the main production branch and rolled out to over half a million members and millions of daily consumers.

Let's get started.

We are going to head over to the Forem source code on Github and fork the repository

Top right, there is a fork button

Hit the fork button and the repo will then fork to your Github account. So let's go there to clone the repository locally.

Now forked to my account

Let's clone the repo locally. Hit the 'clone' button, 'https' should be highlighted by default, hit the clipboard option to grab the URL.

Let's go ahead and clone the repo locally.

mkdir forem
cd forem
git clone https://github.com/{your-github-username}/forem.git
cd forem  

(I am doing this from a Mac, if Git isn't installed then I will get a prompt to install the dependencies)

There we go, we now have the full Forem repo, forked, cloned and git initialised on our local machine. To check that is the case, run 'git remote -v' and the output should display origin fetch and push as the only 2 git remotes.

git remote -v
origin  https://github.com/your-github-account/forem.git (fetch)
origin  https://github.com/your-github-account/forem.git (push)

For the eagle-eyed among you, you may have spotted another button on Forem's Github page named 'Gitpod'.

Gitpod is the shiz

This is an extension available from the Chrome Store. When installed it dynamically adds the Gitpod button to any public repo on Github.

So what does Gitpod do?

Gitpod streamlines developer workflows by providing ready-to-code development environments in your browser - powered by VS Code.

Basically, it allows you to clone, build and work on any Github project within a browser tab, everything built-in. IDE (VS code), embedded test browser, Postgres DB included, full-stack included.

How it works:

1. Install the browser extension
2. Go to one of your projects at GitLab, GitHub or Bitbucket
3. Click on the Gitpod button and start a dev environment for your project.

Pretty cool right? You could completely avoid cloning and setting up the Forem repo locally and instead, do it in a browser in a few minutes, even from an iPad - yep full stack web development on open source projects on an iPad in a tab, amazing.

Right, back to our local forked Forem repository. If you just interested in the CI / CD portion of this article you can skip this bit.

We need to set up Forem to run locally and there are a few prerequisites outlined in the Forem documentation that are needed to successfully build and start it, I am not going to cover those here as they are already detailed out very well in Forem's own getting started documentation.

If this is a struggle, then I highly recommend the Gitpod option. Once the repo is cloned, it is just a question of running the 2 following commands and you are up and running with a local version of Forem.

bin/setup
bin/startup

Alright, time to get into some CI/CD.

First thing I need to do here is to create a new branch so that my changes are isolated from the main branch, there are a number of reasons for this:

1. I want to keep my changes separate from my main branch, so I can see them, track them and bin the whole branch of I need too.

2. I need to push this branch back to my forked repository on Github to then create a pull request so that Github can auto-fire it's CI/CD steps before the core Forem team review and approve the code and merge into the main branch.
   

git checkout -b leewynne/generalize-the-sidebar-campaign-article-published-timeframe

Okay, I am on my new branch - any changes I make to the code base are now isolated here.

The feature I developed is detailed here which allows smaller communities to set the timeframe that posts are counted towards stories highlighted in the campaign sidebar (when enabled) the code is available to view in the pull request.

Once I made the changes in my new branch, it was time to then run a TDD (test-driven deployment) sequence as I included a test in the code I wrote. Forem uses rspec for integration testing (from the perspective of the developer, not the end-user). This is the first introduction to using TDD as part of the overall CI/CD process.

bundle exec rspec spec/requests

This command runs through all the predefined tests (including mine). I can review the output at the end to decide whether my code passes the tests and everything is working as expected as this test will run again, remotely, when I submit the PR back to Forem - if anything fails then it may not make it to a code review until the failing tests are fixed. In addition, the testing tools used that make the process as robust as possible are:

• RSpec for testing the Rails backend
• Capybara with webdrivers for acceptance testing
• guard-rspec for automated testing
• Jest for testing the frontend
• jest-axe for detecting basic a11y regressions
• preact-testing-library for testing Preact components
• SimpleCov for tracking overall test coverage on the backend

As you can see even at this stage the foundations are set to ensure that any code that might cause disruption or isn't in line with the overall roadmap gets flagged way before you get to the stage where you might want to submit a PR. This is critical with any software that runs at scale, especially in the public domain. This will give you some insight into how critical a solid CI/CD pipeline is and why TDD (test-driven deployment) and CI/CD is so crucial to performance, stability, rapid evolution, transparent change management, efficient and productive collaboration and quick to market feature integration.

All tests passed, so I am going to commit the code in my local branch in preparation for pushing it back to my forked version of the Forem repository.

Again, this is where the CI/CD pipeline kicks in.

git add .
git commit -m "generalize the sidebar campaign article published timeframe"

At this point, a few cool things happen locally before I even push the branch. Rubocop and other linters review the code and feedback on any recommendations that I might want to make before I push the code to my fork and create a pull request.

Also, the NPM package 'Husky' kicks off 'Git hooks'. A pre-commit hook is run first before you even type in a commit message. This used to inspect the git snapshot that you're about to commit, to see if you’ve forgotten something, to make sure tests run, or to examine whatever you need to inspect in the code.

There is so much cool stuff happening locally here that ultimately streamline the review and merge process when this code hits the review process, you can see the CI/CD process taking place here locally, on my Mac before I even get close to committing the branch.

There is nothing that I need to change so I am going to go ahead and push the code. Staying in my newly created branch:

git push -u origin leewynne/generalize-the-sidebar-campaign-article-published-timeframe

Up it goes. Now when I visit the forked version of Forem on my own Github account, I am notified that the push has been successful and I can now create a pull request to kick off the CI/CD and TDD process and hopefully get this feature gets integrated into Forem's main production branch and safely rolled out.

I hit the 'create PR' button and fill in the form to ensure the core Forem team are given as much information as possible as part of the pull request. Forem also recommends that you use the draft PR option if needed.

I submit the PR and then switch over to Forem's Github page to view the automated Github actions that run my new feature through the CI/CD pipeline and robust TDD process.

Github actions that auto fire the CI/CD pipeline

This is where we first get introduced to [Travis] CI(https://www.travis-ci.com/).

Travis does something similar to Gitpod in the way it pulls the code and builds it out in an isolated container but then runs all the tests I highlighted previously.

• RSpec for testing the Rails backend
• Capybara with webdrivers for acceptance testing
• guard-rspec for automated testing
• Jest for testing the frontend
• jest-axe for detecting basic a11y regressions
• preact-testing-library for testing Preact components
• SimpleCov for tracking overall test coverage on the backend

So I ran all these tests locally with my 'bundle exec rspec spec/requests' command so I am confident that I would get a pass here but this is Forem running the same set of tests to ensure the code is safe and passes before it is reviewed. This helps streamline work, no point is reviewing code if all the tests are failing.

Here are all the tests that my feature went through before then passing to 2 members of the core Forem team for a review:

There are all sorts of cool things happening here:

Codeclimate does an automated code review, scoring the code submitted in the PR prior to a human reviewing it.

Code coverage is also taking place. Code coverage is a metric that can help you understand how much of your source code is actually tested.

Jest is testing front end Javascript code which is also integrated to test the preact library.

Once all these tests have passed (and I may need to revisit my feature to make changes if they don't) I can look forward to a member of the core Forem team to perform the first review of the code and advise on any changes. Once the first reviewer is happy then a second review is requested, again the core Forem team will work with you collaboratively to assist in getting you PR over the line.

That's it! Once all the checks are green you are through the CI/CD pipeline with TDD all checked off, you have been through a review with 2 core Forem team members, all that is left is for your code to merged into the main branch and away it goes, safely into production.

The git branching element of this process follows the 'Gitflow' methodology, you can learn more about that here

Okay so what about infrastructure?

Forem hosted Dev(dev.to) runs on one of the coolest abstractions of AWS you'll ever get to use. Heroku.

Heroku takes the application code we write, performs a series of transformation steps to "build" it ready for deployment, packages it up inside an isolated container, and launches it in a managed runtime environment. It then routes Web traffic from your customers to the servers running the code.

Here is a post from Ben Halpern that explains why Dev choose to use Heroku

It is so easy, especially with Rails. But how does CI/CD work with Heroku?

Well, Dev uses Heroku release and release phase enables you to run certain tasks before a new release of your app is deployed. Release phase can be useful for tasks such as:

• Sending CSS, JS, and other assets from your app’s slug to * a CDN or S3 bucket
• Priming or invalidating cache stores
• Running database schema migrations
• If a release phase task fails, the new release is not deployed, leaving your current release unaffected.

If this all sounds complicated - it is! But the whole point of Heroku is that you don't have to worry about this complexity.

You can watch the Travis output when your code gets past the initial CI/CD and TDD stage, committed to the main branch and then initiated into Heroku release phase. If the tests aren't passed when the tests are run, then the app doesn't get deployed into production.

I hope that this article gives you an insight into the importance of CI/CD including TDD and that you can see that many organisation do it at scale and in public, you can tune in and watch the process happen, learn from it and contribute to it.

Personally, I am excited to see how CI/CD and TDD evolves in the future, especially in the public domain.

Thanks for reading,
Lee

Comments

[Dr Jemba O Jemba]

Thanks a lot for this article, very insightful and practical.

[Lee Wynne]

You’re welcome Dr Jemba!!