DEV Community: Joel Wasserman

Migrating Flossbank from Vercel to Netlify

Joel Wasserman — Mon, 26 Apr 2021 16:05:11 +0000

Hi everyone! I work on a project called Flossbank, which distributes Open Source funds through the Interledger network to Open Source maintainers all over the world. We just completed our migration from Vercel to Netlify, and I’m going to share how it happened, why it happened, and our immediate findings and first impressions (TL;DR: we’re stoked).

Up until about a month ago, we barely gave a second thought to what company we used for deployments. Vercel caught our attention a while ago with their easy CLI deployments. It was awesome to use, and the company felt very developer-centric. We loved that. Unfortunately, what follows led us to reconsider our host, and ultimately migrate to Netlify.

Why

Since all of our sites use Next.js, you may wonder why we would ever migrate away from Vercel, since they are the core maintainers of Next.js, and so must have the most helpful integrations, right? In summary, we didn’t appreciate where Vercel as a business was headed.

Vercel introduced a concept of “teams” recently, and restricted the ability to deploy GitHub organizations’ repos in a Hobby Vercel account. This makes sense in theory, but there are countless organizations on GitHub that still want to be able to deploy to Vercel. After all, a GitHub organization does not have a 1:1 relationship with profit-seeking entities, contrary to where Vercel seems to be headed. This was a short-sighted barrier to entry for anyone wanting to quickly get up and running with a deployed repo that exists in a GitHub organization, and actually blocked us from a deployment.

To be clear, we don’t mind paying for great tech products. Vercel is a phenomenal product with great tech support that chatted with us about the options. The final straw for us was once we created a team, the team and the Hobby account couldn’t share resources. This meant we had a single domain: flossbank.com that couldn’t be accessed from both our existing Hobby account and the new Team account. We couldn’t subdomain on a paid team and keep our existing, working (live in production) subdomains in the Hobby account up and running. If we wanted to continue working on Vercel, we now had to migrate all of our resources and projects to the team account, and for what? Just to be able to deploy a GitHub organizations repository. Was there an easy way to do this? Maybe a button that says “transfer to team”? No. It was a huge pain just to attempt. This felt much more business-oriented (vs developer-oriented) than we had come to know working with Vercel.

This led us to explore our other options and ultimately settle on Netlify, an equally phenomenal product that has no GitHub organization barriers to deployments.

How

I’m going to outline the steps we took to migrate from Vercel to Netlify, the issues we ran into, and how we subsequently resolved them.

Deployed our repos using Netlify

This was a straight forward step. We opened a Netlify account, hooked it up to GitHub, and deployed all of our repos. This was smooth. One caveat: for our configuration, we needed to change the default build command to next build and the output directory to be .next (these are default for Next.js sites so we’d love if Netlify auto-detected this).

Debug

The first issue we encountered was that the deployments didn’t succeed. We have Next.js sites that are built with next build and they didn’t work out of the box. We quickly realized we needed to add the key/value pair of target: ‘serverless’ in our next.config.js, as well as add a custom Netlify plugin to each of the Next.js projects (The Essential Next.js plugin, which is now auto-installed to Next.js projects).

The next issue (ha, pun) we ran into was CSS being overwritten on one of our websites. After reverting react and next versions, this was no longer happening. This was odd because it was unique to Netlify, and searching through the forums we found numerous other CSS conflict issues. We have yet to find a root to the issue but if you run into CSS issues, definitely look in their forums; chances are you’re not alone.

Copy DNS records

Once we verified all of our projects were up and running (and their CSS worked!), we copied over all of our environment variables and DNS records to the new projects. This included records required for AWS SES, Google Workspace (for email), as well as other custom DNS records for our API. This was crucial to ensure that once we transferred the domain to Netlify, there would be no downtime for all of our services.

Transfer the domain

Transferring our domain required a custom support ticket to Vercel. As promised by their support, the domain transfer was initiated by updating our name servers and took about 8 hours. Thankfully we had set up all of our environment variables and DNS records on Netlify’s side prior to the domain transfer, so we experienced 0 downtime.

Netlify First Impressions

Our first impression of Netlify was astounding. Netlify's UI is very intuitive, and although Netlify has a slightly less minimalist UI, it turns out the lack of minimalism is for a reason: it’s not a black box you have to stumble around to figure out. Everything was easy to find and quick to navigate through. Let alone the fact that Netlify’s site is blazing fast. The best part? Netlify has no B.S. requirement that you need to deploy GitHub organization repos using a paid Pro Account so all of your repositories can happily share resources you purchase through Netlify.

And to be clear, we don’t mind paying for products! In fact, we want to reward great engineering products with our business. Flossbank's focus is to compensate engineers for the work they do. Open Source maintainers maintain packages downloaded billions of times and are used by 99% of learning developers. If you use Open Source software in any way, consider using Flossbank to implicitly (no cost, commitment, or maintenance) support the OSS maintainers you use every day or encourage your company to support OSS maintainers through our enterprise program.

As far as web hosting, if you’re looking to quickly deploy GitHub organization repositories, Next.js or otherwise, Netlify is the easiest, most developer-friendly solution.

AWS Lambda and VPC magic

Joel Wasserman — Sun, 07 Mar 2021 21:37:49 +0000

Hey everyone! Pete and I recently set out to create a somewhat unconventional AWS setup. Now, to be totally transparent, we are almost certainly doing this incorrectly, but it was the only way we got it working! Enjoy.

Our goal was to create an Elastic Beanstalk application that is open to the internet (port 80, 443) but also open to a single other AWS resource (lambda) on a specific single other port.

Specifically, we needed an EB (Elastic Beanstalk) instance that is open to the world, but also needs to accept traffic on port 6543 only from this one specific lambda. We didn't want port 6543 open to the world or publicly accessibly in any way.

Solution: a custom VPC where the EB application, a custom load balancer, and a lambda that will live in.

Now you're probably thinking, why didn't y'all just use a security group to accomplish this? We couldn't just use security groups because by default, an EB app has an application load balancer, which can only accept http / https traffic, not TCP traffic on custom ports. This meant that no matter what our security group was, port 6543 was rejected by the load balancer when our lambda reached out to it... Sad.. I know.

We then found out about network load balancers that enable us to route traffic by port. We then thought, ok, we can spin up the lambda within a new VPC and create a custom network load balancer that routes our 6543 traffic to our Elastic Beanstalk server instance group. So we set out to do just that.

Now bear with me, this is going to get very VPC specific. If you're interested in spinning up VPC's, read on! We finished up creating a custom VPC with 2 CIDRs: 1 public, 1 private. When you create a VPC, you must specify a range of IPv4 addresses for the VPC in the form of a Classless Inter-Domain Routing (CIDR) block. In our case, the CIDR was 11.0.0.0/24 (public) and 192.168.70.0/24 (private) (https://docs.aws.amazon.com/directoryservice/latest/admin-guide/gsg_create_vpc.html). After creating a VPC, you can add one or more subnets in each Availability Zone. We created the two private subnets and the two public subnets. When you create a subnet, you specify the CIDR block for the subnet, which is a subset of the VPC CIDR block. we created 3 subnets and associated them with the VPC: 11.0.1.0/24 in AZ us-west-2a, 11.0.0.0/24 in AZ us-west-2b (2 publics) and 192.168.70.0/24 in AZ us-west-2a (the 2b and 2a ended up being important since you aren't allowed to spin up t2.micros in us-west-2d). Subnets are used by the load balancer to serve traffic. The public subnets need to be associated with an internet gateway. Here is where we got all this info. If a subnet's traffic is routed to an internet gateway, the subnet is known as a public subnet.. the private subnets need to be associated with a nat gateway to access the internet. Each subnet must be associated with a route table, which specifies the allowed routes for outbound traffic leaving the subnet.. So we then created 2 route tables, the public one which routes traffic destined for the internet through the internet gateway; the private one which routes traffic destined for the internet through the NAT gateway; all of the above we got from this random AWS article about how to allow your VPC to access the internet while retaining a private subnet.

We then associated the public and private subnets with the above route tables, created a lambda configured to be inside the VPC, on the private subnet, with a security group that allows all outbound and nothing inbound (default). Unfortunately, we then realized that even though our EB wasn't specified to be in a VPC, our elastic beanstalk spins up instances by default into the default VPC. This meant that our lambda and custom load balancer was now in a different VPC than the elastic beanstalk app, and they couldn't communicate. UGH. And since you can't change the VPC of an EB application after it's built, we had to tear it down and start a new one. Fret not, this wasn't too serious and we just spun our elastic beanstalk application back up using the now custom VPC.

Once we configured our VPC to allow incoming traffic from anywhere and configured the correct regions for our subnets, we spun up our elastic beanstalk application with the specified VPC we just created. We added a VPC access policy to our lambda execution role: also from this doc. We then configured the custom load balancer, which is also in the custom VPC, to point it's traffic (traffic from port 6543) to the instance groups that are spun up from the elastic beanstalk application. We created a security group that allows 6543 from the VPC private subnets (which is how you allow traffic from your NLB (network load balancer) since NLBs can't be associated with security groups). We added that security group to our elastic beanstalk instance config, and because they're in the same VPC, this traffic works! Next, we created a target group for the NLB which includes our elastic beanstalk instance(s) (TODO: we don't know how to auto add new ec2 instances to this NLB target group when EB autoscales up/down)

Lastly, we told our lambda to connect to the public dns name of our NLB on 6543, which gets translated into a private IP, which is inside the VPC, which ends up hitting our elastic beanstalk server on the proper private port. And... take a breath. It worked.

We now had a lambda that could access the internet and send internet traffic over TCP on port 6543 to an elastic beanstalk application, of which could also access the internet but could only receive traffic from port 6543 from our specific lambda.

Whew! Leave a comment below if you know of a better way to do this, because believe it or not, this is the production setup for Flossbank. Hope this helps a fellow dev in the weeds of VPC's!

Publishing your first open source package

Joel Wasserman — Mon, 03 Aug 2020 17:22:31 +0000

Open source is an absolutely amazing concept. Brilliant developers write code to do all sorts of interesting and useful things, and then anyone can use that code and write even more code on top of it themselves. That is very, very cool. Each time I pull in open source packages to build a new project I realize I'm standing on the shoulders of giants.

If you'd like to contribute code to the open source ecosystem and this is your first time, this is the article for you! In the following paragraphs I'll go through creating an npm package from an empty directory to a published package step by step.

Before we get started, I have to plug Flossbank. Flossbank is a package manager wrapper that compensates the authors and maintainers of the open source packages you install, as you install them. The compensation comes from either monthly donations or ad revenue, whichever you opt-in to. Check it out at https://flossbank.com. It's a must if you use npm or yarn. **Full disclosure: I started Flossbank.

Also quick note: I'm working on a Mac and will be using npm to publish the open source package.

Alright, let's jump in. First, in a new directory run npm init to create a new npm package. When we run npm init, we're really just following npm's CLI through a step by step to fill out our package.json and create a package on our local machine (don't worry, your package can't be seen by anyone until you run npm publish later in this tutorial). A package.json is a blueprint to your project. It outlines the dependencies your project needs, the scripts you can run, the name of the package, the version, and much much more. Once you've gone through the command line instructions prompted from running npm init, if you run ls you should just notice an added package.json file that wasn't there before. Go ahead and cat package.json to see what was created for you!

And just like that, we have an open source package! The package is empty, but none the less, we have a package!

Open source is built on the philosophy that open source software begets technological progress, and any code helps. What this means in practice is, don't be shy to publish a package you think might be too small! There are popular open source packages that literally just return whether a number is even or not. Yes, that's it. And it's still useful to people who don't want to write that code in a bunch of places and run the risk of making a typo! So don't ever be worried your open source package isn't enough. I promise, it is.

Now it's time to write some code. When we ran npm init, we likely chose the default "entry point" to be index.js. If that's the case, then go ahead and create a new file named index.js, or create a file of whatever name you chose as the "entry point". If you don't remember, you can always pop open your package.json and look at the key main, and see what file is listed as its value.

I just created my index.js and plopped in a little algorithm for sorting numerical data that has labels in a kind of double sort. First, it sorts by count so that the items with the highest count come first. Then it sorts alphabetically by name but only if the numerical values are the same. I find this useful when I'm dealing with data I'm going to graph and I want to sort by count and then have the data sorted by name if the counts are the same.

See the code I created here.

Once you write the code, it's recommended to store your code in some version control host. This is so others can learn from your code, copy, paste, etc if they so choose! I chose GitHub, but GitLab and BitBucket are also great choices.

To publish your npm package, simply run npm publish. You'll see some output about packing the package contents into a tarball, and after a few seconds you should see output that your package was published with version 1.0.0!

Now you can navigate to your npmjs account and you'll see your new package! It's worth noting that your README.md is very important with open source packages. This is how people see value in your package! The better the README, the more enticing your project is!

Congratulations on publishing your first open source package! You've contributed to technological progress! It might not seem like much, but even if you save your fellow developer 5 minutes of coding, those 5 minutes spread across the vast open source ecosystem add up and allow engineers to create insanely cool projects in little to no time!

Leave a comment with your first open source package, and dev on, friends!

A new way to support open source

Joel Wasserman — Tue, 28 Jul 2020 17:51:54 +0000

Open source is an amazing concept. Brilliant developers write code to do all sorts of interesting and useful things, and then anyone can use that code and write even more code on top of it themselves. That is very, very cool. As a frequent user of open source, I benefit from this combination of generosity and brainpower. And as a result, I think a lot about ways to give back. I'm not the only one - GitHub sponsors, Open Collective, and Patreon all address this desire in one way or another. But given the nature of open source, where code is written on top of code, these 1:1 ways of giving back just feel a little incomplete. When I give back to a maintainer of a package I'm using, I'm really only giving back to the person who put the last block on top of the tower. Their package was built on other packages which was built on other packages - so why isn't there a way to give back to all the builders? Well, now there is.

Some friends and I built Flossbank to address two concerns: First, we think it should be easy to give back to not just the maintainer of a package, but the full dependency tree of that package. Second, we think there should be a way to give back without having to open up your wallet. (If you have the disposable income to do so, great - but if you don't, we want to give you a way to give back too.)

Flossbank is a package manager wrapper. It takes a dependency tree snapshot when you install an open source package (like when running npm install). Then it distributes revenue to all the packages in the tree. What revenue? With Flossbank you can donate monthly or you can choose to see text-based ads while npm install is spinning away. Either way, all of those "dependencies of dependencies of dependencies" are getting their share.

If you're interested in learning more check us out at https://flossbank.com/how-it-works?rc=dto, or if you have general feedback don't hesitate to comment!