DEV Community: TinyStacks, Inc.

API Gateway REST vs. HTTP API: What Are The Differences?

Francesco Ciulla — Tue, 01 Feb 2022 19:51:06 +0000

Article by Jay Allen

AWS API Gateway is a great technology for managing and securing access to your backend REST APIs. However, AWS currently supports two very different versions of the technology. What are the differences? And which one should you use?

AWS covers the basics of the differences between these two technologies in its documentation. In this article, I plan to dive a little deeper by discussing some of the ways missing features from one version of API Gateway can be supported in the other. I'll also give some proscriptive recommendations around which version to use and when.

V1 vs. V2: Avoiding A Nasty Shock

AWS released the first version of API Gateway in 2015 with support for REST APIs. Over the next several years, AWS added numerous features to its REST API support. These included support for authentication via Cognito user pools, exposing private APIs publicly via VpcLink, and canary deployment support, among many others.

Then in 2019, AWS announced that, based on customer feedback, it had developed a new version of API Gateway. This V2 version included support for "HTTP APIs" (effectively REST APIs) as well as WebSocket APIs. A major goal of the change, AWS said, was to simplify the API Gateway model and make it easier to develop and deploy new APIs.

However, Amazon sowed a lot of confusion with this "new" API Gateway. First off, "HTTP API" is something of an odd naming, given that REST is a framework built on top of the HTTP protocol. I'm at a loss to understand why AWS chose a naming convention that makes them sound like diametric opposites.

Second, the name obfuscates that these are two separate versions of the same technology. This only really becomes clear if you look in one of two places:

CloudFormation syntax, where REST API and HTTP API syntax have separate namespaces (AWS::ApiGateway and AWS::ApiGatewayV2).
The AWS Console, where creating a REST API versus an HTTP API gives you two completely separate user experiences.

Above are screenshots of the REST API (first image) and the HTTP API (second image) user interface. As you can see, there's quite a change between V1 and V2. The changes extend, not just to UI organization, but to what features are available - and even down to the price and performance of each system.

In other words, before deciding which version of API Gateway to use, you should understand the differences between V1 and V2 in detail. And when you're researching information on the Web, be careful to identify whether the feature you're reading about is supported in REST APIs, HTTP APIs, or both.

Performance and Price Differences

The major difference between REST APIs and HTTP APIs are in performance and price. In short, HTTP APIs are the winner in both.

Both REST APIs and HTTP APIs only charge for the number of requests actually made plus data transferred out of AWS. However, the difference in pricing is steep. REST APIs will run you USD $3.50 per one million requests plus charges for data transferred out. By contrast, HTTP APIs cost a mere $1.00 per request for the first million requests and then $0.90 per million requests after that. That's a whopping 71% price differential.

On top of that, AWS says that V2 HTTP APIs contain significant performance improvements over their V1 REST brethren. Andreas Wittig at Cloudonaut ran some numbers and found a 14 to 16% improvement in latency in HTTP APIs compared to REST APIs.

As Andreas notes, the latency differential isn't that great. And odds are most of it will be wiped out by dependencies on other components, such as your database. So HTTP APIs are a clear winner in price and a small winner in performance.

Features in REST APIs (But Not in HTTP APIs)

So HTTP APIs are a clear winner when it comes to pricing. But, as I've noted before, price isn't everything. You can justify a higher cost if you're getting something in return for it.

Both REST APIs and HTTP APIs have features the others don't. Let's take a look at each, starting with the features in REST APIs that HTTP APIs lack.

Canary Support

Truth be told, my motivation for writing this article was that I wanted to build an API Gateway deployment with canary support. I'd heard that API Gateway supported canaries. Since I'm a big fan of API Gateway and its capabilities, I rushed into coding.

Unfortunately, what I started creating was an HTTP API. Imagine my shock and disappointment when I realized that HTTP APIs have no support for canary deployments! This is strictly a feature of REST APIs that V2 lacks.

One workaround is to incorporate an Application Load Balancer into your architecture. ALBs also support weighted routing, which allows you to implement canary-style deployments. Using an ALB with API Gateway is an established pattern that provides extra security, as you can host your ALB in a private subnet. You can then use private integrations and VPC Link in API Gateway to route API requests to endpoints in this private subnet.

This pattern carries the added benefit of limiting your Docker container's exposure to the Internet. You can host your container completely in a private subnet and expose only those REST endpoints you want made publicly available. For more information on implementing this pattern, see this article on the AWS Web site, which comes complete with an out of the box CloudFormation template.

Another workaround is to use Route 53's weighted routing feature. (AWS has docs on how to do this in the context of blue/green deployments.) In this case, you'd create another stage (e.g., canary) in your API Gateway. You'd then use weighted routing to route a percentage of traffic to the canary stage, gradually shifting over traffic as you verified the new version's performance. This will work but rollout may be slow due to DNS propagation delays.

Web Application Firewall (WAF) Support

AWS's WAF provides an additional level of security for Web apps. Using WAF, you can apply both pre-made and custom traffic security rules that filter out bots and known exploit vectors. WAF can both keep your application more secure as well as reduce illegitimate, bandwidth-wasting traffic.

API Gateway supports WAF. That is, if you use the REST API. HTTP APIs do not currently support WAF and there's no indication when they might.

If you use the architecture I mention above, you can work around this by turning on WAF on your private Application Load Balancer. ALB supports WAF, which means you can get the benefits of WAF while still enjoying the lower cost and higher performance of HTTP APIs.

Support for AWS X-Ray

X-Ray is AWS's service to add tracing and debugging instrumentation to your code. With X-Ray, you can monitor code and service performance, report errors, and troubleshoot the root cause of issues affecting your callers.

REST APIs have check-button support for adding X-Ray to your API Gateway calls. Sadly, as of this writing, this feature doesn't exist in HTTP APIs.

For teams that use their one home-brewed tracing option or a commercial one like New Relic, this won't be a huge deal. And others can still use X-Ray directly from their code through their programming language's AWS SDK or via the AWS CLI. So, while this is a nice-to-have, it's not necessarily a deal breaker.

Features in REST APIs (But Not in HTTP APIs)

Better Programmatic Model

The programmatic model is one area where HTTP APIs shine. One of AWS's avowed motivations in CREATING HTTP APIs was that the REST API was too complicated. HTTP APIs use a simplified programming model and a new and improved user interface in the console.

Additionally, HTTP APIs support several ease of use dev features that REST APIs don't, including direct support for CORS configuration, automatic deployments, and a default stage and route. However, REST APIs do support a couple of dev features, such as request body transformation, that aren't directly supported as of this writing in HTTP APIs.

However, REST APIs make development easier in one crucial way: they support importing API definitions from OpenAPI definitions supported by Swagger and other API definition/documentation tools. HTTP APIs can only export to OpenAPI.

Private Integrations

Private integrations allow API Gateway to expose resources hosted in a private VPC. Using a private integration, you can host your API sources (e.g., Docker containers) inside a private subnet while exposing only the endpoints you want to expose publicly through API Gateway. This results in enhanced security.

HTTP APIs contain full-fledged support for Application Load Balancers, Network Load Balancers, and AWS Cloud Map. Support is enabled via VPC Link, which allows you to create a route from your API Gateway to a private subnet. VPC Link is easy to configure and assign in both the console and in CloudFormation. (You can find a working CloudFormation example here.)

REST APIs support only Network Load Balancers. Additionally, the configuration isn't nearly as straightforward as it is with VPC Link.

Private integrations shouldn't be confused with private APIs. With private APIs, you can use API Gateway to define an API that's only available via a VPC. Calls to the API stay within the VPC and never route through the public Internet. Only REST APIs support private APIs.

Native OpenID Connect / OAuth 2.0

Finally, HTTP APIs have native support for OpenID Connect and OAuth 2.0. This is the one authentication framework that REST APIs don't natively support.

It's certainly possible to implement your own OAuth 2.0 support in REST APIs. But this is a heavier lift than just using HTTP API's native support.

Which to Use?

There are a few other feature differences I didn't cover here. It's best to familiarize yourself with the documentation and see what you would get - and what you'd miss - by picking one over the other. To make things easier, here's a quick run-down in table format:

As you can see, HTTP APIs have some catching up to do in terms of feature support. On the other hand, HTTP APIs are drastically more economical, performant, and easy to use.

In my view, you may choose to go with REST APIs if they choose some critical features that will make administration easier and reduce time to market (e.g., OpenAPI import). However, the overall improvements in HTTP APIs make them the default choice for most projects. You can directly implement most of the features unique to REST APIs without great effort. Plus, you'll get the benefit of future usability, price, and performance improvements as AWS continues investing in this new service.

Header image credit: Unsplash

Aiven VS AWS

Francesco Ciulla — Mon, 24 Jan 2022 21:02:30 +0000

Francesco on Twitter

Author of the article: Jay Allen.

Aiven is a new company that aims to simplify data storage and management in the cloud. In this article, I look at the benefits Aiven provides, its pricing model, and how their pricing compares to directly hosting your data services on AWS. I also consider when it makes sense to use Aiven vs. hosting on AWS directly.

The Problem of "Cloud Sprawl"

As we've discussed on this blog before, modern cloud providers have become insanely complex. Infrastructure as a Service (IaaS) companies like AWS continue to add an impressive array of features and services every month.

However, while that's made cloud services more useful, it's also made them harder to understand. Developers new to the cloud have to understand a huge host of similar-looking services and features before they can even make fundamental architecture decisions.

At the same time, this sprawl has made cloud dashboards much harder to use. Many AWS users complain about how hard it is to navigate the AWS Console in its current state.

In response, we've seen the rise of Platform as a Service (PaaS). PaaS companies like Heroku and service like Google App Engine aim to reduce the complexity of deploying software applications by providing an out of the box application stack consisting of data storage, virtual servers, virtual networking, and other foundational services.

What is Aiven?

Aiven is a PaaS product focused on data management. With Aiven, users can spin up a vast array of data storage and search services on various IaaS providers, including AWS, Google Cloud, and Microsoft Azure.

After you create an account on Aiven, you can spin up one or more of a larger number of data storage services, including event messaging (Kafka), relational and object-relational data storage (MySQL, PostgreSQL), time series databases (InfluxDB, M3DB), in-memory caching (Redis), and several others.

Once you select a service, you can select to host your infrastructure on a number of cloud providers: AWS, Google Cloud, Microsoft Azure, DigitalOcean, and UpCloud. You can also select a hosting plan, which determines how much memory and processor power your service can access.

Once your service is up and running, you can see the details on your Aiven dashboard. From here, you can access connection information and connect to your data host. Aiven also provides easy access to additional information about your service, including logs, connection pools, metrics, and backups.

Aiven Features

One of Aiven's greatest features is its ease of use. The Aiven UI is far and away easier to navigate than most cloud consoles. Creating a new data service generally takes a few clicks. And Aiven offers easy access to data (metrics, users, etc.) that would require a lot of customized setup if you were creating the service directly on a cloud provider.

Aiven also offers hosting flexibility. With support for five major IaaS providers, teams that use Aiven can easily locate their data hosting in the same cloud provider and even the same region as their application.

Since it's a PaaS, Aiven generally offers "black box" hosting. In other words, data services are hosted on cloud service accounts owned and operated by Aiven. However, customers with over $5000/month of spend can contact Aiven to arrange for direct hosting on their own cloud service accounts.

Finally, Aiven supports a number of advanced features for migration and monitoring The company supports its own aiven-db-migrate tool for migrating from an existing PostgreSQL database to Aiven. Aiven can also integrate with a number of different alerting and monitoring systems, including AWS CloudWatch Logs and Metrics, DataDog, Promethseus, and Syslog. (You can also set up your own metrics dashboarding easily with an Aiven-hosted Grafana dashboard.)

Aiven Pricing vs. AWS Pricing

Beyond features, however, we were also interested in the pricing model. How much does it cost to run a database service in Aiven versus, say, running it directly on AWS?

You won't be surprised to learn that Aiven costs more than vanilla AWS. That's only natural: it's a business providing a service. In this case, the service includes automation of data storage service creation, a slick management user interface, and the ability to create and manage resources cross-cloud.

But what's the cost? And is it worth it? The answer, as always, is: it depends on your scenario.

We ran the numbers on PostgreSQL hosting and compared using Aiven to running an equivalent-sized PostgreSQL instance directly on AWS. For example, Aiven's Startup-4 plan gives you 2 CPUs and 4GB of RAM and a single database instance . So we correlated this with an RDS PostgreSQL db.t4g.medium instance, which supports the same hardware configuration, hosted in a single Availability Zone. All RDS hardware specification data was derived from the AWS Web site and all AWS prices were calculated using the pricing calculator.

Below is a brief summary of the pricing differences for Aiven's startup plans:

Aiven	Aiven pricing	AWS alt	AWS pricing	Monthly savings	% raw savings
Startup-4	$99.00	db.t4g.medium	$56.65	$42.35	42.78%
Startup-8	$195.00	db.t4g.large	$114.30	$80.70	41.38%
Startup-16	$310.00	db.r6g.large	$204.50	$105.50	34.03%
Startup-32	$640.00	db.r6g.xlarge	$409.00	$231.00	36.09%
Startup-64	$1,200.00	db.r6g.2xlarge	$771.27	$428.73	35.73%
Startup-120	$2,140.00	db.r6g.4xlarge	$1,473.54	$666.46	31.14%
Startup-240	$4,280.00	db.r6g.8xlarge	$2,947.81	$1,332.19	31.13%
Startup-360	$8,700.00	db.m5.24xlarge	$6,582.12	$2,117.88	24.34%

Setting aside implementation costs for a moment, the charge for hosting your database directly on AWS will cost around 38% less than hosting it on Aiven. That price difference does include some networking costs that Aiven covers on your behalf. However, many of those charges can be avoided by proper placement of your AWS resources (e.g., running your RDS instance in the same VPC as your application, or using VPC peering to avoid Internet data transfer charges).

Of course, the bill is only half the story. You can't - and shouldn't - ignore how many person-hours or vendor dollars it might take to implement a direct IaaS solution. This will depend on how seasoned your staff is at data storage management and what reusable deployment and configuration solutions you already have available. If your team is starting from scratch with little cloud data management experience, Aiven will likely pay for itself.

When Direct Hosting on AWS Makes Sense

Does that mean you shouldn't use Aiven? Far from it. If your team doesn't have a data expert who's skilled in the various technologies that Aiven supports, its ease of use can save you significant time and money. Aiven's direct logging and metrics support may also save you dev dollars. And if you're pursuing a multi-cloud deployment strategy, Aiven's ability to deploy to all major cloud providers is a huge point in its favor.

However, if you don't have a multi-cloud strategy, the cost of Aiven may be more than it's worth. One way to answer this question for your team is to consider how much data you're storing.

Aiven charges a flat rate for data storage even if you don't use the entire allocation. By contrast, AWS only charges you for the data storage you actually use. And AWS charges far less than Aiven for the same amount of data.

On Aiven, you'll pay around $5 per every extra 5GB of storage on Aiven. Aiven gives small discounts the more storage you buy; e.g., an extra 80GB costs around $42/mo. instead of $50. But this still contrasts sharply with AWS, where an extra 10GB of storage only costs a little over $1 a month.

For example, under Aiven's Startup-4 plan, you receive up to 80GB of storage. If you use less than this on AWS, you'll save a few extra dollars a month. But you'll also have a lot more room to grow on AWS. In this configuration, you can store up to 450GB on a single-AZ configuration of PostgreSQL before you're paying as much as you pay to use Aiven.

In short, if your storage needs will fit within Aiven's default data tiers for your service level, it may well be worth the spend. But that value quickly decreases as your storage needs increase. If you expect large data growth, you may either want to consider hosting on AWS directly from the start, or ensuring you have a plan to migrate from Aiven to direct AWS hosting as your needs change.

Backup Storage Costs

One point we didn't address in the above is the cost of backup storage. On Starter plans, Aiven gives you two days of backups (14 days for Business plans). By contrast, you only get one included backup when you host directly on AWS.

However, AWS backup storage costs are (as of this writing) a scant US $0.095 per GiB. So, even in the case of the Starter-4 plan, adding a second backup on AWS only costs an additional $7.60 a month for 80 GB on AWS. Therefore, backups shouldn't be much of a factor in your cost calculations.

TinyStacks and Aiven

Like the folks at Aiven, we here at TinyStacks also think the cloud is too complicated! That's why we've built a service that provides full DevOps deployment pipeline automation. (You can see it in action on our YouTube channel!) We also include the ability to create an RDS PostgreSQL database - or use any other existing Amazon RDS instance - as part of each stack.

If you need to pursue a multi-cloud strategy, or use another data service outside of Amazon RDS, you can use any of your Aiven-hosted services easily from TinyStacks. Just pass the information for your Aiven resource - such as DNS name, port, and credentials - into your TinyStacks-hosted Docker app. Your application can read these secrets and connect to your Aiven data assets as it would any other data storage resource.

Conclusion

Aiven is an advanced and easy to use interface to various cloud data services. Whether it's worth the premium, however, depends on your use case. For multi-cloud deployments and teams without a data expert, Aiven can be a wise investment. However, AWS-only shops with high data storage needs will want to weigh their usage carefully before deciding whether that investment will yield dividends.

Banner photo by benjamin lehman on Unsplash

What Does a DevOps Engineer Do?

Francesco Ciulla — Thu, 06 Jan 2022 21:41:43 +0000

Hiring a DevOps Engineer for the first time? Knowing what to look for in a talented engineer can be a challenge. In this article, I discuss what you can expect from a DevOps Engineer in today's marketplace. I share some of my own experiences hiring DevOps Engineers in today's competitive labor market. Finally, I talk about cheaper alternatives to hiring a full-time DevOps Engineer.

When Do You Need a DevOps Engineer?

In my past articles, I've discussed DevOps release pipelines, stacks, and stages in-depth. A release pipeline is a software-driven process that development teams use to promote application changes from development into production. The pipeline creates multiple stacks - full versions of your application - across multiple stages of deployment.

A development team usually starts a pipeline automatically via a push to a source code control system, such as Git. The team then pushes the change set gradually through each stage (dev->test->staging->prod), testing and validating their changes along the way.

What I haven't discussed (directly, at least) is how complicated this process is. A DevOps release pipeline is itself a piece of software. It requires code to run - and that code needs to be tested, debugged, and maintained.

Many teams and small development shops get started without a dedicated DevOps engineer. Yours may be one of them! In these situations, a few team members generally own pieces of the pipeline and keep it running. Pipelines at this point are usually a mix of automated promotion and old-school manual deployment.

However, as your application and requests from your customers grow, you may realize the lack of a dedicated DevOps engineer is slowing your team down. Some of the signs include:

Your team's velocity slows under the weight of its current (mostly manual) deployment processes.
You have a somewhat automated deployment process but maintaining it is consuming more and more of the team's time.
You realize after a high-profile failure that your release procedures need professional help.
You know you should improve your deployment process but your team is so crushed with feature work that no one has time to spend on it.

If you're facing down one or more of these issues, it may be time to hire a part-time or full-time DevOps Engineer.

Responsibilities of a DevOps Engineer

A DevOps Engineer's role will likely look slightly different at every company. However, the following broad-based responsibilities tend to be common and consistent.

Automate the Full Release Pipeline

A good release pipeline eliminates unnecessary manual steps and reduces the time required to deploy changes to your application. Building and maintaining this pipeline is the DevOps Engineer's primary job.

DevOps Engineers usually craft release pipelines using a Continuous Integration/Continuous Development tool. Tools such as Jenkins, Atlassian, GitLab, and Azure DevOps integrate with source code control tools (usually Git) and handle triggering automated actions in response to repository check-ins. If your team already uses such a tool and is committed to it, you'll want to find someone proficient in your specific CI/CD toolset.

Many CI/CD toolsets offer a set of predefined actions to assist with the CI/CD process. However, other actions will be specific to your team's application. A DevOps engineer uses one or more scripting languages to automate complicated deployment tasks your team may have been executing manually. Python, JavaScript, shell scripting, and PowerShell (on Windows) are some of the more popular scripting languages that DevOps Engineers use.

For cloud-deployed software, a DevOps Engineer is also responsible for setting up the entire stack on which the application runs using Infrastructure as Code. A DevOps Engineer should be able to design and implement a stack deployment that can be deployed multiple times to any stage of your release pipeline.

Some engineers implement Infrastructure as Code using a scripting language such as Python. However, it's more common to use a templating language, such as CloudFormation on AWS or Azure Resource Manager (ARM) Templates on Azure.

Setting Best Practices for Software Development

As part of setting up the build and release pipeline, your DevOps guru will also define best practices for coding and validation of changes. In other words, they're the point person for your team's change management approval process.

For example, a DevOps Engineer may work with their team to devise the best way to manage the overall work process. For most teams, this usually means adopting an Agile approach to software development such as Scrum or Kanban. It could also mean defining a code review process and teaching the team how to conduct good reviews.

Monitor Builds and Deployments

The DevOps Engineer is responsible for ensuring the continued health of the team's CI/CD pipeline. This includes:

Monitoring build progress and logs from your team's CI/CD tool
Moving quickly to resolve broken builds and keep changes flowing through the pipeline
Observing dashboard metrics as new instances of the application come online
Staying alert for errors as your deployment shifts more users over to the new version of your application

Monitoring should occur in all stages of the pipeline. As Atlassian points out, pre-production monitoring means you can stomp out critical errors before they ever reach customers.

Depending on the size of your organization, the DevOps Engineer may supervise all of this themselves. They may also work in conjunction with a Sustained Engineering or Support team that's ultimately responsible for maintaining application health. In either case, your DevOps Engineer should take the lead in defining what the team needs to monitor.

Be the Git Guru

Ahhh, Git. The free source code control system is a marvelous invention. You can't be a developer nowadays and not know at least the basics of Git. And yet even seasoned developers will sometimes find themselves mired in Merge Conflict Hell.

A team's DevOps Engineer should know Git inside and out. They should understand, for example, the difference between a merge and a rebase - and which one to use when. They are the person primarily responsible for defining the team's branching and merging strategy - and maintaining quality internal documentation for other team members.

What to Look for in a DevOps Engineer

As an engineering manager, I've hired multiple DevOps engineers. During the interview process, my loops focus on validation a combination of technical and soft skills:

DevOps knowledge

Does the candidate have the basics of CI/CD down pat? What successes have they accumulated in developing successful pipelines? What setbacks have they encountered - and how have they overcome them?

Cloud platform and DevOps tools

In what DevOps tools is your candidate most experienced? Do they know the tools your team is already using?

A DevOps Engineer will also need to make numerous decisions on whether to buy or build certain parts of the DevOps process. For example, does your team roll its own artifact storage features? Or does it leverage a tool like Artifactory? DevOps Engineers need to remain up to speed on the tools marketplace so they can make these critical buy vs. build decisions.

Leadership

A DevOps Engineer needs to do more than build a pipeline. They need to convince a (sometimes reluctant) team of engineers and stakeholders to change the way they develop software. Does your candidate have experience talking a tough crowd into adopting new processes?

As a manager, I like to use STAR (Situation-Task-Action-Result) questions to determine a candidate's experience with being a technical leader. So I might ask something like, "Tell me about a time when you received pushback from your team on a process change. What was it and how did you resolve it?"

Growth mindset

The DevOps and cloud spaces are changing constantly. So it's important that a DevOps Engineer not get overly set in their ways.

I also like to use STAR questions to gauge a candidate's willingness to grow. For example, what's the last thing that they learned just because it looked interesting? Did they end up using it on the job? If so, what was the result?

Alternatively, I may ask when was the last time they received critical feedback from their manager. What was it? And how did they use that feedback to improve their job performance?

Alternatives to Hiring a Full-Time DevOps Engineer

You've determined that you need more DevOps savvy in your org. But that doesn't mean you need to start off with a full-time position out of the gate. Maybe you can't afford a full-time position at the moment. Or perhaps you'd just like to test the waters before diving in with both feet.

Fortunately, there are a couple of alternatives to hiring someone full-time.

Hire a Part-Time DevOps Engineer

You may not need (nor even desire) a full-time team member. It may be enough to hire someone on a part-time basis to construct and maintain your build and release pipeline.

In this scenario, you'd want to find a DevOps Engineer who's good at building self-service solutions. Your team should be able to kick off builds, perform releases, and monitor rollouts without having a full-time DevOps Engineer on call to oversee a successful outcome.

Migrate to TinyStacks

Another option? Forego the engineer! You can potentially save both time and money by adopting a DevOps tool that essentially provides you "DevOps as a service".

TinyStacks is one such tool. Built by a team with deep experience building out the Amazon Web Services console, TinyStacks provides an automated approach to DevOps. Using a simple UI Web interface, your team can migrate its application into a full release pipeline - complete with AWS cloud architecture - within the week.

Read a little more on what TinyStacks can do for you and contact us below to start a discussion!

Article by Jay Allen

Approval Workflow: Manual and Automated Approvals in CI/CD

Francesco Ciulla — Tue, 28 Dec 2021 20:11:50 +0000

Recently, I've gone into detail on stacks and stages. I've also examined the importance of dev stacks for both teams and individual developers. Building on these topics, I wanted to talk today about approvals.

How do you promote changes to your stacks to production? More importantly, how do you gate promotions to ensure quality code? I'll look at the two major approaches to approvals - manual vs. automatic - and when and how to use each approach.

What is a Change Management Approval Process?

Traditionally, nothing strikes more fear in the heart of a dev team than pushing a change to production. Change promotion is usually an "all hands on deck" affair. Engineers and support personnel often stand at the ready, testing lives sites and monitoring dashboards for the slightest hint of trouble.

What can go wrong when pushing a change?

A code change that wasn't thoroughly tested or reviewed can break on release.
A code change that worked in dev might not work in production.
A configuration change could break a production server or not be distributed to all instances in a cluster.
A new part of your cloud infrastructure could fail to deploy correctly.
...and any number of other things that keep developers awake at night.

The question isn't "What can go wrong?" during a deployment. It's more like, "What can't go wrong?"

Because of this, software teams don't just shove a change into production and hope for the best. Most teams have some sort of change management approval process.

As I discussed previously, a deployment pipeline consists of a number of stages. Each stage - dev, test, staging, prod - is used to widen a change's availability and vet its quality. A change management approval process sets guidelines for when a change can flow from one stage to the next.

Types of Approvals

Traditionally, there are two types of approvals. Often, both types are used at different stages of the development process.

Manual Approvals

With a manual approval, a change requires some sort of human intervention to progress to the next stage. Often, this takes the form of a code review or buddy test, in which another member of your team reviews your changes before approving them. Once approved, the change migrates to the next stage.

A manual approval is also a good way to await feedback from stakeholders and customers. For example, you may make changes or a new feature available in a staging or demo environment that internal stakeholders and other teams can access. Once the changes have passed all tests and have secured stakeholder approval, you can approve and push them into production.

Manual approval doesn't mean that your release pipeline contains zero automation. You will likely still have steps in your deployment pipeline where you're running unit tests, smoke tests, service health checks, and other automated quality checks.

Automated Approvals

With an automated approval, a change migrates to the next stage if it passes a set of automated checks. These can include but are not limited to unit tests, service health checks, and security checks.

Automated approvals are typical in earlier stages of a release pipeline - e.g., moving from dev to test, or test to stage. They're harder to achieve in production, as they require a high degree of automated testing and verification to ensure users don't get broken bits. Automated delivery into production is often referred to as continuous delivery.

Typically, an automated approval into production will use some sort of phased release strategy. For example, you may deploy code changes to a single server (a canary). You would then test/monitor the results before deploying to all machines in a fleet. Or you may do a rolling deployment in which you deploy new code to a small percentage of your servers or serverless endpoints. If the change doesn't produce any errors (HTTP server errors, virtual machine connectivity issues, etc.), the system continues the promotion process. If there are errors, it rolls back the changes and stops the rollout.

Implementing Manual Approvals on a Pipeline in AWS

Most pipeline technologies provide some way to switch easily between manual and automated approvals.

For example, AWS CodePipeline structures a pipeline in a series of stages. Each step consists of a series of actions. In AWS, you can add a Manual Approval action to a stage.

The manual approval action will stop pipeline execution until someone approves it. AWS sends approval requests to an Amazon SNS (Simple Notification Service) topic. This means you can send the request to one or multiple potential reviewers. You can also configure the message to include a URL link. This is helpful if your team uses a code review software system like Review Board.

TinyStacks Makes Approvals Easy

At TinyStacks, our goal is to make DevOps easy. Our simplified pipeline creation tools will flow approvals automatically from stage to stage. Adding a manual approval is as simple as clicking a checkbox! Your teammates can then easily view and approve the migration to the next stage from the TinyStacks dashboard. Contact us today to see how TinyStacks can simplify your journey to DevOps maturity!

Article by Jay Allen

Dev Environments: An Essential Tool for Software Quality

Francesco Ciulla — Tue, 14 Dec 2021 19:39:39 +0000

There are many steps on the road to DevOps maturity. Recently, I've been covering some of the most basic concepts, such as stacks, stages, and Infrastructure as Code. Today. I'll stick to these foundational steps and talk about on-demand dev stacks. I'll focus on why dev stacks are perhaps the most important first step teams can take on their DevOps journey.

Your Application, On Demand

First, let's recap some concepts from my last article. One of the great benefits of moving to a cloud platform like AWS is Infrastructure as Code. With Infrastructure as Code, you can spin up the architecture your application needs - network topology, Web servers, databases, file storage, load balancers, etc. - by programming it.

Before Infrastructure as Code, standing up a new version of an app usually meant manually configuring and tending to every component of the system. It was tedious and error-prone. Defining your architecture in a programming language like Python or in a declarative language like AWS CloudFormation means you can deploy and re-deploy your application over and over, consistently and without fear.

Stacks, Stages, and Environments

Before I dive in, let's get clear on our terminology.

Using Infrastructure as Code, you can deploy a stack - your application plus all its supporting infrastructure - quickly and easily. Once you can deploy a stack, you can deploy multiple stacks - stages - for various purposes - e.g., a single stack for production, plus other stacks for development and testing.

People in software development talk a lot about environments - e.g., production environments vs. dev environments. In our view, "environment" encompasses a specific runtime for your application that may or may not be hosted in the cloud. For many development teams, dev environments reside on a developer's desktop or laptop.

Production Stacks and Dev Stacks

Many teams are drawn to Infrastructure as Code to streamline their production deployments. And indeed, repeatable production employments can greatly enhance application quality. Standing up new production stacks opens the door to numerous advanced deployment strategies such as canary testing and blue/green deployments.

But Infrastructure as Code can improve quality before your team even pushes to production. You can use the same code you use to stand up a production stacks to stand up development stacks as well!

Why a Dev Stack?

Personal dev environments are becoming increasingly standardized with tools such as Gitpod and Codespaces. As your team moves more toward standing up stacks, the difference between personal dev environments and dev stacks starts to fade.

Dev stacks allow development teams to test their changes end to end before they're ever pushed to production. Using Infrastructure as Code, teams are assured that what they're testing is (apart from a few small config changes) identical to what will run in production.

Having a central dev stack for your team is great. However, giving developers their own fully deployed stacks makes it even easier to test changes before they ever hit main.

With individual dev stacks, your developers can deploy individual changes faster. This leads to greater flexibility and reliability over grouping many changes together into a single deployment. In addition, when building on cloud services, testing against a live service is better than attempting to replicate that service on a laptop.

The "Official" Dev Stage

If your team hasn't started using dev stacks yet, the first step is to make a shared stack. This will be the start of your application pipeline.

A pipeline is a series of stages through which you can push code changes, with each stage gradually widening your user base. On large and long-running projects, pipelines can involve multiple stages and become fairly complicated. However, a simple pipeline consisting of just a dev and a prod stage is a solid start for teams just dipping their toes into the DevOps waters.

To create a dev stage, you first need to create a full application stack using a language such as AWS CloudFormation. Your stack should define everything that your application needs to run.

If you already have this for your production stack, then you're almost there! You may need to make a few adjustments based on how you want to launch your dev stack. You have a couple of choices here.

Launch in Same AWS Account as Prod

The simplest strategy is launching your dev stack in the same stack as your prod account. To do this, you'll need to parameterize your Infrastructure as Code deployment so that it uses different prefixes or suffixes for resource names. This will avoid naming collisions with your prod stack.

AWS CloudFormation makes this easy through the use of parameters. And actually, you don't even need to define your own parameters! You can use CloudFormation's pseudo-parameters - predefined metadata parameters - to implement this quickly and easily.

For example, assume you are defining an S3 bucket name and want to make sure it's distinct from your production bucket. Using CloudFormation, you can use the name of your stack as a prefix for the bucket name. In the example below (YAML), we use a regular CloudFormation stack parameter, AppVersion, and the full stack name as a pseudo-parameter to construct a unique name.

BucketName: !Sub "{$AWS:StackName}-${AppVersion}"

Launch in Separate AWS Account

However, it's not a great idea to mix stacks in a single account. Ideally, you want your production stack hosted in its own AWS account. This allows you to place additional restrictions on access to production. Such restrictions are almost a necessity if your team handles personally identifiable information on customers in prod.

If you launch your dev stack in a separate account, you don't need to worry about name conflicts. The only thing you should have to parameterize in this context are publicly facing values, such as your application's DNS endpoint.

A Dev Stack Per Developer

Creating a central dev stack is definitely a huge step forward. However, there's still room for improvement!

A central dev stack is fine for integrating changes that are getting close to production quality. Ideally, however, you want devs to be able to test in their own stacks before committing to a common Git branch. This reduces merge conflicts and helps ensure high-quality code early in the development process.

If you already have code for launching a dev stack, launching individual dev stacks for developers shouldn't involve much additional work. The major issue is tracking stacks and controlling costs. Giving your entire dev team unfettered access to an AWS account - even a non-production one - can leave you scrambling to control your cloud spend.

One approach is to use AWS Control Tower. Control Tower works in conjunction with AWS Organizations, which enables the creation and management of multiple AWS accounts under a single master account. You can use Control Tower in conjunction with AWS Service Catalog to offer your dev stack as a service catalog offering that developers can install into their accounts. You can even go one step farther and deploy the stack automatically as part of the account vending process.

Defining Your Branching Strategy with Dev Stacks

As I discussed in my last article, it's important when creating your CI/CD pipeline to work out a branching strategy. One of the simplest strategies is to use feature branches for development work. In feature branching, devs create a branch per feature. Developers use pull requests to request integration of their work into main.

Feature branching has several benefits. By using pull requests, other team members can review and vet a set of changes before they are integrated into the main branch. The entire process keeps your project's main branch clean and in a buildable, deployable state.

Whatever branching strategy you choose, there's little doubt that giving developers their own fully deployed stacks makes it easier to test changes before they ever hit main. The result is faster deployments and more reliable code.

TinyStacks Makes Dev Stacks Easy

Here at TinyStacks, we’re all about helping you deploy and manage your stacks in the cloud. We make it easy to transfer from a personal dev environment on your laptop into a development stage with a stack consistent with your production stack. Contact us today to find out more!

A Guide to Stacks and Stages on AWS

Francesco Ciulla — Tue, 23 Nov 2021 22:25:53 +0000

Article by Jay Allen

Learning AWS is complicated enough. But learning AWS is made more challenging when you're also still grappling with some of the major concepts of DevOps software deployments. In this article, I discuss two key concepts: stacks and stages. I also address how you can manage stacks and stages in AWS, along with other factors you need to consider when managing them in practice.

Stacks

In simplest terms, a stack is a unit of application deployment. Using stacks, developers can organize all of the resources required by an application or an application component as a single unit. This enables devs to deploy, tear down, and re-deploy their applications at will.

Stacks can be stood up manually. However, it's better on cloud platforms to program the creation of your stack - e.g., using a scripting language such as Python. The ability to script stack deployments is known as Infrastructure as Code and is a hallmark of cloud computing platforms. Scripting your application deployments and bundling them into stacks reaps multiple benefits:

Once your stack code is fully debugged, you can deploy your application repeatably and reliably. Scripting stack deployments eliminates the errors that inevitably occur in manual deployments.
You can tear down stacks that aren't being used with a single script or command. This saves your team and company money.
You can parameterize stacks to deploy different resources or use different configuration values. This lets you deploy multiple versions of your application. (Remember this - it'll be important soon!)

Stages

A stage, by contrast, is a deployment of your application for a particular purpose. With stages, you can deploy your application multiple times to vet its functionality with an increasingly larger number of users. Typical stages can include:

Dev for developer coding and experimentation (only available to your dev team)
Test for running unit tests (available to dev, test, and internal stakeholders)
Stage for user acceptance testing (available to external alpha/beta testers)
Prod for your publicly facing application (available to all customers)

Stages are part of CI/CD pipelines, which I've discussed in detail before. By constructing your application as a pipeline, you can "flow" app changes from one stage to the next as you test them in each environment. This lets you vet changes multiple times in limited, controlled environments before releasing them to your users.

Stacks and Stages: Better Together

Stacks and stages are a powerful one-two combination. With a properly parameterized stack, you can create whatever stages your application needs. Because you create each stage using the same source code, each stage's stack will contain the same resources and perform the same way as every other stage.

Stacks on AWS

AWS fully embraces Infrastructure as Code. Nearly anything you can accomplish manually with the AWS Management Console can also be created programmatically.

On AWS, you have several options for creating stacks.

AWS CloudFormation

AWS CloudFormation is the official "AWS way" of creating stacks. Using CloudFormation, you can write templates using either JSON or YAML that specify which AWS resources your stack contains.

CloudFormation isn't an imperative programming language like Python. Instead, it uses a declarative format for creating resources. This simplifies creating your infrastructure, as you don't need to be an expert in a particular programming language to stand up resources. Many CloudFormation templates can be constructed by making small tweaks to publicly available templates. ( AWS itself hosts many such sample templates and snippets .)

A key feature of CloudFormation is its support for parameters. Rather than hard-code values, you can declare them as parameters and supply them at run time when you create the stack in AWS. For example, the template snippet below (taken from AWS's sample template for deploying Amazon EC2 instances) defines the parameters KeyPair, InstanceType, and SSHLocation. By parameterizing these values, the same template can be used multiple times to create different EC2 instances of different sizes, in different networks, and with different security credentials.

The great thing about CloudFormation templates is that they make stacks both easy to turn on and easy to turn off. Deleting an instance of a CloudFormation template automatically cleans up the entire stack and deactivates all of its resources.

Your Favorite Programming Language

Not everyone wants to learn a new declarative language to create stacks. And some stacks might require the fine-grained control that an imperative programming language offers.

Fortunately, AWS also produces software development kits (SDKs) for a variety of languages. Developers can use Python Go, Node.js, .NET, and a variety of other languages to automate the creation and deletion of their stack.

Which is Better?

CloudFormation's major advantage is simplicity. Particularly, CloudFormation makes deleting stacks a breeze. By contrast, with a programming language, you need to program the deletion of every resource.

However, using a programming language for stack management offers much greater control than CloudFormation. For example, let's say that a resource fails to create. This can happen sometimes, not because you did anything wrong, but due to an underlying error in AWS, or a lack of available resources in your target region.

Using CloudFormation, a failed resource will result in the stack stopping and everything you've created rolling back. Using a programming language, however, you could detect the failure and handle it more gracefully. For example, you may decide to retry the operation multiple times using incremental backoff.

Your choice between CloudFormation and programming language may also be affected by feature parity. In the past, some AWS teams have released features with SDK support but no initial CloudFormation support.

Many of these issues with CloudFormation can be addressed using a hybrid CloudFormation/code approach. Using CloudFormation custom resources, you can run code in AWS Lambda that orchestrates the creation of both AWS and non-AWS resources. You can also perform other programming-related tasks that might be required for your stack, such as database migration.

In the end, both approaches work fine. My personal recommendation would be to use AWS CloudFormation in conjunction with custom resources when needed. CloudFormation is well-supported and can easily be leveraged by other AWS features (as we will see shortly).

Stages on AWS

The easiest way to manage stages on AWS is by using AWS CodePipeline.

CodePipeline performs two major services. First, it orchestrates multiple AWS services to automate every critical part of your application deployment process. Using CodePipeline, you can ingest code from your code repository (such as GitHub), compile it using AWS CodeBuild, and deploy your application's resources using (you guessed it) AWS CloudFormation.

Second (and most important for today's discussion), CodePipeline supports defining separate stages for your application. When you create a CodePipeline, you create stages that handle importing your source code from source control and building the code. From there, you can add additional deployment stages for dev, test, stage, prod, etc.

In the screenshot below, you can see a minimal deployment pipeline. The third step after the CodeBuild project is a dev stage, intended for developer vetting of new changes.

We could easily add a new stage to our pipeline by clicking Edit and then clicking the Add Stage button.

After you add a stage, you can add one or more action groups. Action groups support a large number of AWS services, including AWS CloudFormation. For our test group, for example, we could add two action groups:

A manual approval. This would stop changes from the dev branch from flowing to test automatically until someone approved the change in the AWS Management Console (e.g., after performing a code review).
An AWS CloudFormation template to deploy our infrastructure stack for the test stage.

When using a CloudFormation script with CodePipeline, you can specify a configuration file that passes in the parameters the CloudFormation script needs to build that stage properly. This might be as simple as prefixing created resources with the name "test" instead of "dev", or as complicated as specifying a data set to load into your database for testing.

Managing Stacks and Stages in Practice

In theory, stacks and stages are pretty simple concepts. In practice, however, it takes a lot of work and fine-tuning to get your CI/CD pipeline to the point where you can deploy your application reliably across multiple stages. Your team also needs to make some up-front decisions about how it's going to manage its source code and work product.

Below are just a few factors to consider when devising your approach to stacks and stages on AWS.

Source Code Branching

A key up-front decision with stacks and stages is how your team will flow changes from development into production. A big part of this decision is how you manage branches in source control.

There are multiple possible branching patterns. On his Web site, programming patterns guru Martin Fowler has documented the key strategies in excruciating detail. On their Web site, Microsoft offers a simpler, more prescriptive approach:

Define feature branches that represent a single feature per branch.
Use pull requests in source control to merge feature branches into your main branch for deployment.
Keep your main branch clean and up to date.

This is, of course, only one way to do things. The important thing is that your branching strategy is clean, simple, and easy to manage. Complex branching strategies that require multiple merges and resolution of merging conflicts end up becoming a nightmare for development teams and slow down deployment velocity.

Unit of Deployment

Another fundamental consideration is the unit of deployment - i.e., how much of your application do you deploy at a time?

Many legacy applications deploy an application's entire stack with every deployment. This so-called monolithic architecture is easy to implement. However, it lacks flexibility and tends to result in hard-to-maintain systems.

The popular alternative to monoliths is microservices. In a microservices architecture, you break your application into a set of loosely-coupled services that your application calls. You can get incredible deployment flexibility with microservices, as you can bundle each service as its own stack. However, managing versions and service discovery in a complex Web of microservices can be daunting.

You can also take an in-between approach. Some teams divide their apps up into so-called "macroservices" or "miniservices" - logical groupings of services and apps that can each be deployed as a single unit. Such deployments avoid the downsides of monolithic deployment while also steering clear of the complexity of microservices.

Data Management

Next, there's how you'll manage data. At a minimum, your team needs to consider how to handle:

Loading data into a dev/test/staging system for testing purposes.
Managing schema changes to your data store (e.g., adding new tables/fields to relational database tables with a new release).

Some development frameworks, such as Django, use an Object Data Model (ODM) or Object Relational Model (ORM) that automates database migrations. In these cases, your application simply needs a way to trigger a migration using the relevant scripts. The AWS Database Blog has some detailed tips for incorporating database migrations into a pipeline.

Managing Secrets

While automation is great, it introduces a devilish problem: managing secrets. Your application can access most AWS services using an AWS Identity and Access Management (IAM) role. However, it will likely also need to connect to other resources - databases, source control systems, dependent services - that require some sort of authentication information, such as access and secret keys.

It can't be said clearly enough: storing secrets in source code is a huge no-no. And storing them in plain text somewhere (like an Amazon S3 bucket) isn't any better.

Fortunately, AWS created the AWS Secrets Manager for just this purpose. Using Secrets Manager, you can authorize your application via IAM to read sensitive key/value pairs over a secure connection. You can even use CloudFormation to store secrets for resources such as databases into Secrets Manager as part of building a stack.

Conclusion

Stacks and stages are cornerstone concepts of DevOps deployments. Once you can deploy your application as a single unit or collection of units, you can spin up any environment you need at any time. The payoff? Faster deployments and more reliable applications - and, as a consequence, happy customers!

Flask CRUD API

Francesco Ciulla — Fri, 19 Nov 2021 14:39:17 +0000

Welcome back on the Docker and AWS series by TinyStacks

In this Article, we will create a simple CRUD API using a Flask Application , Docker, Postgres

Video Version:

Steps:

Create a folder
Create requirements.txt
Create app (~50 loc)
Create Dockerfile
Create docker-compose.yml
Run database
Check database
Run python app
Check that the table has been created
Test endpoints (Postman)
Test Get All endpoint
Create a record(x3)
Get a record
Update record
Delete record

Create folder and step into it

You can create a folder in anyway that you prefer. If you use a terminal you can type:

mkdir  flask-crud-api

Then, step into the folder:

cd flask-crud-api

Then, open this folder with your favorite IDE. If you use Visual Studio Code, you can type:

code .

Now we’re ready to get started coding our Flask Crud API application with the help of GitHub Copilot!

Create requirements.txt

First of all, we need to define the dependent Python libraries for our application. The standard method in Python is to create a requirements.txt file and list our dependencies there.

So create this file called "requirements.txt". (If you have the Material Icon Theme, it will show a nice little Python icon. It’s a nice to spot typos!)

Then we can type the dependencies for our project:

flask
psycopg2-binary
Flask-SQLAlchemy

Those dependencies are:

flask: The Python Framework
psycopg2-binary: To create the connection with the Postgres Database
Flask-SQLAlchemy: Help generate SQL queries without writing them manually

Create app (~50 loc)

At the root level, create a file called app.py. We will write our crud API app in about 50 lines of code!

Let’s specify the libraries we’ll use:

from flask import Flask, request, jsonify
from flask_sqlalchemy import SQLAlchemy
import os

Next, define the Flask app and how to run it:

app = Flask(__name__)

if __name__ == '__main__':
    app.run(debug=True)

Define an environment variable as a string and initialize the SQLAlchemy instance to handle the Postgres database:

app.config['SQLALCHEMY_DATABASE_URI'] = os.environ.get('DATABASE_URL')
db = SQLAlchemy(app)

Now let's define our data model. We’ll create a class named Item with just title and content as properties. We’ll also add an auto-incremental Integer named id. This will act as the primary key for our table.

class Item(db.Model):
  id = db.Column(db.Integer, primary_key=True)
  title = db.Column(db.String(80), unique=True, nullable=False)
  content = db.Column(db.String(120), unique=True, nullable=False)

  def __init__(self, title, content):
    self.title = title
    self.content = content

Now a little bit of magic: with this line we let SQLAlchemy to synchronize with the Postgres database. This will create our databasetable automatically for us!

db.create_all()

Define REST endpoints

Now we need to implement our CRUD endpoints. CRUD stands for:

CREATE
READ
UPDATE
DELETE

These are the basic functions of every application.

To retrieve a singleitem, we define this function:

@app.route('/items/<id>', methods=['GET'])
def get_item(id):
  item = Item.query.get(id)
  del item.__dict__['_sa_instance_state']
  return jsonify(item.__dict__)

To get all the items in the database, we define this function:

@app.route('/items', methods=['GET'])
def get_items():
  items = []
  for item in db.session.query(Item).all():
    del item.__dict__['_sa_instance_state']
    items.append(item.__dict__)
  return jsonify(items)

To create a new item:

@app.route('/items', methods=['POST'])
def create_item():
  body = request.get_json()
  db.session.add(Item(body['title'], body['content']))
  db.session.commit()
  return "item created"

To update an existing item:

@app.route('/items/<id>', methods=['PUT'])
def update_item(id):
  body = request.get_json()
  db.session.query(Item).filter_by(id=id).update(
    dict(title=body['title'], content=body['content']))
  db.session.commit()
  return "item updated"

To delete an existing item:

@app.route('/items/<id>', methods=['DELETE'])
def delete_item(id):
  db.session.query(Item).filter_by(id=id).delete()
  db.session.commit()
  return "item deleted"

That's it! and in less than 60 lines of coding (included new lines)!

Create Dockerfile

A Dockerfile is a text file to define a set of commands to create an image. Starting from this image, we will run our python containers

Let's create a file called Dockerfile (capital D, no extension).

We could create of course a file with a different name.But this is the default one that Docker uses. If we use it, we don't have to specify a name for the file when we build our Docker container image.

This is the final file:

FROM python:3.6-slim-buster

COPY requirements.txt .

RUN pip install -r requirements.txt

COPY . .

EXPOSE 80

CMD ["flask", "run", "--host=0.0.0.0", "--port=80"]

Let's explain briefly what's going on here:

FROM: Set the baseImage to use for subsequent instructions. FROM must be the first instruction in a Dockerfile.
COPY: Copy files or folders from source to the dest path in the image's filesystem. The first COPY copies the requirements.txt file inside the filesystem of the image; the second one copies everything else.
RUN: Execute any commands on top of the current image as a new layer and commit the results. In this case, we are runningpip` to install the Python libraries we need.
EXPOSE: Informs Docker of the port we will use at runtime. (PRO tip: this line is not really needed. It makes the intent of the Dockerfile clear and facilitates the translation to the docker-compose.yml file)
CMD: Provide defaults for an executing container. If an executable is not specified, then ENTRYPOINT must be specified as well. There can only be one CMD instruction in a Dockerfile.

Create docker-compose.yml

Now that we have created the Dockerfile, let's create the docker-compose.yml file to make our life easier.

This is the final file:

`yml
version: '3.9'

services:
pythonapp:
container_name: pythonapp
image: pythonapp
build: .
ports:
- "80:80"
environment:
- DATABASE_URL=postgresql://postgres:postgres@db:5432/postgres
depends_on:
- db

db:
container_name: db
image: postgres:12
ports:
- "5432:5432"
environment:
- POSTGRES_PASSWORD=postgres
- POSTGRES_USER=postgres
- POSTGRES_DB=postgres
volumes:
- pgdata:/var/lib/postgresql/data

volumes:
pgdata: {}
`

Let's explain what's happening line by line:

version: '3.9' is the current version of the docker-compose.yml file.
services: The top-level entry of our docker-compose.yml file. The services are basically the containers.
pythonapp: The Python application we just wrote
container_name: Defines a custom name for our application. It’s the equivalent of using the --name option at the command line when we run docker run.
image: Rhe image for this service (container). Here, we are defining a custom name just to use it locally. If we want to push our containerto a public or private registry (a place to store Docker Images, e.g. Docker hub), we need to change the tag of the image (basically the name). We don’t need to do that now.
build: We need this option if we are using our custom image and not an existing one. The dot after the semicolon is the path of the Dockerfile, and it means "here is where I’mrunning the docker-compose.yml file". Please note that the docker-compose.yml file and the Dockerfile are at the same level.
ports: A list of ports we want to expose to the outside. A good practice is to make the content a quoted string.
environment: Key-value pairs. Here, we use them to define our custom URL to connect to the Postgres database.
depends_on: Express dependency between services. Service dependencies cause the following behaviors:
- docker-compose up starts services in dependency order. In the following example, db and redis are started before web.
- docker-compose up automatically includes a service’s dependencies. In the example below, docker-compose up web also creates and starts dband redis.
- docker-compose stop stops services in dependency order. In the following example, web is stopped before db and redis.
db: Service for the Postgres database.
container_name: The default name for this service, also called db
image: postgres:12 : We will not use our custom image in this case but an existing one, the one the Postgres team has created and pushed for us on Docker Hub.
ports: A list of ports we want to expose to the outside. A good practice is to wrap this content in a quoted string.
environment: Here we define three environment variables for the Postgres service. The keys are not arbitrary, but are the ones defined in the official Postgres image. We can, of course, define the values of these environment variables (this is why the Postgres team has given them to us, to use them!).
volumes: Here we use a named volume called pgdata. the part before the ':' is the name of the volume, and the part on the right of the ':' is the destination path

At the end of the file, we define the actual volume named pgdata.

Run the database service locally

To run the database service locally, we can type:

bash docker compose up -d db

the -d option stands for detached, to leave out terminal available after running this container.

You can check the status of the running container by typing:

docker ps

Check the database

To step inside the Postgres container we will use 2 different approaches.

First approach: directly from the Command line

docker exec -it db psql -U postgres

But if we type

\dt

We will see this:

did not find any relations

This is correct, because we haven't run our Python container yet.

To exit the psql process, type:

exit

Or alternatively, just

\q

Run Python app

To run your Python application, type:

docker compose up --build pythonapp

The --build option is to build your application before running it. It's not useful the first time you run this command because Docker is going to build the image anyway. It becomes useful after you run docker compose up multiple times and you’ve made some changes on your app.

Note: If you build an image using the same tag (name), the previous image will become a so-called "dangling image", with <none> <none> as the repository and tag. To remove them, you can type docker image prune and then y to confirm.

if you see something like this, you have successfully launched your Python Flask application:

Now you can once again check the running containers:

docker ps -a

(Side note: don't mind the "created" value, it's just me removing/stopping the containers for demo-purposes! You should see them both running with a status of some minutes ago.)

Check the table has been created

If you step again inside the Postgres container now, using the command:

docker exec -it db psql -U postgres

and you type:

\dt

you will see the table has been created automatically, without calling any endpoint!

This was possible because of the line (around 22) on the app.py file:

python db.create_all()

Test endpoints (using Postman)

Let's test this simple application! We will use Postman, but you can use any REST API testing tool that you prefer.

Get All

Let's get all the items:

Create one (x3)

Now let's create some new items

Apologies for my lack of imagination :)

Get one

To get a single item, you can just make a GET request at the endpoint /item/<id>, where <id> is the unique ID of the item that you previously created.

For example, to get the item with id 2:

(Please note that we are not handling errors correctly in this example. If that id doesn't exist, we’ll get an error directly from the application and we won’t show an error message to the end-user.)

Update one

To update an existing item, you can make a PUT request using the <id> of the item in the body:

Delete one

Finally, we can delete an existing item from the database. We can make a DELETE request and appending an existing <id> at the end of the url:

If you get all the items again, this will be the result:

Test the final status using the prompt

You can test the final status also directly on the db:

docker exec -it db psql -U postgres

and then run the psql command (don't forget the final ';'):

select * from item;

All the code is available at this url: https://github.com/tinystacks/aws-docker-templates-flask/tree/flask-local-postgres

Video Version:

Video offered by TinyStacks

Migrating DigitalOcean database to AWS

Francesco Ciulla — Thu, 14 Oct 2021 12:54:40 +0000

Video Version: https://youtu.be/3zLWCNn0Vqk

In this article, we’ll look at how to migrate an existing Postgres database on DigitalOcean created through their "managed database" function to a Relational Database Service (RDS) instance on AWS.

For an introduction to RDS, you can read my previous article on migrating a local database to RDS, or watch the video.

Prerequisites:

Managed database on DigitalOcean with some data in it
AWS account
(optional) TablePlus or any other tool to manage a Postgresql DB

Steps:

Check DigitalOcean and download the connection certificate
Install TablePlus (GUI tool for managing relational databases)
Check DigitalOcean database
Create RDS instance
Test empty RDS instance
Backup DigitalOcean DB
Restore RDS database
Final Test

Check DigitalOcean and Download CA certificate

First, let's visit our DigitalOcean account’s Database page. We should see something like this:

Download the CA certificate locally. We need this because managed databases on DigitalOcean don't allow insecure connections.

TablePlus

To access the DB, we can use whatever tool we want (command line interface, Pgadmin, etc.). In this demo, we will use TablePlus (available on Mac/Windows), so if you want to follow along exactly I suggest you download it. We’ll use the free version.

Check DigitalOcean Database

Let's create a new connection on TablePlus:

Add the details for your DigitalOcean database:

Host
Port
Username
Password
Database name
SSL mode: REQUIRED

Remember also to add the certificate we just downloaded.

Click Connect and you will see the database with your data. In this case, we have just 2 tables and 3 inserts.

Create RDS Instance

Go on AWS Console and search "RDS"

Click Create Database.

Select Postgres and version 12, so we will have access to a free Tier (read the conditions before accepting).

Choose a name for the database, as well as a username and password to access the DB.

Make the instance accessible from the Internet:

Double-check that this is a Free Tier (with limitations- please read them)Then, click Create Database.

This will take a few minutes to complete.

Let's check that our security group is configured correctly. Our machine should have access to the instance:

Specifically, check if the inbound rules are set properly. In our case, they’re as follows:

Test Empty RDS Instance

Now let's test connecting to the RDS instance using TablePlus:

Click Connect. As you can see, the DB is empty for now:

Backup DigitalOcean DB

Now let’s use Tableplus to make a backup of the DigitalOcean database:

Choose a folder and save the file called defaultdb.dump (it will have the name of your database):

If you see this, it worked:

Restore RDS Database

To restore the database, click Restore.

Select the aws database. Then, select the postgres database and click Start restore:

Select the dump file, in our case defaultdb.dump:

Final Test

As a final test, let's access the RDS database again:

And here we can see our tables and inserts again:

And we’re done!

Video Version: https://youtu.be/3zLWCNn0Vqk

Choosing Between AWS Lambda and Docker

Francesco Ciulla — Thu, 23 Sep 2021 19:10:05 +0000

Article by Jay Allen

One of the great things about AWS is the vast array of features available to software developers. Sadly, one of the most confusing things about AWS is...the vast array of features available to developers!

AWS provides multiple methods for deploying applications into the cloud. Two of these methods - AWS Lambda and Docker - have grown rapidly in popularity over the past several years. In this article, we compare the benefits of each and discuss when you might want to choose one over the other.

AWS Lambda

AWS Lambda is a "serverless" service that enables running code in the cloud. With Lambda, application developers can package code written in a variety of programming languages - including Java, Go, C#, Python, Powershell, Node.js, and Ruby - into a callable function that complies with their language's Lambda interface. They can then upload these Lambda functions to their AWS accounts, where they can be executed from anywhere over the Internet.

The word "serverless" is a bit of a misnomer here; obviously, AWS didn't find some magical way to run code without compute capacity! "Serverless" here means that the compute power used to run this code doesn't run in your AWS account. Rather, it's executed on one of a series of computing clusters run by AWS itself. This frees development teams up to focus on the business logic of their application rather than on managing compute capacity.

Lambda functions can be called, or invoked, through a variety of methods. One of the most common is by connecting your Lambda functions to AWS API Gateway, which exposes them as REST API calls. Lambda functions can also be used to implement customization and back-end processing logic for a large number of AWS services, including Amazon DynamoDB, Amazon Kinesis, and Amazon Simple Queue Service, among others. Lambda functions may also execute as scheduled tasks, and can even be executed directly from the AWS Command Line Interface (CLI) and the AWS Console.

AWS Lambda can be thought of as the original serverless technology on AWS. It wasn't the first serverless technology on the block. That honor may go to Google' App Engine, which has been doing its thing since 2008. (Lambda, first released in 2015, is comparatively a youngin'.) But it helped inspire a boom in the serverless technology industry that continues to this day.

Docker

In the bad ol' days of software deployment, developers threw their code onto clusters of production servers that might all have wildly different configurations. A web application might work for one user and then fail for a second user if the server to which the request was routed lacked a certain shared library or configuration setting.

Docker was created specifically to resolve this nightmare scenario. A Docker container is a unit of software that contains everything - code, dependent libraries, and configuration files - that an application requires to run. The container is then deployed to and run on a virtual machine.

The utility of Docker containers lies in their "run once, run anywhere" nature. Once you test a Docker container and verify that it functions as expected, that same container will run on any system to which you deploy it.

Unlike Lambda, Docker isn't inherently "serverless". Docker is best thought of as a packaging and deployment mechanism. There are multiple ways on AWS to run a Docker container, including:

Elastic Container Service. ECS is AWS's scalable, enterprise-grade solution for running Docker containers. Containers can be deployed either on an Amazon EC2 cluster hosted in your AWS account or using Fargate, AWS's serverless container deployment solution. (For more, check out my recent article on using EC2 clusters vs. Fargate for your Docker deployments.)
Elastic Beanstalk. AWS's "all-in-one" deployment technology will run your Docker container on a Docker-enabled EC2 instance.
As an AWS Lambda Function . Here's where things get really confusing! Yes, you can implement code in a Docker container and expose it via a Lambda function. I'll talk a little about who you might want to do this below.

Microservices, Served Two Ways

Both AWS Lambda and Docker containers are solid choices for deploying microservices architectures on AWS:

Lambda functions map handily to REST API endpoints. You can use Lambda functions in conjunction with AWS API Gateway to quickly build out a REST API complete with advanced features such as user authentication and API throttling.
Docker makes it easy to implement REST APIs using your favorite REST API framework - such as Node.js, Flask, Django, and many others. Because a Docker container is a deployable unit, you can easily partition your REST APIs into logical units and manage them through separate CI/CD pipelines.

Lambda vs. Docker: Who Wins?

But this raises the perennial question: Which one is better?

The first thing to point out is that this isn't necessarily an either/or question. Both Lambda and Docker are powerful technologies that development teams may choose to combine within a single project. For example, you may decide to implement your microservice as a series of Docker containers, and then use Amazon Simple Queue Service in conjunction with AWS Lambda functions to implement a loosely coupled communications framework between services.

But let's set that aside for now and focus on a narrower question: Which technology should you choose when implementing a microservices architecture?

As with most things in the world of the Cloud, there's no clear-cut answer here. But let's look at a few factors you should consider when making this decision for your own project.

Implementation Languages

When it comes to choice of programming languages and frameworks, Docker is the clear winner. AWS Lambda's support for programming languages is limited to the languages for which it defines an integration API. Docker, meanwhile, can host any language or framework that can run on a Dockerized Linux or Windows operating system.

Portability

The language and framework issue leads me to another issue: cloud lock-in. AWS Lambda isn't an industry standard - it's AWS's proprietary serverless tech. If you need to move to a new cloud provider (Azure, GCP) for any reason, your code may require significant rework to function on the new provider's equivalent serverless solution.

By contrast, Docker is pretty much a de facto standard. A Docker container that works on AWS's ECS will also run on Azure App Service, Google Cloud Run, and Kubernetes.

If you still want to leverage Lambda but are concerned about portability, I'd recommend following AWS's recommendations around Lambda code design. You can easily separate your function's execution logic out from the Lambda execution environment. This reduces your dependency on Lambda and makes your code more portable.

Scaling

If your microservice could potentially be called hundreds of thousands of millions of times a day (or even hour), you'll want to ensure it can scale automatically to meet user demand. Fortunately, both AWS Lambda and Docker offer plenty of options to create a highly scalable microservice.

AWS Lambda creates an instance of your function to serve traffic to users . As that instance reaches capacity, Lambda will automatically create new instances of your function to meet demand. Lambda can "burst" from between 500 up to 3,000 instances per region to handle sudden traffix influxes, and can then scale up to 500 new instances every minute.

AWS also provides multiple options for scaling Docker containers. Containers deployed using Fargate, AWS's serverless container deployment solution, can be configured to scale out based on Amazon CloudWatch alarms. If you're deploying Docker containers to an EC2 cluster in your AWS account, you can even scale out the size of your cluster .

Execution Speeds

In general, both AWS Lambda and Docker containers can be configured to provide the performance required by most applications.

However, I'd be remiss if I didn't note the infamous Lambda cold start issue. Remember above how I said that Lambda will create a new instance of your function when it needs to scale out. This process requires time: the Lambda function code has to be downloaded to an EC2 instance in AWS's Lambda server farm, and the execution environment and its associated dependencies also take time to load and start. This is known as a cold start. It has a particularly hard impact on Java and .NET applications, both of which have weighty runtime environments.

Fortunately, as Mike Roberts at Symphonia points out, cold start isn't an issue for high-demand applications. It only becomes a factor in low-execution environments - e.g., when using a Lambda function as a callback from another AWS service, such as CodePipeline.

Application Dependencies

When it comes to dependency management - libraries that your application depends upon - Docker is king. As I discussed earlier, a Docker container is a self-contained package containing everything your application needs to run.

It's also possible to ship dependencies with your AWS Lambda functions as part of the function's ZIP file. However, things get complicated when you need to package OS-native dependencies. Furthermore, Lambda packages max out at 250MB, which can be an issue when packaging large dependency frameworks.

Fortunately, AWS Lambda's support for Docker containers means you can get the best of both worlds. By implementing your functions as Docker containers, you can package any dependency your application requires and ensure it always runs as intended. Docker containers on AWS Lambda can be up to 10GB in size, which is plenty of space for the vast majority of applications.

Long-Running Tasks

If your code is doing some sort of batch processing - processing DynamoDB events, filtering an Amazon Kinesis stream, generating large images, etc. - you'll need to concern yourself with execution times. Lambda functions can only run for up to 15 minutes before the service will time out. By contrast, Docker containers have no built-in limitations on workload runtimes.

Deployment and Management

As I mentioned earlier, Docker provides a simple and easy-to-understand deployment model that enables packaging a single microservice into a single Docker container. This is where AWS Lambda has often been at a disadvantage: since Lambda is a function-based service, it's proven more challenging to manage an entire service or application as a collection of interconnected Lambda functions.

Fortunately, new tools have come out over the past several years to address exactly this problem. AWS's Serverless Application Model (SAM) enables developers to design, develop, and deploy entire serverless apps directly onto AWS using Lambda and CloudFormation. Other tools, such as the open-source project Serverless, aim to create similar zero-infrastructure deployment experiences for serverless applications on AWS and other cloud providers.

Cost

In general, a "serverless" solution is going to cost you more than a non-serverless solution. We at TinyStacks discovered this recently when we moved all of our container workloads from Fargate to our own ECS EC2 clusters, resulting in a cost savings of 40%.

While we haven't done any direct cost comparisons with AWS Lambda, evidence from others suggests that it's one of the least cost-effective solutions going. An analysis this year by Eoin Shanaghy and Steef-Jan Wiggers on InfoQ found that running a workload on AWS Lambda can cost up to 7.5 times more than running the same workload on AWS Fargate with spot capacity. Given that we manage to run our workloads at a 40% discount on EC2 clusters compared to AWS Fargate, this shows you just how pricey Lambda really is.

Our Recommendation

For large-scale microservice workloads, we've found that running Docker containers on our own tightly managed EC2 cluster using ECS to be the ideal solution.

You may get good mileage from using Lambda selectively for smaller-scale workloads. However, we would recommend implementing your code in Docker containers wherever possible - even when Lambda is your preferred deployment mechanism. Docker containers not only port well across cloud providers but can also be used with numerous AWS services. This makes it easy to change your deployment and hosting strategy in response to your company's changing needs.

CRUD API Express with RDS, ECS and Docker

Francesco Ciulla — Fri, 17 Sep 2021 13:24:54 +0000

Video Version

Do you prefer the Video Version?

In this article, we will see how we can connect an ECS instance, based on an image on ECR, to an RDS Postgres instance.

Prerequisites

Docker installed on your machine
AWS account

Definitions

RDS: Relational Database Service. The AWS service for relational databases such as Postgres. (For more on RDS and Postgres, see my previous article.)
ECR: Elastic Container Registry. Stores Docker images directly on AWS (essentially, an alternative to Docker Hub).
ECS: Elastic Container Service. Deploy and run an application based on an image stored on a registry (it works with both Docker Hub and ECR).

Our Steps Today

Create an RDS Postgres instance
Test the instance
Create the ECR repository using the AWS command line interface
Clone the repository
Create the Docker image
Tag the image accordingly to the ECR repository
Push the image to ECR
Create the ECS based on the ECR repository, setting env variables
Final Test

Create the RDS Postgres Instance

Go on the AWS console and search for RDS:

Then click on Create Database:

Let's create a PostgreSQL instance. We’ll use version 12.5-R1 so we can take advantage of AWS’ free tier:

In Settings, input values for the following:

DB instance identifier (the name)
Master user
Master password + Confirm password (choose a reasonably secure password)

For connectivity, you must be sure that the instance is accessible from the outside. Under Public access, select Yes If you have network issues, check your security group’s inbound rules.

Once you’ve finished, click Create database.

Here is a review of our RDS Postgres instance:

Test the Instance

To test if the RDS instance is accessible, we can use the psql command. You can also test with other command-like tools like pgadmin or your local application.

In the command below, replace RDS_INSTANCE_IP with the one you get from the RDS instance summary:

psql --host RDS_INSTANCE_IP --port 5432 --username postgres

Create the ECR Repository Using the Command Line Interface

ECR stands for Elastic Container Registry, and it's the image registry for AWS. Think about it as a place to store and retrieve your Docker images.

In the AWS Console, type ECR on the search bar and click on Elastic Container Registry:

The UI interface looks like this:

This is a good way to check your existing repositories. But to create one, we’ll use the command-line interface.

Get your credentials using the command:

aws sts get-caller-identity

Then use the credentials and the region you prefer. eplace with the region of your choice, and replace with your AWS account ID (you can get it with the commands).

aws ecr get-login-password --region <REGION> | docker login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com

Let's check if the repository has been created by checking the AWS Console:

Nice! Now let's clone and work on the repository.

Clone the Repository

Clone the aws-express-template repository:

git clone https://github.com/tinystacks/aws-docker-templates-express.git

Now, CD into the directory on the command line:

cd aws-docker-templates-express

and open the project with your favorite IDE. If you have Visual Studio Code, you can type:

code .

Check App and Create the Docker Image

If you want to test the project locally, you can install the dependencies (optional - requires npm installed locally):

npm i

To build the projects:

npm run build

npm run start

Before we build the image, let's check the file inside the config folder called postgres.ts.

Here you can define some environment variables to access your database:

PG_HOST: The address of the database. We’ll use the RDS instance address here later.
PG_PORT: The port of the database. The default one for Postgres is 5432.
PG_USER: The default user of the database
PG_PASSWORD: The password for the user of the database.
PG_DATABASE: The database we want to access. Note that a database called postgres is the default for a Postgres instance

To build the image with Docker, use this command:

docker build -t crud-express .

The name doesn't really matter here, as we will retag the local image in order to push it to the ECR repository we will create soon.

Tag the Image to the ECR Repository

To tag the local image in order to push it to the ECR repository, you need to copy the image URI. For example, you can copy it from the Amazon Console’s list of your repositories in ECR:

docker tag crud-express <AWS_ECR_REPO_URI>

Push the Image to ECR

Just use the same tag as before to push the image tagged locally to your ECR repository:

docker push  <AWS_ECR_REPO_URI>

After this, wait a couple of minutes for the push to complete.

Create and ECS Task from the ECR Repository Image

Now comes the interesting part. Since we have both:

an RDS Postgres instance with public access
an image on the ECR registry
we can create an ECS instance based on the ECR image, and connect it to the RDS instance using the RDS instance's URI by supplying the PG_HOST variable to our application.

In the AWS Console, look for ECS:

Let’s use the Console to configure a custom container:

Choose a container name of your choice. Use the ECR URI as your Docker image:

Set the port to 80:

Now a very important step - set the environment variable as follows:

Key :PG_HOST
Value: Your RDS URI so the ECS app can connect to the RDS instance

Next, click on Update:

On Task definition, you can just click Next:

On Define your service, also click Next:

For the cluster, you can choose a name for your cluster and then click Next:

Then you just have to wait a couple of minutes to let AWS create your resources:

Once it's done, click on the task:

Scroll down and copy the Public IP so we can use with with our favorite API tester:

Final Test

To test our application, we will use Postman. First of all, let's check if the app is up and running. Make a GET request at the endpoint AWS_APP_IP:80/ping:

Now let's make a couple of inserts into the database. Make a PUT request with the following body (title and content) at the endpoint AWS_APP_IP:80/postgresql-item:

Let's make another one:

Now, to get all the items, make a GET request at the endpoint AWS_APP_IP:80/postgresql-item:

To get a single item, make the same request appending the id of the item at the end of the url
(note that we are not handling errors properly here - this is for demo purposes):

To update an existing item, you can make a POST request to the endpoint AWS_APP_IP:80/posgresql-item/1, specifying an id and passing a message body:

Let's check that the values were updated:

You can also delete an existing item, making a DELETE request at the endpoint AWS_APP_IP:80/postgresql-item/ID (e.g. 2):

And with that we’ve successfully validated connecting an ECS task to a Amazon RDS database!

Video Version

Do you prefer the Video Version?

ECS: Serverless or Not? Fargate vs. EC2 Clusters

Francesco Ciulla — Fri, 17 Sep 2021 04:42:44 +0000

When it comes to deploying Docker containers on AWS, developers have two choices: Elastic Container Service (ECS) EC2 clusters and Fargate. But which one is right for your application? In this article, I look at the pros and cons of each - and discuss why we recently made a massive change in our own strategy at Tinystacks.

Article By Jay Allen

ECS EC2 Clusters vs. Fargate

Docker containers have become so popular because they're a great way to package an application with all of the files, libraries, and configuration it needs to operate properly. On AWS, ECS provides an easy way to deploy, run, and manage Docker containers at any scale.

If you're unfamiliar with ECS, you'll want to check out the AWS documentation for an overview of key concepts. In brief, ECS enables running Docker images by defining services that are comprised of one or more tasks, with each task being a running instance of a specific Docker container.

Of course, running a Docker container requires having machines to run them on. In ECS, this is abstracted into the idea of an ECS cluster, a logical grouping of services and tasks. Developers have two choices in how to create and manage ECS clusters.

The first choice is by creating an Amazon EC2 cluster. In this scenario, you use an Amazon EC2 virtual machine image to create one or more VMs that are hosted in your AWS account. You can then run tasks across the instances of your cluster.

The second, more recent choice is Fargate. With Fargate, the hardware and virtual machines on which your Docker containers run are managed completely by AWS as a "serverless" service.

It shouldn't come as a surprise that, as totally different services - one server-based, one serverless - Fargate and EC2 clusters use different pricing models. With EC2 clusters, you pay for only the EC2 compute capacity and Elastic Block Storage (EBS) capacity that you use. By contrast, Fargate charges for usage on a per-minute basis, with charges varying based on the amount of virtual CPU (vCPU) and memory your containers use.

Fargate and EC2 clusters are different means to the same end: running your Docker containers in a scalable manner. But each can have advantages over the other, depending on your specific scenario.

The Advantages of Fargate

As with any serverless service, the allure of Fargate comes in ease of management. If you manage your own EC2 clusters, you have to worry about a whole host of operational issues - VM security, operating system patching and maintenance, and uptime. Since Fargate uses capacity managed by AWS, you needn't worry about ensuring EC2 instances remain healthy and secure - AWS does this for you.

Using Fargate can also lead to operational efficiencies. With EC2 clusters, you run two key operational risks: underprovisioning, or not creating enough instances to meet the demands of your workload; and overprovisioning, or overpaying for too much capacity that you end up not using. With Fargate, you only pay for container runtime - never for unused VM capacity.

The Advantages of EC2 Clusters

However, that doesn't mean that Fargate is always the best choice. There are several compelling reasons why you may opt for using EC2 clusters instead.

The key advantage of EC2 clusters is price. While Fargate is easy and convenient, that convenience comes at a cost. Fargate has come under fire from the developer community for being expensive compared to EC2 clusters. Indeed, AWS itself has stated that, the more you can maximize a cluster's vCPU and memory utilization, the more cost-effective EC2 clusters become.

Additionally, EC2 clusters may bring your customers additional peace of mind in terms of security. While AWS works hard to ensure complete isolation of tasks running on Fargate, companies in sensitive industries such as finance and health care may be wary about their workloads running alongside other arbitrary processes.

Our Experience at TinyStacks

At TinyStacks, we work hard to provide an end-to-end deployment experience on AWS that frees development teams to focus on their application code - not on DevOps infrastructure. Since all TinyStacks-enabled applications are deployed as Docker containers running on ECS, we're very keen on optimizing our ECS usage for performance, scalability, and cost.

Initially, we used Fargate clusters exclusively for our DevOps stack deployments. However, after running some numbers, we concluded that shifting to our own EC2 clusters might be more cost-effective. We ran some tests using EC2 clusters with Amazon ECS cluster auto scaling enabled, scaling out our clusters when instances were maxed at over 75% CPU utilization for 5 minutes, and scaling in when they were under that threshold for the same amount of time. We also configured ECS service scaling and ensured it was synchronized with cluster scaling.

What we found was pretty astounding: by maximizing cluster utilization, we were able to reduce our ECS spend with EC2 clusters by 40% when compared with Fargate. The smallest cost savings came with larger instances. An EC2 m5.xlarge with 4 vCPU and 16GiB of RAM came out to $138.24/month compared to a similar-sized Fargate cluster, which came out to around $167.7888/month - an 18% cost difference. But the smallest instance size we used - a t3.nano with 2 vCPU and 0.5GiB RAM - was a mere $3.744/month. Compare that to Fargate's smallest instance type, a .5 vCPU, 1GiB instance, which cost us a full $17.7732/month. That's a full 79% cost savings.

Based on these results, we moved all of our ECS workloads from Fargate onto our own EC2 clusters. All of our customers will now receive the benefits of EC2 cluster hosting for ECS including, not just reduced cost, but increased security and scalability. We believed these numerous advantages made the decision a no-brainer.

In short, Fargate definitely has some advantages in terms of ease of use and maintenance. But in terms of cost, EC2 cluster hosting for ECS is by far the clear winner.

Migrating DigitalOcean Spaces to an Amazon S3 Bucket

Francesco Ciulla — Thu, 09 Sep 2021 15:28:28 +0000

DigitalOcean Spaces provides Amazon S3-compatible object storage with a simplified pricing model. However, you may at some point find that you need to move your storage off of Spaces and onto Amazon S3. In this post, I'll show how to use the tool Rclone to move your data from Spaces to S3 quickly and easily.

Original Article By Jay Allen

Spaces vs. Amazon S3

Built on the object storage system Ceph, Spaces provides a competitive storage alternative to S3. The base Spaces plan charges a flat $5/month for up to 250GiB of storage and up to 1TiB of data transfer out. That can represent a nice cost saving over Amazon S3, where only the first GiB of data transfer to the Internet is free. And since Spaces is fully S3-compatible, SDK code that works with S3 will work with a Spaces account. Spaces even offers a Content Delivery Network (CDN) at no additional cost.

However, there may be times when you need to bring your data in Spaces over to Amazon S3:

You have data security requirements that are well met by features such as AWS PrivateLink for S3
You need your data in more regions than DigitalOcean supports (five regional endpoints as opposed to AWS's 24), or need to store data in a region supported by AWS to comply with data protection laws
You find that transfer from S3 to other AWS features is faster than transfer from Spaces for your scenario

Whatever the reason, it would be great if you could migrate your data over en masse without having to roll your own script.

Moving from Spaces to S3 with Rclone

Fortunately, the Rclone tool makes this easy. Rclone is a self-described Swiss army knife for storage that supports over 40 different cloud storage products and storage protocols.

Let's walk through this to show just how easy it is. For this walkthrough, I've created a Space on DigitalOcean that contains some random binary files.

We'll want to transfer this into an Amazon S3 bucket in our AWS account. I've created the following bucket for this purpose:

Installing AWS CLI and Rclone

Rclone will make use of the AWS CLI. If you don't have it installed, install and configure it with an access key and secret key that has access to your AWS account.

You'll also need to install Rclone. On Linux/Mac/BSD systems, you can run the following command:

curl https://rclone.org/install.sh | sudo bash

Or, you can install using Homebrew:

brew install rclone

On Windows systems, download and install the appropriate executable from the Rclone site. Make sure to add rclone to your system's PATH afterward so that the subsequent commands in this tutorial work.

Obtaining Your Spaces Connection Information

To use Rclone to perform the copy, you'll need to create an rclone.conf file that enables Rclone to connect to both your AWS S3 bucket and to your Spaces space.

If you've set up your AWS CLI, you already have your access key and secret key for AWS. You will need two pieces of information from Spaces:

The URL to the endpoint for your Space; and
An access key and secret key from DigitalOcean provides access to your Space.

Obtaining your Spaces endpoint is easy: just navigate to your Space in DigitalOcean, where you'll see the URL for your Space. The endpoint you'll use is the regional endpoint without the name of your space (the part highlighted in the red rectangle below):

To create an access key and secret key for Spaces, navigate to the API page on DigitalOcean. Underneath the section Spaces access keys, click Generate New Key.

Give your key a name and then click the blue checkmark next to the name field.

Spaces will generate an access key and a secret token for you, both listed under the column Key. (The actual values in the screenshot below have been blurred for security reasons.) Leave this screen as is - you'll be using these values in just a minute.

Configure rclone.conf File and Perform Copy

Now you need to tell Rclone how to connect to each of the services. To do this, create an rclone.conf file in ~/.config/rclone/rclone.conf (Linux/Mac/BSD) or in C:\Users\<username>\AppData\Roaming\rclone\rclone.conf (Windows). The file should use the following format:

[s3]
type = s3
env_auth = false
access_key_id = AWS_ACCESS_KEY
secret_access_key = AWS_SECRET
region = us-west-2
acl = private

[spaces]
type = s3
env_auth = false
access_key_id = SPACES_ACCESS_KEY
secret_access_key = SPACES_SECRET
endpoint = sfo3.digitaloceanspaces.com
acl = private

Replace AWS_ACCESS_KEY and AWS_SECRET with your AWS credentials, and SPACES_ACCESS_KEY and SPACES_SECRET with your Spaces credentials. Also make sure that:

s3.region lists the correct region for the bucket you plan to copy data into;
spaces.endpoint is pointing to the correct Spaces region.

To test your connection to Amazon S3, save this file and, at a command prompt, type:

rclone lsd s3:

If you configured everything correctly, you should see a list of your Amazon S3 buckets.

Next, test your connection to Spaces with the following command:

rclone lsd spaces:

You should see a list of all Spaces you have created in that region.

If everything checks out, go ahead and copy all of your data from Spaces to Amazon S3 using the following command:

rclone sync spaces:jayallentest s3:jayallen-spaces-test

(Make sure to replace the Space name and S3 bucket name with the values appropriate to your accounts.)

The Rclone command line won't give you any direct feedback even if the operation is successful. However, once it returns, you should see all of the data from your Spaces account now located in your Amazon S3 bucket:

And that's it! With just a little setup and configuration, you can now easily transfer data from DigitalOcean Spaces to Amazon S3.