DEV Community: Albiona

The Complete Beginners Guide to AWS S3

Albiona — Mon, 25 Jan 2021 18:18:34 +0000

When developing your web application, you must think of a place where to store your data, how to back up them, types of data you want to store, such as images, music, and videos, application hosting, data archiving, disaster recoveries. AWS Simple Storage Service (S3) provides you the solutions for these cases. S3 is one of the core services of AWS cloud infrastructure. It's object storage that acts like a regular file system on your personal computer. S3 scales infinitely, with no limit on the amount of data you store.

In this tutorial, we'll get to learn how to use the AWS S3. First, learn what is S3, the core parts of S3 that are the Buckets, Access Point, and Objects. Then we'll get to the practice, by implementing the AWS SDK for Node.js 💻

Finally, we'll provide a cheat sheet on AWS S3 CMD Commands.

Amazon Web Services (AWS) provides multiple types of cloud computing services, one of them is the AWS Storage Service. There are different storage services, such as Simple Storage Service (S3), AWS Elastic File System (EFS), & Elastic Block Store (EBS). For this tutorial, we'll be focusing on the S3 service. S3 is one of the most favorite cloud computing services among all the other services, based on this poll by acloud.guru on Twitter.

What is S3?

You can use S3 to host your static websites, delivering HTML, JavaScript, images, videos, and other files to your website visitors - that doesn't contain server-side code such as Node.js or PHP. Using S3, you can easily deploy your applications in just two to three clicks via the user interface. S3 provides a simple web services interface you can use to store and retrieve any amount of data from anywhere on the web.

We'll now go through the core concepts of S3, such as buckets, access points, and objects.

Buckets

To upload your data to S3, you must create an S3 bucket in one of the AWS Regions, within one bucket you can upload many objects to the bucket. For implementation, buckets and objects are resources, and S3 provides APIs for you to manage them. There are different methods you can use to create buckets such as.

Amazon S3 Console

Follow the guides here to create your first bucket with the S3 console.

REST API

To create buckets using REST API, you must authenticate your requests — follow the PUT Bucket in the S3 API reference. But it's recommended to use the AWS Management Console or AWS SDKs instead.

AWS SDK

To create buckets with the SDK, you first have to create a client and then use the client to send a request to create a bucket. Note: When creating the client and the bucket, use the same region. Here is a dominant source on creating and using AWS S3 Buckets.

Access points

To access the data that you store on S3, you need the S3 Access Point. These are endpoints that are attached to buckets that you used to perform S3 object operations.

Each access point has distinct permissions and network controls S3 applies for any request that is made through the access point. Access points are used to perform operations on objects, but not on buckets. Go through this source to learn how to manage data access with S3 access points.

Objects

We mentioned that AWS S3 is object storage. Each AWS S3 object has data, a key, and metadata. The object key (or key name) uniquely identifies the object in a bucket. Object metadata is a set of name-value pairs.

You can store objects in one or more buckets, and each object can be up to 5 TB in size. For the real-world solutions, let's say you want to share an image or video stored in AWS S3 bucket on your website, that is possible only if you make the object public or use a pre-signed URL on your website. Follow this source on how to work with S3 objects.

Hands-on S3 with AWS SDK

We'll go with the AWS SDK and Node.js to create S3 buckets, uploading an object to a specified bucket and deleting that bucket afterward; we'll provide a How-To on the S3 section where you can learn more about different use-cases commands to run on S3.

In order to continue, you must:

Install Node.js, in case you don't have it, follow the Node.js website.
Set up your user credentials, follow here for more information.

Configuring the SDK

To use the AWS SDK for JavaScript, we must first initiate a node module for our hands-on project.

To do that, first, create a folder named webiny-hands-on-s3 and cd into that folder.

Run the command npm init — this will ask you to provide the project's name and you can name it as you want, in this case, we'll leave it the same as the folder name with an entry point of the index.js file.

Inside our folder, we'll create a couple of files, such as.

createBucket.js
upload.js
listObjects.js
deleteBucket.js

Before we continue to implement any code, we need to install the AWS SDK package by running this command: npm install aws-sdk --save.

Let's dive in.

Creating an AWS S3 Bucket

In order to use the SDK, we have to configure the SDK package by loading it into our file. Open the createBucket.js file, and start writing the below code in your own file. In the snippet we have comments to explain to you what we're doing:

// Load the AWS SDK for Node.js
var AWS = require("aws-sdk");

// Set the region
AWS.config.update({ region: "us-east-1" });

// Create S3 service object
s3 = new AWS.S3({ apiVersion: "2006-03-01" });

// Create the parameters for calling createBucket -- with this part we'll take the bucket name we'll create
var bucketParams = {
  Bucket: process.argv[2],
};

// Call S3 to create the buckets
s3.createBucket(bucketParams, function(err, data) {
  err ? console.log("Error", err) : console.log("Success", data.Location);
});

To create your S3 bucket, with this script, we must run it and give our bucket a name like so: node createBucket.js webiny-s3-bucket-testing. If it's successful, it will console log the Success message together with the location, which will be the bucket name.

Uploading a file to an AWS S3 Bucket

Open the upload.js file and let's dive in. I will repeat some parts of the snippets, such as the SDK configuration, and the AWS S3 service object as shown in the above snippet. The additional part is that we have two command-line arguments, the first one will be the bucket name where you'll upload your file, and the second argument will be the file itself.

Let's dive into the code.

// Load the AWS SDK for Node.js
var AWS = require("aws-sdk");

// Set the region
AWS.config.update({ region: "us-east-1" });

// Create S3 service object
s3 = new AWS.S3({ apiVersion: "2006-03-01" });

// Call S3 to retrieve upload file to specified bucket
var uploadParams = { Bucket: process.argv[2], Key: "", Body: "" };
var file = process.argv[3];

// Configure the file stream and obtain the upload parameters
// The node.js file system module allows you to work (read, create, update, delete, rename files)
// with the file system on your computer.
var fs = require("fs");
var readingFile = fs.createReadStream(file);
readingFile.on("error", function(err) {
  console.log("File Error", err);
});

uploadParams.Body = readingFile;
// The path module provides utilities for working with file and directory paths.
// We can access by using this:
var path = require("path");
uploadParams.Key = path.basename(file);

// Call S3 to retrieve upload file to specified bucket
s3.upload(uploadParams, function(err, data) {
  err ? console.log("Error", err) : console.log("Upload Success!", data.Location);
});

Now, create an index.txt file in your folder, and add some text to it.

Then, run the script by providing two parameters:

The S3 bucket we create in the first snippet webiny-s3-bucket-testing
The local file you want to upload into that S3 bucket.

The full command node upload.js webiny-s3-bucket-testing index.txt

🎉 You just uploaded a file into your S3 bucket!

Listing objects in an AWS S3 Bucket

Now, open the listObjects.js file where you'll list the content of this bucket. It's again a repetitive task of configuring the SDK and creating the AWS S3 service object. What this script will do, is that we'll provide the bucket name from which we want to read the objects and the result will be a list of objects (files) or a failure message.

Let's dive into the code.

// Load the AWS SDK for Node.js
var AWS = require("aws-sdk");

// Set the region
AWS.config.update({ region: "us-east-1" });

// Create S3 service object
s3 = new AWS.S3({ apiVersion: "2006-03-01" });

// Create the parameters for calling listObjects method
var bucketParams = {
  // in here we'll provide the bucket name we created earlier
  Bucket: "webiny-s3-bucket-testing",
};

// Call S3 to obtain a list of the objects in the bucket

s3.listObjects(bucketParams, function(err, data) {
  err ? console.log("Error", err) : console.log("Success", data);
});

Now, let's run the script by running this command: node listObjects.js Check out the result 😄

Deleting an AWS S3 Bucket

Move to the deleteBucket.js file, and configure the SDK and create the AWS S3 service object. What you'll use in this script, comparing with the above script, is the deleteBucket method. But, this case is different, we previously added objects into our bucket, right? We can't delete the AWS S3 buckets if they're not empty. That means you need to delete the objects inside the bucket first, then delete the bucket.

Let's dive in.

// Load the AWS SDK for Node.js
var AWS = require("aws-sdk");

// Set the region
AWS.config.update({ region: "us-east-1" });

// Create S3 service object
s3 = new AWS.S3({ apiVersion: "2006-03-01" });

// Create params for S3.deleteBucket
var bucketParams = {
  // here you'll provide the name of the bucket you want to delete
  Bucket: "webiny-s3-bucket-testing",
};

// We'll first empty the bucket
async function emptyS3Bucket(bucket) {
  const listParams = {
    Bucket: bucket,
    // Prefix: dir,
  };

  const listedObjects = await s3.listObjectsV2(listParams).promise();

  if (listedObjects.Contents.length === 0) return;

  const deleteParams = {
    Bucket: bucket,
    Delete: { Objects: [] },
  };

  listedObjects.Contents.forEach(({ Key }) => {
    deleteParams.Delete.Objects.push({ Key });
  });

  await s3.deleteObjects(deleteParams).promise();

  if (listedObjects.IsTruncated) await emptyS3Bucket(bucket);
}

emptyS3Directory(bucketParams.Bucket);

// Call S3 to delete the bucket
s3.deleteBucket(bucketParams, function(err, data) {
  err ? console.log("Error", err) : console.log("Success", data);
});

`How-to` on S3 - Cheatsheet

Download an entire AWS S3 bucket

1.1 Install the AWS Command Line Tools

sudo easy_install awscli
// or
sudo pip install awscli
// or
brew install awscli

1.2 Run these commands:

aws s3 sync s3://<source_bucket> <local_destination>

Example: aws s3 sync s3://mybucket. Will download all the objects in mybucket to the current directory. And will output: download: s3://mybucket/test.txt to test.txt

Are AWS S3 buckets region-specific?

The user interface shows all your buckets, in all regions. But buckets exist in a specific region and you need to specify that region when you create a bucket.

How to Configure SSL for AWS S3 bucket?

Example:[https://s3.amazonaws.com/bucket_name/images/logo.gif](https://s3.amazonaws.com/bucket_name/images/logo.gif)

If you use a custom domain for your bucket, you can use S3 and CloudFront together with your own SSL certificate (or generate a free one via Amazon Certificate Manager): Amazon CloudFront Custom SSL

Delete AWS S3 buckets

aws s3 rb s3://bucket-name

By default, the bucket must be empty for the operation to succeed. To remove a bucket that's not empty, you need to include the --force option.

aws s3 rb s3://bucket-name --force

Rename AWS S3 Bucket name

There is no rename bucket functionality for S3 because there are technically no folders in S3, so we have to handle every file within the bucket.

aws s3 mb s3://[new-bucket] // 1. Create a new bucket
aws s3 sync s3://[old-bucket] s3://[new-bucket] // 2. Copy files over
aws s3 rb --force s3://[old-bucket] // 3. Delete the old bucket

Quick way to list all files in AWS S3 bucket

aws s3 ls

AWS S3 copy files and folders between two buckets

aws s3 sync s3://DOC-EXAMPLE-BUCKET-SOURCE s3://DOC-EXAMPLE-BUCKET-TARGET

Is it better to have multiple s3 buckets or one bucket with sub folders?
By default, you can create up to 100 buckets in each of your AWS accounts. If you need additional buckets, you can increase your bucket limit by submitting a service limit increase. Source.
The total volume of data and the number of objects you can store are unlimited. Source.

*Object Storage — Also known as object-based storage, is a strategy that manages and manipulates data storage as distinct units, called objects. There are three key components of an object — the content of the object (data stored in the object such as a file or directory), the unique object identifier (ID), and metadata. It stores the metadata as key-pair values and contains information such as name, size, date, security attributes, content type, and URL. Each object has an access control list (ACL) to configure who may access the object.

Now that you've used AWS SDK for S3, you're able to code the solutions that the AWS S3 Console provides via a few clicks, which is faster but, using the SDK you'll be able to continue developing your applications using the AWS services directly by coding. This is a significant advantage for those interested in building applications using AWS services. In this tutorial, we used the AWS SDK to create buckets, upload data, listing data from the buckets, empty, and afterward deleting buckets via AWS SDK for JavaScript for Node.js.

If you learned something new today and are interested to follow up on our blogs, subscribe to our newsletter and we'll provide you the best content of the serverless world!

Thanks for reading! My name is Albiona and I work as a developer relations engineer at Webiny. I enjoy learning new tech and building communities around them = ) If you have questions or just want to say hi, reach out to me via Twitter.

Get Started with Cloud Computing

Albiona — Tue, 19 Jan 2021 14:21:04 +0000

Nowadays, when working on software systems, you have heard about Cloud Computing technology. Understanding cloud fundamentals is an essential part when getting started with Cloud Computing. Amazon Web Services (AWS) is one of the cloud providers that offer over 300 services. Before starting with AWS, understand cloud concepts such as what is cloud computing, the advantages of using cloud computing, cloud computing types, and cloud deployment models.

In this article, we'll cover the concepts you need to build that solid foundation to implement Cloud Computing technologies.

What is Cloud Computing

Cloud computing gives you the option to transform IT infrastructure into a utility - which means you can get into your IT infrastructure via the internet, using computing resources on the cloud. Without the need to install or maintain these resources on-premises.

What is on-premise you might ask? Going with the on-premise way, you must manage your own servers, hire IT people, payor rent the real-estate for the servers.

The opposite of on-premise is the cloud providers, who provide you these services via the internet. The cloud providers are companies who own the servers, hire IT people, pay and manage the real estate for the servers. Those companies give you the option to manage the configuration of the cloud services you use, and your code and they take care of the rest.

Simply put, Cloud Computing offers you on-demand access, through the internet to cloud computing resources, applications, servers (physical and virtual servers), data storage, development tools, networking capabilities, and more — hosted at remote data centers that are managed by cloud service providers.

Advantages of Using Cloud Computing

We learned what on-premise IT is, the traditional way of managing IT infrastructure. Based on the differences between on-premise and cloud computing provider solutions, there are some obvious benefits you get when using cloud computing resources.

Lower total cost of ownership (TCO)

For all the attributes of having your on-premise infrastructure that we mentioned such as managing your own servers, hiring the IT people, paying or renting the real-estate for the servers, those all come up with a price, and with the cloud computing technology, you eliminate the capital expenses.

Agility

In just a few clicks you're able to get up and running the computing resources you need for your applications — As Webiny's CEO Sven noted on the Serverless Chats podcast by answering Jeremy's question "What are the benefits of serverless for both small and big companies?" - "When using serverless, the cost of managing infrastructure will go way down, by releasing a big chunk of the budget or resources or working hours, that you can now focus on product iterations, so your product can grow faster. And if your product grows faster, you can out-innovate potential competitors, which can't afford that same level of innovation."

Scale and Performance

The two most important benefits you get when switching to cloud computing technologies is when you want your customers to have an amazing user experience using your product, anywhere - that means in the lowest or highest spikes of your product's traffic. Cloud computing services run on a worldwide network, so they're able to provide you the ability to scale and deliver the resources when they're needed.

Types of Cloud Computing Services

The cloud computing services fall into four different categories: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Serverless, and Software as a Service (SaaS). They are all built up on top of one another. We'll go through them to understand what they are and the differences between them.

Infrastructure as a Service (IaaS)

IaaS represent the building blocks for cloud IT by providing access to networking features, computers, and data storage space. You can use those resources to build your own products. These are offered by different cloud provider companies on a pay-as-you-go basis. Some examples of cloud providers are Amazon Web Services (AWS), Google Cloud Platform, and Microsoft Azure.

— Intended for Admins

Platform as a Service (PaaS)

I'm assuming you are a software developer? Well, this cloud computing category is designed for you. PaaS offers cloud computing services with an on-demand environment for development, testing, delivering, and managing software applications. This will make it easier for developers to create web or mobile applications, without worrying about the infrastructure, storage, network, and databases needed for development.

Some examples of PaaS are the AWS Lambda Service, Heroku, etc.

— Intended for Developers

Serverless Computing

I'll mention the definition of serverless computing based on Webiny's guide to serverless ⬇️

"Serverless means there are no servers you have to operate to run a particular service or an app. Usually, in Serverless architectures, you rely on services that are managed for you. Be that function-as-a-service, storage-as-a-service, and others. Using managed services means all the maintenance and operational tasks are abstracted away. Yes, there are still servers there, but they are no longer your responsibility to manage."

Some of the examples that offer serverless services are AWS Lambda Service, Cloudflare Workers, Google Cloud Functions, IBM Cloud Functions, Knative, Microsoft Azure Functions, Oracle Functions. When it comes to building serverless applications and APIs on top of these services, Webiny takes the lead! Webiny is a serverless framework — a complete plugin-based admin interface and a set of ready-made serverless applications. Learn more about Webiny features here.

Software as a Service (SaaS)

SaaS takes place when cloud providers offer complete software products. The infrastructure, product maintenance, security its managed by the cloud provider. Some examples of this cloud computing category are Salesforce, Gmail, Office 336.

— Intended for Customers

Cloud Deployment Models

Public Cloud

With this type of deployment model, you'll have access to anything the cloud provider offers, from SaaS applications, virtual machines (VMs), infrastructure, and development platforms — available to users via the public internet. Different cloud providers offer these services in different pricing models such as subscription-based, or pay-per-use.

One of the known characteristics of the Cloud is the multi-tenant* environment—the data centers infrastructure is shared by all public cloud customers. This leads to one or more disadvantages of the Cloud such as limited resources → Even though they have incredible computing power, they share their resources with multiple tenants.

Some examples of companies using the cloud deployment models are Squarespace, Basecamp, and Dropbox.

Multi-tenant → When a single instance of a software application serves multiple tenants (or user accounts).

Tenant → Can be an individual user, but more often is a group of users that share common access and privileges within that instance of a software application.

On-Premise || Private Cloud

The on-premise cloud refers to the cloud computing resources being used by one organization or customer → sometimes is called a private cloud. One reason is the security that the on-premise offers for sensitive data for customers such as Government, Hospitals, Enterprises with different regulations. The on-premise cloud is hosted in the customer's data center, but it has the option to combine the hosting on an independent cloud provider's infrastructure.

Hybrid Cloud

A combination of the public cloud and on-premise → connecting an organization's on-premise cloud services and public clouds into a single infrastructure for all the organization's applications.

The goal is to have an optimal solution for each of the organization's applications to do the workloads between the two — based on the current requirements.

Using a Hybrid cloud gives organizations the freedom to meet their business and technical objectives with benefits such as lower costs and efficiency rather than using just public or on-premise cloud.

Now that you know the cloud computing concepts, I think you are ready to start with some examples of cloud computing services or you can follow the blog on 5 AWS services you need for web development!

If you learned something new today and are interested to follow up on our blogs, subscribe to our newsletter and we'll provide you the best content of the serverless world!

5 AWS Services you need to know for Web Development

Albiona — Wed, 13 Jan 2021 13:41:49 +0000

If you're getting started with AWS and you're interested to build your web applications using AWS services in a short amount of time, then you're in the right place. AWS offers more than 300 services, and to start developing applications with AWS without the right information might be a huge burden. As for today, we'll provide to you only 5 AWS services you need to know, to develop your web application.

Without further ado, let's get started.

Amazon Web Services

Our top 5 picks of AWS services are:

AWS S3

AWS API Gateway

AWS Lambda

AWS DynamoDB

AWS CloudFront

Extra AWS Service - AWS Cognito

For each service, we will go through what is the service, what are the benefits of using one, the pricing, and how to get started.

Follow the article here to get into the details of the services you need for web development!

Now that you have the tools you need, building your web applications with AWS is an amazing experience considering the benefits it offers for developers. And if you want to go one step further, try Webiny — Webiny is an open-source serverless application framework that brings all those tools together. Start building your serverless applications using Webiny. today 🚀

AWS re:Invent - Webiny's Top 5 Picks on Serverless Announcements

Albiona — Mon, 21 Dec 2020 18:29:17 +0000

Today we'll present to you Webiny's top 5 picks on serverless announcements. We've been active during the past three weeks on live announcements of AWS re:Invent, focusing on the serverless news that can affect the way we develop software solutions from the cost and performance perspective.

Without further ado, let's go through the top 5 picks from our team @ Webiny ⬇️

1. AWS Lambda

1.1. Lambda changes billing to per ms

Lambda now will bill your functions by the ms, as opposed to a round of 100ms.

"For example, a function that runs in 30ms on average used to be billed for 100ms. Now, it will be billed for 30ms resulting in a 70% drop in its duration spend."

For more details and information about pricing check out the article here.

1.2. Lambda now supports 10 GB memory and 6 vCPU cores for Lambda Functions

Lambda increased the memory limit by 3x compared to the previous limit, from 3,008 MB up to 10,240 MB (10 GB) 🚀

This helps to perform memory-intensive operations at scale.

Starting today you can configure between 128 MB and 10.140 MB of memory for new or existing Lambda functions.

For more details check out the article here.

1.3. Lambda supports container images as a packaging format

Package the Lambda functions and deploy as a container image up to 10 GB. Now you have the chance to build and use Lambda based applications using container tooling, workflows, and dependencies.

Some advantages of Lambda packaging:

Operational simplicity
Automatic scaling
High availability
Native integrations with 140 AWS services
"With this launch, AWS provides a set of base images for Lambda that are available on ECR Public and Docker Hub."

For more details check out the article here.

1.4. Lambda Runtime Logs API

You can now subscribe to log streams directly from within the Lambda execution environment.

"After receiving the subscription request, the Lambda service streams logs to the extension, and the extension can then process, filter, and send them to any preferred destination."

This extension supersedes the CloudWatch Logs, making it easier for you to use your preferred extensions for diagnostics.

For more details on how to get started with the extension, follow the article here.

2. AWS Aurora Serverless v2

Extending the features of Aurora Serverless, this is one of the biggest announcements on the first week of AWS re:Invent.

One of the key features: "You pay only for the capacity your application consumes, and you can save up to 90% of your database cost compared to the cost of provisioning capacity for peak load."

Amazon Aurora Serverless v2 also provides the full breadth of Amazon Aurora’s capabilities:

Multi-AZ support,
Global Database, and
Read replicas

For more details, check out the in-house AWS article here, or follow the Jeremy Daly's preview on "Aurora Serverless v2: The Good, the Better, and the Possibly Amazing"

3. AWS Proton

"Automated management for container and serverless deployments"

AWS Proton is the first application to fully manage and deploy container and serverless applications.

With AWS Proton, engineering teams can connect and coordinate all the different tools needed for:

Infrastructure Provisioning,
Code Deployments
Monitoring, and
Updates

To learn more about what AWS Proton solves on the software systems complexities, check out the article here.

4. AWS S3

4.1. S3 Replication adds support for multiple destinations

You can replicate data from one source bucket to multiple destination buckets at the same or different AWS regions.

This is intended for you if you're interested to maintain multiple copies of your data in one or more AWS Regions.

"With S3 Replication (multi-destination), you can easily create a shared dataset by replicating data to multiple buckets in the same or different AWS Regions."

For more details on S3 features and the pricing page, check out the article here.

4.2. S3 adds read-after-write consistency automatically for applications

S3 delivers strong read-after-write consistency for any storage request. With the strong consistency, S3 is removing the need to make changes to apps and reduces costs by removing the need for extra infrastructure which provided strong consistency.

This blog post guides you on more details on react-after-write consistency.

4.3. S3 supports two-way replication for object metadata changes

If you want to build shared datasets across multiple regions and keep all object and object metadata changes in sync then two-way replication is important.

Learn more about the S3 Replication here.

5. DynamoDB supports export to S3

You can now export your Amazon DynamoDB table data to your data lake in AWS S3 where you can use different services such as Athena.

Your DynamoDB data added in your AWS S3 data is easily discovered, encrypted at rest, and in transit.

With just a few clicks in the AWS Management Console and a simple API call, you can export DynamoDB tables ranging from a few megabytes to hundreds of terabytes of data.

Learn more about this announcement here.

These were Webiny's top 5 picks on Serverless Announcements @ AWS re:Invent. If you're interested in our future blog posts, subscribe to our newsletter and you'll be notified when we have interesting topics to share!

What is IDaaS and Why Use One?

Albiona — Tue, 15 Dec 2020 18:04:33 +0000

If you're reading this article, you might be interested to learn how to implement user management for your application, understanding the complete picture of how identity management services work. That's what we'll do in this article. You'll learn what is Identity as a Service (IDaaS), why to use such a service, what is Single Sign-On (SSO), and IDaaS platforms.

Let's dive in!

What is an Identity Provider as a Service (IDaaS)?

Manual user identity management

In order to get access to an application, back in the days, you would have to share your credentials such as your username and password with the application maintainers.

For a developer, user identity management is a complex feature to build and manage, especially security-wise. You'd be responsible if someone would gain access (hack) into the users' identities and use those for their purposes.

Let's go through a real-life scenario where your client has requested to implement the Login/Registration feature for the application you're building.

Inspired by the dev.to platform, as seen in the image below, your client wants to give the users an option to access the application using social accounts such as Twitter or GitHub.

Follow the article here to read the below titles 🚀

Open Authentication and OpenID Connect protocols

Identity as a Service provider (IDaaS)

How does IDaaS actually work?

What is SSO?

The key benefits of SSO Login

Why should you use an IDaaS provider?

Customer Benefits of Using an IDaaS

Business Benefits of Using an IDaaS

What are the most popular IDaaS providers?

Now that you clearly understand Identity Providers and what they offer, you'll be able to decide if you want to use such a service and focus on building business value or continue using and maintaining a custom authentication solution. If you're interested in our future blog posts, subscribe to our newsletter and you'll be notified when we have interesting topics to share!

Thanks for reading! My name is Albiona and I work as a developer relations engineer at Webiny. I enjoy learning new tech and building communities around them = ) If you have questions or just want to say hi, reach out to me via Twitter.

Build an E-commerce Website with Webiny Serverless Headless CMS, Next.js, and Stripe

Albiona — Mon, 16 Nov 2020 15:53:50 +0000

In this tutorial, we will create a simple e-commerce website, where you can buy Swag from the best Open Source projects such as Webiny, Next.js, React, etc.

Before we continue, let's go through what you'll learn about building this website.

You will learn how to create the back-end using Webiny Headless CMS and set up two content models, the products, and the categories. Afterward, we'll fetch the data from the Headless CMS to the Next.js project using Apollo GraphQL. Last but not least, we'll integrate Stripe to implement a checkout experience.

First, let's take a look at the diagram of what we'll build ⬇️

In this tutorial, you'll go through the sections below:

1. The e-commerce-starter

2. Webiny Headless CMS

3. Next.js + Apollo GraphQL to fetch data from the backend

4. Next.js + Stripe to create the payment intents

Follow the tutorial here 🚀

Thanks for reading! My name is Albiona and I work as a developer relations engineer at Webiny. I enjoy learning new tech and building communities around them = ) If you have any questions, comments, or just wanna say hi, feel free to reach out to me via Twitter.

Build a Portfolio Website with React, Webiny, and Apollo

Albiona — Tue, 10 Nov 2020 14:54:52 +0000

In this tutorial, you will set up a simple portfolio website to showcase your projects and your blogs using pure React, Webiny Headless CMS, and Apollo GraphQL. After setting up your portfolio website, you will be able to focus all of your creativity on the content for blogs and projects.

These are the features of the Portfolio Website:

2 Content types: Blogs, Projects
6 Created blogs
6 Created projects
Apollo Client to fetch the content models data
Responsive design using Material UI React Components
Front-end deployment with Vercel

Demo

Here is what we'll build:

Or, you can Watch the live demo.

Prerequisites

A Webiny Project

First of all, make sure you follow the prerequisites to create a Webiny project.

Follow the tutorial here 🚀

Creating a blog with Webiny Headless CMS + Gatsby

Albiona — Tue, 10 Nov 2020 14:03:38 +0000

In this tutorial we will learn how to use Gatsby with the Webiny Headless CMS.

All of the code shown in this tutorial is also hosted in our GitHub repository.

Prerequisites

1. Gatsby-cli

Install gatsby-cli using the command below. We will use it to create our Gatsby Blog.

npm install -g gatsby-cli

2. A Webiny Project

Make sure you have a working Webiny project set up.

3. Content Delivery API URL

The Headless CMS app exposes data via the Content Delivery API, which is a simple GraphQL API that dynamically updates its schema on content model changes that you make. Once you've deployed your API stack (using the yarn webiny deploy api command), you should be able to find the Content Delivery API URL in the console output:

4. Content Delivery API Access Token

In order to access the data via the Content Delivery API, we'll need a valid Access Token. These can be created via the Access Tokens form, which you can reach via the main menu:

Create a new token and make sure to copy the actual token string. We'll need it soon.

Creating our first content model

Now that we have all of the prerequisites out of the way, it's time to create our first content model. Let's open the Models section of the Headless CMS app.

Let's create a new content model named Blog Post. Click on the "plus" icon in the lower right corner of the screen and in the dialog that's about to be shown, enter the following:

For the content model group, we'll use the Ungrouped, which is the default group that comes out of the box with every Headless CMS app installation.

Content model groups give you a way to organize the content models inside the main menu, allowing you to build logical sections for your content editors. You can click here to learn more.

Once we've submitted the form in the dialog, we should be redirected to the Content Model Editor. Let's add two fields: title as a Text, and body as a Rich Text field.

INFO: Rich Text field returns a slate object, which is not a valid React element. So, you need to parse that data to render it. You can implement your own parser or use a package created by one of our community members: webiny-richtext-serializer. For usage example, please see this sandbox.

They will match every blog post's title and body (content), respectively.

Save the changes by clicking on the Save button in the top right corner of the screen.

Now it's time to create the actual content. Proceed by clicking on the View content button, located on the left side of the Save button.

You can also reach the content area by clicking on the newly added Blog Post item in the main menu:

Managing Content

As mentioned, navigate to Headless CMS > Ungrouped > Blog Post and create a blog post or two. Feel free to unleash your creativity. 😉

Once you feel happy with the blog post, you can save the changes by clicking the Save button, located at the bottom of the form.

The next and final step is to publish the blog post, which will make it actually visible in the Content Delivery API. To do that, click on the Publish icon, found at the right side in the form header.

Now that we've covered the basics of creating content models and managing content, we can move on to the Gatsby part of this tutorial.

Creating a Gatsby Blog

We can create a new Gatsby app by running gatsby new myBlog command.

This will generate a new folder in your working directory.

Ideally, you should create your Gatsby project in a folder outside of the Webiny project.

Now that we have a new Gatsby app ready to go, let's see what it takes to make a simple page that renders a list of all blog posts that we have just created.

Pulling GraphQL data (Blog Posts) into Gatsby

We will navigate to our myBlog folder created earlier and install NPM package gatsby-source-graphql. This will allow us to fetch the Blog Posts into our Gatsby app.

yarn add gatsby-source-graphql

We will add this plugin in the plugins array found in gatsby-config.js (located in the root of our Gatsby project) and configure its options like below. Most importantly, we will replace CONTENT_DELIVERY_API_URL with our API's URL and <CONTENT_DELIVERY_TOKEN> with the token you created earlier (eg: d3b45980479...)...

{
  resolve: `gatsby-source-graphql`,
  options: {
    typeName: "someTypeName",
    fieldName: "webinyHeadlessCms",
    url: "<CONTENT_DELIVERY_API_URL>/cms/read/production",
    headers: {
      authorization: "<CONTENT_DELIVERY>"
    }
  },
}

We are using the read API in order to pull Blog Post data and the production alias which points to the production environment because that is where we published our content earlier.

Once we have these ready, we can jump to the code. The following snippet shows the code located in the pages/index.js file:

import React from "react"
import { graphql } from 'gatsby'
import Layout from "../components/layout"

// Implement a custom parser or use one from the community: `webiny-richtext-serializer`
function parseRichTextDataAsHTML(data) { ... }

// The IndexPage component that renders our blog posts
const IndexPage = ({data}) => {
  // GraphQL queried data is automatically inserted into the `data` parameter used below
  const blogPosts = data.webinyHeadlessCms.listBlogPosts.data

  // We render a nice list of blog posts
  const BlogPosts = blogPosts.map(post => (
    <div key={`post-${post.id}`}>
      <h1>{post.title}</h1>
      <p
        style={{whiteSpace: "pre-wrap"}}
        dangerouslySetInnerHTML={{ __html: parseRichTextDataAsHTML(post.body) }}
      />
    </div>
  ))

  return (
    {/* We use Gatsby's Layout to make our Blog look nice */}
    <Layout>
      {BlogPosts}
    </Layout>
  )
}

export default IndexPage

// A GraphQL query that fetches our blogs' data
export const query = graphql`{
  webinyHeadlessCms {
    listBlogPosts {
      data {
        id
        createdOn
        title
        body
      }
    }
  }
}`

Previewing the page

Let's run yarn dev in our Gatsby project directory so we can see our page in action:

Note: In order to fetch the latest CMS data and update your content you will need to stop your Gatsby app and run gatsby develop again. If you are in production mode, you need to run gatsby build as well as gatsby serve.

Conclusion

Congratulations! 🎉

We have successfully created a simple page that displays a list of all created blog posts, all powered by Webiny Headless CMS and Gatsby.

The same can also be achieved with other popular tools, like Next.js. To learn more, click here.

FYI: The blog was originally written and published by Ashutosh

Lighting fast search with Elasticsearch

Albiona — Mon, 09 Nov 2020 17:53:33 +0000

If you're reading this blog chances are you really interested in Elasticsearch and the solutions that it provides. This blog will introduce you to Elasticsearch and explain how to get started with implementing a fast search for your app in less than 10 minutes. Of course, we're not going to code up a full-blown production-ready search solution here. But, the below-mentioned concepts will help you get up to speed quickly. So, without further ado, let's start!

What is Elasticsearch?

Elasticsearch is a distributed search and analytics engine. It provides near real-time search and analytics for all types of data. Whether you have structured or unstructured text, numerical data, or geospatial data. One of the key specialties of Elasticsearch is that it can efficiently store and index it in a way that supports fast searches. You can go far beyond simple data retrieval and aggregate information to discover trends and patterns in your data.

Why do you need it?

Elasticsearch is fast. Because Elasticsearch is built on top of Lucene, it excels at full-text search. Elasticsearch is also a near real-time search platform, meaning the latency from the time a document is indexed until it becomes searchable is very short — typically one second. As a result, Elasticsearch is well suited for time-sensitive use cases such as security analytics and infrastructure monitoring.

Elasticsearch is distributed by nature. The documents stored in Elasticsearch are distributed across different containers known as shards, which are duplicated to provide redundant copies of the data in case of hardware failure. The distributed nature of Elasticsearch allows it to scale out to hundreds (or even thousands) of servers and handle petabytes of data.

The speed and scalability of Elasticsearch and its ability to index many types of content mean that it can be used for a number of use cases:

Application search
Website search
Enterprise search
Logging and log analytics And many more...

We at Webiny are building a feature for the upcoming v5 release where we'll use Elasticsearch to perform a super-fast search in our core apps like Page builder, File manager, and Headless CMS. Please check out our Github repo to learn more about it.

Getting started with Elasticsearch

Setup Elasticsearch cluster

You can create a hosted deployment or set up an Elasticsearch cluster on your local machine. For the purpose of this blog, we'll assume that we have an Elasticsearch cluster running at localhost:9200. If you want to go with a local setup please check out this guide.

Setup Elasticsearch Node.js client

We'll be going to use the official Node.js client for Elasticsearch. You can create a new Node.js project or use this example project.

To install the latest version of the client, run the following command:

npm install @elastic/elasticsearch

Using the client is straightforward, it supports all the public APIs of Elasticsearch, and every method exposes the same signature.

Configure the client

The client is designed to be easily configured for your needs. In the example mentioned below, you can see how easy it is to configure it with basic options.

const { Client } = require("@elastic/elasticsearch");

const client = new Client({
  // The Elasticsearch endpoint to use.
  node: "http://localhost:9200",
  // Max number of retries for each request.
  maxRetries: 5,
  // Max request timeout in milliseconds for each request.
  requestTimeout: 60000,
});

NOTE: As we previously mentioned we're assuming that we have an Elasticsearch cluster running locally on localhost:9200

Elasticsearch in action

Before jumping into the core topic of this blog i.e search, we'll need to create the index and add few documents to it.

Create an index

Let's create an index inside our Elasticsearch cluster.

You can use the create index API to add a new index to an Elasticsearch cluster. When creating an index, you can specify the following:

Settings for the index (optional)
Mappings for fields in the index (optional)
Index aliases (optional)

await client.indices.create({
  // Name of the index you wish to create.
  index: "products",
});

We'll be using a dynamic mapping that's why didn't add the settings and mappings in the body here. But, if needed we could have something like this:

await client.indices.create({
  // Name of the index you wish to create.
  index: "products",
  // If you want to add "settings" & "mappings"
  body: {
    settings: {
      number_of_shards: 1,
    },
    mappings: {
      properties: {
        field1: { type: "text" },
      },
    },
  },
});

Index documents

Now that we have created the product index, let's add a few documents so that we can perform a search on those later. There are basically two ways you can do this depending upon the use case.

Index a single document.
Index multiple documents in bulk.

We'll cover both of these use cases in a moment.

Index a single document

Here we're going to use the create method on the client that we created earlier. Let's take a look at the code:

await client.create({
  // Unique identifier for the document.
  // To automatically generate a document ID omit this parameter.
  id: 1,
  type: "doc",
  // The name of the index.
  index: "products",
  body: {
    id: 1,
    name: "iPhone 12",
    price: 699,
    description: "\"Blast past fast\","
  },
});

We can index a new JSON document with the _doc or _create resource. Using _create guarantees that the document is only indexed if it does not already exist. To update an existing document, you must use the _doc resource.

Index multiple documents at once

This is all good. But, sometimes we want to index multiple documents at once. For example, in our case wouldn't it be better if we can index all brand new iPhones at once? Right? We can use the bulk method for this exact use case. Let's take a look at the code:

const dataset = [
  {
    id: 2,
    name: "iPhone 12 mini",
    description: "\"Blast past fast.\","
    price: 599,
  },
  {
    id: 3,
    name: "iPhone 12 Pro",
    description: "\"It's a leap year.\","
    price: 999,
  },
  {
    id: 4,
    name: "iPhone 12 Pro max",
    description: "\"It's a leap year.\","
    price: 1199,
  },
];

const body = dataset.flatMap(doc => [{ index: { _index: "products" } }, doc]);

const { body: bulkResponse } = await client.bulk({ refresh: true, body });

if (bulkResponse.errors) {
  const erroredDocuments = [];
  // The items array has the same order of the dataset we just indexed.
  // The presence of the `error` key indicates that the operation
  // that we did for the document has failed.
  bulkResponse.items.forEach((action, i) => {
    const operation = Object.keys(action)[0];
    if (action[operation].error) {
      erroredDocuments.push({
        // If the status is 429 it means that you can retry the document,
        // otherwise it's very likely a mapping error, and you should
        // fix the document before to try it again.
        status: action[operation].status,
        error: action[operation].error,
        operation: body[i * 2],
        document: body[i * 2 + 1],
      });
    }
  });
  // Do something useful with it.
  console.log(erroredDocuments);
}

The bulk method provides a way to perform multiple indexes, create, delete, and update actions in a single request. Here we are using the index action but you can use the other actions as per your needs.

TIP: bulk performs multiple indexing or delete operations in a single API call. This reduces overhead and can greatly increase indexing speed.

Update an existing document

We often need to update an existing document. We'll use the update method for the same.

It enables you to script document updates. The script can update, delete, or skip modifying the document. To increment the price, you can call the update method with the following script:

await client.update({
  // The name of the index.
  index: "products",
  // Document ID.
  id: -1,
  body: {
    script: {
      source: "ctx._source.price += params.price_diff",
      params: {
        price_diff: 99,
      },
    },
  },
});

The update API also supports passing a partial document, which is merged into the existing document. Let's use it to update the description of the product with id = -1:

await client.update({
  // The name of the index.
  index: "products",
  // Document ID.
  id: -1,
  body: {
    doc: {
      description: "\"Fast enough!\","
    },
  },
});

Delete an existing document

It's a no-brainer that we also need to remove existing documents at some point.

We'll use the delete method to remove a document from an index. For that, we must specify the index name and document ID. Let's take a look at an example:

await client.delete({
  // The name of the index.
  index: "products",
  // Document ID.
  id: -1,
});

The Search

The search API allows us to execute a search query and get back search hits that match the query.

Let's start with a simple query.

// Let's search!
const { body } = await client.search({
  // The name of the index.
  index: "products",
  body: {
    // Defines the search definition using the Query DSL.
    query: {
      match: {
        description: "\"blast\","
      },
    },
  },
});

This query will return all the documents whose description field matches with "blast"

INFO: Learn more about Query DSL here.

Nice and simple right. But, that's not all! We can go for even more specific queries. Let's look at some examples:

Search for exact text like the name of a product

// Let's search for products with the name "iPhone 12 Pro" !
const { body } = await client.search({
    // The name of the index.
  index: "products",
  body: {
        // Defines the search definition using the Query DSL.
    query: {
      term: {
        title.keyword: {
                    value: "iPhone 12 Pro"
                }
      }
    }
  }
});

Search for a range of values like products between a price range

// Let's search for products ranging between 500 and 1000!
const { body } = await client.search({
  // The name of the index.
  index: "products",
  body: {
    // Defines the search definition using the Query DSL.
    query: {
      range: {
        price: {
          gte: 500,
          lte: 1000,
        },
      },
    },
  },
});

Search using multiple conditions

// Let's search for products that are either ranging between 500 and 1000
// or description matching "stunning"
const { body } = await client.search({
  // The name of the index.
  index: "products",
  body: {
    // Defines the search definition using the Query DSL.
    query: {
      // Return result for which this nested condition is TRUE.
      bool: {
        // Acts like an OR operator.
        // Returns TRUE even if one of these conditions is met
        should: [
          {
            range: {
              price: {
                gte: 500,
                lte: 1000,
              },
            },
          },
          {
            match: {
              description: "\"stunning\","
            },
          },
        ],
      },
    },
  },
});

If you need a search query where all the conditions must be matched then you should use the must operator inside the bool It acts like an AND operator and returns TRUE only if all the conditions met. Inside bool, there are also other operators must_not and should_not that you can use as per your needs.

These are just a few examples of search queries, you can perform even more specific and power search queries.

INFO: Learn more about search using Query DSL here.

Sort search results

Elasticsearch allows us to add one or more sorts of specific fields. Each sort can be reversed as well. The sort is defined on a per-field level, with a special field name for _score to sort by score, and _doc to sort by index order.

The order defaults to "desc" when sorting on the _score and defaults to "asc" when sorting on anything else.

Let's take a look at the following example:

// Let's sort the search results!
const { body } = await client.search({
  // The name of the index.
  index: "products",
  body: {
    // Defines the search definition using the Query DSL.
    query: {
      bool: {
        // Acts like an AND operator.
        // Returns TRUE only if all of these conditions are met.
        must: [
          {
            range: {
              price: {
                gte: 500,
                lte: 1100,
              },
            },
          },
          {
            match: {
              name: "iPhone",
            },
          },
        ],
      },
    },
    // Sort the search result by "price"
    sort: [
      {
        price: {
          order: "asc",
        },
      },
    ],
  },
});

Here we've sorted the search result by price in "asc" order.

Paginate search results

Pagination is a must-have feature for every decent real-world app. And Elasticsearch helps us with this as well. Let's see how? 🙂

By default, the search method returns the top 10 matching documents.

To paginate through a larger set of results, you can use the search API’s size and from parameters. The size parameter is the number of matching documents to return. The from parameter is a zero-indexed offset from the beginning of the complete result set that indicates the document you want to start with.

For example, the following search method call sets the from offset to 15, meaning the request offsets, or skips, the first fifteen matching documents.

The size parameter is 15, meaning the request can return up to 15 documents, starting at the offset.

// Let's paginate the search results!
const { body } = await client.search({
  // The name of the index.
  index: "products",
  body: {
    // Starting offset (default: 0)
    from: 15,
    // Number of hits to return (default: 10)
    size: 15,
    // Defines the search definition using the Query DSL.
    query: {
      match: {
        description: "\"blast\","
      },
    },
  },
});

Conclusion

If you're looking to implement a fast search mechanism for your app or website. I would recommend you to consider Elasticsearch as a solution to that.

And if you're interested in building full-stack serverless web applications I would highly recommend you try out Webiny The Easiest Way To Adopt Serverless. We've Elasticsearch along with DynamoDB baked-in for super-fast search in our core apps like Page builder, File manager, and Headless CMS.

I hope this blog will help you in your web development journey, but, of course, if you have any further questions, concerns, or ideas, feel free to ping me 💬 over Twitter or even directly via our community Slack.

Thanks for reading this blog! My name is Ashutosh, and I work as a full-stack developer at Webiny. If you have any questions, comments, or just wanna say hi, feel free to reach out to me via Twitter. You can also subscribe 🍿 to our YouTube channel where we post knowledge sharing every week.

FYI: The blog was originally written and published by Ashutosh

What is Commercial Open Source Software

Albiona — Fri, 06 Nov 2020 10:24:32 +0000

Open source evolution

Open source is the backbone of almost all digital technologies today. Without open-source, most of the modern tech companies that you rely on in your day to day life wouldn't be able to operate or more precisely, wouldn't become what they are in the first place.

What started as a selection of smaller projects back in the 70s created by programming enthusiasts to oppose the unfair profits of closed-source software, has evolved to become the main industry driver of software development in the past decade.

This "recent" acceleration in the number of open-source companies being launched, frequently called The Open Source Renaissance, has been fueled by various circumstances on the software industry market. But one of the main reasons behind this has been the significant innovation around various business models that allowed the open-source companies to commercialize their products while keeping the open-source approach to software development.

Open source has always been, and still is, the best possible way to create new software as it can gather whole communities of motivated, like-minded software developers that can through their contribution not just help speed up the development process, but also help steer it in a direction that would be the most beneficial to the global developer community.

What is Commercial Open Source Software?

The elephant in the (meeting) room of the majority of the open-source project founders has been one common, basic question:

How can we build a business around a product that we are committed to giving away to the community for free?

What started with Red Hat and their paid support and services model over 20 years ago has opened the door for open source communities to start exploring the various ways to monetize their products - which created a new wave of companies that we can group under the name of Commercial Open Source Software.

This commercialization has not only allowed these companies to become sustainable but has also created this perfect symbiosis between the open-source and its proprietary license counterpart that has become a credible new way of funding further technology development and innovation.

COSS business models

The majority of COSS companies have built their business model in a way that allows them to capture part of the value that is delivered by their open-source core product, by offering commercial versions of their products, alternative license options, or add-on proprietary software. While their success still fundamentally depends on their open-source core technology.

The opportunity to commercialize open-source software has increased the appeal of OSS to investors, who in this modern-day also play a fundamental role in the development stage of open source start-ups by providing the crucial funding needed to form a dedicated full-time team to manage the project more effectively, grow the project up to a higher scale, and achieving the product-market fit.

There are numerous different ways the open-source companies have successfully approached the commercialization of their products, but we can group them under five main groups:

Support services
Hosting
Restrictive licensing
Open core
Hybrid licensing

All five of these groups provide a different set of benefits and challenges that fit different types of open source products, and all have found their purpose amongst the COSS companies. But in the past ten years, one of these groups has really been standing out vs. others. The approach referred to as the Open Core model has proven to be the most frequent business model used by highly successful open source companies, that have managed to grow their business to the 100+ million dollar levels.*

COSS company index

The concept of the open core business model is that the majority of the code base is open source - the core - while the smaller part, usually aimed at enterprise clients, is made proprietary - the crust. This allows the COSS company to stay true to the open-source movement and support the developer community while being able to capture a small portion of the delivered value that will fuel the whole project and unlock scale and longevity. The most important task of the company that decides to take the open core approach is to find the right balance between the core and the crust. One that will not restrict the value and performance of the open-source part, while still providing enough incremental value to its paid users that will justify their investment.

Webiny approach to COSS

At Webiny, we have made a deep analysis of all the available and potential business models and have also found the open core to be the best way forward for our product. And we have put a massive focus on making sure we find the perfect balance point between the two parts.

Our commitment to our community is that we will always keep access to our Webiny open-source framework and apps completely free of charge. And rather than separating a part of the code and turning it into proprietary software, we have decided to build a completely separate product that will perfectly complement our Webiny OS by providing additional features that will further improve the Webiny OS usage for larger teams and multi-project circumstances. And most importantly, not taking any value away from our open-source users.

This new product has recently been announced on our product roadmap under the name of the Webiny Control Panel. You can find more details about it here: Webiny product roadmap

The Future

The software has completely changed the world as we know it and is part of almost every aspect of modern life. It is hard to predict what the world will look like in the future and how technology will further develop it, but it is absolutely safe to say that software innovation will be one of the main drivers of that evolution.

The constant and rapid software innovation is unlocking new levels of what is possible every day. And to circle back to what I mentioned at the very beginning - there is no better way to develop new software than by joining the strengths of the global community of developers around an open-source project. Especially when these projects can be turned into credible businesses by implementing modern COSS business models.

References:

COSS company index, by COSS Media
Open Source: From Community to Commercialization, by Peter Levine & Jennifer Li / Andreessen Horowitz

Thanks for reading!

FYI: The blog was originally written and published by Mislav Streicher

Local Development Is Dying 💀 - And It's for the Better

Albiona — Thu, 05 Nov 2020 17:01:11 +0000

Serverless infrastructure is changing the way we architect applications and it’s also changing the way we code and work as developers. The development is moving away from our laptop and into the cloud. It’s the start of cloud-native development.

We at Webiny have been practicing cloud-native development for over a year now, and we believe this is the future. In this article, I will explain why.

A local development environment (LDE) is a way of configuring services on our laptop/desktop to run a website or a web application. This entails installing a web server, a database, and some sort of a language as Node.js, Python, PHP, and others.

This combination we call a stack. Some of the popular stacks are LAMP (linux-apache-mysql-php), MEAN (mongodb-express-angular-node) and others. Today you find these stacks preconfigured as Docker containers. This way you don’t “pollute” your machine with services as they run inside a container.

Advantages and Disadvantages of a Local Development Environment

Advantage: work offline

For years installing these stacks was the first thing a developer would do before checking out some code from git and running it. A local stack has several advantages. One of the main ones is that you can work offline. This is possible because all the services run locally on your machine, there is no need to fetch any data from “outside” the network.

Advantage: simplicity of debugging

Another meaningful advantage is the simplicity of debugging. If your code, database, or network is not working, you access those components directly and try and find the problem. You attach debuggers, inspectors, and profilers along the way to gain additional insight.

Advantage: no need to upload files

Lastly, with a local development environment, each time you change a file there is no need to upload it to a remote server. The moment you save the code it gets compiled locally and you instantly see the result. It provides a faster turnaround time.

Let’s review some of the disadvantages because it’s not all ideal when it comes to having an LDE.

Disadvantage: can be complex to setup

When working in a larger organization the LDE can be large and complex. These environments take a lot of time to get them up and running. When they break, you often need to run the whole setup from the ground up.

Disadvantage: resource-intensive

You are also limited to a fixed number of containers and services you are able to run locally. This is because of the constraints to your CPU and memory. Try having 10 or more Docker containers running in parallel, it will significantly impede the performance of other apps, like your IDE for example.

Disadvantage: "Works on my pc"

Because not everyone has an identical setup, there are differences in how things work. This causes inconsistencies in testing, security, and performance. Those can later cause serious problems. When using an LDE with a team, you need to ensure everyone has the same stack, version and is always updating it with the latest releases.

Disadvantage: reinventing the wheel

Another limitation is you can’t use a closed source or 3rd party services. This is because, in most cases, they don’t provide a way of running their service locally. To keep an LDE, with all the previously mentioned advantages, you might decide not to use a 3rd party service but develop something internally you own. Over time you’ll make several of such decisions that end up wasting a big chunk of your time maintaining them.

This last one is the key reason why a local development environment is dying. — Let me explain.

Introducing Serverless

Development is complex. Over time some of those complexities were solved and solutions were created as new products. Products that all other developers can now use and integrate into their applications. By using those products and services, instead of reinventing the wheel, developers save a lot of time and money and create more resilient and secure solutions.

Let’s take a search engine for an eCommerce website as an example. Coding one is a daunting challenge. You might not have the right expertise in data storage and know the most optimal languages to code such a feature. Luckily services such as Algolia have been created. By leveraging a ready-made service you are certain that it will work, that there are no bugs, and that it can handle the load from your users (in most cases at least).

When you test for resiliency and scale even simple tasks, such as file storage, can pose a challenge to code from the ground up. Try and achieve the 99.999999999% uptime that AWS S3 provides.

Many such examples are available, but the main takeaway is those services are not available inside an LDE. Serverless services in the majority of cases can only be consumed via an API contacting the service in the cloud.

The question is now — would you sacrifice having an LDE over a proven, performant, and secure service with SLAs?

Many developers and businesses have answered NO. This is what’s driving the adoption of building solutions in a serverless way.

With serverless, there are servers behind those services, but you don’t care about them. The infrastructure is managed for you, including all the security patches and backups.

Cloud-Native Development

Developing applications in the serverless way changes how we configure our LDE. We can no longer work offline and run services from our laptop.

Even if we get a part of the services running locally, it’s not a solution. LDE can’t emulate the network of a cloud setup, not to talk about API limits and performance.

Because the LDE is no longer reliable, in my view, the only way forward is to embrace cloud-native development. Our code needs to run in the cloud and the cloud is now part of our development environment and not just the production. The only remaining thing which is running “locally” will be our editor.

It’s clear that LDE is dying, and we won’t miss it. As a tradeoff, we get to create performant, scalable, resilient, reliable, and secure applications in less time. The industry is moving in that direction. There is still a way to go to make it a pleasant experience but it is certainly coming.

Thanks for reading!

FYI: The blog was originally written and published by Sven Al Hamad

5 Tips to Make Your Lambda Functions Run Faster (and Cheaper)

Albiona — Tue, 20 Oct 2020 10:42:35 +0000

The AWS Lambda service allows us to easily deploy and run our own code, without worrying too much about the underlying infrastructure (when compared to non-serverless technologies). It essentially scales infinitely (with great power comes great responsibility), and can be connected with a bunch of other services, like API Gateway, S3, AppSync, DynamoDB, etc.

And usually, the thing people first start creating with the service are good-old HTTP APIs, like for example REST or even GraphQL. In those situations, since the actual users (potential customers) are the ones who will be invoking your Lambda functions, it's important that they are responding as fast as possible - meaning, we want to have function cold starts as short as possible, and afterward, make our code execute necessary logic in the most efficient way.

How to ensure that is the case? Well, that is the topic of this article, in which we'll cover five tips that can help you in that regard. So, without further ado, let's take a look!

1. More RAM = faster execution = same price

Allocating more RAM to a function means faster execution. That's true. But it also means you pay more, right? Well, it depends. Sometimes that's actually not true.

Consider these two 512MB RAM and 1024MB RAM Lambda function CloudWatch logs. The billed durations from the logs are also shown in the following chart:

Comparing function invocation duration - 512MB RAM vs 1024MB RAM.

So, what we can see here is that with the 512MB of RAM Lambda function (blue), the billed duration is most of the times 200ms. But, with 1024MB of RAM (red), which is 2x more, the billed duration gets reduced to 100ms, which is 2x less. Even the initial invocation's duration (the cold start one's) got reduced from 1400ms to 700ms.

Effectively, this means we're getting faster functions, for the same price!

But do note, the results may vary depending on the task the function is performing. For example, in some cases, you might not achieve the reduction big enough for the price reduction to happen. In other words, if you manage to reduce the invocation duration from 140ms to 105ms, this is good, but still, doesn't change the price, since the billed duration on both invocations is 200ms.

Before bumping up RAM, do test your function with different payloads, and then based on the results, determine if there are any actions worth taking.

2. Watch out for function size to reduce the cold start durations

The larger the function in size, the longer the cold start. There is a very nice article written by Mikhail Shilkov, that talks about various factors that impact the duration of cold starts, one of which is the package size. Definitely give it a read if you really want to get into the nitty-gritty.

Based on that fact, the advice I wanted to give here is - watch out for function size. Try to use as few external packages as possible. Be aware that every package you include in your function can also bring additional dependencies with it, making it even worse.

With that, also be careful about how you actually import packages. If there is a way to import just a specific functionality, do that, instead of importing the entire package.

For example, when using the AWS SDK, which we often need to communicate with various AWS services, only import the actual clients you'll be using.

Do not import the whole AWS SDK like so:

import { CloudFront, Lambda } from "aws-sdk";

Instead, do it like this:

import CloudFront from "aws-sdk/clients/cloudfront";
import Lambda from "aws-sdk/clients/lambda";

This approach can significantly affect the final bundle size, so again, watch out!

A couple of useful tools

There are a couple of useful tools I like to use in order to inspect what the 3rd party package is bringing to the table (or should I say - removing from the table).

One of the tools is the (bundlephobia), which will give you an overview of the package size, with all of its dependencies. Definitely useful, especially when comparing similar packages functionality-wise, and deciding which to use.

The other one is the Webpack Visualizer, which gives you an awesome overview of all the packages that are included in your final production Webpack bundle. For example:

3rd party dependencies (node_modules) adding 2MB to the overall function size!

Wait, Webpack bundle?

Yes! Bundling your functions with Webpack is also recommended, since not only that'll make your function a single file, but will also make sure only the code that's actually utilized gets into the build, which naturally reduces the final file size.

Configuring Webpack could be a bit daunting task, but it's definitely worth it at the end of the day. Luckily, with every new Webiny project, the bundling process is pre-configured for you, so you can immediately jump to your new project! 😊

3. Split complex processes into separate functions to save money and gain speed

Some of you might already know that calling another Lambda function within a Lambda function is often considered as an anti-pattern, and yes, this is definitely true in certain cases. The main reason is that, while the other function is processing the invocation payload, the first function is idle and just waiting for the response, meaning you're effectively paying for nothing.

Yes, you can also invoke functions asynchronously, without waiting for the invocation to complete (using the InvocationType: "Event" option), but, here we're talking more about synchronous function invocation, which is definitely needed in case we're interested in the actual response that the invoked function is returning.

Let's consider the following example, where I believe calling another Lambda function from the first one (and waiting for the response) can be considered as a good approach.

To reduce overall cost, use separate 2GB RAM "SSR Lambda" for intensive SSR generation.

The diagram above is showing the SSR implementation we did at Webiny, which consists of a couple of AWS resources: CloudFront, API Gateway, S3, two Lambda functions, and a database.

I won't go too much into the details here (feel free to check the Serverless Side Rendering — The Ultimate Guide article for that, but, what we are doing with the two Lambda functions is essentially splitting work and optimizing for cost. The Web Server Lambda is configured with minimal system resources (256MB RAM), because its only task is to return cached SSR HTML, stored in the database.

But, if there is no SSR HTML in the database, we need to generate it, which is actually an intensive process, and executing it with 256MB of RAM simply won't cut it. So, instead of just bumping the Web Server Lambda's RAM, we are invoking the SSR Lambda function, which contains a lot more system resources (for example - 2 GB RAM), and we're only paying for that when we actually need it. It would simply be a waste of money to have the Web Server Lambda configured with the same amount of resources, just to pull some data out of the database, most of the time. This way we can save a significant amount of money in the long run.

You can also do this if you have a Lambda function that's heavy in size, just because of a 3rd dependency that needs to be there, but still, isn't utilized that often.

For example, one of the apps that Webiny offers out of the box, the Page Builder app, has its own installation process, in which we basically download initial sample pages and images, and respectively, insert them into the database, and save them to user's S3 bucket. This process requires a couple of NPM packages for dealing with files, which can easily add a couple of hundreds of KBs to the overall function size.

Since that's the case, we decided to extract the installation functionality completely from the main Page Builder API function, and place it into a separate Page Builder installation function.

Extract dependency-heavy and rarely-triggered logic into a separate function.

With this organization, we are not burdening the main Page Builder API function with redundant packages, which helps in reducing its size, and naturally, affects the duration of cold starts.

To conclude, whenever you have an intensive process, that either needs more system resources or requires additional packages that increase the overall function size, try to extract it into a separate function, especially if the process is rarely triggered.

4. When possible, execute code in parallel

The following tip is a short one, yet effective one, and can be applied not only when talking about Lambda functions, but coding in general.

If possible, execute code in parallel, not in series. For example, if we needed to do five 100ms operations, doing it in series means it would take a total of 500ms to complete.

const operations = [a, b, c, d, e];
for (let i = 0; i < operations.length; i++) {
  await operations[i](); // 100ms operation.
}

Instead of doing it in series, try doing it in parallel, using the Promise.all method, like so:

const operations = [a, b, c, d, e];
const promises = [];
for (let i = 0; i < operations.length; i++) {
  promises.push(operations[i]());
}

await Promise.all(promises); // 100ms operation.

Doing this will decrease the function invocation duration and naturally, reduce the overall cost!

Nice trick!🙂

5. Reusing connections with Keep-Alive

Last but not least, if you're making HTTP requests within your Lambda function's code, it's useful to enable the "keep-alive" option on the default Node.js HTTP/HTTPS agent, because that avoids establishing the TCP connection on every request, and enables us to always reuse the existing one:

By default, the default Node.js HTTP/HTTPS agent creates a new TCP connection for every new request. To avoid the cost of establishing a new connection, you can reuse an existing connection.

This can be especially handy for clients like DynamoDB, where we really need the HTTP request latencies to be as low as possible.

A good article on the actual performance boost that this option introduces was already written by the great Yan Cui, in his article - Lambda optimization tip - enable HTTP keep-alive. As we can learn, without HTTP keep-alive, the duration of DynamoDB operations averaged around 33ms, while with the option enabled, the average duration dropped to 10ms!

The somehow shocking fact about establishing new TCP connections is that the actual establishment takes more time than the actual client operation we're trying to execute! So when you think about it, deciding whether this option should be enabled or not is almost a no-brainer.

Conclusion

By almost completely abstracting the infrastructure concerns from us and with that, reducing the overall development and maintenance cost, the AWS Lambda (and serverless technologies in general) truly does make our developer lives easier. But as we've seen, there are still a couple of tricks that are useful to know and which can help us in optimizing our workflows even further.

I hope the five tips we've shown here will help you in your serverless journey, but, of course, if you have any further questions, concerns, or ideas, feel free to ping me over Twitter or even directly via our community Slack.

Thanks for reading!

FYI: The blog was originally written and published by Adrian Smijulj

DEV Community: Albiona

The Complete Beginners Guide to AWS S3

What is S3?

Buckets

Access points

Objects

Hands-on S3 with AWS SDK

Configuring the SDK

Creating an AWS S3 Bucket

Uploading a file to an AWS S3 Bucket

Listing objects in an AWS S3 Bucket

Deleting an AWS S3 Bucket

How-to on S3 - Cheatsheet

Get Started with Cloud Computing

What is Cloud Computing

Advantages of Using Cloud Computing

Lower total cost of ownership (TCO)

Agility

Scale and Performance

Types of Cloud Computing Services

Infrastructure as a Service (IaaS)

Platform as a Service (PaaS)

Serverless Computing

Software as a Service (SaaS)

Cloud Deployment Models

Public Cloud

On-Premise || Private Cloud

Hybrid Cloud

5 AWS Services you need to know for Web Development

Amazon Web Services

AWS S3

AWS API Gateway

AWS Lambda

AWS DynamoDB

AWS CloudFront

Extra AWS Service - AWS Cognito

Follow the article here to get into the details of the services you need for web development!

AWS re:Invent - Webiny's Top 5 Picks on Serverless Announcements

1. AWS Lambda

1.1. Lambda changes billing to per ms

1.2. Lambda now supports 10 GB memory and 6 vCPU cores for Lambda Functions

1.3. Lambda supports container images as a packaging format

1.4. Lambda Runtime Logs API

2. AWS Aurora Serverless v2

3. AWS Proton

4. AWS S3

4.1. S3 Replication adds support for multiple destinations

4.2. S3 adds read-after-write consistency automatically for applications

4.3. S3 supports two-way replication for object metadata changes

5. DynamoDB supports export to S3

What is IDaaS and Why Use One?

What is an Identity Provider as a Service (IDaaS)?

Manual user identity management

Follow the article here to read the below titles 🚀

Open Authentication and OpenID Connect protocols

Identity as a Service provider (IDaaS)

How does IDaaS actually work?

What is SSO?

The key benefits of SSO Login

Why should you use an IDaaS provider?

Customer Benefits of Using an IDaaS

Business Benefits of Using an IDaaS

What are the most popular IDaaS providers?

Build an E-commerce Website with Webiny Serverless Headless CMS, Next.js, and Stripe

1. The e-commerce-starter

2. Webiny Headless CMS

3. Next.js + Apollo GraphQL to fetch data from the backend

4. Next.js + Stripe to create the payment intents

Follow the tutorial here 🚀

Build a Portfolio Website with React, Webiny, and Apollo

Demo

Prerequisites

Follow the tutorial here 🚀

Creating a blog with Webiny Headless CMS + Gatsby

Prerequisites

1. Gatsby-cli

2. A Webiny Project

3. Content Delivery API URL

4. Content Delivery API Access Token

Creating our first content model

`How-to` on S3 - Cheatsheet