DEV Community: Renato Nabinger

The ideal VOD Platform build for high-volume Streaming Data

Renato Nabinger — Wed, 24 May 2023 12:48:58 +0000

One of the best ways to streamline your video streaming is by using a combination of Serverless architecture, S3 and CloudFront.

This innovative solution uses a serverless architecture that enables you to run code, integrate applications, and manage data without needing to worry about managing servers. Thanks to its automatic scaling and built-in high availability features, this approach is both more agile and more cost-effective, with a pay-as-you-go billing model that ensures you only pay for what you use.

By eliminating the need for infrastructure management tasks like patching and capacity provisioning, this serverless approach allows you to concentrate on creating code that best serves your customers.

The AWS Lambda event-driven compute service is the starting point for all serverless applications, and it is natively integrated with over 200 AWS services and SaaS applications.
The app can be built in Next.js with the help of serverless next component, can be deployed in AWS with near to zero configuration. All static routes in the application will be pre-rendered and distributed on CloudFront to maximize speed.

This will result is a dynamic application that can run near the speed of a static site because the dynamic parts are being distributed by the CDN as well.

To authenticate users for the web application, this particular solution uses AWS Cognito's User Pools, which control access to the GraphQL API, generating temporary AWS credentials with limited privileges for accessing S3 content.

AWS AppSync, which is based on the GraphQL API, allows client applications to perform CRUD operations on videos and store them in an S3 bucket.

To accomplish all of this, two separate S3 buckets are used: one for storing raw video and another for storing video with different output formats.

Lambda functions are employed for a range of tasks, including triggering MediaConvert jobs, storing metadata in DynamoDB, creating signed URL tokens, and sending notifications when MediaConvert jobs are complete.

MediaConvert, meanwhile, automatically transcodes the uploaded video to HLS format for multiple bitrates and device types, while Secrets Manager securely stores CloudFront's private keys for signing CloudFront URLs. CloudFront itself delivers content securely via CDN Edge locations to authorized users through the application at runtime.

Architecture benefits

The architecture of this solution has numerous benefits, including scalability through the CloudFront CDN, secure video content to prevent unauthorized access, automated transcoding of video content via MediaConvert, support for multiple devices using adaptive bitrate streaming, and secure storage and retrieval of secret keys using Secrets Manager. The solution can be quickly built and deployed using Amplify and CloudFormation, allowing you to get up and running with your video streaming service in no time.

Application testing, the good, the bad and the ugly: from a senior programmer's point of view

Renato Nabinger — Wed, 24 May 2023 12:43:31 +0000

What framework do you use for testing?

This is a commonly asked question, but for me, it's a superficial and potentially hazardous inquiry.

Unit tests alone cannot guarantee that a particular feature will function correctly in production.

How many times I saw code being committed to production, just to witness the system collapse moments after deployment. 🔥

In fact, that is so common that I once prohibited any new releases from being made on Fridays or overnight.

The truth is, a component may work flawlessly in a test environment, but there are other crucial factors to consider such as its performance, safety, target devices, and compatibility with production dependencies.

To illustrate, let's say a junior developer (not you! 😅) creates a function that grows exponentially with the number of requests, an O(2ⁿ) of the worst kind.

This function could pass a unit test despite its suboptimal performance and could automatically make its way into production with an automated CI/CD pipeline.

To prove my point, I'll provide an example below.

If you prefer, you can skip the math and go straight to the result, or simply take my word for it.

You can Skip the math part and go straight to the conclusion

++++++++++++++++++++++++++++++++++++++++++++++++++

Calculate the nth term of a Fibonacci sequence

The Fibonacci sequence, has the numeral 1 as the first and second term of the order, and the following elements are originated by the sum of its two predecessors, so:

1
1 + 1 = 2
1 + 2 = 3
3 + 2 = 5

And so on…

1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181...

In this solution, we use a function that already returns 1 if n is less than 2 (because we already know that the first two terms of a Fibonacci sequence are 1) then we go backwards calculating recursively

function fib(n) {
  if (n <= 2) return 1
  return fib(n - 1) + fib(n - 2)
}

If we run this function in Jest, it will return as correct.

fib(5) must be 5

fib(39) should be 63245986

fib(40) must be 102334155

“There are no failing tests, congratulations!”

Click here to preview in CodeSandbox

The test may not reveal that, while the function produces the expected output, it also leads to an exponential increase in the required computations to solve the problem.

Simplifying this in BIG-O notation, we get an O(2ⁿ) complexity, which is terrible.

If we run fib(n) we get the following:

in fib(5) the calculation time was 0 ms

in fib(39) the time passed to 489 ms

in fib(40) the time passed to 779 ms

Notice that it didn't take much for the running time to double (from 39 to 40).

end of the boring math part
++++++++++++++++++++++++++++++++++++++++++++++++++

And what to do then to minimize errors in production?

It depends!

What do you mean??? 🤯

The testing approach to use depends on various factors, such as the availability of time/money, the complexity of the component, and the risks involved.

Common sense and experience play a significant role in deciding the level of testing required.

For instance, if the project is small and low-risk, there is no need to perform complex tests.

However, if the project involves high-risk factors like money, security, or high volume of requests, it is advisable to use a comprehensive testing arsenal.

Some strategies to reduce risks in production include:

Run performance tests on every commit. By doing this, you decrease the risk of performance regressions early on. The earlier you detect it, the easier it will be to revert.
Do canary tests whenever the functionality has a high impact or risk for the user or the application. Testing the application within a smaller, controlled audience will eliminate a lot of risk without taking away external variables. Netflix, for example, uses its employees' personal accounts to test functionality changes before releasing them to the rest of the public.
Use external error tracking tools. I always recommend Sentry (https://sentry.io/) as it is excellent at detecting errors in real time. Whenever an error happens to a user, you receive a complete report of what happened, the type of browser used and many other insights that will help you to isolate and resolve the bug.
Evaluate not only the inputs and outputs of your components. Make an analysis of the algorithms (I'm a fan of Big O Notation 💘) and apply, whenever possible, software design patterns, the famous design patterns.

Before signing off, I want to share an O(n) solution that is more performant than the example presented initially.

Cheers!

Renato

Bonus:

Correcting the example function through memoization

function fib(n, memo = []) {
  if (memo[n] !== undefined) return memo[n]
  if (n <= 2) return 1
  var res = fib(n - 1, memo) + fib(n - 2, memo)
  memo[n] = res
  return res
}

in fib(5) the calculation time was 0 ms

in fib(39) the calculation time was 0 ms

in fib(40) the calculation time was 0 ms

Answer on CodeSandbox

Fintech goes micro, an aws serverless microservices case study

Renato Nabinger — Wed, 24 May 2023 12:29:23 +0000

"Crie seu banco" is a Brazilian Fintech start-up that focuses on financial services. Their CEO, Julio Cesar, aims to develop the most comprehensive "bank as a service" solution in the market, with an intelligent white label platform that is modular, flexible, and scalable.

Many Fintechs that operate with Banking as a Service use monolithic technologies based on Laravel, which can cause issues with flexibility, modularity, and scalability. This structure requires reproducing a series of steps for each new customer to launch the solution, which creates a nightmare in terms of infrastructure and updates. Implementing a new customer can take up to six months, and each customer requires their own team to perform DevOps tasks, customization, and maintenance, making the product quite expensive.

We asked to create a solution that address those issues. So we create "Sbank," a serverless microservices architecture that minimizes the time and cost required for infrastructure management. we implemented multiple AWS services, including Amazon DynamoDB for data persistence and Amazon API Gateway for API endpoints, with AWS Lambda automating all processes from end to end.

With "Sbank," a person can now submit an online form to request a new account, and an API triggers the account-creation process. By building it on the AWS serverless platform, we achieved the goals of saving time, reducing costs, and strengthening security.
The process of acquiring new clients has gone from taking many months to just one day, reducing the cost of provisioning each account from $8000 to just shy of $500 per new client. The serverless architecture also keeps other costs down, with operational costs barely exceeding $100 a month for all the functions that support "Sbank" across all three AWS Regions. Security is now stronger, too, with "Sbank" ensuring uniform configurations and controls and providing visibility into account security profiles.

Going serverless microservices has relieved the team from managing servers and lets them concentrate on building features, creating real value in terms of time, cost, agility, and innovation.

Using the Iac strategy

Infrastructure as Code (IaC) is the managing and provisioning of infrastructure through code instead of through manual processes. With AWS CDK, we were able to manage infrastructure deployment, destruction, and configuration. Giving us consistent, automated dev, test, and staging environments.

Some Sbank features

Authentication

Sbank allows users to sign in using an email and require the user to upload a document containing his or her photo. We used the AWS Amplify Framework to integrate our Front-End application with Amazon S3 and store this image in a secure and encrypted bucket. Our solution trigger a Lambda function for each new image uploaded to this bucket so that we can index the images inside Amazon Rekognition and save the metadata in a DynamoDB table for later queries.

For authentication, this solution uses Amazon Cognito User Pools combined with Lambda functions to customize the authentication flows together with the Amazon Rekognition CompareFaces API to identify the confidence level between user photos provided during Sign Up and Sign In. Here is the architecture of the solution:

Identity Verification

In-person user identity verification is slow to scale, costly, and high friction for users. Machine learning powered facial biometrics can enable online user identity verification. We used Amazon Rekognition because it offers pre-trained facial recognition and analysis capabilities that we could quickly add to your user onboarding and authentication workflows to verify opted-in users' identity online.

This feature allow Sbank to onboard and authenticate users in seconds while detecting fraudulent or duplicate accounts. As a result, you can grow users faster, reduce fraud, and lower user verification costs.