DEV Community: Maxim Vynohradov

What problem do you want to solve when use Express.js inside AWS Lambda?

Maxim Vynohradov — Fri, 29 Jan 2021 22:11:15 +0000

Hi! Two months ago I have published article Six reasons why you shouldn’t run Express.js inside AWS Lambda.

And looks like there are many people that disagree with this. Maybe somebody here can provide another props and cons of Express.js inside AWS Lambda?

Links:

dev.to - https://dev.to/max_vynohradov/six-reasons-why-you-shouldn-t-run-express-js-inside-aws-lambda-2o88
medium.com - https://medium.com/dailyjs/six-reasons-why-you-shouldnt-run-express-js-inside-aws-lambda-102e3a50f355
hackernoon.com - https://hackernoon.com/6-reasons-why-you-should-not-connect-expressjs-and-aws-lambda-b71n31st
reddit.com - somebody created 3 topics - https://www.reddit.com/search/?q=Six%20reasons%20why%20you%20shouldn%E2%80%99t%20run%20Express.js%20inside%20AWS%20Lambda

The right way to make advanced and efficient MongoDB pagination

Maxim Vynohradov — Mon, 25 Jan 2021 08:57:41 +0000

Onсe upon a time, we had a complex project enough (ride-sharing and taxi application) with stack Node.js and MongoDB. We have chosen this stack because it was preferable by the customer, good known by our team, and at the same time looks like good a suite for project tasks.

Everything was great, the number of users became more than twelve thousand, the number of active drivers was close to three hundred drivers. In one year, the number of rides becomes more than two million.

But once we need to create an admin panel to control and monitor all processes (from the business point of view) in the main application. The huge percent of requirements was to have advanced lists of a variety of entities, with bind statistics over them.

Because we use mongoose, as ODM, first of all, we take a look at its plugins. The most popular of them, that related to pagination are
mongoose-paginate
*Pagination plugin for Mongoose Note: This plugin will only work with Node.js >= 4.0 and Mongoose >= 4.0. Add plugin to…*www.npmjs.com
mongoose-paginate-v2
*A cursor based custom pagination library for Mongoose with customizable labels.*www.npmjs.com
mongoose-aggregate-paginate
*mongoose-aggregate-paginate is a Mongoose plugin easy to add pagination for aggregates. This plugin can be used in…*www.npmjs.com
mongoose-aggregate-paginate-v2
*A cursor based custom aggregate pagination library for Mongoose with customizable labels. If you are looking for basic…*www.npmjs.com

Another requirement was to have a possibility to choose a specific page on-demand, so the approach to use “previous-next”-like pagination, that based on cursors was prohibited immediately — its mongoose-paginate-v2 and *mongoose-aggregate-paginate-v2 *libraries.

The oldest, and probably the simplest in usage is mongoose-paginate — it uses simple search queries, limit, sort, and the skip operations. I guess it’s a good variant for simple pagination — just install a plugin, add few lines of code to your endpoint, and that’s all — work is done. It even can use “populate” of mongoose, — something that emulates joins from SQL world. Technically it just makes additional queries to the database, that probably not the way you want. Even more, when you just have a little bit more complicated query, with any data transformation, it will be totally unusable. I know just one way to normally use it in such cases — first create MongoDB View — technically its pre-saved queries, that MongoDB represents as read-only collections. And just then run pagination using mongoose-paginate over this view. Not bad — you will hide complicated queries under view, but we have a better idea of how to solve this problem.

MongoDB Aggregation Framework is here!

You know, I guess, it was really day for the MongoDB community when Aggregation Framework was released. Probably it allows for most of the queries that you can imagine. So, we think about taking mongoose-aggregate-paginate into use*.*

But the next two things that disappointed us:

**What does this plugin need? **I mean — what task does it help to solve, that cannot be solved without this plugin, with the same effort. Looks like it just one more additional dependency in your project, because it doesn’t bring any profit, even don’t save your time…

Internal codebase, and general approach to making queries. This library makes TWO calls into a database and waits for responses via Promise.all. First — to fetch query result and second — to calculate the count of total records that query returns, without $filter and $limit stages. It needs this to calculate the number of total pages.

How we can avoid additional queries into the database? The worst thing here that we need to run all aggregation pipeline twice, that can be costly enough in terms of memory and CPU usage. Even more, if collection huge, and documents tend to be few megabytes, it can impact Disc I/O usage, that also a big problem.

The good news — Aggregation Framework has a specific stage in its arsenal, that can solve this problem. It’s $facet:

Processes multiple aggregation pipelines within a single stage on the same set of input documents. Each sub-pipeline has its field in the output document where its results are stored as an array of documents.

MongoDB documentation about $facet stage .

Aggregation Pipeline for pagination will have the next shape:

{ 
  $facet: {
      <outputField1>: [ <stage1>, <stage2>, ... ],
      <outputField2>: [ <stage1>, <stage2>, ... ],
      ...

   }
}

Also, pipelines for pagination can be improved by customization for specific cases. Some tips are listed below:

Run all operations, that don’t directly affect on final pagination result, after all possible filters ($match stages). There are stages like $project or $lookup — that don’t change the number or order of result documents. Try to cut off as many documents as you can at once.
Try to make your models as more self-sufficient, to avoid additional $lookups. It’s normal to duplicate some data or make pre-computing fields.
If you have a really huge pipeline, that processes many data, your query will probably use more than 100MB. In this case, you need to use allowDiskUse flag.
Follow Aggregation Pipelines performance optimization guide. This advice helps you to make your queries more efficient.
And technically — you can make dynamic queries on your application code side — depends on conditions you can add, remove or modify specific stages. It can speed up your queries, and moreover, make your code more eloquent.

Coz NDA, I cannot show you real database schema and real queries. But let me show you a small example of such pagination.

Imagine that you have two collections — Statistic and Drivers. Drivers collection is static enough in thinking of types and amount of fields in different documents. But Statistic is polymorphic, can be changed during time, as a result of business requirements updates. Also, some drivers could have big statistic documents and history in general. So you cannot make Statistic as subdocument of Driver.

So code and MongoDB query will have the next shape:

const ridesInfoPaginationPipeline = (filter = {}, skip = 0, limit = 10, sort = {}) => [{
    $match: {
      ...filter,
      active: true,
    }
  },
  {
    $sort: {
      ...sort,
      createdAt: -1,
    }
  },
  {
    $lookup: {
      from: 'statistic',
      localField: '_id',
      foreignField: 'driverId',
      as: 'driver',
    },
  },
  {
    $unwind: {
      path: '$driver',
      preserveNullAndEmptyArrays: true,
    },
  },
  {
    $project: {
      driver: {
        $ifNull: [{
          $concat: ['$driver.firstName', ' ', '$driver.lastName']
        }, 'Technical']
      },
      entityId: 1,
      message: 1,
      meta: 1,
      createdAt: 1,
    },
  },
  {
    $facet: {
      total: [{
        $count: 'createdAt'
      }],
      data: [{
        $addFields: {
          _id: '$_id'
        }
      }],
    },
  },
  {
    $unwind: '$total'
  },
  {
    $project: {
      data: {
        $slice: ['$data', skip, {
          $ifNull: [limit, '$total.createdAt']
        }]
      },
      meta: {
        total: '$total.createdAt',
        limit: {
          $literal: limit
        },
        page: {
          $literal: ((skip / limit) + 1)
        },
        pages: {
          $ceil: {
            $divide: ['$total.createdAt', limit]
          }
        },
      },
    },
  },
];



const executePagination = async () => {
    return Statistic.aggregate(ridesInfoPaginationPipeline());
}

As you see, using Aggregation Framework and $facet stage we can:

make data transformation and complex queries;
fetch data from multiple collections;
get metadata of pagination (total, page, pages)in the one query without additional query execution.

Regarding the main drawbacks of such an approach, I guess just one is major — higher complicity of development and debug process, along with higher entry threshold. It includes performance troubleshooting, knowledge of a variety of stages, and data modeling approaches.

So, pagination, that based on MongoDB Aggregation Framework, is not pretending to be a silver bullet. But after many attempts and pitfalls — we found that this solution is covered all our cases, with no effects and no high coupling to a specific library.

Six reasons why you shouldn't run Express.js inside AWS Lambda

Maxim Vynohradov — Mon, 07 Dec 2020 12:07:16 +0000

Some facts why usage Express.js inside AWS Lambda is pitiful design anti-pattern and how to give it up without pain.

Last few years popularity of NPM packages, that allow you to use Express.js inside AWS Lambda handler, grow up rapidly. These packages provide some functionality that allows you to run Express.js middlewares, controllers with some limitations, instead of plain AWS Lambda handler.
Some examples of such libraries:

aws-serverless-express

Run serverless applications and REST APIs using your existing Node.js application framework, on top of AWS Lambda

serverless-http

But why developers decided to do so? They’re just a few foremost reasons, that I’ve usually met in practice:

No interest to learn new approaches to write handlers for API.For different reasons — want to use serverless architecture, but have not time to adopt and re-write existed Express.js based solution to Lambda handlers.
Wish to use the existing Express.js functionality and ecosystem, mostly it’s about huge numbers of third-party middleware.
Tries to reduce costs using AWS Lambda instead of development server (like EC2, AWS ECS, AWS EKS, etc.)

So below the list of reasons why the usage of Express.js inside AWS Lambda in most cases are redundant, you probably get many drawbacks from this approach.

Increasing node_modules size and cold starts

Simple point — the bigger node_modules do your artifact have, the bigger cold starts of AWS Lambda will you have. With no exceptions. Raw Express.js is near 541.1 KB, but you also need additional dependencies, mostly middleware, that can increase your node_modules several times.

Additional operational time

When you use standalone Express.js on the server (standard way), each HTTP request is some kind of text that the server parses to a well-known request object. Lambdas that people tried to use with Express.js inside, usually runs under API Gateway or AWS Application Load Balancer, and data that come from this event source are already parsed by API GW and ALB! Yes, it’s different, but anyway.
When you use Express.js inside AWS Lambda your “system” make the next thing with input HTTP data:

AWS API GW or AWS ALB parses HTTP request and convert them to the event payload.
The library that wraps the Express.js server maps the lambda event to server request.
Express.js one more time converts this to its request object.
The similar with a response — the library that wraps Express.js converts HTTP response to AWS Lambda response object.

So many supplementary conversions. Sometimes it looks like just wasting processor time.

AWS Lambda has a different limitation, that can be unexpected for your Express.js application:

First of all, lambdas are stateless — each AWS Lambda instance is an AWS Firecracker container, that will shut down in some time after inactivity. So you cannot simply persist data and share them across all lambda instances. The same situation with sessions — to use it with AWS Lambda, you need additional storage, for instance, Redis instance hosted as AWS ElasticCache.

Lambdas container can live during several handler executions (warm lambdas), but in any way, it quits unexpectedly. And this could break some tools or make their behavior unpredictable. The most impressive case is related to buffering, loggers, and any error trackers, like Sentry. Usually, they don’t send all logs, data immediately, they buffering them firstly, and then send several logs items at once, to make this more efficient. But when your lambda’s container quits, time-to-time these buffers do not have time to be flushed into storage or third-party services. For sure, we can disable buffering, but some of the services require another SDKs, that specific for AWS Lambda. And they cannot be re-used simply as Express.js middleware — you should wrap them up as your own middleware, that double work.

Also, you cannot use web-sockets (WebSockets, socket.io) inside the Express.js application, for the same reason — the lifetime of lambda execution container. But at the same time, AWS API GW supports web sockets, but they are implemented in another way, you cannot connect socket.io to them.

Some things that you are used to do in the Express.js app are different in AWS Lambda and has more adequate alternatives

Despite all disadvantages, the embedded middleware pattern in Express.js is probably one of the popular things in the Node.js world. However, there is no need to use Express.js just for this, coz at least one middleware library is suited for AWS Lambda better:
@middy/core

Core component of the middy framework, the stylish Node.js middleware engine for AWS Lambda To install middy you can www.npmjs.com

Also, it implements an onion-like middleware pattern, that is much more flexible than Express.js can provide for you.

Best practices for Express.js and AWS Lambda are different

At least you can easily find out the next point — security protection approaches are different. When Express.js best practice guide proposes to use Helmet.js library, it doesn’t applicable to AWS Lambdas. AWS proposes to use AWS WAF service that:

Protect your web applications from common web exploits

Lost benefits from individual packaging of lambdas

When you write classic AWS Lambda handlers, you usually can package each lambda artifact separately, to reduce each artifact size. But when you use Express.js, you cannot do this — all lambdas require the same dependencies. Technically you can, but all of them will have the same size, which negates their advantages. Also, in this case, serverless-webpack-plugin cannot optimize imports correctly, because technically each lambda will have the same dependencies tree.

Despite all of the above, I believe, that there some cases when the usage of Express.js inside AWS Lambda are valid and justified:

Pet projects — coz great AWS Free Tier, probably you can run them for free.
Your service is not mission-critical, and you are okay with all issues described above — so, okay, you can use it without any doubts (but don’t forget about technical debt).

Hope this information will be useful and you will not forget this when decide to use Express.js inside AWS Lambda the next time.

Tools to Manage NPM Dependency in Your Project as A Professional

Maxim Vynohradov — Mon, 16 Nov 2020 23:16:34 +0000

Why do we talk about project quality and technical debt so much? Because this directly or indirectly affects the speed of development, the complexity of support, the time to implement new functionality, and the possibility of extending the current one.

There is a great number of aspects that affect a project’s quality. Some of them are difficult to understand, hard to check, and require manual checks from highly experienced developers or QA engineers. Others are plain and simple. They can be checked and fixed automatically. Despite this fact, they represent a weighty part of the whole project’s quality.

In this article, you will find NPM packages that can automatically check some critical sides of your project, such as NPM dependencies, their licenses, and validating security issues.

Find Missed or Unused Dependencies

Depcheck is a useful tiny library to check what dependencies are unused and what dependencies are missing from package.json but used in your code base.

depcheck - Check for Vulnerabilities in Your Dependencies

It’s highly recommended to use it locally (for instance, on pre-commit hooks) or in remote CI to avoid the following issues:

Redundant dependencies increase build/bundle size, which leads to these consequences: Docker images become bigger, AWS Lambda handler also has longer cold starts, and an event can surpass lambda size limits.
Missed dependencies can break your applications in totally unexpected ways in production. Moreover, they can crash your CI/CD pipelines if they are dev dependencies.

Installation and usage

npm install -g depcheck
// or
yarn global add depcheck

Example of usage

// usage as npm script
"dependencies:check": "yarn run depcheck",

By running this command, you can see a list of issued dependencies:

List of issued dependencies

npm-audit, yarn audit, and improved-yarn-audit are tools that can find out dependency vulnerabilities. Moreover, they automatically update packages to resolve issues. Both npm audit and yarn audit are pre-installed with package managers, but I prefer improved-yarn-audit. It’s a wrapper around yarn audit that provides some improvements — especially for usage in CI pipelines (from docs):

No way to ignore advisories
Unable to filter out low severity issues
Ongoing network issues with NPM registry cause false positives

improved-yarn-audit - This project aims to improve upon the existing Yarn package manager audit functionality

Installation

npm install -g improved-yarn-audit
// or
yarn global add improved-yarn-audit

Example of usage

"dependencies:audit": "yarn run improved-yarn-audit — min-severity moderate",

Below, you can see the results of using this command in a real project codebase. This tool searches for vulnerabilities in transitive dependencies too:

Check Licenses of Dependencies

For real production projects, dependency licenses are critical to use because the way that you are using the dependency can be prohibited by the package’s license. To avoid this, you should continuously check the licenses of all dependencies that you use in a project. And if your project is a startup, appropriate usage of dependencies according to their licenses is mandatory to get investors to approve your product. license-checker is the best way to do this!

license-checker - Ever needed to see all the license info for a module and its dependencies?

Installation

npm install -g license-checker
// or
yarn global add license-checker

Example of usage

"licenses:check": "yarn run license-checker --exclude 'MIT, MIT OR X11, BSD, ISC'",

Checking licenses of dependencies

But for usage inside CI/CD, I prefer the following variant because it’s much shorter:

"licenses:check:ci": "yarn run license-checker — summary",

I hope this article helped you to address or avoid problems with NPM packages.
Thanks for reading!

What We Have Learned While Using AWS Lambda in Our Production Cycles for More than One Year

Maxim Vynohradov — Tue, 15 Sep 2020 21:38:52 +0000

Over the last few years, serverless approaches have gained decent traction in the web app designing, developing and implementing sectors. In the early days, many engineers treated serverless just like another hype. Still, almost all of those who tried to use it had to admit that the technology turned out to be as good as traditional and standalone virtual machines for hosting web-applications.

To date, we can see that startups tend to utilize serverless technology stack as a part of their systems or even as their primary solution for building products in different domains.

First things first

Our team decided to test out the technology while working on the product during the last year — an on-demand bike taxi app that uses a serverless approach for one of its components. In fact, it is much similar to an Uber app.

Technically, it was mostly a REST API and cron-tasks, anchored by the following technologies (all of these are provided by the Amazon Web Services):

API Gateway as a platform for API management.
CloudWatch Rules for scheduling cron-tasks.
Lambdas as computing units.
S3 buckets to store static files.
CloudWatch Logs with Logs Insights for log management.
Tools for continuous integration and deployment of our application: AWS CodeBuild, AWS CodePipeline and AWS CodeDeploy.

Initially, we used Node.js version 10 to write the code (a few months ago it was upgraded to version 12 without any issues). And all the infrastructure part (I mean all the AWS objects descriptions) is created and managed by an open-source Serverless Framework.

*This guide is not about AWS, FaaS (Function as a Service) or Serverless framework, since there is a lot of such content on the Internet. Here you will only find the things that our team faced during the development and after-launch stages. This info might be helpful if you come up with doubts about which technology to adopt for your next project. *

The Serverless world — the remarkable benefits of using AWS Lambdas

Let's start with the good parts! No matter what any hater says, the Serverless world provides a bunch of excellent features that you cannot achieve in any other way under equal conditions.

When we started this project mostly from scratch, it didn't require any severe capacity in measurements of Memory, CPU or network, to name a few. The same statement can be made not only about the development phase but also about the Staging, QA and Pre-Prod environments.

Traditionally, we need four servers, whether that be virtual machines, docker containers or any other platforms where we can host servers. For sure, it might be quite expensive to keep and maintain servers, even small and low-power ones. Even switching them off at nights and weekends is no way an option.

However, the Serverless world has an alternative solution – the so-called "Pay as you go" payment approach. It means that you pay only for the computing resources and network load that you use, even though the entire infrastructure is deployed and accessible at any moment.

In practice, it means that we were not burdened with any cost savings during the project's development. Moreover, while we remained within the AWS Free Tier limits, the actual cloud usage was charge-free until we reached the production stage.

So here are some advantages of AWS Lambdas worth mentioning here.

Outstanding scalability

The app was designed for the city with more than 13 million people. So it’s no wonder that the number of users started snowballing right after the first release. By "snowballing", I mean thousands of new users per hour in the first few weeks, hence a bunch of rides and ride requests as well.

That's where we felt all the benefits of the AWS Lambdas' incredible scalability and zero management of the scaling process. You know, this feeling when you see a rapidly growing number of requests on the chart (that was automatically provided by AWS). And the greatest part is that you shouldn't even worry about this, since the AWS Lambdas are scaled automatically. All you have to do is set a threshold for the concurrent invocation.

A standard set of monitoring and logging tools

Aside from the automatic scalability feature, AWS provides a basic set of tools for Lambdas. So, you don't have to waste your precious time dealing with the annoying configuration of basic monitoring metrics, such as Memory Usage, Execution time or Errors Count.

Moreover, you can customize your own dashboards in the CloudWatch service that would help you track performance issues and execution errors throughout the entire serverless application.

For sure, you won't come up with as many customizable graphics options, as Grafana or Kibana can provide, but at the same time, the AWS CloudWatch metrics, alarms and dashboards are way cheaper. Besides, you can attune these without much preparation, and last but not least — the cloud provider takes responsibility for the efficiency of the monitoring tools described above.

Isolated environment

Well, let's say you managed to customize a dashboard without any problems. But then you realized that the Lambdas execution process took more time than it should have, and it looked like Lambdas performed some sophisticated calculation. Luckily, it's not a problem for AWS Lambda, since every function-handler runs in an isolated environment, with its own configuration system of Memory or CPU.

In-fact, each instance of Lambda is a separate AWS Firecracker Container that spawns on a trigger (in case of a REST API, the trigger is an HTTP request). That said, all you have to do is just increase CPU units Count or Memory for the specific Lambda, with no need for global updates, as if it were done in a classic server.

Flexible errors management

Another outstanding benefit that you can enjoy while using AWS Lambda is decent error handling.

As said above, each Lambda has an isolated environment, so even if one of your Lambda instances fails for any reason, all other Lambdas will continue to operate normally. It's fantastic when you have just one or two errors from a few hundred possible AWS Lambda invocations, isn't it?

Automated retry attempts

Furthermore, retry attempts is another out-of-the-box feature that AWS provides. Should a Lambda fail for any reason, it would be automatically re-invoked with the same event payload during the pre-configured period. I must say, it’s a quite useful feature if your Lambda is invoked by schedule and is trying to send a request to a third party resource that can be unavailable.

Finally, AWS Lambda supports the Dead letter queue concept that means you can acquire relevant notifications and tracking information about failed Lambdas.

The AWS Lambda disadvantages — a few pain points to learn from

On the flip side of the coin, AWS Lambda and the serverless concept are not entirely perfect yet and have enough unresolved problems and pitfalls that make the development and support processes a little bit harder.

Duration limits

For our project, it was all about limits. For example, we ended up with an execution duration limit — a Lambda can be performed within 15 minutes maximum. Moreover, if a trigger is requested from an API Gateway, the duration must be no more than 30 seconds.

Perhaps, we could accept such limits for the API, but a 15-minutes limit for the cron-tasks was way too tight to execute the particular scope of tasks on time. That said, since the computed intensive tasks couldn't be invoked with Lambdas, we had to create a separate server specifically for long-running tasks.

CloudFormation deployment limitations

Another significant issue we faced was the Lambda deployment via CloudFormation (the AWS service for infrastructure and deployment). At the very beginning of the project, everything was fine. Still, when the number of Lambdas mushroomed into more than 30 CloudFormations, the stack started failing with different errors like "Number of resources exceeded", "Number of outputs exceeded".

Thankfully, the serverless framework and its plugins helped us to tackle this issue early on. There are also a few other ways to solve such kinds of problems, but that'll be a topic for another article.

Failure to expand monitoring and debugging toolset

Even though AWS provides some basic level of monitoring and debugging, it's still impossible to extend this part and make some custom metrics that could be useful for particular cases and projects. This time around, we had to use third-party services that you usually need to integrate as libraries into your code to be able to monitor some specific stuff.

Cold start-related delays

As mentioned above, each Lambda instance is in-fact a tiny Firecracker Container with some basic runtime environment, libraries and your code. It's created temporarily to process any event evoked by the triggers. It's a well-known fact that creating a container or running an executable environment and code takes some operational time called a cold start.

It can take random time between 100 milliseconds to a few minutes. Moreover, if you keep your Lambdas under VPC (Virtual Private Cloud), cold starts will take more time because the system will have to create additional resources for each Lambda, called Elastic Network Interfaces.

This, in turn, results in annoying delays, so the end-users have to wait for the app to respond, which is definitely not good at all, isn't it? The workaround here is to ping your Lambda every 5 minutes to keep containers "warm". The AWS system is smart enough and doesn't kill Lambdas containers immediately, since it is based on the concept that triggers would keep spawning new events.

Database connection pitfalls

In view of the above, it's problematic to manage a database connection for such a system. You cannot just open a connection pool to your MongoDB or MySQL servers at the application startup phase and reuse it during the entire lifecycle.

So there are at least two ways to manage connections:

You should open a connection for each Lambda invocation and close it after your code with logic would be completed; You can try to reuse a connection and keep it in the Lambda memory as a reference in code or field in context — it allows you to keep a connection within the same Lambda containers till closing.

However, both have their own limitations. In the first case, we end up with additional delays since we have to open a connection for each Lambda call. In the second case, we can't be sure for how long Lambda would keep a connection, and consequently — we can't handle a connection shut-down properly.

Local test limitations

Besides, the serverless apps are hard to test locally, because usually there are a lot of integrations between AWS services, like Lambdas, S3 buckets, DynamoDB, etc. For any type of local testing, developers must mock all this stuff, which usually is a formidable and time-consuming task.

Inability to adopt caching in a traditional way

On top of everything else, you can't implement a traditional caching for classic-like servers. Usually, you have to use other services like S3, DynamoDB or ElasticCache (de-facto Redis hosted on AWS) to keep Lambda's state or cache some data between AWS Lambda invocations.

In most cases, it results in extra costs of the entire infrastructure. Not to mention additional operational overhead — you'll have to put and fetch cached data from remote storage, which, in turn, can slow down the performance of your cache.

Complex payment model

The last one worth mentioning is a sophisticated price calculation. Even though AWS Lambda is quite cheap, various supplementary elements can significantly increase the total costs. People tend to think that the pricing for using the AWS Lambda's API is based on its computing resources and duration of code execution. In fact, you should keep in mind that you'll have to pay for additional services, such as:

Network traffic,
API Gateway,
Logs stored in the CloudWatch.

Wrapping up

Summarizing the above, I want to say that the AWS serverless approach is a great way to strengthen your development practices. Nevertheless, you have to keep in mind that it's quite different from traditional servers.

To leverage the life-changing benefits of this technology, you have to get acquainted with all the subtleties and pitfalls in the first place. Besides, you also have to think through the architecture and its specifics for your particular solution.

Otherwise, the serverless approach can bring you rather problems than beneficial features as of insufficient educational background.

Crime and Punishment: Hype Driven Development

Maxim Vynohradov — Sun, 12 Jul 2020 22:28:55 +0000

Disclaimer : long read. About the painful stuff, what I want to share for more than a month. Also does not claim to be complete, it is more about how I try to deal with it.

Introduction: what is Hype Driven Development
HDD definition as a crime
Objective side and consequences of the HDD crime
How to prevent HDD in project
Conclusion and afterword

Introduction: what is Hype Driven Development?

Here I want to share a few thoughts on how to convince the team / manager / product owner to not use the "hype" technology or make proof that this technology fits good in your project.

It is well known that representatives of different professions often invent ways to solve a particular problem. For instance, which approach and constituent materials to use and positions, the choice of a specialist often determines the success of achieving the goals. For example, would the cafe be profitable, and would the bridge withstand the load successfully, would the operated heart beat for many years. And, unfortunately, one wrong decision can lead to fatal consequences and suffer great losses, and even worse – harm the health and life of people.

Engineers, software developers, web developers, and other technical guys are no exception. Moreover, the final software products are extremely dependent on real business, with real people and finances. And the harm that can be caused by badly designed and implemented software can potentially be huge and irreparable damage to the entire business.

Let's look at one of the most common crimes committed by various teams that somehow participate in the design and development of various types of IT projects, named Hype Driven Development (hereinafter referred to as HDD).

This concept is not new, but it does not go back to the past century, and means a decision about a technological stack or software architecture that is taken without technical justification, research, based only on loud and popular words from the Internet or other sources, "buzzwords ", unverified rumors. It's a development based on fashion and trends, without explanation of your choice from a technical point of view and not confirming by checking or evaluating the proposed solution.

What I have most often seen: blockchain, data science, machine / deep / any other learning, MongoDB, GraphQL, Serverless, Clouds, Kubernetes, and more. These "hype" technologies and much more I've seen in different proposals of various technical requirements, in RFPs (Request for proposal), spreadsheets with estimates, and so on. Don't get me wrong, I have a deep respect for the above-listed technologies, and, as an example, I often use MongoDB and Serverless. The problem is that these technologies were proposed to be used and added in the post-production phase, and their connection with business logic seemed as far-fetched as possible.

HDD definition as a crime

Why I started with such a long prelude at the beginning - you can imagine when a builder chooses from what material to build a skyscraper "because it is so fashionable". Or maybe a doctor who prescribes pills according to the principle "well, those people take similar pills abroad, and this may help you too." Personally, I have not, I have never seen this, I wish the same to you. For me, such an approach and attitude to work looks like a crime. Therefore, I propose to consider HDD a crime, at least in the article.

To begin with, I think it's worth remembering what constituent elements of a crime are. Without the presence of corpus delicti, an act cannot be considered criminal.

It's a quite extensive and complex definition, but let's summarize and simplify it a little, coz after all, this is not a jurisprudence article, especially since the laws of countries and states are different enough. We are interested in the following components of constituent elements of a crime:

the object - is the part to which the damage was caused;
the objective side- is the external manifestations of the crime, action or inaction, the consequences of the crime, as well as the connection between them or, for example, the time, method, means of committing;
subject - the one who committed the crime;
the subjective side - is a person's attitude to a crime that he has committed. Was it intent (direct or indirect) or it was negligence (frivolity or negligence), there is a motive and what is the purpose of the crime.

Let's start with the subject. In our case, this is one of the members of the development team in the broad sense of the word: engineers, managers, product owners. That is all those people who in one way or another influence the development process. But they can be divided into two categories. Some of them work on business logic, tasks, and functional requirements - usually the manager, business analyst, and product owner. On the other hand, we have technical specialists - test engineers, developers, architects, team leads. Of course, they can work at the junctions of the technical and non-technical parts of the project, but now it is important that both the former and the latter can offer, perhaps even insist on using the hype technology.

And here we smoothly move on to the subjective side of the HDD crime - did anyone want to harm when choosing an HDD technology? I think nobody would choose technology or design architecture to the detriment of their project or product. Then why?

Based on personal experience, I can say, that typically, customers (meaning people or companies which order development of product, not product users ), product owners choose one of these technologies to attract the attention of investors and potential users as sources of funds and income. The reason is more than clear and transparent - often investors are not technical specialists, so, many of them are being led by beautiful marketing slogans, that promise many profitable things for the product. But at the same time, they usually don't understand the real essence of things. As a result, we get a technical solution based on marketing. There are also cases when a customer already comes with technical requirements for a future product. When asked "why does he want such a stack of technologies?", we usually get one of the options: "one of my projects uses such a stack and everything is fine", "my partner's project uses such a stack - and he is happy with everything". People who say this usually do not think about the fact that if functional and non-functional requirements are different, then the technologies used do not have to be the same.

Sometimes it happens that someone from the technical part of the team initiates the use of "hype" technology. Usually, this is less experienced teammates (juniors), which still cannot always assess the risks and make appropriate time estimations. For them, it all comes down to the fact that they want to try a new technology, which is given in a beautiful wrapper. And also usually they do not look at the license of the solution (let's say it is a library or a framework), or at its maturity.

Objective side and consequences of the HDD crime

The objects of the HDD crime can also be different. It depends on what side of the project we are considering - technical or business. If we are talking about the technical side, there are codebases in general, its architecture, the infrastructure of deployments. On the other side, the business operated by other concepts, like include cash flows, investments, profits, and other business processes.

So, let's say a "hype" technology penetrates the project, seeps into the code base, a lot of logic, additional code and configurations begin to rise around the technology. In this case, what and who will suffer from the introduction of "hype" technology, and how (we are talking about the objective side) will the HDD affect the project and the business as a whole (the object of the crime)?

There are a lot of things, consider that you are playing dice - it all depends on how hype and not properly researched technology is deeply integrated into the business logic of your application, code and development processes in general. The list below presents the most obvious consequences.

🚫 Building a business model around technology, not adopting technology to business.

This happens relatively rarely, but extremely unfortunate. It happens when a "hype" technology is at the very core of the business logic and, accordingly, the code base, when the very idea of the application is to build something over the technology. Accordingly, if the technology breaks down in one way or another (for example, an unobvious bug pops up) - with a high probability, the product that is based on it breaks too.

🚫 Low conversion from "hype" technologies - that is, the discrepancy between the expected profit from the technology and the real one, as they often do not take into account that needs to pay for additional infrastructure along with the support of such a solution, as well as the work of specialists who will configure this solution.

🚫 The black box problem. Usually, the external dependency is perceived as something atomic, as an integral component within our system, with its public API (interfaces, methods, endpoints). But in fact, this is the same program code as the one in which we integrated this component, with its dependencies and bugs, performance, and security problems. You probably say: "But often the code of such components is open source!". Now think, how often do you have enough time, desire, and immediate need to look into the program code of one of your libraries or database (which is also a software product with its source code)? I think not often (but usually it tends to never). For us, a bug (this is how it can harm your project) in any external dependency is a kind of analog of "Schrodinger's cat in a box", which is both there and not. We often find out about the presence of bugs only when this external bug affects the state of our system.

🚫 Lack of professionalism and experience in the introduced technology. It's simple - the devil is in the details. Even if you have over-qualified specialists in your team who love work and are ready to delve into everything new on their way, hardly anyone can comprehend the depth of this or that technology in a short time.

🚫 Increasing the entropy of the program code and the system as a whole. Hmm, okay, but what is entropy? It is a very broad concept, rooted in natural sciences, thermodynamics, and later mechanics and software. The initial idea is that in an isolated system the degree of chaos and uncertainty will not decrease over time but will remain the same or increase. On the other hand, entropy is a measure of the unknown about the system as a whole. The concept first appeared in the context of software in the book Object-Oriented Software Engineering (by Ivar Jacobson, Patrik Jonsson, ACM Press Staff, Magnus Christerson, Gunnar Overgaard). In general, think of entropy as a measure of the dishonesty of the system. The more disorder, uncertainty, the higher the probability of breakage. Compare a hammer and a car, which is a hundred times more complex in its structure - which one is broken down more often and faster? A car has many more constituent components with its level of entropy, and therefore the total entropy is higher. Returning to HDD, the more complex and more unknown technology we involve in the process, the higher the likelihood that the project entropy will increase, and hence the number of bugs and Mean time between failures (MTBF).

🚫 Life cycle and aging of software. Are you sure that the hype framework, the server, will exist, developed, and be supported for a long time, at least as long as the life of your project / product? And that in the next version the public API will not change completely - you need to either translate the API to a new one or live with the deprecated API. Have you assessed these risks correctly? Let's discuss this further.

🚫 Increased risks. New technologies usually mean new risks:

Going out of bounds by time estimation.
Risk of unsuccessful completion of the project.
The additional overhead for infrastructure, development, and support.
Additional vulnerabilities of the system crash.
Nobody guarantees that critical security updates will be issued and fixed on time.

How to prevent HDD in project

Ok, it seems that we have considered all the components of this crime. But how to prevent and avoid the use of HDD on a project without realizing full awareness of what technology the team will live with on the project? Or you can prove the validity of the implementation and evaluate all the pros and cons. Once again, just in case, if the technology is hype, this does not mean at all that it is full of bugs, has not matured, or does not apply to your project. It's just that there should be common sense in everything, namely in the analysis of technology from a technical and business point of view, a sound assessment of risks and additional costs.

Here are some ideas and approaches, the use of which can influence the views and awareness of the team regarding the choice of hype technology.

💡 The opinion of experts, both in the technical part and in the subject area of business - "do we need it?" Before getting into Data Science, Serverless, and the like, consult a specialist with experience with this technology. Probably, you will not give up the technology, but completely rethink its use in the project. Moreover, it has a positive effect on grades.

💡 Proof of Concept creation is as old as the world - before doing a full-scale implementation and implementation, you should make the simplest demo version tied to any technology. Most likely, you will not notice any deep and hidden problems, but you can throw away the technology at the start if you see that it does not suit you in any way.

💡 A collective brainstorm "for" and "against" - decisions should never be made alone or in a narrow circle of employees. It is worth discussing and listening to all opinions, perhaps a sensible thought will come from an unexpected side. Also, interesting is the Three amigos approach, which is rooted in BDD and Agile, and means that there are three parties involved in the process: Business, Development, Testing. Thus, you can try to identify the largest number of problems and corner cases associated with the introduced technology. More details here.

💡 "5 Whys" - and again about how not to drag unnecessary things to your project. Try to apply this technique to the next technology adoption issue. Ask yourself and your colleagues - "why are we doing this," and so on, 5 times according to the list (see the article). It may well be that simpler steps are sufficient to solve the original problem.

Also, consider whether you can achieve the desired results without using this technology. How much more difficult and costly it will be.

💡 KISS (keep it simple stupid) And YAGNI (you ain't gonna need it). A huge number of articles have been written about these principles, and I see no need to retell them. Just make sure you don't break them when you introduce the next technology to your project.

Conclusion and afterword

We're finally here, and I'll try to make the ending shorter! There is a huge difference between "adding and usage of new technology" and "deliberately adding and usage of new technology". It would seem that the phrases differ by one word, but there is a huge gap between them. Hype technologies are often interesting, progressive, the main thing is to choose and apply them deliberately, without risking the project and business, not forgetting about your technical debt.

DEV Community: Maxim Vynohradov

What problem do you want to solve when use Express.js inside AWS Lambda?

The right way to make advanced and efficient MongoDB pagination

Six reasons why you shouldn't run Express.js inside AWS Lambda

Increasing node_modules size and cold starts

Additional operational time

AWS Lambda has a different limitation, that can be unexpected for your Express.js application:

Some things that you are used to do in the Express.js app are different in AWS Lambda and has more adequate alternatives

Best practices for Express.js and AWS Lambda are different

Lost benefits from individual packaging of lambdas

Tools to Manage NPM Dependency in Your Project as A Professional

Find Missed or Unused Dependencies

List of issued dependencies

Check Licenses of Dependencies

What We Have Learned While Using AWS Lambda in Our Production Cycles for More than One Year

First things first

The Serverless world — the remarkable benefits of using AWS Lambdas

Outstanding scalability

A standard set of monitoring and logging tools

Isolated environment

Flexible errors management

Automated retry attempts

The AWS Lambda disadvantages — a few pain points to learn from

Duration limits

CloudFormation deployment limitations

Failure to expand monitoring and debugging toolset

Cold start-related delays

Database connection pitfalls

Local test limitations

Inability to adopt caching in a traditional way

Complex payment model

Wrapping up

Crime and Punishment: Hype Driven Development

Table Of Contents

Introduction: what is Hype Driven Development?

HDD definition as a crime

Objective side and consequences of the HDD crime

How to prevent HDD in project

Conclusion and afterword