Cover image by @designecologist
In this post, I'll show you some of the most powerful and useful ways to use serverless functions.
Why serverless?
Before we get into how to use serverless functions, let's first talk about why you would go "serverless" in the first place!
Where did the term originate?
The term Serverless started being used in talks and discussions around 2010. To the best of my knowledge it was first mentioned in writing in 2010 in this TechCrunch announcement about a startup called PiCloud.
The first thought leadership piece that I could find describing the "Serverless" paradigm was written by Ken Fromm in an October 2012 article titled Why The Future Of Software And Apps Is Serverless.
When AWS Lambda (functions as a service) was launched in November 2014, the term was quickly adopted by users and proponents of the service and "Lambda functions" quickly started being known as "Serverless functions". Since then, many other cloud providers and third party services have also started offering their own functions as a service offerings.
Serverless Paradigm
In the time since serverless functions were introduced the term has evolved and is many times applied to any service or abstraction that does away with the need to manage servers or infrastructure.
In posts by Ben Kehoe (Cloud Robotics Research Scientist at iRobot), he talks about serverless being "a consequence of a focus on business value. It is a trait. It is a direction, not a destination.". He also talks about "The Serverless Spectrum" and that "serverless is not an all-or-nothing proposition. It’s a spectrum — and more than that, it has multiple dimensions along which the degree of serverlessness can vary.".
Basically the idea, in my opinion, is that the more you rely on managed services and functions as a service, your application becomes more serverless. You can have an architecture that is completely serverless, but you can also have architecture that just takes advantage of serverless where and when needed.
Serverless benefits
In my post Full-Stack Development in the Era of Serverless Computing I did a deep dive outlining my thoughts around the benefits of going serverless. Here are the main benefits of using serverless functions and moving to a serverless architecture:
I also like Ben Kehoe's 3 point summary [here])(https://read.acloud.guru/the-serverless-spectrum-147b02cb2292#60b2).
Serverless technologies also help simplify building an event-driven architecture. Serverless functions can be triggered by dozens of event sources.
At the end of the day, the reason I love serverless is because it offers developer velocity, pay per computer, and scale by default. It allows me to experiment and try new things out quickly and easily.
⚡️ 10 Ways to Use Serverless Functions
Now that we've outlined what serverless is and why you would use it, what can you do with serverless functions?
1. Running a web server with routing
When I first started learning how to use serverless functions, one of the things that blew me away was that I was able to run a full express server directly in the function!
That's right, you can run an express server complete with routing and all of the dependencies you're accustomed to having directly in a serverless function and not have to deal with the actual server infrastructure that it's running on.
There are even helpful libraries to help you get this up and running really easily for AWS and Azure.
I wrote a blog post showing how to deploy this type of API pretty quickly, check it out here.
If you're writing Python, you can run a Django or Flask server as well.
2. Creating Alexa skills
If you're interested in developing an Alexa skill you can upload the code to a Lambda function and Lambda does the rest, executing it in response to Alexa voice interactions and automatically managing the compute resources for you.
Here are a few reasons why serverless functions are great for Alexa skills:
- You do not need to administer or manage any of the compute resources for your service.
- You do not need an SSL certificate.
- You do not need to verify that requests are coming from the Alexa service yourself. Access to execute your function is controlled by permissions within AWS instead.
- AWS Lambda runs your code only when you need it and scales with your usage, so there is no need to provision or continuously run servers.
- Alexa encrypts its communications with Lambda utilizing TLS.
- For most developers, the Lambda free tier is sufficient for the function supporting an Alexa skill. The first one million requests each month are free. Note that the Lambda free tier does not automatically expire, but is available indefinitely.
For more in depth information on how to create Alexa skills with Lambda, check out this tutorial.
3. Process images / videos
One of the most popular uses of serverless functions is multimedia processing; the implementation of functions that execute a transformational process in response to a file upload.
Using a storage event from a service like Amazon S3, you can automate this process by configuring a function to be invoked from an image or video upload.
The event argument to the function will have metadata about the file that was uploaded, allowing you to perform processing on it and store it back in the service or do something else with it.
For an example of what this might look like, check out this function that creates a thumbnail for any image uploads.
4. Database events
Like storage events, you can configure a lambda to be invoked from database operations. If you're using AWS, you can invoke a function from Amazon DynamoDB (NoSQL) or Amazon RDS (SQL).
This again enables a very easy way to integrate event-driven architecture without having to do a lot of work.
This comes in handy for a variety of use cases like expanding the base functionality of your database, implementing a stream of logs and analytics, or creating a service that aggregates the data for reports and dashboards
For DynamoDB, an event, or stream of events, from a database action might look like this:
{
"Records": [
{
"eventID": "1",
"eventVersion": "1.0",
"dynamodb": {
"Keys": {
"Id": {
"N": "101"
}
},
"NewImage": {
"Message": {
"S": "New item!"
},
"Id": {
"N": "101"
}
},
"StreamViewType": "NEW_AND_OLD_IMAGES",
"SequenceNumber": "111",
"SizeBytes": 26
},
"awsRegion": "us-west-2",
"eventName": "INSERT",
"eventSourceARN": eventsourcearn,
"eventSource": "aws:dynamodb"
},
{
"eventID": "2",
"eventVersion": "1.0",
"dynamodb": {
"OldImage": {
"Message": {
"S": "New item!"
},
"Id": {
"N": "101"
}
},
"SequenceNumber": "222",
"Keys": {
"Id": {
"N": "101"
}
},
"SizeBytes": 59,
"NewImage": {
"Message": {
"S": "This item has changed"
},
"Id": {
"N": "101"
}
},
"StreamViewType": "NEW_AND_OLD_IMAGES"
},
"awsRegion": "us-west-2",
"eventName": "MODIFY",
"eventSourceARN": sourcearn,
"eventSource": "aws:dynamodb"
}
]
}
As you can see, in the event we have metadata about the item being stored or updated in the database, what type of operation it was (INSERT or MODIFY), the item being stored (NewImage), the size of the item, and the old item if there was one (OldImage).
5. API business logic for interacting with a database
When working with a database, you often will have the business logic for the database operations living in a service somewhere on a server. Serverless functions offer a perfect use case for offloading the infrastructure and enabling you to quickly get your API off the ground.
You will also often have private variables that you don't want exposed to the client, using a serverless function you can easily set these and have them securely available for database access as environment variables.
Combining this with a web server like serverless express allows you an easy way to get up and running with a real scalable, enterprise-ready stack that will give you the foundation for most applications.
You can invoke your function from a GraphQL API layer like AWS AppSync (mutations & queries) or a REST API layer like Amazon API Gateway to send CRUD operations / HTTP event methods to your function.
If you'd like to see an example of how to get this up and running in just a few minutes using AWS Amplify, check out this tutorial.
6. Processing SMS messages
Using services like Twilio or Amazon SNS you can easily process SMS messages in a serverless function.
The event object from the serverless function will look something like this, giving you metadata like the sender's phone number, the message body, and the timestamp of the message:
{
"Records": [
{
"EventVersion": "1.0",
"EventSubscriptionArn": "arn:aws:sns:us-east-2:123456789012:sns-lambda:21be56ed-a058-49f5-8c98-aedd2564c486",
"EventSource": "aws:sns",
"Sns": {
"SignatureVersion": "1",
"Timestamp": "2019-01-02T12:45:07.000Z",
"Signature": "tcc6faL2yUC6dgZdmrwh1Y4cGa/ebXEkAi6RibDsvpi+tE/1+82j...65r==",
"SigningCertUrl": "https://sns.us-east-2.amazonaws.com/SimpleNotificationService-ac565b8b1a6c5d002d285f9598aa1d9b.pem",
"MessageId": "95df01b4-ee98-5cb9-9903-4c221d41eb5e",
"Message": "Hello from SNS!",
"MessageAttributes": {
"Test": {
"Type": "String",
"Value": "TestString"
},
"TestBinary": {
"Type": "Binary",
"Value": "TestBinary"
}
},
"Type": "Notification",
"UnsubscribeUrl": "https://sns.us-east-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-2:123456789012:test-lambda:21be56ed-a058-49f5-8c98-aedd2564c486",
"TopicArn":"arn:aws:sns:us-east-2:123456789012:sns-lambda",
"Subject": "TestInvoke"
}
}
]
}
For an example, check out this function that calls a GraphQL API with a mutation with the SMS body and message sender's information.
7. Authentication events
When managing a user's authentication flow, from signing up to signing in, you may want to have additional logic based on who the user is.
For example, a very popular use case is placing a user in a group and giving them additional permissions based on the user's email or some other attribute. A good example of this is if you have a service that wants to check to see if a user is a student, you may want to place them in a Students group and grant or deny them privileges in your app based on this flag.
Another popular use case is updating the user's metadata or placing the user in a group based on who the user is, for example you may have an array of Admins that you know will be signing up, and based on their email address you may place them in this Admin group or give them a custom attribute based on whether they are an Admin.
You can set various authentication events to trigger a lambda function natively using authentication services like Auth0 or Amazon Cognito.
Here are some examples of events that you can hook into:
- Pre sign-up (invoked when a user submits their information to sign up)
- Pre authentication (invoked when a user submits their information to be authenticated)
- Custom message (send customized MFA message)
- Post authentication (invoked after a user is authenticated, allowing you to add custom logic)
- Post confirmation (invoked after a user is confirmed)
- User Migration (migrate users from an existing directory into a new user directory)
- Pre Token Generation (invoked before the token generation, allowing you to customize the claims in the identity token)
For a look at some real authentication triggers, check out my examples here.
8. Chat bots
Because of the nature of chat bots, using a serverless function to handle the logic makes a lot of sense. You probably don't want to have to build and manage some infrastructure for your chatbot to run, and with a serverless function you won't have to pay for anything until it sees a lot of usage.
If you're authoring a chat bot with AWS services like Amazon Lex, you can even directly integrate a Lambda function to handle things like custom initialization and validation logic.
9. IoT sensor input messages
When working with sensor data, you need to be able to have the ability to respond to messages and scale in response. Provisioning a server(s) at a high capacity to do this doesn't make a ton of sense. Using a serverless function you can easily scale up and down and only pay for the compute resources being used instead of the capital expense of provisioning and maintaining a server(s) for peak scale.
If you're using a managed service like Amazon IOT, you can even use built in integration with Lambda functions to process incoming MQTT messages based on rules set in the service.
10. Payment processing gateway
When working with payment processing APIs, you often have secret keys that you do not want to expose on the client. Using a serverless function you can hide these private keys in environment variables without a lot of work provisioning and managing a server infrastructure. To see an example of how this works, check out this tutorial.
Because of the event-driven nature of processing payments serverless functions are also a perfect fit, allowing you to do things like send emails / notifications, call other functions, or interact with other databases or APIs from a single initial action.
If you're interested in learning more about building applications with serverless technologies, check out my book Full Stack Serverless from O'Reilly Publications.
Top comments (25)
An issue I ran into with Lambda Image processing is the 6MB payload limit. Did you get around that somehow?
The way this is usually worked around is to use a storage service like Amazon S3. From the Lambda you can fetch the image or video and process it from within the function without having to pass the data itself as a payload.
I see. Not sure that works though. The AWS Severless Image Handler template doesn't even support that. I know because I tried. I tried so hard lol. Even pulling from a S3 bucket, it still hits that limit.
Hmm, so the event object with a reference to the S3 image shouldn't be that large, not sure why that wouldn't work, I would have to see more details I think. For reference though, this is essentially the way I usually handle it (the general idea): github.com/dabit3/s3-triggers/blob...
You also could have run into the memory limit, if so you can increase it
Thanks for the info! I tried to dig into my error log to see what the specific error was, but I deleted the Cloud Formation, so those are no longer available. I'm pretty sure it was always the error of "exceeded payload limit" and all I was doing was fetching an S3 object from a bucket and resizing it. I'm pretty sure it wasn't a memory issue because the instances never exceeded the memory limit, but I dunno.
I ended up just using a service because I couldn't get it to work, but maybe I'll keep poking around.
Thanks for sharing your method! It's very helpful.
Do you know if your method works for images over 6Mbs?
What payload were you using to invoke your lambda. The error message is indicative that the JSON is over the 256 KB limit on lambda. If the lambda is invoked by an event in S3, you then can hydrate the images that is in S3 using the payload described here:
docs.aws.amazon.com/AmazonS3/lates...
Here's the doc on creating an S3 event:
docs.aws.amazon.com/AmazonS3/lates...
Cool. Thank you! I'm not entirely sure, but whenever I tried to resize an image that was more than 6mbs, it failed. It was pulling straight from an S3 bucket.
If you can post the code in a gist somewhere, I'll help you out.
Will do. Much appreciated
What about using streams?
You wouldn't have to download the image in its entirety in order to process it.
Wouldn't it solve your problem?
That sounds like it may work. I'll test it out and let you know! Thanks!
Ive been unzipping 3gb+ archives from s3 using streams, so this might help. Ps. i also compress images, tested with 10mb+ so that's probably not an aws issue.
That's good to know. Thank you! Do you have the code available for that?
Not yet. I need to clean it up and then ill be sharing it on my github - just need to find a moment to do that ;)
Cool. Thanks!
Serverless is awesome, and I was fascinated with it..before I started to use it, with limits for functions - you can't use it properly with 3rd party APIs. Cold time, non-trivial deployment, hard debugging, or if you did something wrong - you risk spending all your budget (if you forgot to set the limits in settings). I feel that serverless is good for well-top-notch-experienced developers and teams, enterprise-level companies, not for regular artisans. IMHO for sure, I found that for me much faster, easier, cheaper create a new VPS cluster with nodejs, instead of using serverless functions.
Not really - those were my concerns initially too, but you can get around most of those issues by using the serverless framework and its plugins (e.g. serverless offline plugin). And cold start is easily rectified by setting a cloud watch function to automatically income your lambda every 15 minutes
Cold start can be eliminated by new feature, reserved concurrency or smth. But tbh, i prefer just optimize function code to be fast to boot.
Lambda is also great for web scraping - in combination with CloudWatch Events even more 👍
Or running tests. Someone wrote how they made their tests concurrent and instead of some ridiculous time it os running within 20 seconds on 1000 lambdas ;)
Great article Nader
Thanks!!
You should mention that events are not available if you use Aurora Serverless, which is very annoying because if you need that, this will make your backend more complex and difficult to program.