DEV Community: Eduardo Romero

Serverless Architectural Patterns

Eduardo Romero — Mon, 04 Mar 2019 15:18:52 +0000

Black and white clouds, atmosphere — Photo by Bryan Minear

We’re building more and more complex platforms, at the same time that we are trying to address ever-changing business requirements, and delivering them on time to an increasingly large number of users.

What we deliver is inherently unique. A combination of different services, technologies, features, and teams with their own contexts and competing priorities.

Sometimes it just feels like you’re designing and building an aircraft carrier, while navigating in rough ocean, occupied, and under attack.

Eduardo Romero

@foxteck

"Writing software is like designing and building a new type of aircraft carrier while it is in rough ocean, occupied, and under attack."

21:14 PM - 26 Apr 2017

01

Continuously delivering quality software fast is a core business advantage for our clients. Accelerating the development process with modern architectures, frameworks, and practices is strategic.

The serverless paradigm it’s great at enabling fast, continuous software delivery. You don’t have to think about managing infrastructure, provisioning or planning for demand and scale.

And with the Function as a Service model we structure our code in smaller, simpler units, that are easy to understand, change and deploy to production. Allowing us to deliver business value and iterate quickly.

Serverless is a great enabler for experimenting, learning, and out-experimenting your competition.

Architectural Patterns are powerful way to promote best practices, robust solutions, and a shared architectural vision across our engineering organization.

Working on different projects leveraging serverless, I’ve seen patterns that we can adopt to solve common problems found in modern cloud architectures. While this is not an exhaustive collection, but it can be used as a catalog of architectural level building blocks for the next platform we build.

Serverless Architectural Patterns Catalog

Simple Web Service.
Decoupled Messaging.
Robust API.
Aggregator.
Pub/Sub.
Strangler.
Queue-based Load Leveling.
Read-heavy reporting engine.
Streams and Pipelines.
Fan-in and Fan-out.

Simple Web Service

“A client needs to consume a service via a Public or Internal API.”

A simple web service is the most standard use-case for AWS Lambda as a backend service. It represents the logic or domain layer of a n-tiered or layered architecture.

Public API — Lambda Functions exposed via HTTP(s)

For Public APIs, API Gateway is exposing lambda functions via HTTPs. API Gateway can handle authorization, authentication, routing, and versioning.

For internal APIs, clients invoke lambda functions directly from within the client app using AWS’ SDK.

Internal API — The client invokes Lambda functions via the AWS SDK of your favorite language.

AWS Lambda scales automatically, and can handle fluctuating loads. Execution time is limited to the maximum allowed by API Gateway (29 seconds), or the hard limit of 15 minutes if invoked with the SDK.

It’s highly recommended that functions are stateless, sessions are cached and connections to downstream services are handled appropriately.

Decoupled Messaging

“As break out the monolith or continue to build services for our platform, different services have to interact. We want to avoid bottlenecks, synchronous I/O, and shared state.”

Asynchronous messaging is the foundation for most service integrations. It’s proven to be the best strategy for enterprise architectures. It allows building loosely-coupled architectures that overcome the limits of remote service communication, like latency and unreliability.

Decoupled Messaging with SNS

In the example above Service A is triggering events and SNS distributes these events to other services. When new services are deployed they can subscribe to the channel, and start getting messages as events happen.

Messaging infrastructure is reliable. Offering better encapsulation than a shared database. Perfect for asynchronous, event-based communication.

Different technologies have different constraints and offer specific guarantees. It’s important to be aware of these trade-offs. The most common services for messaging on AWS are SNS, SQS and Kinesis.

Considerations : Besides the known limitation and guarantees of each messaging service there should be conscious considerations for message duplication, message ordering, poisonous messages, sharding, and data retention. Microsoft’s “Asynchronous Messaging Primer” article discuses these topics in detail.

Robust API

“As the number of services of a platform grows clients need to interact with more and more services. Because each service might have different endpoints, belong to different teams, and have different release cycles, having the client manage these connections can be challenging.”

In a this pattern we have a single point of entry that exposes a well defined API, and routes requests to downstream services based on endpoints, routes, methods, and client features.

A Robust API composed of different downstream services, exposed via API Gateway.

The downstream services could be lambda functions, external third-party APIs, Fargate containers, full-blown microservices, even internal APIs.

API Gateway, App Load Balancer or AppSync can be leveraged to implement it as the routing engine.

AppSync (GraphQL) can be seen as an implementation of the Robust API and Aggregator patterns.

Having an abstraction layer between clients and downstream services facilitates incremental updates, rolling releases, and parallel versioning. Downstream services can be owned by different teams, decoupling release cycles, reducing the need for cross-team coordination, and improving API lifecycle and evolvability.

This pattern is also known as the API Gateway or Gateway Router.

Considerations : The Gateway can be a single point of failure. Managing resources and updating the client-facing interface can be tricky. It could become a cross-team bottleneck if it’s not managed via code automation.

Aggregator

“A client might need to make multiple calls to various backend services to perform a single operation. This chattines impacts performance and scale.”

Microservices have made communication overhead a common problem. Applications need to communicate with many smaller services, having a higher amount of cross-service calls.

A service centralizes client requests to reduce the impact of communication overhead. It decomposes and makes the requests to downstream services, collects and stores responses as they arrive, aggregates them, and returns it to the caller as one response.

The invoice lambda function aggregating downstream services.

Microsoft calls this pattern Gateway Aggregation. It can be implemented as a service with some business logic, that is able to cache responses and knows what to do when downstream services fail.

Considerations : Calls to the API should work as one operation. The API of the aggregator should not expose details of the services behind it to the client. The aggregator can be a single point of failure, and if it’s not close enough other services it can cause performance issues. The aggregator is responsible for handling retries, circuit breaking, cache, tracing, and logging.

Pub/Sub

“Services rarely live in isolation. The platform grows and services proliferate. We need services to interact without creating interdependency.

Asynchronous messaging enables services to announce events to multiple interested consumers without coupling.”

In Publisher-Subscriber services publish events through a channel as messages. Multiple interested consumers listen to the events by subscribing to these channels.

The Orders service interacting with the Video Wall, Leaderboard, and Recommendations Engine.

In the example the Orders service will publish an event when an order is created in the mobile application.

The VideoWall service is listening to OrderCreated events. It will take the order, break apart the items in the order — what needs preparation like a v60, or does not like cookies, and update the screen so our Baristas see which coffee to start brewing next.

The Leaderboard service will receive the same event, and will update the tally — Who is the Coffee Afficionado of the month, and what are the Top Selling Coffee Origins, Methods and Goodies.

The Recommendations service will keep track of who is drinking what, things that seem to go great together, and which is the next method you should try. — Hint, try the Japanese Syphon:

https://medium.com/media/c5ae351f5c6bbf46de15cc3c5fa8f498/href

Services are decoupled. They work together by observing and reacting to the environment, and each other — like rappers freestyling.

When new services and features are available they can subscribe, get events, and evolve independently.

Teams can focus on delivering value improving their core capabilities, without having to be focus on the complexity of the platform as a whole.

Considerations: Publisher / Subscriber is a great match for event-driven architectures. With lots of different options for messaging: SNS, Kinesis, Azure’s Service Bus, Google’s Cloud Pub/Sub, Kafka, Pulsar, etc. These messaging services will take care of the infrastructure part of pub/sub, but given the asynchronous nature of messaging all the issues discussed previously — message ordering, duplication, expiration, idempotency, and eventual consistency, should be considered in the implementation.

Strangler

“Systems grow and evolve over time. As complexity increases adding new features can be challenging. Completely replacing a system can take time, and starting from scratch is almost universally a bad idea.

Migrating gradually to a new system, while keeping the old system to handle the features that haven’t been implemented is a better path.

But clients need to know about both systems, and update every time a feature has been migrated.”

The Strangler pattern is a technique to gradually migrate legacy systems that was popularized with the exponential popularity of microservices. In this pattern a service acts as a façade that intercepts requests from the clients, and routes them to either the legacy service or new services.

Clients continue to use the same interface, unaware of the migration minimizing the possible impact — and the risk of the migration.

ALB routing requests between the legacy app and the new microservice.

An Application Load Balancer routes clients’ requests to the Orders Service, the first microservice the team implemented. Everything else continues to go to the Legacy application.

Orders has its own data store, and implements all the business logic for Orders. Because some of the features on the legacy app use orders, we need to push the data back to the legacy app to stay in sync (an Anti-corruption Layer of sorts).

New feature — Leaderboard Microservice now available.

As the project evolves, new features come in and we create new services.

The Leaderboard Service is now available. It’s a completely new feature that the brand new Engagement Team created so there’s no need to interact with the Legacy app.

Teams will continue to create new features and port existing features to new services. When all the required functionality has been phased out of the legacy monolithic app it can be decommissioned.

Considerations: The façade keeps evolving side by side with the new services. Some data stores can potentially be used by both new and legacy services. New services should be structured in a way that they can be intercepted easily. At some point the migration should be complete, and the strangler façade should either go away, or evolve into a gateway or an adaptor.

Queue-based Load Leveling

“Load can be unpredictable, some services cannot scale when there’s an intermittent heavy load. Peaks in demand can cause overload, flooding downstream services causing them to fail.

Introducing a queue between the services that acts as a buffer can alleviate the issues. Storing messages and allowing consumers to process the load at their own pace.”

Buffering services with the help of a queue is very common. In Microsoft’s Cloud Architecture Patterns it’s called the queue-based load leveling pattern, Yan Cui calls it Decoupled Invocation, and Jeremy Daly calls it the Scalable Webhook.

A queue decouples tasks from services, creating a buffer that holds requests for less scalable backends or third-party services.

Regardless of the volume of requests the processing load is driven by the consumers. Low concurrency and batch size can control the workload.

A migration process that consumes items really fast from Elasticsearch, pushes them in a queue while a worker slowly feeds them one by one into a Legacy app.

In the example we have a Migration worker process that reads the content of an Elasticsearch index. Elasticsearch it’s fast. The worker can read thousands of Articles and fetch their dependencies (Authors, Categories, Tags, Shows, Assets, etc.) in less than a second.

On the right side we have a service that needs to ingest all the content, but before we can create an Article we have to create all its relationships in a specific order, double-checking if they exist, and need to be updated which is slower. Even if we scale horizontally the service — and we did — the relational database behind it becomes the bottle neck.

After a point (~100k to 500k Articles) querying the database slows down to a crawl because there’s some locking on the Has-and-Belongs-to-Many relationship tables.

By limiting the batch size and the number of workers running concurrently we can maintain a slow but steady flow that reduces lock contention in the database.

Buffering requests of a busy public API.

Another common example is using a SQS to buffer API requests to amortize spikes in traffic— like in the diagram above.

The endpoint returns 202 — Accepted to the client, with a transaction id and a location for the result. On the client-side the UI can give feedback to the user emulating the expected behavior.

The service can process the requests on the background at its own peace. Even if there are long running process involved an increase in the load on the client-side will never affect the throughput and responsiveness of the system.

Considerations: This pattern is not helpful when services require a synchronous response (waiting for a reply). It’s important to note that not using concurrency limits can diminish the effectiveness of the pattern: AWS Lambda and Kinesis can scale quickly, overwhelming downstream services that are less elastic or slower to scale. Zalando’s API Guidelines includes a full section about Events that talks about this some important considerations for this pattern.

Read-heavy reporting engine

“It’s very common with read-heavy applications to hit the limits of downstream data engines that are not specialized for the different querying patterns that clients use.

Caching data and creating specialized views of the most queried data can help mitigate the load impact of a read-heavy service.”

Previous patterns help address pushing data and events from one service to others, optimizing scale for write-heavy services. This pattern optimizes the data for different accessing and querying patterns that clients need, optimizing scale for read-heavy services.

Most applications are read-intensive. This is particularly true for our Big Media clients where there are far less users generating content than users consuming the content. The ratio between them can be huge like 1:100000.

Caching data and creating specialized views of the most frequent access patterns help services scale effectively.

Caching means temporarily copying frequently used into memory or a shared repository. Caching data is one of the most frequently used strategies to improve performance and scale reads.

In Materialized views the data is replicated and transformed in a way that is optimized for querying and displaying. Materialized views can be new tables or completely different data stores where the data is mapped to be displayed in a new format or limited to a specific subset of the data.

Materialized views are also useful to “bridge” different stores to take advantage of their stronger capabilities.

With Index Tables the data is replicated in a new table using specialized indexes specific to common queries. Specialized indexes can be composite keys, secondary keys, and partially denormalized that.

Using Materialized Views and Index Tables for popular querying patterns on the social app.

Dynamo streams and lambda functions are the perfect tools to create specialized views. In the example we have three endpoints — search, tweet and timeline. Each one needs a slightly different querying pattern where the data needs to be optimized in a particular way.

/search queries the Tweets Index in Elasticsearch. Providing things like phonetic search, typos, related terms, and suggestions. There is no need to index all the data from the original tweet, maybe it only includes the text, location, media url — for a pretty preview, and hashtags. We use the stream to trigger a lambda on TweetCreated that strips all the data we don’t need and indexes the tweet.

/timeline is created from the most interesting tweets on the network, and the activity of my connections.

We use a Dynamo table to keep the Top Tweets — an indexed table limited to the 1000 most viewed items. Tweets are updated via the stream on theTweetViewed event. A lambda function receives the event, queries the Tweets Collection, and saves the result.

Getting the activity of someone’s connection is easier on a Graph Database like Neptune. Another lambda triggered by the TweetCreated event creates a record on Neptune maintaining activities for our connection’s streams of tweets.

In Microsoft’s Cloud Architecture this is a mix of different patterns — Data pipeline, Cache-aside, Materialized views, and Index Table.

Considerations: Handling cache can be hard, remember to follow industry best practices. Maintaining materialized views, and index tables can be complicated. Duplicating the data will add cost, effort, and logic. With very large datasets it can be very difficult to maintain consistency, and keeping the data in sync could slow down the system. Views and Index tables are not fully consistent.

Streams and Pipelines

“A continuous stream processor that captures large volumes of events or data, and distributes them to different services or data stores as fast as they come.”

An important feature we see often is processing streams of data, as fast as the data is being generated and scaling quickly to meet demand of large volumes of events. Downstream services receive the stream and apply business logic to transform, analyze or distribute the data.

Common examples are capturing user-behavior like clickstreams and UI interactions. Data for analytics. Data from IoT sensors, etc.

Using Kinesis Streams, Analytics and Athena to build a data pipeline.

In the example a Kinesis Stream receives events from Service A. The data is transformed with lambda functions, stored in DynamoDB for fast reading, and indexed in Elasticsearch for a good search user-experience.

Kinesis Analytics provides fast querying data in the stream in real time. With the S3 integration all the data is stored for future analysis and real insight. Athena provides querying for all the historical data.

Considerations: Stream processing can be expensive, it might not be worth it if there dataset is small or there are only a few events. All cloud providers have offerings for Data pipelines, Stream processing, and Analytics but they might not integrate well with services that are not part of their ecosystem.

Fan-out and Fan-in

“Large jobs or tasks can easily exceed the execution time limit on Lambda functions. Using a divide-and-conquer strategy can help mitigate the issue.

The work is split between different lambda workers. Each worker will process the job asynchronously and save its subset of the result in a common repository.

The final result can be gathered and stitched together by another process or it can be queried from the repository itself.”

Fan-out and Fan-in refers to breaking a task in subtasks, executing multiple functions concurrently, and then aggregating the result.

They’re two patterns that are used together. The Fan-out pattern messages are delivered to workers, each receiving a partitioned subtask of the original tasks. The Fan-in pattern collects the result of all individual workers, aggregating it, storing it, and sending an event signaling the work is done.

The backend engine of a Digital Assets Management System (DAM) — A classic example of serverless workloads with a twist.

Resizing images is one of the most common examples on using serverless. This is the fan-out approach.

A client uploads a raw image to the Assets S3 Bucket. API Gateway has an integration to handle uploading directly to S3.

A lambda function is triggered by S3. Having one lambda function to do all the work can lead to limit issues. Instead, the lambda pushes an Asset Created event to SNS so our processing lambdas get to work.

There are three lambda functions for resizing — on the right. Each creates a different image size, writing the result on the Renditions bucket.

The lambda on the bottom reads the metadata from the original source — location, author, date, camera, size, etc. and adds the new asset to the DAM’s Assets Table on DynamoDB, but doesn’t mark it as ready for use. Smart auto-tagging, text extraction and content moderation could be added to processing lambdas with Rekognition later.

The Manager App consumes assets via the DAM API.

For the Fan-in part of the DAM, we have a lambda function that’s listening to the renditions bucket, when there’s a change on the bucket it checks if all renditions are ready, and marks the assets ready for use.

With the event-driven nature of serverless, and given the resource limits of lambda functions we favor this type of choreography over orchestration.

Considerations : Not all workloads can be split in small enough pieces for lambda functions. Failure should be considered on both flows, otherwise a task might stay unfinished forever. Leverage lambda’s retry strategies and Dead-Letter Queues. Any task that can take over 15 minutes should use containers instead of lambda functions, sticking to the choreography approach.

More resources 📚

Microsoft’s Azure Architecture Center has an extensive list of Cloud Patterns, a solid guide for best practices, a very good overview of performance anti-patterns and some examples.
AWS also has great resources on Distributed Systems Components, Serverless in general, examples in the Serverless Application Repository, and several webinar videos.
Jeremy Daly has an extensive list of Serverless Patterns. He’s the author of Lambda API a lightweight API framework, and the Serverless MySQL module for managing connections on your lambda functions.
Yan Cui has a tonne of content about serverless. Including Design patterns, performance, lots of points of view, and the best training on serverless. He’s must-follow on the serverless scene.
Mike Roberts writes and talks about serverless in depth.
Sascha ☁ Möllering talks about Serverless Architectural Patterns and Best Practices.
Rob Gruhl’s Event-sourcing at Nordstrom, the post about Hello Retail!, and the video about their architecture.
swardley and Forrest Brazeal discussion about Containers vs Serverless.
Gregor Hohpe’s Enterprise Integration Patterns is a catalog of 65 technology-independent patterns, some of which he’s ported serverless in Google’s cloud.
Twitch’s Design Patterns for Twitch Scale touches on several interesting points about patterns that can apply for scale.
Capital One Tech “When to React vs. Orchestrate” is an excellent post about choreography vs orchestration.

A tale of a dying monolith: The complexity of replacing something simple

Eduardo Romero — Tue, 25 Sep 2018 14:06:02 +0000

A tale of a dying monolith: The complexity of replacing something simple

Photo by Thor Alvis on Unsplash

I’ve been working on Big Media for almost two years now. Two of my favorite projects were building publishing platforms. Both leveraged the Headless CMS architecture and developed a platform around it.

While the stacks were entirely different, the patterns between architectural components seemed the same.

I started to identify the same patterns in older systems I had worked with before, and I wanted to know if I could apply the same patterns to a legacy system.

A friend was building new versions of the Android and iOS Apps for an old CakePHP app in the Healthcare industry I had helped develop almost five years ago. He wanted to add new features to the apps and needed the backend to support the new features.

The CakePHP App interacted with data coming from an external system. It was the source of truth for most of the entities in the app. Both systems were tightly coupled, sharing data via database tables.

CakePHP Monolith tightly-coupled to an external system via the Database.

Data for things like product names, system ids, pricing, discounts, product status, user accounts, membership status, and stock levels come from the external system.

Doctors and practitioners would log in to the CakePHP App and add extra information to the data coming from the external system.

Models for product descriptions, pictures, calendars, procedures, locations, profiles, etc. only existed in the App, and that data was created and managed there. The App would render the content and exposed the data via a REST API to mobile and third-party apps. One big, mighty monolith from the good ole’ days.

I convinced my friend to let me start building experiments around the project. I began strangling the monolith. After several iterations, this is the resulting architecture:

The New Stack — A Highly decoupled architecture with a Headless CMS, an external search engine and real-time API.

The Platform — 🕸

The new Platform is an event-driven, distributed system. A FeathersJS RESTful layer in the middle of the platform. I offloaded all content related responsibilities to Prismic.io, the CMS functionality on CakePHP was deprecated. I used Algolia to power the search engine on all apps, including the API.

Based on the events of the API I can do things like index new content or remove content from the search engine, update pricing, stock levels, enable or disable content, and bake in some default data in case an entity does not exist on the CMS yet.

As a nice bonus, the mobile apps can now give notifications in real time to the users about things that are happening in other parts of the platform.

No shared databases — 💥

I started by removing the database dependency between systems. Now all systems integrate via RESTful API endpoints.

The coupled integration was the source of a lot of pain-points:

Schema changes from external updates : Whenever the external system’s schema was updated the backend needed to be updated too. Some of those changes could tickle all the way up to the mobile apps.
Cascaded down-time : Because the external system was the source of truth for things like membership status and identity when it went down for a reason it would cause partial failures on other systems. Local cache helps but only to some extent.
Batch syncs : Because of the systems integrated via the database some things needed synchronization with daily jobs. Stuff like price lists, stock levels, status (i.e., discontinued items), and search indexing on Lucene all had to run every day at the end or beginning of the day.
Ownership : Because there was very little visibility when something was wrong there was a lot of blame-game between maintainers of the external system and the backend.

Now the external system pushes changes via the REST API. They exposed a SOAP API for User Membership which the Feathers backend consumes and proxies as JSON API for external clients.

The Headless CMS — 🖋

I modeled all Content entities in Prismic. Users can now create and manage content and see it magically appear on all apps in near real-time.

I kept Cake’s frontend as is. Keeping it as the Rendering engine means that no extra effort needed in the frontend. In the future, they want to migrate the to VueJS or React.

Modeling Content in Prismic (sample data model)

I set up webhooks to sync that data back to CakePHP models. The content coming from Prismic will update existing models, handle external assets (backing them up locally) and update current state. Eventually, all content will exist in Prismic.

From previous experience, data migration is always a lot of work. In this case no data migration is needed.

JSON Response from Prismic’s API (from same model above).

The Search Engine — ️🔍

Working with Algolia is a breeze. It’s blazing fast and easy to use. Having an event-driven architecture turned out to be extremely beneficial. Events on the API use NodeJS’ clientto index the data asynchronously.

Search response time — Blazing fast, under 10ms.

In a few milliseconds, the search features on all applications will reflect the changes and show (or hide) the same results. SDKs for PHP, Swift, and Java made the integration easy.

Search Operations — Averages to 88k monthly operations.

For other existing clients that were not updated the CakePHP search continued to work as it was before powered by algolia behind the scenes. This means that other integrations like Chrome’s Omnibox search continued to work.

Continuous Deployment — 🚀

Having more pieces in the architecture means that deploying new versions of the platform can no longer be done by hand. I set up pipelines on Buddy to handle the deployments automatically.

The CakePHP app was already being deployed to an EC2 instance on AWS. I just had to automate the process (and added atomic deployments).

The FeathersJS app is deployed to the cloud with Zeit’s Now. A service for immutable infrastructure that makes it simple to deploy features quickly without hassle (or fear).

NodeJS Deployment Pipeline with Now

Merging a branch will create a new container tied to a new domain name where you can do any tests you need.

If everything works as expected, the last step will alias the current deploy to the backend’s public DNS name and it will automatically start receiving live traffic. It will also automatically scale the app and make sure it’s running in all available regions.

now scale

The configuration for aliasing to production and scaling rules are set in the repository as code. Things like database connection, and the access keys for third-party services are set as environment variables that are injected to now from the pipeline.

MySQL database lives on Amazon’s RDS. The RethinkDB database is hosted in compose.io. Using a three node cluster configuration plus a proxy portal node (and automated backups).

Debugging and Tracing —🕵🐛

Breaking the monolith apart made the overall architecture easy to understand, easier to maintain and evolve. But at the same time made debugging very hard. Trying to figure out what’s going on when an event crosses system boundaries was very challenging.

I had already some idea about observability and traceability because of my experience with AWS Lambda, X-Ray, and IOPipe. I really like IOPipe, but it only works for lambda. I remembered reading great insight on observability via Charity Majors’ twitter. I decided to try Honeycomb.io.

I instrumented the backend and the pipeline.

Request per second (purple) and Releases from GitHub (blue).

Feathers uses ExpressJS under the hood, so I just had to add their beeline for NodeJSand HTTP requests were handled out of the box.

NodeJS Honeycomb Beeline

I used before and after hooks to add extra context to the tracing information. That helped get better insight into what was going on in the platform and made it easier to build meaningful queries and create visualizations with values that made sense business-wise.

There is no beeline for PHP yet. But the Events API can be used with all other languages.

I also instrumented the pipeline so all releases are marked on honeycomb (the blue combs on the image above). I used the marker CLI for that. With pipelines already setup to run when code is merged to master and tagged for release, it was just running one extra step with the following:

$ honeymarker -k $HONEYCOMB\_KEY -d feathers-ws \
    -t release \
    -m "$EXECUTION\_TAG" \
    add

Triggers can be leveraged to keep track of any unexpected behavior across the platform. I expect that in the future Triggers will be used to track behavior after an update performing roll-backs or aliasing the release automatically.

I learned a lot! The platform is robust, decoupled and scalable. It helped me validate my theory around content platforms and helped recognize reusable patterns I can apply in different projects easily.

On the other hand, it made me realize that replacing a monolith with a distributed system helps in the long run but adds a lot of complexity that can be cumbersome if you don’t have the right tools support it like CI/CD, event logs and tracing.

I think observability is a key success factor when building service-oriented architectures like microservices or Lambda-based architectures. I’ll continue to dig more into it.

Finally, the team that will continue to support the platform was also delighted with the outcome. It’s now easier to add new features. Understanding the components is straightforward, more than the original codebase was. They are already thinking of what the next refactor will be — The frontend. The organization has less aversion to change, and new ideas are already on the pipeline, overall they are moving faster, and there’s a new sense of trust in the engineering team.

More resources 📚

How to break a Monolith by Zhamak Dehghani explains how to migrate a monolith to a microservices architecture. The folks from Nginx have a great article series about Microservices, last one explains how to refactor a Monolith into Microservices.
Buddy is my favorite CI/CD tool. It’s easy to use and it has all the cool features like integration with GitHub, deploying to k8s (ECS, GKS), running steps in containers and ephemeral environments they call Sandboxes. Read their guides for more information.
FeathersJS is a Real-time REST API framework for NodeJS. It’s my favorite framework to develop NodeJS APIs. NestJS is second.
Honeycomb’s blog has very good pointers on Observability. They even sponsored this year’s (and first ever) o11ly con.
New Relic has support for Custom Metrics and Distributed tracing, but I haven’t tried. Seems more complicated. I will give it a try because my clients use it.
Martin Fowler has a very good post about Event-driven patterns and a presentation about “headless cms”
Human Made has a good booklet about Headless CMS with WordPress. Concepts apply to any CMS.

Thanks to Felipe Guizar Diaz, Luis Daniel, and Miguel Lomelí for helping proof-read this article.

Fast API Prototyping with Webtask.io and Serverless.

Eduardo Romero — Thu, 28 Sep 2017 14:01:02 +0000

Photo by ShiroÂ hatori

Function as a Service (FaaS) commonly referred as serverless computing, feels like the answer to every engineer’s prayers. Simple functions, doing only one small thing and running in infrastructure that scales automatically for a small price, or even for free ðŸ˜!

Webtask is the Function as a Service implementation from the Auth0 team. It’s perfect for prototyping and easy to get started with. They just released their Serverless integration last week.

I’m working with Serverless on my current project, and I’ve wanted to take Webtask for a spin for a while, so I decided this was the perfect time for it.

In a few minutes, I had a REST API, deployed, and running with the help of Express, RethinkDB, Webtask, and Serverless.

Getting started ðŸ

Install the serverless framework and create a new project:

Go into the project’s folder and add the dependencies with NPM:

express and body-parser will handle routing and JSON responses.
webtask-tools wraps the express app and binds it to the webtask handler.
rethinkdbdash is a RethinkDB driver; it has some cool features like connection pooling, an easier interface, and outstanding performance.

Update serverless.yml. Change the service name and add the IP of the RethinkDB server:

The only differences from the default config should be the service name and the environment and defaults options.

The options tell the provider (webtasks in this case) to set RETHINKDB_SERVER as an environment variable and to take its value from either the environment (env), an option from the command line when deploying (--rethinkdb-server) or, if neither is set, from the defaults section.

The API will be running from Auth0’s FaaS infrastructure. RethinkDB needs to be reachable from the internet. I launched a Digital Ocean instance for that.

The final step is to set up a Webtask account:

Show me the code! ðŸ¤“

At this point, there is already a Function that can be deployed, run and tested. Give it a try:

It is important to run npm update to install all packages before deploying a function. Dependencies will be pulled from node_modules and uploaded with the function.

Running sls deploy will return the URL of the function ðŸŽ‰â€Š–â€ŠReturns a JSON response with a successÂ message.

The service is running already and zero code. Wow! That is really something!

I realized that this was an excellent first approach to the FaaS paradigm. Definitely easier than setting up an account for AWS Î».

Now into the actual code…

Express + RethinkDB

Like most express apps, start by adding express, body-parser, and webtask-tools. Create the app and add the database as a middleware to all routes:

The RethinkDB instance is now available as db through every endpoint.

The API

GET /: Retrieves all documents from RethinkDB.
GET /🆔 Returns a particular document with the given id.

It is a simplistic version with no filtering and basic pagination.

The endpoint will get the response from RethinkDB and return it to the client. If anything goes wrong, it will send the error instead.

POST /: Creates a new document.
PUT /: Updates an existing document.

Creating and modifying data is just as simple.

The endpoint gets the JSON from the request and saves it into the shares table. The code is almost the same for both functions.

RethinkDB has support for upserts. When the conflict option is set to update if the body of the request has an existing id it will replace the document. The default behavior is to throw an error if the document with that id exists, which is what the POSTs method is doing.

With returnChanges, the query will return an Array with the resulting changes of the operation in a special object with two properties: new_val and old_val. We use new_val to return the upserted item in the response. In the case of a POST, the object will include the id of the new document.

RethinkDB accepts an Array or an Item when inserting. Lines 6 and 11 normalize the response. If the request had an array, it would return an array with the changes. If it were just one item, it would return only one item.

Besides returning the actual data, it includes the response from RethinkDB. It contains an object with helpful information that looks like this:

"response": {
    "changes": [
        {
            "new_val": {
                "id": "044ec21c-78b9-47aa-9eb6-e7e5658237db",
                "name": "Tania ðŸ˜"
            },
            "old_val": {
                "id": "044ec21c-78b9-47aa-9eb6-e7e5658237db",
                "name": "Tania"
            }
        }
    ],
    "deleted": 0,
    "errors": 0,
    "inserted": 0,
    "replaced": 1,
    "skipped": 0,
    "unchanged": 0
}

It can be helpful for the requester to know what happened to the data.

DELETE /🆔 Removes the document with the given id from the table.

The endpoint gets the id from the request, and ask RethinkDB to remove that document. If anything goes wrong (i.e., the id doesn’t exist) it sends the error back to the client.

That’s all. A fully working –barebonesâ€Š–â€Šfunction as a service-based REST API. The code is straightforward. Small functions that are easy to read and understand.

Next steps will be adding Authentication to the requests and some business logic so the service is more helpful.

I will continue working on it and will elaborate about it in another post.

Webtask and Serverless make it easy to start “dipping your toes” into the Function as a Service world. It focuses around NodeJS with only two event sources HTTP Requests and Scheduled events. Compared to AWS Lambda it’s faster to get started and easier to approach.

Webtask’s free tier is limited to a request per second. But that should be enough for testing; maybe even for a basic service.

More resources ðŸ“š

Webtask.io, the FaaS from the Auth0 guys. It’s easy and straightforward. Works with Node 8 out of the box and it takes less than a minute to get started. Super useful to get started with Slack hooks, Bots, and APIs. It has a fully-featured Web Editor that makes things even simpler. It has real-time logs, 500k of JSON storage, a cron-like scheduler, and Auth0 support is baked in.
Serverless is a toolkit to work with serverless architectures from any provider (AWS Lambda, Google Functions, Azure Functions, OpenWhisk, Webtask, etc.). With the help of its Event Gateway, you can even combine different providers and make them all work together.
RethinkDB is an Open Source NoSQL Database with a very nice querying language (ReQL) and advanced features like clustering, near-linear scaling, and real-time feeds. It’s my go-to NoSQL data store.
Thinky a Javascript ORM for RethinkDB. An easier way to map skinny JSON objects to RethinkDB documents.

Thanks to Rafa Salazar, David NÃºÃ±ez, Andres Cespedes and Martin Moscosa for helping with this article.

Continuous Engineering with the Serverless Framework

Eduardo Romero — Thu, 03 Aug 2017 19:58:48 +0000

We are working with microservices, and I’m enjoying it. Our implementation is based on AWS Lambda serverless compute. This technology launched in 2014, now in 2017 there are still no well-established engineering best practices for it.

We always follow best practices when we develop a product. In this post, I describe how we apply our workflow to the function as a service paradigm.

Workflow

We use feature branches. Each story gets its branch. When the code is ready, we create a Pull Request (PR) to have changes merged to develop.

The team peer-reviews the code and we suggest changes when needed. We require at least two approvals before merging changes.

Before changes get merged, CI/CD runs. It checks that we stick to our coding standard and runs our tests. If everything passes, you can merge your branch.

When it is merged to develop it gets auto-deployed to our development environment on AWS.

Code Structure

Working with Functions as a Service (FaaS) means coding simpler, easy to understand modules.

Each microservice has its own repository. Inside it we structured our code around the subdomain it belongs to. Clustering all related functions of a service together in the same folder.

We are using the Serverless Framework to manage, build and deploy our functions as a service microservices.

We create one service per subdomain, with its own serverless.yml config file, external packages it needs (package.json) and several JavaScript files that implement its functionality.

Events Definitions

Function as a Service is an event driven paradigm. Functions get spawned in response to an event.

The event definition goes in the serverless.yml config file for each service. It can be an HTTP request through API Gateway, a file uploaded to S3, a record updated in Aurora DB, a document inserted in DynamoDB, etc.

We are currently only using HTTP, Kinesis streams and SNS events for our project. The framework has support for more event sources.

Environment Variables

We use environment variables as much as we can. ARNs, Endpoints, AWS Credentials, AWS Region, etc. Serverless lets you define resources from environment variables. It will let you know if any of the variables are not defined in your environment before it tries to deploy your lambdas.

Serverless Framework complaining about missing environment variables

The team shares environment variables with the help of Torus. Our CI/CD tool reads these environment variables to deploy our lambdas automatically when we approve a PR.

Release / Deploy

We use Continuous Integration and Deployment. Code Linting and Tests runs before a PR can be merged. Deployments to dev and staging are automated.

Deploying with CI/CD + Bash + Serverless (staging)

Our CI/CD pipeline runs a basic NodeJS docker image, finds all our services, goes into each folder, runs yarn to install all dependencies and deploys to AWS. Nothing particularly sophisticated.

A few bash get run commands, et voilÃ ! A fully functional set of lambdas running.

Result of our CI/CD yarn + slsÂ run.

Everything’s been taken care of ðŸŽ‰.

If everything goes well, we will get a message on Slack that everything’s been taken care of. All services are up to date.

At this point we are ready to work on the next feature and continue the development cycle.

More resources ðŸ“š

The Serverless Framework and many examples.
Micro async http node microservices. Great first approach to self-hosted simple functions to build services.
Apex another framework for managing FaaS. It has runtimes for Go, Clojure, and Rust, so you can write functions on those languages that are not supported by AWS Lambda.
WebTasks FaaS by the Auth0 team. Integrates really well with Slack and GitHub. Has Cron-like support for scheduled tasks.
stdlib a slightly different approach. They strive to become a Standard Library for FaaS. Has a Service Directory and functions can be invoked from regular NodeJS, Python or Ruby projects.

DEV Community: Eduardo Romero

Serverless Architectural Patterns

Serverless Architectural Patterns Catalog

Simple Web Service

Decoupled Messaging

Robust API

Aggregator

Pub/Sub

Strangler

Queue-based Load Leveling

Read-heavy reporting engine

Streams and Pipelines

Fan-out and Fan-in

More resources 📚

A tale of a dying monolith: The complexity of replacing something simple

A tale of a dying monolith: The complexity of replacing something simple

The Platform — 🕸

No shared databases — 💥

The Headless CMS — 🖋

The Search Engine — ️🔍

Continuous Deployment — 🚀

Debugging and Tracing —🕵🐛

More resources 📚

Fast API Prototyping with Webtask.io and Serverless.

Getting started ðŸ

Show me the code! ðŸ¤“

Express + RethinkDB

The API

More resources ðŸ“š

Continuous Engineering with the Serverless Framework

Workflow

Code Structure

Events Definitions

Environment Variables

Release / Deploy

More resources ðŸ“š

Getting started ðŸ