Bobby Iliev

Posted on Mar 10, 2022 • Edited on Sep 16 • Originally published at devdojo.com

Blue-green deployments with Materialize

#devops #database #kubernetes #materialize

Introduction

[!WARNING]
This demo includes examples for an unsupported version of Materialize (0.26.x).

For a more up to date guide checkout the documentation here:
https://materialize.com/docs/manage/dbt/development-workflows/

Minimizing downtime during any deployment is a key part of any successful deployment strategy. There are many ways to achieve this, and blue-green deployments are one of them.

Using blue-green deployments allow you to not only minimize downtime but also give you the ability to roll back to a previous version of your application if something goes wrong.

In this article, we will go over how to deploy Materialize in a blue-green deployment strategy.

Prerequisites

Before getting started, you should have the following:

Basic knowledge of Materialize
Materialize Cloud Account - The free trial includes access to two instances, so you should be able to test out a blue/green architecture for free.
psql or mzcli installed on your machine
DNS Service Provider like Cloudflare, AWS Route 53, Google Cloud DNS, or any other DNS service provider

What is a Blue-Green Deployment?

A blue-green deployment strategy allows you to deploy a new version of your application while keeping the old version running. Once the new version is ready, traffic is switched over to it via an update to load balancer rules or DNS. The old version is kept running in the background until the new version has successfully taken over. Once we are sure that the new version is stable, the old instance can be destroyed.

As an example, take a look at the following diagram:

Here, we have two instances of Materialize running on the Materialize Cloud. The blue instance is the one that is currently running and is handling the traffic as this is what we've configured in our DNS. The green instance is the one that is being deployed and would start handling the traffic once the deployment is complete and once we change the DNS to point to the green instance.

The whole process usually goes like this:

You would initially have the blue instance running as follows:

That single instance is the one that is currently handling the traffic and doing all the work.

You would then deploy a new version of Materialize to the green instance:

After the deployment is complete, you would need to create all of the necessary Materialize sources and views.

After the deployment is complete and all of your data has been ingested, you would then switch the DNS to point to the green instance:

Once the DNS is updated, the traffic is switched over to the green instance. However, keep in mind that it might take a while for the DNS to propagate before the green instance starts handling all of the traffic.

During the DNS propagation, the blue instance will continue to handle some of the traffic.

Once the DNS is updated, the traffic is switched over to the green instance:

During that time, the blue instance will no longer be handling any of the traffic and can be shut down. But before shutting down the blue instance, we need to make sure that the green instance is up and running and we've tested it thoroughly.

Finally, we can shut down the blue instance completely:

That way the end-user will not experience any downtime during the deployment as there would have been always an instance running in the background.

Plan Your Deployment

The rest of this article is a hands-on walkthrough of a blue-green deployment of Materialize Cloud taking the following steps:

Create one instance of Materialize via the Materialize Cloud, this will be the blue instance
Create a DNS record pointing to the blue instance
Connect to the blue instance and create all of the necessary Materialize sources and views
Create a new instance of Materialize via the Materialize Cloud, this will be the green instance
At that point, only the blue instance is running and handling the traffic so next, we will switch the DNS to point to the green instance
Once the DNS is updated, the traffic is switched over to the green instance but keep in mind that it might take a while for the DNS to propagate before the green instance starts handling all of the traffic
Finally, we will shut down the blue instance completely once we are sure that the green instance is up and running and we've tested it thoroughly

Note that it is always a good idea to keep the old instance running in the background until you are sure that the new instance is performing well.

Creating Materialize Deployments

Create the blue instance via Materialize Cloud UI:

Click on the 'Create New Deployment' button
Choose the name of your deployment. We will use the name 'blue' for this deployment to make sure that we can easily identify it later
Next, choose the size and the region of your deployment
Finally, click on the 'Create' button

This will take only a few moments to complete.

Once the deployment is ready, we will be able to connect to the instance via any PostgreSQL client.

Connecting to Materialize

For this demo, we will use the psql command to connect to the instance.

Follow the instructions provided in the Cloud UI to download the certificates and connect to your new instance via psql on the command line.

Note: you'll have to change the sslmode=verify-full flag to sslmode=require when not connecting to the deployment hostname itself.

Adding Materialize Sources

For this demo, we will use the PubNub example from the Materialize Cloud documentation.

First, we need to create the PubNub source:

CREATE SOURCE market_orders_raw
FROM PUBNUB
SUBSCRIBE KEY 'sub-c-4377ab04-f100-11e3-bffd-02ee2ddab7fe'
CHANNEL 'pubnub-market-orders';

Then we can create a non-materialized view, which simply provides an alias for the embedded SELECT statement:

CREATE VIEW market_orders AS
SELECT
    ((text::jsonb)->>'bid_price')::float AS bid_price,
    (text::jsonb)->>'order_quantity' AS order_quantity,
    (text::jsonb)->>'symbol' AS symbol,
    (text::jsonb)->>'trade_type' AS trade_type,
    to_timestamp(((text::jsonb)->'timestamp')::bigint) AS ts,
    ''
FROM market_orders_raw;

And finally, create the materialized view that computes the average bid price:

CREATE MATERIALIZED VIEW avg_bid AS
SELECT symbol,
       AVG(bid_price) AS avg
FROM market_orders
GROUP BY symbol;

At last, you can check the results:

SELECT * FROM avg_bid;
  symbol    |        avg
------------+--------------------
Apple       | 199.3392717416626
Google      | 299.40371152970334
Elerium     | 155.04668809209852
Bespin Gas  | 202.0260593073953
Linen Cloth | 254.34273792647863

For more information about the PubNub source, check the Materialize Cloud PubNub example.

Create the route (via DNS or LB) to the blue instance

Once you have created the views, you can add a DNS record or a load balancer rule to point to the instance.

To do so you will need to go to your DNS provider and create the following record:

CNAME record for materialize.your_domain.com pointing to the Materialize instance hostname (e.g. 12345mz.materialize.cloud)

For example, if your instance hostname is my-instance.materialize.cloud, you will need to create a CNAME record pointing to my-instance.materialize.cloud as follows:

materialize.example.com. CNAME  12345mz.materialize.cloud.

Note change the 12345mz.materialize.cloud to the hostname of your instance.

This will allow you to access the instance via your own hostname: materialize.example.com.

Once the CNAME record is created, you can access the instance by using the same psql command as before but instead of the Materialize hostname, you can use the DNS name (e.g. materialize.example.com).

Alternatively, if you already have a load balancer, you can use it to create a route to point to the Materialize cloud instance.

Performing a Blue-Green Deployment

With all that setup, we can now perform a blue-green deployment. We will first create a new instance of Materialize via the Materialize Cloud just like we did for the first deployment.

Once the deployment is ready, we will be able to connect to the instance via any PostgreSQL client and again create the PunMub source and views as follows as we did for the first deployment:

Create the PubNub source:

CREATE SOURCE market_orders_raw
FROM PUBNUB
SUBSCRIBE KEY 'sub-c-4377ab04-f100-11e3-bffd-02ee2ddab7fe'
CHANNEL 'pubnub-market-orders';

Create the non-materialized view:

CREATE VIEW market_orders AS
SELECT
    ((text::jsonb)->>'bid_price')::float AS bid_price,
    (text::jsonb)->>'order_quantity' AS order_quantity,
    (text::jsonb)->>'symbol' AS symbol,
    (text::jsonb)->>'trade_type' AS trade_type,
    to_timestamp(((text::jsonb)->'timestamp')::bigint) AS ts
FROM market_orders_raw;

In most cases, you would proceed with a blue-green deployment when you have to make changes to the source code or the view definitions. One thing that we could do is to add a new column to the avg_bid materialized view. That way we would know which instance is currently handling the traffic:

CREATE MATERIALIZED VIEW avg_bid AS
SELECT symbol,
       trade_type,
       AVG(bid_price) AS avg
FROM market_orders
GROUP BY symbol, trade_type;

That way if we run the SELECT * FROM avg_bid; statement, we will see the new column trade_type which will tell us which instance is currently handling the traffic.

To monitor the status of your instance, you can follow the steps from the Materialize documentation here:

Materialize Cloud Monitoring

Once the results are correct, we can proceed with the blue-green deployment.

Update the route to point to green

Once you are ready, you can head back to your DNS zone and update the DNS record for the instance to point to the new instance hostname. Or in the case of a load balancer, you can update the route to point to the new instance.

For example, if your instance hostname is my-instance.materialize.cloud, you will need to update the CNAME record pointing to my-instance.materialize.cloud as follows:

materialize.example.com. CNAME  54321mz.materialize.cloud.  # Change this to the new hostname

Note change the 54321mz.materialize.cloud to the hostname of your new instance.

In case that you are using a public DNS provider, you can also reduce the TTL of the CNAME record to 1 hour to avoid the DNS provider from caching the record.

This will allow you to access the new instance via your own hostname: materialize.example.com directly once the DNS changes have been propagated.

Verifying the Deployment

To verify that we are successfully connected to the new instance, try to connect to the new instance via the psql command as before by using the DNS name (e.g. materialize.example.com) and run the following query:

SELECT * FROM avg_bid;

If you see the trade_type column, you know that you have successfully connected to the new instance.

Alternative approaches to the DNS change

The described approach using DNS changes and traffic routing is a good one to use but there are other ways to achieve the same result.

With the DNS approach, you will have to wait for the DNS propagation to finish before the green instance starts handling the traffic. During that time both the blue and green instances will be handling some of the traffic based on the DNS propagation.

This means that you don't have much control over the traffic routing and you will have to wait for the DNS propagation to finish before the green instance starts handling the traffic.

As we already mentioned throughout the article, alternatively, you can use a load balancer to route traffic. This is a good approach if you want to have more control over the traffic routing.

With a load balancer, you can do the switchover of the traffic to the green instance at any time and also revert to the blue instance if the green instance is not performing well quickly by making a single change to the load balancer configuration.

Downside of Blue-Green Deployments

Some of the downsides of a blue-green deployment strategy are:

The DNS change might take a while to propagate
The traffic might be routed to the blue instance for a while before the green instance starts handling the traffic
If you have Materialize Sinks you would need to plan how to handle the data that is being sent to those sinks while the two instances are running
During the deployment, you would have to have two instances running during the same time which could add some overhead
In case that you have a lot of data with a lot of changes, you might have to wait for the new Materialize instance to ingest the data which could take a while if there is a lot of backpressure.

Conclusion

This is just a brief overview of the steps that you need to take to deploy Materialize in a blue-green deployment strategy.

In this article, we were using the Materialize Cloud to deploy Materialize but this approach will work with any other Materialize deployment strategy.

If you are not part of the Materialize Slack community, please feel free to join here.

Useful links:

DEV Community