Newsletter Microservice using .NetCore, RedisJSON and Azure

Introduction

In this article, you will be creating a newsletter service based on microservices architecture using RedisJSON, .NET Core and Azure.
You will use RedisJSON as the persistence store for keeping our data related to Subscriptions. RedisJSON is a high performance NoSQL document based storage provided by Redis.
You will be creating REST Api using .NET Core which will provide CRUD operations around managing subscriptions and to create newsletter for the subscribed users.
You will also be creating a serverless function using Azure to process the newsletter and send it out to all users using SendGrid email service. It will be a complete end to end hands on tutorial.

Prerequisites

1. You will be using Redis OM .NET library for handling documents in Redis. For the you will use .NET Core 6.
2. An IDE for writing .NET Code. Visual Studio, VS Code or Rider, any of them will work.
3. An Azure account for Storage Queue and SendGrid for sending emails. You can create a free Azure account here.
4. Azure Functions Core Tools

Design of Newsletter Microservice

In this article, You will be designing a Newsletter microservice which will enable users to subscribe or unsubscribe from a newsletter and allows the owner of the newsletter to send the email when they want to.
You will create a REST API which will provide operations of subscribing/unsubscribing from a newsletter and also to send a newsletter. In order to send the newsletter you will create an asynchronous operation where you will persist the newsletter information on a queue(Azure storage queue) and you will be creating a newsletter processor which will pick the newsletter items from the queue and then send email to the subscribed members using SendGrid.
Here the most important thing is theentities. You need to design some entities for storing our data related to Subscriptions and Newsletter.
Below is the Subscription entity which will serve our purpose to store data related to subscriptions in Redis.

string Id
string Email
DateTime SubscribedOn
bool IsSubscribed

Below is the Newsletter entity which will serve our purpose to keep data related to newsletter and it will be put on the queue when user requests to create newsletter. You will not be storing this info as its just to send the mails.(for keeping design simple)

List<string> Emails
string Subject
string Body

After designing entity, you will be creating 3 REST endpoints for Subscribing, Unsubscribing to newsletter and Sending newsletter. After that you will create an Azure function to process the newsletter i.e. to send the email.
In the next sections you will go through all the steps in details. Lets dive in :)

Setting up Redis Cluster

Signup for Redis cloud using this link. They are providing 200$ free credits to start via coupon code TIGER200.
Once you signup and login. When you sign in, you will be asked to choose cloud provider from Azure, google cloud, aws. Choose any one which you prefer and after that a free subscription will be created for you.
Once the subscription is created, a database will also be created for you which has modules RedisJSON, Redis Search , Redis Bloom, Redis Graph and Redis TimeSeries. For our app you need Redis JSON and Redis Search only. So you will delete the existing one and create a new database.
In order to delete, click on databases on left side of options under Subscriptions. You will get to see the default database which is created for you. After that click on the database name and you will land on database configuration page. Scroll down the page and you will get to see the delete option under Danger Zone at the bottom.

After that you need to create a new database. For that click on Subscription options again. Now under your subscription there is no database and you will be getting an option New Database button to create database. Click on that and you will be getting options to set name for your database and you will be getting the type to set for your database. Here choose Redis instead of Redis Stack and select RedisJSON and RedisSearch. after that click Activate and your database will be ready to use in few minutes.
Once your database is ready, keep the public endpoint and password value handy at one place from general and security sections respective as you will be using it later for creating our connection string.

Setting up Azure account for SendGrid And Storage Queue

Once you sign up and login to your Azure account, go to the Azure Portal, search for "SendGrid" in the top search bar and click on it. Here you will be creating a resource for Twilio SendGrid SaaS (software as a Service). You will land on the resource creation page where you need to fill in some details regarding the name of the resource, plan, etc.
Provide a name for this resource and choose the plan "Free 100 -1 month" as this will be free of cost, and you will be getting 100 emails per day with this plan which is enough for this tutorial. After that, click on Review + Subscribe and you will land on Review + Subscribe tab where you can review the details you filled out. Click on the subscribe button and Azure will create a SendGrid SaaS subscription for you (it will take a few minutes to be ready to use).
When it's ready, you need to configure the SendGrid account, so click on the "Configure account now" button.

Once you click on the "Configure account now" button you will be redirected to a new tab where you need to enter your credentials (same email you have used for Azure account) and after this you will land on the SendGrid app where you need to provide the information to set up your SendGrid account. Fill in all the details and click on get started.
After performing all the above steps, you will land on your SendGrid account's main page as shown below.

You need to get an API key, which you will be using to call SendGrid API to send Email via Azure Functions. For that, you need to click on Settings options on the left side of the SendGrid page and then click on the API keys option from the accordion menu (blue check marked in above image).
After that, click on the Create API Key button. You will move to a page where you need to provide details for your API Key like the Name and permissions for your API key.
Enter any name and select Full Access for your API key Permissions. Then click on the Create & View button at the bottom.

SendGrid now shows you your new API key. Copy your key and save it in a safe place as you won't be able to access the key again for security reasons. Finally, click on Done.
Hooray! You have successfully set up a SendGrid Account via Azure and created an API key for yourself that you will be using later.
Similarly, you need to create an azure storage queue and you can follow the steps mentioned here for the same.

Setting up .NET Core REST API

Open the visual studio and create .NET Core Web API project. You can choose .NET 6 as the target framework and keep rest of the configurations as it is. Once you click on create project , visual studio will bootstrap a project for you.
First you need to add the entity for Subscription which will be used to model document in Redis database.
Below is the entity you will be using:

[Document]
public partial class Subscription
{
[RedisIdField]
public string Id { get; set; }
[Indexed(Sortable = true)]
public string Email { get; set; }
[Indexed(Sortable = true)]
public DateTime SubscribedOn { get; set; }
[Searchable(Sortable = true)]
public bool IsSubscribed { get; set; }
}

After this, you need to add REST endpoints for our Newsletter Service and for that you will first create a Controller for Subscription.

[ApiController]
[Route("[controller]")]
public class SubscriptionController : ControllerBase
{
}

After this you will create below REST endpoints

POST
/subscription
Body: Email 
---------------------------------------------------
DELETE
/subscription
Body: {Email}
---------------------------------------------------
POST
/InitializeStorage
---------------------------------------------------
POST
/newsletter
Body: {string Subject, string Body }

Redis OM .NET library and Azure Storage Queue library installation

Before adding Rest endpoints you need to add Redis OM .NET Nuget which we will be using to connect to Redis database and performing operations on it.
To install Redis OM .NET all you need to do is add the Redis.OM NuGet package to your project. This can be done by running

dotnet add package Redis.OM

Similarly you need to add Nuget for using Azure storage queue library which you will be using to push message to queue on Azure that you have created previously. This can be done by running below command.

dotnet add package Azure.Storage.Queues

Setting up Redis Connection provider dependency
Redis OM .NET uses RedisConnectionProvider class to handle connections to Redis and provides functions by which you can interact with Redis. To use it, you need to inject an instance of the RedisConnectionProvider into your app by using .NET Core dependency injection. For this, you need to put below code in Program.cs file.

builder.Services.AddSingleton(new RedisConnectionProvider(builder.Configuration["REDIS_CONNECTION_STRING"]));

You also need to add REDIS_CONNECTION_STRING in our appsettings.json file. In order to get our connection string you can use the below schema:
"redis://username:password@hostname:port/4"
Username, password and public endpoint(host)can be found in under the configuration tab of database. Replace those values in the above schema and you will get your connection string you need to put under appsettings.json file.
After this the provider will now be available in your controllers/services to use.

Setting up Azure Queue Client dependency
You need to register the Azure storage queue client in the same fashion. You need to put below code in program.cs file in order to register queue client.

builder.Services.AddAzureClients(builder =>
{
builder.AddClient<QueueClient, QueueClientOptions>((options, _, _) =>
{
options.MessageEncoding = QueueMessageEncoding.Base64;
var credential = new DefaultAzureCredential();
var queueUri = new Uri(queueURI);
return new QueueClient(queueUri, credential, options);
});
});

REST Endpoints

The first 3 endpoints are for performing CRUD operations on Subscription and Newsletter resources and the last one is for a specific purpose i.e. to create Index for our Subscription entity on Redis Database that you have created on Redis Cloud in previous step.
1. Create Subscription endpoint
You need to provide an endpoint which will take email address as input for creating a subscription.
You will be creating a POST endpoint which will take SubscribeRequest in which you need to pass email and this will be passed in Body of the POST request.
Below code will create a Subscription with email, SubscribedOn date and isSubscribed flag set to true.

[HttpPost]
public async Task<IActionResult> Subscribe([FromBody]SubscribeRequest subscribeRequest)
{
if (string.IsNullOrWhiteSpace(subscribeRequest.Email))
{
return BadRequest();
}
Subscription subscription = new Subscription();
subscription.Email = subscribeRequest.Email;
subscription.IsSubscribed = true;
subscription.SubscribedOn = DateTime.UtcNow;
var subscriptionCollection = _provider.RedisCollection<Subscription
();
await subscriptionCollection.InsertAsync(subscription);
return Created("/subscription", subscription.Id);
}

2. Delete Subscription(Unsubscribe)
You need to provide an endpoint which allow user to unsubscribe from the newsletter.
You will be creating a DELETE endpoint which will take UnSubscribeRequest in which you will pass email. This will be passed in Body of the DELETE request. With this request you will set the IsSubscribed flag to false if a user with the email exists in our system. Below is the code for the same

[HttpDelete]
public async Task<IActionResult> UnSubscribe([FromBody] UnSubscribeRequest unSubscribeRequest)
{
if (string.IsNullOrWhiteSpace(unSubscribeRequest.Email))
{
return BadRequest();
}
var subscriptionCollection = _provider.RedisCollection<Subscription>();
var existingSubscription = await subscriptionCollection.FirstOrDefaultAsync(x => x.Email == unSubscribeRequest.Email);
if(existingSubscription == null)
{
return NotFound();
}
existingSubscription.IsSubscribed = false;
_provider.Connection.Set(existingSubscription);
return Ok("/unsubscribed");
}

3. Create Newsletter
Next, You will be creating an endpoint which can be used to create newsletter by owner and publish it on the azure queue which later will be pick up by newsletter processor for processing.
You will be creating a POST endpoint which will take NewsletterRequest in which you need to pass subject and body of the newsletter. This data will be passed in POST request's Body.
Below code will create newsletter with the subject and body from the request and fetching emails from Subscriptions which are subscribed. After creating the newsletter you will be publishing it on the Azure queue for asynchronous processing by newsletter processor service.

[HttpPost]
public async Task<IActionResult> SendNewsLetter(NewsletterRequest newsletterRequest)
{
var subscriptionCollection = _provider.RedisCollection<Subscription>();
var existingSubscription = await subscriptionCollection.ToListAsync();
if (existingSubscription != null && existingSubscription.Count > 0)
{
Newsletter newsletter = new Newsletter();
newsletter.Subject = newsletterRequest.Subject;
newsletter.Body = newsletterRequest.Body;
newsletter.Emails = existingSubscription.Where(x => x.IsSubscribed).Select(x => x.Email).ToList();
string serializedNewsLetterObject =
JsonConvert.SerializeObject(newsletter);
await _queueClient.SendMessageAsync(serializedNewsLetterObject);
}
return Accepted();
}

4. Initialize Storage
With the entity in place, the next step is to create the index in Redis database for the Subscription model you have created. This operation needs to be done only one time so you are creating a POST endpoint for this and run this only once.
Below code will create create a RedisConnectionProvider using Redis connection string and then create index of type Subscription in Redis. In case of success it will return 201 (created) else there will be some exception.

[HttpPost("/InitializeStorage")]
public async Task<IActionResult> InitializeStorage()
{
var provider = new RedisConnectionProvider(_configuration["REDIS_CONNECTION_STRING"]);
var isSuccess = await provider.Connection.CreateIndexAsync(typeof(Subscription));
return Created("Storage", null);
}

Setting up Azure Function for Newsletter processing

Once you are done with REST endpoints, you will next work on creating an azure function which will process the newsletter published on Azure queue and send email to the subscribers.
An Azure Function is a serverless architecture of cloud native design, which allows a piece of code to be deployed and executed without any need to configure or maintain server infrastructure or configuration. In simple terms, you can think of Azure Functions as a method that you can directly execute code in the cloud without worrying about what the server is, its OS, runtime, etc. You just need to provide the code and specify the programming platform you are writing the code for like .NET, Java, PHP, Go, etc. All the things related to infrastructure will be taken care by the cloud service provider. For more details, read the documentation on Azure Functions here.
Now, you will create an Azure Function using the Azure Core Command-Line tools, which you have installed as part of the prerequisites.
Open the command prompt shell and run the command below:

func init NewsletterProcessor --dotnet

This command will create a new folder NewsletterProcessor and generate a .NET project for Azure Functions. Navigate to the new folder by running the command below:

cd NewsletterProcessor

This project does not contain any functions yet. Run the following command to generate an Queue trigger based Azure Function.

func new -name NewsletterQueueProcessor -template "Queue trigger"

This command will create a C# function named NewsletterQueueProcessor.
You need to update the function to process the newsletter. For that you will be creating a SendGrid message using the newsletter item that you have received from the queue and this SendGrid message will send email for you.
Below is the code for the same.

public class NewsletterQueueProcessor
{
[FunctionName("NewsletterQueueProcessor")]
[return: SendGrid(ApiKey = "SENDGRID_API_KEY")]
public SendGridMessage Run([QueueTrigger("NewsletterQueue", Connection = "NewsletterQueueURI")] Newsletter newsletter, ILogger log, ExecutionContext context)
{
log.LogInformation($"C# Queue trigger function processed at: {DateTime.UtcNow}");
var config = GetConfiguration(context);
SendGridMessage message = CreateSendGridEmailMessage(newsletter, config);
return message;
}
private IConfiguration GetConfiguration(ExecutionContext context){
var config = new ConfigurationBuilder()
.SetBasePath(context.FunctionAppDirectory)
.AddJsonFile("local.settings.json", optional: true, reloadOnChange: true)
.AddEnvironmentVariables()
.Build();
return config;
}
private SendGridMessage CreateSendGridEmailMessage(Newsletter newsletter, IConfiguration config)
{
var msg = new SendGridMessage()
{
From = new EmailAddress(config["FromEmail"], config["FromName"]),
Subject = newsletter.Subject,
PlainTextContent = newsletter.Body
};
msg.AddTo(config["FromEmail"]);
List<EmailAddress> emailAddresses = new List<EmailAddress>();
foreach (var email in newsletter.Emails)
{
emailAddresses.Add(new EmailAddress(email));
}
msg.AddBccs(emailAddresses);
return msg;
}
}

Adding connection strings for Redis database and Azure Queue

In order to interact with the Redis database and Azure queue, you need to specify connection string for the Redis database and Connection string for Azure queue in our API project. For that you need to add the connection string value for the key REDIS_CONNECTION_STRING and AZURE_STORAGE_QUEUE_URI in appSettings.json file of our .NET Core API project.

Similarly, you need to add SendGrid API key and Azure Queue
ConnectionString in Azure function project as well. For that you need to add values for the key SENDGRID_API_KEY and NewsletterQueueURI in local.settings.json
After setting these config values, you will be able to run the services locally.

Conclusion

In this article you get to learn how you can use RedisJSON with .NET Core along with Azure function.
If you are facing any issue with setting up the project locally you can reach out to me on my email here: [e-mail]
If you have any feedback for me around this Newsletter microservice, you can share on my GitHub repo itself. Till then keep learning and coding :)

• Check out Redis OM, client libraries for working with Redis as a multi-model database.
• Use RedisInsight to visualize your data in Redis.
• Sign up for a free Redis database.

Comments

[Pawan]

Very detailed article though.