A Lightweight Dot Net Realtime Message, Event, HTTP (Request & Response) and Exception Logger For Web (Trying to make centralised logger for Microservice and K8s environment)
I’m searching for a logger and a simple UI to view my HTTP request & response logs in microservice environments. I found something called elastic-search, where all logs are stored & it gives a nice UI to view.
But for my simple APP & Low User base, Elastic search setup & operation cost are so high for my requirements. so I decided to build my simple logger, nothing fancy or complicated.
The idea is to build a simple Middleware that logs all the HTTP requests & responses. Then store the logs in DB for View Later. Create an Extension Method to register this middleware in HTTP Pipeline & Publish this middleware as Nuget Package.
My Requirements are
- I want a simple temporary log storage max 7–14 Days Logs.
- Very Lightweight to set up, and operate and has low memory usage.
- It just does two things record all HTTP request & response data, and exception logs for my microservices which are written in ASP.NET Core (.NET 6).
- A Simple UI where I can search & view logs by service name, request methods, timestamp, etc filter.
- I don’t want to use my primary application, DB, to store temporary logs.
How I build
When searching on the internet I found something called RedisSearch . RediSearch is a Full-Text Search Engine for NoSQL Database.
According to Redis Website, RediSearch is powerful indexing, querying, and full-text search engine for Redis, available on-premises and as a managed service in the cloud.
Everyone knows what is Redis (At least for those who are in the web development field for the last few years). Redis is a very popular cache but Redis also can be used as Primary Database, it supports JSON Documents like MongoDB (No-SQL Database). We can enable these features by installing Redis JSON Module.
This is the perfect choice for my requirements because I want nice search capabilities, I want search data in the HTTP request & response body and I don’t want a strong schema because HTTP request and response bodies vary from service to service but at the same time I also want strong search functionality.
Let’s Get Started
- Let’s set up Redis, we are going to use the Redis cloud for easy setup, and quickly get started with RedisJSON & RedisFullText Search Module.
- Redis Enterprise Cloud provides performance, operational simplicity, data management, and security in the cloud. https://redis.com/redis-enterprise-cloud/overview/. Don’t worry it provides a free tier for quickly building & prototype your solution. You can choose your favorite cloud provider & region. I’m Going to choose AWS and AP-South-1.
Redis Enterprise Cloud
After creating Account & Choosing Your RedisStack Deployment, You will get a public endpoint.
Redis Public Endpoint
We will use RedisInsight . A Redis GUI tool to connect to Redis cloud instance.RedisInsight enables the use of Redis data types including JSON and time series by providing a new user interface, a browser tool, a sophisticated CLI, bespoke data visualization, and built-in tutorials.
Redisinsight
Redis Enterprise enables you to create backups across all database shards quickly and consistently. It achieves quick auto-cluster recovery by rebuilding the cluster from the configuration file and using the same endpoints and database configurations.
Instant disaster recovery is achieved through Active-Active deployment, which allows reading and writing to each replica at any time and is backed up by an academically proven conflict-resolution mechanism.
Quick Overview of Redis JSON & Get Started
We can interact with Redis JSON documents using Redis CLI or Redis CLI available on RedisInsight GUI Tool.
To interact with Redis JSON we most often use JSON.GET & JSON.SET Commands from CLI
Redis Insight CLI
According to Redis DOCS & Website Redis.JSON GET & SET Commands specification.
Redis JSON Supported Data Structure
- Scalar
- Objects (including nested objects)
- Arrays of JSON objects
- JSON nested objects
JSON.GET
Syntax
JSON.GET key [INDENT indent] [NEWLINE newline] [SPACE space] [paths
[paths ...]]
Available in:
Redis Stack / JSON 1.0.0
Time complexity:
O(N) when the path is evaluated to a single value where N is the size of the value, O(N) when the path is evaluated to multiple values, where N is the size of the key
This command accepts multiple path arguments. If no path is given, it defaults to the value's root.
The following subcommands change the reply’s format (all are empty strings by default):
- INDENT sets the indentation string for nested levels
- NEWLINE sets the string that's printed at the end of each line
- SPACE sets the string that's put between a key and a value
JSON.GET Result
JSON.SET
JSON.SET key path value [NX | XX]
Available in: Redis Stack / JSON 1.0.0
Time complexity: O(M+N) when the path is evaluated to a single value where M is the size of the original value (if it exists) and N is the size of the new value, O(M+N) when the path is evaluated to multiple values where M is the size of the key and N is the size of the new value
JSON.SET Command
How to get all keys in Redis JSON
To list the keys in the Redis data store, use the KEYS command followed by a specific pattern. Redis will search the keys for all the keys matching the specified pattern. In our example, we can use an asterisk (*) to match all the keys in the data store to get the keys.
List Keys
To connect Redis from .NET Required Some sort of DB Driver & ORM. StackExchange Redis is popular amongst .NET Developers to connect Redis instances. But for easy use of Redis search & Redis JSON features, we will use Redis OM. NET.
Why Redis OM?
Redis OM provides high-level abstractions for using Redis in .NET, making it easy to model and query your Redis domain objects.
- Declarative object mapping for Redis objects
- Declarative secondary-index generation
- Fluent APIs for querying Redis
- Fluent APIs for performing Redis aggregations
Let’s try to use RedisOM in Our Dotnet Application. It’s simple to use RedisJSON with ASP.NET.
- Create the Logs model. Here you will notice that the fields are decorated with the Indexed attribute. These attributes ( Searchable and Indexed ) tell Redis OM that you want to be able to use those fields in queries when querying your documents in Redis Stack.
[Document(StorageType = StorageType.Json, Prefixes = new []{“WLog”})] Indicates that the data type that Redis OM will use to store the document in Redis is JSON and that the prefix for the keys for the WLog class will be WLog.
The Id property is marked as a RedisIdField. This denotes the field as one that will be used to generate the document’s key name when it’s stored in Redis. e,g Id:”01GC92PD10R0DFVST9CV6Q7SBS”
To insert your object into Redis, use Set on the RedisConnection or Insert in the RedisCollection. Redis OM will automatically set the id for you and you will be able to access it in the object. If the Id type is a string and no IdGenerationStrategy is explicitly overridden on the thing, the ULID will bind to the string.
WLog.cs to RedisJSON
After you’ve finished building the model, you’ll need to construct the index in Redis. The best method to handle this is to spin the index creation out into a Hosted Service that will execute when the app is launched. Make a directory called ‘HostedServices’ and add IndexCreationService.cs to it. Add the following to that file to construct the index on startup.
Redis OM handles connections to Redis and offers the classes you may use to communicate with Redis via the RedisConnectionProvider class. To utilize it, just inject a RedisConnectionProvider object into your application.
Let’s Create a Helper Class That will contain DB Access Logic. One Of the great features I like about Redis OM, we can easily query using LINQ expression. I used Object Mapper to Convert Domain Model to Redis DB Models. In the future, if we want to support other DB, we can easily do that.
public IEnumerable<WatchLog> FilterByMethod(string method){ var logs = _redisWatchLog.Where(x => x.Method == method).ToList(); List<WatchLog> logList = CustomMapper.Mapper.Map<List<WLog>, List<WatchLog>>(logs);
return logList;
}
Import Note About Redis OM .NET
Redis OM may be used in conjunction with normal Redis for object mapping and retrieving objects by ID. Redis OM is dependent on the Redis Stack platform for more sophisticated functionality like indexing, querying, and aggregation. The Redis Stack platform is a collection of modules that expand Redis.
Why is this significant?
You can still use Redis OM to develop declarative models supported by Redis if you don’t have Redis Stack.
We’ll store your model data as Hashes in Redis, and you’ll be able to retrieve models using their main keys.
So, what won’t operate in the absence of Redis Stack?
- Models cannot be nested inside each other.
- You will not be able to discover objects using our expressive queries; you will only be able to query by primary key.
As We are Building A Logger Libray as Middleware, the Injection code is written in Service Collection Extension Method.
We Register the RedisConnection provider as a singleton.
services.AddSingleton(new RedisConnectionProvider(WatchLogExternalDbConfig.ConnectionString));
This will retrieve your connection string from the configuration and initialize the provider. The provider is now accessible for usage in your controllers/services.
Now we will create the leading Middleware , That will intercept the HTTP Request & Response and save it to DB.
Next, To Get Logs in UI, We have to create an API, from which we will fetch logs. Let’s Create an API Controller That looks like the one below.
This is a very basic simple API Idea, We will later extend it with robust Redis FullTextSearch Methods, RealTime Streaming, etc fancy things. Keep in mind this is a Proof of concept type of Project, a very early stage in development.
Now we can use This Logger in the dotnet Web API project.
// Configure DB Connection In API Service
builder.Services.AddWatchLoggerServices(opt =>{ opt.SetExternalDbConnString = builder.Configuration["RedisConnectionString"];});
Now Register WatchLogger In Middleware
// Custom Logger Middleware
app.UseWatchLogger();
Next, We will Build A Sample UI In Angular or a Similar Framework, That will somewhat look like this.
WatchLogger UI
For UI, I am writing a separate second Part. Where we will extend Redis Stack Features, Fully Use RedisFullText Search, and RedisStream also adds Exception Logger Middleware, WebSocket, SignalR, etc.
Don’t forget to try Redis, and we’ll continue the series as a Redis Stack. In forthcoming blogs, we will look at Redis Search, Redis Stream, pub-sub, and many more topics. To learn more about Redis, go to https://redis.io/.
Early Stage Project Available On Github: WatchLogger
This post is in collaboration with Redis.
Learn more about Redis Cool 😎 Features:
Top comments (0)