DEV Community: Chanh Le

Is Unit Test really a MUST?

Chanh Le — Tue, 17 Dec 2024 18:29:32 +0000

I have read the Refactoring book from Martin Fowler, and I am quite confused with the idea of using unit tests for refactoring.

I feel like what we actually need is integration test instead. Because when we do refactoring we will inevitably change the “unit” and then the unit test become outdated and useless and we have to remove them eventually. In the other hand, the integration test is still valid and doing its job, catching bugs and side-effects.

Now, I don’t know whether we still need unit tests for any reason! Yes, running integration tests takes time but we will not run them too frequently and also they will less likely outdated over time or be deleted during refactoring.

Does it make sense? Am I getting anything wrong somewhere?

Is Clean Code really practical?

Chanh Le — Tue, 17 Dec 2024 18:07:00 +0000

Small functions are easy to understand? What if we have too many of them?

Meaningful function names? Is it that easy?

Avoid switch statements, boolean arguments? How ?

When I first read the Clean Code book, I admit it was mind-blowded to me. I was so excited to practice and try to apply the rules to my side projects. But over time, I start to realize that it isn’t easy to achieve.

Does anyone feel the same? Can anyone tell a success story about applying Clean Code?
By the way, the Clean Architecture is also too hard for me. I am quite frustrated!

System Architecture Overview for Message-Oriented Alarm Processing System

Chanh Le — Tue, 10 Sep 2024 15:49:24 +0000

In previous posts, we’ve explored the fundamentals of Message-Oriented Architecture (MOA) and how Redis is utilized as the backbone for our alarm message queue. Now, let’s take a step back and examine the overall architecture. This post provides an overview of the system, highlighting the different components and the reasoning behind key design choices.

Understanding the Big Picture

Our system is designed to handle the reliable delivery of alarm messages from alarm devices to alarm receiving centers (ARCs). The architecture follows a message-oriented pattern, ensuring the system can scale to handle high volumes of messages efficiently while maintaining reliability and low latency.

Here’s a breakdown of the key components:

1. Alarm Devices

These are physical devices that detect and report alarms such as fires or security breaches. Each alarm device is responsible for sending messages with key information to the system:

Message ID: A unique identifier for each alarm event.
Signal Code: A predefined code that specifies the type of alarm being triggered.
Sender ID: The ID of the device that triggered the alarm.
Criteria Index: Parameters specific to the source device that triggered the alarm.

These messages are then pushed to the message queue for further processing.

2. Redis Message Queue

At the core of the architecture, Redis is used as a Message Queue, but with a specific approach:

Redis Key: Instead of placing the full message in the queue, we only push the Redis key (which points to the message hash) into the queue. The Redis key is prefixed by message: followed by the unique message ID.
Message Hash in Redis: Full details about each message—such as its message ID, signal code, criteria index, device ID, destination group, current state, phase, destination information, and TTL—are stored in Redis as a hash. The message hash is assigned an expiration time equal to the TTL of the message to prevent indefinite growth.

3. Processing Layer

This is where alarm messages are taken from the queue and processed by various services in a controlled and scalable manner.

Message Consumers: Consumers retrieve Redis keys from the message queue, fetch the corresponding message hash from Redis, and process the alarm accordingly. The system updates the message state (e.g., "received", "processing", "dispatched") during each stage.
State and Queue Management: The system tracks the message’s current queue and state to handle different stages of the processing workflow.
Timeouts and Retries: If a message isn't processed within its TTL, the system either retries or escalates the issue to ensure critical messages are addressed.

4. Alarm Receiving Centers (ARCs)

Once an alarm message has been processed and routed, it is delivered to an ARC, responsible for handling the final response to the alarm (e.g., dispatching emergency services). The ARCs are decoupled from the processing system, allowing it to scale and process alarms even if some ARCs are temporarily unavailable.

5. Monitoring and Logging

Real-time monitoring is vital to this system. The system tracks:

Queue Length: Ensuring the message queue doesn’t grow uncontrollably.
Processing Time: Monitoring how long each message takes to be processed.
Health Checks: Ensuring the health of each component and alerting the system to potential failures.

Key Design Choices

Message-Oriented Architecture

The system’s message-oriented architecture allows for decoupling between the various components, improving scalability and resilience. Each layer operates independently, allowing flexibility in how messages are processed and how the system scales to handle high message throughput.

Redis as Message Queue and State Store

Redis provides the necessary performance and flexibility. By using Redis for both the message queue and as a hash-based state store, the system ensures efficient, real-time processing of alarm messages. Only the Redis key is pushed into the queue, and the full message is stored in Redis as a hash with additional fields like the message ID, signal code, criteria index, and destination group.

Expiration-Based Message Cleanup

To prevent indefinite message growth in Redis, message hashes have an expiration time set equal to the TTL of the alarm message. This keeps the system efficient and lightweight by removing old messages when they are no longer needed.

Future Enhancements

As the system scales, several improvements could be implemented:

Redis Streams: Transitioning from Redis Lists to Redis Streams could provide better support for advanced features such as consumer groups and message acknowledgment.
Event-Driven Architecture: Implementing an event-driven system could further enhance the responsiveness of the system by integrating external services like analytics.
Redis Clustering: As the message volume increases, Redis clustering can be introduced to ensure consistent performance under high load.

Conclusion

This architectural overview highlights how the system efficiently processes alarm messages in real-time, leveraging Redis to maintain a high level of performance and scalability. By pushing only the Redis key into the message queue and storing the full message in a hash with an expiration, we maintain efficiency and avoid unbounded growth.

In the next post, we will explore the lifecycle of an alarm message and how it transitions through the different states in our system.

Setting Up Redis as a Message Queue: A Step-by-Step Guide

Chanh Le — Tue, 10 Sep 2024 15:29:13 +0000

In our previous discussion, we introduced the concept of Message-Oriented Architecture (MOA) and its importance in building scalable, resilient systems. Today, we’ll delve deeper into one of the key components of our system: Redis as a Message Queue. Specifically, we’ll explore how Redis can be configured to handle alarm messages by queuing essential metadata and storing full message details separately as hashes.

Why Use Redis as a Message Queue?

Redis is a powerful choice for a message queue, particularly in systems where speed and simplicity are paramount. Here’s why Redis excels in this role:

High Performance: Redis operates entirely in-memory, making it capable of handling high volumes of messages with low latency.
Efficient Data Handling: By storing essential metadata (like message IDs) in the queue and full details in separate hashes, Redis minimizes memory usage while maintaining the speed and efficiency of message delivery.
Scalability: Redis supports clustering and sharding, allowing the system to scale as message volumes grow.
Flexibility: Redis can be tailored to fit various queuing models, including FIFO (First-In, First-Out) and more complex workflows.

Understanding the Message Structure

In our project, the Redis message queue is designed to hold minimal information about each alarm message, such as:

Message ID: A unique identifier for the message.
Sender ID: The ID of the device that generated the alarm.
Signal Code: A code representing the type of alarm being handled. This code comes from a predefined list that indicates whether it’s a fire alarm, security breach, or other critical event.
Criteria Index: A reference to criteria on the source device.

The full details of each message, including its state, phase, destination, and TTL (Time-to-Live), are stored in Redis as a hash, with the key being the message ID prefixed by message:. To prevent the full message hash from growing indefinitely, an expiration time is set for the hash equal to the TTL of the message.

Setting Up Redis as a Message Queue

Step 1: Installing Redis

If Redis isn’t already installed on your system, you can easily set it up:

For Ubuntu:

sudo apt-get update
sudo apt-get install redis-server

For macOS:

brew install redis

After installation, start the Redis server:

redis-server

You can interact with Redis using the Redis CLI:

redis-cli

Step 2: Configuring Redis for Queuing and Storing Messages

In this setup, Redis lists are utilized to create a FIFO queue. The list stores minimal metadata about each message, and the full message details are stored in a Redis hash.

Push Message Metadata to the Queue:

Use the LPUSH command to add metadata to the queue:

 LPUSH alarm_queue '{"id":"msg123", "sender_id":"deviceA", "signal_code":"12345", "criteria_index":1}'

Store Full Message Information in a Hash and Set Expiration:
- Use the HMSET command to store the full details of the message in a Redis hash, and then set the expiration for the hash based on the TTL of the message:
```
 HMSET message:msg123 state "new" phase "initial" destination "center1" ttl 3600
 EXPIRE message:msg123 3600
```

This ensures that the full message information will automatically expire and be removed from Redis after 3600 seconds (1 hour), matching the TTL of the message.

Consume Message Metadata from the Queue:
- Retrieve and process message metadata using the RPOP command:
```
 RPOP alarm_queue
```

The metadata retrieved can then be used to fetch the full message details from the corresponding hash.

Step 3: Implementing Producers and Consumers

Message Producer:
Here’s a simple Python script that acts as a producer, pushing metadata into the Redis queue and storing the full message details in a hash with an expiration:

import redis
import json

r = redis.Redis(host='localhost', port=6379, db=0)

# Example metadata to push into the queue
message_metadata = {
    "id": "msg123",
    "sender_id": "deviceA",
    "signal_code": "12345",  # From a predefined list
    "criteria_index": 1
}

# Push metadata to the queue
r.lpush('alarm_queue', json.dumps(message_metadata))

# Store full message details in a hash and set expiration (TTL)
ttl = 3600
r.hmset(f"message:{message_metadata['id']}", {
    "state": "new",
    "phase": "initial",
    "destination": "center1",
    "ttl": ttl
})

# Set expiration for the hash based on TTL
r.expire(f"message:{message_metadata['id']}", ttl)

print("Message metadata pushed to the queue, full message details stored in Redis, and expiration set.")

Message Consumer:
A basic consumer script would look like this:

import redis
import json

r = redis.Redis(host='localhost', port=6379, db=0)

while True:
    # Blocking pop to wait for new messages
    metadata = r.brpop('alarm_queue')
    message_info = json.loads(metadata[1].decode('utf-8'))

    # Retrieve full message details from Redis
    full_message = r.hgetall(f"message:{message_info['id']}")

    print(f"Processing message ID: {message_info['id']} from sender: {message_info['sender_id']}")
    print(f"Full message details: {full_message}")

Step 4: Scaling and Optimization

As your system grows, Redis can be scaled to handle increased message volumes:

Sharding: Distribute your queues across multiple Redis instances.
Clustering: Use Redis clustering to spread the load across multiple nodes.
Optimized Data Handling: Consider using Redis Streams if you need more advanced features like message IDs, groups, and consumer tracking.

Step 5: Monitoring and Maintenance

Monitor your Redis instances regularly using tools like redis-cli, Redis Sentinel, or third-party monitoring solutions to keep track of key metrics such as queue length, memory usage, and latency.

Conclusion

Using Redis as a message queue, particularly when storing minimal metadata in the queue and full message details in hashes, is an efficient way to manage the flow of alarm messages in a message-oriented architecture. By setting expiration on the full message hashes, matching the TTL, you ensure the system remains clean and avoids uncontrolled growth. The signal code, coming from a predefined list, indicates which kind of alarm is being handled, allowing for specific actions based on alarm type. With Redis, the system remains lightweight, scalable, and fast. In the next post, we’ll dive into how to manage the full lifecycle of these messages, including state management, TTL, and delivery guarantees.

Stay tuned!

Understanding Message-Oriented Architecture: The Backbone of Modern Distributed Systems

Chanh Le — Mon, 09 Sep 2024 16:24:18 +0000

In today’s world of complex distributed systems, ensuring that different components communicate efficiently and reliably is crucial. This is where Message-Oriented Architecture (MOA) comes into play. In this blog post, we'll explore what MOA is, its core principles, and why it’s a foundational approach for building scalable, resilient systems.

What is Message-Oriented Architecture?

Message-Oriented Architecture (MOA) is a design paradigm used to build systems where components communicate by sending messages to each other, rather than through direct method calls or function invocations. In an MOA, messages are the primary means of communication, decoupling the sender and receiver and allowing for greater flexibility and scalability.

Why Use MOA?

In a traditional monolithic application, components are tightly coupled, meaning that a failure in one part of the system can have a cascading effect. MOA addresses this issue by introducing a layer of abstraction between components. Here’s why this approach is advantageous:

Decoupling: With MOA, components don’t need to know about each other’s existence or how they’re implemented. They simply send and receive messages. This decoupling allows you to change or replace components without impacting the entire system.
Scalability: Because messages can be queued and processed asynchronously, MOA makes it easier to scale individual components independently. For example, if a service is receiving more requests than it can handle, you can add more instances of that service without affecting the rest of the system.
Resilience: In an MOA, if a component fails, messages can be stored in a queue until the component is back online. This means that the system can continue to function even in the face of individual component failures, leading to higher overall resilience.
Flexibility: MOA supports different communication patterns, such as point-to-point (where one sender communicates with one receiver) and publish-subscribe (where messages are broadcast to multiple receivers). This flexibility makes it easier to implement complex workflows.

How Does MOA Work?

At the heart of an MOA is the message queue or broker, which acts as an intermediary between components. Here’s a simplified workflow:

Message Production: A sender (also known as a producer) creates a message and sends it to the message queue.
Message Queuing: The message queue stores the message until a receiver (also known as a consumer) is ready to process it.
Message Consumption: The receiver retrieves the message from the queue and processes it.
Message Acknowledgment: After processing, the receiver sends an acknowledgment back to the queue, indicating that the message has been handled successfully.

This approach allows components to operate asynchronously and independently, improving the overall efficiency and reliability of the system.

Real-World Applications of MOA

MOA is widely used across various industries and applications, including:

E-commerce: Handling orders, payments, and notifications in a decoupled manner, ensuring that no single component overloads the system.
IoT Systems: Managing communication between a vast number of devices and a central server, where devices send data and receive instructions asynchronously.
Financial Services: Processing transactions, where different services like payment processing, fraud detection, and account updates operate independently and communicate through messages.

Redis as a Message Queue in MOA

Redis, a popular in-memory data structure store, is often used as a message queue in MOA systems. It offers high performance and flexibility, making it an excellent choice for handling message queuing tasks. In upcoming posts, we’ll dive deeper into how Redis can be leveraged in an MOA, particularly in systems where reliability and speed are critical, such as delivering alarm messages from devices to receiving centers.

Conclusion

Message-Oriented Architecture is a powerful design approach that enables building scalable, resilient, and flexible systems. By decoupling components and facilitating asynchronous communication through messaging, MOA allows systems to be more adaptable to change and more robust in the face of failure. As we continue this blog series, we’ll explore practical implementations of MOA using Redis, focusing on real-world scenarios and best practices.

Stay tuned for our next post, where we’ll dive into the specifics of using Redis as a message queue in an MOA system!

Rust ownership: another way to manage memory

Chanh Le — Sat, 02 Mar 2024 16:59:58 +0000

In Java, we have the garbage collector helps us to clean up unused memory. In C++, we have to allocate and de-allocate memory explicitly using pointers. So, what about Rust? Ownership is a core concept in Rust which we developers, would use to ensure memory safety.

What is memory management

Memory in programming has two types: stack and heap. Because stack memory is managed automatically, we will only care about managing data (dynamically sized data) on heap memory.
Memory management is the way we allocate and de-allocate memory for storing variable values, for example values of string or linked-list variables.

JVM garbage collector (Java)

In Java, we don't have to know about how to manage memory because that is the job of JVM's garbage collector. This component of JVM will periodically scan for unused references (variables) and un-reference(de-allocate) them. So, the memory would be clean up automatically.

C++ pointer

In C++, we have to manage the memory manually and explicitly using pointer. When we need memory for a variable, we ask for a memory allocation by using the new keyword. We will receive a pointer to reference to the location where our data would be stored. Then, when the variable is no more needed (out of scope), we will use delete keyword to un-reference the pointer. So, the memory at that location would be cleaned up and released. And if we forgot to delete the pointer, that segment of memory would never be released. Eventually, we would encounter memory leak problems.

new and delete in C++:

int* pointer_name = new int(10); // ask for heap memory allocation
// Code using pointer_name variable
delete pointer_name; // de-allocate the memory segment

Rust ownership

Ownership

Each value in Rust has an owner. The owner here is actually the variable associated with that value. When we assign a variable to another variable, the ownership is transferred.

let hello_str = "hello";
// `hello_str` variable is the owner of the heap segment storing value `hello`

Ownership transferring

When ownership is transferred or moved, the original variable becomes invalid and unusable. Variable assignment and parameter passing would trigger ownership transfer.

let hello_str = "hello";
let new_hello_str = hello_str;
// `new_hello_str` is now the owner, `hello_str` is no more usable

Reference borrowing

In case you just want to use the value and not want to take ownership, you could borrow a reference (&) to that value. If you also want to modify the value, you could borrow a mutable reference(&mut) instead.

fn greeting(name: &str) {
    let mut greeting_str = String::from("hello ");
    greeting_str.push_str(name);
    println!("{}", greeting_str);
}
fn main() {
    let name = "Chanh";

    // Create a reference to the name string
    let name_ref = &name;

    // Pass the reference to the greeting function
    greeting(name_ref);

    // The original name can still be used
    println!("My name is: {}", name);

}

Copy/Clone

In case you do want to have ownership on a value, but in the same time do not want make the original variable invalid, you could copy the value to a new memory location using clone method. You would assign the ownership to a new local variable and have a full control on it without any affect on the original variable.

fn greeting(name: &mut String) {
    name.insert_str(0, "Mr. ");

    let mut greeting_str = String::from("hello ");
    greeting_str.push_str(name);

    println!("{greeting_str}");
}
fn main() {
    let name = String::from("Chanh");

    // Pass the mutable reference of the cloned value to the greeting function
    greeting(&mut name.clone());

    // The original name can still be used
    println!("My name is: {}", name);
}

Summary

As you could see, Rust uses a new approach for managing memory allocation. The developers do not have support on cleaning data up automatically like in Java. They also do not need to manually, explicitly do that like in C++. Instead, they need to participate on helping the Rust Ownership System to know when which memory segment should be cleaned and freed.

Resources

Result in Rust: another way of error handling

Chanh Le — Fri, 01 Mar 2024 18:16:56 +0000

I've just started learning Rust in 2 days. And the first thing I was surprised about was how Rust handles errors. I have used several different programming languages and I thought that the only way to handle errors or exceptions is catching them with try/catch statement. But you know what there is no try/catch at all in Rust. Yeah, that's true I swear!

Traditional way to handle error/exception

In other programming languages like Java, Python, we usually use the Exception Approach for handling errors. More specifically, we would throw exception or raise error when we want to signal the caller of our code that there is an error it needs to handle. And the caller would use try/catch statement to catch the error in order to handle it. That's the way of handling error that we all know I guess. But that's not true with Rust. So, how exactly does Rust handle error?

Java try/catch statement:

try {
  //  Block of code to try
  myFunction();
}
catch(Exception e) {
  //  Block of code to handle errors
}

An Result always returned even in case of error

Yes, I say that. No mistake or typo here. But first, you need to know what Result is. Result in Rust is a built-in enum. It has two variants: Ok (for normal case) and Err (in case of any error occurred during runtime).
In case of no error, the caller of your code would receive a Result object of Ok variant. And if there was an error occurred, the Result object would be an Err variant.
And the caller would use the match statement to handle both variants.

Handling Result using match statement (Guessing Game):

let result = my_function() //guess.trim().parse()
let guess: u32 = match result {
    Ok(num) => num,
    Err(_) => continue,
};

As you can see, Rust uses Result with Err variant to signal the caller of your code about error occurrences. The caller would handle the error using match statement. He could skip the error and retry if the error is recoverable, otherwise he could stop the program from running immediately using panic! marco.

Summary

Error handling in Rust uses a different approach compared to other common programming languages like Java, Python. There is nothing like try/catch statement in Rust. Instead, Rust provides a built-in enum called Result which would carry the error objects for us to handle using match statement.

What's next

Actually, there is another surprise to me about Rust, which is the concept of ownership. There's a complete new world for me to explore there. And that's what I would like to learn and write about next. So, stay tune!

DEV Community: Chanh Le

Is Unit Test really a MUST?

Is Clean Code really practical?

System Architecture Overview for Message-Oriented Alarm Processing System

Understanding the Big Picture

1. Alarm Devices

2. Redis Message Queue

3. Processing Layer

4. Alarm Receiving Centers (ARCs)

5. Monitoring and Logging

Key Design Choices

Message-Oriented Architecture

Redis as Message Queue and State Store

Expiration-Based Message Cleanup

Future Enhancements

Conclusion

Setting Up Redis as a Message Queue: A Step-by-Step Guide

Why Use Redis as a Message Queue?

Understanding the Message Structure

Setting Up Redis as a Message Queue

Step 1: Installing Redis

Step 2: Configuring Redis for Queuing and Storing Messages

Step 3: Implementing Producers and Consumers

Step 4: Scaling and Optimization

Step 5: Monitoring and Maintenance

Conclusion

Understanding Message-Oriented Architecture: The Backbone of Modern Distributed Systems

What is Message-Oriented Architecture?

Why Use MOA?

How Does MOA Work?

Real-World Applications of MOA

Redis as a Message Queue in MOA

Conclusion

Rust ownership: another way to manage memory

What is memory management

JVM garbage collector (Java)

C++ pointer

Rust ownership

Ownership

Ownership transferring

Reference borrowing

Copy/Clone

Summary

Resources

Result in Rust: another way of error handling

Traditional way to handle error/exception

An Result always returned even in case of error

Summary

What's next

References