DEV Community: Sonu Kumar

KUID: Compressed Universally Unique Identifier

Sonu Kumar — Sun, 26 Feb 2023 05:55:05 +0000

KUID, a universally unique identifier, is essentially a shorter version of a UUID and is commonly used to identify records in various applications.

What's UUID?

A UUID (Universally Unique Identifier) is a unique identifier that is widely used across various applications and systems to identify records. It is a 128-bit number typically represented as a sequence of 32 hexadecimal digits, grouped in 5 sections, separated by hyphens, as in the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

UUIDs are designed to be globally unique, meaning that it is extremely unlikely for two UUIDs to be the same, even when generated independently by different systems. This makes UUIDs useful for identifying and tracking records in distributed systems, databases, and other contexts where unique identification is necessary.

There are different variants and versions of UUIDs, each with its own way of generating unique identifiers. Some commonly used variants include UUIDv1, UUIDv4, and UUIDv5. The specific variant and version of a UUID can be determined from the format and content of the identifier.

How to generate KUID?

KUID is generated in two steps

Generate UUID (use any algorithm like V1,V2,V3,V4 etc)
Encode 128-bit number

KUID = Generate UUID and encode it in the higher base like base 32, 62 or even higher.

The length of the KUID depends on the encoding scheme employed. If the base62 encoding scheme ([0–9A-Za-z]) is used, KUID can be encoded to 22 bytes. However, if you wish to store KUID in upper case, then encoding it using the base36 encoding scheme, which utilizes characters [0–9A-Z], will result in an encoded representation that can be stored in 26 bytes. For this post, we'll use base 62 encoding.

Why should we use KUID?

KUID generates a smaller string, which results in less storage space and faster processing of many operations.

NOTE: Following statements are only valid when KUID is treated as a string, not as 128-bit number.

Hashing: Using KUID with its shorter length of 22 bytes reduces the amount of data that needs to be hashed. This results in fewer instructions being required to perform the hashing operation compared to UUID, which requires a hash of 36 bytes.
Comparison: Due to the shorter length of KUID, only 22 bytes need to be compared instead of 36 bytes, resulting in fewer instructions needed for comparison. Read/Write: Using KUID, which can be encoded in 22 bytes, results in a reduction in the memory and disk space required for read and write operations. This is in contrast to UUID, which requires 36 bytes for the same operations. With KUID, fewer instructions are needed to read or write data due to its smaller size.
I/O Calls: Sending a UUID over a network consumes 36 bytes of data (without considering compression), while a KUID only requires 22 bytes. Therefore, KUID is a more efficient choice for network transmission due to its smaller size.
Database indexing: Due to the smaller size of KUID, it leads to a decrease in the size of the database index. In addition, combining KUID with another column for secondary indexing can be beneficial as databases have a limited index width (e.g., 767 bytes in MySQL) that is dependent on the encoding used for the column or table. For instance, MySQL's UTF8 encoding may limit the index width to as little as 255 bytes.
URL safe: Like UUID, KUID is URL safe since it only uses [0–9A-Za-z] characters to represent the data. KUID is commonly used as a unique column, which means that URLs containing KUIDs may be used to represent specific resources, such as /api/v1/products/:productId. As the KUID is included in the URL, it must be URL safe to avoid issues such as encoding errors or unintended interpretation by the web server or browser.
Lower Memory/Disk Usage: Using KUID results in lower memory/disk usage compared to UUID. Since KUID only requires 22 bytes, it uses less storage space than UUID, which uses 36 bytes. This can be beneficial for systems that require frequent read/write operations or have limited storage capacity. For instance, if we consider the storage of 100 million records, assuming each KUID is stored as a 22-byte string, the space consumed by KUIDs would be approximately 2.2 GB. In contrast, the space required for UUIDs would be around 3.6 GB. Therefore, using KUIDs can result in a storage saving of approximately 39%.

KUID Usage



public class Main {
    public static void main(String[] args) {
        for (int i = 0; i < 10; i++) {
            KUID kuid = KUID.randomKUID();
            System.out.println(kuid.toString());
        }

    }
}

This will produce something like this

If you found this post helpful, please share and give a thumbs up.

Asynchronous task execution using Redis and Spring(Boot)

Sonu Kumar — Mon, 31 May 2021 17:42:51 +0000

In this article, we are going to see how to use Spring boot 2.x and Redis to execute asynchronous tasks, with the final code that we will have after following the steps described here.

Spring/Spring Boot

Spring is the most popular framework available for the Java platform. Spring has one of the largest communities in open source. Besides that, Spring provides extensive and up-to-date documentation that covers the inner workings of the framework and sample projects on their blog, there’re 100K+ questions on StackOverflow.
Spring 3.0 was the first version that supported the annotation-based configuration, later in 2014 Spring boot 1.0 was released that completely changed how we look at the Spring framework ecosystems, it provides out of the box configuration and many more. Timeline

Redis

Redis is one of the most popular open-source NoSQL in-memory database. Redis supports different types of data structures e.g. Set, Hash table, List, simple key-value pair just name a few. The latency of Redis call is sub-milliseconds, support of a replica set, etc.

Why Asynchronous task execution

A typical API call consists of five things

Execute one or more database(RDBMS/NoSQL) queries
One or more operations on some cache systems (In-Memory, Distributed, etc )
Some computations (it could be some data crunching doing some math operations)
Calling some other service(s) (internal/external)
Schedule one or more tasks to be executed at a later time or immediately but in the background.

A task can be scheduled at a later time for many reasons for example invoice must be generated after 7 days of order creation or order shipment, similarly, email/notification(s) need not be sent immediately it can be delayed. Sometimes we need to execute tasks in asynchronous to reduce API response time, for example, delete 1K+ records at once if we delete all these records in the same API call then API response time would be increased for sure, to reduce API response time, we can run a task in the background that would delete those records.

Delayed queue

We can run schedule tasks using cron jobs, a cron job can be scheduled using different methods like UNIX style crontabs, Chronos, if we’re using Spring frameworks then it’s out of box Scheduled annotation ❤️. Most of these scheduling mechanisms suffer scaling problems, where we do scan database(s) to find the relevant rows/records. In many situations, this leads to a full table scan which performs very poorly. Imagine the case where the same database is used by a real-time application and this batch processing system. A delayed queue can be used in such cases where as soon as the timer reaches the scheduled time a job would be triggered. There’re many queuing systems/software available, but very few of them provide this feature, like SQS which provides a delay of 15 minutes, not an arbitrary delay like 7 hours or 7 days.

Rqueue

Rqueue is a broker built for the spring framework that stores data in Redis and provides a mechanism to execute a task at any arbitrary delay. Rqueue is backed by Redis since Redis has some advantages over the widely used queuing systems like Kafka, SQS. In most web applications backend, Redis is used to store either cache data orother purposese. In today's world, 8.4% of web applications are using the Redis database.

Generally, for a queue, we use either Kafka/SQS or some other systems these systems bring an additional overhead in different dimensions e.g money which can be reduced to zero using Rqueue and Redis.

Apart from the cost if we use Kafka then we need to do infrastructure setup, maintenance i.e. more ops, as most of the applications are already using Redis so we won’t have ops overhead, in fact, same Redis server/cluster can be used with Rqueue. Rqueue supports an arbitrary delay

Message Delivery
Rqueue guarantee at-least-once message delivery as long data is not wiped out from Redis. Read about it more at Introducing Rqueue

Tools we need:

Any IDE 2. Gradle 3. Java 4. Redis We’re going to use Spring boot for simplicity, we’ll create a Gradle project from spring boot initializer at https://start.spring.io/, for dependency we would need
1. Spring Data Redis 2. Spring Web 3. Lombok and any others.

The directory/folder structure would look like below.

We’re going to use the Rqueue library to execute any tasks with any arbitrary delay. Rqueue is a Spring-based asynchronous task executor, that can execute tasks at any delay, it’s built upon the Spring messaging library and backed by Redis.
We’ll add the Rqueue spring boot starter dependency using com.github.sonus21:rqueue-spring-boot-starter:2.7.0-RELEASE



dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-data-redis'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'com.github.sonus21:rqueue-spring-boot-starter:2.7.0-RELEASE'
    compileOnly 'org.projectlombok:lombok'
    annotationProcessor 'org.projectlombok:lombok'
    providedRuntime 'org.springframework.boot:spring-boot-starter-tomcat'
    testImplementation('org.springframework.boot:spring-boot-starter-test') {
        exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
    }
}

We need to enable Redis spring boot features, for testing purposes we will enable WEB MVC as well.
Update Application file as



@SpringBootApplication
@EnableRedisRepositories
@EnableWebMvc
public class AsynchronousTaskExecutorApplication {
   public static void main(String[] args) {
      SpringApplication.run(AsynchronousTaskExecutorApplication.class, args);
  }
}

Adding tasks using Rqueue is very simple we need to just annotate a method with RqueueListener. RqueuListener annotation has multiple fields that can be set based on the use case, for example, set deadLetterQueue to push tasks to another queue otherwise task will be discarded on failure. We can also set how many times a task should be retried using the numRetries field.
Create a Java file name MessageListener and add some methods to execute tasks.



import com.github.sonus21.rqueue.annotation.RqueueListener;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;

@Component
@Slf4j
public class MessageListener {

  @RqueueListener(value = "${email.queue.name}")
  public void sendEmail(Email email) {
    log.info("Email {}", email);
  }

  @RqueueListener(value = "${invoice.queue.name}")
  public void generateInvoice(Invoice invoice) {
    log.info("Invoice {}", invoice);
  }
}

We would need Email and Invoice classes to store email and invoice data respectively. For simplicity, classes would only have a handful number of fields.

Invoice.java



import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Invoice {
  private String id;
  private String type;
}

Email.java



import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Email {
  private String email;
  private String subject;
  private String content;
}

Task submissions

A task can be submitted using the RqueueMessageEnqueuer bean. This has multiple methods to enqueue tasks depending on the use case, like retry use, retry count, and delay for delayed tasks.

We need to AutoWire RqueueMessageEnqueuer or use constructor to inject this bean.

Create a Controller for testing purpose:
We’re going to schedule invoice generation that would be done in the next 30 seconds, for this we’ll submit a task with 30000 (milliseconds) delay on invoice queue. Also, we’ll try to send an email that will be done in the background. For this purpose, we’ll add two GET methods sendEmail and generateInvoice, we can use POST as well.



@RestController
@RequiredArgsConstructor(onConstructor = @__(@Autowired))
@Slf4j
public class Controller {
  private @NonNull RqueueMessageEnqueuer rqueueMessageEnqueuer;

  @Value("${email.queue.name}")
  private String emailQueueName;

  @Value("${invoice.queue.name}")
  private String invoiceQueueName;

  @Value("${invoice.queue.delay}")
  private Long invoiceDelay;

  @GetMapping("email")
  public String sendEmail(
      @RequestParam String email, @RequestParam String subject, @RequestParam String content) {
    log.info("Sending email");
    rqueueMessageEnqueuer.enqueue(emailQueueName, new Email(email, subject, content));
    return "Please check your inbox!";
  }

  @GetMapping("invoice")
  public String generateInvoice(@RequestParam String id, @RequestParam String type) {
    log.info("Generate invoice");
    rqueueMessageEnqueuer.enqueueIn(invoiceQueueName, new Invoice(id, type), invoiceDelay);
    return "Invoice would be generated in " + invoiceDelay + " milliseconds";
  }
}

Add the following in the application.properties file

email.queue.name=email-queue
invoice.queue.name=invoice-queue

30 seconds delay for invoice

invoice.queue.delay=300000

It’s time to fire up the spring boot application, once the application starts successfully, browse

http://localhost:8080/email?email=xample@exampl.com&subject=%22test%20email%22&content=%22testing%20email%22

In the log, we can see tasks are being executed in the background

Invoice scheduling after 30 seconds
http://localhost:8080/invoice?id=INV-1234&type=PROFORMA

In conclusion, we can schedule tasks using Rqueue without much of the boiler code. We need to consider a few things while configuring the Rqueue library and using them. One of the important is whether a task is a delayed task or not; by default, it’s assumed tasks need to be executed as soon as possible.

Complete code can be found at my Github account https://github.com/sonus21/rqueue-task-exector

Rqueue library code: https://github.com/sonus21/rqueue

If you found this post helpful please share across and give a thumbs up.